Which approach do you think is better for AI and robotics development, and how will this trend evolve?
Please share your data annotation insights and experiences.
Which approach do you think is better for AI and robotics development, and how will this trend evolve?
Please share your data annotation insights and experiences.
PaulShin•8mo ago
For our foundational models (e.g., text summarization), we start with powerful base models like Gemini and fine-tune them. But the real magic happens with our proprietary data, and for that, outsourcing is not an option.
Here's our approach: Our own product, Markhub, is our primary annotation tool.
When our early users give feedback—like circling a button on a screenshot and commenting "This color is wrong"—they are, in effect, creating a perfect piece of labeled data: [Image] + [Area of Interest] + [Instruction].
We call this "Collaborative Annotation" or "In-Workflow Labeling." The data quality is incredibly high because it's generated by domain experts (our users) as a natural byproduct of their daily work, full of real-world context. This is something an external annotation firm can never replicate.
So, to answer your question on how the trend will evolve: I believe the future isn't a binary choice between in-house and outsourced. The next wave will be tools that allow teams to create their own high-context training data simply by doing their work. The annotation process will become invisible, seamlessly integrated into the collaboration flow itself.
yogoism•8mo ago
I completely agree that the first-hand, contextual information you get from actual users is something an external firm can never replicate. It seems like the most effective and efficient way to spin the data flywheel at high velocity.
This leads me to a question I've been struggling to understand: If this approach is so powerful, why do you think even companies with the vast resources of Big Tech still rely on what seems to be a riskier path—using external human evaluators—instead of fully building this feedback loop in-house?
I feel like I'm missing a key piece of the puzzle. I would be very interested to hear if you have any thoughts on this.
PaulShin•8mo ago
1. Foundational Model Training (The Big Tech Approach): To build a base model like GPT-4 or Gemini, you need an unimaginable amount of general, brute-force data. You need millions of images labeled "cat" or "dog," and billions of text examples. For this massive scale of generic data, using large, external teams of human evaluators is often the only feasible way. It's about quantity and breadth.
2. Product-Specific Fine-Tuning (The Markhub Approach): However, once you have that foundational model, the goal changes. To make an AI truly useful for a specific product, you no longer need a million generic data points. You need a thousand high-context, high-quality data points that are specific to your workflow.
For example, an external evaluator can label a button as "a button." But only a real designer using Markhub can provide the critical feedback, "This button's corner radius (8px) is inconsistent with our design system (6px)." This is the kind of nuanced, proprietary data that creates real product value, and it can only be generated "in workflow."
So, I think Big Tech isn't wrong; they're just solving a different problem (building the foundational engine). We, as application-layer startups, have the unique opportunity to build on top of that engine and solve the "last mile" problem by capturing the high-context data that truly makes a product smart.
You're not missing a puzzle piece at all you've just identified the difference between building the engine and building the race car.
yogoism•8mo ago
If you don’t mind me asking, as someone on the front lines of AI product development, what challenges have you found to be even more difficult than data annotation?
I’d really appreciate any insights you can share.