Eval-driven development: Build better AI faster
October 18, 2024
AI changes how we build software. In combination with developers, it creates a positive feedback loop where we can achieve better results faster.
However, traditional testing methods don’t work well with AI’s unpredictable nature. As we’ve been building AI products at Vercel, including v0, we’ve needed a new approach: eval-driven development.
This article explores the ins and outs of evals and their positive impact on AI-native development.
Source: Eval-driven development: Build better AI faster – Vercel
Listening to an interview with Ankur Goyal, the founder of Braintrust a company focussed on AI development tooling I was surprised to learn the most common of their SDK’s used by their customers wasn’t Python it was TypeScript.
Now, I would say every customer and probably north of 75% of the users that are running evals in Braintrust are using the TypeScript SDK. It’s an overwhelming majority. And again, at the time, and still, AI is at least nominally dominated by Python, but product building is dominated by TypeScript. And the real opportunity to our discussion earlier is for product builders to use AI. And so, even if it’s not the majority of typists using AI stuff, writing TypeScript, it worked out to be this magical niche for us that’s led to a lot of, I would say, strong product market fit among product builders
In this article, folks at Vercel explore their approach to AI product development.
