Thoughts On A Month With Devin – Answer.AI
February 24, 2025
As a team at Answer.AI that routinely experiments with AI developer tools, something about Devin felt different. If it could deliver even half of what it promised, it could transform how we work. But while Twitter was full of enthusiasm, we couldn’t find many detailed accounts of people actually using it. So we decided to put it through its paces, testing it against a wide range of real-world tasks. This is our story – a thorough, real-world attempt to work with one of the most hyped AI products of 2024.
One of the areas where LLMs seem to do particularly well is in software engineering. In my experience it can make me far more productive, and as Simon Willison has observed, make developers more adventurous.
I’ve been starting many more projects I know could do, but didn’t know how long they might take, which otherwise I would not even have commenced. I’ve built numerous internal tools for streamlining the production of Conffab content using these technologies and a vector search engine for all of Taylor Swift’s lyrics using Python, a language I had only passing familiarity with. But these are quite constrained, and relatively low stakes problems–the ambition of the likes of of Replit and Devin are far greater-to replace the writing of code almost entirely, not simply to augment existing experienced developers.
The folks at Fast.AI put Devin to the test here–with less than stellar results, at least for now.
