This doesn't take anything away from the technical accomplishment. It doesn't take anything away from the fact that small models are improving, and that maybe one day they'll indeed be good enough for developers to rely on them in their daily work.
But that day is not today.
Thus, I find it spurious to hear developers evaluate their next computer on the prospect of how well it's capable of running local models. Because they all suck! Whether one sucks a little less than the other doesn't really matter. And as soon as you discover this, you'll be back to using the rented models for the vast majority of the work you're doing.
This is actually great news! It means you really don't need a 128GB VRAM computer on your desk. Which should come as a relief now that RAM prices are skyrocketing, exactly because of AI's insatiable demand for more resources. Most developers these days can get by with very little, especially if they're running Linux.
So as an experiment, I've parked my lovely $2,000 Framework Desktop for a while. It's an incredible machine, but in the day-to-day, I've actually found I barely notice the difference compared to a $500 mini PC from Beelink (or Minisforum).
I bet you likely need way less than you think too.
https://world.hey.com/dhh/local-llms-are-how-nerds-now-justify-a-big-computer-they-don-t-need-af2fcb7b
