andy here (headlong post author).
terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them.
w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
Don't use any public benchmarks, every single one is worthless for your own use cases essentially.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
For translations, the score is basically 1 or 0. For some tasks, the least amount of LOC gives the highest score, and so on. Basically, you need to figure out how to score it, so you can compare scores across agents/models.
It looks like they're taking applications for training data (due August 14th), so I think it's safe to say this is just an announcement of intent and a call for involvement vs. something that is readily available. Seems almost quaint in comparison to the strategy of sucking up every piece of data you can find anywhere on the Internet and feeding it to your LLM but I suspect their intent is to be more careful in what they train their model on.
I have no doubt companies like Microsoft, Amazon, and Google will rush to give them all the data they want in order to keep those government contracts flowing.
It's the same code. The only thing it doesn't include is the global scheduling and orchestration, but you don't actually want that locally. (And honestly... that stuff is a bit of a beast.)
Durable Objects are fully supported by workerd (and Cloudflare OS uses them extensively).
There is, however, one catch currently: Durable Objects don't scale out well without the global scheduling. For running Cloudflare OS for one user, this is actually no big deal, but a company-wide instance might not work well.
This is exciting! I've loved the idea of Durable Objects but I've disliked that there's no existing way to use them while scaling to multiple machines outside of Cloudflare.
I am somewhat confident that right now we have crossed a threshold of model capability that we will continue to see such breaches and unsanctioned actions by models in the coming months, some of which would be out in the wild, until someone comes up with some really robust control (keeping the AIs on leash) technique that adequately enforces the sanctioned actions.
Even that guarantees almost nothing about real alignment (making the AIs want to predict and behave how we would have wanted them to behave).
Still figuring it out, but it's been really convenient to have an always-on agent that has access to internal systems and can be triggered by webhooks. Some examples of what we use it for:
- automatically fixing simple CI failures
- getting production alerts and automatically creating RCAs and a fix PR
- periodically checking slow DB queries and finding ways to speed them up.
- creating charts to answer one-off questions about our data
I've tried using it as an on-the-go coding agent as well, but found I prefer more interactive agents, so I can see what the code looks like.
I think the interesting challenge isn’t running agents, it’s reviewing their work. The more code agents produce, the more important provenance, review ergonomics, and trust become. I also suspect repository platforms will need to evolve there over the next few years.
At least for me, I have a couple dozen years of DB experience but robot, given performance metrics, can get really close to optimal on a tactical level (single query or pattern of queries) but can't yet do the full normalize/denormalize level of improvements without supervision. But really solid if you have one misbehaving query and give it explain analyze access on a read only account
It has read only access, so it’s able to check query statistics, find slow queries, then run EXPLAIN ANALYZE to find the root cause and either tweak the query or suggest indexes. A lot of it is low hanging fruit, I just haven’t put in the time to fix it (startup).
Hermes is huge and packed with features you probably don't need. I prefer smaller one I can extend as necessary, there are so many on github now and it is fun to test them but have been impressed with dirge (https://github.com/dirge-code/dirge) not affiliated.
I have one reading my second tier RSS feeds and newsletters and giving me news/market updates filtered for things important to me
I'm not contradicting this but offering a contrast, I like Hermes because it simultaneously lowers barrier of entry and shows you what possibilities are unlocked by agents. I don't think I would have the time, interest or creativity to jump into the deep end by either extending an existing harness or rolling my own from the start. This also isn't an argument for doing just that, I might do so in the future, but critically only after Hermes has shown me what's possible and my preferences are developed.
I completely agree. I started with aichat[0] before the current agent trend and hit all kinds of bumps implementing agentic loops on my own. Then goose[1] showed me what a whole team working toward the same idea could do right before the official Claude Code harness which had all the bells and whistles. Now I know better what I want and its mostly less ram usage and a small set of primitives.
I still use Claude code (and codex and other big contenders) because they know what they are doing and innovate in ways I don't want to miss. And sometimes they are better at tasks.
I had the same line of thought and spent a while with nanoclaw before realizing that adding the features I want back in would have made future updates too painful. I ended up switching to Hermes and aggressively disabling tools/skills and it's been pretty fine so far. I got more use cases set up than I did in my time with nanoclaw.
Totally fair, I still use Claude code for tasks since it is so polished. I think I just dont have that complicated of use cases so in the end performance should reflect my lightweight needs as the priority.
I feel the same but my preference is for Cecli (cecli.dev)
It does what I need it to do and since I invested so much time in setting it up and even contributing to development it is my go to for coding and even managing my VPS as well as business tasks
Hermes is what I was using but I still found it annoying I often wanted to operate 1-2 levels deeper.
Yesterday I just decided to try writing my own version (100% just for me, not open source, no monetization plan, incredibly custom) and I've been enjoying working on it so far (I know, I know, it's been a day, honeymoon period and all that).
Part of it is I like building software (even if I'm not writing every line) and part is I like having full control. Turns out (and many people have said this) the basic agent loop really isn't all that special. There are a million levers to pull and what not outside the base loop but that the fun part for me. Trying out different ways to add on to the core concept.
I'm really enjoying being untethered for things like "how will I monetize?" or "how do I make this generic so others can use it?". If I need functionality I just add it in, I don't need to make it infinitely pluggable, etc.
All that said, I'm thankful to things like nanoclaw and then Hermes for exposing me to the core ideas. I just want to put my own spin on it.
I'm currently using it to help me with my oncall, first responder to our any production alerts. It's not as efficient as coding agent by default, but it's been tremendously helpful to me.
I didn't have good experience with Hermes. Vellum.ai was better, but unfortunately it had a bug where opencode go providers failed to work for a time, so I instead starting writing my own.
hermes is great to get started with, but it's packed to the gills with stuff you'll probably use one time just to test it. and this eats in to your context so if you're hoping to run it on a lighter-weight local model you'll run in to some trouble. if you go into it planning to customize/thin it out it's solid
Sounds like it'll replace v4-flash, v4.1 would be nice to keep both available. On the other hand, it's nice to just get an improvement on anything that asks for "deepseek-v4-flash" without having to change the model string.
I think that's backwards. Anything that changes the performance of a model deserves a minor version bump. A new model has to be qualified before being pushed to production; but we don't get the choice here, just cross your fingers there are no regressions at all on all possible tasks the model might be asked to do.
I should have worded it differently. Having a model selector as `string` seems like the wrong choice. At the very least, I should be able to select 4.X, like how we select dependencies, so people who rely on specific behavior can pin to 4.0 and I can say version >= 4 if I don't care.
That's on you for using a preview/beta. It was properly qualified when it was released. No one complains when ios goes from beta to release; even though it will be ios 27 beta -> ios 27 next month
its an open source model if it is important enough that you have to worry about a new version breaking something then why the fuck was that not running on your own servers this is not an Anthropic or OpenAI closed model that you only have access through an api
