Rendered at 22:43:58 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
seizethecheese 5 hours ago [-]
This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
adi1 5 hours ago [-]
The breakdown with which model, per-task cost, and methodology is in the "Full results and methodology" link in the post, not omitted. Definitely check it out if you haven't.
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
seizethecheese 5 hours ago [-]
I did read "Full results and methodology". It doesn't seem to show which agents you are routing to. Am I missing something? And how do those agents score on SWE-Bench Verified?
alsima 4 hours ago [-]
The benchmark wasn't evaluated with the router, we wanted an apples-to-apples comparison of our harness against other harnesses like mini-swe-agent on the same models to see if we added speed + cost value beyond routing.
seizethecheese 3 hours ago [-]
Okay, that makes more sense, but then this post is very confusing. Why mention the router at all? Or does your live system have a router and the eval doesn't? In which case why not publish an eval with the router?
alsima 3 hours ago [-]
Fair point, yes, the router is only in the live system (we'll try to make that more clear). We thought that comparing the router to other harnesses on the benchmark wouldn't be a fair evaluation because the speed/cost gains from our harness would mostly be from using cheaper and simpler models rather than actually having a better harness.
seizethecheese 5 hours ago [-]
The same thing holds for speed. I could build a system that speeds up Fable on GPQA Diamond ~50%, while improving score, by literally randomly selecting between Fable and Gemini 3.7 Flash. (Solve time for Flash is 0.1min and 0.8min for Fable, with Flash having a better score.)
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
alsima 4 hours ago [-]
Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 Flash wouldn't preserve quality.
seizethecheese 3 hours ago [-]
Mentioned where?
What do you mean it wouldn't preserve quality? I demonstrate in the comments here how it would on a saturated benchmark.
alsima 3 hours ago [-]
Mentioned in my comment above. Sorry, I thought you were talking about Gemini 3.6 Flash, not their new one. Yes, that's the reason why we didn't include the router in the benchmark to have a fair comparison of the harness itself.
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
alsima 12 hours ago [-]
Thanks for bringing this up and trying it out! We've disabled Cmd+Option+I in the new version and also have a message during signup about sharing chats with Bullet, let us know if you run into any other issues!
poly2it 6 hours ago [-]
> We've disabled Cmd+Option+I in the new version
Bulletproof.
xrisk 6 hours ago [-]
Truly built on vibes.
alsima 4 hours ago [-]
We're optimizing every day :)
davidfiala 10 minutes ago [-]
> We've disabled Cmd+Option+I
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
0kk33 5 hours ago [-]
Congrats on the launch.
I can't find which model providers are supported?
Are you calling the claude-code CLI directly and make bullet usable with anthropic subscriptions like orca or herdr?
I guess however this is a harness and it needs to connect to the API?
adi1 5 hours ago [-]
Thanks! We support a variety of model providers (OpenCode/Codex/Grok/Claude) via their subscriptions and API keys. This can be seen in the app.
etchalon 1 hours ago [-]
Since I can't get this thing not to annotate its commits with itself as an author, I can't use it.
That should be an option, not forced.
alsima 54 minutes ago [-]
This is a great point, we're actively working on a fix.
If you’re up for giving it another shot, would love to hear other feedback on the app you have.
myshapeprotocol 7 hours ago [-]
Focusing on execution speed for coding agents is the right bottleneck to tackle. Exciting launch.
adi1 7 hours ago [-]
Thank you!
hmokiguess 6 hours ago [-]
> P.S: we hid a code on the website, see if you can unlock the secret page at the footer, all built with Bullet
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
adi1 6 hours ago [-]
haha fair enough, surprisingly people still struggle to find it, glad you found it and liked the game!
esafak 4 hours ago [-]
I think this adds no value. I would stick to OpenCode.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models.
* Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
alsima 4 hours ago [-]
Thanks for the feedback, we do focus on good tool management, context efficiency, etc. As for routing, ideally it isn't the user who has to route themselves, but a layer that routes for them.
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
retropragma 2 hours ago [-]
does your harness support ACP?
alsima 47 minutes ago [-]
No, we don’t support ACP today. Bullet uses its own internal harness and CLI integration. That said, we can definitely see it being something we work on in the future!
drxcliu 2 hours ago [-]
It just works like magic!
alsima 2 hours ago [-]
Thank you!
7 hours ago [-]
tontinton 6 hours ago [-]
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh, and months ahead of other founders too...
Supermancho 6 hours ago [-]
I'm not sure why anyone would fund "creating an agent".
OpenCode is competing with Codex and Claude already.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
alsima 4 hours ago [-]
I think that if there are many players in the space, it's a problem worth working on! Like I said in the post, our main problem is speed and that's what we're trying to fix. It's been a problem, and it still is a problem. We have tried tools like OpenCode, Pi, etc. but for our speed issue, it did not fix it. If we go down dying on this boat, so be it, at least we went down dying on a problem we care about!
Terretta 17 minutes ago [-]
Meanwhile, Cerebras running Sol at 750 tokens per second, the harness "speed" becomes irrelevant compared to making better use of results from insta-Sol.
alsima 10 minutes ago [-]
Yes, we saw this and it's really exciting, can’t wait to try it in Bullet. But faster inference only speeds up generation. In our own experience, testing, building, searching, and other tool work still dominate plenty of real tasks. We’ll have to test it in practice, but it’s a great development for everyone building agents.
docheinestages 6 hours ago [-]
YCombinator's investments are unfortunately very questionable nowadays.
tomhow 2 hours ago [-]
People were saying that about YC in 2009, when Airbnb and Stripe were funded. YC’s philosophy has always been to invest in a great many companies, accepting that you can’t know in advance which will succeed, most will fail, but the ones that succeed will be successful enough to pay for all the losses.
docheinestages 6 hours ago [-]
Your landing page is very hard to read. The font size is literally 10px for some content, while animations distract the reader.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
KellyCriterion 5 hours ago [-]
++1
Your comment was the solely reason why I clicked on their page! :-D
Looks like "Techno music design of the 90s" or similar :-D
alsima 4 hours ago [-]
We'll get on that :)
waingake 4 hours ago [-]
Website looks great FWIW.
lucasdimarco 5 hours ago [-]
I recently switched over to using primarily Bullet for my projects and the speed of it makes it very nice to get projects off the ground quickly and work properly. I also found that Adi and Alex make updates very regularly based on some of the feedback i've submitted to their feedback tab. Good work on this product guys! I'm excited to see how it develops in the future.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
What do you mean it wouldn't preserve quality? I demonstrate in the comments here how it would on a saturated benchmark.
Cmd+Option+I > Console > 'allow pasting'
const onboarding = document.querySelector('#onboarding'); const app = document.querySelector('#app'); onboarding.style.setProperty('display', 'none', 'important'); app.inert = false; app.removeAttribute('aria-hidden'); document.querySelector('#prompt')?.focus();
Also, warning:
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
Bulletproof.
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
I guess however this is a harness and it needs to connect to the API?
That should be an option, not forced.
If you’re up for giving it another shot, would love to hear other feedback on the app you have.
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models. * Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Looks like "Techno music design of the 90s" or similar :-D