#llm-benchmarks
19 posts
2026 - July
4 posts2026 - June
3 posts2026 - May
1 post2026 - April
4 posts- LLM Benchmarks: Is It Worth ($$) Mixing 2 Models? (Planner + Executor)
- LLM Coding Benchmark (May 2026): DeepSeek v4, Kimi v2.6, Grok 4.3, GPT 5.5
- LLM Benchmarks Part 2: Is It Worth Combining Multiple Models in the Same Project? Claude + GLM??
- Testing Open Source and Commercial LLMs - Can Anyone Beat Claude Opus?
2026 - January
4 posts2025 - May
1 post2025 - April
2 posts2026 - July
4 postsLLM Benchmark: Should I Use the Highest-Scoring Model?
Why the top entry in an LLM ranking is not necessarily the best model, why 90+ is a cluster, and how to use benchmarks without outsourcing your technical judgment.
LLM Benchmark: Has Kimi K3 Reached Claude Opus Level?
I had Kimi K3 build a chat in Rails 8 on its own: it scored 89/A, beat Opus 4.6, and was a cheaper alternative to Opus 4.8. It still lagged behind in architecture, hardening, and tests.
LLM Benchmarks: Grok 4.5 and GPT 5.6 Sol
Grok 4.5 entered Tier A with 87, and GPT 5.6 Sol scored 92. Sol beat GPT 5.5 by 92 to 81 in the blind A/B test, but re-auditing brought it down to 85. With cache-read fixed, Opus 4.8 remains the best API balance.
LLM Benchmark: Sonnet 5 Fails, Gemini Flash Surprises, Sakana Fugu Almost Reaches Tier A
In the coding benchmark, Sonnet 5 scored 58/100 after calling a nonexistent API, while Gemini 3.5 Flash scored 93/100. Sakana Fugu Ultra landed at 79/100, close to Tier A.
2026 - June
3 postsLLM Benchmark: Kimi v2.7 Code, GLM 5.2, MiniMax M3 Local
In the benchmark, GLM 5.2 scored 87 and Kimi K2.7 Code scored 86. The open MiniMax M3 doesn’t fit in 128 GB, while serious programming still calls for Opus 4.8 or GPT 5.5.
LLM Benchmark: Fable 5 and the Anthropic Soap Opera
Fable 5 scored 94/100, practically tied with Opus 4.8, but costs ten times more and redirects sensitive tasks. Since Mythos isn’t auditable, I’m sticking with 4.8.
LLM Benchmarks - An Update on Grok 4.3, MiniMax v3, and Opus 4.8
In the Rails 8 benchmark, Opus 4.8 keeps the lead at 95/100, while Grok 4.3 and MiniMax M3 finally become usable but remain in Tier B, behind GPT and Opus.
2026 - May
1 postLLM Benchmarks: DeepSeek Unlocked! Use DeepClaude
I swapped opencode’s harness for Claude Code’s loop through DeepClaude, and DeepSeek V4 Pro rose from 69/B to 89/A in 18 minutes and $3.14. The controller’s multi-turn bug remains.
2026 - April
4 postsLLM Benchmarks: Is It Worth ($$) Mixing 2 Models? (Planner + Executor)
Three rounds show multi-agent doesn’t beat solo Opus 4.7 in opencode, which scored 97/100 in 18 minutes for about $4. GPT 5.4 xHigh with a medium executor saves money but loses 3 points.
LLM Coding Benchmark (May 2026): DeepSeek v4, Kimi v2.6, Grok 4.3, GPT 5.5
I reaudited 24 LLMs on the same Rails app with RubyLLM: Opus 4.7 and GPT 5.4 tie at 97, GPT 5.5 reaches 96, Kimi and Gemini become Tier A, and DeepSeek V4 Pro reaches Tier A only through DeepClaude.
LLM Benchmarks Part 2: Is It Worth Combining Multiple Models in the Same Project? Claude + GLM??
I tested seven combinations in Claude Code, opencode, and Codex, but no sub-agent was called. Variance across runs and harnesses keeps solo Opus as the default for greenfield projects.
Testing Open Source and Commercial LLMs - Can Anyone Beat Claude Opus?
This historical benchmark compared 33 LLMs on a Rails app: Opus, Sonnet, and GLM 5 worked, while Qwen 3.6 35B came close on an RTX 5090 after a fix. The rankings were later revised.
2026 - January
4 postsVibe Code: Which LLM Is the BEST?? Let's Talk for REAL
I ran the same extensive analysis pass through Opus 4.5, GPT 5.2 Codex, Kimi 2.5, Gemini 3 Pro and MiniMax v2.1 on a tiny app, and every single one still left things on the table.
AI Agents: Which One Is Best? OpenCode, Crush, Claude Code, GPT Codex, Copilot, Cursor, Windsurf, Antigravity?
A practical take on the current state of AI coding agents, why proprietary harnesses matter, and why I am sticking with Crush.
AI Agents: Is GLM 4.7 Flash really that good?
Running GLM 4.7 Flash locally on an RTX 5090 via Ollama to see if it can finally match commercial LLMs on a hard coding challenge.
AI Agents: Comparing the Top LLMs of 2026 on the Zig Challenge
I put the top commercial and open source LLMs of early 2026 through a nasty Zig + llama.cpp migration challenge to see which ones actually deliver.
2025 - May
1 postRant - Will LLMs Evolve Forever? Demystifying LLMs in Programming
A hype-busting rant on LLM benchmarks, the myth of exponential AI progress, and my hands-on experience showing why Vibe Coding does not work.
2025 - April
2 postsTesting the Newly Released Open Source LLM - Qwen3 (with Aider and Ollama)
First hands-on impressions of Qwen3 running locally via Ollama and remotely via OpenRouter, tested with Aider on real refactoring tasks.
Testing LLMs with Aider on RunPod - which one to use for code?
Hands-on testing of Qwen2.5 Coder, DeepSeek, Codestral and others with Aider, locally on an RTX 4090 and on RunPod, comparing them against Claude and Gemini.