#llms
79 posts
2026 - October
14 postsLLM Benchmark v4: Gemini 4 Argon, GPT-6.1 Sol and Terra, Kimi K3.1 Pro, Claude Sonnet 5.5
Muse Wonder closes out v4 at 97.0 with excellent vigilance, but quotas and hangs turned 126 productive minutes into nearly 15 days. The leaders remain very close.
LLM Benchmark: What Are the Best Open Source LLMs for Home Hardware?
I compared Basalt and Strata serving Qwen3.8 Flash-Next on an RTX 5090. Q2_0 scored 100 in all six hard runs and reached 400 tok/s on Basalt, versus 259 on Strata.
LLM Benchmark for Accountants and Lawyers (Experimental - Open Source LLMs)
I tested Sol 6 and four open-model configurations across 38 synthetic accounting and legal cases. The RTX 5090 was competitive, but every result still requires professional review.
LLM Benchmark v4: the open source verdict (as of Oct 2026)
Almost 70 runs put the best open models close to Sonnet 5. Even so, 159 tok/s on an RTX 5090 is still slow for my work, while faster options already cost as much as a server.
Best Open Source LLM: Strata with Qwen 3.8 Flash Next
Qwen3.8 Flash-Next served by the Strata engine scored 91, Tier A, on an RTX 5090, 1.5 points behind the B300 and at about 159 tokens per second. I explain how Strata fits a 125B MoE on a PC by using VRAM as an expert cache and RAM as the warehouse, why that matters with memory costing twice as much, and why it isn't magic: the 5090's memory bandwidth is seven times the Strix Halo's and almost 30 times PCIe's, and even on my machine it takes up almost all the RAM and VRAM. I show the install, a service that loads and unloads the model on its own, the OpenCode configuration, why I consider my machine the minimum for real programming work, and why, even a bit less responsive than a commercial model, it's already almost good enough to replace the cloud day to day.
LLM Benchmark v4: consumer-grade open source on RTX 5090 or Strix Halo. And Strata?
I went back to consumer hardware after the datacenter tests. Qwen3.8 Flash-Next scored 91.0, Tier A, on an RTX 5090 with the Strata engine, 1.5 points behind the B300 at 159 tokens per second. It's a different model from the Qwen 3.8 27B I had called slow: a MoE with 6B active per token, served with the hottest experts on the card. Strix Halo scored 80.0 with the 27B, but at 7.7 tokens per second. It's the first local model to make Tier A on the v4, and it's worth testing in day-to-day programming, if you can find a 5090: NVIDIA denies the end of production but admits tight supply.
LLM Benchmark v4: can my team fit in a 4x b300/h200?
After ranking the big models, the leftover question: how many developers fit on a 4x B300 or 4x H200 pod? We measured real concurrency, with a latency SLO, and the answer changes everything: DeepSeek V4.1 Flash holds 32 comfortable users on B300 and only 8 on H200. Cost per comfortable user: US$ 1.13/hour on B300 versus US$ 1.80 on H200. When your own infra closes the math against a frontier subscription, and when it's just expensive vanity.
LLM Benchmark v4: open source that only fits in 4x B300!!
I rented 4x NVIDIA B300 on Runpod and ran my v4 sabotage benchmark on the open models that don't fit on a single card. DeepSeek V4.1 Flash, at 748B, tied the leaders at 92.5, on the level of hosted GLM 5.3 and GPT-6 Sol. GLM 5.3 full fell behind, the 1T Kimi K2 Thinking disappointed, and a 397B model plummeted from 94 in the cloud to 27 at home. The math: US$ 36 per pod hour, US$ 864 per day. When owning your own infra makes sense, and whether a big open model is already enough for your day-to-day programming.
LLM Benchmark v4: the top open source LLMs on a B300
I rented a 288 GB NVIDIA B300 on Runpod and ran my v4 sabotage benchmark on 14 open-weight models that don't fit at home. None of them serves fast the way it ships: GLM 5.3 Flash went from 10 to 141 tokens per second on configuration alone. Two open models reached 92.5, Sonnet 5 level, and a 27B fine-tune scored 88. A 397B flagship let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 at home, and Nemotron cheated on a test. The math: the cloud API of the same model costs a dollar per run and the pod costs 218 a day. When owning your own infra is worth it, and whether a team fits on one card.
I Trained My Own AI Model (Spoiler: It's Just for Study)
One exercise was missing: training an LLM from scratch. I started from Karpathy's microgpt, 200 lines of pure Python, and expanded it into a lab with a tokenizer, pretraining, SFT, an OpenAI-compatible server and OpenCode integration. Under two weeks, one RTX 5090, 10 billion tokens, 137 hours of training, at most around R$ 80 of electricity. The final model has 337 million parameters, gets 58 out of 100 simple Python functions right and scores zero on anything that requires acting in a repository. I explain each step, the stumbles, the dimensions against GPT-2 and frontier models, and why I think the future is smaller, specialized models.
LLMs + Agents = competent enough
A synthesis of what I've been measuring and writing: LLMs converged. The transformer architecture got near the ceiling, the parameter war is over, and the leap everyone felt in 2025 and 2026 came from the agentic harness (tool calling, prompt caching, long sessions), not from a base intelligence orders of magnitude larger. I retell the timeline from 2019 onward with sources, explain why the agent allows a less dense model, and why that's already competent enough to automate real software engineering.
Are There Better Languages for Programming with LLMs?
DHH converted his Campfire to several languages with AI and started posting that Rust won. Instead of fighting on X, I built a reproducible benchmark: 7 languages, 4 LLM configurations, 112 implementation attempts, and a queue-under-pressure test with Elixir, Rust, Rails and plain Ruby. Result: no language ranking holds up, concise Ruby cost more tokens than Rust, strong typing spared no repair at all, and under load the four stacks tied on throughput. What matters is testing and discipline, not language.
Which Jev-like AI Decision/Classifier Model Is Best?
I ran my own decision-model benchmark: Jev, Clef, OpenAI Decisions and Microsoft Decision-1 in the cloud, plus eighteen open configurations (Perplexity Decider 27B, Decision 3.0 and Edge, Clef-flash, Kev, Jebadiah, Mapika, Eikos, Laya, Julia, GLiNER, Simple-Jev) on my Strix Halo, across more than 22 thousand classifications with cases in English and Portuguese. The top now has two open-weights models: Perplexity and Decision 3.0 tied at 99.4%, with identical labels on the 1,024 classifications and 100% in Portuguese. Microsoft Decision-1 tied Jev at 97.6% while being the cheapest hosted one. Full table, cost in support emails, gotchas, and where the data converges.
Understanding the Jev Hype: What Is It For?
Jev is a hosted classifier, and the shock of '193x faster' only exists because so many people use a chat LLM as an expensive classifier. I publish Paulo Câmara's essay (LUA Vision) with his charts, explain what a classifier is and why it's the first thing a computer scientist reaches for, compare the open alternatives Clef and Laya, show the public Decision Index ranking, and close with how to think about this kind of tool in your own project.
2026 - September
9 postsThe Mystery of LUA Vision's Genesys PI - A Brazilian Frontier Model??
After the first Genesys PI test, LUA Vision fixed bugs in the API and I ran everything again: cost dropped 85%, House climbed to 95.5 and Enterprise stayed flat. I also ran a black-box forensic analysis to answer whether the model is a rebadged Qwen, a disguised gpt-oss or an API proxy. I separate what is checked from what is still only their word.
The Limits of LLMs: Good at Reproducing What Exists. Expensive for What Doesn't
Every LLM was trained only on what's public, but 81.5% of GitHub activity happens in private repositories, and the source code of every proprietary software never made it into training. In my retrocomputing projects I felt firsthand where the machine stalls: with no reference and no oracle, it spins in directed brute force.
LLM Benchmark v4: Genesys PI, a New Brazilian Contender!
LUA Vision reached out on LinkedIn offering test access to Genesys PI, a Brazilian model trained from scratch. I ran both tiers, House and Enterprise, through my v4 sabotage benchmark, compared them against Opus, GPT-6, Grok and the Chinese competition, and looked at price, advantages and where it makes sense in the Brazilian market.
LLM Benchmark v4: Opus 5.5, GPT Sol/Luna 6, Mimo 2.6, Grok 4.7
Five new models entered my v4 sabotage benchmark: Opus 5.5, GPT 6 sol and luna, MiMo V2.6 Pro, and Grok 4.7. I compare against external benchmarks, answer whether upgrading is worth it, and Grok 4.7 regresses badly against its own predecessor.
Talking a Bit About My AI Skills
Agent skills are just prompts in text files. I keep 25 in my my-skills repo, from pr-audit to fact-check, and they have already closed over a thousand PRs and issues across my projects. The recommendation: don't copy somebody else's skill, build your own.
Why Things Like TypeSafe AI Don't Interest Me
A handful of people showed up in the comments asking 'have you tried TypeSafe?'. I use that as a hook to lay out my rule about new AI tools, actually investigate the case, and explain why I don't waste time optimizing token spend.
New LLM Benchmark v4: Retesting 39 LLMs (Part 2)
The full v4 table, nearly forty models ranked by vigilance, not completeness: six ties at the top, a free model beating half of Claude, GLM winning on a flat-rate plan, and the test I ran to find out whether Kimi or DeepSeek are secretly hiding a connection to Claude under the hood.
New LLM Benchmark v4: Retesting ALL the Top LLMs (Part 1)
I rebuilt my benchmark from scratch for the third time. I threw out an entire version, built a Rails app sabotaged with real CVEs, and spent over $4,000 in 9 days. In this Part 1: the process, the cost, and the why.
You're an Idiot if You Believe the Misleading Ads from OpenAI, Anthropic, NVIDIA. Here's Why
Nvidia and OpenAI declared that AGI has arrived, OpenAI brute-forced a Millennium Problem and threatened a mathematician's career, and an Anthropic employee quit crying Skynet. I take the propaganda apart.
2026 - August
4 postsLLM Benchmarks: The latest Deepseek v4, stop asking
I ran the new Deepseek v4 snapshots through the benchmark: Flash jumped from 80 to 90 and Pro from 82 to 91, both Tier A, the cheapest in the pack. I still prefer Kimi K3 and GLM 5.3, and I explain why a close score does not mean an equivalent model.
Hot Take: Harness, Loop Engineering, Graph Engineering Are Bullshit
I posted that Harness Engineering, Loop Engineering, and Graph Engineering are bullshit to sell consulting hours and courses. Here's my receipt: 30+ public repos, an AI marathon, and a whole benchmark — none of it using any of those 'disciplines'.
Understanding Anthropic's AI Watermark: How to Beat It
Claude now stamps everything it writes with an invisible statistical watermark, a requirement of European AI law. How it works, what the AI Act demands, and the rewrite in another LLM that wipes the signal.
LLM Benchmarks: Qwen 3.8, GLM 5.3, Gemini 3.7, Grok 4.6
GLM 5.3 scored 94 and closed in on the leading trio. Qwen 3.8 Max jumped from 51 to 92 after it stopped hallucinating the RubyLLM API. Gemini 3.7 Flash scored 93 — after its first run was voided for cheating. And the local 27B Qwen showed that context is now the bottleneck.
2026 - July
7 postsNew LLM Benchmark: I Reran Every Test!
I reran the LLM Coding Benchmark with a harder test, three phases, native harnesses, and new tiers. Fable led, Terra tied Sol as the best GPT, and 24 models proved suitable for programming.
LLM Benchmark: Is Opus 5 Any Good?
Opus 5 scored 95/100 on the Rails benchmark, tying Opus 4.8 and edging Fable 5 by one point. Its API rate is half Fable’s, but this narrow test does not define the best LLM.
LLM Benchmark: Should I Use the Highest-Scoring Model?
Why the top entry in an LLM ranking is not necessarily the best model, why 90+ is a cluster, and how to use benchmarks without outsourcing your technical judgment.
LLM Benchmark: Has Kimi K3 Reached Claude Opus Level?
I had Kimi K3 build a chat in Rails 8 on its own: it scored 89/A, beat Opus 4.6, and was a cheaper alternative to Opus 4.8. It still lagged behind in architecture, hardening, and tests.
LLM Benchmarks: Grok 4.5 and GPT 5.6 Sol
Grok 4.5 entered Tier A with 87, and GPT 5.6 Sol scored 92. Sol beat GPT 5.5 by 92 to 81 in the blind A/B test, but re-auditing brought it down to 85. With cache-read fixed, Opus 4.8 remains the best API balance.
Frank GO: Playing Go with AI
I built Frank GO as a Sabaki fork with KataGo, bringing together more than 4,700 tsumego, historical games, and 15 of Hikaru’s Go games. Overlays make territory and scoring visible to beginners.
LLM Benchmark: Sonnet 5 Fails, Gemini Flash Surprises, Sakana Fugu Almost Reaches Tier A
In the coding benchmark, Sonnet 5 scored 58/100 after calling a nonexistent API, while Gemini 3.5 Flash scored 93/100. Sakana Fugu Ultra landed at 79/100, close to Tier A.
2026 - June
6 postsWhy LLMs Will Fail at Your Company
The deliberately speculative thesis is that LLMs amplify processes where people outsource decisions. The proposed alternative is daily cycles of decision, implementation, testing, and review close to the outcome.
ai-memory: long-term memory (Karpathy Wiki) and self-improvement (Hermes) for your projects
ai-memory turns agent sessions into a searchable Markdown wiki with hooks, MCP, and handoffs between tools. A Hermes-inspired loop promotes memories with validation, evidence, and optional review.
The Rio 3.5 LLM Controversy. Plagiarism?
Public evidence indicates that Rio 3.5’s initial checkpoint was a blend of Nex-N2-Pro and Qwen, without credit to Nex. The case points to an attribution failure, not legal plagiarism.
LLM Benchmark: Kimi v2.7 Code, GLM 5.2, MiniMax M3 Local
In the benchmark, GLM 5.2 scored 87 and Kimi K2.7 Code scored 86. The open MiniMax M3 doesn’t fit in 128 GB, while serious programming still calls for Opus 4.8 or GPT 5.5.
LLM Benchmark: Fable 5 and the Anthropic Soap Opera
Fable 5 scored 94/100, practically tied with Opus 4.8, but costs ten times more and redirects sensitive tasks. Since Mythos isn’t auditable, I’m sticking with 4.8.
LLM Benchmarks - An Update on Grok 4.3, MiniMax v3, and Opus 4.8
In the Rails 8 benchmark, Opus 4.8 keeps the lead at 95/100, while Grok 4.3 and MiniMax M3 finally become usable but remain in Tier B, behind GPT and Opus.
2026 - May
4 postsComplete Manga Solution: Frank Manga+, Frank Yomik, and the Prettify-Manga Extension
Frank Manga+ brings the MANGA Plus catalog to the desktop, while Prettify improves reading on websites and Kindle, and Frank Yomik adds furigana or translation to Japanese and Korean pages.
I Built a Waybar Widget for Omarchy to Monitor LLM Plan Usage: ai-usagebar
ai-usagebar, a Rust port of claudebar, brings Claude, Codex, Z.AI, and OpenRouter into a Waybar widget or standalone TUI that works in any terminal, including over SSH.
AI Agent Memory: Karpathy LLM Wiki and agentmemory in Practice
Akita compares compaction in Claude Code, Codex, and opencode, explains Karpathy’s LLM Wiki, then tests agentmemory before dropping it over structural bugs and pointing to ai-memory.
LLM Benchmarks: DeepSeek Unlocked! Use DeepClaude
I swapped opencode’s harness for Claude Code’s loop through DeepClaude, and DeepSeek V4 Pro rose from 69/B to 89/A in 18 minutes and $3.14. The controller’s multi-turn bug remains.
2026 - April
7 postsLLM Benchmarks: Is It Worth ($$) Mixing 2 Models? (Planner + Executor)
Three rounds show multi-agent doesn’t beat solo Opus 4.7 in opencode, which scored 97/100 in 18 minutes for about $4. GPT 5.4 xHigh with a medium executor saves money but loses 3 points.
LLM Coding Benchmark (May 2026): DeepSeek v4, Kimi v2.6, Grok 4.3, GPT 5.5
I reaudited 24 LLMs on the same Rails app with RubyLLM: Opus 4.7 and GPT 5.4 tie at 97, GPT 5.5 reaches 96, Kimi and Gemini become Tier A, and DeepSeek V4 Pro reaches Tier A only through DeepClaude.
LLM Benchmarks Part 2: Is It Worth Combining Multiple Models in the Same Project? Claude + GLM??
I tested seven combinations in Claude Code, opencode, and Codex, but no sub-agent was called. Variance across runs and harnesses keeps solo Opus as the default for greenfield projects.
Why LLMs Aren't Giving You the Result You Expect | Why I Prefer Claude Code Today
Every week somebody tells me 'I canceled my Claude plan, it just doesn't perform as well as GPT for me.' I've got 500+ hours in Claude Code and Codex, 400k lines generated, and both deliver. The difference isn't the model. It's how people are talking to it.
Is RAG Dead? Long Context, Grep, and the End of the Mandatory Vector DB
Frontier models went from 200k to 1M tokens of context. Does it still make sense to wire up a whole vector DB stack to do RAG, when you can just grep and dump the entire document into the window?
Testing Open Source and Commercial LLMs - Can Anyone Beat Claude Opus?
This historical benchmark compared 33 LLMs on a Rails app: Opus, Sonnet, and GLM 5 worked, while Qwen 3.6 35B came close on an RTX 5090 after a fix. The rankings were later revised.
Turning YouTube into a Karaoke App | Frank Karaoke
I built a Flutter Android app that overlays live scores on YouTube karaoke videos, using audio filtering and YIN in four modes. It’s fun, but scoring is still experimental.
2026 - March
6 postsTeaching People to Question the News | Frank Investigator
I built Frank Investigator to compare coverage, track omitted sources, and analyze news in 15 stages with three LLMs. The report flags omissions and reframing, but doesn’t issue final verdicts.
I Rewrote OpenClaw in Rust. Did It Work? | FrankClaw
I rewrote OpenClaw’s core in Rust as FrankClaw, with 56,586 lines, 7 channels, and fixes for the 7 audited critical vulnerabilities. It handles simple conversations, but complex workflows remain untested.
Going After Email Fraud | Frank FBI
I built Frank FBI to analyze forwarded emails in six layers, combining authentication, reputation, OSINT, and three LLMs. The result is a self-hosted risk report, with the data under your control.
My First Vibe Code Failure and How I Fixed It | Frank Yomik
How I spent days building a manga speech bubble detection system with OpenCV, threw it all out and rebuilt it in hours with a pretrained model. Lessons on vibe coding, productive failure, and knowing when to change course.
I Built a Data Mining System for My Influencer Girlfriend — Tips and Tricks
How I built a Rails 8 + SQLite + Discord bot data mining system for my influencer girlfriend in 3 days, 58 commits, with 40+ tools accessible via LLM tool calling.
Software Is Never 'Done' — 4 Projects, Life After Deploy, and Why One-Shot Prompting Is a Myth
125 post-production commits across 4 projects in 10 days. Real bugs, real users, real iteration. Why one-shot prompting is a myth and software is never 'done'.
2026 - February
7 postsRANT: Did Akita Bend Over for AI??
I have been 'bending over' for AI since 2023. If you only watch out-of-context podcast clips, let me spell it out for you.
Vibe Code: I Built a Smart Image Indexer with AI in 2 Days | Frank Sherlock
How I built Frank Sherlock, a local AI-powered image catalog desktop app, in a weekend using Agile Vibe Coding with Claude Code.
From Zero to Post-Production in 1 Week - How to Use AI on Real Projects | Behind The M.Akita Chronicles
How 274 commits in 8 days shipped a full newsletter, podcast, blog, and Discord bot to production using Extreme Programming with Claude Code as the pair.
AI Agents: What Would Be the Best Programming Language for LLMs?
A thought experiment on what a programming language designed for LLM coding agents, rather than human programmers, would look like.
RANT: Did AI Kill Programmers?
Akita's rant on why AI won't replace real programmers, why the programming bubble already popped, the S-curve ceiling of current LLM architecture, and what actually changes for engineers in the LLM era.
Vibe Code: I Built a Markdown Editor From Scratch With Claude Code (FrankMD) PART 2
Part 2 of the FrankMD story: refactoring, i18n, syntax highlighting, tests, and the hard-earned lessons from 30 hours of vibe coding with Claude.
Vibe Code: I Built a Markdown Editor From Scratch With Claude Code (FrankMD) PART 1
How I vibe-coded FrankMD, a self-hosted Markdown editor tailored to my blogging workflow, in three days with Claude Code.
2026 - January
3 postsVibe Code: Which LLM Is the BEST?? Let's Talk for REAL
I ran the same extensive analysis pass through Opus 4.5, GPT 5.2 Codex, Kimi 2.5, Gemini 3 Pro and MiniMax v2.1 on a tiny app, and every single one still left things on the table.
AI Agents: Comparing the Top LLMs of 2026 on the Zig Challenge
I put the top commercial and open source LLMs of early 2026 through a nasty Zig + llama.cpp migration challenge to see which ones actually deliver.
Omarchy 3 - One of the Best Coding Agents Out There: Crush
I used Crush with Claude Opus 4.5 on a Hugo blog and a Zig chat using llama.cpp, CUDA, and Qwen3. It refactored scripts, updated APIs, fixed a segmentation fault, and delivered a working binary for nearly $8.
2025 - June
1 postAGI or Skynet Isn't Coming Anytime Soon
Why the current LLM architecture cannot reach AGI, and why the hype from Big Tech CEOs and media-anointed experts is mostly backroom negotiation noise.
2025 - May
5 postsFinal Attempt to Train an LLM with LoRA. Cannon Shot, But Missing the Fly.
Renting an 80GB H100 on RunPod to fine-tune Qwen3-32B with a LoRA on Zig 0.14, and discovering that even cannon-class hardware can still miss the target.
Teaching the Latest Zig to Your LLM - Training LoRAs (Sort Of)
A hands-on walkthrough of training a LoRA adapter on Qwen3-8B to teach it about Zig 0.14, covering datasets, fine-tuning, vLLM serving, and the limits of the experiment.
RANT - LLMs are LOOT BOXES!
Why LLMs for real coding behave like gacha loot boxes, and why every incentive in the industry pushes you to burn more tokens.
When Do LLMs Fail at Programming? A More Realistic Use Case.
A real-world attempt to build a Zig program using llama.cpp with Gemini 2.5 Pro and Aider, showing where LLMs crumble on new languages and libraries.
Rant - Will LLMs Evolve Forever? Demystifying LLMs in Programming
A hype-busting rant on LLM benchmarks, the myth of exponential AI progress, and my hands-on experience showing why Vibe Coding does not work.
2025 - April
6 postsDissecting an Ollama Modelfile - Tuning Qwen3 for Code
A walkthrough of sampling parameters (temperature, top_p, top_k, min_p, repeat_penalty) and how to build a custom Ollama Modelfile to tune Qwen3 for software development tasks.
Testing the Newly Released Open Source LLM - Qwen3 (with Aider and Ollama)
First hands-on impressions of Qwen3 running locally via Ollama and remotely via OpenRouter, tested with Aider on real refactoring tasks.
Destroying the ChatGPT 4o "Personality"
How to disable ChatGPT's pseudo-human chatter and get straight, dry technical answers by tweaking the custom traits settings.
Testing LLMs with Aider on RunPod - which one to use for code?
Hands-on testing of Qwen2.5 Coder, DeepSeek, Codestral and others with Aider, locally on an RTX 4090 and on RunPod, comparing them against Claude and Gemini.
Your Own Free Universal Co-Pilot Running Local: AIDER-OLLAMA-QWEN
How to run Aider with Ollama and Qwen 2.5 Coder locally to build a free, universal Co-Pilot alternative that works with any editor.
LLM Hello World: Building Your Own Local AI Chat
A hands-on Hello World experiment explaining how LLMs actually work under the hood, building a tiny local chat CLI with Qwen 2.5 Coder running on a single GPU.