AI Fatigue and Vendor Lock-in

“What is the best AI tool to use: ChatGPT or Claude?” “Codex or Claude Code?”

“The new best AI agent has arrived.” “Claude Code just killed {insert any tool or workflow}.” “Codex has removed the need for {insert another tool or workflow}.” Fill in the blanks yourself.

Everyone selling an AI opinion has a reason to keep you unsettled. Their business depends on keeping you engaged and ready to switch. The industry keeps crowning a new model, then publishing a chart to justify it. Whatever you used yesterday is quickly presented as hopelessly behind.

The fitness and productivity industries have the same problem. Fitness keeps inventing a new answer to the same question, while conflicting studies make each recommendation look provisional. Productivity culture turns a simple task into a workflow you have to maintain. A new hack promises a min-maxed improvement, but getting it into your routine can cost more time than the improvement ever returns. AI is moving even faster, so you can waste your entire working life configuring tools that were supposed to save you time.

Eventually, the noise becomes background. Constantly reacting wastes your time; ignoring releases leaves you stuck with tools long after they stop earning their place. I need a filter for what deserves my attention and the freedom to replace anything that no longer does.

You can waste a stupid amount of time chasing model releases. You can lose even more by choosing one company and handing it your entire workflow. Claude Code could cost $500 a month tomorrow. A lot of programmers would complain and pay it because they have already built too much around it.

The Model Is Only Part of the Setup

I use Claude and Codex constantly. Claude is especially good at code review and conversational output. It is also just as proficient at ignoring instructions and lying about it. I can repeat one instruction five times, watch it ignore me, and still pay enough to wonder why I am paying for the privilege. Codex follows instructions far better for me, but its CLI is so horrible that you kind of have to use the desktop app if you care what the agent is doing. The desktop app is built with Electron (which is honestly ridiculous in 2026, especially from OpenAI. You literally make coding agents. If any company can scaffold a fast native desktop app, it should be the one selling us agents that write code. There is legitimately no excuse left). Good luck running that while doing really anything else on your computer. I will keep using both for now. I also want to replace either one the second it stops earning its place.

How I Personally Verify the “Best” Model

How do I know when either one has stopped earning its place? I am obviously not rebuilding my workflow on the whims of people on Twitter trying to go viral and drive engagement. I use benchmarks to decide what is worth testing myself.

For anyone new to this, a benchmark is simply a standardized test for an AI system. The organizer gives each model the same task and records whether the result meets the requirements. The resulting score tells you how that model performed under that test, not how it will perform in your workflow.

The harness is part of the test. It is the software around the model, controlling what the model can use, what it can see, and how the work is evaluated. Put the same model in a different harness and you can get a very different result, often with a different token cost. A direct API call may tell you more about the model than the product around it, but it may tell you less about how the model performs in your actual workflow. When a company posts a victory chart, I want to know what it ran before I care who came first.

Expand

DeepSWE's published leaderboard, captured from the DeepSWE benchmark on May 26, 2026.

More importantly, I want to know who paid for it. If a cigarette company funded a study that found cigarettes extend your life, nobody would accept the result because the graphic looked professional. I treat a benchmark created by the same lab whose model somehow wins it with the same suspicion. I do not look at the victory graphic. Show me the problems and the harness.

Expand

DeepSeek's benchmark chart from the DeepSeek-V4-Pro model card, published alongside the April 24, 2026 preview release. It is a useful claim to inspect, not an independent verdict.

People outside this field should not need to understand benchmark methodology to avoid being misled, yet they are often the exact audience these charts are made for. When DeepSeek released, the claims about its strength looked ridiculous to me once I used it. I did not need to build a formal benchmark suite to see that it was nowhere near as accurate as advertised. Someone who does not work with these models regularly might hear the claim repeated after seeing the chart. Of course they might take it seriously.

DeepSWE is the public benchmark I take most seriously because its tasks are new. Its reference solutions are written from scratch and kept out of upstream repositories, so the model cannot pass by recalling a patch it saw during training. That matters more to me than another score on a recycled GitHub issue. For the problems I am solving day to day, the agent is usually working inside private repositories on code that did not exist when the model was trained. There is no answer hiding in its weights. It has to understand the code in front of it. That is the number I want from a benchmark.

DeepSWE narrows the field. I then evaluate promising models against a fixed suite drawn from completed engineering work. For a bug-repair case, I reset the repository and give the model the original report without the accepted patch or regression check. Each model runs under the same fixed conditions. A run succeeds only if it recovers the correct behavior and leaves the existing suite green. I track human intervention and the total cost of reaching an acceptable patch. Repeating these cases across releases shows me when an open-weight model has become capable enough to take on work that previously required a closed one.

We’ve Been Here Before

That matters because a vendor can change the deal whenever it wants. Anthropic already did. Developers were using Claude subscriptions through third-party tools and custom harnesses. Anthropic shut that down. Its current Claude Code documentation says subscription OAuth is only for Anthropic’s apps. Third-party tools have to use the separately billed API.

I get why. A flat subscription gets expensive when someone runs agents through it all night. From my side, the outcome is simple: the subscription I already pay for cannot follow me into my own harness. I have to use Anthropic’s interface or start paying a separate meter. If Claude stops making financial sense there, I need another model ready.

Anthropic is following a playbook Apple spent years perfecting. Apple kept Lightning on the iPhone long after USB-C was available because controlling the connector still paid. The EU eventually made USB-C a condition of selling new phones there. Keeping Lightning now meant building a separate iPhone for Europe or walking away from those sales. Either option cost more than switching, so Apple switched. Claude is heading in the same direction. My subscription works through Anthropic’s interface, but plugging Claude into my own harness means paying a second bill. We’ve been here before, and I’d rather not wait for regulations to arrive. I’ll opt out by opting into alternatives.

Always Retain the Ability to Leave

Having a replacement model ready only matters if I can move my workflow to it. In production, each layer of the stack had its own format and interface. Claude Code looks for CLAUDE.md; Codex follows AGENTS.md; other CLIs use their own manifests. A skill that worked in one environment could disappear in another because its frontmatter fields or directory layout differed. Plugin manifests drifted too. Hooks created another seam: one might run as a shell script, another as a Node or TypeScript process, and ours as a Rust executable. They still expected different event names and payloads.

We were dealing with twenty different CLIs, forty skill formats, twenty plugin formats, and forty harnesses. We built one internal harness that standardizes those boundaries. We can declare an agent once, translate skills into the format a tool expects, and route hooks through a controlled implementation. Trying a new model is now a configuration change instead of a workflow rewrite. When something breaks, we control the bug fix.

A technical team can absorb the upfront work; for everyone else, building a harness may become the new problem. The worst part of the current tools is that each one has functionality the others inexplicably lack. Claude Code gives me a useful status line; its desktop app is rough. Codex is easier to use on the desktop, although it is made with Electron. Its CLI offers little visibility into agents, especially when you want status-line updates. The open-weight tools I have tried are often too minimal to scale to what I need. Many are written in TypeScript, which limits how far I can push their speed and functionality. Bouncing between them makes me context-switch constantly, which is exactly the slowdown agents were supposed to remove. Building your own harness lets you reduce both AI fatigue and vendor lock-in. If you have the technical capacity, I strongly suggest doing it.

My current harness is a Tauri application written in Rust. (As one should do in the modern era.)

If I want to test a feature, I build it. Owning the place where all of this comes together lets me build what I need. Anthropic and OpenAI cannot even standardize skills between the two biggest coding agents. Fine! I wrote an internal compiler that takes my format and emits whatever each agent currently expects. Hooks get the same treatment, also in Rust, so I am not running another Node.js process on each agent event.

Hooks are part of this too because I use them for correctness as much as style. They block patterns I have already decided do not belong in the codebase, regardless of which model is writing. The guardrails come with my harness. I covered the mechanics in Correctness by Construction and Correctness by Construction in the AI Age, so I will not re-litigate them here.

Owning it costs time up front, and I think that cost is worth paying. The trade-off changes from task to task, and my harness lets me decide without accepting one company’s defaults.

An Apple-versus-Samsung future for AI would be awful. Choosing a model should not determine the rest of your setup.

As long as I can afford them and they’re the most powerful agents, I’ll use them (albeit begrudgingly).

If an open-weight model from China wins, I’ll use it. If a closed Chinese model wins, glory to the Chinese Communist Party, I suppose. If open source takes the crown, however, don’t tread on me. A dollar saved is a dollar earned.