TL;DR - VentureBeat Intelligence surveyed 100 people who build, run or use AI at organisations of 100 employees or more. 47% of them had swapped the model inside a production system in the past year while keeping the application around it. Of that group, 69% spent a few weeks or more on the swap, 36% spent more than a month, and only 13% said it was close to a config change. 44% either watched performance drop and reworked the application or went back to the old model. This is a U-letter story: the model under your application is software you depend on, and a change to it needs an evaluation set, a rollback plan and a change window, exactly like a patch. What you need to do: build an evaluation set from your own real requests, keep your business rules outside the model, test the candidate inside the harness it will actually run in, run the old and new versions side by side on live traffic, and watch the cost per good result instead of the price per token.
| Figure | Value | Source |
|---|---|---|
| Qualified respondents | 100 (from 106 surveyed, October 2026) | VB Pulse, agentic orchestration report |
| Swapped a model in production, kept the app | 47% | VB Pulse |
| Swap took a few weeks or more | 69% | VB Pulse, 45 respondents who swapped |
| Swap took more than a month | 36% (12% in September) | VB Pulse |
| Swap was close to a config change | 13% | VB Pulse |
| Performance dropped and app reworked, or reverted | 44% (58% in September) | VB Pulse |
| Agent or workflow failure affecting the business, past 12 months | 36% | VB Pulse |
| Could not say whether one had failed | 22% | VB Pulse |
| Failures that reached customers or a financial, regulatory or board report | 56% | VB Pulse, 34 respondents |
| Same model, different harness, cost per passed task | $1.05 to $18.34 (17.5x spread) | FrontierHarness Eval, Kimi K3 |
| Claude Fable 5.2 price cut | 20% ($8 and $40 per million tokens, down from $10 and $50) | Anthropic release notes, 9 November 2026 |
I run my own agent harnesses, and I swapped the model in them this year the way you would swap a lightbulb. One line of config, four hours of testing, done. That experience has quietly misled me into thinking model swaps are cheap everywhere. It was cheap for me because I built the harness around swappable seats and I had already made the calls that make a swap boring.
Last week I read a survey of 100 people who run AI at organisations of 100 employees or more, and it says my experience is the exception, not the rule. When they swapped the model in a production system and kept the application around it, 69% spent at least a few weeks of engineering time. 36% spent more than a month. 13% said it was close to a config change.
Then there is the number that stayed with me. 44% of that group either saw the new model perform worse, reworked their application around it, or went back to the original model. That is nearly half of the organisations that tried this, landing a new model and then undoing it.
The interesting part is not that swapping is hard. It is that the industry has been selling swaps as easy, and the people paying for the mistake are the ones who built the application, not the ones who shipped the model.
What the survey actually asked
The question is worth quoting, because the easy version of this conversation is "companies are switching to cheaper models" and the real question is much harder.
VentureBeat Intelligence asked whether the respondent's organisation had replaced the model in a production AI system in the past 12 months while keeping the surrounding application in place. Not a new app with a new model. The same app, new engine.
Of the 100 qualified respondents, 47% said yes. Of the 45 who said yes, they were asked how much engineering effort the swap took beyond changing the model itself:
- 13% said almost none, close to a config change
- 9% said a few days
- 33% said a few weeks
- 36% said more than a month
- 9% did not know
Add those up and you get the shape of the problem. Roughly seven in ten organisations treated a model change as a project with weeks attached.
There is a second number in the same report that deserves more attention than the swap. 36% of all respondents said an AI agent or automated workflow had failed in the past year in a way that affected a customer, an employee or a business process. Another 22% could not say whether one had. Of the 34 who reported a failure, 56% said the failure reached customers or a financial, regulatory or board report. 41% said customers received incorrect output. 38% said it was caught internally.
That is the cost of not knowing what your AI system is doing. It sits underneath the swap question.
Where the weeks go
The model is the part everyone talks about. The application around it is where the work lives.
An enterprise AI system wraps a model in prompts, tools, tests and an agent loop. Each of those was tuned against how one particular model behaves. When you drop a different engine under that stack, the behaviour changes, and everything tuned to the old behaviour has to move.
Ben Kus, chief technology officer at Box, put it plainly on VentureBeat's Beyond the Pilot podcast on 30 October. Executives tend to assume a newer model means better results, and that assumption is often wrong. His prediction is that the most severe differences you will see come from the harness rather than the model, because the current frontier models are all very good. Box now tests harnesses and models in combination rather than picking a model alone.
PayPal's Ren Zhang, vice president and global head of AI product, described the same problem from the other side of the switch. PayPal had moved its commerce agent to a smaller, faster model and fine-tuned it, and accuracy improved, though not enough. The bigger lesson was what happened next. The company decided to fine-tune less, because fine-tuning takes a specialist team and every new model means fine-tuning again. Fine-tuning binds your work to a specific model, and that binding is what turns a config change into a project.
The same week, JPMorgan Chase was making a new model a gated event. Anthropic's own account of the rollout says new Claude models go through the firm's approval process before anyone there can use them, and that Claude Code runs in controlled environments including Devspace, the firm's containerised development platform, specifically so agents do not get direct access to employee credentials or internal systems. Access widens in stages. That is a model change treated as a production change, because it is one.
T-Mobile's approach is the one I would copy. When the customer service agents change, they run the old and new versions at the same time on real customer calls, and they scale the new one only if it works better. They also run a small model that monitors every chat and voice session and flags answers that look wrong, because testing before launch cannot cover every way a customer asks a question. Their chief information officer, Jeff Simon, said plainly that in customer service they are not trying to optimise for cost. They are trying to give customers fast, accurate answers.
The harness is the part you control
Here is the number that convinced me the swap maths is not what most people assume.
Runta's FrontierHarness Eval ran the same Kimi K3 model through nine coding harnesses, 12 configurations in total, on the same 30 software engineering tasks. Same model, same tasks, same runtime. What changed was the harness, and the cost moved by 17.5 times:
| Harness | Pass rate | Median cost per passed task |
|---|---|---|
| Codex | 66.7% | $3.47 |
| DeepSeek Harness (Creator) | 63.3% | $3.28 |
| Claude Code | 63.3% | $18.34 |
| Pi | 60.0% | $2.43 |
| Exo Harness | 53.3% | $1.05 |
| Hermes | 50.0% | $2.90 |
Claude Code and the DeepSeek Harness configuration scored the same pass rate on the same tasks, and one cost 5.6 times more per passed task than the other. The README for that project puts it as a headline: similar pass rate, 17.5x cost differences.
Two things to note, and I will be honest about both. Thirty tasks is a small test, so treat the ranking as a rough guide. And my own Hermes harness sits at the bottom of that pass-rate column at 50%, with a median cost per passed task of $2.90. That is a humbling place to find your own work in a public leaderboard, and it is the reason I trust the finding more than I would have before.
Micah Hill-Smith, co-founder of Artificial Analysis, made the point that lands hardest for anyone planning a cheap migration. For the largest models, the harness mattered less than expected. For smaller models it matters enormously: a small model can swing 20, 30 or 40 points in coding agent results depending on whether it was trained on that harness's style. A harness built for a frontier model may suit a small model badly. Moving down to a smaller model is how a task gets dramatically cheaper, and the price is paid in engineering time.
If a cheap model produces the wrong answer more often, the cheaper model is not cheaper. Upwork's Sudhanshu Bist gave the audience at VentureBeat's Agentic Infrastructure Stack Summit a worked example on 28 October. A small model that costs 10 cents a run and succeeds 20% of the time needs five runs on average, so 50 cents per good result. A larger model at 30 cents a run succeeding 90% of the time costs about 33 cents per good result. The larger model costs three times as much per run and less per good result.
His gateway does not even see the full cost, because it tracks spend per team on model calls and does not see the GPU time spent training and fine-tuning, the time a person spends fixing an agent's output, or the processor cost of a tool call.
Keep the knowledge out of the model
The most useful thing PayPal said was not about models at all.
PayPal's internal data science assistant could reach vetted, high-quality data and still could not reliably answer business questions, because answering well required the definitions, query patterns and analytical procedures PayPal's own people use. A "golden query" specifies how to retrieve data for a particular purpose. An analytical playbook specifies how to investigate a question, including which segments, products or behaviours to look at, such as login failures and passkey use.
When PayPal added that knowledge layer, accuracy on internal business questions went from 43% to 95% without changing the model. Those are PayPal's own reported numbers, with no evaluation method published and no outside verification, so treat them as one company's account rather than a benchmark. The design decision behind them is the part worth copying. Zhang was deliberate: "If the model is getting better, I'll just plug in and play. I should not change my knowledge."
A company that ties its knowledge to one model tunes that knowledge again for every new model. A company that keeps the knowledge in a layer it controls can swap models without redoing the intellectual work.
Databricks customers run the same sequence. They build the first version on a frontier model with a simple text index, which Bryan Clark, director of product management at Databricks, called a smart agent and not a smart back end. They save the frontier model's answers as an evaluation set. Then they move to a cheaper open-weight model, add vector search to help it, and check the new model's answers against the evaluation set.
Box does the same thing with three numbers. Kus said the approach does not matter, what matters is accuracy, cost and speed measured on an evaluation set built from what customers actually ask Box's agents to do. Collecting that evaluation data is usually the hard part.
There is a governance version of this too. The report notes that 42% of respondents cannot point to anything that automatically stops their agents' token spending: 28% do not know how spending is controlled, 9% have no specific controls and 5% monitor only after the fact. Another 33% use limits built into their orchestration platform. So the same report that says swaps take weeks also says most of the people in these organisations have not finished the basics.
What to do about it
I want to be careful here. If you run a production AI system, most of this list is not about the model.
1. Build an evaluation set before you need it. Take 30 to 50 real requests that matter, with the answers you consider correct. Real traffic, not synthetic examples. This is the single thing that turns a swap from a project into a test. If you build it after the new model is live, you have nothing to compare against.
2. Keep your business rules outside the model. Definitions, playbooks, approved queries and policy belong in a layer you own, not in a prompt tuned to one model or in a fine-tune. PayPal's 43% to 95% move happened without touching the model at all.
3. Test the candidate in the harness it will run in. If your application runs the model inside a specific agent framework, that is the configuration you need to measure. The FrontierHarness Eval spread of 17.5x on cost per passed task came from the harness alone, with the model held constant. Test a second harness if you can afford it before you blame the model.
4. Run old and new side by side on live traffic. T-Mobile does this on real customer calls. It is the only comparison that catches the cases your tests never imagined, and it is how you scale the new version on evidence rather than on hope.
5. Measure cost per good result, not price per token. Add the failed attempts and the human fix time. A 10 cent model that succeeds one time in five costs more per good answer than a 30 cent model that succeeds nine times in ten.
6. Give the change a window and a rollback. JPMorgan Chase routes new models through its own approval process and widens access in stages. A model swap with no approval step and no way back is an untested dependency change in production.
7. Decide what the model is not allowed to do. T-Mobile runs a small model watching every session and flagging answers that look wrong. Anthropic's account of the JPMorgan Chase rollout says the containerised environment exists so agents do not get direct access to employee credentials or internal systems. The model that reads your data should not decide what it may read.
8. Write down what changed. When the model changes, record it the way you record a patch. Version, date, what broke, what you changed in the prompts, what the evaluation set said. The next swap starts from that record instead of from scratch.
Key takeaways
- 47% of the organisations in the survey swapped a model in production without replacing the application, and 69% of them spent a few weeks or more on it.
- 44% of the organisations that swapped saw performance drop and reworked the app, or reverted to the old model.
- 36% said an AI agent or workflow had failed in a way that reached a customer, an employee or a business process, and another 22% could not say whether one had.
- The same model in a different harness moved cost per passed task from $1.05 to $18.34, a 17.5x spread with the model held constant.
- 43% to 95% accuracy at PayPal, on PayPal's own unverified numbers, came from a knowledge layer with no model change at all.
- Cost per good result beats price per token: 50 cents against 33 cents in Upwork's example.
- A new model inside your application is a software change. It gets an evaluation set, an approval step and a rollback plan.
FAQ
Does this mean I should stop upgrading my AI tools?
No. The system that never changes is the one you fall behind with. The point is that the change costs more than the config line suggests, and you find out which cost before you commit, not after.
I am a small business with one AI feature. Does any of this apply to me?
The evaluation set part, yes, and it is cheap. Twenty real questions with the answers you like is a perfectly good evaluation set. The approval and rollback steps shrink to a single person deciding to switch back, which is still a decision worth making on purpose.
Should I fine-tune the new model to match the old one's behaviour?
PayPal stopped doing that, for a reason worth copying. Fine-tuning needs a specialist team and every new model means doing it again. Keeping your knowledge in a layer you control was cheaper and survived model changes.
How do I know which harness my application is actually using?
Look at what wraps the model call: the agent framework, the tool layer, the retry and permission logic, the prompt assembly. If you cannot name those in one sentence, you do not know your harness, and you cannot test a swap in it.
Is a smaller model worth the engineering time?
Micah Hill-Smith's numbers say small models swing 20 to 40 points on coding results depending on the harness they were trained against. If a small model is 5 times cheaper per token and swings 30 points on your task, the arithmetic may still work out, but only if you measure cost per good result on your own traffic.
Our gateway already routes automatically. Is that the same as a swap?
No, and this is the distinction the whole survey turns on. A router picks a model per request inside an application you already tuned. A swap is replacing the model under an application whose prompts, tools and tests were tuned to the old one. Tetrate's Varun Talwar described the router state honestly: the company is not yet fully able to know which model is right for which task, so customers enable automatic selection by default only after checking the answers stay as good and the costs come down.
My Take
The survey is small. 100 qualified respondents, self-selected, 28% of them consultants answering about a client, and VentureBeat's own methodology section says the results describe these respondents and may not hold for all enterprises. I would not hang a strategy on the precise percentages. The direction is hard to argue with: nearly half the organisations that swapped a model in production ended up reworking their application or going back to the old one, and almost nobody treated it as a configuration change.
What I take from it is a change in how I think about model selection. For two years the interesting question has been which model is best. The evidence in this report says the better question is what sits around the model and what I can measure before I commit. The 17.5x cost spread from the harness alone is the part I cannot unsee. We spent years comparing models on benchmarks while the wrapper we already controlled was moving the bill by more than an order of magnitude.
There is also a governance thread here that I do not think has landed yet. If a model change is a production change, then every control you already have for production changes should apply to it. Approval, staged rollout, monitoring, a way back. JPMorgan Chase is treating a new model as something that has to pass their own approval process before a single employee can use it. Most organisations will do the opposite, because the config change feels too small to need a process.
The cheap-model arithmetic will keep getting better, and the cheap-model failure mode will get more expensive, because the cost of a bad answer does not go down when the model does. That asymmetry is the part worth planning around.
Mathew Clark Founder, SecureInSeconds Currently: rebuilding my evaluation set so the next swap is a test and not a project Drafted with AI assistance - read, verified, and fixed by a human.
Further Reading
- VB Pulse Agentic Orchestration Report, October 2026 - the survey behind the numbers, including the full methodology and sample notes
- VentureBeat: Swapping AI models is easier said than done - the reporting, with the practitioner interviews from Uber, PayPal, Box, T-Mobile and Upwork
- FrontierHarness Eval results - the same model across nine harnesses, with pass rates and cost per passed task
- PayPal on building an AI assistant that went from 43% to 95% accuracy without changing the model - the knowledge layer design
- Anthropic: JPMorganChase expands Claude Code across the firm - model approval gates and controlled environments
- Anthropic model pricing - Claude Fable 5.2 at $8 and $40 per million tokens
- Harness over frontier: running production AI agents on open-weights models - my earlier write-up on why the harness is the part you own
- AI Token Consumption Is the New Uptime Problem - the metering and alert thresholds that make the cost numbers visible
