We finally learned to switch LLM providers without rewriting half the app

The easiest way to switch LLM providers is to stop wiring Anthropic, OpenAI, or Gemini directly into your business logic. Keep one model interface, normalize tool schemas, isolate retries and fallbacks, and use an OpenAI-compatible endpoint where possible. Google Gemini can get surprisingly close with just three code changes: API key, base URL, and model name.

The moment this finally clicked for me, it was almost embarrassing.

We weren’t blocked by prompts. We weren’t blocked by evals. We weren’t even blocked by model quality. We were blocked by the dumbest thing possible: our app had quietly grown a second codebase made entirely of provider quirks.

One branch existed for OpenAI function calls. Another existed for Anthropic tool_use blocks. Then there was a weird little side path for Gemini because somebody wanted to test a deepseek api alternative after a nasty week of latency and pricing anxiety.

That’s when I realized the problem wasn’t “how do we switch models?”

The problem was that we had built the app as if the model was the app.

And that’s exactly backward.

Your app does not have a model problem

It has a dependency-boundary problem.

LangChain says this more cleanly than most people do: Agent = Model + Harness. That framing is useful because it forces you to separate the actual model from everything wrapped around it — prompts, tools, middleware, memory, retries, structured output, and routing.

If you don’t do that, every provider migration turns into a quarter-long archaeology project.

You start with a simple goal — maybe move from GPT-5 to Claude, or test Gemini 3.8 Flash because the pricing looks attractive — and suddenly you’re tracing weird assumptions through n8n workflows, OpenClaw agents, Zapier steps, and custom Python handlers.

The text generation part usually survives.

The pain lives everywhere else.

So where does provider migration actually break?

Not where people think.

Basic chat completion is the easy part. The trap is everything that made your app useful in the first place.

Tool calling is where the bodies are buried

OpenAI-style APIs typically use function tools with JSON Schema and a call_id-style flow.

Anthropic’s native API does not. It uses tool_use and tool_result content blocks, plus input_schema. That is not a cosmetic difference. If your agent loop assumes one shape, swapping providers is not a config change. It’s surgery.

Gemini sits in an interesting middle spot. Google documents an OpenAI-library migration path where you can keep the OpenAI Python client and change just three things:

from openai import OpenAI
client = OpenAI(
    api_key="GEMINI_API_KEY",
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)
response = client.chat.completions.create(
    model="gemini-3.8-flash",
    messages=[{"role": "user", "content": "Explain how AI works"}]
)

That’s real. It works. And if all you need is message-in, text-out, it feels magical.

But the magic fades the second your app depends on structured output, tool semantics, or provider-specific reasoning controls.

That’s the part nobody mentions in the “just change the base URL” demos.

Structured output is the second trap

A lot of teams think they have “LLM integration” when what they actually have is “a fragile JSON-parsing ritual.”

The better pattern is to define schemas once and let an abstraction layer handle the ugly parts. LangChain’s structured output support is useful here because it can prefer provider-native schemas when available and fall back to tool-calling when they’re not.

That’s not just convenience. That’s migration insurance.

Because once your codebase is full of provider-specific flags and hand-rolled parser hacks, every deepseek price increase or model deprecation becomes a fire drill.

The highest-leverage fix is boring on purpose

Normalize the schema.

Seriously. This is the least glamorous change and probably the most valuable one.

OpenRouter explicitly talks about normalizing schema across models and providers. LiteLLM does something similar by mapping OpenAI-style parameters by provider and model, and it can quietly drop unsupported params with drop_params=True instead of blowing up.

That one feature tells you everything.

A mature LLM stack assumes your app will ask for things some providers don’t support.

Here’s the kind of call that saves a shocking amount of pain:

from litellm import completion
response = completion(
    model="command-r",
    messages=[{"role": "user", "content": "Hey, how's it going?"}],
    response_format={"key": "value"},
    drop_params=True
)

Would I trust drop_params=True forever? No. You still need tests. You still need evals.

But it’s a fantastic escape hatch when your codebase has accumulated provider-specific assumptions like barnacles.

Why are retries and fallbacks still living in app code?

This is the part that drives me crazy.

I keep seeing production apps where business logic knows way too much about rate limits, context windows, and backup providers. That’s like putting your database failover logic inside your checkout form.

LiteLLM has already done a lot of the heavy lifting here: num_retries, ordered fallbacks, context-window fallbacks, even switching API keys or API bases when a deployment fails. OpenRouter exposes provider ordering, fallback controls, and routing constraints like require_parameters, preferred_max_latency, and max_price.

That stuff belongs in the harness.

Not in your refund workflow. Not in your Discord support bot. Not in the n8n branch that updates Salesforce.

If your application logic knows the difference between “Azure deployment in Europe failed” and “fallback to OpenAI-compatible endpoint in Canada,” you didn’t build an app. You built a hostage situation.

The alias trick more teams should steal

One of my favorite LiteLLM patterns is using a stable alias while infrastructure changes underneath.

model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-4o-eu
      api_base: https://my-endpoint-europe.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY_EU"
      rpm: 6
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-4o-ca
      api_base: https://my-endpoint-canada.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY_CA"
      rpm: 6
router_settings:
  fallbacks:
    - {"gpt-5.6-luna": ["gpt-5.6-terra"]}

Your app keeps calling gpt-5.6-terra. Infra swaps regions, keys, deployments, or providers behind the curtain.

That is how you stop provider changes from becoming product work.

But aren’t OpenAI-compatible APIs enough?

Enough for some migrations. Not enough for all of them.

Here’s the honest version.

OpenAI-compatible layers are real. They are useful. And they are absolutely not identical.

Anthropic still has native concepts that don’t map perfectly. Gemini still has its own pricing and capability quirks. OpenAI still ships features that become the de facto shape other providers imitate later, imperfectly.

So yes, use compatibility layers.

Just don’t confuse “compatible” with “interchangeable.”

The pricing rabbit hole is exactly why abstraction wins

This gets more obvious the moment finance shows up.

Google’s Gemini 3.8 Flash pricing through December 31, 2026 is $0.75 per 1M input tokens and $3.75 per 1M output tokens. Starting January 1, 2027, that doubles to $1.50 input and $7.50 output. Google’s Batch API cuts that in half to $0.375 input and $1.875 output per 1M tokens during the lower-priced period.

Anthropic adds a completely different dimension with prompt caching. For Claude Fable 5.1, Anthropic lists $10/MTok base input, $12.50/MTok for 5-minute cache writes, $20/MTok for 1-hour cache writes, $0.25/MTok for cache hits or refreshes, and $50/MTok output. The default cache lifetime is 5 minutes, and it refreshes at no additional cost when reused.

That’s not just pricing. That’s architecture pressure.

Once your team starts optimizing around batch discounts, prompt caching, fallback routing, latency ceilings, or a sudden deepseek price increase, you need a clean abstraction or you’ll end up re-plumbing the app every quarter.

And yes, this is the uncomfortable tradeoff: abstraction can hide useful provider-specific features.

If Anthropic prompt caching saves you real money, you probably should expose it deliberately. If OpenRouter’s routing controls improve reliability, you probably should use them. A good harness doesn’t erase these features. It gives them a controlled place to live.

What should you separate first if you only have one sprint?

If I had to do this under time pressure, I’d separate things in this order:

1. Prompt templates from provider request objects
   Stop building prompts inline inside OpenAI or Anthropic request code.

2. Tool definitions from provider syntax
   Define one internal schema, then compile it into OpenAI functions, Anthropic input_schema, or Gemini-compatible shapes.

3. Structured output parsing from model calls
   Your app should ask for CustomerIntent or InvoiceFields, not hand-parse JSON strings.

4. Retries, fallbacks, and rate-limit handling into middleware
   Keep this out of business logic completely.

5. A stable internal model alias
   Let the app call primary-agent-model, not gpt-5, claude-opus, or gemini-3.8-flash directly.

6. One OpenAI-compatible edge where possible
   This is especially useful for n8n, Make, Zapier, and custom agent stacks because existing HTTP clients and SDKs often keep working.

That last one matters a lot in automation land.

n8n’s OpenAI node now supports the OpenAI Responses API in node V2, but when an operation isn’t supported, n8n recommends using the HTTP Request node with predefined credentials for custom API operations. That’s a very practical migration escape hatch: keep the workflow logic, swap the endpoint behind an OpenAI-style request, and avoid rebuilding the whole thing from scratch.

The weirdly liberating part

Once you do this properly, switching providers stops feeling dramatic.

You can test GPT-5 against Claude. You can route some extraction jobs to Gemini 3.8 Flash. You can keep a Qwen or Llama path around for niche workloads. You can compare quality without first spending three weeks untangling your own integration code.

That’s the real win.

Not “multi-model strategy.” Not “future-proofing.” Those phrases are too clean.

The real win is much simpler: when provider drama hits — pricing changes, outages, deprecations, weird tool-call regressions — you get to respond like an adult instead of a hostage.

And if your app still treats one vendor SDK as part of your product architecture, that rewrite you’re dreading has probably already started. You just haven’t named it yet.