Designing AI API Integrations That Survive Provider Churn

The naive way to integrate an AI API is to read a comparison post, pick whichever model wins on the day, and write that model's name into your service. It works, and it keeps working right up until the provider publishes a retirement date. Then you discover how many places that string appears in, how much of your prompt tuning was specific to it, and that nobody on the team remembers why the timeout was set the way it was.

This article used to be that comparison post. It listed models, per-token prices and rate limits in tables. Most of those models have since been retired, several of the prices changed, and at least one of the rate-limit rows was inverted by a tier revision — advice that pointed readers in exactly the wrong direction. The tables were the least durable thing in it and the fastest to become actively misleading.

So the tables are gone. What follows is the part that did not rot: how to structure an AI integration so that a model retirement is a configuration change, a provider outage is a failover, and a price rise is a routing decision rather than a rewrite. Where you need a number, this article links the vendor's own page rather than copying it — because a copied number is wrong the moment the vendor edits it, and you have no way to tell when that happened.

What rots, and what doesn't

Before designing anything, it helps to sort your decisions by how long they stay true. The instinct is to treat all of it as equally solid. It isn't.

Rots quickly — never hard-code, never transcribe into docs:
  model IDs and version suffixes
  per-token prices and billing units
  rate limits and tier thresholds
  context window sizes
  "which provider is best at X" rankings
  SDK class and method names

Durable — safe to build structure around:
  the shape of a request (system prompt + messages + output cap + sampling)
  429 means slow down; 5xx and timeouts mean try somewhere else
  what actually drives your bill (tokens in, tokens out, how often)
  the need for a second provider you have already tested
  the compliance questions you must be able to answer in writing

Everything in the first list belongs in configuration, in a vendor link, or in a test you run on a schedule. Everything in the second list belongs in your code, and will still be there in three years.

Put a seam between your application and the provider

The single highest-leverage thing you can do is refuse to let a vendor SDK type appear anywhere outside one package. Not because you expect to switch providers every quarter — you probably won't — but because the seam is what makes a retirement, an outage or a price change survivable at all.

Keep the interface small. Every method you add is a feature you have to implement for every provider, and the temptation is to widen it until it is just the union of every SDK, at which point it has stopped being an abstraction.

public interface AiClient {

    /** Returns the model's text response, or throws AiUnavailable / AiRateLimited. */
    String complete(String systemPrompt, String userMessage);

    /** For logging, metrics and routing decisions — not for branching business logic. */
    String providerId();
}

Two exception types is usually enough, and the distinction matters more than the names: AiRateLimited means the provider is telling you to slow down and that waiting will help, and AiUnavailable means waiting will not help and you should ask somebody else. Everything else — a malformed prompt, an invalid key — is your bug and should fail loudly rather than trigger a retry storm.

An Anthropic implementation

Note that the model ID is injected, not written into the class. That is the whole point: when a model is retired, you change a config value and redeploy, and the diff is one line.

import com.anthropic.client.AnthropicClient;
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
import com.anthropic.models.messages.Message;
import com.anthropic.models.messages.MessageCreateParams;

public class AnthropicAiClient implements AiClient {

    // fromEnv() reads ANTHROPIC_API_KEY from the environment.
    private final AnthropicClient client = AnthropicOkHttpClient.fromEnv();
    private final String modelId;      // injected from configuration
    private final long maxTokens;      // injected from configuration

    public AnthropicAiClient(String modelId, long maxTokens) {
        this.modelId = modelId;
        this.maxTokens = maxTokens;
    }

    @Override
    public String complete(String systemPrompt, String userMessage) {
        Message message = client.messages().create(MessageCreateParams.builder()
                .model(modelId)
                .maxTokens(maxTokens)
                .system(systemPrompt)
                .addUserMessage(userMessage)
                .build());

        // content() is a list of blocks; only some of them are text.
        return message.content().stream()
                .flatMap(block -> block.text().stream())
                .map(textBlock -> textBlock.text())
                .collect(Collectors.joining());
    }

    @Override
    public String providerId() {
        return "anthropic";
    }
}

An OpenAI implementation behind the same interface

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.chat.completions.ChatCompletion;
import com.openai.models.chat.completions.ChatCompletionCreateParams;

public class OpenAiAiClient implements AiClient {

    // fromEnv() reads OPENAI_API_KEY from the environment.
    private final OpenAIClient client = OpenAIOkHttpClient.fromEnv();
    private final String modelId;      // injected from configuration

    public OpenAiAiClient(String modelId) {
        this.modelId = modelId;
    }

    @Override
    public String complete(String systemPrompt, String userMessage) {
        ChatCompletion completion = client.chat().completions().create(
                ChatCompletionCreateParams.builder()
                        .model(modelId)
                        .addSystemMessage(systemPrompt)
                        .addUserMessage(userMessage)
                        .build());

        // content() is an Optional — a refusal or a tool call leaves it empty.
        return completion.choices().stream()
                .flatMap(choice -> choice.message().content().stream())
                .findFirst()
                .orElse("");
    }

    @Override
    public String providerId() {
        return "openai";
    }
}

The two bodies look nothing alike, which is exactly why the seam earns its keep. One returns a list of typed blocks, the other an optional string per choice. If that shape leaks into your service layer, you have coupled your business logic to a vendor's response envelope.

Model IDs and provider choice live in configuration

ai:
  primary:
    provider: anthropic
    model: ${AI_PRIMARY_MODEL}      # set per environment, not in the repo
    max-tokens: ${AI_MAX_OUTPUT_TOKENS}
  fallback:
    provider: openai
    model: ${AI_FALLBACK_MODEL}
  bulk:
    provider: anthropic
    model: ${AI_BULK_MODEL}         # the small, cheap model

Reading the model ID from the environment rather than committing it means you can roll a replacement model through staging, then production, without a code review cycle — and roll it back the same way when the outputs turn out to be subtly different. They usually are.

Tiered routing: the cheap model does the bulk

The most reliable cost lever is not negotiating rates. It is noticing that most requests in a typical product are easy, and that you were sending all of them to the model you picked for the hard ones. Classification, extraction, tagging, routing, short summarisation — a small model handles these. Ambiguous input, long multi-step reasoning, anything a human would have to think about — that is what the expensive model is for.

The hard part is not the routing. It is deciding which bucket a request is in before you have paid for an answer. Two rules keep this honest:

• Route on a property you can check cheaply. Input length, whether the request came from a paying tier, whether the document parsed cleanly, whether a prior step already extracted the fields you need. These are all knowable before the call.
• Or try cheap first and escalate on a verifiable failure. Not "does the answer look good" — you cannot tell, and asking a model to grade itself is not a check. A real check is one your code can run: the JSON parses against your schema, the extracted date is a date, the classification is one of your known labels, the model explicitly returned an "unsure" sentinel you told it to emit when it cannot answer.

public class TieredAiClient implements AiClient {

    private final AiClient bulk;         // small, cheap model
    private final AiClient escalation;   // large, expensive model
    private final Predicate<String> acceptable;   // your verifiable check

    public TieredAiClient(AiClient bulk, AiClient escalation, Predicate<String> acceptable) {
        this.bulk = bulk;
        this.escalation = escalation;
        this.acceptable = acceptable;
    }

    @Override
    public String complete(String systemPrompt, String userMessage) {
        String draft = bulk.complete(systemPrompt, userMessage);

        if (acceptable.test(draft)) {
            return draft;
        }

        // Escalation is not free: this request now costs both calls. Track the rate.
        escalationCounter.increment();
        return escalation.complete(systemPrompt, userMessage);
    }

    @Override
    public String providerId() {
        return "tiered";
    }
}

Instrument the escalation rate from day one and alarm on it. Try-cheap-first only pays if escalations are the minority; if the rate climbs past the point where you are paying for two calls more often than one, the small model is no longer suited to the workload and you should route up front instead. That crossover is specific to your prompts and your data, which is why you measure it rather than take a number from an article.

Retry the rate limit, fail over the outage

These are different failures and deserve different responses, and conflating them is how a bad afternoon becomes an incident. A 429 means the provider is throttling you: retrying the same provider after a wait is correct, and failing over immediately just moves the load to a provider you were not planning to pay for. A connection timeout or a 5xx means retrying quickly will not help, and that is when the fallback earns its cost.

public class ResilientAiClient implements AiClient {

    private static final int MAX_ATTEMPTS = 3;
    private static final Duration BASE_DELAY = Duration.ofSeconds(1);

    private final AiClient primary;
    private final AiClient fallback;

    @Override
    public String complete(String systemPrompt, String userMessage) {
        for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
            try {
                return primary.complete(systemPrompt, userMessage);

            } catch (AiRateLimited e) {
                // Waiting genuinely helps here. Honour Retry-After when the provider
                // sends one; it knows when your budget resets and your backoff curve
                // does not. Add jitter, or every throttled caller retries in lockstep.
                if (attempt == MAX_ATTEMPTS) {
                    return fallback.complete(systemPrompt, userMessage);
                }
                sleep(e.retryAfter().orElse(backoffWithJitter(attempt)));

            } catch (AiUnavailable e) {
                // Waiting does not help. Go somewhere else on the first failure.
                log.warn("Primary provider {} unavailable, using fallback {}",
                        primary.providerId(), fallback.providerId(), e);
                return fallback.complete(systemPrompt, userMessage);
            }
        }
        throw new IllegalStateException("unreachable");
    }

    private Duration backoffWithJitter(int attempt) {
        long millis = BASE_DELAY.toMillis() * (1L << (attempt - 1));
        return Duration.ofMillis(ThreadLocalRandom.current().nextLong(millis / 2, millis));
    }
}

Three things this snippet is quietly doing, all of which are easy to leave out and painful to add later:

• Jitter. Without it, everything you throttled retries at the same instant and you throttle yourself again. Deterministic backoff synchronises your own fleet against you.
• A bounded attempt count. Retries consume the caller's request thread and the caller's patience. If the upstream HTTP timeout is shorter than your total retry budget, you are doing work nobody will receive.
• A fallback that has actually been exercised. A fallback path that runs for the first time during an outage is not a fallback, it is a second incident. Send a small share of live traffic to it continuously, or run it against your evaluation set on a schedule, so you know its outputs still satisfy your checks.

If you would rather not hand-roll this, Spring Retry and Resilience4j both express the same policy declaratively, and both are a reasonable choice. The logic above is shown longhand because the decisions — which exception retries, which one fails over, where the jitter goes — are yours regardless of which library applies them.

Spring AI or the vendor SDK directly?

Spring AI gives you a single ChatClient over many providers with Spring Boot auto-configuration, so the abstraction you would otherwise write by hand comes for free, along with retries, observability and a common options model.

@Service
public class SummaryService {

    private final ChatClient chatClient;

    public SummaryService(ChatClient.Builder builder,
                          @Value("${ai.bulk.model}") String modelId) {
        this.chatClient = builder
                .defaultOptions(OpenAiChatOptions.builder()
                        .model(modelId)
                        .temperature(0.2)
                        .build())
                .build();
    }

    public String summarise(String document) {
        return chatClient.prompt()
                .system("Summarise the document. If it is not a document, say so.")
                .user(document)
                .call()
                .content();
    }
}

The tradeoff has nothing to do with which version is current. It is this: a framework abstraction is defined by the intersection of what every provider supports, and the interesting features live outside that intersection. Extended thinking, provider-specific caching controls, structured-output modes and new tool-calling shapes all land in the vendor SDK first and reach the framework later, if the framework can express them at all. Meanwhile the framework's own API is one more surface that changes underneath you — the builder methods on its options classes have been renamed before, and code written against the old names does not compile against the new ones.

A workable rule: if you are using several providers in one application and want Spring idioms, uniform observability and no bespoke plumbing, take the framework. If your product depends on a capability that only exists in one vendor's SDK, use that SDK directly behind your own narrow interface — which you are writing anyway, because that interface is what makes the framework replaceable too.

The JavaScript side, and a lesson about packages

Model IDs are not the only thing that gets retired. Client libraries do too: Google's original @google/generative-ai package is deprecated in favour of the unified @google/genai SDK, and the old repository has been renamed to mark it as such. Code written against the old package still runs until it doesn't, and no compiler warns you.

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

// Same discipline as the Java side: the model ID is configuration, not a literal.
const MODEL = process.env.GEMINI_MODEL;

export async function summarise(document) {
  const response = await ai.models.generateContent({
    model: MODEL,
    contents: document,
  });
  return response.text;
}

Wrap this in the same kind of narrow function your Java services use. The value is identical: when the package is superseded again, one module changes and the rest of the codebase does not notice.

Where the money actually goes

Cost surprises rarely come from the headline rate. They come from volume you did not realise you were generating. None of the following depends on current prices, which is why it is worth internalising rather than looking up.

• The system prompt is billed on every single request. It feels like configuration, so it feels free. It is input tokens, multiplied by your request rate, forever. A system prompt that grew organically as people appended edge-case instructions is often the largest single line in an input bill. Measure it, trim it, and put the stable parts where caching can reach them.
• Embedding pipelines that reprocess unchanged documents. The classic version is a nightly job that re-embeds the entire corpus because that was the easiest thing to write. Key your embeddings on a content hash and skip anything unchanged. The cost of a full rebuild scales with your corpus; the cost of an incremental update scales with what actually changed, which is almost always a rounding error by comparison.
• Structured output inflates output tokens. Asking for JSON means paying for braces, quotes, field names and indentation on every response, and field names are repeated for every element of every array. Short keys and flat schemas are not micro-optimisation at volume. Verify whether your provider's constrained-decoding mode changes this before assuming it does.
• Cached, batch and standard requests are billed differently.Treat these as three distinct concepts. Standard is the synchronous call in your request path. Cached input applies when a stable prefix — a system prompt, a document you are asking many questions about — is reused within the cache's lifetime, so the provider does not reprocess it. Batchapplies when you can tolerate an asynchronous turnaround, which most background work can. The rates and the eligibility rules differ per provider and change; the structural point is that work you never restructured is silently paying the most expensive of the three.
• Retries and escalations multiply everything above. A tiered router that escalates often, or a retry policy that fires frequently, is paying for the same request more than once. This is fine and expected — as long as it is on a dashboard.

Instrument token counts per request and per feature, not rupees. Token counts are a property of your application and stay comparable across months; currency amounts mix in price changes and exchange rates, and you lose the ability to tell whether your usage grew or the rate did. Multiply by the current published rates when you need a figure for finance: OpenAI, Anthropic, Google Gemini, Amazon Bedrock.

Rate limits: read them, don't memorise them

Every provider limits requests and tokens per unit time, and every provider raises those limits as your account matures — automatically on spend, or on request. The specific thresholds are the least stable numbers in this entire domain. They differ by model, by tier and by account, they are revised without ceremony, and a tier restructure can invert which provider is the most generous.

What is durable is the operational advice. Find your current limits in your own dashboard rather than an article, before launch rather than after. Estimate your peak requests and peak tokens per minute — peak, not average, because the limit is enforced against the peak. Ask for an increase ahead of a launch, since approval is not instant. And handle 429 properly regardless of the headroom you think you have, because a retry bug is only ever discovered under load. Current limits: OpenAI, Anthropic, Google Gemini.

Data residency: a region name is not a guarantee

This is the section where getting it wrong is most expensive, and where the intuitive answer is wrong.

The intuition goes: I create the client in ap-south-1, therefore my prompts are processed in Mumbai. That inference does not hold. On Amazon Bedrock, the model ID you pass is frequently an inference profile, and the profile — not the client's region — determines which Regions can serve the request. AWS offers geographic profiles, which route within a geography such as US, EU or APAC, and global profiles, which do not. Of global cross-Region inference, AWS's own documentation states that "your requests can be processed by compute across supported commercial AWS Regions worldwide rather than within a single geography", and warns that "organizations with data residency or compliance requirements should assess whether Global cross-Region inference fits their compliance framework, since requests may be processed in other supported AWS commercial Regions."

The region you construct the client in still matters — AWS calculates price from the source Region, and it is where your API call is authenticated and metered. It is simply not, on its own, a statement about where inference runs.

// The client region and the routing scope are two different decisions.
var client = BedrockRuntimeClient.builder()
        .region(Region.AP_SOUTH_1)     // where you call from; NOT where inference runs
        .build();

// This is the value that decides which Regions may serve the request.
// Geographic profile  -> routed within one geography
// Global profile      -> routed across supported commercial Regions worldwide
var modelId = System.getenv("BEDROCK_INFERENCE_PROFILE");

var message = Message.builder()
        .content(ContentBlock.fromText(prompt))
        .role(ConversationRole.USER)
        .build();

ConverseResponse response = client.converse(request -> request
        .modelId(modelId)
        .messages(message));

var responseText = response.output().message().content().getFirst().text();

If you have a real residency obligation — a fintech or healthcare product, a contractual commitment to a customer, an auditor to satisfy — treat the following as the minimum:

• Do not infer residency from a region name, an endpoint hostname or an SDK constant. Read the provider's current routing documentation for the specific profile or endpoint you are using, and note that it is versioned and revised.
• Make the routing scope explicit and enforceable. On Bedrock, choose the geographic profile deliberately rather than inheriting a default, and constrain it with IAM or service control policies so a well-meaning change cannot silently widen it.
• Distinguish where inference runs from where data is retained.Processing location, logging, and any retention for abuse monitoring or model improvement are separate terms with separate answers. A satisfactory answer on one tells you nothing about the others.
• Get it in writing, dated. A commitment you can show an auditor is a contractual term or a documented service commitment, not a blog post, not a support chat, and not this article. Re-check it when you renew, because these terms change.
• If you cannot get the commitment, change the architecture, not the paperwork. Self-hosting an open-weights model on infrastructure you control is the option that makes the question answerable by inspection. It costs more engineering and usually more money; that is the honest trade, and it is a business decision rather than a technical one.

AWS's own guidance is the right starting point: Increase throughput with cross-Region inference and Global cross-Region inference. Read the version that is live when you make the decision, not the version quoted here.

Keeping the integration alive

An AI integration is not a thing you finish. It is a dependency on several services that change on their own schedule, so give it the same operational treatment you would give a database version.

• Subscribe to model deprecation notices for every provider you use, and make sure they reach a rota rather than one person's inbox. Retirements are announced with notice; missing the notice is what turns them into outages.
• Keep an evaluation set. A few dozen real inputs with the outputs you consider correct, checked by code rather than by eye. This is what turns "swap the model ID" from a gamble into a fifteen-minute job, and it is the only thing that catches a replacement model being quietly worse at your specific task.
• Log the model ID and provider on every request. When quality changes, the first question is what actually served the traffic, and you want that answerable from data rather than from memory of what was deployed.
• Re-derive your cost model periodically from live token metrics and current published rates, rather than trusting a spreadsheet built at launch.
• Exercise the fallback on purpose. Schedule it. An untested failover path is an assumption, not a control.