AI Translation

How we pick the AI model behind your translations

Sasho Savkov,Updated on November 16, 2026·5 min read
How a model earns the right to touch your content

Every few weeks a customer asks whether we support the newest AI model yet. Behind the question is a reasonable worry: AI models improve monthly, and customers want those gains to reach their translations. Meeting that expectation comes down to how quickly models reach production - and what has to be true before they do. Providers ship new models constantly, sometimes with only a couple of weeks between versions. At Lokalise, every release starts as a candidate, not an automatic upgrade. Public benchmarks grade models on somebody else's problem set, so each release has to be evaluated on its own merit, against ours.

Translation at our scale makes this harder: 959 language pairs as of this writing. This comes with a unique challenge most AI products don't have. Since we don't speak most of these languages, the people responsible for quality can't see it directly. The decision therefore has to be automated, with the evidence picking the winners.

Below is what that process looks like at Lokalise.

Before shipping a model: Three criteria to screen for

We don't route between every model on the market. Instead, we pick the best model from each provider, which holds a slot in our routing. When a provider ships a new release, the question is: does it take over the slot?

At Lokalise, we make this decision by screening models based on three criteria:

  1. Language coverage: Does the model handle the language well?
    We do this by translating into the target language, translating the results back again, and then measuring how much meaning survives the round trip. Back translation is a rough check and not a quality verdict since a fluent round trip can hide awkward output. But it's useful since it is fast and cheap enough to run on all 1,140 locales. Each locale comes back as a pass, a fail, or borderline. A model can excel at English, but still fail the test for Japanese, which is why you’ll see that not all pairs offer the same models in Lokalise’s product.
  2. Regression suite: How does the model deal with predictable challenges?
    We test what support flags, what customers report, and what has burned us before: right-to-left rendering, forbidden terms, locale punctuation conventions. Test cases are built for each, with every candidate running through the same checks. None of this is revolutionary. Regression testing is standard engineering, and the work is done for every release against a test set that keeps growing as production finds new ways to fail.
  3. Business impact: Which trade-off would customers feel more?
    Sometimes a candidate does better on one check and worse on another. For example, a new model might keep translations within character limits better than the current one, but fail to recognise brand names or product terms. When that happens, scores alone don’t decide. Instead, we make a judgment call about which failure would hurt our customers more. In this case, a translation that runs a few characters over is easy to spot and fix, while a translated brand name can ship unnoticed until the user finds it in the live product.
Every new model passes the language floor, the regression suite, and Shadow Tasks before it takes the slot and earns its traffic across 959 language pairs

Every release takes the same path: two screens, a rehearsal on real traffic, and a market where it earns its place.

Shadow Tasks: the rehearsal on real traffic

The screens only test what we know to ask, so the final check before a model touches real traffic is production.

Shadow Tasks take real customer requests and run them through the candidate in parallel, with the same context the current model receives, but the output stays internal.

Shadow traffic plays by the same rules as live traffic, and measure two things:

  • Cost. What a model costs on your traffic: tokenizers, verbosity, and context overhead all vary between models. Per-token price is not per-translation cost, and the only way to learn the second is to run the traffic. A model can clear every quality screen and still lose the upgrade on cost. The release call weighs both.
  • Latency. AI suggestions appear in the editor while a translator works, so a slow model makes them wait. Automations translate in large batches, so a slower model compounds delays in the processing queue. Published speed figures don’t reflect our prompts or volume, so we measure speed on real requests.

After launch: the competition starts

Clearing the three screens earns a model its provider's slot. But the traffic still has to be won per language pair, continuously.

The mechanism is a multi-armed bandit, an algorithm that balances exploitation (send traffic to what performs best) against exploration (keep giving alternatives a chance). The competitors are the providers, each with its best model; the traffic is AI suggestions in the editor; the signal is whether the editor keeps a suggestion unchanged.

Thompson sampling: for each language pair, we track how often each provider’s suggestions are kept, and how sure we are of that number. We’re confident where a provider has handled many requests, and unsure where it hasn’t. Each request takes one random draw from every provider's distribution and routes to the highest draw. Exploration happens automatically, in proportion to what's left to learn, and pairs with little traffic simply move more slowly. Its parameters are recomputed regularly from recent traffic and shipped like any other production change.

The result is a routing table that disagrees with itself, on purpose. For example, the best AI model for English–German is not the same as English–Japanese, and last quarter's winner isn't always the same as this.

Share of AI suggestions for one language pair, January to September 2026, across Provider A, Provider B, and a new model from Provider C
Real production data from one language pair, January to August 2026. The winners change, and traffic is earned rather than assigned.

As traffic grows, we plan to let every model from every provider compete, and to learn what works best for each organization as well as each language pair. Both split traffic into smaller groups, so each needs more volume before we can trust the results.

Two speeds of change

Routing picks the model in two places in Lokalise, and each place updates at its own pace.

AI suggestions in the editor update continuously

A translator reviews each suggestion before it’s published, correcting a weak suggestion or rejecting it. That human review gives us room to adjust the routing as new evidence comes in.

Tasks and automations remain stable

Since these translations go straight into use, these model changes need more care. For each language pair, we have a primary provider and a fallback option. We choose those based on the same evidence the bandit collects, but we update it only after review. Here, consistency matters more than always having the latest winner.

 AI suggestionsTasks & automations
Who sees the outputAn editor, every timeNo one, until it's in use
How routing changesContinuously, from live evidenceRarely, through reviewed updates
What it optimizes forBeing currentBeing consistent

So, do we support the newest AI model yet?

The honest answer is either a yes within weeks, or a no we can back with test results. If the question is if the field's monthly gains reach your translations, the answer is a hard yes. That is what we built this system for.

New models will keep shipping. Our job is to keep scoring, and evaluating each candidate to ensure the improvements arrive without you having to ask.

If you’re interested in what this means for your content and language pairs, our team can walk you through it. Contact sales →

AI Translation

Author

2.webp

Engineering Manager, AI

I'm Sasho, Engineering Manager at Lokalise where I manage the AI team, which mostly means I get to argue about evaluation metrics for a living and occasionally ship something that translates the internet slightly better.

Before this I spent a bunch of years in clinical NLP, including a PhD at Sussex teaching computers to read doctors' notes — which back then took years of research and is now roughly a weekend and a good prompt. Somewhere in there I picked up a handful of papers and patents, mostly on things like evaluating medical note generation and measuring how similar two sentences really are.

These days at Lokalise I help build the AI behind translation quality — RAG-based translation, evaluation pipelines, the unglamorous plumbing that keeps the AI-harness in check. It's going fine, mostly.

I studied Computational Linguistics at the University of Tübingen before my PhD at the University of Sussex, and I've spent 14 years across NLP research and AI engineering, with stops in academia, clinical AI, and now localization.

You can also find me at sasho.io and on Google Scholar.

AI translation SEO: a six-step workflow from strong source page to indexed translated page

AI translation SEO: Will Google penalize your AI-translated pages?

Google doesn't penalize pages for being AI-translated. It penalizes large numbers of low-value pages. Here's what Google has actually said, the five real AI translation SEO risks, and how to fix each one before you index.

Read more AI translation SEO: Will Google penalize your AI-translated pages?

Three stacks of coins, each taller than the last, illustrating the rising cost of building in-house AI translation

The real cost of building in-house AI translation

The model API is the cheap part. This is what it actually costs to automate AI translation, give it your company's context, learn to trust the output, and keep the system running.

Read more The real cost of building in-house AI translation

AI translation quality.webp

How to get data-backed proof that AI translation works — on your own content

Here's a stat that should make every localization leader uncomfortable: 57% of localization teams say their number one barrier to using AI more is that they don't trust the quality. Not that the quality is bad. That they don't trust it. The distinction matters. AI translation has improved dramatically. Models are better, context-aware systems like

Read more How to get data-backed proof that AI translation works — on your own content

Stop wasting time with manual localization tasks.

Launch global products days from now.

  • Lokalise_Arduino_logo_28732514bb (1).svg
  • mastercard_logo2.svg
  • 1273-Starbucks_logo.svg
  • 1277_Withings_logo_826d84320d (1).svg
  • Revolut_logo2.svg
  • hyuindai_logo2.svg