How to get data-backed proof that AI translation works — on your own content

The AI trust ladder: four steps from doubt to confidence

Before diving into the product, it's worth understanding the framework Marta introduced during the webinar — what she called the AI Trust Ladder. It maps the four stages every team goes through when moving from “Can AI translate?” to “We trust it enough to ship.”

Step 1: Context

Does the AI know your brand, or is it starting from zero every time?

Without a strong context layer — glossaries, style guides, tone of voice instructions, past approved translations — every AI output is a fresh guess. The result is translations that are technically correct but sound flat. Generic. Off-brand.

Real readiness starts with feeding the AI your accumulated knowledge instead of treating every translation as a cold start.

Step 2: Proof

Can you verify AI works on your content — not someone else's benchmark?

This is where most companies stall. They either skip verification entirely, roll out AI on faith, or lean on a vendor's benchmark that doesn't reflect their actual content, terminology, or quality bar. Real proof means testing AI against your own historical, approved translations and getting a measurable answer.

Step 3: Transparency

When you get a quality score, can you understand why?

A good number alone won't build trust — it just relocates the doubt. Teams trust systems they can inspect. You need to see the reasoning, the side-by-side comparisons, the edge cases. Not “this score looks fine,” but “I understand it well enough to defend it to my VP.”

Step 4: Ownership

Who makes sure the system keeps working as your data evolves?

This is the step most organizations skip — and it's exactly where governance breaks down. Someone needs to own the process: watching for drift, reproving value as usage grows, and making sure that a massive new translation job doesn't silently degrade your AI context.

Context first: why generic AI sounds flat (and how RAG fixes it)

Alesia opened the product walkthrough with an analogy that stuck with me. A generic LLM translating your content is like onboarding a junior translator. They're competent. They'll produce decent work. But they don't know your product, your style preferences, or the terminology your team spent years refining.

A Custom AI profile powered by RAG is the senior translator who's worked with you for years and knows exactly how you want things to sound.

Here's how that plays out in practice. Take a simple string: “Create a support ticket,” translated into Polish. A generic LLM produces Utwórz bilet wsparcia — technically correct, but generic. Your team, however, has consistently translated “support ticket” as zgłoszenie do pomocy technicznej across your product. With RAG, the AI retrieves that pattern from your translation memory and produces output that matches your established terminology.

The mechanism is straightforward. RAG doesn't retrain the model (which is slow and expensive). Instead, it retrieves relevant past translations from your translation memory at inference time and uses them as context for the new translation. The more you translate and review, the more context the AI has — and the better it gets. It's a flywheel, not a one-time setup.

Custom AI profiles in Lokalise let you control this context precisely:

• Choose your LLM routing. For the AI Proof of Value evaluation, we pin the model so the comparison is deterministic rather than based on a non-deterministic model choice. For other translation workflows, you can use Lokalise's intelligent routing (which selects the best-performing model per language pair and content type) or lock to a specific model if your organization has compliance requirements.
• Select which languages provide context. Translation memories aren't always homogeneous. Choose only languages with reliable, up-to-date translations. If your German TM is pristine but your Italian TM has legacy quality issues, exclude Italian until that data is cleaned or replaced; otherwise, those lower-quality examples can teach the profile the wrong patterns.
• Filter by date. If recent translations are higher quality than older ones (common after switching vendors), you can limit context to newer TM entries.
• Create multiple profiles. Your legal content, marketing copy, and product strings each have different quality standards and terminology. Separate profiles mean each content type gets the right treatment.

All of this is powerful. But it raises the obvious follow-up question: how do you know it's actually working better?

Proving the value: testing AI on your own data

This is the problem Proof of Value was built to solve. And it's the part Alesia demonstrated during the webinar.

The feature runs a controlled experiment on your own data. It takes a set of your approved, high-quality translations as the reference standard. Then it retranslates the source content three different ways:

1. Custom AI profile — your RAG-powered profile, using your translation memory as context.
2. Base AI profile — a standard LLM with glossary and style guide support, but without access to your existing translations.
3. Google Translate — a general-purpose machine translation baseline.

It then compares all three outputs against your approved translations and shows you, quantitatively, how similar each one is to your established quality bar.

The entire evaluation runs in minutes. No human reviewers needed. No external audit. No waiting weeks for results.

In the demo, the results for one customer's French and Russian content showed that the Custom AI profile produced at least 13% more edit-free translations (perfect matches) than the base profile — and 31% less editing effort overall. For some longer, more complex translations, the Custom profile produced output that was identical to the human-approved reference. Zero edits required.

That's not a benchmark from a research paper. That's a customer's own data, evaluated against their own quality standard, in their own Lokalise project.

How to set up an evaluation

The setup is deliberately simple:

1. Select the project you want to evaluate.
2. Choose your trusted reference translations. Because translation memories can contain mixed-quality content, Proof of Value asks you to specify which translations represent your quality bar. You can filter by review status or by tag (e.g., a “quality-example” tag your team has applied to verified translations).
3. Select target languages. The feature shows you which languages have enough reference data for a meaningful evaluation.
4. Run it. You'll get a notification when results are ready — typically within a few minutes.

💡 Pro tip

For teams that want even more granular control, you can create additional profiles that use only tagged or reviewed translations as AI context (rather than your full TM). This is useful when your TM has mixed content types — say, creative marketing copy alongside strict software strings — and you want to evaluate each separately.

Under the hood: how the metrics work

As the engineer who built much of this system, I want to be transparent about what the numbers mean and, just as importantly, what they don't.

We use four metrics, each telling you something slightly different.

How we split the data

When you provide your trusted dataset, we divide it into two sets:

• Reference set: the “exam.” All three models translate these source strings, and we compare their output against your approved translations.
• Context set: the “study material.” Only the RAG-powered Custom AI profile can access this set during translation — because that's exactly how RAG works in production.

We keep these sets separate by key ID to prevent data leakage.

💡 Pro tip

We require a minimum of 500 keys for tag-based and review-based evaluations, and 200 project keys plus 500 TM keys for TM-based evaluations. Below those thresholds, noise starts to dominate and the results lose meaning.

What the results tell you (and what they don't)

A deliberate design decision: we don't label results as “good” or “bad.” We rank the models and show you the delta.

Why? Because absolute interpretation depends on too many factors — the target language, the content type, the length of your strings, the quality of your reference data. What we can tell you objectively is which model performs best on your data, and whether the gap is meaningful.

We also can't project exact cost savings from choosing one model over another. The evaluation is a snapshot — a controlled comparison at a point in time. Your real-world data will vary, your TM will grow, and performance will shift accordingly. That's why we built the re-evaluation capability: run it again after a major translation job, a TM cleanup, or a vendor switch to see how the landscape has changed.

Data quality is everything

I have to be direct about this: the quality of your input data determines the quality of your results. Bad data hurts the evaluation in two ways.

First, it teaches the RAG model the wrong patterns. If your TM contains inconsistent or incorrect translations, the Custom AI profile will faithfully replicate those mistakes. If you see a suspiciously small gap between Custom and Base profiles, your context data may be the problem.

Second, it corrupts the judge. Your reference translations are the source of truth in this experiment. If the reference says “apple” should be translated as “peach,” every model that correctly translates “apple” will score poorly. If all three models show unusually low scores, it's worth questioning the reference data before questioning the models.

Share of AI translations identical to your approved reference. Higher is better. Custom ahead in 7/7 customers (32/32 pairs), median gap +12.5 pts

Values are each customer's median across their language pairs. The spread shows the minimum-to-maximum custom-vs-base gap across pairs. Every production evaluation, April–August 2026. Standard means generic machine translation; base and custom profiles ran the identical model, so the gap isolates the customer's own data.

Proof of Value vs. Analytics: different questions, different tools

Alesia made an important distinction during the webinar that's worth reinforcing: Proof of Value and Lokalise's Analytics feature serve different purposes. Conflating them will confuse your stakeholders.

Proof of Value answers: Which AI approach works best for my content? It's a controlled experiment — same source strings, same moment in time, three different translation methods, compared against the same reference. It's the tool you use to justify adopting (or upgrading) a Custom AI profile.

Analytics answers: How is my chosen AI performing in production? It shows live data on editing rates, acceptance rates, and effort over time. It's the tool you use to prove ongoing value to stakeholders — “here's how much effort our localization program is saving this quarter.”

Analytics will never give you a reliable apples-to-apples comparison between AI approaches, because production conditions constantly change: different glossary states, different TM sizes, different reviewers, different project instructions at different points in time.

💡 Pro tip

Use Proof of Value to make the decision. Use Analytics to prove the decision is paying off.

Real results: what customers are seeing

One example from the webinar stands out. A European FinTech company ran Proof of Value on their German and Italian engagement communications — their most personalized, culturally nuanced content. The kind of content that typically requires a three-cycle human review process.

The result? A 49% perfect match rate with a Custom AI Profile, compared with 13% using generic AI.

This is counterintuitive. You'd expect AI to struggle most with highly creative, heavily localized content. But the logic makes sense: the more personalized your content is, the more nuance exists in your approved translations — and the more signal RAG has to work with. When your human translators have invested effort in getting the tone exactly right, that effort becomes the AI's best training data.

For content with minimal human editing, the gap between Custom and Base is smaller — because there's less nuance for the AI to pick up. The takeaway: Custom AI profiles add the most value precisely where the stakes are highest.

Here's what customers are reporting:

• “The results were definitive. AI with RAG clearly outperformed generic AI — it would be silly not to use it.” - Senior Localization PM, European fintech
• “The AI profile really calmed down a lot of issues our linguists were having. The more we translate, the more it adapts to preferences, and we stop making the same post-edits repeatedly.” - Technical Writer, Supply Chain Management
• “It's getting closer to our preferred style. Once corrections are made, the AI seems to accept the changes and starts producing the correct format in similar cases.” - Localization Lead, Consumer mobile app

What this means for your team

The bigger point isn't the feature — it's the shift in how localization teams can operate. For years, the conversation around AI translation quality has been stuck in a loop of subjective impressions, vendor benchmarks that don't reflect your reality, and expensive external audits that deliver results too late to influence decisions.

Proof of Value breaks that loop. It gives you an objective, repeatable, data-backed answer to the question your leadership is already asking: Does this actually work for us?

Here's how I'd recommend approaching it:

1. Start with your highest-value content. Don't test on throwaway strings. Run the evaluation on the content where quality matters most — that's where the delta between Custom and Base will be most meaningful, and where the business case is strongest.
2. Invest in your reference data. Tag or review your best translations. The evaluation is only as good as your reference set. If your TM is messy, clean a subset first — even 500 high-quality keys are enough to get meaningful results.
3. Re-evaluate after major changes. Ran a large translation job? Switched vendors? Cleaned up your TM? Run the evaluation again. Your AI's performance landscape shifts with your data.
4. Use the results to build your business case. A 13% improvement in edit-free translations or 31% reduction in editing effort isn't abstract — it's directly translatable to time saved, cost avoided, and faster time to market. Pair Proof of Value results with Analytics data to tell the complete story.

The gap between “AI can translate” and “we trust AI to translate our content” isn't a technology problem anymore. It's a measurement problem. And now you have the tool to close it.

Proof of Value is available on high tier plans. Contact your CSM to get started, or speak with Lokalise about running your first evaluation directly from the AI Profiles section in your Lokalise project.