What is the best LLM for translation? A comparison of top AI translation models

The results: LLMs vs traditional machine translation

The research team at Lokalise conducted an extensive blind-comparison study covering three language pairs (English to German, Polish, and Russian) across five translation models to test their hypothesis that LLMs have surpassed ‘more traditional’ MT systems for translation:

• LLMs: Claude Sonnet 3.5 and GPT-4o
• Traditional machine translation: Google Translate, DeepL, and Microsoft Translator

Our hypothesis holds: LLMs demonstrated superior performance across all tested language pairs, with translation quality marked as ‘good’ between 55.7% and 80% of the time, even without any contextual information.

There was a clear winner: Claude Sonnet 3.5 ranked #1 across all languages tested, achieving ‘good’ translations approximately 78% of the time across Polish, German, and Russian.

This aligns with the WMT24 conference findings, which identified Claude 3.5-Sonnet as the top-performing system, winning in 9 out of 11 language pairs tested.

WMT24 conference validates LLM superiority

The 2024 Conference on Machine Translation (WMT24), the main event for machine translation and machine translation research, provides independent validation of our findings.

Their comprehensive evaluation of 242 translation systems, including 8 large language models, and online MT systems for all language pairs, confirmed that LLMs are outperforming traditional machine translations.

Curious to learn how AI translation works? Read the linked jargon-free guide.

How Lokalise tested LLMs for translation

To ensure scientific rigor, we used multiple evaluation methodologies, including the Bradley-Terry model, one of the most respected statistical methods for ranking items based on pairwise comparisons. It’s a probabilistic framework that models the probability that translation A will be preferred over translation B using strength parameters.

This approach allowed us to establish a clear hierarchy of translation quality based on maximum likelihood estimation.

The language pairs we tested

We focused on high-resource languages to test, not only the hypothesis that LLMs have surpassed ‘more traditional’ MT systems for translation but also that LLM translation is already ‘good enough’ for high-resource languages.

We used Large Language Models to translate from English into three languages:

• English to German
• English to Polish
• English to Russian

Then we asked human annotators to evaluate translation quality through pairwise comparisons of translations from different engines. That means, for every translation, native speakers compared the variants from different engines and highlighted the best one.

600+ pairwise comparisons were carried out by multiple human annotators for each language pair.

Interestingly, while Russian is technically a high-resource language, our ‘goodness’ scores were significantly lower than for German and Polish, suggesting that resource availability doesn’t always correlate directly with translation performance.

So, hypothesis 2 holds somewhat.

Curious to learn more? Discover what’s the difference between NLP vs. LLM.

Key finding: Human vs machine agreement

One of the most significant discoveries from our research involves evaluator agreement.

We found that the gap between human inter-annotator agreement and AI ranking system agreement is minimal, whether measured by Cohen’s Kappa (a statistical measure that evaluates inter-rater reliability while accounting for chance agreement) or Average Jaccard Similarity (which measures overlap between evaluation sets).

In some cases, human annotators actually showed stronger agreement with AI ranking models than with fellow human annotators.

This suggests that AI systems can now evaluate translation quality with near-human reliability, a milestone that has significant implications for automated translation quality assessment.

Why LLMs outperform traditional translation tools

LLMs like Claude Sonnet 3.5 and GPT-4o bring several advantages to translation tasks that traditional machine translation tools don’t:

1. Contextual understanding: Unlike traditional translation systems, LLMs grasp the broader context of text, enabling more natural-sounding outputs
2. Cultural nuance: These models can better preserve idioms, cultural references, and tone across languages
3. Adaptability: LLMs demonstrate greater flexibility when handling specialized terminology or uncommon language patterns
4. Consistency: Our testing showed remarkable consistency in quality, with the best LLM achieving good translations 78% of the time across multiple language pairs

As large language models (LLMs) evolve and new models are released, the gap between LLM and traditional machine translation quality is only going to get bigger.

The WMT24 research confirms this trend, showing that traditional neural machine translation (NMT) is increasingly being integrated with or replaced by LLM-based approaches, typically via fine-tuning or as part of data cleaning/post-editing pipelines

Today, LLM code translation is increasingly common, which speaks volumes about how far the technology has come.

The time has come for machine translation to move over and Large Language Model translation to fill its bionic boots!

Current LLM and AI translation rankings

Based on the latest research findings and WMT24 results, here’s how the leading translation systems currently rank:

How to choose the right LLM for translation

While Claude Sonnet 3.5 emerged as the clear winner in our testing, the best solution may depend on factors such as:

• Language pairs required
• Content type (technical documentation, marketing copy, legal text)
• Integration capabilities (e.g. with a translation and localization platform)
• Budget constraints
• Volume and consistency requirements

With Large Language Models constantly evolving and releasing new versions, it’s hard to stay on top of which translation model to choose depending on your needs. 

That’s where Lokalise comes in. Our proprietary AI orchestration tool blends multiple AI engines and automatically picks the best one for your language pair and content type, resulting in an 80% first-pass acceptance rate. In other words, translations that are ready to publish without human review.

At Lokalise, we’re LLM agnostic, ensuring our customers always get the best performing LLM per language pair, without being locked into one model as AI technology continues to improve.

We also integrate with 60+ modern tools, so you can plug AI orchestration into your workflow in a matter of minutes.

When it comes to budget, customers have registered savings of up to 80% using AI translation with Lokalise instead of going the traditional linguistic route.

Stay tuned for more LLM translation research and insights

Our latest research, combined with findings from WMT24, clearly demonstrates that LLM-based solutions outperform traditional machine translation.

Remember though that LLMs need context for translation, in the same way that humans need context. That said, LLM-powered translations beat traditional machine translations even without context.

Without context, LLM-powered translations were rated “good” 78% of the time in our testing.

For even higher accuracy, consider a localization platform that already integrates with LLMs like Claude and GPT-4, allowing you to leverage the best model for each specific translation task. This approach ensures you’re always using the most capable system available while managing the entire translation process in one place.

As AI technology continues advancing, we expect to see even greater improvements in translation quality, with LLMs becoming the standard for professional translation workflows.

We’ll continue to update our LLM translation insights, so stay tuned.