Localization metrics that matter: How to track, improve, and prove impact

Full transcript

Welcome and introductions

Maria (host): Welcome, everybody, and thank you for joining us from all over the world. With us today is Alex Terehov, Senior Product Manager here at Lokalise. He has been speaking with lots of customers and users of Lokalise Analytics to understand the main problems and scenarios you face daily. We are also joined by Kevin O’Donnell, Founder and CEO of Global10x, who helps companies with international expansion and advises them on their strategy to go global.

Why are we here? Sometimes we have lots of data, but we get lost in the localization data maze. So we are here to define which metrics really matter and how we can track them, improve them, and finally prove the impact of the localization work we all do.

First, we want to ask you: how do you track your localization data right now? Are some of you using spreadsheets? I know some of you might be using the magic ball. Or maybe some of you have already started with Lokalise Analytics, which I see — that’s great. A very mixed audience, with almost twenty percent of people using the magic ball, so I think you’ll find this webinar very interesting.

Data means nothing if we don’t know how to apply it and in which scenarios it becomes relevant. So here with me is Alex, who will uncover the real scenarios where data makes it real and relevant. Alex, the floor is yours.

Scenario 1: Making the case for budget and headcount

Alex Terehov: Hello. Today, Kevin and I will share three scenarios where we see a very good fit between Lokalise Analytics and some of the most common problems we observe. Often, people reach out to product when there is a problem and something needs to change. Consider this example: a games company with multiple studios. For the last two years they have had the same number of games, but more languages, more content types, and more content both inside the game and around it — knowledge bases, frequently asked questions, and so on. Yet they were operating on the same budget with the same headcount. How can they communicate that even though the number of games in the portfolio is the same, the amount of work is different? That is what we believe can be answered with a localization report.

A localization report helps with three main challenges. The first is budgeting: how do we say that next year we may need more budget, in a way that doesn’t overcommit or inflate the number, but still accounts for all the unplanned campaigns we now have to support in localization? Second, when we try to communicate how translation volumes are changing, we often feel more pressure — tighter deadlines, more incoming requests — but that emotional feeling is hard to communicate without hard data, and vendor invoices or spreadsheets are often disconnected, so they don’t show the full picture. The third challenge: with everything happening in AI and machine translation, can we do more with the same budget and headcount? Or should we secure more budget to use AI and translate content we didn’t even dare to translate before? Without seeing how things went before, it’s hard to communicate what we plan to optimize and what comes next. Kevin, how do localization reports help with these kinds of challenges?

Kevin O’Donnell: Thank you, Alex. I love this question because it’s so important for every localization team, no matter what size you are or what maturity level you’re at. If you want to be more business-focused and think about the impact of localization on the business, you have to start by creating a localization source of truth. Your dashboard should be the one place to understand everything that’s happening in your localization operations. That matters so much when you’re communicating upwards, with stakeholders and senior leadership — it gives you credibility, it allows you the room to scale your operations, and it gives you one place to observe everything that’s happening. Think about meeting with stakeholders across the business and being able to review, project by project, what’s happening — the efficiencies, the delays, the bottlenecks. It’s essential to come to that meeting highly credible, with rich data. I always encourage teams to find any opportunity to share this in highly relatable forms: put it in your newsletters, your business reviews, your QBRs.

In terms of what to track, I always start with the simple metrics. Think about source updates and the impact of source changes — whether they come from code, content updates, or something else — and what that means for your operations. Think about time spent on tasks, because this often gets lost: localization teams can be superheroes at just getting stuff done with minimal resources, but those hidden time sinks really matter when it comes to making improvements. And think about historical trends: which projects tend to run late, which tend to have a glut of changes near the end — you might want to change that.

I worked with a marketing localization team recently. They were very efficient but tended to be very ad hoc: localization requests came in quite late, there were a lot of late-breaking changes that required a lot of churn, and they were getting burnt out. They found it really difficult to get included in strategic planning or upfront decisions. What we decided to do was change the nature of the communication entirely. Instead of reporting how many words they translated last quarter or what the cost per word was, they started talking about time to market, project priorities, where to allocate budget, and which localization methods were most advantageous for each project. By shifting that focus, and by having a couple of dashboards that really demonstrated business impact, they were able to get a seat at the table. They were taken more seriously, their insights showed up in company all-hands and business reviews, and they got the influence they needed with senior marketing leadership. That helped shift them from a transactional partner to a strategic partner — and it all started with a central source-of-truth dashboard.

Alex Terehov: And on how to actually start — this is where Lokalise can jump in. First, historical trends: with Analytics you have up to three years of history, so you can observe seasonality and spikes, and map events you remember — like that launch that felt tricky — to specific volumes. That way you can predict how much volume and time a similar launch will need. But when you look at historical trends, what often slips through the cracks is that people are conscious of new content — a new blog post, a new feature — while iterations on existing content get much less attention. In this example, the customer had almost the same amount of content coming from iterations on existing features, which wasn’t getting much awareness in the organization, even though for the localization team it was about the same volume. Bringing this to attention and sharing it in town halls helps communicate the real amount of work.

Next is time spent on tasks. We have heard multiple times that people use averages: on average, it takes us this amount of time to finish translation or review. Instead, we suggest using the eightieth percentile, which means that in eighty percent of cases a task will be finished within that time. For example, one customer’s average was twenty days, and it always felt like they were late, with constant deadline pressure. When they looked at the eightieth percentile, they saw they actually needed at least one more week — so they set the expectation that they really need a month. A month is still a lot, but at least it’s a realistic expectation. At the same time, you can’t use the longest possible time — for this customer that was about three months, which is unreasonable to communicate; those are outliers. In Analytics we try to give you the tools to set realistic expectations and build a localization report of what happened before, so you can bring past insights into your planning sessions.

Scenario 2: Choosing the right translation methods

Alex Terehov: The next case Kevin mentioned is choosing the most advantageous translation methods, and decisions about changing them often come in the form of a business case. Here’s an example one localization manager shared with us. They have a user analytics platform and do in-house reviews. Their colleagues reviewing into German and French were complaining that they were wasting their time making really trivial changes. So the question the localization manager had was: what should we do about this? They were using Google or DeepL — traditional machine translation. This kind of challenge often comes in three forms. The first is choosing the best method for the purpose, because you cannot use one method for everything — you need to decide where to employ people, where to employ machines, and in what combinations. The second is finding the right cases: should we use AI for the knowledge base, real-time translation for something like chat messages, and so on? Choosing the right fit should be informed — that’s how you get a bit more credibility, like Kevin said, or at least feel more in control: you stop guessing. And the last part, which we see in the questions that come to us, is about editing: can we drop post-editing and just use AI? That kind of question is worrisome — can we do this, or is it too risky? Kevin, how do these business cases help us answer this question?

Kevin O’Donnell: It’s such a timely topic, Alex. The central way to think about this is to make an informed choice about your translation method. You’re absolutely right — there is no one-size-fits-all, and there should not be. So often a CEO or senior leader will say, why don’t we just put everything through ChatGPT? Why don’t we use AI translation for everything? And it’s a surprisingly difficult question to answer, because technically you can do that — if you spend a bit of money, you can get it all done. The real question is: what is the impact if we do this, and where is it appropriate? That’s a hard question, because nobody knows straight away where it fits and where it doesn’t. It’s all about balancing speed, cost, and quality. The first step is to move away from the subjective — stop this being opinion-based and move to data, have benchmarks, and have targets. Starting with something like Lokalise Analytics gives you the means to monitor this over time, so instead of somebody’s opinion that AI localization should come first, you can look at the data and make data-informed decisions.

Introducing new translation methods is rarely done lightly, because it can have a profound impact on quality, so you need to be very careful. But by introducing them carefully — testing across different projects and different languages — you can identify where each one is a good fit: pure human translation, MT plus post-editing, pure AI translation, or some other combination. Many companies use a variety of MT engines with different tunings, and by having one dashboard to track the impact, you can identify which is most appropriate. The results will probably vary across languages, projects, and certainly content types.

I worked with a company recently that was under severe pressure from leadership to slash the translation budget and introduce pure AI translation for everything — the CEO was on a buzz to bring AI everywhere. There was a big dilemma for the localization team. They didn’t want to push back entirely or be seen as resistant to change, but at the end of the day they were accountable for quality, and they needed to make sure the quality of the localization was fit for the business purpose and good enough to maintain brand recognition and awareness. So they said: yes, we will adopt AI translation, but we’ll do it where the data allows us to. They did some testing and identified that AI translation was perfect for developer documentation and for support — very few changes were needed in post-editing. It was not perfect for marketing: for some of the highest-profile pieces, extensive changes were needed and there were very few savings to be gained, because they were using a generic engine that wasn’t well tuned. By having this data — and it took a long time to compile — they were able to build a business case: we are embracing AI translation in these cases; over here we are not yet, we have more work to do, we’ll proceed with this type of localization while we spend time tuning the engine, improving the context, and testing again. And they had one dashboard to return to, where they could demonstrate improvements over time and keep iterating and refining.

Alex Terehov: And having this dashboard is where Lokalise Analytics kicks in. Here’s an example similar to what Kevin was sharing. A customer had a dashboard filtered to their knowledge base, where machine translation was the primary translation method. Then in September — under similar pressure to what Kevin just described — they decided to try AI and see how it would perform. Before, they had an edit rate of about twenty percent, sometimes more. When they were building the case, they found that initially, in September, with a lot of attention and a bit of resistance from multiple parties, the edit rate was almost forty percent — the generic model didn’t work well enough. After fine-tuning and improving the model and providing more context — in Lokalise you can provide context within tasks and key descriptions, and that context is shipped to the AI — things improved dramatically. They reduced the edit rate to six percent. Where before one in five keys had to be touched by a reviewer, now only six percent do — a major speed-up, and the customer support team freed up quite a bit of time. This case was a successful start, and from there they carefully expanded their adoption of AI. They built an image of making informed decisions — following the trends rather than just jumping ahead — and there were no dramatic errors or problems for the brand or the image of the company. And capturing this data is simply enabled: you do your edits in Lokalise and that’s enough — no special configuration, no getting out of your usual ways of working. That’s how you build informed decisions about methods.

Scenario 3: Building better vendor partnerships

Alex Terehov: The last scenario we want to cover is when we still need humans: how do we communicate that improvements are needed without causing conflict or sounding too aggressive — being constructive and actually leading a change? Here’s an example. A company in the ecommerce space had challenges with some language pairs — not systematic, but with worrisome delays quite often in the last couple of months — and they were wondering how to package their message to articulate the potential risk as a red flag. This kind of question often comes from three different sides. One is setting expectations: can we trust that this piece or this campaign will be translated by this time? Can we trust the vendor’s service-level agreements? We need to know how it went before to build that trust. The next is about languages: are our tier-one languages on time? Our tier-two languages? And if we’re not on time, is it a language-specific issue? Sometimes the same tasks don’t take long on average, but there are outliers that are suspiciously common yet not bound to a specific language or campaign — and we want to investigate what’s going on and work together with the vendor. Kevin, how do we work together with a vendor instead of getting into a conflict?

Kevin O’Donnell: It’s a really interesting topic. I think the best way to build a successful partnership with your vendor is to have transparent goals and to drive for a collaborative relationship. This really starts, again, with having one source of truth that you can stand over and maintain, rather than relying on your vendor to bring that data once a month or once a quarter to show you the throughput and the type of work you’ve been doing. If you’re in control of that, you can set the agenda in the partnership — particularly in a QBR. I always advise localization teams: you should be deciding what you want to cover in a QBR, not your vendor. It’s your relationship to manage and to own. Identify projects that are not running effectively, or languages that might be running slow, and use the QBR to dive into that in greater detail and understand what’s behind the delay. Are they missing some context? Is there a resourcing issue? Was there an inefficiency in how we delivered it? By having the data ready, you can bring this to your vendor and have a productive discussion.

I also think it’s key to have shared goals that you can track over time. These goals might be around quality, turnaround time, use of different translation methods, introduction of new technology, or something different. By having clear goals, tracking them, and having the data to monitor trends over time, you effectively keep everybody honest and avoid surprises for either party. It helps drive a very fair relationship and can really improve performance. In terms of operational excellence, localization teams are brilliant at continuous improvement, but it only works well when they monitor in real time and identify the projects and languages where blockages or delays are happening, so they can investigate rather than waiting for somebody to complain or to notice at the end of a project that something hasn’t been delivered. If you can proactively see that and step in, you can avoid this type of problem. And continuous improvement is only as good as the relationship with your vendors, because it is such a dependent partnership — you have to rely on them, particularly when you’re up against deadlines. This is all in aid of going from ad hoc to a more planned type of localization.

To give an example: I worked at Microsoft previously, and we had three localization vendors and over a hundred languages — a lot to juggle. We used our QBRs not for transactional matters, to see what’s happening this week on this project, but to look at our long-term goals. We created a dashboard where we would look at turnaround time, quality targets, and translation methods used, and we were able to discuss languages where there was a dip in quality or projects where there was a delay and maybe some quality gaps. Then we’d focus on why that was the case, what we needed to do to improve, and what the vendor needed to do to improve. It became more of a collaborative troubleshooting session, which turned into a productive, very long-term, enduring partnership — wonderful for everybody. By having quarter-over-quarter continuous improvement backed by data, you can invest in your vendor partnership and move towards a strategic way of working.

Alex Terehov: And I cannot hold myself back from saying: with Lokalise, you can get yourself to be like Microsoft. How? The main part is this chart — we have multiple charts, but I want to focus on time to close. To be fair with a vendor: in this specific case, a customer was working on improving their time to close, and you only need to glance at the chart once to see the improvement. Before, there were a lot of outliers that took almost a year to be closed. Some tasks were forgotten; some were never actually translated and got lost, leading to complaints and customer support requests; some were never even picked up by the vendor because of integration problems. All of these things were resolved. It took a year, but they went from being unpredictable and not trusting each other to something they can agree on and improve. It’s still fifty days — there is room for improvement — but they started there and are working together.

When working with a vendor, it’s important to be fair. Sometimes there is only one outlier, which can be painful and dramatic — but if it’s only one outlier, you can frame the conversation that way. Instead of pressuring the vendor and getting them into defensive mode, you become a bit more collaborative: we are working together to improve it. And when you need to investigate outliers, each of them inside Lokalise has details — hover over it and you’ll see the task, the language, how many words there were, and so on. In this case, for example, it was twenty-eight words that were lost because of a problem between the vendor’s system and the integration. Now you can bring that specific task ID, share something tangible with the vendor, and agree that you’re working on solving this specific problem this time. Doing this each quarter, each month, or each week will improve things tremendously — in half a year or a year, you won’t recognize how well operations are running. Kevin, we’ve covered three scenarios. Can you summarize them for us before we go to questions?

Recap: Three steps to prove localization impact

Kevin O’Donnell: Absolutely. First, the place to begin is always to create your source of truth for localization. It is so essential for localization teams to have one place to track what’s happening, to gain those insights, and to be in control of your own messaging — for your internal team, your internal stakeholders and leadership, and your external vendors. Secondly, understand the impact of translation methods, turnaround time, and quality, and understand the interplay between them. Have the control to affect each lever, understand what happens to different projects and languages when you change any of these factors, and then monitor trends over time. And thirdly, drive continuous improvement — which all localization teams tend to do — but most importantly, drive it through localization insights. Direct your focus towards where the priority is and where the business impact has its maximum potential. Instead of taking a peanut-butter approach and improving every single localization project and task, focus on where you can have the biggest impact, track it over time, and use your central source of truth to celebrate your wins, share them, and continue in that vein.

Maria (host): Great — thank you, Kevin and Alex. It was great hearing all the scenarios and connecting data to business. Everyone, if you want to reach out to Kevin, I’m putting the Global10x web page in the chat. Now is your time to ask questions — you can use the Q&A panel — and I’ll start with some that we already have.

Q&A

Q: Can I filter to a specific brand or subproduct in our portfolio?
Alex Terehov: Yes. In Lokalise you can filter by project and key. For all plans in Lokalise, there are filters by date and language. Enterprise customers can dive deeper and do something like what Microsoft does: dive into specific projects and specific products, and use project tags to group projects. For example, for a game, different types of content can be connected to separate projects, and you can filter the game overall, or just the knowledge base, or just quests. And with key tags, you can even track how a specific launch or a specific feature is going. So in Lokalise Analytics you can definitely do the filters — it’s available for enterprise customers. If you’re an enterprise customer, definitely go to your CSM and make sure you’re getting the most value you can. And if you’re not an enterprise customer, you know what to do — upgrade.

Q: Can I see which of my internal reviewers is having trouble or being slower, so I can ping them?
Alex Terehov: Yes, this case is very well covered in the dashboard. You should be using tasks, because with tasks you can attribute activity to a specific person. On a chart like this you will see outliers; hover over them and you’ll see the specific task and specific target language. So you’ll know that this person was doing a German review at this time and it took unusually long. Going to Analytics in these situations helps you see, like Kevin said, who it’s impactful to ping. Instead of going to public or shared channels and causing drama — hey, people, please be on time — you can send a direct message: I would appreciate it a lot if you finish sooner; we’re just waiting for your contribution. That’s how you can dive down to the specific person and address it effectively.

Q: How do we measure our UX writing copy in an ecommerce app, for instance?
Kevin O’Donnell: It’s a really interesting question, and there are a few ways I’d think about it. First, measure the source updates happening on that particular UX copy — track how effective your writing and content teams are at creating copy that’s suitable for efficient localization. Are they updating it many times and delivering several different iterations towards the end, causing a lot of repeat localization work that can be time-consuming and very expensive? Having a close understanding of source updates gives you a lot of information to bring back to your writing team. Secondly, try to understand the impact of different translation methods: if you put that UX writing through different methods — human translation, MT, or AI plus review — you can understand the level of effort needed to achieve high-quality localization. That tells you how effective your style guides are for human translators and how well tuned your MT engines are for the context you need. Depending on the results, you can either improve the context and preparation for your localization methods, or work with your writing team to improve how effectively they deliver high-quality source content.

Q: Could Lokalise provide custom reports where we could enter a translator’s rate and have costs computed, so we don’t need to combine analytics and spreadsheets?
Alex Terehov: In the tasks dashboard in Lokalise, you can download all of the tasks in a very detailed manner — for each task and specific language, which person did which part of the task and how much they did. Once you have all the languages, you just multiply by the person’s rate to see the monthly cost. That’s currently the best way to address costs inside Lokalise using analytics.

Q: To monitor my own work, do I have to create tasks for myself? (from a solo third-party translator)
Alex Terehov: This is interesting. Analytics usually focuses on people who oversee the program. For your own work, you have tasks: go into the tasks dashboard in a specific project and monitor whether specific tasks are on time. Analytics is focused on seeing the bigger picture; for solo work, the tasks dashboard should be enough.

Q: Will analytics be available in smaller time intervals than monthly?
Alex Terehov: Data in analytics is updated once a day, and you can filter to smaller ranges. The only caveat is that the columns you see will be monthly. You can still filter out part of a month and see partial data — it just won’t be automatically divided into weeks or days. For example, if you know you ran a pilot in a specific week to build a case, adjust your filters to that week and analytics will show the data for that week specifically, even though the column will be labeled, for example, March 2025.

Q: By edit rate, do you literally mean how many critical or necessary changes were made to the text? Should we have a baseline — for example, five errors per thousand words?
Alex Terehov: Edit rate depends entirely on what your reviewer decided to change. It may or may not be necessary — it depends on how aligned you are with the reviewer on what should be changed. What we observe, though, is that when people know they’re reviewing AI or machine translation, they tend to edit more than they would against human translations, because there’s a little bit of prejudice against AI and MT. So edit rate shows all edits, and it’s up to you to align with the reviewer so that only necessary changes are made. Kevin, can you share an insight on what a good baseline is?
Kevin O’Donnell: I’m going to push back on that a little, because it’s such a highly subjective topic that I don’t think any one number is the right answer. What’s far more important than coming up with one number — whether it’s five errors, ten errors, or two errors per thousand words — is to determine, for your localization project and, more importantly, for your content type, what your expectation of quality is and what you’re prepared to live with. That will change depending on whether you’re producing developer documentation — which isn’t selling your product; it has to be accurate and understandable, but doesn’t need to be perfect — or legal documentation or medical translations, where you want to be very careful. Understanding the nature of the content and what you’re prepared to live with is far more important, and that’s why one simple baseline isn’t effective. Tune your baseline and your expectations based on the project, the importance of the language, and the job the content is doing for you.

Q: What is the “other” translation method we can see in analytics?
Alex Terehov: In Lokalise, there are some technical things you can do with content — applying pseudo-localization, copying a project, restoring a project from a snapshot, copying source to target. Lokalise still tracks all of these changes, but they’re not real translations per se. We group them under “other” so you can see what was going on with the content while knowing these are technical changes that don’t necessarily need your attention.

Maria (host): Those were all the questions. I just want to say that we also created a report template for your localization metrics, which you’ll receive after the webinar — so you can start using analytics to report and show stakeholders the power and the impact of localization. We hope you enjoy it. Thank you, Kevin and Alex, for joining us in this session, and thank you everybody for being with us today. Let’s keep in touch. Bye!