The buzz around large language models (LLMs) often highlights their incredible linguistic capabilities. We hear claims of models being “multilingual,” capable of understanding and generating text in dozens, even hundreds, of languages. And to a large extent, that’s true. For major global languages, LLMs have indeed made remarkable strides, bridging communication gaps in ways we only dreamed of a decade ago. But what happens when you step off the well-trodden paths of English, Spanish, or Mandarin?
What about languages like Basque, Kazakh, Amharic, or Sundanese? These aren’t obscure dialects; they’re vibrant languages spoken by millions, rich in unique cultural contexts and linguistic structures. Do our “multilingual” LLMs truly serve these communities with the same fidelity? My recent deep dive into benchmarking 11 mid- and low-resource languages revealed some fascinating, and at times, sobering truths. It turns out, “multilingual” is not always synonymous with “cross-lingual,” especially when the stakes involve genuine understanding beyond simple translation.
Beyond the English Echo Chamber: Unpacking Multilingual vs. Cross-Lingual
Before diving into the nuts and bolts of my research, let’s clarify a crucial distinction that often gets overlooked: the difference between multilingual and cross-lingual. A multilingual model, in its simplest form, can process multiple languages. It might have seen data in Basque, Kazakh, or Hausa during its training.
However, true cross-lingual performance implies something deeper. It’s the ability to transfer knowledge, reasoning, and even cultural understanding from one language to another. It means a model isn’t just regurgitating information it’s seen in a particular language; it’s applying generalized reasoning and contextual awareness to new problems presented in that language, especially when those problems are culturally specific or require nuanced understanding.
Most existing benchmarks for “multilingual” LLMs primarily rely on tasks translated from English. While useful, this approach inherently biases evaluations towards content and concepts prevalent in English-speaking cultures. It’s like testing a chef on how well they can adapt a French recipe using local ingredients – important, but it doesn’t tell you how well they understand the local cuisine itself or how to create an entirely new dish from scratch within that tradition.
My goal was to move beyond this English-centric view and rigorously test LLMs on their intrinsic understanding and performance within these less-resourced linguistic contexts. I wanted to see if they could truly “think” in Basque, or reason effectively about specific cultural concepts in Amharic, rather than just provide a translated approximation.
Inside My Benchmark: Languages, Datasets, and LASS
To achieve this, I built a comprehensive evaluation pipeline designed specifically to probe the depths of LLM performance on mid- and low-resource languages. The selection of languages wasn’t arbitrary; it included a diverse mix from different language families and geographic regions: Basque, Kazakh, Amharic, Hausa, Sundanese, and six others, totaling eleven distinct languages.
Native Data for Native Understanding
A core principle of this benchmark was to evaluate models using native-language datasets, not just translated content. This is where many existing multilingual benchmarks fall short. To remedy this, I leveraged datasets like KazMMLU for Kazakh (a Kazakh-specific version of the popular MMLU benchmark), BertaQA for Basque, and BLEnD for various low-resource languages. These datasets present questions and challenges rooted in the cultural and linguistic fabric of their respective communities, providing a more authentic test of understanding.
I combined these rich datasets with a robust evaluation methodology: zero-shot chain-of-thought (CoT) prompts. This approach forces the LLM to “think step-by-step” and demonstrate its reasoning process, rather than just guessing an answer. It provides a clearer window into how the model arrives at its conclusions, or fails to.
Introducing LASS: The Language-Aware Semantic Score
One of the most exciting aspects of this benchmark is the introduction of a new metric: the Language-Aware Semantic Score (LASS). Why a new metric? Because traditional accuracy scores often don’t tell the whole story. An LLM might output a semantically correct answer but in the wrong language, or it might get the gist but miss crucial nuances. LASS was designed to reward:
- Semantic correctness: Is the answer truly accurate?
- Language adherence: Is the answer provided in the requested language?
- Nuance and context: Does the answer reflect a deep understanding of the query’s cultural or linguistic context?
LASS moves us beyond a simple pass/fail by incorporating a more nuanced evaluation of output quality, ensuring that models are assessed not just on what they say, but also on how and in what language they say it.
The Revealing Truths: What My Data Showed
After running 11 LLMs through this rigorous gauntlet, a few key findings emerged that paint a clearer picture of the current state of “multilingual” AI:
Scale Helps, But with Diminishing Returns
Unsurprisingly, larger models generally perform better. However, there’s a clear point of diminishing returns. Beyond a certain parameter count, simply adding more data or more layers doesn’t lead to proportionate gains in performance on these specific tasks. It suggests that sheer scale alone isn’t the silver bullet for truly robust multilingual capabilities.
Reasoning-Optimized Models Punch Above Their Weight
Perhaps the most compelling finding was that models specifically optimized for reasoning, even if they weren’t the largest in terms of parameters, frequently outperformed much larger models that lacked similar reasoning enhancements. This underscores the importance of architectural design and training methodologies focused on cognitive abilities over brute-force data ingestion. It’s not just about how much data an LLM sees, but how well it can process and reason with that data.
Open-Weight Models are Catching Up, But a Gap Remains
For the open-source community, there’s good news: the best open-weight model in my benchmark performed remarkably well, closing the gap significantly. It was, on average, only about 7% behind the best closed-source model. This indicates the incredible progress being made in the open-source AI space and offers hope for more accessible and customizable solutions for diverse linguistic communities.
The “Multilingual” Illusion on Cross-Lingual Tasks
This was arguably the most critical takeaway. Models touted as “multilingual” consistently underperformed on culturally specific cross-lingual tasks when the evaluations moved beyond simply translated English content. When models were asked to reason about concepts deeply embedded in, say, Amharic culture or to solve problems requiring an understanding of Basque social norms, their performance dipped significantly. This highlights that simply training on a wide range of languages doesn’t automatically confer deep cultural understanding or the ability to transfer knowledge effectively across diverse cultural contexts.
Navigating the Nuances: The Road Ahead for Truly Global AI
My benchmark isn’t just a set of numbers; it’s a mirror reflecting the current state of LLM development for the vast majority of the world’s languages. It tells us that while LLMs are indeed powerful, their “multilingual” capabilities often remain surface-level for mid- and low-resource languages, especially when cultural nuance and genuine cross-lingual reasoning are required.
The implications are clear: if we want to build truly inclusive AI that serves all humanity, we need to move beyond English-centric evaluations and invest heavily in native-language data, culturally sensitive training, and models explicitly designed for reasoning. This means supporting researchers and communities working with these languages, creating more robust native datasets, and developing evaluation metrics like LASS that can truly assess deep linguistic and cultural understanding.
The journey towards genuinely cross-lingual AI is long, but it’s a vital one. It’s about empowering billions, preserving linguistic diversity, and ensuring that the future of artificial intelligence is truly global, not just English-speaking with an accent. My code and data are openly available (see the GitHub link in the reproducibility section of my full paper), and I invite researchers and developers to build upon this work as we collectively strive for a more equitable AI landscape.
