# How to Evaluate Arabic LLMs Before Production in MENA

Choosing an Arabic large language model from a leaderboard is easy. Deciding whether it can safely handle customer messages, internal documents, or public services across MENA is much harder. A benchmark score can establish a useful baseline, but it cannot reproduce the language mix, failure costs, traffic patterns, and operational constraints of a real product.

A production evaluation should therefore answer a narrower question than “Which model is best?” It should ask: “Which model, prompt, retrieval setup, and review process meets the requirements of this use case, for these users, at an acceptable cost and speed?”

*Cover visual: original SultanByte editorial artwork.*

## Start with ArabicMMLU, but understand its boundary

[ArabicMMLU](https://arxiv.org/html/2402.12840v2) is a valuable knowledge benchmark. It contains 14,575 multiple-choice questions across 40 tasks, all in Modern Standard Arabic (MSA), and its authors evaluated 35 models. The subjects range from school-level material to professional topics, so the benchmark can expose gaps that are easy to miss in English-first evaluations.

It is not, however, a simulation of an Arabic production system. The paper explicitly identifies two important limitations: dialectical Arabic is excluded, and the benchmark is text-only. ArabicMMLU can indicate how well a model answers structured knowledge questions in MSA. It does not tell a bank whether the model understands a Saudi customer’s colloquial complaint, whether a retailer’s chatbot can interpret Egyptian Arabizi, or whether a voice assistant works over a noisy telephone line.

A model can score well on clean multiple-choice questions and still fail when it must follow a response template, cite a retrieved policy, refuse a harmful request, or respond within a latency budget. Treat ArabicMMLU as one lane in an evaluation suite, not as a procurement decision.

## Build the language test around actual users

Arabic evaluation should reflect the language people use with the product, not an abstract idea of “Arabic support.” At minimum, build separate slices for MSA and every target dialect. A service launching in the UAE may need Emirati or broader Gulf Arabic, plus language from expatriate communities. A regional support product may need Egyptian, Levantine, Saudi, and Maghrebi varieties, each weighted by expected usage.

Useful public resources can help construct these slices. [MADAR](https://aclanthology.org/L18-1535/) provides an Arabic dialect corpus and lexicon designed to support work across dialects. [CAMeL Tools](https://aclanthology.org/2020.lrec-1.868/) offers open-source components for Arabic preprocessing, morphology, dialect identification, named-entity recognition, and sentiment analysis. These are useful building blocks for dataset preparation and diagnostics, although neither replaces examples drawn from the product domain.

The test set should also contain code-switching between Arabic and English, or Arabic and French where relevant. Include Arabizi, where Arabic is written with Latin characters and numerals, as well as spelling variation, missing diacritics, local abbreviations, typos, emoji, and right-to-left text mixed with account numbers or URLs. Preserve these features instead of normalising every example into formal Arabic. The variation is part of the task.

[LAraBench](https://arxiv.org/html/2305.14982v2) shows why breadth matters. It spans 33 tasks, 61 public datasets, 98 evaluation setups, roughly 296,000 data points, and 46 hours of speech. Its coverage includes MSA and dialectal Arabic from news, tweets, meetings, telephony, and broadcast material. The study also found that state-of-the-art task-specific systems often outperformed zero-shot LLMs, while three-shot prompting improved many tasks. The practical lesson is not that one architecture always wins. It is that task, domain, input format, and prompting can materially change the result.

![Four-stage Arabic LLM production evaluation workflow: define the target, map ArabicMMLU and LAraBench evidence, build product tests, and set release gates.](https://cdn.hashnode.com/uploads/covers/60ecf4a0fc37a15ec15655e8/e1ff1433-3f3a-47e2-8877-f294eecf570d.png)
*Sources and reporting periods: ArabicMMLU (2024) and LAraBench (2024 revision). Original infographic: SultanByte editorial, 2026.*

## Test product behaviour, not only language knowledge

Once the language slices are defined, evaluate the complete behaviour expected from the application.

Formatting deserves its own tests. If an API must return valid JSON, measure schema validity, required-field completion, enum compliance, and whether Arabic strings preserve direction and punctuation. For customer-facing text, inspect headings, lists, dates, currencies, phone numbers, Arabic and Western numerals, and mixed-direction rendering. A linguistically correct answer can still break a workflow if it adds commentary outside the required schema.

Safety testing should use realistic adversarial prompts in MSA, dialects, code-switched language, and Arabizi. Test direct harmful requests, indirect requests, role-play, obfuscation, and multi-turn attempts to bypass policy. Also measure false refusals. A model that blocks ordinary health, finance, or civic questions may be unusable even if it catches risky prompts. The [Jais model paper](https://arxiv.org/html/2308.16149v2) advises adopters to conduct additional safety testing, not assume equal proficiency across dialects, and apply oversight rather than relying on the model in sensitive or high-stakes uses. That is a sensible standard for any model under consideration, not only Jais.

Retrieval-augmented generation needs end-to-end evaluation. Test whether the retriever finds the correct Arabic passage, whether the model answers from that passage, whether citations point to the supporting text, and whether it declines to invent an answer when evidence is absent. Include Arabic document structures, OCR noise, transliterated names, and queries whose terminology differs from the source. Report retrieval recall separately from answer quality so a weak retriever is not mistaken for a weak generator.

## Define the matrix before comparing models

A compact evaluation specification keeps teams from changing the rules after seeing the scores. The weights below are illustrative. A support assistant may place more weight on dialect coverage and latency, while an internal research tool may emphasise retrieval and citation quality.

```yaml
production_evaluation:
  language: {weight: 25, slices: [MSA, target_dialects, code_switching, Arabizi]}
  task_quality: {weight: 20, metrics: [accuracy, completeness, formatting]}
  retrieval: {weight: 15, metrics: [recall_at_k, groundedness, citation_accuracy]}
  safety: {weight: 15, metrics: [unsafe_compliance, false_refusal, policy_consistency]}
  human_review: {weight: 10, metrics: [fluency, local_fit, severity_weighted_errors]}
  operations: {weight: 15, metrics: [p50_latency, p95_latency, cost_per_completed_task]}
release_gates:
  critical_safety_failures: 0
  valid_schema_rate: ">= product threshold"
  dialect_slice_minimum: ">= product threshold"
```

Use hard gates for requirements that averages can hide. A high overall language score should not compensate for severe safety failures, consistently poor performance in one launch market, or invalid output that crashes a downstream service. Record results by model version, system prompt, sampling settings, retrieval configuration, and date. Model names alone are not reproducible configurations.

The open-source [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) provides a unified way to run public evaluations, define custom tasks and metrics, and test local or API-backed models. It can help automate repeatable benchmark runs, but production teams still need custom datasets, system-level load tests, and human review.

## Measure latency and cost as user outcomes

Provider pricing and tokens per second are useful inputs, but neither is the final business metric. Measure latency from the application boundary, including retrieval, prompt assembly, queueing, generation, moderation, and any retry. Track median latency and tail latency, especially p95, under expected concurrency. Run long Arabic prompts and outputs because tokenisation and response length can affect both speed and cost.

Cost should be calculated per completed task, not only per million tokens. Include input and output tokens, embeddings, reranking, safety calls, retries, failed requests, hosting, and human escalation. A cheaper model that requires repeated prompts or frequent manual correction may cost more per resolved support case. Conversely, a smaller model may be the better choice if retrieval supplies the needed facts and the task is tightly constrained.

Run load tests with the actual orchestration stack. Capture timeouts, rate limits, malformed outputs, and fallback behaviour. If the product has a latency service-level objective, define the allowed quality trade-off before launch. For example, the system might route simple classification to a smaller model while reserving a stronger model for complex Arabic generation.

## Put qualified human reviewers in the loop

Automated metrics are necessary for regression testing, but they miss pragmatic and cultural errors. Recruit reviewers who understand the target dialect and the product domain. A fluent MSA reviewer should not be expected to judge every regional expression, and linguistic fluency alone is insufficient for specialist tasks.

Use a short rubric with observable criteria: factual correctness, instruction following, fluency, dialect fit, tone, groundedness, formatting, and safety. Ask reviewers to label error severity rather than provide only a preference. A mistranslated greeting and a fabricated eligibility requirement should not count equally. For pairwise comparisons, randomise model order and hide provider identity where practical.

Review disagreements are useful data. They may reveal an unclear policy, a contested local expression, or a prompt that permits several valid answers. Resolve those cases through adjudication and update the rubric. Keep a protected holdout set so prompt and model changes are not repeatedly tuned against the same examples.

Human review should continue after release. Sample live interactions with privacy controls, track escalations and user corrections, and convert confirmed failures into regression cases. Monitor results by language slice and task rather than relying on a single satisfaction average.

## A practical path from shortlist to launch

First, turn the product specification into measurable release gates. Identify target countries, dialects, channels, document types, risk levels, peak traffic, response-time targets, and escalation rules.

Second, create a representative dataset. Combine licensed public resources with carefully governed product examples. Remove duplicates and prevent train-test contamination where it can be identified. Separate development, validation, and holdout sets, then tag every item by dialect, task, difficulty, risk, and input type.

Third, establish simple baselines. Compare candidate LLMs with the current workflow, a smaller model, and task-specific systems where available. LAraBench is a useful reminder that a general LLM should not be assumed to beat a specialised system.

Fourth, evaluate the full stack. Run the exact system prompt, tools, retrieval pipeline, safety controls, and output parser intended for production. Repeat runs where sampling introduces variance. Then conduct human review on both random samples and high-risk slices.

Finally, pilot with limited traffic and clear rollback criteria. Watch quality, safety, retrieval, latency, cost, and escalation rates together. Re-run the suite whenever the model, prompt, knowledge base, moderation policy, or orchestration layer changes.

## The production decision is a set of thresholds

There is no single “best Arabic LLM” for MENA. There is a model and system configuration that either clears or misses the thresholds for a defined product. ArabicMMLU can test MSA knowledge, but it excludes dialectical Arabic and remains text-only. Production readiness requires evidence across MSA, target dialects, code-switching, Arabizi, formatting, safety, retrieval, latency, cost, and qualified human review.

Write the launch decision in specific terms: name the passing configuration, the markets and tasks it covers, the slices where it remains weak, the conditions that trigger human escalation, and the metrics that will stop or roll back deployment. That record is more useful than any leaderboard position because it connects model performance to the users, risks, and economics of the actual service.