# Building Arabic Search That Users Can Trust

Arabic search often fails in ways that English-language test data will not reveal. A customer types `تأمين` and the catalogue contains the same word with diacritics. A merchant name arrives with an alef variant. A support article uses Modern Standard Arabic, while the query is Gulf or Egyptian dialect. The search engine returns nothing, or produces a long list of loosely related results.

The fix is not a single Arabic tokenizer. Reliable search needs separate treatment for character normalization, morphology, typing errors, names and semantic intent. This guide lays out a practical architecture for product teams serving Arabic-speaking users across MENA.

*Cover: original SultanByte technical illustration. Arabic text in the visual reads "health insurance".*

## Keep the original text and build search-specific fields

Never replace the text that users or content teams entered. Store it as the display value, then derive one or more fields for retrieval. That lets you improve the search pipeline without corrupting names, legal text or carefully edited Arabic.

A useful document shape is:

```json
{
  "title": "تأمين صحي للعائلة",
  "title_exact": "تأمين صحي للعائلة",
  "title_ar": "تامين صحي للعائله",
  "title_trigram": "تامين صحي للعائله"
}
```

The exact field supports identifiers, quoted text and heavily weighted phrase matches. The Arabic field can remove diacritics and apply conservative letter folding. The trigram field catches limited spelling variation. If the product also uses embeddings, keep the vector beside these lexical fields rather than treating it as a replacement.

This separation matters most for names. Mapping `ة` to `ه`, or `ى` to `ي`, may improve recall but also merges spellings that are not interchangeable in every context. A normalized key is useful for candidate retrieval. The original form should still drive display, auditing and final ranking.

## Normalize characters before analysing words

Unicode can represent visually equivalent text with different code-point sequences. [Unicode Standard Annex #15](https://www.unicode.org/reports/tr15/) defines normalization forms so equivalent strings can share a consistent binary representation. NFC is a safe baseline for stored text. A search-only field may use NFKC when compatibility characters, including Arabic presentation forms, need folding, but test it against identifiers and mixed-script data before adopting it.

Arabic search normalization commonly adds four operations:

1. remove tatweel (`ـ`);
2. remove optional vowel marks and Quranic annotation marks where the corpus does not depend on them;
3. fold alef variants such as `أ`, `إ` and `آ` to `ا`;
4. optionally fold `ى` to `ي` and `ة` to `ه` in a recall-oriented field.

[Lucene's Arabic normalizer](https://lucene.apache.org/core/9_12_1/analysis/common/org/apache/lucene/analysis/ar/ArabicNormalizer.html) documents a concrete implementation: it folds several alef forms, dotless yeh, teh marbuta and Persian heh/yeh variants, and removes tatweel and harakat. [Elasticsearch's built-in Arabic analyser](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#arabic-analyzer) combines character normalization, lowercasing, stop-word removal and light stemming.

That is a good starting point, not a universal specification. Qur'anic search, Arabic-language learning, legal archives and name matching can all require different rules. Make normalization a versioned function and retain the original field so that a future change does not require reconstructing source data.

A small JavaScript implementation for a general catalogue might look like this:

```js
const ARABIC_MARKS = /[\u0610-\u061A\u064B-\u065F\u0670\u06D6-\u06ED]/gu;

export function arabicSearchKey(input) {
  return input
    .normalize("NFKC")
    .replace(ARABIC_MARKS, "")
    .replace(/ـ/gu, "")
    .replace(/[إأآٱ]/gu, "ا")
    .replace(/ى/gu, "ي")
    .replace(/ة/gu, "ه")
    .replace(/\s+/gu, " ")
    .trim();
}
```

Keep punctuation policy outside this function. Product codes, model numbers, email addresses and Arabic text mixed with English can be damaged by blanket punctuation removal.

## Tokenization is not segmentation

Splitting on spaces is only the first step. Arabic attaches conjunctions, prepositions, articles and pronouns to words. A query may contain `وبالسيارة`, while the indexed text contains `سيارة`. A search engine that treats the surface form as one indivisible token misses the connection.

[Unicode Standard Annex #29](https://www.unicode.org/reports/tr29/) defines default word boundaries, but it also notes that language-specific behaviour may require tailoring. Arabic retrieval usually needs a morphological layer on top of Unicode-safe boundary detection.

There are two practical levels:

- **Light stemming** removes common prefixes and suffixes without trying to recover a dictionary root. Lucene's [Arabic stemmer](https://lucene.apache.org/core/9_12_1/analysis/common/org/apache/lucene/analysis/ar/ArabicStemmer.html) follows this approach, and Elasticsearch exposes it in the standard Arabic analysis chain.
- **Morphological segmentation** separates clitics and analyses richer word structure. [CAMeL Tools](https://camel-tools.readthedocs.io/en/latest/) provides open-source Arabic NLP components, while its [LREC 2020 paper](https://aclanthology.org/2020.lrec-1.868/) describes tools for preprocessing, morphology, dialect identification, named-entity recognition and sentiment. [Farasa](https://aclanthology.org/L16-1170/) is another established Arabic segmenter; its paper describes splitting words into constituent clitics and reports substantially faster processing than the systems it compared against in 2016.

Use light stemming for broad product and content search first. Add full segmentation when clitic-heavy queries, dialect processing or downstream NLP justify the operational cost. Root stemming is often too aggressive for brands, people and specialist terms.

## Use different retrieval paths for different fields

One Arabic analyser applied to every field is a design smell. Names, product descriptions, addresses and help-centre articles behave differently.

| Field | Primary treatment | Useful fallback | Avoid |
|---|---|---|---|
| Person or company name | NFC, conservative folding, phrase boost | Trigram candidates | Aggressive stemming |
| Product title | Arabic normalization, light stemming | Trigram and synonyms | Vector-only ranking |
| SKU or policy number | Exact keyword field | Prefix match | Diacritic or punctuation stripping |
| Address | Normalized Arabic plus Latin alias | Local abbreviation dictionary | Assuming one regional format |
| Long support content | Light stemming and BM25 | Hybrid semantic retrieval | Returning chunks without article context |
| User-generated listings | Conservative normalization | Typo tolerance with a threshold | Unbounded fuzzy matching |

PostgreSQL's [`pg_trgm`](https://www.postgresql.org/docs/current/pgtrgm.html) measures text similarity from shared trigrams and provides GiST and GIN index support. It can be a practical typo-tolerance layer for smaller systems. Trigrams are character based, so apply the same Arabic search-key function to both the indexed value and the query. Set thresholds from real relevance judgments; a low global threshold can turn short Arabic queries into noise.

For Elasticsearch or OpenSearch, create separate exact, analysed and n-gram subfields. Run the original and normalized query against them with different boosts, then inspect which clause produced each match. Explainability saves time when a merchandising or compliance team asks why a result ranked first.

## Dialect and code-switching need their own tests

A model or analyser trained on Modern Standard Arabic does not automatically cover Gulf, Egyptian, Levantine and Maghrebi usage equally. The differences include vocabulary, spelling, transliteration and how Arabic mixes with English or French. Country-aware synonym sets can help, but they should come from observed queries rather than a generic regional dictionary.

Semantic models can recover some intent that lexical search misses. [AraBERT](https://aclanthology.org/2020.osact-1.2/) showed how Arabic-specific pretraining and preprocessing improved performance across several Arabic NLP tasks. That finding supports language-aware model selection, but it does not prove that any Arabic embedding model will improve a particular search product.

Evaluate semantic retrieval on your own query-document pairs. Include dialect, Arabizi, mixed Arabic and Latin text, misspellings, short queries, names and safety-sensitive queries. Keep lexical ranking in the path for exact terms, regulated product names and identifiers. Hybrid search is useful when the semantic layer adds candidates and the lexical layer preserves precision.

## Build a relevance set before tuning

Offline evaluation does not need to start with thousands of examples. Take a few hundred anonymised queries from each target market, remove sensitive data, and have people familiar with the domain grade the results. Do not pool Saudi, UAE, Egyptian and Moroccan traffic into one score if the vocabulary and catalogue differ.

Track metrics by query class:

- exact names and identifiers;
- normalized spelling variants;
- clitic and morphological variants;
- dialect and code-switched queries;
- typos;
- queries with no relevant result.

Measure recall at a candidate depth and ranking quality near the top. Also track zero-result rate, reformulation rate and clicks on low-ranked results in production. A lower zero-result rate is not automatically better; broad fuzzy matching can hide failure by returning irrelevant items.

Every normalization, synonym or model change should run against a fixed regression set. Include counterexamples where over-normalization produces a false match. For financial services, healthcare and government products, add human review for queries whose wrong answer carries a higher cost.

## A practical rollout sequence

Start with raw text preservation, NFC and a separate normalized Arabic field. Add exact-field boosts and light stemming. Then introduce trigram candidates for misspellings, with stricter thresholds for short terms and names.

The next investment should follow the error log. If attached clitics dominate failures, test segmentation. If users phrase needs rather than product names, add a hybrid semantic path. If Maghrebi or Gulf vocabulary fails, build market-specific evaluation sets and synonyms. Do not add a larger model to compensate for broken Unicode handling or missing exact matches.

Arabic search quality comes from controlled layers, not one clever component. Preserve the source, normalize for retrieval, keep field-specific analysers, and test every change against the Arabic people actually type in each market. That architecture is easier to debug and much harder to fool with a polished demo query.