Search for a command to run...
A practical architecture for normalization, morphology, typo tolerance and hybrid retrieval across Arabic-speaking markets.
No comments yet. Be the first to comment.
DataVolt says construction could begin within months, but power, financing and customers remain unresolved.
IBM’s Saudi and UAE sample shows that lost business costs far more than breach notification. Recovery and identity still matter most.
The fintech sector in the Middle East has experienced remarkable growth, driven by technological innovation, supportive regulatory frameworks, and substantial investments. This expansion is reshaping the region's financial landscape, fostering financ...
Vercel Ship 2024, the annual celebration of the frontend cloud, took place in New York City on May 23rd, 2024. The event brought together nearly 1,000 developers and tech enthusiasts to explore the latest advancements in frontend development, AI, and...
Sultan Byte
10 posts
Articles addressing day-to-day development challenges
On this page
Arabic search often fails in ways that English-language test data will not reveal. A customer types تأمين and the catalogue contains the same word with diacritics. A merchant name arrives with an alef variant. A support article uses Modern Standard Arabic, while the query is Gulf or Egyptian dialect. The search engine returns nothing, or produces a long list of loosely related results.
The fix is not a single Arabic tokenizer. Reliable search needs separate treatment for character normalization, morphology, typing errors, names and semantic intent. This guide lays out a practical architecture for product teams serving Arabic-speaking users across MENA.
Cover: original SultanByte technical illustration. Arabic text in the visual reads "health insurance".
Never replace the text that users or content teams entered. Store it as the display value, then derive one or more fields for retrieval. That lets you improve the search pipeline without corrupting names, legal text or carefully edited Arabic.
A useful document shape is:
{
"title": "تأمين صحي للعائلة",
"title_exact": "تأمين صحي للعائلة",
"title_ar": "تامين صحي للعائله",
"title_trigram": "تامين صحي للعائله"
}
The exact field supports identifiers, quoted text and heavily weighted phrase matches. The Arabic field can remove diacritics and apply conservative letter folding. The trigram field catches limited spelling variation. If the product also uses embeddings, keep the vector beside these lexical fields rather than treating it as a replacement.
This separation matters most for names. Mapping ة to ه, or ى to ي, may improve recall but also merges spellings that are not interchangeable in every context. A normalized key is useful for candidate retrieval. The original form should still drive display, auditing and final ranking.
Unicode can represent visually equivalent text with different code-point sequences. Unicode Standard Annex #15 defines normalization forms so equivalent strings can share a consistent binary representation. NFC is a safe baseline for stored text. A search-only field may use NFKC when compatibility characters, including Arabic presentation forms, need folding, but test it against identifiers and mixed-script data before adopting it.
Arabic search normalization commonly adds four operations:
ـ);أ, إ and آ to ا;ى to ي and ة to ه in a recall-oriented field.Lucene's Arabic normalizer documents a concrete implementation: it folds several alef forms, dotless yeh, teh marbuta and Persian heh/yeh variants, and removes tatweel and harakat. Elasticsearch's built-in Arabic analyser combines character normalization, lowercasing, stop-word removal and light stemming.
That is a good starting point, not a universal specification. Qur'anic search, Arabic-language learning, legal archives and name matching can all require different rules. Make normalization a versioned function and retain the original field so that a future change does not require reconstructing source data.
A small JavaScript implementation for a general catalogue might look like this:
const ARABIC_MARKS = /[\u0610-\u061A\u064B-\u065F\u0670\u06D6-\u06ED]/gu;
export function arabicSearchKey(input) {
return input
.normalize("NFKC")
.replace(ARABIC_MARKS, "")
.replace(/ـ/gu, "")
.replace(/[إأآٱ]/gu, "ا")
.replace(/ى/gu, "ي")
.replace(/ة/gu, "ه")
.replace(/\s+/gu, " ")
.trim();
}
Keep punctuation policy outside this function. Product codes, model numbers, email addresses and Arabic text mixed with English can be damaged by blanket punctuation removal.
Splitting on spaces is only the first step. Arabic attaches conjunctions, prepositions, articles and pronouns to words. A query may contain وبالسيارة, while the indexed text contains سيارة. A search engine that treats the surface form as one indivisible token misses the connection.
Unicode Standard Annex #29 defines default word boundaries, but it also notes that language-specific behaviour may require tailoring. Arabic retrieval usually needs a morphological layer on top of Unicode-safe boundary detection.
There are two practical levels:
Use light stemming for broad product and content search first. Add full segmentation when clitic-heavy queries, dialect processing or downstream NLP justify the operational cost. Root stemming is often too aggressive for brands, people and specialist terms.
One Arabic analyser applied to every field is a design smell. Names, product descriptions, addresses and help-centre articles behave differently.
| Field | Primary treatment | Useful fallback | Avoid |
|---|---|---|---|
| Person or company name | NFC, conservative folding, phrase boost | Trigram candidates | Aggressive stemming |
| Product title | Arabic normalization, light stemming | Trigram and synonyms | Vector-only ranking |
| SKU or policy number | Exact keyword field | Prefix match | Diacritic or punctuation stripping |
| Address | Normalized Arabic plus Latin alias | Local abbreviation dictionary | Assuming one regional format |
| Long support content | Light stemming and BM25 | Hybrid semantic retrieval | Returning chunks without article context |
| User-generated listings | Conservative normalization | Typo tolerance with a threshold | Unbounded fuzzy matching |
PostgreSQL's pg_trgm measures text similarity from shared trigrams and provides GiST and GIN index support. It can be a practical typo-tolerance layer for smaller systems. Trigrams are character based, so apply the same Arabic search-key function to both the indexed value and the query. Set thresholds from real relevance judgments; a low global threshold can turn short Arabic queries into noise.
For Elasticsearch or OpenSearch, create separate exact, analysed and n-gram subfields. Run the original and normalized query against them with different boosts, then inspect which clause produced each match. Explainability saves time when a merchandising or compliance team asks why a result ranked first.
A model or analyser trained on Modern Standard Arabic does not automatically cover Gulf, Egyptian, Levantine and Maghrebi usage equally. The differences include vocabulary, spelling, transliteration and how Arabic mixes with English or French. Country-aware synonym sets can help, but they should come from observed queries rather than a generic regional dictionary.
Semantic models can recover some intent that lexical search misses. AraBERT showed how Arabic-specific pretraining and preprocessing improved performance across several Arabic NLP tasks. That finding supports language-aware model selection, but it does not prove that any Arabic embedding model will improve a particular search product.
Evaluate semantic retrieval on your own query-document pairs. Include dialect, Arabizi, mixed Arabic and Latin text, misspellings, short queries, names and safety-sensitive queries. Keep lexical ranking in the path for exact terms, regulated product names and identifiers. Hybrid search is useful when the semantic layer adds candidates and the lexical layer preserves precision.
Offline evaluation does not need to start with thousands of examples. Take a few hundred anonymised queries from each target market, remove sensitive data, and have people familiar with the domain grade the results. Do not pool Saudi, UAE, Egyptian and Moroccan traffic into one score if the vocabulary and catalogue differ.
Track metrics by query class:
Measure recall at a candidate depth and ranking quality near the top. Also track zero-result rate, reformulation rate and clicks on low-ranked results in production. A lower zero-result rate is not automatically better; broad fuzzy matching can hide failure by returning irrelevant items.
Every normalization, synonym or model change should run against a fixed regression set. Include counterexamples where over-normalization produces a false match. For financial services, healthcare and government products, add human review for queries whose wrong answer carries a higher cost.
Start with raw text preservation, NFC and a separate normalized Arabic field. Add exact-field boosts and light stemming. Then introduce trigram candidates for misspellings, with stricter thresholds for short terms and names.
The next investment should follow the error log. If attached clitics dominate failures, test segmentation. If users phrase needs rather than product names, add a hybrid semantic path. If Maghrebi or Gulf vocabulary fails, build market-specific evaluation sets and synonyms. Do not add a larger model to compensate for broken Unicode handling or missing exact matches.
Arabic search quality comes from controlled layers, not one clever component. Preserve the source, normalize for retrieval, keep field-specific analysers, and test every change against the Arabic people actually type in each market. That architecture is easier to debug and much harder to fool with a polished demo query.