Arabic text input in production: normalization and security

Arabic text input in production: normalization, length and security

Arabic text fields fail in quiet ways. A name looks identical on screen but misses a database lookup. A 30-character limit rejects one spelling and accepts another. A username passes the client check, then collides with a different account after normalization. Copy and paste adds an invisible direction mark that nobody can see in the support screenshot.

The fix is not an "Arabic regex." Treat text as several related representations: what the user typed, what the interface displays, what the product compares, and what a security-sensitive identifier is allowed to contain.

Store the original and derive comparison forms

Unicode can represent equivalent text with different code-point sequences. The Unicode Normalization Forms specification defines NFC and NFD for canonical equivalence, plus NFKC and NFKD for compatibility equivalence.

For ordinary names, addresses and messages, preserve the user's original text. Derive an NFC form for stable storage or comparison where the field contract allows it. Do not overwrite the original with aggressive cleanup.

NFKC needs more care. It removes compatibility distinctions, including some presentation forms and width variants. That can be useful for identifiers or search keys, but it can also erase a distinction that matters in a specialist field. Choose the form per field rather than applying one global middleware rule.

export function prepareArabicText(raw: string) {
  return {
    original: raw,
    canonical: raw.normalize("NFC"),
    comparisonKey: raw.normalize("NFKC"),
  };
}

A comparison key is derived data. Version the rule that produced it. If the rule changes, rebuild the key deliberately and check for collisions before adding a unique index.

Count what the user perceives

JavaScript's string.length counts UTF-16 code units. It does not reliably count code points, and neither measure is necessarily the number of characters a person sees.

Arabic letters can carry combining marks. Emoji can contain several code points. Unicode Annex #29 defines grapheme clusters as user-perceived characters. ECMA-402 exposes that model through Intl.Segmenter.

const segmenter = new Intl.Segmenter("ar", { granularity: "grapheme" });

export function graphemeLength(value: string) {
  return Array.from(segmenter.segment(value)).length;
}

Use grapheme counts for visible counters and product limits such as a display name. Keep byte and code-unit limits as separate backend safeguards for storage, indexes and downstream protocols.

The browser's maxlength is not a substitute for a product rule. The HTML Standard defines its own form-control behaviour, while pasted text, mobile clients and API calls still need server validation. Return a clear field error instead of truncating. Silent truncation can split a grapheme or create two values that appear to be the same.

Do not strip every invisible character

Some invisible characters are legitimate. Arabic and Persian shaping may depend on U+200C ZERO WIDTH NON-JOINER or U+200D ZERO WIDTH JOINER. Bidirectional controls can affect how mixed Arabic, English, numbers and punctuation are displayed.

The W3C Arabic Layout Requirements explains script joining, bidirectional text and punctuation behaviour. Use it as a rendering baseline, not just a typography reference.

Classify controls instead of deleting all of them:

• Preserve characters that the field genuinely needs.
• Reject unexpected bidi overrides in identifiers and machine-facing keys.
• Make invisible controls visible in an internal diagnostics view.
• Log code points and reason codes, not the user's complete private text.

For free-form content, safe rendering still depends on context-aware output encoding. The OWASP Input Validation Cheat Sheet explicitly warns that validation is not the main defence against XSS. A person's name may legitimately contain punctuation that a denylist considers suspicious.

Separate display names from identifiers

A display name should allow natural language. A username, tenant slug, coupon code or account-recovery identifier needs a narrower contract because it participates in equality, uniqueness or authorization.

Build that contract in stages:

1. Decode input strictly and reject malformed sequences.
2. Normalize with the field's documented form.
3. Apply a script and character policy appropriate to the identifier.
4. Derive the comparison key.
5. Check uniqueness on the comparison key inside the same transaction that creates the record.
6. Store the original form for display when it is safe to do so.

Unicode Technical Standard #39 documents confusable detection and mixed-script security. It is useful for risk signals, not an automatic ban on every mixed-script string. Arabic products routinely handle Latin brand names, model numbers and email addresses. A blanket single-script rule creates more support tickets than security.

For high-risk identifiers, show the normalized value back to the user before confirmation and flag unexpected script mixtures for review. Never use a display label as the authorization key.

Search keys need a different pipeline

Search often removes more distinctions than storage. A product may decide that common Arabic letter variants or diacritics should match for discovery. That is a search policy, not Unicode normalization itself.

Keep the layers explicit:

type TextRecord = {
  original: string;
  canonical: string;
  searchKey: string;
  ruleVersion: number;
};

Do not reuse searchKey for login, deduplication or legal records. Search can tolerate a broad match; account identity cannot.

The same rule applies to database collation. Test the actual collation used by production, including unique indexes. Application equality, database equality and search-engine analysis can disagree even when they receive the same string.

Build a fixture set from real failure classes

A good Arabic text test suite is not a list of translated English names. It covers representations and transitions:

• precomposed and decomposed canonical equivalents;
• letters with one and several combining marks;
• Arabic mixed with Latin text, digits and punctuation;
• Arabic-Indic and European digits where the field allows them;
• zero-width joiner and non-joiner;
• leading, trailing and repeated whitespace;
• bidi controls and copied rich-text fragments;
• emoji and multi-code-point graphemes;
• confusable and mixed-script identifiers;
• text at the grapheme, code-unit, byte and database-index limits.

Run each fixture through web, iOS, Android, API, queue, database and export paths. Assert the stored original, canonical form, comparison key, visible counter and rendered direction independently.

Ship one field contract at a time

There is no safe universal sanitizeText() function. A biography, an Arabic personal name, a username and an address do not share the same risk or meaning.

For every field, write down the allowed purpose, maximum graphemes, storage limit, normalization form, comparison rule, control-character policy, script policy and output contexts. Then enforce the same contract at the API boundary and test every client against it.

That turns Arabic text support from a collection of regex patches into a versioned part of the product model.

Sources: Unicode UAX #15, UAX #29 and UTS #39; W3C Arabic Layout Requirements; ECMA-402; WHATWG HTML; OWASP Input Validation Cheat Sheet. Graphic: SultanByte editorial artwork.

Cover credit: SultanByte editorial artwork based on Unicode, W3C, ECMA-402, WHATWG and OWASP specifications.