screening system checks a reputation in opposition to a watchlist, it faces a silent failure mode that no person talks about. Sort “Владимир Путин” right into a system listed on “Vladimir Putin” and most name-matching approaches return nothing. The 2 strings share zero characters, so edit distance is meaningless, phonetic codes fail (they assume Latin), and BM25 provides up solely.
This isn’t an obscure edge case. Immigration databases, hospital file techniques, and monetary compliance pipelines cope with this every day. And but, the dominant approaches to this drawback are both classical (edit distance, Soundex variants) or heavyweight (fine-tune a multilingual LLM on just a few hundred manually labeled pairs). On this put up, I’ll stroll you thru how we educated a compact transformer encoder from scratch on uncooked UTF-8 bytes, with no tokenizer, no pretrained spine, and no script detection, to unravel cross-script phonetic title retrieval. We achieved 0.775 MRR and 0.897 R@10 throughout 8 non-Latin scripts, decreasing the efficiency hole between Latin and non-Latin queries by 10x over the perfect classical baseline.
The complete code is on GitHub. This put up covers the concepts and the engineering.
Why is this difficult?
The issue sits on the intersection of three issues that don’t cooperate:
Scripts are disjoint image units. “Schwarzenegger” and “שוורצנגר” (Hebrew) haven’t any shared characters. Edit distance, the go-to for fuzzy matching, produces a maximum-distance rating each time a script boundary is crossed. Phonetic hashing (Double Metaphone, Soundex) encodes approximate English pronunciation, so it’s ineffective for non-Latin queries by design.
Romanization isn’t a perform. The Chinese language title written as “张” maps to Zhang, Chang, and Cheung relying on dialect, romanization commonplace, and historic conference. The Korean “박” maps to Park, Pak, and Bak. Any method that tries to normalize to a canonical Latin type (like ICU transliterate) will get the best reply for one conference and fail for the others.
Names carry no semantic context. Dense retrieval strategies like DPR and BGE-M3 are highly effective for sentence-level duties as a result of surrounding phrases present semantic grounding. For a 2-word individual title there isn’t a context to compensate for floor mismatch. Chari et al. (2025) confirmed that even robust multilingual retrievers degrade severely when queries are transliterated fairly than written of their native script.
The perception behind our method: each Unicode character decomposes deterministically into 1 to 4 bytes from a hard and fast 256-symbol alphabet. “Владимир” and “Vladimir” are totally different byte sequences, however a mannequin educated contrastively on sufficient phonetic pairs can study to map them to close by vectors. The vocabulary is common by development.
Constructing Coaching Information at Scale
You’ll be able to’t prepare this mannequin with out knowledge, and there’s no dataset of 4 million cross-script phonetic title pairs mendacity round. We constructed one with a 4-stage LLM pipeline.
Stage 1: Stratified sampling from Wikidata
We began with 2 million person-name entities from Wikidata, which gives canonical English names plus partial cross-script labels (some entities have Russian or Arabic names of their Wikidata file, most don’t). Naively sampling from this produces a dataset dominated by English-only names. We stratified by script-coverage bucket (0, 1-2, 3-4, 5+ non-English labels) and sampled proportionally inside every bucket, yielding 119,040 entities with balanced protection.
Stage 2: Phonetic Latin variants (Llama-3.1-8B)
For every English anchor title, we requested Llama-3.1-8B-Instruct to generate 4 phonetic spelling variants — the sorts of mishearings and misspellings actual folks produce. The immediate was strict:
Generate 4 DISTINCT phonetic spelling variants of this title
because it sounds when spoken: "Catherine"
Guidelines:
- Every variant should be spelled in a different way from all others and from the unique
- Simulate how totally different folks may mishear or misspell the title phonetically
- Do NOT use nicknames, abbreviations, or shortened types
- Do NOT change language (keep in Latin script)
Return a JSON array of precisely 4 strings, no clarification:
["variant1", "variant2", ...]End result for “Catherine”: ["Kathryn", "Katerin", "Kathrin", "Katharine"]
Stage 3: Cross-script transliteration (Qwen3-30B)
For every English title and every of its Latin variants, we generated transliterations into 8 scripts: Arabic, Russian, Chinese language, Japanese, Hebrew, Hindi, Greek, Korean. We used Qwen3-Coder-30B-A3B-Instruct-FP8:
{
"Catherine": {"ar": "كاثرين", "ru": "Катрин", "he": "קתרין", ...},
"Kathryn": {"ar": "كاثرين", "ru": "Катрин", ...},
"Katharine": {"ar": "...", "ru": "...", ...}
}Each stage is independently resumable: it reads present output, builds a set of already-processed entity IDs, and skips them. A crash loses at most one in-flight batch.
Stage 4: Merge and tag
The ultimate stage merges Wikidata ground-truth labels with LLM output, deduplicates, and tags every constructive pair by sort:
- phonetic: Latin spelling variant of the English anchor (“Catherine” → “Kathryn”)
- script: direct transliteration right into a non-Latin script (“Catherine” → “كاثرين”)
- mixed: a phonetic Latin variant that was then transliterated (“Katharine” → “كاثرين”)
Positives are saved per entity; negatives should not saved in any respect, they’re mined dynamically throughout coaching. Splits are assigned on the entity stage (80/10/10, deterministic MD5 hash of entity ID) so all variants of an identification go to at least one partition.
Remaining dataset: 119,040 entities, 4.67 million constructive pairs.
The Mannequin
The encoder is genuinely small: 6 transformer layers, 8 consideration heads, hidden dim 256, FFN dim 1024, dropout 0.1, max size 256 bytes. Complete parameters: ~4M.
class ByteLevelEncoder(PreTrainedModel):
def __init__(self, config: ByteEncoderConfig):
tremendous().__init__(config)
self.embedding = nn.Embedding(
config.vocab_size, # 256 — uncooked UTF-8 bytes
config.hidden_dim,
padding_idx=config.pad_token_id,
)
self.pos_embedding = nn.Embedding(config.max_len, config.hidden_dim)
encoder_layer = nn.TransformerEncoderLayer(
d_model=config.hidden_dim,
nhead=config.n_heads,
dim_feedforward=config.ffn_dim,
dropout=config.dropout,
batch_first=True,
norm_first=True, # pre-norm: extra secure when coaching from scratch
)
self.transformer = nn.TransformerEncoder(
encoder_layer, num_layers=config.n_layers,
enable_nested_tensor=False,
)
def ahead(self, input_ids, attention_mask):
B, L = input_ids.form
positions = torch.arange(L, system=input_ids.system).unsqueeze(0)
x = self.embedding(input_ids) + self.pos_embedding(positions)
padding_mask = ~attention_mask # TransformerEncoder makes use of True = ignore
x = self.transformer(x, src_key_padding_mask=padding_mask)
# imply pool over actual tokens solely
mask_f = attention_mask.unsqueeze(-1).float()
pooled = (x * mask_f).sum(dim=1) / mask_f.sum(dim=1).clamp(min=1)
return F.normalize(pooled, p=2, dim=-1) # unit vectorsWhy pre-norm (norm_first=True)? When coaching a transformer from scratch (no pretrained initialization), pre-norm stabilizes gradient move in early coaching. Publish-norm tends to diverge except you might be cautious with studying charge warmup and initialization. For a fine-tuning state of affairs, you most likely don’t want to consider this, however right here it mattered.
The output is a unit vector in 256 dimensions. Cosine similarity = inside product on unit vectors, so retrieval is only a dot product.
Coaching: InfoNCE and Exhausting Detrimental Mining
The InfoNCE loss
The loss is commonplace: an (anchor, constructive) pair ought to have a excessive inside product; the anchor’s inside product with each different constructive within the batch (the in-batch negatives) ought to be low.
def infonce_loss(anchor, constructive, temperature=0.07):
# anchor, constructive: (B, D), L2-normalized
logits = (anchor @ constructive.T) / temperature # (B, B)
labels = torch.arange(len(anchor), system=anchor.system) # diagonal = appropriate
return F.cross_entropy(logits, labels)With batch dimension 256 and temperature 0.07, that is 255 negatives per anchor per step. The temperature controls how peaked the distribution is: too excessive and the loss ignores arduous negatives, too low and coaching turns into unstable.
Why in-batch negatives aren’t sufficient
In-batch negatives are low-cost however shallow: they’re random names from the dataset, which are typically straightforward to separate. A mannequin that has been coaching for just a few hundred steps can distinguish “Catherine” from “Zhao Wei” effortlessly. What it struggles with is “Katarina” vs “Katherine” — names which might be phonetically shut however check with totally different folks. These are the instances the place the gradient sign is definitely informative.
That is the motivation for ANCE (Approximate Nearest Neighbour Contrastive Estimation): periodically rebuild a FAISS index from the present mannequin’s embeddings, then for every anchor, discover the present nearest non-matching neighbors and use these as negatives. They’re arduous exactly as a result of the mannequin presently thinks they’re comparable.
The arduous detrimental schedule
class ANCEBatchSampler(Sampler):
def _current_mix_ratio(self) -> float:
if self._step < self.warmup or self.index is None:
return 0.0
steps_past_warmup = self._step - self.warmup
# ramp from 0 → target_mix_ratio over mix_ramp_steps
return min(
self.target_mix_ratio,
self.target_mix_ratio * steps_past_warmup / max(1, self.mix_ramp_steps)
)Through the first 200 steps: random batches solely. The mannequin has no significant construction but; a FAISS index over random embeddings would produce ineffective arduous negatives.
After step 200: the FAISS index is rebuilt periodically from contemporary embeddings (each refresh_every steps). Every batch is constructed by taking a seed anchor, discovering its nearest neighbors within the present index, filling n_hard = batch_size * mix_ratio slots with these neighbors, and padding the remainder with random samples. The combo ratio ramps linearly from 0 to 0.7 over 500 steps after warmup, so the transition is gradual.
The coaching loop:
for batch in train_loader:
anchor = mannequin(batch["anchor"].to(system), batch["anchor_mask"].to(system))
constructive = mannequin(batch["positive"].to(system), batch["positive_mask"].to(system))
loss = loss_fn(anchor, constructive)
loss.backward()
optimizer.step()
optimizer.zero_grad()
scheduler.step()
if global_step % refresh_every == 0:
embs, ids = encode_all(mannequin, train_ds, train_batch_size, system)
train_sampler.update_index(embs, ids)Analysis
The retrieval setup is a typical dense IR analysis. The corpus is all 11,974 test-split anchor names, every encoded to a unit vector and saved in a FAISS FlatIP index. Every constructive variant within the check set is issued as a question; retrieval succeeds if the proper anchor seems within the top-k outcomes.
We report MRR, R@1, R@5, R@10, and NDCG@10, damaged down 3 ways: total, by question sort, and by script.
Total outcomes:
The classical baselines (Levenshtein, Double Metaphone, BM25) cluster at MRR ~0.09. This seems to be horrible, but it surely’s an artifact of what’s being measured: 70% of the analysis queries are cross-script (script or mixed sort), on which these strategies rating close to zero as a result of they share no characters with Latin-indexed names. On Latin-only queries, Levenshtein achieves 0.894 MRR — a wonderfully respectable quantity for a classical baseline.
Why total MRR misleads
The mixed sort is each the toughest and the most typical (70% of queries): the question is a phonetic variant of the anchor that was then transliterated right into a non-Latin script (“Katharine” → “كاثرين”, English anchor “Catherine”). Breaking down by question sort reveals the place every technique really fails.
The mannequin must deal with phonetic variation and script change concurrently. Transliterate, which applies a hard and fast canonical romanization, drops to 0.485 right here as a result of a hard and fast mapping can not account for phonetic variants within the question.
The byte encoder maintains robust efficiency throughout all three varieties (0.937 / 0.827 / 0.738). The contrastive coaching sign, which sees all three pair varieties, efficiently aligns phonetically equal byte sequences no matter script.
The script hole
The script hole is the R@10 distinction between Latin and non-Latin queries. Classical baselines have gaps of 0.88 to 0.94: they retrieve effectively inside Latin script however fail solely throughout script boundaries. The byte encoder reduces this to 0.096.
Importantly, the mannequin additionally improves Latin R@10 from 0.944 to 0.983. The contrastive goal generalizes within-script in addition to throughout scripts.
The remaining hole (0.096) is nearly solely defined by two scripts:
Scripts with constant romanization conventions (Arabic, Russian, Hebrew, Hindi, Greek) attain above 0.95. Chinese language (0.666) and Korean (0.728) are the outliers. Each have extreme romanization ambiguity: “张” maps to Zhang, Chang, and Cheung; “박” maps to Park, Pak, and Bak. The LLM-generated coaching knowledge accommodates all of those as positives for a similar entity, which produces conflicting gradient sign. The mannequin can not totally resolve which embedding area a reputation belongs to when its romanization is genuinely ambiguous.
Discover additionally that BM25 performs barely higher on Chinese language and Korean than different baselines. This isn’t as a result of BM25 understands phonetics. When the question is already within the goal script (Chinese language querying a Chinese language-indexed corpus), an identical CJK characters might seem in each question and doc, producing incidental character n-gram overlap. This impact disappears for true cross-script retrieval (Latin question, CJK corpus) and shouldn’t be mistaken for phonetic matching.
FAISS index ablation
HNSW matches precise search recall (0.896 vs 0.897 R@10) at 5.7x decrease latency. For deployment, HNSW is the selection: the small recall penalty is negligible and the latency enchancment compounds at scale. IVF-PQ cuts index dimension by 96% at a 6.4% R@10 penalty — value contemplating in the event you’re indexing hundreds of thousands of entities and reminiscence is constrained.
At 11,974 entities the distinction between 0.03 ms and 0.17 ms is tutorial. At 50 million entities in an actual deployment, HNSW’s recall benefit over IVF-Flat turns into extra pronounced because the variety of index partitions grows.
What doesn’t work (and why)
The mannequin fails to totally shut the hole on Chinese language and Korean, and the reason being value dwelling on. The pipeline generates non-Latin variants completely by transliterating from Latin: “Catherine” → Latin variant → Arabic/Chinese language/and so on. It by no means generates native-script spelling variation. Different Arabic orthographies, Korean spacing conventions, or variant Chinese language character types that check with the identical title don’t seem in coaching knowledge. The mannequin learns to map Latin byte sequences to non-Latin byte sequences, but it surely hasn’t seen non-Latin spelling variation inside a single script.
It is a identified limitation. The repair could be a fifth pipeline stage: given a generated Chinese language or Arabic title, ask the LLM to provide native-script phonetic variants of it. We didn’t do that, so the mannequin is probably going underperforming on queries that signify real-world native-script variation.
A second limitation: 99.5% of constructive pairs are LLM-generated. The analysis makes use of the identical LLM-generated pairs. If the LLM systematically mistransliterates a category of names, each coaching and analysis sign could be fallacious in the identical route, and we might not catch it. The 0.5% Wikidata floor reality gives a sanity examine however not a whole one.
Key takeaways
Byte-level tokenization is an underused software for multilingual duties. It eliminates out-of-vocabulary tokens by development, requires no language-specific tokenizer, and provides you a common 256-symbol vocabulary that covers each Unicode character. For duties the place floor type issues greater than semantics — like title matching — it’s a pure match.
LLMs are a viable knowledge engine for low-resource retrieval duties. We generated 4.67 million constructive pairs throughout 8 scripts utilizing two open-weight fashions. The pipeline is 4 phases, every independently resumable. This method is generalizable to different low-resource entity matching issues the place ground-truth labels are scarce however a succesful LLM can synthesize sensible variation.
ANCE arduous detrimental mining issues. The transition from random negatives to ANN-mined arduous negatives noticeably sharpens the embedding house. With out it, the mannequin would study to separate straightforward instances (totally different names in the identical script) however wrestle on the arduous ones (phonetically comparable names throughout scripts).
Report outcomes by question sort and script, not simply total MRR. An total MRR of 0.775 masks enormous variation: 0.937 on phonetic queries, 0.738 on mixed. A system that appears mediocre on headline metrics could also be near-perfect for one use case and damaged for one more.
The code, dataset pipeline, educated checkpoint, and analysis scripts are at github.com/vedant-jumle/cross-language-phonetic-text-alignment.
Word about Wikidata: Wikidata is launched below CC0 1.0 Common (public area) — no restrictions on use, together with industrial.
