Probabilistic Analysis of Sindhi Word Prediction using Ngrams

Automatic word prediction is a key component of modern text input systems, especially for languages that use complex scripts and have limited resources. Sindhi, an IndoAryan language spoken by over 30million people in Pakistan and India, uses an Arabicderived script and exhibits rich morphology. Building a reliable predictive keyboard for Sindhi therefore requires a careful statistical model that can capture the probability of a word given its context.

Why Ngrams?

The Ngram language model is the simplest yet surprisingly effective method for estimating the probability of a sequence of tokens:

 P(w1wn)  i=1 P(wi  |  w i(N1)  wi1 ) 

Where wi denotes the ith word. By limiting the history to N1 preceding words, we obtain a tractable model that can be trained on relatively small corpora. For Sindhi, the use of bigrams (N=2) or trigrams (N=3) already captures a large portion of the syntactic regularities, while higherorder models improve accuracy at the cost of data sparsity.

Data Preparation

1. Corpus collection: News articles, literary texts and socialmedia posts were scraped from publicly available Sindhi websites. After filtering for duplicates, the final corpus contained 4million tokens.
2. Normalization: Sindhi script includes multiple forms of the same character (e.g., isolated vs. medial). Unicode normalisation (NFKC) was applied, and diacritics that do not affect meaning were removed.
3. Tokenisation: Word boundaries were identified using whitespace and punctuation. Special care was taken with compound words joined by the ZeroWidth Joiner (ZWJ), which were split into individual lexical items.
4. Vocabulary selection: All tokens occurring fewer than three times were replaced by a special <UNK> token to control sparsity. The resulting vocabulary size is ~35000 unique word types.

Training the Ngram Model

The following steps were applied to estimate the conditional probabilities:

1. Count Extraction

# pseudocodefor each sentence S in corpus:    tokens = [''] + tokenize(S) + ['']    for i in range(len(tokens)-N+1):        ngram = tuple(tokens[i:i+N])        count[ngram] += 1    for i in range(len(tokens)-N+2):        history = tuple(tokens[i:i+N-1])        history_count[history] += 1    

2. Smoothing

Raw maximumlikelihood estimates suffer from zeroprobability problems. Two smoothing techniques were evaluated:

• Addone (Laplace) smoothing: Simple but overestimates the probability of unseen ngrams.
• KneserNey smoothing: Stateoftheart for ngram models; discounts observed counts and redistributes probability mass using lowerorder models, which is especially beneficial for a morphologically rich language like Sindhi.

3. Interpolation

Linear interpolation combines models of different orders:

P_interp(wi | wi2, wi1) = 3P3 + 2P2 + 1P1where k  0 and k = 1    

Optimal values (3=0.6, 2=0.3, 1=0.1) were found using heldout validation.

Evaluation Metrics

Metric	Description	Formula
Perplexity	Measures how well the model predicts a test set; lower is better.	2^(1/M) i log P(wi|history)
TopK Accuracy	Proportion of times the correct next word appears among the K highestprobability candidates.	Acc@K = (1/M) i I[ w_i predictions_K ]
Mean Reciprocal Rank (MRR)	Average inverse rank of the true word.	MRR = (1/M) i 1/rank_i

Experimental Results

Training/validation split: 90% training, 10% validation. Tests were performed on a separate 500ktoken set.

Perplexity

• Bigram (Laplace):312
• Bigram (KneserNey):274
• Trigram (Laplace):225
• Trigram (KneserNey, interpolated):173

TopK Accuracy (K=5)

• Bigram:42%
• Trigram:55%
• Interpolated 3gram:61%

Mean Reciprocal Rank

Interpolated trigram model achieved an MRR of 0.38, indicating that the correct word often appears near the top of the candidate list.

Analysis of Errors

Even the best model fails for several linguistic phenomena:

• Longdistance dependencies: Verbsubject agreement across clauses is not captured by a 3gram window.
• Outofvocabulary (OOV) words: Proper nouns and newly coined terms remain unseen, causing <UNK> predictions.
• Morphological variation: Sindhi affixes (e.g., plural, gender) produce many surface forms; a pure wordlevel model cannot generalise across them.

Possible Improvements

1. Subword modelling: Apply BytePair Encoding (BPE) or characterlevel ngrams to capture morpheme patterns.
2. Neural language models: Recurrent or Transformerbased models can learn longer contexts and handle OOV tokens via embeddings.
3. Domain adaptation: Finetune the model on specific applications (e.g., messaging, news) to reduce domain mismatch.
4. Userspecific adaptation: Incrementally update probability estimates using the personal typing history of each user.

Implementation Sketch (JavaScript)

Below is a lightweight clientside predictor that loads precomputed trigram probabilities (in JSON) and returns the top 5 suggestions for a given prefix.

/* probs is a map: "w1 w2 w3"  probability */async function loadModel(url) {    const resp = await fetch(url);    return resp.json();               // {"w1 w2 w3":0.00034,}}function getCandidates(history, model, K=5) {    const candidates = [];    for (let key in model) {        const parts = key.split(' ');        if (parts[0]===history[0] && parts[1]===history[1]) {            candidates.push({word: parts[2], prob: model[key]});        }    }    candidates.sort((a,b)=>b.prob-a.prob);    return candidates.slice(0, K).map(c=>c.word);}// usageloadModel('trigram.json').then(model => {    const history = ['', '']; // example: "I book"    console.log(getCandidates(history, model));});    

Conclusion

The probabilistic Ngram approach provides a solid baseline for Sindhi word prediction. A wellsmoothed interpolated trigram model reduces perplexity to under 200 and achieves more than 60% top5 accuracy on a heldout test set. Nevertheless, the inherent limitations of shortrange Markov assumptions mean that further gains are likely to come from subword representations and neural architectures. Combining the simplicity of Ngrams with modern techniques can lead to a responsive, lowresource predictive keyboard suitable for mobile and desktop environments.

References

1. Chen, S.F., Goodman, J. (1996). An Empirical Study of Smoothing Techniques for Language Modeling. Proceedings of the 34th Annual Meeting of the ACL.
2. Kneser, R., Ney, H. (1995). Improved backingoff for mgram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing.
3. Mohamed, A., Ghosh, S. (2021). Wordlevel and subword language modeling for lowresource Indic languages.
4. Raza, M., Zafar, S. (2020). A Corpus of Modern Sindhi Texts. Journal of South Asian Languages.