CorpusBased Finite State Morphological Analyzer for Pashto

Pashto () is an IndoIranian language spoken by over 50million people across Afghanistan and Pakistan. Its rich inflectional system, extensive use of prefixes, suffixes, and cliticisation, makes automatic morphological analysis a challenging but essential task for downstream applications such as machine translation, speechtotext, and information retrieval.

Why a Finite State Approach?

Finite State Transducers (FSTs) have become the defacto standard for modelling morphology because they combine:

• Determinism: The same input word always yields the same analysis.
• Efficiency: Lineartime processing and low memory consumption.
• Expressiveness: Ability to encode concatenative and nonconcatenative processes using regular relations.

For Pashto, where regular suffixes convey tense, aspect, gender, and number, an FST can directly encode these alternations while still allowing for exceptions and lexical irregularities.

CorpusBased Development

The analyzer described here is built on a sizeable, annotated Pashto corpus (approximately 2million word tokens). The workflow follows three major steps:

1. Corpus Collection and Normalisation

Texts were gathered from newspapers, literary works, and online forums. Unicode Normalisation Form C (NFC) was applied to ensure consistent encoding of diacritics and character variants (e.g., vs ).

2. Lexicon Extraction

From the normalized corpus a highfrequency lemma list was extracted. Lemmas were manually verified and tagged with POS information (Noun, Verb, Adj, etc.). For each lemma the following morphological features were recorded:

• Stem type (root, derived)
• Paradigm ID (determines the set of suffixes applicable)
• Irregular forms (if any)

3. Rule Induction

Statistical alignment between lemmas and their surface forms identified recurrent affix patterns. These patterns were then encoded as regular expressions that feed into the FST compiler. For example, the verb stem (to do) combines with the presenttense suffix - to produce (you do). The rule is represented as:

VerbStem+PresentSuffix -> VerbStem+PresentSuffix

Irregularities such as vowel deletion or gemination are captured by separate rewrite rules.

Finite State Implementation

The analysis engine is built with FOMA, an opensource toolkit for constructing weighted FSTs. The architecture consists of three layers:

Lexicon Layer

A symbol table maps each lemma to a unique identifier. Each entry includes a set of morphological tags encoded as feature bundles (e.g., +N+SG+ACC).

Affix Layer

All productive affixes are listed together with their feature specifications. For instance:

+V+PAST+3SG   ->   -+N+PL+GEN    ->   -+Adj+COMP    ->   -    

Rewrite Layer

Contextual rewrite rules handle phonological alternations. Example rule for vowel harmony:

[ae] -> [i] / _[+Vowel]  # fronting before a high front vowel    

The three layers are compiled into a single transducer that maps an input surface form to a set of possible analyses.

Evaluation

Using a heldout test set of 10000 manually annotated words, the analyzer achieved:

• Precision: 93.2%
• Recall: 89.7%
• Fscore: 91.4%

Most errors stem from:

• Undocumented loanwords.
• Ambiguous orthography where the same string can be both a noun and a verb.
• Rare morphological patterns not represented in the training corpus.

Sample Analyses

Below are illustrative outputs produced by the analyzer.

Input:      Analysis 1: +N+PL+ACCAnalysis 2: +N+PL+OBL    

Input:      Analysis:   +V+PAST+3SG    

Input:      Analysis:   +Adj+SUPERL+DEF    

Integration with Other Tools

The analyzer can be combined with:

• POS Taggers: The morphological tags serve as features for a Conditional Random Field tagger, improving disambiguation.
• Syntax Parsers: Morphological information feeds into a dependency parser, allowing better handling of casemarking.
• Speech Recognition: Pronunciation models benefit from explicit stemsuffix decomposition.

Future Work

Planned enhancements include:

• Incorporating a probabilistic ranking of analyses using a language model trained on the same corpus.
• Extending the lexicon with dialectspecific variants from southern Pashto.
• Adding a bidirectional transducer for generation (lemma surface) to support machine translation.

Conclusion

The presented corpusbased finite state morphological analyzer delivers highaccuracy analysis of Pashto word forms while remaining computationally lightweight. By grounding the rule set in a large, authentic corpus, the system captures both regular inflectional patterns and frequent irregularities. Its modular architecture allows straightforward extension and integration with downstream NLP components, providing a solid foundation for the continued development of Pashto language technologies.

For the source code, data, and detailed documentation visit the project repository.