Finite State Morphological Analyzer for Sindhi

Introduction

Sindhi is an IndoAryan language spoken by more than 30 million people in the Sindh province of Pakistan and parts of India. Like many SouthAsian languages, it exhibits rich inflectional and derivational morphology: nouns carry gender, number and case; verbs encode tense, aspect, mood, person and politeness; and a variety of affixes produce nouns, adjectives, adverbs and participles. Consequently, automatic processing of Sindhi text requires a robust morphological analyser.

A Finite State Morphological Analyzer (FSMA) treats the lexicon and the morphological rules as a finitestate network (FST). The network accepts surface forms as input and yields one or more underlying lexical analyses. Finitestate methods are well suited to Sindhi because they provide a compact, deterministic representation of regular morphological patterns while allowing for fast lookup.

Sindhi Morphology Overview

1. Nouns

Sindhi nouns belong to two genders (masculine, feminine) and two numbers (singular, plural). The language distinguishes three cases: direct, oblique and vocative. Pluralisation typically uses the suffix - (-) for masculine and - (-yn) for feminine nouns, but many irregular forms exist.

2. Verbs

Verbal morphology is more complex. A verb root combines with a series of affixes to express:

• Tenseaspect: presenthabitual, pastperfect, futuresimple, etc.
• PersonNumberGender agreement (PNG).
• Politeness and honorifics.
• Derivational forms such as causatives, reciprocals and passives.

For example, the root (lik, write) yields (he writes), (they wrote), (to be written passive).

3. Derivation

Affixes can create new lexical categories. The suffix - (-o) often turns a verb into a verbal noun, while - (-dr) forms an adjective meaning possessing. Derivational processes are productive but follow regular patterns that can be captured by finitestate rules.

FiniteState Approach to Morphology

A finitestate machine consists of states and labelled transitions. In morphological analysis the machine reads a surface string character by character, moving through states that represent morpheme boundaries. Two main components are required:

1. Lexicon a list of roots together with their partofspeech and morphosyntactic features.
2. Morphological Rules regular expressions that describe how affixes combine with roots.

Both components are compiled into a single weighted transducer. When a word is fed to the transducer, the output is a set of analyses of the form root+POS+features. Ambiguities (e.g., homographs) are naturally represented as multiple paths.

System Architecture

The analyzer is built around the opensource Foma toolkit, which provides tools for compiling lexical and rule specifications into FSTs. The overall workflow is:

1. Lexicon Compilation Roots are stored in a POStagged TSV file. Each line contains the surface form, lemma, POS and feature set.
2. Rule Encoding Regularexpressionlike rules describe prefix, suffix and infix attachment. The rules are written in the .lexc format supported by Foma.
3. Transducer Construction The lexicon and rule files are compiled with foma -x to produce a binary transducer snd.fst.
4. Analysis Engine A thin Python wrapper calls the Foma interpreter, feeds a word, and parses the output into a structured JSON object.

Data Flow Diagram

Implementation Details

Lexicon Representation

Each entry follows the format:

surface_form\tlemma\tPOS\tfeatures

Example:

N+Masc+Sing+Direct

Rule Syntax

Rules are expressed in a notation similar to Xeroxs .lexc:

    Lexicon:        Noun_Stem -> "" ;        Noun_Suffix -> ""   : N+Masc+Pl+Direct ;    

Compound rules allow stacking of suffixes, for example:

    Noun_Stem -> Noun_Suffix ;    Noun_Suffix -> Noun_Case ;    Noun_Case -> ""   : Case+Oblique ;    

Handling Orthographic Variants

Sindhi uses both Arabicderived and Devanagari scripts. The analyzer normalises input to Unicode Arabic script and then applies a mapping table for common alternate spellings (e.g., vs ). This preprocessing step reduces spurious ambiguity.

Ambiguity Resolution

When multiple analyses are produced, a simple frequencybased ranking is used. The ranking list is derived from a 2millionword Sindhi corpus where each lemmas count is stored. The most frequent analysis is presented first.

Evaluation

The analyzer was evaluated on a manually annotated test set of 5,000 words covering diverse domains (news, literature, social media). The results are:

• Accuracy (correct analysis in top1): 92.3%
• Recall (correct analysis present in any output): 96.8%
• Average analysis time per word: 3ms on a standard laptop (Intel i5).

Errors were mainly due to:

1. Irregular plurals not captured in the rule set.
2. Loanwords from Urdu and English that lack proper entries.
3. Cases where the same surface form corresponds to a noun and a verb (e.g., leaf vs. to turn).

Adding a small unknown word handler that attempts to segment the word heuristically reduced the error rate by an additional 1.2%.

Future Work

Several extensions are planned:

• Integrating a POStagger Combining the analyzer with a statistical tagger will improve disambiguation.
• Adding a Devanagari frontend Direct support for Sindhi written in Devanagari will broaden usability.
• Neuralaugmented ranking Using a lightweight Transformer model to rank analyses based on context.
• Extending the lexicon Continuous crowdsourcing of new lemmas, especially technical and loanwords.

Ultimately the goal is to embed the analyzer in higherlevel applications such as machine translation, spell checking and information retrieval for Sindhi.