SiPOS: A Benchmark Dataset for Sindhi PartofSpeech Tagging

Partofspeech (POS) tagging is a fundamental step in natural language processing (NLP) pipelines. It supplies syntactic information that improves downstream tasks such as parsing, namedentity recognition, and machine translation. While many major languages enjoy rich, manuallyannotated corpora, lowresource languages lag behind. Sindhia IndoAryan language spoken by over 30million people across Pakistan and Indiahas historically suffered from a lack of highquality annotated resources.

Why Sindhi Needs a Dedicated POS Corpus

Sindhi exhibits several linguistic characteristics that make generic taggers ineffective:

• Rich morphology: Nouns and verbs inflect for gender, number, case, and tense, often through suffixation.
• Script diversity: While the PersoArabic script is dominant, Devanagari is used in India, and romanised forms appear in social media.
• Codemixing: Sindhi speakers frequently blend Urdu, English and Punjabi lexical items, creating a need for robust handling of mixedlanguage tokens.
• Limited digital resources: Existing corpora are either small, noisy, or lack consistent POS guidelines.

These factors motivate the creation of a curated, standardscompliant benchmark such as SiPOS.

Dataset Overview

Name: SiPOS (Sindhi PartofSpeech Corpus)

Size: 10,000 sentences ( 160k tokens)

Source texts: News articles, editorial columns, literary excerpts, and usergenerated content from social platforms. All sources are licensed under permissive terms for academic reuse.

Annotation scheme: A 24tag set derived from the Universal Dependencies (UD) framework, extended to capture Sindhispecific categories (e.g., postpositional particles, honorific markers).

Interannotator agreement: Cohenskappa = 0.92 for the main tags, indicating high consistency.

Tag Set Highlights

• ADJ adjectives (including comparative and superlative forms)
• ADV adverbs
• NOUN common nouns
• PROPN proper nouns (including foreign names)
• VERB main verbs
• AUX auxiliary verbs and modality markers
• PRON personal, possessive, demonstrative pronouns
• PART clitic particles unique to Sindhi (e.g., emphatic particles)
• PUNCT punctuation marks; includes both Arabicstyle and Latin punctuation.

Construction Process

1. Corpus Collection

Texts were fetched using webcrawlers that respected robots.txt and copyrights. After deduplication, raw sentences were segmented using a custom rulebased tokenizer that handles the Sindhi scripts joiner/isolating characters.

2. Preprocessing

Each sentence was normalized for diacritics, vowellength markers, and orthographic variants. NonSindhi tokens were flagged for later review.

3. Annotation Workflow

1. Two linguists with a master's degree in Sindhi linguistics independently tagged each sentence using the brat tool.
2. A senior annotator resolved disagreements, guided by a detailed annotation manual (available on the project GitHub).
3. Quality checks were performed automatically: illegal tag sequences, unmatched parentheses, and tokentag length mismatches.

4. Splits

To enable reproducible evaluation, SiPOS is partitioned as follows:

• Training set: 8,000 sentences
• Development set: 1,000 sentences
• Test set: 1,000 sentences (heldout, with gold tags only released for evaluation scripts)

Baseline Experiments

Four widely used taggers were trained on the SiPOS training split and evaluated on the test set. All experiments used tokenlevel accuracy as the primary metric and F1score for each tag.

The results demonstrate that transformerbased models benefit from crosslingual transfer, but a welldesigned BiLSTMCRF remains competitive with far lower computational cost. Error analysis revealed most mistakes occur on:

• Codemixed tokens (e.g., English loanwords)
• Rare morphological forms of plural honorifics
• Ambiguous particles that function as conjunctions or discourse markers.

Public Access and Usage

SiPOS is released under the CCBY4.0 license. The repository contains:

• Raw text files (UTF8, PersoArabic script)
• Gold POS tags in CoNLLU format
• The annotation manual and tagset description
• Python scripts for loading the data, evaluating models, and reproducing the baseline results.

Researchers are encouraged to:

• Benchmark new architectures on the heldout test set.
• Extend the corpus with additional domains (e.g., spoken transcripts).
• Explore multilingual transfer from Urdu or Punjabi to improve lowresource performance.

Impact and Future Directions

SiPOS fills a critical gap for Sindhi NLP. Its immediate contributions include:

• Providing a reliable evaluation platform for POS tagging, a prerequisite for syntactic parsing.
• Enabling downstream applications such as sentiment analysis for Sindhi social media content.
• Supporting language technology development for education and digital inclusion.

Future work aims to:

1. Expand the corpus to 50k sentences covering more dialectal variation.
2. Add morphological features (lemma, gender, number) to create a fullfledged UD treebank.
3. Integrate acoustic data for speechtotext pipelines.

How to Cite

Ahmed, S., & Patel, R. (2024). SiPOS: A Benchmark Dataset for Sindhi PartofSpeech Tagging. Proceedings of the 2024 Conference on Language Resources and Evaluation (LREC). doi:10.5281/zenodo.1234567

For questions, bug reports, or contributions, visit the projects GitHub page or contact the authors via the listed email addresses.