Automatic PreProcessing of Marathi Text for Summarization

1. Introduction

Marathi, spoken by more than 80 million people in India, possesses a rich literary tradition that ranges from classical poetry to modern journalism. Automatic text summarization can help readers quickly grasp the essence of long articles, news items, or research papers written in Marathi. However, reliable summarization depends heavily on the quality of the input text. Unlike English, Marathi exhibits a high degree of morphological variation, flexible word order and the frequent use of diacritics, making preprocessing a critical step.

2. Why PreProcessing Matters

Summarization modelswhether extractive (selecting salient sentences) or abstractive (generating new sentences)are trained on tokenised, normalised data. Poorly tokenised or noisy input leads to:

• Incorrect sentence boundaries, causing fragmented extracts.
• Vocabulary mismatch, which reduces the effectiveness of word embeddings.
• Increased outofvocabulary (OOV) rates, especially for inflected forms.
• Misinterpretation of named entities and numeric expressions.

Consequently, a systematic preprocessing pipeline dramatically boosts summarization performance.

3. Core PreProcessing Steps

3.1. Normalisation

Marathi text can be written in Devanagari script or a mixture of Devanagari and Latin characters (codeswitching). Normalisation includes:

• Unicode normalisation (NFKC) to ensure that visually identical characters have a single code point.
• Conversion of fullwidth digits and punctuation to ASCII equivalents.
• Standardising common orthographic variations, e.g., . versus .

3.2. Tokenisation

Word tokenisation in Marathi is nontrivial because spaces often appear inside compound words and after punctuation marks. Two approaches are common:

• Rulebased tokenisers using regular expressions for punctuation, numerals and common suffixes (,,).
• Statistical tokenisers trained on annotated corpora such as the IndicNLP Tokenizer, which learn probable word boundaries.

3.3. Sentence Segmentation

Marathi uses (Danda) as the primary sentence delimiter, but periods, question marks, and exclamation points are also common, especially in digital media. A robust segmenter must:

• Detect . used in abbreviations like . or . and prevent premature splitting.
• Handle quoted dialogue where punctuation appears inside quotation marks.

3.4. StopWord Removal (Optional)

While stopword elimination is useful for extractive methods that rely on term frequency, it may hurt abstractive models that need full context. A curated stopword list (250 words) is available from the StopwordsISO project.

3.5. Stemming and Lemmatisation

Marathi is highly inflectional; a single lemma can appear in dozens of surface forms. Two techniques are employed:

• Light stemming removal of common suffixes (,,,) without altering the root meaning.
• Lemmatisation mapping inflected forms to their dictionary lemma using resources such as the Indic NLP Library. Lemmatisation yields higher recall for OOV reduction.

3.6. Named Entity Recognition (NER) Normalisation

Named entities (persons, locations, organisations) often appear in various spellings. Normalising them helps the summarizer preserve essential information. Techniques include:

• Dictionarybased lookup using a gazetteer of common Marathi names.
• Contextual NER models (e.g., multilingual BERT finetuned on Marathi NER data).

3.7. Handling Numerics and Dates

Numbers may be written in Devanagari digits (, ) or Arabic digits (1, 2). Converting all numerics to a unified format simplifies later processing. Dates are expressed in multiple patterns (e.g., , 25-07-2023). Regular expressions can standardise them to ISO format (YYYYMMDD).

4. Sample Pipeline Implementation (Python)

The following snippet demonstrates a compact pipeline using indic_nlp_library and regex:

import refrom indicnlp.normalize.indic_normalize import IndicNormalizerFactoryfrom indicnlp.tokenize import sentence_tokenizefrom indicnlp.tokenize import indic_tokenize  def normalise(text):    factory = IndicNormalizerFactory()    normalizer = factory.get_normalizer('mr')    return normalizer.normalize(text)def split_sentences(text):    # First use Danda    sentences = re.split(r'|\?|!|\.', text)    # Clean empty strings    return [s.strip() for s in sentences if s.strip()]def tokenise(sentence):    return list(indic_tokenize.trivial_tokenize(sentence))def stem(tokens):    suffixes = ['','','','','','','','']    stemmed = []    for t in tokens:        for suf in suffixes:            if t.endswith(suf) and len(t) > len(suf)+2:                t = t[:-len(suf)]                break        stemmed.append(t)    return stemmeddef preprocess(text):    text = normalise(text)    sentences = split_sentences(text)    processed = []    for s in sentences:        toks = tokenise(s)        toks = stem(toks)        processed.append(' '.join(toks))    return processedsample = ".               ."print(preprocess(sample))

This code normalises Unicode, splits sentences, tokenises, and applies a light stemming step. For productiongrade systems, replace the simplistic stemmer with the Lemmatiser from indic_nlp_library and integrate a pretrained NER model.

5. Evaluation of PreProcessing Impact

Empirical studies on Marathi summarization (e.g., using the IndicXSUM dataset) show that:

• Applying Unicode normalisation reduces ROUGE1 error by ~2%.
• Accurate sentence segmentation improves extractive summarizer recall by 45%.
• Lemmatisation lowers the OOV rate from 12% to 6%, leading to a 3% boost in BLEU for abstractive models.

These gains underline that even modest preprocessing can have a measurable effect on downstream summarization quality.

6. Challenges and Open Issues

Despite progress, several challenges remain:

• Codeswitching: Social media posts frequently mix Marathi and English. Detecting language boundaries and applying languagespecific preprocessing is still an active research area.
• Dialectal variation: Words like vs. appear in rural texts, requiring dialectaware lexicons.
• Scarcity of annotated corpora: Highquality sentencesegmented and lemmatised corpora are limited, affecting the training of statistical tokenisers.

7. Future Directions

To further improve Marathi summarization, researchers can explore:

• Joint modelling of tokenisation and summarization using transformerbased encoders that learn subword units (e.g., SentencePiece trained on Marathi).
• Multitask learning where NER, POS tagging, and summarization share a common backbone.
• Incorporating phonetic transliteration to handle mixedscript inputs.

8. Conclusion

Automatic preprocessing of Marathi text is a prerequisite for any reliable summarization system. By addressing Unicode normalisation, accurate tokenisation, sentence segmentation, and morphological analysis, developers can significantly reduce noise, improve vocabulary coverage, and ultimately produce more coherent, informative summaries. Continuous refinement of these stepsespecially in the face of codeswitching and dialectal diversitywill enable richer information access for Marathi speakers.