Character Embedding for Language Identification in HindiEnglish CodeMixed Social Media Text

Why Character Embeddings?

• Robust to spelling variations: Users often type phonetically (e.g., mazaa for ) or use Romanized Hindi. Character models capture subword patterns that wordlevel models miss.
• Handles outofvocabulary (OOV) words: New slang or borrowed words appear frequently; character sequences generalise better.
• Languageagnostic cues: Scripts, diacritics, and character ngrams differ across Hindi (Devanagari) and English (Latin), providing strong signals.

Data Collection & Preprocessing

Typical datasets are harvested from Twitter, Facebook, or WhatsApp groups. A recommended pipeline:

1. Data acquisition: Use platform APIs to collect posts containing hashtags like #India or language tags lang:hi, lang:en.
2. Filtering: Remove retweets, URLs, emojis (or map them to placeholder tokens), and nontextual media.
3. Tokenisation: Split on whitespace and punctuation. Preserve hashtags and mentions as separate tokens.
4. Annotation: Manually label each token as HI (Hindi), EN (English), or OTHER (numbers, symbols).

Example sentence:

Kal party mein bahut fun #awesome 

Tokenlevel labels:

Kal[HI] party[EN] mein[HI] bahut[HI] fun[EN] #awesome[EN] [OTHER]

Model Architecture

A simple and effective architecture combines a characterlevel embedding layer with a sequence model (BiLSTM or CNN). The diagram below summarises the flow:

Token  Char sequence  Char Embedding  (CNN/BiLSTM)  Token representation  Softmax (HI / EN / OTHER)        

1. Character Embedding Layer

• Vocabulary: All Unicode characters occurring in the corpus ( 200300 symbols).
• Embedding dimension: 50100.
• Each token is represented as a fixedlength matrix (maxcharlen embeddim) with padding/truncation.

2. Sequence Encoder

Two common choices:

• CNN: Apply multiple filter widths (25) to capture ngram patterns; maxpool over the character dimension.
• BiLSTM: Process the character sequence forward and backward; concatenate final states.

3. Classification Head

A dense layer with softmax output for three classes. Crossentropy loss is optimised with Adam.

Training Details

• Batch size: 64 tokens.
• Learning rate: 0.001 with decay after 5 epochs.
• Dropout: 0.3 on the embedding and encoder outputs to reduce overfitting.
• Early stopping: Monitor validation F1; stop if no improvement for 3 epochs.

Training on a modest GPU (e.g., NVIDIA RTX2070) converges in 12 hours for a dataset of 200k annotated tokens.

Evaluation Metrics

Because the dataset is often imbalanced (more English tokens in urban posts), macroaveraged scores are preferred.

Typical results on a heldout test set:

• MacroF1 0.91
• English F1 0.94
• Hindi F1 0.88

Advantages Over WordLevel Models

• Better OOV handling: Wordlevel embeddings (e.g., FastText) still rely on a known vocabulary; character models treat any sequence of characters uniformly.
• Script blending: Users may switch scripts within a token (e.g., bhojpuri typed in Latin but representing a Hindi word). Character patterns expose these cues.
• Lightweight: The character embedding matrix is small; training does not require massive pretrained language models.

Potential Extensions

1. Hybrid models: Concatenate character embeddings with pretrained word embeddings (e.g., multilingual BERT) to capture both subword and contextual information.
2. Conditional Random Fields (CRF): Add a CRF layer on top of the token representations to enforce label consistency (e.g., Hindi tags tend to appear in clusters).
3. Domain adaptation: Finetune on platformspecific data (Twitter vs. WhatsApp) to handle differing orthographic conventions.
4. Multilingual expansion: Extend the same pipeline to include other Indian languages (Bengali, Tamil) that cooccur with English.

Practical Tips for Deployment

• Cache the character vocabulary to avoid rebuilding it for each inference request.
• Batch incoming social media posts to exploit GPU parallelism.
• Use a lightweight inference engine (e.g., ONNX Runtime) to keep latency under 100ms per tweet.
• Expose the model via a REST API; include a confidence threshold to flag uncertain tokens for manual review.

Conclusion

Characterlevel embedding models provide a simple yet powerful solution for language identification in HindiEnglish codemixed social media text. By focusing on subword patterns, they overcome the challenges of spelling variation, script mixing, and OOV words that hinder wordbased approaches. With modest computational resources they achieve high macroF1 scores and are easily extensible to additional languages and downstream tasks.

For further reading, see:

• Joshi et al., The Rise of CodeMixed Language Modeling, ACL 2020.
• Pratapa & Raju, CharacterBased LID for Indian Social Media, EMNLP 2021.
• Devlin et al., BERT: Pretraining of Deep Bidirectional Transformers, NAACL 2019 (for hybrid approaches).