Machine Learning with Ngrams for Sentiment Analysis of CodeMixed Texts

1. Introduction

Codemixing (or codeswitching) occurs when speakers alternate between two or more languages within a single sentence or discourse. Social media platforms, chat applications, and online forums generate massive amounts of codemixed data, especially in multilingual communities (e.g., HindiEnglish, SpanishEnglish, ArabicFrench). Sentiment analysis on such texts is challenging because lexical resources, syntactic rules, and sentiment cues are spread across languages.

A practical approach combines traditional ngram features with modern machinelearning classifiers. Ngrams capture shortrange lexical patterns that often carry sentiment information (e.g., muy bueno, not bad). When paired with languageidentification and preprocessing steps tailored to codemixed data, they become a strong baseline and sometimes rival deeplearning models.

2. Why Ngrams?

• Languageagnostic: Ngrams treat text as a sequence of characters or tokens, so they work regardless of language boundaries.
• Capture sentiment cues: Bigrams like very good, no me, or not happy are strong polarity indicators.
• Simplicity & speed: Feature extraction is fast, models are lightweight, and they can be trained on modest hardware.
• Interpretability: Important ngrams can be inspected directly, helping researchers understand linguistic phenomena in codemixed data.

3. Data Preparation

3.1 Collection

Typical sources: Twitter API (search with language filters), YouTube comments, Reddit threads, or WhatsApp chat dumps. When gathering data, retain the original Unicode characters to preserve diacritics and script variations.

3.2 Preprocessing Steps

1. Normalization: Convert fullwidth characters to halfwidth, unify quotation marks, and normalize emojis.
2. Language identification (tokenlevel): Tools such as langdetect, fastText language IDs, or custom CRF models can tag each token with its language.
3. Tokenization:
   • Use whitespace tokenization as a baseline.
   • Apply languagespecific tokenizers (e.g., jieba for Chinese, Moses for IndoEuropean languages) after language tags are known.
4. Cleaning: Remove URLs, user mentions, hashtags (or keep them as separate tokens), and replace numbers with a generic token <NUM>.
5. Handling transliteration: For codemixed scripts that use Romanized forms (e.g., bhalo for Bengali), convert to a canonical transliteration using libraries like Aksharamukha.

4. Feature Extraction

4.1 Tokenlevel Ngrams

Generate ngrams (n = 13) from the token list. Example for the sentence I love bahut accha:

unigrams:   [I, love, bahut, accha]bigrams:    [I love, love bahut, bahut accha]trigrams:   [I love bahut, love bahut accha]        

Store frequencies or TFIDF weights. TFIDF helps downweight ubiquitous ngrams such as the or common function words across languages.

4.2 Characterlevel Ngrams

Especially useful for mixed scripts or transliterated words. For the word sper, 3grams are s, p, per. Character ngrams capture morphological clues and misspellings that often appear in informal text.

4.3 Languageaware Features

• Proportion of tokens per language (e.g., 60% Hindi, 40% English).
• Switching points: count of language switches per sentence.
• Presence of languagespecific sentiment lexicon hits.

5. MachineLearning Models

After feature extraction, the data is represented as a sparse matrix (e.g., scipy.sparse.csr_matrix) and fed into a classifier.

5.1 Baseline Linear Models

• Logistic Regression with L2 regularization fast, works well with highdimensional ngram vectors.
• Linear Support Vector Machines (SVM) robust to class imbalance and can be calibrated for probability outputs.

5.2 Ensemble Methods

• Random Forest captures nonlinear interactions but may overfit on very sparse data.
• Gradient Boosting (XGBoost, LightGBM) effective when combined with TFIDF features and additional metadata (e.g., user location).

5.3 Neural Alternatives (Optional)

If larger datasets are available, a shallow CNN or a bidirectional LSTM that consumes the same ngram embeddings can be added for comparison. However, the focus of this page is the ngram + traditional ML pipeline.

6. Evaluation

Typical metrics for sentiment classification:

• Accuracy
• Macroaveraged F1score (important when classes are imbalanced)
• Precision & Recall per class (positive, neutral, negative)

Use stratified Kfold crossvalidation to ensure each fold respects the language distribution. Report confusion matrices to highlight systematic errors such as negative being confused with neutral in the minority language.

7. Practical Tips & Common Pitfalls

• Vocabulary size: Limit to the top 2030k ngrams; too many features increase memory consumption and degrade performance.
• Handling rare languages: If a token appears in only one language, consider adding a language prefix (e.g., en_good, hi_achha) to avoid accidental merging.
• Stopword removal: In codemixed data, stopwords can be sentiment carriers (e.g., not, nahi). Remove only languagespecific stopwords that are known to be neutral.
• Emoji and emoticon processing: Map them to textual sentiment tokens (e.g., POS_EMOJI).
• Class imbalance: Apply oversampling (SMOTE) or classweight adjustments in the loss function.

8. Sample Python Pipeline

import pandas as pdfrom sklearn.model_selection import train_test_split, StratifiedKFoldfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import classification_report, confusion_matrix# 1. Load data (columns: 'text', 'sentiment')df = pd.read_csv('code_mixed_sentiment.csv')# 2. Simple preprocessingdef clean(txt):    txt = txt.lower()    txt = re.sub(r'http\S+', ' ', txt)          # URLs    txt = re.sub(r'@\w+', ' ', txt)             # mentions    txt = re.sub(r'#\w+', ' ', txt)             # hashtags    txt = re.sub(r'[^\\w\\s]', ' ', txt)        # punctuation    return txt.strip()df['clean'] = df['text'].apply(clean)# 3. Tokenlevel TFIDF with ngrams (13)vectorizer = TfidfVectorizer(ngram_range=(1,3),                             max_features=30000,                             tokenizer=str.split,                             sublinear_tf=True)X = vectorizer.fit_transform(df['clean'])y = df['sentiment']# 4. Traintest splitX_train, X_test, y_train, y_test = train_test_split(        X, y, test_size=0.2, stratify=y, random_state=42)# 5. Classifierclf = LogisticRegression(max_iter=1000, n_jobs=-1, class_weight='balanced')clf.fit(X_train, y_train)# 6. Evaluationy_pred = clf.predict(X_test)print(classification_report(y_test, y_pred))print(confusion_matrix(y_test, y_pred))

The script demonstrates a minimal yet effective workflow. Extending it with languagetagged tokens or characterlevel vectors follows the same patternjust create additional TfidfVectorizer objects and horizontally stack the resulting matrices.

9. Future Directions

• Multilingual embeddings: Combine ngram features with word embeddings like FastText multilingual vectors for richer semantic info.
• Domain adaptation: Use adversarial training to align feature distributions between languages.
• Explainability: Apply SHAP or LIME on ngram models to visualize which mixedlanguage phrases drive sentiment decisions.
• Data augmentation: Generate synthetic codemixed sentences via backtranslation or languagemodel sampling to enlarge training sets.

10. Conclusion

Ngram based machinelearning pipelines remain a powerful tool for sentiment analysis of codemixed texts. By carefully preprocessing multilingual content, extracting both token and characterlevel ngrams, and leveraging robust linear classifiers, researchers can achieve high accuracy with modest computational resources. The approach offers transparency, ease of implementation, and a solid baseline for more complex neural architectures.