Embedding Coherence Test (ECT)

The Embedding Coherence Test (ECT) is a diagnostic tool used to evaluate how well a set of vector embeddings preserves the semantic relationships present in the original data. It is especially relevant for modern language models, multimodal embeddings, and any system that transforms raw inputs into continuous vector spaces. By measuring coherence, researchers can identify whether embeddings capture meaningful structure or whether they are noisy, biased, or overly generic.

Why Coherence Matters

Vector embeddings are the backbone of many downstream tasks: similarity search, clustering, classification, and even generative models. If embeddings are incoherentmeaning that similar items are not clustered together or that unrelated items appear artificially closethe performance of all downstream pipelines degrades. Coherence assesses two fundamental properties:

Local consistency: items that belong to the same semantic class should have high mutual similarity.
Global structure: the overall geometry should reflect the hierarchy or topology of the source data.

Core Idea of the Test

ECT typically follows these steps:

Define a groundtruth taxonomy. This can be a labeled dataset, a hierarchy (e.g., WordNet), or a set of humangenerated similarity judgments.
Compute pairwise similarities. Using cosine similarity (or Euclidean distance) on the embeddings, produce a similarity matrix.
Compare to ground truth. Quantify how well the similarity matrix aligns with the taxonomy using statistical measures such as Spearmans rank correlation, average precision, or clustering metrics.
Report a coherence score. The final valueusually between 0 and 1summarizes the degree of alignment.

Key Metrics Used in ECT

Spearmans Rank Correlation

Ranks the similarity scores and compares them to the ranked groundtruth similarity. A value close to 1 indicates that higher-ranked pairs in the embedding space correspond to higher semantic similarity.

Mean Average Precision (MAP)

For each query item, retrieve its nearest neighbours and compute the precision at each rank where a true positive appears. The mean over all queries yields MAP, reflecting the quality of the nearestneighbour ranking.

Adjusted Rand Index (ARI)

If a clustering is performed on the embeddings, ARI measures the agreement between the induced clusters and the groundtruth class labels, correcting for chance.

Normalized Mutual Information (NMI)

Another clusteringbased metric that quantifies the shared information between predicted clusters and actual classes.

Practical Implementation

The following example uses Python with numpy, scikitlearn, and scipy to compute an ECT score based on Spearman correlation.

import numpy as npfrom scipy.spatial.distance import cosinefrom scipy.stats import spearmanr# embeddings: shape (n_samples, dim)# labels:   list or array of class ids for each sampleembeddings = np.load('embeddings.npy')labels      = np.load('labels.npy')# 1. build groundtruth similarity matrix (1 if same label, else 0)gt_sim = (labels[:, None] == labels[None, :]).astype(int)# 2. compute cosine similarity matrix for embeddingsnorms = np.linalg.norm(embeddings, axis=1, keepdims=True)normed = embeddings / normssim_matrix = np.dot(normed, normed.T)# 3. flatten uppertriangular part (exclude diagonal)triu_idx = np.triu_indices_from(sim_matrix, k=1)gt_vals   = gt_sim[triu_idx]emb_vals  = sim_matrix[triu_idx]# 4. calculate Spearman correlationrho, p_val = spearmanr(emb_vals, gt_vals)print(f"ECT Spearman score: {rho:.4f} (p={p_val:.2e})")

This script returns a single coherence number. Higher scores imply that the embedding space respects the labeling structure.

Extensions and Variations

Multilingual ECT: When embeddings are trained across languages, evaluate coherence per language and across language pairs to detect alignment gaps.
Temporal ECT: For models that evolve (e.g., continual learning), compute coherence at successive checkpoints to monitor drift.
Taskspecific ECT: Customize the groundtruth taxonomy to reflect the downstream task (e.g., sentiment classes for sentiment embeddings).
Negative Sampling: Include deliberately hard negatives in the groundtruth matrix to make the test more discriminative.

Common Pitfalls

Dataset bias. If the groundtruth taxonomy does not represent the diversity of the data, coherence scores can be misleading.

Dimensionality effects. Highdimensional embeddings may produce artificially high similarity scores; consider dimensionality reduction (e.g., PCA) before computing ECT.

Metric selection. Cosine similarity is standard, but Euclidean distance or Mahalanobis distance may be more appropriate for certain embeddings.

Interpreting Results

A typical interpretation guide (subject to domain specifics) might look like:

Score	Interpretation
0.80 1.00	Excellent coherence; embeddings faithfully encode the taxonomy.
0.60 0.79	Good but room for improvement; consider finetuning or adding contrastive loss.
0.40 0.59	Moderate coherence; model may be undertrained or data noisy.
Below 0.40	Poor coherence; revisit architecture, training data, or embedding objective.

Use Cases

Model selection. Compare several embedding models (BERT, RoBERTa, CLIP, etc.) on the same dataset to pick the most coherent.
Feature debugging. When a downstream task underperforms, run ECT to see if the root cause is incoherent embeddings.
Fairness auditing. Compute coherence separately for demographic subgroups to uncover bias.
Continuous monitoring. In production pipelines, schedule periodic ECT runs to detect embedding drift.

Conclusion

The Embedding Coherence Test provides a systematic, quantitative way to gauge whether vector representations retain the semantic structure of their source data. By aligning embeddings with a trusted taxonomy and measuring the agreement through wellunderstood statistical metrics, practitioners gain actionable insight into model quality, bias, and readiness for downstream applications. Whether you are finetuning language models, building crossmodal embeddings, or maintaining a live retrieval system, incorporating ECT into your evaluation toolkit can significantly improve robustness and interpretability.