Word & Sentence Embeddings
Meaning as vectors: the distributional idea, word2vec and co-occurrence embeddings, analogies and bias, contextual embeddings, and sentence embeddings for search and clustering.
"You shall know a word by the company it keeps"
Words that appear in similar contexts tend to have similar meanings: "metformin" and "insulin" both appear near "diabetes", "dose", "script". Embeddings turn this into geometry: each word becomes a dense vector (typically 100 to 1,000 numbers) so that similar words are close, measured by cosine similarity.
Two classic ways to learn them:
- Count-based: build a word–context co-occurrence matrix over a corpus, reweight it with PPMI (positive pointwise mutual information: how much more often two words co-occur than chance), and compress it with SVD.
- Prediction-based: word2vec trains a tiny network to predict context words from a word (skip-gram) using negative sampling; GloVe fits vectors to the log co-occurrence counts. Both give similar results to well-tuned count methods.
Embeddings famously support analogies by vector arithmetic (king − man + woman ≈ queen), and they also absorb biases in the text (e.g. associating professions with genders), which then flow into downstream models.
Try it: a map of word meanings
Embeddings learned from patient messages, projected to 2-D. Pick a word to see its nearest neighbours.
Interactive: this part runs in your browser.
Contextual embeddings
Static embeddings give one vector per word, but meaning depends on context: "discharge" from hospital vs "discharge" from a wound. A transformer encoder (BERT and its descendants) gives a different vector for each occurrence, computed from the whole sentence through attention. These contextual embeddings are what fine-tuned classifiers and taggers build on.
Sentence embeddings for search and clustering
To compare whole texts, use a model trained to produce one vector per sentence or paragraph so that texts with similar meaning are close: sentence-transformers (e.g. all-MiniLM, BGE, E5 models), trained contrastively on pairs of related texts. Uses: semantic search and RAG retrieval, duplicate detection, clustering messages into themes, and few-shot classification (a logistic regression on embeddings). Averaging raw BERT token vectors works much worse than a model trained for sentence similarity.
Evaluate retrieval on your own queries: recall@k (is the right document in the top k?) and MRR. Check multilingual performance if your users write in several languages.
Code example: embeddings from co-occurrence
A worked example: read it step by step (every part is explained), then run it and experiment.
Interactive: this part runs in your browser.
Code: semantic search with sentence-transformers
Embed policy paragraphs once, then search them with a question.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["Missed appointments without 24 hours notice may incur a fee.",
"Repeat prescriptions take two working days to process.",
"Call 000 for chest pain or difficulty breathing."]
D = model.encode(docs, normalize_embeddings=True)
q = model.encode("is there a charge if I don't turn up", normalize_embeddings=True)
scores = util.cos_sim(q, D)[0]
print(docs[int(scores.argmax())], scores.tolist())Quiz
1. What does cosine similarity between two embeddings measure?
2. PPMI reweights co-occurrence counts by…
3. Why do contextual embeddings help with "discharge"?
4. Best default for semantic search over clinic policies?
5. An embedding places "nurse" closer to "she" than "he". This is…
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Efficient Estimation of Word Representations (Mikolov et al., 2013) ↗ The word2vec paper
- Sentence-BERT (Reimers & Gurevych, 2019) ↗ the classic paper on sentence embeddings for semantic search
- Speech and Language Processing (Jurafsky & Martin, 3rd ed. draft) ↗ The standard NLP textbook, free online