MIS 752 · Lab 5 · kenneth-holstad

Speak the Patient's Language: semantic search vs keyword search

A 20-sentence clinical corpus embedded with all-MiniLM-L6-v2, the full cosine similarity matrix, and a scored comparison of semantic search against a keyword baseline on a written gold standard. Runs locally on a free CPU: no API key, no PHI, $0.

The 30-second read

THE MODEL encoder : sentence-transformers/all-MiniLM-L6-v2 dimensions : 384 numbers per sentence corpus : 20 sentences, 4 topics, 2 voices embed time : 367 ms for the whole corpus API cost : $0.00 (runs locally, no key, no network after load) DOES IT CAPTURE MEANING? mean cosine within a topic : 0.235 mean cosine across topics : 0.129 separation : +0.106 RETRIEVAL vs THE GOLD STANDARD (12 queries, top-3) keyword (word overlap) precision 0.38 · recall 0.32 · MRR 0.50 semantic (embeddings) precision 0.42 · recall 0.54 · MRR 0.92 overall, semantic beats keyword on MRR by +0.42 THE FINDING - MRR SPLIT BY WHETHER THE PATIENT GUESSED OUR WORDS patient guessed our words keyword 0.86 · semantic 1.00 patient's own words keyword 0.00 · semantic 0.80 EXAMPLE QUERY "I get really tired if I skip breakfast" 1. men1 [mental health/patient] cosine 0.449 2. dia1 [diabetes/patient] cosine 0.429 3. dia3 [diabetes/patient] cosine 0.390

Do embeddings capture meaning?

The 20×20 cosine similarity matrix, rows and columns sorted by topic and boxed into the four topic blocks. Each label shows the sentence id and its voice (pat = patient, cli = clinical). The diagonal is blanked and the colour scale spans the off-diagonal range: every sentence scores 1.00 against itself, and leaving that in flattens everything else into one shade. A dark cell spanning both voices is the whole point of the week: same meaning, no shared words. A pale row inside a box is a sentence this model cannot place.

Meaning beats spelling only where spelling runs out

Scored against the written-down gold standard, not by eyeball. MRR asks how high the first right answer was, and it is the metric a patient on a phone actually feels: 1.0 means rank one. The two groups differ by whether the patient happened to guess the corpus's own vocabulary — a split computed from the text, not labelled by hand. In the left group keyword search is fine, because there are words to match. In the right group there are none and it collapses. Semantic search never needed the words to match in the first place.

Per-query detail

Where each method wins and loses, query by query. The keyword baseline is deliberately naive word overlap, so this is not the strongest possible keyword system — BM25 would do better, and a reranker better still. It is enough to show that the vocabulary gap, not the algorithm, is what defeats word matching.

Blue chip = relevant per the gold standard · outlined = retrieved but not relevant · P = precision@k, R = recall@k

Query	Keyword search	Semantic search
patient guessed our words
chest hurts when i climb stairs	car1car2first hit at #1 · P 1.00 · R 0.67	car1car2car4first hit at #1 · P 0.67 · R 0.67
my refill is not ready at the pharmacy	med5first hit at #1 · P 1.00 · R 0.50	med5dia4car5first hit at #1 · P 0.33 · R 0.50
is it safe to take ibuprofen with warfarin	med1dia1med2first hit at #1 · P 0.67 · R 1.00	med1med2med5first hit at #1 · P 0.67 · R 1.00
i feel low and cannot sleep	men1men3first hit at #1 · P 0.50 · R 0.33	men1dia3dia5first hit at #1 · P 0.33 · R 0.33
would the anti-inflammatory clash with what thins my blood	car5med1first hit at #2 · P 0.50 · R 0.50	med1med2men3first hit at #1 · P 0.67 · R 1.00
i have been down and wide awake at 3am for weeks	med5men1men5first hit at #2 · P 0.33 · R 0.33	men1dia3men2first hit at #1 · P 0.67 · R 0.67
my feet feel like pins and needles	dia5men3first hit at #1 · P 0.50 · R 0.50	dia5car1car5first hit at #1 · P 0.33 · R 0.50
patient's own words
it aches to climb to the second floor	nothing returnedno hit · P 0.00 · R 0.00	car1dia5men1first hit at #1 · P 0.33 · R 0.33
the reading on my meter jumped since supper	nothing returnedno hit · P 0.00 · R 0.00	car5dia5car2no hit · P 0.00 · R 0.00
i cannot pay for the name brand any more	nothing returnedno hit · P 0.00 · R 0.00	med3med5car5first hit at #1 · P 0.33 · R 0.50
the drug store says come back next week	nothing returnedno hit · P 0.00 · R 0.00	med5men5dia4first hit at #1 · P 0.33 · R 0.50
i get winded carrying the groceries in	nothing returnedno hit · P 0.00 · R 0.00	car4car1dia1first hit at #1 · P 0.33 · R 0.50

How this was measured

Sentences were embedded with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, ungated, run locally on a free Colab CPU -- no API key and no network traffic after the model loads). Cosine similarity is computed directly as E @ E.T on L2-normalised rows, not via a library convenience method, so every number is reproducible from the saved embeddings.npy. Retrieval is scored against 12 queries carrying 29 written relevance judgements (GOLD in the notebook). The keyword baseline counts shared content words (stopwords dropped, length > 2). Metrics are precision@k, recall@k and mean reciprocal rank at k=3. This is a 20-sentence teaching corpus with a 12-query gold set -- a demonstration of method, not a clinical validation. The corpus is teaching text I wrote: no PHI, no real records, nothing to act on clinically.