cs-135 — The Split That Leaks

Problem

A team builds a study-hours → pass/fail predictor for 40 students and reports: "test accuracy computed on 20 held-out rows." Their pipeline:

1. Standardise the feature: scaled = (hours − mean) / sd, with mean and sd computed over all 40 rows.
2. Rule: predict pass if scaled hours ≥ 0.
3. Evaluate on a random 20-row test split carved from the same 40.

Your deliverables:

1. In three sentences a non-technical dean would understand: why is the reported number optimistic?
2. The minimal correct pipeline order (what is computed where).
3. The worked demonstration below, by hand: one student whose pass/fail decision flips between the leaky and the correct pipeline.
4. One further leak in this setup, beyond the scaling one.

The eight-row teaching dataset (feature = study hours; label in parentheses):

2 (fail)  4 (fail)  4 (fail)  6 (fail)
7 (pass)  10 (pass)  12 (pass)  14 (pass)

For the demonstration, take the test split to be the rows 2, 7, 12, 14 and the training split the other four rows (4, 4, 6, 10). Standardise the test row 7 twice — once with the mean and sd computed over all eight rows (leaky), once over the four training rows (correct) — and apply the rule each time.

If the problem is ambiguous anywhere, state the assumption you are making and carry on — a stated assumption is part of a good answer.

Examples

• Input: the eight-row dataset; the two standardisation choices
• Output: leaky: scaled −0.0945 → fail — correct: scaled +0.4082 → pass — the same student, two decisions

Solve for the worked instance first. Then ask what would change if one number, name, or condition were replaced by something stranger.

What to notice

• Scaling is not cosmetic: the rule's threshold lives in scaled space, so whoever computes the mean controls the verdict.
• The leak is information flow: eight rows include the test row, so the "unseen" case helped set its own yardstick.
• A decision flip is the strongest form of evidence — one traced student persuades a dean better than an abstract warning.

Discussion starter

After the solution is revealed, the class spends five minutes on this prompt: "The team answers: 'the flip is one student out of forty — immaterial.' Draft your two-sentence reply, using the pipeline order from deliverable 2, not rhetoric."

Constraints and assumptions

• Difficulty: Expert (subtle pipeline reasoning with stakeholder framing).
• Topics: leakage, evaluation, communication.
• Use the population standard deviation (divide by n); rounding at four decimals only at the very end.
• Classroom-scale inputs unless the problem says otherwise.
• Time-box: about five minutes for a first attempt; partial answers with clear reasoning are worth more than silent struggling.
• Optional stretch (not required): describe an alternative approach and say when it beats the obvious one.

Hints and extension

• First nudge: list the quantities the leaky pipeline computes from all rows before anything is "held out".
• If stuck after three minutes, write the input/output line you expect and trace the smallest example by hand.
• After the reveal, attempt the extension below before the next lecture.
• Extension challenge: with the same eight rows, choose a test split for which the two pipelines agree — and characterise (no exhaustive search needed) which test rows can never flip.