The Model Picks the Code, the Rules Decide: Guardrails for an LLM-Driven Medical Interview
Hosted by Piotr Obolewicz / CIC Fukuoka Member
Wednesday, September 30, 2026
9:00 AM–10:00 AM ET
About
MedBot is a research prototype that conducts structured medical interviews with synthetic patients in simulated cases, then hands the resulting report to a physician evaluator. It never gives advice to patients.
What you'll see:
- Closed catalogs instead of free-text safety decisions. The model can select only red-flag codes from a YAML catalog. Urgency levels and escalation language come from deterministic rule files; out-of-catalog values are discarded as invalid.
- Structured output generated from the rules. JSON Schema enum values are generated from the same rule files used during runtime validation.
- Detect, then audit. One LLM call selects red-flag codes; a second call audits each code against its formal condition. Giving the condition to the detector itself dropped sensitivity from 1.00 to 0.91, so the two jobs stay separate. If either call returns no verdict, the turn is rolled back.
- Measure, don't guess. A phrase matcher achieved 0.81 sensitivity on the tuning set but only 0.60 on a sealed holdout set. Across a 220-case evaluation, an open-weight model detected every labeled alarm for approximately $0.05 per run, although it produced more false positives. The tested Sonnet configuration missed five alarms.
- Safety gates we removed. A dosage filter could not reliably distinguish medication history ("the patient takes aspirin") from treatment advice ("take aspirin"). A fact-grounding gate rejected 25 of 40 first-pass outputs.
- Prompt caching in a multi-turn interview. Why the framework's default caching strategy yields zero cache reads, and costs more than running without caching.
Who it's for: engineers building LLM features in regulated or high-stakes domains, including healthcare, finance, and legal services. The implementation uses Java and Spring AI with Anthropic and open-weight models; the patterns are stack-agnostic.
In person, 16 seats, 11th floor (Singapore room). Doors 8:45 AM; we start at 9:00 sharp and end at 10:00 AM.
Register on Luma: https://luma.com/h3mw9ge8
Hosted by Piotr Obolewicz · https://obolewicz.com
Speakers
- Piotr Obolewicz
Host
Piotr Obolewicz / CIC Fukuoka Member
Location
- CIC Cambridge, One Broadway, 11th floor (Singapore room) - Doors 8:45 AM; we start at 9:00 sharp and end at 10:00 AM - 16 seats. If your plans change, please cancel your RSVP so someone on the waitlist can take your seat.
1 Broadway, Cambridge, MA 02142
Open in Google Maps ↗