Clinical NLP Prototype

A two-stage pipeline for processing clinical text in swiss context:

1. De-identification : detect and redact personally identifiable information (PII) with language auto-detection (en/fr/de/it).
2. Entity extraction : extract clinical entities (diseases, medications, dosages, frequencies, lab values) from the de-identified text and emit them as structured JSON.

---

Models

Both PII and NER models are hard-coded. Here is the list of the models used.

Stage	Language	Model
PII (de-id)	DE	OpenMed/OpenMed-PII-German-SuperClinical-Small-44M-v1
PII (de-id)	FR	OpenMed/OpenMed-PII-French-SuperClinical-Small-44M-v1
PII (de-id)	IT	OpenMed/OpenMed-PII-Italian-SuperClinical-Small-44M-v1
PII (de-id)	EN	OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1
Disease NER	all	OpenMed/OpenMed-NER-DiseaseDetect-SuperClinical-434M
Medication NER	all	OpenMed/OpenMed-NER-PharmaDetect-SuperClinical-434M

All models are downloaded automatically from HuggingFace on first run. The clinical NER models are English-trained but provide partial cross-lingual coverage through shared Latin medical vocabulary.

---

Architecture overview

Key design choices :

• Hybrid ML + regex. Transformers are good for fuzzy clinical entities such as diseases, symptoms, and drug names. Regex is better for entities with fixed formats, such as AHV numbers, dosages like 5 mg, dates, routes, and frequencies. A practical prototype can combine both: use ML for ambiguous medical language, and regex for structured patterns.
• Uniform Span type across all detectors. Whether a span came from a regex or a transformer, the downstream merge/redaction/linking code is identical.
• Multilingual. Switzerland operates clinically in DE/FR/IT/EN. The PII stage uses language-specific OpenMed models (one per language). The clinical NER models (disease, medication) are primarily English-trained; Latin medical vocabulary provides partial cross-lingual coverage, but German compound words or French clinical phrasing can degrade quality. The sentence_language parameter is set per document to improve sentence segmentation.
• Auditability. Every span carries a source field (regex:ahv_ch, transformer:OpenMed/...) so any extraction can be traced back to the rule or model that produced it.

---

Reflection

The design rationale, alternatives considered, human-review points, and production failure modes are documented in REFLECTION.md.

---

Setup

Requires Python ≥ 3.10, <3.14.
This project was developed and tested using uv, which is the recommended way to install and run the pipeline. uv provides fast, reproducible dependency resolution and environment management, making it easier to reproduce the prototype exactly as tested.

Recommended: using uv

# Install uv if you don't have it
# Linux/MacOS
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"


# Clone the clinical-nlp repository and enter
git clone https://github.com/girochat/clinical-nlp.git
cd clinical-nlp

# Create environment and install (this resolves dependencies)
uv sync

Alternatively with pip:

python -m venv .venv
source .venv/bin/activate
pip install -e .[dev]

Model downloads. On first run, ~2 GB of HuggingFace models are downloaded and cached under ~/.cache/huggingface/. Subsequent runs use the local cache.

---

Usage

CLI

# Process a file
uv run clinical-nlp process data/note_en_01.txt

# Write JSON to a file
uv run clinical-nlp process data/note_de_01.txt --output result.json

# Use a different redaction strategy
uv run clinical-nlp process data/note_fr_01.txt --strategy replace

# Only run Stage 1
uv run clinical-nlp deid-only data/note_en_01.txt

CLI options:

Option	Default	Description
--output, -o	(stdout)	Write JSON output to this file
--strategy, -s	mask	mask | replace
--min-score-pii	0.5	Minimum confidence score required to keep transformer-extracted PII entities (0–1).
--min-score-ner	0.5	Minimum confidence score required to keep transformer-extracted NER entities (0–1).
--lang, -l	None (auto-detect)	Text language, when known. Only en, fr, de, it supported. Language auto-detection is run by default.
--seed	None	Random seed for reproducible 'replace' surrogates.
--pretty/--compact	--pretty	Pretty-print JSON
--verbose, -v	off	Print INFO-level logs

HTTP API

uv run uvicorn api.app:app --reload

Then:

# Full pipeline
curl -X POST http://localhost:8000/process \
    -H "Content-Type: application/json" \
    -d '{"text": "John Doe, born 12.03.1958, has diabetes and takes metformin 500mg BID."}'

# De-identification only
curl -X POST http://localhost:8000/deidentify \
    -H "Content-Type: application/json" \
    -d '{"text": "Patient: Mary Schmidt, +41 44 255 11 11"}'

# Health check
curl http://localhost:8000/health

OpenAPI docs are auto-generated at http://localhost:8000/docs.

---

Example

Input (examples/example_input_de.txt):

Verlaufseintrag
Patient: Max Mustermann, geb. 14.03.1978
AHV-Nr.: 756.9217.0769.85
Telefon: +41 44 255 11 11
 
Diagnosen: Diabetes mellitus Typ 2, arterielle Hypertonie.
Medikation: Metformin 1000 mg 1-0-1, Ramipril 5 mg 1-0-0.
Labor: HbA1c 7.8%, Kreatinin 95 µmol/L, CRP 3.2 mg/L.
 
Hausarzt: Dr. med. Anna Weber, 1700 Fribourg, Tel. +41 26 555 12 34.

De-identified output using the following command: (examples/example_deidentified_output_de_replace.txt) clinical-nlp process examples/example_input_de.txt --strategy replace --seed 42

Verlaufseintrag
Patient: Noémie Bucher, geb. 31.05.1971
AHV-Nr.: 756.0000.0000.00
Telefon: +41 43 332 18 19
 
Diagnosen: Diabetes mellitus Typ 2, arterielle Hypertonie.
Medikation: Metformin 1000 mg 1-0-1, Ramipril 5 mg 1-0-0.
Labor: HbA1c 7.8%, Kreatinin 95 [CURRENCYSYMBOL]/L, CRP 3.2 mg/L.

Hausarzt: Dr. med. Irfan Bär, 1133 Veyrier, Tel. +41 33 863 79 40.

Names, dates, AHV, phone numbers, and postal codes are replaced with Swiss-locale Faker surrogates. When the PII model emits a label not supported by the replace strategy (e.g. CURRENCYSYMBOL), the pipeline falls back to mask-style output ([CURRENCYSYMBOL]). Here, µmol is misclassified as currency (score 0.62) — a false positive on a clinical unit. This illustrates the need for further work on label mapping for greater precision.

Structured extraction (abbreviated, full output examples/example_output_de.json):

{
  "medications": [
    {"name": "Metformin", "dosage": "1000 mg"},
    {"name": "Ramipril", "dosage": "5 mg"}
  ],
  "lab_values": [
    {"name": "HbA1c", "value": "7.8", "unit": "%"},
    {"name": "Kreatinin", "value": "95"},
    {"name": "CRP", "value": "3.2"}
  ],
  "grouped": {
    "disease": ["Diabetes mellitus", "arterielle Hypertonie"],
    "medication": ["Metformin", "Ramipril"]
  }
}

---

Synthetic data

The data/ folder contains synthetic clinical notes in four languages for testing as well as small comments about edge cases.

---

Project layout

clinical-nlp/
├── README.md
├── pyproject.toml
├── src/clinical_nlp/
│   ├── schema.py          ← pydantic data contracts
│   ├── regex_rules.py     ← Swiss PII regex + clinical regex
│   ├── merge.py           ← span-overlap resolution
│   ├── deid.py            ← Stage 1: de-identification
│   ├── ner.py             ← Stage 2: clinical NER
│   ├── linking.py         ← medication ↔ dosage linking, lab values
│   ├── pipeline.py        ← end-to-end orchestrator
│   └── cli.py             ← typer CLI
├── api/app.py             ← FastAPI
├── tests/                 ← pytest suite (runs without model downloads)
├── data/                  ← synthetic clinical notes (EN/DE/FR/IT)
└── examples/              ← example I/O for documentation

---

Tests

uv run pytest

The test suite (90 tests) intentionally covers only the deterministic, model-free components: regex rules, span merging, redaction strategies, medication linking, lab value extraction.

---

What this prototype does NOT do

List of out-of-scope items:

• No negation / hypothetical detection. "Patient denies chest pain" currently produces a positive chest pain extraction. NegEx-style detection is not implemented.
• No fine-tuning on Swiss clinical text. The models are used off-the-shelf. Swiss clinical notes differ from the English biomedical text these models were trained on: different drug brand names (Dafalgan instead of Tylenol), German compound medical terms (Belastungsdyspnoe, Niereninsuffizienz), Swiss-specific abbreviations, and multilingual mixing within documents. Fine-tuning would require annotated Swiss clinical data, accessible through institutional partnerships (e.g. SPHN — Swiss Personalized Health Network).
• No terminology linking. The pipeline outputs text as found in the document (e.g. "Diabetes mellitus", "Metformin"). It does not map these to standardized medical codes such as SNOMED-CT, ICD-10, or ATC, which would be required for integration with hospital IT systems or billing.
• No formal evaluation on annotated test sets. Synthetic notes used here are illustrative, not benchmarking material.
• Lab extraction is regex-based and minimal. Only a small list of common lab names have been used for demonstration; production would go further and use LOINC-driven lookup.
• No Human Review flag. The prototype does not implement a requires_review flag on PII entities because it would not make sense at this stage (all de-identified output should be reviewed by a clinician before use). Similarly, the clinical extraction stage (medications, diseases, lab values) produces results that require systematic review: lab analytes misclassified as medications, temperatures matched as dosages, and the absence of negation detection all mean that structured output cannot be trusted without human validation.

---

AI Assistance Disclosure

AI assistants (ChatGPT and Claude) were used during development for architectural discussion, implementation support, debugging, test generation, and documentation drafting. Final design decisions, evaluation of model behavior and failure modes, and the overall integration of the pipeline were reviewed and validated by the author.

---

License

Apache 2.0