MHB is a benchmark for evaluating whether large language models resist medical hallucination traps in complex clinical tasks. It contains two subsets:

• MHB-Dialogue: adversarial multi-turn medical conversations derived from realistic doctor-patient dialogues.
• MHB-Report: clinical case reports paired with hallucination-inducing questions.

The benchmark follows the paper "MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks". The release contains 4,695 samples and 20,288 rubric items.

.
├── data/
│   ├── mhb_dialogue.jsonl
│   ├── mhb_report.jsonl
│   ├── dataset_stats.json
│   ├── taxonomy.json
│   └── README.md
├── src/mhb/
│   ├── respond.py
│   ├── evaluate.py
│   └── summarize.py
├── configs/models.example.json
└── requirements.txt

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No package installation step is required. Run the Python scripts directly from the repository root. The model-call scripts use OpenAI-compatible chat-completion APIs, so set your endpoint and key before running them:

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://your-provider.example/v1"

Each JSONL row contains only the fields needed for evaluation: a stable id, a task, hallucination labels, rubrics, and task-specific input.

Dialogue sample:

{
  "id": "mhb_dialogue_00001",
  "task": "dialogue",
  "hallucination_types": ["A1"],
  "input": {"messages": [{"role": "user", "content": "..."}]},
  "rubrics": ["The model must identify ..."]
}

Report sample:

{
  "id": "mhb_report_00001",
  "task": "report",
  "input": {"clinical_report": "...", "question": "..."},
  "rubrics": ["The model should not fabricate ..."]
}

Use mhb_dialogue.jsonl and mhb_report.jsonl to reproduce the paper-scale sample count.

Generate target-model responses:

python src/mhb/respond.py \
  --data data/mhb_dialogue.jsonl data/mhb_report.jsonl \
  --out outputs/target_model.responses.jsonl \
  --model your-target-model \
  --temperature 0

Judge the responses against the expert rubrics:

python src/mhb/evaluate.py \
  --data data/mhb_dialogue.jsonl data/mhb_report.jsonl \
  --responses outputs/target_model.responses.jsonl \
  --out outputs/target_model.judgments.jsonl \
  --judge-model your-judge-model \
  --temperature 0

The main metric is rubric-level hallucination rate:

hallucination_rate = failed_rubrics / total_rubrics

where a failed rubric is one for which the judge returns criteria_met=false.

@inproceedings{lu2026mhb,
  title={MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks},
  author={Lu, Jianrong and Liu, Junwei and Zheng, Xingyun and Yang, Minghui and Wang, Jian and Wang, Ping and Zhang, Yechao},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2026}
}

Code is released under the MIT License. Dataset users should follow the licenses and usage terms of the original source datasets cited in the paper, including HealthBench and PMC-Patients, and use MHB for research and safety evaluation only.