LLM Residual Leak Detection in Anonymized Medical Reports

Context

When dealing with medical data for secondary analytical use, it is important to secure a good pseudonymization or anonymization pipeline that would keep all personal identifiers and sensitive information out of exports. This can be trivial for structured fields like personal names or addresses, but challenging for free-form textual reports, as there can be undesirable snippets embedded into long text chunks that vary per report type or even per medical professional. For that reason, such reports are usually not provided for secondary use because it is hard to guarantee they are clean.

I tried experimenting with using local LLMs to check for potential sensitive information and personal identifiers in texts, since intuitively they should be a stronger and more complex solution compared to regex, pattern matching, key lookups, and other common heuristics and tools. The goal was to build a sort of secondary audit layer (not an automatic anonymizer) that could triage already-anonymized reports, flag suspicious leftovers, and make manual review feasible instead of hopelessly broad.

What I Built

A series of experiments and local tooling to understand the behavior of locally run LLM models. This was initially a hobby project that later turned into a full-on research project, with benchmark datasets, model comparisons, manual review tooling, and paper-shaped evaluation scripts. We expect to turn this into a journal paper publication.

• Synthetic dataset generator — initial experiments were conducted on a dataset crafted around common report structures and patterns. Intentional leaks were introduced into the text, while keeping the exact spans as metadata to form a well-labeled dataset. The dataset included leaks such as personal names, addresses, birth dates, government identifiers, or strong contextual information pinpointing the individual.
• Experiment suite — resumable experiment harness against locally run LM Studio backends with classification and extraction tasks, strict/relaxed matching metrics, weighted stratum estimates, rule baseline comparison, and report table generators. Several experiments were set up both on synthetic data and on the local unlabeled dataset.
• Manual review tooling — the experiments needed tooling for results review, classification error handling, annotating datasets, comparing different models, and checking extracted spans. I chose a target-user oriented approach and crafted the tooling specifically for the case, so experts could move through blind annotation, mismatches, extraction audits, label export, and error taxonomy work as efficiently as possible.

Interesting Problems

• Utilization of local LLMs — due to the nature of data, it is impossible to utilize modern powerful models running as a cloud service. The data needs to stay on-premise, so the only viable option is to look into the capabilities of weaker models that can be run on common desktop hardware with a reasonable response speed or throughput. This alone is quite challenging, as the response quality is quite frequently correlated to the time spent generating the answer, so we are dealing with an optimization task in a space with various competing metrics. This also led to the idea of utilizing faster models with high recall and then either validating with slower models or combining several weaker local models into a practical model-agreement review queue.
• Unlabeled dataset — while we had quite a big local corpus of text reports, it was challenging to properly handle since labeling it would be extremely time-consuming and expensive. I needed to switch to stratified sampling of reports for manual annotation and work with estimated metrics describing the behavior. This made the evaluation into a small statistical workflow instead of a single leaderboard score. I looked at review burden, weighted estimates, model agreement, rule baseline comparison, and various edge cases around what should count as a residual identifier.
• Long-running experiments — as the experiments were constructed to simulate the expected hardware configurations, the batch jobs quite frequently took several days to complete, and careful preplanning and logging were needed not to waste the compute time.
• Custom-made tooling — I opted for custom-made annotation and review tooling instead of utilizing existing solutions that were not made for the exact purpose. This made it possible to navigate the results more easily and see how different models evaluated the case side by side, or quickly look up just the errors of a particular model in a given experiment. Due to the usage of an agentic coding workflow, the tooling was prepared in a matter of days, and additional feature requests were handled in a matter of hours. The iterative feedback with users was also rapid thanks to that. Compared to traditional SW development, this was created maybe at 1/10 of the time I'd expect it to take.