AI in healthcare: Study shows risks to data privacy

Health data is particularly sensitive. A recent study highlights risks associated with the use of AI in healthcare.
At the NeurIPS 2025 conference, Sana Tonekaboni and colleagues from MIT and the Broad Institute presented an investigation into the safety of Foundation Models in healthcare. The work focuses on the risk of so-called memorization of training data by Artificial Intelligence and the associated dangers for data protection. Memorization refers to a process similar to learning data by heart.

The Conflict: generalization vs. memorization

Foundation Models for the medical field are often trained with extensive electronic health records. The goal of training is generalization: the model should learn medical relationships to predict diagnoses or progressions. Example: Coughing and weight loss suggest tuberculosis. A technical problem arises when the model instead stores and reproduces specific patient data from the training set. This can lead to sensitive information of a real person being output instead of a general prediction when a query is made.

Contextual assessment of risks

The research emphasizes that technical data leaks do not automatically mean a high risk for the patient. The authors distinguish based on the sensitivity of the data:

1. Low Risk: The model reveals general information such as blood values or age, which often do not allow identification on their own.
2. High Risk: The model reveals specific diagnoses (e.g., HIV, drug abuse) or data from patients with rare diseases.

Because health data is often anonymized, the risk increases significantly if an attacker already has partial information such as age and gender and uses the model to reconstruct the missing, sensitive parts of the record. The investigation shows: The more information contained in the prompt, the more likely a successful reconstruction of the training data.

Methodology: a framework for privacy tests

To make these risks measurable, the researchers developed a test framework comprising six different testing procedures. The framework tests the model as a black box, meaning only via inputs and outputs and without direct access to the model weights.

• Generative Tests: Here, it is checked whether the model exactly reproduces training data. A sensitivity test specifically examines whether the model predicts sensitive attributes such as drug abuse even if this information was omitted in the input.
• Embedding Tests: These analyze the internal vector representations of the model. It is tested whether conclusions can be drawn about a person's membership in the training dataset from these numerical values, the so-called Membership Inference.
• Distinction by Perturbation: To check whether the model generalizes knowledge or just retrieves facts, the researchers changed details in the input, e.g., the patient's age. If the model continues to make the same specific diagnosis despite the changed age, this indicates generalization: the disease matches the symptoms. However, if the prediction changes significantly with a minimal change in personal data, this can be an indication that the model had previously merely memorized a specific data record.
• Subgroup Analysis: The framework specifically checks vulnerable groups, such as patients with rare diagnoses or very old people. Because their data patterns in the training set are unique, the risk that the model memorizes these specific cases is significantly higher.

Results and benefit

Applying the tests to a publicly available benchmark model showed that while the model learned statistical patterns, in certain constellations it was susceptible to revealing sensitive attributes, especially when the input prompt contained many details. The researchers provided an open-source toolkit that allows developers to systematically check medical AI models for such privacy gaps before release.