The original article is in Chinese. The English version is automatically translated by AI.
Preface
When federated learning comes up, there’s usually a point where someone says the data stays local. I pause there. Suppose training really is done on the clients and the server receives only their updates. How do we tell how much that helps with privacy?
I want to start with what’s visible to the server. In another setting, whoever is looking might have prediction scores from an API, the model file, or gradients. Once we know what they can see, we can talk about the risk.
Let’s go through a few of those situations. Someone might want to know whether a record was in training, or they might be interested in an attribute. We’ll see what signals are available, how to read the results, and what the protections do about them. The numerical examples all use synthetic data.
Establishing the Threat Model First
I start with four questions. Who can see the results, what exactly can they see, what do they already know, and what are they trying to learn? Keep those in mind and it’s easier to place each paper in the table.
| Scenario | Attacker’s Observation | Possible Additional Knowledge |
|---|---|---|
| User of a classification service | Final label, confidence, or full probability vector | Candidate samples and their true labels, data from the same distribution |
| Person obtaining model files | Parameters, structure, intermediate representations, or gradients | Training pipeline, preprocessing methods, partial training data |
| Federated learning server | Models from each round, client updates, or aggregation results | Client identities, participation records, optimizer configurations |
| Collaborating clients | Received global model and their own local updates | Data they hold, some other publicly available information |
From the word black box alone I’d be tempted to assume two papers are using the same interface. Look at what actually comes back. A single label and the full list of class probabilities give someone very different amounts of information.
Another detail worth checking is whether the person observing the results can change what happens. An honest-but-curious participant follows the protocol and analyzes what it returns. An active adversary may also alter the model, participant selection, or other steps. I check which of those the security result assumes.
What Are the Different Attack Types Asking?
| Attack Type | Objective | Objects Not to Confuse |
|---|---|---|
| Membership Inference | Determine whether a candidate sample participated in training | Does not equal recovering the full content of that sample |
| Attribute Inference | Infer sensitive attributes of a sample, user, or training set | Does not necessarily require knowing exact membership status |
| Model Inversion / Gradient Reconstruction | Recover input information from model outputs, representations, or gradients | Representative images of a class are not necessarily real training samples |
| Training Data Extraction | Recover actual training content from the model | Distinguish between ordinary generation, similar content, and verbatim reproduction |
When I see an experiment labeled an attack, I look at what it’s measuring at the end. Model theft goes after the model’s capabilities or parameters. Adversarial examples are about changing predictions. For the questions here, we’re trying to understand what the model can reveal about its training data.
Membership Inference: Why Do Predictions Reveal Training Identity?
A Simple Loss Threshold Attack
Let the candidate sample be $(x,y)$. The attacker can obtain the target model’s prediction and knows its label. One of the simplest approaches is to compute the loss and make a judgment based on a threshold:
where $m=1$ denotes a training set member. This attack relies on an empirical signal: models often fit training samples better, so members may have lower loss.
Think of two easy cases. A sample the model has never seen can still have low loss if it’s easy to classify. A hard training sample can have high loss. To measure the rule itself I’d use separate data for calibrating the threshold and for reporting the result.
Shokri and colleagues give us a way to check that intuition. They used shadow models and a classifier to look for differences between member and non-member predictions, and demonstrated a risk in their black-box setting. With another model or dataset, I’d check whether those differences are still there. 1
Why Is Average Accuracy Insufficient?
Definition
I start with the denominators when reading these two abbreviations. TPR counts the share of actual members we find. FPR counts the share of non-members we call members by mistake. Suppose the member share among the candidates is $\pi$. Of everything labeled a member, how much actually is?
It’s easier to picture with 10,000 candidates. Take $\pi=1%$, a TPR of 80%, and an FPR of 1%. That gives 80 actual members detected and 99 non-members flagged by mistake. Put both counts in the denominator and precision comes out to about 44.7%.
When someone reports accuracy with half members and half non-members, I ask whether that split matches the setting we’re interested in. Carlini and colleagues’ LiRA work also looks at how many members are detected at low false-positive rates, for example TPR at 0.1% FPR. 2
There’s one more issue when measuring a false-positive rate that low. We need enough non-members to see it. If we have a few hundred and happen to get no false positives, all we’ve observed is zero in that run. Check the sample count before reporting a stable estimate at the one-in-a-thousand scale.
Overfitting Is Not the Only Explanation
Overfitting can give us a membership signal. Even when the average training and test losses are close, I’d go back and look at the unusual records. Heavily repeated, rare, or anomalous ones can get hidden in the average.
For the comparison I’d first see whether the two groups came from comparable sources. Match the classes, preprocessing, and collection process as closely as possible. If those differ, check whether the classifier is finding membership or simply telling the data sources apart.
Attribute Inference and Model Inversion
For attributes the question gets a little narrower. Someone may already know a few features and want to infer one more from the output. Or they may be looking at a client’s update to infer something about the distribution on that client.
I look separately at how accurate the inference is and whether the model remembered a particular record. Inferring a sensitive attribute from public features can already affect privacy. Showing that the model memorized this person’s training record takes more evidence.
When looking at a model inversion image I first ask what it matches. It may be very convincing to the classifier, or it may actually match a training record. To claim the latter, compare it with the training records and state the matching criteria and reconstruction quality.
Why Gradients Might Leak Inputs?
Analytical Example with a Single-Sample Linear Model
Let’s start with the simplest model:
Let the residual be $r=f_{w,b}(x)-y$, then
If an attacker observes these two single-sample gradients and $g_b\ne0$, they can divide coordinate by coordinate:
The original input was not uploaded, yet the gradients are sufficient to recover it. Below, we verify this using only three dimensions of synthetic data:
| |
This is not a formula that holds for all models or all batch settings. If $g_b=0$, division is not applicable; if one observes the average gradient of multiple samples, the numerator and denominator become weighted sums, and typically one cannot recover each input using this ratio.
From Analytical Recovery to Gradient Matching
For neural networks, reconstruction can be understood as searching for a set of candidate inputs whose generated gradients are close to the observed gradients:
where $R$ represents a prior or regularization term; this expression is merely illustrative for the single-sample case. Deep Leakage from Gradients demonstrates the possibility of recovering training inputs via gradient matching; subsequent work, Inverting Gradients, used direction-dependent objectives and stronger optimization strategies to further investigate image reconstruction. 3, Inverting Gradients4
When I look back at a reconstruction result, I check what the experiment actually provided. Batch size, model, parameters, known labels, preprocessing, normalization, and whether several updates are visible can all matter. A failed reconstruction is an observation about that setup. A strict privacy guarantee after changing batch size, compression, or clipping still needs its own analysis.
Training Data Extraction: Do Models Really Recite the Original Text?
If the question becomes whether we can get the actual content back, we have to go beyond membership. When a language model generates something familiar, I ask whether it’s a common phrase, public material, or an exact passage we can match to a training record.
Carlini and colleagues did that comparison, checking text returned by a model against training text. Results that can be matched to records show the risk that comes with memorization. 5
In a controlled experiment we can use a unique synthetic canary with no real sensitive information. Record whether it entered training, how often it appeared, and what the model produced. I’d read the result in light of those conditions. Real-data risk takes an assessment of its own.
This brings us back to the first question. If we see no verbatim repetition, we can still ask what someone might infer about membership, attributes, or other information. Those outcomes have to be evaluated separately.
Why Federated Learning Does Not Automatically Solve Privacy Issues?
Back to federated learning. Moving training to clients can reduce how much raw data has to be collected centrally. Then I’d look at what they send back. Those updates can still carry information about their local data.
I first check whether the server gets one client’s update or a combined result. Seeing just the aggregate changes what’s visible. Then look at who participated, possible collusion, information across rounds, and whether the server can change the training conditions.
For secure aggregation, follow what the server sees. Under the protocol’s assumptions it gets the aggregate, while the individual plaintext inputs are hidden during computation. How much the resulting aggregate or model says about an individual is the next question to ask. 6
I keep track of which inputs secure aggregation hides during computation and how much influence DP allows a protected unit to have on the released result. The two can be used together. For an implementation I’d check the actual protocol and its assumptions on participation, dropouts, and collusion.
One related detail from the Dirichlet non-IID split7 is that some clients can have very few samples or highly concentrated labels. With that setup I’d also check what’s observed and how the test was run before interpreting the privacy result.
What Do Common Defenses Protect?
| Method | Primary Effect | Boundaries to Retain |
|---|---|---|
| Data minimization, sensitive field handling, and deduplication | Reduces sensitive content entering the system before training and opportunities for repeated memorization | Cannot guarantee by itself that remaining data will not leak |
| Training measures such as regularization and early stopping | Improves generalization and may reduce some membership inference signals | Good average generalization does not equate to per-sample privacy guarantees |
| Reduce probability outputs, query limits, and access control | Limit attackers’ observation and invocation capabilities | Must cover model files, logs, and other accessible outputs |
| Clipping, quantization, compression, or larger batch sizes | Change the information carried by updates and the difficulty of attacks | Cannot claim a DP guarantee without privacy analysis |
| Secure aggregation, encrypted computation, or trusted execution environments | Protect the computation process under specified trust and protocol assumptions | Released outputs may still enable inference |
| Differential Privacy | Limit output distribution changes caused by altering protected units | Adjacency relations, sampling methods, and total privacy budget must be explicitly defined |
DP-SGD is not just about adding a bit of noise to gradients
Follow one update when reading DP-SGD. Clip each sample’s gradient first, aggregate the result, then add calibrated noise. The expression below puts those steps together.
In the expression, $C$ is the clipping threshold, $\sigma$ is the noise multiplier, and $\mathcal B$ is the batch. If you’re asking what $(\varepsilon,\delta)$ comes out to, first pin down the neighboring datasets, the sampling mechanism, and the privacy accountant. 8
When reading the code I check where clipping happens and which results are exposed. Moving clipping from individual samples to a whole batch changes the sensitivity and accounting. If unnoised gradients are also released, account for that view separately. Check what the optimizer, checkpoints, data-dependent tuning, and additional released statistics reveal too.
With federated data, remember how many records one user can contribute. To protect the user, control the effect at the level of their contribution and account for how they participate and what gets released. If all I have is a record-level parameter, I go back and check which unit it covers.
Patterns in the population and public information can still support inference. I go back and ask what change the neighboring-dataset definition makes and what the observer already knows. I pick that up again in the post on information theory in privacy9.
How to Design a Persuasive Privacy Experiment?
If I were reading an experiment, I’d first write down what’s exposed and what unit is protected. Are we measuring the risk from final predictions or from training gradients? Is the claim about one record or all the records from a user?
Then I’d look at the numbers. For membership inference I’d find TPR at low FPR and check the assumed member prevalence. For reconstruction I’d check the match to the actual inputs, the errors, and the success rate. For extraction I’d count how many matches can be verified. A good-looking picture helps me follow one example. The full set of samples needs those numbers too.
I’d also check that calibration and final evaluation are separate and that the two groups have comparable distributions. The seeds for training, attack optimization, and splitting can change the result. Along with the average I’d look for variation and failures.
For a defense I’d look at several reasonable attack baselines and ask whether they know the defense. An attack failing in this run is one empirical result. For a theoretical guarantee, I’d go to the proof or the privacy accounting from a trusted implementation.
Conclusion
At the end I’d ask the opening question a little more precisely. Are we worried about someone learning that a sample was in training, inferring an attribute, or getting the input or original text back? Once we name the secret, we know which observations to look at.
Next time I hear that the raw data stays local, I want to ask which data we can leave out from the start, which intermediate results need to stay hidden, and how much one record can influence what we release. Then I can go back to the threat model and the experiments and see which step each guarantee covers.
References and Further Reading
Shokri et al.: Membership Inference Attacks against Machine Learning Models. ↩︎
Carlini et al.: Membership Inference Attacks From First Principles. ↩︎
Carlini et al.: Extracting Training Data from Large Language Models. ↩︎
Bonawitz et al.: Practical Secure Aggregation for Privacy-Preserving Machine Learning. ↩︎

