The original article is in Chinese. The English version is automatically translated by AI.

Preface

When federated learning comes up, there’s usually a point where someone says the data stays local. I pause there. Suppose training really is done on the clients and the server receives only their updates. How do we tell how much that helps with privacy?

I want to start with what’s visible to the server. In another setting, whoever is looking might have prediction scores from an API, the model file, or gradients. Once we know what they can see, we can talk about the risk.

Let’s go through a few of those situations. Someone might want to know whether a record was in training, or they might be interested in an attribute. We’ll see what signals are available, how to read the results, and what the protections do about them. The numerical examples all use synthetic data.

Establishing the Threat Model First

I start with four questions. Who can see the results, what exactly can they see, what do they already know, and what are they trying to learn? Keep those in mind and it’s easier to place each paper in the table.

ScenarioAttacker’s ObservationPossible Additional Knowledge
User of a classification serviceFinal label, confidence, or full probability vectorCandidate samples and their true labels, data from the same distribution
Person obtaining model filesParameters, structure, intermediate representations, or gradientsTraining pipeline, preprocessing methods, partial training data
Federated learning serverModels from each round, client updates, or aggregation resultsClient identities, participation records, optimizer configurations
Collaborating clientsReceived global model and their own local updatesData they hold, some other publicly available information

From the word black box alone I’d be tempted to assume two papers are using the same interface. Look at what actually comes back. A single label and the full list of class probabilities give someone very different amounts of information.

Another detail worth checking is whether the person observing the results can change what happens. An honest-but-curious participant follows the protocol and analyzes what it returns. An active adversary may also alter the model, participant selection, or other steps. I check which of those the security result assumes.

What Are the Different Attack Types Asking?

Attack TypeObjectiveObjects Not to Confuse
Membership InferenceDetermine whether a candidate sample participated in trainingDoes not equal recovering the full content of that sample
Attribute InferenceInfer sensitive attributes of a sample, user, or training setDoes not necessarily require knowing exact membership status
Model Inversion / Gradient ReconstructionRecover input information from model outputs, representations, or gradientsRepresentative images of a class are not necessarily real training samples
Training Data ExtractionRecover actual training content from the modelDistinguish between ordinary generation, similar content, and verbatim reproduction

When I see an experiment labeled an attack, I look at what it’s measuring at the end. Model theft goes after the model’s capabilities or parameters. Adversarial examples are about changing predictions. For the questions here, we’re trying to understand what the model can reveal about its training data.

Membership Inference: Why Do Predictions Reveal Training Identity?

A Simple Loss Threshold Attack

Let the candidate sample be $(x,y)$. The attacker can obtain the target model’s prediction and knows its label. One of the simplest approaches is to compute the loss and make a judgment based on a threshold:

$$ \widehat m(x,y)=\mathbf 1\{\ell(f_w(x),y)\le\tau\}, $$

where $m=1$ denotes a training set member. This attack relies on an empirical signal: models often fit training samples better, so members may have lower loss.

Think of two easy cases. A sample the model has never seen can still have low loss if it’s easy to classify. A hard training sample can have high loss. To measure the rule itself I’d use separate data for calibrating the threshold and for reporting the result.

Shokri and colleagues give us a way to check that intuition. They used shadow models and a classifier to look for differences between member and non-member predictions, and demonstrated a risk in their black-box setting. With another model or dataset, I’d check whether those differences are still there. 1

Why Is Average Accuracy Insufficient?

Definition

$$ \operatorname{TPR}=P(\widehat m=1\mid m=1), \qquad \operatorname{FPR}=P(\widehat m=1\mid m=0). $$

I start with the denominators when reading these two abbreviations. TPR counts the share of actual members we find. FPR counts the share of non-members we call members by mistake. Suppose the member share among the candidates is $\pi$. Of everything labeled a member, how much actually is?

$$ P(m=1\mid\widehat m=1)= \frac{\pi\operatorname{TPR}} {\pi\operatorname{TPR}+(1-\pi)\operatorname{FPR}}. $$

It’s easier to picture with 10,000 candidates. Take $\pi=1%$, a TPR of 80%, and an FPR of 1%. That gives 80 actual members detected and 99 non-members flagged by mistake. Put both counts in the denominator and precision comes out to about 44.7%.

When someone reports accuracy with half members and half non-members, I ask whether that split matches the setting we’re interested in. Carlini and colleagues’ LiRA work also looks at how many members are detected at low false-positive rates, for example TPR at 0.1% FPR. 2

There’s one more issue when measuring a false-positive rate that low. We need enough non-members to see it. If we have a few hundred and happen to get no false positives, all we’ve observed is zero in that run. Check the sample count before reporting a stable estimate at the one-in-a-thousand scale.

Overfitting Is Not the Only Explanation

Overfitting can give us a membership signal. Even when the average training and test losses are close, I’d go back and look at the unusual records. Heavily repeated, rare, or anomalous ones can get hidden in the average.

For the comparison I’d first see whether the two groups came from comparable sources. Match the classes, preprocessing, and collection process as closely as possible. If those differ, check whether the classifier is finding membership or simply telling the data sources apart.

Attribute Inference and Model Inversion

For attributes the question gets a little narrower. Someone may already know a few features and want to infer one more from the output. Or they may be looking at a client’s update to infer something about the distribution on that client.

I look separately at how accurate the inference is and whether the model remembered a particular record. Inferring a sensitive attribute from public features can already affect privacy. Showing that the model memorized this person’s training record takes more evidence.

When looking at a model inversion image I first ask what it matches. It may be very convincing to the classifier, or it may actually match a training record. To claim the latter, compare it with the training records and state the matching criteria and reconstruction quality.

Why Gradients Might Leak Inputs?

Analytical Example with a Single-Sample Linear Model

Let’s start with the simplest model:

$$ f_{w,b}(x)=w^{\mathsf T}x+b, \qquad \ell=\frac12(f_{w,b}(x)-y)^2. $$

Let the residual be $r=f_{w,b}(x)-y$, then

$$ g_w=\nabla_w\ell=rx, \qquad g_b=\frac{\partial\ell}{\partial b}=r. $$

If an attacker observes these two single-sample gradients and $g_b\ne0$, they can divide coordinate by coordinate:

$$ x=\frac{g_w}{g_b}. $$

The original input was not uploaded, yet the gradients are sufficient to recover it. Below, we verify this using only three dimensions of synthetic data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import numpy as np

x = np.array([0.2, 0.7, -0.4])
w = np.array([0.3, -0.2, 0.5])
b, y = 0.1, 1.0

residual = w @ x + b - y
weight_gradient = residual * x
bias_gradient = residual
assert bias_gradient != 0

reconstructed = weight_gradient / bias_gradient
assert np.allclose(reconstructed, x)
print(reconstructed)

This is not a formula that holds for all models or all batch settings. If $g_b=0$, division is not applicable; if one observes the average gradient of multiple samples, the numerator and denominator become weighted sums, and typically one cannot recover each input using this ratio.

From Analytical Recovery to Gradient Matching

For neural networks, reconstruction can be understood as searching for a set of candidate inputs whose generated gradients are close to the observed gradients:

$$ \min_{\widetilde x,\widetilde y} \left\|\nabla_w\ell(f_w(\widetilde x),\widetilde y)-g_{\mathrm{obs}}\right\|^2 +\lambda R(\widetilde x). $$

where $R$ represents a prior or regularization term; this expression is merely illustrative for the single-sample case. Deep Leakage from Gradients demonstrates the possibility of recovering training inputs via gradient matching; subsequent work, Inverting Gradients, used direction-dependent objectives and stronger optimization strategies to further investigate image reconstruction. 3, Inverting Gradients4

When I look back at a reconstruction result, I check what the experiment actually provided. Batch size, model, parameters, known labels, preprocessing, normalization, and whether several updates are visible can all matter. A failed reconstruction is an observation about that setup. A strict privacy guarantee after changing batch size, compression, or clipping still needs its own analysis.

Training Data Extraction: Do Models Really Recite the Original Text?

If the question becomes whether we can get the actual content back, we have to go beyond membership. When a language model generates something familiar, I ask whether it’s a common phrase, public material, or an exact passage we can match to a training record.

Carlini and colleagues did that comparison, checking text returned by a model against training text. Results that can be matched to records show the risk that comes with memorization. 5

In a controlled experiment we can use a unique synthetic canary with no real sensitive information. Record whether it entered training, how often it appeared, and what the model produced. I’d read the result in light of those conditions. Real-data risk takes an assessment of its own.

This brings us back to the first question. If we see no verbatim repetition, we can still ask what someone might infer about membership, attributes, or other information. Those outcomes have to be evaluated separately.

Why Federated Learning Does Not Automatically Solve Privacy Issues?

Back to federated learning. Moving training to clients can reduce how much raw data has to be collected centrally. Then I’d look at what they send back. Those updates can still carry information about their local data.

I first check whether the server gets one client’s update or a combined result. Seeing just the aggregate changes what’s visible. Then look at who participated, possible collusion, information across rounds, and whether the server can change the training conditions.

For secure aggregation, follow what the server sees. Under the protocol’s assumptions it gets the aggregate, while the individual plaintext inputs are hidden during computation. How much the resulting aggregate or model says about an individual is the next question to ask. 6

I keep track of which inputs secure aggregation hides during computation and how much influence DP allows a protected unit to have on the released result. The two can be used together. For an implementation I’d check the actual protocol and its assumptions on participation, dropouts, and collusion.

One related detail from the Dirichlet non-IID split7 is that some clients can have very few samples or highly concentrated labels. With that setup I’d also check what’s observed and how the test was run before interpreting the privacy result.

What Do Common Defenses Protect?

MethodPrimary EffectBoundaries to Retain
Data minimization, sensitive field handling, and deduplicationReduces sensitive content entering the system before training and opportunities for repeated memorizationCannot guarantee by itself that remaining data will not leak
Training measures such as regularization and early stoppingImproves generalization and may reduce some membership inference signalsGood average generalization does not equate to per-sample privacy guarantees
Reduce probability outputs, query limits, and access controlLimit attackers’ observation and invocation capabilitiesMust cover model files, logs, and other accessible outputs
Clipping, quantization, compression, or larger batch sizesChange the information carried by updates and the difficulty of attacksCannot claim a DP guarantee without privacy analysis
Secure aggregation, encrypted computation, or trusted execution environmentsProtect the computation process under specified trust and protocol assumptionsReleased outputs may still enable inference
Differential PrivacyLimit output distribution changes caused by altering protected unitsAdjacency relations, sampling methods, and total privacy budget must be explicitly defined

DP-SGD is not just about adding a bit of noise to gradients

Follow one update when reading DP-SGD. Clip each sample’s gradient first, aggregate the result, then add calibrated noise. The expression below puts those steps together.

$$ \overline g_i=\frac{g_i}{\max(1,\|g_i\|_2/C)}, \qquad \widetilde g=\frac1{|\mathcal B|} \left(\sum_{i\in\mathcal B}\overline g_i+\xi\right), \qquad \xi\sim\mathcal N(0,\sigma^2 C^2 I). $$

In the expression, $C$ is the clipping threshold, $\sigma$ is the noise multiplier, and $\mathcal B$ is the batch. If you’re asking what $(\varepsilon,\delta)$ comes out to, first pin down the neighboring datasets, the sampling mechanism, and the privacy accountant. 8

When reading the code I check where clipping happens and which results are exposed. Moving clipping from individual samples to a whole batch changes the sensitivity and accounting. If unnoised gradients are also released, account for that view separately. Check what the optimizer, checkpoints, data-dependent tuning, and additional released statistics reveal too.

With federated data, remember how many records one user can contribute. To protect the user, control the effect at the level of their contribution and account for how they participate and what gets released. If all I have is a record-level parameter, I go back and check which unit it covers.

Patterns in the population and public information can still support inference. I go back and ask what change the neighboring-dataset definition makes and what the observer already knows. I pick that up again in the post on information theory in privacy9.

How to Design a Persuasive Privacy Experiment?

If I were reading an experiment, I’d first write down what’s exposed and what unit is protected. Are we measuring the risk from final predictions or from training gradients? Is the claim about one record or all the records from a user?

Then I’d look at the numbers. For membership inference I’d find TPR at low FPR and check the assumed member prevalence. For reconstruction I’d check the match to the actual inputs, the errors, and the success rate. For extraction I’d count how many matches can be verified. A good-looking picture helps me follow one example. The full set of samples needs those numbers too.

I’d also check that calibration and final evaluation are separate and that the two groups have comparable distributions. The seeds for training, attack optimization, and splitting can change the result. Along with the average I’d look for variation and failures.

For a defense I’d look at several reasonable attack baselines and ask whether they know the defense. An attack failing in this run is one empirical result. For a theoretical guarantee, I’d go to the proof or the privacy accounting from a trusted implementation.

Conclusion

At the end I’d ask the opening question a little more precisely. Are we worried about someone learning that a sample was in training, inferring an attribute, or getting the input or original text back? Once we name the secret, we know which observations to look at.

Next time I hear that the raw data stays local, I want to ask which data we can leave out from the start, which intermediate results need to stay hidden, and how much one record can influence what we release. Then I can go back to the threat model and the experiments and see which step each guarantee covers.

References and Further Reading