| homeworld | species | name |
|---|---|---|
| Naboo | Gungan | Jar Jar Binks |
| Naboo | Droid | R2-D2 |
6 Disclosure Risk Metrics
NoteOverview
In this chapter, you will learn about:
- The different types of disclosure risk metrics used to evaluate synthetic data,
- How to apply these metrics through both conceptual questions and hands-on computational exercises, and
- Key considerations and best practices when interpreting and using these metrics in real-world contexts.
6.1 Evaluating Synthetic Data Privacy
6.1.1 Motivations for Privacy Evaluation
TipDefinition: Disclosure Risk Metrics
Disclosure risk metrics are metrics that measure the degree to which a statistical output reveals information about its inputs, i.e., individual records in confidential data.
Disclosure risk refers to the ability to infer information about data subjects’ confidential datasets based on the synthetic data. Measures of disclosure risk depend on the type of inference someone is attempting to make, the amount of additional information we assume is available about the target individuals and the population, and the computational power available for the attack.
ImportantIMPORTANT: Fundamental law of information recovery
All non-trivial statistical outputs induce some non-zero disclosure risks. The only way to ensure perfect disclosure risk protection is to not release data.
Synthetic data does not inherently guarantee disclosure protection; risk depends on how the data are generated. For example, a “synthetic” with 99% identical records to the confidential data still poses high disclosure risk.
Disclosure risk metrics evaluate how statistical outputs may reveal information about individuals.
- All non-trivial outputs carry some disclosure risk.
- Zero risk is only possible by not releasing any data or information.
Disclosure risk is context-specific and multidimensional.
- No universal metric or method fits all scenarios.
- Curators may prioritize different risks (e.g., inferring record presence, assessing individual contributions).
- No synthetic data solution can fully eliminate all dimensions of disclosure risk.
Code and documentation used to generate synthetic data must be reviewed for disclosure risks.
- Greater transparency increases risk.
- Review all supplemental materials (e.g., code, data collection details) carefully.
- Avoid publishing:
- Data collection frames and response rates (can reveal sample demographics).
- Pseudo-random number generation code (can enable replication and re-identification).
The mere presence of synthetic data does not automatically ensure adequate disclosure protection. Simply put, synthetic data can be generated in many different ways, but synthetic data does not eliminate all possible disclosure risk harms. The level of disclosure risk protection afforded by synthetic data is a function of how the synthetic data are generated, not the mere fact that the resulting dataset is named a “synthetic” dataset. For example, consider a “synthetic” dataset where 99% of records are identical to the confidential dataset. This dataset may be labeled “synthetic,” but it still poses significant disclosure risks.
Disclosure risk metrics measure the extent to which a statistical output may reveal information about individual records in confidential data. A key consideration when selecting metrics for disclosure risk evaluation is understanding that all non-trivial statistical outputs carry some level of disclosure risk. The only way to guarantee zero disclosure risk is to not release any data or information from the confidential data.
In other words, disclosure risk is contextually specific and multidimensional, by necessity, there are no standardized “one-size-fits-all” metrics or evaluation techniques for disclosure risk measurement. For example, data curators may care about the ability to infer the presence or absence of particular records more than others. Similarly, data curators may wish to assess the influence of certain data curator’s contributions on a resulting synthetic dataset. By construction, no synthetic data solution can ever address every dimension of this problem simultaneously and with certainty.
Additionally, the code and documentation used to generate synthetic data should be reviewed independently for disclosure risks. In general, greater methodological transparency increases disclosure risk for all data products. Therefore, all code, documentation, and other supplemental materials related to data product publication should be reviewed for potential disclosure. For example, information about data collection frames and response rates can be used to reverse engineer sample demographics and should be published with discretion. Similarly, code details related to pseudo-random number generation should be excluded to prevent replication of randomness and potential re-identification.
6.1.2 Mapping Synthetic Data onto Disclosure Risk Harms
Randomization in synthetic data allows us to consider inferential harms. In other words,
- What can one infer from synthetic data about confidential data records?
- With what certainty could one draw said inference?
Recall that disclosure risk assessments typically focus on three broad kinds of inferential harms:
- Identity inference: can I infer whether a particular record in a dataset belongs to a specific person?
- Example 1: “Does this record in a dataset of government employees describe me?”
- Example 2: “Does this tax record have the same social security number as me?”
- Attribute inference: can I infer the value of a particular attribute for a record in the confidential dataset?
- Example 1: “Does the dataset I’m in let users infer when I graduated from UC Berkeley?”
- Example 2: “Can the synthetic data based on my tax records enable accurate inferences about my income?”
- Membership inference: can I infer the presence or absence of an individual in a particular dataset?
- Example 1: “Does the data reveal whether I was ever a student in the UC school system?”
- Example 2: “Can you infer whether or not my information is in a database of current Medicaid enrollees?”
TipDefinitions: Direct and Indirect Identification
Direct identifiers are data elements that a priori correspond to a natural person, like names, addresses, unique government identifiers, etc.
Indirect identifiers are any other data elements that, in combination, could be used to isolate or single out an individual. Sometimes, sets of indirect identifiers are referred to as quasi-identifiers.
6.1.3 Identity Inference and Indirect Identifiers
- Fully synthetic data typically includes only indirect identifiers, not direct identifiers.
- Identity inference usually requires a mapping between indirect and direct identifiers (e.g., demographic data linked to SSNs).
- Linkages using such mappings can lead to identity disclosure.
- For example, if a mapping exists between demographic traits and SSNs, one could potentially infer a respondent’s SSN.
- Privacy evaluations for fully synthetic data often assess combinations of indirect identifiers as proxies for direct identification.
6.1.4 Quasi-Identifiers
- Membership inference using quasi-identifiers (indirect identifier sets) is the most common disclosure risk assessment for randomization-based PETs (e.g., synthetic data, differential privacy).
- Choosing quasi-identifier sets involves trade-offs:
- Smaller sets: More plausible attackers, harder membership inference.
- Larger sets: Less plausible attackers, easier membership inference.
- Attribute inference depends on membership inference:
- It is most relevant when a record is likely to belong to the confidential dataset.
- Otherwise, attribute inference can apply to any arbitrary record.
NoteNote: Two Key Disclosure Risk Metrics for Fully Synthetic Data
- Attribute inference: measures how well someone can infer information about data subjects within the confidential dataset.
- With sufficient background knowledge, successful attribute inference may also indicate high data utility.
- High-utility synthetic models often capture real-world relationships, not just those in the confidential sample.
- Membership inference: measures how well someone can infer the presence or absence of data subjects within the confidential dataset.
6.2 Identity Disclosure Metrics
TipDefinition: Identity Disclosure Metrics
Identity disclosure metrics evaluate how often we correctly re-identify confidential records in the synthetic data. i.e., can I infer whether a particular record in a dataset belongs to a natural person?
Note: These metrics require substantial assumptions about attacker information.
WarningWARNING: Not for fully synthetic data!
For fully synthetic datasets, there is no one-to-one relationship between individuals and records so identity disclosure risk is ill-defined. In other words, identity disclosure is only considered for partially synthetic data, because the synthetic records correspond to the same set of records in the confidential data, resulting in a one-to-one mapping between the datasets. In fully synthetic data, no records correspond to any records in the confidential data, and therefore an identify disclosure will not be possible and is not considered.
NoteNote: Evaluating risk through simulated attacks
Most disclosure risk metrics involve data stewards or maintainers simulating attacks on their own synthetic data to assess how successfully individuals can be identified.
6.2.1 Basic Matching Approaches
- Applicable to partially synthetic data and traditional PETs with one-to-one record mapping.
- Begins with assumptions about attacker knowledge (e.g., access to external public data).
- For each confidential record, attacker identifies potential matches in the output data using external variables.
- Some matching methods include:
- Distance-based algorithms.
- Probability-based algorithms.
- Matching can be exact or relaxed:
- Continuous variables within a radius.
- Adjacent categorical variables.
- Re-identification accuracy is evaluated using various metrics.
When working with partially synthetic data or other traditional PETs that maintain one-to-one mapping of records, basic matching approaches can help assess identity disclosure risk. The process begins by making assumptions about the attacker’s knowledge; typically external, publicly accessible data. For each confidential record, the attacker identifies a set of records from the output data they believe may contain the target (i.e., potential matches), using external variables as matching criteria. Matching can be performed using distance-based or probability-based algorithms, which may rely on exact variable matches or relaxed criteria, such as matching continuous variables within a certain radius of the target record or matching adjacent categorical variables. Finally, the accuracy of the re-identification process is evaluated using a variety of metrics.
6.2.2 Matching Example
As a simple example, imagine a data attacker has access to the following external data:
And imagine that the partially synthetic released data looks like this:
| homeworld | species | skin_color |
|---|---|---|
| Tatooine | Human | fair |
| Tatooine | Droid | gold |
| Naboo | Droid | white, blue |
| Tatooine | Human | white |
| Alderaan | Human | light |
| Tatooine | Human | light |
Note that the released partially synthetic data does not have names. But using some basic matching rules in combination with the external data, an attacker is able to identify the following potential matches for Jar Jar Binks and R2D2, two characters in the Starwars universe:
| homeworld | species | skin_color |
|---|---|---|
| Potential Jar Jar matches | ||
| Naboo | Gungan | orange |
| Naboo | Gungan | grey |
| Naboo | Gungan | green |
| Potential R2-D2 Matches | ||
| Naboo | Droid | white, blue |
And since we are the data maintainers, we can take a look at the confidential data and know that the highlighted rows are “true” matches.
| homeworld | species | skin_color |
|---|---|---|
| Potential Jar Jar matches | ||
| Naboo | Gungan | orange |
| Naboo | Gungan | grey |
| Naboo | Gungan | green |
| Potential R2-D2 Matches | ||
| Naboo | Droid | white, blue |
These matches above are counted in various ways to evaluate identity disclosure risk. Below are some of those specific metrics. Generally for a good synthesis, we want a low expected match rate and true match rate, and a high false match rate.
Expected Match Rate: On average, how likely is it to find a “correct” match among all potential matches? Essentially, the expected number of observations in the confidential data expected to be correctly matched by an intruder.
Higher expected match rate = higher identification disclosure risk.
The two other risk metrics below focus on the subset of confidential records for which the intruder identifies a single match.
In our example, this is \(\frac{1}{3} + 1 = 1.333\).
True Match Rate: The proportion of true unique matches among all confidential records. Higher true match rate = higher identification disclosure risk.
Assuming there are 100 rows in the confidential data in our example, this is \(\frac{1}{100} = 1\%\).
False Match Rate: The proportion of false matches among the set of unique matches. Lower false match rate = higher identification disclosure risk.
In our example, this is \(\frac{0}{1} = 0\%\).
6.3 Attribute Disclosure Risk Metrics
TipDefinition: Attribute Disclosure Metrics
Attribute disclosure occurs if the data intruder determines new characteristics (or attributes) of an individual based on the information available through public data or statistics to infer the confidential value of target record(s) in the confidential data.
For example, if a dataset shows that all people age 50 or older in a city are on Medicaid, then the data adversary knows that any person in that city above age 50 is on Medicaid. This information is learned without identifying a specific individual in the data!
NoteNote: For Both Partially and Fully Synthetic Data
Privacy researchers consider attribute disclosure for both partially synthetic and fully synthetic data.
TipDefinition: Attribute Inference Test
Attribute inference tests measure how well someone can infer sensitive characteristics about confidential data using the synthetic data (and possibly external data).
Recall, attribute inferences can be either an indicator of statistical utility or a source of potential harm.
6.3.1 Predictive Accuracy
TipDefinition: Predictive Accuracy
Predictive accuracy measures how well an attacker can learn about attributes in the confidential data using the synthetic data (and possibly external data).
We can build a predictive model using the synthetic data to make predictions on the confidential data.
Key variables: Variables that an attacker already knows about a record and can use to match.
Target variables: Variables that an attacker wishes to know more or infer about using the synthetic data.
Pick a sensitive variable in the confidential data and use the synthetic data to make predictions. Evaluate the accuracy of the predictions.
6.3.2 Copy Protection
TipDefinition: Copy Protection Metrics
Copy protection metrics measure how often the synthesizer memorizes or inadvertantly duplicates confidential records.
Exact Copies
Checking for exact copies is a useful start but runs into two issues:
- You have to make assumptions about the variables used for linking. If you use too many variables, then the probabilities of exact matches approaches zero.
- Copying common records may be perfectly normal for a synthesizer.
We use three adjustments:
- Systematically add or drop linking variables to see how sensitive the metric is to the selection of linking variables.
- Only count joins on unique confidential records or even unique synthetic records to unique confidential records.
- Join on all records but weight the metric by the inverse frequency (or inverse frequency squared) of the observation in the confidential data.
DCR
Another approach is to relax the assumption that copying needs to be an exact match and use distance-based calculations.
- Distance to Closest Record (DCR): Measures distance between each real record (\(r\)) and the closest synthetic record (\(s_i\)), as determined by a distance calculation.
- Many common distance metrics are used in the literature, including Euclidean distance, cosine distance, Gower distance, or Hamming distance. We frequently use Gower distance.
- The goal of this metric is to easily expose exact copies or simple perturbations of the real records that exist in the synthetic dataset.
WarningWARNING: Computational Cost for Distance Calculations
Computation time needed for distance calculations rapidly increases with more observations and/or more variables.
- Note that having DCR = 0 doesn’t necessarily mean a high disclosure risk because in some datasets the “space” spanned by the variables in scope is relatively small.
6.4 Membership Inference Tests
TipDefinition: Memebership Inference Test
Membership inference (MI) tests measure how well someone can infer the presence or absence of data subjects within the confidential dataset used to train the synthetic data.
- MI tests the ability to distinguish between two hypotheses:
\[ H_0: \text{ record i is not in the confidential dataset, } \qquad H_1: \text{otherwise.} \]
The goal of MI is to make it intentionally hard to distinguish between \(H_0\) and \(H_1\).
Key implementation choices:
- What constitutes a record? Ideally, the full set of confidential records; MI results may vary by record.
- Test statistic selection. Preferably statistics that fully reflect the synthesis process; may use incomplete or external data.
- Assumed relationship: Ideally, the exact mathematical link between synthetic and confidential data; often modeled or approximated.
Distance metrics help define “close enough” thresholds:
- Examples: edit distance for text, numeric distance for continuous variables.
- Try different thresholds to see how results vary.
MI is critical when membership itself is sensitive (e.g., HIV status, homelessness).
Especially useful for fully synthetic data, where identity and attribute disclosure metrics may not apply.
Assumes attacker has access to part of the confidential data and tries to detect if specific records were used in synthesis.
Data maintainers can evaluate attacker success using true/false positives and negatives.
MI is a method used to evaluate whether an attacker can distinguish between two hypotheses:
\[ H_0: \text{ record i is not in the confidential dataset, } \qquad H_1: \text{otherwise.} \]
The goal of MI is to make it intentionally difficult to differentiate between these two scenarios, thereby protecting the confidentiality of individuals in the dataset.
Implementing MI involves several key decisions. First, it requires defining what constitutes a “record,” ideally considering the full set of confidential records, though results may vary across individual records. Second, a test statistic must be chosen to differentiate between \(H_0\) and \(H_1\). Ideally, this statistic should provide complete modeling information about the synthesis process, though it may be based on incomplete data or supplemented with external sources. Third, assumptions must be made about the relationship between the test statistic and the confidential data. Ideally, this relationship is known exactly, but in practice, it is often modeled or approximated.
A critical part of MI implementation is determining what counts as “close enough” when comparing records. This threshold is usually defined by a custom distance metric, such as edit distance for text variables or numeric distance for continuous variables. It is often useful to experiment with different distance thresholds to observe how the results change.
MI is particularly important when membership in a dataset is itself sensitive. For example, datasets involving individuals who are HIV positive or have experienced homelessness. It is also especially relevant for fully synthetic data, where traditional identity and attribute disclosure metrics may not be applicable. MI assumes that an attacker has access to a subset of the confidential data and seeks to determine whether specific records were used to generate the synthetic data. Because data maintainers know the true membership status of each record, they can evaluate the accuracy of the attacker’s guesses using metrics such as true positives, true negatives, false positives, and false negatives.
6.4.1 Empirical Membership Inference via Simulated Membership Inference Attacks
We can empirically measure the quality of membership inference tests by simulating the test itself.
WarningWARNING: Infeasibility of ideal membership inference evaluation
In an ideal scenario, we would:
- Evaluate membership inference for each record in the confidential dataset.
- Generate synthetic data with and without the target record included in the confidential data.
- Use the best possible test statistic to perform the inference.
However, this approach is likely to be computationally infeasible!
6.4.2 syntheval snippet: Nearest-Neighbor Membership Inference Test
disc_mit() implements a nearest-neighbor membership inference test.
- Combine the confidential and holdout data, which becomes a blended dataset.
- Calculate the distance from each observation in the blended data to the synthetic data.
- Pick a threshold. Every blended observation with a distance below the threshold is assumed to be in the training data for the synthesizer.
- Evaluate the accuracy of this model. Accuracy close to chance level (e.g., 50%) is good, since it means an attacker can’t reliably distinguish members from non-members.
Code
eval_data_lr <- eval_data(
conf_data = acs_conf,
synth_data = acs_lr_synths[[1]],
holdout_data = acs_holdout
)
disc_mit(eval_data_lr)$precision
[1] 0.512
$recall
[1] 0.512
$auc
[1] 0.517971
$conf_mat
Truth
Prediction training holdout
training 512 488
holdout 488 512Code
eval_data_hr <- eval_data(
conf_data = acs_conf,
synth_data = acs_hr_synths[[1]],
holdout_data = acs_holdout
)
disc_mit(eval_data_hr)$precision
[1] 0.511
$recall
[1] 0.511
$auc
[1] 0.5256895
$conf_mat
Truth
Prediction training holdout
training 511 489
holdout 489 5116.5 Holdout Dataset
TipDefinition: Holdout Dataset
Membership inference tests and copy protection metrics are informative but lack context. When possible, create a holdout dataset similar to the training data. Then calculate membership inference tests and copy protections metrics replacing the synthetic data with the hold out data. The results are useful for benchmarking the original membership inference tests and copy protection metrics.
Holdout data is useful for most utility and disclosure risk metrics. Holdout data should ideally come from the same population as the GSDS. For example, holdout data can be a separate sample from the confidential data. Alternatively, holdout data can come from a very similar population. Holdout data is useful for DCR because it gives a sense of the naturally occurring distances between observations.
We also summarize the ratio of the closest and second closest distance. High ratios when DCR is low suggest that a synthetic observation is close to one, and only one, observation.
ImportantImportant: Decision Point for Holdout Data
How much of your data will you reserve as holdout data before the synthesis process starts?
6.6 Exercises
6.6.1 Exercise 1: Is Attribute Inference a Privacy Violation? (Conceptual)
Suppose person 1’s data are in a dataset, and person 2’s data are not in a dataset. This dataset yields a strong correlation between variables A and B. Using this, someone who knows person 1’s or person 2’s variable A could infer their variable B.
Is this a privacy problem?
Suppose person 1’s data are in a dataset, and person 2’s data are not in a dataset. This dataset yields a strong correlation between variables A and B. Using this, someone who knows person 1’s or person 2’s variable A could infer their variable B.
Is this a privacy problem?
Privacy scholars STRONGLY disagree over this topic!
Technical-oriented privacy scholars, typically computer scientists, statisticians, and engineers, generally argue the answer is NO.
Policy-oriented privacy scholars, typically lawyers, social scientists, and policymakers, generally argue the answer is YES.
The reason these communities disagree is that, depending on the context, attribute inference can be either an indicator of statistical utility or a source of potential harm.
6.6.2 Exercise 2: Simulated Membership Inference Attack (Calculation)
Suppose you had a confidential dataset with \(100,000\) records, each of which required fitting and sampling from \(100\) joint and/or conditional models. Suppose you wanted to follow the ideal procedure listed above for MI. Excluding the original synthesis, how many different models would you need to fit to perform the ideal MI?
Suppose you had a confidential dataset with \(100,000\) records, each of which required fitting and sampling from \(100\) joint and/or conditional models. Suppose you wanted to follow the ideal procedure listed above for MI. Excluding the original synthesis, how many different models would you need to fit to perform the ideal MI?
(\(10^5\) synthetic data processes, one per record) * (\(10^2\) models per process) = \(10^7\) models.
How do we make this problem tractable?
MI is most critical to assess for…
- Records with outlying characteristics.
- Records corresponding to smaller demographic sub-populations.
Why do this?
- Different groups have different privacy-utility trade-offs.
- Less common records are likely to have MI results closer to formal upper bounds.
When theoretical assessments of Type I (i.e., rejecting the null hypothesis when it is true) and Type II (i.e., not rejecting the null hyptohtehsis when it is false) error are impractical, use holdout data to empirically estimate them.
When possible, create a holdout dataset similar to the training data. In other words…
- Holdout data refers to data not used in the creation of synthetic data.
- Ideally, holdout data should be similar in distribution to the confidential data (i.e., internally heterogeneous but mutually homogeneous, like cluster sampling).
When theoretically ideal test statistics are infeasible, treat membership inference as a black-box classification problem.
- Depends on access to holdout data and the ability to model differences between confidential and holdout records.
- Models range significantly in complexity.
6.6.3 Exercise 3: Disclosure Metrics (Conceptual)
In 2018, the U.S. Census Bureau performed a disclosure risk assessment of its 2010 Decennial Census published statistics. Without going into the complete details, the following analyses were performed:
{#fig-census-reid fig-align=“center” width = “100%”}
1. First, Census used published 2010 statistics to create a collection of quasi-identifiers for U.S. residents that plausibly completed the 2010 Decennial Census. 2. Next, Census purchased third-party PII with demographic information and attempted to link these statistics to the plausible set of quasi-identifiers from Step 1.
What kinds of inference are being performed in steps 1 and 2?
In 2018, the U.S. Census Bureau performed a disclosure risk assessment of its 2010 Decennial Census published statistics. Without going into the complete details, the following analyses were performed:
{#fig-census-reid fig-align=“center” width = “100%”}
1. First, Census used published 2010 statistics to create a collection of quasi-identifiers for U.S. residents that plausibly completed the 2010 Decennial Census. 2. Next, Census purchased third-party PII with demographic information and attempted to link these statistics to the plausible set of quasi-identifiers from Step 1.
What kinds of inference are being performed in steps 1 and 2?
- Membership inference.
- Identity inference.
6.6.4 Exercise 4: Disclosure Metrics (Conceptual)
| Name | Year | Elective |
|---|---|---|
| Adam | 2009 | Chorus |
| Betsy | 2010 | Band |
Attacker Information
| Year | Elective | Synthetic SAT |
|---|---|---|
| 2008 | Chorus | 1100 |
| 2008 | Chorus | 1420 |
| 2009 | Chorus | 900 |
| 2009 | Band | 1100 |
| 2010 | Band | 1420 |
| 2010 | Band | 900 |
| 2010 | Band | 1200 |
Partially Synthetic Data
- Are there any matches for Adam?
- Are there any matches for Betsy?
- What risks are created by the release?
| Name | Year | Elective |
|---|---|---|
| Adam | 2009 | Chorus |
| Betsy | 2010 | Band |
Attacker Information
| Year | Elective | Synthetic SAT |
|---|---|---|
| 2008 | Chorus | 1100 |
| 2008 | Chorus | 1420 |
| 2009 | Chorus | 900 |
| 2009 | Band | 1100 |
| 2010 | Band | 1420 |
| 2010 | Band | 900 |
| 2010 | Band | 1200 |
Partially Synthetic Data
- Are there any matches for Adam?
Using Year and Elective as key variables, Adam has a unique match (highlighted in pink)
- Are there any matches for Betsy?
Using Year and Elective as key variables, Betsy has a three matches (highlighted in yellow)
- What risks are created by the release?
It is tough to say without context but here are a few considerations:
- Is SAT easily observable outside of the data?
- Are the values of
Synthetic SATclose to the true values forSAT? - Are
SATandSynthetic SATlikely to be close under random guessing because it has low sample variance?
6.6.5 Exercise 5: Disclosure Metrics (Conceptual)
What is an example of a utility metric you might design to ensure your synthetic data are meeting the needs of those users well?
Suppose you have a colleague that generated fully synthetic data and wants to measure identity disclosure risk. Your colleague proposes using the expected match rate, false match rate, and true match rate to evaluate how well the synthetic data protects the identity of people in the data. Why is their approach flawed and what disclosure metrics would you suggest instead?
What is an example of a utility metric you might design to ensure your synthetic data are meeting the needs of those users well?
- Is the metric you proposed above a general or specific utility metric?
- Specific Utility Metric
- General Utility Metric
Suppose you have a colleague that generated fully synthetic data and wants to measure identity disclosure risk. Your colleague proposes using the expected match rate, false match rate, and true match rate to evaluate how well the synthetic data protects the identity of people in the data. Why is their approach flawed and what disclosure metrics would you suggest instead?
These metrics apply only to partially synthetic data. Consider instead a metric like the membership inference test.