In this chapter, you will learn about:
We will assume the original, confidential, and GSDS are identical for this set of notes. In practice, it is necessary to pick the appropriate comparison dataset for evaluating the synthetic dataset.
Data utility, quality, accuracy, or usefulness is how useful or accurate the data are for research and analysis purposes.
Utility metrics are metrics that measure a synthetic dataset’s degree of usefulness for downstream data processing.
Why produce utility metrics?
Reason #1 describes what synthetic data can and cannot do as a mathematical description.
Reason #2 describes what synthetic data should or should not do as a policy choice for data curators.
A main goal of synthesis is to learn the data generation process of the population or superpopulation from an observed dataset while minimizing the amount of information learned about individual observations in that data. In this way, avoiding memorizing information about individual records aids both generalization and disclosure risk protections.
It is important to measure data utility to understand what synthetic data can and can’t do and to understand where the synthetic data fall in the privacy-utility trade-off. However, it is also important to understand that targeting what synthetic data should do is a policy choice for data curators.
Two broad categories of data use:
Statistical data use refers to using summaries to infer statistical features of an implicit or explicit population or superpopulation.
Administrative data use refers to using summaries to describe features of the data for direct use of the data.
Why does this distinction matter? Many policies and technologies attempt to draw hard lines between these two!
For example…
Confidential Information Protection and Statistical Efficiency Act (CIPSEA): “To ensure that information supplied by individuals or organizations to an agency for statistical purposes under a pledge of confidentiality is used exclusively for statistical purposes.”
Differential Privacy’s risk measures: “Statistical data uses rely on ‘data about you’ and DON’T concern privacy, whereas administrative data uses rely on ‘your data’ and DO concern privacy.”
The same summaries that enable statistical data uses will always enable some administrative data uses, and contextual evaluation is necessary to strike the right balance between the two.
General utility metrics measure differences between synthetic and confidential data independent of pre-specified use cases.
General utility metrics…
Example questions:
Example metrics:
Comparing univariate distributions is a basic approach to evaluating utility. For example, Figure 5.3 shows a comparison of means for tax variables for a synthetic file released by the IRS Statistics of Income division.
Categorical variables: frequencies, relative frequencies.
Numeric variables means, standard deviations, skewness, kurtosis (i.e., first four moments), percentiles, and number of zero/non-zero values.
For the univariate case, we could calculate the frequencies and relative frequencies of the categorical variables in the synthetic and confidential data. When the variables are numeric, we could compute the means, standard deviations, skewness, kurtosis, percentiles, and number of zero/non-zero values. We can also visually compare the results of the univariate distributions from the synthetic and confidential data using a histogram, density plots, or empirical cumulative distribution function (eCDF) plots.
The eCDF comparison of the synthetic data and confidential data is useful for identifying differences in both central values and the tails. In other words, discrepancies may arise not only in the central region but also in the tails; specifically where the eCDF approaches 0 or 1.
Woo et al. (2009b) propose the use of the eCDF as a global utility metric, which is particularly useful for univariate numeric variables. While the metric can be easily extended to multivariate distributions, implementing the eCDF estimation itself in multivariate cases is not straightforward. Therefore, the eCDF global utility metric is rarely used beyond univariate cases.
It is also useful to visually compare univariate distributions using histograms (Figure 5.4), density plots (Figure 5.5), or empirical cumulative distribution function plots (Figure 5.6). The following examples use the Palmer penguins data.
compare_penguins |>
select(
data_source,
bill_length_mm,
flipper_length_mm
) |>
pivot_longer(-data_source, names_to = "variable") |>
ggplot(aes(x = value, fill = data_source)) +
geom_histogram(alpha = 0.3, color = NA, position = "identity") +
facet_wrap(~ variable, scales = "free") +
scatter_grid()
compare_penguins |>
select(
data_source,
bill_length_mm,
flipper_length_mm
) |>
pivot_longer(-data_source, names_to = "variable") |>
ggplot(aes(x = value, fill = data_source)) +
geom_density(alpha = 0.3, color = NA) +
facet_wrap(~variable, scales = "free") +
scatter_grid()
compare_penguins |>
select(
data_source,
bill_length_mm,
flipper_length_mm
) |>
pivot_longer(-data_source, names_to = "variable") |>
ggplot(aes(x = value, color = data_source)) +
stat_ecdf() +
facet_wrap(~ variable, scales = "free") +
scatter_grid()
Consider the following two syntheses of x. Which synthesis do you think has higher utility?
Consider the following two syntheses of x. Which synthesis do you think has higher utility?
set.seed(20230710)
bind_rows(
synth1 = tibble(
x_conf = rnorm(n = 1000),
x_synth = rnorm(n = 1000, mean = 0.2)
),
synth2 = tibble(
x_conf = rnorm(n = 1000),
x_synth = rnorm(n = 1000, sd = 0.5)
),
.id = "synthesis"
) |>
pivot_longer(-synthesis, names_to = "variable") |>
ggplot(aes(x = value, color = variable)) +
stat_ecdf() +
facet_wrap(~ synthesis) +
scatter_grid()
Consider the following two syntheses of x. Which synthesis do you think has higher utility?
Both syntheses have utility issues; what do you think are the issues?
synth1 to be slightly higher utility than synth2 based on the large vertical distances between the lines for synth2.synth1 matches the variance of the confidential data but the mean is a little larger. synth2 matches the mean but has lower variance with fewer observations in the tails of the synthetic data.A synthetic dataset can do a great job of recreating every univariate or marginal distribution while failing to capture the joint distribution. The rest of these notes prioritize evaluating the relationships between variables.
Many analyses rely on relationships between variables. Reviewing pairwise relationships is challenging because it quickly becomes a high-dimensional problem. If a dataset has \(p\) variables, then there are \(\frac{p(p - 1)}{2}\) pairwise relationships in the data. Here, visualization and numeric summaries are important.
Correlation fit measures how well the synthetic dataset recreates the linear relationships between variables in the confidential dataset.
Correlation fit uses the lower triangle of correlation matrices for the synthetic data and confidential data and their difference.
Those differences are often summarized across all variables using L1 or L2 distance. Figure 5.7 shows the creation of a difference matrix. Let’s summarize the difference matrix using mean absolute error (MAE). This will give us a sense of how off the average correlation will be in the synthetic data compared to the confidential data.
\[MAE_{dist} = \frac{1}{n}\sum_{i = 1}^n |dist|\]
\[MAE_{dist} = \frac{1}{6} \left(|-0.15| + |0.01| + |0.1| + |-0.15| + |0.15| + |0.02|\right) \approx 0.0966667\] Note: one can alternatively substitute different correlation measures (for example, rank correlation) or different correlation matrix distances (for example, Frobenius norm).
Pearson’s correlation coefficient is ever present but is limited to numeric variables. There is less consensus about measures for quantifying relationships between categorical variables. Here, we will use relative mutual information.
Relative mutual information (RMI) measures the reduction in entropy (i.e., uncertainty) in one variable when observing another variable. It quantifies the relationship between two variables, generalizes to categorical variables, and is in the interval [0,1].
Relative mutual information fit measures how well the synthetic dataset recreates the relative mutual information between variables in the confidential dataset. The element-wise differences are evaluated, and the numeric summaries of the differences are calculated. Higher utility is demonstrated by the difference close to 0.
Let’s walk through a simple example using the Palmer penguins data.
The confidential RMI matrix shows the relationships between the three categorical variables in the confidential data.
species island sex
species 1.00 0.50 0.01
island 0.52 1.00 0.01
sex 0.01 0.01 1.00Synthetic RMI matrix shows the relationships between the three categorical variables in the synthetic data.
species island sex
species 1.00 0.45 0
island 0.49 1.00 0
sex 0.00 0.00 1The difference matrix shows that the confidential RMI matrix and synthetic RMI matrix are fairly similar.
species island sex
species 0.00 -0.05 -0.01
island -0.03 0.00 -0.01
sex -0.01 -0.01 0.00The mean absolute error (L1 norm) between the two measures is just 0.02.
Part 1: Calculate the correlation fit between the synthetic and confidential data. Fill in the blanks and run the code below.
penguins_conf <- read_csv(here::here("data", "penguins_synthetic_and_confidential.csv")) |>
filter(data_source == "confidential")
penguins_synth <- read_csv(here::here("data", "penguins_synthetic_and_confidential.csv")) |>
filter(data_source == "synthetic")
# The cor() function can take in a dataframe and compute correlations
# between all columns in the dataframe and spit out a correlation matrix
conf_data_corr <- cor(###)
synth_data_corr <- cor(###)
conf_data_corr <- conf_data_corr[lower.tri(conf_data_corr)]
synth_data_corr <- synth_data_corr[lower.tri(synth_data_corr)]
correlation_diff <- conf_data_corr - synth_data_corr
# Correlation fit is the sum of the sqrt of the squared differences between each correlation in the difference matrix.
cor_fit <- sum(sqrt( ### ^2))
cor_fitpenguins_conf <- read_csv(here::here("data", "penguins_synthetic_and_confidential.csv")) |>
filter(data_source == "confidential")
penguins_synth <- read_csv(here::here("data", "penguins_synthetic_and_confidential.csv")) |>
filter(data_source == "synthetic")
# The cor() function can take in a dataframe and compute correlations
# between all columns in the dataframe and spit out a correlation matrix
conf_data_corr <- cor(select(penguins_conf, where(is.numeric)))
synth_data_corr <- cor(select(penguins_synth, where(is.numeric)))
conf_data_corr <- conf_data_corr[lower.tri(conf_data_corr)]
synth_data_corr <- synth_data_corr[lower.tri(synth_data_corr)]
correlation_diff <- conf_data_corr - synth_data_corr
# Correlation fit is the sum of the sqrt of the squared differences between each correlation in the difference matrix.
cor_fit <- sum(sqrt(correlation_diff ^2))
cor_fit[1] 0.6178178Part 2: Compare the univariate distributions for mass and height in the confidential and synthetic data using density plots. Fill in the blanks and run the code below.
conf_data <- read_csv(here::here("data/lesson_03_conf_data.csv"))
synth_data <- read_csv(here::here("data/lesson_03_synth_data.csv"))
combined_data <- bind_rows(
"synthetic" = synth_data,
"confidential" = conf_data,
.id = "type"
)
# Create a density plot of the mass distributions
combined_data %>%
ggplot(aes(x = ###,
fill = type,),
position = "dodge",
color = "white") +
geom_density(alpha = 0.4)
# Create a density plot of the height distributions
combined_data %>%
ggplot(aes(x = ###,
fill = type,),
position = "dodge",
color = "white") +
geom_density(alpha = 0.4)conf_data <- read_csv(here::here("data/lesson_03_conf_data.csv"))
synth_data <- read_csv(here::here("data/lesson_03_synth_data.csv"))
combined_data <- bind_rows(
"synthetic" = synth_data,
"confidential" = conf_data,
.id = "type"
)
# Create a density plot of the mass distributions
combined_data %>%
ggplot(aes(x = mass,
fill = type),
position = "dodge",
color = "white") +
geom_density(alpha = 0.4)
# Create a density plot of the height distributions
combined_data %>%
ggplot(aes(x = height,
fill = type),
position = "dodge",
color = "white") +
geom_density(alpha = 0.4)
Discriminant based methods measure how well a predictive model can distinguish (i.e., discriminate) between records from the confidential and synthetic data. Simply put, the harder it is for the predictive model to distinguish records from one another, the higher the general utility of the synthetic data.
The basic idea is to combine (stack) the GSDS and synthetic data and see how well a predictive model distinguishes (i.e., discriminates) between synthetic observations and confidential observations.
Poor model performance in distinguishing records indicates high-utility synthesis.
Discriminant modeling involves:
General strategies:
Discriminant based models can assess the quality of synthetic data, which essentially tests how well a predictive model can distinguish between synthetic and confidential records. This approach assumes that both datasets are drawn from the same superpopulation, meaning they should reflect similar underlying distributions.
The basic idea is to combine (stack) the confidential data and synthetic data and see how well a predictive model distinguishes (i.e., discriminates) between synthetic observations and confidential observations. If the model struggles to distinguish between the two, this suggests the synthetic data closely mimics the confidential data, indicating a high quality data synthesis.
Modeling Techniques While logistic regression can be used for this binary classification task, more commonly used methods include: decision trees, random forests, and boosted trees. These models are often preferred due to their flexibility and ability to capture complex patterns.
Discriminant model metrics have two stages:
Model Training: Fit a black-box model (known as a discriminator) trained on a subset of combined confidential and synthetic data as to whether each record originated from the confidential or synthetic data (i.e., binary classification).
Model Evaluation: Assess the success (or lack of) with this model on out-of-sample data. If the model performs poorly (i.e., struggles to distinguish between the records), this indicates that the synthetic data are of high quality and closely resembles the confidential data.
General strategies:
Most discriminant based methods are propensity score based, allowing the method to compare the similarity of two datasets of the same structure of any dimension without making assumptions on the distributions of the attributes. Mathematically, these methods use the following steps. Let \(\mathbf{Y}\) be the confidential dataset with \(n\) observations and \(p\) variables.
What is done with the propensity scores next depends on the discriminant based method. Woo et al. (2009a) computes the mean squared error (MSE) of the propensity score against the true proportion of synthetic cases. Snoke et al. (2018b) enhances Woo et al. (2009a)’s approach by computing the average MSE between the propensity scores and the expected probabilities called the propensity score mean squared error (pMSE). Essentially, pMSE normalizes the MSE statistic by its expected null value and standard deviation, helping with its interpretability and differentiating the synthetic dataset apart from the confidential dataset.
Snoke et al. (2018b) also develops the pMSE ratio, which is one of the most popular discriminant based methods. The pMSE ratio is the average pMSE score across all records, divided by the null model, where the null model is the expected value of the pMSE score under the best case scenario when the model used to generate the data reflects the confidential data perfectly. Sakshaug and Raghunathan (2010) discretizes the propensity scores based on how the Chi-squared test is formulated.
Finally, Bowen, Liu, and Su (2021) calculates the eCDFs of the propensity scores of the synthetic and confidential data and then computes the KS (Kolmogorov-Smirnov) distance, a method called SPECKS. In other words, the SPECKS method considers the worst-case separation between the synthetic dataset and the confidential dataset.
What the discriminant based metrics actually measure for assessing the synthetic data quality varies depending on the method and the classification algorithm. For instance, Bowen and Snoke (2021) compares several utility metrics, such as the pMSE ratio and SPECKS, to evaluate differentially private synthetic datasets2 for a data challenge. The authors find that the utility metric algorithms produce mixed results in ranking the best performing differentially private synthetic data method. Conducting a study to analyze what features of the synthetic data are captured by various discriminant based methods using different classification models would be invaluable to the field (Drechsler 2022). However, to the best of our knowledge, no such study exists for synthetic data with and without differential privacy or formal privacy guarantee.
Multivariate and discriminant based methods are high dimensional. To simplify learning, let’s focus on a two-dimensional case in Figure 5.8. In the first panel, the confidential data and synthetic data have the same population parameters. In the second panel, the means differ significantly.
