Synthetic Inspector: Utility and Privacy Checks for Tabular Data
A stateless Cloud Run application that compares real and synthetic tabular data across marginals, correlations, subgroups, downstream utility, and nearest-neighbor privacy risk.
- Python
- Pandas
- scikit-learn
- Plotly
- Flask
- Google Cloud Run
Evaluates synthetic data from several angles, including train-synthetic-test-real utility and a nearest-neighbor memorization heuristic, while keeping application instances stateless through object storage.
Contents
The Problem
Visual similarity is not enough to establish that synthetic tabular data is useful or safe. A dataset can match marginal distributions while breaking correlations, underrepresenting subgroups, failing downstream models, or copying real rows too closely.
Why This Approach?
The project uses multiple evaluators because fidelity, utility, and privacy are different properties. No single score is treated as a complete verdict.
Architecture
Uploads and generated data are stored in Google Cloud Storage under a job identifier. The Cloud Run service remains stateless and produces an HTML report containing evaluator findings and Plotly charts.
Implementation
The downstream evaluator compares a real-trained baseline with train-synthetic-test-real and train-real-test-synthetic Random Forest ROC-AUC. The privacy heuristic normalizes numeric columns, measures each real row’s nearest non-self neighbor, then flags synthetic rows closer to real data than the fifth percentile of real-to-real distances.
The repository also includes marginal, correlation, and subgroup evaluators plus a smoke test that runs all five against a small perturbed Pima-like dataset.
Experiments and Results
The committed smoke test verifies that the evaluators execute end to end on 50 generated rows. It does not publish a benchmark report or evidence that the thresholds are calibrated across domains, so no general quality score is claimed.
What Failed?
The implementation specializes several assumptions to a Pima-style dataset, including an Outcome target. Nearest-neighbor distance is only a heuristic for memorization, not a formal privacy guarantee. Small or single-class samples can make ROC-AUC unstable or uninformative.
What I Learned
Synthetic-data evaluation is a matrix, not a leaderboard number. A system should expose where a dataset passes or fails and preserve the details needed to challenge each evaluator.
Next Steps
- Make target columns and task types configurable.
- Add repeated splits and uncertainty intervals for downstream utility.
- Add membership-inference testing and documented threat models.
- Build regression fixtures from several public tabular datasets.