Reproducible benchmarks

How to read these numbers

SynthForge is schema-driven, not data-fitted. For each dataset we hand-author a schema from the public data dictionary and never show the generator the real CSV. CTGAN trains on the real data. The benchmark exists to measure that asymmetry, not to hide it.

So the fidelity and utility tables below are where a trained model wins, and it does. The privacy and constraint tables are where a generator that never saw the data wins: it cannot memorise rows it never read, and every row it emits satisfies the schema by construction.

A higher DCR means synthetic rows are far from real ones. An NNDR near 1 means no synthetic row hugs one specific real row. A validity rate of 1.000 means every row passed every range and enum constraint.

Headline

Four numbers per dataset, SynthForge against CTGAN. Constraint validity and privacy distance are where a generator that never saw the data wins. Fidelity and utility are where a model trained on it does.

adult

Constraint conformance

Privacy

ML utility (TSTR vs TRTR, AUC)

Statistical fidelity

credit

Constraint conformance

Privacy

ML utility (TSTR vs TRTR, AUC)

Statistical fidelity

Run history

Every benchmark run is appended here so you can see how the numbers change over time. We rerun whenever the generator changes meaningfully, or on a schedule. One run so far; future runs append below.

Full per-run reports: results-history.json.

Methodology

What we measure

The caveat about schema authoring, and how to read the privacy numbers, are at the top of the page.

What we run

• SynthForge, with the same generator the product runs.
• CTGAN via SDV's CTGANSynthesizer, default hyperparameters, 300 epochs, seed 42.

Both synthesizers produce a synthetic dataset of the same size as the real dataset, then every metric is run on the (real, synthetic) pair.

Datasets

• UCI Adult (Census Income, 1994). After dropping rows with missing values: 30,162 rows, 14 features, binary income target.
• UCI Credit Card Default (Taiwan, 2005). 30,000 rows, 23 features, binary default target.

Reproducibility

• The real datasets are the public UCI downloads linked above.
• All randomness uses seed 42.
• The numbers above come from the most recent run, and the full output of that run is in results.json.

What we explicitly do not measure (and why)

• Differential privacy guarantees. Differential privacy (DP) is a mathematical bound on how much any single record in the training data can change the output, expressed as numbers like "(epsilon=2, delta=1e-6)." SynthForge does not yet ship a DP-aware path; SmartNoise integration is on the roadmap. Claiming DP numbers without the math behind them would be misleading, so we don't.
• Image, text, time-series datasets. Out of scope for tabular synthetic data benchmarks.
• Membership-inference attacks against DP baselines. Same gap as DP.

Want to inspect the raw numbers? Download results.json. The field-type catalog and these result snapshots are also staged as an MIT package under github.com/synthforgeio/benchmarks (MIT; field catalog + results). Full harness source is still private; reach out at support@synthforge.io to review methodology.