Output Format

Consumer-facing specification for generated data. This is a contract document — downstream users can rely on the guarantees described here.

DatasetBundle (in-memory)

Each generated dataset is returned as a DatasetBundle with these fields:

Target dtype: int64 for classification, float for regression.

Feature dtype: matches the configured torch dtype (float32 or float64).

Feature type encoding

Each entry in feature_types is one of:

• "num" — continuous feature. After postprocessing, values are clipped and standardized to approximately zero mean and unit variance.
• "cat" — categorical feature. Observed values are integer indices in the range 0 .. cardinality - 1. When missingness is enabled, missing values are encoded as NaN.

feature_types[i] describes column index i in X_train and X_test.

On-disk directory structure

out_dir/
  shard_00000/
    train.parquet
    test.parquet
    metadata.ndjson
    lineage/
      adjacency.bitpack.bin
      adjacency.index.json
  shard_00001/
    ...

Shard naming: shard_{id:05d} — five-digit zero-padded shard ID. Default: 128 datasets per shard.

Shard ID calculation: dataset_index // shard_size.

Request-run layout (dagzoo request)

Request-driven runs use output_root as a stable handoff root:

output_root/
  handoff_manifest.json
  generated/
    shard_00000/
    effective_config.yaml
    effective_config_trace.yaml
  filter/
    filter_manifest.ndjson
    filter_summary.json
  curated/
    shard_00000/

generated/, filter/, and curated/ reuse the same shard and summary contracts documented on this page. handoff_manifest.json is the downstream entrypoint for consumers such as tab-foundry.

Request handoff manifest JSON

handoff_manifest.json uses this versioned top-level contract:

Current artifacts keys:

• run_root
• generated_dir
• filter_dir
• filtered_corpus_dir
• effective_config_path
• effective_config_trace_path
• filter_manifest_path
• filter_summary_path

Current diversity_artifacts keys:

• summary_json_path
• summary_md_path

throughput.generation_stage and throughput.filter_stage report request-run wall-clock stage timing from the dagzoo request workflow. filter_summary.json remains the underlying deferred-filter artifact and retains its own timing semantics.

dagzoo request does not run a diversity audit automatically, so the diversity_artifacts values are currently null unless a separate workflow persists request-associated diversity outputs alongside the run.

Parquet column schema

Shard-level train.parquet and test.parquet both use packed row-wise records:

Compression: zstd (default).

Feature typing metadata remains per-dataset in metadata.ndjson records.

Metadata NDJSON structure

Each shard writes one metadata.ndjson file with one JSON record per dataset. Each line contains:

metadata contains the dataset-level generation metadata described below.

Top-level keys

For canonical generation (generate_one, generate_batch, generate_batch_iter, and dagzoo generate), replay later bundles with the shared seed, run_num_datasets, and dataset_index by regenerating the canonical batch and selecting that index. dataset_seed preserves the per-bundle child seed for deferred replay and diagnostics. Exact keyed subtree replay uses seed together with the keyed_replay paths.

keyed_replay sub-object

Present for canonical generation outputs. These paths are interpreted relative to KeyedRng(metadata["seed"]):

mechanism_families sub-object

Present for canonical fixed-layout generation outputs.

Current variant labels are additive observability fields, not separate public family config names. Today they are emitted for widened gp execution as gp.standard, gp.periodic, and gp.multiscale.

Coverage, diversity-audit, and filter-calibration summary artifacts reuse these counts inside mechanism_family_summary and add dataset-level presence-rate fields:

• dataset_presence_rate_by_family
• dataset_presence_rate_by_variant

Shift sub-object

Present for all generated bundles. When shift is disabled, scales are 0.0 and multipliers are 1.0.

Noise Distribution sub-object

Present for all generated bundles.

Filter sub-object

Class Structure sub-object (classification only)

Present only for classification datasets.

Fixed-layout metadata

Present for all canonical generation outputs. These bundles share one sampled layout per run and preserve emitted column alignment (feature count, column order, and lineage feature-to-node mapping) within that run.

Under chunk_batched_v1, canonical fixed-layout outputs are deterministic for the same run seed and realized run shape. Internal plan metadata records the shared sampled layout and execution-plan fingerprint used for that run, while keyed_replay records the exact keyed subtree roots needed for internal replay.

Missingness sub-object (optional)

Present only when missingness is enabled.

Lineage schema

Schema name: dagzoo.dag_lineage

Version 1.0.0 (dense, in-memory)

Used in the in-memory metadata lineage field during generation. When lineage is persisted to disk, payloads are rewritten to compact version 1.1.0.

{
  "schema_name": "dagzoo.dag_lineage",
  "schema_version": "1.0.0",
  "graph": {
    "n_nodes": 8,
    "adjacency": [[0, 1, 0, ...], ...]
  },
  "assignments": {
    "feature_to_node": [2, 3, 5, 7],
    "target_to_node": 7
  }
}

• adjacency is an n_nodes x n_nodes list of lists. Entries are 0 or 1. Upper-triangular only; diagonal is always 0. Direction convention is adjacency[src][dst] (src -> dst), so parents of node j are found from column j.
• feature_to_node[i] is the DAG node index that produces feature i.
• target_to_node is the DAG node index that produces the target.

Version 1.1.0 (compact, on-disk)

Used in metadata.ndjson dataset records when lineage artifacts are written to disk. Replaces the dense adjacency matrix with a reference to bitpacked binary data.

{
  "schema_name": "dagzoo.dag_lineage",
  "schema_version": "1.1.0",
  "graph": {
    "n_nodes": 8,
    "edge_count": 12,
    "adjacency_ref": {
      "encoding": "upper_triangle_bitpack_v1",
      "blob_path": "lineage/adjacency.bitpack.bin",
      "index_path": "lineage/adjacency.index.json",
      "dataset_index": 0,
      "bit_offset": 0,
      "bit_length": 28,
      "sha256": "a1b2c3..."
    }
  },
  "assignments": {
    "feature_to_node": [2, 3, 5, 7],
    "target_to_node": 7
  }
}

Adjacency encoding: upper_triangle_bitpack_v1

• Packs the n_nodes * (n_nodes - 1) / 2 upper-triangle bits into bytes.
• Bit order: little-endian.
• bit_offset and bit_length locate this dataset’s bits within the shared shard-level blob file.
• sha256 is a hex-encoded SHA-256 checksum of the packed bytes for this dataset’s adjacency data.

Lineage index file

Each shard contains lineage/adjacency.index.json with:

• schema_name and schema_version — echo the lineage schema identifiers.
• encoding — always "upper_triangle_bitpack_v1".
• records — array of per-dataset offset/length/checksum entries.

Contract guarantees

Determinism — seed derivation is deterministic. For fixed seed and configuration, runs are expected to reproduce metadata and numerical outputs within tolerance. Strict byte-identical tensors/files are not guaranteed across all backends.

Feature alignment — feature_types[i] in each metadata record describes feature index i inside packed x row vectors and tensor column index i in X_train / X_test.

Lineage integrity — each dataset’s bitpacked adjacency data is protected by a SHA-256 checksum recorded in the metadata.

Postprocessing invariants:

• Canonical generation (generate_one, generate_batch, generate_batch_iter) is fixed-layout-backed and preserves emitted feature schema across the run: constant-column removal and feature-column permutation are disabled.
• Numeric features are clipped and standardized (approximately zero mean, unit variance).
• Classification target classes are randomly permuted (label indices carry no ordinal meaning).