Diagnostics and DataFrames#

The previous pages look at one configuration at a time. This one looks at a space in aggregate: what a batch of draws reports before the space is trusted, and what a columnar view of those draws looks like. The running example is a solver portfolio.

import designspace as ds

SOLVERS = ("cplex", "gurobi", "heuristic")

space = (
    ds.space(
        ds.param("enabled_solvers").subset(SOLVERS, min_size=1),
        ds.param("solver_mode").choice(
            "single",
            ensemble=ds.space(ds.param("n_workers").integer(2, 8)),
        ),
        ds.param("priority").ordinal("low", "normal", "high"),
        ds.param("warm_start").bool(),
        ds.param("warm_start_frac").real(0.05, 0.9).when(ds.param("warm_start")),
        ds.param("time_limit_s").integer(10, 3600),
        # A static count gives an Array column; a dynamic one gives a List.
        ds.param("weights").real(-1.0, 1.0).repeat(4),
        ds.param("n_checkpoints").integer(0, 3),
        ds.param("checkpoints").real(0.0, 1.0).repeat(ds.param("n_checkpoints")),
    )
    .encourage(
        ds.param("warm_start_frac") + ds.param("time_limit_s") / 3600.0 <= 1.0,
        tags=("budget-unguarded",),
    )
    .encourage(
        ds.param("warm_start_frac").if_inactive(0.0)
        + ds.param("time_limit_s") / 3600.0 <= 1.0,
        tags=("budget-guarded",),
    )
)
print(space)

Space: 10 params, 2 conditional, 2 constraints
  enabled_solvers       subset   subset of {'cplex', 'gurobi', 'heuristic'}, size 1..3
  solver_mode           choice   one of single, ensemble(...)
    ensemble.n_workers  integer  [2, 8]
  priority              ordinal  ('low' < 'normal' < 'high')
  warm_start            bool     {False, True}
  warm_start_frac       real     [0.05, 0.9]  when warm_start
  time_limit_s          integer  [10, 3600]
  weights               list     count = 4, of real [-1.0, 1.0]
  n_checkpoints         integer  [0, 3]
  checkpoints           list     count = n_checkpoints, of real [0.0, 1.0]

  encourage  warm_start_frac + time_limit_s / 3600.0 <= 1.0
  encourage  warm_start_frac.if_inactive(0.0) + time_limit_s / 3600.0 <= 1.0

Columnar output#

space.sample(n) returns a polars.DataFrame and needs the polars extra. sample_dicts and sample_one need no extra and are unaffected.

df = space.sample(6, seed=0)
df

shape: (6, 10)

enabled_solvers	solver_mode	solver_mode.ensemble	priority	warm_start	warm_start_frac	time_limit_s	weights	n_checkpoints	checkpoints
list[str]	str	struct[1]	str	bool	f64	i64	array[f64, 4]	i64	list[f64]
["gurobi", "heuristic"]	"single"	null	"high"	false	null	2188	[0.458993, 0.08725, … 0.631707]	0	[]
["gurobi"]	"single"	null	"high"	false	null	1086	[-0.154626, -0.943361, … 0.341249]	2	[0.615385, 0.383678]
["heuristic"]	"single"	null	"high"	false	null	1124	[-0.028329, 0.778976, … -0.28441]	2	[0.321869, 0.5943]
["cplex", "gurobi"]	"single"	null	"normal"	true	0.757748	2836	[-0.521261, 0.752968, … -0.327766]	0	[]
["cplex", "heuristic"]	"single"	null	"normal"	true	0.12714	2093	[-0.402608, 0.34399, … 0.884226]	1	[0.105495]
["heuristic"]	"ensemble"	{5}	"normal"	false	null	3583	[0.897887, -0.07991, … -0.005155]	2	[0.785786, 0.414656]

The dtype table is visible in the schema. Scalars map to Boolean, Float64 and Int64; a choice becomes a Utf8 discriminator plus one nullable Struct per parameterized variant; a static-count lift becomes Array(dtype, n) and a dynamic-count one becomes List(dtype).

dict(df.schema)

{'enabled_solvers': List(String),
 'solver_mode': String,
 'solver_mode.ensemble': Struct({'n_workers': Int64}),
 'priority': String,
 'warm_start': Boolean,
 'warm_start_frac': Float64,
 'time_limit_s': Int64,
 'weights': Array(Float64, shape=(4,)),
 'n_checkpoints': Int64,
 'checkpoints': List(Float64)}

An inactive parameter has no columnar analogue for the dict form’s “absent”, so it becomes null:

df.select("warm_start", "warm_start_frac")

shape: (6, 2)

warm_start	warm_start_frac
bool	f64
false	null
false	null
false	null
true	0.757748
true	0.12714
false	null

reject_soft=True additionally rejects declared violations. It is off by default, since declared constraints do not affect feasibility.

space.sample(6, seed=0, reject_soft=True).height

Sampling diagnostics#

sampling_report() draws from the unconditioned measure, before any rejection, and aggregates what happened. Drawing unconditioned is the point: two pathologies are invisible once rejection has hidden them.

report = space.sampling_report(n=500, seed=0)
round(report.acceptance_rate, 3)

1.0

for row in report.constraints:
    tags = ", ".join(sorted(row.constraint.tags)) or "-"
    print(f"{row.constraint.kind:10} [{tags:18}] "
          f"applicable={row.applicable:.3f} satisfied={row.satisfied:.3f} "
          f"violation_rate={row.violation_rate:.3f}")

encourage  [budget-unguarded  ] applicable=0.490 satisfied=0.592 violation_rate=0.408
encourage  [budget-guarded    ] applicable=1.000 satisfied=0.800 violation_rate=0.200

Unknown-swallowing#

A constraint that cannot be evaluated is inapplicable, and inapplicable means accepted. That is the permissive direction, and it is silent.

The two budget constraints above are the same aggregate over the same draws, differing only in the guard. The unguarded one goes Unknown wherever warm_start is off; its .if_inactive(0.0) twin stays evaluable throughout.

{
    ", ".join(row.constraint.tags): round(row.applicable, 3)
    for row in report.constraints
}

{'budget-unguarded': 0.49, 'budget-guarded': 1.0}

by_tag = {", ".join(r.constraint.tags): r for r in report.constraints}
assert by_tag["budget-guarded"].applicable == 1.0
assert by_tag["budget-unguarded"].applicable < 1.0

applicable is the only signal that this is happening. Nothing in sample()’s output would report it.

round(report.activity["warm_start_frac"], 3)

0.49

activity gives the fraction of draws in which each parameter was active, which is what the unguarded constraint’s applicable is tracking.

Funnels#

Unknown-swallowing has a second consequence beyond a constraint quietly not enforcing. A constraint that is inapplicable on part of the space biases the conditioned measure toward that part, since rejection accepts those draws unconditionally.

This is what require is defined to do: it conditions the declared measure. The effect is not visible from the resulting sample, which is the reason the report draws unconditioned.

Reading the report correctly#

satisfied is conditioned on applicability, not on all draws. A constraint applicable in 1% of draws and always satisfied there reports 1.0, not 0.01. The pair is read together: applicable says how often the question was asked, satisfied how often the answer was yes.

satisfied is also raw, so it means opposite things for opposite verbs. violation_rate folds the polarity in and always means the unhealthy fraction, whichever verb produced the row.

Bound tightening#

The reference sampler can fold an already-assigned bound-origin coupling into the draw instead of drawing and rejecting. For sample() that is unobservable, since truncation and conditioning agree. For a report it is observable, which is why it defaults to off: on a bound-coupled space it would collapse exactly the rows most likely to carry a pathology.

tightened = space.sampling_report(n=500, seed=0, tighten_bounds=True)
round(report.acceptance_rate, 3), round(tightened.acceptance_rate, 3)

(1.0, 1.0)

This space has no bound-origin coupling to tighten, so the two agree. The three sampling entry points take no such flag at all, because tightening cannot change the distribution they return.

Comparing two configurations#

a = space.sample_one(seed=0)
b = space.sample_one(seed=1)
[(d.param, d.old, d.new) for d in ds.config_diff(a, b, space)][:5]

[('enabled_solvers', ['gurobi', 'heuristic'], ['heuristic']),
 ('solver_mode', 'single', 'ensemble'),
 ('priority', 'high', 'normal'),
 ('time_limit_s', 2188, 1479),
 ('weights[0]', 0.4589931219679968, 0.09918737534611899)]

Where to go next#

Identity, serialization and solver hand-off covers what a consumer stores and how it plugs in.