Ablation

Measure how the rows that fail validation steps affect a model trained on the data.

Usage

Ablation(
    validation,
    evaluate=None,
    metric=None,
    higher_is_better=True,
    repeats=5,
    seed=0,
    control="size_matched",
    duplicates="keep_first",
    duplicates_order_by=None,
    max_removed=0.5,
    conf_level=0.95,
    adjust="holm",
    label=None,
    evaluate_version=None,
    cache=None,
)

A validation says which rows of a training table fail its checks, but not whether those rows matter to a model trained on the table. An ablation finds out: it trains and evaluates the model on the full table and on versions of the table with the rows that failed some of the steps removed, and compares the results. Pointblank never trains anything itself: you provide an evaluate= function that takes a training table and returns a score (e.g., the accuracy of a model trained on the table, measured on a fixed holdout set).

Removing rows can change a score just because there are fewer rows to train on, so each variant is also compared to a control that removes the same number of rows at random. The difference between the two is the effect of the rows that failed the steps, as opposed to the effect of having less data. Training is often random too (e.g., the initial weights of a model), so every version is evaluated repeats= times with different seeds. The r-th evaluation of every version gets the same seed, so the differences between versions are compared seed by seed (paired), which cancels out much of the noise.

An Ablation is built in two parts, like a validation: the strategy methods add variants (drop_each(), drop_all(), drop_steps(), drop_all_but_each(), and drop_groups()), and nothing is evaluated until run() is called. Use plan() before that to see how many rows each variant removes and how many evaluations the run needs. When training happens elsewhere (e.g., on a cluster), export() writes out the versions of the table to train on, and load_results() reads back the metrics.

Parameters

validation: Validate: An interrogated validation (a Validate object). Its target table is the training table, and its row-level steps (the same ones that get_row_flags() uses) say which rows to remove.
evaluate: Callable[…, Any] | None = None: A function that trains and evaluates a model. It’s called as evaluate(train, seed=seed, variant=variant), where train is the training table with some rows removed (of the same type as the validation’s target table; a lazy or database table is passed as a lazy table), seed is an integer to seed any randomness in training with, and variant is an AblationVariant describing the version of the table. The function can leave out the variant= parameter if it doesn’t need it. It must return a number, or a dictionary of named numbers (one per metric). The evaluation data should be a fixed holdout set that the function holds on to: it must not depend on train. For scikit-learn models, sklearn_evaluator() makes this function for you. It can be left out (None) if the evaluations are done elsewhere, with export() and load_results().
metric: str | None = None: The name of the metric that verdicts are based on. It’s needed when evaluate= returns a dictionary with more than one metric (the others are still reported). When evaluate= returns a number, this is the name it’s given ("score" by default).
higher_is_better: bool = True: Is a higher value of metric= better (e.g., accuracy) or worse (e.g., an error)? This decides whether a variant that raises the metric helps or hurts.
repeats: int = 5: The number of times each version of the table is evaluated, each time with a different seed. At least 3 are needed for verdicts. More repeats give narrower intervals but take longer to run.
seed: int = 0: The seed of the first repeat: repeat r (counting from 0) passes seed + r to evaluate=. The random rows removed by the controls also depend on it, so a run can be reproduced exactly.
control: str | Stratified | None = "size_matched": The control that variants are compared to. With "size_matched" (the default), each variant is compared to the full table with the same number of rows removed at random (drawn anew for each repeat). Variants that remove the same number of rows share their control. A Stratified control (e.g., pb.ablation.Stratified(by="source")) removes the random rows in proportion to the groups of one or more columns. Use None to compare variants to the full table only.
duplicates: str | None = "keep_first": How the duplicate rows found by rows_distinct() steps are removed. With "keep_first" (the default) or "keep_last", one copy of each group of duplicates is kept and the others are removed. "flag_all" removes every copy, and None leaves rows_distinct() steps out of the ablation. These are the same options as in get_row_flags().
duplicates_order_by: str | list[str] | None = None: One or more columns that order the copies of a duplicated row, to decide which copy is kept (e.g., "updated_at" with duplicates="keep_last" keeps the newest record).
max_removed: float = 0.5: A variant that removes more than this share of the rows (from 0 to 1) gives a warning, as a comparison of such different tables says more about the size of the training table than about the quality of the removed rows.
conf_level: float = 0.95: The confidence level of the intervals, and of the verdicts (a variant helps or hurts if its adjusted p-value is below 1 - conf_level).
adjust: str | None = "holm": How p-values are adjusted for comparing several variants: "holm" (the default) uses the Holm method, which keeps the chance of any false verdict among the variants below 1 - conf_level. Use None to use the unadjusted p-values.
label: str | None = None: An optional label for the ablation, shown in its reports.
evaluate_version: str | None = None: An optional version of the evaluate= function (e.g., "v2", or a commit hash). It’s recorded with the results and is part of the cache key, so changing it when the model or its training changes makes the cache evaluate everything again.
cache: str | os.PathLike[str] | None = None: A directory to keep the results of evaluations in. Each result is saved as soon as it’s made, under a key made from the data, the rows removed, the seed, and evaluate_version=. An evaluation with the same key is never repeated, so a run that stops partway can be started again without losing work, and adding a variant only evaluates what’s new. The evaluate= function itself isn’t part of the key, so change evaluate_version= (or use another directory) when it changes.

Returns

Ablation: An Ablation object, with no variants yet.

Details

For each variant and metric, the results give the mean and standard deviation of the metric across the repeats, and two paired comparisons: with the full table (delta_vs_full) and with the control (delta_vs_control). Each comparison has a t-interval at conf_level= and a p-value from a paired t-test, along with the p-value adjusted across the variants. The verdict on a variant is based on the metric named in metric= and the comparison with the control (or with the full table, if control=None):

"helps": removing the variant’s rows makes the metric better, with an adjusted p-value below 1 - conf_level
"hurts": removing the rows makes the metric worse, with an adjusted p-value below 1 - conf_level
"inconclusive": the difference could be due to chance

The variants of drop_all_but_each() are judged on what their left-out step adds on top of the others instead (the delta_added columns), and can also be "redundant": the step’s rows help on their own, but make no clear difference once the other steps’ rows are removed.

With fewer than 3 repeats, there are no verdicts. The intervals aren’t adjusted for multiple comparisons, so an interval can leave out zero while the verdict is "inconclusive".

A variant that removes no rows (e.g., a step that no row failed) is left out of the ablation, with a note in the reports. Steps that can’t be matched to the rows of the table, such as steps with a pre= function not marked with row_preserving(), are left out too.

Before a long run, check_determinism() checks that evaluate= gives the same result for the same table and seed, which the paired comparisons rely on.

Examples

Let’s take a table of temperature readings where some readings were recorded in tenths of a degree (so they’re ten times too high) and some are missing. The ‘model’ trained on it is simple: it estimates the average temperature from a bootstrap sample of the readings, and it’s evaluated by its error against the known average of 20 degrees.

import pointblank as pb
import polars as pl

temps = [20 + (k * 37 % 11 - 5) * 0.6 for k in range(300)]
for k in range(0, 300, 25):
    temps[k] *= 10
for k in range(7, 300, 40):
    temps[k] = None
readings = pl.DataFrame({"temp": temps})

def evaluate(train, *, seed, variant):
    sample = train.drop_nulls().sample(fraction=1.0, with_replacement=True, seed=seed)
    return {"error": abs(sample["temp"].mean() - 20)}

The validation has two steps: one for missing readings and one for readings out of range.

validation = (
    pb.Validate(data=readings)
    .col_vals_not_null(columns="temp")
    .col_vals_between(columns="temp", left=-10, right=45, na_pass=True)
    .interrogate()
)

Now we can set up the ablation, with one variant for each step (drop_each()) and one for both steps (drop_all()). The error is the metric, and lower errors are better.

ablation = (
    pb.Ablation(validation, evaluate=evaluate, metric="error", higher_is_better=False)
    .drop_each()
    .drop_all()
)

ablation.run()

Variant	Removed	error	Δ vs random
Ablation report
error on the full table (300 rows): 7.179 ± 2.440 (lower is better). Each version was evaluated 5 times; Δ is the mean paired difference from removing the same number of random rows.
Step 1 Expect that all values in `temp` should not be Null.	8 2.7%	7.179 ± 2.440 Δ vs full +0.000	+0.563 [−4.216, +5.342] inconclusive
Step 2 Expect that values in `temp` should be between `-10` and `45`.	12 4.0%	0.069 ± 0.062 Δ vs full −7.109	−7.974 [−10.556, −5.393] helps
All steps 1: Expect that all values in `temp` should not be Null. 2: Expect that values in `temp` should be between `-10` and `45`.	20 6.7%	0.069 ± 0.062 Δ vs full −7.109	−6.649 [−9.249, −4.049] helps
Intervals are 95% paired t-intervals, not adjusted for multiple comparisons; verdicts use Holm-adjusted p-values.

Removing the rows that failed step 2 (the readings out of range) cuts the error by much more than removing the same number of random rows, so the verdict is that it helps. The missing readings were dropped by the model anyway, so removing them makes no real difference and the verdict on step 1 is inconclusive. The numbers behind the report are available as a table from get_effects().

Parameters

Returns

Details

Examples

See Also