How-To Guide

The following how-to guides illustrate specific tasks that can be performed with PyScat. They assume some familiarity with the package and its basic concepts. For introductory tutorials, please refer to the Getting Started section.

How to create a parameter ensemble from an optimization run?

Ensemble-based uncertainty quantification can be a relatively cheap way to estimate prediction uncertainty (ref). One way to create a parameter ensemble is to use the parameter sets encountered during an optimization run that meet a certain objective function threshold.

Here is an example of how to create a pypesto.Ensemble parameter ensemble from a SacessOptimizer optimization run:

[1]:

import logging

from scipy.stats import chi2

from pyscat import EvalLogger, SacessOptimizer, ThresholdSelector
from pyscat.examples import problem_schwefel

problem = problem_schwefel

# Let's pretend the objective function is a negative log-likelihood
#  and set a threshold for the 95% confidence region accordingly.
threshold = chi2.ppf(0.95, df=1) / 2
print(f"Using threshold: {threshold:.2f}")
selector = ThresholdSelector(mode="abs", threshold=threshold, dim=problem.dim)

with EvalLogger(selector).attach(problem):
    optimizer = SacessOptimizer(
        problem=problem,
        num_workers=4,
        max_walltime_s=2,
        sacess_loglevel=logging.WARNING,
    )
    result = optimizer.minimize()

ensemble = selector.to_ensemble()
print(f"Created ensemble with {ensemble.n_x} members.")

Using threshold: 1.92
Created ensemble with 1 members.

Note that this will record all objective function evaluations during the optimization run within the given threshold. It does not check for admissibility of the parameters. Make sure any local optimizers respect the problem bounds to avoid inadmissible parameter sets in the ensemble.