{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89f0cefdc311a192",
   "metadata": {},
   "source": [
    "# How-To Guide\n",
    "\n",
    "The following how-to guides illustrate specific tasks that can be performed with PyScat. They assume some familiarity with the package and its basic concepts. For introductory tutorials, please refer to the [Getting Started](getting_started.ipynb) section."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483f52dc17dd713",
   "metadata": {},
   "source": [
    "## How to create a parameter ensemble from an optimization run?\n",
    "\n",
    "Ensemble-based uncertainty quantification can be a relatively cheap way to estimate prediction uncertainty ([ref](https://doi.org/10.1109/TCBB.2022.3213914)).\n",
    "One way to create a parameter ensemble is to use the parameter sets encountered during an optimization run that meet a certain objective function threshold.\n",
    "\n",
    "Here is an example of how to create a `pypesto.Ensemble` parameter ensemble from a `SacessOptimizer` optimization run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial_id",
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "from scipy.stats import chi2\n",
    "\n",
    "from pyscat import EvalLogger, SacessOptimizer, ThresholdSelector\n",
    "from pyscat.examples import problem_schwefel\n",
    "\n",
    "problem = problem_schwefel\n",
    "\n",
    "# Let's pretend the objective function is a negative log-likelihood\n",
    "#  and set a threshold for the 95% confidence region accordingly.\n",
    "threshold = chi2.ppf(0.95, df=1) / 2\n",
    "print(f\"Using threshold: {threshold:.2f}\")\n",
    "selector = ThresholdSelector(mode=\"abs\", threshold=threshold, dim=problem.dim)\n",
    "\n",
    "with EvalLogger(selector).attach(problem):\n",
    "    optimizer = SacessOptimizer(\n",
    "        problem=problem,\n",
    "        num_workers=4,\n",
    "        max_walltime_s=2,\n",
    "        sacess_loglevel=logging.WARNING,\n",
    "    )\n",
    "    result = optimizer.minimize()\n",
    "\n",
    "ensemble = selector.to_ensemble()\n",
    "print(f\"Created ensemble with {ensemble.n_x} members.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba45e6c2f608405a",
   "metadata": {},
   "source": "Note that this will record all objective function evaluations during the optimization run within the given threshold. It does not check for admissibility of the parameters. Make sure any local optimizers respect the problem bounds to avoid inadmissible parameter sets in the ensemble."
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}