Add option to control figure generation in composite experiments (#1240)

### Summary

Closes #1239. An init parameter has been added to composite analysis,
`generate_figures`, to control figure generation. By default,
``generate_figures`` is ``always``, meaning figures will always be
generated. If ``generate_figures`` is set to ``selective``, then only
figures for analysis results of bad quality will be generated. If
``generate_figures`` is set to ``never``, then figures will never be
generated. This behavior can still be overridden for individual analyses
by setting the analysis option ``plot``, which is now unset by default
instead of `True`.

### Details and comments
- I didn't end up implementing the proposal discussed in the issue where
all figures would still be generated in `selective` mode if there are 10
or fewer figures. If this is preferable, we can add the behavior, but it
seems a bit messy to keep a counter of all component experiments and how
many figures each generates.
- Added how to access child analysis classes in the Getting Started
tutorial.

### Timing benchmark
I ran a parallel T1 experiment over 127 qubits then timed only the
analysis.
- ``always``: ~120 seconds
- ``selective``: ~100 seconds (generated 60/127 figures)
- ``never``: ~15 seconds

docs/howtos/figure_generation.rst

@@ -0,0 +1,45 @@
+Control figure generation 
+=========================
+
+Problem
+-------
+
+You want to change the default behavior where figures are generated with every experiment.
+
+Solution
+--------
+
+For a single non-composite experiment, figure generation can be switched off by setting the analysis
+option ``plot`` to ``False``:
+
+.. jupyter-input::
+
+    experiment.analysis.set_options(plot = False)    
+
+For composite experiments, there is a ``generate_figures`` analysis option which controls how child figures are
+generated. There are three options:
+
+- ``always``: The default behavior, generate figures for each child experiment.
+- ``never``: Never generate figures for any child experiment.
+- ``selective``: Only generate figures for analysis results where ``quality`` is ``bad``. This is useful
+  for large composite experiments where you only want to examine qubits with problems.
+
+This parameter should be set on the analysis of a composite experiment before the analysis runs:
+
+.. jupyter-input::
+
+    parallel_exp = ParallelExperiment(
+        [T1(physical_qubits=(i,), delays=delays) for i in range(2)]
+    )
+    parallel_exp.analysis.set_options(generate_figures="selective")
+
+Discussion
+----------
+
+These options are useful for large composite experiments, where generating all figures incurs a significant
+overhead.
+
+See Also
+--------
+
+* The `Visualization tutorial <visualization.html>`_ discusses how to customize figures

docs/tutorials/getting_started.rst

@@ -238,6 +238,8 @@ supports can be set:
 
   exp.set_run_options(shots=1000,
                       meas_level=MeasLevel.CLASSIFIED)
+  print(f"Shots set to {exp.run_options.get('shots')}, " 
+        "measurement level set to {exp.run_options.get('meas_level')}")
 
 Consult the documentation of the run method of your
 specific backend type for valid options.
@@ -253,6 +255,7 @@ before execution:
   exp.set_transpile_options(scheduling_method='asap',
                             optimization_level=3,
                             basis_gates=["x", "sx", "rz"])
+  print(f"Transpile options are {exp.transpile_options}")
 
 Consult the documentation of :func:`qiskit.compiler.transpile` for valid options.
 
@@ -267,14 +270,15 @@ upon experiment instantiation, but can also be explicitly set via
     exp = T1(physical_qubits=(0,), delays=delays)
     new_delays=np.arange(1e-6, 600e-6, 50e-6)
     exp.set_experiment_options(delays=new_delays)
+    print(f"Experiment options are {exp.experiment_options}")
 
 Consult the :doc:`API documentation </apidocs/index>` for the options of each experiment
 class.
 
 Analysis options
 ----------------
 
-These options are unique to each analysis class. Unlike the other options, analyis
+These options are unique to each analysis class. Unlike the other options, analysis
 options are not directly set via the experiment object but use instead a method of the
 associated ``analysis``:
 
@@ -295,7 +299,7 @@ Running experiments on multiple qubits
 ======================================
 
 To run experiments across many qubits of the same device, we use **composite
-experiments**. A composite experiment is a parent object that contains one or more child
+experiments**. A :class:`.CompositeExperiment` is a parent object that contains one or more child
 experiments, which may themselves be composite. There are two core types of composite
 experiments:
 
@@ -323,7 +327,7 @@ Note that when the transpile and run options are set for a composite experiment,
 child experiments's options are also set to the same options recursively. Let's examine
 how the parallel experiment is constructed by visualizing child and parent circuits. The
 child experiments can be accessed via the
-:meth:`~.ParallelExperiment.component_experiment` method, which indexes from zero:
+:meth:`~.CompositeExperiment.component_experiment` method, which indexes from zero:
 
 .. jupyter-execute::
 
@@ -333,6 +337,16 @@ child experiments can be accessed via the
 
     parallel_exp.component_experiment(1).circuits()[0].draw(output='mpl')
 
+Similarly, the child analyses can be accessed via :meth:`.CompositeAnalysis.component_analysis` or via
+the analysis of the child experiment class:
+
+.. jupyter-execute::
+
+    parallel_exp.component_experiment(0).analysis.set_options(plot = True)
+
+    # This should print out what we set because it's the same option
+    print(parallel_exp.analysis.component_analysis(0).options.get("plot"))
+
 The circuits of all experiments assume they're acting on virtual qubits starting from
 index 0. In the case of a parallel experiment, the child experiment
 circuits are composed together and then reassigned virtual qubit indices:

docs/tutorials/index.rst

@@ -10,11 +10,14 @@ They're suitable for beginners who want to get started with the package.
 The Basics
 ----------
 
+.. This toctree is hardcoded since Getting Started is already included in the sidebar for more visibility.
+
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
 
     intro
-
+    getting_started
+
 Exploring Modules
 -----------------
 

qiskit_experiments/curve_analysis/base_curve_analysis.py

@@ -160,8 +160,8 @@ def _default_options(cls) -> Options:
                 the analysis result.
             plot_raw_data (bool): Set ``True`` to draw processed data points,
                 dataset without formatting, on canvas. This is ``False`` by default.
-            plot (bool): Set ``True`` to create figure for fit result.
-                This is ``True`` by default.
+            plot (bool): Set ``True`` to create figure for fit result or ``False`` to
+                not create a figure. This overrides the behavior of ``generate_figures``.
             return_fit_parameters (bool): Set ``True`` to return all fit model parameters
                 with details of the fit outcome. Default to ``True``.
             return_data_points (bool): Set ``True`` to include in the analysis result
@@ -213,7 +213,6 @@ def _default_options(cls) -> Options:
 
         options.plotter = CurvePlotter(MplDrawer())
         options.plot_raw_data = False
-        options.plot = True
         options.return_fit_parameters = True
         options.return_data_points = False
         options.data_processor = None
@@ -333,7 +332,7 @@ def _evaluate_quality(
         Returns:
             String that represents fit result quality. Usually "good" or "bad".
         """
-        if fit_data.reduced_chisq < 3.0:
+        if 0 < fit_data.reduced_chisq < 3.0:
             return "good"
         return "bad"
 

qiskit_experiments/curve_analysis/composite_curve_analysis.py

@@ -332,6 +332,15 @@ def _run_analysis(
         experiment_data: ExperimentData,
     ) -> Tuple[List[AnalysisResultData], List["matplotlib.figure.Figure"]]:
 
+        # Flag for plotting can be "always", "never", or "selective"
+        # the analysis option overrides self._generate_figures if set
+        if self.options.get("plot", None):
+            plot = "always"
+        elif self.options.get("plot", None) is False:
+            plot = "never"
+        else:
+            plot = getattr(self, "_generate_figures", "always")
+
         analysis_results = []
         figures = []
 
@@ -355,6 +364,10 @@ def _run_analysis(
             else:
                 quality = "bad"
 
+            # After the quality is determined, plot can become a boolean flag for whether
+            # to generate the figure
+            plot_bool = plot == "always" or (plot == "selective" and quality == "bad")
+
             if self.options.return_fit_parameters:
                 # Store fit status overview entry regardless of success.
                 # This is sometime useful when debugging the fitting code.
@@ -429,7 +442,7 @@ def _run_analysis(
         else:
             composite_results = []
 
-        if self.options.plot:
+        if plot_bool:
             self.plotter.set_supplementary_data(
                 fit_red_chi={k: v.reduced_chisq for k, v in fit_dataset.items() if v.success},
                 primary_results=composite_results,

qiskit_experiments/curve_analysis/curve_analysis.py

@@ -496,6 +496,15 @@ def _run_analysis(
         analysis_results = []
         figures = []
 
+        # Flag for plotting can be "always", "never", or "selective"
+        # the analysis option overrides self._generate_figures if set
+        if self.options.get("plot", None):
+            plot = "always"
+        elif self.options.get("plot", None) is False:
+            plot = "never"
+        else:
+            plot = getattr(self, "_generate_figures", "always")
+
         # Prepare for fitting
         self._initialize(experiment_data)
 
@@ -507,6 +516,10 @@ def _run_analysis(
         else:
             quality = "bad"
 
+        # After the quality is determined, plot can become a boolean flag for whether
+        # to generate the figure
+        plot_bool = plot == "always" or (plot == "selective" and quality == "bad")
+
         if self.options.return_fit_parameters:
             # Store fit status overview entry regardless of success.
             # This is sometime useful when debugging the fitting code.
@@ -565,7 +578,7 @@ def _run_analysis(
                 )
             )
 
-        if self.options.plot:
+        if plot_bool:
             if fit_data.success:
                 self.plotter.set_supplementary_data(
                     fit_red_chi=fit_data.reduced_chisq,

qiskit_experiments/curve_analysis/standard_analysis/decay.py

@@ -98,13 +98,13 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared lower than three
+            - a reduced chi-squared lower than three and greater than zero
             - tau error is less than its value
         """
         tau = fit_data.ufloat_params["tau"]
 
         criteria = [
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             curve.utils.is_error_not_significant(tau),
         ]
 

qiskit_experiments/curve_analysis/standard_analysis/error_amplification_analysis.py

@@ -185,14 +185,14 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared lower than three,
+            - a reduced chi-squared lower than three and greater than zero,
             - a measured angle error that is smaller than the allowed maximum good angle error.
               This quantity is set in the analysis options.
         """
         fit_d_theta = fit_data.ufloat_params["d_theta"]
 
         criteria = [
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             abs(fit_d_theta.nominal_value) < abs(self.options.max_good_angle_error),
         ]
 

qiskit_experiments/curve_analysis/standard_analysis/gaussian.py

@@ -126,7 +126,7 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared less than 3,
+            - a reduced chi-squared less than 3 and greater than zero,
             - a peak within the scanned frequency range,
             - a standard deviation that is not larger than the scanned frequency range,
             - a standard deviation that is wider than the smallest frequency increment,
@@ -149,7 +149,7 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
             fit_data.x_range[0] <= fit_freq.n <= fit_data.x_range[1],
             1.5 * freq_increment < fit_sigma.n,
             fit_width_ratio < 0.25,
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             curve.utils.is_error_not_significant(fit_sigma),
             snr > 2,
         ]

qiskit_experiments/curve_analysis/standard_analysis/oscillation.py

@@ -111,15 +111,15 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared lower than three,
+            - a reduced chi-squared lower than three and greater than zero,
             - more than a quarter of a full period,
             - less than 10 full periods, and
             - an error on the fit frequency lower than the fit frequency.
         """
         fit_freq = fit_data.ufloat_params["freq"]
 
         criteria = [
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             1.0 / 4.0 < fit_freq.nominal_value < 10.0,
             curve.utils.is_error_not_significant(fit_freq),
         ]
@@ -260,15 +260,15 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared lower than three
+            - a reduced chi-squared lower than three and greater than zero
             - relative error of tau is less than its value
             - relative error of freq is less than its value
         """
         tau = fit_data.ufloat_params["tau"]
         freq = fit_data.ufloat_params["freq"]
 
         criteria = [
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             curve.utils.is_error_not_significant(tau),
             curve.utils.is_error_not_significant(freq),
         ]

qiskit_experiments/curve_analysis/standard_analysis/resonance.py

@@ -126,7 +126,7 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
         """Algorithmic criteria for whether the fit is good or bad.
 
         A good fit has:
-            - a reduced chi-squared less than 3,
+            - a reduced chi-squared less than 3 and greater than zero,
             - a peak within the scanned frequency range,
             - a standard deviation that is not larger than the scanned frequency range,
             - a standard deviation that is wider than the smallest frequency increment,
@@ -149,7 +149,7 @@ def _evaluate_quality(self, fit_data: curve.CurveFitResult) -> Union[str, None]:
             fit_data.x_range[0] <= fit_freq.n <= fit_data.x_range[1],
             1.5 * freq_increment < fit_kappa.n,
             fit_width_ratio < 0.25,
-            fit_data.reduced_chisq < 3,
+            0 < fit_data.reduced_chisq < 3,
             curve.utils.is_error_not_significant(fit_kappa),
             snr > 2,
         ]

qiskit_experiments/framework/__init__.py

@@ -101,6 +101,7 @@
 .. autosummary::
     :toctree: ../stubs/
 
+    CompositeExperiment
     ParallelExperiment
     BatchExperiment
     CompositeAnalysis
@@ -143,6 +144,7 @@
 from .composite import (
     ParallelExperiment,
     BatchExperiment,
+    CompositeExperiment,
     CompositeAnalysis,
 )
 from .json import ExperimentEncoder, ExperimentDecoder

qiskit_experiments/framework/composite/__init__.py

@@ -13,6 +13,7 @@
 """Composite Experiments"""
 
 # Base classes
+from .composite_experiment import CompositeExperiment
 from .composite_analysis import CompositeAnalysis
 
 # Composite experiment classes