Finish first draft of basic usage

Mercury.ipynb

@@ -806,12 +806,12 @@
     "\n",
     "# Run and get results\n",
     "with clock_time():\n",
-    "    results, results_seq = mercury.run(scenarios=scenarios,\n",
-    "                                       case_studies=case_studies,\n",
-    "                                        paras_sc_iterated=paras_sc_iterated,\n",
-    "                                        paras_sc_fixed=paras_sc_fixed,\n",
-    "                                        paras_simulation=ps\n",
-    "                                      )"
+    "        results, results_seq = mercury.run(scenarios=scenarios,\n",
+    "                                           case_studies=case_studies,\n",
+    "                                            paras_sc_iterated=paras_sc_iterated,\n",
+    "                                            paras_sc_fixed=paras_sc_fixed,\n",
+    "                                            paras_simulation=ps\n",
+    "                                          )"
    ]
   },
   {

docs/Advanced usage.rst

@@ -7,6 +7,7 @@ This sections covers the advanced usage of Mercury, in particular how to define
 messaging system, and how to expand the model door-to-door.
 
 .. toctree::
+    Parametriser and aggregator.rst
     code_organisation.rst
     Modules.rst
     External messaging system.rst

docs/Basic usage.rst

@@ -7,11 +7,10 @@ This sections covers the basic usage of Mercury, in particular how to run it, ho
 get the output from the model.
 
 .. toctree::
-    How to use Mercury.rst
-    CLI.rst
-    Notebook.rst
-    GUI.rst
-    Simulation parameters.rst
-    Scenario parameters.rst
+    How to run Mercury.rst
     Data Input.rst
     Data Output.rst
+    CLI.rst
+    Programmatic use.rst
+    GUI.rst
+

docs/CLI.rst

@@ -10,12 +10,63 @@ following:
 
     ./mercury.py -id -1
 
-This will for instance run one iteration of the basic scenario "-1", reading the default `config/mercury_config.toml`
+This will for instance run one iteration of the basic scenario "-1", reading the default ``config/mercury_config.toml``
 config file.
 
 The CLI can be used in a very agile way. Indeed, all parameters from the mercury config are accessible through the
-interface (the list of parameter is built at runt-time, so even new custom parameters will be accessible. All parameters
+interface, as well as the parameters defined in the scenario (the list of parameter is built at runtime, so even new custom parameters will be accessible). All parameters
 are listed in :ref:`simulation_parameters` but some have shortcuts and the most important ones are explained here:
 
 - "-id": the id of the scenario to run. If "-id X" is run, there must be a "scenario=X" folder in the input folder.
+  Compulsory.
+- "-psi": path to the toml mercury config file. Default is ``config/mercury_config.toml``.
+- "-cs": the id of the case study to run. If "-id X -cs Y" is run, there must be a "scenario=X/case_studies/case_study=Y"
+  folder in the input directory.
+- "-n": number of iterations to run a given set of parameters.
+- "-fl": once the input data is stable, this option can be used to speed up the data loading.
+- "-pc": number of core to use in parallel for multiple iterations (a single run of Mercury is always single-core and
+  single thread)
+
+On top of that, oen can use the cli to set and iterate over parameters. Iterable parameters include scenario ids, case
+study ids. For instance, this command performs one iteration on scenario -1 and one iteration on scenarios -2:
+
+.. code:: bash
+
+    ./mercury.py -id -1 -2
+
+Any parameter defined in the scenario (or case_study) parameter file can also be fixed or iterated through the CLI.
+For instance, if the price of fuel is defined as:
+
+.. code:: toml
+
+    [paras]
+        [paras.airlines]
+        fuel_price = 0.1
+
+in the scenario config file, then one can iterate over it like this:
+
+.. code:: bash
+
+    ./mercury.py -id -1 -airlines__fuel_price 0.3 0.5 0.7
+
+Note that by specifying iterations over several parameters, one ends up always with the combination of parameters, for
+instance:
+
+.. code:: bash
+
+    ./mercury.py -id -1 -2 -airlines__fuel_price 0.3 0.5 0.7 -n 2
+
+will perform 2 iterations on scenario -1 with a fuel price of 0.3, 2 iterations of scenario -1 with a fuel price
+of 0.5, etc.
+
+By default, the CLI will save two kinds of results:
+
+- some aggregated results, by default saved in ``../results/results.csv``
+- some detailed results on each flight, passenger, airport, etc, by default in the
+  ``../results/[model version]_[scenario id]_[case study id]_[iteration number]`` folder
+
+More details on the output data can be found in :ref:`data_output`.
+
+Note: iteration over string parameters will not work through the CLI.
+
 

docs/Data Input.rst

@@ -1,19 +1,85 @@
+.. _data_input:
+
 Data Input
-----------
+==========
+
+Running Mercury requires to input some data in the right format.
+
+Note: a sample of data is provided with Mercury and can be downloaded
+`here <https://zenodo.org/records/11384379/files/Mercury_data_sample.zip?download=1>`_.
+
+The input data is organised in scenarios and case studies. Scenarios can be seen typically as a set of schedules,
+passenger itineraries, as well as the definition other agents like the AMAN. A case study is usually represented by a
+subset of flights and/or different operational configuration. When running Mercury, one has to specify as least the id
+of the scenario, and optionally the if of the case study. If no case study is chosen, Mercury will run the case "0",
+coinciding exactly with the data and parameters defined by APIthe scenario itself.
 
-- input
+The scenarios are read from the input folder, defined in the ``mercury_config.toml`` file, by default ``../input``. The
+input folder should follow the following structure:
 
-  - scenario=-1
+- ``input``
 
-    - scenario_config.toml
-    - data -> with all base data
-    - case_studies -> with all data replacing the base data.
+  - ``scenario=-1``
 
-      - case_study=0
+    - ``scenario_config.toml``
+    - ``data`` -> with all base data
+    - ``case_studies`` -> with all data replacing the base data.
 
-        - case_study_config.toml
-        - data
-  - scenario=0
+      - ``case_study=0``
+
+        - ``case_study_config.toml``
+        - ``data``
+  - ``scenario=0``
 
     - etc...
 
+The first important file in each scenario is the ``scenario_config.toml`` file. This file is organised in two sections.
+The first one, under the header ``data``, is a list of all the table that are needed for the simulation, and what is their
+specific names in this scenario folder, for instance:
+
+.. code:: toml
+
+    [data.delay]
+    input_delay_paras = 'delay_parameters'
+
+    [data.network_manager]
+    input_atfm_delay = 'iedf_atfm_static'
+    input_atfm_prob = 'prob_atfm_static'
+
+The second part of this file is composed by parameters and their values, for instance:
+
+.. code:: toml
+
+    [paras.modules]
+    modules_to_load = ['CM']
+    path = 'modules'
+
+    [paras.airlines]
+    non_ATFM_delay_loc =  0.0
+    compensation_uptake = 0.11
+    delay_estimation_lag = 60
+
+Like in the first part, the parameters are organised in different sections and subsections, here for instance "modules"
+and "airlines". This important when using the CLI or the programmatic interface, because the parameters have to be called
+based on their subsections, for instance "airlines__non_ATFM_delay_loc", i.e. the name of the section, two underscores
+``__``, then the name of the parameter. A full description of the parameters can be found here:
+:ref:`scenarios_parameters`.
+
+The tables listed in the first part have to be included in the scenario folder, in ``data``, following the same structure
+than the toml file. All tables must be in parquet format.
+
+A detailed description of the all the input tables can be found here: :ref:`input_tables`.
+
+Finally, note that the GUI version of Mercury provides an easy way of exploring the different types of data, modifying
+them, creating scenarios, etc. More information here: :ref:`gui`.
+
+
+
+
+
+
+
+
+
+
+

docs/Data Output.rst

@@ -1,4 +1,38 @@
 .. _data_output:
 
 Data Output
-===========
+===========
+
+Mercury is an agent-based model with a high number of agents, therefore producing potentially high volume of data
+on each agents.
+
+The main output of Mercury is a set of tables, stored in the output directory defined by the variable
+``write_profile.path`` in the mercury config file (by default ``../results``).
+These tables (saved as ``.csv.gz`` files) are stored in a folder whose name is structured as
+``[model version]_[scenario id]_[case study id]_[iteration number]``. One can also insert a timestamp in folder name
+by setting the ``outputs_handling.insert_time_stamp`` parameter to True in the mercury config file.
+
+The two most important tables saved by Mercury are the following:
+
+- output_flights: compiles important information the flights, including fuel consumption, delay, etc.
+- output_pax: gathers information related to passengers, in particular their final delay, whether they missed
+  connections etc.
+
+Other types of data are also saved at the save time, including:
+
+- output_dci: information related to Dynamic Cost Indexing when used.
+- output_eaman: information related to the optimisation process followed by the EAMAN.
+- output_events: information related to various events in the simulation, mostly used for benchmarking.
+- output_general_simulation: information related to the simulation itself.
+- output_hotspot: information related to the resolution of hotspots in the airspace.
+- output_messages: information related to internal messages during the simulation, mostly used for benchmarking.
+- output_RNG: information related to the Random Number Generator (probably not working at the moment).
+- output_wfp: information related to the "Wait For Passenger" process, whereby airlines may wait for late connecting
+  passengers
+
+Finally, a copy of all the parameters used for this simulation is saved as a pickled dictionary ``paras.pic``, and
+a script ``unzip_results.py`` is included to easily unzip all the files.
+
+The description of all fields appearing in the output tables can be found here: :ref:`output_tables`.
+
+

docs/How to use Mercury.rst → docs/How to run Mercury.rst

@@ -1,22 +1,22 @@
-.. _how_to_use_mercury:
+.. _how_to_run_mercury:
 
-How to use Mercury
+How to run Mercury
 ==================
 
 There are three entry points to the model, all in the root folder:
 
-- mercury.py: CLI interface. Useful for running the simulator on a cluster for instance. See :ref:`cli`.
-- Mercury.ipynb: Jupyter notebook. Useful to play with the Mercury object, importable in other scripts. See :ref:`notebook`.
-- mercury_gui.py: Dash interface. Useful to explore the input data and prepare new datasets. See :ref:`gui`.
+- ``mercury.py``: CLI interface. Useful for running the simulator on a cluster for instance. See :ref:`cli`.
+- ``Mercury.ipynb``: Jupyter notebook. Useful to play with the Mercury object, importable in other scripts. See :ref:`notebook`.
+- ``mercury_gui.py``: Dash interface. Useful to explore the input data and prepare new datasets. See :ref:`gui`.
 
 To run the simulator all of them use the same underlying engine. The behaviour of the engin can be driven via parameter
-files and input data. The main configuration file for Mercury can be found in `config/mercury_config.toml`. This file
+files and input data. The main configuration file for Mercury can be found in ``config/mercury_config.toml``. This file
 contains parameters related to **how** Mercury runs, for instance the location of the input data, their format,
 parallelisation, etc. By default, the three interfaces will read this file for the parameters. You can also use another
-parameter file, for instance passing to the CLI interface the option `-psi path/to/my_custom_profile_file.toml`. More
+parameter file, for instance passing to the CLI interface the option ``-psi path/to/my_custom_profile_file.toml``. More
 details can be found in :ref:`cli`. All parameters are listed here: :ref:`config_parameter_file`.
 
-The config parameter includes in particular the path to the data, by default `../input`. The input folder needs to be
+The config parameter includes in particular the path to the data, by default ``../input``. The input folder needs to be
 organised as explained in :ref:`data_input` to be readable by Mercury. In particular, it needs to include a file called
 "scenario_config.py", which is compiling all the necessary information to run this particular scenario. It is organised
 in two parts:
@@ -26,11 +26,14 @@ in two parts:
 
 All scenario parameters are described here: :ref:`scenario_parameter_file`.
 
-The mercury config also includes the path to where the results will be saved, but default in `../results`. The structure
+The mercury config also includes the path to where the results will be saved, but default in ``../results``. The structure
 of the results is discussed and explained in :ref:`data_output`.
 
 All parameters both from the mercury_config.toml file and the scenario parameter file can be set at runtime, in the cli
-version, with the Mercury object, or in the GUI (e.g. using the argument `--airlines__fuel_price 1.2` in the CLI).
+version, with the Mercury object, or in the GUI (e.g. using the argument ``--airlines__fuel_price 1.2`` in the CLI).
+
+Finally, the three entry points (CLI, Mercury object, GUI) are available as Docker images, which requires only the
+installation of the docker third-party app. More information can be found here: :ref:`docker`.
 
 
 

docs/IO specifications.rst

@@ -0,0 +1,13 @@
+.. _io_specs:
+
+Input/Output specifications
+===========================
+
+This sections describes in details all the fields present in input or output tables, as well as all the simulation and
+scenario parameters included in the simulator.
+
+.. toctree::
+    Simulation parameters.rst
+    Scenario parameters.rst
+    Input tables.rst
+    Output tables.rst

docs/Input tables.rst

@@ -0,0 +1,4 @@
+.. _intput_tables:
+
+Input table specifications
+==========================

docs/Output tables.rst

@@ -0,0 +1,4 @@
+.. _output_tables:
+
+Output table specifications
+===========================

docs/Overview.rst

@@ -31,9 +31,12 @@ Mercury is particularly well suited to answer to problem related to:
 - passenger centric-issues, like modifications to passenger compensation.
 
 If you want to have a quick start, go there: :ref:`quickstart`. If you want have more details about how to run Mercury,
-like how the options to run it and the input/output data specs, go there: :ref:`basic_usage`. If you need more details
-about advanced usage of Mercury like Module creation, go there: :ref:`advanced_usage`. If you want to have a look at the
-underlying model, go there: :ref:`model_presentation`. If you want to access the API, go there: :ref:`api`.
+like the options and the input/output data specs, go there: :ref:`basic_usage`. If you need more details
+about advanced usage of Mercury like Module creation, go there: :ref:`advanced_usage`. If you need a detailed description
+of all tables in input and output, as well as all the parameters of the model, go there: :ref:`io_specs`.
+If you want to know more about the underlying model, go there: :ref:`model_presentation`.
+If you want more details about the code itself, go there: :ref:`api`.
+
 
 Quick FAQ
 ---------

docs/Parametriser and aggregator.rst

@@ -0,0 +1,13 @@
+.. _parametriser_aggregator:
+
+Parametriser and Aggregator
+===========================
+
+
+Parametriser
+------------
+
+
+Aggregator
+----------
+