Merge branch 'dev' into 'master'

Dev

Closes #233, #232, #231, #230, #229, #221, #223, #215, and #220

See merge request it/syncopy!87

README.rst

@@ -10,7 +10,7 @@ electrophysiology data. The goals of the project are:
    data analysis. 
 2. Syncopy is *scalable* for large datasets. It will automatically make use of the
    available computing resources by being developed with built-in parallelism. 
-3. Syncopy is *compatible with Fieldtrip*. 
+3. Syncopy is *compatible with FieldTrip*. 
     - data and results can loaded into MATLAB and Python
     - parameter names and function call syntax are as similar as possible
 

doc/source/README.rst

@@ -10,8 +10,16 @@
 Welcome to Syncopy's Documentation!
 ===================================
 
-Syncopy (Systems Neuroscience COmputing in Python) is a
-Python toolkit for user-friendly, large-scale electrophysiology data analysis.
+Syncopy (Systems Neuroscience COmputing in Python) is a Python toolkit for
+user-friendly, large-scale electrophysiology data analysis. The goals of the
+project are:
+
+1. Syncopy provides a fully *open source* Python environment for reproducable
+   electrophysiology data analysis.
+2. Syncopy is *scalable* for large datasets. It automatically makes use of the
+   available computing resources by being developed with built-in parallelism.
+3. Syncopy is *compatible* with the MATLAB toolbox FieldTrip.
+
 
 Contents
 ========

doc/source/developer/datatype.rst

@@ -1,15 +1,13 @@
 .. _syncopy-data-classes:
 
-Syncopy data classes 
+Syncopy data classes
 ====================
-
-
 
 The data structure in Syncopy is based around the idea that all
 electrophysiology data can be represented as multidimensional arrays. For
 example, a multi-channel local field potential can be stored as a
 two-dimensional `float` array with the dimensions being time (sample) and
-channel. Hence, 
+channel. Hence,
 
 .. note:: Each Syncopy data object is simply an anotated multi-dimensional array.
 

doc/source/developer/developers.rst

@@ -3,14 +3,16 @@ Developer's guide to Syncopy
 ****************************
 
 The following information is meant for advanced users with an understanding of
-class hierarchies.
+class hierarchies that 
 
 .. toctree::
     :glob:
 
     datatype
     io
     tools
+    compute_kernels
     developer_api
+
 
 

doc/source/developer/tools.rst

@@ -38,6 +38,8 @@ corresponding stages here are
 2. Wrapper class: :class:`syncopy.specest.MultiTaperFFT`
 3. Metafunction: :func:`syncopy.freqanalysis` 
 
+.. image:: ../_static/ComputationalRoutine.png
+
 For a detailed walk-through explaining the intricacies of writing an analysis
 routine, please refer to the :doc:`compute_kernels`.
 

doc/source/quickstart.rst

@@ -4,16 +4,77 @@ Getting started with Syncopy
 Installing Syncopy
 ------------------
 
+Until the end of the early alpha development phase, Syncopy is only hosted on
+`TestPyPI <https://test.pypi.org/project/syncopy/>`_. It can be installed using
+`Pip <https://pypi.org/project/pip/>`_:
 
+.. code-block:: bash
+
+    pip install -i https://test.pypi.org/simple/ syncopy
+
+Eventually, Syncopy will be hosted on `PyPI <https://pypi.org/>`_ and
+`conda-forge <https://conda-forge.org/>`_. If you're working on the ESI cluster
+installing Syncopy is only necessary if you create your own environment.
 
-Setting up your environment
----------------------------
+Setting up your Python environment
+----------------------------------
+
+On the ESI cluster, ``/opt/ESIsoftware/python/envs/syncopy`` provides a
+pre-configured and tested Conda environment with the most recent Syncopy
+version. This environment can be easily started using the `ESI JupyterHub
+<http://jupyterhub.esi.local>`_
+
+Syncopy makes heavy use of temporary files, which may become large (> 100 GB).
+The storage location can be set using the `environmental variable
+<https://linuxhint.com/bash-environment-variables/>`_ :envvar:`SPYTMPDIR`, which
+by default points to your home directory:
 
 .. code-block:: bash
 
     SPYTMPDIR=~/.spy
 
+The performance of Syncopy is strongly depends on the read and write speed into
+this folder. On the `ESI JupyterHub <http://jupyterhub.esi.local>`_, the
+variable is set to use the high performance storage:
+
+.. code-block:: bash
+
+    SPYTMPDIR=/mnt/hpx/home/$USER/.spy
 
 
 Importing Syncopy
------------------
+-----------------
+
+To start using Syncopy you have to import it in your Python code:
+
+.. code-block:: python
+
+    import syncopy as spy
+
+All :doc:`user-facing functions and classes <user/user_api>` can then be
+accessed with the ``spy.`` prefix, e.g.
+
+.. code-block:: python
+
+    spy.load("~/testdata.spy")
+
+
+Starting up parallel workers
+----------------------------
+
+In Syncopy all computations are designed to run in parallel taking advantage of
+modern multi-core system architectures. However, before any computation you
+have to initialize your "cluster" of parallel workers, which are the CPU cores
+if run on a single machine (laptop or workstation) or compute jobs if run on a
+computing cluster (e.g. SLURM).
+
+To initialize the cluster, use the :func:`~syncopy.esi_cluster_setup` function.
+For example,
+
+.. code-block:: python
+
+    spy.esi_cluster_setup(n_jobs=10)
+
+will start 10 parallel workers on the SLURM cluster of the ESI. If the
+:func:`~syncopy.esi_cluster_setup` function is not run before any computation,
+all following computations will be computed sequentially.

doc/source/user/data_handling.rst

@@ -1,12 +1,22 @@
 Handling data in Syncopy
 ========================
 
-Coming soon
+Syncopy utilizes a simple data format based on `HDF5
+<https://portal.hdfgroup.org/display/HDF5/HDF5>`_ and `JSON
+<https://en.wikipedia.org/wiki/JSON>`_ (see :doc:`../developer/io` for details).
+These formats were chosen for their *ubiquity* as they can be handled well in
+virtually all popular programming languages, and for allowing *streaming,
+parallel access* enabling computing on parallel architecures.
+
+Currently, data in other formats (e.g. from  a recording system) have to be
+converted first. For this purpose, later versions of Syncopy will include
+importing and exporting engines, for example based on `Neo
+<https://neo.readthedocs.io/en/latest/>`_ or `NWB <https://www.nwb.org/>`_.
 
 
 Reading and saving Syncopy (``*.spy``) data
 -------------------------------------------
-.. autosummary::    
+.. autosummary::
 
     syncopy.load
     syncopy.save
@@ -17,8 +27,7 @@ Functions for editing data in memory
 ------------------------------------
 These functions are useful for editing and slicing data:
 
-.. autosummary::      
-
-    syncopy.selectdata
+.. autosummary::
+
     syncopy.definetrial
     syncopy.padding

doc/source/user/fieldtrip.rst

@@ -1,21 +1,25 @@
-Syncopy for Fieldtrip users
+Syncopy for FieldTrip users
 ===========================
 
 Syncopy is written in the `Python programming language
 <https://www.python.org/>`_ using the `NumPy <https://www.numpy.org/>`_ and
-`SciPy <https://scipy.org/>`_ libraries. However, it's call signatures and
+`SciPy <https://scipy.org/>`_ libraries for computing as well as `Dask
+<https://dask.org>`_ for parallelization. However, it's call signatures and
 parameter names are designed to mimick the `MATLAB <https://mathworks.com>`_
-analysis toolbox `Fieldtrip <http://www.fieldtriptoolbox.org>`_. 
+analysis toolbox `FieldTrip <http://www.fieldtriptoolbox.org>`_.
 
-The scope of Syncopy is limited to only parts of Fieldtrip, in particular
+The scope of Syncopy is limited to only parts of FieldTrip, in particular
 spectral analysis of electrophysiology data. Therefore, M/EEG-specific routines
-such as loading M/EEG file types, source localization, ..., are not part of
-Syncopy. For a Python toolbox tailored to M/EEG data analysis, see for example
-the `MNE Project <https://www.martinos.org/mne/>`_.
+such as loading M/EEG file types, source localization, ..., are currently not
+covered by Syncopy. For a Python toolbox tailored to M/EEG data analysis, see
+for example the `MNE Project <https://www.martinos.org/mne/>`_.
 
+.. contents::
+    Contents
+    :local:
 
-Translating MATLAB code Python
-------------------------------
+Translating MATLAB code to Python
+---------------------------------
 
 For translating code from MATLAB to Python there are several guides, e.g.
 
@@ -24,23 +28,25 @@ For translating code from MATLAB to Python there are several guides, e.g.
 * `MATLAB to Python - A Migration Guide by Enthought <https://www.enthought.com/white-paper-matlab-to-python>`_
 
 Key differences between Python and MATLAB
------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 While the above links cover differences between Python and MATLAB to a great
-extent, we highlight here what we think are the most important differences: 
+extent, we highlight here what we think are the most important differences:
 
-* Python array indexing starts at 0
-* The end of a range in Python is not included
+* Indexing is different: Python array indexing starts at 0. The end of a range
+  in Python is not included
 * Data in Python is not necessarily copied and may be manipulated in-place.
-* Namespaces
+* The powerful `import system of Python <https://docs.python.org/3/reference/import.html>`_
+  allows simple function names (e.g., :func:`~syncopy.load`) without worrying
+  about overwriting built-in functions.
+* `Project-specific environments <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>`_
+  allow reproducable and customizable working environments.
 
-
-Translating Fieldtrip calls to Syncopy
+Translating FieldTrip calls to Syncopy
 --------------------------------------
 
-Using a Fieldtrip function in MATLAB usually works via constructing a ``cfg``
-``struct`` that contains all configured parameters, e.g. frequency and time
-limits for spetral analysis:
+Using a FieldTrip function in MATLAB usually works via constructing a ``cfg``
+``struct`` that contains all configured parameters:
 
 .. code-block:: matlab
 
@@ -63,9 +69,48 @@ with the defaults of any function.
     cfg.option2 = [10, 20]
     result = spy.something(cfg=cfg)
 
+A FieldTrip power spectrum in Syncopy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For example, a power spectrum calculated with FieldTrip via
+
+.. code-block:: matlab
+      
+    cfg = [];
+    cfg.method = 'mtmfft';
+    cfg.foilim = [1 150];
+    cfg.output = 'pow';
+    cfg.taper = 'dpss';
+    cfg.tapsmofrq = 10;
+    spec = ft_freqanalysis(cfg, data)
+
+can be computed in Syncopy with
+
+.. code-block:: matlab
+      
+    cfg = spy.get_defaults(spy.freqanalysis)
+    cfg.method = 'mtmfft';
+    cfg.foilim = [1, 150];
+    cfg.output = 'pow';
+    cfg.taper = 'dpss';
+    cfg.tapsmofrq = 10;
+    spec = spy.freqanalysis(cfg, data)
+
+
+Key differences between FieldTrip and Syncopy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* FieldTrip has more features. Syncopy is still in early development and will
+  never cover the rich featureset of FieldTrip.
+* FieldTrip supports many data formats. Syncopy
+* Syncopy data objects are never fully loaded into memory.
+
+
 
-Key differences between Fieldtrip and Syncopy
+Exchanging data between FieldTrip and Syncopy
 ---------------------------------------------
 
-* Data is only loaded into memory on demand
-* ...
+Data created with Syncopy can be loaded into MATLAB using the `matlab-syncopy
+<http://git.esi.local/it/matlab-syncopy>`_ interface. It's still in early
+development and supports only a subset of data classes. Also, the MATLAB
+interface does not support loading data larger than local memory.

doc/source/user/users.rst

@@ -7,7 +7,8 @@ who primarily want to apply existing analysis functions on their data. This
 usually entails writing analysis scripts operating on a given list of data
 files.
 
-.. toctree :: 
+.. toctree ::
+    :maxdepth: 2
 
     data_handling
     fieldtrip

syncopy/datatype/base_data.py

@@ -4,7 +4,7 @@
 # 
 # Created: 2019-01-07 09:22:33
 # Last modified by: Stefan Fuertinger [[email protected]]
-# Last modification time: <2019-09-25 16:57:28>
+# Last modification time: <2019-10-08 14:08:32>
 
 # Builtin/3rd party package imports
 import getpass
@@ -1163,6 +1163,8 @@ class FauxTrial():
         must slice `data` correctly so that ``obj.data[idx] == obj.trials[k]``
     dtype : :class:`numpy.dtype`
         Datatype of source trial array
+    dimord : list
+        Dimensional order of source trial array
         
     Returns
     -------
@@ -1178,10 +1180,11 @@ class FauxTrial():
     syncopy.continuous_data.ContinuousData._preview_trial : makes use of this class
     """
 
-    def __init__(self, shape, idx, dtype):
+    def __init__(self, shape, idx, dtype, dimord):
         self.shape = tuple(shape)
         self.idx = tuple(idx)
         self.dtype = dtype
+        self.dimord = dimord
 
     def __str__(self):
         msg = "Trial placeholder of shape {} and datatype {}"
@@ -1198,15 +1201,15 @@ def squeeze(self):
         shp = list(self.shape)
         while 1 in shp:
             shp.remove(1)
-        return FauxTrial(shp, self.idx, self.dtype)
+        return FauxTrial(shp, self.idx, self.dtype, self.dimord)
 
     @property
     def T(self):
         """
         Return a new `FauxTrial` instance with reversed dimensions
         (parroting the NumPy original :func:`numpy.transpose`)
         """
-        return FauxTrial(self.shape[::-1], self.idx[::-1], self.dtype)
+        return FauxTrial(self.shape[::-1], self.idx[::-1], self.dtype, self.dimord[::-1])
 
 
 class Selector():
@@ -1418,7 +1421,7 @@ def trials(self, dataselect):
         except Exception as exc:
             raise exc
         if not set(trials).issubset(trlList):
-            lgl = "List/array of values b/w 0 and {}".format(trlList[-1])
+            lgl = "list/array of values b/w 0 and {}".format(trlList[-1])
             act = "Values b/w {} and {}".format(min(trials), max(trials))
             raise SPYValueError(legal=lgl, varname=vname, actual=act)
         self._trials = trials
@@ -1748,7 +1751,7 @@ def _selection_setter(self, data, select, dataprop, selectkey):
                 else:
                     targetArr = np.arange(target.size)
                 if not set(selection).issubset(targetArr):
-                    lgl = "List/array of {} names or indices".format(dataprop)
+                    lgl = "list/array of {} existing names or indices".format(dataprop)
                     raise SPYValueError(legal=lgl, varname=vname)
 
                 # Preserve order and duplicates of selection - don't use `np.isin` here!