Skip to content
This repository has been archived by the owner on Apr 26, 2023. It is now read-only.

Creating a Plugin

Jump to bottom
Yunior C. Fonseca Reyna edited this page Feb 20, 2019 · 19 revisions
Scipion Logo

A Plugin for Scipion is a Python module that have some extra requirements. Each Plugin contains classes for protocols, viewers, wizards, etc…​allowing to use the functionality of a given EM package within the Scipion framework. Prior to version XXX, the plugins code was not completely isolated, it was under the scipion/pyworkflow/em/packages folder and hosted in the same central repository.

For the release XXX ("the pluginization"), we worked hard to make the entire system more de-centralized and more standard. For that we re-factored the plugins to make them standard Python modules and host their code in independent repositories and with the possibility different release cycles.

This document is organized in two main parts: (1) Explanation of the structure of a plugin as a Python module and the required submodules and functionality for Scipion. (2) Necessary files structure and the process for distributing a Plugin in the Python de-facto repository PyPI. We will use Relion as an example, so we recommend to clone and take a look at the repository here.

Naming Conventions

Similar to style guides, conventions are about consistency. Consistency is crucial for a project such as Scipion and its distributed nature. Despite the final decision is up to the developer of a package, we STRONGLY recommend the following conventions:

  • Repository name: scipion-em-packagename

  • Python package name: scipion-em-packagename

  • Python module with package code: packagename

This means that it will be a folder named scipion-em-package with the structure of a standard Python package (explained below in the second part) and it will contains a folder package that is a Python module with functions and classes required by Scipion (explained below).

Plugin Overview

Files structure

The file and folder structure follows the convention established for Python modules. For example, the Relion plugin has the following structure:

$ cd ~/work/development/scipion-em-plugins/scipion-em-relion
$ tree
.
├── LICENSE
├── .gitignore
├── CHANGES.txt
├── README.rst
├── MANIFEST.in
├── setup.py
└── relion
    ├── __init__.py
    ├── bibtex.py
    ├── constants.py
    ├── convert
    │   ├── __init__.py
    │   ├── convert.py
    │   └── dataimport.py
    ├── protocols
    │   ├── __init__.py
    │   ├── protocol_autopick.py
    │   ├── protocol_autopick_v2.py
    │   ├── protocol_base.py
    │   ├── protocol_center_averages.py
    │   ├── protocol_classify2d.py
    │   ├── protocol_classify3d.py
    │   ├── protocol_create_mask3d.py
    │   ├── ...
    ├── tests
    │   ├── __init__.py
    │   ├── test_convert_relion.py
    │   ├── test_protocols_relion.py
    │   └── test_workflow_relion.py
    ├── viewers.py
    └── wizards.py
    └── protocols.conf

Standard submodules

Scipion plugins are usually composed by the following submodules:

__init__.py

This file is required in Python modules and it is executed when the module is imported. It is common to add here imports related with Scipion’s pyworkflow, and to import values from the constants.py file. We also define the logo and plugin-level references:

import os
import pyworkflow.em
import pyworkflow.utils as pwutils

from .constants import RELION_HOME, V2_0, V2_1, RELION_CUDA_LIB

_logo = "relion_logo.png"
_references = ['Scheres2012a', 'Scheres2012b', 'Kimanius2016']
define the Plugin class

Additionally, it is necessary to add a Plugin class (subclass from pyworkflow.em.Plugin), which contains much of the logic related with the plugin’s variables, environment, associated binaries and paths.

class Plugin(pyworkflow.em.Plugin):
    _homeVar = RELION_HOME
    _supportedVersions = [V2_0, V2_1]
_homeVar

In the case of scipion-em-relion, the plugin is associated to some binaries. In _homeVar, we point to the variable that has the path to the binaries. It is a good practice to define this in constants.py, as it is done here. As we will see later, this has a default value, but it can also be overwritten by the user.

_supportedVersions

Here we store which versions of the binaries are supported by this plugin.

_defineVariables

Here is where we give the Plugin’s environment variables a default value. In the case of Relion, we only have the RELION_HOME, which points to the binaries of the plugin and by default would be 'relion-2.1' 

  @classmethod
  def _defineVariables(cls):
      cls._defineEmVar(RELION_HOME, 'relion-2.1')
There are two functions defined in pyworkflow/plugin.py that may be useful here: _defineEmVar and _defineVar. The first one will add the path to software/em to the variable (which is the default place to install binaries). The second will store the value as is. We only need defineEmVar in Relion, since the binary location is the variable we’ll declare.
getEnviron

We can also overwrite the function getEnviron if there are any modifications we need to do in the environment variables in order to run the plugin. 

  @classmethod
  def getEnviron(cls):
      """ Setup the environment variables needed to launch Relion. """

      environ = pwutils.Environ(os.environ)
      binPath = cls.getHome('bin')
      libPath = cls.getHome('lib') + ":" + cls.getHome('lib64')

      if binPath not in environ['PATH']:
          environ.update({'PATH': binPath,
                          'LD_LIBRARY_PATH': libPath,
                          'SCIPION_MPI_FLAGS': os.environ.get('RELION_MPI_FLAGS', ''),
                          }, position=pwutils.Environ.BEGIN)

      # Take Scipion CUDA library path
      cudaLib = environ.getFirst((RELION_CUDA_LIB, 'CUDA_LIB'))
      environ.addLibrary(cudaLib)

      return environ
Implement validateInstallation() (Optional)

In the plugin class, we can overwrite the validateInstallation function. In the case of Relion this is not overwritten, so this plugin will use Scipion’s default validate installation. You can check the current implementation in pyworkflow/plugin.py.

Defining the plugin binaries (Optional)

The next step is to add the code responsible for the installation of the binaries. We redefine defineBinaries in our RelionPlugin class in the init.py. Please note how we have added default=True to the latest relion binaries. - this means that this binary will be installed automatically when we get this plugin (unless specified otherwise). 

# this goes inside class RelionPlugin(Plugin):
    def defineBinaries(self, env):
        relion_commands = [('./INSTALL.sh -j %d' % env.getProcessors(),
                                  ['relion_build.log',
                                   'bin/relion_refine'])]

        env.addPackage('relion', version='1.4',
                       tar='relion-1.4.tgz',
                       commands=relion_commands)

        env.addPackage('relion', version='1.4f',
                       tar='relion-1.4_float.tgz',
                       commands=relion_commands)

        # Define FFTW3 path variables
        relion_vars = [('FFTW_LIB', SW_LIB),
                       ('FFTW_INCLUDE', SW_INC)]

        relion2_commands = [('cmake -DGUI=OFF -DCMAKE_INSTALL_PREFIX=./ .', []),
                            ('make -j %d' % env.getProcessors(), ['bin/relion_refine'])]

        env.addPackage('relion', version='2.0',
                       tar='relion-2.0.4.tgz',
                       commands=relion2_commands,
                       updateCuda=True,
                       vars=relion_vars)

        env.addPackage('relion', version='2.1',
                      tar='relion-2.1.tgz',
                      commands=relion2_commands,
                      updateCuda=True,
                      vars=relion_vars,
                      default=True)
register the plugin

To finish, we must register the plugin with the following line. This is very important so that Scipion distinguishes this from other python modules as a plugin. 

pyworkflow.em.Domain.registerPlugin(__name__)

bibtex.py (Optional?)

This submodule is not supposed to be imported directly, it should contain the bibtex string literal as the Python doc string. Scipion will take care of parse the bibtex reference and incorporate into the plugin module. 

"""

@article{Scheres2012a,
title = "A Bayesian View on Cryo-EM Structure Determination ",
journal = "JMB",
volume = "415",
number = "2",
pages = "406 - 418",
year = "2012",
issn = "0022-2836",
doi = "http://dx.doi.org/10.1016/j.jmb.2011.11.010",
url = "http://www.sciencedirect.com/science/article/pii/S0022283611012290",
author = "Scheres, Sjors H.W.",
keywords = "cryo-electron microscopy, three-dimensional reconstruction, maximum a posteriori estimation "
}

@article{Scheres2012b,
title = "RELION: Implementation of a Bayesian approach to cryo-EM structure determination ",
journal = "JSB",
volume = "180",
number = "3",
pages = "519 - 530",
year = "2012",
issn = "1047-8477",
doi = "http://dx.doi.org/10.1016/j.jsb.2012.09.006",
url = "http://www.sciencedirect.com/science/article/pii/S1047847712002481",
author = "Scheres, Sjors H.W.",
keywords = "Electron microscopy, Single-particle analysis, Maximum likelihood, Image processing, Software development "
}
[. . .]

constants.py

This submodule should contain all the constants that can be later imported in protocols etc. If there are only few of them, there is no need for a separate constants.py file. 

from collections import OrderedDict

import pyworkflow.em.metadata as md

RELION_HOME = 'RELION_HOME'
RELION_CUDA_LIB = 'RELION_CUDA_LIB'

# Supported versions:
V2_0 = '2.0'
V2_1 = '2.1'

MASK_FILL_ZERO = 0
MASK_FILL_NOISE = 1
[. . .]

convert

This submodule might contain two files: convert.py with all functions used for conversion between base classes and programs inside the plugin; dataimport.py with import classes that are used in pyworkflow/em/protocol/protocol_import/. In cases when there are only few conversion functions, the submodule folder can be replaced by a single convert.py file.

protocols

In this submodule all the protocols of the plugin should be implemented. Usually a plugin provides many protocols, so the most common case is to have a submodule folder with its own __init__.py and one .py file per each protocol class. You can read more detailed information on the implementation of protocols here.

viewers

A plugin can also define viewers for existing objects or new protocols. Since many built-in viewers are provided by Scipion, plugins do not define many viewers, so a viewers.py will serve as submodule.

wizards

Wizards need to be defined for protocols/parameters, but many base classes are already provided. Here again the wizards.py submodule is usually enough.

tests

We strongly recommend to follow Test-Driven-Development, so this is the place where all plugin tests should go. It is important to create different test cases from the beginning of the plugin development.

logo.png

.gitignore

this file is required for Git. Here is an example: 

#### Eclipse and so on
.project
.cproject
.pydevproject
.classpath
.idea

#### Python
build/
dist/
*.egg-info/
*.egg
*.py[cod]
__pycache__/
*.so
*~

PyPI-related files

These files are required for PyPI distribution. More information can be found here and examples of the contents of these files can be found in Publishing the Plugin to PyPI

  • LICENSE: license file for plugin code

  • CHANGES.txt: version history

  • README.rst: long description of your plugin (for PyPI catalog)

  • MANIFEST.in: includes links to README and LICENSE files

  • setup.py: this is a build script for PyPI distribution, containing important information about your plugin. See below for an example of such file.

Testing

  • In your terminal, add the plugin directory to PYTHONPATH. This will make our plugin available as a python module when we launch scipion. While we develop and change our code, every time we launch Scipion we will have our changes available. 

    export PYTHONPATH=/path/to/scipion-em-relion

  • Check if all submodules are imported correctly 

    scipion run python scripts/inspect_plugins.py relion

  • List your tests and copy the one you want to run: 

    scipion test --show --grep relion

Publishing the Plugin to PyPI

We’ll explain below the steps followed to convert the package into a pip module that we can upload to pypi. Most of these steps are not scipion-specific, so it is recommended to check an external source (like this one or this one ) if you have doubts about pip packaging.

Adding PyPI files

We add four files to the folder scipion-em-relion: CHANGES.txt, setup.py, MANIFEST.in, README.rst.

setup.py

This is the most important one. It needs to call the setup function with, at least, the required arguments. Here we present a synthesized version: 

  from setuptools import setup, find_packages
  from codecs import open
  from os import path
  
  here = path.abspath(path.dirname(__file__))
  
  # Get the long description from the README file
  with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
      long_description = f.read()
  
  setup(
      name='scipion-em-relion',  # Required
      version='1.0.0a',  # Required
      description='A python wrapper to use relion within Scipion',  # Required
      long_description=long_description,  # Optional
      url='https://github.com/scipion-em/scipion-em-relion',  # Optional, but very important
      author='Relion authors',  # Optional
      author_email='[email protected]',  # Optional
      keywords='scipion cryoem imageprocessing scipion-1.2',  # Optional
      packages=find_packages(),
      package_data={  #!!!!!! Required if we have a logo!!!!!
         'relion': ['logo.png'],
      }
  
  )

CHANGES.txt (optional)

This file records a short description of the modifications made with each release of the pip package. 

v1.0.0, 23/04/2018 -- First commit

MANIFEST.in (optional)

The MANIFEST.in is needed so that our .txt file is included when we do the distribution (or many other non .py extensions, please check the official pip packaging guide if you need to include such files). IMPORTANT: if you have non-python files like images (except the logo), docs, scripts, you have to specify them here, otherwise they will be excluded from PyPi distribution! An example below recursively includes all files in spider/scripts folder. See more info here.

include *.txt
recursive-include spider/scripts *

Also, you will need to add/uncomment the following line into setup.py: include_package_data=True

Installing via pip locally

Remove the previous installation from Scipion

Remove binaries - if it applies. If you didn’t have a prior binary installation (i.e. you’re building this plugin new from scratch), go to next step. 

rm -rf $SCIPION_HOME/software/em/relion*

Working with a plugin in devel mode

If you want to use the sources of a plugin in a "live" way (meaning that changes in the plugin code will be reflected) you can use the PYTHONPATH as described above. Additionally, if you want to test the whole plugin, as a pip package (not only a python module) you can alternatively:

1.- git clone the plugin repository to any local folder, in case of a third party plugin

git clone [email protected]:scipion-em/scipion-em-myplugin.git /home/me/myplugin

2.- install it in "devel" mode:

$SCIPION_HOME/scipion installp -p /home/me/myplugin --devel

The PYTHONPATH approach will provide you with all execution features (protocols, wizzards, all should work). The only additional thing you are getting with this is testing the installation of the plugin as a pip package, or for convenience, to forget about the PYTHONPATH and, still have a live code reacting to latest git pulls or branch changes.

Get plugins.json

Scipion requests a json list of available plugins from http://scipion.i2pc.es/getplugins and uses metadata from pypi to filter which packages are available for the current Scipion version. Since we want to test our pip plugin before we upload it to pypi, we will read locally a file like the one provided in the website, with our plugin added.

  • In a directory of your choice, add a plugins.json file with the appropriate info for your plugin - you can save Scipion’s plugins.json and add your plugin’s data.

    plugins.json:

    {
    "scipion-em-relion": {
        "name":"relion",
        "pipName": "scipion-em-relion",
        "pluginSourceUrl":"/path/to/your/scipion-em-relion"
    }
}

    Note that when you add the key pluginSourceUrl, Scipion will use pip to install the plugin from that directory (i.e. pip will copy the relion folder to python’s site-packages folder). If this key is missing, Scipion will try to install from https://pypi.org/. Once you do this installation, changes made in your development folder will NOT be present in the copy used by Scipion. You would have to uninstall and go back to development mode using the variable PYTHONPATH.

  • In the VARIABLES section of your ~/.config/scipion/scipion.conf, add variable SCIPION_PLUGIN_JSON. If you don’t add this variable, Scipion will read the json from http://scipion.i2pc.es/getplugins instead of reading your local json copy. If you use pycharm to run Scipion, you can also add it as environment variable in your run configuration. Remember to replace the example provided with the right path:

    [VARIABLES]
SCIPION_NOTES_PROGRAM =
SCIPION_NOTES_ARGS =
SCIPION_NOTES_FILE = notes.txt
SCIPION_NOTIFY = False
SCIPION_PLUGIN_JSON=/home/desktop/yaiza/plugins.json

Installation script

Scipion has a script to handle plugin installation / uninstallation. Use this script in a new terminal or reset the PYTHONPATH variable that we defined at the beginning. We have a few choices (un)installation choices:

  • Installing plugin and default binaries: 

    $SCIPION_HOME/scipion installp -p scipion_grigoriefflab
    This command does two things:

    1. Gets the package from pypi

    2. Installs the default binaries (those that had default=True in the registerPluginBinaries function). If no errors happen, we get an output similar to this one: 

      /home/yaiza/git/scipion/software/bin/python /home/yaiza/git/scipion/scipion installp -p scipion-em-relion

Scipion  pluginization_install_config (2018-04-11) 0ee533a

python  /home/yaiza/git/scipion/install/install-plugin.py /home/yaiza/git/scipion/scipion installp -p scipion-em-relion
Building scipion-em-relion ...
python /home/yaiza/git/scipion/software/lib/python2.7/site-packages/pip install /home/yaiza/git/scipion-em-relion
Processing /home/yaiza/git/scipion-em-relion
Installing collected packages: scipion-em-relion
  Running setup.py install for scipion-em-relion: started
    Running setup.py install for scipion-em-relion: finished with status 'done'
Successfully installed scipion-em-relion-1.0a0
Done (1.01 seconds)
[. . .]
Building relion-2.1 ...
...Relion binaries installation log goes here
...
Done (0.20 seconds)

Process finished with exit code 0

  • Uninstalling plugin and all binaries installed 

    $SCIPION_HOME/scipion installp -p scipion-em-relion

  • We can use the flag --noBin to both install and uninstall without binaries: 

    $SCIPION_HOME/scipion installp -p scipion-em-relion --noBin

  • Install specific plugin binaries (only works if we have done installp first). 

    $SCIPION_HOME/scipion installb relion-2.1

  • Uninstall specific plugin binaries 

    $SCIPION_HOME/scipion uninstallb relion-2.0

Testing

  • With your plugin and binaries installed, it is recommended to run some of your plugin’s tests to check everything is in order: 

    scipion test em.packages.relion.tests.test_***

  • Open the test project: 

    scipion last
    First, inspect the protocol output to make sure there’s nothing weird; then, open the protocol box to see if our logo and references are there. It’s important to do this step because if we don’t open the GUI we won’t be able to detect logo related issues.

Add your own DataSet

If you need an additional dataset you can do this and host it where ever you want /can. Let’s assume you need a new dataset…​

  • usually you work first locally until you are happy with your data set.

  • Decide where to host it and upload it. For that scipion will:

  • Generate a MANIFEST file

  • rsync it to your server, you will need to provide a login info (like [email protected], and a remote folder location.

  • type something like: 

    scipion testdata --upload myplugin123_testdata -l [email protected] -rf /path/at/the/server/for/your/datasets

    Please note that the dataset name must be unique, so better prefix it with the plugin name. -l is the login for your server and -rf is the remote folder where to rsync your files.

  • Refer to if in your tests, at you tests folder/init.py: 

    DataSet(name='myplugin123_testdata', 
        folder='myplugin123_testdata',
        files={
               'file1': 'file1.txt'
               ...},
        url='http://wwww.server.com/datasets')
    NOTE: url parameter should be a valid url where your dataset is being published. TIP: I haven’t tried, but doing the upload yourself, to generate the MANIFEST and then adding your datasets + MANIFEST to github might also work if you later point to the gitraw url?? (disclaimer…​I haven’t tried it.)

Create and upload distribution

To upload your distribution to pypi, you’ll need to create an account.

  • Install twine if you don’t have it 

    pip install twine

  • Create the source distribution (at least! You can also create a Built distribution. Read more in the official [packaging guide]()) 

    cd $PLUGIN_HOME
python setup.py sdist
    It is convenient to check your *egg-info/SOURCES.TXT and see if you miss any file (pay special attention to non-python files that you might have forgot to include in MANIFEST.in or in your setup.py, like the logo).

  • Upload the distribution WITH EARLIEST COMPATIBLE SCIPION VERSION IN THE COMMENTS. 

    cd $PLUGIN_HOME && twine upload dist/* -c "scipion-1.2"
    This means that this release we’re uploading will be available for Scipion version 1.2 or higher. The scipion version must follow the pattern used above (scipion-X.Y(.Z)) Now our plugin is on [pypi](https://pypi.org/project/scipion-em-relion/).

Install from pip

  • Uninstall plugin: 

    $SCIPION_HOME/scipion uninstallp -p scipion-em-relion

  • Remove SCIPION_PLUGIN_JSON from ~/.config/scipion/scipion.conf (we exit development mode)

  • Install 

    $SCIPION_HOME/scipion installp -p scipion-em-relion

  • Test again 

    scipion test em.packages.relion.tests.test_***
Clone this wiki locally