Faster import part2: Delay import of heavy packages

[danielhollas]

Delay import of disk_objectstore, jsonschema, requests, plumpy and paramiko. Obviously, many of these will be eventually imported for many tasks, but right now they are imported indiscriminately for everything, making everything slower, see benchmarks below.

There are other avenues for improvement, but I wanted to get the most impactful stuff first.

Timings against the main branch

import aiida

Benchmark 1: PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida'
  Time (mean ± σ):     102.4 ms ±  10.5 ms    [User: 89.6 ms, System: 12.4 ms]
  Range (min … max):    85.6 ms … 127.2 ms    23 runs
 
Benchmark 2: python -c 'import aiida'
  Time (mean ± σ):     263.4 ms ±  16.1 ms    [User: 236.3 ms, System: 26.6 ms]
  Range (min … max):   240.9 ms … 298.2 ms    11 runs
 
Summary
  'PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida'' ran
    2.57 ± 0.31 times faster than 'python -c 'import aiida''

import aiida.orm

Benchmark 1: PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida.orm'
  Time (mean ± σ):     333.8 ms ±  40.1 ms    [User: 960.0 ms, System: 2257.9 ms]
  Range (min … max):   302.6 ms … 410.4 ms    10 runs
 
Benchmark 2: python -c 'import aiida.orm'
  Time (mean ± σ):     586.0 ms ±  24.1 ms    [User: 1246.8 ms, System: 2339.1 ms]
  Range (min … max):   560.4 ms … 620.8 ms    10 runs
 
Summary
  'PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida.orm'' ran
    1.76 ± 0.22 times faster than 'python -c 'import aiida.orm''

import aiida.cmdline.commands

Benchmark 1: PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida.cmdline.commands'
  Time (mean ± σ):     280.0 ms ±  14.6 ms    [User: 256.8 ms, System: 22.6 ms]
  Range (min … max):   248.1 ms … 300.0 ms    12 runs
 
Benchmark 2: python -c 'import aiida.cmdline.commands'
  Time (mean ± σ):     346.3 ms ±  32.9 ms    [User: 315.1 ms, System: 30.6 ms]
  Range (min … max):   318.2 ms … 421.6 ms    10 runs
 
Summary
  'PYTHONPATH=/home/bl22441/atmospec/aiida-core python -c 'import aiida.cmdline.commands'' ran
    1.24 ± 0.13 times faster than 'python -c 'import aiida.cmdline.commands''

[danielhollas]

@sphuber this is ready for review

[danielhollas]

Note: About half of the cost of importing plumpy (30ms) comes from importing asyncio

[danielhollas]

NOTE: This is a breaking API change in principle, but since this only seems to be used in aiida/manage/manager.py I think it is okay. This is AFAIK the only way how to avoid importing plumpy upon every import, since the ProcessLauncher uses plumpy.ProcessLauncher as a parent class.

[sphuber]

Thanks @danielhollas . The changes look good to me. You are right that the removal of the export of ProcessLauncher to higher modules is technically breaking, but I agree that it is unlikely that it affects anyone, so I think this is an acceptable change. Are these the last of the changes related to the import speed? Or do you have more lined up?

[danielhollas]

Thanks @sphuber ! I have more lined up, specifically targeting verdi.

[sphuber]

Thanks @danielhollas