-
Notifications
You must be signed in to change notification settings - Fork 4
Ska3 runtime environment for users
The Ska3 runtime environment consists of an integrated suite of Python-3 packages and associated applications. The installation and update of these packages is done using the Anaconda Python distribution and the Conda package manager. This allows simple installation on linux machines or Mac laptops. Similar support for Windows is under consideration.
- Maintaining Ska3 runtime environment
- Installing Ska3 runtime environment
- Configuration hints
- Testing Ska3
- Ska Data
- Installing and maintaining Ska3 development (masters) environment
For discussion about motivation for Python 3, new features, and converting from Python 2.7 to Python 3, see:
If you have a Ska3 runtime environment already installed, maintaining your environment is straightforward. When there is a new release (or just periodically) do:
mamba update ska3-flight --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight
If you are using Ska3 on the HEAD or GRETA linux networks or via MATLAB tools, then there is no need to install Ska3.
However, you may wish to look through the [Configuration hints][#configuration-hints] section.
Power users note: on the HEAD linux network there can be a substantial improvement in package import
times if you install and maintain your own Ska3 installation on a local disk. The production
installation on /proj/sot is served via a remote NetApp which can be quite slow.
The basis for installing, managing, and using Ska3 is the Conda package
manager and this needs to be installed before anything
else can be done. If you already have the conda package manager installed
with version 4.8 or higher (conda --version), then you can skip these steps and go to the
Create Ska3 environment section.
Important note about older conda
If you have a version of conda older than 4.8, this is not compatible with ska3. We recommend deleting that old conda and the associated environment.
The steps for creating a new conda environment are as follows:
- Get one of the following files by URL:
https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh
https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-MacOSX-x86_64.sh
https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-Windows-x86_64.exe
For Windows follow the instructions for Installing on Windows or consult your local Windows expert.
For linux and MacOSX proceed with the following steps.
-
Open a terminal window and change to the downloads directory for your browser.
-
Ensure that you have
gitinstalled in your system by typinggit --versionon the command line. If not, install it via https://git-scm.com/downloads. -
The installer is called something like
Miniconda-py310_23.1.0-1-<OS>.sh. Find it and then enter the following (without the%, which is used as the linux prompt in all these examples):
bash Miniconda3-py310_23.1.0-1-MacOSX-x86_64.sh
-
Hit ENTER, then agree to the license terms, and finally specify the installation directory. This will be created by the installer and needs to be a directory that doesn t currently exist. A fine choice is the default of
~/miniconda3(which is what is assumed for the rest of the installation), but you may want to install to a different directory depending on the details of your system. -
At the end it asks about updating your
.bashrcor.bash-profileor.cshrcfile. We recommend accepting (say yes). -
Now create a new terminal window.
This gives you a minimal working Python along with the package manager utility conda. Check hat this is working by doing:
which python
--> shows the expected path, e.g. $HOME/miniconda3/bin/python
Now you can install all the Ska3 packages and required dependencies to an environment
named ska3 by copy-pasting this long command:
conda create -n ska3 ska3-flight --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight
On linux and MacOSX you can optionally install ska3-perl as well. This is mostly for aspect-specific apps like starcheck.
After you see the list of packages, accept, and see that the install finished. Then you can get into the Ska3 environment
conda activate ska3
It's done and now you have a ska3 flight environment! The name ska3 is not magic, and you
could have called it my-ska or whatever you want.
If you make a new shell window then you need to conda activate ska3 to get into Ska3
again.
NOTE: sometimes, the conda create... command above fails with a long message about conflicts. If that is the case, try the following instead:
conda create -n ska3 ska3-flight
conda activate ska3-flight
conda install --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight mamba
mamba install --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight ska3-flight
It is useful to define aliases to get into the Ska3 production environment on HEAD or GRETA and adjust your prompt to indicate that you are using Ska3. The command and file to modify depends on the shell you are using and the network. First if you don't know what shell you are using then do:
echo $0
This should say either csh or tcsh or bash.
============= ============= =============
Shell Network File
============= ============= =============
csh or tcsh HEAD ~/.cshrc.user
csh or tcsh GRETA ~/.cshrc
bash HEAD or GRETA ~/.bashrc
============= ============= =============
Now put the appropriate line at the end of the indicated file:
# Csh or tcsh
alias ska3 'source /proj/sot/ska3/flight/bin/ska_envs.csh; set prompt="ska3-$prompt:q"'
# Bash
alias ska3='source /proj/sot/ska3/flight/bin/ska_envs.sh; export PS1="ska3-$PS1"'
The ska3 alias is a one-way ticket. Once you
get into the Ska3 environment the recommended way to get out or change to a
different version is to start a new window.
The interactive Python tool ipython has a nice feature that you can
define custom "magic" commands to perform actions with a single command. One
useful thing to do is to import a number of common Ska3 packages with a single
command %impska at the ipython command line. To do this, follow the
instructions below.
First locate your default IPython profile directory with:
ipython profile locate
In that directory, if there is not already a sub-directory startup then create it now.
Finally, in the startup sub-directory within your default IPython profile directory make
a new file called 10_import_ska3.py and copy/paste the following into that file. You
can easily customize yourself following the obvious pattern.
from IPython.core.magic import Magics, magics_class, line_magic
from IPython import get_ipython
ip = get_ipython()
@magics_class
class MyMagics(Magics):
@line_magic
def impska(self, line):
ip.ex('import cheta.fetch_eng as fetch')
ip.ex('from cheta import fetch_sci')
ip.ex('from Chandra.Time import DateTime')
ip.ex('from cxotime import CxoTime')
ip.ex('from Ska.Matplotlib import plot_cxctime, cxctime2plotdate')
ip.ex('from kadi import events')
ip.ex('from astropy.table import Table')
ip.ex('import numpy as np')
ip.ex('import matplotlib.pyplot as plt')
ip.ex('from Ska.tdb import msids as TDB')
ip.ex('import kadi')
ip.ex('import cxotime')
ip.ex('print(f"cheta version={fetch.__version__}")')
ip.ex('print(f"kadi version={kadi.__version__}")')
ip.ex('print(f"cxotime version={cxotime.__version__}")')
ip.register_magics(MyMagics)
Now start up ipython and enter %impska. You should see something like this:
(ska3) ➜ ~ ipython
Python 3.8.3 (default, Jul 2 2020, 11:26:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: %impska
cheta version=4.50.1
kadi version=5.4.0
cxotime version=3.2.3
Congratulations, you should now have all the Ska3 packages installed!
As a quick smoke test that things are working, confirm that the following works and makes a line plot:
ipython --matplotlib
>>> import matplotlib.pyplot as plt
>>> from Chandra.Time import DateTime
>>> import ska_path
>>>
>>> plt.plot([1, 2])
>>> DateTime(100).date
'1998:001:00:00:36.816'
A comprehensive check that everything is working can be done by running the full Ska3 integration test suite. This will take up to a half hour. Before starting this you must ensure that the standard Ska data are installed.
# Change to a directory for git repos, or a temporary directory
cd ~/git # Or whatever
git clone https://github.com/sot/ska_testr.git
cd ska_testr
run_testr --test-spec test_spec_standalone
This will provide a summary of test results at the end, with detailed logs available in the outputs/logs directory.
To make full use of the Ska environment you need various package data, e.g.:
- Ska engineering archive
- Kadi databases
- Commanded states databases
- AGASC star catalog
- Telemetry database (MSIDs)
The very first thing to do is define the disk location where you want to store the Ska data. Packages and applications in the Ska runtime environment use an environment variable SKA to define the root directory of data and other shared resources. On the HEAD or GRETA networks this root is the familiar /proj/sot/ska3/flight.
For a machine (e.g. your laptop) not on the HEAD or GRETA networks you need to make a directory that will hold the Ska data. A reasonable choice is putting this in a ska directory in your home directory:
mkdir ~/ska
setenv SKA ${HOME}/ska # csh / tcsh
export SKA=${HOME}/ska # bash
If you have made a local Ska environment on a machine on the HEAD or GRETA networks, you can just set the SKA environment variable to point at the existing root:
setenv SKA /proj/sot/ska # csh / tcsh
export SKA=/proj/sot/ska # bash
For a machine not on the HEAD or GRETA networks you need to get at least a subset of the data on your machine. FOT users may want to start up MATLAB which will run a task to update the Ska data used by those tools. Other users or FOT users with different data needs will use a script called ska_sync that is installed as part of Ska. This assume you have set the SKA environment variable and created a directory as shown above.
The first thing is to try running it and getting the online help:
% ska_sync --help
usage: ska_sync [-h] [--user USER] [--install] [--force]
Synchronize data files for Ska runtime environment
Arguments
=========
optional arguments:
-h, --help show this help message and exit
--user USER User name for remote host
--install Install sync config file in default location
--force Force overwrite of sync config file
Next you need to install a copy of the template configuration file into your Ska root directory. This will let you customize which data you want and how to get it. This installation is done with:
% echo $SKA # Check the SKA root directory
/Users/aldcroft/ska
% ska_sync --install
Wrote ska sync config file to /Users/aldcroft/ska/ska_sync_config
Now you need to edit the file it just created and set the remote host for getting data and the remote host user name. Choose either kadi.cfa.harvard.edu (HEAD) or the IP address for cheru (OCC) and edit accordingly. Likewise put in the corresponding user name:
# Host machine to supply Ska data (could also be chimchim but kadi works
# from OCC VPN just as well).
host: kadi.cfa.harvard.edu
# Remote host user name. Default is local user name.
# user: name
If you want Ska engineering archive access then following the instructions in Syncing Ska engineering archive data
Finally do the sync step:
% ska_sync
Loaded config from /Users/aldcroft/ska/ska_sync_config
This prints something like below:
COPY and PASTE the following at your terminal command line prompt:
rsync -arzv --progress --size-only --files-from="/Users/aldcroft/ska/ska_sync_files" \
[email protected]:/proj/sot/ska/ "/Users/aldcroft/ska/"
As instructed copy and paste the rsync line, after which you will need to enter your password. At this point it will sync the relevant Ska data files into your local Ska root.
NOTE for Windows users (but not on FOT laptops): one possibility for running the rsync
command is to use Windows Subsystem for Linux. This has been successfully demonstrated
for a standalone Windows machine.
As long as you don t change your config file, you can just re-run that same command to re-sync as needed.
To test that the data are really there make sure you can reproduce the following:
% ipython --matplotlib
>>> from Ska.tdb import msids
>>> msids.find('tephin')
[<MsidView msid="TEPHIN" technical_name="EPHIN SENSOR HOUSING TEMP">]
>>> from kadi import events
>>> events.normal_suns.filter('2014:001')
<NormalSun: start=2014:207:07:04:09.331 dur=65207>
>>> from Chandra.cmd_states import fetch_states
>>> fetch_states('2011:100', '2011:101', vals=['obsid'])
...
SOME WARNINGS WHICH ARE OK and will get patched up later
...
[ ('2011:100:11:53:12.378', '2011:101:00:26:01.434', 418823658.562, 418868827.618, 13255)
('2011:101:00:26:01.434', '2011:102:13:39:07.421', 418868827.618, 419002813.605, 12878)]
Syncing Ska engineering data to your standalone installation (laptop) is feasible. Circa mid-2019, the archive about 170 Gb. Copying the entire archive over VPN network will take a while, but it has been done.
Keeping the archive up to date should be done using cheta_sync, which is documented
in the Cheta telemetry archive tutorial.
rsync -av --prune-empty-dirs --exclude='/*/*/*/' --include='*/' \
--include='???????/ofls?/*.backstop' --include='???????/ofls?/*.or' --exclude='*' \
kady:/data/mpcrit1/mplogs/2019/ $SKA/data/mpcrit1/mplogs/2019/
Users that want to work in an environment that contains the latest development version
(the master branch) of all Ska3 packages can do so as follows. This takes advantage of a distribution channel
named masters that has the latest conda build for the master branch of all Ska3 packages.
The command below will select packages from masters where the version is greater than
the release version.
conda create -n ska3-dev --override-channels \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
ska3-core-latest ska3-flight-latest ska3-perl-latest
conda create -n ska3-dev --override-channels `
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters `
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight `
ska3-core-latest ska3-flight-latest
For a Ska3-dev environment you can easily update a single Ska3 package. For instance if
you wanted to revert to version 3.1.0 of dea_check (since master of dea_check is ahead of that),
you would do the following:
conda install -c $ska3conda/flight --override-channels dea_check=3.1.0
The commands below to update all packages to the latest would undo this.
Getting a bit braver, if you wanted to install numpy 1.19 then this would work:
conda install --override-channels \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight -c defaults \
numpy=1.19
If you have your own development version of a package and you want to install it
into your dev environment, this is easily done with pip:
pip install .
This will override the existing conda package (if it exists). Doing a conda list
will now show the package as being pip-installed and will show the appropriate
dev version.
You can update to the latest version in the master branch for all Ska3 packages with the following.
mamba update ska3-flight-latest \
--all \
--strict-channel-priority \
--override-channels \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight
mamba update ska3-flight-latest `
--all `
--strict-channel-priority `
--override-channels `
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters `
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test `
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight