cli_v2

SBGrid command line installation manager - v2.1.1

How do I get it?

LINUX : sbgrid-cli_2.1.1_linux.tar.gz

macOS : sbgrid-cli_2.1.1_macos.tar.gz

Supported OSes

Linux (CentOS/RHEL 7 or 8 recommended) or Apple OSX computers running MacOSX v10.12 - 10.15.

MacOS 10.15 Catalina can be used, but a work-around is required to add the /programs path. See here for more info :

Pre-installation requirements

Admin access required on your computer. The SBGrid installation manager will install the software tree to /opt/sbgrid and create a symlink from this directory to /programs. While admin privileges are not required to run the application, you will likely need admin privileges to create /opt/sbgrid and /programs .

  $ ./sbgrid-cli 
Usage: sbgrid-cli [command] [options]

Commands:
  help [command]                                         command-specific help

  activate [options] <site user key> | <credential-key>  activate a new installation
  reactivate [options] <configFilePath> [packagesPath]   reactivate installation using config or package list

  install [options] [title] [otherTitles...]             install package(s)
  remove|uninstall [options] [title] [otherTitles...]    remove package(s)
  reinstall [options] [title] [otherTitles...]           reinstall package(s)
  update|upgrade [options] [title] [otherTitles...]      update package(s) to the latest release/version

  list [options] [title]                                 list available software titles
  installed                                              list installed software titles
  updates                                                list available updates
  pending                                                list pending installations
  obsolete                                               list obsolete software titles
  search <query>                                         search availble titles
  collections                                            list available package collections
  grids                                                  list available software trees
  info [options] [title]                                 print title information
  shell                                                  start SBGrid Shell

  clean [options]                                        remove obsolete software versions and titles
  admin [options]                                        run periodic installation, update and clean up
  rebuild                                                rebuild installation environment
  crontab [options]                                      basic script for crontab
  save [filename]                                        save current configuration

  check-connection                                       connectivity checks
  diag [options]                                         troubleshooting and diagnostics
  verify                                                 list missing software versions and .rc files

  Options:
  -V, --version                                          version
  --verbose                                              verbose output
  --quiet                                                disable output
  -h, --help                                             usage information
  --linux                                                specify linux as platform
  --darwin                                               specify macOS as platform
  --no-color                                             disable output colors

General Usage

The SBGrid command line installation manager takes subcommands and arguments to those commands to perform various tasks. These can be found in the usage info shown above. Each subcommand can also take a -h or --help argument to show usage info.

Activation

To install the software, you must have valid credentials provided by SBGrid.

Activate with the 'activate' subcommand

activate

Usage: activate [options] [site] [user] [key]

activate new installation using credentials provided
(credentials are sent via email and can either be a single credential-key, or a site username key triplet)

Options:
  -k, --credential-key <credential-key>  activate with activation key
  -h, --help                             output usage information

Trouble activating? See Troubleshooting options further down this page.

reactivate

reactivate can restore from a saved config file. See below.

  $ ./sbgrid-cli reactivate -h
Usage: reactivate [options] <configFilePath> [packagesPath]

reactivate a new instance of an installation using a “saved” configuration file
to repair an existing installation see “reinstall --all-versions”

Options:
  --skip-folder-checking  skip checking if folder with programs already exists
  --just-activation       skip software titles installation
  -h, --help              output usage information

Listing available titles, status

The sbgrid list command will show all available titles. They are normally colored by their installation status. Available options :

list

Usage: list [options] [title] [otherTitles...]

list available software titles

Options:
  -a, --all-versions      list all available software versions
  -d, --default-versions  list default software versions
  -c, --collections       list available collections
  -o, --one-column        list software in one column
  -l, --long-format       list software versions in long format
  -h, --help              output usage information

updates

Shows available updated version and titles

./sbgrid-cli updates

obsolete

Shows obsolete versions that will be removed. Obsolete versions are previously installed versions no longer included in the SBGrid installation.

./sbgrid-cli obsolete

Getting Information about titles

info

The info flag shows information about each title, including a description and links to documentation can be show with the sbgrid info command. See example below.

Usage: info [options] [title] [otherTitles...] 

print software title(s) metadata

Options:
  -v, --versions     list available and default versions for <title> on current platform
  -c, --collections  list available collections with software count
  -h, --help         output usage information

For example, info for RELION :

  $ ./sbgrid-cli info relion 
Fetching requested info...
------------------------------------------------------------------------------------
Package information for relion (i386-mac):

Package: RELION
Title: relion
Architecture: i386-mac

Available version(s):
	* 1.3 (installed) depends on: openmpi (1.8.4)
	* 1.4 (installed) depends on: openmpi (1.8.4)
	* 2.1 (installed) depends on: openmpi (2.1.2)
	* 3.0.6 (installed) depends on: openmpi (2.1.3)
	* 3.0.7 (installed) depends on: openmpi (2.1.3)
	* 3.0.8 (installed, default)
	* 3.1-beta (installed)

Grid(s): SBGrid
Collection(s): Electron Microscopy

Description:
(REgularised LIkelihood OptimisatioN) a stand-alone computer program for Maximum A Posteriori (MAP) refinement of (multiple) 3D reconstructions or 2D class averages in cryo-electron microscopy.

Technical notes:
**GPU-accelerated RELION :  ** RELION versions > 2.0 are GPU-accelerated using Nvidia CUDA on Linux. 
For general information on running GPU accelerated applications from SBGrid, please see here : https://sbgrid.org/wiki/gpu</a>

CUDA builds of RELION are designated by a '_cu<cuda_version>' suffix in the version of the application. 
For example, 2.0.4_cu7.5 is linked against CUDA v7.5 libraries and is not recommended for Pascal architecture Nvidia hardware (GTX 1080 and higher). 2.0.4_cu8.0 is built with CUDA 8.0 which supports Pascal architecture GPUs. The '_SP' designation indicates 'single precision' CPU builds. All builds are single precision on the GPU and can be run on 'consumer-grade' GPU hardware. 

**RELION and MPI :  **
RELION uses the 'mpirun' executable to manage MPI ranks. SBGrid includes two OpenMPI versions, 2.1.2 (default) and 1.8.4 (legacy).
The OpenMPI mpirun executable must match the version of OpenMPI used to build RELION. 
For example, RELION versions 2.1_cu*.0 use OpenMPI version 2.1.2 and must use mpirun from 2.1.2. 

Older RELION versions require an override in $HOME/.sbgrid.conf to set the older OpenMPI version to 1.8.4. Add 

OPENMPI_X=1.8.4"

to $HOME/.sbgrid.conf and open a new terminal to use the correct mpirun. 
You can also use the 'mpirun.relion' executable in place of mpirun to use the MPI that matches the currently configured version of RELION.

Links:
	 Website: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
	 Manual: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
	 Forumhelp: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=CCPEM

Installing and removing

Install and remove titles with the sbgrid install and sbgrid remove command. Each of these commands can take multiple titles with versions at one time.

install

install software title(s)

Options:
  -d, --all-default              install all default versions of all available software titles
  -a, --all-versions             install all versions of all available software titles
  -c, --collection <collection>  install all software titles from collection
  title@all                      install all versions of a software title
  title@x.y.z                    install "x.y.z" version of a software title
  -p, --progress-bar             show progress bar during installation
  -y, --yes                      continue without confirmation
  -h, --help                     output usage information

Install examples :

Install a single default version of an application :

./sbgrid-cli install relion 

Install a specific version of a relion :

 ./sbgrid-cli install relion@3.0.8

Install all versions of relion :

 ./sbgrid-cli install relion@all

Install the 'Electron Microscopy' collection of titles (see above list command to list collections):

$ ./sbgrid-cli install -c 'Electron Microscopy'

Install all default versions of all titles :

 ./sbgrid-cli install --all-defaults

Install all versions of all titles ( The full collection):

 ./sbgrid-cli install --all-versions

remove

Same options as install flag above.

remove software title(s)

Options:
  -d, --all-default              remove all default versions of all installed software titles
  -n, --non-default              remove all non-default versions of all installed software titles
  -a, --all-versions             remove all versions of all installed software titles
  -c, --collection <collection>  remove all software titles from collection
  title@all                      remove all installed versions of a software title
  title@x.y.z                    remove "x.y.z" version of a software title
  -y, --yes                      continue without confirmation
  -h, --help                     output usage information

Updating titles

Updates are performed when default titles change to a new version or bugs are fixed in existing verisons. To update, use the update flag.

update

./sbgrid-cli update relion

Remove obsolete titles

Use the obsolete flag to list or remove obsolete versions or titles

obsolete

  $ ./sbgrid-cli obsolete -h
Usage: obsolete [options]

list obsolete software versions

Options:
  -r, --remove  remove the obsolete software versions
  -y, --yes     continue without confirmation
  -h, --help    output usage information

Advanced modes

The advanced modes perform multiple actions and are designed for automated use for a managed 'rolling release' installation.

clean

Removes all obsolete software versions and titles

admin

Run periodic installation, update and removal. Installs all new titles, updates where appropriate and cleans up (removes) obsolete versions

Usage: admin [options]

periodic update and installation

Options:
  -a, --all-versions  install all versions of all available software titles
  -r, --reinstall     reinstall all of installed software titles
  -h, --help          output usage information

For example running

./sbgrid-cli admin 

will install all default, update all installed titles and remove all obsolete versions while

./sbgrid-cli admin -a

will install all versions of all titles, add any new titles, update any pending updates, and remove obsolete versions. This

rebuild

rebuild rebuilds the installation environment configuration files.

crontab

Writes a basic script for crontab for admin functionality

save

The save flag saves the current configuration so it can be restored at a later stage or replicated on another machine (or in the cloud).

Troubleshooting and diagnostics

check-connection
connectivity checks

diag troubleshooting and diagnostics

verify
list missing software versions and .rc files

Installing software for multiple platforms

On linux, add the --darwin command to install / remove for macOS . Conversely, use the --linux command to install / remove for Linux while on macOS.

--linux                                                specify linux as platform
--darwin                                               specify macOS as platform
The platform flag is implied for the platform used to install the software.

Software Version Management in the SBGrid environment

If you have a single version of an application installed, explicit version selection in your environment is not needed.

In the case of multiple installed versions, the SBGrid Default version will be the version available at the terminal when it is installed, unless configured otherwise in ~/.sbgrid.conf. This is standard for the SBGrid software environment.

In the case of multiple non-default versions installed, the latest release will be version available in your shell. If you would like to have more than one version of a title installed, and prefer to use an older release or the non-default version when a default version is installed, you must set this explicitly in your ~/.sbgrid.conf file. This is the same as in the standard SBGrid environment.

Version priority in the shell

  1. ~/.sbgrid.conf
  2. Installed single version
  3. Installed Default version
  4. Latest installed release

There is more info on how to do that here --> SBGrid version overrides

Running SBGrid software

To use the software at the terminal in bash, open a new terminal and run

$ source /programs/sbgrid.shrc

or in tcsh

$ source /programs/sbgrid.cshrc

Known Issues

  • rsync required on 873 or 8080. The application will fail if outbound port 873 and port 8080 are blocked by your institution or are otherwise not available. You can check if you can access port 873 here and port 8080 here

For help or to report issues, please email bugs@sbgrid.org.