sbgrid-cli

SBGrid command line installation manager, sbgrid-cli

sbgrid-cli is a command line installation manager for the SBGrid software collection.

SBGrid can be installed on computers running Linux or macOS operating systems. CentOS/RHEL 7 or 8 is recommended for linux, or MacOSX v10.13 - macOS 10.15 for Macs.. CentOS 6 has reached "End-of-Life" and is no longer supported. macOS 10.15 "Catalina" and macOS 11.0 "Big Sur" can be used, but an /etc/syntetic.conf file is required to create the /programs path. See here for more info :

The SBGrid installation manager requires an account. Set up an account by registering here : SBGrid registration

Requirements and recommendations

sbgrid-cli is single binary executable and should work on most modern Linux distributions.

The SBGrid installation manager will install the software collection to /opt/sbgrid by default (though this can be changed). A symlink from /programs to the installation directory is required and this step does require admin priveleges. Admin privileges are not required to run the applications. This is only for the initial installation, specifically the creation of /opt/sbgrid (if desired) and for /programs (required).

The sbgrid-cli install manager uses 'sudo' for priveleged installation tasks. As such, it should be used from an account with sudo privileges if this functionality is desired. Alternatively, software installation can be competed without sudo and the /programs link can be added manually later. The software will not be functional without this path.

While not strictly required, we recommend a dedicated 'sbgrid' user account for software management, particularly on multiuser systems. This ensures the software can't be modified by user accounts and that data won't be mistakenly saved into the software stack and lost inadvertantly during software updates. The sbgrid-cli program will add and remove software. If data files are accidentally saved into the software collection, they are likely to be lost.

Latest version : v2.2.2

LINUX : sbgrid-cli_2.2.2_linux.tar.gz

macOS : sbgrid-cli_2.2.2_macos.tar.gz

Installation

The sbgrid-cli program is a single executable and does not have any dependencies or requirements. Untar the archive on the command line with :

tar -zxf <archive_file.tar.gz>

where the "archive file" is the name of the tar.gz file downloaded from the links above.

MAC USERS PLEASE NOTE If you encounter a pop-up warning about an "unidentified developer", you need to strip apple's quarantine extended attribute from the sbgrid-cli executable. This is easy to do.

First, cancel the dialog and then run this in your terminal:

xattr -d com.apple.quarantine sbgrid-cli

You should then be able to run the application normally without warnings.


General Usage

The SBGrid command line installation manager takes subcommands and arguments to those commands to perform various tasks. These can be found in the usage info shown above. Each subcommand can also take a -h or --help argument to show more detailed usage info. For example :

$ sbgrid-cli install --help

will produce the help info for the install subcommand.

Activation

To install the software, you must activate your installation with valid credentials provided by SBGrid. This process is only required for the initial installation of the software and is not required to add/remove/upgrade titles.

Activate with the activate subcommand

$ sbgrid-cli activate --help 

SBGrid Installer 2.2.2 f593932a17 — 
                 _______  _____    _    __
                / __/ _ )/ ___/___(_)__/ /
               _\ \/ _  / (_ / __/ / _  /
              /___/____/\___/_/ /_/\_,_/
                  C o n s o r t i u m


USAGE — activate

  ▸ sbgrid-cli activate [site] [user] [key] [OPTIONS...]


ARGUMENTS

  [site]                               The installation site provided in your activation      
                                       credentials                                            
  [user]                               The username provided in your activation credentials   
  [key]                                The activation key provided in your activation         
                                       credentials                                            

OPTIONS

  -k, --credential-key <key>           Activate with a single credential key                  
  --darwin                             Specify macOS as a platform                            
  --no-link                            No symlink creation, no sudo prompt.                   
  --linux                              Specify Linux as a platform                            
  --skip-folder-checking               Skips checking if a folder with programs already exists
  -t, --target <target>                Specify a custom installation target

Some useful options here are :

--no-link

skips creatation of a symlink at /programs. Useful for installs without sudo

--skip-folder-checking

skips the check for software that is already installed. Will overwrite existing installations.

--target

Set the install directory to a different path than /opt/sbgrid

Trouble activating? See Troubleshooting options further down this page.

Restoring from a saved config : reactivate

reactivate subcommand can restore from a saved config file. This config includes all titles and versions from a saved installation. Useful for moving installations or reproducing installations.

 $ sbgrid-cli reactivate <configPath> [OPTIONS...]

Listing available titles, status

The sbgrid-cli list command will show all available titles. They are normally colored by their installation status. Available options :

list

-a, --all-versions                   List all available versions for a title                
-c, --collections                    List the available software collections                
-d, --default-versions               List only the default version for a title              
--long-format                        Use the long format to display the title list          
--one-column                         Format the output into a single column  

The list subcommand has several options

USAGE — list

  ▸ sbgrid-cli list [title...] [OPTIONS...]


ARGUMENTS

  [title...]                           The names of the software titles to list               

OPTIONS

  -a, --all-versions                   List all available versions for a title                
  -c, --collections                    List the available software collections                
  --darwin                             Specify macOS as a platform                            
  -d, --default-versions               List only the default version for a title              
  --linux                              Specify Linux as a platform                            
  --long-format                        Use the long format to display the title list          
  --one-column                         Format the output into a single column  

Note --collections can be used to see subsets of software as listed at https://sbgrid.org/software/.

updates

sbgrid-cli updates Shows available updated titles and versions

$ sbgrid-cli updates
------------------------------------------------------------------------------------
info: Retrieving package metadata...
The following installed packages have updates available (x86_64-linux):
- cryodrgn@0.3.0b => 0.3.1 (new default)
- imod@4.11.0 (patch for installed default version)
- ligplotplus@2.2 (patch for installed default version)
- r@3.6.2 (patch for installed version)

Updates are either new default versions (new default), or updates to an existing version (patch for installed version). The patch versions are usually bug or configuration fixes.

obsolete

Shows obsolete versions that will be removed. Obsolete versions are previously installed versions no longer included in the SBGrid installation.

 $ sbgrid-cli obsolete [OPTIONS...]

Getting Information about titles

info

The info subcommand shows information about each title, including a description and links to documentation can be show with the sbgrid-cli info <title> command. See example below.

 $ sbgrid-cli info <title...> [OPTIONS...]

For example, info for RELION :

 $ sbgrid-cli info relion 
------------------------------------------------------------------------------------
Package information for relion (x86_64-linux):

Package: RELION
Name slug: relion
Architecture: x86_64-linux

Available version(s):
        * 1.3 (installed) depends on: openmpi (1.8.4)
        * 1.4 (installed) depends on: openmpi (1.8.4)
        * 1.4b (installed) depends on: openmpi (1.8.4)
        * 1.4-randomphase3d (installed) depends on: openmpi (1.8.4)
        * 2.1_cu8.0 (installed) depends on: cuda (8.0), openmpi (2.1.2)
        * 3.0.4_cu9.0 (installed) depends on: cuda (9.0), openmpi (2.1.2)
        * 3.0.8 (installed) depends on: openmpi (2.1.3)
        * 3.0.8_cu10.1 (installed) depends on: cuda (10.1), openmpi (2.1.2)
        * 3.0.8_cu8.0 (installed) depends on: cuda (8.0), openmpi (2.1.2)
        * 3.0.8_cu8.0_slurm20 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_slurm-20.02.03)
        * 3.0.8_cu9.0 (installed) depends on: cuda (9.0), openmpi (2.1.2)
        * 3.0.8_cu9.2 (installed) depends on: cuda (9.2), openmpi (2.1.2)
        * 3.1.0 (installed) depends on: openmpi (3.1.6)
        * 3.1.0_cu10.2 (installed) depends on: cuda (10.2.89), openmpi (3.1.6)
        * 3.1.0_cu8.0 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_compat)
        * 3.1.0_cu8.0_compat (installed) depends on: cuda (8.0.44), openmpi (3.1.6_compat)
        * 3.1.0_cu8.0_slurm20 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_slurm-20.02.03)
        * 3.1.0_cu9.2 (installed) depends on: cuda (9.2.88), openmpi (3.1.6)
        * 3.1.1 (installed) depends on: openmpi (3.1.6)
        * 3.1.1_cu9.2 (installed, default) depends on: cuda (9.2.88), openmpi (3.1.6)

Grid(s): sbgrid
Collection(s): Electron Microscopy

Description:
(REgularised LIkelihood OptimisatioN) a stand-alone computer program for Maximum A Posteriori (MAP) refinement of (multiple) 3D reconstructions or 2D
class averages in cryo-electron microscopy.

Technical notes:
## SBGrid Usage Info RELION versions > 2.0 are GPU-accelerated using Nvidia CUDA on Linux. For general information on running GPU accelerated applications from SBGrid, please see here : https://sbgrid.org/wiki/gpuCUDA builds of RELION re designated by a '_cu' suffix in the version of the application. For example, 3.1.0_cu9.2 is linked against CUDA v9.2 libraries. All builds are single precision on the GPU and can be run on 'consumer-grade' GPU hardware.RELION builds without CUDA support are compiled with Intel 2019.5 with accelation options and Intel math libraries. ### RELION and MPI versionsRELION uses the 'mpirun' executable to manage MPI "ranks" for parallel processing. SBGrid includes several OpenMPI versions which are matched to specific RELION versions. When called from the SBGrid capsule environment, the relion GUI is configured to use the correct version of OpenMPI to manage parallel processing. When calling RELION binaries from the command line, the `mpirun.relion` executable can be used. `mpirun.relion` is configured to used the correct version of OpenMPI for each version of RELION. The 3.1.* versions for RELION use the default configuration of OpenMPI (3.1.6 on Linux, 2.1.3 on macOS) and do not require the version to be defined. Older releases use OpenMPI 2.1.2 which should be specified in $HOME/.sbgrid.conf. To set the OpenMPI version, add : OPENMPI_X=2.1.2"to $HOME/.sbgrid.conf and open a new terminal to use the correct mpirun. As noted above, you can also use the `mpirun.relion` executable in place of `mpirun` to use the MPI that matches the currently configured version of RELION. In this case, no other configuration is needed.Version 3.1.0_cu8.0 is compiled to support legacy CPUs (pre-Haswell). OpenMPI 3.1.6_compat is also needed for this version.Versions 3.1.0_cu8.0_slurm20 3.1-beta_cu8.0_slurm20 and 3.0.8_cu8.0_slurm20 support Slurm v20 PMI library. ## Recent Changes2.2.2111 : Add 3.1.1 RELION versions20200703 : Add 3.1.0 release version20200629 : reconfigure RELION gui to automatically use correct 'mpirun' in capsules.20200610 : rebuild of 3.1-beta releases with OpenMPI 3.1.6 from commit 5997001.

## Known Issues20200610 : Empty values in GUI during autopicking can be interpreted as Text instead of Float values. ## Deprecated versionsThe following verions are deprecated and will be removed in a future SBGrid update.20200706: 3.0.8_cu10.1 3.0.8_cu9.2 3.0.8_avx2 3.1-beta_cu9.2 3.1-beta_cu10.1

## (c) Copyright 2008, Structural Biology Grid (www.sbgrid.org)## Last Update to this document# Jason Key 0a6b896 2020-11-15 18:39:38 -0500

Links:
         Website: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
         Manual: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
         Forumhelp: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=CCPEM

The info subcommand includes all version info, dependencies ( which are automatically installed), technical notes and relevant links.

Installing and removing

Install and remove titles with the sbgrid-cli install <title> and sbgrid-cli remove <title> command. Each of these commands can take multiple titles with versions at one time.

install

sbgrid-cli install [title...] [OPTIONS...]

ARGUMENTS

  [title...]                           The name(s) of the software title(s) to install. To    
                                       specify a non-default version, append @ and the version
                                       number to the name of the title (e.g. xyz@1.2.3).      

OPTIONS

  --all-titles                         Installs all the available titles                      
  --all-versions                       This will install all the available versions for a     
                                       specified title or collection                          
  -c, --collection <title>             The title of a software collection to install          
  --darwin                             Specify macOS as a platform                            
  --linux                              Specify Linux as a platform                            
  -j, --num-workers <num-workers>      Run installation tasks in parallel                     
                                       number, default: 1                                     
  --reinstall                          Force the reinstallation of already installed packages 

Installation Examples

Install a single default version of an application, adxv :

$ sbgrid-cli install adxv
 - Install adxv@1.9.14 (x86_64-linux)
 - Installation setup (x86_64-linux).
Continue? … yes
info: (1/2) Installing adxv@1.9.14 (x86_64-linux)...
info: Installation complete for adxv@1.9.14 (x86_64-linux)
info: (2/2) Setting up your installation...
$

Install a specific version of a adxv :

 $ sbgrid-cli install adxv@1.9.11

Install all versions of adxv :

 $ sbgrid-cli install adxv --all-versions

Install the 'Electron Microscopy' collection of titles (see above list command to list collections):

 $ sbgrid-cli install -c 'Electron Microscopy'

Install all default versions of all titles :

 $ sbgrid-cli install --all-titles

Install all versions of all titles ( The full collection):

 $ sbgrid-cli install --all-titles --all-versions

Note the --reinstall flag can be added to all commands to reinstall titles that have already been installed.

remove

Same options as install subcommand above.

$ sbgrid-cli remove [title...] [OPTIONS...]

Updating titles

Updates are performed when default titles change to a new version or bugs are fixed in existing verisons. To update, use the update subcommand.

update

$ sbgrid-cli update relion

Remove obsolete titles

Use the clean subcommand to list or remove obsolete versions or titles

clean

$ sbgrid-cli clean  [OPTIONS...]

Additional subcommands

sbgrid-cli provides additional subcommands for automated software colletion maintenance and troubleshooting.

admin

Run periodic installation, update and removal. Installs all new titles, updates where appropriate and removes obsolete versions

$ sbgrid-cli admin

For example running

$ sbgrid-cli admin 

will install all default, update all installed titles and remove all obsolete versions while

$ sbgrid-cli admin --all-versions

will install all versions of all titles, add any new titles, update any pending updates, and remove obsolete versions. This

rebuild

rebuild rebuilds the installation environment configuration files. Usually this step is automatic but can be run manually for troubleshooting.

$ sbgrid-cli rebuild

crontab

Writes a basic script for crontab for admin functionality

save

The save subcommand saves the current configuration so it can be restored at a later stage or replicated on another machine or in the cloud.

Troubleshooting and diagnostics

check-connection

Provides network diagnostics for connectivity to SBGrid at Harvard Medical School.

  $ sbgrid-cli check-connection 
info: (1/1) Checking connection...
info: Checking services...
info: Checking sync2.sbgrid.org (port: 8080) ...
info: Checking sync2.sbgrid.org (port: 873) ...
info: Checking sync2.sbgrid.org (port: 27010) ...
info: Checking sbgrid.org (HTTPS) ...
info: Checking sbgrid.org (HTTP) ...
info: Connectivity test succeeded, all remote services are reachable

diag Run troubleshooting and diagnostic tests for the software collection.

verify
Lists missing software versions and .rc files.

Installing software for multiple platforms

On linux, add the --darwin command to install / remove for macOS . Conversely, use the --linux command to install / remove for Linux while on macOS. These flags can be added to many of the subcommands to limit to Linux or macOS. Useful for servers that share a software stack over NFS.

 $ sbgrid-cli list --darwin
This will list all software available for macOS. The platform flag ( --linux or --darwin) is implied for the platform used to install the software, i.e. on linux, --linux is redundant.

Logging

sbgrid-cli writes log files to $HOME/.sbgrid/logs

Software Version Management in the SBGrid environment

If you have a single version of an application installed, explicit version selection in your environment is not needed.

In the case of multiple installed versions, the SBGrid Default version will be the version available at the terminal when it is installed, unless configured otherwise in ~/.sbgrid.conf.

In the case of multiple non-default versions installed, the latest release will be version available in your shell. If you would like to have more than one version of a title installed, and prefer to use an older release or the non-default version when a default version is installed, you must set this explicitly in your ~/.sbgrid.conf file. This is the same as in the standard SBGrid environment.

Version priority in the shell

  1. ~/.sbgrid.conf
  2. Installed single version
  3. Installed Default version
  4. Latest installed release

There is more info on how to manage versions here --> SBGrid version overrides

Running SBGrid software

To use the software at the terminal in bash, open a new terminal and run

$ source /programs/sbgrid.shrc

or in tcsh

$ source /programs/sbgrid.cshrc

For more info on system configuration for SBGrid software, please check here ---> Workstation setup.

Known Issues

rsync required on 873 or 8080. The application will fail if outbound port 873 and port 8080 are blocked by your institution or are otherwise not available. You can check if you can access port 873 here and port 8080 here

Deprecated releases

for the sbgrid-cli version 1, see here

For help or to report issues, please email bugs@sbgrid.org.