sbgrid-cli-admin

SBGrid site installation

One of the primary benefits of an SBGrid software installation is that it is a managed software stack. New titles are added, obsolete versions are removed, and updates happen automatically. This makes software updates and bug fixes available to users by an email request (bugs@sbgrid.org), all without requiring local administrators or system admistration expertise. While there are other installation modes available (GUI, default versions, select titles, etc), the site installation is the recommended mode of installation for most sites.

Installation Steps:

1. Create an 'sbgrid' user on the machine that will host your software installation. This step is not strictly required, but is highly recommended.

adduser sbgrid

By default the SBGrid installation manager, sbgrid-cli, will install at /opt/sbgrid. This path typically requires sudo/root to create Linux and macOS. The sudo requirement can be skipped by using the --target flag to specify an alternate installation location. This is usually preferred as /opt may be on a filesystem with insufficient capacity. The sbgrid user must have write access to this target directory.

2. Log in as the 'sbgrid' user

su sbgrid

3. Download the sbgrid-cli application

sbgrid-cli for linux

sbgrid-cli for macOS

To download directly to a server or workstation from the command line : For Linux:

curl -LO https://sbgrid.org/downloads/latest/sbgrid-cli_linux.tar.gz

For Mac:

curl -LO https://sbgrid.org/downloads/latest/sbgrid-cli_macos.tar.gz

4. Untar the archive

tar -zxvf sbgrid-cli_linux.tar.gz

5. Activate the installation

Activating the installation will create the required installation directories and core libraries for installing the software. Alternate installation paths can be used with the '--target flag'. The entire software collection will not be installed here - just an initial the core without the titles. Note that "activation" here is different from other tools, such as conda. For SBGrid, this is only necessary at initial installation.

./sbgrid-cli activate-site --target <path to software installation location>

where the target path is writable. With --target, activation does not require sudo. The application will prompt for a site name and key which was provided by email. After this is entered and confirmed the software will be installed. A cron task is created for update checks.

The manual creation of the "/programs" symlink to the target installation directory will be required, but only for machines running the software. /programs is not needed for updates or servers that host the programs as a network share. The '--skip-folder-checking' flag can be used to install without creation of /programs or directly into an existing /programs path.

Post-Install Steps:

Once the download has completed, you'll need to add a "/programs" symlink in the root directory of each machine that wants to use the software. Typically, this directory is shared via NFS to each of your computers.

On Linux:

ln -s /path/to/sbgrid/install/target /programs

On macOS, root level symlinks must be defined in the /etc/synthetic.conf file. This is a simple configuration file, More info on synthetic.conf can be found here.
Any user that wishes to use SBGrid needs to configure their shell to initialize the SBGrid shell environment:

For bash: add source /programs/sbgrid.shrc to ~/.bashrc
For tcsh: add source /programs/sbgrid.cshrc to ~/.cshrc
Configure your workstations for compatibility: https://sbgrid.org/wiki/software/workstation_setup NOTE: XQuartz is required for many applications on macOS

SBGrid installation manager admin mode

sbgrid-cli admin mode replaces the sbgrid-admin script installation method used previously to install and automatically update SBGrid site installations. sbgrid-cli admin offers a some advantages:

parallel updates for fast, more efficient transfers
default version only for limited disk space
rolling updates for immediate releases

The sbgrid-cli admin command will:

Install all new titles added to SBGrid
Update existing versions when newer versions are added to SBGrid
Remove obsolete verisons that are no longer supported

sbgrid-cli admin Examples :

sbgrid-cli admin commands are typically run periodically from cron.

To install all default versions of all titles, add new titles when available, update all installed when available and remove obsolete versions, run :
```
sbgrid-cli admin
```
To do the same but to include all versions of all titles :
```
sbgrid-cli admin --all-versions
```
To make this automatic, add a crontab entry. This can be done from sbgrid-cli with this command:
```
sbgrid-cli crontab
```
Full auto mode - all versions, all titles mac and linux, 4 processes. This is the recommened installation configuration.
```
$ sbgrid-cli crontab --all-versions --darwin --linux -j 4
```

More crontab examples

Only update the titles that have been explicitly installed, no new additions

*/15 * * * * /programs/x86_64-linux/sbgrid_installer/latest/sbgrid-cli update --all-titles --linux --darwin --yes --quiet

Run a daily cleanup to preserve space

* 1 * * * /programs/x86_64-linux/sbgrid_installer/latest/sbgrid-cli clean --yes --quiet

For more info on cron configuration, see https://www.man7.org/linux/man-pages/man5/crontab.5.html

Migrating existing script-based "sbgrid-admin" site installations to sbgrid-cli

To migrate an existing site-install that uses our sbgrid-admin script, use the sbgrid-cli migrate-admin command as your sbgrid user. This will convert the legacy sbgrid-admin installation.

sbgrid@computer $  sbgrid-cli migrate-admin

Will produce

warn: The RSYNC_ARGS in your sbgridrc will not be copied over to the new configuration file.
warn: (in most cases sbgrid-cli doesn't require any custom arguments for rsync, so this should be fine)
info: Retrieving package metadata...
info: (1/9) Checking connection...
info: Checking services...
info: Checking sbgrid.org (HTTPS) ...
info: Checking sbgrid.org (HTTP) ...
info: (2/9) Checking rsync credentials...
info: (3/9) Calling activate API endpoint...
info: Site activation credentials provided, skipping API activation
info: (4/9) Initializing installation directories...
info: Please note that a link from the default installation directory /opt/sbgrid to the install target is required to run the software.
A link from /programs to /opt/sbgrid is also required.
info: (5/9) Synchronizing...
info: (6/9) Synchronizing...
info: (7/9) Installing sbgrid-installer@2.5.7 (x86_64-linux)...
info: Installation complete for sbgrid-installer@2.5.7 (x86_64-linux)
info: (8/9) Saving installation sitename file...
info: (9/9) Setting up your installation...
info: Setting all packages to installed & synchronizing your installation...
info: (1/2) Installing the admin cron job...
info: Cron job successfully installed to the current user's crontab
info: (2/2) Setting up your installation...
info: Migration to sbgrid-cli complete. The legacy configuration file has been renamed to /home/sbgrid/.sbgridrc_migrated

When migrating from an sbgrid-admin "site" installation to a sbgrid-cli installation, the symlinks are likely to be already in place and the info about creating them can usually be ignored

The default crontab is added by this process. You can set that with the crontab -l command. This will check for updates every 15 minutes for the default software versions.

sbgrid@computer $ crontab -l
*/15 * * * * /programs/x86_64-linux/sbgrid_installer/latest/sbgrid-cli admin

To check for updates for all versions (recommended), run sbgrid-cli crontab --all-versions which will create a crontab with the proper --all-versions flag.

sbgrid@computer $ crontab -l
*/15 * * * * /programs/x86_64-linux/sbgrid_installer/latest/sbgrid-cli admin --all-versions

Add the relevant --darwin or --linux flags to the crontab command if your installation should include software for macOS on linux or vice-versa, i.e. for an SBGrid installation shared across Mac and Linux workstations.

The sbgrid-admin Utility (legacy method, deprecated)

The sbgrid-admin utility is our deprecated legacy method to install and manage the software collection. This script can be used to install the complete software collection in what we refer to as a "Site installation", but has a number of limitations. We are no longer using this mode for new installations but older sbgrid-admin installations are supported.

Usage

sbgrid-admin includes built-in help:

 $ sbgrid-admin -h
sbgrid-admin: a tool for administering an SBGrid Software suite installation

Usage: sbgrid-admin [-chirstu]

  -c Cron mode.  Used for automated updates of the software suite.
  -h Help message.
  -i Install the software suite on this machine.
  -p Print a launchd plist for updating the software on OS X machines.
  -r Regenerate sh/csh shell configuration files.
  -s Install the software branch for SGI IRIX.
  -t Test the configuration file.
  -u Force manual update of an existing installation.

The most commonly used option is sbgrid-admin -u to force a manual update. This will synchronize your local installation with the SBGrid master installation hosted at Harvard Medical School.

The SBGrid software configuration files use both static and dynamically generated files. The main shell initialization files, /programs/sbgrid.cshrc and /programs/sbgrid.shrc are static files, but the operating-system specific configurations are dynamically generated and customized for the packages available at your site. sbgrid-admin -r will regenerate the branch-specific configuration files should you accidentally delete or alter them.

Configuration Options

The sbgrid-admin script has some additional configuration options to accommodate the installation requirements at different sites.

The ~sbgrid/.sbgridrc configuration file for the sbgrid-admin script must contain a sitename and site key. All other configurations are optional.

Options take the form of OPTION_NAME="setting".

Option	Example	Description
SB_SITENAME	sbgrid-bene	The site name for your installation provided by SBGrid
SB_SITEKEY	xyxyxyxyxyxyxy	The installation key for your site provided by SBGrid
INSTALL_TARGET	/nfs/programs	The installation directory for the software
RSYNC_ARGS	"--port 8080"	Advanced arguments to rsync. Can be used to workaround blocked rsync ports.
HTTP_PROXY	host.name:port	Proxy hostname and port for use by rsync and curl
POST_UPDATE_SCRIPT	/absolute/path/to/script	Optional script to be run after a successful software update
CRON_SCHEDULE	5 1 * * *	Set a custom cron schedule

Rsync Transaction Logging

If you want a date-stamped log of all rsync downloads, deletions and updates, you can add a line like this to the ~sbgrid/.sbgridrc file:

RSYNC_ARGS="--log-file=$HOME/logs/sbgrid-admin-`date +%Y%m%d`.log"

This will create a date-stamped log file any time an actual change to the software tree happens. In general this should not be necessary, but for troubleshooting purposes it can be useful.