installation cleanup, more sbgrid-cli admin mode

+

+## ALPHAFOLD2

+<!-- TOC -->

+

+- [ALPHAFOLD2](#alphafold2)

+ - [Preparing to run Alphafold](#preparing-to-run-alphafold)

+ - [Running the python script run_alphafold.py](#running-the-python-script-run_alphafoldpy)

+ - [GPU Memory](#gpu-memory)

+ - [Web portal](#web-portal)

+ - [Changes in AlphaFold 2.1.1 multimer](#changes-in-alphafold-211-multimer)

+ - [Examples](#examples)

+ - [Known issues](#known-issues)

+

+<!-- /TOC -->

+### Preparing to run Alphafold

+The ALPHAFOLD2 source an implementation of the inference pipeline of AlphaFold v2.0. using a completely new model that was entered in CASP14. This is not a production application per se, but a reference that is capable of producing structures from a single amino acid sequence.

+

+From the developers' original publication: "The provided inference script is optimized for predicting the structure of a single protein, and it will compile the neural network to be specialized to exactly the size of the sequence, MSA, and templates. For large proteins, the compile time is a negligible fraction of the runtime, but it may become more significant for small proteins or if the multi-sequence alignments are already precomputed. In the bulk inference case, it may make sense to use our make_fixed_size function to pad the inputs to a uniform size, thereby reducing the number of compilations required."

+

+The SBGrid installation of Alphafold2 does not require Docker to run, but does require a relatively recent NVidia GPU and updated driver.

+

+AlphaFold requires a set of (large) genetic databases that must be downloaded separately. See https://github.com/deepmind/alphafold#genetic-databases for more information.

+

+These databases can be downloaded with the included download script and the aria2c program, both of which are available in the SBGrid collection. Note that these databases are large in size (> 2Tb) and may require a significant amount of time to download.

+

+```

+/programs/x86_64-linux/alphafold/2.1.1/alphafold/scripts/download_all_data.sh <destination path>

+```

+

+The database directory shouuld look like this :

+

+```

+├── bfd

+│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_a3m.ffdata

+│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_a3m.ffindex

+│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_cs219.ffdata

+│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_cs219.ffindex

+│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_hhm.ffdata

+│   └── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_hhm.ffindex

+├── mgnify

+│   ├── mgy_clusters_2018_12.fa

+├── params

+│   ├── LICENSE

+│   ├── params_model_1_multimer.npz

+│   ├── params_model_1.npz

+│   ├── params_model_1_ptm.npz

+│   ├── params_model_2_multimer.npz

+│   ├── params_model_2.npz

+│   ├── params_model_2_ptm.npz

+│   ├── params_model_3_multimer.npz

+│   ├── params_model_3.npz

+│   ├── params_model_3_ptm.npz

+│   ├── params_model_4_multimer.npz

+│   ├── params_model_4.npz

+│   ├── params_model_4_ptm.npz

+│   ├── params_model_5_multimer.npz

+│   ├── params_model_5.npz

+│   └── params_model_5_ptm.npz

+├── pdb70

+│   ├── md5sum

+│   ├── pdb70_a3m.ffdata

+│   ├── pdb70_a3m.ffindex

+│   ├── pdb70_clu.tsv

+│   ├── pdb70_cs219.ffdata

+│   ├── pdb70_cs219.ffindex

+│   ├── pdb70_hhm.ffdata

+│   ├── pdb70_hhm.ffindex

+│   └── pdb_filter.dat

+├── pdb_mmcif

+│   ├── mmcif_files

+│   └── obsolete.dat

+├── pdb_seqres

+│   └── pdb_seqres.txt

+├── small_bfd

+│   └── bfd-first_non_consensus_sequences.fasta

+├── uniclust30

+│   └── uniclust30_2018_08

+├── uniprot

+│   └── uniprot.fasta

+```

+

+### Running the python script run_alphafold.py

+The run_alphafold.py script requires all parameters to be set explicitly.

+Pass --helpshort or --helpfull to see help on flags. See examples below.

+

+### GPU Memory

+Memory is going to be an issue with larger protein sizes. The original publication suggests some things to try:

+

+"Inferencing large proteins can easily exceed the memory of a single GPU. For a V100 with 16 GB of memory, we can predict the structure of proteins up to ~1,300 residues without ensembling and the 256- and 384-residue inference times are using a single GPU’s memory. "

+

+"The memory usage is approximately quadratic in the number of residues, so a 2,500 residue protein involves using unified memory so that we can greatly exceed the memory of a single V100. In our cloud setup, a single V100 is used for computation on a 2,500 residue protein but we requested four GPUs to have sufficient memory."

+

+The following environment variable settings may help with larger polypeptide calculations (> 1,200 aa).

+

+```

+TF_FORCE_UNIFIED_MEMORY=1

+XLA_PYTHON_CLIENT_MEM_FRACTION=0.5

+XLA_PYTHON_CLIENT_ALLOCATOR=platform

+```

+*Thanks Ci Ji Lim at Wisconsin for suggesting and testing these.*

+

+### Web portal

+It is possible to run alphafold through a web portal. See

+https://colab.research.google.com/github/deepmind/alphafold/blob/main/notebooks/AlphaFold.ipynb .

+

+### Changes in AlphaFold 2.1.1 multimer

+On 4 Nov 2021 we added vesion 2.1.1 to the installation. This version allows prediction of multimers from fasta files containing multiple sequences.

+

+**NOTE**

+- Databases must be redownloaded, specifically UniProt and PDB seqres databases.

+ - Run `scripts/download_uniprot.sh <DOWNLOAD_DIR>`.

+ - Remove/rename `<DOWNLOAD_DIR>/pdb_mmcif`. It is needed to have PDB SeqRes and PDB from exactly the same date. Failure to do this step will result in potential errors when searching for templates when running AlphaFold-Multimer.

+ - Run `scripts/download_pdb_mmcif.sh <DOWNLOAD_DIR>`.

+ - Run `scripts/download_pdb_seqres.sh <DOWNLOAD_DIR>`.

+- Update the model parameters.

+ - Remove/rename the old model parameters in `<DOWNLOAD_DIR>/params`.

+ - Download new model parameters using `scripts/download_alphafold_params.sh <DOWNLOAD_DIR>`.

+

+Some command line flags have changed since version 2.0.0. We recommend running the `run_alphafold.py` command directly and we are not providing a wrapper script ( as we did for 2.0.0) at the present time.

+

+### Examples

+**Standard prediction example :**

+```

+/programs/x86_64-linux/alphafold/2.1.1/bin.capsules/run_alphafold.py \

+--data_dir=/programs/local/alphafold/ \

+--output_dir=/scratch/data/sbgrid/alphafold/test_monomer \

+--fasta_paths=test_monomer.fasta \

+--max_template_date=2020-05-14 \

+--db_preset=full_dbs \

+--bfd_database_path=/programs/local/alphafold/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \

+--uniclust30_database_path=/programs/local/alphafold//uniclust30/uniclust30_2018_08/uniclust30_2018_08 \

+--uniref90_database_path=/programs/local/alphafold//uniref90/uniref90.fasta \

+--mgnify_database_path=/programs/local/alphafold//mgnify/mgy_clusters_2018_12.fa \

+--template_mmcif_dir=/programs/local/alphafold//pdb_mmcif/mmcif_files \

+--pdb70_database_path=/programs/local/alphafold//pdb70/pdb70 \

+--obsolete_pdbs_path=/programs/local/alphafold//pdb_mmcif/obsolete.dat

+```

+

+where test_monomer.fasta is

+

+```

+>T1083

+GAMGSEIEHIEEAIANAKTKADHERLVAHYEEEAKRLEKKSEEYQELAKVYKKITDVYPNIRSYMVLHYQNLTRRYKEAAEENRALAKLHHELAIVED

+```

+

+**Multimer example :**

+```

+/programs/x86_64-linux/alphafold/2.1.1/bin.capsules/run_alphafold.py \

+--data_dir=/programs/local/alphafold \

+--output_dir=/scratch/data/sbgrid/alphafold/test_multimer \

+--fasta_paths=test_multimer.fasta \

+--max_template_date=2020-05-14 \

+--db_preset=full_dbs \

+--bfd_database_path=/programs/local/alphafold/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \

+--uniclust30_database_path=/programs/local/alphafold/uniclust30/uniclust30_2018_08/uniclust30_2018_08 \

+--uniref90_database_path=/programs/local/alphafold/uniref90/uniref90.fasta \

+--mgnify_database_path=/programs/local/alphafold/mgnify/mgy_clusters_2018_12.fa \

+--template_mmcif_dir=/programs/local/alphafold/pdb_mmcif/mmcif_files --model_preset=multimer \

+--uniprot_database_path=/programs/local/alphafold/uniprot/uniprot.fasta \

+--pdb_seqres_database_path=/programs/local/alphafold/pdb_seqres/pdb_seqres.txt \

+--obsolete_pdbs_path=/programs/local/alphafold/pdb_mmcif/obsolete.dat

+```

+

+where test_multimer.fasta is

+

+```

+>T1083

+GAMGSEIEHIEEAIANAKTKADHERLVAHYEEEAKRLEKKSEEYQELAKVYKKITDVYPNIRSYMVLHYQNLTRRYKEAAEENRALAKLHHELAIVED

+>T1084

+MAAHKGAEHHHKAAEHHEQAAKHHHAAAEHHEKGEHEQAAHHADTAYAHHKHAEEHAAQAAKHDAEHHAPKPH

+```

+

+### Known issues

+- Unified memory across GPUs does not appear to work in the current version.

+- The `ptxas` executable is required to be in PATH in some cases, but not all. We can not redistribute this binary since it is part of the NVIDIA SDK. It must be installed separetely and added to the environment PATH variable, typically in `/usr/local/cuda/bin`. Version 11.0.3 works well in our hands, but other versions should work. [You can download the SDK here : https://developer.nvidia.com/cuda-toolkit-archive](https://developer.nvidia.com/cuda-toolkit-archive)

+- Clashes bewtween monomers have been reported in some cases in multimer mode

+

+The software collection can be installed using either our graphical or command-line installation manager:

+- [SBGrid installation manager command-line interface](sbgrid-cli)

+

+We also have a script-based installation for Unix systems that are not compatible with the Linux and macOS installatino managers(BSD, AIX, etc).

+

+[An SBGrid installation that includes the complete software collection for Linux and/or MacOS.](sbgrid-cli-admin)

+One of the primary benefits of an SBGrid software installation is that it is a curated and actively managed software stack. New titles are added, obsolete versions are removed, and updates happen automatically. New software and bug fixes are only an email request away (bugs@sbgrid.org) without required system administration. While the installation configuration is flexible (CLI/GUI, title/versions, Linux/macOS, etc), this is the recommended mode of installation for most workstations and clusters, large labs, or shared departmental installations using network file storage for their program stack.

+The MacOS GUI installation manager allows users to select the applications they need when they need them through a graphical application. Applications are updated on a rolling basis and are available to the user at their convenience. Updates are reported when available, but are managed by the user.

+The SBGrid CLI tool 'sbgrid-cli' allows users to choose the titles they wish to install using a command-line driven package manager. Applications can be installed and updated as convenient. Updates are released on a rolling basis and can be installed by the user as necessary. This program can be made [automatic](sbgrid-cli-admin) or used in manual mode.

+The SBGrid Software Suite can run on computers running Linux and macOS/OS X operating systems.

+ - [OS X / macOS](#os-x--macos)

+**GPUs**

+We support a number of titles that run on nVidia GPU hardware. We include the required libraries for these applications and typically only the latest driver for the hardware is required.

+### OS X / macOS

+We build and test programs for macOS under the most five most recent OS X/macOS releases.

+Currently Supported:

+- OS X 10.13 - macOS 12

+- As above with earlier versions of linux, there are many software applications that are fully functional on earlier versions of MacOSX in the SBGrid tree. We work to maintain an environment compatible with these OS versions, though they may not receive the latest applications.

+- "Apple Silicon" macs can run most of the software in the collection. Few native applications are currently available, but most 64-bit software works through Apple's Rosetta2 compatility environment. Notable exceptions are applications compiled with Intel compilers (older Rosetta and RELION verions).

+Windows provides Linux compatibility environment called "Windows Subsystem for Linux" (WSL), based on Ubuntu. WSL is still under active development. We suspect it is possible to get the SBGrid applications working with WSL, but we have no experience with Windows can can't provide support. If you have success with SBGrid and WSL, let us know.

+The applications are archived and available on request. SGI was fun.

+One of the primary benefits of an SBGrid software installation is that it is a managed software stack. New titles are added, obsolete versions are removed, and updates happen automatically. New software and bug fixes are an email request away (bugs@sbgrid.org) without requiring local administrators or system admistration expertise. While there are other installation modes available (GUI, default versions, select titles, etc), this is the recommended mode of installation for most sites.

+| Installation Requirements | |

+| ------------------------- | ------------------------------------------ |

+| Hard Drive Space | ~1+ Tb recommended for full installation |

+| Operating System | Linux, macOS |

+| Privileges | sudo / root for initial install only |

+| network | Outbound https, rsync |

+| Branch | size |

+| ------------ | ------ |

+| x86_64-linux | ~750 Gb |

+By default the SBGrid installation manager, `sbgrid-cli`, will install at `/opt/sbgrid`. This path typically requires sudo/root to create Linux and macOS. The sudo requirement can be skipped by using the `--target` flag to specify an alternate installation location. This is usually preferred as /opt may be on a filesystem with insufficient capacity. The *sbgrid* user must have write access to this target directory.

+**2. Log in as the 'sbgrid' user**

+[sbgrid-cli for linux](https://sbgrid.org/downloads/latest/sbgrid-cli_linux.tar.gz)

+[sbgrid-cli for macOS](https://sbgrid.org/downloads/latest/sbgrid-cli_macos.tar.gz)

+

+

+To download directly to a server or workstation from the command line :

+curl -LO https://sbgrid.org/downloads/latest/sbgrid-cli_linux.tar.gz

+

+curl -LO https://sbgrid.org/downloads/latest/sbgrid-cli_macos.tar.gz

+tar -zxvf sbgrid-cli_linux.tar.gz

+Activating the installation will create the required installation directories and core libraries for installing the software. Alternate installation paths can be used with the '--target flag'. The entire software collection will not be installed here - just an initial the core without the titles.

+./sbgrid-cli activate-site --target <path to software installation location>

+where the target path is writable. With --target, activation does not require sudo.

+The application will prompt for a site name and key which was provided by email. After this is entered and confirmed the software will be installed. A cron task is created for update checks.

+The manual creation of the "/programs" symlink to the target installation directory will be required, but only for machines running the software. /programs is not needed for updates or servers that host the programs as a network share.

+1. Once the download has completed, you'll need to add a "/programs" symlink in

+

+ On Linux:

+ `ln -s /path/to/sbgrid/install/target /programs`

+

+ On macOS, root level symlinks must be defined in the /etc/synthetic.conf file. This is a simple configuration file, [More info on synthetic.conf can be found here.](catalina#manual-creation-of-the-programs-path-after-macos-upgrade)

+3. Configure your workstations for compatibility:

+### SBGrid installation manager admin mode

+

+`sbgrid-cli admin mode replaces the sbgrid-admin script installation method used previously to install and automatically update SBGrid site installations. `sbgrid-cli admin` offers a some advantages:

+ - parallel updates for fast, more efficient transfers

+ - default version only for limited disk space

+ - rolling updates for immediate releases

+

+The `sbgrid-cli admin` command will:

+

+- Install all new titles added to SBGrid

+- Update existing versions when newer versions are added to SBGrid

+- Remove obsolete verisons that are no longer supported

+

+### sbgrid-cli admin Examples :

+sbgrid-cli admin commands are typically run periodically from cron.

+4. Full auto mode - all versions, all titles mac and linux, 4 processes. **This is the recommened installation configuration**.

+### More crontab examples

+1. Only update the titles that have been explicitly installed, no new additions

+FIX ME

+

+2. Run a daily cleanup to preserve space

+FIX ME

+

+For more info on cron configuration, see [https://www.man7.org/linux/man-pages/man5/crontab.5.html](https://www.man7.org/linux/man-pages/man5/crontab.5.html)

+

+## Migrating existing script-based "sbgrid-admin" site installations

+To migrate an existing site-install that uses our sbgrid-admin script, use the `sbgrid-cli migrate-admin` command as your *sbgrid* user. This will convert the legacy sbgrid-admin installation.