-
Notifications
You must be signed in to change notification settings - Fork 31
Legacy BiG SCAPE 1 Documentation
Legacy BiG-SCAPE 1 Documentation
To select the domains predicted in each BGC for the distance calculation, BiG-SCAPE uses two strategies. In the global mode, all domains from both BGCs are taken into account, whereas in the glocal mode, only a subset of the domains is chosen. The logic in glocal mode is the following:
During pairwise distance calculation, genes are represented as a concatenation of (Pfam) domain ids, and each BGC in the pair is represented as a list of those domain concatenations (strandedness is not taken into account).
BiG-SCAPE then uses the SequenceMatcher method from Python's difflib library to to find the longest match (internally called the LCS or "Longest Common Subcluster"). The second BGC is tried in the reverse orientation and the orientation with the largest LCS is kept.
To proceed to the next step, the LCS must be either 3 genes long, or contain at least one gene marked by antiSMASH as "Core Biosynthetic"
In the extension stage the selection of domains is extended for the BGC with least amount of genes up(down)stream. If the amount of genes up(down)stream is the same, each expansion will be attempted and scored, and the one with the best score will be kept. The remaining BGC domain selection (per side) will be tried to be expanded according to the following scoring algorithm in the Alignment Stage: for every gene in the reference BGC, a gene with the same domain organization is searched for in the remaining BGC. If such gene is found, the score will be added a bonus (match=5) plus a penalty proportional to the distance from the current position (number of genes * gap penalty where gap=-2) and the current position will be moved to the position of the matching gene. If a gene with the same domain organization is not found, the score will be decreased with a penalty (mismatch=-3)
This file contains a list of domains which will be given a special weight in the DSS index. The file should contain at least one column with the hmm model id and, optionally, a second column (separated by a tab) with a comment. By default, BiG-SCAPE will use the included anchor_domains.txt file but the user can specify her own file with the --anchorfile ANCHORFILE option.
By default, BiG-SCAPE will try to separate clusters into eight classes. This is done based on the product annotation from antiSMASH (see here). New labels introduced in antiSMASH 5 and 6 are annotated with a superscript.
The rules followed are currently:
| antiSMASH annotation | BiG-SCAPE class |
|---|---|
t1pks, T1PKS5
|
PKS I |
transatpks, t2pks, t3pks, otherks, hglks, transAT-PKS5, transAT-PKS-like5, T2PKS5, T3PKS5, PKS-like5, hglE-KS5, prodigiosin and combinations of these with {t1pks, T1PKS5} or themselves |
PKS other |
nrps, NRPS5, NRPS-like5, thioamide-NRP5, NAPAA6
|
NRPS |
lantipeptide, thiopeptide, bacteriocin, linaridin, cyanobactin, glycocin, LAP, lassopeptide, sactipeptide, bottromycin, head_to_tail, microcin, microviridin, proteusin, guanidinotides, lanthipeptide5, lipolanthine5, RaS-RiPP5, fungal-RiPP5, thioamitides5.1, lanthipeptide-class-i6, lanthipeptide-class-ii6, lanthipeptide-class-iii6, lanthipeptide-class-iv6, lanthipeptide-class-v6, ranthipeptide6, redox-cofactor6, RRE-containing6, epipeptide6, cyclic-lactone-autoinducer6, spliceotide6, crocagin and combinations of these |
RiPPs |
amglyccycl, oligosaccharide, cf_saccharide, saccharide5 and combinations of these |
Saccharides |
terpene |
Terpene |
| any of {PKS I} + any of {NRPS} | PKS/NRPS Hybrids |
acyl_amino_acids, arylpolyene, aminocoumarin, ectoine, butyrolactone, nucleoside, melanin, phosphoglycolipid, phenazine, phosphonate, other, cf_putative, resorcinol, indole, ladderane, PUFA, furan, hserlactone, fused, cf_fatty_acid, siderophore, blactam, fatty_acid5, PpyS-KS5, CDPS5, betalactone5, PBDE5, tropodithietic-acid5, NAGGN5, halogenated5, pyrrolidine6, mycosporine-like and any combined annotation |
Others |
| * | < mix > |
If the hybrids mode is enabled, some clusters could be analyzed in different classes (if those are valid classes):
If cluster is the PKS-NRP_Hybrids BiG-SCAPE class: the cluster will also be put in the NRPS class and one of the PKS classes (PKS I or PKS other). If the cluster contains the t1pks annotation, it will always be put on the PKS I class.
If the cluster is classified as Others and BiG-SCAPE detects that it's because of a multiple annotation (e.g. terpene-t1pks), BiG-SCAPE will also put the cluster in every different individual class.
Once the distance matrix is calculated for the data set, Gene Cluster Family (GCF) assignment is performed for every cutoff distance selected by the --cutoffs parameter.
The interactive visualization of BiG-SCAPE output will show the one with the largest number.
For every cutoff, BiG-SCAPE creates a network using all distances lower or equal than the current cutoff. The Affinity Propagation clustering algorithm is applied to each subnetwork of connected components that emerge from this procedure. The similarity matrix for Affinity Propagation includes all distances between members of the subnetwork (i.e. it includes those with distance greater than the current cutoff)
Gene Cluster Clan (GCC) setting (enabled by default) will perform a second layer of clustering on the GCFs. For this, Affinity Propagation will be applied again (i.e. on a network of subconnected components) but network nodes are represented by the GCFs defined at the cutoff level specified in the first value of the --clan_cutoff parameter (Default: 0.3). Clustering will be applied to the network of all GCFs connected by a distance lower or equal than the GCC cutoff (the second value of the --clan_cutoff parameter; larger distances are discarded. Default: 0.7). Inter-GCF distance is calculated as an average distance between the BGCs within both families.
Affinity propagation parameters used in both clustering layers: damping=0.9, max_iter=1000, convergence_iter=200
Starting from the --inputdir folder, BiG-SCAPE will recursively look for files with the .gbk extension. The following files are excluded:
- Filenames that include the string(s) specified in
--exclude_gbk_str(default: 'final'. This is in order to exclude the summary GenBank file produced by antiSMASH, which ends with <clustername>.final.gbk) - Files with spaces in their path (including the filename). Spaces don't work well with hmmer
- Files with the string '_ORF', which is used internally by BiG-SCAPE
- Files with duplicated names (e.g. in different folders)
- Files where no protein sequences could be extracted
- Files whose sequence (summed between all records) is shorter than
min_bgc_size - Files with format issues not parseable by BioPython
By default, only the following files are included:
- Files with 'cluster' in their name (antiSMASH 4)
- Files with 'region' in their name (antiSMASH 5)
If you need to exclude or include files with certain strings in their name, use the --exclude_gbk_str and
If two CDS features overlap (e.g. splicing events), BiG-SCAPE's behaviour is to allow for a maximum of 10% of the shortest CDS. If more overlap is detected, BiG-SCAPE will discard the smallest feature from the analysis.
The file's name (without extension) will be used in the following as the BGC name.
Note that at the time being, BiG-SCAPE does not do any particular analysis for a given taxon (i.e. bacterial, archeal, fungal or plant BGCs are treated the same)
-
cache: Holds precalculated data from the analysis. If BiG-SCAPE is run again and pointed to the same output folder, it will try to read and re-use files from this directory.
-
domains For each domain found in the analysis, three files are generated:
- fasta file: Contains the sequences of the same domain from all the proteins from all BGCs
- stk file: Alignment of each of the sequences using
hmmalignin stockholm format - algn file: Fasta file with the aligned domains in fasta format (parsed from the stockholm file). These are the sequences that will be used in DSS
-
domtable Raw output from domain prediction on each BGC's proteins sequences using
hmmscan - fasta Protein sequences from each BGC. Extracted from the CDS features in the GenBank file.
- pfd Parsed results from the domtable file in a tab-separated format. These results have already been filtered for overlapping domains. Columns: Cluster name, (per-domain) score, gene id (if present), envelope coordinate from, envelope coordinate to (of the domain prediction, in amino acids), pfam id, pfam descriptor, start coordinate gene, end coordinate gene, internal cds header.
- pfs A list of predicted domains for each BGC file.
- .dict files: internal files
-
domains For each domain found in the analysis, three files are generated:
- html_content: All the code necessary for the interactive visualization
- logs: Currently only holds the parameters used in each run (that points to this output folder) and the run time.
- network files: See more information in the next section.
- SVG: Arrow figures in svg format for every BGC in the analysis. Every figure has boxes representing the predicted domains. This are given random colors but can be changed by the user by modifying the domains_color_file.tsv file
Each run will generate its own set of output files which can be used for analysis using other tools (e.g. Cytoscape):
- Network_Annotations_Full.tsv a tab-separated file with information about each BGC that was successfully processed in the input. This includes: BGC name, the original accesion ID from the GenBank file, the description from the original GenBank file, the antiSMASH product prediction, the BiG-SCAPE class, the organism tag from the original GenBank file, and finally, the taxonomy string also from the GenBank file.
- Folders for each BiG-SCAPE class which contain:
- The .network file. One file for each cutoff selected.
- The Network Annotation file with the BGCs used for this particular class
- The clustering files. These contain, for each cutoff, a first column with the BGC name, and a second column, separated by a tab, with the label representing the cluster (GCF number) that the BGC was put in.
All of these files can be opened in a text editor.
Launch the interactive output by clicking on the index.html file or opening the file with any web browser. This file is located in the root of the output folder.
When opening the visualization page, you will be shown an overview page.
Click on the dropdown menu on the right to select the run that you want to visualize.
Download and decompress your favorite release of the code. In a terminal, this would be done like:
$ wget https://github.com/medema-group/BiG-SCAPE/archive/refs/tags/v1.1.9.zip
$ unzip BiG-SCAPE-1.1.9.zip
Although each library could be installed on its own, the use of a virtual environment is highly recommended. Here is a quick guide of BiG-SCAPE installation using Miniconda
Install Miniconda.
We recommend that you install the Python 3 version as default for all new conda environments.
$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh
Create a new environment and install all the packages listed above using the environment.yml file (see additional documentation about conda here).
$ conda env create -f bigscape_dependencies.yml
Activate new environment (you should see a change in the prompt)
$ conda activate bigscape
(bigscape) $
For simplicity, it will be put in the same BiG-SCAPE folder but it can be located elsewhere. BiG-SCAPE has been tested with versions 31 and 32, and newer versions should work fine. Note that in this example, version 32 is used. Browse the different versions here.:
(bigscape) $ cd BiG-SCAPE
(bigscape) $ wget https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.gz && gunzip Pfam-A.hmm.gz
(bigscape) $ hmmpress Pfam-A.hmm
(bigscape) $ python bigscape.py --version
(see the BiG-SCAPE v1 parameters section for an overview of the options)
To deactivate the environment do source deactivate.
If you use Docker, download the run_bigscape file from the repository using a web browser or from the command line:
$ curl -q https://raw.githubusercontent.com/medema-group/BiG-SCAPE/97d616c56930f27cf66424f81242564a7ed0d501/run_bigscape > run_bigscape
Then, change permissions to be able to execute the file, and run it:
$ chmod a+x run_bigscape
$ run_bigscape
If you don't (or can't) use conda, you will need to download all the files from the repository (e.g. from here, download and prepare the Pfam database (see above) and have the following packages installed in your system:
- Python (BiG-SCAPE development will focus on Python 3. We will try to keep compatibility with Python 2 as long as we can)
- The HMMER suite (3.1b2+)
- The Pfam database
- Biopython
- Numpy
- Fasttree
- scipy
- scikit-learn (tested with 0.19.1)
- NetworkX (v2.x)
Type python bigscape.py -h to display a list of all available parameters. See also the following sections:
Type -h or --help to display all current available options
--label <string>
By default, BiG-SCAPE runs will have a name such as YYYY-MM-DD_HH-MM-SS_[extra] where the extra string is hybrids (if activated) and the mode (i.e. glocal). With the --label option, it is possible to add an extra string to the run name. This will be reflected in the dropdown menu in the visualization page.
-i INPUTDIR, --inputdir INPUTDIR
Specify a path with the starting point to look for .gbk files. If empty, the search will start where the BiG-SCAPE files are located. The search is recursive. See more information here
-o OUTPUTDIR, --outputdir OUTPUTDIR
Output directory, this will contain all output data files. See its structure and more details about each type of result here
--include_gbk_str INCLUDE_GBK_STR
If any string in this list occurs in the filename, this file will be included in the analysis. (default: cluster, region)
--exclude_gbk_str EXCLUDE_GBK_STR
If any string in this list occurs in the filename, this file will not be used for the analysis. (default: final)
--pfam_dir PFAM_DIR
Location of Pfam files. Default is same location of BiG-SCAPE. See how to prepare these files in the installation instructions.
-c CORES, --cores CORES
BiG-SCAPE will try to paralellize some steps in the analysis like domain prediction and distance calculation. Use this option to set the number of cores the script may use. If not specified, BiG-SCAPE will use all available cores.
-v, --verbose
Prints more detailed information of each step in the analysis. Toggle to activate. Because of the amount of information, it might be a good idea to redirect output to to a file e.g.:
$> python bigscape.py <options> --verbose > run.log &
--include_singletons
Toggle to activate. This will include BGCs that don't have a distance lower than the cutoff distance specified.
-d DOMAIN_OVERLAP_CUTOFF, --domain_overlap_cutoff DOMAIN_OVERLAP_CUTOFF
Specify at which overlap percentage domains are considered to overlap. Domain with the best score is kept (default=0.1).
-m MIN_BGC_SIZE, --min_bgc_size MIN_BGC_SIZE
Provide the minimum size of a BGC to be included in the analysis. Default is 0 base pairs. This includes the sum of all loci in a multi-record GenBank file.
--mix
By default, BiG-SCAPE separates the analysis according to the BGC product and will create network directories for each class (see BiG-SCAPE classes). Toggle to include an analysis mixing all classes. As BiG-SCAPE needs to calculate an all-vs-all distance network, this might use a lot of memory.
--no_classify
By default, BiG-SCAPE classifies the output files analysis based on the BGC product. Toggle to deactivate (note that if the --mix parameter is not activated, BiG-SCAPE will not create any network file but all intermetiate files will be processed)
--banned_classes {PKSI, PKSother, NRPS, RiPPs, Saccharides, Terpene, PKS-NRP_Hybrids, Others}
BiG-SCAPE Classes that should NOT be included in the classification. E.g. "--banned_classes PKSI PKSOther". Strings in lowercase are also allowed.
--cutoffs {0.0-1.0}
Generate networks using multiple raw distance cutoff values, example: --cutoffs 0.1, 0.25, 0.5, 1.0. Default: c=0.3. For every cutoff value, a different network file will be generated. Regarding the interactive visualization, only the highest cutoff will be shown. Automatic clustering of Gene Cluster Families will be done using each cutoff.
--clans-off
By default, BiG-SCAPE will perform a second layer of clustering to group GCFs into GCCs. Toggle to deactivate this.
--clans_cutoff {0.0-1.0} {0.0-1.0}
Cutoff Parameters for which clustering families into clans will be performed in raw distance. First value is the cutoff used for finding GCFs that will be used for Clan calling (default: 0.3). If this GCF cutoff value is not included within --cutoffs, it will be added automatically. Second value is the GCC cutoff value for clustering families into clans (default: 0.7). Average distance between members of each pair of GCFs are used as inter-GCF distance. Every pair of GCFs linked with a distance of GCC cutoff value or less will be taken into account. Example: --clan_cutoff 0.5 0.8)
--hybrids-off
By default, BGCs with hybrid predicted products from the PKS/NRPS Hybrids and Others classes will be included into each subclass (e.g. a terpene-nrps BGC that will usually be classified in Others would be added to both the Terpene and NRPS classes). This means that the same cluster may appear in different classes. Toggle to deactivate
--mode {global,glocal,auto}
Alignment mode used when comparing each pair of gene clusters. global: the whole list of domains of each BGC are compared; glocal (default): Longest Common Subcluster mode. In it, the subset of the domains used to calculate distance is redefined by finding the longest slice of common domain content per gene in both BGCs, then expands each slice. auto: use glocal mode when at least one of the BGCs in each pair has the contig_edge annotation from antiSMASH v4+, otherwise use global mode on that pair.
--anchorfile ANCHORFILE
Point to a custom anchor file. Default is anchor_domains.txt, included in with the repository.
--force_hmmscan
Force domain prediction using hmmscan even if BiG-SCAPE finds processed domtable files (e.g. to use a new version of the Pfam database).
--skip_ma
Skip multiple alignment of domains' sequences. Use if alignments have been generated in a previous run. Domain sequence alignment will also be skipped if BiG-SCAPE reutilizes an output directory and no new BGCs are found within the input folder
--mibig, --mibig14, --mibig13
Use included BGCs from the MIBiG database. Currently, versions 2.1 (--mibig), 1.4 (--mibig14) and 1.3 (--mibig13) of the database are included in the BiG-SCAPE project as a compressed file, which will be unzipped the first time these options are used.
Note that this sets are different from the bundle found in the downloads section of the MIBiG site; these GenBank files have been processed by antiSMASH to annotate the BGC type. Additionally, the latest version is an unofficial version of the current content of MIBiG 'repository' page (i.e. it contains a few more BGCs than the official 2.0 bundle)
--query_bgc QUERY_BGC
Instead of making an all-VS-all comparison of all the input BGCs, choose one BGC to compare with the rest of the set (one-VS-all). The query BGC does not have to be within inputdir. The distances that will be used for the GCF and GCC analysis are all that are equal or lower than the maximum cutoff value. This will only take into account the BiG-SCAPE class(es) that the Query BGC belongs to.
--domain_includelist
Only include BGCs which include (any) domains with the Pfam accessions found in the domain_includelist.txt file. In this file, each line contains a single Pfam accession (with an optional comment, separated by a tab). Lines starting with "#" are ignored. See the file for an example using the P450 domain. Pfam accessions are case-sensitive.
--version
Show program's version number and exit