Merge branch 'dev' into 'main'

Finalization of v2.1.0

See merge request cdd/QSPRpred!127

Author: martin-sicho

.gitignore

@@ -5,12 +5,15 @@
 !.pre-commit-config.yaml
 /build/
 /docs/_build
+/qsprpred/_version.py
 *.pkg
 *.tgz
 *.gzip
 *.tar.gz
 *.out
-/**/test_files/qspr/
+/**/test_files/generated/
+/**/tutorial/test_files/
+/**/tutorial/cli_tutorial/
 
 ### Python template
 # Byte-compiled / optimized / DLL files

.gitlab-ci.yml

@@ -1,5 +1,9 @@
 # This workflow will install Python dependencies, run tests, and write documentation
 
+variables:
+   GIT_STRATEGY: clone # clone entire repo instead of reusing workspace
+   GIT_DEPTH: "0" # avoid shallow clone to give sonar all the info it needs
+
 stages:
    - test
    - deploy
@@ -25,16 +29,16 @@ stages:
       - cd clitest && ./run.sh
       - cd ../tutorial && ./run_all.sh
 
+# # Run tests on Python version 11 (linux image)
+# test:python3.11:
+#    extends: .test
+#    image: python:3.11
+
 # Run tests on Python version 10 (linux image)
 test:python3.10:
    extends: .test
    image: python:3.10
 
-# Run tests on Python version 9 (linux image)
-#test:python3.9:
-#   extends: .test
-#   image: python:3.9
-
 # Write documentation on main branch
 pages:
    stage: deploy
@@ -49,4 +53,4 @@ pages:
       paths:
       - public
    only:
-      - main
+      - main

CHANGELOG.md

@@ -1,14 +1,49 @@
 # Change Log
 
-From v2.0.0 to v2.0.1
+From v2.0.1 to v2.1.0.a2
 
 ## Fixes
 
-- Requirement python version in pyproject.toml updated to 3.10, as older version of python don't support the type hinting used in the code.
-- Corrected type hinting for `QSPRModel.handleInvalidsInPredictions`, which resulted in an error when importing the package in google colab.
-- The `predictMols` method returned random predictions in v2.0.0 due to unpatched shuffling code. This has now been fixed.
+- fixed error with serialization of the `DataFrameDescriptorSet` (#63)
+- Papyrus descriptors are not fetched by default anymore from the `Papyrus`  adapter, which caused fetching of unnecessary data.
+- A [potential bug in new version of pandas](https://github.com/pandas-dev/pandas/issues/55009)  broke scaffold generation so a workaround was implemented.
 
 ## Changes
+- `QSPRModel.evaluate` moved to a separate class `EvaluationMethod` in `qsprpred.models.interfaces`, with subclasses for cross-validation and making predictions on a test set in `qsprpred.models.evaluation_methods` (`CrossValidation` and `EvaluateTestSetPerformance` respectively).
+- `QSPRModel` attribute `scoreFunc` is removed.
+- 'qspr/models' is no longer added to the output path of `QSPRModel.save`, allowing for complete control over the output path.
+- `SKlearnMetrics.supportsTask` now uses a dictionary like dict[ModelTasks, list[str]] to map tasks to supported metric names. (#53)
+- `GBMTRandomSplit` and `ScaffoldSplit` now use the `GBMTDataSplit` to create balanced splits. `RandomSplit` still functions the same way as a completely random test split.
+- `PCMSplit` replaces `StratifiedPerTarget` and is compatible with `RandomSplit`, `ScaffoldSplit` and `ClusterSplit`.
+- `DuplicatesFilter` refactored to`RepeatsFilter`, as it also captures scenarios where triplicates/quadruplicates are found in the dataset. These scenarios are now also covered by the respective UnitTest.
+- The versioning scheme of development snapshots has changed from `devX` to `alphaX`/`betaX`, where `X` is an integer that increments with each release.
+- The following model class have been renamed and moved:
+    - `models.models.QSPRsklearn` > `models.sklearn.SklearnModel`
+    - `deep.models.QSPRDNN` > `extra.gpu.models.dnn.DNNModel`
+    - `extra.models.pcm.ModelPCM` > `extra.models.pcm.PCMModel`
+    - `extra.models.pcm.QSPRsklearnPCM` > `extra.models.pcm.SklearnPCMModel`
+- The command line interface modules now use input and output file paths instead
+  of automatically placing all files in a subfolder `qspr`, allowing for more
+  control over the output and input paths.
 
 ## New Features
-- raise error if search space for optuna optimization is missing search space type annotation or if type not in list
+- `GBMTDataSplit` - parent class to create globally balanced splits with the [gbmt-split](https://github.com/sohviluukkonen/gbmt-splits) package.
+- `ClusterSplit` - splits data based clustering of molecular fingerprints (uses `GBMTDataSplit`).
+- Raise error if search space for optuna optimization is missing search space type annotation or if type not in list.
+- When installing package with pip, the commit hash and date of the installation is saved into `qsprpred._version`
+- `HyperParameterOptimization` classes now accept a `evaluation_method` argument, which is an instance of `EvaluationMethod` (see above). This allows for hyperparameter optimization to be performed on a test set, or on a cross-validation set. (#11)
+- `HyperParameterOptimization` now accepts `score_aggregation` argument, which is a function that takes a list of scores and returns a single score. This allows for the use of different aggregation functions, such as `np.mean` or `np.median` to combine scores from different folds. (#45)
+- A new tutorial `adding_new_components.ipynb` has been added to the `tutorials` folder, which demonstrates how to add new model to QSPRpred.
+- A new function `Metrics.checkMetricCompatibility` has been added, which checks if a metric is compatible with a given task and a given prediction methods (i.e. `predict` or `predictProba`)
+- In `EvaluationMethod` (see above), an attribute `use_proba` has been added, which determines whether the `predict` or `predictProba` method is used to make predictions (#56).
+- Add new descriptorset `SmilesDesc` to use the smiles strings as a descriptor.
+- New module `early_stopping` with classes `EarlyStopping` and `EarlyStoppingMode` has been added. This module allows for more control over early stopping in models that support it.
+- Add new descriptorset `SmilesDesc` to use the smiles strings as a descriptor.
+- Refactoring of the test suite under `qsprpred.data` and improvement of temporary file handling (!114).
+- `PyBoostModel` - QSPRpred wrapper for py-boost models. Requires optional `pyboost` dependencies.
+- `ChempropModel` - QSPRpred wrapper for Chemprop models. Requires optional `deep` dependencies.
+- The `data_CLI` argument `--log_transform` (`-lt`) has been changed to `--transform_data` (`-t`), which now accepts a number of transformations to apply to the target data. Available transformations are `log`, `log10`, `log2`, `sqrt`, `cbrt`, `exp`, `exp2`, `exp10`, `square`, `cube`, `reciprocal`.
+- New `data_CLI`, `model_CLI` and `predict_CLI` argument `--skip_backup` (`-sb`) to skip the backup of the output files. WARNING: This will overwrite existing files.
+
+## Removed Features
+- `StratifiedPerTarget` is replaced by `PCMSplit`.

README.md

@@ -24,19 +24,18 @@ QSPRpred can be installed with pip like so (with python >= 3.10):
 pip install git+https://github.com/CDDLeiden/QSPRPred.git@main
 ```
 
-Note that this will install the basic dependencies, but not the optional dependencies. If you want to use the optional dependencies, you can install the package with the following installation options:
+Note that this will install the basic dependencies, but not the optional dependencies. If you want to use the optional dependencies, you can install the package with an option:
 
 ```bash
-# include deep neural network models and their dependencies
-pip install git+https://github.com/CDDLeiden/QSPRPred.git@main#egg=qsprpred[deep] 
-
-# include extra dependencies for PCM models and extra descriptor sets from packages other than RDKit
-pip install git+https://github.com/CDDLeiden/QSPRPred.git@main#egg=qsprpred[extra]
-
-# include all optional dependencies
-pip install git+https://github.com/CDDLeiden/QSPRPred.git@main#egg=qsprpred[full]
+pip install git+https://github.com/CDDLeiden/QSPRPred.git@main#egg=qsprpred[<option>]
 ```
 
+The following options are available:
+- extra : include extra dependencies for PCM models and extra descriptor sets from packages other than RDKit
+- deep : include deep learning models (torch and chemprop)
+- pyboost : include pyboost model (requires cupy, `pip install cupy-cudaX`, replace X with your [cuda version](https://docs.cupy.dev/en/stable/install.html))
+- full : include all optional dependecies (requires cupy, `pip install cupy-cudaX`, replace X with your [cuda version](https://docs.cupy.dev/en/stable/install.html))
+
 ### Multiple Sequence Alignment Provider for Protein Descriptors
 
 If you plan to optionally use QSPRPred to calculate protein descriptors for PCM, make sure to also install Clustal Omega. You can get it via `conda`:
@@ -45,20 +44,21 @@ If you plan to optionally use QSPRPred to calculate protein descriptors for PCM,
 
 conda install -c bioconda clustalo
 ```
-
-This is needed to provide multiple sequence alignments for the PCM descriptors. Clustal Omega does not, however, work on Windows. In that case, you can use MAFFT instead: 
+or install MAFFT instead:
 
 ```bash
 conda install -c biocore mafft
 ```
+This is needed to provide multiple sequence alignments for the PCM descriptors.
+At the moment, we do not support protein descriptor calculation for PCM on Windows.
 
 ## Use
 After installation, you will have access to various command line features, but you can also use the Python API directly (see [Documentation](https://cddleiden.github.io/QSPRPred/docs/)). For a quick start, you can also check out the  [Jupyter notebook tutorials](./tutorial), which documents the use of the Python API to build different types of models. [This tutorial](./tutorial/tutorial_training.ipynb) shows how a QSAR model can be trained. [This tutorial](./tutorial/tutorial_usage.ipynb) shows how to use a QSAR model to predict the bioactivity of a set of molecules. The tutorials as well as the [documentation](https://cddleiden.github.io/QSPRPred/docs/use.html) are still work in progress, and we will be happy for any contributions where it is still lacking.
 
 To use the commandline to train the same QSAR model as in the tutorial use (run from tutorial folder):
 ```bash
-python -m qsprpred.data_CLI -i parkinsons_pivot.tsv -pr GABAAalpha -r true -sf 0.15 -fe Morgan
-python -m qsprpred.model_CLI -pr GABAAalpha -r true -m PLS -s -o bayes -nt 10 -me
+python -m qsprpred.data_CLI -i ./data/parkinsons_pivot.tsv -o qspr/data -pr GABAAalpha -pr NMDA -r true -sp random -sf 0.15 -fe Morgan
+python -m qsprpred.model_CLI -dp ./qspr/data/GABAAalpha_REGRESSION_df.pkl -o ./qspr/models -m PLS -o bayes -nt 5 -me -s
 ```
 
 Workflow

clitest/run.sh

@@ -6,7 +6,7 @@ export PYTHONPATH=".."
 
 # input data and base directory
 export TEST_BASE="."
-export TEST_DATA='test_data_large.tsv'
+export TEST_DATA='data/test_data_large.tsv'
 
 function cleanup() {
   rm -rf ${TEST_BASE}/data/backup_*;
@@ -25,21 +25,21 @@ export TRAIN_BATCH=32
 export TRAIN_GPUS=0
 export N_CPUS=2
 export OPTIMIZATION='bayes'
-export SEARCH_SPACE='data/search_space/search_space_test'
+export SEARCH_SPACE='./data/search_space/search_space_test.json'
 export N_TRIALS=2
 
 ###############
 # DATA #
 ###############
 python -m qsprpred.data_CLI \
--b ${TEST_BASE} \
 -de \
--i ${TEST_DATA} \
+-i ${TEST_BASE}/${TEST_DATA} \
+-o ${TEST_BASE}/qspr/data/ \
 -ncpu ${N_CPUS} \
 -sm  ${SMILES} \
 -pr  CL fu \
 -th '{"CL":[6.5],"fu":[0.3]}' \
--lt '{"CL":true,"fu":false}' \
+-tr '{"CL":"log"}' \
 -sp 'time' \
 -stc 'Year of first disclosure' \
 -st 2000 \
@@ -54,13 +54,13 @@ python -m qsprpred.data_CLI \
 # MODELLING #
 ###############
 python -m qsprpred.model_CLI \
--b ${TEST_BASE} \
+-o ${TEST_BASE}/qspr/models \
 -de \
--dp CL_fu_SINGLECLASS \
+-dp ${TEST_BASE}/qspr/data/CL_fu_SINGLECLASS_df.pkl \
 -ncpu ${N_CPUS} \
 --model_types RF \
 -s \
--o ${OPTIMIZATION} \
+-op ${OPTIMIZATION} \
 -ss ${SEARCH_SPACE} \
 -nt ${N_TRIALS} \
 -me
@@ -69,11 +69,11 @@ python -m qsprpred.model_CLI \
 # PREDICTING #
 ###############
 python -m qsprpred.predict_CLI \
--b ${TEST_BASE} \
 -de \
 -i ${TEST_DATA} \
+-o ${TEST_BASE}/qspr/predictions.tsv \
 -ncpu ${N_CPUS} \
--mp ./qspr/models/RF_CL_fu_SINGLECLASS/RF_CL_fu_SINGLECLASS_meta.json \
+-mp ${TEST_BASE}/qspr/models/RF_CL_fu_SINGLECLASS/RF_CL_fu_SINGLECLASS_meta.json \
 -pr \
 -fv 0.0
 

docs/api/qsprpred.data.utils.rst

@@ -12,6 +12,14 @@ Subpackages
 Submodules
 ----------
 
+qsprpred.data.utils.data\_clustering module
+-------------------------------------------
+
+.. automodule:: qsprpred.data.utils.data_clustering
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 qsprpred.data.utils.datafilters module
 --------------------------------------
 

docs/api/qsprpred.deep.models.rst

@@ -0,0 +1,53 @@
+qsprpred.extra.gpu.models package
+=================================
+
+Submodules
+----------
+
+qsprpred.extra.gpu.models.chemprop module
+-----------------------------------------
+
+.. automodule:: qsprpred.extra.gpu.models.chemprop
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+qsprpred.extra.gpu.models.dnn module
+------------------------------------
+
+.. automodule:: qsprpred.extra.gpu.models.dnn
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+qsprpred.extra.gpu.models.neural\_network module
+------------------------------------------------
+
+.. automodule:: qsprpred.extra.gpu.models.neural_network
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+qsprpred.extra.gpu.models.pyboost module
+----------------------------------------
+
+.. automodule:: qsprpred.extra.gpu.models.pyboost
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+qsprpred.extra.gpu.models.tests module
+--------------------------------------
+
+.. automodule:: qsprpred.extra.gpu.models.tests
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: qsprpred.extra.gpu.models
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/api/qsprpred.extra.gpu.models.rst

@@ -1,18 +1,18 @@
-qsprpred.deep package
-=====================
+qsprpred.extra.gpu package
+==========================
 
 Subpackages
 -----------
 
 .. toctree::
    :maxdepth: 4
 
-   qsprpred.deep.models
+   qsprpred.extra.gpu.models
 
 Module contents
 ---------------
 
-.. automodule:: qsprpred.deep
+.. automodule:: qsprpred.extra.gpu
    :members:
    :undoc-members:
    :show-inheritance:

docs/api/qsprpred.deep.rst docs/api/qsprpred.extra.gpu.rstdocs/api/qsprpred.deep.rst renamed to docs/api/qsprpred.extra.gpu.rst

@@ -4,6 +4,14 @@ qsprpred.extra.models package
 Submodules
 ----------
 
+qsprpred.extra.models.models module
+-----------------------------------
+
+.. automodule:: qsprpred.extra.models.models
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 qsprpred.extra.models.pcm module
 --------------------------------