Merge pull request #13 from PageUpPeopleOrg/feature/OSC-1238_ci-cd

[OSC-1238] Add CI/CD
Overview

Integration with Travis CI! 🎉
Add Pipenv support (highly recommended to manage dependencies/venv)
Add PyTest for when we do write unit tests
Changes

Add build status to README.md
Add .travis.yml
Add Pipfile and Pipfile.lock
Add tests folder directory
Contains integration and unit tests
Rename psycopg2 dependency to psycopg2-binary
Update setup.py to reflect changes
Notes

CI/CD

Travis builds will perform two types of testing:

Unit testing
Integration testing
Integration tests perform the following:

Create stub load models
a. load_model_1.json, load_model_2.json
Initialise a new execution
Compare load models and assert that the models changed are * (as no models have been compared)
Complete execution and assert that last successful execution ID is the one returned from init
Initalise another execution
Modify load_model_1.json
Compare load models and assert that models changed are load_model_1.json
Complete execution and assert that last successful execution ID is the one returned from init

Author: ChintanRaval

.gitignore

@@ -109,4 +109,5 @@ venv.bak/
 # JetBrains
 .idea/
 
-test-models/
+test-models/
+tests/integration/models/

.travis.yml

@@ -0,0 +1,15 @@
+dist: xenial
+language: python
+python: 3.7
+env:
+  - PIPENV_VERBOSITY=-1
+
+services: docker
+before_install:
+  - docker pull postgres
+  - docker run -p 5432:5432 -e POSTGRES_PASSWORD=travisci -d postgres
+
+install: make install_deps
+script:
+  - make test_unit
+  - make test_integration

Makefile

@@ -0,0 +1,13 @@
+# Install dependencies
+install_deps:
+	pip install pipenv --upgrade
+	pipenv install --dev
+
+# Run unit tests
+test_unit:
+	pipenv run pytest
+
+# Run integration tests
+test_integration:
+	./tests/integration/test_integration.sh
+	

Pipfile

@@ -0,0 +1,14 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+pytest = "*"
+
+[packages]
+psycopg2-binary = "==2.7.7"
+SQLAlchemy = "==1.2.17"
+
+[requires]
+python_version = "3.7"

Pipfile.lock

@@ -1,20 +1,13 @@
 # Model Change Detector
 
-## About
+[![Build Status](https://travis-ci.com/PageUpPeopleOrg/model-change-detector.svg?branch=master)](https://travis-ci.com/PageUpPeopleOrg/model-change-detector)
 
 A utility that persists state of a data pipeline execution and uses them to detect changes in models.
 
 ## Usage
 
-```commandline
-py mcd.py [options] <command> [command-parameters]
 ```
-
-To get help,use:
-
-```commandline
-py mcd.py -h
-py mcd.py <db-connection-string> <command> -h
+$ python mcd.py [options] {db-connection-string} <command> [command-parameters]
 ```
 
 - `options` include:
@@ -38,121 +31,113 @@ py mcd.py <db-connection-string> <command> -h
   - `complete-execution`: Marks the completion of an existing execution by updating a record for the same in the given database. Returns nothing unless there's an error.
     - `execution-id`: a GUID identifier of an existing data pipeline execution as returned by the `init` command.
 
-### As a script
-
-- Use a local isolated/virtual python environment for this project
-- Install project dependencies
-- `py mcd.py [options] <command> [command-parameters]`
+To get help, use:
 
-_Windows example:_
+```
+$ python mcd.py --help
+$ python mcd.py <command> --help
+```
 
-```commandline
-py -m venv new-env --clear
-new-env\scripts\activate
+### Usage Example
 
-py -m pip install -r requirements.txt
+```
+$ pipenv install
+$ pipenv shell
 
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname init-execution
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname init-execution
 
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname get-last-successful-execution
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname get-execution-last-updated-timestamp id-as-returned-by-get-last-successful-execution-command
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname get-last-successful-execution
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname get-execution-last-updated-timestamp id-as-returned-by-get-last-successful-execution-command
 
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command load ./relative/path/to/load/models **/*.json
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command load
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command load ./relative/path/to/load/models **/*.json
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command load
 
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command transform C:/absolute/path/to/transform/models group1/*.csv ./group2/**/*.sql
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command transform
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command transform C:/absolute/path/to/transform/models group1/*.csv ./group2/**/*.sql
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command transform
 
-py mcd.py postgresql+psycopg2://user:password@host:port/dbname complete-execution id-as-retured-by-init-command
+$ python mcd.py postgresql+psycopg2://user:password@host:port/dbname complete-execution id-as-retured-by-init-command
 ```
 
-### As a package
+## Prerequisites
 
-- Use/create an empty directory
-- Use a local isolated/virtual python environment for this project
-- [Install](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs) this package
-  - `pip install -e path/to/ProjectX`
-  - `pip install -e git+git://github.com/ProjectX.git#egg=ProjectX`
-- `py -m mcd [options] <command> <command-options>`
+- [Python 3](https://www.python.org/downloads/)
+- [Pipenv](https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv)
 
-_Windows example:_
+### Verify Installation
 
-```commandline
-mkdir new-dir
-cd new-dir
+Verify dependencies are installed by running the following commands:
 
-py -m venv new-env --clear
-new-env\scripts\activate
-
-pip install -e git+git://github.com/PageUpPeopleOrg/model-change-detector.git#egg=mcd
-
-py -m mcd postgresql+psycopg2://user:password@host:port/dbname init-execution
+```
+$ python --version
+$ pipenv --version
 ```
 
-## Setup
+## Getting Started
 
-1. Install pre-requisites
-2. Use a local isolated/virtual python environment for this project
-3. Install project dependencies
-4. Develop and test code changes
-5. Once done, deactivate the virtual environment
+### Install dependencies
 
-### Install pre-requisites
+To install project dependencies, run the following command:
 
-#### Python 3
+```
+$ pipenv install
+```
 
-Install from [here](https://www.python.org/) _(or your choice of safe software provider)_. During installation, choose the option to _Add to PATH_ and _Custom Installation_ so you can provide the install path to not be the default `C:\Program Files\Python37\` and be outside the `Program Files` directory to say, `C:\Python37\`. This is just a suggestion since there have been issues with updating key python packages once Python is installed within `Program Files` on Windows.
+### Activate Virtual Environment
 
-Verify your installation by running the below commands.
+To activate a virtual environment, run the following command:
 
-```powershell
-py --version
-python --version
-pip --version
+```
+$ pipenv shell
 ```
 
-If you end up with multiple accidental/purposeful python installations, use the below in Windows Commandline to figure out where the executables are located.
+### Using MCD
 
-```cmd
-where py
-where python
-where pip
-```
+Once the virtual environment has been activated, please refer to [usage](#Usage) for how to use MCD.
+
+## Testing
+
+To run integration tests locally, it is highly recommended that [Docker](https://www.docker.com/) is installed.
+
+### Unit Tests
 
-### Use a local isolated/virtual python environment for this project
+To run unit tests, run the following command:
 
-`py -m venv /path/to/new/virtual/environment` _e.g._ `py -m venv new-env`
+```
+$ python pytest
+```
 
-If you build with `--system-site-packages` directory, your virtual environment will be allowed access to packages from your global site-packages directory. Although, if you want isolation from the global system, do not use this flag. Once you've created a new environment, you need to activate the same.
+### Integration Tests
 
-On Windows:
+To run integration tests, please ensure the following information is configured correctly:
 
-`path\to\environment\scripts\activate` _e.g._ `new-env\scripts\activate`
+- `tests/integration/test_integration.sh:9`
 
-On Linux / Mac OS
+Please ensure that the database connection string points to a valid PostgreSQL instance.
 
-`source path/to/environment/bin/activate` _e.g._ `source new-env/bin/activate`
+#### Docker
 
-You should see the name of your virtual environment in brackets on your terminal line, e.g.:
+If Docker is installed, running tests is as simple as running the following commands:
 
-```commandline
-C:\path\to\working\dir: new-env\scripts\activate
-(new-env) C:\path\to\working\dir: _
+```
+$ docker pull postgres
+$ docker run --name stubdatabase -p 5432:5432 -e POSTGRES_PASSWORD=travisci -d postgres
+$ make test_integration
+$ docker stop stubdatabase
+$ docker remove stubdatabase
 ```
 
-Any python commands you use will now, work within your virtual environment only.
+#### Local PostgreSQL
 
-### Install project dependencies
+If a local instance of PostgreSQL is installed, run integration tests with the following command:
 
-```powershell
-pip install -r requirements.txt
+```
+$ make test_integration
 ```
 
-### Deactivate the virtual environment
+#### Notes
 
-Once done, deactivate the virtual environment with a simple `decativate` command, e.g.:
+If you do not have `make` installed, you can substitute `make` with:
 
-```commandline
-(new-env) C:\path\to\working\dir: deactivate
-C:\path\to\working\dir: _
+```
+$ ./tests/integration/test_integration.sh
 ```

README.md

@@ -4,7 +4,7 @@
       version='0.0.2',
       packages=find_packages(),
       install_requires=[
-          'psycopg2==2.7.7',
+          'psycopg2-binary==2.7.7',
           'SQLAlchemy==1.2.17'
       ]
       )