Merge pull request #19 from PageUpPeopleOrg/feature/OSC-1290_rename_mcd_to_dpo

ChintanRaval · web-flow · commit 3a83e6ae8c44 · 2019-04-03T14:15:04.000+11:00
OSC-1290 Rename to Data Pipeline Orchestrator / DPO
diff --git a/README.md b/README.md
@@ -1,13 +1,13 @@
-# Model Change Detector
+# Data Pipeline Orchestrator
 
-[![Build Status](https://travis-ci.com/PageUpPeopleOrg/model-change-detector.svg?branch=master)](https://travis-ci.com/PageUpPeopleOrg/model-change-detector)
+[![Build Status](https://travis-ci.com/PageUpPeopleOrg/data-pipeline-orchestrator.svg?branch=master)](https://travis-ci.com/PageUpPeopleOrg/data-pipeline-orchestrator)
 
 A utility that persists state of a data pipeline execution and uses them to detect changes in models.
 
 ## Usage
 
 ```
-$ python -m mcd [options] {db-connection-string} <command> [command-parameters]
+$ python -m dpo [options] {db-connection-string} <command> [command-parameters]
 ```
 
 - `options` include:
@@ -34,8 +34,8 @@ $ python -m mcd [options] {db-connection-string} <command> [command-parameters]
 To get help, use:
 
 ```
-$ python -m mcd --help
-$ python -m mcd <command> --help
+$ python -m dpo --help
+$ python -m dpo <command> --help
 ```
 
 ### Usage Example
@@ -44,18 +44,18 @@ $ python -m mcd <command> --help
 $ pipenv install
 $ pipenv shell
 
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname init-execution
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname init-execution
 
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname get-last-successful-execution
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname get-execution-last-updated-timestamp id-as-returned-by-get-last-successful-execution-command
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname get-last-successful-execution
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname get-execution-last-updated-timestamp id-as-returned-by-get-last-successful-execution-command
 
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command load ./relative/path/to/load/models **/*.json
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command load
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command load ./relative/path/to/load/models **/*.json
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command load
 
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command transform C:/absolute/path/to/transform/models group1/*.csv ./group2/**/*.sql
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command transform
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname persist-models id-as-retured-by-init-command transform C:/absolute/path/to/transform/models group1/*.csv ./group2/**/*.sql
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname compare-models id-as-retured-by-get-last-successful-execution-command id-as-retured-by-init-command transform
 
-$ python -m mcd postgresql+psycopg2://user:password@host:port/dbname complete-execution id-as-retured-by-init-command
+$ python -m dpo postgresql+psycopg2://user:password@host:port/dbname complete-execution id-as-retured-by-init-command
 ```
 
 ## Prerequisites
@@ -90,9 +90,9 @@ To activate a virtual environment, run the following command:
 $ pipenv shell
 ```
 
-### Using MCD
+### Using DPO
 
-Once the virtual environment has been activated, please refer to [usage](#Usage) for how to use MCD.
+Once the virtual environment has been activated, please refer to [usage](#Usage) for how to use DPO.
 
 ## Testing
 
@@ -103,17 +103,23 @@ To run integration tests locally, it is highly recommended that [Docker](https:/
 To run unit tests, run the following command:
 
 ```
-$ python pytest
+$ pytest
 ```
 
 ### Integration Tests
 
-To run integration tests, please ensure the following information is configured correctly:
+Before running integration tests, please ensure the following information is configured correctly:
 
-- `tests/integration/test_integration.sh:9`
+- `tests/integration/test_integration.sh:7`
 
 Please ensure that the database connection string points to a valid PostgreSQL instance with valid parameters.
 
+To run integration tests, run the following command:
+
+```
+$ ./tests/integration/test_integration.sh
+```
+
 #### Docker
 
 If Docker is installed, running tests is as simple as running the following commands:
diff --git a/dpo/BaseObject.py b/dpo/BaseObject.py
diff --git a/dpo/DataPipelineOrchestrator.py b/dpo/DataPipelineOrchestrator.py
@@ -1,18 +1,18 @@
 import argparse
 import logging
 
-from mcd import Shared
-from mcd.BaseObject import BaseObject
-from mcd.Shared import Constants
-from mcd.commands.InitialiseExecutionCommand import InitialiseExecutionCommand
-from mcd.commands.GetLastSuccessfulExecutionCommand import GetLastSuccessfulExecutionCommand
-from mcd.commands.GetExecutionLastUpdateTimestampCommand import GetExecutionLastUpdateTimestampCommand
-from mcd.commands.PersistModelsCommand import PersistModelsCommand
-from mcd.commands.CompareModelsCommand import CompareModelsCommand
-from mcd.commands.CompleteExecutionCommand import CompleteExecutionCommand
-
-
-class ModelChangeDetector(BaseObject):
+from dpo import Shared
+from dpo.BaseObject import BaseObject
+from dpo.Shared import Constants
+from dpo.commands.InitialiseExecutionCommand import InitialiseExecutionCommand
+from dpo.commands.GetLastSuccessfulExecutionCommand import GetLastSuccessfulExecutionCommand
+from dpo.commands.GetExecutionLastUpdateTimestampCommand import GetExecutionLastUpdateTimestampCommand
+from dpo.commands.PersistModelsCommand import PersistModelsCommand
+from dpo.commands.CompareModelsCommand import CompareModelsCommand
+from dpo.commands.CompleteExecutionCommand import CompleteExecutionCommand
+
+
+class DataPipelineOrchestrator(BaseObject):
     def __init__(self, logger=None):
         self.args = self.__get_arguments()
         Shared.configure_root_logger(self.args.log_level)
@@ -51,10 +51,10 @@ def __process_complete_execution_command(self):
     def __get_arguments(self):
         parser = argparse.ArgumentParser(
             description=Constants.APP_NAME,
-            usage='mcd [options] <db-connection-string> <command> [command-parameters]\n\n'
+            usage='dpo [options] <db-connection-string> <command> [command-parameters]\n\n'
                   'To see help text, you can run\n'
-                  '  mcd --help\n'
-                  '  mcd <db-connection-string> <command> --help\n\n',
+                  '  dpo --help\n'
+                  '  dpo <db-connection-string> <command> --help\n\n',
             parents=[Shared.get_default_arguments()])
 
         parser.add_argument(
diff --git a/dpo/DataRepository.py b/dpo/DataRepository.py
@@ -1,11 +1,11 @@
 from sqlalchemy import desc, select
 from sqlalchemy.sql import func
 from sqlalchemy.orm import sessionmaker
-from mcd import Shared
-from mcd.BaseObject import BaseObject
-from mcd.Shared import Constants
-from mcd.entities.DataPipelineExecutionEntity import DataPipelineExecutionEntity
-from mcd.entities.ModelChecksumEntity import ModelChecksumEntity
+from dpo import Shared
+from dpo.BaseObject import BaseObject
+from dpo.Shared import Constants
+from dpo.entities.DataPipelineExecutionEntity import DataPipelineExecutionEntity
+from dpo.entities.ModelChecksumEntity import ModelChecksumEntity
 
 
 class DataRepository(BaseObject):
diff --git a/dpo/Shared.py b/dpo/Shared.py
@@ -6,7 +6,7 @@
 
 
 class Constants:
-    APP_NAME = 'model-change-detector'
+    APP_NAME = 'data-pipeline-orchestrator'
     DATA_PIPELINE_EXECUTION_SCHEMA_NAME = 'data_pipeline'
     NO_LAST_SUCCESSFUL_EXECUTION = 'NO_LAST_SUCCESSFUL_EXECUTION'
 
diff --git a/dpo/__init__.py b/dpo/__init__.py
diff --git a/dpo/__main__.py b/dpo/__main__.py
@@ -0,0 +1,3 @@
+from dpo.DataPipelineOrchestrator import DataPipelineOrchestrator
+
+DataPipelineOrchestrator()
diff --git a/dpo/commands/BaseCommand.py b/dpo/commands/BaseCommand.py
@@ -1,6 +1,6 @@
 from sqlalchemy import create_engine
-from mcd.DataRepository import DataRepository
-from mcd.BaseObject import BaseObject
+from dpo.DataRepository import DataRepository
+from dpo.BaseObject import BaseObject
 
 
 class BaseCommand(BaseObject):
diff --git a/dpo/commands/CompareModelsCommand.py b/dpo/commands/CompareModelsCommand.py
@@ -1,5 +1,5 @@
-from mcd.commands.BaseCommand import BaseCommand
-from mcd.Shared import Constants
+from dpo.commands.BaseCommand import BaseCommand
+from dpo.Shared import Constants
 
 
 class CompareModelsCommand(BaseCommand):
diff --git a/dpo/commands/CompleteExecutionCommand.py b/dpo/commands/CompleteExecutionCommand.py
@@ -1,6 +1,6 @@
 import hashlib
 from pathlib import Path
-from mcd.commands.BaseCommand import BaseCommand
+from dpo.commands.BaseCommand import BaseCommand
 
 
 class CompleteExecutionCommand(BaseCommand):
diff --git a/dpo/commands/GetExecutionLastUpdateTimestampCommand.py b/dpo/commands/GetExecutionLastUpdateTimestampCommand.py
@@ -1,5 +1,5 @@
-from mcd.commands.BaseCommand import BaseCommand
-from mcd.Shared import Constants
+from dpo.commands.BaseCommand import BaseCommand
+from dpo.Shared import Constants
 
 
 class GetExecutionLastUpdateTimestampCommand(BaseCommand):
diff --git a/dpo/commands/GetLastSuccessfulExecutionCommand.py b/dpo/commands/GetLastSuccessfulExecutionCommand.py
@@ -1,5 +1,5 @@
-from mcd.commands.BaseCommand import BaseCommand
-from mcd.Shared import Constants
+from dpo.commands.BaseCommand import BaseCommand
+from dpo.Shared import Constants
 
 
 class GetLastSuccessfulExecutionCommand(BaseCommand):
diff --git a/dpo/commands/InitialiseExecutionCommand.py b/dpo/commands/InitialiseExecutionCommand.py
@@ -1,4 +1,4 @@
-from mcd.commands.BaseCommand import BaseCommand
+from dpo.commands.BaseCommand import BaseCommand
 
 
 class InitialiseExecutionCommand(BaseCommand):
diff --git a/dpo/commands/PersistModelsCommand.py b/dpo/commands/PersistModelsCommand.py
@@ -1,6 +1,6 @@
 import hashlib
 from pathlib import Path
-from mcd.commands.BaseCommand import BaseCommand
+from dpo.commands.BaseCommand import BaseCommand
 
 
 class PersistModelsCommand(BaseCommand):
diff --git a/dpo/commands/__init__.py b/dpo/commands/__init__.py
diff --git a/dpo/entities/DataPipelineExecutionEntity.py b/dpo/entities/DataPipelineExecutionEntity.py
@@ -2,8 +2,8 @@
 from sqlalchemy.sql import func
 from sqlalchemy.dialects.postgresql import UUID
 import uuid
-from mcd.Shared import Constants
-from mcd import Shared
+from dpo.Shared import Constants
+from dpo import Shared
 
 TABLE_NAME = 'data_pipeline_execution'
 PRIMARY_KEY_COL_NAME = 'id'
diff --git a/dpo/entities/ModelChecksumEntity.py b/dpo/entities/ModelChecksumEntity.py
@@ -1,9 +1,9 @@
 from sqlalchemy import Column, DateTime, Integer, String, ForeignKey
 from sqlalchemy.sql import func
 from sqlalchemy.dialects.postgresql import UUID
-from mcd import Shared
-from mcd.Shared import Constants
-from mcd.entities import DataPipelineExecutionEntity
+from dpo import Shared
+from dpo.Shared import Constants
+from dpo.entities import DataPipelineExecutionEntity
 
 TABLE_NAME = 'model_checksum'
 
diff --git a/dpo/entities/__init__.py b/dpo/entities/__init__.py
diff --git a/mcd/__main__.py b/mcd/__main__.py
diff --git a/setup.py b/setup.py
@@ -1,6 +1,6 @@
 from setuptools import setup, find_packages
 
-setup(name='mcd',
+setup(name='dpo',
       version='0.0.2',
       packages=find_packages(),
       install_requires=[
diff --git a/tests/integration/test_integration.sh b/tests/integration/test_integration.sh
@@ -4,7 +4,7 @@
 set -e
 
 # Bootstrap
-mcd="pipenv run python -m mcd postgresql+psycopg2://postgres:travisci@localhost:5432/postgres"
+dpo="pipenv run python -m dpo postgresql+psycopg2://postgres:travisci@localhost:5432/postgres"
 modelDirectory="./tests/integration/models"
 loadModelDirectory="$modelDirectory/load"
 transformModelDirectory="$modelDirectory/transform"
@@ -58,7 +58,7 @@ AssertAreEqual () {
 }
 
 InitExecution () {
-    local executionId=$($mcd init-execution)
+    local executionId=$($dpo init-execution)
 
     if [ ${#executionId} != 36 ]
     then
@@ -69,7 +69,7 @@ InitExecution () {
 }
 
 GetLastSuccessfulExecution () {
-    local executionId=$($mcd get-last-successful-execution)
+    local executionId=$($dpo get-last-successful-execution)
 
     if [ ${#executionId} != 36 ] && [ $executionId != 'NO_LAST_SUCCESSFUL_EXECUTION' ]
     then
@@ -81,7 +81,7 @@ GetLastSuccessfulExecution () {
 
 GetExecutionLastUpdatedTimestamp () {
     local executionId=$1
-    local executionLastUpdatedTimestamp=$($mcd get-execution-last-updated-timestamp $executionId)
+    local executionLastUpdatedTimestamp=$($dpo get-execution-last-updated-timestamp $executionId)
 
     if ! [[ $executionLastUpdatedTimestamp =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{6}[\+,-][0-9]{2}:[0-9]{2}$ ]]
     then
@@ -96,23 +96,23 @@ PersistModels () {
     local modelType=$2
     local basePath=$3
 
-    $mcd persist-models $executionId $modelType $basePath "**/*.json" "**/*.csv" "**/*.sql"
+    $dpo persist-models $executionId $modelType $basePath "**/*.json" "**/*.csv" "**/*.sql"
 }
 
 CompareModels () {
     local previousExecutionId=$1
     local currentExecutionId=$2
     local modelType=$3
 
-    local result=$($mcd compare-models $previousExecutionId $currentExecutionId $modelType)
+    local result=$($dpo compare-models $previousExecutionId $currentExecutionId $modelType)
 
     echo "$result"
 }
 
 CompleteExecution () {
     local executionId=$1
 
-    $mcd complete-execution $executionId
+    $dpo complete-execution $executionId
 
     local lastSuccessfulExecId=$(GetLastSuccessfulExecution)
     AssertAreEqual "ExecutionID $executionId completed successfully; next LastSuccessfulExecutionID" "$lastSuccessfulExecId" "$executionId"

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+from dpo.DataPipelineOrchestrator import DataPipelineOrchestrator`
	`2`	`+`
	`3`	`+DataPipelineOrchestrator()`