Add scheduler components and installation instructions #1503
Conversation
| @@ -0,0 +1,24 @@ | |||
| # Installing the Usage Collection Scheduler | |||
|
|
|||
There was a problem hiding this comment.
moving the markdown inside
/docs/remorph/docs/assessment folder will help for auto document generation
There was a problem hiding this comment.
Great idea 👍
| """A scheduler that executes Pipeline steps according to predefined schedules.""" | ||
|
|
||
| def __init__(self, pipelines: list[PipelineClass], | ||
| db_path: str = "pipeline_steps.duckdb", |
There was a problem hiding this comment.
instead of the path, can we pass a connection object?
There was a problem hiding this comment.
or use a singleton for duckdb connection management and statemanagement.
There was a problem hiding this comment.
Added a class, DuckDBManager to manage the connection to the DuckDB database. But perhaps we should move this to the connections module instead and even inherit from the DatabaseManager parent class. While it isn't necessarily a source database/data warehouse all of the abstract methods make sense. What do you think?
|
|
||
| from sqlalchemy import create_engine | ||
| from sqlalchemy.engine import Engine, Result, URL | ||
| from snowflake.sqlalchemy import URL as SnowflakeURL |
There was a problem hiding this comment.
I always go back and forth on this, using Snowflake Connector Lib vs. using vanilla pyodbc with Alchemy.
lesser library management and other advantages.
There was a problem hiding this comment.
Should I pull this out of this PR for now? I only added it since I was adding a Postgres connector anyway.
There was a problem hiding this comment.
Leave it as is, we will add it in Notice in a separate PR.
sundarshankar89
changed the title
|
@goodwillpunning can you recreate the PR direclty not from a fork, so our CI runs. |
|
@goodwillpunning can we close this one now? |
3 participants
This commit adds a new scheduler component for executing a usage collection against a target Enterprise Data Warehouse (EDW).
The scheduler module contains:
usage_collector.py) for instantiating thePipelineClassand running usage collection queries