Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ This Demo launches Bronze and Silver pipelines with following activities:
7. ```commandline
python demo/launch_dais_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```
- uc_catalog_name : Unity catalog name
- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.

![dais_demo.png](../docs/static/images/dais_demo.png)
Expand Down Expand Up @@ -86,7 +86,7 @@ This demo will launch auto generated tables(100s) inside single bronze and silve
7. ```commandline
python demo/launch_techsummit_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```
- uc_catalog_name : Unity catalog name
- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token

![tech_summit_demo.png](../docs/static/images/tech_summit_demo.png)
Expand Down Expand Up @@ -128,7 +128,7 @@ This demo will perform following tasks:
7. ```commandline
python demo/launch_af_cloudfiles_demo.py --uc_catalog_name=<<uc catalog name>> --source=cloudfiles --profile=<<DEFAULT>>
```
- uc_catalog_name : Unity Catalog name
- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token

![af_am_demo.png](../docs/static/images/af_am_demo.png)
Expand Down Expand Up @@ -178,7 +178,7 @@ This demo will perform following tasks:
- Create databricks secrets to store producer and consumer keys using the scope created in step 2

- Following are the mandatory arguments for running EventHubs demo
- uc_catalog_name : unity catalog name e.g. ravi_dlt_meta_uc
- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- eventhub_namespace: Eventhub namespace e.g. dltmeta
- eventhub_name : Primary Eventhubname e.g. dltmeta_demo
- eventhub_name_append_flow: Secondary eventhub name for appendflow feed e.g. dltmeta_demo_af
Expand Down Expand Up @@ -231,7 +231,8 @@ This demo will perform following tasks:
```commandline
python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```


- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.

a. Databricks Workspace URL:
Expand Down Expand Up @@ -296,6 +297,9 @@ This demo will perform following tasks:
```commandline
python demo/launch_acfs_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```
- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.

![acfs.png](../docs/static/images/acfs.png)

# Lakeflow Declarative Pipelines Sink Demo
Expand Down Expand Up @@ -350,6 +354,10 @@ This demo will perform following tasks:
```commandline
python demo/launch_dlt_sink_demo.py --uc_catalog_name=<<uc_catalog_name>> --source=kafka --kafka_source_topic=<<kafka source topic name>>>> --kafka_sink_topic=<<kafka sink topic name>> --kafka_source_servers_secrets_scope_name=<<kafka source servers secret name>> --kafka_source_servers_secrets_scope_key=<<kafka source server secret scope key name>> --kafka_sink_servers_secret_scope_name=<<kafka sink server secret scope key name>> --kafka_sink_servers_secret_scope_key=<<kafka sink servers secret scope key name>> --profile=<<DEFAULT>>
```

- uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.
- you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.

![dlt_demo_sink.png](../docs/static/images/dlt_demo_sink.png)
![dlt_delta_sink.png](../docs/static/images/dlt_delta_sink.png)
![dlt_kafka_sink.png](../docs/static/images/dlt_kafka_sink.png)
Expand Down Expand Up @@ -406,6 +414,8 @@ This demo will perform following steps:
```commandline
python demo/generate_dabs_resources.py --source=cloudfiles --uc_catalog_name=<your_catalog_name> --profile=<your_profile>
```
> Note: uc_catalog_name : UC catalog name. Can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_'). Must also not start with a digit.

> Note: If you don't specify `--profile`, you'll be prompted for your Databricks workspace URL and access token.

7. Deploy and run the DAB bundle:
Expand Down
6 changes: 6 additions & 0 deletions demo/generate_dabs_resources.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
f"{run_id}/demo-output.csv"
),
)
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)

return runner_conf

Expand Down
6 changes: 6 additions & 0 deletions demo/launch_acfs_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
env="demo"
)
runner_conf.uc_catalog_name = self.args['uc_catalog_name']
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)
return runner_conf

def launch_workflow(self, runner_conf: DLTMetaRunnerConf):
Expand Down
7 changes: 7 additions & 0 deletions demo/launch_af_cloudfiles_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,13 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
),
)

if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)

return runner_conf

def launch_workflow(self, runner_conf: DLTMetaRunnerConf):
Expand Down
6 changes: 6 additions & 0 deletions demo/launch_af_eventhub_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,12 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
eventhub_port=self.args["eventhub_port"]
)
runner_conf.uc_catalog_name = self.args['uc_catalog_name']
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)
runner_conf.runners_full_local_path = 'demo/notebooks/afam_eventhub_runners'
return runner_conf

Expand Down
6 changes: 6 additions & 0 deletions demo/launch_dais_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,12 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
)
if self.args['uc_catalog_name']:
runner_conf.uc_catalog_name = self.args['uc_catalog_name']
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)
runner_conf.uc_volume_name = f"{runner_conf.uc_catalog_name}_dais_demo_{run_id}"

return runner_conf
Expand Down
7 changes: 7 additions & 0 deletions demo/launch_dlt_sink_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
),
)

if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)

return runner_conf

def launch_workflow(self, runner_conf: DLTMetaRunnerConf):
Expand Down
6 changes: 6 additions & 0 deletions demo/launch_silver_fanout_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@ def init_runner_conf(self) -> DLTMetaRunnerConf:
env="demo"
)
runner_conf.uc_catalog_name = self.args['uc_catalog_name']
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)
runner_conf.uc_volume_name = f"{runner_conf.uc_catalog_name}_dlt_meta_fout_demo_{run_id}"
return runner_conf

Expand Down
7 changes: 7 additions & 0 deletions demo/launch_techsummit_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,14 @@ def init_runner_conf(self) -> TechsummitRunnerConf:
)
if self.args['uc_catalog_name']:
runner_conf.uc_catalog_name = self.args['uc_catalog_name']
if '-' in runner_conf.uc_catalog_name or runner_conf.uc_catalog_name[0].isdigit():
print(
"\nERROR: 'uc_catalog_name' can only contain ASCII letters ('a' - 'z', 'A' - 'Z'),"
" digits ('0' - '9'), and underbar ('_'). Must also not start with a digit. Exiting."
)
exit(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not put into base class instead of duplicating this validation in every demo class?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can. I checked the page https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-identifiers, we need then to implement a more robust check. Are we also supporting delimited identifiers?

runner_conf.uc_volume_name = f"{self.args['uc_catalog_name']}_volume_{run_id}"

return runner_conf

def create_bronze_silver_dlt(self, runner_conf: DLTMetaRunnerConf):
Expand Down
Loading