Skip to content

Commit b4e8d3a

Browse files
ohriteohrite
authored
Analyst sees incremental NTD API retrievals (#4327)
* Run tests when airflow files change * Analyst sees incremental NTD API retrievals Signed-off-by: Erika Pacheco <[email protected]> --------- Signed-off-by: Erika Pacheco <[email protected]>
1 parent 90a341e commit b4e8d3a

File tree

51 files changed

+21535
-1015
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+21535
-1015
lines changed

‎.github/workflows/test-airflow.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,12 @@ on:
66
- main
77
paths:
88
- .github/workflows/test-airflow.yml
9+
- 'airflow/**'
910
- 'warehouse/**'
1011
pull_request:
1112
paths:
1213
- .github/workflows/test-airflow.yml
14+
- 'airflow/**'
1315
- 'warehouse/**'
1416

1517
env:

‎airflow/dags/sync_ntd_data_api/annual_reporting/breakdowns.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: breakdowns
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/amkt-4ehs.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/breakdowns/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: amkt-4ehs

‎airflow/dags/sync_ntd_data_api/annual_reporting/breakdowns_by_agency.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: breakdowns_by_agency
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/fk8n-qvag.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/breakdowns_by_agency/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: fk8n-qvag

‎airflow/dags/sync_ntd_data_api/annual_reporting/capital_expenses_by_capital_use.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: capital_expenses_by_capital_use
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/fphd-jyyj.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/capital_expenses_by_capital_use/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: fphd-jyyj

‎airflow/dags/sync_ntd_data_api/annual_reporting/capital_expenses_by_mode.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: capital_expenses_by_mode
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/2667-vitc.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/capital_expenses_by_mode/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: 2667-vitc

‎airflow/dags/sync_ntd_data_api/annual_reporting/capital_expenses_for_existing_service.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: capital_expenses_for_existing_service
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/7kqv-yqbn.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/capital_expenses_for_existing_service/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: 7kqv-yqbn

‎airflow/dags/sync_ntd_data_api/annual_reporting/capital_expenses_for_expansion_of_service.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: capital_expenses_for_expansion_of_service
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/nvgd-g6pj.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/capital_expenses_for_expansion_of_service/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: nvgd-g6pj

‎airflow/dags/sync_ntd_data_api/annual_reporting/employees_by_agency.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: employees_by_agency
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/brbd-9azc.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/employees_by_agency/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: brbd-9azc

‎airflow/dags/sync_ntd_data_api/annual_reporting/employees_by_mode.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: employees_by_mode
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/wsxw-2rpq.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/employees_by_mode/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: wsxw-2rpq

‎airflow/dags/sync_ntd_data_api/annual_reporting/employees_by_mode_and_employee_type.yml‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
1-
operator: operators.NTDToGCSOperator
1+
operator: operators.soda_to_gcs_operator.SODAToGCSOperator
22

3-
year: multi_year
4-
product: employees_by_mode_and_employee_type
5-
bucket: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
6-
endpoint: resource/uyv8-9jek.json
7-
parameters:
8-
$limit: 5000000
3+
object_path: multi_year/employees_by_mode_and_employee_type/dt={{ dag_run.logical_date.date().isoformat() }}/execution_ts={{ dag_run.logical_date.isoformat() }}
4+
bucket_name: "{{ env_var('CALITP_BUCKET__NTD_API_DATA_PRODUCTS') }}"
5+
resource: uyv8-9jek

0 commit comments

Comments
 (0)