You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**This PR will address the following Issue/Feature:**
3
+
4
+
**This PR will result in the following new package version:**
5
+
<!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
6
+
7
+
**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
8
+
9
+
## PR Checklist
10
+
### Basic Validation
11
+
Please acknowledge that you have successfully performed the following commands locally:
12
+
-[ ] dbt compile
13
+
-[ ] dbt run –full-refresh
14
+
-[ ] dbt run
15
+
-[ ] dbt test
16
+
-[ ] dbt run –vars (if applicable)
17
+
18
+
Before marking this PR as "ready for review" the following have been applied:
19
+
-[ ] The appropriate issue has been linked and tagged
20
+
-[ ] You are assigned to the corresponding issue and this PR
21
+
-[ ] BuildKite integration tests are passing
22
+
23
+
### Detailed Validation
24
+
Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
25
+
-[ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
26
+
-[ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
27
+
-[ ] You have provided details below around the validation steps performed to gain confidence in these changes.
28
+
<!--- Provide the steps you took to validate your changes below. -->
29
+
30
+
### Standard Updates
31
+
Please acknowledge that your PR contains the following standard updates:
32
+
- Package versioning has been appropriately indexed in the following locations:
33
+
-[ ] indexed within dbt_project.yml
34
+
-[ ] indexed within integration_tests/dbt_project.yml
35
+
-[ ] CHANGELOG has individual entries for each respective change in this PR
36
+
<!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry. -->
37
+
-[ ] README updates have been applied (if applicable)
38
+
<!--- Remember to check the following README locations for common updates. →
39
+
<!--- Suggested install range (needed for breaking changes) →
40
+
<!--- Dependency matrix is appropriately updated (if applicable) →
41
+
<!--- New variable documentation (if applicable) -->
42
+
-[ ] DECISIONLOG updates have been updated (if applicable)
43
+
-[ ] Appropriate yml documentation has been added (if applicable)
44
+
45
+
### dbt Docs
46
+
Please acknowledge that after the above were all completed the below were applied to your branch:
47
+
-[ ] docs were regenerated (unless this PR does not include any code or yml updates)
48
+
49
+
### If you had to summarize this PR in an emoji, which would it be?
50
+
<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908) -->
<!--- Please tell us your name, title and company -->
1
+
**Please provide your name and company**
3
2
4
-
**What change(s) does this PR introduce?**
5
-
<!--- Describe what changes your PR introduces to the package and how to leverage this new feature. -->
3
+
**Link the issue/feature request which this PR is meant to address**
4
+
<!--- If an issue was not created, please create one first so we may discuss the PR prior to opening one. -->
6
5
7
-
**Did you update the CHANGELOG?**
6
+
**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.**
7
+
8
+
**How did you validate the changes introduced within this PR?**
9
+
10
+
**Which warehouse did you use to develop these changes?**
11
+
12
+
**Did you update the CHANGELOG?**
8
13
<!--- Please update the new package version’s CHANGELOG entry detailing the changes included in this PR. -->
9
14
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
10
15
-[ ] Yes
11
16
12
-
**Does this PR introduce a breaking change?**
13
-
<!--- Does this PR introduce changes that will cause current package users' jobs to fail or require a `--full-refresh`? -->
14
-
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
15
-
-[ ] Yes (please provide breaking change details below.)
16
-
-[ ] No (please provide an explanation as to how the change is non-breaking below.)
17
-
18
-
**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
17
+
**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
19
18
<!--- The dbt_project.yml and the integration_tests/dbt_project.yml files contain the version number. Be sure to upgrade it accordingly -->
20
19
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
21
20
-[ ] Yes
22
21
23
-
**Is this PR in response to a previously created Bug or Feature Request**
24
-
<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
25
-
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
26
-
-[ ] Yes, Issue/Feature [link bug/feature number here]
27
-
-[ ] No
28
-
29
-
**How did you test the PR changes?**
30
-
<!--- Proof of testing is required in order for the PR to be approved. -->
31
-
<!--- To check a box, remove the space and insert an x in the box (eg. [x] BuildKite). -->
32
-
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
33
-
-[ ] BuildKite <!--- BuildKite testing is only applicable to Fivetran employees. -->
34
-
-[ ] Local (please provide additional testing details below)
35
-
36
-
**Select which warehouse(s) were used to test the PR**
37
-
<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). -->
38
-
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
39
-
-[ ] BigQuery
40
-
-[ ] Redshift
41
-
-[ ] Snowflake
42
-
-[ ] Postgres
43
-
-[ ] Databricks
44
-
-[ ] Other (provide details below)
45
-
46
22
**Provide an emoji that best describes your current mood**
47
23
<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908) -->
48
24
:dancer:
49
25
50
26
**Feedback**
51
27
52
28
We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,14 @@
1
+
# dbt_reddit_ads_source v0.2.0
2
+
[PR #5](https://github.com/fivetran/dbt_reddit_ads_source/pull/5) includes the following updates:
3
+
## Feature update 🎉
4
+
- Unioning capability! This adds the ability to union source data from multiple reddit_ads connectors. Refer to the [Union Multiple Connectors README section](https://github.com/fivetran/dbt_reddit_ads_source/blob/main/README.md#union-multiple-connectors) for more details.
5
+
6
+
## Under the hood 🚘
7
+
- Updated tmp models to union source data using the `fivetran_utils.union_data` macro.
8
+
- To distinguish which source each field comes from, added `source_relation` column in each staging model and applied the `fivetran_utils.source_relation` macro.
9
+
- Updated tests to account for the new `source_relation` column.
10
+
- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job.
- Materializes [Reddit Ads staging tables](https://fivetran.github.io/dbt_reddit_ads_source/#!/overview/reddit_ads_source/models/?g_v=1&g_e=seeds) which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/reddit-ads#schemainformation). These staging tables clean, test, and prepare your reddit_ads data from [Fivetran's connector](https://fivetran.com/docs/applications/reddit-ads) for analysis by doing the following:
16
-
-Names columns for consistency across all packages and for easier analysis
17
-
-Adds freshness tests to source data
18
-
-Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
16
+
-Naming the columns for consistency across all packages and for easier analysis
17
+
-Adding freshness tests to source data
18
+
-Adding column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
19
19
- Generates a comprehensive data dictionary of your Reddit Ads data through the [dbt docs site](https://fivetran.github.io/dbt_reddit_ads_source/).
20
20
- These tables are designed to work simultaneously with our [Reddit Ads transformation package](https://github.com/fivetran/dbt_reddit_ads).
21
21
22
22
# 🎯 How do I use the dbt package?
23
23
## Step 1: Prerequisites
24
24
To use this dbt package, you must have the following:
25
-
- At least one Fivetran Reddit Ads connector syncing data into your destination.
26
-
- A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
25
+
- At least one Fivetran Reddit Ads connector syncing data into your destination
26
+
- A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination
27
27
28
28
### Databricks Dispatch Configuration
29
-
If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.
29
+
If you are using a Databricks destination with this package, you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils`, then the `dbt-labs/dbt_utils` packages, respectively.
30
30
```yml
31
31
dispatch:
32
32
- macro_namespace: dbt_utils
@@ -39,10 +39,10 @@ Include the following reddit_ads_source package version in your `packages.yml` f
39
39
```yaml
40
40
packages:
41
41
- package: fivetran/reddit_ads_source
42
-
version: [">=0.1.0", "<0.2.0"]
42
+
version: [">=0.2.0", "<0.3.0"]
43
43
```
44
44
## Step 3: Define database and schema variables
45
-
By default, this package runs using your destination and the `reddit_ads` schema. If this is not where your Reddit Ads data is (for example, if your reddit_ads schema is named `reddit_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
45
+
By default, this package runs using your destination and the `reddit_ads` schema. If this is not where your Reddit Ads data is (for example, if your `reddit_ads` schema is named `reddit_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
46
46
47
47
```yml
48
48
vars:
@@ -51,12 +51,22 @@ vars:
51
51
```
52
52
53
53
## (Optional) Step 4: Additional configurations
54
-
<details><summary>Expand for configurations</summary>
54
+
### Union multiple connectors
55
+
If you have multiple reddit_ads connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `reddit_ads_union_schemas` OR `reddit_ads_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
56
+
57
+
```yml
58
+
vars:
59
+
reddit_ads_union_schemas: ['reddit_ads_usa','reddit_ads_canada'] # use this if the data is in different schemas/datasets of the same database/project
60
+
reddit_ads_union_databases: ['reddit_ads_usa','reddit_ads_canada'] # use this if the data is in different databases/projects but uses the same schema name
61
+
```
62
+
Please be aware that the native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
63
+
64
+
To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
55
65
56
66
### Passing Through Additional Metrics
57
67
By default, this package will select `clicks`, `impressions`, and `spend` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the following configurations to your `dbt_project.yml` file. These variables allow the pass-through fields to be aliased (`alias`) if desired, but not required. Use the following format for declaring the respective pass-through variables:
58
68
59
-
> **Note** Ensure you exercised due diligence when adding metrics to these models. The metrics added by default (clicks, impressions, and cost) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example, metric averages, which may be inaccurately represented at the grain for reports created in this package. You want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
69
+
> **NOTE** Ensure you exercised due diligence when adding metrics to these models. The metrics added by default (clicks, impressions, and cost) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example, metric averages, which may be inaccurately represented at the grain for reports created in this package. You want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
## (Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Core™
97
105
<details><summary>Expand for more details</summary>
98
106
@@ -127,4 +135,4 @@ We highly encourage and welcome contributions to this package. Check out [this d
127
135
# 🏪 Are there any resources available?
128
136
- If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_reddit_ads_source/issues/new/choose) section to find the right avenue of support for you.
129
137
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
130
-
- Have questions or want to just say hi? Book a time during our office hours [on Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at [email protected].
138
+
- Have questions or want to just say hi? Book a time during our office hours [on Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at [email protected].
0 commit comments