Skip to content

[Bug] Incremental runs on daily_issue_field_history bringing in fewer records than a --full-refresh #155

@fivetran-avinash

Description

@fivetran-avinash

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

When testing incremental behavior on jira__daily_issue_field_history on our internal data in deploying the v1.1.0 release by doing a full-refresh on a filtered of date/incremental run with filtered date removed, we noticed that there was a drop in 10k+ records compared to doing a standard full-refresh.

Relevant error log or model output

Expected behavior

We would expect there to be a near equivalent amount of records in both models.

Possible solution

We would have to guess the incremental logic might need to be reworked to find the missing records.

dbt Project configurations

Attempted with these vars:

  jira_using_components: false
  issue_field_history_columns: ['team', 'assignee', 'labels', 'collaborators', 'quarter',  'project']
  jira_using_teams: true

Package versions

tested on both the existing v1.0.0 main release and the v1.1.0 release branch (to ensure this wasn't introducing the issue), where both 10k+ fewer records are brought in via an incremental run.

What database are you using dbt with?

bigquery

How are you running this dbt package?

dbt Core™

dbt Version

1.10.11

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance.
  • No.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugSomething is broken or incorrect

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions