Coder Social home page Coder Social logo

fivetran / dbt_apple_store_source Goto Github PK

View Code? Open in Web Editor NEW
2.0 38.0 2.0 1.68 MB

Fivetran's Apple App Store source dbt package

Home Page: https://fivetran.github.io/dbt_apple_store_source/

License: Apache License 2.0

Shell 100.00%
apple-app-store itunes-connect dbt dbt-packages fivetran

dbt_apple_store_source's People

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbt_apple_store_source's Issues

[Bug] `seeds/apple_store_country_codes.csv`(ISO-3166 country codes) needs a refresh.

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

Would it be possible to refresh the apple_store_country_codes to reflect the most recent version of ISO-3166 country codes?

The staleness seems to affect downstream data models such as [apple_store__territory_report](https://github.com/fivetran/dbt_apple_store/blob/9735693405f64101beae5ef8129681363ca8b8b4/models/apple_store__territory_report.sql), with territory_short columns being NULL for certain countries such as the following:
image

I would be happy to update the country codes, if you can provide me with the source for the data ๐Ÿ˜.

Relevant error log or model output

No response

Expected behavior

Preferrably, the territory_short column of the model [apple_store__territory_report](https://github.com/fivetran/dbt_apple_store/blob/9735693405f64101beae5ef8129681363ca8b8b4/models/apple_store__territory_report.sql) should reflect the latest ISO-3166 country codes.

dbt Project configurations

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: "droom_analytics_bq"
version: "1.0.0"
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: "droom_analytics_bq"

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
docs-paths: ["docs"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"

# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  apple_store:
    +schema: itunes_connect # leave blank for just the target_schema
  apple_store_source:
    +schema: itunes_connect # leave blank for just the target_schema
  google_play:
    +schema: google_play # leave blank for just the target_schema
  google_play_source:
    +schema: google_play # leave blank for just the target_schema

    # Config indicated by + and applies to all files under models/example/
    #

vars:
  # enable subscriptions and earnings models (https://github.com/fivetran/dbt_app_reporting#step-4-disable-and-enable-source-tables)
  apple_store__using_subscriptions: true # by default this is assumed to be false
  google_play__using_subscriptions: true # by default this is assumed to be false
  google_play__using_earnings: true # by default this is assumed to be false

Package versions

packages:
  - package: fivetran/apple_store
    version: 0.2.0
  - package: fivetran/google_play
    version: 0.2.0

What database are you using dbt with?

bigquery

dbt Version

Core:
  - installed: 1.4.4
  - latest:    1.4.5 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 1.4.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Feature] Update collect_freshness config in src_apple_store.yml to account for dbt macro logic

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Our fivetran_utils collect_freshness macro was designed to override dbt's equivalent macro to allow for the enabling/disabling of variables when running dbt source snapshot-freshness. Thus if a source table does not exist, dbt will not run (and error on) a freshness test on the table.

dbt has since updated its macro in recent versions to basically provide the same functionality. So warnings are being thrown to upgrade the package logic.

We will want to modify all versions of the meta: is_enabled config in our src_apple_store.yml package ([example here])(https://github.com/fivetran/dbt_apple_store_source/blob/main/models/src_apple_store.yml#L188-L189) to match the config: enabled dbt logic. Once those dependencies are either updated or removed, we can further explore updates to our own internal collect_freshness macro.

Describe alternatives you've considered

At the moment, we will be proceeding forward with updating dbt_fivetran_utils to match the logic of dbt's collect_freshness macro. That way customers will no longer experience that warning.

More details can be found in this ticket.

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

Customers raising this issue can be found in this dbt slack thread https://getdbt.slack.com/archives/C03SAHKKG2Z/p1683127164993329

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.