dbt-spark 1.3.0rc2

dbt-spark 1.3.0-rc2 - October 03, 2022

Features

• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Contributors

• @colin-rogers-dbt (#480)

dbt-spark 1.3.0rc1

dbt-spark 1.3.0-rc1 - September 28, 2022

Features

• merge exclude columns for spark models (#5260, #390)
• Array macros (#453, #454)
• implement testing for type_boolean in spark (#470, #471)
• Support job cluster in notebook submission method, remove requirement for user for python model submission (#444, #467)

Fixes

• python incremental model tmp table using correct schema (#441, #445)
• change to get_columns_in_relation to fix cache inconsistencies to fix cache issues in incremental models causing failure on on_schema_change (#447, #451)

Under the Hood

• Submit python model with Command API by default. Adjusted run name (#424, #442)
• Better interface for python submission (#452, #452)
• ignore mypy typing issues (#461, #462)
• Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (#468, #469)
• Convert df to pyspark DataFrame if it is koalas before writing (#473, #474)

Dependency

• Bump pyodbc from 4.0.32 to 4.0.34 (#417, #459)
• Bump black from 22.3.0 to 22.8.0 (#417, #458)
• Update click requirement from ~=8.0.4 to ~=8.1.3 (#417, #457)
• Bump mypy from 0.950 to 0.971 (#417, #456)
• Bump thrift-sasl from 0.4.1 to 0.4.3 (#417, #455)

Contributors

• @chamini2 (#469)
• @colin-rogers-dbt (#462)
• @dave-connors-3 (#390)
• @dbeatty10 (#454, #469, #474)
• @graciegoheen (#454)
• @jpmmcneill (#471)
• @ueshin (#474)

dbt-spark 1.3.0b2

dbt-spark 1.3.0-b2 - August 30, 2022

Features

• Add changie to dbt-spark (#416, #418)

Under the Hood

• specify supported_languages for materialization that support python models (#437, #440)

dbt-spark 1.3.0b1

dbt-spark 1.3.0b1 (July 29, 2022)

Features

• Support python model through notebook, currently supported materializations are table and incremental. (#377)

Fixes

• Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)

Under the hood

• Support core incremental refactor (#394)

Contributors

• @barberscott (#398)

dbt-spark 1.2.0

dbt-spark 1.2.0 (July 26, 2022)

Features

• Add grants to materializations (#366, #381)

Fixes

• Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)
• Incremental materialization updated to not drop table first if full refresh for delta lake format, as it already runs create or replace table (#286, #287)
• Apache Spark version upgraded to 3.1.1 (#348, #349)
• adapter.get_columns_in_relation (method) and get_columns_in_relation (macro) now return identical responses. The previous behavior of get_columns_in_relation (macro) is now represented by a new macro, get_columns_in_relation_raw (#354, #355)

Under the hood

• Update SparkColumn.numeric_type to return decimal instead of numeric, since SparkSQL exclusively supports the former (#380)
• Initialize lift + shift for cross-db macros (#359)
• Add invocation env to user agent string (#367)
• Use dispatch pattern for get_columns_in_relation_raw macro (#365)

Contributors

• @barberscott (#398)
• @grindheim (#287)
• @nssalian (#349)
• @ueshin (#365)
• @dbeatty10 (#359)

dbt-spark 1.2.0rc1

dbt-spark 1.2.0rc1 (July 12, 2022)

Fixes

• Incremental materialization updated to not drop table first if full refresh for delta lake format, as it already runs create or replace table (#286, #287)
• Apache Spark version upgraded to 3.1.1 (#348, #349)

Features

• Add grants to materializations (#366, #381)

Under the hood

• Update SparkColumn.numeric_type to return decimal instead of numeric, since SparkSQL exclusively supports the former (#380)

Contributors

• @grindheim (#287)
• @nssalian (#349)

dbt-spark 1.2.0b1

dbt-spark 1.2.0b1 (June 24, 2022)

Fixes

• adapter.get_columns_in_relation (method) and get_columns_in_relation (macro) now return identical responses. The previous behavior of get_columns_in_relation (macro) is now represented by a new macro, get_columns_in_relation_raw (#354, #355)

Under the hood

• Add DBT_INVOCATION_ENV environment variable to ODBC user agent string (#366)
• Initialize lift + shift for cross-db macros (#359)
• Add invocation env to user agent string (#367)
• Use dispatch pattern for get_columns_in_relation_raw macro (#365)

Contributors

• @ueshin (#365)
• @dbeatty10 (#359)

dbt-spark 1.1.0

dbt-spark 1.1.0 (April 28, 2022)

Features

• Add session connection method (#272, #279)
• Adds new integration test to check against new ability to allow unique_key to be a list. (#282), #291)

Fixes

• Closes the connection properly (#280, #285)

Under the hood

• get_response -> AdapterResponse (#265)
• Adding stale Actions workflow (#275)
• Update plugin author name (fishtown-analytics → dbt-labs) in ODBC user agent (#288)
• Configure insert_overwrite models to use parquet (#301)
• Use dbt.tests.adapter.basic in test suite (#298, #299)
• Make internal macros use macro dispatch to be overridable in child adapters (#319, #320)
• Override adapter method 'run_sql_for_tests' (#323, #324)
• when a table or view doesn't exist, 'adapter.get_columns_in_relation' will return empty list instead of fail ([#328]#328)

Contributors

• @amychen1776 (#288)
• @ueshin (#285), (#320)
• @JCZuurmond ( #279)

dbt-spark 1.0.1

dbt-spark 1.0.1 (April 19, 2022)

Fixes

• Closes the connection properly (#280, #285)
• Make internal macros use macro dispatch to be overridable in child adapters (#319, #320)

Under the hood

• Configure insert_overwrite models to use parquet (#301)

Contributors

• @ueshin (#285, #320)

dbt-spark 1.1.0rc1

dbt-spark 1.1.0rc1 (April 13 2022)

Features

• Add session connection method (#272, #279)

Under the hood

• Use dbt.tests.adapter.basic in test suite (#298, #299)
• Make internal macros use macro dispatch to be overridable in child adapters (#319, #320)
• Override adapter method 'run_sql_for_tests' (#323, #324)
• when a table or view doesn't exist, 'adapter.get_columns_in_relation' will return empty list instead of fail ([#328]#328)

Contributors

• @JCZuurmond ( #279)
• @ueshin (#320)