Dataflow Templates 2024-09-10-00_RC01

Release Week of 9/10/24

Improvements

• Add support for OAuth authentication to schema registry in the Kafka to Big Query Flex and Kafka to GCS flex templates. (#1816)
• feat: Adds SourceDBToSpanner PostgreSQL Adapter (#1754)

Bug Fixes

• BigTable to Parquet: only set numShards for >0 (#1820)
• Fix for custom logical types in DatastreamIO (#1767)

Contributors

Astha Mohta
Damon
Deep1998
Dylan Hercher
Jeff Kinard
Ron Gal
Shuran Zhang
Thiago Nunes
Vardhan Vinay Thigle
Yi Hu
aksharauke
atask-g
darshan-sj
fozzie15
shreyakhajanchi
yair-harel

Dataflow Templates 2024-09-10-00_RC00

Release week of 2024-09-09

Dataflow Templates 2024-09-03-00_RC00

Release week of 2024-09-02

Dataflow Templates 2024-08-30-00_RC00

Release week of 2024-08-26

Dataflow Templates 2024-08-28-03_RC00

Release week of 2024-08-26

Dataflow Templates 2024-08-28-02_RC00

Release week of 2024-08-26

Dataflow Templates 2024-08-27-00_RC00

Release week of 2024-08-26

Dataflow Templates 2024-08-20-00_RC00

Release Week of 2024-08-20

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Add support for the PLACEMENT KEY column constraint in Spanner import/export templates (#1792)
• Bump hadoop version (#1791)
• DLQ implementation (#1799)
• Feat: Deploy Spanner Migration Tool for PostgreSQL with Terraform (#1788)
• Finetuning DbCp2 Parameters for Bulk Reader. (#1786)
• KafkaToBigQueryFlex Template: Expose source message offset in DLQ metadata (for Json format only)
• Move UDF usage docs back to main README (#1790)
• Support PLACEMENT schema objects in Spanner's import/export templates (#1779)
• Support Timestamp in proto to bigquery schema conversion (#1789)
• TF: Update permissions and check if IAM API is enabled (#1802)

Contributors

• Aditya
• DemoYeti
• Jeff Kinard
• Manit Gupta
• Prayas Purusottam
• Vardhan Vinay Thigle
• Yi Hu
• aksharauke
• fozzie15
• sgorse123
• wazi55

Dataflow Templates 2024-08-13-00_RC00

Release week of 2024-08-12

Dataflow Templates 2024-08-06-00_RC00

Release Week of 2024-08-06

Improvements

• Add App Profile ID to CBT Dataflow Templates: Bigtable to Avro/Parquet/JSON (#1762)
• Add string mapper case for IT (#1757)
• Add support for ALO in SpannerChangeStreamsToBigQuery template (#1750)
• Adding autoReconnect parameters to URL (#1760)
• DatastreamToSQL - log schema map details (#1747)
• Handling for String Primary Keys in Bulk Reader. (#1743)
• Spanner to Source basic impl (#1707)
• Terraform template updates for custom transformation (#1746)
• [BULK] Convert MySQL Time to unified TIME_INTERVAL types (#1741)
• docs: adding file for load testing data generation (#1733)
• fix CdcToBigQueryChangeApplierPipelineIT debezium wait logic (#1729)
• pipeline parameter updates (#1740)

Bug Fixes

• Fixing Index Discovery for 5.7 and removing innodb_parallel_read_threads from init sequences for MySql5.7 compliance. (#1758)
• Fixing exception in String .isSplittable (#1755)

Contributors

Aditya
Astha Mohta
Deep1998
Derek Yau
Dip Patel
Dylan Hercher
Jeff Kinard
Vardhan Vinay Thigle
aksharauke
shreyakhajanchi