✨Destination Bigquery 1s1t: handle cursor change by edgao · Pull Request #28721 · airbytehq/airbyte

Edward Gao (edgao) · 2023-07-25T23:24:07Z

Handle this scenario:

Source declares a cursor old_cursor; we run a sync
Source changes its schema to add a new_cursor column, and changes its cursor to new_cursor; run a sync

During the second sync, all old records have a null new_cursor (because that field didn't exist previously). But we still want to be able to overwrite them in the second sync. Therefore, we need to treat NULL cursors as older than non-NULL cursors.

Update the base T+D tests to verify the schema change case.
Update bigquery sqlgenerator integration test to verify the NULL cursor handling
Update bigquery sqlgenerator to sort NULLS LAST

This is essentially happening with CDC sources right now.

github-actions · 2023-07-25T23:24:29Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Evan Tahler (evantahler) · 2023-07-25T23:59:09Z

                SELECT `_airbyte_raw_id` FROM (
                  SELECT `_airbyte_raw_id`, row_number() OVER (
-                    PARTITION BY ${pk_list} ORDER BY ${cursor_name} DESC, `_airbyte_extracted_at` DESC
+                    PARTITION BY ${pk_list} ORDER BY ${cursor_name} DESC NULLS LAST, `_airbyte_extracted_at` DESC


Snowflake can do this!

Redshift can do this!

Postgres can do this!

Clickhouse can do this!

MySQL can do this (in a very odd way)!

Databrikcs can do this!

Seems like this is a thing for enough databases to roll with!

In MySQL NULL values are considered lower than any non-NULL value

so we don't strictly need to do anything, order by <cursor> desc is inherently putting nulls last

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T00:07:55Z

destination-bigquery test report (commit `043b8f5ac9`) - ✅

⏲️ Total pipeline duration: 40mn40s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T00:30:09Z

destination-bigquery-denormalized test report (commit `043b8f5ac9`) - ✅

⏲️ Total pipeline duration: 22mn02s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T01:29:40Z

destination-snowflake test report (commit `043b8f5ac9`) - ✅

⏲️ Total pipeline duration: 59mn17s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T02:31:32Z

destination-snowflake test report (commit `dfd2b58423`) - ✅

⏲️ Total pipeline duration: 59mn47s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T02:54:23Z

destination-bigquery-denormalized test report (commit `dfd2b58423`) - ✅

⏲️ Total pipeline duration: 22mn37s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T03:32:49Z

destination-bigquery test report (commit `dfd2b58423`) - ✅

⏲️ Total pipeline duration: 38mn13s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-26T23:33:10Z

destination-bigquery test report (commit `face39e39e`) - ✅

⏲️ Total pipeline duration: 31mn50s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-27T00:17:05Z

destination-snowflake test report (commit `face39e39e`) - ✅

⏲️ Total pipeline duration: 43mn40s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

Octavia Squidington III (octavia-squidington-iii) · 2023-07-27T00:36:09Z

destination-bigquery-denormalized test report (commit `face39e39e`) - ✅

⏲️ Total pipeline duration: 18mn53s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

Edward Gao (edgao) added 2 commits July 25, 2023 16:12

handle new cursor column

3a855b0

sync2 is actualy weird, apparently

b397319

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/destination/bigquery labels Jul 25, 2023

logistics

043b8f5

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Jul 25, 2023

Edward Gao (edgao) marked this pull request as ready for review July 25, 2023 23:26

Edward Gao (edgao) requested a review from a team as a code owner July 25, 2023 23:26

Automated Commit - Format and Process Resources Changes

dfd2b58

Evan Tahler (evantahler) reviewed Jul 25, 2023

View reviewed changes

Evan Tahler (evantahler) requested a review from a team July 25, 2023 23:59

Evan Tahler (evantahler) approved these changes Jul 26, 2023

View reviewed changes

Merge branch 'master' into edgao/dest_bigquery_null_order

face39e

Edward Gao (edgao) changed the title ~~Destination Bigquery: handle cursor change~~ ✨Destination Bigquery: handle cursor change Jul 26, 2023

Edward Gao (edgao) changed the title ~~✨Destination Bigquery: handle cursor change~~ ✨Destination Bigquery 1s1t: handle cursor change Jul 26, 2023

Edward Gao (edgao) enabled auto-merge (squash) July 26, 2023 23:00

Edward Gao (edgao) added the checklist-action-run label Jul 26, 2023

Edward Gao (edgao) merged commit 9f6963c into master Jul 27, 2023

Edward Gao (edgao) deleted the edgao/dest_bigquery_null_order branch July 27, 2023 00:36

Edward Gao (edgao) mentioned this pull request Jul 27, 2023

Destination bigquery: fix new test case after updating test schema #28802

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨Destination Bigquery 1s1t: handle cursor change#28721

✨Destination Bigquery 1s1t: handle cursor change#28721
Edward Gao (edgao) merged 5 commits into
masterfrom
edgao/dest_bigquery_null_order

Edward Gao (edgao) commented Jul 25, 2023 •

edited

Loading

github-actions Bot commented Jul 25, 2023 •

edited by edgao

Loading

Evan Tahler (evantahler) Jul 25, 2023

Edward Gao (edgao) Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 27, 2023

Octavia Squidington III (octavia-squidington-iii) commented Jul 27, 2023

Labels

3 participants

Conversation

Edward Gao (edgao) commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions Bot commented Jul 25, 2023 • edited by edgao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Evan Tahler (evantahler) Jul 25, 2023

Choose a reason for hiding this comment

Edward Gao (edgao) Jul 26, 2023

Choose a reason for hiding this comment

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-bigquery test report (commit 043b8f5ac9) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-bigquery-denormalized test report (commit 043b8f5ac9) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-snowflake test report (commit 043b8f5ac9) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-snowflake test report (commit dfd2b58423) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-bigquery-denormalized test report (commit dfd2b58423) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-bigquery test report (commit dfd2b58423) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 26, 2023

destination-bigquery test report (commit face39e39e) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 27, 2023

destination-snowflake test report (commit face39e39e) - ✅

Octavia Squidington III (octavia-squidington-iii) commented Jul 27, 2023

destination-bigquery-denormalized test report (commit face39e39e) - ✅

Labels

3 participants

Edward Gao (edgao) commented Jul 25, 2023 •

edited

Loading

github-actions Bot commented Jul 25, 2023 •

edited by edgao

Loading

destination-bigquery test report (commit `043b8f5ac9`) - ✅

destination-bigquery-denormalized test report (commit `043b8f5ac9`) - ✅

destination-snowflake test report (commit `043b8f5ac9`) - ✅

destination-snowflake test report (commit `dfd2b58423`) - ✅

destination-bigquery-denormalized test report (commit `dfd2b58423`) - ✅

destination-bigquery test report (commit `dfd2b58423`) - ✅

destination-bigquery test report (commit `face39e39e`) - ✅

destination-snowflake test report (commit `face39e39e`) - ✅

destination-bigquery-denormalized test report (commit `face39e39e`) - ✅