Skip to content

🐛 Destination bigquery 1s1t: respect dataset location#28580

Merged
Edward Gao (edgao) merged 2 commits into
masterfrom
edgao/1s1t/bigquery/dataset_location
Jul 21, 2023
Merged

🐛 Destination bigquery 1s1t: respect dataset location#28580
Edward Gao (edgao) merged 2 commits into
masterfrom
edgao/1s1t/bigquery/dataset_location

Conversation

@edgao
Copy link
Copy Markdown
Contributor

@edgao Edward Gao (edgao) commented Jul 21, 2023 •
edited
Loading

as title. the test is super hacky because of a bigquery api quirk.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 21, 2023 •
edited by edgao
Loading

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.
@edgao Edward Gao (edgao) marked this pull request as ready for review July 21, 2023 21:53
@edgao Edward Gao (edgao) requested a review from a team as a code owner July 21, 2023 21:53
@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Jul 21, 2023
@edgao Edward Gao (edgao) changed the title set dataset location during creation Destination bigquery 1s1t: respect dataset location Jul 21, 2023
@edgao Edward Gao (edgao) changed the title Destination bigquery 1s1t: respect dataset location 🐛 Destination bigquery 1s1t: respect dataset location Jul 21, 2023
edgao
Edward Gao (edgao) commented Jul 21, 2023
View reviewed changes
Comment thread ...e/integrations/destination/bigquery/typing_deduping/BigQuerySqlGeneratorIntegrationTest.java

// Empirically, it sometimes takes Bigquery nearly 30 seconds to propagate the dataset's existence.
// Give ourselves 2 minutes just in case.
for (int i = 0; i < 120; i++) {
Copy link
Copy Markdown
Contributor Author

@edgao Edward Gao (edgao) Jul 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is dumb, but we're running it concurrently with other tests that take longer (e.g. most TypingDedupingTests testcases need to run two full syncs).
@edgao Edward Gao (edgao) enabled auto-merge (squash) July 21, 2023 22:13
@octavia-squidington-iii
Copy link
Copy Markdown
Collaborator

Octavia Squidington III (octavia-squidington-iii) commented Jul 21, 2023

destination-bigquery test report (commit 6e73d69925) - ✅

⏲️ Total pipeline duration: 37mn15s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test
@octavia-squidington-iii
Copy link
Copy Markdown
Collaborator

Octavia Squidington III (octavia-squidington-iii) commented Jul 21, 2023

destination-bigquery-denormalized test report (commit 6e73d69925) - ✅

⏲️ Total pipeline duration: 21mn51s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test
@edgao Edward Gao (edgao) merged commit adf8870 into master Jul 21, 2023
@edgao Edward Gao (edgao) deleted the edgao/1s1t/bigquery/dataset_location branch July 21, 2023 22:54
@octavia-squidington-iii
Copy link
Copy Markdown
Collaborator

Octavia Squidington III (octavia-squidington-iii) commented Jul 21, 2023

destination-bigquery-denormalized test report (commit 9b0c51ba05) - ✅

⏲️ Total pipeline duration: 17mn21s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test
@octavia-squidington-iii
Copy link
Copy Markdown
Collaborator

Octavia Squidington III (octavia-squidington-iii) commented Jul 21, 2023

destination-bigquery test report (commit 9b0c51ba05) - ✅

⏲️ Total pipeline duration: 26mn44s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test
Efim Matytsin (efimmatytsin) pushed a commit to scentbird/airbyte that referenced this pull request Jul 27, 2023
@edgao @efimmatytsin
🐛 Destination bigquery 1s1t: respect dataset location (airbytehq#28580)
f8571e2 
* set dataset location during creation

* logistics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation checklist-action-run connectors/destination/bigquery

3 participants

@edgao @octavia-squidington-iii @cynthiaxyin