Skip to content

F/aws kinesis firehose delivery stream support database source #44227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 8 commits into from
Closed

F/aws kinesis firehose delivery stream support database source #44227

wants to merge 8 commits into from

Conversation

psantus
Copy link
Contributor

@psantus psantus commented Sep 10, 2025 •
edited
Loading

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the library.

Changes to Security Controls

No changes to security controls. This PR adds support for configuring database source authentication via AWS Secrets Manager, which follows existing security patterns in the provider.

Description

Adds support for database_source_configuration to the aws_kinesis_firehose_delivery_stream resource and data source. This enables Firehose to replicate data from MySQL and PostgreSQL databases.

Key features:

  • Database connection configuration (type, endpoint, port, SSL mode)
  • Database, table, and column filtering with include/exclude patterns
  • Authentication via AWS Secrets Manager
  • VPC configuration for private database connectivity
  • Snapshot watermark table configuration
  • Surrogate keys support

Also fixes missing warehouse_location field in iceberg_configuration per AWS API requirements.

Relations

Closes #41577

References

Output from Acceptance Testing

TF_ACC=1 go test ./internal/service/firehose/ -run TestAccFirehoseDeliveryStream_databaseSourceConfiguration -v -timeout 10m          

2025/09/10 22:15:42 Creating Terraform AWS Provider (SDKv2-style)...
2025/09/10 22:15:42 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccFirehoseDeliveryStream_databaseSourceConfiguration
=== PAUSE TestAccFirehoseDeliveryStream_databaseSourceConfiguration
=== CONT  TestAccFirehoseDeliveryStream_databaseSourceConfiguration
--- PASS: TestAccFirehoseDeliveryStream_databaseSourceConfiguration (9.00s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/firehose   13.600s

TF_ACC=1 go test ./internal/service/firehose/ -run TestAccFirehoseDeliveryStreamDataSource_databaseSourceConfiguration -v -timeout 10m                 

2025/09/10 22:17:08 Creating Terraform AWS Provider (SDKv2-style)...
2025/09/10 22:17:08 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccFirehoseDeliveryStreamDataSource_databaseSourceConfiguration
=== PAUSE TestAccFirehoseDeliveryStreamDataSource_databaseSourceConfiguration
=== CONT  TestAccFirehoseDeliveryStreamDataSource_databaseSourceConfiguration
--- PASS: TestAccFirehoseDeliveryStreamDataSource_databaseSourceConfiguration (8.69s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/firehose   13.396s

 ⋮
@psantus psantus requested a review from a team as a code owner September 10, 2025 18:52
@github-actions GitHub Actions
Copy link
Contributor

Community Guidelines

This comment is added to every new Pull Request to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

  • Please vote on this Pull Request by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
  • Please see our prioritization guide for additional information on how the maintainers handle prioritization.
  • Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Pull Request and do not help prioritize the request.

Pull Request Authors

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.
@github-actions github-actions bot added needs-triage Waiting for first response or review from a maintainer. documentation Introduces or discusses updates to documentation. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/firehose Issues and PRs that pertain to the firehose service. service/glue Issues and PRs that pertain to the glue service. size/XL Managed by automation to categorize the size of a PR. labels Sep 10, 2025
@psantus psantus force-pushed the f/aws_kinesis_firehose_delivery_stream-support-database-source branch 5 times, most recently from 6143718 to 0fda4ab Compare September 10, 2025 21:01
@psantus
Add database source configuration support to firehose delivery stream
1ccd246 
- Add database_source_configuration block with MySQL/PostgreSQL support
- Include authentication via Secrets Manager
- Add VPC configuration for database connectivity
- Support database, table, and column filtering
- Add warehouse_location field to iceberg configuration
- Extend data source to read database source config

Fixes hashicorp#41577
@psantus psantus force-pushed the f/aws_kinesis_firehose_delivery_stream-support-database-source branch from 0fda4ab to 1ccd246 Compare September 10, 2025 21:36
@psantus psantus force-pushed the f/aws_kinesis_firehose_delivery_stream-support-database-source branch from e51a8d2 to 322e0d3 Compare September 10, 2025 22:16
@ewbankkit ewbankkit added enhancement Requests to existing resources that expand the functionality or scope. and removed service/glue Issues and PRs that pertain to the glue service. needs-triage Waiting for first response or review from a maintainer. labels Sep 11, 2025
@ewbankkit ewbankkit self-assigned this Sep 11, 2025
@github-actions github-actions bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Sep 11, 2025
ewbankkit
ewbankkit previously approved these changes Sep 11, 2025
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccFirehoseDeliveryStream_basic\|TestAccFirehoseDeliveryStream_icebergUpdates' PKG=firehose
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
make: Running acceptance tests on branch: 🌿 HEAD 🌿...
TF_ACC=1 go1.24.6 test ./internal/service/firehose/... -v -count 1 -parallel 20  -run=TestAccFirehoseDeliveryStream_basic\|TestAccFirehoseDeliveryStream_icebergUpdates -timeout 360m -vet=off
2025/09/11 14:10:04 Creating Terraform AWS Provider (SDKv2-style)...
2025/09/11 14:10:04 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccFirehoseDeliveryStream_basic
=== PAUSE TestAccFirehoseDeliveryStream_basic
=== RUN   TestAccFirehoseDeliveryStream_icebergUpdates
=== PAUSE TestAccFirehoseDeliveryStream_icebergUpdates
=== CONT  TestAccFirehoseDeliveryStream_basic
=== CONT  TestAccFirehoseDeliveryStream_icebergUpdates
--- PASS: TestAccFirehoseDeliveryStream_basic (104.08s)
--- PASS: TestAccFirehoseDeliveryStream_icebergUpdates (162.10s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/firehose	167.624s
% make testacc TESTARGS='-run=TestAccFirehoseDeliveryStreamDataSource_basic' PKG=firehose       
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
make: Running acceptance tests on branch: 🌿 HEAD 🌿...
TF_ACC=1 go1.24.6 test ./internal/service/firehose/... -v -count 1 -parallel 20  -run=TestAccFirehoseDeliveryStreamDataSource_basic -timeout 360m -vet=off
2025/09/11 14:16:11 Creating Terraform AWS Provider (SDKv2-style)...
2025/09/11 14:16:11 Initializing Terraform AWS Provider (SDKv2-style)...
=== RUN   TestAccFirehoseDeliveryStreamDataSource_basic
=== PAUSE TestAccFirehoseDeliveryStreamDataSource_basic
=== CONT  TestAccFirehoseDeliveryStreamDataSource_basic
--- PASS: TestAccFirehoseDeliveryStreamDataSource_basic (80.84s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/firehose	86.293s
@psantus
Copy link
Contributor Author

psantus commented Sep 11, 2025 •
edited
Loading

@ewbankkit thanks for reviewing so quickly I appreciate.
You didn't like the data source part? Don't think it's any useful to get the source configuration from the firehose?
[EDIT - Answered on Slack]removed the additions to the corresponding data source and it had non of the other source or destination configurations.[/EDIT]
@psantus
Copy link
Contributor Author

psantus commented Sep 25, 2025

Put on hold, it seems the service feature won't go from Preview to GA 😭 😞
@ewbankkit ewbankkit removed their assignment Sep 25, 2025
@tabito-hara tabito-hara mentioned this pull request Sep 28, 2025
Open
@psantus
Copy link
Contributor Author

psantus commented Sep 29, 2025 •
edited
Loading

Won't be merged, AWS has confirmed they're retiring this tomorrow
@psantus psantus closed this Sep 29, 2025
@github-actions GitHub Actions
Copy link
Contributor

Warning

This Issue has been closed, meaning that any additional comments are much easier for the maintainers to miss. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. service/firehose Issues and PRs that pertain to the firehose service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
2 participants
@psantus @ewbankkit