fix(clickpipes): document that SQS queues / Pub/Sub subscriptions must not be shared across pipes#6519
Open
whites11 wants to merge 1 commit into
Open
fix(clickpipes): document that SQS queues / Pub/Sub subscriptions must not be shared across pipes#6519whites11 wants to merge 1 commit into
whites11 wants to merge 1 commit into
Conversation
SQS queues and Pub/Sub subscriptions deliver each message to a single consumer, so sharing one across multiple ClickPipes causes silent, random file loss: a pipe that receives an event for a file outside its path pattern discards it and no other pipe ever sees it. Add explicit warnings to both the S3 and GCS unordered-mode guides. Also make the direct S3->SQS instructions explicit about selecting "All object create events" (s3:ObjectCreated:*), since subsets like Put-only silently miss multipart uploads. Prompted by support-escalation #8195. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
3 Skipped Deployments
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prompted by support-escalation #8195: a customer pointed two S3 ClickPipes (different path patterns) at the same SQS queue. SQS delivers each message to exactly one consumer, so each
ObjectCreatedevent was received by only one of the two pipes — and a pipe receiving an event for a file outside its path pattern silently discards (and deletes) the message. Result: random, silent file loss on both pipes, with no error signal anywhere. The same failure mode exists for GCS pipes sharing a single Pub/Sub subscription.Changes
S3 unordered-mode guide:
:::warningin the Create an Amazon SQS queue step: one dedicated queue per ClickPipe; sharing a queue across pipes (or any other consumer) loses files, even with different path patterns; use EventBridge fan-out (one rule per queue) to feed multiple pipes from the same bucket.s3:ObjectCreated:*): selecting individual event types (e.g. onlyPut) silently misses files uploaded via other methods such as multipart uploads (CompleteMultipartUpload).GCS unordered-mode guide:
:::warningfor Pub/Sub: one dedicated subscription per ClickPipe; fan out with one subscription per pipe on the same topic.Spell dictionary: add
multipart.🤖 Generated with Claude Code