• Design and implement a solution using Terraform and Terragrunt to capture live traffic from a source ALB and forward it to a fleet of collector instances for analysis.

• Zero Latency Impact: The inspection process must happen "out-of-band." It cannot add even 1ms of latency to the main production API.

• Aggressive Cost Optimization: As a non-revenue-generating support service, the inspection infrastructure must be highly cost-efficient.

• Dynamic Scale: The production Application Load Balancer (ALB) scales in and out frequently. The inspection solution must adapt to these changes automatically.

• The Security Team has issued a critical mandate: they must inspect 100% of inbound HTTP traffic for zero-day threats and anomalies using a tool called goreplay.

flowchart LR
  client[Client] --> alb[Public ALB]
  alb --> app_asg[Demo Service ASG]

  alb -. mirror .-> eni[ALB ENIs]
  eni -. |vxlan udp 4789| .-> nlb[NLB mirror target]
  nlb --> collector_asg[Collector ASG - Graviton Spot]
  collector_asg --> goreplay[goreplay]
  collector_asg --> healthd[healthd]

  artifacts[S3 Artifacts] --> goreplay
  artifacts --> healthd

  eventbridge[EventBridge]
  reconciler[ENI reconciler Lambda]
  eventbridge --> reconciler
  reconciler -. updates .-> eni

flowchart TD
  root[Terragrunt root.hcl] --> stacks[Terragrunt stacks]

  stacks --> bootstrap[bootstrap]
  stacks --> network[network]
  stacks --> artifacts[artifacts]
  stacks --> alb_stack[alb]
  stacks --> demo[demo-service]
  stacks --> shadow[shadow-traffic]

  bootstrap --> bootstrap_s3[S3 state bucket]
  bootstrap --> bootstrap_ddb[DynamoDB lock table]

  network --> network_vpc[VPC]
  network --> network_subnets[Public and private subnets]
  network --> network_routes[Route tables and IGW]
  network --> network_endpoints[S3 gateway endpoint]

  artifacts --> artifacts_bucket[S3 artifacts bucket]
  artifacts --> artifacts_versioning[S3 bucket versioning]
  artifacts --> artifacts_block[S3 public access block]

  alb_stack --> alb_lb[ALB]
  alb_stack --> alb_sg[ALB security group]
  alb_stack --> alb_tg[ALB target group]
  alb_stack --> alb_listener[ALB listener]

  demo --> demo_asg[Demo ASG]
  demo --> demo_lt[Launch template]
  demo --> demo_sg[Instance security group]
  demo --> demo_iam[IAM role and instance profile]

  shadow --> shadow_asg[Collector ASG]
  shadow --> shadow_lt[Launch template]
  shadow --> shadow_sg[Collector security group]
  shadow --> shadow_nlb[NLB]
  shadow --> shadow_tg[NLB target group]
  shadow --> shadow_listener[NLB listener]
  shadow --> shadow_mirror[Traffic mirror filter and sessions]
  shadow --> shadow_iam[IAM role and instance profile]

NB. There is a fundamental business requirement conflict. We require 100% traffic scanning, and we require traffic capture occur at the ALB, however AWS does not provide the capabilities to guarantee we will capture 100% of traffic via ALB mirroring. A potential solution to this conflict is to instead capture traffic from application instance ENIs behind the ALB, assuming this is the target for API traffic.

• Implementation is managed with Terragrunt and Terraform on AWS

• Mirrored traffic is being used for inline enforcement (collector data will eventually be used to take action)

• Mirror Source: traffic will be captured from the ALB ENI.

• Mirror Sink: traffic will be sent to a Network Load Balancer (NLB) target.

• Mirror filtering: The current implementation will employ naive filtering to mirror all API traffic, however filtering or sampling in the future will create an opportunity to manage and tune traffic to reduce work and thus cost. (Just document what is necessary for this change)

• NLB forwards the mirrored VXLAN traffic directly over UDP 4789, so collectors can bind to VXLAN without a GENEVE decapsulation step.

• Because of encapsulation overhead, our mirror traffic path needs to be capable of handling approximately 1.6 to 2x the source traffic.

• AWS traffic mirroring is best effort:

  • if the source ENI is busy, mirror packets drop first
  • if the NLB/GWLB or collector is saturated, packets are dropped

• Handling ALB Changes We don't need to implement this in V1, only document how it will be achieved.

  • ALB ENIs can change due to the following:

    • Scaling events
    • AZ enable/disable
    • Maintenance or failover
    • Internal rebalancing

  • How can we get ahead of some of these changes in state?

    • Let's start with Event-driven ALB ENI change handling to reattach the mirror to a new ALB. This is a fast method to capture ALB changes, however can potentially miss some.
    • NB. This strategy relies on an EventBridge to Lambda pipeline, and delays are inevitable. The EventBridge is not instantly updated with ENI changes before or when they happen, and lambdas are accompanied by cold startup latencies. There are additional strategies which can be emloyed to reduce these latencies, some of which come with additional complexity or financial cost.
    • Later, we can add a tight reconciliation loop to capture any event misses on top of that.
    • We can further reduce overall ALB churn by creating Load balancer Capacity Unit (LCU) reservations which allow reserving a static minimum capacity for load balancer. Doing this before anticipated traffic events should reduce ALB churn.
    • Mirror reserved ENIs so that we are already capturing their traffic should the ALB switch to them.

• Use flow-consistent hashing to distribute traffic to collectors, in order to maintain coherent streams on any given collector. Best practice shard key: 5-tuple flow hashing + stickiness src IP, dst IP, src port, dst port, protocol

• Collectors are implemented and scaled in an EC2 ASG. We will use spot instances to balance cost efficiency and service availability. It may not be implemented in V1, but we should document possible methods to observe and notify operators of spot instance constraints (such as potential upcoming shortages or price spikes). Also document methods to fallback outside of Spot instances to ensure availability.

• We'll scale the collector EC2 ASG using NetworkIn bytes/sec and PacketsDropped as control signals.

• Our collector is a faux process. We only need to worry about setting up the host with EC2 user-data that:

  • Securely installs a simple noop go script called "goreplay"
  • Sets up the system to accept VXLAN traffic on id 0. goreplay must demux from there.

  ip link add vxlan0 type vxlan \
  id 0 \
  dstport 4789 \
  local <collector-ip> \
  nolearning
  ip link set vxlan0 up
  

  • Reports itself as unhealthy if the goreplay script dies

• Collector instances don't need outbound access.

• Document where we can observe ongoing costs. In a future revision, we should also alert and potentially take action if costs increase beyond acceptable measures (whether that be a budget constraint, significant change over time, or other).

In V1, mirror_source_eni_ids is a manual input. This is a placeholder for a reconciler that keeps the list aligned with the ALB as it scales.

Proposed V2 approach:

• Event-driven path: EventBridge rule on ALB scale events -> Lambda that:

  • Calls DescribeLoadBalancers to identify the ALB.
  • Resolves current ENIs via DescribeNetworkInterfaces (filter by ALB name/ARN).
  • Diff/patches Traffic Mirror Sessions to match the current ENI set.

• Periodic reconciliation: a short-interval job (Lambda/Step Functions) that re-lists ENIs and re-applies sessions to catch missed events.

• Drift visibility: emit metrics on “expected ENIs vs mirrored ENIs” and alarm if coverage drops below 100%.

• Add functionality to the goreplay script to die randomly, in order to exercise the health check functionality.

When goreplay and healthd are fetched from S3, bootstrap pulls a sidecar .sha256 file and validates it before installation:

• Upload scripts generate a checksum file in sha256sum -c format (<hash> <filename>).
• User-data downloads <artifact>.sha256 and runs sha256sum -c to verify integrity.
• If validation fails, bootstrap exits and the instance will not become healthy.