Mastering Data Compaction in Apache Iceberg: A Production Deep Dive

Introduction

The relentless growth of data in modern applications presents a significant engineering challenge: maintaining query performance on ever-increasing datasets. We recently faced this at scale while building a real-time analytics platform for a large e-commerce company. Ingesting clickstream data at 100K events/second, coupled with historical data totaling over 500TB, resulted in query latencies exceeding acceptable thresholds for dashboarding and reporting. The root cause wasn’t compute or storage capacity, but the proliferation of small files and inefficient data layout within our data lake. This led us to deeply investigate and optimize data compaction strategies within Apache Iceberg, a table format designed for large analytic datasets. This post details our journey, focusing on architectural considerations, performance tuning, and operational best practices for Iceberg compaction. We’ll cover how it fits into a modern data lakehouse architecture leveraging Spark, Flink, and cloud storage (AWS S3).

What is Data Compaction in Big Data Systems?

Data compaction, in the context of Iceberg, is the process of rewriting smaller data files into larger, more efficient files. Iceberg’s architecture relies on metadata layers (manifest lists, manifest files, data files) to track data. Frequent small writes, common in streaming or micro-batch scenarios, create a large number of data files. This leads to:

• Metadata Overhead: Listing and processing a vast number of metadata files becomes a bottleneck for query planning.
• I/O Inefficiency: Reading numerous small files introduces significant seek overhead, drastically reducing query performance.
• Storage Costs: A large number of small files can increase storage costs due to metadata storage and potential inefficiencies in cloud storage billing.

Iceberg’s compaction process leverages its snapshot isolation and metadata management to rewrite data files without disrupting concurrent reads and writes. It operates at the file level, rewriting data in formats like Parquet or ORC, and updating the metadata to reflect the new file locations. The protocol-level behavior involves reading source files, applying any necessary transformations (e.g., partitioning), writing new files, and atomically updating the table metadata.

Real-World Use Cases

1. Clickstream Analytics: High-velocity clickstream data ingested via Kafka requires frequent compaction to maintain query performance on user behavior dashboards.
2. IoT Sensor Data: Similar to clickstream, IoT data streams necessitate continuous compaction to handle the constant influx of new data.
3. Change Data Capture (CDC) Pipelines: CDC pipelines often write small batches of changes. Compaction consolidates these changes into larger, queryable files.
4. Log Analytics: Aggregating and analyzing large volumes of log data requires efficient file sizes for fast querying.
5. Machine Learning Feature Stores: Frequent updates to feature values necessitate compaction to ensure feature pipelines operate on optimized data.

System Design & Architecture

graph LR
    A[Kafka] --> B(Flink CDC);
    B --> C{Iceberg Table (S3)};
    C --> D[Spark Batch Compaction];
    C --> E[Iceberg Compactor (Scheduled)];
    D --> C;
    E --> C;
    F[Presto/Trino] --> C;
    G[Dashboarding/Reporting] --> F;
    style C fill:#f9f,stroke:#333,stroke-width:2px

This diagram illustrates a typical architecture. Flink consumes CDC events from Kafka and writes them to an Iceberg table stored in S3. Two compaction mechanisms are employed: a scheduled Iceberg compactor (for continuous, smaller compaction) and a Spark batch compaction job (for larger, more aggressive compaction). Presto/Trino queries the Iceberg table, and the results are consumed by dashboards and reporting tools. The Iceberg table itself manages the metadata and data file organization.

We leverage AWS EMR for our Spark and Flink clusters, utilizing S3 as the underlying storage layer. The Iceberg metadata is stored in AWS Glue as a Hive Metastore catalog.

Performance Tuning & Resource Management

Effective compaction requires careful tuning. Key parameters include:

• iceberg.compactor.target-file-size: The desired size of compacted files (e.g., 128MB, 256MB). Larger files generally improve read performance but increase compaction latency.
• iceberg.compactor.max-files-per-compaction: Limits the number of files rewritten in a single compaction task. Helps control resource consumption.
• spark.sql.shuffle.partitions: Crucial for Spark compaction jobs. Increasing this value (e.g., to 2000) can improve parallelism, but excessive values can lead to small tasks and increased overhead.
• fs.s3a.connection.maximum: Controls the number of concurrent connections to S3. Increasing this value (e.g., to 1000) can improve I/O throughput.
• iceberg.compactor.concurrency: Number of concurrent compaction tasks.

We found that a target file size of 256MB, combined with a spark.sql.shuffle.partitions value of 1500 and fs.s3a.connection.maximum of 500, provided the best balance between compaction latency and query performance. Monitoring compaction duration and query latency is critical for iterative tuning.

Failure Modes & Debugging

Common failure modes include:

• Data Skew: Uneven data distribution can lead to some compaction tasks taking significantly longer than others. Monitor task durations in the Spark UI. Consider salting or bucketing to mitigate skew.
• Out-of-Memory Errors: Compaction tasks can consume significant memory, especially when dealing with wide tables or complex transformations. Increase executor memory or reduce the number of files processed per task.
• Job Retries: Transient network errors or S3 throttling can cause compaction jobs to fail and retry. Implement exponential backoff and retry mechanisms.
• DAG Crashes: Errors in the compaction logic or metadata corruption can lead to DAG crashes. Examine the Spark logs for detailed error messages.

The Spark UI and Flink dashboard are invaluable for debugging. Datadog alerts based on compaction duration, task failure rates, and query latency provide proactive monitoring. Iceberg’s metadata consistency guarantees allow for safe retries and recovery from failures.

Data Governance & Schema Management

Iceberg’s schema evolution capabilities are crucial for maintaining data quality. We use a schema registry (Confluent Schema Registry) to enforce schema compatibility. Iceberg’s metadata layer tracks schema changes, allowing for backward and forward compatibility. We leverage the Hive Metastore (via Glue) to store table metadata and schema information. Data quality checks are integrated into the compaction process using Great Expectations to validate data integrity.

Security and Access Control

We utilize AWS Lake Formation to manage access control to our Iceberg tables. Lake Formation allows us to define granular permissions at the table and column level. Data is encrypted at rest using S3 encryption and in transit using TLS. Audit logging is enabled to track data access and modifications.

Testing & CI/CD Integration

We’ve implemented a comprehensive testing framework for our Iceberg compaction pipelines. This includes:

• Unit Tests: Testing individual compaction logic components.
• Integration Tests: Validating the end-to-end compaction process.
• Regression Tests: Ensuring that compaction performance remains consistent after code changes.

We use DBT for data transformation testing and Great Expectations for data quality validation. Our CI/CD pipeline automatically deploys compaction jobs to staging environments for testing before promoting them to production.

Common Pitfalls & Operational Misconceptions

1. Ignoring Compaction: Assuming that storage is cheap and compaction is unnecessary. This leads to performance degradation and increased costs.
2. Over-Compaction: Compacting too frequently can consume excessive resources and negatively impact query performance.
3. Incorrect Target File Size: Choosing a target file size that is too small or too large.
4. Lack of Monitoring: Failing to monitor compaction performance and identify bottlenecks.
5. Ignoring Data Skew: Not addressing data skew, which can lead to uneven compaction performance.

Enterprise Patterns & Best Practices

• Data Lakehouse Architecture: Embrace a data lakehouse architecture combining the benefits of data lakes and data warehouses.
• Batch vs. Streaming Compaction: Utilize a combination of batch and streaming compaction to optimize performance.
• File Format Selection: Choose Parquet or ORC based on query patterns and data characteristics.
• Storage Tiering: Leverage storage tiering to reduce costs for infrequently accessed data.
• Workflow Orchestration: Use Airflow or Dagster to orchestrate compaction pipelines.

Conclusion

Mastering data compaction in Apache Iceberg is critical for building reliable, scalable Big Data infrastructure. By carefully tuning compaction parameters, monitoring performance, and implementing robust testing and CI/CD pipelines, we were able to significantly improve query performance and reduce costs for our real-time analytics platform. Next steps include benchmarking new compaction configurations, introducing schema enforcement, and migrating to a more granular partitioning scheme. Continuous optimization is key to maintaining a high-performing data lakehouse.