Skip to content

feat(pageserver): create image layers at L0-L1 boundary #12023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 28, 2025

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented May 27, 2025

Problem

Previous attempt #10548 caused some issues in staging and we reverted it. This is a re-attempt to address #11063.

Currently we create image layers at latest record LSN. We would create "future image layers" (i.e., image layers with LSN larger than disk consistent LSN) that need special handling at startup. We also waste a lot of read operations to reconstruct from L0 layers while we could have compacted all of the L0 layers and operate on a flat level of historic layers.

Summary of changes

  • Run repartition at L0-L1 boundary.
  • Roll out with feature flags.
  • Piggyback a change that downgrades "image layer creating below gc_cutoff" to debug level.
@skyzh skyzh requested review from erikgrinaker and problame May 27, 2025 07:06
@skyzh skyzh requested a review from a team as a code owner May 27, 2025 07:06
Copy link

github-actions bot commented May 27, 2025

8503 tests run: 7932 passed, 0 failed, 571 skipped (full report)


Flaky tests (3)

Postgres 17

Postgres 14

Code coverage* (full report)

  • functions: 32.3% (9062 of 28021 functions)
  • lines: 48.6% (79858 of 164178 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
82669fe at 2025-05-28T07:10:24.480Z :recycle:
@skyzh skyzh force-pushed the skyzh/image-generation branch from c9ae00b to 6ce1a88 Compare May 28, 2025 06:01
Signed-off-by: Alex Chi Z <[email protected]>
@skyzh skyzh enabled auto-merge May 28, 2025 06:06
@skyzh skyzh added this pull request to the merge queue May 28, 2025
Merged via the queue into main with commit 67ddf1d May 28, 2025
100 checks passed
@skyzh skyzh deleted the skyzh/image-generation branch May 28, 2025 07:07
@skyzh skyzh mentioned this pull request May 29, 2025
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants