Skip to content

feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166)#222

Open
jagannalla wants to merge 2 commits into
Netflix:mainfrom
jagannalla:feature/local-memory-support
Open

feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166)#222
jagannalla wants to merge 2 commits into
Netflix:mainfrom
jagannalla:feature/local-memory-support

Conversation

@jagannalla

@jagannalla jagannalla commented Jun 14, 2026

Copy link
Copy Markdown

Description

This PR addresses three core enhancement and bug-fix issues across Maestro's runtime, engine, and database layers:

  1. Step Local Memory Support (Maestro add local-memory support #134):

    • Added transient, step-instance-scoped local memory with a strict capacity constraint of 64KB (serialized JSON bytes).
    • Integrated lifecycle hooks in MaestroTask to ensure local memory is cleared when the step terminates or is canceled.
    • Exposed getLocalMemory() in StepRuntimeSummary for child step execution contexts to read/write.
  2. Generation Guard Data Protection (Leverage generation_id to prevent reverting step data #128):

    • Added a generation_id column to the maestro_step_instance schema (migration included).
    • Enforced database-level protection checks on step updates/upserts, preventing stale flow executions/workers from overwriting updates from newer runs (generation_id <= EXCLUDED.generation_id and generation_id <= :flowGeneration).
    • Propagated flow generation context from the event loop down through database synchronization layer.
    • Preserved backwards-compatible signatures to prevent breaks in legacy tests and services.
  3. Human-in-the-Loop While Loop Breakpoint Support (Human in the loop support for while loop #166):

    • Added PAUSED to the StepRuntime.State state machine, allowing step execution to yield back a pause state.
    • Updated StepRuntimeManager to handle PAUSED step status by cleanly calling runtimeSummary.markPaused(...).
    • Updated WhileStepRuntime to check stepBreakpointDao.createPausedStepAttemptIfNeeded(...) between iterations. If a breakpoint exists, it yields a PAUSED state with timeline event logs.
    • Modified WhileStepRuntime.start(...) to return the existing WhileArtifact directly, allowing a seamless resume from the same attempt/iteration without resetting counters or the inner loop's run policy.
    • Wired database breakpoint DAO dependency in MaestroStepRuntimeConfiguration.

Verification

  • Unit and Integration Tests: Added extensive test coverage for all three features. Verified with ./gradlew :maestro-engine:test (all pass).
  • Compile/Build Checks: Verified compilation and formatting with ./gradlew build -x test and ./gradlew spotlessCheck.
@jagannalla jagannalla changed the title feat: Implement step local-memory (#134) and flow generation ID protection (#128) feat: Add Step Local Memory (#134), Generation ID Protection (#128), and While Loop Breakpoint Support (#166) Jun 14, 2026
…data protection (Netflix#128)

- Added transient, step-instance-scoped local memory with 64KB limit and automated cleanup.
- Added generation_id database column to maestro_step_instance and enforced checks on UPSERT/UPDATE to prevent stale worker writes.
- Added unit and integration tests for local memory and generation ID protection.
- Configured SpotBugs exclusions filter and updated configuration.

Signed-off-by: Jagan Nalla <jagannalla1@gmail.com>
Signed-off-by: Jagan Nalla <jagannalla1@gmail.com>
@jagannalla jagannalla force-pushed the feature/local-memory-support branch from ef2e098 to 97b11fa Compare June 15, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant