[codex] Observe remote exec-server lifecycle by richardopenai · Pull Request #27470 · openai/codex

richardopenai · 2026-06-10T21:02:06Z

Summary

Record bounded duration and outcome metrics for remote environment registration and Noise rendezvous connection attempts.
Count reconnects by bounded reason: disconnect, connection failure, or rejected registration.
Trace registration at the owning client boundary without exporting raw environment or registration identifiers.
Replace the stale pre-Noise WebSocket observability design with the current remote transport model.

Stack

Review and land this stack in order:

[codex] Trace exec-server JSON-RPC requests #27466 — trace exec-server JSON-RPC requests
[codex] Record exec-server lifecycle metrics #27467 — record bounded connection, request, and process lifecycle metrics
[codex] Observe remote exec-server lifecycle #27470 — observe remote registration and Noise rendezvous lifecycle (this PR)

Validation

just test -p codex-exec-server --lib (149 passed)
just test -p codex-cli --test exec_server (4 passed)
just argument-comment-lint
just bazel-lock-check
just fix -p codex-exec-server -p codex-cli
just fmt

chatgpt-codex-connector · 2026-06-10T21:13:13Z

Review source: Codex Cloud Agents (CCA)

Codex Cloud Agents (CCA) couldn't complete this review. The original Codex Review is unaffected.

## Why Exec-server JSON-RPC calls can cross local and remote transports, but trace context stopped at the RPC boundary. That made client and server work difficult to correlate when diagnosing latency or failures. ## What changed - Propagate the current W3C trace context on outbound JSON-RPC requests. - Parent inbound request spans from received trace context. - Record the received JSON-RPC method on server spans and keep each span open through response enqueue. - Add only the OTEL dependencies required by the exec-server crate. ## Stack Review and land this stack in order: 1. #27466 — trace exec-server JSON-RPC requests **(this PR)** 2. #27467 — record bounded connection, request, and process lifecycle metrics 3. #27470 — observe remote registration and Noise rendezvous lifecycle ## Validation - `just test -p codex-exec-server --lib` (153 passed) - `just bazel-lock-check` - `just fix -p codex-exec-server`

## Summary - Record bounded connection, request, and process lifecycle metrics. - Report active gauges from callbacks on every collection, including delta exports. - Serialize active-count updates so concurrent starts and finishes cannot publish stale values. - Serialize process exit, explicit termination, and shutdown through the process registry so exactly one completion result wins. - Keep the implementation small with single-owner RAII guards and one real OTLP/HTTP integration test using the existing `wiremock` dependency. ## Root cause Process exit and session shutdown previously used cloned completion state. That avoided duplicate emission, but it duplicated lifecycle ownership and made the ordering harder to reason about. The process registry mutex already defines the lifecycle ordering, so the final implementation stores the metric guard and termination flag directly on the process entry. Whichever path claims the entry first owns the completion result. Production metric export uses delta temporality. Event-only synchronous gauge recordings disappear after the next collection when no count changes, so active counts now use observable callbacks that report current state on every collection. The cleanup also removes the constant `result="accepted"` connection tag, redundant route and response assertions, a custom HTTP collector, and fallback initialization machinery that did not add behavior. ## Stack Review and land this stack in order: 1. #27466 — trace exec-server JSON-RPC requests 2. #27467 — record bounded connection, request, and process lifecycle metrics **(this PR)** 3. #27470 — observe remote registration and Noise rendezvous lifecycle ## Validation - `just test -p codex-exec-server --lib` (158 passed) - `just test -p codex-cli --test exec_server` (3 passed) - `just test -p codex-otel observable_gauge_is_collected_on_every_delta_snapshot` (1 passed) - `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server` - `just fmt` - `git diff --check`

anp-oai · 2026-06-25T20:26:47Z

    Ok(())
 }

+#[cfg(unix)]


Could we also have a test that covers the windows path? I'm not sure what tokio's ctrl_c handler waits for on Windows but should be possible to exercise it here?

I investigated this. Tokio’s Windows path waits for CTRL_C_EVENT, but Windows cannot target that event to only the spawned process: group 0 broadcasts to every process sharing the test console, while a nonzero group does not deliver Ctrl-C. CREATE_NEW_PROCESS_GROUP also disables Ctrl-C. Tokio’s own tests therefore stub its internal handler instead of exercising the OS path. A safe integration test here would require a separate console and helper process, or expanding production behavior to accept targetable Ctrl-Break. Both seem disproportionate for this PR, so I’d prefer to rely on Tokio’s signal implementation and Windows CI compilation unless we want to add Ctrl-Break semantics.

This was referenced Jun 10, 2026

[codex] Observe remote exec-server WebSocket lifecycle #27471

Closed

[codex] Record exec-server process lifecycle metrics #27469

Closed

richardopenai marked this pull request as ready for review June 10, 2026 21:13

richardopenai added the code-reviewed label Jun 10, 2026 — with ChatGPT Codex Connector

richardopenai force-pushed the codex/exec-server-remote-registration-observability branch from cb15fe9 to 08f9dd1 Compare June 16, 2026 19:21

richardopenai requested a review from a team as a code owner June 16, 2026 19:21

richardopenai changed the title ~~[codex] Observe remote exec-server registration~~ [codex] Observe remote exec-server lifecycle Jun 16, 2026

richardopenai changed the base branch from codex/exec-server-process-metrics to codex/exec-server-connection-metrics June 16, 2026 19:22

richardopenai mentioned this pull request Jun 16, 2026

[codex] Record exec-server lifecycle metrics #27467

Merged

richardopenai force-pushed the codex/exec-server-connection-metrics branch from 7a28c0a to 5a94cf1 Compare June 18, 2026 22:11

richardopenai force-pushed the codex/exec-server-remote-registration-observability branch from 69e10d5 to 97d99d1 Compare June 18, 2026 22:21

richardopenai force-pushed the codex/exec-server-connection-metrics branch from 5a94cf1 to 1128e27 Compare June 18, 2026 22:39

richardopenai force-pushed the codex/exec-server-remote-registration-observability branch from 97d99d1 to f8dd96f Compare June 18, 2026 23:23

richardopenai force-pushed the codex/exec-server-connection-metrics branch from 1128e27 to 2d7885f Compare June 19, 2026 00:15

richardopenai force-pushed the codex/exec-server-remote-registration-observability branch from f8dd96f to c475918 Compare June 19, 2026 00:15

richardopenai mentioned this pull request Jun 19, 2026

[codex] Trace exec-server JSON-RPC requests #27466

Merged

anp-oai reviewed Jun 23, 2026

View reviewed changes

Comment thread codex-rs/cli/src/exec_server_telemetry.rs

Base automatically changed from codex/exec-server-connection-metrics to main June 25, 2026 18:02

richardopenai added 4 commits June 25, 2026 11:04

exec-server: observe remote lifecycle

bacc15e

exec-server: complete remote telemetry instrumentation

b1af93e

exec-server: wait for readiness before SIGTERM test

5b56bbd

exec-server: register shutdown signal before startup

5916ef7

richardopenai force-pushed the codex/exec-server-remote-registration-observability branch from 44642f8 to 5916ef7 Compare June 25, 2026 18:11

exec-server: trim remote observability scope

5509635

richardopenai requested a review from anp-oai June 25, 2026 19:22

anp-oai approved these changes Jun 25, 2026

View reviewed changes

richardopenai merged commit 3b22498 into main Jun 25, 2026
31 checks passed

richardopenai deleted the codex/exec-server-remote-registration-observability branch June 25, 2026 20:42

github-actions Bot locked and limited conversation to collaborators Jun 25, 2026

Navigation Menu

Uh oh!

[codex] Observe remote exec-server lifecycle - #27470

[codex] Observe remote exec-server lifecycle#27470
richardopenai merged 5 commits into
mainfrom
codex/exec-server-remote-registration-observability

richardopenai commented Jun 10, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Jun 10, 2026

Uh oh!

anp-oai Jun 25, 2026

richardopenai Jun 25, 2026

Uh oh!

Labels

2 participants

Uh oh!

Conversation

richardopenai commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Validation

chatgpt-codex-connector Bot commented Jun 10, 2026

Uh oh!

anp-oai Jun 25, 2026

Choose a reason for hiding this comment

richardopenai Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Labels

2 participants

richardopenai commented Jun 10, 2026 •

edited

Loading