E2B Interactive PTY Execution, Streaming I/O, Resize, and Reconnect
Source:
docs/enhancements/0023-e2b-interactive-pty/README.md· Status:implemented· Area:runtime· Milestone:v0.5| Field | Value | | :--- | :--- | | CEP Number |0023| | Stage |alpha| | Status |implemented| | Created |2026-11-15| | Updated |2026-11-15| | Owning Area |runtime(podd),gateway(cubepod-gateway) |
Release Signoff Checklist
Section titled “Release Signoff Checklist”- Enhancement scope, Goals, and Non-Goals are defined
- ConnectRPC wire compatibility with official E2B 2.64 Python and TypeScript SDKs is specified
- Dual-runtime behavior (Kubernetes
Kata + CSI Blockviapoddand Local DevelopmentDocker / Gateway Local Fallback) is documented - Security boundaries (fd lifetime,
O_CLOEXEC, sandbox isolation, env sanitization, credential separation) are reviewed - Unit and concurrency/race tests cover PTY lifecycle, isolation, and edge cases
- Production Readiness Review (PRR) questionnaire is complete
Summary
Section titled “Summary”This proposal specifies E2B-compatible interactive pseudo-terminal (PTY) execution across CubePod’s in-guest daemon (podd) and API gateway (cubepod-gateway). Official E2B 2.64 Python and TypeScript SDKs can create real OS PTY sessions, send keystrokes (SendInput, StreamInput), resize terminal windows (Update), reconnect to running or recently exited PTYs with bounded output replay (Connect), list running processes (List), and deliver signals (SendSignal).
Client stream disconnects do not terminate the PTY process; the PTY continues running until it exits or receives an explicit SendSignal, matching upstream envd semantics.
Motivation
Section titled “Motivation”Prior to this enhancement, CubePod’s ConnectRPC /process.Process/* surface only supported non-interactive pipe execution (Start with stdout/stderr pipes) and lacked PTY allocation, bidirectional input, window resizing, and live reconnect replay. Interactive AI coding agents (e.g., terminal-based coding assistants, vim, htop, REPLs) require a controlling TTY (isatty() == true), SIGWINCH resize propagation, and session reconnects.
- Real OS PTY Allocation: Allocate a real master/slave PTY pair with
setsid()andTIOCSCTTY, defaultingTERMtoxterm-256colorand default dimensions to80x24. - Official E2B 2.64 Wire Compatibility: Support
Start(withpty.size),SendInput,StreamInput,Update,Connect,List, andSendSignalover ConnectRPC (application/connect+jsonandapplication/json), emittingexitCode(camelCase) inendframes. - Disconnect-Safe Lifecycle & Bounded Replay: Keep PTYs alive when
StartorConnectHTTP streams disconnect; maintain a 64 KiB replay tail and per-subscriber bounded queues (64chunks) that fail slow consumers withresource_exhaustedrather than blocking the PTY. - Dual-Runtime Parity: Implement the same semantics in
podd(for KubernetesKata + CSI Blockand Docker withpodd) and incubepod-gatewaylocal fallback mode.
Non-Goals
Section titled “Non-Goals”CloseStdinfor PTY: Closing stdin half-duplex on a single bidirectional PTY master fd is not supported;CloseStdinreturns501 Not Implemented(unimplemented).- Server-Side Idle Kill Timeout:
timeoutMsin ConnectRPC requests is a client stream deadline, not a server-side process kill timer. - Persistent PTY State Across Sandbox Pause/Snapshot: Running PTYs are in-memory processes and are terminated on
prepare_stop/kill_all_managed.
Proposal
Section titled “Proposal”User Stories
Section titled “User Stories”- Story 1 (Interactive Agent Terminal): As an AI agent using the official E2B SDK (
sandbox.pty.create), I can launch/bin/bash -i -l, send commands viaSendInputorStreamInput, resize the terminal viaUpdate, and receive raw terminal output frames (event.data.pty). - Story 2 (Reconnect Without Data Loss): As a client whose network connection drops or whose
Starttimeout expires, I can callConnectwith the PTY’spidortagto receive the last 64 KiB of output plus seamless live streaming without gaps or duplicates. - Story 3 (Multi-Tenant Local Gateway): As a developer running
cubepod-gatewayin local mode with multiple sandboxes, sandbox B cannot list, read, type into, resize, or kill sandbox A’s PTY processes.
Notes / Constraints / Caveats
Section titled “Notes / Constraints / Caveats”- Dual-Runtime Awareness:
- Production Kubernetes (
Kata + CSI Block):cubepod-gatewayforwards all/process.Process/*ConnectRPC requests directly to the sandbox pod’spoddHTTP daemon (shouldForwardConnectRPC).poddallocates the PTY inside the Kata MicroVM guest, appliesconfigure_command_credentials(defaulting tocubeplex1000:1000), and managesOwnedFdlifetimes. - Local Development (
Docker + Loopback ext4/ Gateway Local Mode): When a sandbox runs withpodd(Docker runtime), requests forward topoddidentically. When running in gateway-local fallback mode (nopoddURL),cubepod-gatewayallocates the OS PTY on the host via/dev/ptmx(openOSPTY), scopes all lookups bysandboxID, and sanitizes the child environment viasafeSandboxEnv(never inheritingos.Environ()).
- Production Kubernetes (
- Zero-Trust Secret Safety: Neither
poddnor gateway local mode injects host/gateway environment secrets (os.Environ()) into PTY processes. OnlyPATH,HOME,LANG=C.UTF-8,TERM=xterm-256color, and explicitly requestedenvs(which may containcpodref_placeholders) are passed. - No Gzip Magic-Byte Sniffing:
unpack_all_connect_frameshonors only declaredContent-Encoding: gzipand the Connect per-frame compression flag (flag & 0x01 != 0), never sniffing0x1f 0x8bmagic bytes.
Risks and Mitigations
Section titled “Risks and Mitigations”| Risk / Failure Mode | Impact | Mitigation |
|---|---|---|
Raw fd closed while another thread calls write/ioctl/read |
Input or ioctl sent to a reused fd belonging to another client or file | podd wraps the PTY master in Arc<OwnedFd> behind a Mutex<Option<Arc<OwnedFd>>>; SendSignal only signals, and the fd is closed only when the last Arc drops. |
Background child (sleep &) holds slave fd open after shell exits |
Reader never sees EIO, zombie never reaped, End event never sent |
podd reaps child.wait() in a dedicated waiter thread; once child exits, the reader drains buffered bytes until EAGAIN, publishes End, and clears the master. |
Concurrent fork/exec inherits PTY slave fds |
Other child processes pin the PTY open indefinitely | All PTY fds (posix_openpt, slave open, and F_DUPFD_CLOEXEC) are opened with O_CLOEXEC atomically. |
| Cross-sandbox PID/tag lookup in gateway local mode | Tenant B can read, write, resize, or kill Tenant A’s PTY | findLocalCommandEntry and List strictly filter by entry.sandboxID == rec.SandboxID. |
Gateway os.Environ() leaked into local PTYs |
Host credentials (E2B_API_KEY, KUBECONFIG) exposed to sandbox |
safeSandboxEnv constructs a minimal safe environment instead of copying os.Environ(). |
| Slow subscriber blocks PTY reader or drops bytes silently | PTY stalls or corrupts output for other subscribers | Each subscriber has a bounded queue (64); overflow drops only that subscriber and ends its stream with resource_exhausted. |
| Unbounded process registry growth | Memory leak from 64 KiB replay tails of exited PTYs | Exited entries are pruned after a 5-minute grace period or when exited count exceeds 256. |
Design Details
Section titled “Design Details”API / Wire Changes
Section titled “API / Wire Changes”All endpoints are served under /process.Process/* (and /sandboxes/{sandboxID}/process.Process/* on the gateway):
POST /process.Process/Start: Acceptsprocess.{cmd, args, envs, cwd, uid, gid, user}, optionalid,tag, andpty.size.{cols, rows}. Rejects duplicate client-suppliedidwith409 Conflict(already_exists). Streams Connect frames:event.start.pid,event.data.pty(base64),event.end.{exitCode, exited: true}, and end-of-stream{}.POST /process.Process/SendInput: Acceptsprocessselector (pidortag) andinput.pty/input.stdin(base64). Rejects invalid base64 with400 invalid_argument. Writes all bytes with non-blockingEAGAIN+pollretry (5s timeout inpodd), returning500 internalon failure.POST /process.Process/StreamInput: Client-streaming ConnectRPC frames (event.startselector,event.database64 input). Buffers HTTP/1.1 request body, unpacks Connect frames honoring per-frame compression (flag & 1), and writes all bytes.POST /process.Process/Update: Acceptsprocessselector andpty.size.{cols, rows}. AppliesTIOCSWINSZand sendsSIGWINCHto the process group and pid.POST /process.Process/Connect: Acceptsprocessselector. Atomically snapshots the 64 KiB replay tail and subscribes under the same locks as publish (no gaps, no duplicates). For running non-PTY processes, streams new output and waits for actual exit before emittingevent.end.POST /process.Process/List: Returns running processes (scoped torec.SandboxIDin gateway local mode) withpid,tag, andconfig.{cmd, args, envs, cwd}.POST /process.Process/SendSignal: SendsSIGTERMorSIGKILLto-pidandpidwithout prematurely markingrunning = falseor closing the master fd.POST /process.Process/CloseStdin: Returns501 Not Implemented(unimplemented).
State Machine & Data Flow
Section titled “State Machine & Data Flow”sequenceDiagram
participant C as E2B Client
participant G as cubepod-gateway
participant P as podd (in-guest)
participant W as Waiter Thread
participant R as Reader Thread
C->>G: POST /process.Process/Start (pty.size)
G->>P: Forward ConnectRPC Start
P->>P: posix_openpt(O_CLOEXEC) + F_DUPFD_CLOEXEC + spawn
P-->>C: event.start { pid }
loop PTY Running
R->>P: read(master) -> publish_pty_chunk (64KiB tail + subs)
P-->>C: event.data.pty (base64)
C->>G: SendInput / Update (TIOCSWINSZ + SIGWINCH)
end
C->>G: Disconnect (timeoutMs)
Note over P: PTY keeps running
C->>G: POST /process.Process/Connect { pid }
P-->>C: Snapshot (<=64KiB) + live stream
W->>W: child.wait() -> exitCode
R->>R: Drain until EAGAIN, publish_pty_end(exitCode), clear_master()
P-->>C: event.end { exitCode, exited: true }Boundary Conditions & Failure Recovery
Section titled “Boundary Conditions & Failure Recovery”- Client Disconnect: Dropping the
StartorConnectTCP stream removes only that subscriber; the PTY process continues running andConnectcan reattach later. - Process Ignores
SIGTERM:SendSignal(SIGTERM)delivers the signal only;runningremainstrueuntilchild.wait()observes exit. - Background Child Holding Slave: When the main shell exits, the waiter thread unblocks
child.wait(), causing the reader thread to drain buffered output untilEAGAIN, publishEnd, and drop the masterArc<OwnedFd>. - Sandbox Stop (
prepare_stop):kill_all_managedsendsSIGKILLto all managed process groups and signals readers to exit.
Test Plan
Section titled “Test Plan”- Unit Tests (
cargo test -p podd --locked):test_pty_start_isatty_resize_bidirectional_and_killtest_pty_reconnect,test_pty_start_disconnect_does_not_kill,test_pty_exit_code_nonzerotest_pty_stream_input,test_pty_process_list,test_pty_kill_all_managedtest_pty_replay_bounded_snapshot_has_no_gap_or_dup,test_pty_subscriber_overflow_sets_flag,test_pty_slow_start_consumer_ends_with_resource_exhaustedtest_pty_background_child_holding_slave_publishes_end_and_reapstest_pty_concurrent_send_input_and_kill_safetest_pty_send_signal_trapped_sigterm_stays_runningtest_process_connect_running_non_pty_waits_for_exittest_pty_rejects_invalid_base64_and_duplicate_id,test_unpack_all_connect_frames_no_gzip_sniff,test_pty_close_stdin_returns_unimplemented
- Gateway Tests (
go test -race ./pkg/gateway/... -count=1):TestConnectRPC_PTY_Connect_And_Multibyte_And_ExitCode,TestConnectRPC_PTY_StreamInput,TestConnectRPC_PTY_ForwardingTestConnectRPC_PTY_SandboxIsolation_And_EnvSanitizationTestConnectRPC_PTY_ConcurrentSendInputAndKill_NoRaceTestConnectRPC_Process_ConnectRunningNonPTY_WaitsForExit
Graduation Criteria
Section titled “Graduation Criteria”- Alpha (Current): All unit and
-racetests pass inpoddandcubepod-gateway; official E2B 2.64 SDK wire shapes verified. - Beta: Live Kubernetes (
Kata + CSI Block) and Docker E2E suite exercisessandbox.pty.create/connect/resize/killend-to-end. - Stable: Multi-day soak testing with concurrent interactive terminals and
prepare_stopcleanup.
Upgrade / Downgrade Strategy
Section titled “Upgrade / Downgrade Strategy”- Purely additive in-memory process management and ConnectRPC handlers; no CRD schema or
/workspace/rootfs-upperPVC layout changes. Rolling backpoddorcubepod-gatewayhas zero impact on persisted storage.
Production Readiness Review Questionnaire (PRR)
Section titled “Production Readiness Review Questionnaire (PRR)”- Feature Enablement and Rollback: Always enabled as part of the E2B ConnectRPC
/process.Process/*surface. Rollback is a standard binary downgrade with no persistent state migration. - Rollout, Upgrade and Rollback Planning: In-flight PTYs live in
podd(inside the pod) orcubepod-gatewaymemory and terminate if the daemon restarts;/workspacedata is unaffected. - Monitoring Requirements: ConnectRPC HTTP status codes (
200,400,404,409,500,501) andresource_exhaustedstream termination frames. - Dependencies: Linux
/dev/ptmx(devpts) andposix_openpt/TIOCSWINSZ/TIOCSCTTYioctls (standard in both Kata guest kernels and host Linux). - Scalability: Bounded by
64 KiBreplay buffer per PTY and256retained exited entries (5-minute grace). - Troubleshooting: Use
POST /process.Process/Listto inspect active PTYs (pid,tag,cmd,cwd) andSendSignal(SIGKILL) to terminate stuck sessions.
Implementation History
Section titled “Implementation History”2026-11-15: Initial implementation incrates/poddandpkg/gateway, followed by review hardening (OwnedFdlifetime, atomicO_CLOEXEC, waiter thread reaping, gateway sandbox isolation,safeSandboxEnv, and regression tests).
Drawbacks
Section titled “Drawbacks”StreamInputinpoddandcubepod-gatewaybuffers the HTTP/1.1 request body before parsing Connect frames, which is suitable for interactive keystroke bursts but not for multi-megabyte continuous binary streaming over a singleStreamInputcall.- Exited PTY replay buffers (
<= 64 KiB) remain in memory for up to 5 minutes (capped at 256 entries) to support post-exitConnect.
Alternatives
Section titled “Alternatives”- WebSocket-Only PTY: Rejected because official E2B 2.64 SDKs use ConnectRPC over HTTP (
POST /process.Process/*), not WebSockets. - Closing Master Fd on
SendSignal: Rejected because it races with concurrentSendInput/Update/readcallers (risking fd-reuse corruption) and turnsSIGTERMinto an immediateSIGHUP.