E2B Filesystem Directory Watching and MakeDir Compatibility
此内容尚不支持你的语言。
Source:
docs/enhancements/0024-e2b-filesystem-watch-and-makedir/README.md· Status:implemented· Area:gateway· Milestone:v0.5| Field | Value | | :--- | :--- | | CEP Number |0024| | Status |implemented| | Stage |alpha| | Owning Area |gateway| | Participating Areas |runtime| | Authors | @xfgong | | Created | 2026-11-15 | | Updated | 2026-11-15 |
Release Signoff Checklist
Section titled “Release Signoff Checklist”- Enhancement scope, Goals, and Non-Goals are agreed upon
- ConnectRPC wire compatibility for
WatchDir,CreateWatcher,GetWatcherEvents,RemoveWatcher, andMakeDirverified - Dual-runtime behavior (Kubernetes
Kata + CSI Blockand LocalDocker + Loopback ext4/ gateway local fallback) documented - Security invariants verified: symlink containment,
/proc//dev//sysand network-mount refusal,cubeplex(1000:1000) directory ownership, per-sandbox watcher caps, and idle poll TTL - Unit and multi-process tests added in Rust (
podd) and Go (pkg/gateway) - Production Readiness Review (PRR) section complete
Summary
Section titled “Summary”This proposal specifies CubePod’s implementation of the official E2B filesystem.Filesystem ConnectRPC directory-watching and directory-creation RPCs:
- Streaming watch:
POST /filesystem.Filesystem/WatchDir(application/connect+jsonserver stream) - Polling watch:
POST /filesystem.Filesystem/CreateWatcher,GetWatcherEvents, andRemoveWatcher(unary ConnectRPC) - Directory creation:
POST /filesystem.Filesystem/MakeDir(unary ConnectRPC)
Both podd (inside the sandbox guest) and cubepod-gateway (forwarding to podd in live sandbox mode, or executing in local fallback mode when poddURL is absent) implement identical wire semantics, inotify event mapping, overflow handling, and path-safety rules.
Motivation
Section titled “Motivation”Official E2B Python (e2b) and TypeScript (@e2b/code-interpreter, e2b) SDKs implement sandbox.files.watch_dir(...) / sandbox.files.watchDir(...) and sandbox.files.make_dir(...) / sandbox.files.makeDir(...) over ConnectRPC (/filesystem.Filesystem/*). Without these endpoints:
- Agent frameworks and IDE integrations that watch
/workspaceor/home/userfor live file changes fail with404/unimplemented. - SDKs that call
make_dirbefore writing files fail or hit permission mismatches if directories are created asrootinstead ofcubeplex(1000:1000).
- ConnectRPC Wire Compatibility: Support
WatchDir,CreateWatcher,GetWatcherEvents,RemoveWatcher, andMakeDirwith officialapplication/connect+jsonframing and unary Connect JSON error codes (not_found,invalid_argument,already_exists,resource_exhausted). - Accurate Overflow & Lifecycle Semantics: Never silently drop events. Surface kernel
IN_Q_OVERFLOW, poll-ring overflow, subscriber queue overflow, and inotifyENOSPC/EMFILE/ENFILEasresource_exhausted. Emit a clean{}end-stream frame on server-side stop and periodic{"keepalive":{}}frames on quiet streams. - Security & Privilege Separation: Prevent symlink escapes and
/proc//dev//sysor unauthorized network/FUSE watches; chown newly createdMakeDirdirectories inpoddto the resolved user (defaulting tocubeplex1000:1000); enforce per-sandbox watcher caps and idle poll expiration.
Non-Goals
Section titled “Non-Goals”- Synthesizing missed events during the Linux inotify recursive
mkdir+inotify_add_watchrace window. - Watching network or FUSE mounts by default without
allowNetworkMounts: true. - Replacing
podd’s/v1/filesupload/download REST endpoints (covered by CEP-0018).
Proposal
Section titled “Proposal”User Stories
Section titled “User Stories”- As an E2B SDK user, I can call
files.watch_dir("/home/user", recursive=True)orfiles.watch_dir("/workspace")and receiveCREATE,WRITE,REMOVE,RENAME, andCHMODevents as files are modified. - As an E2B SDK user, I can call
files.make_dir("/home/user/project/src")and immediately write files into the created directory as the defaultcubeplex(1000:1000) user without permission errors. - As a cluster operator, a misbehaving sandbox or abandoned poll watcher cannot exhaust host or guest
fs.inotify.max_user_instancesor memory because watchers are capped per sandbox (16), idle poll watchers expire (300s), andWatchDirstreams do not buffer into a poll ring.
Notes / Constraints / Caveats
Section titled “Notes / Constraints / Caveats”- Linux
inotifyrecursive race: When a new subdirectory is created under a recursive watch,inotify_add_watchis called afterIN_CREATE | IN_ISDIRis read. Files created inside the new subdirectory beforeinotify_add_watchreturns are not queued by the kernel and are not synthesized by scanning. - Renamed watched subdirectories: If a watched subdirectory is renamed while armed, the
watchesmap retains the original relative prefix untilIN_IGNOREDor watcher removal. - Extra JSON fields:
MakeDirandentrypayloads includeisDirandis_diralongside proto fields (name,type,path,size,mode,permissions); Connect JSON clients withignoreUnknownFieldsignore them.
Risks and Mitigations
Section titled “Risks and Mitigations”| Risk | Impact | Mitigation |
|---|---|---|
Stream WatchDir killed after 256 events |
Long-lived streams fail with resource_exhausted even when client keeps up |
Poll ring (pending, cap 4096) is only populated for poll: true (CreateWatcher) watchers; WatchDir (poll: false) delivers strictly via subscriber channels. |
inotify fd close/reuse TOCTOU race |
stop() closing fd while reader thread is between fd check and poll/read can read an unrelated socket |
Reader thread/goroutine exclusively owns and closes fd on loop exit. stop() signals shutdown via eventfd (EFD_CLOEXEC | EFD_NONBLOCK). |
MakeDir creates root:root directories in podd |
cubeplex (1000:1000) agent processes cannot write into SDK-created directories (Invariant 4) |
podd handle_make_dir resolves ownership via resolve_file_ownership and calls chown_if_root on each newly created directory. |
Symlink escape out of /workspace or into /proc |
Agent creates symlink /workspace/l -> /etc or /proc and watches/creates outside allowed tree |
Deepest existing ancestor is canonicalized (canonicalize_deepest_ancestor / EvalSymlinks) before containment, /proc//dev//sys, and statfs mount checks. |
Host fs.inotify.max_user_instances exhaustion in gateway local mode |
One tenant opens many poll watchers and starves other tenants | Per-sandbox watcher cap (16), lazy reaping of idle poll watchers (300s TTL), and ENOSPC/EMFILE/ENFILE/ENOMEM mapped to resource_exhausted. |
Design Details
Section titled “Design Details”API / Wire Changes
Section titled “API / Wire Changes”All five endpoints are served under /filesystem.Filesystem/* on both podd (:49983) and cubepod-gateway:
POST /filesystem.Filesystem/MakeDir(unary):- Request:
{"path": "<dir>"} - Response (
200 OK):{"entry": {"name", "type": "FILE_TYPE_DIRECTORY", "path", "size", "mode", "permissions", "isDir": true, "is_dir": true}} - Errors:
already_exists(409) if directory exists;invalid_argument(400) if path or parent is a non-directory file, empty, or escapes confinement.
- Request:
POST /filesystem.Filesystem/CreateWatcher(unary,poll: true):- Request:
{"path", "recursive", "includeEntry", "allowNetworkMounts"} - Response (
200 OK):{"watcherId": "w<n>"}
- Request:
POST /filesystem.Filesystem/GetWatcherEvents(unary):- Request:
{"watcherId": "w<n>"} - Response (
200 OK):{"events": [{"name", "type", "entry"?}]}(drains poll ring, updateslast_poll) - Errors:
not_found(404) if missing/expired/stopped;resource_exhausted(429) if ring (4096) or kernelIN_Q_OVERFLOWoccurred.
- Request:
POST /filesystem.Filesystem/RemoveWatcher(unary):- Request:
{"watcherId": "w<n>"} - Response (
200 OK):{}
- Request:
POST /filesystem.Filesystem/WatchDir(server stream,poll: false,application/connect+json):- First frame (
flag=0x00):{"start":{}} - Event frames (
flag=0x00):{"filesystem":{"name","type","entry"?}} - Keepalive frames (
flag=0x00):{"keepalive":{}}every15swhen idle - Clean server-stop end frame (
flag=0x02):{}onRemoveWatcher,prepare_stop,stop_all, or sandbox deletion - Error end frame (
flag=0x02):{"error":{"code":"...","message":"..."}}
- First frame (
Inotify Event Mapping (aligned with fsnotify / envd)
Section titled “Inotify Event Mapping (aligned with fsnotify / envd)”Linux inotify Mask |
E2B Event Type | Notes |
|---|---|---|
IN_CREATE | IN_MOVED_TO |
EVENT_TYPE_CREATE |
Matches fsnotify (destination of rename is Create) |
IN_MOVED_FROM |
EVENT_TYPE_RENAME |
Source of rename |
IN_DELETE | IN_DELETE_SELF |
EVENT_TYPE_REMOVE |
|
IN_ATTRIB |
EVENT_TYPE_CHMOD |
|
IN_MODIFY |
EVENT_TYPE_WRITE |
IN_CLOSE_WRITE is not subscribed to avoid duplicate WRITE events |
State Machine & Data Flow
Section titled “State Machine & Data Flow”flowchart LR
C[E2B SDK Client] -->|ConnectRPC| GW[cubepod-gateway]
GW -->|Live poddURL present| PODD[podd in guest]
GW -->|Local fallback| LOC[gateway watch.go]
PODD -->|inotify + eventfd| K1[Guest Kernel]
LOC -->|inotify + eventfd| K2[Host Kernel]
K1 --> R1[Reader Thread]
K2 --> R2[Reader Goroutine]
R1 -->|poll=true| RING1[Poll Ring cap 4096]
R1 -->|poll=false| SUB1[Stream Channel cap 64]
R2 -->|poll=true| RING2[Poll Ring cap 4096]
R2 -->|poll=false| SUB2[Stream Channel cap 64]Path Resolution, Symlink Containment, and Ownership
Section titled “Path Resolution, Symlink Containment, and Ownership”podd(in-guest):poddruns inside the sandbox trust boundary. Relative paths and/workspace/...paths are confined to canonicalizedworkdir(default/workspace); explicit non-/workspaceabsolute paths (e.g./home/user) are permitted just likefiles.write. Deepest existing ancestor is canonicalized before checking/proc,/dev,/sysandstatfsnetwork/FUSE mount magic (nfs,cifs,smb,smb2,fuse,9p,cephrefused unlessallowNetworkMounts: true). Newly createdMakeDirdirectories are chowned viachown_if_root(path, uid, gid)usingresolve_file_ownership(defaulting tocubeplex1000:1000).cubepod-gateway(local fallback): MirrorsresolveFilePathby re-rooting non-/workspaceabsolute paths (such as/home/user) underrec.WorkspaceDir, while rejecting/workspace/..and relative..escapes. Deepest existing ancestor is canonicalized viafilepath.EvalSymlinksand verified to remain inside canonicalizedWorkspaceDir, preventing container-written symlinks from resolving on the host.rec.WorkspaceDirinitialization is synchronized unders.mu.
Dual-Runtime Awareness
Section titled “Dual-Runtime Awareness”- Production Kubernetes (
Kata + CSI Block):cubepod-gatewayproxies all/filesystem.Filesystem/*ConnectRPC requests directly topoddinside the Kata MicroVM over the pod network.poddwatches the guest’s OverlayFS/workspace(CSI Block volume) or guest rootfs (/home/user) using the guest kernel’sinotifysubsystem. On/v1/prepare-stop,poddcallswatchers.stop_all(), cleanly ending activeWatchDirstreams beforesync(2)and snapshot/quiesce. - Local Development (
Docker + Loopback ext4): When a Docker container runspodd,cubepod-gatewayforwards topoddjust as in Kubernetes. When running in gateway local-fallback mode (withoutpoddURL),pkg/gateway/watch.goruns in-process on the host againstrec.WorkspaceDir(with non-/workspacepaths re-rooted underWorkspaceDir) and cleans up all sandbox watchers onDELETE /sandboxes/{id}.
Boundary Conditions & Failure Recovery
Section titled “Boundary Conditions & Failure Recovery”- Slow stream consumer: Per-subscriber bounded queue (
64) fails only that subscriber withresource_exhausted; other subscribers and the watcher remain healthy. - Idle poll watchers:
POLL_WATCHER_IDLE_TTL = 300slazily reaps abandonedCreateWatcherinstances oncreateandgetcalls. - Subdirectory
EACCESduring recursive walk: Skipped without failing the recursive watch;ENOSPC/EMFILE/ENFILE/ENOMEMpropagate asresource_exhausted.
Test Plan
Section titled “Test Plan”- Rust unit & HTTP tests (
crates/podd/src/watch.rs,crates/podd/src/tests.rs):- Stream watcher >
WATCH_RING_CAPevents without overflow; poll ring overflow (resource_exhausted); kernelIN_Q_OVERFLOW; slow subscriber isolation. - Clean
{}end-stream frame onRemoveWatcher/stop_all;prepare_stopcleanup. MakeDirofficial error semantics (already_exists,invalid_argumenton file/file-parent), explicit absolute paths, symlink escape rejection,/proc//sys//devrefusal.- Watcher cap + idle poll TTL reaping;
IN_MOVED_TO->EVENT_TYPE_CREATEandIN_CLOSE_WRITEexclusion.
- Stream watcher >
- Go unit & race tests (
pkg/gateway/local_watch_pty_test.go,pkg/gateway/connectrpc_test.go):- Same coverage under
go test -race: stream >watchRingCap, poll ring overflow,resolveWatchPath/home/userre-rooting + symlink escape rejection, watcher cap + TTL,mapInotify, sandbox delete cleanup.
- Same coverage under
Graduation Criteria
Section titled “Graduation Criteria”- Alpha (current): Implemented in
poddandcubepod-gatewaywith unit/multi-process tests passing (cargo test -p podd,go test -race ./pkg/gateway/...). - Beta: Verified against official Python and TypeScript E2B SDKs in CI and live Kubernetes Kata cluster.
- Stable: Baked in CubePlex production with Prometheus watcher/stream metrics.
Upgrade / Downgrade Strategy
Section titled “Upgrade / Downgrade Strategy”No CRD or persistent on-disk schema changes. Watchers are ephemeral in-memory kernel resources; rolling podd or cubepod-gateway cleanly closes streams and clients reconnect.
Production Readiness Review Questionnaire (PRR)
Section titled “Production Readiness Review Questionnaire (PRR)”- Feature Enablement and Rollback: Always enabled as part of E2B ConnectRPC compatibility; no
/workspaceformat changes, so rollback is a standard binary downgrade. - Rollout, Upgrade and Rollback Planning: In-flight
WatchDirstreams receive connection close or clean{}end-stream frames duringprepare_stop/ pod termination. - Monitoring Requirements: ConnectRPC HTTP status/error codes (
429 resource_exhausted,404 not_found,400 invalid_argument) and gateway logs. - Dependencies: Linux kernel
inotifyandeventfdsyscalls (standard in both Kata guest kernel and Docker host kernel). - Scalability: Bounded to
16watchers per sandbox,4096events per poll ring, and64events per stream subscriber;300sidle poll TTL bounds leaked poll watchers. - Troubleshooting:
resource_exhaustedwith"too many active watchers"indicates sandbox watcher cap (16);"inotify_init1: Too many open files"indicates host/guestfs.inotify.max_user_instancessaturation.
Implementation History
Section titled “Implementation History”2026-11-15: Initial implementation incrates/podd/src/watch.rs,pkg/gateway/watch.go, andpkg/gateway/connectrpc.go; review hardening (stream/poll separation,eventfdfd-ownership,MakeDirchown, symlink containment, path alignment, watcher caps/TTL, clean end-stream/keepalive frames,fsnotifyevent mapping) and CEP-0024 documented.
Drawbacks
Section titled “Drawbacks”- Maintaining parallel inotify implementations in Rust (
podd) and Go (pkg/gatewaylocal fallback) requires keeping event mapping and path semantics synchronized across both codebases.
Alternatives
Section titled “Alternatives”- Polling directory trees (
statloop): Rejected due to high CPU/I/O overhead and missed transient events. - Host
fanotify: Rejected becausefanotifyrequiresCAP_SYS_ADMINand does not work inside unprivileged containers or without elevated privileges. - Using
notify/fsnotifyexternal crate/module: Rejected inpoddto preservepodd’s zero-external-dependency Rust build (libconly);golang.org/x/sys/unixin gateway avoids extra transitive dependencies while matchingfsnotifyevent semantics.