Official E2B Secrets API and Zero-Trust Secret Store Integration
Source:
docs/enhancements/0027-e2b-secrets-api/README.md· Status:implementable· Area:gateway· Milestone:v0.5| Stage | Status | Created | Updated | Owning Area | | :--- | :--- | :--- | :--- | :--- | |alpha|implementable|2026-11-15|2026-11-15|gateway|
Table of Contents
Section titled “Table of Contents”- Release Signoff Checklist
- Summary
- Motivation
- Proposal
- Design Details
- Production Readiness Review Questionnaire (PRR)
- Implementation History
- Drawbacks
- Alternatives
Release Signoff Checklist
Section titled “Release Signoff Checklist”- Enhancement is scoped clearly with explicit Goals and Non-Goals.
- API / OpenAPI / CRD changes preserve backward compatibility with existing
extension.secretBindingsandpkg/egress. - Security boundary (MicroVM isolation, UID/GID privilege separation, Zero-Trust
cpodref_egress secret safety) is reviewed. - Dual-runtime behavior (Production Kubernetes
Kata + CSI Blockand Local DevelopmentDocker + Loopback ext4) is specified. - Test plan includes unit tests (
go test -race) and integration/E2E verification of write-only semantics, rotation, revocation, and HTTPS header injection. - Graduation criteria for Alpha, Beta, and Stable are defined.
Summary
Section titled “Summary”CEP-0027 adds the official E2B Secrets management API (POST /secrets, GET /secrets, GET /secrets/{secretID}, POST /secrets/{secretID}, DELETE /secrets/{secretID}) and a tenant-scoped, encrypted durable secret store (pkg/secret) to cubepod-gateway. It bridges E2B’s Secret.create / Secret.fill('name') (${e2b.secrets.<name>}) workflow directly into CubePod’s existing Zero-Trust Egress Vault (pkg/egress.Vault) and transparent mTLS egress sidecar (cubepod-egress).
Real secret values remain strictly write-only at the control plane and never enter the sandbox Pod, PodBox CRD, container environment variables, policy.json, or logs. Instead, references to stored secrets are resolved at POST /sandboxes into per-sandbox cpodref_ placeholders scoped to explicit destination SNI hosts and HTTP headers, and swapped in-flight by cubepod-egress over per-sandbox mTLS.
Motivation
Section titled “Motivation”CubePod already implements zero-trust egress secret swapping via pkg/egress.Vault (cpodref_ placeholders) and extension.secretBindings (CEP-0005, CEP-0009, CEP-0020). However, two gaps prevent drop-in compatibility with official E2B SDKs (e2b Python/TS Secret / AsyncSecret classes) and production multi-sandbox secret management:
- Missing
/secretsCRUD Surface: Official E2B clients expect/secretsendpoints to store, list, rotate (versionedPOST /secrets/{secretID}), and delete tenant secrets once, rather than embeddingrawSecretin everyPOST /sandboxespayload. - Missing
${e2b.secrets.<name>}Resolution & E2Bnetwork.rulesTransform Support: E2B SDKs reference stored secrets viaSecret.fill('name')(${e2b.secrets.<name>}) insidenetwork.rules.<host>.transform.headers(and/orextension.secretBindings). CubePod’s gateway andcubepod-egressdo not yet resolve tenant secret references intocpodref_placeholders or applytransform.headerson outbound HTTPS requests.
- Official E2B
/secretsWire Compatibility: ImplementPOST /secrets,GET /secrets(withlimit+nextToken/X-Next-Tokenpagination),GET /secrets/{secretID},POST /secrets/{secretID}(versioned rotation), andDELETE /secrets/{secretID}matching E2B OpenAPI semantics. - Strict Write-Only & Zero-Trust Invariant: Never return plaintext secret values from any API, error, log,
PodBoxCRD, or sandbox process environment. Onlycpodref_<32-char-base32>placeholders may cross intoPodBox.Spec.NetworkPolicy,policy.json, or sandboxenvVars. - Durable, Encrypted, Tenant-Isolated Storage (
pkg/secret):- Tenant isolation keyed by
sha256(managementAPIKey)(matchingpkg/volume). - AES-256-GCM encrypted file-backed store for local Docker/dev and single-gateway PVC deployments, plus a Kubernetes
Secretbackend for multi-replica HA gateways (CEP-0022).
- Tenant isolation keyed by
- Seamless
egress.VaultIntegration:- Resolve
${e2b.secrets.<name>}inPOST /sandboxes(network.rulesheader transforms) andsecretName/secretIDinextension.secretBindingsinto per-sandboxcpodref_bindings inegress.Vault. - Propagate secret rotation (
POST /secrets/{secretID}) and revocation (DELETE /secrets/{secretID}) to active sandboxegress.Vaultbindings in real time.
- Resolve
- Dual-Runtime Parity: Support both local Docker/loopback mode and Production Kubernetes (
Kata + CSI Block + cubepod-egresssidecar).
Non-Goals
Section titled “Non-Goals”- Reading back plaintext secret values via any API or CLI command (
write-onlyby design). - Injecting plaintext secrets into sandbox
/etc/environment,/proc/<pid>/environ, or containerenv(explicitly forbidden by CubePod Security Invariant #3). - Plaintext HTTP secret injection (secrets are injected strictly into outbound HTTPS requests after TLS SNI verification; plain HTTP requests fail closed).
- Replacing
pkg/egresscpodref_mTLS exchange protocol orextension.secretBindingsbackwards compatibility.
Proposal
Section titled “Proposal”User Stories
Section titled “User Stories”- Story 1 (E2B SDK Developer): As an agent developer using the official E2B SDK, I call
await Secret.create('openai_api_key', 'sk-...')once, then create sandboxes referencingSecret.fill('openai_api_key')innetwork.rules['api.openai.com']header transforms. My sandbox code callshttps://api.openai.com/v1/...without ever seeingsk-...in its environment or memory. - Story 2 (Security Administrator — Rotation & Revocation): As a platform operator, when an API key is compromised, I call
POST /secrets/{secretID}to rotate it (incrementingversionto2) orDELETE /secrets/{secretID}to revoke it. Running sandboxes immediately use the new value (or are denied with403on exchange) without restarting MicroVMs. - Story 3 (Multi-Tenant Isolation): As Tenant A, listing
GET /secretsor passingSecret.fill('tenant_b_secret')inPOST /sandboxesnever sees or resolves Tenant B’s secrets.
Notes / Constraints / Caveats
Section titled “Notes / Constraints / Caveats”- Header Injection & Hop-by-Hop Safety:
- Secret values used in HTTP header transforms MUST pass
httpguts.ValidHeaderFieldValue(rejecting\r,\n, and\x00to prevent CRLF injection). - Header names in
network.rules.*.transform.headersMUST passhttpguts.ValidHeaderFieldName, and sensitive routing/hop-by-hop headers (Host,Content-Length,Transfer-Encoding,Connection,Proxy-Authorization) MUST be rejected (400 Bad Request). serveMITMTLSMUST applyHeaderTransformsviaHeader.Set(overwriting any sandbox-supplied header of the same name rather thanHeader.Add) before runningreplaceHeaderPlaceholders.
- Secret values used in HTTP header transforms MUST pass
- Size Limit: Maximum secret value size is
64 KiB(65536UTF-8 bytes), matching E2B’s documented limit. - Egress Sidecar Dependency & WarmPool Exclusion:
- On Kubernetes Kata pods,
transform.headersandcpodref_substitution requireEgress.Enabled=true(cubepod-egresssidecar with selective HTTPS MITM enabled, per CEP-0020). If a sandbox requests${e2b.secrets.<name>}header transforms when the template/cluster has egress sidecar disabled,POST /sandboxesfails closed (400 Bad Requestor501 Not Implemented). - Sandboxes with
network.rules/HeaderTransforms(or non-emptyNetworkPolicy) MUST skip WarmPool auto-claim (matchinglen(plannedMounts) > 0inpkg/gateway/sandboxes.go), because WarmPool pods already runcubepod-egresswith an empty{}policy.jsonand do not receiveSIGHUPon claim.
- On Kubernetes Kata pods,
- Dual-Runtime Behavior:
- Production Kubernetes (
Kata + CSI Block): Tenant secrets are stored encrypted in KubernetesSecretobjects (or encrypted PVC store) in the CubePod namespace.POST /sandboxesbindscpodref_placeholders inegress.Vaultbefore callingreconciler.Reconcile, writing onlycpodref_intoPodBox.Spec.NetworkPolicy.HeaderTransforms->policy.json.cubepod-egressinside the Pod injects thecpodref_header on matching HTTPS hosts and exchanges it over mTLS withcubepod-gateway. - Local Development (
Docker + Loopback ext4): Tenant secrets are stored in<gateway-data-dir>/secrets/secrets.enc.json(AES-256-GCM,0600), and/secretsCRUD +POST /sandboxes${e2b.secrets.<name>}resolution +egress.Vaultoperate identically. Note thatpkg/runtime/docker.godoes not currently inject acubepod-egresssidecar into Docker containers; Alpha local E2E verification exercisesegress.EgressProxy+Vaultdirectly in-process against mock HTTPS upstreams.
- Production Kubernetes (
Risks and Mitigations
Section titled “Risks and Mitigations”| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
Plaintext secret leaked in gateway logs, API errors, or PodBox CRD |
Low | Critical | Dedicated SecretValue wrapper type implementing fmt.Stringer / slog.LogValuer returning "[REDACTED]"; PodBox CRD only ever receives cpodref_ tokens. |
Cross-tenant secret access via secretID guessing or Secret.fill(name) |
Low | Critical | All /secrets routes and POST /sandboxes secret lookups require matching tenantHash = sha256(APIKey). Non-owned secretID returns 404 Not Found. |
| CRLF / header smuggling via malformed secret value or header name | Medium | High | Validate header names (ValidHeaderFieldName), deny hop-by-hop headers, reject \r/\n/\x00 in values, and use Header.Set in serveMITMTLS. |
Multi-replica gateway HA desync (egress.Vault is in-memory) |
Medium | High | PodBox.Spec.NetworkPolicy.SecretRefBindings stores non-secret {placeholder, tenantHash, secretID, allowedHosts, allowedHeaders} metadata so any replica can re-hydrate Vault on cache miss from PodBox + KubeSecretBackend. KubeSecretBackend watches K8s Secret MODIFIED/DELETED events to propagate rotation/revocation to all replicas. |
mTLS header spoofing on /internal/v1/egress/exchange |
Low | Critical | In production K8s mode (ca != nil), /internal/v1/egress/exchange MUST be served on a tls.RequireAndVerifyClientCert mTLS listener (or reject non-mTLS header/body sandboxID fallbacks). |
Lost KEK (CUBEPOD_SECRETS_KEY) on restart in file-store mode |
Low | High | Production Helm chart provisions/mounts a persistent K8s Secret for CUBEPOD_SECRETS_KEY; local dev auto-generates and persists <dir>/.kek with 0600 permissions if unset. |
Design Details
Section titled “Design Details”API / CRD / Wire Changes
Section titled “API / CRD / Wire Changes”1. Official E2B /secrets REST Endpoints (pkg/gateway/secrets.go & api/openapi/cubepod-v1.yaml)
Section titled “1. Official E2B /secrets REST Endpoints (pkg/gateway/secrets.go & api/openapi/cubepod-v1.yaml)”All /secrets endpoints require X-API-Key (via requireManagementKey(w, r)), scoped to tenantHash = sha256(apiKey).
-
POST /secrets— Create a new secret- Request:
{"name": "openai_api_key","value": "sk-proj-..."}
- Validation:
namematches^[a-zA-Z0-9_.-]{1,128}$;valueis non-empty and<= 65536bytes; tenant secret count< MaxSecretsPerTenant(default100,429 Too Many Requestsif exceeded); duplicatenamefor same tenant returns409 Conflict. - Response (
201 Created):{"id": "sec_01j9x8k2m4n6p8q0r2s4t6v8w0","name": "openai_api_key","version": 1,"createdAt": "2026-11-15T18:35:00Z"}
- Request:
-
GET /secrets— List secrets (metadata only, nevervalue)- Query params:
limit(1..100, default100),nextToken(opaque base64 cursor). - Response (
200 OK, optionalX-Next-Tokenheader if truncated):[{"id": "sec_01j9x8k2m4n6p8q0r2s4t6v8w0","name": "openai_api_key","version": 1,"createdAt": "2026-11-15T18:35:00Z"}]
- Query params:
-
GET /secrets/{secretID}— Get secret metadata- Accepts
secretID(sec_...); returns404 Not Foundif missing or owned by another tenant. - Response (
200 OK):SecretInfo(id,name,version,createdAt, optionalupdatedAt).
- Accepts
-
POST /secrets/{secretID}— Rotate / update secret value- Request:
{ "value": "sk-proj-new..." } - Increments
version(version + 1), updatesupdatedAt, persists encrypted envelope, and updates all activeegress.Vaultbindings linked to(tenantHash, secretID). - Response (
200 OK): updatedSecretInfo.
- Request:
-
DELETE /secrets/{secretID}— Delete secret- Removes secret from durable store and immediately revokes all active
egress.Vaultbindings linked to(tenantHash, secretID)so in-flight sandboxes can no longer exchange itscpodref_token. - Response (
204 No Content).
- Removes secret from durable store and immediately revokes all active
2. POST /sandboxes Integration (pkg/gateway/sandboxes.go)
Section titled “2. POST /sandboxes Integration (pkg/gateway/sandboxes.go)”Support two seamless ways to reference /secrets:
-
Official E2B
network.rulesHeader Transforms:{"templateID": "base","network": {"allowOut": ["api.stripe.com"],"denyOut": ["0.0.0.0/0"],"rules": {"api.stripe.com": [{"transform": {"headers": {"Authorization": "Bearer ${e2b.secrets.stripe_api_key}"}}}]}}}- Request Parsing & CIDR Mapping (
createSandboxReq.Network):- Add
Network *e2bNetworkSpec(allowOut,denyOut,rules) tocreateSandboxReqinpkg/gateway/sandboxes.goand updateapi/openapi/cubepod-v1.yaml. - Map
"0.0.0.0/0"or"*"indenyOuttoDefaultAction: v1alpha1.NetworkDefaultDeny(and inallowOuttov1alpha1.NetworkDefaultAllow), rather than passing"0.0.0.0/0"intoDenyHosts(sinceegress.MatchHostperforms hostname/wildcard matching, not CIDR matching). Map remaining host/wildcard entries intoAllowHosts/DenyHostsand orderedEgressrules, and ensure every host present innetwork.rulesis allowed byCheckEgressAllowed.
- Add
- Placeholder Resolution, Aggregation & Ordering:
- The gateway parses all
${e2b.secrets.<name>}tokens innetwork.rules[host][].transform.headers, looks up<name>inSecretStorefortenantHash, and mints per-sandboxcpodref_<32-char-base32>placeholders vias.vault.BindManagedSecret(...). - Multiple
${e2b.secrets.<name>}tokens within a single header value (e.g.,"${e2b.secrets.user}:${e2b.secrets.pass}") mint distinctcpodref_placeholders; if the same secret is referenced across multiple rules/headers in one sandbox,BindManagedSecreteither mints a per-rule placeholder or aggregatesAllowedHosts/AllowedHeaders. - Critical Ordering & Rollback: Secret resolution and
s.vault.BindManagedSecret(...)MUST execute befores.reconciler.Reconcile(ctx, id, spec)(and skip WarmPool auto-claim whenevernetwork/HeaderTransformsis non-empty) soPodBox.Spec.NetworkPolicyand<box>-egress-mtls(policy.json) receive the mintedcpodref_placeholders. IfReconcileorstartRuntimeSandboxfails,rollbackCreatedSandboxMUST calls.vault.RevokeSandbox(id). - Update
cloneNetworkPolicyinpkg/gateway/sandboxes.goto deep-copyHeaderTransforms(includingHeadersmaps) andSecretRefBindings.
- The gateway parses all
- Request Parsing & CIDR Mapping (
-
Extended
extension.secretBindingsReference:- Allow
secretNameorsecretIDas an alternative torawSecretinextension.secretBindingsso CubePod-native clients can also reference stored/secretswithout transmitting plaintext onPOST /sandboxes.
- Allow
3. CRD & Egress Sidecar Policy (api/v1alpha1/types.go, cubepod.io_podboxes.yaml & pkg/egress/proxy.go)
Section titled “3. CRD & Egress Sidecar Policy (api/v1alpha1/types.go, cubepod.io_podboxes.yaml & pkg/egress/proxy.go)”Add HeaderTransforms and SecretRefBindings (non-secret metadata for HA cache-miss re-hydration) to v1alpha1.NetworkPolicySpec in api/v1alpha1/types.go and update the CRD OpenAPI v3 schema in deploy/helm/cubepod/crds/cubepod.io_podboxes.yaml (under spec.properties.networkPolicy.properties) so Kubernetes does not prune these fields:
// HeaderTransformRule injects or rewrites HTTP headers on outbound HTTPS requests matching Host.// Header values must contain only cpodref_ placeholders, never plaintext secrets.type HeaderTransformRule struct { Host string `json:"host"` Headers map[string]string `json:"headers"`}
// SecretRefBinding records non-secret metadata on PodBox so any HA gateway replica// can re-hydrate egress.Vault from KubeSecretBackend on cache miss.type SecretRefBinding struct { Placeholder string `json:"placeholder"` TenantHash string `json:"tenantHash"` SecretID string `json:"secretID"` AllowedHosts []string `json:"allowedHosts"` AllowedHeaders []string `json:"allowedHeaders"`}Added to NetworkPolicySpec:
HeaderTransforms []HeaderTransformRule `json:"headerTransforms,omitempty"`SecretRefBindings []SecretRefBinding `json:"secretRefBindings,omitempty"`In pkg/egress/proxy.go (serveMITMTLS and shouldMITM):
- In
serveMITMTLS, whenpolicy.HeaderTransformsmatches the TLS SNItargetHost(viaMatchHost), apply configured headers viareq.Header.Set(k, v)(overwriting any sandbox-supplied header of the same name) beforereplaceHeaderPlaceholders(...)runs, socpodref_is exchanged over mTLS for the real secret in flight. - Update
shouldMITM(policy, sniHost)to returntruewhenhostMatchesHeaderTransform(policy, sniHost)is true, and gate the fallbackhasExchangecheck onlen(policy.SecretBindings) == 0 && len(policy.HeaderTransforms) == 0so hosts without transforms or secret bindings are not unnecessarily MITM’d.
4. Durable Encrypted Store (pkg/secret/) & HA Sync
Section titled “4. Durable Encrypted Store (pkg/secret/) & HA Sync”New package pkg/secret:
Storeinterface / implementation with:FileBackend: AES-256-GCM encrypted JSON envelope (nonce || ciphertext) stored atomically (temp+fsync+rename, mode0600) under<dataDir>/secrets/secrets.enc.json.KubeSecretBackend: Stores each secret as a KubernetesSecret(cubepod-e2b-sec-<id>) withData["value"],Data["meta"], and labelscubepod.io/resource=e2b-secret,cubepod.io/tenant=<tenantPrefix>for HA multi-replica gateway deployments.
egress.Vault& HA synchronization enhancements:- Track optional
(TenantHash, SecretID)metadata onSecretBindingsoVault.UpdateSecretByRef(tenantHash, secretID, newRawSecret)andVault.RevokeSecretByRef(tenantHash, secretID)update/revoke all live sandboxes using that secret in O(active bindings). - In HA K8s mode (
KubeSecretBackend), gateway replicas watch KubernetesSecretMODIFIED/DELETEDevents (labelcubepod.io/resource=e2b-secret) to propagate rotations and revocations across all replicas, andVaultExchangeHandlerre-hydrates cache misses usingPodBox.Spec.NetworkPolicy.SecretRefBindings+KubeSecretBackend.
- Track optional
State Machine & Data Flow
Section titled “State Machine & Data Flow”sequenceDiagram
autonumber
actor User as E2B Client / SDK
participant GW as cubepod-gateway
participant Store as pkg/secret Store (AES-GCM / K8s Secret)
participant Vault as egress.Vault (in-memory)
participant Egress as cubepod-egress (Pod Sidecar)
participant Upstream as api.stripe.com (HTTPS)
User->>GW: POST /secrets {name: "stripe_key", value: "sk_live_..."}
GW->>Store: Encrypt & persist (tenantHash, id="sec_...", v=1)
GW-->>User: 201 {id, name, version: 1, createdAt}
User->>GW: POST /sandboxes {network.rules["api.stripe.com"].transform.headers: {"Authorization": "Bearer ${e2b.secrets.stripe_key}"}}
GW->>Store: Resolve (tenantHash, "stripe_key")
GW->>Vault: BindManagedSecret(sandboxID, secID, rawSecret, ["api.stripe.com"], ["Authorization"])
Vault-->>GW: placeholder = "cpodref_XYZ..."
Note over GW,Egress: PodBox.Spec.NetworkPolicy.HeaderTransforms carries ONLY "Bearer cpodref_XYZ..."
GW-->>User: 201 Sandbox {sandboxID}
Note over Egress: Sandbox sends HTTPS request to api.stripe.com
Egress->>Egress: Selective MITM + inject HeaderTransform ("Bearer cpodref_XYZ...")
Egress->>GW: mTLS POST /internal/v1/egress/exchange (CN=sandboxID, cpodref_XYZ..., host=api.stripe.com, header=Authorization)
GW->>Vault: Exchange(...) -> "sk_live_..."
Vault-->>GW: "sk_live_..."
GW-->>Egress: 200 {secret: "sk_live_..."}
Egress->>Upstream: HTTPS GET /v1/charges (Authorization: Bearer sk_live_...)Boundary Conditions & Failure Recovery
Section titled “Boundary Conditions & Failure Recovery”- Unknown Secret Reference in
POST /sandboxes: If${e2b.secrets.missing}does not exist fortenantHash,POST /sandboxesfails immediately with400 Bad Request(unknown secret "missing") before creating anyPodBox, PVC, or container. - Secret Rotated While Sandbox Is Running:
POST /secrets/{secretID}updates the encrypted store first; on success, it callsvault.UpdateSecretByRef(...). If the gateway restarts in K8s HA mode,VaultExchangeHandleror sandbox resync re-hydratesSecretBindings fromPodBoxannotations/spec +SecretStore(or fails closed403if the secret was deleted). - Secret Deleted While Sandbox Is Running:
DELETE /secrets/{secretID}removes the secret fromStoreand callsvault.RevokeSecretByRef(...). Subsequent outbound HTTPS requests carryingcpodref_fail closed (403 Forbiddenfrom exchange,502/403to the sandbox caller) without leaking any credential. - Gateway Crash During Write:
FileBackendwrites tosecrets.enc.json.tmp, callsSync(), and atomicallyRename()s oversecrets.enc.json; partial writes never corrupt existing secrets.
Test Plan
Section titled “Test Plan”- Unit Tests (
go test -race):pkg/secret: CRUD, duplicate name409, tenant isolation, AES-256-GCM encryption on disk (asserting plaintext never appears in the file on disk), corruption handling, quota enforcement (100), atomic write,SecretValueredaction infmt/slog.pkg/egress:BindManagedSecret,UpdateSecretByRefrotation,RevokeSecretByRefrevocation,HeaderTransformsinjection in selective HTTPS MITM (proxy_test.go), CRLF rejection, plain HTTP fail-closed.pkg/gateway:/secretsHTTP handlers (POST,GETlist +limit/nextToken/X-Next-Tokenpagination,GET /{id},POST /{id}rotation,DELETE /{id}), cross-tenant404,POST /sandboxesresolving${e2b.secrets.<name>}intocpodref_without leaking raw values intoPodBoxorSandboxRecord.
- Integration / E2E Tests (
make test-e2e):- Create secret -> boot sandbox with
network.rulesheader transform -> verify sandbox env has no secret -> make HTTPS request viaEgressProxy-> verify mock upstream receives real secret -> rotate secret viaPOST /secrets/{id}-> verify next request receives new secret -> delete secret viaDELETE /secrets/{id}-> verify next request is denied (403).
- Create secret -> boot sandbox with
Graduation Criteria
Section titled “Graduation Criteria”- Alpha (
v0.5):E2BSecretsAPIfeature gate / gateway config enabled when secrets store is configured.- All 5
/secretsendpoints +pkg/secretAES-256-GCM file backend +egress.Vaultrotation/revocation +HeaderTransformsMITM injection implemented and passingmake testandmake test-e2e.
- Beta (
v0.6):KubeSecretBackendenabled by default for multi-replica K8s gateways with Helm chart RBAC andCUBEPOD_SECRETS_KEYSecret management.- Python & TypeScript SDK helpers (
Secret/AsyncSecret,Secret.fill) verified against live gateway.
- Stable (
v1.0):- Production bake time in CubePlex with Prometheus alerting on secret exchange errors and store latency.
Upgrade / Downgrade Strategy
Section titled “Upgrade / Downgrade Strategy”- Upgrade: Additive CRD field
NetworkPolicySpec.HeaderTransforms(omitempty) and new/secretsroutes; zero impact on existing/workspaceor/rootfs-upperPVCs or runningPodBoxresources. - Downgrade: Removing
E2BSecretsAPIdisables/secretsroutes (501 Not Implementedor404); existingPodBoxobjects withoutheaderTransformscontinue running unchanged.
Production Readiness Review Questionnaire (PRR)
Section titled “Production Readiness Review Questionnaire (PRR)”- Feature Enablement and Rollback: Controlled via
CUBEPOD_SECRETS_DIR/CUBEPOD_SECRETS_BACKEND(file|k8s|disabled) andCUBEPOD_SECRETS_KEY. Disabling stops new/secretsoperations without touching/workspaceorrootfs-upperPVCs. - Rollout, Upgrade and Rollback Planning: Existing
PausedorRunningsandboxes withoutheaderTransformsare unaffected. Rolling gateways withKubeSecretBackendpreserves secret availability across replicas. - Monitoring Requirements:
cubepod_secrets_operations_total{op,status}cubepod_secrets_store_count- Existing
cubepod_egress_secret_exchanges_total{result}(ok,denied,error).
- Dependencies: Existing
cubepod-egresssidecar (CEP-0020) for Kubernetes Kata pods;crypto/aes+crypto/cipher(Go stdlib) for envelope encryption. - Scalability:
MaxSecretsPerTenant(default100) bounds per-tenant storage and list responses;GET /secretssupports cursor pagination (limit <= 100). - Troubleshooting: Operators inspect
cubepod_secrets_operations_totalandcubepod-egressstructured logs (placeholder=cpodref_..., never raw values) to diagnose403exchange denials (host mismatch, header mismatch, or revoked secret).
Implementation History
Section titled “Implementation History”2026-11-15: CEP-0027 drafted and markedimplementable.
Drawbacks
Section titled “Drawbacks”- Requires
cubepod-egressselective HTTPS MITM to be enabled for pods that usenetwork.rules.*.transform.headers, adding one TLS termination hop for those specific matched hosts. - File-backed encryption requires operators to manage (or persist)
CUBEPOD_SECRETS_KEYcleanly across gateway upgrades in non-K8s-Secret deployments.
Alternatives
Section titled “Alternatives”- Inject
/secretsdirectly as container environment variables: Rejected. Violates CubePod Security Invariant #3 (zero-trust egress secret safety); untrusted LLM-generated code can trivially exfiltrateos.environ. - Pass
rawSecretinsidePodBox.Specor KubernetesSecretmounted into the Kata Pod: Rejected. If the guest MicroVM is compromised orpodd/ container reads mounted files, the raw secret is exposed inside the untrusted boundary. - Store tenant secrets only in
egress.Vaultmemory: Rejected. Secrets would be lost on every gateway restart and could not be shared across HA gateway replicas.