debug-openshell-cluster
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseDebug OpenShell Gateway Deployment
调试 OpenShell Gateway 部署
Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, the experimental VM driver, or an operator-managed out-of-tree compute driver.
Use first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver: , , /, or VM driver logs.
openshelldockerpodmankubectlhelm诊断 gateway 及其选定的 compute platform。不要假定 OpenShell 会置备 Kubernetes 或运行 k3s container。OpenShell 面向的是一个可达的 gateway endpoint,其后端可以是 Docker、Podman、Kubernetes、实验性 VM driver,或由 operator 管理的 out-of-tree compute driver。
首先使用 识别当前活动的 endpoint。然后使用与 gateway 的 compute driver 相匹配的 platform 工具:、、/,或 VM driver 日志。
openshelldockerpodmankubectlhelmOverview
概览
The target deployment flow is:
- Operator starts or deploys the gateway with system packages, systemd, or Helm. The CLI does not start, stop, or destroy gateway services.
- Operator configures the compute driver.
- Operator provides the CLI and supervisor authentication material required by the deployment mode: edge or OIDC user auth, optional CLI mTLS, and gateway-minted sandbox JWTs.
- The CLI registers a reachable gateway endpoint with .
openshell gateway add - The gateway creates sandboxes through the selected compute driver.
The composition crate explicitly installs its compiled
Docker, Podman, Kubernetes, and VM registrations at startup;
does not link compute-driver crates. Custom gateway binaries may include a
subset of those registrations. With no configured driver, the gateway probes only
installed registrations in priority order (Kubernetes, Podman, then Docker);
VM has no probe and remains opt-in. Confirm the binary's registered drivers
when auto-detection reports that no suitable driver is available. If
configuration selects a driver that was not compiled in, the gateway treats
the name as an external driver and reports a missing unless an
endpoint is configured.
openshell-gatewayopenshell-serversocket_pathOn Windows, custom binaries can include MXC independently. Registrations for
Docker, Podman, Kubernetes, and VM are rejection stubs when included; they do
not enable those runtimes on Windows.
See the compute driver reference
for selective-build options and external-driver configuration.
For local evaluation only, TLS may be disabled and the gateway can be reached through .
http://127.0.0.1:<port>目标部署流程如下:
- Operator 使用 system packages、systemd 或 Helm 启动或部署 gateway。CLI 不负责启动、停止或销毁 gateway services。
- Operator 配置 compute driver。
- Operator 根据部署模式提供 CLI 和 supervisor 所需的认证材料:edge 或 OIDC 用户认证、可选的 CLI mTLS,以及由 gateway 签发的 sandbox JWT。
- CLI 使用 注册一个可达的 gateway endpoint。
openshell gateway add - Gateway 通过选定的 compute driver 创建 sandbox。
openshell-gatewayopenshell-serversocket_path在 Windows 上,自定义 binary 可以独立包含 MXC。包含 Docker、Podman、Kubernetes 和 VM 注册时,它们只是 rejection stub,并不会在 Windows 上启用这些 runtime。
有关 selective-build 选项和 external driver 配置,请参阅 compute driver reference。
仅限本地评估时,可以禁用 TLS,并通过 访问 gateway。
http://127.0.0.1:<port>Prerequisites
前置条件
- The CLI must be available for endpoint checks.
openshell - Know the active gateway name and endpoint, or be able to inspect local gateway metadata.
- Know the compute platform: Docker, Podman, Kubernetes, VM, or an out-of-tree driver.
- For Kubernetes: must target the cluster that hosts OpenShell and Helm version 3 or later must be available.
kubectl - For Docker or Podman: the runtime socket must be reachable from the gateway host.
Use and nested output as the authority for the installed CLI version. Use the published installation guide, compute-driver reference, gateway configuration reference, and Kubernetes setup guide as the authority for deployment and configuration behavior.
openshell --help--help- 必须可以使用 CLI 检查 endpoint。
openshell - 知道当前活动的 gateway 名称和 endpoint,或者能够检查本地 gateway metadata。
- 知道 compute platform:Docker、Podman、Kubernetes、VM 或 out-of-tree driver。
- 对于 Kubernetes:必须指向承载 OpenShell 的集群,并且可以使用 Helm 3 或更高版本。
kubectl - 对于 Docker 或 Podman:runtime socket 必须可从 gateway host 访问。
应以 及嵌套的 输出作为已安装 CLI 版本的权威依据。部署和配置行为应以已发布的 installation guide、compute-driver reference、gateway configuration reference 和 Kubernetes setup guide 为准。
openshell --help--helpWorkflow
工作流
Run diagnostics in order and stop once the root cause is clear.
按顺序运行诊断命令,确定根本原因后即可停止。
Step 1: Check CLI Reachability
第 1 步:检查 CLI 可达性
bash
openshell gateway list --output json
openshell gateway info
openshell statusFor a one-off endpoint check that bypasses stored gateway selection and metadata:
bash
openshell --gateway-endpoint <url> statusCommon findings:
- : register one with
No active gateway.openshell gateway add <endpoint> - Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active.
- TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener.
- from an edge or OIDC gateway: refresh stored credentials with
Unauthenticated, then retry. Useopenshell gateway login [name]only when intentionally clearing local credentials.gateway logout - A direct development endpoint with a private or self-signed certificate can be isolated with ; do not persist or recommend insecure verification for shared gateways.
--gateway-endpoint <url> --gateway-insecure
bash
openshell gateway list --output json
openshell gateway info
openshell status如需执行一次性 endpoint 检查,并绕过已存储的 gateway 选择和 metadata,可使用:
bash
openshell --gateway-endpoint <url> status常见发现:
- :使用
No active gateway注册一个 gateway。openshell gateway add <endpoint> - Connection refused:gateway process 未运行、service 暴露方式错误,或 port-forward/proxy 未运行。
- TLS/certificate 错误:endpoint scheme 或 trust chain 错误、本地 mTLS bundle 与 gateway CA 不匹配,或 TLS termination 与 gateway listener 不匹配。
- edge 或 OIDC gateway 返回 :使用
Unauthenticated刷新已存储的凭据,然后重试。只有在明确需要清除本地凭据时才使用openshell gateway login [name]。gateway logout - 若要隔离使用私有或自签名证书的直接开发 endpoint,可以使用 ;不要为共享 gateway 持久化或建议使用不安全的证书验证方式。
--gateway-endpoint <url> --gateway-insecure
Step 2: Identify the Compute Platform
第 2 步:确定 Compute Platform
Use gateway metadata, deployment values, or the user's setup notes to identify the driver.
| Platform | Primary checks |
|---|---|
| Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. |
| Podman | Podman socket, rootless networking, sandbox containers, image pulls. |
| Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. |
| OpenShift | Same as Kubernetes, plus SecurityContextConstraints (SCCs) and, for external access, an OpenShift |
| VM | VM driver logs, rootfs availability, host virtualization support. |
| Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. |
使用 gateway metadata、deployment values 或用户提供的设置说明确定 driver。
| Platform | 主要检查项 |
|---|---|
| Docker | Gateway process 日志、Docker daemon 健康状态、sandbox containers、镜像拉取。 |
| Podman | Podman socket、rootless networking、sandbox containers、镜像拉取。 |
| Kubernetes | Helm release、gateway workload、service、secrets、sandbox pods、events。 |
| OpenShift | 与 Kubernetes 相同,另外检查 SecurityContextConstraints(SCC),以及外部访问所需的 OpenShift |
| VM | VM driver 日志、rootfs 可用性、host virtualization 支持情况。 |
| Extension | External driver process、Unix socket 所有权/权限、已配置的 driver 名称、capability handshake、gateway 日志。 |
Step 3: Check Gateway Startup Dependencies
第 3 步:检查 Gateway 启动依赖
Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready.
For out-of-tree compute drivers, confirm the selected driver name and socket agree across CLI flags or , and that the operator-owned driver is running before the gateway starts:
gateway.tomlbash
rg -n '^version|compute_driver|socket_path|guest_tls_' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200Gateway configuration requires , a singular
selector, and driver-owned settings under
. The gateway rejects legacy and
selectors rather than silently migrating them. One valid, nonempty
value remains a deprecated environment-only alias when the
canonical selector is absent; the gateway selects that driver with a warning.
Empty, invalid, comma-delimited, or conflicting values fail startup. Homebrew
and RPM package startup migrates only exact package-generated v1 defaults. If
an upgraded package still reports an unsupported version, inspect the active prefix or ; an edited v1 file must
follow the published schema-v2 migration steps and must not be overwritten.
Guest TLS CA, certificate, and key paths are the exception to driver ownership:
configure the complete bundle under , and the gateway
injects it only into the selected local driver. TLS-enabled Docker, Podman, and
VM drivers fail startup when neither those paths nor the package-managed local
bundle is available; Kubernetes projects its bundle through a Secret.
[openshell] version = 2compute_driver[openshell.drivers.<name>]compute_drivers--driversOPENSHELL_DRIVERS~/.config/openshell/gateway.toml[openshell.gateway]Custom names use . A launch-time override may also use , , , or ; the endpoint then takes precedence over built-in construction. First-party standalone drivers require the socket parent directory to be owned by the driver's effective UID, force its mode to , create the socket with mode , and accept only peers with that same UID. Check the parent and socket separately with ; a gateway running under a different UID cannot connect even when filesystem permissions or group membership would otherwise allow it. Operator-supplied drivers must provide equivalent access control appropriate to their implementation. Check gateway logs for connection errors, failures, or an unexpected advertised driver name. The advertised name is diagnostic metadata; negotiated features control optional behavior. The gateway does not create or supervise operator-supplied driver processes or sockets.
[openshell.drivers.<name>].socket_path--compute-driver-socketdockerpodmankubernetesvm07000600statGetCapabilitiesFor configured gateway interceptors, inspect , their Unix or network endpoints, and gateway startup logs:
[[openshell.gateway.interceptors]]bash
rg -n 'interceptors|provider_profile_sources|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|binding_policy|failure_policy|gateway_jwt' /etc/openshell/gateway.toml
stat /run/openshell/interceptors/<name>.sock
journalctl -u <interceptor-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200The gateway calls each interceptor's RPC and validates its manifest at startup. Check for unreachable endpoints, invalid RPC/phase bindings, strict or mismatches, and bindings that resolve to . If gateway JWT signing is enabled, authenticated network interceptors require HTTPS and a valid bearer token; check the private CA path, endpoint hostname, expected audience, issuer, , and interceptor logs for token rejection. explicitly preserves unauthenticated plaintext behavior. If names an interceptor, that interceptor must advertise provider-profile capability and return a valid, duplicate-free catalog. A selected interceptor-only source is authoritative; include or sources explicitly when composition is intended.
Describeallowlistexactpost_commitfail_closedkidallow_insecure_transport = trueprovider_profile_sourcesbuiltinuserFor operator-run supervisor middleware, inspect , service reachability, and both gateway and supervisor logs:
[[openshell.supervisor.middleware]]bash
rg -n 'supervisor|middleware|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|max_payload_bytes|timeout|gateway_jwt' /etc/openshell/gateway.toml
journalctl -u <middleware-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
openshell logs <sandbox-name> --tail --source sandboxThe middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if is unavailable, a manifest exposes duplicate operation/phase bindings, the registration claims the reserved namespace, or payload and timeout limits are invalid. Supported V1 bindings are and . When gateway JWT signing is disabled, supervisors preserve the legacy unauthenticated connector and do not request extension credentials. When signing is enabled, credential acquisition and verification failures are fail closed: check HTTPS trust and hostname validation, audience and issuer agreement, the token , gateway errors, and middleware logs. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its config.
Describeopenshell/HTTP_REQUEST/PRE_CREDENTIALSWEBSOCKET_MESSAGE/PRE_CREDENTIALSkidRefreshSandboxTokennetwork_middlewaresAt request time, distinguish attachment, binding selection, coverage, denial, and failure. A host-matched HTTP-only attachment can inspect the upgrade GET but does not join the WebSocket chain; the connection proceeds under either mode and emits coverage. A selected WebSocket stage receives text messages only. Binary messages pass under both modes, emit coverage, and consume a session sequence without an RPC. An explicit result is always enforced. WebSocket preflight returns , voluntary , or authoritative ; rejects the upgrade before upstream contact under both modes. A selected-stage failure follows the policy-local : blocks the HTTP request or closes the WebSocket, while bypasses only that stage and emits a detection finding. A fail-open per-message capacity failure bypasses that message without disabling the stage. A timeout, transport failure, stream closure, missing or invalid response, duplicate or regressed sequence, or other failure that makes an established WebSocket stream unreliable disables that stage for later messages on the connection and emits . Confirm preflight, session-start, and session-end in service logs. OpenShell best-effort sends at most one session-end to each still-writable opened stage, including a preflight that terminates before session start; distinguish from . WebSocket message sequences are allocated session-wide; each stage receives a strictly increasing subset, so gaps are valid when binary messages or other units are not delivered to that stage. Zero, duplicate, or regressed sequences are protocol errors. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
on_errorbinding_not_selectedunsupported_message_typemiddleware_deniedINSPECTSKIPDENYDENYon_erroron_errorfail_closedfail_openopenshell.middleware.websocket_stage_disabledMIDDLEWARE_DENIALMIDDLEWARE_FAILUREFor network policy validation failures, first distinguish a gateway mutation
rejection from a supervisor runtime rejection. Direct policy updates,
incremental merges and approvals, provider attachments, and provider-profile
fanout are validated against the complete effective policy before persistence
when the gateway knows the affected sandbox scope. A
ambiguity response means no invalid revision or partial fanout was stored.
Supervisor validation remains defense in depth for startup, races, and policy
sources outside those mutation paths.
FAILED_PRECONDITIONRuntime rejection behavior is configured only in :
gateway.tomltoml
[openshell.gateway]
policy_validation_failure_mode = "fail_closed"The default mode deactivates the previous generation, closes
pinned relays, and quarantines new egress until a valid generation loads.
explicitly keeps the previous valid policy active; without
one it still fails closed. Restart the gateway after changing this field.
Inspect sandbox OCSF configuration and finding events for the validation
rationale, configured and effective modes, active generation, and the explicit
state.
fail_closedretain_last_validprevious_policy_active调试 compute platform 之前,应检查 gateway 日志中 listener ready 之前初始化的依赖是否存在失败。
对于 out-of-tree compute driver,请确认 CLI flag 或 中选择的 driver 名称与 socket 一致,并确保 operator 管理的 driver 在 gateway 启动前已经运行:
gateway.tomlbash
rg -n '^version|compute_driver|socket_path|guest_tls_' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200Gateway 配置要求 、单个 selector,并将 driver 自有设置放在 下。Gateway 会拒绝旧版的 和 selector,而不会静默迁移。只有一个有效且非空的 值仍可作为已弃用的仅环境变量别名使用;canonical selector 缺失时,gateway 会选择该 driver 并给出警告。空值、无效值、以逗号分隔的值或冲突值都会导致启动失败。Homebrew 和 RPM package 的启动迁移仅处理由 package 生成的精确 v1 默认配置。如果升级后的 package 仍报告不支持的 version,应检查当前 prefix 或 ;手工编辑过的 v1 文件必须遵循已发布的 schema-v2 迁移步骤,且不得被覆盖。
[openshell] version = 2compute_driver[openshell.drivers.<name>]compute_drivers--driversOPENSHELL_DRIVERS~/.config/openshell/gateway.tomlGuest TLS CA、certificate 和 key path 是 driver ownership 的例外:应在 下配置完整 bundle,gateway 仅会将其注入选定的本地 driver。启用 TLS 的 Docker、Podman 和 VM driver 在既无这些 path、也无 package 管理的本地 bundle 时会启动失败;Kubernetes 则通过 Secret 投射该 bundle。
[openshell.gateway]自定义名称使用 。启动时的 override 也可以使用 、、 或 ;此时 endpoint 的优先级高于内置构造。第一方 standalone driver 要求 socket 的父目录归 driver 的 effective UID 所有,强制将其 mode 设为 ,以 mode 创建 socket,并且只接受具有相同 UID 的 peer。应分别使用 检查父目录和 socket;以不同 UID 运行的 gateway 即使在文件权限或组成员关系本来允许的情况下也无法连接。Operator 提供的 driver 必须实施适合其实现的同等访问控制。检查 gateway 日志中的连接错误、 失败或意外声明的 driver 名称。声明的名称属于诊断 metadata;可选行为由协商后的 features 决定。Gateway 不会创建或监管 operator 提供的 driver process 或 socket。
[openshell.drivers.<name>].socket_path--compute-driver-socketdockerpodmankubernetesvm07000600statGetCapabilities对于已配置的 gateway interceptor,请检查 、其 Unix 或 network endpoint 以及 gateway 启动日志:
[[openshell.gateway.interceptors]]bash
rg -n 'interceptors|provider_profile_sources|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|binding_policy|failure_policy|gateway_jwt' /etc/openshell/gateway.toml
stat /run/openshell/interceptors/<name>.sock
journalctl -u <interceptor-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200Gateway 会在启动时调用每个 interceptor 的 RPC,并验证其 manifest。检查 endpoint 不可达、RPC/phase binding 无效、严格的 或 不匹配,以及解析为 的 binding。如果启用了 gateway JWT signing,经过认证的 network interceptor 必须使用 HTTPS 和有效的 bearer token;应检查 private CA path、endpoint hostname、预期 audience、issuer、 以及 interceptor 日志中的 token rejection。 会明确保留未经认证的明文行为。如果 指定了某个 interceptor,该 interceptor 必须声明 provider-profile capability,并返回有效且无重复项的 catalog。仅选择 interceptor 的 source 是权威来源;如果需要组合,应明确包含 或 source。
Describeallowlistexactfail_closedpost_commitkidallow_insecure_transport = trueprovider_profile_sourcesbuiltinuser对于 operator 运行的 supervisor middleware,请检查 、service 可达性以及 gateway 和 supervisor 日志:
[[openshell.supervisor.middleware]]bash
rg -n 'supervisor|middleware|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|max_payload_bytes|timeout|gateway_jwt' /etc/openshell/gateway.toml
journalctl -u <middleware-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
openshell logs <sandbox-name> --tail --source sandboxMiddleware service 必须在 gateway 之前启动,并且 gateway 与 sandbox supervisor 都能访问它。如果 不可用、manifest 暴露了重复的 operation/phase binding、注册声明了保留的 namespace,或者 payload 与 timeout limit 无效,则 gateway 启动失败。支持的 V1 binding 为 和 。禁用 gateway JWT signing 时,supervisor 会保留旧版未经认证的 connector,也不会请求 extension credential。启用 signing 后,凭据获取和验证失败均采用 fail-closed 策略:检查 HTTPS trust 与 hostname validation、audience 和 issuer 是否一致、token 、gateway 错误以及 middleware 日志。修改注册后需要重启 gateway。如果选定的实现拒绝 配置,policy update 也可能在持久化之前失败。
Describeopenshell/HTTP_REQUEST/PRE_CREDENTIALSWEBSOCKET_MESSAGE/PRE_CREDENTIALSkidRefreshSandboxTokennetwork_middlewares在请求时,应区分 attachment、binding selection、coverage、denial 和 failure。匹配 host 的纯 HTTP attachment 可以检查 upgrade GET,但不会加入 WebSocket chain;无论采用哪一种 模式,连接都会继续,并发出 coverage。被选中的 WebSocket stage 只接收 text message。Binary message 在两种模式下都会直接通过,发出 coverage,并消耗一个 session sequence,但不会产生 RPC。显式的 结果始终会被执行。WebSocket preflight 会返回 、主动 或权威性的 ;在两种 模式下, 都会在联系上游之前拒绝 upgrade。被选中的 stage 发生失败时遵循 policy-local : 会阻止 HTTP request 或关闭 WebSocket; 只会绕过该 stage 并发出 detection finding。Fail-open 模式下的单条消息 capacity 失败只会绕过该消息,不会禁用整个 stage。Timeout、transport failure、stream closure、响应缺失或无效、sequence 重复或倒退,以及其他导致已建立 WebSocket stream 不可靠的失败,都会为该连接后续消息禁用该 stage,并发出 。应在 service 日志中确认 preflight、session-start 和 session-end。OpenShell 会尽力向每个仍可写入且已打开的 stage 最多发送一次 session-end,包括在 session start 前终止的 preflight;应区分 和 。WebSocket message sequence 按 session 范围分配;每个 stage 接收严格递增的子序列,因此当 binary message 或其他单元未传递给某个 stage 时,出现间隔是正常的。Sequence 为零、重复或倒退均属于 protocol error。如果运行中的 supervisor 无法安装新的 registry,它会保留最后一个已知正常的 generation,并发出 configuration failure event。
on_errorbinding_not_selectedunsupported_message_typemiddleware_deniedINSPECTSKIPDENYon_errorDENYon_errorfail_closedfail_openopenshell.middleware.websocket_stage_disabledMIDDLEWARE_DENIALMIDDLEWARE_FAILURE对于 network policy validation failure,首先应区分 gateway mutation rejection 和 supervisor runtime rejection。当 gateway 已知受影响的 sandbox scope 时,direct policy update、incremental merge 和 approval、provider attachment 以及 provider-profile fanout,都会在持久化之前针对完整的 effective policy 进行验证。 ambiguity response 表示没有存储无效 revision,也没有执行部分 fanout。Supervisor validation 仍作为纵深防御,用于处理启动、竞态以及 mutation path 之外的 policy source。
FAILED_PRECONDITIONRuntime rejection 行为只能在 中配置:
gateway.tomltoml
[openshell.gateway]
policy_validation_failure_mode = "fail_closed"默认的 模式会停用上一 generation、关闭 pinned relay,并隔离新的 egress,直到有效的 generation 加载完成。 会明确保留上一个有效 policy;如果不存在有效 policy,仍采用 fail closed。修改此字段后应重启 gateway。检查 sandbox OCSF configuration 和 finding events,以了解 validation rationale、已配置及 effective mode、active generation 以及明确的 状态。
fail_closedretain_last_validprevious_policy_activeStep 4: Check Docker-Backed Gateways
第 4 步:检查 Docker 后端的 Gateway
bash
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
openshell statusFor Docker GPU failures, check CDI support and NVIDIA CDI discovery separately:
bash
docker info --format '{{json .CDISpecDirs}}'
docker info --format '{{json .DiscoveredDevices}}'
for dir in /etc/cdi /var/run/cdi; do
if [ -d "$dir" ]; then
find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print
else
echo "$dir missing"
fi
done
systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50
journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100When the NVIDIA Container Toolkit CDI refresh units are not enabled or no NVIDIA CDI spec has been generated, enable them and trigger a refresh:
bash
sudo systemctl enable --now nvidia-cdi-refresh.path
sudo systemctl enable --now nvidia-cdi-refresh.service
sudo systemctl restart nvidia-cdi-refresh.service
docker info --format '{{json .DiscoveredDevices}}'Common findings:
- Docker daemon unavailable: start Docker Desktop or Docker Engine.
- Gateway process stopped: inspect exit status and logs.
- Sandbox image missing or pull denied: verify image reference and registry credentials.
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI and matching
USERand/etc/passwdentries, or explicitly set both process identity fields in policy. Numeric workload identities/etc/groupthrough1are accepted; root, the invalid identity sentinel, and missing identities are rejected.4294967294 - Sandbox fails before readiness with an OCI workspace validation error: inspect the image's using the immutable image ID reported by the gateway. Empty,
WorkingDir, and explicit/use the managed/sandboxcompatibility workspace. Any other workdir must be an absolute normalized directory with no symlink components; the final policy UID, primary GID, and supplementary groups must pass the kernel's effective traverse/write checks, including POSIX ACL and LSM decisions. OpenShell does not create, chown, or chmod a non-default image workdir./sandbox - Docker also rejects an image that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the
VOLUMEbelow the workspace or remove the declaration.VOLUME - A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
- Docker driver cannot initialize because it cannot find : verify
openshell-sandbox, the sibling binary next toOPENSHELL_DOCKER_SUPERVISOR_BIN, or the configured supervisor image containsopenshell-gateway./openshell-sandbox - Sandbox never registers: check gateway logs and supervisor callback endpoint.
- On macOS, repeated messages with a Homebrew gateway bound to
Policy fetch failed after 5 attemptsindicate that the Docker[::1]:17670IPv4 route has no matching callback listener. Current releases leavehost-gatewayunset in the Homebrew config, use the built-inbind_addressprimary listener, and reuse it for authenticated sandbox callbacks. On an older release, set127.0.0.1:17670or upgrade.bind_address = "127.0.0.1:17670" - Supervisor image exits before printing : verify the configured supervisor image contains a static executable at
openshell-sandbox --version./openshell-sandbox - A sandbox with explicit endpoints fails before workload readiness: confirm the Docker or Podman driver supplied the
protocol: tcpruntime capability and inspect supervisor logs for missingpolicy-dns-transparent-tcp, synthetic-route overlap, or namespace-local DNS/TCP listener bind failures. Kubernetes, VM, sidecar, and out-of-tree drivers must reject this policy until they provide the complete substrate; use omitted protocol with an explicit proxy on those runtimes.nft - A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify contains entries such as
.DiscoveredDevices, verifynvidia.com/gpu=allor/etc/cdicontains a generated NVIDIA spec, and check that/var/run/cdiandnvidia-cdi-refresh.servicefrom NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, sonvidia-cdi-refresh.pathcan be normal after a successful run; useinactive (dead)andsystemctl statusto distinguish success from a skipped or failed refresh. Restartjournalctlto regenerate missing or stale CDI specs, then restart or reload Docker and re-checknvidia-cdi-refresh.service.docker info
During a graceful gateway restart, Docker, Podman, and VM sandboxes with
running intent should stop before the gateway exits and restart after it
returns. Check for and in gateway logs. A sandbox explicitly stopped
through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do
not follow this local gateway lifecycle. Internal and external drivers follow
the same rule: must be true for the
gateway to run shutdown and startup sweeps.
Stopped sandbox during gateway shutdownStarted sandbox during gateway startupGetCapabilities.gateway_manages_lifecyclebash
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
openshell status对于 Docker GPU 故障,应分别检查 CDI 支持和 NVIDIA CDI discovery:
bash
docker info --format '{{json .CDISpecDirs}}'
docker info --format '{{json .DiscoveredDevices}}'
for dir in /etc/cdi /var/run/cdi; do
if [ -d "$dir" ]; then
find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print
else
echo "$dir missing"
fi
done
systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50
journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100如果 NVIDIA Container Toolkit CDI refresh unit 未启用,或尚未生成 NVIDIA CDI spec,应启用它们并触发 refresh:
bash
sudo systemctl enable --now nvidia-cdi-refresh.path
sudo systemctl enable --now nvidia-cdi-refresh.service
sudo systemctl restart nvidia-cdi-refresh.service
docker info --format '{{json .DiscoveredDevices}}'常见发现:
- Docker daemon 不可用:启动 Docker Desktop 或 Docker Engine。
- Gateway process 已停止:检查 exit status 和日志。
- Sandbox 镜像缺失或拉取被拒:验证 image reference 和 registry credential。
- Sandbox 在 ready 之前因 identity-resolution error 失败:检查镜像的 OCI 以及匹配的
USER和/etc/passwd条目,或者在 policy 中明确设置两个 process identity 字段。接受/etc/group到1的数值型 workload identity;拒绝 root、无效 identity sentinel 和缺失的 identity。4294967294 - Sandbox 在 ready 之前因 OCI workspace validation error 失败:使用 gateway 报告的不可变 image ID 检查镜像的 。空值、
WorkingDir和显式的/使用受管理的/sandboxcompatibility workspace。任何其他 workdir 都必须是绝对路径下的规范目录,且不能包含 symlink component;最终 policy UID、primary GID 和 supplementary group 必须通过 kernel 的 effective traverse/write 检查,包括 POSIX ACL 和 LSM 决策。OpenShell 不会创建、chown 或 chmod 非默认镜像 workdir。/sandbox - 如果镜像的 覆盖 workdir 或其父目录,Docker 也会拒绝该镜像,因为 runtime 会在验证前屏蔽该不可变路径。应将
VOLUME移到 workspace 下方或移除该声明。VOLUME - 如果 workdir 因特殊 filesystem 或 OpenShell control-path collision 被拒绝,无法通过权限修复。应将镜像 workdir 移 away from kernel-backed mount,以及错误信息中列出的具体 supervisor、TLS、token、runtime 和 socket path。
- Docker driver 因找不到 而无法初始化:验证
openshell-sandbox、位于OPENSHELL_DOCKER_SUPERVISOR_BIN旁的同级 binary,或已配置的 supervisor image 中包含openshell-gateway。/openshell-sandbox - Sandbox 始终未注册:检查 gateway 日志和 supervisor callback endpoint。
- 在 macOS 上,如果 Homebrew gateway 绑定到 ,但反复出现
[::1]:17670,说明 DockerPolicy fetch failed after 5 attemptsIPv4 route 没有匹配的 callback listener。当前版本在 Homebrew config 中不设置host-gateway,使用内置的bind_addressprimary listener,并将它用于经过认证的 sandbox callback。旧版本应设置127.0.0.1:17670或进行升级。bind_address = "127.0.0.1:17670" - Supervisor image 在输出 之前退出:验证已配置的 supervisor image 中包含位于
openshell-sandbox --version的 static executable。/openshell-sandbox - 显式使用 endpoint 的 sandbox 在 workload ready 之前失败:确认 Docker 或 Podman driver 提供了
protocol: tcpruntime capability,并检查 supervisor 日志中缺少policy-dns-transparent-tcp、synthetic-route overlap 或 namespace-local DNS/TCP listener bind failure。在 Kubernetes、VM、sidecar 和 out-of-tree driver 提供完整 substrate 之前,必须拒绝该 policy;这些 runtime 应省略 protocol 并使用显式 proxy。nft - 如果 GPU sandbox 因 Docker 报告未发现 NVIDIA CDI device 而失败:验证 包含
.DiscoveredDevices等条目,验证nvidia.com/gpu=all或/etc/cdi中包含已生成的 NVIDIA spec,并检查 NVIDIA Container Toolkit 的/var/run/cdi和nvidia-cdi-refresh.service是否已启用且健康。该 service 是 one-shot unit,因此成功运行后显示nvidia-cdi-refresh.path可能属于正常情况;应使用inactive (dead)和systemctl status区分成功、跳过和失败。重启journalctl以重新生成缺失或过期的 CDI spec,然后重启或 reload Docker,并再次检查nvidia-cdi-refresh.service。docker info
在 gateway 平滑重启期间,具有 running intent 的 Docker、Podman 和 VM sandbox 应在 gateway 退出前停止,并在 gateway 返回后重启。在 gateway 日志中检查 和 。通过 CLI 显式停止的 sandbox 会保持停止状态。Kubernetes sandbox 由 cluster 持有,不遵循这一本地 gateway lifecycle。Internal 和 external driver 遵循相同规则:只有 为 true 时,gateway 才会执行 shutdown 和 startup sweep。
Stopped sandbox during gateway shutdownStarted sandbox during gateway startupGetCapabilities.gateway_manages_lifecycleStep 5: Check Podman-Backed Gateways
第 5 步:检查 Podman 后端的 Gateway
bash
podman info
podman ps --filter name=openshell
podman logs <container> --tail=200
openshell statusCommon findings:
- Podman socket unavailable: start or expose the user socket.
- Rootless networking unavailable: inspect Podman network configuration.
- Sandbox image missing or pull denied: verify image reference and registry credentials.
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI and matching
USERand/etc/passwdentries, or explicitly set both process identity fields in policy. Numeric workload identities/etc/groupthrough1are accepted; root, the invalid identity sentinel, and missing identities are rejected.4294967294 - Supervisor cannot call back: check callback endpoint and gateway logs.
- A sandbox with explicit endpoints fails before readiness: inspect supervisor logs for policy DNS port-53 binding, synthetic-route, or nftables redirect failures. Rootless Podman must provide these primitives inside the supervisor-owned nested network namespace; setup fails closed.
protocol: tcp - Gateway exits before becoming healthy with a callback-listener discovery
error: inspect , the configured Podman network, and the host's IPv4 default route. Rootless pasta uses the private source address selected by that route; rootful Podman uses the bridge gateway address.
podman info --debug - Current gateways reuse the primary listener when it covers Podman's callback address. If the primary does not cover that address, inspect the gateway startup logs for the additional callback-only listener and its provenance.
- Rootless slirp4netns, another named helper, or missing helper metadata
requires an explicitly remote . An explicit
grpc_endpointcannot bypass slirp4netns host-loopback isolation. Do not work around discovery failures by broadening the primary gateway listener tohost_gateway_ip.0.0.0.0
When is configured (e.g. or ):
usernsuserns = "auto"userns = "keep-id"- Supervisor delivery uses bind-mount fallback instead of image volumes because
overlay mounts do not support mounts. The supervisor binary is extracted from the supervisor image and cached at
idmapped(typically$XDG_DATA_HOME/openshell/podman-supervisor/).~/.local/share/openshell/podman-supervisor/ - Stale cache: if the supervisor image is updated but the cached binary is not refreshed, sandbox creation may fail with an ELF validation error or version mismatch. Remove the cache directory and retry.
- mode requires subuid/subgid ranges for the current user in
autoand/etc/subuid. If missing, Podman returns a user-namespace mapping error at container creation./etc/subgid - mode requires explicit
privateanduidmaparrays in the TOML config. Without both, the gateway rejects the config at startup. Rootless Podman uses intermediate IDs (e.g.gidmap); rootful Podman uses absolute host IDs (e.g.uidmap = ["0:0:1", "1:1:65535"]).uidmap = ["0:1000:1", "1:100000:65536"] - (without hyphen) is accepted as input but canonicalized to
nomapfor Podman's API.no-map - A workload remains in until Podman resorts to
stopping: inspect supervisor logs forSIGKILL. The supervisor must retainfailed to signal entrypoint process groupso its root process can forwardCAP_KILLto a workload that runs as the sandbox user.SIGTERM
bash
podman info
podman ps --filter name=openshell
podman logs <container> --tail=200
openshell status常见发现:
- Podman socket 不可用:启动或暴露 user socket。
- Rootless networking 不可用:检查 Podman network configuration。
- Sandbox 镜像缺失或拉取被拒:验证 image reference 和 registry credential。
- Sandbox 在 ready 之前因 identity-resolution error 失败:检查镜像的 OCI 以及匹配的
USER和/etc/passwd条目,或者在 policy 中明确设置两个 process identity 字段。接受/etc/group到1的数值型 workload identity;拒绝 root、无效 identity sentinel 和缺失的 identity。4294967294 - Supervisor 无法 callback:检查 callback endpoint 和 gateway 日志。
- 显式使用 endpoint 的 sandbox 在 ready 之前失败:检查 supervisor 日志中的 policy DNS port-53 binding、synthetic-route 或 nftables redirect failure。Rootless Podman 必须在 supervisor 持有的嵌套 network namespace 内提供这些 primitive;setup 采用 fail-closed 策略。
protocol: tcp - Gateway 在达到健康状态之前因 callback-listener discovery error 退出:检查 、已配置的 Podman network 和 host 的 IPv4 default route。Rootless pasta 使用该 route 选出的 private source address;rootful Podman 使用 bridge gateway address。
podman info --debug - 当前 gateway 会在 primary listener 覆盖 Podman callback address 时复用它。如果 primary 未覆盖该 address,应检查 gateway 启动日志中的额外 callback-only listener 及其来源。
- Rootless slirp4netns、其他命名 helper 或缺失 helper metadata 要求显式配置 remote 。显式的
grpc_endpoint无法绕过 slirp4netns 的 host-loopback isolation。不要通过将 primary gateway listener 扩大到host_gateway_ip来绕过 discovery failure。0.0.0.0
配置 时,例如 或 :
usernsuserns = "auto"userns = "keep-id"- Supervisor delivery 使用 bind-mount fallback,而不是 image volume,因为 overlay mount 不支持 mount。Supervisor binary 会从 supervisor image 中提取并缓存在
idmapped(通常为$XDG_DATA_HOME/openshell/podman-supervisor/)。~/.local/share/openshell/podman-supervisor/ - 缓存过期:如果 supervisor image 已更新,但缓存的 binary 未刷新,sandbox creation 可能因 ELF validation error 或 version mismatch 而失败。删除 cache directory 后重试。
- 模式要求当前用户在
auto和/etc/subuid中具有 subuid/subgid range。如果缺失,Podman 会在创建 container 时返回 user-namespace mapping error。/etc/subgid - 模式要求在 TOML config 中显式提供
private和uidmaparray。缺少任意一个都会导致 gateway 在启动时拒绝配置。Rootless Podman 使用中间 ID,例如gidmap;rootful Podman 使用绝对 host ID,例如uidmap = ["0:0:1", "1:1:65535"]。uidmap = ["0:1000:1", "1:100000:65536"] - (不带连字符)可作为输入接受,但会针对 Podman API 规范化为
nomap。no-map - Workload 会一直保持 ,直到 Podman 最终使用
stopping:检查 supervisor 日志中的SIGKILL。Supervisor 必须保留failed to signal entrypoint process group,使其 root process 能以 sandbox user 身份向 workload 转发CAP_KILL。SIGTERM
Step 6: Check Kubernetes Helm Gateways
第 6 步:检查 Kubernetes Helm Gateway
bash
helm -n openshell status openshell
helm -n openshell get values openshell
kubectl -n openshell get deployment,statefulset,pod,svc,pvc
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
kubectl -n openshell rollout status deployment/openshell
kubectl -n openshell rollout status statefulset/openshellUse the log and rollout commands for the workload kind that exists in the
release. Look for failed installs, unexpected values, missing namespace, wrong
image tag, TLS settings that do not match the registered endpoint, and
scheduling failures.
The chart mounts the ConfigMap key directly at
as a read-only file. This avoids the
atomic-writer symlink exposed by a ConfigMap directory mount because the gateway
rejects symlinked configuration. A checksum pod-template annotation rolls the
workload when the ConfigMap changes. If config preflight reports a symlink or
nonregular path, inspect the rendered mount and confirm the workload rolled to
the current chart revision:
gateway.toml/etc/openshell/gateway.tomlsubPathbash
kubectl -n openshell get deployment,statefulset -o yaml | rg -n 'gateway-config|mountPath|subPath|checksum/gateway-config'
kubectl -n openshell rollout status <deployment-or-statefulset>/openshell
kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200server.telemetryEnabledOPENSHELL_TELEMETRY_ENABLEDWhen no external credential driver is enabled, the Helm chart uses the
gateway's default encrypted database credential storage. The chart creates a
retained Kubernetes Secret for the shared KEK, injects it into gateway pods, and
stores encrypted credential envelopes in the OpenShell database. For
or multi-replica gateways, confirm
points at a shared database. A render/install error
mentioning means the values selected multiple
external credential backends.
workload.kind=deploymentserver.externalDbSecretserver.credentialDriversFor HA or PostgreSQL-backed installs, also check the external database Secret
referenced by and the PostgreSQL workload when it is
deployed in-cluster:
server.externalDbSecretbash
kubectl -n <namespace> get secret <external-db-secret> -o yaml
kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<postgres-workload>
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200Check required Helm deployment secrets:
bash
kubectl -n openshell get secret \
openshell-server-tls \
openshell-server-client-ca \
openshell-client-tls \
openshell-jwt-keysWhen , the chart intentionally omits
and the mount, even with built-in PKI or
cert-manager. That is expected; do not treat a missing pod
mount as a defect ( may still exist from PKI).
User auth is OIDC or trusted proxy ();
supervisor transport still uses .
server.tls.clientCaSecretName=""client_ca_pathtls-client-catls-client-caopenshell-server-client-caserver.auth.allowUnauthenticatedUsers=trueopenshell-client-tlsIn cert-manager installs, makes cert-manager own TLS
generation. The Helm chart should still render the
pre-install/pre-upgrade hook in JWT-only mode to create ,
even if remains true.
If the gateway pod is pending with and is absent, inspect the rendered
output and the hook Job logs; cert-manager creates TLS
Secrets but does not create the sandbox JWT signing Secret.
certManager.enabled=trueopenshell-certgenopenshell-jwt-keyspkiInitJob.enabledMountVolume.SetUp failed for volume "sandbox-jwt"openshell-jwt-keystemplates/certgen.yamlIf the gateway exits with , verify that contains
, , and , and that the gateway workload mounts the
secret at . The sandbox JWT mount is required
even when local Helm values disable TLS.
failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pemopenshell-jwt-keyssigning.pempublic.pemkidsandbox-jwt/etc/openshell-jwtIf points the server certificate at an external
Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted
cert on an OpenShift with TLS passthrough — see
), the chart creates two server certificates: an
internal one (chart CA, internal SANs) and an external one (from the configured
issuer, external SANs only). The gateway uses SNI to present the right cert.
certManager.serverIssuerRefRouteopenshiftRoute.enabledCheck the external // resources
directly when the external secret never becomes Ready:
CertificateCertificateRequestChallengebash
kubectl -n openshell get certificate,certificaterequest,challenge
kubectl -n openshell describe certificate openshell-server-external
oc -n openshell get routeACME issuers reject certificate requests that include internal-only names
(, , loopback IPs) and require the
to also be a SAN — the external only requests the
hostnames in , for exactly this reason.
*.svc.cluster.locallocalhostcommonNameCertificatecertManager.serverDnsNamesIf sandbox supervisors fail their TLS handshake to the gateway with
after configuring , the most likely cause is
set to the external hostname. This forces supervisors
to connect via the external hostname, receiving the ACME cert (via SNI) which
they cannot verify against the chart CA. Remove or set
it to the internal service name so supervisors receive the internal cert:
UnknownCAserverIssuerRefserver.grpcEndpointserver.grpcEndpointbash
helm -n openshell get values openshell | grep -E 'grpcEndpoint|clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'bash
helm -n openshell status openshell
helm -n openshell get values openshell
kubectl -n openshell get deployment,statefulset,pod,svc,pvc
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
kubectl -n openshell rollout status deployment/openshell
kubectl -n openshell rollout status statefulset/openshell应根据 release 中实际存在的 workload 类型使用对应的 log 和 rollout command。查找 install 失败、意外 values、namespace 缺失、image tag 错误、与已注册 endpoint 不匹配的 TLS 设置以及调度失败。
Chart 会将 ConfigMap key 以只读 file 的形式直接挂载到 。这样可以避免 ConfigMap directory mount 暴露的 atomic-writer symlink,因为 gateway 会拒绝符号链接形式的配置。Checksum pod-template annotation 会在 ConfigMap 变化时滚动 workload。如果 config preflight 报告 symlink 或 nonregular path,应检查渲染后的 mount,并确认 workload 已滚动到当前 chart revision:
gateway.tomlsubPath/etc/openshell/gateway.tomlbash
kubectl -n openshell get deployment,statefulset -o yaml | rg -n 'gateway-config|mountPath|subPath|checksum/gateway-config'
kubectl -n openshell rollout status <deployment-or-statefulset>/openshell
kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200server.telemetryEnabledOPENSHELL_TELEMETRY_ENABLED未启用 external credential driver 时,Helm chart 使用 gateway 默认的 encrypted database credential storage。Chart 会创建一个保留型 Kubernetes Secret 用于共享 KEK,将其注入 gateway pod,并把 encrypted credential envelope 存储在 OpenShell database 中。对于 或多副本 gateway,应确认 指向共享 database。如果 render/install error 中提到 ,说明 values 选择了多个 external credential backend。
workload.kind=deploymentserver.externalDbSecretserver.credentialDrivers对于 HA 或使用 PostgreSQL 的安装,还应检查 引用的 external database Secret,以及 PostgreSQL workload 在 cluster 内部署时的状态:
server.externalDbSecretbash
kubectl -n <namespace> get secret <external-db-secret> -o yaml
kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<postgres-workload>
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200检查必需的 Helm deployment secret:
bash
kubectl -n openshell get secret \
openshell-server-tls \
openshell-server-client-ca \
openshell-client-tls \
openshell-jwt-keys当 时,即使使用内置 PKI 或 cert-manager,chart 也会有意省略 和 mount。这属于预期行为;不要将缺失的 pod mount 视为缺陷(PKI 仍可能创建 )。用户认证使用 OIDC 或 trusted proxy();supervisor transport 仍使用 。
server.tls.clientCaSecretName=""client_ca_pathtls-client-catls-client-caopenshell-server-client-caserver.auth.allowUnauthenticatedUsers=trueopenshell-client-tls在 cert-manager 安装中, 表示由 cert-manager 管理 TLS 生成。即使 仍为 true,Helm chart 在 JWT-only 模式下仍应渲染 pre-install/pre-upgrade hook,以创建 。如果 gateway pod 一直处于 pending 状态,并出现 ,同时缺少 ,应检查渲染后的 输出和 hook Job 日志;cert-manager 会创建 TLS Secret,但不会创建 sandbox JWT signing Secret。
certManager.enabled=truepkiInitJob.enabledopenshell-certgenopenshell-jwt-keysMountVolume.SetUp failed for volume "sandbox-jwt"openshell-jwt-keystemplates/certgen.yaml如果 gateway 因 退出,应验证 包含 、 和 ,并确认 gateway workload 将 secret 挂载到 。即使本地 Helm values 禁用了 TLS,也必须挂载 sandbox JWT。
failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pemopenshell-jwt-keyssigning.pempublic.pemkidsandbox-jwt/etc/openshell-jwt如果 将 server certificate 指向 external Issuer 或 ClusterIssuer,例如在启用 TLS passthrough 的 OpenShift 上使用面向公众受信任的 certificate——参见 ——chart 会创建两个 server certificate:一个内部 certificate(chart CA、内部 SAN),一个外部 certificate(来自配置的 issuer,仅含外部 SAN)。Gateway 使用 SNI 提供正确的 certificate。
certManager.serverIssuerRefRouteopenshiftRoute.enabled当 external secret 始终未进入 Ready 状态时,应直接检查 external // resource:
CertificateCertificateRequestChallengebash
kubectl -n openshell get certificate,certificaterequest,challenge
kubectl -n openshell describe certificate openshell-server-external
oc -n openshell get routeACME issuer 会拒绝包含仅供内部使用名称的 certificate request,例如 、 和 loopback IP,并要求 同时也是 SAN;external 只请求 中的 hostname,正是出于这一原因。
*.svc.cluster.locallocalhostcommonNameCertificatecertManager.serverDnsNames如果配置 后,sandbox supervisor 在 TLS handshake 期间因 失败,最可能的原因是 被设置为 external hostname。这会强制 supervisor 通过 external hostname 连接,从而收到无法针对 chart CA 验证的 ACME certificate。应删除 ,或将其设置为内部 service name,使 supervisor 获取内部 certificate:
serverIssuerRefUnknownCAserver.grpcEndpointserver.grpcEndpointbash
helm -n openshell get values openshell | grep -E 'grpcEndpoint|clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'server.grpcEndpoint should be unset or point to internal service name
server.grpcEndpoint should be unset or point to internal service name
Less commonly, `UnknownCA` can occur if the gateway's client-verification CA
is misconfigured. The default `clientCaFromServerTlsSecret=true` is correct
for all configurations — the internal server certificate is always signed by
the chart CA (the same CA that signs the client cert), so its `ca.crt` is
the right trust anchor. Only override this if you intentionally mount a
separate client CA via `server.tls.clientCaSecretName`. Verify the mounted
client CA matches the CA that signed the client certificate:
```bash
kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}' | jq .
较少见的情况是 gateway 的 client-verification CA 配置错误导致 `UnknownCA`。默认的 `clientCaFromServerTlsSecret=true` 适用于所有配置——内部 server certificate 始终由 chart CA 签名,该 CA 与 client certificate 的签名 CA 相同,因此其 `ca.crt` 是正确的 trust anchor。只有在明确通过 `server.tls.clientCaSecretName` 挂载单独的 client CA 时才应覆盖此默认值。验证挂载的 client CA 与签署 client certificate 的 CA 相匹配:
```bash
kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}' | jq .Should show items filter for ca.crt from openshell-server-tls
Should show items filter for ca.crt from openshell-server-tls
If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still
render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret.
SPIRE is used by both the gateway and sandbox supervisors for dynamic provider
token grants. The gateway pod must mount the `spiffe-workload-api` CSI volume
and set `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`; sandbox pods must
receive the matching Workload API socket from the Kubernetes driver config.
The gateway verifies supervisor JWT-SVIDs from JWT bundles fetched through this
Workload API socket, not from the SPIRE OIDC discovery endpoint.
Verify that SPIRE is installed, the CSI driver is available, and the Kubernetes
driver config includes `provider_spiffe_workload_api_socket_path`:
```bash
helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath'
kubectl get pods -A | grep -E 'spire|spiffe'
kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{.items[*].spec.containers[*].env[?(@.name==\"OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET\")].value}{\"\n\"}"Sandbox pods using provider token grants should have an
annotation, an
label, supervisor env vars and
, plus both the projected
volume and the CSI volume.
openshell.ai/sandbox-idopenshell.ai/managed-by=openshellOPENSHELL_K8S_SA_TOKEN_FILEOPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKETopenshell-sa-tokenspiffe-workload-apiCheck the image references currently used by the gateway deployment:
bash
kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'The gateway and supervisor images should use the same release tag. A stale supervisor image can make sandbox behavior lag behind gateway policy or protocol changes.
For vulnerability reports, record the running image digest and scan that exact
artifact. The gateway includes a pinned Distroless base; the supervisor includes
Alpine packages updated at image build time. A dependency or base-image fix only
reaches deployed containers after rebuilding, publishing, and redeploying the
images. Compare findings against the SBOM for that digest, not just its mutable
or tag.
latestdevFor plaintext local evaluation, confirm the chart has:
bash
helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint'Expected shape:
yaml
server:
disableTls: true
grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080Check service exposure:
bash
kubectl -n openshell get svc openshell -o wide
kubectl -n openshell get endpoints openshellFor local port-forward testing:
bash
kubectl -n openshell port-forward service/openshell 8080:8080Leave the port forward running. In another terminal, register the local endpoint if needed and verify it:
bash
openshell gateway add http://127.0.0.1:8080 --local --name local-kubernetes
openshell statusIf the gateway is healthy but sandbox creation fails:
bash
kubectl -n openshell get pods
kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200Check the configured sandbox namespace:
bash
helm -n openshell get values openshell | grep sandboxNamespaceThen inspect sandbox resources in that namespace.
For a split release, the gateway values should have
, and the target namespace should contain a
separate release:
workspaceResources.enabled=falseopenshell-workspacebash
helm -n openshell get values openshell | grep -A2 workspaceResources
helm -n <sandbox-namespace> status openshell-workspace
kubectl -n <sandbox-namespace> get serviceaccount,role,rolebinding,networkpolicy \
-l app.kubernetes.io/instance=openshell-workspace
kubectl auth can-i create sandboxes.agents.x-k8s.io \
--namespace <sandbox-namespace> \
--as system:serviceaccount:openshell:openshellIf the gateway cannot create or watch sandboxes, verify the workspace
RoleBinding subject matches the gateway ServiceAccount name and namespace.
If SSH relay connections fail, verify the workspace NetworkPolicy selects the
gateway's actual and
labels.
app.kubernetes.io/nameapp.kubernetes.io/instanceCheck the configured sandbox service account when TokenReview bootstrap or
sandbox registration fails. Helm creates a dedicated sandbox service account by
default and writes it to ;
the selected Kubernetes compute driver rejects projected tokens from other
service accounts. For an external driver, inspect its logs and confirm it
advertises ; the gateway delegates the opaque
credential over the driver socket and never interprets Kubernetes settings.
[openshell.drivers.kubernetes].service_account_namesupports_sandbox_authenticationbash
helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'If is rendered under ,
sandbox pods should have an init container running
, an container running
, and an
container running . The init container owns nftables setup and
should be the only sidecar topology container with . It also needs
/ to hand shared emptyDir state to the effective sidecar UID. The
default binary-aware network sidecar runs as UID 0 with primary GID
and adds plus . When
, it runs as the configured
non-root without those inspection capabilities. That dedicated
proxy UID must remain at least and must not match the workload UID
because the pod egress fence exempts its traffic. The pod is set to
in both modes.
topology = "sidecar"[openshell.drivers.kubernetes]openshell-network-init--mode=network-initagentopenshell-sandbox --mode=processopenshell-supervisor-network--mode=networkNET_ADMINCHOWNFOWNERsandbox_gidSYS_PTRACEDAC_READ_SEARCHprocess_binary_aware_network_policy = falseproxy_uid1000fsGroupsandbox_gidIn sidecar topology only the network sidecar should mount the gateway bootstrap
credentials ( and ). The process
container should not receive , gateway TLS env vars, the
sandbox token file, or those credential mounts. Instead, the network sidecar
serves policy and provider environment state over the Unix control socket from
( by
default). The process supervisor must be the first and only client. After
validating its peer UID, GID, and PID, the sidecar unlinks the listener. If the
connection later closes, the network sidecar exits non-zero so Kubernetes can
restart it with a fresh listener. If the process supervisor fails before
launching the workload,
inspect both containers for control-socket bind, connect, bootstrap, or update
errors. If new SSH/exec sessions do not pick up refreshed provider environment,
inspect the network sidecar settings-poll logs and the process container logs
for provider environment update handling; the process container should consume
newer provider-env revisions without receiving gateway credentials.
openshell-sa-tokenopenshell-client-tlsOPENSHELL_ENDPOINTOPENSHELL_SIDECAR_CONTROL_SOCKET/run/openshell-sidecar/control.sockThe process container reports the workload entrypoint PID over the same control
socket, and the network sidecar uses that PID for binary-scoped policy
decisions through . If rules with are unexpectedly
denied, inspect the sidecar control logs and confirm the pod has
.
The shared state directory should preserve inheritance
(). Sidecar SSH uses the Linux abstract socket
; the network sidecar verifies its peer PID before
bridging gateway relay requests. No file should appear in the shared
state directory.
Inspect all three when sandbox registration or egress enforcement fails:
/procpolicy.binariesshareProcessNamespace: truesandbox_gid02775@openshell-sidecar-sshssh.sockbash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-network-init --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200
如果 `server.providerTokenGrants.spiffe.enabled=true`,gateway 仍应渲染 `[openshell.gateway.gateway_jwt]` 并挂载 `sandbox-jwt` Secret。SPIRE 同时供 gateway 和 sandbox supervisor 用于 dynamic provider token grant。Gateway pod 必须挂载 `spiffe-workload-api` CSI volume,并设置 `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`;sandbox pod 必须从 Kubernetes driver config 获得匹配的 Workload API socket。Gateway 通过此 Workload API socket 获取的 JWT bundle 验证 supervisor JWT-SVID,而不是通过 SPIRE OIDC discovery endpoint。验证 SPIRE 已安装、CSI driver 可用,并且 Kubernetes driver config 包含 `provider_spiffe_workload_api_socket_path`:
```bash
helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath'
kubectl get pods -A | grep -E 'spire|spiffe'
kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{.items[*].spec.containers[*].env[?(@.name==\"OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET\")].value}{\"\n\"}"使用 provider token grant 的 sandbox pod 应具有 annotation、 label、supervisor 环境变量 和 ,以及投射的 volume 和 CSI volume。
openshell.ai/sandbox-idopenshell.ai/managed-by=openshellOPENSHELL_K8S_SA_TOKEN_FILEOPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKETopenshell-sa-tokenspiffe-workload-api检查 gateway deployment 当前使用的 image reference:
bash
kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'Gateway 和 supervisor image 应使用相同的 release tag。过期的 supervisor image 可能导致 sandbox 行为落后于 gateway policy 或 protocol 变更。
对于 vulnerability report,应记录运行中 image 的 digest,并扫描该确切 artifact。Gateway 包含固定版本的 Distroless base;supervisor 包含在 image build 时更新的 Alpine package。Dependency 或 base-image 修复只有在重新构建、发布并重新部署 image 后才能进入已部署的 container。应针对该 digest 的 SBOM 比较发现,而不能只使用可变的 或 tag。
latestdev对于明文本地评估,确认 chart 中存在:
bash
helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint'预期结构:
yaml
server:
disableTls: true
grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080检查 service exposure:
bash
kubectl -n openshell get svc openshell -o wide
kubectl -n openshell get endpoints openshell对于本地 port-forward 测试:
bash
kubectl -n openshell port-forward service/openshell 8080:8080保持 port forward 运行。在另一个 terminal 中,根据需要注册本地 endpoint 并验证:
bash
openshell gateway add http://127.0.0.1:8080 --local --name local-kubernetes
openshell status如果 gateway 健康但 sandbox creation 失败:
bash
kubectl -n openshell get pods
kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200检查已配置的 sandbox namespace:
bash
helm -n openshell get values openshell | grep sandboxNamespace然后检查该 namespace 中的 sandbox resource。
对于 split release,gateway values 应包含 ,并且目标 namespace 中应存在独立的 release:
workspaceResources.enabled=falseopenshell-workspacebash
helm -n openshell get values openshell | grep -A2 workspaceResources
helm -n <sandbox-namespace> status openshell-workspace
kubectl -n <sandbox-namespace> get serviceaccount,role,rolebinding,networkpolicy \
-l app.kubernetes.io/instance=openshell-workspace
kubectl auth can-i create sandboxes.agents.x-k8s.io \
--namespace <sandbox-namespace> \
--as system:serviceaccount:openshell:openshell如果 gateway 无法创建或 watch sandbox,应验证 workspace RoleBinding subject 与 gateway ServiceAccount 名称和 namespace 相匹配。如果 SSH relay connection 失败,应验证 workspace NetworkPolicy 是否选择了 gateway 实际使用的 和 label。
app.kubernetes.io/nameapp.kubernetes.io/instance当 TokenReview bootstrap 或 sandbox registration 失败时,检查已配置的 sandbox service account。Helm 默认创建专用的 sandbox service account,并将其写入 ;选定的 Kubernetes compute driver 会拒绝来自其他 service account 的 projected token。对于 external driver,应检查其日志并确认它声明了 ;gateway 通过 driver socket 传递 opaque credential,且绝不解释 Kubernetes 设置。
[openshell.drivers.kubernetes].service_account_namesupports_sandbox_authenticationbash
helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'如果在 下渲染出 ,则 sandbox pod 应具有一个运行 的 init container、一个运行 的 container,以及一个运行 的 container。Init container 负责 nftables setup,并且应是 sidecar topology container 中唯一拥有 的 container。它还需要 /,以便将共享的 emptyDir state 移交给 effective sidecar UID。默认的 binary-aware network sidecar 以 UID 0 运行,primary GID 为 ,并添加 和 。当 时,它以配置的 non-root 运行,且不具备这些 inspection capability。该专用 proxy UID 必须至少为 ,并且不能与 workload UID 相同,因为 pod egress fence 会豁免它的流量。两种模式下的 pod 均设置为 。
[openshell.drivers.kubernetes]topology = "sidecar"--mode=network-initopenshell-network-initopenshell-sandbox --mode=processagent--mode=networkopenshell-supervisor-networkNET_ADMINCHOWNFOWNERsandbox_gidSYS_PTRACEDAC_READ_SEARCHprocess_binary_aware_network_policy = falseproxy_uid1000fsGroupsandbox_gid在 sidecar topology 中,只有 network sidecar 应挂载 gateway bootstrap credential( 和 )。Process container 不应接收 、gateway TLS 环境变量、sandbox token file 或这些 credential mount。相反,network sidecar 通过 (默认为 )上的 Unix control socket 提供 policy 和 provider environment state。Process supervisor 必须是第一个且唯一的 client。验证 peer UID、GID 和 PID 后,sidecar 会 unlink listener。如果 connection 随后关闭,network sidecar 会以非零状态退出,让 Kubernetes 使用新的 listener 重启它。如果 process supervisor 在启动 workload 前失败,应检查两个 container 中的 control-socket bind、connect、bootstrap 或 update error。如果新的 SSH/exec session 未获取更新后的 provider environment,应检查 network sidecar 的 settings-poll 日志和 process container 的 provider environment update handling 日志;process container 应消费较新的 provider-env revision,而无需接收 gateway credential。
openshell-sa-tokenopenshell-client-tlsOPENSHELL_ENDPOINTOPENSHELL_SIDECAR_CONTROL_SOCKET/run/openshell-sidecar/control.sockProcess container 通过同一个 control socket 报告 workload entrypoint PID,network sidecar 使用该 PID 通过 做出 binary-scoped policy decision。如果包含 的 rule 意外被拒绝,应检查 sidecar control log,并确认 pod 设置了 。
/procpolicy.binariesshareProcessNamespace: true共享 state directory 应保留 inheritance()。Sidecar SSH 使用 Linux abstract socket ;network sidecar 在桥接 gateway relay request 前会验证 peer PID。共享 state directory 中不应出现 file。
sandbox_gid02775@openshell-sidecar-sshssh.sock当 sandbox registration 或 egress enforcement 失败时,检查以下三个 container:
bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-network-init --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200Corporate upstream proxy
Corporate upstream proxy
When the deployment routes sandbox egress through a corporate HTTP forward
proxy, the operator-owned settings render under
from the Helm values. Absent proxy configuration preserves
direct-dial egress; any present-but-invalid value fails closed at gateway
startup () rather than silently reverting to a
direct connection. Confirm the rendered configuration first:
[openshell.drivers.kubernetes]upstreamProxyvalidate_upstream_proxy_configbash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname'
helm -n openshell get values openshell | grep -A8 upstreamProxyOnly forward proxies are supported; proxy URLs and
plain-HTTP egress are out of scope and rejected. Proxy credentials require
— combined topology shares the credential mount with the
workload, so the gateway rejects credentials there. The credential Secret named
by must exist in the sandbox namespace with the key
named by , and Kubernetes will not create keys longer
than 253 bytes or named /.
http://host:porthttps://topology = "sidecar"proxy_auth_secret_nameproxy_auth_secret_key...The proxy arguments and credential mount are injected only into the container
that runs network supervision (the container in combined topology, the
sidecar in sidecar topology). The one-shot
container and the process container in sidecar
topology must never receive them. The credential is projected read-only as the
volume at
and passed as ; it must never appear in env,
annotations, or command arguments.
agentopenshell-supervisor-networkopenshell-network-initagentopenshell-upstream-proxy-auth/run/openshell/upstream-proxy-auth--upstream-proxy-auth-filebash
kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20A missing Secret or wrong key leaves the pod stuck with a
/ event. If the pod starts but
egress still fails, the corporate proxy itself is the next suspect: policy-
approved TLS CONNECT requests that time out after policy evaluation usually mean
the proxy URL is unreachable from the sandbox namespace, or a cluster-internal
destination that should be direct is missing from . Inspect the
network supervisor logs for CONNECT and upstream-proxy decisions:
MountVolume.SetUp failedsecret ... not foundno_proxybash
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy'当 deployment 通过 corporate HTTP forward proxy 路由 sandbox egress 时,operator 管理的设置会根据 Helm values 渲染到 下。未配置 proxy 时会保留 direct-dial egress;任何存在但无效的值都会在 gateway startup 时 fail closed(),而不会静默回退到 direct connection。首先确认渲染后的配置:
upstreamProxy[openshell.drivers.kubernetes]validate_upstream_proxy_configbash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname'
helm -n openshell get values openshell | grep -A8 upstreamProxy仅支持 forward proxy;不支持 proxy URL 和 plain-HTTP egress,这些配置会被拒绝。Proxy credential 要求 ——combined topology 会让 credential mount 与 workload 共享,因此 gateway 会拒绝该配置。 指定的 credential Secret 必须存在于 sandbox namespace 中,并包含 指定的 key;Kubernetes 不会创建 key 长度超过 253 bytes 或名称为 / 的 Secret。
http://host:porthttps://topology = "sidecar"proxy_auth_secret_nameproxy_auth_secret_key...Proxy argument 和 credential mount 仅注入运行 network supervision 的 container:在 combined topology 中为 container,在 sidecar topology 中为 sidecar。One-shot container 以及 sidecar topology 中的 process container 绝不能接收它们。Credential 会以只读方式投射为 volume,路径为 ,并通过 传入;它绝不能出现在环境变量、annotation 或 command argument 中。
agentopenshell-supervisor-networkopenshell-network-initagentopenshell-upstream-proxy-auth/run/openshell/upstream-proxy-auth--upstream-proxy-auth-filebash
kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20如果 Secret 缺失或 key 不正确,pod 会停留在 / event 状态。如果 pod 已启动但 egress 仍然失败,下一个排查对象是 corporate proxy 本身:policy 批准的 TLS CONNECT request 在 policy evaluation 后超时,通常意味着 proxy URL 无法从 sandbox namespace 访问,或者应直连的 cluster-internal destination 没有包含在 中。检查 network supervisor 日志中的 CONNECT 和 upstream-proxy decision:
MountVolume.SetUp failedsecret ... not foundno_proxybash
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy'Step 7: Check VM-Backed Gateways
第 7 步:检查 VM 后端的 Gateway
Use the VM driver logs and host diagnostics available in the user's environment. Verify:
- The VM driver process is running and reachable by the gateway.
- The runtime rootfs exists and matches the expected architecture.
- or
mke2fsandmkfs.ext4from e2fsprogs are installed; explicitdebugfs/sandbox_uiddoes not remove this prerequisite.sandbox_gid - A persisted overlay identity error is resolved from its owner marker, overlay
upper layer, prepared rootfs, explicit config, or current image. Do not assign
unless the persisted state reports that legacy identity.
10001:10001 - Host virtualization support is enabled.
- The sandbox supervisor can establish its callback connection to the gateway.
Then run:
bash
openshell status
openshell logs <sandbox-name>使用用户环境中可用的 VM driver 日志和 host diagnostic。验证:
- VM driver process 正在运行且 gateway 可以访问。
- Runtime rootfs 存在,并且与预期 architecture 匹配。
- 已安装 e2fsprogs 中的 或
mke2fs和mkfs.ext4;显式设置debugfs/sandbox_uid并不会取消此前置条件。sandbox_gid - Persisted overlay identity error 应根据其 owner marker、overlay upper layer、prepared rootfs、explicit config 或 current image 解决。除非 persisted state 报告了 legacy identity,否则不要分配 。
10001:10001 - Host virtualization support 已启用。
- Sandbox supervisor 可以建立到 gateway 的 callback connection。
然后运行:
bash
openshell status
openshell logs <sandbox-name>Corporate upstream proxy
Corporate upstream proxy
When VM sandbox egress routes through a corporate HTTP forward proxy, the
operator-owned settings live under and the gateway
forwards them to the subprocess as ,
, ,
,
, and . Both the gateway and
the driver validate them at startup, so any present-but-invalid value fails
closed with an error naming the key rather than reverting to a direct dial.
Confirm the configuration and the resulting driver argv first:
[openshell.drivers.vm]openshell-driver-vm--upstream-proxy--upstream-no-proxy--upstream-proxy-auth-file--upstream-proxy-auth-allow-insecure--upstream-proxy-connect-by-hostname--upstream-proxy-ca-bundlebash
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy'Reachability is the most common failure, and it depends on the VM backend.
On libkrun (non-GPU sandboxes) guest egress leaves through gvproxy, so a proxy
bound to the gateway host's loopback is not reachable at from
inside the guest: it must be addressed as
, which gvproxy NATs from
to the host's . A pointing at a
loopback URL produces policy-approved CONNECT attempts that time out while
public destinations still work.
127.0.0.1http://host.openshell.internal:<port>192.168.127.254127.0.0.1https_proxyGPU sandboxes run on QEMU/TAP, where no gateway-host proxy is reachable at
all: resolves to the TAP host address, and the
driver's nftables chain accepts only the gateway port from the guest.
The driver rejects such a configuration at launch — a create failing with
means the proxy must move to an address routable from the
guest's masqueraded egress (or the sandbox must run without a GPU).
host.openshell.internalinputhttps_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ... cannot reachThe settings reach the supervisor through a driver-written argument file in
the per-sandbox overlay, not through the guest environment. The credential and
CA bundle are staged into the same overlay at fixed guest paths. Inspect the
guest side from the VM console log, which records how many driver-supplied
arguments the init script read:
bash
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40FATAL: supervisor argument list ... is not readableFATAL: empty entry in supervisor argument listgateway.tomlhttps_proxy[openshell.drivers.vm].driver_dir当 VM sandbox egress 通过 corporate HTTP forward proxy 路由时,operator 管理的设置位于 下,gateway 会将其作为 、、、、 和 转发给 subprocess。Gateway 和 driver 都会在启动时验证这些设置,因此任何存在但无效的值都会 fail closed,并返回指明具体 key 的错误,而不会回退到 direct dial。首先确认配置和最终生成的 driver argv:
[openshell.drivers.vm]--upstream-proxy--upstream-no-proxy--upstream-proxy-auth-file--upstream-proxy-auth-allow-insecure--upstream-proxy-connect-by-hostname--upstream-proxy-ca-bundleopenshell-driver-vmbash
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy'可达性是最常见的故障,并且取决于 VM backend。在 libkrun(non-GPU sandbox)上,guest egress 通过 gvproxy 离开,因此绑定到 gateway host loopback 的 proxy 无法从 guest 内部通过 访问:必须使用 ,gvproxy 会将 NAT 到 host 的 。指向 loopback URL 的 会产生经过 policy 批准的 CONNECT attempt 并发生超时,同时 public destination 仍可能正常工作。
127.0.0.1http://host.openshell.internal:<port>192.168.127.254127.0.0.1https_proxyGPU sandbox 运行在 QEMU/TAP 上,其中 gateway host proxy 完全不可达: 会解析到 TAP host address,driver 的 nftables chain 只允许来自 guest 的 gateway port。Driver 会在启动时拒绝此类配置;create 失败并显示 ,意味着 proxy 必须迁移到 guest 的 masqueraded egress 可路由的地址,或者 sandbox 必须在不使用 GPU 的情况下运行。
host.openshell.internalinputhttps_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ... cannot reach这些设置通过 driver 在每个 sandbox overlay 中写入的 argument file 传递给 supervisor,而不是通过 guest environment。Credential 和 CA bundle 会被暂存到同一 overlay 的固定 guest path 中。从 VM console log 检查 guest side,该日志会记录 init script 读取了多少个 driver-supplied argument:
bash
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40FATAL: supervisor argument list ... is not readableFATAL: empty entry in supervisor argument listgateway.tomlhttps_proxy[openshell.drivers.vm].driver_dirCommon Failure Patterns
常见故障模式
| Symptom | Likely cause | Check |
|---|---|---|
| Gateway endpoint unreachable or auth mismatch | |
| The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart |
| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
| Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, |
| Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
| Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | |
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | |
| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default | |
| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | |
OpenShift gateway pod fails to start with an SCC/ | Chart's default | |
OpenShift sandbox pod fails to start ( | The | |
| CLI TLS error | Local mTLS bundle does not match server cert/CA | Check |
Edge or OIDC gateway returns | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | |
| Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; |
| Authenticated interceptor or middleware rejects gateway calls | Private CA or hostname mismatch, expected audience or issuer mismatch, stale/unknown | |
| Provider profiles disappear after enabling an interceptor catalog | | Inspect source list and interceptor |
| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid payload/timeout limit | Middleware service and gateway logs; |
Policy update rejects | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of | Policy error, gateway logs, middleware |
Policy mutation returns | Equally specific effective endpoint selectors disagree on connection or request-processing metadata | CLI error, base and provider-composed policy, affected profile attachments; confirm no new revision was stored |
| Supervisor enters policy quarantine | A runtime candidate failed validation while | Sandbox OCSF config/finding events, validation rationale, active generation, |
HTTP request returns | Selected stage failed or explicitly denied admitted traffic | Sandbox OCSF logs; policy-local middleware config; service availability; binding operation; |
| WebSocket upgrades but a host-matched middleware receives no preflight or message RPC | The implementation did not advertise | |
| Binary WebSocket message passes without a middleware RPC | Binary is unsupported by the V1 text-message binding under both | |
| WebSocket messages stop reaching middleware after one failure | A fail-open stage stream was disabled for the rest of the connection | |
| Supervisor repeatedly fails to install middleware after enabling gateway JWT signing | Extension credential minting, distribution, or authenticated service connection failed; last-known-good registry remains active | Gateway |
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or selected name does not match its endpoint/config key | Socket ownership/mode, driver service logs, gateway |
Sandbox remains | Driver stop/start failed, retained resource is missing, or a fresh supervisor has not connected | Gateway and driver logs; |
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
Gateway API resources fail with | Optional Gateway API resources were applied without Envoy Gateway CRDs | Install Envoy Gateway and enable |
HTTPS ingress ( | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set |
HTTPS ingress returns | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure |
External server | ACME issuer rejected internal-only SANs, a loopback IP, or a | |
Sandbox supervisors fail TLS handshake with | | Remove |
Browser | Listener client-CA verification still enabled ( | Set |
| 症状 | 可能原因 | 检查项 |
|---|---|---|
| Gateway endpoint 不可达或 auth 不匹配 | |
| 本地 gateway 启动时配置了 OTLP,但后续 collector forwarding task 已停止,或者配置是手工创建的 | 重启 |
| Gateway 启动但 sandbox create 失败 | Compute driver 无法访问 runtime | Docker/Podman/Kubernetes/VM driver 日志 |
| Gateway 在解析 compute-driver listener requirement 时退出 | Callback alias topology 不受支持、无法检查 Podman network,或选定地址不是 private/authorized | Gateway startup error、 |
| 在额外的 Docker/Podman callback-only listener 上,admin、health、reflection 或 HTTP request 被拒绝 | 额外 callback listener 有意仅公开 sandbox 可调用的 gRPC method | 通过 gateway 的 primary endpoint 重试;如果地址不符合预期,检查 listener-purpose startup log |
| Docker 或 Podman sandbox 始终未注册 | Callback endpoint 错误或 supervisor startup failure | Gateway 日志和 sandbox container 日志 |
| Docker GPU sandbox 在启动前失败 | NVIDIA CDI spec 缺失,或 Docker 尚未发现它们 | |
| Kubernetes gateway pod 处于 pending 状态 | PVC 未绑定、taint、selector 或资源不足 | |
| Kubernetes sandbox pod 一直处于 pending 状态,workspace PVC 未绑定 | Cluster 没有默认 | |
| Kubernetes gateway pod 陷入 crash loop | 缺少 secret、DB URL 错误或 TLS 配置错误 | |
OpenShift gateway pod 因 SCC/ | Chart 默认的 | |
OpenShift sandbox pod 无法启动,显示 | | |
| CLI TLS error | 本地 mTLS bundle 与 server cert/CA 不匹配 | 检查 |
Edge 或 OIDC gateway 返回 | 已存储的 login 已过期、audience/scopes 不匹配,或 gateway auth 配置发生变化 | |
| 启用 interceptor 后 gateway 在提供 health service 前失败 | Interceptor endpoint 不可用,或 manifest/binding validation 失败 | Gateway 和 interceptor 日志;interceptor socket; |
| Authenticated interceptor 或 middleware 拒绝 gateway call | Private CA 或 hostname 不匹配、预期 audience 或 issuer 不匹配、 | |
| 启用 interceptor catalog 后 provider profile 消失 | | 检查 source list 和 interceptor |
| 注册 supervisor middleware 后 gateway 失败 | Service 不可用、manifest 无效、binding 重复、保留名称或 payload/timeout limit 无效 | Middleware service 和 gateway 日志; |
Policy update 拒绝 | Unknown middleware name、implementation-owned config 无效、order 重复、host selector 过宽或无效,或 fail-closed coverage 包含 | Policy error、gateway 日志、middleware |
Endpoint ambiguity 导致 policy mutation 返回 | 匹配同等具体程度的 effective endpoint selector 在 connection 或 request-processing metadata 上存在分歧 | CLI error、base 和 provider-composed policy、受影响的 profile attachment;确认没有存储新 revision |
| Supervisor 进入 policy quarantine | Runtime candidate validation 失败,同时 | Sandbox OCSF config/finding event、validation rationale、active generation、 |
HTTP request 返回 | 被选中的 stage 失败或显式拒绝已准入流量 | Sandbox OCSF 日志;policy-local middleware 配置;service 可用性;binding operation; |
| WebSocket upgrade 成功,但匹配 host 的 middleware 未收到 preflight 或 message RPC | 实现未声明 | |
| Binary WebSocket message 未触发 middleware RPC 即通过 | 在两种 | |
| 一次失败后 WebSocket message 不再到达 middleware | Fail-open stage stream 已在该 connection 的剩余生命周期中被禁用 | |
| 启用 gateway JWT signing 后,supervisor 反复无法安装 middleware | Extension credential minting、distribution 或 authenticated service connection 失败;最后一个已知正常的 registry 仍处于活动状态 | Gateway |
| Custom compute driver 不可用 | Driver process/socket 缺失或不可访问,或者选定名称与 endpoint/config key 不匹配 | Socket ownership/mode、driver service 日志、gateway |
Sandbox 一直处于 | Driver stop/start 失败、保留的 resource 缺失,或新的 supervisor 尚未连接 | Gateway 和 driver 日志; |
| Image pull failure | Gateway 或 sandbox image 无法拉取 | Runtime event 和 image pull credential |
Gateway API resource 失败并显示 | 应用 optional Gateway API resource 时未安装 Envoy Gateway CRD | 安装 Envoy Gateway,并在应用 optional ingress resource 前启用 |
HTTPS ingress( | Envoy 终止 TLS,但 gateway pod 仍预期 TLS,导致明文 backend hop 失败 | 设置 |
HTTPS ingress 连接后返回 | TLS 在 Envoy 处终止,因此 gateway 看不到 client certificate;也没有为 identity 配置 OIDC issuer | 配置 |
设置 | ACME issuer 拒绝仅供内部使用的 SAN、loopback IP,或拒绝未同时出现在 SAN 中的 | |
配置 | | 删除 |
浏览器显示 | Listener client-CA verification 仍处于启用状态( | 设置 |
Reporting
结果报告
When handing results back to the user, include:
- Active gateway endpoint and auth mode.
- Compute platform and driver.
- Gateway process or workload status.
- Recent gateway log summary.
- Missing or malformed TLS, OIDC/mTLS, or sandbox JWT material.
- Service exposure status.
- Sandbox workload status.
- The exact command that failed and the shortest fix.
向用户返回结果时,应包括:
- 当前活动的 gateway endpoint 和 auth mode。
- Compute platform 和 driver。
- Gateway process 或 workload 状态。
- 最近的 gateway 日志摘要。
- 缺失或格式错误的 TLS、OIDC/mTLS 或 sandbox JWT material。
- Service exposure 状态。
- Sandbox workload 状态。
- 发生失败的确切命令和最短修复方案。
Package Configuration Preflight
Package Configuration Preflight
For a Debian, Ubuntu, or Snap gateway that stops before certificate generation or
daemon startup, validate the selected configuration without starting the service:
shell
openshell-gateway config preflight [--path PATH | -- GATEWAY_ARGS...]Without a path, preflight validates a nonempty or an
auto-discovered XDG config; no config succeeds. An explicit missing path, legacy
schema-v1 file, malformed TOML, symlink, or nonregular file fails before gateway
startup. It also applies read-only effective-config checks for driver selection
and configuration, sockets, rate limits, TLS, interceptors, and supervisor
middleware. If the file omits the selector, preflight validates configured tables
for auto-detectable drivers without running socket or process-based detection
probes. Arguments after validate the effective daemon invocation,
including its command-line overrides. Preflight preserves every failed file. Do
not advise users to delete or rewrite it automatically; back it up and follow the
manual schema-v2 migration in the Gateway Configuration reference.
OPENSHELL_GATEWAY_CONFIG--对于在 certificate generation 或 daemon startup 之前停止的 Debian、Ubuntu 或 Snap gateway,应在不启动 service 的情况下验证所选配置:
shell
openshell-gateway config preflight [--path PATH | -- GATEWAY_ARGS...]未指定 path 时,preflight 会验证非空的 ,或自动发现的 XDG config;不存在配置时验证成功。显式指定但不存在的 path、legacy schema-v1 file、格式错误的 TOML、symlink 或 nonregular file 都会在 gateway startup 前失败。它还会对 driver selection 和 configuration、socket、rate limit、TLS、interceptor 及 supervisor middleware 执行只读 effective-config check。如果文件省略 selector,preflight 会在不运行 socket 或基于 process 的 detection probe 的情况下,验证可自动检测 driver 的已配置 table。 之后的 argument 会针对 effective daemon invocation 进行验证,包括 command-line override。Preflight 会保留每个失败的文件。不要建议用户自动删除或重写这些文件;应先备份,然后按照 Gateway Configuration reference 中的说明手工完成 schema-v2 migration。
OPENSHELL_GATEWAY_CONFIG--