debug-openshell-cluster

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Debug OpenShell Gateway Deployment

调试 OpenShell Gateway 部署

Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, the experimental VM driver, or an operator-managed out-of-tree compute driver.
Use
openshell
first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver:
docker
,
podman
,
kubectl
/
helm
, or VM driver logs.
诊断 gateway 及其选定的 compute platform。不要假定 OpenShell 会置备 Kubernetes 或运行 k3s container。OpenShell 面向的是一个可达的 gateway endpoint,其后端可以是 Docker、Podman、Kubernetes、实验性 VM driver,或由 operator 管理的 out-of-tree compute driver。
首先使用
openshell
识别当前活动的 endpoint。然后使用与 gateway 的 compute driver 相匹配的 platform 工具:
docker
、
podman
、
kubectl
/
helm
,或 VM driver 日志。

Overview

概览

The target deployment flow is:
  1. Operator starts or deploys the gateway with system packages, systemd, or Helm. The CLI does not start, stop, or destroy gateway services.
  2. Operator configures the compute driver.
  3. Operator provides the CLI and supervisor authentication material required by the deployment mode: edge or OIDC user auth, optional CLI mTLS, and gateway-minted sandbox JWTs.
  4. The CLI registers a reachable gateway endpoint with
    openshell gateway add
    .
  5. The gateway creates sandboxes through the selected compute driver.
The
openshell-gateway
composition crate explicitly installs its compiled Docker, Podman, Kubernetes, and VM registrations at startup;
openshell-server
does not link compute-driver crates. Custom gateway binaries may include a subset of those registrations. With no configured driver, the gateway probes only installed registrations in priority order (Kubernetes, Podman, then Docker); VM has no probe and remains opt-in. Confirm the binary's registered drivers when auto-detection reports that no suitable driver is available. If configuration selects a driver that was not compiled in, the gateway treats the name as an external driver and reports a missing
socket_path
unless an endpoint is configured.
On Windows, custom binaries can include MXC independently. Registrations for Docker, Podman, Kubernetes, and VM are rejection stubs when included; they do not enable those runtimes on Windows.
See the compute driver reference for selective-build options and external-driver configuration.
For local evaluation only, TLS may be disabled and the gateway can be reached through
http://127.0.0.1:<port>
.
目标部署流程如下:
  1. Operator 使用 system packages、systemd 或 Helm 启动或部署 gateway。CLI 不负责启动、停止或销毁 gateway services。
  2. Operator 配置 compute driver。
  3. Operator 根据部署模式提供 CLI 和 supervisor 所需的认证材料:edge 或 OIDC 用户认证、可选的 CLI mTLS,以及由 gateway 签发的 sandbox JWT。
  4. CLI 使用
    openshell gateway add
    注册一个可达的 gateway endpoint。
  5. Gateway 通过选定的 compute driver 创建 sandbox。
openshell-gateway
composition crate 会在启动时明确安装其编译时包含的 Docker、Podman、Kubernetes 和 VM driver 注册;
openshell-server
不会链接 compute-driver crate。自定义 gateway binary 可能只包含其中一部分注册。未配置 driver 时,gateway 只会按照优先级探测已安装的注册:Kubernetes、Podman,然后是 Docker;VM 不参与探测,必须显式启用。当自动检测报告没有合适的 driver 时,应确认 binary 中实际注册的 driver。如果配置选择了未编译进 binary 的 driver,gateway 会将该名称视为 external driver,并在未配置 endpoint 时报告缺少
socket_path
。
在 Windows 上,自定义 binary 可以独立包含 MXC。包含 Docker、Podman、Kubernetes 和 VM 注册时,它们只是 rejection stub,并不会在 Windows 上启用这些 runtime。
有关 selective-build 选项和 external driver 配置,请参阅 compute driver reference。
仅限本地评估时,可以禁用 TLS,并通过
http://127.0.0.1:<port>
访问 gateway。

Prerequisites

前置条件

  • The
    openshell
    CLI must be available for endpoint checks.
  • Know the active gateway name and endpoint, or be able to inspect local gateway metadata.
  • Know the compute platform: Docker, Podman, Kubernetes, VM, or an out-of-tree driver.
  • For Kubernetes:
    kubectl
    must target the cluster that hosts OpenShell and Helm version 3 or later must be available.
  • For Docker or Podman: the runtime socket must be reachable from the gateway host.
Use
openshell --help
and nested
--help
output as the authority for the installed CLI version. Use the published installation guide, compute-driver reference, gateway configuration reference, and Kubernetes setup guide as the authority for deployment and configuration behavior.
  • 必须可以使用
    openshell
    CLI 检查 endpoint。
  • 知道当前活动的 gateway 名称和 endpoint,或者能够检查本地 gateway metadata。
  • 知道 compute platform:Docker、Podman、Kubernetes、VM 或 out-of-tree driver。
  • 对于 Kubernetes:
    kubectl
    必须指向承载 OpenShell 的集群,并且可以使用 Helm 3 或更高版本。
  • 对于 Docker 或 Podman:runtime socket 必须可从 gateway host 访问。
应以
openshell --help
及嵌套的
--help
输出作为已安装 CLI 版本的权威依据。部署和配置行为应以已发布的 installation guide、compute-driver reference、gateway configuration reference 和 Kubernetes setup guide 为准。

Workflow

工作流

Run diagnostics in order and stop once the root cause is clear.
按顺序运行诊断命令,确定根本原因后即可停止。

Step 1: Check CLI Reachability

第 1 步:检查 CLI 可达性

bash
openshell gateway list --output json
openshell gateway info
openshell status
For a one-off endpoint check that bypasses stored gateway selection and metadata:
bash
openshell --gateway-endpoint <url> status
Common findings:
  • No active gateway
    : register one with
    openshell gateway add <endpoint>
    .
  • Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active.
  • TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener.
  • Unauthenticated
    from an edge or OIDC gateway: refresh stored credentials with
    openshell gateway login [name]
    , then retry. Use
    gateway logout
    only when intentionally clearing local credentials.
  • A direct development endpoint with a private or self-signed certificate can be isolated with
    --gateway-endpoint <url> --gateway-insecure
    ; do not persist or recommend insecure verification for shared gateways.
bash
openshell gateway list --output json
openshell gateway info
openshell status
如需执行一次性 endpoint 检查,并绕过已存储的 gateway 选择和 metadata,可使用:
bash
openshell --gateway-endpoint <url> status
常见发现:
  • No active gateway
    :使用
    openshell gateway add <endpoint>
    注册一个 gateway。
  • Connection refused:gateway process 未运行、service 暴露方式错误,或 port-forward/proxy 未运行。
  • TLS/certificate 错误:endpoint scheme 或 trust chain 错误、本地 mTLS bundle 与 gateway CA 不匹配,或 TLS termination 与 gateway listener 不匹配。
  • edge 或 OIDC gateway 返回
    Unauthenticated
    :使用
    openshell gateway login [name]
    刷新已存储的凭据,然后重试。只有在明确需要清除本地凭据时才使用
    gateway logout
    。
  • 若要隔离使用私有或自签名证书的直接开发 endpoint,可以使用
    --gateway-endpoint <url> --gateway-insecure
    ;不要为共享 gateway 持久化或建议使用不安全的证书验证方式。

Step 2: Identify the Compute Platform

第 2 步:确定 Compute Platform

Use gateway metadata, deployment values, or the user's setup notes to identify the driver.
PlatformPrimary checks
DockerGateway process logs, Docker daemon health, sandbox containers, image pulls.
PodmanPodman socket, rootless networking, sandbox containers, image pulls.
KubernetesHelm release, gateway workload, service, secrets, sandbox pods, events.
OpenShiftSame as Kubernetes, plus SecurityContextConstraints (SCCs) and, for external access, an OpenShift
Route
. Detect OpenShift by the presence of the
route.openshift.io
API group (
oc api-resources --api-group=route.openshift.io
).
VMVM driver logs, rootfs availability, host virtualization support.
ExtensionExternal driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs.
使用 gateway metadata、deployment values 或用户提供的设置说明确定 driver。
Platform主要检查项
DockerGateway process 日志、Docker daemon 健康状态、sandbox containers、镜像拉取。
PodmanPodman socket、rootless networking、sandbox containers、镜像拉取。
KubernetesHelm release、gateway workload、service、secrets、sandbox pods、events。
OpenShift与 Kubernetes 相同,另外检查 SecurityContextConstraints(SCC),以及外部访问所需的 OpenShift
Route
。可通过
route.openshift.io
API group 的存在检测 OpenShift:
oc api-resources --api-group=route.openshift.io
。
VMVM driver 日志、rootfs 可用性、host virtualization 支持情况。
ExtensionExternal driver process、Unix socket 所有权/权限、已配置的 driver 名称、capability handshake、gateway 日志。

Step 3: Check Gateway Startup Dependencies

第 3 步:检查 Gateway 启动依赖

Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready.
For out-of-tree compute drivers, confirm the selected driver name and socket agree across CLI flags or
gateway.toml
, and that the operator-owned driver is running before the gateway starts:
bash
rg -n '^version|compute_driver|socket_path|guest_tls_' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
Gateway configuration requires
[openshell] version = 2
, a singular
compute_driver
selector, and driver-owned settings under
[openshell.drivers.<name>]
. The gateway rejects legacy
compute_drivers
and
--drivers
selectors rather than silently migrating them. One valid, nonempty
OPENSHELL_DRIVERS
value remains a deprecated environment-only alias when the canonical selector is absent; the gateway selects that driver with a warning. Empty, invalid, comma-delimited, or conflicting values fail startup. Homebrew and RPM package startup migrates only exact package-generated v1 defaults. If an upgraded package still reports an unsupported version, inspect the active prefix or
~/.config/openshell/gateway.toml
; an edited v1 file must follow the published schema-v2 migration steps and must not be overwritten. Guest TLS CA, certificate, and key paths are the exception to driver ownership: configure the complete bundle under
[openshell.gateway]
, and the gateway injects it only into the selected local driver. TLS-enabled Docker, Podman, and VM drivers fail startup when neither those paths nor the package-managed local bundle is available; Kubernetes projects its bundle through a Secret.
Custom names use
[openshell.drivers.<name>].socket_path
. A launch-time
--compute-driver-socket
override may also use
docker
,
podman
,
kubernetes
, or
vm
; the endpoint then takes precedence over built-in construction. First-party standalone drivers require the socket parent directory to be owned by the driver's effective UID, force its mode to
0700
, create the socket with mode
0600
, and accept only peers with that same UID. Check the parent and socket separately with
stat
; a gateway running under a different UID cannot connect even when filesystem permissions or group membership would otherwise allow it. Operator-supplied drivers must provide equivalent access control appropriate to their implementation. Check gateway logs for connection errors,
GetCapabilities
failures, or an unexpected advertised driver name. The advertised name is diagnostic metadata; negotiated features control optional behavior. The gateway does not create or supervise operator-supplied driver processes or sockets.
For configured gateway interceptors, inspect
[[openshell.gateway.interceptors]]
, their Unix or network endpoints, and gateway startup logs:
bash
rg -n 'interceptors|provider_profile_sources|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|binding_policy|failure_policy|gateway_jwt' /etc/openshell/gateway.toml
stat /run/openshell/interceptors/<name>.sock
journalctl -u <interceptor-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
The gateway calls each interceptor's
Describe
RPC and validates its manifest at startup. Check for unreachable endpoints, invalid RPC/phase bindings, strict
allowlist
or
exact
mismatches, and
post_commit
bindings that resolve to
fail_closed
. If gateway JWT signing is enabled, authenticated network interceptors require HTTPS and a valid bearer token; check the private CA path, endpoint hostname, expected audience, issuer,
kid
, and interceptor logs for token rejection.
allow_insecure_transport = true
explicitly preserves unauthenticated plaintext behavior. If
provider_profile_sources
names an interceptor, that interceptor must advertise provider-profile capability and return a valid, duplicate-free catalog. A selected interceptor-only source is authoritative; include
builtin
or
user
sources explicitly when composition is intended.
For operator-run supervisor middleware, inspect
[[openshell.supervisor.middleware]]
, service reachability, and both gateway and supervisor logs:
bash
rg -n 'supervisor|middleware|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|max_payload_bytes|timeout|gateway_jwt' /etc/openshell/gateway.toml
journalctl -u <middleware-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
openshell logs <sandbox-name> --tail --source sandbox
The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if
Describe
is unavailable, a manifest exposes duplicate operation/phase bindings, the registration claims the reserved
openshell/
namespace, or payload and timeout limits are invalid. Supported V1 bindings are
HTTP_REQUEST/PRE_CREDENTIALS
and
WEBSOCKET_MESSAGE/PRE_CREDENTIALS
. When gateway JWT signing is disabled, supervisors preserve the legacy unauthenticated connector and do not request extension credentials. When signing is enabled, credential acquisition and verification failures are fail closed: check HTTPS trust and hostname validation, audience and issuer agreement, the token
kid
, gateway
RefreshSandboxToken
errors, and middleware logs. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its
network_middlewares
config.
At request time, distinguish attachment, binding selection, coverage, denial, and failure. A host-matched HTTP-only attachment can inspect the upgrade GET but does not join the WebSocket chain; the connection proceeds under either
on_error
mode and emits
binding_not_selected
coverage. A selected WebSocket stage receives text messages only. Binary messages pass under both modes, emit
unsupported_message_type
coverage, and consume a session sequence without an RPC. An explicit
middleware_denied
result is always enforced. WebSocket preflight returns
INSPECT
, voluntary
SKIP
, or authoritative
DENY
;
DENY
rejects the upgrade before upstream contact under both
on_error
modes. A selected-stage failure follows the policy-local
on_error
:
fail_closed
blocks the HTTP request or closes the WebSocket, while
fail_open
bypasses only that stage and emits a detection finding. A fail-open per-message capacity failure bypasses that message without disabling the stage. A timeout, transport failure, stream closure, missing or invalid response, duplicate or regressed sequence, or other failure that makes an established WebSocket stream unreliable disables that stage for later messages on the connection and emits
openshell.middleware.websocket_stage_disabled
. Confirm preflight, session-start, and session-end in service logs. OpenShell best-effort sends at most one session-end to each still-writable opened stage, including a preflight that terminates before session start; distinguish
MIDDLEWARE_DENIAL
from
MIDDLEWARE_FAILURE
. WebSocket message sequences are allocated session-wide; each stage receives a strictly increasing subset, so gaps are valid when binary messages or other units are not delivered to that stage. Zero, duplicate, or regressed sequences are protocol errors. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
For network policy validation failures, first distinguish a gateway mutation rejection from a supervisor runtime rejection. Direct policy updates, incremental merges and approvals, provider attachments, and provider-profile fanout are validated against the complete effective policy before persistence when the gateway knows the affected sandbox scope. A
FAILED_PRECONDITION
ambiguity response means no invalid revision or partial fanout was stored. Supervisor validation remains defense in depth for startup, races, and policy sources outside those mutation paths.
Runtime rejection behavior is configured only in
gateway.toml
:
toml
[openshell.gateway]
policy_validation_failure_mode = "fail_closed"
The default
fail_closed
mode deactivates the previous generation, closes pinned relays, and quarantines new egress until a valid generation loads.
retain_last_valid
explicitly keeps the previous valid policy active; without one it still fails closed. Restart the gateway after changing this field. Inspect sandbox OCSF configuration and finding events for the validation rationale, configured and effective modes, active generation, and the explicit
previous_policy_active
state.
调试 compute platform 之前,应检查 gateway 日志中 listener ready 之前初始化的依赖是否存在失败。
对于 out-of-tree compute driver,请确认 CLI flag 或
gateway.toml
中选择的 driver 名称与 socket 一致,并确保 operator 管理的 driver 在 gateway 启动前已经运行:
bash
rg -n '^version|compute_driver|socket_path|guest_tls_' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
Gateway 配置要求
[openshell] version = 2
、单个
compute_driver
selector,并将 driver 自有设置放在
[openshell.drivers.<name>]
下。Gateway 会拒绝旧版的
compute_drivers
和
--drivers
selector,而不会静默迁移。只有一个有效且非空的
OPENSHELL_DRIVERS
值仍可作为已弃用的仅环境变量别名使用;canonical selector 缺失时,gateway 会选择该 driver 并给出警告。空值、无效值、以逗号分隔的值或冲突值都会导致启动失败。Homebrew 和 RPM package 的启动迁移仅处理由 package 生成的精确 v1 默认配置。如果升级后的 package 仍报告不支持的 version,应检查当前 prefix 或
~/.config/openshell/gateway.toml
;手工编辑过的 v1 文件必须遵循已发布的 schema-v2 迁移步骤,且不得被覆盖。
Guest TLS CA、certificate 和 key path 是 driver ownership 的例外:应在
[openshell.gateway]
下配置完整 bundle,gateway 仅会将其注入选定的本地 driver。启用 TLS 的 Docker、Podman 和 VM driver 在既无这些 path、也无 package 管理的本地 bundle 时会启动失败;Kubernetes 则通过 Secret 投射该 bundle。
自定义名称使用
[openshell.drivers.<name>].socket_path
。启动时的
--compute-driver-socket
override 也可以使用
docker
、
podman
、
kubernetes
或
vm
;此时 endpoint 的优先级高于内置构造。第一方 standalone driver 要求 socket 的父目录归 driver 的 effective UID 所有,强制将其 mode 设为
0700
,以 mode
0600
创建 socket,并且只接受具有相同 UID 的 peer。应分别使用
stat
检查父目录和 socket;以不同 UID 运行的 gateway 即使在文件权限或组成员关系本来允许的情况下也无法连接。Operator 提供的 driver 必须实施适合其实现的同等访问控制。检查 gateway 日志中的连接错误、
GetCapabilities
失败或意外声明的 driver 名称。声明的名称属于诊断 metadata;可选行为由协商后的 features 决定。Gateway 不会创建或监管 operator 提供的 driver process 或 socket。
对于已配置的 gateway interceptor,请检查
[[openshell.gateway.interceptors]]
、其 Unix 或 network endpoint 以及 gateway 启动日志:
bash
rg -n 'interceptors|provider_profile_sources|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|binding_policy|failure_policy|gateway_jwt' /etc/openshell/gateway.toml
stat /run/openshell/interceptors/<name>.sock
journalctl -u <interceptor-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
Gateway 会在启动时调用每个 interceptor 的
Describe
RPC,并验证其 manifest。检查 endpoint 不可达、RPC/phase binding 无效、严格的
allowlist
或
exact
不匹配,以及解析为
fail_closed
的
post_commit
binding。如果启用了 gateway JWT signing,经过认证的 network interceptor 必须使用 HTTPS 和有效的 bearer token;应检查 private CA path、endpoint hostname、预期 audience、issuer、
kid
以及 interceptor 日志中的 token rejection。
allow_insecure_transport = true
会明确保留未经认证的明文行为。如果
provider_profile_sources
指定了某个 interceptor,该 interceptor 必须声明 provider-profile capability,并返回有效且无重复项的 catalog。仅选择 interceptor 的 source 是权威来源;如果需要组合,应明确包含
builtin
或
user
source。
对于 operator 运行的 supervisor middleware,请检查
[[openshell.supervisor.middleware]]
、service 可达性以及 gateway 和 supervisor 日志:
bash
rg -n 'supervisor|middleware|grpc_endpoint|tls_ca_cert_path|audience|allow_insecure_transport|max_payload_bytes|timeout|gateway_jwt' /etc/openshell/gateway.toml
journalctl -u <middleware-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
openshell logs <sandbox-name> --tail --source sandbox
Middleware service 必须在 gateway 之前启动,并且 gateway 与 sandbox supervisor 都能访问它。如果
Describe
不可用、manifest 暴露了重复的 operation/phase binding、注册声明了保留的
openshell/
namespace,或者 payload 与 timeout limit 无效,则 gateway 启动失败。支持的 V1 binding 为
HTTP_REQUEST/PRE_CREDENTIALS
和
WEBSOCKET_MESSAGE/PRE_CREDENTIALS
。禁用 gateway JWT signing 时,supervisor 会保留旧版未经认证的 connector,也不会请求 extension credential。启用 signing 后,凭据获取和验证失败均采用 fail-closed 策略:检查 HTTPS trust 与 hostname validation、audience 和 issuer 是否一致、token
kid
、gateway
RefreshSandboxToken
错误以及 middleware 日志。修改注册后需要重启 gateway。如果选定的实现拒绝
network_middlewares
配置,policy update 也可能在持久化之前失败。
在请求时,应区分 attachment、binding selection、coverage、denial 和 failure。匹配 host 的纯 HTTP attachment 可以检查 upgrade GET,但不会加入 WebSocket chain;无论采用哪一种
on_error
模式,连接都会继续,并发出
binding_not_selected
coverage。被选中的 WebSocket stage 只接收 text message。Binary message 在两种模式下都会直接通过,发出
unsupported_message_type
coverage,并消耗一个 session sequence,但不会产生 RPC。显式的
middleware_denied
结果始终会被执行。WebSocket preflight 会返回
INSPECT
、主动
SKIP
或权威性的
DENY
;在两种
on_error
模式下,
DENY
都会在联系上游之前拒绝 upgrade。被选中的 stage 发生失败时遵循 policy-local
on_error
:
fail_closed
会阻止 HTTP request 或关闭 WebSocket;
fail_open
只会绕过该 stage 并发出 detection finding。Fail-open 模式下的单条消息 capacity 失败只会绕过该消息,不会禁用整个 stage。Timeout、transport failure、stream closure、响应缺失或无效、sequence 重复或倒退,以及其他导致已建立 WebSocket stream 不可靠的失败,都会为该连接后续消息禁用该 stage,并发出
openshell.middleware.websocket_stage_disabled
。应在 service 日志中确认 preflight、session-start 和 session-end。OpenShell 会尽力向每个仍可写入且已打开的 stage 最多发送一次 session-end,包括在 session start 前终止的 preflight;应区分
MIDDLEWARE_DENIAL
和
MIDDLEWARE_FAILURE
。WebSocket message sequence 按 session 范围分配;每个 stage 接收严格递增的子序列,因此当 binary message 或其他单元未传递给某个 stage 时,出现间隔是正常的。Sequence 为零、重复或倒退均属于 protocol error。如果运行中的 supervisor 无法安装新的 registry,它会保留最后一个已知正常的 generation,并发出 configuration failure event。
对于 network policy validation failure,首先应区分 gateway mutation rejection 和 supervisor runtime rejection。当 gateway 已知受影响的 sandbox scope 时,direct policy update、incremental merge 和 approval、provider attachment 以及 provider-profile fanout,都会在持久化之前针对完整的 effective policy 进行验证。
FAILED_PRECONDITION
ambiguity response 表示没有存储无效 revision,也没有执行部分 fanout。Supervisor validation 仍作为纵深防御,用于处理启动、竞态以及 mutation path 之外的 policy source。
Runtime rejection 行为只能在
gateway.toml
中配置:
toml
[openshell.gateway]
policy_validation_failure_mode = "fail_closed"
默认的
fail_closed
模式会停用上一 generation、关闭 pinned relay,并隔离新的 egress,直到有效的 generation 加载完成。
retain_last_valid
会明确保留上一个有效 policy;如果不存在有效 policy,仍采用 fail closed。修改此字段后应重启 gateway。检查 sandbox OCSF configuration 和 finding events,以了解 validation rationale、已配置及 effective mode、active generation 以及明确的
previous_policy_active
状态。

Step 4: Check Docker-Backed Gateways

第 4 步:检查 Docker 后端的 Gateway

bash
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
openshell status
For Docker GPU failures, check CDI support and NVIDIA CDI discovery separately:
bash
docker info --format '{{json .CDISpecDirs}}'
docker info --format '{{json .DiscoveredDevices}}'
for dir in /etc/cdi /var/run/cdi; do
  if [ -d "$dir" ]; then
    find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print
  else
    echo "$dir missing"
  fi
done
systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50
journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100
When the NVIDIA Container Toolkit CDI refresh units are not enabled or no NVIDIA CDI spec has been generated, enable them and trigger a refresh:
bash
sudo systemctl enable --now nvidia-cdi-refresh.path
sudo systemctl enable --now nvidia-cdi-refresh.service
sudo systemctl restart nvidia-cdi-refresh.service
docker info --format '{{json .DiscoveredDevices}}'
Common findings:
  • Docker daemon unavailable: start Docker Desktop or Docker Engine.
  • Gateway process stopped: inspect exit status and logs.
  • Sandbox image missing or pull denied: verify image reference and registry credentials.
  • Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI
    USER
    and matching
    /etc/passwd
    and
    /etc/group
    entries, or explicitly set both process identity fields in policy. Numeric workload identities
    1
    through
    4294967294
    are accepted; root, the invalid identity sentinel, and missing identities are rejected.
  • Sandbox fails before readiness with an OCI workspace validation error: inspect the image's
    WorkingDir
    using the immutable image ID reported by the gateway. Empty,
    /
    , and explicit
    /sandbox
    use the managed
    /sandbox
    compatibility workspace. Any other workdir must be an absolute normalized directory with no symlink components; the final policy UID, primary GID, and supplementary groups must pass the kernel's effective traverse/write checks, including POSIX ACL and LSM decisions. OpenShell does not create, chown, or chmod a non-default image workdir.
  • Docker also rejects an image
    VOLUME
    that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the
    VOLUME
    below the workspace or remove the declaration.
  • A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
  • Docker driver cannot initialize because it cannot find
    openshell-sandbox
    : verify
    OPENSHELL_DOCKER_SUPERVISOR_BIN
    , the sibling binary next to
    openshell-gateway
    , or the configured supervisor image contains
    /openshell-sandbox
    .
  • Sandbox never registers: check gateway logs and supervisor callback endpoint.
  • On macOS, repeated
    Policy fetch failed after 5 attempts
    messages with a Homebrew gateway bound to
    [::1]:17670
    indicate that the Docker
    host-gateway
    IPv4 route has no matching callback listener. Current releases leave
    bind_address
    unset in the Homebrew config, use the built-in
    127.0.0.1:17670
    primary listener, and reuse it for authenticated sandbox callbacks. On an older release, set
    bind_address = "127.0.0.1:17670"
    or upgrade.
  • Supervisor image exits before printing
    openshell-sandbox --version
    : verify the configured supervisor image contains a static executable at
    /openshell-sandbox
    .
  • A sandbox with explicit
    protocol: tcp
    endpoints fails before workload readiness: confirm the Docker or Podman driver supplied the
    policy-dns-transparent-tcp
    runtime capability and inspect supervisor logs for missing
    nft
    , synthetic-route overlap, or namespace-local DNS/TCP listener bind failures. Kubernetes, VM, sidecar, and out-of-tree drivers must reject this policy until they provide the complete substrate; use omitted protocol with an explicit proxy on those runtimes.
  • A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify
    .DiscoveredDevices
    contains entries such as
    nvidia.com/gpu=all
    , verify
    /etc/cdi
    or
    /var/run/cdi
    contains a generated NVIDIA spec, and check that
    nvidia-cdi-refresh.service
    and
    nvidia-cdi-refresh.path
    from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so
    inactive (dead)
    can be normal after a successful run; use
    systemctl status
    and
    journalctl
    to distinguish success from a skipped or failed refresh. Restart
    nvidia-cdi-refresh.service
    to regenerate missing or stale CDI specs, then restart or reload Docker and re-check
    docker info
    .
During a graceful gateway restart, Docker, Podman, and VM sandboxes with running intent should stop before the gateway exits and restart after it returns. Check for
Stopped sandbox during gateway shutdown
and
Started sandbox during gateway startup
in gateway logs. A sandbox explicitly stopped through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do not follow this local gateway lifecycle. Internal and external drivers follow the same rule:
GetCapabilities.gateway_manages_lifecycle
must be true for the gateway to run shutdown and startup sweeps.
bash
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
openshell status
对于 Docker GPU 故障,应分别检查 CDI 支持和 NVIDIA CDI discovery:
bash
docker info --format '{{json .CDISpecDirs}}'
docker info --format '{{json .DiscoveredDevices}}'
for dir in /etc/cdi /var/run/cdi; do
  if [ -d "$dir" ]; then
    find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print
  else
    echo "$dir missing"
  fi
done
systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true
systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50
journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100
如果 NVIDIA Container Toolkit CDI refresh unit 未启用,或尚未生成 NVIDIA CDI spec,应启用它们并触发 refresh:
bash
sudo systemctl enable --now nvidia-cdi-refresh.path
sudo systemctl enable --now nvidia-cdi-refresh.service
sudo systemctl restart nvidia-cdi-refresh.service
docker info --format '{{json .DiscoveredDevices}}'
常见发现:
  • Docker daemon 不可用:启动 Docker Desktop 或 Docker Engine。
  • Gateway process 已停止:检查 exit status 和日志。
  • Sandbox 镜像缺失或拉取被拒:验证 image reference 和 registry credential。
  • Sandbox 在 ready 之前因 identity-resolution error 失败:检查镜像的 OCI
    USER
    以及匹配的
    /etc/passwd
    和
    /etc/group
    条目,或者在 policy 中明确设置两个 process identity 字段。接受
    1
    到
    4294967294
    的数值型 workload identity;拒绝 root、无效 identity sentinel 和缺失的 identity。
  • Sandbox 在 ready 之前因 OCI workspace validation error 失败:使用 gateway 报告的不可变 image ID 检查镜像的
    WorkingDir
    。空值、
    /
    和显式的
    /sandbox
    使用受管理的
    /sandbox
    compatibility workspace。任何其他 workdir 都必须是绝对路径下的规范目录,且不能包含 symlink component;最终 policy UID、primary GID 和 supplementary group 必须通过 kernel 的 effective traverse/write 检查,包括 POSIX ACL 和 LSM 决策。OpenShell 不会创建、chown 或 chmod 非默认镜像 workdir。
  • 如果镜像的
    VOLUME
    覆盖 workdir 或其父目录,Docker 也会拒绝该镜像,因为 runtime 会在验证前屏蔽该不可变路径。应将
    VOLUME
    移到 workspace 下方或移除该声明。
  • 如果 workdir 因特殊 filesystem 或 OpenShell control-path collision 被拒绝,无法通过权限修复。应将镜像 workdir 移 away from kernel-backed mount,以及错误信息中列出的具体 supervisor、TLS、token、runtime 和 socket path。
  • Docker driver 因找不到
    openshell-sandbox
    而无法初始化:验证
    OPENSHELL_DOCKER_SUPERVISOR_BIN
    、位于
    openshell-gateway
    旁的同级 binary,或已配置的 supervisor image 中包含
    /openshell-sandbox
    。
  • Sandbox 始终未注册:检查 gateway 日志和 supervisor callback endpoint。
  • 在 macOS 上,如果 Homebrew gateway 绑定到
    [::1]:17670
    ,但反复出现
    Policy fetch failed after 5 attempts
    ,说明 Docker
    host-gateway
    IPv4 route 没有匹配的 callback listener。当前版本在 Homebrew config 中不设置
    bind_address
    ,使用内置的
    127.0.0.1:17670
    primary listener,并将它用于经过认证的 sandbox callback。旧版本应设置
    bind_address = "127.0.0.1:17670"
    或进行升级。
  • Supervisor image 在输出
    openshell-sandbox --version
    之前退出:验证已配置的 supervisor image 中包含位于
    /openshell-sandbox
    的 static executable。
  • 显式使用
    protocol: tcp
    endpoint 的 sandbox 在 workload ready 之前失败:确认 Docker 或 Podman driver 提供了
    policy-dns-transparent-tcp
    runtime capability,并检查 supervisor 日志中缺少
    nft
    、synthetic-route overlap 或 namespace-local DNS/TCP listener bind failure。在 Kubernetes、VM、sidecar 和 out-of-tree driver 提供完整 substrate 之前,必须拒绝该 policy;这些 runtime 应省略 protocol 并使用显式 proxy。
  • 如果 GPU sandbox 因 Docker 报告未发现 NVIDIA CDI device 而失败:验证
    .DiscoveredDevices
    包含
    nvidia.com/gpu=all
    等条目,验证
    /etc/cdi
    或
    /var/run/cdi
    中包含已生成的 NVIDIA spec,并检查 NVIDIA Container Toolkit 的
    nvidia-cdi-refresh.service
    和
    nvidia-cdi-refresh.path
    是否已启用且健康。该 service 是 one-shot unit,因此成功运行后显示
    inactive (dead)
    可能属于正常情况;应使用
    systemctl status
    和
    journalctl
    区分成功、跳过和失败。重启
    nvidia-cdi-refresh.service
    以重新生成缺失或过期的 CDI spec,然后重启或 reload Docker,并再次检查
    docker info
    。
在 gateway 平滑重启期间,具有 running intent 的 Docker、Podman 和 VM sandbox 应在 gateway 退出前停止,并在 gateway 返回后重启。在 gateway 日志中检查
Stopped sandbox during gateway shutdown
和
Started sandbox during gateway startup
。通过 CLI 显式停止的 sandbox 会保持停止状态。Kubernetes sandbox 由 cluster 持有,不遵循这一本地 gateway lifecycle。Internal 和 external driver 遵循相同规则:只有
GetCapabilities.gateway_manages_lifecycle
为 true 时,gateway 才会执行 shutdown 和 startup sweep。

Step 5: Check Podman-Backed Gateways

第 5 步:检查 Podman 后端的 Gateway

bash
podman info
podman ps --filter name=openshell
podman logs <container> --tail=200
openshell status
Common findings:
  • Podman socket unavailable: start or expose the user socket.
  • Rootless networking unavailable: inspect Podman network configuration.
  • Sandbox image missing or pull denied: verify image reference and registry credentials.
  • Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI
    USER
    and matching
    /etc/passwd
    and
    /etc/group
    entries, or explicitly set both process identity fields in policy. Numeric workload identities
    1
    through
    4294967294
    are accepted; root, the invalid identity sentinel, and missing identities are rejected.
  • Supervisor cannot call back: check callback endpoint and gateway logs.
  • A sandbox with explicit
    protocol: tcp
    endpoints fails before readiness: inspect supervisor logs for policy DNS port-53 binding, synthetic-route, or nftables redirect failures. Rootless Podman must provide these primitives inside the supervisor-owned nested network namespace; setup fails closed.
  • Gateway exits before becoming healthy with a callback-listener discovery error: inspect
    podman info --debug
    , the configured Podman network, and the host's IPv4 default route. Rootless pasta uses the private source address selected by that route; rootful Podman uses the bridge gateway address.
  • Current gateways reuse the primary listener when it covers Podman's callback address. If the primary does not cover that address, inspect the gateway startup logs for the additional callback-only listener and its provenance.
  • Rootless slirp4netns, another named helper, or missing helper metadata requires an explicitly remote
    grpc_endpoint
    . An explicit
    host_gateway_ip
    cannot bypass slirp4netns host-loopback isolation. Do not work around discovery failures by broadening the primary gateway listener to
    0.0.0.0
    .
When
userns
is configured (e.g.
userns = "auto"
or
userns = "keep-id"
):
  • Supervisor delivery uses bind-mount fallback instead of image volumes because overlay mounts do not support
    idmapped
    mounts. The supervisor binary is extracted from the supervisor image and cached at
    $XDG_DATA_HOME/openshell/podman-supervisor/
    (typically
    ~/.local/share/openshell/podman-supervisor/
    ).
  • Stale cache: if the supervisor image is updated but the cached binary is not refreshed, sandbox creation may fail with an ELF validation error or version mismatch. Remove the cache directory and retry.
  • auto
    mode requires subuid/subgid ranges for the current user in
    /etc/subuid
    and
    /etc/subgid
    . If missing, Podman returns a user-namespace mapping error at container creation.
  • private
    mode requires explicit
    uidmap
    and
    gidmap
    arrays in the TOML config. Without both, the gateway rejects the config at startup. Rootless Podman uses intermediate IDs (e.g.
    uidmap = ["0:0:1", "1:1:65535"]
    ); rootful Podman uses absolute host IDs (e.g.
    uidmap = ["0:1000:1", "1:100000:65536"]
    ).
  • nomap
    (without hyphen) is accepted as input but canonicalized to
    no-map
    for Podman's API.
  • A workload remains in
    stopping
    until Podman resorts to
    SIGKILL
    : inspect supervisor logs for
    failed to signal entrypoint process group
    . The supervisor must retain
    CAP_KILL
    so its root process can forward
    SIGTERM
    to a workload that runs as the sandbox user.
bash
podman info
podman ps --filter name=openshell
podman logs <container> --tail=200
openshell status
常见发现:
  • Podman socket 不可用:启动或暴露 user socket。
  • Rootless networking 不可用:检查 Podman network configuration。
  • Sandbox 镜像缺失或拉取被拒:验证 image reference 和 registry credential。
  • Sandbox 在 ready 之前因 identity-resolution error 失败:检查镜像的 OCI
    USER
    以及匹配的
    /etc/passwd
    和
    /etc/group
    条目,或者在 policy 中明确设置两个 process identity 字段。接受
    1
    到
    4294967294
    的数值型 workload identity;拒绝 root、无效 identity sentinel 和缺失的 identity。
  • Supervisor 无法 callback:检查 callback endpoint 和 gateway 日志。
  • 显式使用
    protocol: tcp
    endpoint 的 sandbox 在 ready 之前失败:检查 supervisor 日志中的 policy DNS port-53 binding、synthetic-route 或 nftables redirect failure。Rootless Podman 必须在 supervisor 持有的嵌套 network namespace 内提供这些 primitive;setup 采用 fail-closed 策略。
  • Gateway 在达到健康状态之前因 callback-listener discovery error 退出:检查
    podman info --debug
    、已配置的 Podman network 和 host 的 IPv4 default route。Rootless pasta 使用该 route 选出的 private source address;rootful Podman 使用 bridge gateway address。
  • 当前 gateway 会在 primary listener 覆盖 Podman callback address 时复用它。如果 primary 未覆盖该 address,应检查 gateway 启动日志中的额外 callback-only listener 及其来源。
  • Rootless slirp4netns、其他命名 helper 或缺失 helper metadata 要求显式配置 remote
    grpc_endpoint
    。显式的
    host_gateway_ip
    无法绕过 slirp4netns 的 host-loopback isolation。不要通过将 primary gateway listener 扩大到
    0.0.0.0
    来绕过 discovery failure。
配置
userns
时,例如
userns = "auto"
或
userns = "keep-id"
:
  • Supervisor delivery 使用 bind-mount fallback,而不是 image volume,因为 overlay mount 不支持
    idmapped
    mount。Supervisor binary 会从 supervisor image 中提取并缓存在
    $XDG_DATA_HOME/openshell/podman-supervisor/
    (通常为
    ~/.local/share/openshell/podman-supervisor/
    )。
  • 缓存过期:如果 supervisor image 已更新,但缓存的 binary 未刷新,sandbox creation 可能因 ELF validation error 或 version mismatch 而失败。删除 cache directory 后重试。
  • auto
    模式要求当前用户在
    /etc/subuid
    和
    /etc/subgid
    中具有 subuid/subgid range。如果缺失,Podman 会在创建 container 时返回 user-namespace mapping error。
  • private
    模式要求在 TOML config 中显式提供
    uidmap
    和
    gidmap
    array。缺少任意一个都会导致 gateway 在启动时拒绝配置。Rootless Podman 使用中间 ID,例如
    uidmap = ["0:0:1", "1:1:65535"]
    ;rootful Podman 使用绝对 host ID,例如
    uidmap = ["0:1000:1", "1:100000:65536"]
    。
  • nomap
    (不带连字符)可作为输入接受,但会针对 Podman API 规范化为
    no-map
    。
  • Workload 会一直保持
    stopping
    ,直到 Podman 最终使用
    SIGKILL
    :检查 supervisor 日志中的
    failed to signal entrypoint process group
    。Supervisor 必须保留
    CAP_KILL
    ,使其 root process 能以 sandbox user 身份向 workload 转发
    SIGTERM
    。

Step 6: Check Kubernetes Helm Gateways

第 6 步:检查 Kubernetes Helm Gateway

bash
helm -n openshell status openshell
helm -n openshell get values openshell
kubectl -n openshell get deployment,statefulset,pod,svc,pvc
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
kubectl -n openshell rollout status deployment/openshell
kubectl -n openshell rollout status statefulset/openshell
Use the log and rollout commands for the workload kind that exists in the release. Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures.
The chart mounts the
gateway.toml
ConfigMap key directly at
/etc/openshell/gateway.toml
as a read-only
subPath
file. This avoids the atomic-writer symlink exposed by a ConfigMap directory mount because the gateway rejects symlinked configuration. A checksum pod-template annotation rolls the workload when the ConfigMap changes. If config preflight reports a symlink or nonregular path, inspect the rendered mount and confirm the workload rolled to the current chart revision:
bash
kubectl -n openshell get deployment,statefulset -o yaml | rg -n 'gateway-config|mountPath|subPath|checksum/gateway-config'
kubectl -n openshell rollout status <deployment-or-statefulset>/openshell
kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200
server.telemetryEnabled
renders
OPENSHELL_TELEMETRY_ENABLED
on the gateway pod, and the gateway propagates the effective value to sandbox supervisors.
When no external credential driver is enabled, the Helm chart uses the gateway's default encrypted database credential storage. The chart creates a retained Kubernetes Secret for the shared KEK, injects it into gateway pods, and stores encrypted credential envelopes in the OpenShell database. For
workload.kind=deployment
or multi-replica gateways, confirm
server.externalDbSecret
points at a shared database. A render/install error mentioning
server.credentialDrivers
means the values selected multiple external credential backends.
For HA or PostgreSQL-backed installs, also check the external database Secret referenced by
server.externalDbSecret
and the PostgreSQL workload when it is deployed in-cluster:
bash
kubectl -n <namespace> get secret <external-db-secret> -o yaml
kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<postgres-workload>
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200
Check required Helm deployment secrets:
bash
kubectl -n openshell get secret \
  openshell-server-tls \
  openshell-server-client-ca \
  openshell-client-tls \
  openshell-jwt-keys
When
server.tls.clientCaSecretName=""
, the chart intentionally omits
client_ca_path
and the
tls-client-ca
mount, even with built-in PKI or cert-manager. That is expected; do not treat a missing
tls-client-ca
pod mount as a defect (
openshell-server-client-ca
may still exist from PKI). User auth is OIDC or trusted proxy (
server.auth.allowUnauthenticatedUsers=true
); supervisor transport still uses
openshell-client-tls
.
In cert-manager installs,
certManager.enabled=true
makes cert-manager own TLS generation. The Helm chart should still render the
openshell-certgen
pre-install/pre-upgrade hook in JWT-only mode to create
openshell-jwt-keys
, even if
pkiInitJob.enabled
remains true. If the gateway pod is pending with
MountVolume.SetUp failed for volume "sandbox-jwt"
and
openshell-jwt-keys
is absent, inspect the rendered
templates/certgen.yaml
output and the hook Job logs; cert-manager creates TLS Secrets but does not create the sandbox JWT signing Secret.
If the gateway exits with
failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pem
, verify that
openshell-jwt-keys
contains
signing.pem
,
public.pem
, and
kid
, and that the gateway workload mounts the
sandbox-jwt
secret at
/etc/openshell-jwt
. The sandbox JWT mount is required even when local Helm values disable TLS.
If
certManager.serverIssuerRef
points the server certificate at an external Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted cert on an OpenShift
Route
with TLS passthrough — see
openshiftRoute.enabled
), the chart creates two server certificates: an internal one (chart CA, internal SANs) and an external one (from the configured issuer, external SANs only). The gateway uses SNI to present the right cert.
Check the external
Certificate
/
CertificateRequest
/
Challenge
resources directly when the external secret never becomes Ready:
bash
kubectl -n openshell get certificate,certificaterequest,challenge
kubectl -n openshell describe certificate openshell-server-external
oc -n openshell get route
ACME issuers reject certificate requests that include internal-only names (
*.svc.cluster.local
,
localhost
, loopback IPs) and require the
commonName
to also be a SAN — the external
Certificate
only requests the hostnames in
certManager.serverDnsNames
, for exactly this reason.
If sandbox supervisors fail their TLS handshake to the gateway with
UnknownCA
after configuring
serverIssuerRef
, the most likely cause is
server.grpcEndpoint
set to the external hostname. This forces supervisors to connect via the external hostname, receiving the ACME cert (via SNI) which they cannot verify against the chart CA. Remove
server.grpcEndpoint
or set it to the internal service name so supervisors receive the internal cert:
bash
helm -n openshell get values openshell | grep -E 'grpcEndpoint|clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'
bash
helm -n openshell status openshell
helm -n openshell get values openshell
kubectl -n openshell get deployment,statefulset,pod,svc,pvc
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
kubectl -n openshell rollout status deployment/openshell
kubectl -n openshell rollout status statefulset/openshell
应根据 release 中实际存在的 workload 类型使用对应的 log 和 rollout command。查找 install 失败、意外 values、namespace 缺失、image tag 错误、与已注册 endpoint 不匹配的 TLS 设置以及调度失败。
Chart 会将
gateway.toml
ConfigMap key 以只读
subPath
file 的形式直接挂载到
/etc/openshell/gateway.toml
。这样可以避免 ConfigMap directory mount 暴露的 atomic-writer symlink,因为 gateway 会拒绝符号链接形式的配置。Checksum pod-template annotation 会在 ConfigMap 变化时滚动 workload。如果 config preflight 报告 symlink 或 nonregular path,应检查渲染后的 mount,并确认 workload 已滚动到当前 chart revision:
bash
kubectl -n openshell get deployment,statefulset -o yaml | rg -n 'gateway-config|mountPath|subPath|checksum/gateway-config'
kubectl -n openshell rollout status <deployment-or-statefulset>/openshell
kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200
server.telemetryEnabled
会在 gateway pod 上渲染
OPENSHELL_TELEMETRY_ENABLED
,gateway 会将 effective value 传播给 sandbox supervisor。
未启用 external credential driver 时,Helm chart 使用 gateway 默认的 encrypted database credential storage。Chart 会创建一个保留型 Kubernetes Secret 用于共享 KEK,将其注入 gateway pod,并把 encrypted credential envelope 存储在 OpenShell database 中。对于
workload.kind=deployment
或多副本 gateway,应确认
server.externalDbSecret
指向共享 database。如果 render/install error 中提到
server.credentialDrivers
,说明 values 选择了多个 external credential backend。
对于 HA 或使用 PostgreSQL 的安装,还应检查
server.externalDbSecret
引用的 external database Secret,以及 PostgreSQL workload 在 cluster 内部署时的状态:
bash
kubectl -n <namespace> get secret <external-db-secret> -o yaml
kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<postgres-workload>
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200
检查必需的 Helm deployment secret:
bash
kubectl -n openshell get secret \
  openshell-server-tls \
  openshell-server-client-ca \
  openshell-client-tls \
  openshell-jwt-keys
当
server.tls.clientCaSecretName=""
时,即使使用内置 PKI 或 cert-manager,chart 也会有意省略
client_ca_path
和
tls-client-ca
mount。这属于预期行为;不要将缺失的
tls-client-ca
pod mount 视为缺陷(PKI 仍可能创建
openshell-server-client-ca
)。用户认证使用 OIDC 或 trusted proxy(
server.auth.allowUnauthenticatedUsers=true
);supervisor transport 仍使用
openshell-client-tls
。
在 cert-manager 安装中,
certManager.enabled=true
表示由 cert-manager 管理 TLS 生成。即使
pkiInitJob.enabled
仍为 true,Helm chart 在 JWT-only 模式下仍应渲染
openshell-certgen
pre-install/pre-upgrade hook,以创建
openshell-jwt-keys
。如果 gateway pod 一直处于 pending 状态,并出现
MountVolume.SetUp failed for volume "sandbox-jwt"
,同时缺少
openshell-jwt-keys
,应检查渲染后的
templates/certgen.yaml
输出和 hook Job 日志;cert-manager 会创建 TLS Secret,但不会创建 sandbox JWT signing Secret。
如果 gateway 因
failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pem
退出,应验证
openshell-jwt-keys
包含
signing.pem
、
public.pem
和
kid
,并确认 gateway workload 将
sandbox-jwt
secret 挂载到
/etc/openshell-jwt
。即使本地 Helm values 禁用了 TLS,也必须挂载 sandbox JWT。
如果
certManager.serverIssuerRef
将 server certificate 指向 external Issuer 或 ClusterIssuer,例如在启用 TLS passthrough 的 OpenShift
Route
上使用面向公众受信任的 certificate——参见
openshiftRoute.enabled
——chart 会创建两个 server certificate:一个内部 certificate(chart CA、内部 SAN),一个外部 certificate(来自配置的 issuer,仅含外部 SAN)。Gateway 使用 SNI 提供正确的 certificate。
当 external secret 始终未进入 Ready 状态时,应直接检查 external
Certificate
/
CertificateRequest
/
Challenge
resource:
bash
kubectl -n openshell get certificate,certificaterequest,challenge
kubectl -n openshell describe certificate openshell-server-external
oc -n openshell get route
ACME issuer 会拒绝包含仅供内部使用名称的 certificate request,例如
*.svc.cluster.local
、
localhost
和 loopback IP,并要求
commonName
同时也是 SAN;external
Certificate
只请求
certManager.serverDnsNames
中的 hostname,正是出于这一原因。
如果配置
serverIssuerRef
后,sandbox supervisor 在 TLS handshake 期间因
UnknownCA
失败,最可能的原因是
server.grpcEndpoint
被设置为 external hostname。这会强制 supervisor 通过 external hostname 连接,从而收到无法针对 chart CA 验证的 ACME certificate。应删除
server.grpcEndpoint
,或将其设置为内部 service name,使 supervisor 获取内部 certificate:
bash
helm -n openshell get values openshell | grep -E 'grpcEndpoint|clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'

server.grpcEndpoint should be unset or point to internal service name

server.grpcEndpoint should be unset or point to internal service name


Less commonly, `UnknownCA` can occur if the gateway's client-verification CA
is misconfigured.  The default `clientCaFromServerTlsSecret=true` is correct
for all configurations — the internal server certificate is always signed by
the chart CA (the same CA that signs the client cert), so its `ca.crt` is
the right trust anchor.  Only override this if you intentionally mount a
separate client CA via `server.tls.clientCaSecretName`.  Verify the mounted
client CA matches the CA that signed the client certificate:

```bash
kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}' | jq .

较少见的情况是 gateway 的 client-verification CA 配置错误导致 `UnknownCA`。默认的 `clientCaFromServerTlsSecret=true` 适用于所有配置——内部 server certificate 始终由 chart CA 签名,该 CA 与 client certificate 的签名 CA 相同,因此其 `ca.crt` 是正确的 trust anchor。只有在明确通过 `server.tls.clientCaSecretName` 挂载单独的 client CA 时才应覆盖此默认值。验证挂载的 client CA 与签署 client certificate 的 CA 相匹配:

```bash
kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}' | jq .

Should show items filter for ca.crt from openshell-server-tls

Should show items filter for ca.crt from openshell-server-tls


If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still
render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret.
SPIRE is used by both the gateway and sandbox supervisors for dynamic provider
token grants. The gateway pod must mount the `spiffe-workload-api` CSI volume
and set `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`; sandbox pods must
receive the matching Workload API socket from the Kubernetes driver config.
The gateway verifies supervisor JWT-SVIDs from JWT bundles fetched through this
Workload API socket, not from the SPIRE OIDC discovery endpoint.
Verify that SPIRE is installed, the CSI driver is available, and the Kubernetes
driver config includes `provider_spiffe_workload_api_socket_path`:

```bash
helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath'
kubectl get pods -A | grep -E 'spire|spiffe'
kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{.items[*].spec.containers[*].env[?(@.name==\"OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET\")].value}{\"\n\"}"
Sandbox pods using provider token grants should have an
openshell.ai/sandbox-id
annotation, an
openshell.ai/managed-by=openshell
label, supervisor env vars
OPENSHELL_K8S_SA_TOKEN_FILE
and
OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET
, plus both the projected
openshell-sa-token
volume and the
spiffe-workload-api
CSI volume.
Check the image references currently used by the gateway deployment:
bash
kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'
The gateway and supervisor images should use the same release tag. A stale supervisor image can make sandbox behavior lag behind gateway policy or protocol changes.
For vulnerability reports, record the running image digest and scan that exact artifact. The gateway includes a pinned Distroless base; the supervisor includes Alpine packages updated at image build time. A dependency or base-image fix only reaches deployed containers after rebuilding, publishing, and redeploying the images. Compare findings against the SBOM for that digest, not just its mutable
latest
or
dev
tag.
For plaintext local evaluation, confirm the chart has:
bash
helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint'
Expected shape:
yaml
server:
  disableTls: true
  grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080
Check service exposure:
bash
kubectl -n openshell get svc openshell -o wide
kubectl -n openshell get endpoints openshell
For local port-forward testing:
bash
kubectl -n openshell port-forward service/openshell 8080:8080
Leave the port forward running. In another terminal, register the local endpoint if needed and verify it:
bash
openshell gateway add http://127.0.0.1:8080 --local --name local-kubernetes
openshell status
If the gateway is healthy but sandbox creation fails:
bash
kubectl -n openshell get pods
kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
Check the configured sandbox namespace:
bash
helm -n openshell get values openshell | grep sandboxNamespace
Then inspect sandbox resources in that namespace.
For a split release, the gateway values should have
workspaceResources.enabled=false
, and the target namespace should contain a separate
openshell-workspace
release:
bash
helm -n openshell get values openshell | grep -A2 workspaceResources
helm -n <sandbox-namespace> status openshell-workspace
kubectl -n <sandbox-namespace> get serviceaccount,role,rolebinding,networkpolicy \
  -l app.kubernetes.io/instance=openshell-workspace
kubectl auth can-i create sandboxes.agents.x-k8s.io \
  --namespace <sandbox-namespace> \
  --as system:serviceaccount:openshell:openshell
If the gateway cannot create or watch sandboxes, verify the workspace RoleBinding subject matches the gateway ServiceAccount name and namespace. If SSH relay connections fail, verify the workspace NetworkPolicy selects the gateway's actual
app.kubernetes.io/name
and
app.kubernetes.io/instance
labels.
Check the configured sandbox service account when TokenReview bootstrap or sandbox registration fails. Helm creates a dedicated sandbox service account by default and writes it to
[openshell.drivers.kubernetes].service_account_name
; the selected Kubernetes compute driver rejects projected tokens from other service accounts. For an external driver, inspect its logs and confirm it advertises
supports_sandbox_authentication
; the gateway delegates the opaque credential over the driver socket and never interprets Kubernetes settings.
bash
helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
If
topology = "sidecar"
is rendered under
[openshell.drivers.kubernetes]
, sandbox pods should have an
openshell-network-init
init container running
--mode=network-init
, an
agent
container running
openshell-sandbox --mode=process
, and an
openshell-supervisor-network
container running
--mode=network
. The init container owns nftables setup and should be the only sidecar topology container with
NET_ADMIN
. It also needs
CHOWN
/
FOWNER
to hand shared emptyDir state to the effective sidecar UID. The default binary-aware network sidecar runs as UID 0 with primary GID
sandbox_gid
and adds
SYS_PTRACE
plus
DAC_READ_SEARCH
. When
process_binary_aware_network_policy = false
, it runs as the configured non-root
proxy_uid
without those inspection capabilities. That dedicated proxy UID must remain at least
1000
and must not match the workload UID because the pod egress fence exempts its traffic. The pod
fsGroup
is set to
sandbox_gid
in both modes.
In sidecar topology only the network sidecar should mount the gateway bootstrap credentials (
openshell-sa-token
and
openshell-client-tls
). The process container should not receive
OPENSHELL_ENDPOINT
, gateway TLS env vars, the sandbox token file, or those credential mounts. Instead, the network sidecar serves policy and provider environment state over the Unix control socket from
OPENSHELL_SIDECAR_CONTROL_SOCKET
(
/run/openshell-sidecar/control.sock
by default). The process supervisor must be the first and only client. After validating its peer UID, GID, and PID, the sidecar unlinks the listener. If the connection later closes, the network sidecar exits non-zero so Kubernetes can restart it with a fresh listener. If the process supervisor fails before launching the workload, inspect both containers for control-socket bind, connect, bootstrap, or update errors. If new SSH/exec sessions do not pick up refreshed provider environment, inspect the network sidecar settings-poll logs and the process container logs for provider environment update handling; the process container should consume newer provider-env revisions without receiving gateway credentials.
The process container reports the workload entrypoint PID over the same control socket, and the network sidecar uses that PID for binary-scoped policy decisions through
/proc
. If rules with
policy.binaries
are unexpectedly denied, inspect the sidecar control logs and confirm the pod has
shareProcessNamespace: true
. The shared state directory should preserve
sandbox_gid
inheritance (
02775
). Sidecar SSH uses the Linux abstract socket
@openshell-sidecar-ssh
; the network sidecar verifies its peer PID before bridging gateway relay requests. No
ssh.sock
file should appear in the shared state directory. Inspect all three when sandbox registration or egress enforcement fails:
bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-network-init --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200

如果 `server.providerTokenGrants.spiffe.enabled=true`,gateway 仍应渲染 `[openshell.gateway.gateway_jwt]` 并挂载 `sandbox-jwt` Secret。SPIRE 同时供 gateway 和 sandbox supervisor 用于 dynamic provider token grant。Gateway pod 必须挂载 `spiffe-workload-api` CSI volume,并设置 `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`;sandbox pod 必须从 Kubernetes driver config 获得匹配的 Workload API socket。Gateway 通过此 Workload API socket 获取的 JWT bundle 验证 supervisor JWT-SVID,而不是通过 SPIRE OIDC discovery endpoint。验证 SPIRE 已安装、CSI driver 可用,并且 Kubernetes driver config 包含 `provider_spiffe_workload_api_socket_path`:

```bash
helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath'
kubectl get pods -A | grep -E 'spire|spiffe'
kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path
kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{.items[*].spec.containers[*].env[?(@.name==\"OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET\")].value}{\"\n\"}"
使用 provider token grant 的 sandbox pod 应具有
openshell.ai/sandbox-id
annotation、
openshell.ai/managed-by=openshell
label、supervisor 环境变量
OPENSHELL_K8S_SA_TOKEN_FILE
和
OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET
,以及投射的
openshell-sa-token
volume 和
spiffe-workload-api
CSI volume。
检查 gateway deployment 当前使用的 image reference:
bash
kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}"
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'
Gateway 和 supervisor image 应使用相同的 release tag。过期的 supervisor image 可能导致 sandbox 行为落后于 gateway policy 或 protocol 变更。
对于 vulnerability report,应记录运行中 image 的 digest,并扫描该确切 artifact。Gateway 包含固定版本的 Distroless base;supervisor 包含在 image build 时更新的 Alpine package。Dependency 或 base-image 修复只有在重新构建、发布并重新部署 image 后才能进入已部署的 container。应针对该 digest 的 SBOM 比较发现,而不能只使用可变的
latest
或
dev
tag。
对于明文本地评估,确认 chart 中存在:
bash
helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint'
预期结构:
yaml
server:
  disableTls: true
  grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080
检查 service exposure:
bash
kubectl -n openshell get svc openshell -o wide
kubectl -n openshell get endpoints openshell
对于本地 port-forward 测试:
bash
kubectl -n openshell port-forward service/openshell 8080:8080
保持 port forward 运行。在另一个 terminal 中,根据需要注册本地 endpoint 并验证:
bash
openshell gateway add http://127.0.0.1:8080 --local --name local-kubernetes
openshell status
如果 gateway 健康但 sandbox creation 失败:
bash
kubectl -n openshell get pods
kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
检查已配置的 sandbox namespace:
bash
helm -n openshell get values openshell | grep sandboxNamespace
然后检查该 namespace 中的 sandbox resource。
对于 split release,gateway values 应包含
workspaceResources.enabled=false
,并且目标 namespace 中应存在独立的
openshell-workspace
release:
bash
helm -n openshell get values openshell | grep -A2 workspaceResources
helm -n <sandbox-namespace> status openshell-workspace
kubectl -n <sandbox-namespace> get serviceaccount,role,rolebinding,networkpolicy \
  -l app.kubernetes.io/instance=openshell-workspace
kubectl auth can-i create sandboxes.agents.x-k8s.io \
  --namespace <sandbox-namespace> \
  --as system:serviceaccount:openshell:openshell
如果 gateway 无法创建或 watch sandbox,应验证 workspace RoleBinding subject 与 gateway ServiceAccount 名称和 namespace 相匹配。如果 SSH relay connection 失败,应验证 workspace NetworkPolicy 是否选择了 gateway 实际使用的
app.kubernetes.io/name
和
app.kubernetes.io/instance
label。
当 TokenReview bootstrap 或 sandbox registration 失败时,检查已配置的 sandbox service account。Helm 默认创建专用的 sandbox service account,并将其写入
[openshell.drivers.kubernetes].service_account_name
;选定的 Kubernetes compute driver 会拒绝来自其他 service account 的 projected token。对于 external driver,应检查其日志并确认它声明了
supports_sandbox_authentication
;gateway 通过 driver socket 传递 opaque credential,且绝不解释 Kubernetes 设置。
bash
helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
如果在
[openshell.drivers.kubernetes]
下渲染出
topology = "sidecar"
,则 sandbox pod 应具有一个运行
--mode=network-init
的
openshell-network-init
init container、一个运行
openshell-sandbox --mode=process
的
agent
container,以及一个运行
--mode=network
的
openshell-supervisor-network
container。Init container 负责 nftables setup,并且应是 sidecar topology container 中唯一拥有
NET_ADMIN
的 container。它还需要
CHOWN
/
FOWNER
,以便将共享的 emptyDir state 移交给 effective sidecar UID。默认的 binary-aware network sidecar 以 UID 0 运行,primary GID 为
sandbox_gid
,并添加
SYS_PTRACE
和
DAC_READ_SEARCH
。当
process_binary_aware_network_policy = false
时,它以配置的 non-root
proxy_uid
运行,且不具备这些 inspection capability。该专用 proxy UID 必须至少为
1000
,并且不能与 workload UID 相同,因为 pod egress fence 会豁免它的流量。两种模式下的 pod
fsGroup
均设置为
sandbox_gid
。
在 sidecar topology 中,只有 network sidecar 应挂载 gateway bootstrap credential(
openshell-sa-token
和
openshell-client-tls
)。Process container 不应接收
OPENSHELL_ENDPOINT
、gateway TLS 环境变量、sandbox token file 或这些 credential mount。相反,network sidecar 通过
OPENSHELL_SIDECAR_CONTROL_SOCKET
(默认为
/run/openshell-sidecar/control.sock
)上的 Unix control socket 提供 policy 和 provider environment state。Process supervisor 必须是第一个且唯一的 client。验证 peer UID、GID 和 PID 后,sidecar 会 unlink listener。如果 connection 随后关闭,network sidecar 会以非零状态退出,让 Kubernetes 使用新的 listener 重启它。如果 process supervisor 在启动 workload 前失败,应检查两个 container 中的 control-socket bind、connect、bootstrap 或 update error。如果新的 SSH/exec session 未获取更新后的 provider environment,应检查 network sidecar 的 settings-poll 日志和 process container 的 provider environment update handling 日志;process container 应消费较新的 provider-env revision,而无需接收 gateway credential。
Process container 通过同一个 control socket 报告 workload entrypoint PID,network sidecar 使用该 PID 通过
/proc
做出 binary-scoped policy decision。如果包含
policy.binaries
的 rule 意外被拒绝,应检查 sidecar control log,并确认 pod 设置了
shareProcessNamespace: true
。
共享 state directory 应保留
sandbox_gid
inheritance(
02775
)。Sidecar SSH 使用 Linux abstract socket
@openshell-sidecar-ssh
;network sidecar 在桥接 gateway relay request 前会验证 peer PID。共享 state directory 中不应出现
ssh.sock
file。
当 sandbox registration 或 egress enforcement 失败时,检查以下三个 container:
bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}'
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-network-init --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200

Corporate upstream proxy

Corporate upstream proxy

When the deployment routes sandbox egress through a corporate HTTP forward proxy, the operator-owned settings render under
[openshell.drivers.kubernetes]
from the Helm
upstreamProxy
values. Absent proxy configuration preserves direct-dial egress; any present-but-invalid value fails closed at gateway startup (
validate_upstream_proxy_config
) rather than silently reverting to a direct connection. Confirm the rendered configuration first:
bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname'
helm -n openshell get values openshell | grep -A8 upstreamProxy
Only
http://host:port
forward proxies are supported;
https://
proxy URLs and plain-HTTP egress are out of scope and rejected. Proxy credentials require
topology = "sidecar"
— combined topology shares the credential mount with the workload, so the gateway rejects credentials there. The credential Secret named by
proxy_auth_secret_name
must exist in the sandbox namespace with the key named by
proxy_auth_secret_key
, and Kubernetes will not create keys longer than 253 bytes or named
.
/
..
.
The proxy arguments and credential mount are injected only into the container that runs network supervision (the
agent
container in combined topology, the
openshell-supervisor-network
sidecar in sidecar topology). The one-shot
openshell-network-init
container and the process
agent
container in sidecar topology must never receive them. The credential is projected read-only as the
openshell-upstream-proxy-auth
volume at
/run/openshell/upstream-proxy-auth
and passed as
--upstream-proxy-auth-file
; it must never appear in env, annotations, or command arguments.
bash
kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20
A missing Secret or wrong key leaves the pod stuck with a
MountVolume.SetUp failed
/
secret ... not found
event. If the pod starts but egress still fails, the corporate proxy itself is the next suspect: policy- approved TLS CONNECT requests that time out after policy evaluation usually mean the proxy URL is unreachable from the sandbox namespace, or a cluster-internal destination that should be direct is missing from
no_proxy
. Inspect the network supervisor logs for CONNECT and upstream-proxy decisions:
bash
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy'
当 deployment 通过 corporate HTTP forward proxy 路由 sandbox egress 时,operator 管理的设置会根据 Helm
upstreamProxy
values 渲染到
[openshell.drivers.kubernetes]
下。未配置 proxy 时会保留 direct-dial egress;任何存在但无效的值都会在 gateway startup 时 fail closed(
validate_upstream_proxy_config
),而不会静默回退到 direct connection。首先确认渲染后的配置:
bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E 'https_proxy|no_proxy|proxy_auth_secret_(name|key)|proxy_auth_allow_insecure|proxy_connect_by_hostname'
helm -n openshell get values openshell | grep -A8 upstreamProxy
仅支持
http://host:port
forward proxy;不支持
https://
proxy URL 和 plain-HTTP egress,这些配置会被拒绝。Proxy credential 要求
topology = "sidecar"
——combined topology 会让 credential mount 与 workload 共享,因此 gateway 会拒绝该配置。
proxy_auth_secret_name
指定的 credential Secret 必须存在于 sandbox namespace 中,并包含
proxy_auth_secret_key
指定的 key;Kubernetes 不会创建 key 长度超过 253 bytes 或名称为
.
/
..
的 Secret。
Proxy argument 和 credential mount 仅注入运行 network supervision 的 container:在 combined topology 中为
agent
container,在 sidecar topology 中为
openshell-supervisor-network
sidecar。One-shot
openshell-network-init
container 以及 sidecar topology 中的 process
agent
container 绝不能接收它们。Credential 会以只读方式投射为
openshell-upstream-proxy-auth
volume,路径为
/run/openshell/upstream-proxy-auth
,并通过
--upstream-proxy-auth-file
传入;它绝不能出现在环境变量、annotation 或 command argument 中。
bash
kubectl -n <sandbox-namespace> get secret <proxy-auth-secret> -o jsonpath='{.data}' >/dev/null && echo "secret present"
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-'
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth
kubectl -n <sandbox-namespace> get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20
如果 Secret 缺失或 key 不正确,pod 会停留在
MountVolume.SetUp failed
/
secret ... not found
event 状态。如果 pod 已启动但 egress 仍然失败,下一个排查对象是 corporate proxy 本身:policy 批准的 TLS CONNECT request 在 policy evaluation 后超时,通常意味着 proxy URL 无法从 sandbox namespace 访问,或者应直连的 cluster-internal destination 没有包含在
no_proxy
中。检查 network supervisor 日志中的 CONNECT 和 upstream-proxy decision:
bash
kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy'

Step 7: Check VM-Backed Gateways

第 7 步:检查 VM 后端的 Gateway

Use the VM driver logs and host diagnostics available in the user's environment. Verify:
  • The VM driver process is running and reachable by the gateway.
  • The runtime rootfs exists and matches the expected architecture.
  • mke2fs
    or
    mkfs.ext4
    and
    debugfs
    from e2fsprogs are installed; explicit
    sandbox_uid
    /
    sandbox_gid
    does not remove this prerequisite.
  • A persisted overlay identity error is resolved from its owner marker, overlay upper layer, prepared rootfs, explicit config, or current image. Do not assign
    10001:10001
    unless the persisted state reports that legacy identity.
  • Host virtualization support is enabled.
  • The sandbox supervisor can establish its callback connection to the gateway.
Then run:
bash
openshell status
openshell logs <sandbox-name>
使用用户环境中可用的 VM driver 日志和 host diagnostic。验证:
  • VM driver process 正在运行且 gateway 可以访问。
  • Runtime rootfs 存在,并且与预期 architecture 匹配。
  • 已安装 e2fsprogs 中的
    mke2fs
    或
    mkfs.ext4
    和
    debugfs
    ;显式设置
    sandbox_uid
    /
    sandbox_gid
    并不会取消此前置条件。
  • Persisted overlay identity error 应根据其 owner marker、overlay upper layer、prepared rootfs、explicit config 或 current image 解决。除非 persisted state 报告了 legacy identity,否则不要分配
    10001:10001
    。
  • Host virtualization support 已启用。
  • Sandbox supervisor 可以建立到 gateway 的 callback connection。
然后运行:
bash
openshell status
openshell logs <sandbox-name>

Corporate upstream proxy

Corporate upstream proxy

When VM sandbox egress routes through a corporate HTTP forward proxy, the operator-owned settings live under
[openshell.drivers.vm]
and the gateway forwards them to the
openshell-driver-vm
subprocess as
--upstream-proxy
,
--upstream-no-proxy
,
--upstream-proxy-auth-file
,
--upstream-proxy-auth-allow-insecure
,
--upstream-proxy-connect-by-hostname
, and
--upstream-proxy-ca-bundle
. Both the gateway and the driver validate them at startup, so any present-but-invalid value fails closed with an error naming the key rather than reverting to a direct dial. Confirm the configuration and the resulting driver argv first:
bash
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy'
Reachability is the most common failure, and it depends on the VM backend. On libkrun (non-GPU sandboxes) guest egress leaves through gvproxy, so a proxy bound to the gateway host's loopback is not reachable at
127.0.0.1
from inside the guest: it must be addressed as
http://host.openshell.internal:<port>
, which gvproxy NATs from
192.168.127.254
to the host's
127.0.0.1
. A
https_proxy
pointing at a loopback URL produces policy-approved CONNECT attempts that time out while public destinations still work.
GPU sandboxes run on QEMU/TAP, where no gateway-host proxy is reachable at all:
host.openshell.internal
resolves to the TAP host address, and the driver's nftables
input
chain accepts only the gateway port from the guest. The driver rejects such a configuration at launch — a create failing with
https_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ... cannot reach
means the proxy must move to an address routable from the guest's masqueraded egress (or the sandbox must run without a GPU).
The settings reach the supervisor through a driver-written argument file in the per-sandbox overlay, not through the guest environment. The credential and CA bundle are staged into the same overlay at fixed guest paths. Inspect the guest side from the VM console log, which records how many driver-supplied arguments the init script read:
bash
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40
FATAL: supervisor argument list ... is not readable
or
FATAL: empty entry in supervisor argument list
means the overlay is broken or was tampered with, and the guest deliberately aborts rather than starting a supervisor with a truncated egress configuration. If the guest logs no driver arguments at all while
gateway.toml
sets
https_proxy
, the running driver predates the configuration — check that the gateway spawned the driver binary you expect (
[openshell.drivers.vm].driver_dir
).
当 VM sandbox egress 通过 corporate HTTP forward proxy 路由时,operator 管理的设置位于
[openshell.drivers.vm]
下,gateway 会将其作为
--upstream-proxy
、
--upstream-no-proxy
、
--upstream-proxy-auth-file
、
--upstream-proxy-auth-allow-insecure
、
--upstream-proxy-connect-by-hostname
和
--upstream-proxy-ca-bundle
转发给
openshell-driver-vm
subprocess。Gateway 和 driver 都会在启动时验证这些设置,因此任何存在但无效的值都会 fail closed,并返回指明具体 key 的错误,而不会回退到 direct dial。首先确认配置和最终生成的 driver argv:
bash
grep -A20 '^\[openshell.drivers.vm\]' <gateway.toml> | grep -E 'https_proxy|no_proxy|proxy_auth_file|proxy_auth_allow_insecure|proxy_connect_by_hostname|proxy_ca_bundle'
ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy'
可达性是最常见的故障,并且取决于 VM backend。在 libkrun(non-GPU sandbox)上,guest egress 通过 gvproxy 离开,因此绑定到 gateway host loopback 的 proxy 无法从 guest 内部通过
127.0.0.1
访问:必须使用
http://host.openshell.internal:<port>
,gvproxy 会将
192.168.127.254
NAT 到 host 的
127.0.0.1
。指向 loopback URL 的
https_proxy
会产生经过 policy 批准的 CONNECT attempt 并发生超时,同时 public destination 仍可能正常工作。
GPU sandbox 运行在 QEMU/TAP 上,其中 gateway host proxy 完全不可达:
host.openshell.internal
会解析到 TAP host address,driver 的 nftables
input
chain 只允许来自 guest 的 gateway port。Driver 会在启动时拒绝此类配置;create 失败并显示
https_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ... cannot reach
,意味着 proxy 必须迁移到 guest 的 masqueraded egress 可路由的地址,或者 sandbox 必须在不使用 GPU 的情况下运行。
这些设置通过 driver 在每个 sandbox overlay 中写入的 argument file 传递给 supervisor,而不是通过 guest environment。Credential 和 CA bundle 会被暂存到同一 overlay 的固定 guest path 中。从 VM console log 检查 guest side,该日志会记录 init script 读取了多少个 driver-supplied argument:
bash
grep -E 'supervisor arguments from driver|supervisor argument list' <state_dir>/sandboxes/<id>/rootfs-console.log
grep -Ei 'upstream|connect|proxy' <state_dir>/sandboxes/<id>/rootfs-console.log | tail -n 40
FATAL: supervisor argument list ... is not readable
或
FATAL: empty entry in supervisor argument list
意味着 overlay 已损坏或遭到篡改;guest 会故意中止,而不是使用被截断的 egress configuration 启动 supervisor。如果
gateway.toml
设置了
https_proxy
,但 guest 日志完全没有 driver argument,则说明当前运行的 driver 早于该配置;检查 gateway 是否启动了预期的 driver binary(
[openshell.drivers.vm].driver_dir
)。

Common Failure Patterns

常见故障模式

SymptomLikely causeCheck
openshell status
fails
Gateway endpoint unreachable or auth mismatch
openshell gateway info
, gateway logs
BatchSpanProcessor.ExportError
repeatedly reports connection refused on
127.0.0.1:4317
The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manuallyRestart
gateway:docker
,
gateway:podman
, or
gateway:vm
so it re-detects the listener; inspect the generated
gateway.toml
for
[openshell.gateway.otlp]
Gateway starts but sandbox create failsCompute driver cannot reach runtimeDocker/Podman/Kubernetes/VM driver logs
Gateway exits while resolving compute-driver listener requirementsCallback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorizedGateway startup error,
podman info --debug
, Podman network inspection, host IPv4 default route
Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listenerAdditional callback listeners intentionally expose only sandbox-callable gRPC methodsRetry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected
Docker or Podman sandbox never registersWrong callback endpoint or supervisor startup failureGateway logs and sandbox container logs
Docker GPU sandbox fails before startupNVIDIA CDI specs are missing or Docker has not discovered them
docker info --format '{{json .DiscoveredDevices}}'
,
/etc/cdi
,
/var/run/cdi
,
nvidia-cdi-refresh.service
Kubernetes gateway pod pendingPVC unbound, taint, selector, or insufficient resources
kubectl -n openshell describe pod <pod>
Kubernetes sandbox pod stuck pending, workspace PVC unboundCluster has no default
StorageClass
and OpenShell does not set
storageClassName
on the workspace PVC (clusters with a default
StorageClass
bind fine without it)
kubectl -n openshell describe pvc
; set
server.workspaceStorageClass
(gateway config
workspace_storage_class
) to a valid
StorageClass
Kubernetes gateway pod crash loopsMissing secret, bad DB URL, bad TLS config
kubectl -n openshell logs deployment/openshell -c openshell-gateway
or
kubectl -n openshell logs statefulset/openshell -c openshell-gateway
OpenShift gateway pod fails to start with an SCC/
runAsUser
error (e.g.
unable to validate against any security context constraint
)
Chart's default
podSecurityContext
/
securityContext
hardcodes
runAsUser
/
fsGroup
, which the restricted-v2 SCC rejects; it must instead inject the namespace-assigned UID/GID range
oc -n openshell describe pod <pod>
; deploy with
podSecurityContext: null
and clear
securityContext.runAsUser
(see
deploy/helm/openshell/ci/values-openshift-scc.yaml
)
OpenShift sandbox pod fails to start (
unable to validate against any security context constraint
)
The
openshell-sandbox
service account lacks the privileged SCC it needs
oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell
; remove with
remove-scc-from-user
when done
CLI TLS errorLocal mTLS bundle does not match server cert/CACheck
~/.config/openshell/gateways/<name>/mtls/
Edge or OIDC gateway returns
Unauthenticated
Stored login expired, audience/scopes mismatch, or gateway auth configuration changed
openshell gateway info
,
openshell gateway login <name>
, gateway auth logs
Gateway fails before serving health after enabling an interceptorInterceptor endpoint unavailable or manifest/binding validation failedGateway and interceptor logs; interceptor socket;
binding_policy
, phases, and failure policy
Authenticated interceptor or middleware rejects gateway callsPrivate CA or hostname mismatch, expected audience or issuer mismatch, stale/unknown
kid
, or malformed extension token
tls_ca_cert_path
, registration
audience
, service verifier config and logs; fetch well-known metadata only through the already-trusted gateway TLS endpoint
Provider profiles disappear after enabling an interceptor catalog
provider_profile_sources
selected only an authoritative interceptor or returned invalid/duplicate IDs
Inspect source list and interceptor
Describe
/catalog logs; include
builtin
and
user
when intended
Gateway fails after registering supervisor middlewareService unavailable, invalid manifest, duplicate binding, reserved name, or invalid payload/timeout limitMiddleware service and gateway logs;
[[openshell.supervisor.middleware]]
;
Describe
response
Policy update rejects
network_middlewares
Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of
tls: skip
Policy error, gateway logs, middleware
ValidateConfig
, selector and order fields
Policy mutation returns
FAILED_PRECONDITION
for endpoint ambiguity
Equally specific effective endpoint selectors disagree on connection or request-processing metadataCLI error, base and provider-composed policy, affected profile attachments; confirm no new revision was stored
Supervisor enters policy quarantineA runtime candidate failed validation while
policy_validation_failure_mode = "fail_closed"
Sandbox OCSF config/finding events, validation rationale, active generation,
previous_policy_active
HTTP request returns
middleware_failed
or
middleware_denied
, or WebSocket closes with
1008
Selected stage failed or explicitly denied admitted trafficSandbox OCSF logs; policy-local middleware config; service availability; binding operation;
on_error
WebSocket upgrades but a host-matched middleware receives no preflight or message RPCThe implementation did not advertise
WEBSOCKET_MESSAGE/PRE_CREDENTIALS
WEBSOCKET_MIDDLEWARE_COVERAGE state=binding_not_selected
; service
Describe
; the upgrade GET may still have used its HTTP binding
Binary WebSocket message passes without a middleware RPCBinary is unsupported by the V1 text-message binding under both
on_error
modes
WEBSOCKET_MIDDLEWARE_COVERAGE state=unsupported_message_type
; the next text RPC may have a valid sequence gap
WebSocket messages stop reaching middleware after one failureA fail-open stage stream was disabled for the rest of the connection
openshell.middleware.websocket_stage_disabled
; middleware timeout/stream/protocol logs. A per-message capacity bypass alone leaves the stage active. Reconnect to create a fresh stream after a genuine stream failure
Supervisor repeatedly fails to install middleware after enabling gateway JWT signingExtension credential minting, distribution, or authenticated service connection failed; last-known-good registry remains activeGateway
RefreshSandboxToken
logs, sandbox configuration events, service token-verification logs, registration TLS/audience settings
Custom compute driver is unavailableDriver process/socket missing, inaccessible, or selected name does not match its endpoint/config keySocket ownership/mode, driver service logs, gateway
GetCapabilities
logs
Sandbox remains
Stopping
or
Starting
Driver stop/start failed, retained resource is missing, or a fresh supervisor has not connectedGateway and driver logs;
docker inspect
,
podman inspect
, Agent Sandbox status/PVC, or VM state marker and launcher process
Image pull failureGateway or sandbox image cannot be pulledRuntime events and image pull credentials
Gateway API resources fail with
the server could not find the requested resource
Optional Gateway API resources were applied without Envoy Gateway CRDsInstall Envoy Gateway and enable
grpcRoute
before applying the optional ingress resources
HTTPS ingress (
grpcRoute.gateway.listener.protocol=HTTPS
) connection resets or TLS handshake hangs
Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop failsSet
server.disableTls=true
so Envoy forwards plaintext to the pod; verify the listener
certificateRefs
Secret exists in the release namespace and
openshell status
over
https://<host>
HTTPS ingress returns
Unauthenticated
after connecting
TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identityConfigure
server.oidc.issuer
and register with
openshell gateway add https://<host> --oidc-issuer <url>
, or set
server.auth.allowUnauthenticatedUsers=true
for a trusted-proxy/dev cluster
External server
Certificate
never becomes Ready with
certManager.serverIssuerRef
set
ACME issuer rejected internal-only SANs, a loopback IP, or a
commonName
absent from the SANs
kubectl -n openshell describe certificate openshell-server-external
; confirm
certManager.serverDnsNames
lists only real, externally-resolvable hostnames
Sandbox supervisors fail TLS handshake with
UnknownCA
after configuring
certManager.serverIssuerRef
server.grpcEndpoint
is set to the external hostname, forcing supervisors to receive the ACME cert (via SNI) which they can't verify against chart CA
Remove
server.grpcEndpoint
or set it to the internal service name; supervisors should connect via internal service name to receive the internal cert
Browser
ERR_BAD_SSL_CLIENT_AUTH_CERT
or gateway logs show client cert verification when OIDC or direct HTTPS is expected
Listener client-CA verification still enabled (
clientCaSecretName
unset or
client_ca_path
in ConfigMap)
Set
server.tls.clientCaSecretName=""
, upgrade chart, confirm ConfigMap omits
client_ca_path
症状可能原因检查项
openshell status
失败
Gateway endpoint 不可达或 auth 不匹配
openshell gateway info
、gateway 日志
BatchSpanProcessor.ExportError
反复报告无法连接
127.0.0.1:4317
本地 gateway 启动时配置了 OTLP,但后续 collector forwarding task 已停止,或者配置是手工创建的重启
gateway:docker
、
gateway:podman
或
gateway:vm
,使其重新检测 listener;检查生成的
gateway.toml
中的
[openshell.gateway.otlp]
Gateway 启动但 sandbox create 失败Compute driver 无法访问 runtimeDocker/Podman/Kubernetes/VM driver 日志
Gateway 在解析 compute-driver listener requirement 时退出Callback alias topology 不受支持、无法检查 Podman network,或选定地址不是 private/authorizedGateway startup error、
podman info --debug
、Podman network inspection、host IPv4 default route
在额外的 Docker/Podman callback-only listener 上,admin、health、reflection 或 HTTP request 被拒绝额外 callback listener 有意仅公开 sandbox 可调用的 gRPC method通过 gateway 的 primary endpoint 重试;如果地址不符合预期,检查 listener-purpose startup log
Docker 或 Podman sandbox 始终未注册Callback endpoint 错误或 supervisor startup failureGateway 日志和 sandbox container 日志
Docker GPU sandbox 在启动前失败NVIDIA CDI spec 缺失,或 Docker 尚未发现它们
docker info --format '{{json .DiscoveredDevices}}'
、
/etc/cdi
、
/var/run/cdi
、
nvidia-cdi-refresh.service
Kubernetes gateway pod 处于 pending 状态PVC 未绑定、taint、selector 或资源不足
kubectl -n openshell describe pod <pod>
Kubernetes sandbox pod 一直处于 pending 状态,workspace PVC 未绑定Cluster 没有默认
StorageClass
,且 OpenShell 未在 workspace PVC 上设置
storageClassName
(具有默认
StorageClass
的 cluster 无需设置也可正常绑定)
kubectl -n openshell describe pvc
;将
server.workspaceStorageClass
(gateway config
workspace_storage_class
)设置为有效的
StorageClass
Kubernetes gateway pod 陷入 crash loop缺少 secret、DB URL 错误或 TLS 配置错误
kubectl -n openshell logs deployment/openshell -c openshell-gateway
或
kubectl -n openshell logs statefulset/openshell -c openshell-gateway
OpenShift gateway pod 因 SCC/
runAsUser
错误而无法启动,例如
unable to validate against any security context constraint
Chart 默认的
podSecurityContext
/
securityContext
硬编码了
runAsUser
/
fsGroup
,restricted-v2 SCC 会拒绝这些值;它应当改为注入 namespace 分配的 UID/GID range
oc -n openshell describe pod <pod>
;使用
podSecurityContext: null
部署,并清除
securityContext.runAsUser
(参见
deploy/helm/openshell/ci/values-openshift-scc.yaml
)
OpenShift sandbox pod 无法启动,显示
unable to validate against any security context constraint
openshell-sandbox
service account 缺少所需的 privileged SCC
oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell
;完成后使用
remove-scc-from-user
移除
CLI TLS error本地 mTLS bundle 与 server cert/CA 不匹配检查
~/.config/openshell/gateways/<name>/mtls/
Edge 或 OIDC gateway 返回
Unauthenticated
已存储的 login 已过期、audience/scopes 不匹配,或 gateway auth 配置发生变化
openshell gateway info
、
openshell gateway login <name>
、gateway auth 日志
启用 interceptor 后 gateway 在提供 health service 前失败Interceptor endpoint 不可用,或 manifest/binding validation 失败Gateway 和 interceptor 日志;interceptor socket;
binding_policy
、phase 和 failure policy
Authenticated interceptor 或 middleware 拒绝 gateway callPrivate CA 或 hostname 不匹配、预期 audience 或 issuer 不匹配、
kid
过期或未知,或 extension token 格式错误
tls_ca_cert_path
、注册
audience
、service verifier 配置和日志;仅通过已经信任的 gateway TLS endpoint 获取 well-known metadata
启用 interceptor catalog 后 provider profile 消失
provider_profile_sources
仅选择了 authoritative interceptor,或返回了无效/重复 ID
检查 source list 和 interceptor
Describe
/catalog 日志;按需包含
builtin
和
user
注册 supervisor middleware 后 gateway 失败Service 不可用、manifest 无效、binding 重复、保留名称或 payload/timeout limit 无效Middleware service 和 gateway 日志;
[[openshell.supervisor.middleware]]
;
Describe
response
Policy update 拒绝
network_middlewares
Unknown middleware name、implementation-owned config 无效、order 重复、host selector 过宽或无效,或 fail-closed coverage 包含
tls: skip
Policy error、gateway 日志、middleware
ValidateConfig
、selector 和 order 字段
Endpoint ambiguity 导致 policy mutation 返回
FAILED_PRECONDITION
匹配同等具体程度的 effective endpoint selector 在 connection 或 request-processing metadata 上存在分歧CLI error、base 和 provider-composed policy、受影响的 profile attachment;确认没有存储新 revision
Supervisor 进入 policy quarantineRuntime candidate validation 失败,同时
policy_validation_failure_mode = "fail_closed"
Sandbox OCSF config/finding event、validation rationale、active generation、
previous_policy_active
HTTP request 返回
middleware_failed
或
middleware_denied
,或 WebSocket 以
1008
关闭
被选中的 stage 失败或显式拒绝已准入流量Sandbox OCSF 日志;policy-local middleware 配置;service 可用性;binding operation;
on_error
WebSocket upgrade 成功,但匹配 host 的 middleware 未收到 preflight 或 message RPC实现未声明
WEBSOCKET_MESSAGE/PRE_CREDENTIALS
WEBSOCKET_MIDDLEWARE_COVERAGE state=binding_not_selected
;service
Describe
;upgrade GET 可能仍使用了 HTTP binding
Binary WebSocket message 未触发 middleware RPC 即通过在两种
on_error
模式下,V1 text-message binding 均不支持 binary
WEBSOCKET_MIDDLEWARE_COVERAGE state=unsupported_message_type
;下一条 text RPC 可能存在有效的 sequence gap
一次失败后 WebSocket message 不再到达 middlewareFail-open stage stream 已在该 connection 的剩余生命周期中被禁用
openshell.middleware.websocket_stage_disabled
;middleware timeout/stream/protocol 日志。仅单条消息 capacity bypass 不会禁用 stage;发生真正的 stream failure 后应重连以创建新 stream
启用 gateway JWT signing 后,supervisor 反复无法安装 middlewareExtension credential minting、distribution 或 authenticated service connection 失败;最后一个已知正常的 registry 仍处于活动状态Gateway
RefreshSandboxToken
日志、sandbox configuration event、service token-verification 日志、registration TLS/audience 设置
Custom compute driver 不可用Driver process/socket 缺失或不可访问,或者选定名称与 endpoint/config key 不匹配Socket ownership/mode、driver service 日志、gateway
GetCapabilities
日志
Sandbox 一直处于
Stopping
或
Starting
Driver stop/start 失败、保留的 resource 缺失,或新的 supervisor 尚未连接Gateway 和 driver 日志;
docker inspect
、
podman inspect
、Agent Sandbox status/PVC,或 VM state marker 和 launcher process
Image pull failureGateway 或 sandbox image 无法拉取Runtime event 和 image pull credential
Gateway API resource 失败并显示
the server could not find the requested resource
应用 optional Gateway API resource 时未安装 Envoy Gateway CRD安装 Envoy Gateway,并在应用 optional ingress resource 前启用
grpcRoute
HTTPS ingress(
grpcRoute.gateway.listener.protocol=HTTPS
)连接被重置或 TLS handshake 卡住
Envoy 终止 TLS,但 gateway pod 仍预期 TLS,导致明文 backend hop 失败设置
server.disableTls=true
,使 Envoy 将明文转发到 pod;验证 listener
certificateRefs
Secret 存在于 release namespace,并通过
https://<host>
执行
openshell status
HTTPS ingress 连接后返回
Unauthenticated
TLS 在 Envoy 处终止,因此 gateway 看不到 client certificate;也没有为 identity 配置 OIDC issuer配置
server.oidc.issuer
并使用
openshell gateway add https://<host> --oidc-issuer <url>
注册,或为 trusted-proxy/dev cluster 设置
server.auth.allowUnauthenticatedUsers=true
设置
certManager.serverIssuerRef
后,external server
Certificate
始终未进入 Ready 状态
ACME issuer 拒绝仅供内部使用的 SAN、loopback IP,或拒绝未同时出现在 SAN 中的
commonName
kubectl -n openshell describe certificate openshell-server-external
;确认
certManager.serverDnsNames
仅列出真实且可从外部解析的 hostname
配置
certManager.serverIssuerRef
后,sandbox supervisor 因
UnknownCA
导致 TLS handshake 失败
server.grpcEndpoint
设置为 external hostname,迫使 supervisor 获取 ACME certificate(通过 SNI),但无法针对 chart CA 验证
删除
server.grpcEndpoint
或将其设置为内部 service name;supervisor 应通过内部 service name 连接,以获取内部 certificate
浏览器显示
ERR_BAD_SSL_CLIENT_AUTH_CERT
,或 gateway 日志在预期使用 OIDC 或 direct HTTPS 时显示 client cert verification
Listener client-CA verification 仍处于启用状态(
clientCaSecretName
未设置,或 ConfigMap 中存在
client_ca_path
)
设置
server.tls.clientCaSecretName=""
,升级 chart,并确认 ConfigMap 中省略了
client_ca_path

Reporting

结果报告

When handing results back to the user, include:
  • Active gateway endpoint and auth mode.
  • Compute platform and driver.
  • Gateway process or workload status.
  • Recent gateway log summary.
  • Missing or malformed TLS, OIDC/mTLS, or sandbox JWT material.
  • Service exposure status.
  • Sandbox workload status.
  • The exact command that failed and the shortest fix.
向用户返回结果时,应包括:
  • 当前活动的 gateway endpoint 和 auth mode。
  • Compute platform 和 driver。
  • Gateway process 或 workload 状态。
  • 最近的 gateway 日志摘要。
  • 缺失或格式错误的 TLS、OIDC/mTLS 或 sandbox JWT material。
  • Service exposure 状态。
  • Sandbox workload 状态。
  • 发生失败的确切命令和最短修复方案。

Package Configuration Preflight

Package Configuration Preflight

For a Debian, Ubuntu, or Snap gateway that stops before certificate generation or daemon startup, validate the selected configuration without starting the service:
shell
openshell-gateway config preflight [--path PATH | -- GATEWAY_ARGS...]
Without a path, preflight validates a nonempty
OPENSHELL_GATEWAY_CONFIG
or an auto-discovered XDG config; no config succeeds. An explicit missing path, legacy schema-v1 file, malformed TOML, symlink, or nonregular file fails before gateway startup. It also applies read-only effective-config checks for driver selection and configuration, sockets, rate limits, TLS, interceptors, and supervisor middleware. If the file omits the selector, preflight validates configured tables for auto-detectable drivers without running socket or process-based detection probes. Arguments after
--
validate the effective daemon invocation, including its command-line overrides. Preflight preserves every failed file. Do not advise users to delete or rewrite it automatically; back it up and follow the manual schema-v2 migration in the Gateway Configuration reference.
对于在 certificate generation 或 daemon startup 之前停止的 Debian、Ubuntu 或 Snap gateway,应在不启动 service 的情况下验证所选配置:
shell
openshell-gateway config preflight [--path PATH | -- GATEWAY_ARGS...]
未指定 path 时,preflight 会验证非空的
OPENSHELL_GATEWAY_CONFIG
,或自动发现的 XDG config;不存在配置时验证成功。显式指定但不存在的 path、legacy schema-v1 file、格式错误的 TOML、symlink 或 nonregular file 都会在 gateway startup 前失败。它还会对 driver selection 和 configuration、socket、rate limit、TLS、interceptor 及 supervisor middleware 执行只读 effective-config check。如果文件省略 selector,preflight 会在不运行 socket 或基于 process 的 detection probe 的情况下,验证可自动检测 driver 的已配置 table。
--
之后的 argument 会针对 effective daemon invocation 进行验证,包括 command-line override。Preflight 会保留每个失败的文件。不要建议用户自动删除或重写这些文件;应先备份,然后按照 Gateway Configuration reference 中的说明手工完成 schema-v2 migration。