eve-deploy-debugging

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Eve Deploy and Debug

Eve 部署与调试

Use these steps to deploy and diagnose app issues quickly.

按照以下步骤快速部署应用并排查问题。

Environment Setup

环境设置

Get the staging API URL from your admin.
Create and use a profile:

bash

eve profile create staging --api-url https://api.eve.example.com
eve profile use staging

从管理员处获取staging API URL。
创建并使用配置文件：

bash

eve profile create staging --api-url https://api.eve.example.com
eve profile use staging

Infrastructure Change Policy

基础设施变更策略

Never run

kubectl apply

helm install

, or any direct Kubernetes resource creation against shared infrastructure. All infrastructure changes go through Terraform. Use the Eve CLI (

eve env

eve env deploy

) to manage application deployments — the platform handles the underlying k8s resources.

切勿针对共享基础设施直接运行

kubectl apply

、

helm install

或任何直接创建Kubernetes资源的命令。所有基础设施变更都需通过Terraform完成。使用Eve CLI（

eve env

、

eve env deploy

）管理应用部署——平台会处理底层的k8s资源。

Deploy Flow (Staging)

部署流程（Staging环境）

bash

undefined

bash

undefined

Create env if needed

如有需要，创建环境

eve env create staging --project proj_xxx --type persistent

Deploy (requires --ref with 40-char SHA or a ref resolved against --repo-dir)

部署（需要--ref参数，值为40字符的SHA，或基于--repo-dir解析的引用）

eve env deploy staging --ref main --repo-dir .

When environment has a pipeline configured, the above triggers the pipeline.

当环境配置了流水线时，上述命令会触发流水线。

Use --direct to bypass pipeline and deploy directly:

使用--direct参数绕过流水线直接部署：

eve env deploy staging --ref main --repo-dir . --direct

Pass inputs to pipeline:

向流水线传递输入参数：

eve env deploy staging --ref main --repo-dir . --inputs '{"key":"value"}'

undefined

eve env deploy staging --ref main --repo-dir . --inputs '{"key":"value"}'

undefined

Deploy Polling Flow

部署轮询流程

When

eve env deploy

is called:

Direct deploy (no pipeline): Returns
```
deployment_status
```
directly. Poll health endpoint until
```
ready === true
```
.
Pipeline deploy: Returns
```
pipeline_run_id
```
. Poll
```
GET /pipelines/{name}/runs/{id}
```
until all steps complete, then check health.

Deploy is complete when:

ready === true

AND

active_pipeline_run === null

调用

eve env deploy

时：

直接部署（无流水线）：直接返回
```
deployment_status
```
。轮询健康端点直到
```
ready === true
```
。
流水线部署：返回
```
pipeline_run_id
```
。轮询
```
GET /pipelines/{name}/runs/{id}
```
直到所有步骤完成，然后检查健康状态。

当

ready === true

且

active_pipeline_run === null

时，部署完成。

Observe the Deploy

监控部署状态

bash

eve job list --phase active
eve job follow <job-id>              # Real-time SSE streaming
eve job watch <job-id>               # Poll-based status updates
eve job diagnose <job-id>            # Full diagnostic
eve job result <job-id>              # Final result
eve job runner-logs <job-id>         # Raw worker logs

bash

eve job list --phase active
eve job follow <job-id>              # 实时SSE流
eve job watch <job-id>               # 基于轮询的状态更新
eve job diagnose <job-id>            # 完整诊断信息
eve job result <job-id>              # 最终结果
eve job runner-logs <job-id>         # 原始工作节点日志

Real-Time Debugging (3-Terminal Approach)

实时调试（三终端方法）

bash

undefined

bash

undefined

Terminal 1: Pipeline/job progress

终端1：流水线/任务进度

eve job follow <job-id>

Terminal 2: Environment health

终端2：环境健康状态

eve env diagnose <project> <env>

Terminal 3: System-level logs

终端3：系统级日志

eve system logs

undefined

eve system logs

undefined

Debugging Workflows

调试工作流

Job Won't Start

任务无法启动

Check dependencies:
```
eve job dep list <job-id>
```
Check if blocked:
```
eve job show <job-id>
```
→ look at
```
blocked_by
```
Verify environment readiness:
```
eve env show <project> <env>
```
Check orchestrator:
```
eve system orchestrator status
```

检查依赖：
```
eve job dep list <job-id>
```
检查是否被阻塞：
```
eve job show <job-id>
```
→ 查看
```
blocked_by
```
字段
验证环境就绪状态：
```
eve env show <project> <env>
```
检查编排器：
```
eve system orchestrator status
```

Job Failed

任务失败

Get the error:
```
eve job diagnose <job-id>
```

Check logs:

eve job follow <job-id>

eve job runner-logs <job-id>

If build failure:
```
eve build diagnose <build-id>
```
If secret failure:
```
eve secrets list --project <project_id>
```

获取错误信息：
```
eve job diagnose <job-id>
```

查看日志：

eve job follow <job-id>

或

eve job runner-logs <job-id>

如果是构建失败：
```
eve build diagnose <build-id>
```
如果是密钥失败：
```
eve secrets list --project <project_id>
```

Job Stuck Active

任务停滞在活跃状态

Check if waiting for input:
```
eve job show <job-id>
```
→
```
effective_phase
```
Check thread messages:
```
eve thread messages <thread-id>
```
Check runner pod:
```
eve system pods
```

检查是否在等待输入：
```
eve job show <job-id>
```
→ 查看
```
effective_phase
```
字段
查看线程消息：
```
eve thread messages <thread-id>
```
检查工作节点Pod：
```
eve system pods
```

System Issues

系统问题

API health:
```
eve system health
```
Orchestrator:
```
eve system orchestrator status
```
Recent events:
```
eve system events
```

API健康状态：
```
eve system health
```
编排器状态：
```
eve system orchestrator status
```
近期事件：
```
eve system events
```

Deploy Failed (Structured Diagnosis)

部署失败（结构化诊断）

When a deploy fails, prefer structured output over raw HTTP errors:

bash

eve env diagnose <project> <env>

eve env diagnose

surfaces a typed

last_deploy_failure

(kind, service, pod, namespace, message), the live K8s state, and

manifest_hash

of the deployed release versus the latest sync — use this to spot applied-release drift before re-running. The CLI no longer hides failures behind a bare

HTTP request failed

当部署失败时，优先使用结构化输出而非原始HTTP错误：

bash

eve env diagnose <project> <env>

eve env diagnose

会展示类型化的

last_deploy_failure

（类型、服务、Pod、命名空间、消息）、实时K8s状态，以及已部署版本与最新同步版本的

manifest_hash

——可用于在重新运行部署前发现已应用版本的偏移。CLI不再将错误隐藏在单纯的

HTTP request failed

之后。

Custom Domains

自定义域名

Custom hostnames are declared in the manifest under

x-eve.ingress.domains

and bound on first deploy. To operate them:

bash

eve domain list --env <env>          # what is bound where
eve domain verify <hostname>         # DNS check + cert state + next steps
eve domain status <hostname>         # which env owns it today
eve domain transfer <hostname> --to <env>   # move ownership across envs in same project
eve domain unbind <hostname>         # release so the next deploy claims it

Ownership is env-scoped with first-bind-wins: the first env to deploy with a hostname owns it, and other envs referencing the same hostname log

owned by environment "<A>"

and skip rendering. Use

transfer

+ redeploy (or scope per-env via

environments.<env>.overrides

) to move it.

自定义主机名在清单的

x-eve.ingress.domains

下声明，并在首次部署时绑定。操作方式如下：

bash

eve domain list --env <env>          # 查看域名绑定情况
eve domain verify <hostname>         # DNS检查 + 证书状态 + 后续步骤
eve domain status <hostname>         # 查看当前该域名所属环境
eve domain transfer <hostname> --to <env>   # 在同一项目的不同环境间转移域名所有权
eve domain unbind <hostname>         # 解绑域名，以便下次部署时重新绑定

所有权遵循环境范围的先绑定先得原则：首个部署时使用该主机名的环境拥有其所有权，其他引用同一主机名的环境会记录

owned by environment "<A>"

并跳过渲染。使用

transfer

+ 重新部署（或通过

environments.<env>.overrides

按环境划分范围）来转移所有权。

Platform Sentinel Alerts

平台哨兵警报

The platform runs a continuous environment sentinel that posts Slack alerts when envs degrade. Treat a sentinel ping as a starting point: pull the project/env from the alert, then run

eve env diagnose <project> <env>

to confirm the current state before reacting — alerts can lag a self-heal.

The daily Sentinel summary may include a monthly cost section. Use

eve system env-cost --all

for the full breakdown; it reads stored environment cost snapshots and labels stale or unavailable estimates. Do not query OpenCost directly from the Slack-summary path.

平台运行一个持续的环境哨兵，当环境降级时会发送Slack警报。将哨兵提醒作为排查起点：从警报中获取项目/环境信息，然后运行

eve env diagnose <project> <env>

确认当前状态后再采取行动——警报可能滞后于自我修复。

每日哨兵摘要可能包含月度费用部分。使用

eve system env-cost --all

查看完整明细；该命令会读取存储的环境费用快照，并标记过期或不可用的估算值。请勿通过Slack摘要中的路径直接查询OpenCost。

Common Error Messages

常见错误信息

Error	Cause	Fix
`401 Unauthorized`	Token expired	`eve auth login`
`git clone failed`	Missing credentials	Set `github_token` or `ssh_key` secret
`service not provisioned`	Environment not created	`eve env create <env>`
`image pull backoff`	Registry auth failed	If using BYO/custom registry, verify `REGISTRY_USERNAME` + `REGISTRY_PASSWORD` ; for managed apps use `registry: "eve"`
`healthcheck timeout`	App not starting	Check app logs, verify ports in manifest

错误	原因	修复方法
`401 Unauthorized`	令牌过期	`eve auth login`
`git clone failed`	缺少凭据	设置 `github_token` 或 `ssh_key` 密钥
`service not provisioned`	环境未创建	`eve env create <env>`
`image pull backoff`	镜像仓库认证失败	如果使用自定义仓库，验证 `REGISTRY_USERNAME` + `REGISTRY_PASSWORD` ；对于托管应用，使用 `registry: "eve"`
`healthcheck timeout`	应用未启动	查看应用日志，验证清单中的端口配置

Build Failures

构建失败

If a deploy pipeline fails at the build step:

bash

eve build list --project <project_id>
eve build diagnose <build_id>
eve build logs <build_id>
eve secrets list --project <project_id>     # Required for BYO/custom registry: REGISTRY_USERNAME, REGISTRY_PASSWORD

Common build failures:

Registry auth: For BYO/custom registry, verify
```
REGISTRY_USERNAME
```
and
```
REGISTRY_PASSWORD
```
secrets
Dockerfile not found: Check
```
build.context
```
path in manifest
Multi-stage build failure: BuildKit handles these correctly; Kaniko may have issues
Workspace errors: Build context not available — check
```
eve build diagnose
```

如果部署流水线在构建步骤失败：

bash

eve build list --project <project_id>
eve build diagnose <build_id>
eve build logs <build_id>
eve secrets list --project <project_id>     # 使用自定义仓库时需要：REGISTRY_USERNAME, REGISTRY_PASSWORD

常见构建失败原因：

仓库认证问题：使用自定义仓库时，验证
```
REGISTRY_USERNAME
```
和
```
REGISTRY_PASSWORD
```
密钥
未找到Dockerfile：检查清单中的
```
build.context
```
路径
多阶段构建失败：BuildKit可正确处理此类构建；Kaniko可能存在问题
工作区错误：构建上下文不可用——查看
```
eve build diagnose
```
结果

Public Runner Image

公共工作节点镜像

The public runner is

public.ecr.aws/w7c4v0w3/eve-horizon/worker:<platform-version>

. It is one of the seven images published by

release-v*

alongside

api

sso

gateway

agent-runtime

orchestrator

, and

dashboard

. Pin the same platform version used by the rest of the deployment through

EVE_RUNNER_IMAGE

There are no supported public

worker-full

worker-python

, or other worker variant release tags. Toolchains are separate images and are provisioned only for jobs that declare them.

公共工作节点镜像为

public.ecr.aws/w7c4v0w3/eve-horizon/worker:<platform-version>

。它是

release-v*

发布的七个镜像之一，另外六个为

api

、

sso

、

gateway

、

agent-runtime

、

orchestrator

和

dashboard

。通过

EVE_RUNNER_IMAGE

固定与部署其他部分相同的平台版本。

目前不支持公共的

worker-full

、

worker-python

或其他工作节点变体的发布标签。工具链是独立的镜像，仅为声明了它们的任务提供。

Platform Environment Variables

平台环境变量

Eve automatically injects these into every deployed service container:

Variable	Purpose
`EVE_API_URL`	Internal cluster URL for server-to-server calls
`EVE_PUBLIC_API_URL`	Public ingress URL for browser-facing apps (when configured)
`EVE_SSO_URL`	SSO broker URL for user authentication (when configured)
`EVE_PROJECT_ID`	Current project ID
`EVE_ORG_ID`	Current organization ID
`EVE_ENV_NAME`	Current environment name

Use

EVE_API_URL

for backend calls. Use

EVE_PUBLIC_API_URL

for browser/client-side code. Services can override any of these by defining them explicitly in their manifest

environment

section.

Eve会自动将以下变量注入每个已部署的服务容器：

变量	用途
`EVE_API_URL`	用于服务器间调用的集群内部URL
`EVE_PUBLIC_API_URL`	面向浏览器的应用使用的公共入口URL（已配置时）
`EVE_SSO_URL`	用户认证用的SSO代理URL（已配置时）
`EVE_PROJECT_ID`	当前项目ID
`EVE_ORG_ID`	当前组织ID
`EVE_ENV_NAME`	当前环境名称

后端调用使用

EVE_API_URL

。浏览器/客户端代码使用

EVE_PUBLIC_API_URL

。服务可通过在清单的

environment

部分显式定义这些变量来覆盖默认值。

Access URLs

访问URL

URL pattern:

{service}.{orgSlug}-{projectSlug}-{env}.{domain}

Local dev default domain:
```
lvh.me
```
Ask the admin for the correct domain (staging vs production).

URL格式：

{service}.{orgSlug}-{projectSlug}-{env}.{domain}

本地开发默认域名：
```
lvh.me
```
向管理员询问正确的域名（staging环境 vs 生产环境）。

Environment-Specific Debugging

特定环境的调试方法

Environment	How to Debug
Local (k3d)	Direct service access via ingress, `eve system logs`
Docker Compose	`docker compose logs <service>` , dev-only (no production use)
Kubernetes	Ingress-based access, `kubectl -n eve logs` as last resort

环境	调试方式
Local (k3d)	通过入口直接访问服务，使用 `eve system logs`
Docker Compose	`docker compose logs <service>` ，仅用于开发（不用于生产）
Kubernetes	基于入口的访问，最后可使用 `kubectl -n eve logs`

Private Endpoints (Tailscale)

私有端点（Tailscale）

Connect services on private networks (home lab GPUs, internal APIs, dev machines) to the Eve cluster. The platform creates K8s ExternalName services backed by Tailscale egress proxies.

bash

undefined

将私有网络中的服务（家庭实验室GPU、内部API、开发机器）连接到Eve集群。平台会创建由Tailscale出口代理支持的K8s ExternalName服务。

bash

undefined

Register a private endpoint

注册私有端点

eve endpoint add
--name lmstudio
--provider tailscale
--tailscale-hostname mac-mini.tail12345.ts.net
--port 1234
--org org_xxx

List and inspect

列出并查看端点信息

eve endpoint list --org org_xxx eve endpoint show lmstudio --org org_xxx

Diagnose connectivity

诊断连通性

eve endpoint diagnose lmstudio

Remove

删除端点

eve endpoint remove lmstudio --org org_xxx


Each endpoint gets a stable in-cluster DNS name: `http://{orgSlug}-{name}.eve-tunnels.svc.cluster.local:{port}`. Wire it into apps/agents via secrets:

```bash
eve secrets set LLM_BASE_URL \
  "http://myorg-lmstudio.eve-tunnels.svc.cluster.local:1234/v1" \
  --scope project

Diagnostics check: operator status, K8s service existence, DNS resolution, TCP connectivity, and HTTP health.

eve endpoint remove lmstudio --org org_xxx


每个端点都会获得一个稳定的集群内DNS名称：`http://{orgSlug}-{name}.eve-tunnels.svc.cluster.local:{port}`。通过密钥将其连接到应用/Agent：

```bash
eve secrets set LLM_BASE_URL \
  "http://myorg-lmstudio.eve-tunnels.svc.cluster.local:1234/v1" \
  --scope project

诊断内容包括：操作员状态、K8s服务存在性、DNS解析、TCP连通性和HTTP健康状态。

Worker Toolchain-on-Demand

按需工作节点工具链

The canonical worker contains the runner and harnesses. Toolchains (Python, Rust, Java, Kotlin, media) are separate images injected on demand via init containers rather than bundled into alternate worker variants.

Deployment impact: If an agent job needs toolchains, the runner pod starts init containers that copy toolchain binaries from small pre-built images. First pull adds ~5-10s; subsequent jobs on the same node use cached images.

Debugging toolchain issues:

bash

undefined

标准工作节点包含运行器和工具包。工具链（Python、Rust、Java、Kotlin、媒体处理等）是独立的镜像，通过初始化容器按需注入，而非捆绑到不同的工作节点变体中。

部署影响：如果Agent任务需要工具链，工作节点Pod会启动初始化容器，从预构建的小型镜像中复制工具链二进制文件。首次拉取会增加约5-10秒；同一节点上的后续任务会使用缓存的镜像。

工具链问题调试：

bash

undefined

Check if toolchains are declared in agent config

检查Agent配置中是否声明了工具链

agents.yaml: toolchains: [python]

If a toolchain binary is missing at runtime:

如果运行时缺少工具链二进制文件：

1. Verify agent config has the toolchain declared

1. 验证Agent配置中已声明该工具链

2. Check init container logs on the runner pod

2. 查看工作节点Pod上的初始化容器日志

3. Verify the configured toolchain image prefix/tag resolves in the registry

3. 验证配置的工具链镜像前缀/标签在仓库中可解析


Do not work around a missing toolchain by selecting a worker variant. Verify
the agent/workflow declaration, inspect runner init-container logs and
`runtime_meta.toolchains`, then check `EVE_TOOLCHAIN_IMAGE_PREFIX` and
`EVE_TOOLCHAIN_IMAGE_TAG`.


请勿通过选择工作节点变体来解决工具链缺失问题。请验证Agent/工作流声明，检查工作节点初始化容器日志和`runtime_meta.toolchains`，然后检查`EVE_TOOLCHAIN_IMAGE_PREFIX`和`EVE_TOOLCHAIN_IMAGE_TAG`。

App Undeploy/Delete Lifecycle

应用卸载/删除生命周期

Remove environments and clean up resources:

bash

undefined

移除环境并清理资源：

bash

undefined

Undeploy services from an environment (stops pods, keeps env record)

从环境中卸载服务（停止Pod，保留环境记录）

eve env undeploy <project> <env>

Delete the environment entirely (removes env record, managed DB, secrets)

完全删除环境（移除环境记录、托管数据库、密钥）

eve env delete <project> <env>


When a managed DB is attached, `eve env delete` deprovisions it. Secrets scoped to the environment are cleaned up. The environment's pipeline history remains in the audit log.

For app-level cleanup, remove the project:

```bash
eve project delete <project-id>

This cascades: environments, secrets, pipeline history, and build artifacts are removed.

eve env delete <project> <env>


当环境附加了托管数据库时，`eve env delete`会销毁该数据库。环境范围的密钥会被清理。环境的流水线历史会保留在审计日志中。

如需应用级别的清理，删除项目：

```bash
eve project delete <project-id>

此操作会级联删除：环境、密钥、流水线历史和构建产物。

Workspace Janitor

工作区清理器

Production disk management for agent workspaces:

```
EVE_WORKSPACE_MAX_GB
```
— total workspace budget
```
EVE_WORKSPACE_MIN_FREE_GB
```
— trigger cleanup threshold
```
EVE_SESSION_TTL_HOURS
```
— auto-evict stale sessions
LRU eviction when approaching budget; TTL cleanup for idle sessions
K8s: per-attempt PVCs deleted on completion

用于Agent工作区的生产环境磁盘管理：

```
EVE_WORKSPACE_MAX_GB
```
— 工作区总容量上限
```
EVE_WORKSPACE_MIN_FREE_GB
```
— 触发清理的阈值
```
EVE_SESSION_TTL_HOURS
```
— 自动驱逐闲置会话的时长
接近容量上限时使用LRU驱逐策略；闲置会话按TTL清理
Kubernetes：每次尝试使用的PVC会在完成后删除

Managed DB TLS

托管数据库TLS

Managed Postgres now ships a trusted CA chain to apps. Do not set

rejectUnauthorized: false

ssl: { rejectUnauthorized: false }

in service code — verified TLS is the default. If a client errors on cert verification, check that the service is reading

${managed.<db>.url}

rather than a hand-crafted DSN.

托管Postgres现在会向应用提供可信CA链。请勿在服务代码中设置

rejectUnauthorized: false

或

ssl: { rejectUnauthorized: false }

——验证TLS是默认设置。如果客户端在证书验证时出错，请检查服务是否读取的是

${managed.<db>.url}

而非手动编写的DSN。

Stable Egress (Allowlisted Source IPs)

稳定出口（允许列表源IP）

When a vendor requires fixed source IPs, opt the service into platform-managed stable egress in the manifest:

yaml

services:
  poller:
    x-eve:
      networking:
        egress: stable      # default is 'nat'

The deployer schedules the pod on the stable-egress node group with

hostNetwork: true

so traffic exits via that node's IGW path. Only opt in when needed (it bypasses NAT and constrains scheduling). See

eve-manifest-authoring

for the full field shape.

当供应商要求固定源IP时，在清单中将服务配置为使用平台托管的稳定出口：

yaml

services:
  poller:
    x-eve:
      networking:
        egress: stable      # 默认值为'nat'

部署器会将Pod调度到启用

hostNetwork: true

的稳定出口节点组，这样流量会通过该节点的IGW路径流出。仅在必要时启用此配置（它会绕过NAT并限制调度）。完整字段格式请参考

eve-manifest-authoring

。