lancedb

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Building LanceDB Pipelines

构建LanceDB管道

Use this skill to produce LanceDB pipelines that are portable between local and remote tables (for LanceDB Enterprise/Cloud) and idiomatic for the selected SDK.

使用本技能可构建能在本地表与远程表（适用于LanceDB Enterprise/Cloud）之间移植、且符合所选SDK语言习惯的LanceDB管道。

LanceDB Table Modes

LanceDB表模式

LanceDB has two common execution modes:

Local table: embedded, open source, in-process LanceDB. The client opens data from a local path or object storage URI and executes queries in the application process.
Remote table: LanceDB Enterprise/Cloud table opened through a
```
db://...
```
URI. The data may be very large, commonly backed by object storage, and queried through a remote service.

Do NOT assume local-only table helpers exist on remote tables. If the user asks for LanceDB Enterprise, Cloud,

db://...

, production remote access, or a remote table, focus on the remote table path: use

search()

query()

, keep reads bounded with

select()

and

limit()

, and avoid table-level full materialization APIs.

LanceDB有两种常见执行模式：

本地表：嵌入式开源LanceDB，在进程内运行。客户端从本地路径或对象存储URI读取数据，并在应用进程内执行查询。
远程表：通过
```
db://...
```
URI访问的LanceDB Enterprise/Cloud表。数据量可能极大，通常由对象存储提供支持，并通过远程服务进行查询。

请勿假设远程表支持仅本地表可用的工具方法。如果用户提及LanceDB Enterprise、Cloud、

db://...

、生产环境远程访问或远程表，请专注于远程表的实现路径：使用

search()

query()

，通过

select()

和

limit()

限制读取范围，避免使用表级全表物化API。

Workflow

工作流程

Identify the SDK: Python, TypeScript, or both.
Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path. If the task involves jobs in any way (listing, inspecting, creating, or canceling jobs), it is always the remote path and requires a remote server connection — see "Connecting to the LanceDB remote server" below before doing anything else.
Read the matching language branch before writing or changing code:
- Python patterns:
```
references/python/patterns.md
```
- Python API quick reference:
```
references/python/api_reference.md
```
- Python performance guidance:
```
references/python/performance.md
```
- TypeScript patterns:
```
references/typescript/patterns.md
```
- TypeScript API quick reference:
```
references/typescript/api_reference.md
```
- TypeScript performance guidance:
```
references/typescript/performance.md
```
- Column metadata authoring (both SDKs):
```
references/column_metadata.md
```
- Branch operations (both SDKs):
```
references/branch_ops.md
```
- Remote server connection resolution (jobs, raw REST):
```
references/remote_connect.md
```
- Job operations REST API (list/describe/cancel/query_events):
```
references/remote_jobs.md
```
Start with
```
patterns.md
```
for the selected SDK. Read
```
api_reference.md
```
when choosing method names or return collectors. Read
```
performance.md
```
when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read
```
column_metadata.md
```
when the task is documenting, tagging, classifying, or grouping table columns (field descriptions,
```
lancedb:tag:*
```
tags, logical column families). Read
```
branch_ops.md
```
when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read
```
remote_connect.md
```
when the task involves jobs or direct REST access to an Enterprise deployment, and
```
remote_jobs.md
```
for the job REST methods themselves (list, describe, cancel, query_events).
For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
Prefer
```
search()
```
or
```
query()
```
builders with explicit
```
select()
```
and
```
limit()
```
for reads.
Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
After a successful embedded OSS ingestion, call
```
table.optimize()
```
. Do not call it for Enterprise/Cloud; remote maintenance is automatic.
For remote Enterprise/Cloud writes, never drop-then-reuse or
```
mode="overwrite"
```
the same table name — see "Enterprise: never drop-then-reuse the same table name" below. This is the main local-vs-remote write pitfall.
If reviewing an existing file or repo, run
```
scripts/check_materialization.py
```
on the relevant paths and inspect each finding before editing.
Cross-check unfamiliar or non-trivial API claims against the source tree instead of relying on memory.

确定SDK类型：Python、TypeScript或两者兼用。
确定表模式：本地/嵌入式OSS、远程Enterprise/Cloud，或需在两者间移植。如果用户指定"LanceDB Enterprise"，则选择远程表路径。如果任务涉及任何任务操作（列出、检查、创建或取消任务），则始终采用远程路径，且需要先建立远程服务器连接——请先阅读下文的「连接LanceDB远程服务器」部分，再执行其他操作。
在编写或修改代码前，阅读对应语言的参考文档：
- Python模式：
```
references/python/patterns.md
```
- Python API速查：
```
references/python/api_reference.md
```
- Python性能指南：
```
references/python/performance.md
```
- TypeScript模式：
```
references/typescript/patterns.md
```
- TypeScript API速查：
```
references/typescript/api_reference.md
```
- TypeScript性能指南：
```
references/typescript/performance.md
```
- 列元数据编写（适用于两种SDK）：
```
references/column_metadata.md
```
- 分支操作（适用于两种SDK）：
```
references/branch_ops.md
```
- 远程服务器连接配置（任务、原生REST）：
```
references/remote_connect.md
```
- 任务操作REST API（列出/描述/取消/查询事件）：
```
references/remote_jobs.md
```
从所选SDK的
```
patterns.md
```
开始阅读。选择方法名称或返回收集器时，阅读
```
api_reference.md
```
。任务涉及数据摄入、索引、过滤、查询调优、诊断或大型数据集时，阅读
```
performance.md
```
。任务涉及文档编写、标记、分类或分组表列（字段描述、
```
lancedb:tag:*
```
标签、逻辑列族）时，阅读
```
column_metadata.md
```
。任务涉及分支生命周期（列出/创建/删除）、向非主分支写入或验证变更未进入主分支时，阅读
```
branch_ops.md
```
。任务涉及任务操作或直接REST访问Enterprise部署时，阅读
```
remote_connect.md
```
；任务操作REST方法本身（列出、描述、取消、查询事件）则阅读
```
remote_jobs.md
```
。
对于Python模式，优先使用Pydantic模型，并在写入前验证记录。当数据为Arrow原生、流式或高度动态时，使用PyArrow模式更合适。
读取操作优先使用带显式
```
select()
```
和
```
limit()
```
的
```
search()
```
或
```
query()
```
构建器。
在远程或可移植代码中，避免表级全表物化操作。这是本地与远程读取操作的主要陷阱。
嵌入式OSS数据摄入成功后，调用
```
table.optimize()
```
。Enterprise/Cloud无需调用此方法，远程维护是自动进行的。
对于远程Enterprise/Cloud写入操作，切勿删除后复用同一表名或使用
```
mode="overwrite"
```
——请阅读下文的「Enterprise：切勿删除后复用同一表名」部分。这是本地与远程写入操作的主要陷阱。
审查现有文件或仓库时，在相关路径上运行
```
scripts/check_materialization.py
```
，并在编辑前检查每个检测结果。
对于不熟悉或非 trivial 的API用法，对照源码树进行交叉验证，而非依赖记忆。

Core Portability Rule

核心可移植性规则

Do not write code that assumes a local table API will exist on a remote table. Remote tables can be very large, so whole-table materialization helpers are intentionally unavailable or unsafe.

This does not mean result conversion is forbidden. Bounded query/search result collection is normal:

Python:

table.search(...).select([...]).limit(10).to_pandas()

TypeScript:

await table.search(...).select([...]).limit(10).toArray()

The unsafe pattern is table-level or unbounded collection, plus local-only dataset escape hatches in remote code:

Python:

table.to_pandas()

table.to_arrow()

table.to_polars()

;

table.to_lance()

is local/OSS-only dataset access, not materialization

TypeScript:

await table.toArrow()

await table.query().toArray()

without

limit()

不要编写假设本地表API在远程表上也存在的代码。远程表的数据量可能极大，因此全表物化工具方法被有意设置为不可用或不安全。

这并不意味着禁止结果转换。有限范围的查询/搜索结果收集是正常操作：

Python：

table.search(...).select([...]).limit(10).to_pandas()

TypeScript：

await table.search(...).select([...]).limit(10).toArray()

不安全的模式是表级或无限制的结果收集，以及在远程代码中使用仅本地可用的数据集逃逸方法：

Python：
```
table.to_pandas()
```
、
```
table.to_arrow()
```
、
```
table.to_polars()
```
；
```
table.to_lance()
```
是仅本地/OSS可用的数据集访问方法，不属于物化操作

TypeScript：

await table.toArrow()

、未加

limit()

的

await table.query().toArray()

Enterprise: never drop-then-reuse the same table name

Enterprise：切勿删除后复用同一表名

LanceDB Enterprise/Cloud splits a control plane (DDL: create/drop/rename) from a data plane (query nodes that serve reads). Query nodes cache the resolved dataset for a table name for up to

table_cache_ttl

— default 300 seconds (5 minutes). After you drop or overwrite a table, the control plane updates immediately but the data plane keeps serving the old dataset until that cache entry expires. During the window the two planes disagree.

The failure this causes: you

drop_table("t")

then immediately

create_table("t", ...)

(or

create_table("t", ..., mode="overwrite")

). The DDL returns success, but every query against

returns 500 Internal Server Error
(the query node resolves the stale/deleted dataset), and a fresh

describe

may still show the old schema/version. It looks like your write silently failed; it didn't — the name is cached.

mode="overwrite"
has the same problem — it is a drop+create of the same name under the hood.

Rules for portable Enterprise ingestion:

Never reuse a table name you just dropped/overwrote within the cache TTL. Do not use
```
mode="overwrite"
```
to replace an existing Enterprise table in place.
To (re)load data, write to a fresh table name (e.g.
```
<table>_v2
```
, or a run-stamped suffix). A brand-new name has no cached data-plane entry, so writes and reads work immediately.
Before creating,
```
list_tables()
```
and fail loudly if the name already exists rather than overwriting — prompt for a new name.
To land on a specific final name that is currently occupied by an old table: drop the old table, wait out the TTL (~5 min), then
rename_table(fresh_name, final_name)
. Renaming onto a name whose old dataset is still cached hits the same race, so the wait is mandatory.
```
rename_table
```
is a supported control-plane op.
When you hand a table name back to a human, tell them which step still needs the propagation wait (usually: "the old
```
t
```
was dropped; run the rename in ~5 minutes").

This is Enterprise/Cloud-specific. Local/OSS tables have no separate data plane, so

mode="overwrite"

and immediate same-name reuse are fine there.

LanceDB Enterprise/Cloud将控制平面（DDL：创建/删除/重命名）与数据平面（提供读取服务的查询节点）分离。查询节点会缓存表名对应的解析数据集，缓存时长最长为

table_cache_ttl

——默认300秒（5分钟）。删除或覆盖表后，控制平面会立即更新，但数据平面会继续提供旧数据集，直到缓存条目过期。在此期间，两个平面的状态不一致。

这种情况会导致的故障：你执行

drop_table("t")

后立即执行

create_table("t", ...)

（或

create_table("t", ..., mode="overwrite")

）。DDL操作返回成功，但对

的所有查询都会返回**

500 Internal Server Error

**（查询节点解析到的是已过期/已删除的数据集），而新执行的

describe

可能仍显示旧的模式/版本。看起来你的写入操作静默失败了，但实际上并没有——只是表名被缓存了。

mode="overwrite"
存在同样的问题——它本质上是先删除再创建同名表。

适用于Enterprise的可移植数据摄入规则：

切勿在缓存TTL内复用刚删除/覆盖的表名。不要使用
```
mode="overwrite"
```
原地替换现有的Enterprise表。
要（重新）加载数据，写入全新的表名（例如
```
<table>_v2
```
，或带运行时间戳的后缀）。全新的表名没有缓存的数据集条目，因此写入和读取操作可立即生效。
创建表前，执行
```
list_tables()
```
检查，如果表名已存在则直接报错，而非覆盖——提示用户输入新表名。
若要使用当前被旧表占用的特定最终表名：删除旧表，等待TTL过期（约5分钟），然后执行
rename_table(fresh_name, final_name)
。将新表重命名到仍缓存旧数据集的表名会遇到同样的竞争问题，因此等待是必须的。
```
rename_table
```
是受支持的控制平面操作。
当你将表名交付给用户时，告知他们仍需等待传播完成的步骤（通常是："旧表
```
t
```
已删除；约5分钟后执行重命名操作"）。

这是Enterprise/Cloud特有的规则。本地/OSS表没有独立的数据平面，因此

mode="overwrite"

和立即复用同名表是可行的。

Connecting to the LanceDB remote server

连接LanceDB远程服务器

LanceDB Enterprise/Cloud deployments are served by a server implementing the lance-namespace OpenAPI spec (https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml). Every remote (

db://...

) connection talks to such a server, and some operations exist only there. In particular, all operations around jobs (listing, inspecting, creating, or canceling jobs) run server-side — there is no local/OSS equivalent. Before any job work, or any direct REST call to an Enterprise deployment, read

references/remote_connect.md

to resolve the base URL, credentials, and database header and to validate the connection. Then use the four job REST methods documented in

references/remote_jobs.md

(list, describe, cancel, query_events).

LanceDB Enterprise/Cloud部署由实现lance-namespace OpenAPI规范的服务器提供服务（https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml）。所有远程（

db://...

）连接都会与这类服务器通信，且部分操作仅在此处支持。特别是，所有与任务相关的操作（列出、检查、创建或取消任务）都在服务器端运行——没有本地/OSS等效操作。在执行任何任务操作或直接REST访问Enterprise部署前，请阅读

references/remote_connect.md

以配置基础URL、凭据和数据库头，并验证连接。然后使用

references/remote_jobs.md

中记录的四个任务REST方法（列出、描述、取消、查询事件）。

Script

脚本

Run the scanner when reviewing or modifying an existing codebase:

bash

python skills/lancedb/scripts/check_materialization.py path/to/file_or_dir

The script reports likely unsafe full-table materialization in Python and TypeScript. Treat results as review prompts, not automatic proof of a bug.

审查或修改现有代码库时，运行扫描器：

bash

python skills/lancedb/scripts/check_materialization.py path/to/file_or_dir

该脚本会报告Python和TypeScript中可能存在的不安全全表物化操作。将检测结果视为审查提示，而非自动判定的bug。