Vector Database Operations

向量数据库运维

Run production vector databases for AI-powered search, RAG, and recommendation systems.

为基于AI的搜索、RAG和推荐系统运行生产级向量数据库。

When to Use This Skill

何时使用此技能

Use this skill when:

Setting up a vector database for a RAG or semantic search application
Choosing between Qdrant, Weaviate, pgvector, or Pinecone
Managing collections, indexes, and data migrations
Optimizing query performance and indexing for production loads
Implementing multi-tenant vector search with namespace isolation

在以下场景使用此技能：

为RAG或语义搜索应用搭建向量数据库
在Qdrant、Weaviate、pgvector或Pinecone之间做选型
管理集合、索引和数据迁移
针对生产负载优化查询性能与索引
实现带命名空间隔离的多租户向量搜索

Vector Database Comparison

向量数据库对比

Database	Best For	Hosting	Filtering	Scale
Qdrant	High-performance, rich filtering, self-hosted	Self / Cloud	Excellent	Very High
Weaviate	Schema-first, hybrid search, multi-modal	Self / Cloud	Good	High
pgvector	Already on Postgres, simple use cases	Self	Good	Medium
Pinecone	Zero-ops managed, serverless	Managed only	Good	Very High
Chroma	Local dev, prototyping	Self only	Basic	Low-Medium

数据库	适用场景	部署方式	过滤能力	扩展性
Qdrant	高性能、丰富过滤能力、自托管	自托管/云托管	优秀	极高
Weaviate	Schema优先、混合搜索、多模态	自托管/云托管	良好	高
pgvector	已使用PostgreSQL、简单场景	自托管	良好	中等
Pinecone	零运维托管、无服务器	仅托管	良好	极高
Chroma	本地开发、原型验证	仅自托管	基础	低-中等

Qdrant — Production Deployment

Qdrant — 生产环境部署

bash

undefined

bash

undefined

Docker (single node)

Docker (单节点)

docker run -d
--name qdrant
-p 6333:6333
-p 6334:6334
-v $(pwd)/qdrant-data:/qdrant/storage
qdrant/qdrant:latest

With custom config

自定义配置启动

docker run -d
--name qdrant
-p 6333:6333
-v $(pwd)/qdrant-data:/qdrant/storage
-v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml
qdrant/qdrant:latest


```yaml

docker run -d
--name qdrant
-p 6333:6333
-v $(pwd)/qdrant-data:/qdrant/storage
-v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml
qdrant/qdrant:latest


```yaml

qdrant-config.yaml

storage: storage_path: /qdrant/storage on_disk_payload: true # store payload on disk (saves RAM)

service: max_request_size_mb: 32

hnsw_index: m: 16 # graph connections per node ef_construct: 100 # accuracy vs build time trade-off full_scan_threshold: 10000 # switch to brute force below this

quantization: scalar: type: int8 quantile: 0.99 always_ram: true # keep quantized index in RAM

telemetry_disabled: true

undefined

storage: storage_path: /qdrant/storage on_disk_payload: true # 将负载存储在磁盘（节省内存）

service: max_request_size_mb: 32

hnsw_index: m: 16 # 每个节点的图连接数 ef_construct: 100 # 精度与构建时间的权衡 full_scan_threshold: 10000 # 低于此数量时切换为暴力搜索

quantization: scalar: type: int8 quantile: 0.99 always_ram: true # 将量化索引保留在内存中

telemetry_disabled: true

undefined

Qdrant Collection Management

Qdrant 集合管理

python

from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
)

client = QdrantClient("http://localhost:6333")

python

from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
)

client = QdrantClient("http://localhost:6333")

Create optimized collection

创建优化后的集合

client.create_collection( collection_name="documents", vectors_config=VectorParams( size=1536, # OpenAI ada-002 / text-embedding-3-small distance=Distance.COSINE, on_disk=True, # save RAM — vectors stored on disk ), hnsw_config=HnswConfigDiff( m=32, # higher = better recall, more RAM ef_construct=200, on_disk=False, # keep HNSW graph in RAM for speed ), quantization_config=QuantizationConfig( scalar=ScalarQuantizationConfig( type=ScalarType.INT8, quantile=0.99, always_ram=True, ) ), )

client.create_collection( collection_name="documents", vectors_config=VectorParams( size=1536, # OpenAI ada-002 / text-embedding-3-small distance=Distance.COSINE, on_disk=True, # 节省内存 — 向量存储在磁盘 ), hnsw_config=HnswConfigDiff( m=32, # 值越高召回率越好，占用内存越多 ef_construct=200, on_disk=False, # 将HNSW图保留在内存以提升速度 ), quantization_config=QuantizationConfig( scalar=ScalarQuantizationConfig( type=ScalarType.INT8, quantile=0.99, always_ram=True, ) ), )

Create payload index for fast filtering

创建负载索引以实现快速过滤

client.create_payload_index( collection_name="documents", field_name="tenant_id", field_schema="keyword", ) client.create_payload_index( collection_name="documents", field_name="created_at", field_schema="datetime", )

Collection info

查看集合信息

info = client.get_collection("documents") print(f"Vectors: {info.vectors_count}, Status: {info.status}")

undefined

info = client.get_collection("documents") print(f"向量数量: {info.vectors_count}, 状态: {info.status}")

undefined

Qdrant Filtered Search

Qdrant 过滤搜索

python

from qdrant_client.models import Filter, FieldCondition, MatchValue, Range

python

from qdrant_client.models import Filter, FieldCondition, MatchValue, Range

Tenant-isolated search (multi-tenant RAG)

租户隔离搜索（多租户RAG）

results = client.query_points( collection_name="documents", query=query_embedding, query_filter=Filter( must=[ FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")), FieldCondition(key="doc_type", match=MatchValue(value="contract")), ], should=[ FieldCondition(key="created_at", range=Range(gte="2024-01-01")), ], ), limit=10, with_payload=True, )

undefined

results = client.query_points( collection_name="documents", query=query_embedding, query_filter=Filter( must=[ FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")), FieldCondition(key="doc_type", match=MatchValue(value="contract")), ], should=[ FieldCondition(key="created_at", range=Range(gte="2024-01-01")), ], ), limit=10, with_payload=True, )

undefined

pgvector — PostgreSQL Extension

pgvector — PostgreSQL 扩展

sql

-- Enable extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- Create HNSW index (faster queries, more memory)
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- Create IVFFlat index (less memory, slower build)
-- CREATE INDEX ON documents
-- USING ivfflat (embedding vector_cosine_ops)
-- WITH (lists = 100);

-- Semantic search with metadata filtering
SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
ORDER BY embedding <=> $1::vector
LIMIT 10;

bash

undefined

sql

-- 启用扩展
CREATE EXTENSION IF NOT EXISTS vector;

-- 创建带向量列的表
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- 创建HNSW索引（查询更快，占用内存更多）
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- 创建IVFFlat索引（占用内存更少，构建更慢）
-- CREATE INDEX ON documents
-- USING ivfflat (embedding vector_cosine_ops)
-- WITH (lists = 100);

-- 带元数据过滤的语义搜索
SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
ORDER BY embedding <=> $1::vector
LIMIT 10;

bash

undefined

Deploy pgvector via Docker

通过Docker部署pgvector

docker run -d
--name pgvector
-e POSTGRES_PASSWORD=secret
-e POSTGRES_DB=vectordb
-p 5432:5432
-v pgvector-data:/var/lib/postgresql/data
pgvector/pgvector:pg16

undefined

docker run -d
--name pgvector
-e POSTGRES_PASSWORD=secret
-e POSTGRES_DB=vectordb
-p 5432:5432
-v pgvector-data:/var/lib/postgresql/data
pgvector/pgvector:pg16

undefined

Weaviate Deployment

Weaviate 部署

yaml

undefined

yaml

undefined

docker-compose for Weaviate

Weaviate的docker-compose配置

services: weaviate: image: semitechnologies/weaviate:latest ports: - "8080:8080" - "50051:50051" environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false" AUTHENTICATION_APIKEY_ENABLED: "true" AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}" AUTHENTICATION_APIKEY_USERS: "admin" PERSISTENCE_DATA_PATH: /var/lib/weaviate ENABLE_MODULES: text2vec-openai,generative-openai OPENAI_APIKEY: "${OPENAI_API_KEY}" CLUSTER_HOSTNAME: node1 volumes: - weaviate-data:/var/lib/weaviate restart: unless-stopped

volumes: weaviate-data:

undefined

services: weaviate: image: semitechnologies/weaviate:latest ports: - "8080:8080" - "50051:50051" environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false" AUTHENTICATION_APIKEY_ENABLED: "true" AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}" AUTHENTICATION_APIKEY_USERS: "admin" PERSISTENCE_DATA_PATH: /var/lib/weaviate ENABLE_MODULES: text2vec-openai,generative-openai OPENAI_APIKEY: "${OPENAI_API_KEY}" CLUSTER_HOSTNAME: node1 volumes: - weaviate-data:/var/lib/weaviate restart: unless-stopped

volumes: weaviate-data:

undefined

Backup and Restore

备份与恢复

bash

undefined

bash

undefined

Qdrant — snapshot backup

Qdrant — 快照备份

curl -X POST "http://localhost:6333/collections/documents/snapshots"

Download snapshot

下载快照

curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"

Restore

恢复快照

curl -X POST "http://localhost:6333/collections/documents/snapshots/recover"
-H "Content-Type: application/json"
-d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'

pgvector — standard pg_dump

pgvector — 使用标准pg_dump备份

pg_dump -h localhost -U postgres -d vectordb
--table=documents --format=custom > documents-backup.dump

Restore

恢复备份

pg_restore -h localhost -U postgres -d vectordb documents-backup.dump

undefined

pg_restore -h localhost -U postgres -d vectordb documents-backup.dump

undefined

Performance Tuning

性能调优

python

undefined

python

undefined

Qdrant — optimize collection after bulk load

Qdrant — 批量加载后优化集合

client.update_collection( collection_name="documents", optimizer_config={"indexing_threshold": 0}, # force indexing now )

client.update_collection( collection_name="documents", optimizer_config={"indexing_threshold": 0}, # 立即强制索引 )

Wait for optimization to complete

等待优化完成

import time while True: info = client.get_collection("documents") if info.status.value == "green": break time.sleep(5) print(f"Optimizing... segments: {info.segments_count}")

undefined

import time while True: info = client.get_collection("documents") if info.status.value == "green": break time.sleep(5) print(f"优化中... 段数: {info.segments_count}")

undefined

Common Issues

常见问题

Issue	Cause	Fix
Slow queries	No HNSW index built yet	Wait for indexing; check `status == green`
High RAM usage	Vectors in memory	Enable `on_disk=True` for vectors
Poor recall	Low `ef` search param	Increase `ef` in search request (at query time)
pgvector slow	Using IVFFlat without vacuum	Run `VACUUM ANALYZE documents`
Weaviate OOM	Too many objects	Enable async indexing; increase heap

问题	原因	解决方法
查询缓慢	尚未构建HNSW索引	等待索引完成；检查 `status == green`
内存占用过高	向量存储在内存中	启用 `on_disk=True` 将向量存储到磁盘
召回率低	搜索参数 `ef` 设置过低	在查询时提高 `ef` 参数
pgvector查询缓慢	使用IVFFlat但未执行vacuum	运行 `VACUUM ANALYZE documents`
Weaviate内存溢出	对象数量过多	启用异步索引；增加堆内存

Best Practices

最佳实践

Use cosine distance for normalized embeddings; dot product for unnormalized.
Always create payload indexes on filter fields (
```
tenant_id
```
,
```
doc_type
```
).
For datasets >10M vectors, use
```
on_disk
```
vectors +
```
always_ram
```
quantization.
Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
Snapshot before any bulk delete or migration operation.

归一化嵌入使用余弦距离；非归一化嵌入使用点积。
务必为过滤字段（
```
tenant_id
```
、
```
doc_type
```
）创建负载索引。
对于超过1000万条向量的数据集，使用
```
on_disk
```
向量 +
```
always_ram
```
量化。
在选择IVFFlat与HNSW之前，用实际查询模式做基准测试。
在执行批量删除或迁移操作前创建快照。

vector-database-ops

Original

Translation

Vector Database Operations

向量数据库运维

When to Use This Skill

何时使用此技能

Vector Database Comparison

向量数据库对比

Qdrant — Production Deployment

Qdrant — 生产环境部署

Docker (single node)

Docker (单节点)

With custom config

自定义配置启动

qdrant-config.yaml

qdrant-config.yaml

Qdrant Collection Management

Qdrant 集合管理

Create optimized collection

创建优化后的集合

Create payload index for fast filtering

创建负载索引以实现快速过滤

Collection info

查看集合信息

Qdrant Filtered Search

Qdrant 过滤搜索

Tenant-isolated search (multi-tenant RAG)

租户隔离搜索（多租户RAG）

pgvector — PostgreSQL Extension

pgvector — PostgreSQL 扩展

Deploy pgvector via Docker

通过Docker部署pgvector

Weaviate Deployment

Weaviate 部署

docker-compose for Weaviate

Weaviate的docker-compose配置

Backup and Restore

备份与恢复

Qdrant — snapshot backup

Qdrant — 快照备份

Download snapshot

下载快照

Restore

恢复快照

pgvector — standard pg_dump

pgvector — 使用标准pg_dump备份

Restore

恢复备份

Performance Tuning

性能调优

Qdrant — optimize collection after bulk load

Qdrant — 批量加载后优化集合

Wait for optimization to complete

等待优化完成

Common Issues

常见问题

Best Practices

最佳实践

Related Skills

相关技能