kafka-connector-review

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Kafka Connect Configuration Review

Kafka Connect配置检查

Reviews Kafka Connect connector configurations for common misconfigurations. Connectors are defined as JSON/YAML and are entirely language-agnostic.

检查Kafka Connect连接器配置中的常见错误配置。连接器以JSON/YAML格式定义，完全与语言无关。

Workflow

工作流程

Copy this checklist and track your progress:

Connector Review Progress:
- [ ] Step 1: List all connectors
- [ ] Step 2: Inspect each connector's configuration
- [ ] Step 3: Validate configurations against plugin schemas
- [ ] Step 4: Audit for common misconfigurations
- [ ] Step 5: Generate report

List all connectors with status and task states
Inspect each connector's configuration in detail
Validate configurations against plugin schemas
Audit for common misconfigurations
Report findings with current and recommended configs

复制此检查清单并跟踪进度：

Connector Review Progress:
- [ ] Step 1: List all connectors
- [ ] Step 2: Inspect each connector's configuration
- [ ] Step 3: Validate configurations against plugin schemas
- [ ] Step 4: Audit for common misconfigurations
- [ ] Step 5: Generate report

列出所有连接器及其状态和任务状态
详细检查每个连接器的配置
对照插件架构验证配置
审核常见错误配置
生成报告，包含当前配置与推荐配置

Step 1: List All Connectors

步骤1：列出所有连接器

Use the Lenses MCP

list_kafka_connectors

tool to get all connectors with:

Connector name, class and type (source/sink)
Status (RUNNING, PAUSED, FAILED, UNASSIGNED)
Task states and count
Cluster info

Flag immediately:

Critical: Connectors in FAILED state
Critical: Tasks in FAILED state
Warning: Connectors in PAUSED state with no obvious reason

Expected output: List of all connectors with name, class, status and task states.

Validation: If no connectors are returned, report that no Connect cluster is configured and stop.

使用Lenses MCP的

list_kafka_connectors

工具获取所有连接器的以下信息：

连接器名称、类和类型（源/ sink）
状态（RUNNING、PAUSED、FAILED、UNASSIGNED）
任务状态和数量
集群信息

立即标记以下情况：

严重问题：处于FAILED状态的连接器
严重问题：处于FAILED状态的任务
警告：无明显原因处于PAUSED状态的连接器

预期输出：包含所有连接器名称、类、状态和任务状态的列表。

验证：如果未返回任何连接器，需报告未配置Connect集群并停止操作。

Step 2: Inspect Configurations

步骤2：检查配置

Use the Lenses MCP

get_kafka_connector_target_definition

tool to get the full configuration YAML for each connector.

使用Lenses MCP的

get_kafka_connector_target_definition

工具获取每个连接器的完整YAML配置。

Step 3: Validate Configurations

步骤3：验证配置

Use the Lenses MCP

validate_connector_configuration

tool to validate each connector's config against its plugin's schema. This catches:

Missing required configuration fields
Invalid configuration values
Type mismatches

使用Lenses MCP的

validate_connector_configuration

工具对照插件架构验证每个连接器的配置。这将捕获：

缺失的必填配置字段
无效的配置值
类型不匹配

Step 4: Audit Common Misconfigurations

步骤4：审核常见错误配置

Error Handling

错误处理

Critical:

errors.tolerance=all

without

errors.deadletterqueue.topic.name

(silently drops messages)

Warning: Missing
```
errors.tolerance
```
(defaults to
```
none
```
, connector stops on any error)
Warning: Missing
```
errors.log.enable=true
```
(errors not logged)

Suggestion: Add

errors.deadletterqueue.context.headers.enable=true

for richer DLQ metadata

严重问题：
```
errors.tolerance=all
```
但未设置
```
errors.deadletterqueue.topic.name
```
（会静默丢弃消息）
警告：缺失
```
errors.tolerance
```
配置（默认值为
```
none
```
，连接器遇到任何错误都会停止）
警告：未启用
```
errors.log.enable=true
```
（错误不会被记录）

建议：添加

errors.deadletterqueue.context.headers.enable=true

以获取更丰富的DLQ元数据

Converters

转换器

Warning:
```
key.converter
```
or
```
value.converter
```
mismatch with topic serialisation format
Warning: Using
```
org.apache.kafka.connect.storage.StringConverter
```
for structured data (should use Avro/JSON/Protobuf converter)
Warning: Missing
```
schemas.enable
```
setting where schemas are expected

警告：
```
key.converter
```
或
```
value.converter
```
与主题序列化格式不匹配
警告：对结构化数据使用
```
org.apache.kafka.connect.storage.StringConverter
```
（应使用Avro/JSON/Protobuf转换器）
警告：在需要架构的场景中缺失
```
schemas.enable
```
设置

Transforms (SMTs)

转换操作（SMTs）

Warning: Complex transform chains (> 3 SMTs) that may be hard to debug
Suggestion: Consider Kafka Streams or ksqlDB for complex transformations
Verify transform class names are valid and in the correct order

警告：复杂的转换链（超过3个SMT）可能难以调试
建议：对于复杂转换，考虑使用Kafka Streams或ksqlDB
验证转换类名称是否有效且顺序正确

Task Count

任务数量

Warning:
```
tasks.max=1
```
on high-throughput connectors (limits parallelism)
Suggestion: Source connectors -
```
tasks.max
```
should align with source partitioning
Suggestion: Sink connectors -
```
tasks.max
```
should align with topic partition count

警告：高吞吐量连接器设置
```
tasks.max=1
```
（限制并行处理能力）
建议：源连接器 -
```
tasks.max
```
应与源分区数量匹配
建议：Sink连接器 -
```
tasks.max
```
应与主题分区数量匹配

Connection and Retry

连接与重试

Warning: Missing retry configuration for connectors that interact with external systems
Warning: Missing connection timeout settings
Suggestion: Set explicit
```
consumer.override.*
```
or
```
producer.override.*
```
for performance tuning

警告：与外部系统交互的连接器缺失重试配置
警告：缺失连接超时设置
建议：设置明确的
```
consumer.override.*
```
或
```
producer.override.*
```
以优化性能

Naming Conventions

命名规范

Suggestion: Connector names should follow a consistent pattern (e.g.,
```
{source/sink}-{system}-{entity}
```
)

建议：连接器名称应遵循一致的模式（例如：
```
{source/sink}-{system}-{entity}
```
）

Success Criteria

成功标准

Quantitative

量化标准

Triggers on 90% of connector-related queries (test with 10-20 varied phrasings)
Completes review in under 10 MCP tool calls
Catches 100% of validation errors reported by the plugin schema

90%的连接器相关查询可触发此工具（使用10-20种不同表述测试）
完成检查所需的MCP工具调用次数少于10次
捕获插件架构报告的100%验证错误

Qualitative

定性标准

Failed connectors are flagged immediately in the status overview
Error handling gaps (missing DLQ, silent drops) are always identified
Report is useful for both connector operators and developers

在状态概览中立即标记失败的连接器
始终识别错误处理漏洞（缺失DLQ、静默丢弃消息）
生成的报告对连接器运维人员和开发人员均有帮助

Examples

示例

Example 1: Routine connector review

示例1：常规连接器检查

User says: "Review all connectors in staging"

Actions:

List all connectors with status
Inspect and validate each connector's configuration
Audit for common misconfigurations Result: Full report covering all connectors with prioritised findings

用户说：“检查 staging 环境中的所有连接器”

操作：

列出所有连接器及其状态
检查并验证每个连接器的配置
审核常见错误配置结果：包含所有连接器及优先级问题的完整报告

Example 2: Investigating a failed connector

示例2：排查失败的连接器

User says: "My sink connector is failing, can you check why?"

Actions:

List connectors and identify those in FAILED state
Get the full configuration for the failed connector
Validate config against plugin schema
Check for common issues (DLQ, converters, error handling) Result: Diagnosis of the specific failure with remediation steps

用户说：“我的sink连接器失败了，能帮忙检查原因吗？”

操作：

列出连接器并识别处于FAILED状态的连接器
获取失败连接器的完整配置
对照插件架构验证配置
检查常见问题（DLQ、转换器、错误处理）结果：针对具体故障的诊断及修复步骤

Example 3: Single connector deep dive

示例3：单个连接器深度检查

User says: "Review the config for the elasticsearch-sink connector"

Actions:

Fetch the target definition for the named connector
Validate against its plugin schema
Audit error handling, converters, task count Result: Focused report on a single connector

用户说：“检查elasticsearch-sink连接器的配置”

操作：

获取指定连接器的目标定义
对照其插件架构验证配置
审核错误处理、转换器、任务数量结果：针对单个连接器的聚焦报告

Troubleshooting

故障排除

No connectors returned

未返回任何连接器

Cause: No Kafka Connect cluster is configured in the environment or no connectors are deployed. Solution: Verify that a Connect cluster exists via Lenses UI. Check

get_deployment_targets

for available Connect clusters.

原因：环境中未配置Kafka Connect集群或未部署任何连接器。解决方案：通过Lenses UI验证Connect集群是否存在。使用

get_deployment_targets

工具查看可用的Connect集群。

Validation fails with unknown plugin

验证失败，提示未知插件

Cause: The connector plugin class is not installed on the Connect cluster. Solution: Report the missing plugin. This is a deployment issue, not a configuration issue.

原因：Connect集群上未安装连接器插件类。解决方案：报告缺失的插件。这属于部署问题，而非配置问题。

Connector shows RUNNING but tasks are FAILED

连接器显示RUNNING但任务处于FAILED状态

Cause: The connector framework is running but individual tasks have encountered errors. Solution: Check each task's status. Common causes include authentication failures, network issues or schema mismatches with the external system.

原因：连接器框架正在运行，但个别任务遇到错误。解决方案：检查每个任务的状态。常见原因包括认证失败、网络问题或与外部系统的架构不匹配。

Output Format

输出格式

undefined

undefined

Connector Review Report

Environment: {name}

Connector Status Overview

Connector	Class	Status	Tasks	Failed Tasks
name	class	RUNNING	3/3	0

Connector	Class	Status	Tasks	Failed Tasks
name	class	RUNNING	3/3	0

Critical (must fix)

[connector-name] Description of the issue Current: {current config} | Recommended: {recommended config}

[connector-name] Description of the issue Current: {current config} | Recommended: {recommended config}

Warning (should fix)

[connector-name] Description of the issue Current: {current config} | Recommended: {recommended config}

[connector-name] Description of the issue Current: {current config} | Recommended: {recommended config}

Validation Errors

[connector-name] {field}: {validation error message}

[connector-name] {field}: {validation error message}

Suggestion (consider improving)

[connector-name] Description of the suggestion Recommendation: {how to improve}

[connector-name] Description of the suggestion Recommendation: {how to improve}

Summary

X connectors reviewed
Y critical issues found
Z warnings found
Failed connectors: N
Failed tasks: M

undefined

X connectors reviewed
Y critical issues found
Z warnings found
Failed connectors: N
Failed tasks: M

undefined