Kafka Topic Configuration Audit
Audits all topic configurations against production best practices. Misconfigured topics are the #1 cause of Kafka data loss - engineers create topics and forget to tune them.
Workflow
Copy this checklist and track your progress:
Audit Progress:
- [ ] Step 1: Check environment health
- [ ] Step 2: Fetch all topics
- [ ] Step 3: Audit configurations against best practices
- [ ] Step 4: Check metadata completeness
- [ ] Step 5: Detect orphaned topics
- [ ] Step 6: Run consistency checks
- [ ] Step 7: Generate report
- Check environment health for a high-level summary
- Fetch all topics and their configurations
- Audit each topic against best practices (see
references/audit-rules.md
- Cross-reference metadata for completeness
- Detect orphaned topics with no consumers
- Report findings with prioritised recommendations
Step 1: Environment Overview
Use the Lenses MCP
tool to get a quick summary:
- Broker count, topic count, consumer count, connector count
- Any existing issues flagged by Lenses
Expected output: Environment health summary with broker, topic and consumer counts.
Validation: If the environment is unhealthy or unreachable, stop and report the connection issue before proceeding.
Step 2: Fetch All Topics
Use the Lenses MCP
tool to retrieve all topics with their configurations in one call.
For topics that need deeper inspection, use:
- for detailed config including partitions and consumers
- for broker-level config overrides
- for partition-level message counts and bytes
Expected output: Full list of topics with their configurations. If zero topics are returned, report this and stop.
Step 3: Audit Configurations
For each topic, check against the thresholds in
references/audit-rules.md
:
- Replication factor - RF=1 is critical, RF=2 is a warning in production
- Retention policies - unbounded growth, too short or excessively long
- Partition count - single-partition bottlenecks or excessive partitions
- Compaction settings - compact without keys, delete for state topics
- Naming conventions - must follow
{domain}.{entity}.{event}
Step 4: Metadata Completeness
Use the Lenses MCP
tool to check:
- Topics missing descriptions
- Topics missing tags
- Topics without registered schemas (key or value)
Use
with filters (
,
) to find anomalies.
Step 5: Orphan Detection
For each topic, use
list_consumer_groups_by_topic
to check for active consumers.
- Warning: Topics with zero consumer groups (may be orphaned/dead)
- Suggestion: Topics with only inactive/empty consumer groups
Step 6: Consistency Checks
- Flag topics in the same domain with different retention policies
- Flag topics in the same domain with different replication factors
- Flag topics with inconsistent serialisation formats within a domain
Success Criteria
Quantitative
- Triggers on 90% of topic-related queries (test with 10-20 varied phrasings)
- Completes full audit in under 15 MCP tool calls
- 0 failed MCP calls per run
Qualitative
- Report is actionable without follow-up questions from the user
- Consistent severity categorisation (critical/warning/suggestion) across runs
- Every finding includes a concrete remediation step
Examples
Example 1: Routine weekly audit
User says: "Run a topic audit on the staging environment"
Actions:
- Check staging environment health via Lenses MCP
- Fetch all topics and configs
- Audit each topic against rules in
references/audit-rules.md
- Check metadata completeness and orphaned topics
Result: Full audit report with prioritised findings
Example 2: Pre-deployment check
User says: "Check if my topic configs are production-ready"
Actions:
- Audit all topics for RF < 3, unbounded retention, single partitions
- Flag any critical issues that would block a production deployment
Result: Report highlighting critical issues that must be fixed before go-live
Example 3: Investigate a specific topic
User says: "Is the orders.payment.completed topic configured correctly?"
Actions:
- Fetch detailed config for the specific topic using
- Check broker-level overrides with
- Verify metadata and consumer groups
Result: Focused report on a single topic with all findings
Troubleshooting
Lenses MCP connection failed
Cause: Environment name is incorrect or Lenses agent is offline.
Solution: Run
first. Verify the environment name matches what
returns.
No topics returned
Cause: Environment exists but has no topics or permissions are restricted.
Solution: Confirm the cluster has topics via the Lenses UI. Check that the Lenses agent has read access.
Metadata endpoint returns empty
Cause: Schema Registry is not configured or topics have no registered schemas.
Solution: This is a valid finding - report it as missing metadata rather than treating it as an error.
Output Format
## Topic Audit Report
### Environment: {name}
- Brokers: X | Topics: Y | Consumer groups: Z
### Critical (must fix)
- [topic-name] Description of the issue
Current: {current value} | Recommended: {recommended value}
### Warning (should fix)
- [topic-name] Description of the issue
Current: {current value} | Recommended: {recommended value}
### Suggestion (consider improving)
- [topic-name] Description of the issue
Recommendation: How to fix it
### Summary
- X critical issues found
- Y warnings found
- Z suggestions found
- Topics audited: N
- Orphaned topics: M