elk-stack

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

ELK Stack

Centralize and analyze logs with Elasticsearch, Logstash, and Kibana.

使用Elasticsearch、Logstash和Kibana集中化并分析日志。

When to Use This Skill

使用场景

Use this skill when:

Centralizing logs from multiple sources
Building log search and analytics platforms
Creating log-based dashboards and alerts
Implementing full-text search for logs
Processing and transforming log data

在以下场景中使用本技能：

集中化来自多数据源的日志
构建日志搜索与分析平台
创建基于日志的仪表板与告警
为日志实现全文搜索
处理与转换日志数据

Prerequisites

前提条件

Docker or server infrastructure
Sufficient disk space for log storage
Network access from log sources

Docker或服务器基础设施
足够的磁盘空间用于日志存储
日志源的网络访问权限

Docker Deployment

Docker部署

yaml

undefined

yaml

undefined

docker-compose.yml

version: '3.8'

services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0 environment: - discovery.type=single-node - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms1g -Xmx1g" ports: - "9200:9200" volumes: - elasticsearch-data:/usr/share/elasticsearch/data

logstash: image: docker.elastic.co/logstash/logstash:8.11.0 volumes: - ./logstash/pipeline:/usr/share/logstash/pipeline - ./logstash/config:/usr/share/logstash/config ports: - "5044:5044" - "5000:5000" depends_on: - elasticsearch

kibana: image: docker.elastic.co/kibana/kibana:8.11.0 ports: - "5601:5601" environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 depends_on: - elasticsearch

filebeat: image: docker.elastic.co/beats/filebeat:8.11.0 user: root volumes: - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro - /var/lib/docker/containers:/var/lib/docker/containers:ro - /var/run/docker.sock:/var/run/docker.sock:ro depends_on: - logstash

volumes: elasticsearch-data:

undefined

version: '3.8'

kibana: image: docker.elastic.co/kibana/kibana:8.11.0 ports: - "5601:5601" environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 depends_on: - elasticsearch

volumes: elasticsearch-data:

undefined

Elasticsearch Configuration

Elasticsearch配置

Index Templates

索引模板

json

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "trace_id": { "type": "keyword" }
      }
    }
  }
}

json

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "trace_id": { "type": "keyword" }
      }
    }
  }
}

Index Lifecycle Management

索引生命周期管理

json

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

json

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Logstash Pipeline

Logstash管道

Basic Pipeline

基础管道

ruby

undefined

ruby

undefined

logstash/pipeline/main.conf

input { beats { port => 5044 }

tcp { port => 5000 codec => json_lines } }

filter {

Parse JSON logs

if [message] =~ /^{/ { json { source => "message" } }

Parse timestamp

date { match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"] target => "@timestamp" }

Add environment tag

mutate { add_field => { "environment" => "production" } }

Grok pattern for nginx logs

if [type] == "nginx" { grok { match => { "message" => '%{IPORHOST:client_ip} - %{USER:user} [%{HTTPDATE:timestamp}] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes}' } } } }

output { elasticsearch { hosts => ["elasticsearch:9200"] index => "logs-%{+YYYY.MM.dd}" } }

undefined

input { beats { port => 5044 }

tcp { port => 5000 codec => json_lines } }

filter {

解析JSON日志

if [message] =~ /^{/ { json { source => "message" } }

解析时间戳

date { match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"] target => "@timestamp" }

添加环境标签

mutate { add_field => { "environment" => "production" } }

Nginx日志的Grok模式

output { elasticsearch { hosts => ["elasticsearch:9200"] index => "logs-%{+YYYY.MM.dd}" } }

undefined

Advanced Filtering

高级过滤

ruby

filter {
  # Parse application logs
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}"
    }
  }
  
  # Extract trace ID from message
  if [log_message] =~ /trace_id=/ {
    grok {
      match => { "log_message" => "trace_id=%{UUID:trace_id}" }
    }
  }
  
  # GeoIP lookup
  if [client_ip] {
    geoip {
      source => "client_ip"
      target => "geoip"
    }
  }
  
  # Drop debug logs in production
  if [level] == "DEBUG" and [environment] == "production" {
    drop {}
  }
  
  # Enrich with lookup
  translate {
    field => "status"
    destination => "status_description"
    dictionary => {
      "200" => "OK"
      "404" => "Not Found"
      "500" => "Internal Server Error"
    }
  }
}

ruby

filter {
  # 解析应用日志
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}"
    }
  }
  
  # 从日志消息中提取trace ID
  if [log_message] =~ /trace_id=/ {
    grok {
      match => { "log_message" => "trace_id=%{UUID:trace_id}" }
    }
  }
  
  # GeoIP查询
  if [client_ip] {
    geoip {
      source => "client_ip"
      target => "geoip"
    }
  }
  
  # 生产环境中丢弃调试日志
  if [level] == "DEBUG" and [environment] == "production" {
    drop {}
  }
  
  # 通过查询丰富字段
  translate {
    field => "status"
    destination => "status_description"
    dictionary => {
      "200" => "OK"
      "404" => "Not Found"
      "500" => "Internal Server Error"
    }
  }
}

Filebeat Configuration

Filebeat配置

yaml

undefined

yaml

undefined

filebeat/filebeat.yml

filebeat.inputs:

type: container paths:
- '/var/lib/docker/containers//.log' processors:
- add_docker_metadata: host: "unix:///var/run/docker.sock"
type: log enabled: true paths:
- /var/log/nginx/*.log tags: ["nginx"] fields: type: nginx

output.logstash: hosts: ["logstash:5044"]

logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7

undefined

filebeat.inputs:

type: container paths:
- '/var/lib/docker/containers//.log' processors:
- add_docker_metadata: host: "unix:///var/run/docker.sock"
type: log enabled: true paths:
- /var/log/nginx/*.log tags: ["nginx"] fields: type: nginx

output.logstash: hosts: ["logstash:5044"]

logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7

undefined

Elasticsearch Queries

Elasticsearch查询

Basic Queries

基础查询

json

// Search all logs
GET logs-*/_search
{
  "query": {
    "match_all": {}
  }
}

// Search by keyword
GET logs-*/_search
{
  "query": {
    "match": {
      "message": "error"
    }
  }
}

// Filter by field
GET logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "service": "api-gateway" } }
      ]
    }
  }
}

json

// 搜索所有日志
GET logs-*/_search
{
  "query": {
    "match_all": {}
  }
}

// 按关键词搜索
GET logs-*/_search
{
  "query": {
    "match": {
      "message": "error"
    }
  }
}

// 按字段过滤
GET logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "service": "api-gateway" } }
      ]
    }
  }
}

Aggregations

聚合查询

json

// Count by log level
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "log_levels": {
      "terms": { "field": "level" }
    }
  }
}

// Error rate over time
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "5m"
      },
      "aggs": {
        "error_count": {
          "filter": { "term": { "level": "ERROR" } }
        }
      }
    }
  }
}

json

// 按日志级别统计数量
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "log_levels": {
      "terms": { "field": "level" }
    }
  }
}

// 按时间统计错误率
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "5m"
      },
      "aggs": {
        "error_count": {
          "filter": { "term": { "level": "ERROR" } }
        }
      }
    }
  }
}

Kibana Setup

Kibana设置

Index Patterns

索引模式

Go to Stack Management → Index Patterns
Create pattern:
```
logs-*
```
Set time field:
```
@timestamp
```

进入 Stack Management → Index Patterns
创建模式：
```
logs-*
```
设置时间字段：
```
@timestamp
```

Saved Searches

已保存搜索

Create saved searches for common queries:

```
level:ERROR
```
- All errors
```
service:api-gateway AND level:ERROR
```
- API gateway errors
```
response_time:>1000
```
- Slow requests

为常见查询创建已保存搜索：

```
level:ERROR
```
- 所有错误日志
```
service:api-gateway AND level:ERROR
```
- API网关错误日志
```
response_time:>1000
```
- 慢请求日志

Visualizations

可视化图表

Common visualization types:

Line Chart: Error rate over time
Pie Chart: Distribution by log level
Data Table: Top error messages
Metric: Total error count

常见可视化类型：

折线图：按时间统计错误率趋势
饼图：日志级别分布
数据表：高频错误消息
指标卡：错误日志总数

Dashboard Example

仪表板示例

Create dashboard with:

Total log count (Metric)
Error rate trend (Line chart)
Logs by service (Pie chart)
Recent errors (Data table)
Log stream (Discover panel)

创建包含以下组件的仪表板：

日志总数（指标卡）
错误率趋势（折线图）
按服务划分的日志分布（饼图）
近期错误日志（数据表）
日志流（Discover面板）

Alerting

告警

Watcher (X-Pack)

Watcher（X-Pack）

json

PUT _watcher/watch/error_alert
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "match": { "level": "ERROR" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 100 } }
  },
  "actions": {
    "notify_slack": {
      "webhook": {
        "scheme": "https",
        "host": "hooks.slack.com",
        "port": 443,
        "method": "post",
        "path": "/services/xxx",
        "body": "{\"text\": \"High error rate detected: {{ctx.payload.hits.total.value}} errors in last 5 minutes\"}"
      }
    }
  }
}

json

PUT _watcher/watch/error_alert
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "match": { "level": "ERROR" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 100 } }
  },
  "actions": {
    "notify_slack": {
      "webhook": {
        "scheme": "https",
        "host": "hooks.slack.com",
        "port": 443,
        "method": "post",
        "path": "/services/xxx",
        "body": "{\"text\": \"检测到高错误率：过去5分钟内出现{{ctx.payload.hits.total.value}}条错误日志\"}"
      }
    }
  }
}

Common Issues

常见问题

Issue: High Disk Usage

问题：磁盘占用过高

Problem: Elasticsearch consuming too much disk Solution: Implement ILM policies, reduce retention

现象：Elasticsearch占用过多磁盘空间 解决方案：实施ILM策略，缩短日志保留周期

Issue: Slow Searches

问题：搜索速度缓慢

Problem: Queries taking too long Solution: Optimize index settings, add more shards, use filters

现象：查询耗时过长 解决方案：优化索引设置，增加分片数量，使用过滤器

Issue: Log Parsing Failures

问题：日志解析失败

Problem: Logs not parsed correctly Solution: Test grok patterns, check for log format changes

现象：日志未被正确解析 解决方案：测试Grok模式，检查日志格式是否变更

Issue: Memory Pressure

问题：内存压力过大

Problem: Elasticsearch OOM errors Solution: Increase heap size (max 50% of RAM), limit field data

现象：Elasticsearch出现OOM错误 解决方案：增加堆内存大小（最大为内存的50%），限制字段数据量

Best Practices

最佳实践

Implement index lifecycle management
Use index templates for consistent mappings
Parse logs at ingestion time
Limit stored fields to reduce storage
Use data streams for time-series data
Monitor cluster health
Implement proper security (X-Pack)
Regular index maintenance

实施索引生命周期管理
使用索引模板保证映射一致性
在日志摄入阶段完成解析
限制存储字段以减少磁盘占用
为时间序列数据使用数据流
监控集群健康状态
启用适当的安全机制（X-Pack）
定期进行索引维护

elk-stack

Original

Translation

ELK Stack

ELK Stack

When to Use This Skill

使用场景

Prerequisites

前提条件

Docker Deployment

Docker部署

docker-compose.yml

docker-compose.yml

Elasticsearch Configuration

Elasticsearch配置

Index Templates

索引模板

Index Lifecycle Management

索引生命周期管理

Logstash Pipeline

Logstash管道

Basic Pipeline

基础管道

logstash/pipeline/main.conf

logstash/pipeline/main.conf

Parse JSON logs

Parse timestamp

Add environment tag

Grok pattern for nginx logs

解析JSON日志

解析时间戳

添加环境标签

Nginx日志的Grok模式

Advanced Filtering

高级过滤

Filebeat Configuration

Filebeat配置

filebeat/filebeat.yml

filebeat/filebeat.yml

Elasticsearch Queries

Elasticsearch查询

Basic Queries

基础查询

Aggregations

聚合查询

Kibana Setup

Kibana设置

Index Patterns

索引模式

Saved Searches

已保存搜索

Visualizations

可视化图表

Dashboard Example

仪表板示例

Alerting

告警

Watcher (X-Pack)

Watcher（X-Pack）

Common Issues

常见问题

Issue: High Disk Usage

问题：磁盘占用过高

Issue: Slow Searches

问题：搜索速度缓慢

Issue: Log Parsing Failures

问题：日志解析失败

Issue: Memory Pressure

问题：内存压力过大

Best Practices

最佳实践

Related Skills

相关技能