audit-logging

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Audit Logging

审计日志记录

Implement comprehensive audit logging for compliance, security monitoring, and forensic analysis across infrastructure and applications.
为基础设施与应用系统实现全面的审计日志记录,以满足合规要求、安全监控与取证分析需求。

When to Use

适用场景

  • Setting up centralized logging for compliance frameworks (SOC 2, HIPAA, PCI DSS)
  • Implementing security event monitoring and alerting
  • Building audit trails for regulatory requirements
  • Configuring log retention and tamper-proof storage
  • Integrating application logs with SIEM platforms
  • 为合规框架(SOC 2、HIPAA、PCI DSS)搭建集中式日志系统
  • 实现安全事件监控与告警
  • 构建符合监管要求的审计追踪链路
  • 配置日志保留与防篡改存储
  • 将应用日志与SIEM平台集成

Log Categories

日志分类

yaml
audit_events:
  authentication:
    - Login attempts (success and failure)
    - MFA enrollment and verification events
    - Session creation, renewal, and termination
    - Password changes and resets
    - API key and token generation

  authorization:
    - Access grants and denials
    - Permission changes and role assignments
    - Privilege escalation events
    - Resource sharing modifications
    - Policy evaluation results

  data_access:
    - Read operations on sensitive data
    - Write and update operations
    - Delete and purge operations
    - Bulk export and download events
    - Data classification changes

  administrative:
    - Configuration changes
    - User and group management
    - System startup and shutdown
    - Backup and restore operations
    - Network and firewall rule changes

  system:
    - Service health state changes
    - Resource provisioning and deprovisioning
    - Certificate and key rotation events
    - Scheduled job execution results
    - Integration and webhook events
yaml
audit_events:
  authentication:
    - Login attempts (success and failure)
    - MFA enrollment and verification events
    - Session creation, renewal, and termination
    - Password changes and resets
    - API key and token generation

  authorization:
    - Access grants and denials
    - Permission changes and role assignments
    - Privilege escalation events
    - Resource sharing modifications
    - Policy evaluation results

  data_access:
    - Read operations on sensitive data
    - Write and update operations
    - Delete and purge operations
    - Bulk export and download events
    - Data classification changes

  administrative:
    - Configuration changes
    - User and group management
    - System startup and shutdown
    - Backup and restore operations
    - Network and firewall rule changes

  system:
    - Service health state changes
    - Resource provisioning and deprovisioning
    - Certificate and key rotation events
    - Scheduled job execution results
    - Integration and webhook events

Rsyslog Configuration for Centralized Logging

集中式日志的Rsyslog配置

bash
undefined
bash
undefined

/etc/rsyslog.d/50-audit.conf

/etc/rsyslog.d/50-audit.conf

Load imfile module to read application logs

Load imfile module to read application logs

module(load="imfile")
module(load="imfile")

Forward auth logs

Forward auth logs

input(type="imfile" File="/var/log/auth.log" Tag="auth" Severity="info" Facility="auth" )
input(type="imfile" File="/var/log/auth.log" Tag="auth" Severity="info" Facility="auth" )

Forward application audit logs

Forward application audit logs

input(type="imfile" File="/var/log/app/audit.log" Tag="app-audit" Severity="info" Facility="local0" )
input(type="imfile" File="/var/log/app/audit.log" Tag="app-audit" Severity="info" Facility="local0" )

Structured JSON template

Structured JSON template

template(name="json-audit" type="list") { constant(value="{") constant(value=""@timestamp":"") property(name="timereported" dateFormat="rfc3339") constant(value="","host":"") property(name="hostname") constant(value="","severity":"") property(name="syslogseverity-text") constant(value="","facility":"") property(name="syslogfacility-text") constant(value="","tag":"") property(name="syslogtag" format="json") constant(value="","message":"") property(name="msg" format="json") constant(value=""}\n") }
template(name="json-audit" type="list") { constant(value="{") constant(value=""@timestamp":"") property(name="timereported" dateFormat="rfc3339") constant(value="","host":"") property(name="hostname") constant(value="","severity":"") property(name="syslogseverity-text") constant(value="","facility":"") property(name="syslogfacility-text") constant(value="","tag":"") property(name="syslogtag" format="json") constant(value="","message":"") property(name="msg" format="json") constant(value=""}\n") }

Forward to central syslog server over TLS

Forward to central syslog server over TLS

action( type="omfwd" target="syslog.internal.example.com" port="6514" protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" template="json-audit" queue.type="LinkedList" queue.size="50000" queue.filename="fwd_audit" queue.saveonshutdown="on" action.resumeRetryCount="-1" )
undefined
action( type="omfwd" target="syslog.internal.example.com" port="6514" protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" template="json-audit" queue.type="LinkedList" queue.size="50000" queue.filename="fwd_audit" queue.saveonshutdown="on" action.resumeRetryCount="-1" )
undefined

Journald Configuration for Persistent Logging

持久化日志的Journald配置

ini
undefined
ini
undefined

/etc/systemd/journald.conf

/etc/systemd/journald.conf

[Journal] Storage=persistent Compress=yes Seal=yes SplitMode=uid MaxRetentionSec=365d MaxFileSec=30d SystemMaxUse=10G SystemKeepFree=2G ForwardToSyslog=yes

```bash
[Journal] Storage=persistent Compress=yes Seal=yes SplitMode=uid MaxRetentionSec=365d MaxFileSec=30d SystemMaxUse=10G SystemKeepFree=2G ForwardToSyslog=yes

```bash

Query journald for audit events

Query journald for audit events

journalctl _TRANSPORT=audit --since "24 hours ago" --output json-pretty
journalctl _TRANSPORT=audit --since "24 hours ago" --output json-pretty

Filter by specific audit types

Filter by specific audit types

journalctl _AUDIT_TYPE=1112 --since today # user login events journalctl _AUDIT_TYPE=1100 --since today # user auth events
journalctl _AUDIT_TYPE=1112 --since today # user login events journalctl _AUDIT_TYPE=1100 --since today # user auth events

Export for offline analysis

Export for offline analysis

journalctl --since "7 days ago" --output export > /backup/journal-export.bin
undefined
journalctl --since "7 days ago" --output export > /backup/journal-export.bin
undefined

Application Logging with Structured JSON

基于结构化JSON的应用日志记录

python
import logging
import json
import hashlib
from datetime import datetime, timezone
from functools import wraps

class AuditLogger:
    def __init__(self, service_name, logger_name="audit"):
        self.service = service_name
        self.logger = logging.getLogger(logger_name)
        handler = logging.FileHandler("/var/log/app/audit.log")
        handler.setFormatter(logging.Formatter("%(message)s"))
        self.logger.addHandler(handler)
        self.logger.setLevel(logging.INFO)
        self._prev_hash = None

    def log_event(self, event_type, user, resource, action, result,
                  metadata=None, source_ip=None):
        log_entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "service": self.service,
            "event_type": event_type,
            "user": user,
            "resource": resource,
            "action": action,
            "result": result,
            "source_ip": source_ip,
            "metadata": metadata or {},
        }
        # Chain hash for tamper detection
        raw = json.dumps(log_entry, sort_keys=True)
        log_entry["prev_hash"] = self._prev_hash
        log_entry["hash"] = hashlib.sha256(
            f"{self._prev_hash}:{raw}".encode()
        ).hexdigest()
        self._prev_hash = log_entry["hash"]
        self.logger.info(json.dumps(log_entry))

    def log_auth(self, user, action, success, source_ip=None, mfa=False):
        self.log_event(
            event_type="authentication",
            user=user,
            resource="auth-service",
            action=action,
            result="success" if success else "failure",
            metadata={"mfa_used": mfa},
            source_ip=source_ip,
        )

    def log_data_access(self, user, resource, operation, record_count=0,
                        source_ip=None):
        self.log_event(
            event_type="data_access",
            user=user,
            resource=resource,
            action=operation,
            result="success",
            metadata={"record_count": record_count},
            source_ip=source_ip,
        )


def audit_trail(audit_logger, resource_name):
    """Decorator to automatically audit function calls."""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            user = kwargs.get("current_user", "system")
            try:
                result = func(*args, **kwargs)
                audit_logger.log_event(
                    event_type="operation",
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result="success",
                )
                return result
            except Exception as e:
                audit_logger.log_event(
                    event_type="operation",
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result="failure",
                    metadata={"error": str(e)},
                )
                raise
        return wrapper
    return decorator
python
import logging
import json
import hashlib
from datetime import datetime, timezone
from functools import wraps

class AuditLogger:
    def __init__(self, service_name, logger_name="audit"):
        self.service = service_name
        self.logger = logging.getLogger(logger_name)
        handler = logging.FileHandler("/var/log/app/audit.log")
        handler.setFormatter(logging.Formatter("%(message)s"))
        self.logger.addHandler(handler)
        self.logger.setLevel(logging.INFO)
        self._prev_hash = None

    def log_event(self, event_type, user, resource, action, result,
                  metadata=None, source_ip=None):
        log_entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "service": self.service,
            "event_type": event_type,
            "user": user,
            "resource": resource,
            "action": action,
            "result": result,
            "source_ip": source_ip,
            "metadata": metadata or {},
        }
        # Chain hash for tamper detection
        raw = json.dumps(log_entry, sort_keys=True)
        log_entry["prev_hash"] = self._prev_hash
        log_entry["hash"] = hashlib.sha256(
            f"{self._prev_hash}:{raw}".encode()
        ).hexdigest()
        self._prev_hash = log_entry["hash"]
        self.logger.info(json.dumps(log_entry))

    def log_auth(self, user, action, success, source_ip=None, mfa=False):
        self.log_event(
            event_type="authentication",
            user=user,
            resource="auth-service",
            action=action,
            result="success" if success else "failure",
            metadata={"mfa_used": mfa},
            source_ip=source_ip,
        )

    def log_data_access(self, user, resource, operation, record_count=0,
                        source_ip=None):
        self.log_event(
            event_type="data_access",
            user=user,
            resource=resource,
            action=operation,
            result="success",
            metadata={"record_count": record_count},
            source_ip=source_ip,
        )


def audit_trail(audit_logger, resource_name):
    """Decorator to automatically audit function calls."""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            user = kwargs.get("current_user", "system")
            try:
                result = func(*args, **kwargs)
                audit_logger.log_event(
                    event_type="operation",
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result="success",
                )
                return result
            except Exception as e:
                audit_logger.log_event(
                    event_type="operation",
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result="failure",
                    metadata={"error": str(e)},
                )
                raise
        return wrapper
    return decorator

Fluentd / Fluent Bit Log Aggregation

Fluentd / Fluent Bit日志聚合

yaml
undefined
yaml
undefined

fluent-bit.conf - lightweight agent on each node

fluent-bit.conf - lightweight agent on each node

[SERVICE] Flush 5 Daemon Off Log_Level info Parsers_File parsers.conf
[INPUT] Name tail Path /var/log/app/audit.log Parser json Tag audit.app Refresh_Interval 5 Rotate_Wait 30
[INPUT] Name systemd Tag audit.system Systemd_Filter _TRANSPORT=audit
[FILTER] Name modify Match audit.* Add cluster ${CLUSTER_NAME} Add node ${NODE_NAME}
[OUTPUT] Name es Match audit.* Host elasticsearch.internal.example.com Port 9200 Index audit-logs Type _doc tls On tls.verify On Retry_Limit 5
[OUTPUT] Name s3 Match audit.* region us-east-1 bucket audit-logs-archive total_file_size 50M upload_timeout 10m s3_key_format /logs/%Y/%m/%d/$TAG/%H_%M_%S.gz compression gzip
undefined
[SERVICE] Flush 5 Daemon Off Log_Level info Parsers_File parsers.conf
[INPUT] Name tail Path /var/log/app/audit.log Parser json Tag audit.app Refresh_Interval 5 Rotate_Wait 30
[INPUT] Name systemd Tag audit.system Systemd_Filter _TRANSPORT=audit
[FILTER] Name modify Match audit.* Add cluster ${CLUSTER_NAME} Add node ${NODE_NAME}
[OUTPUT] Name es Match audit.* Host elasticsearch.internal.example.com Port 9200 Index audit-logs Type _doc tls On tls.verify On Retry_Limit 5
[OUTPUT] Name s3 Match audit.* region us-east-1 bucket audit-logs-archive total_file_size 50M upload_timeout 10m s3_key_format /logs/%Y/%m/%d/$TAG/%H_%M_%S.gz compression gzip
undefined

Elasticsearch Index Lifecycle for Retention

用于日志保留的Elasticsearch索引生命周期

json
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}
json
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Retention Policy by Compliance Framework

基于合规框架的保留策略

yaml
retention_requirements:
  soc2:
    minimum: 1 year
    recommended: 3 years
    notes: "Based on audit period and report requirements"

  hipaa:
    minimum: 6 years
    notes: "From date of creation or last effective date"

  pci_dss:
    minimum: 1 year
    immediately_available: 3 months
    notes: "Req 10.7 - retain for at least one year, 3 months immediately available"

  gdpr:
    minimum: "As long as necessary for processing purpose"
    notes: "Apply data minimization; delete when no longer needed"

  fedramp:
    minimum: 3 years
    notes: "AU-11 control requirement"

  iso27001:
    minimum: "Defined by organization policy"
    recommended: 3 years
    notes: "A.12.4.1 - retention period must be defined"
yaml
retention_requirements:
  soc2:
    minimum: 1 year
    recommended: 3 years
    notes: "Based on audit period and report requirements"

  hipaa:
    minimum: 6 years
    notes: "From date of creation or last effective date"

  pci_dss:
    minimum: 1 year
    immediately_available: 3 months
    notes: "Req 10.7 - retain for at least one year, 3 months immediately available"

  gdpr:
    minimum: "As long as necessary for processing purpose"
    notes: "Apply data minimization; delete when no longer needed"

  fedramp:
    minimum: 3 years
    notes: "AU-11 control requirement"

  iso27001:
    minimum: "Defined by organization policy"
    recommended: 3 years
    notes: "A.12.4.1 - retention period must be defined"

Log Integrity Verification Script

日志完整性验证脚本

bash
#!/usr/bin/env bash
bash
#!/usr/bin/env bash

verify-log-integrity.sh - Verify log file checksums against stored hashes

verify-log-integrity.sh - Verify log file checksums against stored hashes

LOG_DIR="/var/log/app" HASH_FILE="/var/log/app/.checksums" ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"
verify_logs() { local failures=0 while IFS=' ' read -r stored_hash filename; do if [ -f "$filename" ]; then current_hash=$(sha256sum "$filename" | awk '{print $1}') if [ "$stored_hash" != "$current_hash" ]; then echo "TAMPER DETECTED: $filename" failures=$((failures + 1)) curl -s -X POST "$ALERT_WEBHOOK"
-H "Content-Type: application/json"
-d "{"text":"ALERT: Audit log tamper detected on $(hostname): $filename"}" fi else echo "MISSING: $filename" failures=$((failures + 1)) fi done < "$HASH_FILE"
return $failures }
update_checksums() { find "$LOG_DIR" -name "*.log" -type f -exec sha256sum {} ; > "$HASH_FILE" chmod 440 "$HASH_FILE" }
case "${1:-verify}" in verify) verify_logs ;; update) update_checksums ;; *) echo "Usage: $0 {verify|update}" ;; esac
undefined
LOG_DIR="/var/log/app" HASH_FILE="/var/log/app/.checksums" ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"
verify_logs() { local failures=0 while IFS=' ' read -r stored_hash filename; do if [ -f "$filename" ]; then current_hash=$(sha256sum "$filename" | awk '{print $1}') if [ "$stored_hash" != "$current_hash" ]; then echo "TAMPER DETECTED: $filename" failures=$((failures + 1)) curl -s -X POST "$ALERT_WEBHOOK"
-H "Content-Type: application/json"
-d "{"text":"ALERT: Audit log tamper detected on $(hostname): $filename"}" fi else echo "MISSING: $filename" failures=$((failures + 1)) fi done < "$HASH_FILE"
return $failures }
update_checksums() { find "$LOG_DIR" -name "*.log" -type f -exec sha256sum {} ; > "$HASH_FILE" chmod 440 "$HASH_FILE" }
case "${1:-verify}" in verify) verify_logs ;; update) update_checksums ;; *) echo "Usage: $0 {verify|update}" ;; esac
undefined

SIEM Integration Checklist

SIEM集成检查清单

yaml
siem_integration:
  log_sources:
    - [ ] Operating system auth logs (syslog, journald)
    - [ ] Application audit logs (structured JSON)
    - [ ] Cloud provider audit trails (CloudTrail, Activity Log, Audit Logs)
    - [ ] Database query and access logs
    - [ ] Network flow logs and firewall logs
    - [ ] Container and orchestrator logs (Kubernetes audit)
    - [ ] WAF and CDN access logs
    - [ ] VPN and remote access logs

  normalization:
    - [ ] Common event format (CEF) or OCSF schema
    - [ ] Consistent timestamp format (ISO 8601 / UTC)
    - [ ] Unified user identity fields
    - [ ] Standardized severity levels

  alerting_rules:
    - [ ] Multiple failed login attempts (brute force)
    - [ ] Login from unusual location or device
    - [ ] Privilege escalation events
    - [ ] Sensitive data bulk export
    - [ ] Administrative action outside change window
    - [ ] Service account anomalous activity
    - [ ] Log forwarding gap or interruption

  operational:
    - [ ] Log pipeline health monitoring
    - [ ] Storage capacity alerting
    - [ ] Retention policy enforcement verified
    - [ ] Backup of log archives confirmed
    - [ ] Access to log systems restricted and audited
yaml
siem_integration:
  log_sources:
    - [ ] Operating system auth logs (syslog, journald)
    - [ ] Application audit logs (structured JSON)
    - [ ] Cloud provider audit trails (CloudTrail, Activity Log, Audit Logs)
    - [ ] Database query and access logs
    - [ ] Network flow logs and firewall logs
    - [ ] Container and orchestrator logs (Kubernetes audit)
    - [ ] WAF and CDN access logs
    - [ ] VPN and remote access logs

  normalization:
    - [ ] Common event format (CEF) or OCSF schema
    - [ ] Consistent timestamp format (ISO 8601 / UTC)
    - [ ] Unified user identity fields
    - [ ] Standardized severity levels

  alerting_rules:
    - [ ] Multiple failed login attempts (brute force)
    - [ ] Login from unusual location or device
    - [ ] Privilege escalation events
    - [ ] Sensitive data bulk export
    - [ ] Administrative action outside change window
    - [ ] Service account anomalous activity
    - [ ] Log forwarding gap or interruption

  operational:
    - [ ] Log pipeline health monitoring
    - [ ] Storage capacity alerting
    - [ ] Retention policy enforcement verified
    - [ ] Backup of log archives confirmed
    - [ ] Access to log systems restricted and audited

Best Practices

最佳实践

  • Use structured logging (JSON) with consistent field names across all services
  • Ship logs to a centralized platform with write-once storage for tamper protection
  • Implement hash chaining or digital signatures for log integrity verification
  • Define and enforce retention policies per compliance framework requirements
  • Set up real-time alerting for high-severity security events
  • Separate audit logs from application debug logs to reduce noise
  • Never log sensitive data (passwords, tokens, PII) in audit entries
  • Monitor the logging pipeline itself to detect gaps in coverage
  • Regularly test log restoration from archives to verify recoverability
  • Rotate and compress logs to manage storage while meeting retention windows
  • 在所有服务中使用结构化日志(JSON)并保持字段名称一致
  • 将日志发送至具备写入即存储特性的集中式平台,以实现防篡改保护
  • 实现哈希链或数字签名机制以验证日志完整性
  • 根据合规框架要求定义并执行日志保留策略
  • 针对高优先级安全事件设置实时告警
  • 将审计日志与应用调试日志分离,以减少无效信息干扰
  • 切勿在审计条目中记录敏感数据(密码、令牌、个人可识别信息)
  • 监控日志采集链路本身,以检测覆盖缺口
  • 定期测试从归档中恢复日志的能力,验证可恢复性
  • 对日志进行轮转与压缩,在满足保留周期的同时优化存储管理