• Cluster-wide networking impact (CNI misconfiguration can break entire cluster)
• Security policy errors (accidentally block critical traffic or allow unauthorized access)
• Service mesh failures (break mTLS, observability, load balancing)
• Network performance degradation (inefficient policies, resource exhaustion)
• Data plane disruption (eBPF program failures, kernel compatibility issues)

• CNI Configuration: Cilium as Kubernetes CNI, IPAM modes, tunnel overlays (VXLAN/Geneve), direct routing
• Network Policies: L3/L4 policies, L7 HTTP/gRPC/Kafka policies, DNS-based policies, FQDN filtering, deny policies
• Service Mesh: Cilium Service Mesh, mTLS, traffic management, canary deployments, circuit breaking
• Observability: Hubble for flow visibility, service maps, metrics (Prometheus), distributed tracing
• Security: Zero-trust networking, identity-based policies, encryption (WireGuard, IPsec), network segmentation
• eBPF Programs: Understanding eBPF datapath, XDP, TC hooks, socket-level filtering, performance optimization
• Multi-Cluster: ClusterMesh for multi-cluster networking, global services, cross-cluster policies
• Integration: Kubernetes NetworkPolicy compatibility, Ingress/Gateway API, external workloads

• Secure: Zero-trust by default, least-privilege policies, encrypted communication
• Performant: eBPF-native, kernel bypass, minimal overhead, efficient resource usage
• Observable: Full flow visibility, real-time monitoring, audit logs, troubleshooting capabilities
• Reliable: Robust policies, graceful degradation, tested failover scenarios

1. TDD First: Write connectivity tests and policy validation before implementing network changes
2. Performance Aware: Optimize eBPF programs, policy selectors, and Hubble sampling for minimal overhead
3. Zero-Trust by Default: All traffic denied unless explicitly allowed with identity-based policies
4. Observe Before Enforce: Enable Hubble and test policies in audit mode before enforcement
5. Identity Over IPs: Use Kubernetes labels and workload identity, never hard-coded IP addresses
6. Encrypt Sensitive Traffic: WireGuard or mTLS for all inter-service communication
7. Continuous Monitoring: Alert on policy denies, dropped flows, and eBPF program errors

• Installation: Helm charts, cilium CLI, operator deployment, agent DaemonSet
• IPAM Modes: Kubernetes (PodCIDR), cluster-pool, Azure/AWS/GCP native IPAM
• Datapath: Tunnel mode (VXLAN/Geneve), native routing, DSR (Direct Server Return)
• IP Management: IPv4/IPv6 dual-stack, pod CIDR allocation, node CIDR management
• Kernel Requirements: Minimum kernel 4.9.17+, recommended 5.10+, eBPF feature detection
• HA Configuration: Multiple replicas for operator, agent health checks, graceful upgrades
• Kube-proxy Replacement: Full kube-proxy replacement mode, socket-level load balancing
• Feature Flags: Enable/disable features (Hubble, encryption, service mesh, host-firewall)

• L3/L4 Policies: CIDR-based rules, pod/namespace selectors, port-based filtering
• L7 Policies: HTTP method/path filtering, gRPC service/method filtering, Kafka topic filtering
• DNS Policies: matchPattern for DNS names, FQDN-based egress filtering, DNS security
• Deny Policies: Explicit deny rules, default-deny namespaces, policy precedence
• Entity-Based: toEntities (world, cluster, host, kube-apiserver), identity-aware policies
• Ingress/Egress: Separate ingress and egress rules, bi-directional traffic control
• Policy Enforcement: Audit mode vs enforcing mode, policy verdicts, troubleshooting denies
• Compatibility: Support for Kubernetes NetworkPolicy API, CiliumNetworkPolicy CRDs

• Sidecar-less Architecture: eBPF-based service mesh, no sidecar overhead
• mTLS: Automatic mutual TLS between services, certificate management, SPIFFE/SPIRE integration
• Traffic Management: Load balancing algorithms (round-robin, least-request), health checks
• Canary Deployments: Traffic splitting, weighted routing, gradual rollouts
• Circuit Breaking: Connection limits, request timeouts, retry policies, failure detection
• Ingress Control: Cilium Ingress controller, Gateway API support, TLS termination
• Service Maps: Real-time service topology, dependency graphs, traffic flows
• L7 Visibility: HTTP/gRPC metrics, request/response logging, latency tracking

• Hubble Deployment: Hubble server, Hubble Relay, Hubble UI, Hubble CLI
• Flow Monitoring: Real-time flow logs, protocol detection, drop reasons, policy verdicts
• Service Maps: Visual service topology, traffic patterns, cross-namespace flows
• Metrics: Prometheus integration, flow metrics, drop/forward rates, policy hit counts
• Troubleshooting: Debug connection failures, identify policy denies, trace packet paths
• Audit Logging: Compliance logging, policy change tracking, security events
• Distributed Tracing: OpenTelemetry integration, span correlation, end-to-end tracing
• CLI Workflows:

  hubble observe

  ,

  hubble status

  , flow filtering, JSON output

• Identity-Based Policies: Kubernetes identity (labels), SPIFFE identities, workload attestation
• Encryption: WireGuard transparent encryption, IPsec encryption, per-namespace encryption
• Network Segmentation: Isolate namespaces, multi-tenancy, environment separation (dev/staging/prod)
• Egress Control: Restrict external access, FQDN filtering, transparent proxy for HTTP(S)
• Threat Detection: DNS security, suspicious flow detection, policy violation alerts
• Host Firewall: Protect node traffic, restrict access to node ports, system namespace isolation
• API Security: L7 policies for API gateway, rate limiting, authentication enforcement
• Compliance: PCI-DSS network segmentation, HIPAA data isolation, SOC2 audit trails

• eBPF Efficiency: Minimize program complexity, optimize map lookups, batch operations
• Resource Tuning: Memory limits, CPU requests, eBPF map sizes, connection tracking limits
• Datapath Selection: Choose optimal datapath (native routing > tunneling), MTU configuration
• Kube-proxy Replacement: Socket-based load balancing, XDP acceleration, eBPF host-routing
• Policy Optimization: Reduce policy complexity, use efficient selectors, aggregate rules
• Monitoring Overhead: Tune Hubble sampling rates, metric cardinality, flow export rates
• Upgrade Strategies: Rolling updates, minimize disruption, test in staging, rollback procedures
• Troubleshooting: High CPU usage, memory pressure, eBPF program failures, connectivity issues

Empty ingress/egress = deny all

ingress: [] egress: []

• toEndpoints:
  • matchLabels: io.kubernetes.pod.namespace: kube-system k8s-app: kube-dns toPorts:
  • ports:
    • port: "53" protocol: UDP rules: dns:
      • matchPattern: "*" # Allow all DNS queries

• toEndpoints:
  • matchLabels: app: backend io.kubernetes.pod.namespace: production toPorts:
  • ports:
    • port: "8080" protocol: TCP rules: http:
      • method: "GET|POST" path: "/api/.*"

**Key Points**:
- Start with default-deny, then allow specific traffic
- Always allow DNS (kube-dns) or pods can't resolve names
- Use namespace labels to prevent cross-namespace traffic
- Test policies in audit mode first (`policyAuditMode: true`)

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-gateway-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-gateway
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        # Only allow specific API endpoints
        - method: "GET"
          path: "/api/v1/(users|products)/.*"
          headers:
          - "X-API-Key: .*"  # Require API key header
        - method: "POST"
          path: "/api/v1/orders"
          headers:
          - "Content-Type: application/json"
  egress:
  - toEndpoints:
    - matchLabels:
        app: user-service
    toPorts:
    - ports:
      - port: "3000"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/users/.*"
  - toFQDNs:
    - matchPattern: "*.stripe.com"  # Allow Stripe API
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

• L7 policies require protocol parser (HTTP/gRPC/Kafka)
• Use regex for path matching:

  /api/v1/.*

• Headers can enforce API keys, content types
• Combine L7 rules with FQDN filtering for external APIs
• Higher overhead than L3/L4 - use selectively

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: external-api-access
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-processor
  egress:
  # Allow specific external domains
  - toFQDNs:
    - matchName: "api.stripe.com"
    - matchName: "api.paypal.com"
    - matchPattern: "*.amazonaws.com"  # AWS services
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
  # Allow Kubernetes DNS
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      rules:
        dns:
        # Only allow DNS queries for approved domains
        - matchPattern: "*.stripe.com"
        - matchPattern: "*.paypal.com"
        - matchPattern: "*.amazonaws.com"
  # Deny all other egress
  - toEntities:
    - kube-apiserver  # Allow API server access

• toFQDNs

  uses DNS lookups to resolve IPs dynamically
• Requires DNS proxy to be enabled in Cilium

• matchName

  for exact domain,

  matchPattern

  for wildcards
• DNS rules restrict which domains can be queried
• TTL-aware: updates rules when DNS records change

• port: 8080 protocol: TCP

• toEndpoints:
  • matchLabels: app: backend io.kubernetes.pod.namespace: production

    Matches pods in ANY connected cluster

  toPorts:
  • ports:
    • port: "8080" protocol: TCP

**Key Points**:
- Each cluster needs unique `cluster.id` and `cluster.name`
- ClusterMesh API server handles cross-cluster communication
- Global services automatically load-balance across clusters
- Policies work transparently across clusters
- Supports multi-region HA and disaster recovery

• fromEndpoints:
  • matchLabels: io.kubernetes.pod.namespace: production egress:
• toEndpoints:
  • matchLabels: io.kubernetes.pod.namespace: production

**Key Points**:
- WireGuard: modern, performant (recommended for kernel 5.6+)
- IPsec: older kernels, more overhead
- Transparent: no application changes needed
- Node-to-node encryption for cross-node traffic
- Verify with `hubble observe --verdict ENCRYPTED`
- Minimal performance impact (~5-10% overhead)

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: host-firewall
spec:
  nodeSelector: {}  # Apply to all nodes
  ingress:
  # Allow SSH from bastion hosts only
  - fromCIDR:
    - 10.0.1.0/24  # Bastion subnet
    toPorts:
    - ports:
      - port: "22"
        protocol: TCP

  # Allow Kubernetes API server
  - fromEntities:
    - cluster
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP

  # Allow kubelet API
  - fromEntities:
    - cluster
    toPorts:
    - ports:
      - port: "10250"
        protocol: TCP

  # Allow node-to-node (Cilium, etcd, etc.)
  - fromCIDR:
    - 10.0.0.0/16  # Node CIDR
    toPorts:
    - ports:
      - port: "4240"  # Cilium health
        protocol: TCP
      - port: "4244"  # Hubble server
        protocol: TCP

  # Allow monitoring
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: monitoring
    toPorts:
    - ports:
      - port: "9090"  # Node exporter
        protocol: TCP

  egress:
  # Allow all egress from nodes (can be restricted)
  - toEntities:
    - all

• Use

  CiliumClusterwideNetworkPolicy

  for node-level policies
• Protect SSH, kubelet, API server access
• Restrict to bastion hosts or specific CIDRs
• Test carefully - can lock you out of nodes!
• Monitor with

  hubble observe --from-reserved:host

• Default Deny: All traffic denied unless explicitly allowed
• Least Privilege: Grant minimum necessary access
• Identity-Based: Use workload identity (labels), not IPs
• Encryption: All inter-service traffic encrypted (mTLS, WireGuard)
• Continuous Verification: Monitor and audit all traffic

undefined

• fromEndpoints:
  • matchLabels: app: frontend env: production # Require specific identity

Policy logged but not enforced

undefined

• fromEndpoints:
  • matchLabels: io.kubernetes.pod.namespace: tenant-a # Same namespace only egress:
• toEndpoints:
  • matchLabels: io.kubernetes.pod.namespace: tenant-a
• toEntities:
  • kube-apiserver
  • kube-dns

**Environment Isolation** (dev/staging/prod):

```yaml

• fromEndpoints:
  • matchLabels: env: production # Only prod can talk to prod ingressDeny:
• fromEndpoints:
  • matchLabels: env: development # Explicit deny from dev

undefined

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set authentication.mutual.spire.enabled=true \
  --set authentication.mutual.spire.install.enabled=true

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: mtls-required
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: api-gateway
    authentication:
      mode: "required"  # Require mTLS authentication

• See

  references/network-policies.md

  for advanced policy examples
• See

  references/observability.md

  for security monitoring with Hubble

• name: curl image: curlimages/curl:latest command: ["sleep", "infinity"] EOF

• fromEndpoints:
  • matchLabels: app: frontend # Only frontend allowed, not test-client toPorts:
  • ports:
    • port: "8080" protocol: TCP

undefined

• toCIDR:
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16

**Good** - Identity-based rules with eBPF map caching:
```yaml

• toEndpoints:
  • matchLabels: app: backend io.kubernetes.pod.namespace: production
• toEntities:
  • cluster # Pre-cached entity

undefined

Default: all flows captured

**Good** - Sampling for production workloads:
```yaml

Reduce cardinality

• toPorts:
  • ports:
    • port: "8080" rules: http:
      • method: ".*"

**Good** - Selective L7 policy for specific services:
```yaml

• fromEndpoints:
  • matchLabels: app: frontend toPorts:
  • ports:
    • port: "8080" rules: http:
      • method: "GET|POST" path: "/api/v1/.*"

undefined

• name: curl image: curlimages/curl:latest command: ["sleep", "infinity"]

• name: nginx image: nginx:alpine ports:
  • containerPort: 80 EOF

• fromEndpoints:
  • matchLabels: app: client toPorts:
  • ports:
    • port: "80" protocol: TCP EOF

• Read existing policies - Understand current network policy state
• Check Cilium version -

  cilium version

  for feature compatibility
• Verify kernel version - Minimum 4.9.17, recommend 5.10+
• Review PRD requirements - Identify security and connectivity requirements
• Plan test strategy - Define connectivity tests before implementation
• Enable Hubble - Required for policy validation and troubleshooting
• Check cluster state -

  cilium status

  and

  cilium connectivity test

• Identify affected workloads - Map services that will be impacted
• Review release notes - Check for breaking changes if upgrading

• Write failing tests first - Create connectivity tests before policies
• Use audit mode - Deploy with

  cilium.io/policy-audit-mode: "true"

• Always allow DNS - Include kube-dns egress in every namespace
• Allow kube-apiserver - Use

  toEntities: [kube-apiserver]

• Use identity-based selectors - Labels over CIDR where possible
• Verify selectors -

  kubectl get pods -l app=backend

  to test
• Monitor Hubble flows - Watch for AUDIT/DROPPED verdicts
• Validate incrementally - Apply one policy at a time
• Document policy purpose - Add annotations explaining intent

• Run full connectivity test -

  cilium connectivity test

• Verify no unexpected drops -

  hubble observe --verdict DROPPED

• Check policy enforcement - Remove audit mode annotation
• Test rollback procedure - Ensure policies can be quickly removed
• Validate performance - Check eBPF map usage and agent resources
• Run helm validation -

  helm template --validate

  for chart changes
• Document exceptions - Explain allowed traffic paths
• Update runbooks - Include troubleshooting steps for new policies
• Peer review - Have another engineer review critical policies

• Backup ConfigMaps - Save cilium-config before changes
• Test upgrades in staging - Never upgrade Cilium in prod first
• Plan maintenance window - For disruptive upgrades
• Verify eBPF features -

  cilium status

  shows feature availability
• Monitor agent health -

  kubectl -n kube-system get pods -l k8s-app=cilium

• Check endpoint health - All endpoints should be in ready state

• Default-deny policies - Every namespace should have baseline policies
• Enable encryption - WireGuard for pod-to-pod traffic
• mTLS for sensitive services - Payment, auth, PII-handling services
• FQDN filtering - Control egress to external services
• Host firewall - Protect nodes from unauthorized access
• Audit logging - Enable Hubble for compliance
• Regular policy reviews - Quarterly review and remove unused policies
• Incident response plan - Procedures for policy-related outages

• Use native routing - Avoid tunnels (VXLAN) when possible
• Enable kube-proxy replacement - Better performance with eBPF
• Optimize map sizes - Tune based on cluster size
• Monitor eBPF program stats - Check for errors, drops
• Set resource limits - Prevent OOM kills of cilium agents
• Reduce policy complexity - Aggregate rules, simplify selectors
• Tune Hubble sampling - Balance visibility vs overhead

1. Configures Cilium CNI for high-performance, secure Kubernetes networking
2. Implements network policies at L3/L4/L7 with identity-based, zero-trust approach
3. Deploys service mesh features (mTLS, traffic management) without sidecars
4. Enables observability with Hubble for real-time flow visibility and troubleshooting
5. Hardens security with encryption, network segmentation, and egress control
6. Optimizes performance with eBPF-native datapath and kube-proxy replacement
7. Manages multi-cluster networking with ClusterMesh for global services
8. Troubleshoots issues using Hubble CLI, flow logs, and policy auditing

• Zero-trust by default: Deny all, then allow specific traffic
• Identity over IPs: Use labels, not IP addresses
• Observe first: Enable Hubble before enforcing policies
• Test in audit mode: Never deploy untested policies to production
• Encrypt sensitive traffic: WireGuard or mTLS for compliance
• Monitor continuously: Alert on policy denies and dropped flows
• Performance matters: eBPF is fast, but bad policies can slow it down

• references/network-policies.md

  - Comprehensive L3/L4/L7 policy examples

• references/observability.md

  - Hubble setup, troubleshooting workflows, metrics