Kubernetes Log Collection with Purl

Deploy Purl on your Kubernetes cluster with Helm and collect logs from all pods automatically using Fluent Bit or OpenTelemetry Collector.

Prerequisites

✓ Kubernetes cluster (1.24+)
✓ Helm 3 installed
✓ kubectl configured with cluster access

Deploy Purl via Helm

Add the Purl Helm repository and install Purl with ClickHouse in the monitoring namespace.

bash

# Add Purl Helm repo
helm repo add purl https://charts.purlogs.com
helm repo update

# Create namespace
kubectl create namespace monitoring

# Install Purl with ClickHouse
helm install purl purl/purl \
  --namespace monitoring \
  --set clickhouse.enabled=true \
  --set purl.apiKeys="your-api-key" \
  --set purl.retention.days=90

# Verify deployment
kubectl get pods -n monitoring

Deploy Fluent Bit DaemonSet

Fluent Bit runs as a DaemonSet on every node, tailing container logs and forwarding them to Purl. It automatically enriches logs with Kubernetes metadata (pod name, namespace, labels).

fluent-bit-values.yaml

# values.yaml for Fluent Bit Helm chart
config:
  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        Refresh_Interval  10
        Mem_Buf_Limit     5MB

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off

  outputs: |
    [OUTPUT]
        Name          http
        Match         *
        Host          purl.monitoring.svc.cluster.local
        Port          3000
        URI           /api/v1/logs
        Format        json
        Header        X-API-Key your-api-key
        Header        Content-Type application/json
        Retry_Limit   3

bash

# Install Fluent Bit
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  --namespace monitoring \
  -f fluent-bit-values.yaml

Alternative: OpenTelemetry Collector

If you prefer the OpenTelemetry ecosystem, deploy the OTel Collector as a DaemonSet to forward logs via OTLP to Purl's native endpoint.

otel-collector.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: purl-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      filelog:
        include:
          - /var/log/pods/*/*/*.log
        operators:
          - type: container
            id: container-parser

    processors:
      k8sattributes:
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
            - k8s.deployment.name

    exporters:
      otlphttp:
        endpoint: http://purl.monitoring:3000/api/v1/otlp
        headers:
          X-API-Key: your-api-key

    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [k8sattributes]
          exporters: [otlphttp]

Verify and Monitor

Check that logs are flowing by opening the Purl dashboard or querying the API.

bash

# Port-forward to access Purl dashboard
kubectl port-forward -n monitoring svc/purl 3000:3000

# Verify logs via API
curl -s http://localhost:3000/api/v1/search \
  -H "X-API-Key: your-api-key" \
  -G -d 'q=*' -d 'limit=5' | jq .

# Check Fluent Bit logs for errors
kubectl logs -n monitoring -l app.kubernetes.io/name=fluent-bit --tail=50

Advanced: Namespace Filtering

To collect logs only from specific namespaces, add an exclude filter to Fluent Bit:

fluent-bit.conf

[FILTER]
    Name    grep
    Match   kube.*
    Exclude $kubernetes['namespace_name'] kube-system|kube-public

← Back to all guides