Kubernetes Log Collection with Purl

Deploy Purl on your Kubernetes cluster with Helm and collect logs from all pods automatically using Fluent Bit or OpenTelemetry Collector.

Prerequisites

✓ Kubernetes cluster (1.24+)
✓ Helm 3 installed
✓ kubectl configured with cluster access

Deploy Purl via Helm

Add the Purl Helm repository and install Purl with ClickHouse in the monitoring namespace.

bash

# Add Purl Helm repo
helm repo add purl https://charts.purlogs.com
helm repo update

# Create namespace
kubectl create namespace monitoring

# Install Purl with ClickHouse
helm install purl purl/purl \
  --namespace monitoring \
  --set clickhouse.enabled=true \
  --set purl.apiKeys="your-api-key" \
  --set purl.retentionDays=90

# Verify deployment
kubectl get pods -n monitoring

Deploy Fluent Bit DaemonSet

Fluent Bit runs as a DaemonSet on every node, tailing container logs and forwarding them to Purl. It automatically enriches logs with Kubernetes metadata (pod name, namespace, labels).

fluent-bit-values.yaml

# values.yaml for Fluent Bit Helm chart
config:
  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        Refresh_Interval  10
        Mem_Buf_Limit     5MB

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off

  outputs: |
    [OUTPUT]
        Name          http
        Match         *
        Host          purl.monitoring.svc.cluster.local
        Port          3000
        URI           /api/logs
        Format        json
        Header        X-API-Key your-api-key
        Header        Content-Type application/json
        Retry_Limit   3

bash

# Install Fluent Bit
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  --namespace monitoring \
  -f fluent-bit-values.yaml

Recommended: Built-in Vector Collector

The Purl Helm chart ships with its own Vector DaemonSet that maps pod labels to the service field and stores Kubernetes metadata (namespace, pod, node) in meta, so it stays queryable in Purl:

bash

helm upgrade purl purl/purl \
  --namespace monitoring \
  --reuse-values \
  --set vector.enabled=true

With an external Fluent Bit, the enriched kubernetes metadata object is not mapped into Purl's meta field and logs may arrive with service unknown.

Alternative: OpenTelemetry Collector

If you prefer the OpenTelemetry ecosystem, deploy the OTel Collector as a DaemonSet to forward logs via OTLP to Purl's native endpoint.

otel-collector.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: purl-collector
  namespace: monitoring
spec:
  mode: daemonset
  config: |
    receivers:
      filelog:
        include:
          - /var/log/pods/*/*/*.log
        operators:
          - type: container
            id: container-parser

    processors:
      k8sattributes:
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
            - k8s.deployment.name

    exporters:
      otlphttp:
        # logs_endpoint sets the exact path — the exporter does NOT
        # append /v1/logs to it (unlike the base `endpoint` option)
        logs_endpoint: http://purl.monitoring.svc.cluster.local:3000/api/v1/otlp/logs
        headers:
          X-API-Key: your-api-key

    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [k8sattributes]
          exporters: [otlphttp]

Verify and Monitor

Check that logs are flowing by opening the Purl dashboard or querying the API.

bash

# Port-forward to access Purl dashboard
kubectl port-forward -n monitoring svc/purl 3000:3000

# Verify logs via API
curl -s -H "X-API-Key: your-api-key" \
  "http://localhost:3000/api/logs?range=1h&limit=5" | jq .

# Check Fluent Bit logs for errors
kubectl logs -n monitoring -l app.kubernetes.io/name=fluent-bit --tail=50

Advanced: Namespace Filtering

To collect logs only from specific namespaces, add an exclude filter to Fluent Bit:

fluent-bit.conf

[FILTER]
    Name    grep
    Match   kube.*
    Exclude $kubernetes['namespace_name'] kube-system|kube-public

← Back to all guides