Migrating from ELK Stack to Purl: A Complete Guide

Why Migrate from ELK?

The ELK Stack (Elasticsearch, Logstash, Kibana) has been the de facto standard for log management for over a decade. But it comes with significant operational overhead:

• Resource hungry: Elasticsearch requires 3+ nodes for production, each needing 16GB+ RAM
• JVM tuning nightmares: Heap size, garbage collection pauses, circuit breakers
• Shard management: Index lifecycle policies, shard allocation, rebalancing
• Complex pipeline: Logstash → Elasticsearch → Kibana = 3 separate systems to maintain

Purl replaces all of this with a single binary + ClickHouse. The whole stack runs in one container pair, and the defaults we ship give the Purl process 1 CPU and 1GB of RAM — a fraction of what a single Elasticsearch data node expects before it will start.

Architecture Mapping

Here's how ELK components map to Purl:

• Logstash → Purl Ingest API (/api/logs) or OTLP endpoint
• Beats/Filebeat → Fluent Bit or OpenTelemetry Collector pointing to Purl
• Elasticsearch → ClickHouse (columnar storage, ZSTD-compressed message columns)
• Kibana → Purl Dashboard (built-in web UI)
• Elasticsearch Query DSL → Purl also supports ES-compatible _search endpoint

Step 1: Prepare Your Purl Instance

Install Purl alongside your existing ELK stack. Both can run simultaneously during migration.

curl -fsSL https://purlogs.com/install.sh | sudo bash -s -- -i

This installs Purl + ClickHouse via Docker Compose. Verify it's running:

curl http://localhost:3000/api/health
# {"status":"ok","clickhouse":"connected"}

Step 2: Dual-Write Log Pipeline

Configure your log pipeline to send to both ELK and Purl simultaneously. Note that Filebeat can't do this — it has no HTTP output and only supports a single output at a time. Use Fluent Bit for the dual-write phase instead; it fans out one stream to both systems:

# fluent-bit.conf — dual output: Elasticsearch + Purl
[INPUT]
    Name          tail
    Path          /var/log/app/*.log
    Tag           app.*
# Keep feeding Elasticsearch during the migration
[OUTPUT]
    Name          es
    Match         *
    Host          your-es-host
    Port          9200
    Index         logs-app
# Send the same stream to Purl
[OUTPUT]
    Name          http
    Match         *
    Host          your-purl-host
    Port          3000
    URI           /api/logs
    Format        json
    Header        X-API-Key your-api-key
    Header        Content-Type application/json

Once you cut over in Step 5, simply delete the es output.

Step 3: Migrate Saved Searches and Alerts

Map your Kibana saved searches to Purl:

• KQL queries → Purl supports single-filter KQL (level:error, service:api, meta.namespace:prod); combine with the service, host, and level query params for multi-field filtering
• Kibana dashboards → Purl custom dashboards (drag-and-drop builder)
• Watcher alerts → Purl alerts with Telegram, Slack, and Webhook channels

Step 4: Verify Data Parity

Run the same queries on both systems and compare results:

# Purl
curl -H "X-API-Key: key" "http://purl:3000/api/logs?q=level:error&range=1h&limit=100"
# Elasticsearch
curl "http://es:9200/logs-*/_search?q=level:error&size=100"

Step 5: Cut Over

Once you've verified data parity for 24-48 hours:

1. 1Update all log shippers to point only to Purl
2. 2Update team bookmarks from Kibana to Purl dashboard
3. 3Disable Logstash and Elasticsearch
4. 4Reclaim the RAM the Elasticsearch nodes were reserving

What to Expect — and What We Cannot Tell You

We have not benchmarked Elasticsearch. Not on our hardware, not on yours, not at all — so this guide has no speed-up multiplier and no cost-saving percentage in it. Any number we published there would be a guess dressed up as a measurement, and an untuned competitor is trivially easy to beat on a slide.

What we can say honestly:

• We measured Purl, on the box we ship: ingest and search figures on the default docker-compose.yml are in our ClickHouse architecture post, method and caveats included
• The operational surface shrinks: three services become one, and shard allocation, index lifecycle policies and JVM heap tuning stop being your problem — that is a structural difference, not a benchmark
• Recovery is docker compose up: there is no cluster state to reconcile, because there is no cluster
• The cost question is arithmetic you can do yourself: compare what your Elasticsearch nodes cost you per month against one VPS, and you will get a number that is true for you rather than one that was true for someone else

The only comparison worth trusting is the one you run on your own logs. Both stacks can sit side by side through Step 2 for exactly that reason — dual-write, then measure, then decide.