ELK Stack Setup Guide: Elasticsearch, Logstash, and Kibana

The ELK Stack combines Elasticsearch for storing and searching data, Logstash for ingesting and transforming events, and Kibana for exploration, dashboards, and administration. Elastic now uses the broader name “Elastic Stack,” which can also include Elastic Agent, Beats, Fleet, APM, and security capabilities, but people still search for ELK when they want a centralized logging pipeline.

A production-ready setup is more than installing three services. You need a data contract, secure transport and identities, buffering, mappings, retention, monitoring, backups, and a failure plan. This guide gives you an implementation sequence that works whether you choose Elastic Cloud, another supported managed option, containers for evaluation, or a self-managed cluster.

ELK architecture at a glance

Applications, servers, cloud services
        |
        |  Elastic Agent / Beats / syslog / HTTP / queue
        v
Logstash input -> parse, normalize, enrich -> Elasticsearch data stream
                                                    |
                                                    v
                                      Kibana search and dashboards

Logstash is optional when an integration or agent can send correctly structured events directly to Elasticsearch. Keep it when you need complex parsing, conditional routing, enrichment, protocol conversion, or output to multiple destinations. Do not insert it merely because the acronym includes an L.

Step 1: choose a deployment model

For a tutorial or local evaluation, a single node can prove ingestion and queries. It does not prove production resilience. Elastic’s production guidance covers deployment-specific availability, fault tolerance, scaling, and operational considerations. Check the applicability badges because some instructions differ across Hosted, Serverless, ECE, ECK, and self-managed deployments.

Step 2: define the event contract

Before installing Logstash, collect representative events from every intended source. For each source, document:

• Owner, producer, environment, and expected daily and peak volume
• Transport protocol and behavior during destination failure
• Timestamp field, time zone, precision, and late-arrival expectations
• Stable fields, optional fields, nested objects, and maximum message size
• Secrets, personal data, tokens, and fields that require removal or restriction
• Retention, search latency, recovery, and audit requirements
• Schema version and the process for producer changes

Adopt the Elastic Common Schema where it fits. Consistent names for host, service, user, source, destination, event, and error fields make dashboards and correlation reusable. Preserve the raw message during early rollout so parsing mistakes can be investigated, but set a retention and access policy for that raw copy.

Step 3: install and secure Elasticsearch first

Elasticsearch is the system of record for the indexed events. Establish it before building dashboards or complex pipelines. Use the current installation instructions for your deployment type; package names, container tags, enrollment workflows, and defaults change across releases.

At minimum, verify:

• Every node has a stable identity, persistent data path, and synchronized time.
• Transport between nodes and client connections uses approved encryption.
• Administrative access is separated from application ingestion and read access.
• Cluster endpoints are not exposed directly to the public internet.
• Heap, memory locking where applicable, storage latency, and disk watermarks are understood.
• Snapshots write to a repository outside the cluster’s failure domain.
• Monitoring alerts cover cluster health, unassigned shards, disk, JVM pressure, and ingestion failures.

Create narrowly scoped credentials for each producer or Logstash pipeline. Do not place an all-powerful username and password in a configuration file committed to source control. Use your deployment’s secret store or orchestration secret mechanism and define a rotation process.

Step 4: use data streams for append-only logs

Logs, events, and metrics are usually time-series data: new documents arrive continuously and old documents are rarely edited. Elasticsearch data streams provide a stable named resource over automatically managed backing indices. Every document needs an @timestamp field.

The official Elastic data-stream documentation explains when data streams fit and when an index alias is more appropriate. A data stream is a strong default when events are append-only and mostly searched rather than repeatedly updated by document ID.

Use a predictable naming strategy, such as type, dataset, and namespace. Keep production and nonproduction data separated through names, permissions, or projects. Define component and index templates before the first write so mappings and settings do not depend on accidental dynamic inference.

Step 5: create a minimal Logstash pipeline

A Logstash configuration has an input, optional filters, and an output. Start with the smallest pipeline that proves transport and authentication, then add parsing one operation at a time. Elastic’s pipeline tutorial documents the structure and execution flow.

input {
  beats {
    port => 5044
  }
}

filter {
  if [message] {
    # Add a parser only after testing representative messages.
    # Preserve failed events instead of silently dropping them.
  }
}

output {
  elasticsearch {
    hosts   => ["${ELASTICSEARCH_URL}"]
    api_key => "${ELASTICSEARCH_API_KEY}"
    # Configure data-stream behavior for your plugin and deployment version.
  }
  stdout { codec => rubydebug }
}

This is an architecture example, not a copy-paste production file. Confirm plugin options against the documentation installed for your Logstash version. Remove the stdout output after validation if it could expose sensitive events.

Parsing rules that prevent silent data loss

• Parse timestamps explicitly and test multiple time zones and malformed values.
• Use conditionals by source or dataset; one universal grok pattern becomes brittle.
• Tag parse failures and route them to a reviewable failure stream or queue.
• Convert numeric and boolean fields deliberately rather than indexing every value as text.
• Redact credentials, session tokens, authorization headers, and prohibited personal data before output.
• Keep a version field for parser and schema changes.
• Test oversized, multiline, truncated, duplicated, late, and out-of-order events.

Step 6: add buffering and backpressure

Ask what happens when Elasticsearch is slow or unavailable. A producer that blocks indefinitely can turn a logging problem into an application outage; a producer that drops immediately creates an investigation gap.

Choose a failure strategy based on the importance and volume of the data:

• Agent-side disk queues for temporary destination outages
• Logstash persistent queues for pipeline buffering
• A durable message broker when multiple consumers, replay, or stronger decoupling are required
• A dead-letter or failure stream for events that cannot be indexed
• Rate limits and load shedding for low-value telemetry

Set an explicit maximum buffer age and capacity. Alert before the queue fills, test recovery throughput, and measure whether the pipeline can drain a backlog faster than new events arrive.

Step 7: connect Kibana and create a data view

Kibana should connect over the secured cluster endpoint with its own service identity. Human users should authenticate through the organization’s approved identity system where supported. Map roles to the minimum data and Kibana spaces each team needs.

After events arrive:

1. Confirm the data stream and document count in Elasticsearch.
2. Create or select a Kibana data view matching the intended streams.
3. Choose @timestamp as the time field.
4. Use Discover to inspect field types, missing values, and parse-failure tags.
5. Save a narrow search for a known service and environment.
6. Create a dashboard only after validating the underlying event contract.

Build operational dashboards around decisions: error rate by service and version, slow requests by endpoint, authentication failures by source, queue lag, and deployment regressions. A wall of charts without owners or thresholds is decoration, not observability.

Step 8: configure retention before volume grows

Retention is a business and compliance requirement, not just a storage setting. Define how long each dataset must remain searchable, restorable, or deleted, and who approves exceptions.

Elastic provides data stream lifecycle and index lifecycle management options depending on the deployment and use case. The data stream lifecycle guide explains rollover, retention, and downsampling for supported data streams. Elastic Serverless and versioned Stack deployments have different lifecycle controls, so do not apply an ILM tutorial blindly across both.

Estimate daily indexed size after parsing, replication, expected growth, and retention. Test queries against old and current data. Deleting old data is not the same as maintaining recoverable snapshots, and snapshots are not a substitute for tested retention deletion.

Step 9: secure the entire path

Logs commonly contain data more sensitive than the application team realizes: tokens, email addresses, internal URLs, database errors, request bodies, customer identifiers, and security events. Inventory fields and prevent collection when the organization has no legitimate need to store them.

Step 10: test production failure modes

A green dashboard during normal traffic proves little. Before declaring the stack production-ready, test:

• Loss and recovery of an Elasticsearch node
• Disk pressure and watermark behavior
• Expired or rotated ingestion credentials
• A malformed event surge and mapping conflict
• Logstash restart with queued events
• Network interruption between every major component
• Snapshot creation and restoration into an isolated environment
• A retention-policy change and deletion approval
• Kibana or identity-provider unavailability
• Recovery from an accidental dashboard, pipeline, or template change

Record loss, duplication, recovery time, backlog growth, and operator actions. Turn the results into alerts and runbooks. Elastic’s Elasticsearch production guidance covers resilience, data architecture, lifecycle, security, and monitoring topics that should feed this review.

Common ELK setup mistakes

• Exposing Elasticsearch or Kibana directly to the internet
• Using one administrator credential for ingestion, dashboards, and automation
• Letting dynamic mappings define production field types from the first event
• Creating too many tiny indices or shards without a lifecycle strategy
• Keeping every log indefinitely because no owner chose retention
• Dropping parse failures instead of routing and measuring them
• Failing to redact secrets before they reach durable storage
• Treating snapshots as valid without a restoration test
• Upgrading all components without compatibility and rollback checks
• Building dashboards before validating timestamps and schema

Frequently asked questions

Is ELK Stack free?

Elastic software and services have different licenses, subscription tiers, and included features. Check the current Elastic licensing and subscription documentation for the exact deployment and capabilities you plan to use.

Do I need Logstash?

No. Use it when you need its inputs, filters, enrichment, routing, or outputs. Elastic Agent, Beats, application clients, or other shippers can send suitable events directly when complex transformation is unnecessary.

Should logs use indices or data streams?

Append-only time-series logs are usually a strong data-stream use case. Use a different design when documents require frequent updates or last-write-wins behavior by explicit ID.

Can I run ELK on one server?

One server can support learning, development, or a carefully bounded low-risk use case. It creates a shared failure domain and does not demonstrate production availability.

What should I monitor first?

Monitor ingestion success and delay, parse failures, queue depth, rejected writes, cluster health, unassigned shards, disk, memory pressure, snapshot results, and the freshness of each critical data source.

Bottom line

A reliable ELK setup follows this order: define events, secure Elasticsearch, choose data streams and mappings, build a minimal pipeline, handle backpressure, validate data in Kibana, enforce retention, and test failure recovery. Installations are easy to demonstrate; trustworthy logging systems are designed around ownership, schema, security, lifecycle, and evidence that no important event silently disappears.