Getting a lean, reliable log analysis pipeline in place does not require expensive closed-source SIEMs. With careful design you can build a detection and investigation stack using open-source components that scale from small SOCs to large enterprises. This tutorial walks through a pragmatic architecture, recommended components, deployment patterns, and day-to-day playbooks to get analysts productive fast.

Why open-source for log analysis

Open-source stacks give you control over data, flexibility for custom parsing and enrichment, and lower entry cost for experimentation and proof of value. Licensing is a key consideration when choosing a search/index layer, so evaluate licensing and community activity before you standardize on a distribution. Elastic announced changes in 2024 that affect how Elasticsearch and Kibana are licensed, so factor license alignment into procurement and architecture choices.

Core components and what they do

  • Collectors and agents: gather logs, telemetry and endpoint state. Tools such as osquery provide deep host telemetry via SQL style queries.
  • Network sensors: for session and protocol context use Zeek to generate rich network logs that complement host logs.
  • Packet capture and indexing: for full-packet forensics deploy Arkime to index PCAP metadata and provide fast session search.
  • Storage and search: OpenSearch or Elasticsearch style backends index and serve logs for search and dashboards. OpenSearch is a maintained Apache 2.0 project and is a widely used alternative if you prefer an Apache licensed stack.
  • Detection and rules: Wazuh offers an open-source detection and correlation platform that integrates host telemetry and log analysis for alerting. Use Sigma as a vendor agnostic rule format to share and convert detections into your target query language.

Quick reference architecture (minimal viable SOC)

  1. Central indexer cluster - OpenSearch or Elasticsearch. Store parsed logs, Zeek output and Arkime session metadata. Consider separate clusters or indices for short hot retention and longer cold retention.
  2. Wazuh manager - receives agent logs and runs correlation rules and alerts. Dashboard for analyst workflows.
  3. Hosts - Wazuh agent and osquery deployed for visibility into processes, scheduled queries and file events.
  4. Network sensors - Zeek on TAP/SPAN ports to produce conn, http, dns, ssl logs; Arkime to capture and index full packets for forensic pulls.
  5. Alerting and investigations - convert Sigma rules into the query language used by your search backend, push alerts to Wazuh or your alerting layer, and use Arkime/Zeek to enrich any alert with network context.

Step-by-step: a pragmatic build plan

Step 1 - Plan retention and sources

  • List the log sources you need: host logs, syslog, cloud audit, firewall, Zeek, Arkime session metadata, osquery results.
  • Decide retention policy for each source (hot search window, warm archive, long term storage). Keep raw PCAP off the index cluster and store PCAP on Arkime sensors.

Step 2 - Deploy the indexer and dashboard

  • For a quick lab pick OpenSearch or an Elasticsearch distribution you are comfortable with. Use official install docs and start with 3 nodes for HA. OpenSearch is Apache 2.0 licensed and community maintained.
  • Expose the dashboard to your analysts over HTTPS and restrict write access to ingestion systems only.

Step 3 - Install Wazuh manager and agents

  • Follow the Wazuh quickstart to install a manager and begin onboarding agents. Wazuh integrates endpoint telemetry, file integrity monitoring and detection rules, and can forward alerts into the indexer for dashboarding.

Step 4 - Host and endpoint visibility

  • Deploy osquery on Windows, macOS and Linux endpoints where possible. Use a fleet manager for scheduled queries and to collect results centrally. Use Wazuh to surface osquery findings into your alert pipeline.

Step 5 - Network visibility and packet capture

  • Deploy Zeek on network taps to produce protocol level logs that are highly useful for detection and hunting. Use Arkime collectors for full packet capture and index metadata into your search backend for fast retrieval.

Step 6 - Rules, conversions and tuning

  • Use Sigma as your canonical detection format. Convert Sigma rules into your backend query language using pySigma or an appropriate converter and load converted queries into Wazuh or your alert engine. Start with low volume, high fidelity rules and tune thresholds.

Daily SOC playbook examples

  • Alert triage
    1. Review the alert in Wazuh or your alert inbox. Confirm source fields and timestamps.
    2. Run analytics: search the index for related host and network events across the alert window. Pull Zeek conn/http/dns logs for lateral movement signs.
    3. If network context is needed, search Arkime for the session and export PCAP for deep analysis.
    4. If host artifacts are required run an osquery scheduled query against the host or check recent osquery results.
  • Threat hunting
    • Build Sigma hunting rules for suspicious patterns and run as broad searches during off peak times. Use Zeek logs to pivot on network observables and Arkime to confirm packet-level activity.

Scaling and operational tips

  • Separate ingestion from query load. Use hot-warm index tiers to reduce cost and maintain query speed.
  • Snapshot indices regularly and validate restores. Keep Arkime PCAP retention separate from indexed metadata and use a size based policy for PCAP disks.
  • Plan for parser maintenance. A small SOC can get large coverage by investing in robust parsing/enrichment for common sources like Windows Event Logs, Sysmon, Zeek logs and cloud audit trails.
  • Watch licensing and community support. Elastic licensing changes in 2021 and the 2024 licensing announcements mean you should pick a distribution and license model that fits your organization. OpenSearch is a community Apache 2.0 project to consider.

Security, privacy and hardening

  • Lock down index endpoints and limit which services can write to your cluster. Use API keys and network controls.
  • Restrict access to Arkime sensors; PCAPs are very sensitive. Harden the sensor OS and use encrypted links for UI access.
  • Sanitize and control PII within logs to meet privacy obligations before sharing dashboards externally.

Wrap up and next steps

Start small and iterate. Onboard a single host group, one network zone with Zeek, and one Arkime sensor as proof of concept. Use Sigma as a common detection language so you can share and improve rules with the community. As the SOC gains confidence, expand hosts and sensors, tune Sigma conversions and build runbooks that link Wazuh alerts to Zeek and Arkime evidence for fast investigation. The components in this tutorial are actively maintained and community driven, so you can prototype and scale without vendor lock in.

Checklist - deployable in a long weekend

  • Deploy indexer cluster (OpenSearch or chosen Elasticsearch distribution) and secure it.
  • Install Wazuh manager and onboard 5 agents.
  • Deploy Zeek on one network span port.
  • Stand up Arkime capture on a sensor and index into the cluster.
  • Convert a handful of Sigma rules and create your first alerts.

If you want, I can turn this into a runnable playbook with specific commands and a docker-compose lab manifest that wires Wazuh, Zeek, osquery and Arkime together for a hands-on SOC lab. Say the word and I will draft the playbook and a small test dataset to validate your pipelines.