Loki is a horizontally scalable log aggregation system from Grafana Labs that indexes labels rather than full log content — making it far cheaper to run than Elasticsearch for pure log storage. Combined with Promtail for log shipping and Grafana for visualization, it gives you a full security logging stack for a fraction of the cost of an ELK setup.

Architecture

Servers (auth.log, syslog, app logs)
    └─ Promtail (log shipper, runs on each server)
            └─ Loki (storage + query engine)
                    └─ Grafana (dashboards + alerts)

Prerequisites

• One server to run Loki and Grafana (4GB RAM minimum for production)
• Promtail on each server you want to collect logs from
• Docker Compose (simplest deployment method)

Step 1: Deploy Loki and Grafana with Docker Compose

mkdir -p /opt/loki && cd /opt/loki
mkdir -p data/loki data/grafana
chown 10001:10001 data/loki

Create /opt/loki/docker-compose.yml:

version: "3.8"

services:
  loki:
    image: grafana/loki:2.9.8
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml:ro
      - ./data/loki:/loki
    command: -config.file=/etc/loki/config.yml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - "3000:3000"
    volumes:
      - ./data/grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.example.com
    restart: unless-stopped
    depends_on:
      - loki

Create /opt/loki/loki-config.yml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 30d
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h

Start the stack:

cd /opt/loki
docker compose up -d
docker compose logs -f

Step 2: Install Promtail on Log Sources

On each server whose logs you want to ship:

# Download Promtail binary
curl -LO "https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail

Create /etc/promtail/config.yml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://LOKI_SERVER_IP:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          host: web-01
          __path__: /var/log/syslog

  - job_name: auth
    static_configs:
      - targets: [localhost]
        labels:
          job: auth
          host: web-01
          __path__: /var/log/auth.log

  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: web-01
          __path__: /var/log/nginx/{access,error}.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)'
      - labels:
          status:
          method:

Create a systemd service:

sudo useradd -r -s /bin/false promtail
sudo mkdir -p /var/lib/promtail
sudo chown promtail:promtail /var/lib/promtail

sudo tee /etc/systemd/system/promtail.service > /dev/null <<'EOF'
[Unit]
Description=Promtail log shipper
After=network.target

[Service]
User=promtail
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yml
Restart=on-failure
SupplementaryGroups=adm systemd-journal

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now promtail

The SupplementaryGroups=adm systemd-journal gives Promtail permission to read system logs without running as root.

Step 3: Configure Grafana Data Source

1. Open Grafana at http://your-server:3000
2. Log in with admin / changeme (change immediately)
3. Go to Connections > Add new data source
4. Select Loki
5. Set URL: http://loki:3100 (or the actual IP if not using Docker networking)
6. Click Save & Test

Step 4: Writing Security Queries with LogQL

LogQL is Loki’s query language. For security monitoring:

# All failed SSH login attempts
{job="auth"} |= "Failed password"

# Failed logins by source IP (rate over 5 minutes)
sum by (ip) (
  rate({job="auth"} |= "Failed password"
  | regexp "from (?P<ip>[0-9.]+)"
  [5m])
)

# Nginx 5xx errors in last hour
{job="nginx", status=~"5.."}

# sudo usage (privilege escalation)
{job="auth"} |= "sudo"

# Successful SSH logins
{job="auth"} |= "Accepted publickey" OR "Accepted password"

# Top IPs hitting 404s
topk(10,
  sum by (remote_addr) (
    count_over_time({job="nginx", status="404"}[1h])
  )
)

Step 5: Create Security Alerts

In Grafana, navigate to Alerting > Alert Rules > New Rule.

Example: Alert on 5+ failed SSH attempts from the same IP in 5 minutes:

# Alert query
sum by (ip) (
  rate({job="auth"} |= "Failed password"
  | regexp "from (?P<ip>[0-9.]+)"
  [5m])
) > 0.016

0.016/s = ~5 events per 5-minute window. Configure the alert to fire when this threshold is exceeded and send to a notification channel (Slack, email, PagerDuty).

Step 6: Securing Loki

By default Loki has no authentication. Add basic auth via Nginx or Caddy reverse proxy:

# Caddyfile snippet
loki.internal.example.com {
    basicauth {
        promtail $2a$14$...bcrypt-hash-of-promtail-password...
    }
    reverse_proxy localhost:3100
}

Then update Promtail config:

clients:
  - url: https://loki.internal.example.com/loki/api/v1/push
    basic_auth:
      username: promtail
      password: your-promtail-password

Step 7: Log Retention and Storage Management

Loki’s compactor handles retention automatically if configured. Check current storage usage:

# Check how much disk Loki is using
du -sh /opt/loki/data/loki/chunks/
du -sh /opt/loki/data/loki/

# Check Loki's internal stats via API
curl -s http://localhost:3100/loki/api/v1/status/buildinfo | jq .
curl -s http://localhost:3100/metrics | grep loki_ingester_chunks_stored_total

To adjust retention without redeploying, update loki-config.yml and restart:

limits_config:
  retention_period: 90d   # Change from 30d to 90d
  max_query_lookback: 90d # Must match retention_period

For production systems generating high log volumes, enable chunk compression:

common:
  storage:
    filesystem:
      chunks_directory: /loki/chunks
  compactor:
    working_directory: /loki/compactor
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

chunk_store_config:
  chunk_cache_config:
    embedded_cache:
      enabled: true
      max_size_mb: 500

Step 8: High-Value Security Dashboards

Import or build these dashboards in Grafana to make the security data actionable:

SSH Brute Force Dashboard panels:

# Panel 1: Failed SSH attempts per minute (rate graph)
sum(rate({job="auth"} |= "Failed password" [1m])) by (host)

# Panel 2: Top attacking IPs (table)
topk(20,
  sum by (src_ip) (
    count_over_time(
      {job="auth"} |= "Failed password"
      | regexp "from (?P<src_ip>[\\d.]+)"
      [24h]
    )
  )
)

# Panel 3: Successful logins after failures (potential compromise indicator)
{job="auth"} |= "Accepted" | line_format "{{.message}}"

Web Application Attack Dashboard panels:

# HTTP 400-499 rate (client errors including scanner noise)
sum(rate({job="nginx", status=~"4.."}[5m])) by (host)

# Top paths receiving 404s (scanner enumeration detection)
topk(10,
  sum by (path) (
    count_over_time({job="nginx", status="404"}[1h])
  )
)

# POST requests with 200 response to unusual paths
{job="nginx", method="POST", status="200"}
| line_format "{{.path}}"
| regexp "^(?!.*(login|api|upload|submit))(?P<suspicious_path>.+)"

To import a dashboard JSON:

1. Grafana → Dashboards → Import
2. Paste a dashboard JSON or enter a Grafana.com dashboard ID (e.g., 15141 for Loki logs overview)

Related Reading

• How to Set Up Promtail for Log Shipping
• How to Use syslog-ng for Centralized Logging

• How to Monitor File Changes with inotifywait

• Setting Up ClamAV Antivirus on Linux
• AI Coding Assistant Session Data Lifecycle

• How to Audit What Source Code AI Coding Tools Transmit

Related Articles

• How to Set Up Promtail for Log Shipping
• What VPN Logs Actually Mean No Log Policy Explained
• What VPN Logs Actually Mean: No-Log Policy Explained
• Opt Out of Data Sharing Under Connecticut Data Privacy Act
• Cloud Storage Security Breach History: Compromised Built by theluckystrike — More at zovo.one