🚀 Installing and Running Loki (Grafana) on Linux 🖥️

Loki is a log aggregation system by Grafana, designed to store and query logs efficiently. Follow the steps below to download, install, and run Loki on your Linux system.

👥 Step 1: Download Loki Binary

curl -O -L "https://github.com/grafana/loki/releases/download/v3.4.2/loki-linux-amd64.zip"

👉 This command does the following:

-O → Saves the file with the same name as in the URL.
-L → Follows redirects (useful if the URL redirects to another location).
Downloads Loki v3.4.2 for Linux AMD64 architecture.

📂 Step 2: Extract the Loki Binary

unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/bin/loki

👉 This extracts the loki-linux-amd64 binary from the ZIP file.

If unzip is not installed, install it using:

Debian/Ubuntu: sudo apt install unzip
CentOS/RHEL: sudo yum install unzip
Arch Linux: sudo pacman -S unzip

📝 Step 3: Download Loki Configuration File

sudo mkdir -p /etc/loki
wget https://raw.githubusercontent.com/grafana/loki/v3.4.2/cmd/loki/loki-local-config.yaml -O /etc/loki/config.yaml

👉 This downloads the default Loki configuration file (loki-local-config.yaml) from the Grafana Loki GitHub repository.

🔹 Why do we need this?

It provides the basic settings required for running Loki.
You can modify it later based on your requirements.

Example config to auto-delete logs after 24 hours:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: debug
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2025-04-08
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /tmp/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

limits_config:
  retention_period: 24h

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

🚀 Step 4: Create a Systemd Service for Loki

Create a systemd service file for Loki at /etc/systemd/system/loki.service:

sudo tee /etc/systemd/system/loki.service > /dev/null <<EOF
[Unit]
Description=Loki Log Aggregation System
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/loki -config.file /etc/loki/config.yaml

Restart=always
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

📝 NOTE: If the service fails to start, it might be because Loki is already running manually. Kill it using:
ps aux | grep loki
kill <PID>

⚠️ Important

By default, the Loki configuration uses /tmp for storage:

storage:
  filesystem:
    chunks_directory: /tmp/loki/chunks
    rules_directory: /tmp/loki/rules

You should change the path from the /tmp directory in your config.yaml file. Otherwise, logs will be deleted after the server restarts because /tmp is cleared on reboot.

✅ Recommended Example

storage:
  filesystem:
    chunks_directory: /var/lib/loki/chunks
    rules_directory: /var/lib/loki/rules

Make sure the directories exist and are writable:

sudo mkdir -p /var/lib/loki/chunks /var/lib/loki/rules
sudo chown -R loki:loki /var/lib/loki

🛠️ Reload and Start Loki Service

sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki
sudo systemctl status loki

🔍 Verify If Loki Is Running

Check Loki readiness on port 3100:

curl -s http://localhost:3100/ready

✅ If Loki is working correctly, it should return:

"ready"

🌐 Why Use a Domain for Loki with NGINX Reverse Proxy?

Setting up Loki behind an NGINX reverse proxy and using a domain name instead of a raw IP address is a powerful and flexible approach — especially when sending logs from multiple agents like Promtail. Here’s a detailed breakdown of the benefits, along with explanations and some cool ✨ perks:

🛠️ 1. IP Independence — "Set it and forget it!"

Problem without domain: If the Loki server IP changes, you’ll have to update all 100+ Promtail agents manually — one by one! 😩

Solution with domain:

Point a domain (e.g., logs.mydomain.com) to your server IP.
Even if the server IP changes, just update the DNS A record.
Promtail clients will still connect to the domain, and you don’t have to touch them at all. 🙌

✅ Benefit: Easy infrastructure changes, zero downtime for agents.

🔐 2. Add Basic Authentication — "Secure access with user/pass"

Using NGINX, you can enable Basic Auth (username & password) in front of Loki.

location / {
  auth_basic "Protected Loki";
  auth_basic_user_file /etc/nginx/.htpasswd;
  proxy_pass http://localhost:3100;
  ...
}

Then, in your promtail-config.yaml, use:

client:
  url: https://logs.mydomain.com/loki/api/v1/push
  basic_auth:
    username: lokiuser
    password: lokipass

✅ Benefit: Prevents unauthorized users from pushing/viewing logs.

🛡️ 3. Add Security Headers — "Boost security with best practices"

NGINX allows you to set security headers to protect your Loki endpoint from certain web vulnerabilities.

add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options DENY;
add_header Content-Security-Policy "default-src 'none';";

✅ Benefit: Hardens your logging endpoint against basic attack vectors.

🚪 4. Avoid Exposing Loki Port Directly — "Hide internal stuff from the outside"

Loki typically runs on port 3100. Exposing this port publicly isn’t ideal.

With NGINX:

Loki stays bound to localhost:3100 internally.
You only expose standard ports like 443 (HTTPS) externally.
Promtail connects securely through NGINX.

✅ Benefit: Smaller attack surface + more professional setup.

✅ Summary Table

Feature	With Domain + NGINX	Without Domain
IP-independent config	✅	❌
Basic Auth protection	✅	❌
Security headers possible	✅	❌
Port hiding & HTTPS	✅	❌

📌 Final Thought

Using a domain with an NGINX reverse proxy in front of Loki is more than a convenience — it’s a DevOps best practice. It simplifies scaling, improves security, and keeps your logging pipeline professional, flexible, and future-proof. 🚀

🔐 Setup HTTPS with Self-Signed Certificate for Loki behind NGINX Reverse Proxy

📁 Step 1: Create Certificate Directory

sudo mkdir -p /etc/nginx/certs
cd /etc/nginx/certs

📝 Step 2: Create `loki-cert.conf` Configuration File

[req]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_ca
prompt             = no

[req_distinguished_name]
C  = IN
ST = State
L  = City
O  = Company
OU = Department
CN = loki-staging.abcfintech.com

[req_ext]
subjectAltName = @alt_names

[v3_ca]
subjectAltName = @alt_names
basicConstraints = critical,CA:TRUE,pathlen:0
keyUsage = critical,keyCertSign,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
DNS.1 = loki-staging.abcfintech.com

🔑 Step 3: Generate SSL Certificate and Key

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout loki.key \
  -out loki.crt \
  -config loki-cert.conf

🌐 Step 4: NGINX Configuration for Loki HTTPS Proxy

Create or update your NGINX site config:

server {
    listen 443 ssl;
    server_name loki-staging.abcfintech.com;

    ssl_certificate     /etc/nginx/certs/loki.crt;
    ssl_certificate_key /etc/nginx/certs/loki.key;

    # 🔐 Allow only /loki/api/v1/push with basic auth
      location = /loki/api/v1/push {
          proxy_pass http://localhost:3100;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        auth_basic "Push Access Only";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }

    # ✅ Allow /ready endpoint without auth
    location = /ready {
        proxy_pass http://localhost:3100/ready;
    }

    # 🚫 Deny all other routes
    location / {
        return 403;
    }
}

🔐 Step 5: Create Basic Auth User

sudo htpasswd -c /etc/nginx/.htpasswd pushuser
# Enter password: pushpassword

🌍 Step 6: Test the Endpoint (with `--insecure` for self-signed cert)

curl -v https://loki-staging.abcfintech.com/ready --insecure

🏢 Step 7: Add Certificate to System Trust Store

sudo cp loki.crt /usr/local/share/ca-certificates/loki.crt
sudo update-ca-certificates

🔄 Step 8: Restart Promtail to Apply Changes

sudo systemctl restart promtail

✅ Now, Loki is served securely over HTTPS with self-signed certs and access control via NGINX.

🎉 Pro Tip: Replace the self-signed cert with a real one using Let's Encrypt for production use!

👥 Step 1: Download Loki Binary​

📂 Step 2: Extract the Loki Binary​

📝 Step 3: Download Loki Configuration File​

Example config to auto-delete logs after 24 hours:​

🚀 Step 4: Create a Systemd Service for Loki​

⚠️ Important​

✅ Recommended Example​

🛠️ Reload and Start Loki Service​

🔍 Verify If Loki Is Running​

🌐 Why Use a Domain for Loki with NGINX Reverse Proxy?

🛠️ 1. IP Independence — "Set it and forget it!"​

🔐 2. Add Basic Authentication — "Secure access with user/pass"​

🛡️ 3. Add Security Headers — "Boost security with best practices"​

🚪 4. Avoid Exposing Loki Port Directly — "Hide internal stuff from the outside"​

✅ Summary Table​

📌 Final Thought​