Caddy开启自身监控，使用prometheus获取数据，同时使用promtail和loki监控caddy日志文件，记录请求来源等信息。

使用到的工具Prometheus、Grafana

• Grafana面板
• Loki日志抓取

Caddy开启监控和日志输出

修改Caddyfile，增加监控和日志输出

{
    log {
            output file /var/log/caddy/caddy_main.log {
                      roll_size 500MiB
                      roll_keep 5
                      roll_keep_for 100d
            }
            format json
            level DEBUG
     }
    servers {
            metrics
    }
}

live.vio.vin {
  reverse_proxy 10.115.15.25:3001
}

访问 curl 127.0.0.1:2019/metrics测试是否配置成功，应该输出prometheus格式的数据

安装loki

尝试过安装最新版loki，3.3.2版本，无法启动，安装2.7.0版本没有问题

sudo mkdir /opt/loki
cd /opt/loki

sudo wget -qO /opt/loki/loki.gz "https://github.com/grafana/loki/releases/download/v2.7.0/loki-linux-amd64.zip"

sudo gunzip /opt/loki/loki.gz
sudo chmod a+x /opt/loki/loki

sudo ln -s /opt/loki/loki /usr/local/bin/loki

使用 loki -version测试是否安装成功

下载配置文件

sudo wget -qO /opt/loki/loki-local-config.yaml "https://raw.githubusercontent.com/grafana/loki/v2.7.0/cmd/loki/loki-local-config.yaml"

测试使用配置文件启动

sudo /opt/loki/loki -config.file=/opt/loki/loki-local-config.yaml

设置自启动

sudo vi /etc/systemd/system/loki.service

[Unit]
Description=Loki log aggregation system
After=network.target

[Service]
ExecStart=/opt/loki/loki -config.file=/opt/loki/loki-local-config.yaml
Restart=always

[Install]
WantedBy=multi-user.target

systemctl start loki
systemctl status loki
systemctl enable loki

安装promtail

wget https://github.com/grafana/loki/releases/download/v2.7.0/promtail-linux-amd64.zip

sudo apt install unzip
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

sudo wget -P /etc/promtail https://raw.githubusercontent.com/grafana/loki/v2.7.0/clients/cmd/promtail/promtail-local-config.yaml

修改配置文件，增加caddy日志查询

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      stream: stdout
- job_name: caddy
  static_configs:
  - targets:
      - localhost
    labels:
      job: caddy
      __path__: /root/caddy/logs/caddy_main.log
      agent: caddy-promtail
  pipeline_stages:
  - json:
      expressions:
        status: status
        duration: duration
  - labels:
      duration:
      status:

设置自启动

vi /etc/systemd/system/promtail.service

[Unit]
Description=Promtail service
After=network.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-local-config.yaml

[Install]
WantedBy=default.target

sudo systemctl start promtail 
sudo systemctl enable promtail
sudo systemctl status promtail

安装Prometheus

sudo apt update
sudo apt install prometheus prometheus-node-exporter

添加caddy输出

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']
  - job_name: caddy
    static_configs:
      - targets: ['localhost:2019']

设置Grafana

添加数据源

添加数据源prometheus和loki

loki地址为对应服务器的3100端口

prometheus地址为对应服务器的9090端口

添加Dashboard

https://grafana.com/grafana/dashboards/20802-caddy-monitoring

ID:20802