快速安装docker环境

🐬 参考：https://blog.csdn.net/Sebastien23/article/details/137086778

被监控端部署exporter

🐬参考：https://gottdeskrieges.blog.csdn.net/article/details/136403810

配置Redis exporter

在被监控端部署exporter：

# 仅主机网络模式部署
docker run -d --name redis_exporter --restart unless-stopped --network host oliver006/redis_exporter 

检查是否采集到数据：

curl -X GET http://localhost:9121/metrics > redis.metrics
cat redis.metrics | grep redis_up

配置MySQL exporter

在数据库中创建监控用户：

CREATE USER 'exporter'@'%' IDENTIFIED BY 'Monpass_XXX' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

在被监控端部署exporter：

# 容器化部署
mkdir -vp /opt/mysqld-exporter/cat > /opt/mysqld-exporter/config.my-cnf << EOF
[client]
host=127.0.0.1
user=exporter
password=Monpass_XXX
EOFcd /opt && chown -R polkitd mysqld-exporter/docker run -d --network host --name mysqld_exporter_1 --restart unless-stopped \
-v /opt/mysqld-exporter/config.my-cnf:/etc/mysql/config.my-cnf \
prom/mysqld-exporter --config.my-cnf=/etc/mysql/config.my-cnf \
--web.listen-address=:9104 \
--no-collect.info_schema.query_response_time \
--no-collect.info_schema.innodb_cmp \
--no-collect.info_schema.innodb_cmpmem \
--collect.info_schema.processlist --collect.binlog_sizedocker ps -a

检查是否采集到数据：

curl -X GET http://localhost:9104/metrics > mysqld.metrics
cat mysqld.metrics | grep mysql_up

部署Prometheus和Grafana

创建安装目录及相关文件：

mkdir -vp /opt/docker-compose/prometheus/data/
mkdir -vp /opt/docker-compose/prometheus/conf/
mkdir -vp /opt/docker-compose/prometheus/conf/rules/#mkdir -vp /opt/docker-compose/grafana/data/
#mkdir -vp /opt/docker-compose/grafana/conf/touch /opt/docker-compose/prometheus/conf/prometheus.yml
touch /opt/docker-compose/prometheus/conf/alertmanager.yml#告警规则文件
touch /opt/docker-compose/prometheus/conf/rules/redis_alerts.yml
touch /opt/docker-compose/prometheus/conf/rules/mysql_alerts.ymltouch /opt/docker-compose/docker-compose.ymlcd /opt/ && chown -R polkitd docker-compose/ 

Prometheus配置文件

配置文件prometheus.yml：

#vi /opt/docker-compose/prometheus/conf/prometheus.ymlglobal:scrape_interval:     15s # Default is every 1 minute.evaluation_interval: 15s # The default is every 1 minute.scrape_timeout:      10s # scrape_timeout is set to the global default (10s).# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets: ['<ALERTMANAGER_IP>:9093']# - alertmanager:9093# load alerting rule files
rule_files:- "rules/*.yml"- "rules/*.yaml"scrape_configs:- job_name: 'prometheus'static_configs:- targets: ['<PROMETHEUS_SERVER_IP>:9090']- job_name: 'redis_exporter'static_configs:- targets: ['<REDIS_EXPORTER_IP>:9121']- job_name: 'mysql_exporter'scrape_interval: 8sstatic_configs:- targets: ['<MYSQL_EXPORTER_IP>:9104']

注：根据实际情况修改尖括号中的IP地址。

Alertmanager配置文件

配置文件alertmanager.yml：

#vi /opt/docker-compose/prometheus/conf/alertmanager.ymlglobal:resolve_timeout: 5m                       # 处理超时时间，默认为5minsmtp_smarthost: 'smtp.qq.com:465'         # 邮箱smtp服务器代理smtp_from: '123456789@qq.com'             # 发送邮箱名称smtp_auth_username: '123456789@qq.com'    # 发邮件的邮箱用户名smtp_auth_password: 'xxxxxx'              # 邮箱密码或授权码smtp_require_tls: false                   # 不进行tls验证# 自定义html模板,发邮件的时候用自定义的模板内容
templates:- 'template/*.tmpl'# 定义路由树信息,这个路由可以接收到所有的告警
route:group_by: ['alertname'] # 报警分组依据group_wait: 10s         # 最初即第一次等待多久时间发送一组警报的通知group_interval: 60s     # 在发送新警报前的等待时间repeat_interval: 1h     # 发送重复警报的周期。对于email配置中，此项不可以设置过低，否则将会由于邮件发送太多频繁，被smtp服务器拒绝receiver: 'email'       # 发送警报的接收者的名称，下面的receivers.name# 定义警报接收者信息
receivers:- name: 'email'                            # 路由中对应的receiver名称email_configs:                           # 邮箱配置- to: '987654321@qq.com'                 # 接收警报的email配置#html: '{{ template "test.html" . }}'  # 设定邮箱的内容模板

注：根据实际情况修改邮箱地址和授权码。

rule_files告警规则文件

Redis实例宕机告警规则：

#vi /opt/docker-compose/prometheus/conf/rules/redis_alerts.ymlgroups:
- name: redis_alert_rulesrules:- alert: redis_down     # name of an alerting rule# alert triggering condition defined using PromSQL expression. Here params must be metrics scraped by redis exporter.expr: up{job="redis_exporter"} == 0for: 1mlabels:severity: "Critical"# alert info that will sent to alertmanagerannotations:summary: "Instance {{ $labels.instance }} down"description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes. Current value: {{ $value }}"

MySQL实例宕机告警规则：

#vi /opt/docker-compose/prometheus/conf/rules/mysql_alerts.ymlgroups:
- name: mysql_alert_rulesrules:- alert: mysql_down# alert triggering condition defined using PromSQL expression. Here params must be metrics scraped by mysql exporter.expr: up{job="mysql_exporter"} == 0for: 1mlabels:severity: "Critical"annotations:summary: "Instance {{ $labels.instance }} down"description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes. Current value: {{ $value }}"

docker-compose部署文件

编写docker compose文件，用于部署prometheus、alertmanager和grafana。

#vi /opt/docker-compose/docker-compose.ymlversion: '3'services:prometheus:image: prom/prometheusnetwork_mode: hostcontainer_name: prometheus_1restart: unless-stopped# if you are running as root then set it to 0, else find the right id with the id -u commanduser: '0'#ports:#  - '9090:9090'environment:TZ: "Asia/Shanghai"#command: ["/etc/prometheus/prometheus.yml"]volumes:- ./prometheus/data/:/prometheus/- ./prometheus/conf/rules/:/etc/prometheus/rules/- ./prometheus/conf/prometheus.yml:/etc/prometheus/prometheus.ymlalertmanager:image: prom/alertmanagernetwork_mode: hostcontainer_name: alertmanager_1restart: unless-stopped# if you are running as root then set it to 0, else find the right id with the id -u commanduser: '0'#ports:#  - '9093:9093'environment:TZ: "Asia/Shanghai"volumes:- ./prometheus/conf/alertmanager.yml:/etc/alertmanager/alertmanager.ymlgrafana:image: grafana/grafananetwork_mode: hostcontainer_name: grafana_1restart: unless-stopped# if you are running as root then set it to 0, else find the right id with the id -u commanduser: '0'  #ports:#  - '3000:3000'environment:- TZ="Asia/Shanghai"#- GF_INSTALL_PLUGINS="grafana-simple-json-datasource"volumes:- 'grafana_storage:/var/lib/grafana'# DECLARE DOCKER VOLUME FOR GRAFANA_STORAGE      
volumes:grafana_storage: {}

运行容器并检查：

# 启动容器
cd /opt/docker-compose/ && docker-compose up -d# 停止容器
cd /opt/docker-compose/ && docker-compose stop

部署后检查：

• 访问http://<Prometheus_SERVER_IP>:9090，检查Prometheus是否部署成功。
• 访问http://<Alertmanager_SERVER_IP>:9093，检查AlertManager是否部署成功。
• 访问http://<Grafana_SERVER_IP>:3000，检查Grafana是否部署成功，默认用户名和密码都是admin，登录后需要修改密码。

常见报错信息

1. Docker-compose文件中，如果没有user: '0'，可能会收到下面的报错：

caller=query_logger.go:86 level=error component=activeQueryTracker msg="Error opening query log file" 
file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied

2. Grafana容器的环境变量赋值只能用等号，如果用冒号会收到如下报错（怀疑跟镜像或docker-compose的版本有关）：

environment:- TZ: "Asia/Shanghai"- GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource"
...# 运行容器时报错
panic: interface conversion: interface {} is map[string]interface {}, not string

3. 如果docker-compose文件中配置了往Grafana容器中安装某些插件（例如grafana-clock-panel），可能会导致grafana容器不断重启。检查容器日志会看到如下报错：

$ docker logs -fn grafana_1
...
Error: ✗ Plugin not found (Grafana v8.3.3 linux-amd64)

配置Grafana仪表盘

添加数据源

在添加数据源中选择Prometheus，在HTTP下的URL栏中粘贴http://<Prometheus_SERVER_IP>:9090并保存。

创建仪表盘

在新建仪表盘（New dashboard）中点击右侧的导入仪表盘（Create --> Import），输入并搜索仪表盘编号，在显示的仪表盘选项（Options）中选择数据源为Prometheus，最后点击导入即可。

• 推荐的REDIS仪表盘：763, 11835
• 推荐的MySQL仪表盘：7362

关于自定义仪表盘，以后将补充文章专门介绍。

References
【1】https://gottdeskrieges.blog.csdn.net/article/details/113645177
【2】https://prometheus.io/docs/prometheus/latest/installation/
【3】https://blog.csdn.net/qq_36306519/article/details/128255913
【4】https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
【5】https://blog.csdn.net/weixin_45697293/article/details/119353915
【6】https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/
【7】https://grafana.com/grafana/dashboards
【8】https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/