This is the prometheus.yml file. Before I configure any rules, It was running perfectly
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# query_log_file: /var/log/prometheus/query.log
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 'localhost:9093'
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- prometheus_rules.yml
This is the log
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.562Z caller=main.go:842 msg="TSDB started"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.562Z caller=main.go:969 msg="Loading configuration file" filename=/usr/local/bin/prometheus/prometheus.yml
Aug 5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.571Z caller=manager.go:956 component="rule manager" msg="loading groups failed" err="/usr/local/bin/prometheus/prometheus_rules.yml: group \"instance_down_alert_rule\", rule 1, \"InstanceDown\": invalid annotation name: d:wqescription"
Aug 5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.571Z caller=main.go:995 msg="Failed to apply configuration" err="error loading rules, previous rule set restored"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:685 msg="Stopping scrape discovery manager..."
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:699 msg="Stopping notify discovery manager..."
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:721 msg="Stopping scrape manager..."
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:695 msg="Notify discovery manager stopped"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:681 msg="Scrape discovery manager stopped"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.572Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.573Z caller=main.go:896 msg="Notifier manager stopped"
Aug 5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.573Z caller=main.go:715 msg="Scrape manager stopped"
Aug 5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.573Z caller=main.go:905 err="error loading config from \"/usr/local/bin/prometheus/prometheus.yml\": one or more errors occurred while applying the new configuration (--config.file=\"/usr/local/bin/prometheus/prometheus.yml\")"
Aug 5 05:38:03 localhost systemd: prometheus.service: main process exited, code=exited, status=1/FAILURE
Aug 5 05:38:03 localhost systemd: Unit prometheus.service entered failed state.
Aug 5 05:38:03 localhost systemd: prometheus.service failed.
"I am getting this same error in Docker prometheus container too"