alert manager not working properly in prometheus

Question

I have some IoT devices connected to a server being monitored by Prometheus. I have it configured in a way that whenever a device shuts it sends an alert. But right now, whenever a device goes offline the alert manager is sending alert of all the devices which are offline and not a single device at a time.

ALERT IoT_online
  IF IoT_online == 0
  LABELS {severity="critical"}
  ANNOTATIONS {description="This device is offline  {{ $value }}.", summary="Instance {{ $labels.instance }} IoT device is offline"}

I just want it to send alerts of the only device which went offline at that particular time. Help Please...

Answer 1

This is the default functionality of the alert manager. Whenever an alert group sends an alert, it includes all alerts that are currently active.

What you can try is to separate alert group into smaller groups by labeling group_by. This may increase the amount of notifications that you'll get. You can also try and throttle how often these alerts are sent because of group changes using grpoup_interval.