Search

Prometheus SMS Alerting

Updated: May 7

Prometheus is a metrics and monitoring workhorse for many of DevOps teams.

A wide set of integrations makes it easy to monitor and visualize critical aspects of IT infrastructures. It is an open-source monitoring system built around a time-series database, able to receive data and store it, in addition to supporting queries, graphs, and in-built alerting.


Alerting with Prometheus is separated into two parts - alerting rules, which are configured in Prometheus servers send alerts to an Alertmanager. Alertmanager is used to handle alerts for client applications. It takes care of silencing, deduplicating, grouping and routes alerts to different receivers such as email, chat platforms (like Slack), and on-call notification systems.


I consider SMS as a common way for notification and would expect Prometheus to have easy integration for it too.

Reality is sometimes cruel as there is no common API that can be used to create one solution for all. Usually, SMS providers have API or SDK to send SMS, but it does not always fit Prometheus or Alertmanager structure.

The good news is that Prometheus gives a method to call Webhooks - web calls to other services.

As such call is generic and contains only alert data it can not be used straight away to call our SMS provider.



For this reason, we've developed an SMS proxy that can handle such calls and send messages to our team members.

SMS Proxy is a lightweight program running in docker, which makes it easy to deploy and integrate with existing installations.


The SMS proxy will get a Webhook from Prometheus and will send SMS using the provider API or SDK.


On the Prometheus side, the new Webhook receiver entry will assure that alerts are being forwarded to the proxy that will handle it.

Our implementation of SMS Proxy can be found at https://github.com/Nandi-Ai/Prometheus-SMS. This SMS proxy code implements a connection to Nexmo, MessageBird, and TeleMessage SMS API. You are more than welcome to clone, use or contibute.


Conclusion

Just like humans progressed after grasping the fire, Prometheus and AlertManager Webhook gives us the endless possibility for the integrations. Using Webhook, it allows trigger anything, as long as it fits Prometheus Alertmanager Reciever. (For further information please check https://prometheus.io/docs/alerting/latest/configuration/#webhook_config)


Hope you find this document informative.

1,579 views0 comments

Recent Posts

See All