Telegram Notifications From AlertManager With Respect to 4096 Chars Limit

Issue

• Telegram API has limit of 4096 characters per message
• AlertManager sending notifications truncates text of message to fit this limit
• As result markdown(or HTML depending on the parse_mode specified in telegram_configs of AlertManager config) may be broken and telegram API responses with Status Code 400

Solution

Telegram notification template should generate message that is always less than 4096 characters and has consistent markdown/html

Template below:

• Builds full notification message with all alerts
• If it is too big falls back to less verbose single alert format (no labels)
• If even with less verbose format the message appears to be too big
  • removes some alerts to fit the size limit (alerts are removed as a whole, so telegram does not complain about broken markdown)
  • add notice about truncation to the message

{{- define "telegram.alert_list" -}}
  {{- $maxLength := 3900 -}}
  {{- $message_tmp := "" -}}
  {{- $message := "" -}}
  {{- $message_is_truncated := false -}}
  {{- $message_no_labels := "" -}}
  {{- $message_no_labels_is_truncated := false -}}
  {{- $escape_regex := "([-_*\\[\\]()~`>#+=|{}.!])" -}}

  {{- $alerts := .Alerts -}}

  {{- /* 
  Hack to output firing alerts first:

  Go Templates does not allow to define array ["firing","resolved"] 
  and iterate over it.

  Instead iterating over first two items in `.Alerts` array ignoring 
  the value and generating only firing alerts for `index == 0` 
  and resolved for `index == 1`. 

  Private case where there is only one alert - just output this alert

  Private case where there are no alerts - output nothing 
  (for consistency, AlertManager does not generate empty notifications)

  */ -}}
  {{- $status := "firing" -}}
  {{- $count := 2 -}}
  {{- if lt (len $alerts) 2 -}}
    {{- $count = len $alerts -}}
  {{- end -}}
  {{- if eq (len $alerts) 1 -}}
    {{- $status = (index $alerts 0).Status -}}
  {{- end -}}

  {{- range $index,$_ := slice $alerts 0 $count -}}
    {{- if eq $index 1 -}}
      {{- $status = "resolved" -}}
    {{- end -}}

    {{- range $alerts -}}
      {{- if eq .Status $status   }}
        {{- /* Build alert text */ -}}
        {{- $alertText := "" }}
        {{- /* Status */ -}}
        {{- if eq .Status "firing" }}
          {{- $alertText = "🔥 "}}
        {{- end -}}
        {{- if eq .Status "resolved" }}
          {{- $alertText = "✅ "}}
        {{- end -}}

        {{- /* Title */ -}}
        {{- $title := .Labels.alertname | reReplaceAll  $escape_regex "\\$1" -}}
        {{- if .Annotations.summary}}
          {{- $title = printf "%s \\- %s" $title (.Annotations.summary | reReplaceAll  $escape_regex "\\$1")  -}}
        {{- end }}

        {{- $alertText = printf "%s***\\<%s\\> %s***\n" $alertText ( or .Labels.severity "no severity") $title -}}
        {{- if .Annotations.description}}
          {{- $alertText = printf "%s_%s_\n" $alertText ( .Annotations.description | reReplaceAll $escape_regex "\\$1" ) -}}
        {{- end }}

        {{- /* Source link */ -}}
        {{- if .GeneratorURL -}}
          {{- $alertText = printf "%s[Source](%s)" $alertText (.GeneratorURL) -}}
        {{- end }}

        {{- /* Runbook link */ -}}
        {{- if .Annotations.runbook_url -}}
          {{- $alertText = printf "%s [Runbook](%s)" $alertText (.Annotations.runbook_url) -}}
        {{- end -}}

        {{- /* Store alert text without labels in case all alerts would not fit message length limit  */ -}}
        {{- if $message_no_labels  -}}
          {{- $message_tmp = printf "%s\n\n%s" $message_no_labels $alertText -}}
        {{- else -}}
          {{- $message_tmp = printf "%s" $alertText -}}
        {{- end -}} 

        {{- if lt ($message_tmp | len)  $maxLength  -}}
          {{- $message_no_labels = $message_tmp -}}
        {{- else -}}
          {{- $message_no_labels_is_truncated = true -}}
        {{- end -}}

        {{- /* Add labels to alert text */ -}}
        {{- range .Labels.SortedPairs -}}
          {{- if (eq .Name "alertname" "runbook_url" "severity" "prometheus" ) | not -}}
          {{- $alertText = printf "%s\n🏷 %s: `%s`" $alertText (.Name | reReplaceAll $escape_regex "\\$1" ) .Value -}}
          {{- end -}}
        {{- end }}

        {{- /* Store full alert text */ -}}
        {{- if $message  -}}
          {{- $message_tmp = printf "%s\n\n%s" $message $alertText -}}
        {{- else -}}
          {{- $message_tmp = printf "%s" $alertText -}}
        {{- end -}}

        {{- if lt ($message_tmp | len)  $maxLength  -}}
          {{- $message = $message_tmp -}}
        {{- else -}}
          {{- $message_is_truncated = true -}}
        {{- end -}}

      {{- end -}}
    {{- end -}}
  {{- end -}}

  {{- $message =  printf "%s\n" $message -}}
  {{- if $message_is_truncated -}}
    {{- $message =  printf "%s\nMessage is truncated to fit telegram API limit" $message -}}
  {{- end -}}

  {{- $message_no_labels =  printf "%s\n" $message_no_labels -}}
  {{- if $message_no_labels_is_truncated -}}
    {{- $message_no_labels =  printf "%s\nMessage is truncated to fit telegram API limit" $message_no_labels -}}
  {{- end -}}

  {{- if $message_is_truncated -}}
    {{- $message_no_labels -}}
  {{- else -}}
    {{- $message -}}
  {{- end -}}
{{- end -}}

{{- define "telegram.message" -}}
🔥 firing: {{ len .Alerts.Firing }} ✅ resolved: {{ len .Alerts.Resolved }}

{{ template "telegram.alert_list" . }}

[See On AlertManager]({{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }})
\#alerts
{{- end -}}

Result

Full Message

Compact Message

Truncated Message

Comments