Documentation

User Documentation

New alarm can be created in three simple steps. First register it using REST API, add the alarm generating code to your program, create a cron that executes your program periodically.

Registering Alarm

Before registering a new kind of alarm, look at the classification tree of the current alarms and decide if there is already a category and subcategory where the new alarm would naturally fit. Once you have decided on category and subcategory, you should create a small json document in this form:

{
  "category": "Analytics",
  "subcategory": "Frontier",
  "event": "Failed queries",
  "description": "Code running every 1h at UC k8s cluster. Checks all servers for: Rejected queries - server is busy and doesn't respond to the query; DB disconnections - the query was processed by the Frontier server but the Oracle DB terminated the connection. The code can be found here: https://github.com/ATLAS-Analytics/AlarmAndAlertService/blob/master/frontier-failed-q.py",
  "template": "Servers with failed queries:\n%{servers}\n. Affected tasks: \n%{tasks}\n\tConsult the following link to get a table with the most relevant taskids https://atlas-kibana.mwt2.org:5601/s/frontier/goto/c72d263c3e2b86f394ab99211c99b613\n"
}

It is really important to write the description field as detailed as possible as that will help people getting alarm know how it got generated. The template field defines text that will appear in the alert email. So it should be brief, but give all the information that a person receiving it could judge how serious the issue is, should they forward it, and have enough information to either act on it, or start digging further into the issue. All the instances of %{variable} will be changed with values of variable field given in alarms "source" field. At this moment only simple \n and \t are available for formatting. Once you have the json alarm description ready, you can simply post it to the Alarm And Alert Service REST endpoint.

Creating Alarm

Once Alarm has been configured, you may start creating them. Alarms are again simple JSON documents, and follow this form:

{
  "category": "network",
  "subcategory": "perfsonar",
  "event": "high packet loss",
  "tag": "ps.sl",
  "body": "ps_1 needs 10000ms",
  "source": {
      "src": "137.222.79.1",
      "avg_value": 1.0,
      "host_src": "dice-io-37-00.acrc.bris.ac.uk",
      "site_src": "UKI-SOUTHGRID-BRIS-HEP"
  }
}

In addition to alarm classification, it contains fields:

• tag - a single keyword or a short array that a user can subscribe to. This helps limit number of alerts that a user will receive.
• body - a very brief description of the event.
• source - this optional field is a short map whose attributes will be placed in the alert template.

The document should be sent to this REST endpoint.

Every alarm has its own page at /alarms/<id>, where id is the alarm's Elasticsearch document id. It shows the alarm, links to Kibana, and lets logged-in users add annotations.

Examples

Registering an alarm category and sending an alarm with curl:

AAAS=https://aaas.atlas-ml.org

curl -X POST $AAAS/alarm/category -H 'Content-Type: application/json' -d '{
"category": "Analytics", "subcategory": "Frontier", "event": "Failed queries",
"description": "What generates this alarm and how.",
"template": "Servers with failed queries:\n%{servers}\n"
}'

curl -X POST $AAAS/alarm/ -H 'Content-Type: application/json' -d '{
"category": "Analytics", "subcategory": "Frontier", "event": "Failed queries",
"tag": "MWT2", "body": "3 servers with failed queries",
"source": {"servers": "frontier01, frontier02, frontier03"}
}'

The same with python (requests):

import requests

AAAS = "https://aaas.atlas-ml.org"

alarm = {
  "category": "Analytics",
  "subcategory": "Frontier",
  "event": "Failed queries",
  "tag": "MWT2",
  "body": "3 servers with failed queries",
  "source": {"servers": "frontier01, frontier02, frontier03"},
}
r = requests.post(f"{AAAS}/alarm/", json=alarm, timeout=30)
r.raise_for_status()

# heartbeats: register once, then send one every few seconds from each instance
requests.post(f"{AAAS}/heartbeat/register", json={
  "category": "SLATE", "subcategory": "Squid", "event": "server down",
  "description": "Squid instance stopped sending heartbeats.",
  "template": "Squid instance %{instance} at %{site} stopped sending heartbeats.",
  "interval": 60, "min_expected": 4,
}, timeout=30)
requests.post(f"{AAAS}/heartbeat/", json={
  "category": "SLATE", "subcategory": "Squid", "event": "server down",
  "source": {"site": "MWT2", "instance": "MWT2_Slate_01"},
}, timeout=30)

REST API

There are three parts of the API: alarms, users and heartbeats.

Alarm

{
    "category": "",
    "subcategory": "",
    "event": "",
    "description": "",
    "template": ""
}

{
    "category": "",
    "subcategory": "",
    "event": "",
    "description": "",
    "template": ""
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "tag": "",
  "body": "",
  "source": {"xxx": "", "yyy": "", ...}
}

{
    "category": "",
    "subcategory": "",
    "event": "",
    "period": 6
}

{
    "category": "",
    "subcategory": "",
    "event": ""
}

User

{
  "vacation": true
}

[
  {
      "category": "",
      "subcategory": "",
      "event": "",
      "tags": ["xxx", "yyy"]
  },
  ...
]

Heartbeat

{
    "category": "",
    "subcategory": "",
    "event": "",
    "description": "",
    "template": "",
    "interval": 60,
    "min_expected": 4
}

{
    "category": "",
    "subcategory": "",
    "event": "",
    "description": "",
    "template": "",
    "interval": 60,
    "min_expected": 4
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "source": {"xxx": "", "yyy": "", ...}
}

{
    "category": "",
    "subcategory": "",
    "event": "",
    "period": 6
}

{
    "category": "",
    "subcategory": "",
    "event": ""
}