Page History

Table of Contents

Credentials and Access

SSH Access

• Members of the AD group GEANT\\SSH-opsdb have ssh access and sudoer rights on the required servers.

Other Access

• Required credentials are in the "GÉANT Dashboard v3" LastPass folder

Problem: New alarms are not appearing in the gui

Possible Cause: Traps not being processed by RabbitMQ

Analysis

1. Open one or more of the following RabbitMQ management consoles.  (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
   • http://net-alarms01.geant.org:15672/
   • http://net-alarms02.geant.org:15672/
   • http://prod-dashboard-storage01.geant.org:15672/
2. Scroll down to the "Nodes" section
3. There should be 3 rows in the table and all status icons should be green.  The expected node hostnames are:
   • prod-noc-alarms01.geant.org
   • prod-noc-alarms02.geant.org
   • prod-dashboard-storage01.geant.org

Solution

1. If one of the 3 nodes is failing or missing from the list, log into the failing server and restart the RabbitMQ service:

   • sudo systemctl restart rabbitmq-server

2. After a minute or two the management consoles should show the cluster is restored.

Possible Cause: Collectors have stopped working

Analysis

1. Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
2. Scroll down to the "Collectors" panel
3. Check that the graph shows a nonzero rate of traps being processes

Solution

1. On each of the following servers:
   • net-alarms01.geant.org
   • net-alarms02.geant.org
2. Log in and execute the following command:

   • sudo systemctl restart trap_collector

Possible Cause: Correlators have stopped working

Analysis

1. Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
2. Scroll down to the "Collectors" panel
3. Check that the graph shows one of the collector processes processing a non-zero rate of traps.
   • note that it is normal for only one of the collectors to be processing traps, the other line should remain at zero.

Solution

1. On each of the following servers:
   • net-alarms01.geant.org
   • net-alarms02.geant.org
2. Log in and execute the following command:

   • sudo systemctl restart trap_correlator

Related articles