Credentials and Access

SSH Access

• Please use your GEANT AD account(GEANT\dante.xxx or GEANT\Firsname.Lastname)to have ssh access and sudoer rights on the required servers.

Other Access

• Required credentials are in the "GÉANT Dashboard v3" LastPass folder

Problems

New alarms are not appearing in the gui

Possible Cause: Traps not being processed by RabbitMQ

Analysis

1. Open one or more of the following RabbitMQ management consoles.  (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
   • https://prod-noc-alarms01.geant.org:15672/
   • https://prod-noc-alarms02.geant.org:15672/
   • https://prod-noc-alarms03.geant.org:15672/
2. Scroll down to the "Nodes" section
3. There should be 3 rows in the table and all status icons should be green (currently - there is a red bar showing a deprecated node - this will be removed when possible).  The expected node names are:
   • rabbit@prod-noc-alarms01
   • rabbit@prod-noc-alarms02
   • rabbit@prod-noc-alarms03

Solution

1. If one of the 3 nodes is failing or missing from the list, log into the failing server via ssh and restart the RabbitMQ service:

   • systemctl restart dashboard-docker.service

2. After a minute or two the management consoles should show the cluster is restored.

Solution #2

1. If all 3 nodes appear in the list, but if the state of the nodes is different when logging into their respective administration gui's
   • follow these instructions to restart/rebootstrap the cluster 

Collectors have stopped working

Analysis

1. Open this Correlation status dashboard
2. Scroll down to the "Collectors" panel
3. Check that the graph shows a nonzero rate of traps being processes

Solution

1. On each of the following servers:
   • prod-noc-alarms01.geant.org
   • prod-noc-alarms02.geant.org
   • prod-noc-alarms03.geant.org
2. Log in via ssh and execute the following command:

   • sudo systemctl restart trap_collector

Possible Cause: Correlators have stopped working

Analysis

1. Open this Correlation status dashboard
2. Scroll down to the "Collectors" panel
3. Check that the graph shows the leader collector processing a non-zero rate of traps.  The current leader can be identified by the FORWARDER with state 2 in the "Raft States" panel.

Solution

1. On each of the following servers:
   • prod-noc-alarms01.geant.org
   • prod-noc-alarms02.geant.org
   • prod-noc-alarms03.geant.org
2. Log in via ssh and execute the following command:

   • sudo systemctl restart trap_correlator

In case production operation isn't restored quickly ...

If the production environment can't be recovered quickly and operation restored, please refer temporarily to the UAT environment.  This UAT environment continually processes the same traps as production, and uses the same IMS instance, so should be useable while production operation is being restored.  To access the UAT environment gui, please navigate to one of the following:

• https://uat-noc-alarms-ui01.geant.org/
• https://uat-noc-alarms-ui02.geant.org/

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.