Versions Compared

Old Version 11

changes.mady.by.user Mandeep Saini

Saved on

compared with

New Version Current

changes.mady.by.user Erik Reid

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Open one or more of the following RabbitMQ management consoles.  (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
  2. Scroll down to the "Nodes" section
  3. There should be 3 rows in the table and all status icons should be green (currently - there is a red bar showing a deprecated node - this will be removed when possible).  The expected node hostnames names are:
    • prodrabbit@prod-noc-alarms01.geant.orgprod
    • rabbit@prod-noc-alarms02.geant.orgprod
    • rabbit@prod-dashboard-storage01.geant.orgnoc-alarms03
Solution
  1. If one of the 3 nodes is failing or missing from the list, log into the failing server via ssh and restart the RabbitMQ service:
    • sudo systemctl restart rabbitmqdashboard-serverdocker.service
  2. After a minute or two the management consoles should show the cluster is restored.

...

Solution #2
  1. If all 3 nodes appear in the list, but if the state of the nodes is different when logging into their respective administration gui's

Collectors have stopped working

Analysis

  1. Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1 this Correlation status dashboard
  2. Scroll down to the "Collectors" panel
  3. Check that the graph shows a nonzero rate of traps being processes

...

  1. On each of the following servers:
    • net-alarms01.geant.org
    • net-alarms02.geant.org
    • net-alarms03.geant.org
  2. Log in via ssh and execute the following command:
    • sudo systemctl restart trap_collector

Possible Cause: Correlators have stopped working

Analysis

  1. Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1 this Correlation status dashboard
  2. Scroll down to the "Collectors" panel
  3. Check that the graph shows one of the leader collector processes processing a non-zero rate of traps.note that it is normal for only one of the collectors to be processing traps, the other line should remain at zero  The current leader can be identified by the FORWARDER with state 2 in the "Raft States" panel.

Solution

  1. On each of the following servers:
    • net-alarms01.geant.org
    • net-alarms02.geant.org
    • net-alarms03.geant.org
  2. Log in via ssh and execute the following command:
    • sudo systemctl restart trap_correlator

...