...
- Open one or more of the following RabbitMQ management consoles. (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
- Scroll down to the "Nodes" section
- There should be 3 rows in the table and all status icons should be green (currently - there is a red bar showing a deprecated node - this will be removed when possible). The expected node hostnames names are:
- prodrabbit@prod-noc-alarms01.geant.orgprod
- rabbit@prod-noc-alarms02.geant.orgprod
- rabbit@prod-noc-alarms03.geant.org
Solution
- If one of the 3 nodes is failing or missing from the list, log into the failing server via ssh and restart the RabbitMQ service:
sudo systemctl restart rabbitmqdashboard-serverdocker.service
- After a minute or two the management consoles should show the cluster is restored.
...
- If all 3 nodes appear in the list, but if the state of the nodes is different when logging into their respective administration gui's
- log into all 3 nodes
- on all 3 nodes type
sudo systemctl stop rabbitmq-server
- wait for the service to stop on all 3 nodes
- then on all 3 nodes type:sudo systemctl start rabbitmq-serverfollow these instructions to restart/rebootstrap the cluster
Collectors have stopped working
Analysis
- Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1 this Correlation status dashboard
- Scroll down to the "Collectors" panel
- Check that the graph shows a nonzero rate of traps being processes
...
Possible Cause: Correlators have stopped working
Analysis
- Open https://prod-dashboardv3-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1 this Correlation status dashboard
- Scroll down to the "Collectors" panel
- Check that the graph shows the leader collector processing a non-zero rate of traps. The current leader can be identified by the FORWARDER with state 2 in the "Raft States" panel.
...