...
Traps not being processed by RabbitMQ
Analysis
- Open one or more of the following RabbitMQ management consoles. (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
...
...
- Scroll down to the "Nodes
...
Solution
...
- " section
- There should be 3 rows in the table and all status icons should be green. The expected node hostnames are:
- prod-noc-alarms01.geant.org
- prod-noc-alarms02.geant.org
- prod-dashboard-storage01.geant.org
Solution
- If one of the 3 nodes is failing or missing from the list, log into the failing server and restart the RabbitMQ service:
sudo systemctl restart rabbitmq-server
- After a minute or two the management consoles should show the cluster is restored.
Possible Cause
Collectors have stopped working
Analysis
- Open
...
...
...
...
- /d/hESYQotZz/correlation-services?orgId=1
- Scroll down to the "Collectors" panel
- Check that the
...
- Collectors
Solution
- graph shows a nonzero rate of traps being processes
Solution
- On each of the following servers:
- net-alarms01.geant.org
- net-alarms02.geant.org
- Log in and execute the following command:
sudo systemctl restart trap_collector
...
Possible Cause
Correlators have stopped working
Analysis
- Open
...
...
...
...
- /d/hESYQotZz/correlation-services?orgId=1
- Scroll down to the "Collectors" panel
- Check
...
- Correlators - received
- Correlators - handled
Solution
...
- that the graph shows one of the collector processes processing a non-zero rate of traps.
- note that is is normal for only one of the collectors to be processing traps, the other line should remain at zero.
Solution
- On each of the following servers:
- Log in and execute the following command:
sudo systemctl restart trap_correlator
Related articles
| Content by Label | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...