Versions Compared

Old Version 9

changes.mady.by.user Erik Reid

Saved on

compared with

New Version 10

changes.mady.by.user Mandeep Saini

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Credentials and Access

SSH Access

  • Members of the AD group GEANT\\SSH-opsdb have ssh Please use your GEANT AD account(GEANT\dante.xxx or GEANT\Firsname.Lastname)to have ssh access and sudoer rights on the required servers.

...

  • Required credentials are in the "GÉANT Dashboard v3" LastPass folder

...

Problems

New alarms are not appearing in the gui

Possible Cause: Traps not being processed by RabbitMQ

Analysis
  1. Open one or more of the following RabbitMQ management consoles.  (Credentials are in the "GÉANT Dashboard v3" LastPass folder)
  2. Scroll down to the "Nodes" section
  3. There should be 3 rows in the table and all status icons should be green.  The expected node hostnames are:
    • prod-noc-alarms01.geant.org
    • prod-noc-alarms02.geant.org
    • prod-dashboard-storage01.geant.org
Solution
  1. If one of the 3 nodes is failing or missing from the list, log into the failing server via ssh and restart the RabbitMQ service:
    • sudo systemctl restart rabbitmq-server
  2. After a minute or two the management consoles should show the cluster is restored.

...

ΩΩCollectors have stopped working

Analysis

  1. Open https://net-alarms-monitoring.geant.org/d/hESYQotZz/correlation-services?orgId=1
  2. Scroll down to the "Collectors" panel
  3. Check that the graph shows a nonzero rate of traps being processes

...