The table below attempts to show some of the existing monitoring tools that are available for SAML entities and federations, what they focus on and what results they deliver. The idea is to look at where we are currently delivering tools, how we present them, where we can combine efforts and, most importantly, if we can combine results in a convenient and coherent report for entities.
Questions to ask:
- Do we currently have all the right tools that we want (and preferably are we cooperating on tool development and not duplicating where appropriate)?
- Have we defined all the tests that we want in the right contexts (e.g. test for eduGAIN compliance vs local federation compliance vs general SAML compliance)?
- Tools vs instances, what do we want, where?
- Who is running the test?
- How are we promoting these to users? Can people find the tools right now?
- What happens when a service instance flags something as red?
- What reports are being delivered to the testers? Can these be standardised / combined into a larger grade report? Something like the AAF report? https://aaf.edu.au/wp-content/uploads/2015/04/AAF_example_sum_report.pdf
| Check Type | Purpose | REFEDS | eduGAIN | Wider | When Run | Report Given | Comments |
|---|---|---|---|---|---|---|---|
| SAML Deployment Profile checks | To check compliance against SAML deployments in given contexts | FedLab: SAML2Int (code) | eduGAIN metadata validator (service) - tests against the eduGAIN Metadata Profile for federation metadata SAML2Int? - no test run against the SAML2Int SHOULD FEDERATION | Fedlab: SAML2Int (code) TestShib (service) | Testing during deployment process by IdPs and SPs. | ??? | metadata validator a different audience, not entity focused. |
| SAML Configuration Check | To test specific elements of the way of SAML deployment is configured and whether it is operational | N/A | eduGAIN metadata validator (service) eduGAIN Connectivity Check (service) USER CENTRIC SITE | Fedlab: MCCS - Metadata Monitoring Service (code) | Testing during service operation to flag operational issues. | Connectivity check gives red / yellow / green warnings. Currently no action taken when flagged. | |
| Verify Entity Categories | To verify that entities are meeting requirements as laid out in entity categories (mostly R&S and CoCo at this stage). | FedLab: Entity Check (code) Need R&S monitor? | CoCo Monitor (service) BOTH eduGAIN Attribute Release Check (in development - service) BOTH | FedLab: Entity Check (code) | Testing during service operation or testing when setting up an entity category | CoCo shows a red / yellow / green flag. Sends automated email to SP admin when it turns red. Can be used by entities or by a service operator (eduGAIN, federation). | |
| Check Attribute Release | Tools to check that IdPs are releasing attributes / what attributes are being released | N/A | N/A | SWITCH Interfederation Attribute Check (service) Foodle has a built in page which highlights what is being released (service) | In service. | SWITCH tool gives a report showing fail / pass and shows other entities that have passed. "Fail" is difficult in some contexts as the result might actually be correct for the implementation - only works locally. | Difficult to get right as the IdP might be deliberately chosing not to release attributes to the SP. |
| Test IdP / Access Check | Check to see if an SP works with a test IdP | N/A | eduGAIN Access Check (service) USER CENTRIC - SP | TestShib (service) Feide OpenIdP and Metadata Edit (shutdown as of 1-Jan-2016) | Testing during deployment process by IdPs and SPs. | ?? | |
Metadata Explorer | Human readable metadata and metadata search | MET | eduGAIN Entities (service) BOTH | SMEV (service) Pyff (service and code) WAYF.dk Cantina (service)
| General overview of metadata at any given time | Shows human readable metadata and reports against them. Current implementations probably right for each environment. | Different than the other tools listed above |
IsFederated | Checks to see if an organisation is federated | N?A | USER CENTRIC - SP | N/A | Shows if a specific domain is using any given federation. | Different than the other tools listed above |