...
- Do we currently have all the right tools that we want (and preferably are we cooperating on tool development and not duplicating)?
- Have we defined all the tests that we want in the right contexts (e.g. test for eduGAIN compliance vs local federation compliance vs general SAML compliance)?
- Tools vs instances, what do we want, where?
- Who is running the test?
- How are we promoting these to users? Can people find the tools right now?
- What happens when a service instance flags something as red?
- What reports are being delivered to the testers? Can these be standardised / combined into a larger grade report? Something like the AAF report? https://aaf.edu.au/wp-content/uploads/2015/04/AAF_example_sum_report.pdf
| Check Type | Purpose | REFEDS | eduGAIN | Wider | When Run | Report Given | Comments |
|---|---|---|---|---|---|---|---|
| SAML Deployment Profile checks | To check compliance against SAML deployments in given contexts | FedLab: SAML2Int (code) | eduGAIN metadata validator (service) - tests against the eduGAIN Metadata Profile for federation metadata SAML2Int? - no test run against the SAML2Int SHOULD | Fedlab: SAML2Int (code) TestShib (service) | Testing during deployment process by IdPs and SPs. | ??? | metadata validator a different audience, not entity focused. |
| SAML Configuration Check | To test specific elements of the way of SAML deployment is configured and whether it is operational | N/A | eduGAIN metadata validator (service) eduGAIN Connectivity Check (service) | Fedlab: MCCS - Metadata Monitoring Service (code) | Testing during service operation to flag operational issues. | Connectivity check gives red / yellow / green warnings. Currently no action taken when flagged. | |
| Verify Entity Categories | To verify that entities are meeting requirements as laid out in entity categories (mostly R&S and CoCo at this stage). | FedLab: Entity Check (code) Need R&S monitor? | CoCo Monitor (service) eduGAIN Attribute Release Check (in development - service) | FedLab: Entity Check (code) | Testing during service operation or testing when setting up an entity category | CoCo shows a red / yellow / green flag. Currently no action taken when flagged. Can be used by entities or by a service operator (eduGAIN, federation). | |
| Check Attribute Release | Tools to check that IdPs are releasing attributes / what attributes are being released | N/A | N/A | SWITCH Interfederation Attribute Check (service) Foodle has a built in page which highlights what is being released (service) | In service. | SWITCH tool gives a report showing fail / pass and shows other entities that have passed. "Fail" is difficult in some contexts as the result might actually be correct for the implementation - only works locally. | Difficult to get right as the IdP might be deliberately chosing not to release attributes to the SP. |
| Test IdP / Access Check | Check to see if an SP works with a test IdP | N/A | eduGAIN Access Check (service) | TestShib (service) Feide OpenIdP and Metadata Edit (shutdown as of 1-Jan-2016) | Testing during deployment process by IdPs and SPs. | ?? | |
Metadata Explorer | Human readable metadata and metadata search | MET | eduGAIN Entities (service) | SMEV (service) Pyff (service and code) WAYF.dk Cantina (service)
| General overview of metadata at any given time | Shows human readable metadata and reports against them. Current implementations probably right for each environment. | Different than the other tools listed above |
IsFederated | Checks to see if an organisation is federated | N?A | eduGAIN isFederated | N/A | Shows if a specific domain is using any given federation. | Different than the other tools listed above |