Test Environment

Conformance IdP

The Conformance IdP is a SimpleSAMLphp v2.1 SAML2 IdP instance with:

default authentication source which automatically authenticates a sample (dummy) user
Conformance module installed which provides
- authentication processing filter that can modify SAML Responses, that is, create invalid ones in order to test SP behavior
- ability to run Nuclei tests from the module UI
- ability to ask SP contacts for consent for running tests on SP
- API which enables programmatic control and execution of tests

IdP metadata: https://conformance-idp.maiv1.incubator.geant.org/module.php/saml/idp/metadata

Admin dashboard: https://conformance-idp.maiv1.incubator.geant.org/module.php/admin/

Conformance module UI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/setup

Conformance module repo: https://github.com/cicnavi/simplesamlphp-module-conformance

Running tests

Prior to running tests, of course, the SAML trust between the Conformance IdP and particular SP has to be established
(SAML metadata exchanged). Check the "SP metadata handling" on how to add the SP metadata you want to test.

Also, this instance of Conformance IdP is configured to require SP consent to run tests on it. When the test for particular SP is initiated,
Conformance IdP will check if it has consent for it. If not, it will send an invitation to accept the test to the SP contacts emails. Just to showcase
this scenario, a catch-all SMTP service is brought up at https://mailbox.maiv1.incubator.geant.org/, where all the invitations can be
inspected / accepted (emails won't actually be sent to SP contacts).

Manual testing

For manual testing and validation, SP can simply initiate the authentication using the Conformance IdP
(SP initiated flow), in which case the Conformance IdP will present the screen with available test options prior to
returning the SAML response:

After the test selection, Conformance IdP will return the (modified) SAML response so one can observe if the SP is
behaving appropriately.

Running single tests from Conformance module UI

On the other hand, administrator can use the Conformance IdP UI to initiate all tests using the Nuclei tool.
To do that, navigate to SimpleSAMLphp administration area > Configuration > Conformance (Details area) > Run Nuclei Test.

Choose the desired SP and click Run. This will invoke the Nuclei tool in the backend and start testing using the
predefined testing templates, and then stream all of its output to the screen.

After the testing is done, results can be seen on the Nuclei Results page.

Bulk testing

Bulk testing can be executed using [SimpleSAMLphp Cron module](https://github.com/simplesamlphp/simplesamlphp/blob/master/modules/cron/docs/cron.md).
As you can see in Cron documentation, a cron tag can be invoked using HTTP or CLI. Of course, with Conformance testing
using CLI is the preferred way, since testing can take a (relatively) long time depending on the number of SPs.
However, you are free to test execution using the HTTP version, in which case the maximum execution time
will correspond to the 'max_execution_time' INI setting.

Only one test runner instance can run at given point in time. By maintaining internal state, test runner can first check
if there is another runner active. If yes, the latter will simply exit and let the active test runner do its work.
This way one is free to invoke the cron tag at any time, since only one test runner will ever be active.

For this test environment, feel free to initiate bult testing from the Cron module UI here:

https://conformance-idp.maiv1.incubator.geant.org/module.php/cron/info

Bulk testing uses the same Nuclei logic, and all test results will be available as described with single tests in UI.

SP metadata handling

Conformance IdP is configured with PDO metadata storage handler (it can use database to store SP metadata) in addition to plain PHP metadata files.

Conformance module exposes an HTML form which can be used to manually add additional SP metadata, either by pasting the SP metadata XML or by uploading the metadata XML file.

The UI form is available here: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/metadata/add

Also, there is an API endpoint which can be used to provision SP metadata dynamically (described below).

API endpoints

API endpoints are protected with Authorization Bearer token. In order to access the API, you must provide the token in the HTTP request as the Authorization header, with Bearer scheme. For example:

GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer sometoken

Test setup for next authentication event

Endpoint to define the next test for particular SP. This will determine the shape of the SAML response in the next
authentication event for the given service provider.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup

HTTP method: GET

Parameters:

testId
- valid values: standardResponse | noSignature | invalidSignature
- example: noSignature
spEntityId
- valid values: any trusted SP Entity ID
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

For example, to specify that the next test for the SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp' should be the one that doesn't sign the SAML Response:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup?testId=noSignature&spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Running Nuclei tests using predefined templates

Endpoint which can be used to run all Nuclei tests on a particular service provider using the predefined templates.
This is the same as running Nuclei tests from the Conformance module UI.
The HTTP response is a streamed output from the Nuclei tool.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run

HTTP method: GET

Parameters:

spEntityId
- valid values: any trusted SP Entity ID
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
acsUrl (optional) - if not provided, a default SP ACS URL will be used
- valid values: ACS URL belonging to the SP defined with spEntityId
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
enableDebug (optional) - parameter forwarded to Nuclei which determines if HTTP requests and responses should be shown
- valid values: 1 (true), 0 (false, default)
- example: 1
enableVerbose (optional) - parameter forwarded to Nuclei which determines if verbose output should be shown
- valid values: 1 (true), 0 (false, default)
- example: 1

For example, to run tests for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`, make an
HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Test results

Endpoint to fetch test results in JSON format. By default, all results from all SPs will be returned. This can be
filter out using parameters below.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

HTTP method: GET

Parameters:

spEntityId (optional): can be used to fetch results for particular SP only
- valid values: any trusted SP Entity ID
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
latestOnly (optional): can be used to only fetch latest result per SP
- valid values: `1` (true), `0` (false, default)
- example: `1`

For example, to fetch all available results, make an HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

To fetch only latest results for all SPs:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?latestOnly=1

To fetch all results for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

To fetch only the latest result for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp&latestOnly=1

SP metadata provisioning

Endpoint to provision SP metadata which will be trusted by the Conformance IdP.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/metadata/persist

HTTP method: POST

Parameters:

xmlData - optional (mandatory if xmlFile not provided)
- valid values: SAML2 SP metadata XML string
- example: <?xml version="1.0" encoding="utf-8"?>
  <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp">
  <md:SPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
  <md:SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/saml/sp/saml2-logout.php/good-sp"/>
  <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/saml/sp/saml2-acs.php/good-sp" index="0"/>
  <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Artifact" Location="https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/saml/sp/saml2-acs.php/good-sp" index="1"/>
  </md:SPSSODescriptor>
  </md:EntityDescriptor>
xmlFile - optional (mandatory if xmlData not provided)
- valid values: SAML SP XML metadata file
- example: default-sp

IdP Initiated Login

IdP initiated login can be performed as per SimpleSAMLphp documentation: https://simplesamlphp.org/docs/2.1/simplesamlphp-idp-more.html

Sample URI to initiate login to SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp':

https://conformance-idp.maiv1.incubator.geant.org/saml2/idp/SSOService.php?spentityid=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

SP Consents

Conformance module has ability to ask SP contacts for consent before running tests for SP.

For testing / demo purposes, in this environment a dummy smtp server was provisioned which is used to catch all outgoing email invitations for consents (emails won't really be sent to SP contacts).

All emails will be available for inspection at: https://mailbox.maiv1.incubator.geant.org/

Sample SPs and Related Apps

SimpleSAMLphp

SimpleSAMLphp v2.1 instance with configured SPs listed below is featuring a code modification to skip signature checks for 'bad' SPs for simulation purposes.

Admin dashboard: https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/admin/

List of apps: https://simplesamlphp-sp.maiv1.incubator.geant.org/

Good SP / App

Metadata: https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/saml/sp/metadata/good-sp

App: https://simplesamlphp-sp.maiv1.incubator.geant.org/php-app-good-ssp-sp/

Bad SP / App

Metadata: https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/saml/sp/metadata/bad-sp

App: https://simplesamlphp-sp.maiv1.incubator.geant.org/php-app-bad-ssp-sp/

KeyCloak

Instance with 'conformance' realm with two SPs in with different configurations regarding signature checking, available here: https://keycloak.maiv1.incubator.geant.org/

Authentication on either SP can be initiated by going to https://keycloak.maiv1.incubator.geant.org/realms/conformance/account > Personal info > Click on the appropriate IdP / SP on the "Or sign in with" section (choose good or bad SP).