Operational Practice Statement

This is a working document, major changes can be expected at any time.

Document structure

• OT tasks

• • Management of core eduGAIN services
  • Supervision of eduGAIN joining process
  • Management or supervision of supplementary eduGAIN services
• eduGAIN services
  
  • Core services
  • Suplementarny services

• OT procedures

  • members registering or modification of supplied information

  • introduction of new eduGAIN metadata requirements

  • introduction of new good practices for metadata

  • handling of aggregation alerts

  • system updates

  • software development, testing and production implementation

  • backup

  • monitoring

• 
  

• 
  

• Service Order

• Problem resolution

• Configuration change

• System update

• Backup

• Disaster recovery

Availability of services

Operational Team tasks

Management of core eduGAIN services

• management of virtual machines (access management, system maintenance - installation and updates, global backups, status monitoring)
• management of the development platform (based on the GEANT git)
• management of eduGAIN core services (maintenance of any software tools required by the services, monitoring of services, specialised backups)
• supervision of the aggregation function - reaction to aggregation errors, supporting federations in location of problems
• technical documentation - maintenance of user documentation of eduGAIN services
• user support - done in cooperation with the eduGAIN support team

Supervision of eduGAIN joining process

Management or supervision of supplementary eduGAIN services

eduGAIN OT directly manages ...

eduGAIN OT supervises ...

eduGAIN services

Core services

Operational Team procedures

Registration and modification of federation information

Federation delegate and deputy are the only federation representatives authorized to submit information, therefore their identity needs to be established in a trusted way, this is however part of the global eduGAIN trust model, not specific to the SAML prifile.

Security levels

Introduction of new requirements for federation metadata feeds

As a principle requirements for federation feeds must be based on either general standards to which eduGAIN SAML profile adheres or on the eduGAIN SAML profile. In the case of standards, the experience shows that certain violations are only discovered when reported by participating federations - not all such violations are reported by standard schema validation tools, ot in fact not all are just schema errors. Whenever a new problem is reported, even if it is in fact a standards violation, it is first raised to the eduGAIN SG. In the same time, eduGAIN OT contacts federations whose metadata feeds cause the problem and points out that the error should be rectified. After the SG has decided that a new requirements should be implemented and a suitable grace period is set, the new validator rule is implemented by the eduGAIN OT, initially as a warning and after the agreed grace period has passed, as an error. 

Every rule is documented in the [eduGAIN-meta] 

When raising an error, the validator points to the specific rule in  [eduGAIN-meta] 

Introduction of new best current practices for federation metadata feeds

Additions to metadata best current practices need to be decided by the eduGAIN SG. Each such good practice needs to needs to be implemented as an eduGAIN validator warning by the eduGAIN OT. Each good practice rule needs to be implemented in [eduGAIN-BCP].

System updates

All virtual machines running eduGAIN services are regularly updated.

Before an update is planned, the local personel at PSNC is notified in case 

Handling of aggregation alerts

As described in [eduGAIN-meta], under certain conditions aggregation alerts are raised. The current practice is that these alerts are sent as e-mails to the eduGAIN OT. Since alerts are relatively rare and federation metadata feeds can be cached for at least 4 days, the eduGAIN OT makes its own decision on how to react to a particular alert. Usually before sending a notification to a federation, the OT waits until the next aggregation run to make sure that the situation has not been rectified by the federation.

If the OT decides that a notification needs to be sent to the federation, the official contact address registered be the given federation is used. If one of the following conditions is met:

• there is a immediate reject form the federation mail server,
• there is no reply within 24 hours,
• the metadata validUntil period is under 14 hours

then the OT sends additional notifications to the federation delegate and the deputy using their e-mail addresses registered by the given federation.

If it becomes likely that a given federation may not be able to react in time and that its cached metadata feed may expire, the eduGAIN OT sends a warning to the eduGAIN SG mailing list.

It must be realised that that the case of all entities supplied by a large federation being deleted form eduGAIN has heavy consequences - other participating federations will naturally have to drop these entities. When the federation metadata feed becomes available again, other federations may be forced into running emergency regeneration of their metadata, service providers may observe limited breaks in their service. Therefore the eduGAIN OT is making all possible effort to avoid such situations. If the eduGAIN OT realises a very special situation it is allowed to temporarily stop aggregation in order to avoid the deletion of of the federation but ir MUST immediately notify the eduGAIN SG that such measures have been taken.

One example of such a special situation may be a real case of an introduction of a valid but a very short-lived metadata file followed by a metadata error causing a aggregation reject. That situation, leaving no time for normal procedures to take place, was caused by a configuration error on the federation side and was rectified in a short time while the eduGAIN aggregation was suspended.

Technical details

eduGAIN database description

The eduGAIN database is central to all eduGAIN core services. The database stores:

• general and contact information about participating and candidate federations
• operational information about participating and candidate federations like metadata URLs, signing keys, registrationAuthority values
• operational information about the metadata aggregation process including details about metadata acquiring form participating federations, results of metadata validation, cache timers for individual participant federations
• operational information about entities published through eduGAIN derived from the metadata
• statistics derived from metadata aggregation, like numbers of entities published by individual federations and much more
• information collected from supporting monitoring services like ECCS, CoCo

The database is placed on a host separated from the external network, accessible only trough a limited numbers of secure hosts. Database access is realised via dedicated user accounts with access right crafted to minimize the possibility of unauthorized changes.

The database is managed mostly via a web interface secured with access passwords. Modification of data on security levels S, 1, 2 can be done without any additional protection. Management of data with security level 3 is protected with on-time passwords mailed to an external mail account of the managing administrator. Management of data with security level 4 requires direct access to the database host. 

For security reasons singing keys can be present only for federations which have been approved to be a member of the eduGAIN SAML Profile.

eduGAIN services

Under the term services listed are utilities as perceived by external users. The internal organisation of services, flow of information and dependencies are not important in this view, but are described in sections further down.

Core Services

eduGAIN Metadata Distribution Service (MDS) is the central component of the eduGAIN service as a whole. For the  detailed description and procedures used in the eduGAIN metadata aggregate distributed by MDS see [eduGAIN-meta]. The eduGAIN metadata aggregate is produced on a separate, secured host (mds-feed) and the 

Organisation and management of services

System maintenance

Operating system and general software components

All eduGAIN core service hosts are 

Custom eduGAIN software

Security considerations

The security of the eduGAIN SAML services is essentially the security of the eduGAIN aggregate. This in turn depends on:

1. validation of federation metadata input data - their originality and integrity - this depends on the safety of federation certificates (stored in the database) and the safety of the signature verification process itself
2. aggregation process - it is crucial that the resulting aggregate contains exactly the data provided by participating federations (after modifications described in the [aggregation])
3. aggregation signature - the eduGAIN signing key and the signing process are the key factors here