Open Cloud Mesh

The Open Cloud Mesh project is coordinated by GÉANT with contributions from CERN, National Research and Education Networks (NRENs), academic institutions and commercial partners.

Interconnected Private Clouds for Universities and Researchers

Open Cloud Mesh (OCM) is a joint international initiative under the umbrella of the GÉANT Association that is built on ownCloud’s open federated cloud sharing application programming interface (API) taking Universal File Access beyond the borders of individual clouds and into a globally interconnected mesh of research clouds - without sacrificing any of the advantages in privacy, control and security an on-premises cloud provides. OCM defines a common file access layer across an organization and/or across globally interconnected organizations, regardless of the user data locations.

Concept document

The Open Cloud Mesh concept document was produced by ownCloud Inc. and first distributed on 23 July 2015

Download from here...

Statement
All leading partners of the Open Cloud Mesh project are fully committed to the open API design principle. This means that - from day one - the OCM sharing API should be discussed, designed and developed as a vendor neutral protocol to be adopted by any on-premise sync&share product vendor or service provider. We acknowledge the fact that the piloting of the first working interface prototype will be carried out in an ownCloud environment that should not effect the adoption of the open API in any other vendor and provider domain.

Community effort - open for participation

A collaborative project was established under the umbrella of GÉANT called the Open Cloud Mesh project on 22 October 2015. The kick-off meeting was held in Vienna, Austria.

The project is co-managed by Peter Szegedi (GÉANT), Jakub Moscicki (CERN) and Christian Schmitz (ownCloud). This combination of project management ensures that all the major stakeholders – GÉANT National Research and Education Networks, CERN research community and ownCloud Inc. as a commercial company with its open source developers community – are equally represented in the project and the technical, management and business aspects are well-covered.

The collaborative project is fully open to any participation and in-kind contributions. Interested parties can subscribe to the mailing list at:

ocm-all@lists.geant.org...

GÉANT Project code P15_225

Key stakeholders

Name	Organization	Interest / Involvement / Role	RACI	Stakeholder Comments
Peter Szegedi	GÉANT	Project management (about 200h/year)	A, R	Committed
Jakub Moscicki Massimo Lamanna	CERN	Project management	A, R	Committed
Christian Schmitz	ownCloud Inc.	Project management	A, R	Committed
Charles du Jeu David Gillard	Pydio	Contribute to the specifications and development	R, C	Committed
Rogier Spoor	SURFnet	Contribute to the specifications and development	R, C
Ron Trompert	SURFsara	Contribute to the specifications and development	R, C	Committed
Christoph Herzog Simon Leinen	SWITCH	Contribute to the specifications and development	R, C	Committed
Guido Aben David Jericho	AARNet	Contribute to the specifications and development	R, C	Committed
Holger Angenent	Sciebo / Uni Münster	Contribute to the specifications and development	R, C	Committed
David Antoš	CESNET	Contribute to the specifications and development	R, C	Committed
Frederik Orellana	DeIC	Contribute to the specifications and development	R, C
Kurt Bauer	ACOnet	Contribute to the specifications and development	R, C
Christian Kracher	University of Vienna	Contribute to the specifications and development	R, C	Committed
Jari Miettinen	CSC/Funet / EUDAT	Contribute to the specifications and development	R, C
Andreas Eckey	Technische Universität Berlin	Contribute to the specifications and development	R, C
Woojin Seok	KISTI	Interest from South Korea	I
Benedikt Wegmann Ralph Krimmel	GWDG	Contribute to the specifications and development	R, C	Committed
Hrachya Astsatryan Andranik Hayrapetyan Wahi	ASNET-AM	Contribute to the specifications and development	R, C	Committed
Christian Sprajc	PowerFolder	Interested	I

The project will be delivered in phases.

Project plan (Phase II.)

The Phase II. objectives and project planning will be defined as we go. A demonstration is expected at TNC'16 in June 2016.

PHASE II.	DESCRIPTION	COMMENT	ASSIGNED TO	START DATE	END DATE	STATUS
1. Pre-project (preparation)	Initiation Pick up the results on Phase I. See section 4.2 Deliverable of Phase I. below...	Create the new structure of two WGs: Strategic WG ocm-admin@lists.geant.org Technical WG ocm-tech@lists.geant.org Assign mailing lists above All: ocm-all@lists.geant.org	Peter Szegedi	19 January 2016	5 February 2016	COMPLETE
2. Initiation
2.1	Call for a kick-off video conference	What are the next steps for OCM? 1. Understanding current usage of Federated Sharing feature - How many ownCloud sites in our community have this feature enabled? - How many admins know about this feature? - What prevents them from enabling it? - If enabled, how many users have already used this functionality. [ACTION] Christian to send a questionnaire to all known ownCloud instance admins. [ACTION] Holger to prepare simple admin instructions on how to check usage and status. 2. General feeling is that we do not have enough understanding of this technology Tilo proposed to setup a test bed where we could check various technical aspects (ref: http://cs3.ethz.ch/presentations/Interoperability/02Moscicki.pdf) and get confidence. [ACTION] Kuba to send preliminary test plan. 3. Involvement of pioneer users Tilo proposed to involve pioneer users early on to get real user feedback. The extent of exposure of the users should be function of progress in point (2). [ACTION] All participants of the call are asked to look for potential users that would be willing to try out this functionality. There should be a real use-case for it e.g. members of the same research group sitting in different locations and using private clouds in their institutes already. Possible candidate users identified already: - Physicists from ETH and CERN. - Others...	Christian Schmitz	19 January 2016	11 February 2016	COMPLETE
2.2	Define objectives, key results and timelines	Repeat the Zurich demo but with more domains and more interesting use cases. Demonstrate the interest and if possible the active participation of vendors other than ownCloud Inc.	Peter Szegedi	11 February 2016	11 March 2016	COMPLETE
3. Subsequent stages (execution)
3.1	Build the OCM community	ownCloud v.8.2 or higher: Uni Münster, Uni Vienna, AARnet, SWITCH, CESNET, GWDG Others very close: SURFnet, CERN Pydio v.x?: ASNET-AM, University of Lausanne (TBC)	Peter Szegedi	19 January 2016		ON GOING
3.2	KEY - Involvement of other vendors...	Locate other vendors and identify user cases, universities with two products and cross-sharing needs. 18-02-2015 News from Christian: Pydio will most likely join OCM. They are actually aiming for test implementation by end of March. 16-03-2016 Charles du Jeu and David Gillard from Pydio (https://pydio.com/) joined the OCM project. Charles, the CEO/CTO of Pydio, confirmed that that are preparing the release of an important dev version, that actually contains the Federated Sharing API implementation. See https://pydio.com/en/community/releases/pydio-core/pydio-core-631-development-release for more info. Pydio is making its best to have this release transformed to a stable one by the end of this month. Press release https://pydio.com/en/blog/federated-sharing-pydio-connects-owncloud-and-joins-opencloudmesh-initiative	Christian Schmitz Peter Szegedi	19 January 2016	16 March 2016	COMPLETE
3.3	Demonstrate inter-vendor OCM functionality	29-03-2016 OCM call at 2pm CET 1) Charles (Pydio) demonstrated via screen-share the OCM functionality that has been implemented by Pydio in their software. Federated sharing of files was shown between two local independent instances of Pydio v.6.4.0.(to be released on 30 March) running locally. Comments: Server name and user name must be defined separately. Should work with username@domain@servername (TBC) Shared files are kept in a tree separate from the local file tree (this topic is also being discussed by the OCM community) 2) Charles (Pydio) and Frank (ownCloud) also tested the interoperability (technical feasibility) of the OCM protocol implementations between Pydio v.6.4 and ownCloud v.9.0 offline. It was reported successful but yet to be seen...	Charles du Jeu David Gillard Frank and Lukas	16 March 2016	29 March 2016	COMPLETE
3.4	Multi-vendor OCM validation by the Community ownCloud - Pydio	1) AARNet (Guido) initiated some initial testing with ASNET-AM (Hrachya) AARNet (Australia) uses ownCloud and ASNET-AM (Armenia) uses Pydio. Some initial issues out of the box. Pydio developers are involved... Results to be seen... 2) More Pydio users and test scenarios to be defined (see 3.1) RENATER and University of Lausanne to be contacted...	Guido Aben Hrachya Astsatryan Andranik Hayrapetyan Wahi			COMPLETE
4. Delivery
4.1	Demonstration at the TNC'16 Conference in Prague, Czech Republic	Booth demos: ownCloud, Pydio: Interoperability demo at GÉANT booth Lightning talk by Guido Aben: submitted and approved	Christian Schmitz Guido Aben Charles du Jeu	19 January 2016	13 June 2016	COMPLETE
4.2	Demo feedback	Collect and made available... DEMO IDEAS for the next step... We can redo the same demo as shown at CS3 but fancier and bigger, also involving more sites. Additional sites which would like to join this time for the demo: SURFNet: currently stuck with the Shibboleth issue with ownCloud 8 CERN: needs ownCloud 9 fully deployed and operational (sharing module is not in maintainable/fixable state as of ownCloud 8) Others...			30 June 2016	ON-GOING
5. Closing	Publish results and define next phase				30 June 2016	ON-GOING

Project plan (Phase I.)

Phase I. aims at demonstrating the first working prototype of the OCM protocol (API v1.0 BETA) functionally working between two separate administrative ownCloud domains (i.e. between two NRENs).

October 2015 - February 2016

PHASE I.	DESCRIPTION	COMMENT	ASSIGNED TO	START DATE	END DATE	STATUS
1. Pre-project (preparation)	Start collecting organizations and people interested in joining the initiative.	Mailing list to be created. Announcements to be made.	Peter Szegedi Christian Schmitz	8 February 2015	15 June 2015	COMPLETE
2. Initiation
2.1	ownCloud to release the first version of the API code and documentation.	DELAYED Code v.0.002 has been released on 27 July 2015 by ownCloud Inc. Comments have been provided by CERN.	Christian Schmitz	8 February 2015	27 July 2015	COMPLETE
2.2	Create a project team, estimate budget and organize a kick-off meeting.	VC for coordination on 24 August 2015. Concluded in the Communique GSec(15)015. Pre-launch meeting organized by ownCloud on 28 August 2015 in Berlin. Peter (GÉANT), Guido (AARnet) and Kuba (CERN) and others. Code v.0.002 released and commented Christian as an interim leader GÉANT to provide the project framework Find the neutral project lead, co-chairs from the community. Approach other vendors: PowerFolder, W3C, Pydio, Cozy Put it in the GÉANT procurement requirements (the support for the API) NIF/PID to be submitted and mailing list migration to be done. Kick-off meeting: 22 October, 2015 in Vienna, Austria The slides of the event can be found here. password edu221015	Peter Szegedi Christian Schmitz	8 February 2015	22 October 2015	COMPLETE
3. Subsequent stages (execution)
3.1	Get the API v.0.004 code and documentation, define the participating domains, initiate the first tests.	DRAFT protocol definition v.0.004 released OpenCloudMesh = ownCloudMesh	Christian Schmitz	15 June 2015	28 Augustus 2015	COMPLETE
3.2	Demonstrate the first working prototype	Uni Münster server-to-server sharing (i.e. federated cloud sharing) feature has been demonstrated by Holger to Kuba and Peter. There was an agreement to plan for a new demonstration between two administrative domains, say SURFnet or CERN and Uni Münster. collect a list of feature requests related to the function (to be discussed with ownCloud) think about trust policies and quality assurances in the context of federated cloud sharing. Comments from Kuba: quality of service: in the current owncloud implementation a remote share enters into the discovery process for synchronization - unstable or poorly-performing remote instance may impact users of the local instance authorization: a service manager should be in some control of which cloud instances can issue external shares requests for their users [there is plenty of room for abuse there, in addition this is amplified by the synchronisation of such injected shares on the user’s devices] security: it is not clear how secure the federated sharing mechanism currently is under-the-hood open protocol: is there a bottom-up interest in OCM being embraced by other sync/share software stacks?	Holger Angenent	22 October 2015	18 November 2015	COMPLETE
3.3	Prepare for a demonstration of the federated cloud sharing feature between two administrative domains, say SURFsara and Uni Münster	We are looking for volunteers with ownCloud server version v.8 or above to test the feature. Ron, SURFsara pointed out the the Federated Cloud Sharing feature does not work together with SAML/Shibboleth based authentication. This is a showstopper for a planned SURFdrive - Uni Münster demonstration. Ticket has been created: https://github.com/owncloud/core/issues/21227 ownCloud is working on a quick work-around. FCS just needs a user name. Partners to demonstrate Federated Cloud Sharing on 19 January 2016: Uni Münster (local users) h_zimm01@uni-muenster.de@uni-muenster.sciebo.de a_wilm04@uni-muenster.de@uni-muenster.sciebo.de AARnet (remote users) david.jericho@aarnet.edu.au@cloudstor.aarnet.edu.au/plus guido.aben@aarnet.edu.au@cloudstor.aarnet.edu.au/plus Uni Vienna (remote user) krachec6@ucloud-test-files.univie.ac.at Demonstration has been successfully done: Holger at Uni Münster shared a folder with Guido at AARnet (federated cloud sharing). Holger at Uni Münster shared a folder with Andres at Uni Münster (local sharing). Christian at Uni Vienna shared a folder with Holger at Uni Münster (federated cloud sharing). Simon at SWITCH shared a folder and sync files with Holger at Uni Münster (federated cloud sharing).	Holger Angenent Andreas Wilmer Ron Trompert David Jericho Guido Aben Christian Kracher Simon Leinen	18 November 2015	19 January 2016	COMPLETE
3.4	Initiate discussions about policies, metadata release, directories, legal issues, etc.	Two main topics have been identified (18 November 2015) Trust policy: What level of trusted relationship is needed to be established and maintained between two administrative cloud domains in order to share files and folders among their users. Quality assurance: What service level assurances are needed to be agreed and verified between two server operators in order to maintain the integrity and scalability of shared files and folder inside or outside of the users' file system. Mind map (15 January 2016): Kuba at CERN talked about the open issues and Simon at SWITCH talked about the standardization aspects (19 January 2016) Kuba: http://cs3.ethz.ch/presentations/Interoperability/02Moscicki.pdf Simon: http://cs3.ethz.ch/presentations/Interoperability/04Leinen.pdf	Jakub Moscicki Simon Leinen All	18 November 2015	19 January 2016	COMPLETE
4. Delivery
4.1 Workshop	Open API v.1.0 documented and released at least in BETA version with the intention to come up v.2.0 vendor agnostic version (IETF WG)	Cloud Services for Synchronization and Sharing (CS3) Workshop *ETH Zürich, Switzerland; January 18-19 2016* http://cs3.ethz.ch/program.html *Slides and presentation/demo materials are available!* *Praying for OCM to work...*	Peter Szegedi Christian Schmitz Kuba Moscicki Simon Leinen	15 November 2015	19 January 2016	COMPLETE
4.2 Deliverable	Come up with recommendations for the development of the API towards a new v.1.0 according to the requirements of an open standard.	Set of recommendations: Integration with Macaroons authz/authn: Google macaroons (http://research.google.com/pubs/pub41892.html) Cloud user lookup service Need @ a @ simple @ username @ structure endpoint discovery: DNS David: Rather than trying to implement a meta-data or negotiation server, why don't we just use the NAPTR/SRV DNS RR method I proposed late last year? Multiple records, one for each protocol, makes discovery and announcement very easy, and allows the sites to use different endpoints for different protocols as suited. Something akin to: _standard._opencloudmesh._tcp.cloudstor.aarnet.edu.au. SRV 0 0 5009 ocm.cloudstor.aarnet.edu.au. _quic._opencloudmesh._udp.cloudstor.aarnet.edu.au. SRV 0 0 5009 quic.cloudstor.aarnet.edu.au. _ftp._opencloudmesh._tcp.cloudstor.aarnet.edu.au. SRV 0 0 5011 ocm.cloudstor.aarnet.edu.au. ...and so on. Discovery and compatibility is known by simple DNS query, and dynamically configurable using existing standards and tools. Supporting new protocols becomes a server side issue, rather than having to make any additional software aware. Protocol negotiation handshake Holger: What would you think of a signaling mechanism, comparable to the handshake when handling the method for an encrypted connection? We think of something like: Server A asks server B for a connection and offers WebDAV and CMIS and asks server B which of these it likes. Server B speaks only WebDAV so they agree to use this. Of course it is clear that now it could happen that both servers do not offer a matching method. But the acceptance threshold to participate at OCM could be lowered since only the signaling would be mandatory. David: At least if the mandate was WebDAV for the most basic method, that would be the lowest barrier to entry. WebDAV has been around since 1996, and it really doesn't extend existing HTTP that far, there's no real excuse for not implementing it. A technical person is able to talk to a service using telnet/netcat/socat and do sharing functions manually, as well as making it simple to do these things programmatically using curl. Metadata against the query is also trivial, because HTTP/(1.1\|2) support headers that can be used to carry all the relevant information. Caching should be a secondary consideration, because it's a non-trivial activity to get right (consider HTTP caching and HTTP 304 return codes). Our primary concern is fast transfer, reliable transfer, and having people join in. In the case of many small files, many alternate protocols allow streaming, batching or the like, akin to the old days of asking a ftp server for a tar of a directory. Security and Trust Trusted curcle of servers vs. open internet policy Verification of architectures and deployments up to a community standard. DeiC's Smashbox idea More admin features are in 9.0 also better addressing and web popup. Shibboteh ticket. Data path for federated shares pass-through: data is pumped from server B to the client A via server A (as currently available) server-cache: as mentioned by Holger (comes with potential consistency&refresh mechanism issues to be addressed) client redirect: as mentioned by Paul Millar (would affect the current usability model of ownCloud) Pick a preferred home for standardization IETF??? - Simon to give a thought All the recommendations have been fed into the Phase II initiation phase.	All	19 January 2016	05 February 2016	COMPLETE
5. Closing	Summary	The OCM team agreed to continue and split into two working groups (WGs): Strategic policy and standardization oversee WG Technical protocol definition and implementation WG The key objective of the Strategic WG will be to reach the ultimate "Open Cloud Mesh standard" and to oversee (not overdrive) the Technical WG making sure that the open API design principles are properly addressed. The key objective of the Technical WG will be to deliver one or more working prototypes of OCM and to provide (not force) input to the Strategic WG making sure that their assumptions are realistic. The detailed objectives and key results of the Phase II will be determined in the initiation phase. Phase I is considered to be CLOSED. News item: http://www.geant.org/News_and_Events/Pages/OpenCloudMesh.aspx	Peter Szegedi	19 January 2016	05 February 2016	COMPLETE

History

Around early 2012, TF-Storage participants started to actively look into data storage software platforms in order to provide on-premise file-based sync&share (aka. Dropbox-like) services to their constituencies.
- Some NRENs even ventured into the development of a proof-of-concept tool called the Trusted Cloud Drive (TCD) under TERENA
By mid 2013, ownCloud appeared to be the most promising one with a growing open-source development community behind.
In December 2013, the GÉANT Association (formerly known as TERENA) and ownCloud Inc. made an agreement that serves to facilitate the desire of various National Research and Education Networking organisations (NRENs) to introduce services based on ownCloud technology and/or to offer the technology to their constituencies.
As part of this collaboration effort, in January 2015, Christian Schmitz from ownCloud initiated an idea (aka. OpenCloudMesh) to interconnect the individual on-premise private cloud domains at the server side in order to provide federated sharing and syncing functionality between the different administrative domains.

Problem space

Make a comment below if you see what the problem is that we are trying to solve on the screenshot of the Box folder view

Space shortcuts

Page tree