Requirement
|
|
Overview
RARE project objective is to provide a routing platform proposing various solutions addressing multiple use cases in the R&E landscape. In the picture below you see in purple the different use case:
As you can notice, each use case will run on different hardware that potentially can have different dataplanes. As we were starting from a clean slate environment without not that much choice especially with P4 programmability the first dataplane or P4 target considered was BMv2. BMv2 is an excellent way to learn P4, it is also the first target we use in order to program and validate new features. After 6 months of practising our "P4-fu" we developed:
- a P4lang repository for ubuntu bionic and focal
- a debian 10 repository
- had our first RARE/FreeRouter prototype powered by a P4 BMv2 dataplane !
Our initial work, considering FreeRouter Java nature, was to wrote a Java P4Runtime GRPC client that would be able to program the entries in the tables exposed by BMv2 via the P4Info file. However, this would have intimately tied FreeRouter code to P4Runtime GRPC code. Even if this is more natural to choose this solution, going into that direction implied that other dataplane than BMv2 would be compliant to P4Runtime. Which turns out that this is not the case. We then opted for a simple message API via a bi-directional raw UNIX socket. We will see what's this means later.
Motivated by this successful experience with BMv2, we then decided to move forward and started to study TOFINO target. We were greedy and eager to apply our P4 code against multi-terabits traffic. After few P4 program compilations, our first impression from my personal perspective was ... mind blowing ! INTEL/BAREFOOT TOFINO effectively opened to us the door to multi-terabits packet processing... Just the idea to be have at the tip of your finger the possibility to process traffic at these traffic levels was exciting !
As a side note, the journey was not without suffering and pains... 
We had to port our BMv2 code and to TOFINO and this was not "Une lettre à la poste"... It is not that TOFINO programming is gratuitously painful. It is just that it is p4c-tofino's job to make sure that our packets are processed at silicon lighting speed. Imagine you are asked to convey parcels by driving from Paris to Amsterdam with a car that has a truck with an infinite size, with an infinite gaz tank and no particular speed constraint along the road. And then you are asked to to the same trip but with an actual real car that has a truck with a fixed size and with a 50 litre gaz tank, and of course you'll have to follow speed signs along the road.
In the first case, you would put as many as parcel you would want and you even won't bother looking at your gaz tank level and maybe you'd set the speed to 200 Km/h. The second case forces you to carefully think about how many parcel you can put in your truck, check to see if one completely full tank can be sufficient for the trip and of course, you would have to follow the speed signs.
If you allow me this comparison, this is where BMv2 and TOFINO programming differs.
But, this pain was not in vain, it was for the greater good... You can't imagine this inherent joy when you see TOFINO compiler displaying the DONE word ! For the veteran who can remember, it is the same feeling when you manage to compile your first program in ADA language. The compiler is not strict that compiling an ADA program is in itself a feat. No wonder why this language is used in Spatial rocket (Ariane).
Back to our dataplane interface story, even TOFINO and BMv2 share some roots, while BMv2 had P4Runtime as a northnound interface, INTEL/BAREFOOT pushed into TOFINO platform with P4_16 their GRPC interface counterpart: BfRuntime.
Our best bet, paid off as FreeRouter message API was unchanged and without not very hard effort we could add a new dataplane "wingman" to FreeRouter control plane.
As a recap:
- For BMv2: Our interface yield P4Runtime RPC calls. This program is called: forwarder.py
- For TOFINO: Our interface yield BfRuntime RPC calls. This program is called witout too much originality: bf_forwader.py
At that point we were starting to have a decent LSR/LER router for CORE and Aggregation use cases.
But we still had nothing at the EDGE/AGGREGATION layer in term of solution proposal, deploying P4 hardware might be way too expensive in specific context sich as small R&E institutions like primary schools or small R&E labs. To that purapose we started to study new targets such as VMWARE XDP and a very promising projet: T4P4S ELTE. While we could not use XDP without a lot of P4 code rewriting and compromise, T4P4S ELTE was from our perpective very promising. But due to a compilation issue, we could not move forward.
FPGA was also a solution that we considered but had no access to any FPGA hardware that was P4 compliant.
As a result, we were a little bit bitter and started to read DPDK library. And we started to play with DPDK examples... These examples were tremendously useful as it sparked some DPDK development into the RARE team. Csaba FreeRouter lead developer, step by step came up with this GENIUS idea: why don't we just use emulate P4 RARE P4 dataplane program ? We can still revert to using T4P4S ELTE when it will be ready ?
P4emu/P4dpdk was then born !
To conclude this short story, RARE/FreeRouter has now 3 completely different dataplanes: (in order of appearance)
- BMv2
- TOFINO
- DPDK
Unique RARE/FreeRouter feature
However, please note that FreeRouter message API is common to the three dataplanes listed above. You'll see further how this structure make the solution: an open modular, interchangeable solution.
Article objective
In this article, let's present RARE/FreeRouter platform structure and focus on the interface(S) between FreeRouter control plane and various dataplane.
Diagram
[ #001 ] - Modular design
As said in the beginning of the article, freeRouter control plane would have to deal with dataplane of different nature. And we concluded in mentioning that for now, freeRouter has three dataplanes. Each of these dataplanes have their own northbound interface, whether this is P4Runtime for BMv2, BfRuntime for TOFINO or P4DPDK for system compatible with DPDK and having DPDK complinnt NIC.
For BMv2 we just had to write an interface that would translate freeRouter API message into P4Runtime GRPC calls. For BMv2 this interface is called forwarder.py:
For TOFINO we just had to write an interface that would translate freeRouter API message into BfRuntime GRPC calls. For TOFINO this interface is called bf_forwarder.py:
For DPDK we just had to write an interface that would translate freeRouter API message into DPDK primitives. This interface is included into DPDK dataplane bundled into freeRouter binaries: p4dpdk.bin
It is just as simple as that !
Discussion
One thing that we did not experienced, but this can be maybe one day a reality.
What if you have your own control plane and that you absolutely want to keep it, but would like to re-use BMv2/TOFINO or DPDK RARE dataplane ?
Well this is possible !
Long time ago I met Thomas MANGIN (yet another cool and nice French guy ) which is the author of Exa-BGP, i did not talk about this and I don't want to give him bad idea, but what if if would like to hook a TOFINO P4 dataplane to exaBGP ?
Well, he actually would just have to teach exaBGP to handle entry ADD/DELETE/MODIFY message according to the message API above.
I also love the work DONE at the SoNIC project level and I know that SoNIC has already a P4 dataplane called switch.p4. I doubt it will be the case one day but, what if SoNIC project wanted to re-use RARE dataplane for especially for Service Provider capability ?
OK, this sounds crazy, but the modular design we proposed here is valid and can make the RARE dataplane available for other control plane.
Of course, we strongly suggest you to stick with FreeRouter as you will just realize IMHO that in the TELCO Service Provider space there is no match. You'll have the venerable IOS-XR and JUNOS, but these are not Open Source counterparts.
Conclusion
In this 1st article you:
- had a 10K feet view description of RARE/FreeRouter modular design
- This design allow rapid dataplane addtion without altering whatsoever FreeRouter code base
- In case you would like to re-use BMv2/TOFINO/P4DPDK dataplane, this has been never implemented but this is possible !
Message API documentation
From the time being this API message is not yet publicly documented. However, it is available and buried inside forwarder.py or bf_forwarder.py source code. This is work in progress but if you feel an urgent need to use it feel free to read the code.
PS: We will publish this document ASAP, but time plays against us ...