InfiniBand
InfiniBand is a high-performance network architecture developed by the IBTA (InfiniBand Trade Association). IBTA is a group of 220 or more companies founded in 1999 with the purpose of developing a technology which would substantially outstrip the capabilities of industry-standard I/O systems.
The InfiniBand architecture (IBA) is mainly based on switching fabric and is designed for use in I/O networks, such as storage area networks (SAN) or cluster networks, where low latency plays an extremely important role.
A large IBA network is composed of different subnets joined together by routers.
IBA Subnet
The elements of a subnet are end nodes, switches, links, and a subnet manager.
End nodes exchange messages routed by switches over links. No routing exists in IBA. A subnet discovery is performed at the beginning by the Subnet Manager.
IBA offers point-to-point bidirectional links with five different signalling data rates:
- SDR - Single Data Rate with signalling rate equal to 2.5 Gbit/s in each direction per connection;
- DDR - Double Data Rate with 5 Gbit/s per lane;
- QDR - Quad Data Rate with 10 Gbit/s per lane;
- FDR - Fourteen Data Rate with 14.0625 Gbit/s per lane;
- EDR - Enhanced Data Rated with 25.78125 Gbit/s per lane.
The physical links can be aggregated in parallel (4X, 8X or 12X) to achieve greater bandwidth.
- The IBTA‘s InfiniBand™ roadmap:
In IBA, QoS is supported through Virtual Lanes (VL) which also provide a mechanism to avoid Head of Line (HoL) blocking. VLs are separate logical communication links which share a single physical link. As a packet traverses the subnet, a Service Level (SL) is defined to ensure its QoS level. Each link along a path can have a different VL, and the SL provides each link with a desired priority of communication.
Each switch/router has an SL to VL mapping table that is set by the subnet manager to keep the proper priority with the number of VLs supported on each link. Therefore, the InfiniBand Architecture can ensure end-to-end QoS through switches, routers and over the long haul.
References
- An Introduction to the Infiniband Architecture - pdf file
- Mellanox Introduction to Infiniband - white paper
– Main.AlessandraScicchitano - 03 May 2012