Router Architectures

Basic Functions

Basic functions of an IP router can be divided into two main groups:

• The forwarding plane is responsible for packet forwarding (or packet switching), which is the act of receiving packets on the router's interfaces and sending them out on (usually) other interfaces.
• The control plane gathers and maintains network topology information, and passes it to the forwarding plane so that it knows where to forward the received packets.

Architectures

CPU-based vs. hardware forwarding support

Small routers usually have similar architectures to general purpose computers. They have a CPU, operational memory, some persistent storage device (to store configuration settings and the operating system software), and network interfaces. Both forwarding and control functions are carried out by the CPU, and being different processes is the only separation between the forwarding and the control plane. Network interfaces in these routers are treated just like NICs in general purpose computers, or as any other peripheral device.

High performance routers however, in order to achieve multi-Gbps throughput, use specialized hardware to forward packets. (The control plane is still very similar to a general-purpose computer architecture.) This way the forwarding and the control plane are far more separated. They do not contend for shared resources like processing power, because they both have their own.

Centralized vs. distributed forwarding

Another difference between low-end and high-end routers is that in low-end routers, packets from all interfaces are forwarded using a central forwarding engine, while some high-end routers decentralize forwarding across line cards, so that packets received on an interface are handled by a forwarding engine local to that line card. Distributed forwarding is especially attractive where one wants to scale routers to large numbers of line cards (and thus interfaces). Some routers allow line cards with and without forwarding engines to be mixed in the same chassis - packets arriving on engine-less line cards are simply handled by the central engine.

On distributed architectures, the line cards typically have their own copy of the forwarding table, and run very few "control-plane" functions - usually just what's necessary to communicate with the (central) control-plane engine. So even when forwarding is CPU-based, a router with distributed forwarding behaves much like a hardware-based router, in that forwarding performance is decoupled from control-plane performance.

There are examples for all kinds of combinations of CPU-/hardware-based and centralized/distributed forwarding, even within the products of a single vendor, for example Cisco:

%TABLE{ sort="on" tableborder="0" cellpadding="1" cellspacing="3" headerbg="#000099" headercolor="#FFFFCC" databg="#C8CB8F, #DBDDB5" }%

Effects on performance analysis

Because of the separation of the forwarding and the control plane in high performance routers, the performance characteristics of traffic passing through the router and traffic destined to the router may be significantly different. The reason behind the difference is that transit traffic may be handled completely by the separated forwarding plane (for which this is the function it is optimized for), while traffic destined to the router is passed to the control plane and handled by the control plane CPU, which may have more important tasks at the moment (e.g. running routing protocols, calculating routes), and even if it's free, it cannot process traffic as efficiently as the forwarding plane.

This means that performance metrics of intermediate hops obtained from ping or traceroute-like tools may be misleading, as they may show significantly worse metrics than those of the analyzed "normal" transit traffic. In other words, it may happen easily that a router is forwarding transit traffic correctly, fast, without any packet loss, etc., while a traceroute through the router or pinging the router shows packet loss, high round-trip time, high delay variation, or some other bad things.

However, the control plane CPU of a high performance router normally is not always so busy, and in quiet periods it can deal well with the probe traffic directed to it. So intermediate hop probe traffic measurement results should be usually interpreted by dropping the occasional bad values, based on the assumption that the intermediate router's forwarding plane CPU had more important things to do than dealing with the probe traffic.

Beware the slow path

Routers with hardware support for forwarding often restrict this support to a common subset of traffic and configurations. "Unusual" packets may be handled in a "slow path" using the general-purpose CPU which is otherwise mostly dedicated to control-plane tasks. This often includes packets with IP options.

Some routers also revert from hardware-forwarding to CPU-based when certain complex functions are configured on the device - for example when the router has to do NAT or other payload-inspecting features. Another reason for reverting to CPU-based forwarding is when some resources in the forwarding hardware become exhausted, such as the forwarding table ("hardware" equivalent of the routing table) or access control tables.

To effectively use routers with hardware-based forwarding, it is therefore essential to know the restrictions of the hardware, and to ensure that the majority of traffic can indeed be hardware-switched. Where this is supported, it may be a good idea to limit the amount of "slow-path" traffic using rate-limits such as Cisco's "Control Plane Policing" (CoPP).

– Main.AndrasJako - 06 Mar 2006