This site has been archived. For information on the GN Project’s eduPERT initiative please visit https://archive.geant.org/projects/gn3/geant/services/edupert/Pages/Home.html

Web10G

Web10G is a follow-up of the Web100 project, which had given us things like the web100 TCP instrumentation in the Linux kernel (underlying tools such as ndt), RFC 4898, and contributions to TCP buffer auto-tuning. The US National Science Foundation (NSF) granted over a million dollars of funding for this project to PSC/3NOX and NCSA in September 2010.

The main reasons behind the migration to Web10G are related to strong limitation of Web100.

In Web100 the TCP instrumentation and /proc ABI are combined into a single kernel patch. This causes unsustainable overhead and limits the host to under 30,000 TCP connection. Considering that the /proc interface doesn't scale and has been deprecated, Web100 will never become part of the mainline linux kernel codebase. Moreover Web100 predates RFC4898. Its Kernel instrument set (KIS) doesn't conform it and only supports RENO.

Web10G gets rid of these limitations. The TCP Extended Statistics MIB instrumented in Web10G are defined in RFC4898. The userspace API is simple and lightweight and it uses only 4 calls to give access to all instruments in the stack. The ABI is a netlink based communication with KIS to bring metrics to userspace. It uses the Netlink/Genetlink framework. This allows deployment on high volume production hosts. Web10G should allow millions of concurrent connections.

Netlink is a communication channel used between kernel and user space processes, as well as between user space processes alone (like e.g., UNIX sockets) or a mixture of multiple user space and kernel space processes. It brings kernel data to the user using a socket model. Generic Netlink or Genetlink is the resulting concern that Netlink family numbers would soon be exhausted. Genetlink is a conservative extension of Netlink, in that once a family is created, communication proceeds similarly to that of Netlink. Its use is really simple: Open a socket, read/write the instruments.

Web10G Progress

In May 2011, the project made its first code releases. This includes kernel patches for 2.6.38. On 18 August, 2011, a patch for the Linux 3.0 kernel was released.

In December 2014, Chris Rapier submitted a new setof patches for inclusion in the "mainline" Linux kernel. However, there was some opposition, and it is unclear whether or when this will be accepted. A common view was that the existing tcp_info mechanism could be extended to accommodate this kind of information. Eric Dumazet (Google) also expressed doubts that the amount of instrumentation might be overkill for the general-purpose kernel. Following discussions at the Linux NetDev 0.1 conference in February 2015, the tcpinst-wg (TCP Instrumentation) mailing list was created to discuss a way forward.

The project's web site provides the usual general information, news, access to the software, as well as discussion fora. The source code is also available through git .

Web10G code

According to the developer section on the web10g.org site, the kernel code from web100 was split into two parts:

  1. The TCP instrumentation - a set of counters and other variables that are maintained by TCP. This is implemented as a kernel patch, just like it was in web100.
  2. The interface for accessing that instrumentation, i.e. the /proc interface. This is now a loadable kernel module.
    A declared goal of the project is to get the Web10G improvements integrated into the "mainline" Linux kernel source:

The goal of the Web10G project is to create a production-quality implementation of the RFC 4898 standard TCP instrumentation, and to have it included in all Linux kernel releases as a loadable kernel module. Inclusion in the main-line Linux kernel will greatly improve the ease of deployment, support, and third party development.

Splitting the code up should help making the code easier to integrate.

The code archives can be found here.

References

– Main.SimonLeinen - 2011-03-25 - 2015-02-26

– Alessandra Scicchitano - 2012-08-10

  • No labels