You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

There are many factors that affect the performance of data transfers. To achieve a high level of performance, ensuring the DTN elements of a Science DMZ deployment are well configured and well-tuned is of key importance.

Each use case should be examined in terms of the specific requirements involved. It is important to keep in mind that changes that improve performance in one data transfer scenario might negatively affect another, as well as to understand what effect these changes have on the end to end transfer performance as a whole. For example, TCP tuning parameters for long-distance transfer are generally different to those for short-distance. Specific values for such parameters are available from various sources. ESnet provides general performance advice [FASTERDATA] which includes settings suitable for certain DTN setups, e.g. Linux tuning information [ESnet]. Here, areas of interest are discussed with references on where to get additional information.

 

DTNs usually mount to a connected file system, whether it is a Storage Area Network (SAN) or High Performance Computing (HPC) network, with a network interface to either transmit or receive data files. Dedicated software transfer tools like GridFTP [GridFTP], XRootd [XRootd], XDD [XDD], FDT [FDT], BBCP [BBCP], FTS [FTS-CERN], etc. are best installed on a DTN instance to achieve better input/output performance concerning data transfer.

Since DTNs are placed in a high-performance network “on ramp” at sites, as shown in Figure 1.1 above, for security reasons, only software for dedicated data transfers are installed on the servers with "allow" access only to the endpoint sites (not open to the normal Internet traffic), and any in-network filtering is typically performed by efficient ACLs, rather than full, heavyweight stateful firewalls (of a type that would protect the rest of the site network for its day to day business traffic).

Please note that while following the next practices for tuning on the DTN endpoints for a data transfer, it is also important that the local network architecture is also appropriately designed.

In the following sections, we describe examples of DTN tuning for Linux DTNs. More information and detail, for example on pacing and MTU settings, can be found in [FASTERDATA2].

Networking

Networking

Storage

Storage

Architecture

Architecture









Networking

Various kernel parameters affect network settings. These kernel parameters can be inspected and modified using the sysctl tool or the files under /proc/sys/. Below, they are referred to using the sysctl name.

Socket Parameters

The following are some of the settings that affect socket networking parameters for all protocols.

net.core.rmem_default

net.core.wmem_default

net.core.rmem_max

net.core.wmem_max

The rmem parameters refer to the socket receive buffer size and the wmem refer to the send buffer size. Under TCP socket parameters, a sending host will typically need to buffer data to support the bandwidth delay product (BDP) of the link, i.e. enough memory for all bits in flight; the higher the RTT of the link, the more memory that is required (see the TCP Throughput Calculator [TTC] resource for an example of a BDP and buffer size calculator). As the names imply, those ending in default set the default value used, and those ending in max set the maximum value that can be requested by a program. A privileged program can set a buffer size beyond the maximum value [socket].


TCP Socket Parameters

The following parameters affect socket sizes for the Transmission Control Protocol (TCP) protocol in particular.

net.ipv4.tcp_mem

net.ipv4.tcp_rmem

net.ipv4.tcp_wmem

The tcp_rmem and tcp_wmem parameters are similar to the socket parameters in that they influence the socket buffer size but they are set differently. Each takes three values: a minimum, a default and a maximum. The default value overrides the value set by rmem_default and wmem_default. The maximum values do not override the settings of rmem_max and wmem_max. The minimum and maximum values set the range in which TCP can dynamically adjust the buffer size. This has no effect on the buffer size that a program can request. The setting tcp_mem affects how TCP manages its memory usage. It takes three values: low, pressure and high. Below the value of low, TCP will stop regulating memory usage (if it was previously). Above the value of pressure, TCP will start regulating its memory usage. The high value is the maximum amount of memory that TCP will use. These are not set in bytes but in the number of pages of memory. They refer to global memory usage not individual sockets [tcp].

Typically, larger buffer sizes are better for higher throughput or data transfers with a high round-trip-time. However, unnecessarily high values will waste RAM.

Another TCP parameter which can have a significant effect on performance is the congestion control algorithm used. The following parameters influence which algorithm is used.

net.ipv4.tcp_available_congestion_control

net.ipv4.tcp_allowed_congestion_control

net.ipv4.tcp_congestion_control

The tcp_available_congestion_control parameter is read-only and shows what algorithms are available on the system. The tcp_congestion_control parameter sets the default algorithm used; the Fasterdata site recommends htcp, but the newer TCP BBR developed by Google and being standardised in the IETF is certainly worth considering due to its greater resilience to packet loss). A program may instead use any of the algorithms set in tcp_allowed_congestion_control and a privileged program can use any of the available algorithms [tcp].

In order for a particular algorithm to be available it may require that a module is loaded into the kernel. While choosing the right congestion control algorithm can improve performance it also affects fairness. On shared links, some algorithms may negatively impact the network performance of other systems.  For example, TCP BBR is more “aggressive” and thus may dominate some traditional TCP algorithms.

The ss command can be used to get detailed information about current TCP connections including what congestion control algorithm they are using:

ss --tcp --info


NIC Driver Settings

In addition to kernel parameters, network performance can be tuned by adjusting settings that affect the actual network hardware. NIC manufacturers may supply proprietary tools to adjust settings. However, some network driver settings can be modified using the ethtool command. Two important settings that should be looked at are the size of the send and receive rings and the offload features.

ethtool --show-ring INTERFACE

ethtool --show-features INTERFACE

Larger rings typically improve throughput but can have a negative impact on latency or jitter. Offloading features allow some work that would need to be done by the CPU to be done on the NIC instead. This can lead to lower CPU utilization and higher throughput.


MTU Settings

The Maximum Transmission Unit (MTU) size can have a significant effect on throughput.  For most typical Ethernet networks, the default MTU is 1500 bytes.  This means each Ethernet frame sent will have a maximum size of 1500 bytes.  However, some networks may support larger MTUs, often up to 9000 bytes.  Where such MTUs are available end to end, greater throughput can be achieved; see the TCP Throughput Calculator [TTC] to explore the effect.  A higher MTU also puts less pressure on devices on the path in terms of their requirements to support high packet per second processing. 

Many NREN backbone networks support 9000 MTU, some support slightly higher to allow for some framing overhead.  Where a Science DMZ network is connected directly on to an NREN backbone through its site router, it may be possible to configure some or all of the Science DMZ to a higher MTU.

In Appendix A, the example tuning for the GTS DTN setup uses an MTU of 8986 bytes, using a simple ifconfig to set the MTU value.


  • No labels