Page History

...

Expand

title	create freeRouter /rtr folder

In this setup we will create a freeRouter folder at the filesystem root directory

Code Block

language	bash
theme	Midnight
title	Create freeRouter folder at filesystem root directory

mkdir /rtr

Code Block

language	bash
theme	Midnight
title	get freeRouter control plane software

cd /rtr 
wget http://freerouter.nop.hu/rtr.jar

Code Block

language	bash
theme	Midnight
title	get freeRouter net-tools tarball

cd /rtr 
tar xvf rtr.tar -C /rtr
rm rtr.tar

Expand

freeRouter systemd startup script

title	Disable host networking (One time installation)

As freeRouter is handling the networking task, we has to disable the appliance networking. Forgetting to do so will result in conflicts and unpredictable behaviour.

Code Block

language	bash
theme	Midnight
title	freeRouter systemd startup script	Disable networking from systemd perspective

systemctl set-default multi-user.target
rm /usr/cat /lib/systemd/systemnetwork/rtr.service

[Unit]
Description=router processes
Wants=network.target
After=network-pre.target
Before=network.target

[Service]
Type=forking
ExecStart=/rtr/hwdet-all.sh

[Install]
WantedBy=multi-user.target*
SVC="network-manager NetworkManager ModemManager systemd-network-generator systemd-networkd systemd-networkd-wait-online systemd-resolved wpa_supplicant"
systemctl disable $SVC
systemctl mask $SVC

Expand

title	freeRouter systemd startup script

Code Block

language	bash
theme	Midnight
title	/rtr/hwdet-all.sh freeRouter systemd startup script

cat /lib/systemd/system/rtr/hwdet-all.shservice

[Unit]
Description=router processes
Wants=network.target
After=network-pre.target
Before=network.target

[Service]
Type=forking
ExecStart=/rtr/hwdet-all.sh

[Install]
WantedBy=multi-user.target

Code Block

language	bash
theme	Midnight
title	/rtr/hwdet-all.sh script

cat /rtr/hwdet-all.sh

#!/bin/sh

cd /rtr
echo 1#!/bin/sh

cd /rtr
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/loall/disable_ipv6
ipecho link1 set lo up > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6
ip link set lo up mtu 65535
ip addr add 127.0.0.1/8 dev lo
ip addr add ::1/128 dev lo

# DPDK
echo 96 > /proc/sys/vm/nr_hugepages
modprobe uio_pci_generic

dpdk-devbind.py -b uio_pci_generic 01:00.0 
dpdk-devbind.py -b uio_pci_generic 02:00.0 
dpdk-devbind.py -b uio_pci_generic 05:00.0 
dpdk-devbind.py -b uio_pci_generic 06:00.0 
dpdk-devbind.py -b uio_pci_generic 07:00.0 
dpdk-devbind.py -b uio_pci_generic 08:00.0 

#VETH for CPU_PORT and OOBM_PORT
ip link add veth0a type veth peer name veth0b

ip link set veth0a multicast on
ip link set veth0a allmulti on
ip link set veth0a promisc on
ip link set veth0a mtu 8192
ip link set veth0a up

ip link set veth0b multicast on
ip link set veth0b allmulti on
ip link set veth0b promisc on
ip link set veth0b mtu 8192
ip link set veth0b up

ethtool -K veth0a rx off
ethtool -K veth0a tx off
ethtool -K veth0a sg off
ethtool -K veth0a tso off
ethtool -K veth0a ufo off
ethtool -K veth0a gso off
ethtool -K veth0a gro off
ethtool -K veth0a lro off
ethtool -K veth0a rxvlan off
ethtool -K veth0a txvlan off
ethtool -K veth0a ntuple off
ethtool -K veth0a rxhash off
ethtool --set-eee veth0a eee off

ethtool -K veth0b rx off
ethtool -K veth0b tx off
ethtool -K veth0b sg off
ethtool -K veth0b tso off
ethtool -K veth0b ufo off
ethtool -K veth0b gso off
ethtool -K veth0b gro off
ethtool -K veth0b lro off
ethtool -K veth0b rxvlan off
ethtool -K veth0b txvlan off
ethtool -K veth0b ntuple off
ethtool -K veth0b rxhash off
ethtool --set-eee veth0b eee off

ip link add veth1a type veth peer name veth1b

ip link set veth1a multicast on
ip link set veth1a allmulti on
ip link set veth1a promisc on
ip link set veth1a mtu 1500
ip link set veth1a up

ip link set veth1b multicast on
ip link set veth1b allmulti on
ip link set veth1b promisc on
ip link set veth1b mtu 8192
ip link set veth1b up

ip link set wlan0 up

ethtool -K veth1a rx off
ethtool -K veth1a tx off
ethtool -K veth1a sg off
ethtool -K veth1a tso off
ethtool -K veth1a ufo off
ethtool -K veth1a gso off
ethtool -K veth1a gro off
ethtool -K veth1a lro off
ethtool -K veth1a rxvlan off
ethtool -K veth1a txvlan off
ethtool -K veth1a ntuple off
ethtool -K veth1a rxhash off
ethtool --set-eee veth1a eee off

ethtool -K veth1b rx off
ethtool -K veth1b tx off
ethtool -K veth1b sg off
ethtool -K veth1b tso off
ethtool -K veth1b ufo off
ethtool -K veth1b gso off
ethtool -K veth1b gro off
ethtool -K veth1b lro off
ethtool -K veth1b rxvlan off
ethtool -K veth1b txvlan off
ethtool -K veth1b ntuple off
ethtool -K veth1b rxhash off
ethtool --set-eee veth1b eee off

ip addr flush dev veth1a 
ip addr add 192.168.128.254/24 dev veth1a

#ADD DEFAULT ROUTE to OOBM SDN999
route add default gw 192.168.128.1

# START RTR !
start-stop-daemon -S -b -x /rtr/hwdet-main.sh

Code Block

language	bash
theme	Midnight
title	make hwdet-main.sh executable

chmod u+x /rtr/hwdet-main.sh

Warning

title	A bit of explanation

Code Block

language	bash
theme	Midnight
title	Disable IPv6

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6
ip link set lo up mtu 65535

Note that IPv6 operation would occur on the host it self, IPv6 will be handle at freeRouter level

Code Block

language	bash
theme	Midnight
title	Disable IPv6

echo 96 > /proc/sys/vm/nr_hugepages
modprobe uio_pci_generic

dpdk-devbind.py -b uio_pci_generic 01:00.0 
dpdk-devbind.py -b uio_pci_generic 02:00.0 
dpdk-devbind.py -b uio_pci_generic 05:00.0 
dpdk-devbind.py -b uio_pci_generic 06:00.0 
dpdk-devbind.py -b uio_pci_generic 07:00.0 
dpdk-devbind.py -b uio_pci_generic 08:00.0

In the stanza above, we configure DPDK (required)

Configure HugePages

In this case we use 96 hugepages, this value can be different if you are using a BOX with different characteristic (# of ports, memory etc.) The objective is to configure a value that is not too high (waste of resources) and not too small. otherwise p4dpdk won't run. In this case this let 10 Free HugePages.

Code Block

language	bash
theme	Midnight
title	HugesPages Verification

grep HugePages_ /proc/meminfo
HugePages_Total:      96
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

Activate UIO_PCI_GENERIC driver
Bind the interfaces to DPDK, DPDK will controlled them now. Keep in mind that now they will be invisible from the linux kernel.

This command use device PCI ID. In order to check device PCI ID just issue the below command:

Code Block

language	bash
theme	Midnight
title	List PCI device ID list ready to be use (or not by DPDK)

 dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:05:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:06:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:07:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:08:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb

Network devices using kernel driver
===================================
0000:09:00.0 'AR928X Wireless Network Adapter (PCI-Express) 002a' if=wlan0 drv=ath9k unused=uio_pci_generic 

No 'Baseband' devices detected
==============================

Other Crypto devices
====================
0000:00:1a.0 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine 0f18' unused=uio_pci_generic

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

Configure the appliance OOBM via veth pair (as all physical port are handle by dpdk and will be invisible from the Linux kernel)

Code Block

language	bash
theme	Midnight
title	Disable IPv6

#VETH for CPU_PORT and OOBM_PORT
ip link add veth0a type veth peer name veth0b

ip link set veth0a multicast on
ip link set veth0a allmulti on
ip link set veth0a promisc on
ip link set veth0a mtu 8192
ip link set veth0a up

ip link set veth0b multicast on
ip link set veth0b allmulti on
ip link set veth0b promisc on
ip link set veth0b mtu 8192
ip link set veth0b up

ethtool -K veth0a rx off
ethtool -K veth0a tx off
ethtool -K veth0a sg off
ethtool -K veth0a tso off
ethtool -K veth0a ufo off
ethtool -K veth0a gso off
ethtool -K veth0a gro off
ethtool -K veth0a lro off
ethtool -K veth0a rxvlan off
ethtool -K veth0a txvlan off
ethtool -K veth0a ntuple off
ethtool -K veth0a rxhash off
ethtool --set-eee veth0a eee off

ethtool -K veth0b rx off
ethtool -K veth0b tx off
ethtool -K veth0b sg off
ethtool -K veth0b tso off
ethtool -K veth0b ufo off
ethtool -K veth0b gso off
ethtool -K veth0b gro off
ethtool -K veth0b lro off
ethtool -K veth0b rxvlan off
ethtool -K veth0b txvlan off
ethtool -K veth0b ntuple off
ethtool -K veth0b rxhash off
ethtool --set-eee veth0b eee off

So the above section is pretty straightforward:

It creates veth0a / veth0b pair , for those familiar with P4 this is similar to the channel betwwen the control plane (freeRouter) and p4dpdk (dataplane) using CPU_PORT
It sets for veth0a/veth0b: multicast/allmulti/promisc flag + mtu=8192
It disable TCP offload for veth0a/veth0b

We are doing the same thing for the wireless network

Code Block

language	bash
theme	Midnight
title	veth1a/veth1b for Wireless network

ip link add veth1a type veth peer name veth1b

ip link set veth1a multicast on
ip link set veth1a allmulti on
ip link set veth1a promisc on
ip link set veth1a mtu 1500
ip link set veth1a up

ip link set veth1b multicast on
ip link set veth1b allmulti on
ip link set veth1b promisc on
ip link set veth1b mtu 8192
ip link set veth1b up

ip link set wlan0 up

ethtool -K veth1a rx off
ethtool -K veth1a tx off
ethtool -K veth1a sg off
ethtool -K veth1a tso off
ethtool -K veth1a ufo off
ethtool -K veth1a gso off
ethtool -K veth1a gro off
ethtool -K veth1a lro off
ethtool -K veth1a rxvlan off
ethtool -K veth1a txvlan off
ethtool -K veth1a ntuple off
ethtool -K veth1a rxhash off
ethtool --set-eee veth1a eee off

ethtool -K veth1b rx off
ethtool -K veth1b tx off
ethtool -K veth1b sg off
ethtool -K veth1b tso off
ethtool -K veth1b ufo off
ethtool -K veth1b gso off
ethtool -K veth1b gro off
ethtool -K veth1b lro off
ethtool -K veth1b rxvlan off
ethtool -K veth1b txvlan off
ethtool -K veth1b ntuple off
ethtool -K veth1b rxhash off
ethtool --set-eee veth1b eee off

ip addr flush dev veth1a 
ip addr add 192.168.128.254/24 dev veth1a

Add default route to SDN999 for OOBM return traffic (192.168.128.1 is freeRouter sdn999: we will see the full config later)

Code Block

language	bash
theme	Midnight
title	#ADD DEFAULT ROUTE to OOBM SDN999

route add default gw 192.168.128.1

Effectively start freeRouter main loop

Code Block

language	bash
theme	Midnight
title	Start freeRouter inside main loop

start-stop-daemon -S -b -x /rtr/hwdet-main.sh

This main loop is triggered by the script hwdet-main.sh below:

Code Block

language	bash
theme	Midnight
title	/rtr/hwdet-all.sh script

cat /rtr/hwdet-main.sh 

#!/bin/sh

while (true); do
  cd /rtr/
  stty raw < /dev/tty
  java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-
  if [ $? -eq 4 ] ; then
    sync
    reboot -f
  fi
  stty cooked < /dev/tty
  sleep 1
done

Warning

title	A bit of explanation

Requirement considerations:

The box should run 24x7
It must survive a power cut, i.e the service should be restored each time the power is cut for any reasons
If no power cut but freeRouter has crashed for any reason it should be restarted

Let me re-assure you, freeRouter usually don't crash, most often freeRouter has manual or better: auto-upgrades

Code Block

language	bash
theme	Midnight
title	freeRouter infinite loop: freeRouter autoupgrade process restarts and self-restarts

while (true); do
  ...
done

The appliance has 8Gb RAM which is enough for JVM running freeRouter. (Full routing IPv4/IPv6 at the control plane is possible at home ! ← ok this is useless but cool no ? :3 )
- RAM allocation is for JVM and it's tables
- Additional RAM allocation is for p4dpdk and p4emu as we have to store also the table once for the native code too
- Lastly the kernel needs also memory so it's good idea to leave some free ram and not give everything to JVM.
The appliance has 8Gb RAM which is enough for JVM running freeRouter. (Full routing IPv4/IPv6 at the control plane is possible at home ! ← ok this is useless but cool no ? :3 )

Code Block

language	bash
theme	Midnight
title	#ADD DEFAULT ROUTE to OOBM SDN999

java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-

We will see later a very cool and unique feature of freeRouter: auto-upgrade freeRouter "Cold reboot"

Code Block

language	bash
theme	Midnight
title	#ADD DEFAULT ROUTE to OOBM SDN999

if [ $? -eq 4 ] ; then
  sync
  reboot -f
fi

...

Expand

title	Design choice considerations

All the choices have been made in order to make the appliance resilient as much as possible and provide an enjoyable user experience. We will see in later article, a feature that I love: auto-upgrade. This will keep your appliance up to date obvr over the network with the latest freeRouter train during low traffic period. Of course, for ISP P/PE core router we don't want this, but hey ! why not ? As soon as all customers are dual homed to 2 different PE reachable via 2 direct core path this can be achieve during low traffic period after having set the metric to infinity on all the PE/P box to be upgraded. (use IS-IS overload bit or OSPF max-metric router-lsa)

...

Blog

Versions Compared

Old Version 2

New Version 3

Key