I previously wrote a “How-To” article detailing how to setup VLAN Tagging (802.1q) using FreeBSD and a Cisco 2924-XL-EN. You can read the article for more information, but basically VLAN tagging is a way isolate traffic to particular ports on a switch. In addition that How-To also covers how to use the FreeBSD machine to provide rate limiting and firewall protection to those VLANs.
Anyways… In that article I touch on the fact that we must use device polling to avoid context switching of the kernel back and forth between userland applications and kernel processes. To truly understand the benefits of device polling you should read the device polling web page which explains the reasons behind device polling better than I ever could hope to.
So I never got a chance to go back and write the information on how to set up device polling like I wanted to… Until now.
First some quick and pretty graphs of a P4 server pushing about ~50Mbps (core router for a data center) without device polling enabled:
1 users Load 0.00 0.00 0.00 Aug 29 22:54
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 16496 4784 24640 5304 645424 count
All 375240 5984 2742020 7080 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 16713 total
1 6 10 1 6016714 3 3 103724 wire stray irq7
16412 act em1 irq5
1.9%Sys 21.5%Intr 0.0%User 0.0%Nice 76.6%Idl 255080 inact em2 irq12
| | | | | | | | | | 24 cache 7133 em3 irq10
=+++++++++++ 645400 free 1 ata0 irq14
daefr 1355 mux irq11
Namei Name-cache Dir-cache prcfr fdc0 irq6
Calls hits % hits % react 1000 clk irq0
pdwak 128 rtc irq8
zfod pdpgs 3 mux irq5
Disks ad0 ofod intrn 7093 mux irq12
KB/t 2.00 %slo-z 113712 buf
tps 1 tfree 3 dirtybuf
MB/s 0.00 69954 desiredvnodes
% busy 0 59766 numvnodes
26 freevnodes
Note the long line of “+” signs. Those are IRQ interrupts that the CPU has to handle. At the time this screen scrape was taken, over 20% of the CPU was spent handling IRQ requests from device em3 (right column 7153 IRQ requests).
Now the same exact server with the same exact data flow:
1 users Load 0.00 0.00 0.00 Aug 29 22:55
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 17504 4784 26020 5304 645424 count
All 375240 5984 2742020 7080 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 1124 total
8 9 1 50 1124 4 3 103724 wire stray irq7
16408 act em1 irq5
3.1%Sys 0.8%Intr 0.0%User 0.0%Nice 96.2%Idl 255084 inact em2 irq12
| | | | | | | | | | 24 cache em3 irq10
== 645400 free ata0 irq14
daefr mux irq11
Namei Name-cache Dir-cache prcfr fdc0 irq6
Calls hits % hits % react 996 clk irq0
pdwak 128 rtc irq8
zfod pdpgs mux irq5
Disks ad0 ofod intrn mux irq12
KB/t 0.00 %slo-z 113712 buf
tps 0 tfree 4 dirtybuf
MB/s 0.00 69954 desiredvnodes
% busy 0 59766 numvnodes
26 freevnodes
Now note the difference. The CPU is now free to handle other things instead of IRQ requests. That is the power of device polling.
Here is how you set it up.
First you have to make sure that the network interface you are using supports device polling. The author’s web site linked above lists support for “fxp” (Intel 10/100 cards), “sis” (SiS based network cards – not sure which), and “dc” (Some sort of DEC card). One card which is not listed on that page, but I know supports device polling is the “em” card (Intel Gigabit cards). If there are other cards that support device polling please let me know. I have only used fxp and em cards so I know for a fact that they work.
So now that we know we have a network interface that is going to support polling (it is really the device driver that supports polling) we can get started on adding the support to the kernel. This is really simple and is comprised of two directive in the kernel config file:
options DEVICE_POLLING
options HZ=1000
Once you have those in your kernel config, you can recompile your kernel and boot it up. If you need help doing that part consult the FreeBSD How-To for creating a custom kernel (and you are running a custom kernel with FreeBSD, right?).
Now you have a computer that has a network card that supports device polling, a kernel that is device polling enabled as well, a real time clock that is more responsive (the HZ=1000 line), and the kernel is booted and we are ready to rock and roll. Now we just need to issue:
sysctl kern.polling.enable=1
Now we have it enabled and you are all set to go forth into that dark night and let your devices be polled.
I hope that sheds some light on a subject that can be a of great use for high network traffic servers (routers, firewalls, proxies, etc).
Let me know if you found it useful.
Comments (0)