FreeBSD Device Polling

I previously wrote a “How-To” article detailing how to setup VLAN Tagging (802.1q) using FreeBSD and a Cisco 2924-XL-EN. You can read the article for more information, but basically VLAN tagging is a way isolate traffic to particular ports on a switch. In addition that How-To also covers how to use the FreeBSD machine to provide rate limiting and firewall protection to those VLANs.

Anyways… In that article I touch on the fact that we must use device polling to avoid context switching of the kernel back and forth between userland applications and kernel processes. To truly understand the benefits of device polling you should read the device polling web page which explains the reasons behind device polling better than I ever could hope to.

So I never got a chance to go back and write the information on how to set up device polling like I wanted to… Until now.

First some quick and pretty graphs of a P4 server pushing about ~50Mbps (core router for a data center) without device polling enabled:

   1 users    Load  0.00  0.00  0.00                  Aug 29 22:54

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER       Tot   Share      Tot    Share    Free         in  out     in  outAct   16496    4784    24640     5304  645424 countAll  375240    5984  2742020     7080         pages                                                                InterruptsProc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow   16713 total    1        6        10    1   6016714    3    3 103724 wire        stray irq7                                                   16412 act         em1 irq51.9%Sys  21.5%Intr  0.0%User  0.0%Nice 76.6%Idl   255080 inact       em2 irq12|    |    |    |    |    |    |    |    |    |         24 cache  7133 em3 irq10=+++++++++++                                       645400 free      1 ata0 irq14                                                         daefr  1355 mux irq11Namei         Name-cache    Dir-cache                     prcfr       fdc0 irq6   Calls     hits    %     hits    %                     react  1000 clk irq0                                                         pdwak   128 rtc irq8                                         zfod            pdpgs     3 mux irq5Disks   ad0                               ofod            intrn  7093 mux irq12KB/t   2.00                               %slo-z   113712 buftps       1                               tfree         3 dirtybufMB/s   0.00                                         69954 desiredvnodes% busy    0                                         59766 numvnodes                                                      26 freevnodes

Note the long line of “+” signs. Those are IRQ interrupts that the CPU has to handle. At the time this screen scrape was taken, over 20% of the CPU was spent handling IRQ requests from device em3 (right column 7153 IRQ requests).

Now the same exact server with the same exact data flow:

   1 users    Load  0.00  0.00  0.00                  Aug 29 22:55

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER       Tot   Share      Tot    Share    Free         in  out     in  outAct   17504    4784    26020     5304  645424 countAll  375240    5984  2742020     7080         pages                                                                InterruptsProc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    1124 total             8         9    1   50 1124    4    3 103724 wire        stray irq7                                                   16408 act         em1 irq53.1%Sys   0.8%Intr  0.0%User  0.0%Nice 96.2%Idl   255084 inact       em2 irq12|    |    |    |    |    |    |    |    |    |         24 cache       em3 irq10==                                                 645400 free        ata0 irq14                                                         daefr       mux irq11Namei         Name-cache    Dir-cache                     prcfr       fdc0 irq6   Calls     hits    %     hits    %                     react   996 clk irq0                                                         pdwak   128 rtc irq8                                         zfod            pdpgs       mux irq5Disks   ad0                               ofod            intrn       mux irq12KB/t   0.00                               %slo-z   113712 buftps       0                               tfree         4 dirtybufMB/s   0.00                                         69954 desiredvnodes% busy    0                                         59766 numvnodes                                                      26 freevnodes

Now note the difference. The CPU is now free to handle other things instead of IRQ requests. That is the power of device polling.

Here is how you set it up.

First you have to make sure that the network interface you are using supports device polling. The author’s web site linked above lists support for “fxp” (Intel 10/100 cards), “sis” (SiS based network cards – not sure which), and “dc” (Some sort of DEC card). One card which is not listed on that page, but I know supports device polling is the “em” card (Intel Gigabit cards). If there are other cards that support device polling please let me know. I have only used fxp and em cards so I know for a fact that they work.

So now that we know we have a network interface that is going to support polling (it is really the device driver that supports polling) we can get started on adding the support to the kernel. This is really simple and is comprised of two directive in the kernel config file:

options DEVICE_POLLING

options HZ=1000

Once you have those in your kernel config, you can recompile your kernel and boot it up. If you need help doing that part consult the FreeBSD How-To for creating a custom kernel (and you are running a custom kernel with FreeBSD, right?).

Now you have a computer that has a network card that supports device polling, a kernel that is device polling enabled as well, a real time clock that is more responsive (the HZ=1000 line), and the kernel is booted and we are ready to rock and roll. Now we just need to issue:

sysctl kern.polling.enable=1

Now we have it enabled and you are all set to go forth into that dark night and let your devices be polled.

I hope that sheds some light on a subject that can be a of great use for high network traffic servers (routers, firewalls, proxies, etc).

Let me know if you found it useful.

Comments (0)

› No comments yet.

Pingbacks (0)

› No pingbacks yet.