[SDBUG] OpenBSD on Dell issues ?

Thu Aug 16 08:17:23 PDT 2007

While OpenBSD has made progress on their SMP support, FreeBSD's work over
the past few years in the development and stabilization of the 5.0 and 6.0
branches was primarily about highly scalable SMP support.  If you're working
with a dual core system, it wouldn't suprise me that FreeBSD is better able
to schedule interrupts to service your network traffic.

I'm joining the conversation late and I don't see anything in the quotes for
this email regarding which ethernet chipset you're talking about, but what
you're looking for in this situation is interrupt moderation.  Polling
interrupts is not the solution.  If you have an intel chipset based driver
"em", that driver does support interrupt moderation.  This will allow you to
reach much higher speeds by not overloading your CPU with interrupts.  The
key here is to not have one interrupt per packet, which is what happens
without interrupt moderation.  Since the driver will see *all* the packets
the network card has received when an interrupt is serviced, it makes no
sense to have one interrupt per packet.  So on cards that support interrupt
moderation, depending on settings and the overall traffic level, 3,000-4,000
packets can be queued for each interrupt that fires.  This will cause a huge
drop in CPU load.

This is probably also related to the performance difference you see in
FreeBSD, because I am guessing the em driver defaults interrupt moderation
to "on".  Even if your ethernet is not an Intel chipset (Broadcoms are
really common on my Dells right now), most of the mainstream drivers should
support interrupt moderation.  For more information on interrupt moderation
see this link:

http://download.intel.com/design/network/applnots/ap450.pdf

I would say that using FreeBSD for your firewall is a perfectly good
solution.  If I was building a firewall personally and I was worried about
performance, I would probably buy a dual or quad port Intel gigabit
PCI-Express ethernet card, although it might not be necessary.  If your
system is PCI-Express based (which it can be even if you have PCI / PCI-X
ports, Dell sells mixed riser cards), then your built in gigabit ethernet
(probably Broadcom) is already linked through PCI-Express.  If that's the
case, don't add a PCI ethernet card and use that.  If your system has
PCI-Express slots, you can add on a PCI-Express Intel card, but again
probably not needed until you're pushing the performance levels or need more
ports.

----- Original Message ----- 
From: "Michael J McCafferty" <mike at m5computersecurity.com>
To: <sdbug at sdbug.org>
Sent: Thursday, August 16, 2007 12:55 AM
Subject: Re: [SDBUG] OpenBSD on Dell issues ?

Can,
    Yes, I know that the 32-bit PCI slot will be the bottleneck before
the 1.4G extrapolated max. It wouldn't be much fun to have the
firewalls at 100% at any time anyway. I'd be looking for an upgrade
long before the daily peak CPU was even 70%. Daily we are at 40% now,
and I am feverishly working to make the upgrade.
    ...and this is combined bandwidth on all interfaces, full-duplex.
Currently the networks plugged in to these firewalls are maxed at
600Mbps for now... and we aren't using near that yet. When we do get
get closer to that, we will start upgrading the Internet connections
to GigE.
    The problem is that even a moderate DoS causes too many
interrupts. I mentioned polling because ultiamtely there has to be
some way to mitigate the interrupts. Can't go 10G if the interrupts
stop you at 1.4G
    Now, the other night I decided to try FreeBSD 6.2-RELEASE in place
of the OpenBSD snapshot box. It reported that it was about HALF the
CPU interrupt time as the OpenBSD boxes. Is this because the kernel is
letting both cores handle interrupts ? Or is this an error in the way
that the kernel is reporting interrupt time at the CPU ? I wasn't able
to generate enough pps or mbps with my 4 traffic generating boxes
(realtek NICs) to get it over 50% so I can't tell if it's reporting
correctly or if it's actually doing it.
    Since FreeBSD has PF, CARP and pfsync, that might be an option. No ?

    Thanks, your comments are appreciated.

Mike

Quoting Can Erkin Acar <canacar at gmail.com>:

> On 8/13/07, Michael J McCafferty <mike at m5computersecurity.com> wrote:
>> Well... I got the hardware, and the results are in:
>
> Nice, and good results. There is a problem with your extrapolation
> calculations though. The peak theoretical bandwidth of PCI (32 bit) is
> about 128MB/sec which is equivalent to 1Gb/s, so you will not be able
> to reach 1.4Gb using 32 bit slots. It is probably ok though, since the
> interrupts are the real problem that limit the pps rate, usually at a
> much lower bandwidth with small packets. And this is what matters with
> DoS attacks.
>
> For a good overview of PCI technologies you can have a look at:
> http://arstechnica.com/articles/paedia/hardware/pcie.ars/1
>
>
>> I have a more complete spreadsheet, but here are the highlights:
>>
> [snip]
>>         I think it is relevant to note that interrupts were handled  by a 
>> single
>> core of the dual core CPUs in each case.
>
> Yes, only userland benefits from multiple processors at the moment.
>
>>         What I have learned from this is that when it comes to PCIe or 
>> PCI-X
>> and 133MHz 64-bit vs. 32-bit PCI... it's not really that relevant for
>> x86/x86_64/amd64 firewalls with OpenBSD. The bottleneck is hardware
>> interrupts to the CPU. I actually already knew that the interrupts were
>> the bottleneck, but what I didn't know was how 1333MHz FSB vs. 1066MHz
>> FSB, or 32-bit PCI vs. PCIe, or OpenBSD 3.8, OpenBSD 4.1 or the latest
>> snapshot would affect the final performance. Of course I also didn't
>> know the difference between Xeon 5130 and Core2 Duo E6600 for interrupts
>> either.
>>         There is one faster model of CPU available for the Dell system. 
>> The
>> 3GHz Woodcrest CPU is about $500 MORE than the CPU tested, each. *If* it
>> scales linearly, this would bring the performance to ~2Gbps and
>> 400,000pps. This is still 100,000pps short of my desire to handle over
>> 500,000pps (about 48Mbps of 12Byte UDP packets, a relatively small DDoS
>> attack). Really, I'd like to be able to handle 100Mbps+ of 12Byte
>> packets, but I realize a million pps is pretty steep.
>>
>>         Why oh why won't someone smarter than me work on device polling 
>> for
>> OpenBSD ? Or is that even the solution ?
>
> I dont claim to be an expert, but claudio@ says polling is not the
> solution (it may actually increase latency). A good network card and a
> good matching driver would probably perform much better.
>
> Some developers are working on improving the network performance.
> There are also several 10Gb card drivers being worked on, so we really
> aim to support higher speeds. The latest profiling and tuning effort
> at the last hackathon was a start.
>
>
> Enjoy your new setup :)
>
> Can
> _______________________________________________
> SDBUG mailing list
> SDBUG at sdbug.org
> http://lists.sdbug.org/mailman/listinfo/sdbug
>

-- 
************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
858-576-7325 Voice
http://www.m5hosting.com
************************************************************

_______________________________________________
SDBUG mailing list
SDBUG at sdbug.org
http://lists.sdbug.org/mailman/listinfo/sdbug