Very interesting story from the front lines here where a lot of effort finally discovered the TOE was causing a major problem:
This guy probably spent hundreds of hours testing and researching this problem. He finally admitted to a rather drastic solution, removing the TOE chip from the NICs in multiple servers. From my own research, the firmware on the card can be “problematic” and when the kernel driver is enabled (in Linux or VMware), odd behavior can sometimes be observed, including dropped packets, resets or suboptimal performance. But there’s lots of controversy surrounding this issue.
via Revenge of the TOE - Packet Pushers.
I’m hearing a lot of reports of the problems with TOE drivers and hardware. A recent podcast with Jim Gettys about Bufferbloat also was a problem:
NIC Offload engines generate bursts of line rate packet streams at multi-gigabit rates. These features are now “on” by default even in cheap consumer hardware including home routers, and certainly in data centers. Whether this is advisable (it is not…) is orthogonal to the reality of deployed hardware and current device drivers and default settings.
I’m beginning to think that TOE might be something to avoid. It’s also worth noting the latest generation Intel processors with DPDK make TOE unnecessary. And CNAs for FibreChannel.
The times are changing.