Musing: Ethernet Connected Hard Drives, Network Designs

I’ve been having thought exercises around the impact of hard disk drives that have ethernet interfaces. The basic concept of connecting a a multi-terabye HDD directly to an Ethernet switch is powerful since it provides options to those very expensive arrays in the data centre. There is obvious value in this idea.

  1. Switch silicon is fast, lossless and reliable.
  2. Ethernet fabric chips make for high speed bus backplanes inside some vendor arrays already.
  3. Disk drives are using I/O standards that hopelessly out of date like FibreChannel or SATA.
  4. magnetic disk drives are moving to low cost, low value solutions while SSDs will take over critical loads.

Possible use cases.

  1. disk spindle could be used for near line backup or long term storage.
  2. house Ethernet drives in low cost cases
  3. consider a Backblaze approach.

Replace the Array with A Ethernet Switch

At a basic level, a storage array is simply a network bus between the disk drives and controller motherboard with a CPU. The controller runs software that presents the disks to other software – like servers as LUNs, or data stream for Dedpulication, or blocks for tape drives.

So, connect HDD to a switch, allocate some x86 servers as a controller. Since we are using off the shelf hard are, the hardest work is mostly done. The relatively “easy” part, of course, is the software that turns this into something useful.


The controller needs to deliver a few key functions.

  1. stripe data across disks for redundancy,
  2. monitor status/integrity
  3. determine the fastest or best access for read/write
  4. value add functions like deduplications.

Comparing with Distributed Storage

Distributed storage is the current direction for low end storage using software CEPH, Lustre or GFS. A X86 server acts as an “Ethernet – SATA” bridge for the disk drives. Enhancements like SSD caching are done locally since it simplifies the software. Storage startups are only capable of limited amount in the few years they have to prove viability and producing a complex SSD caching for this type of system is possibly beyond them.

Switch Inside a x 86 Server

As Ethernet silicon becomes more available, it’s possible that we could see an Ethernet switch inside an x86 server. The Intel FM7000 has excellent networking features and Intel could conceivably provide motherboard that has a switch onboard. The server metalwork could hold a bunch of drives connected over an Ethernet bus.

Could be interesting.

Network View

From a networking point of view, this changes the network design. Most storage arrays are barnacles on the network architecture with a couple of “fat Ethernet” connections and large amounts of traffic that creates congestion points. An improved network design would have the load more equally distributed over many interfaces. Current storage technology creates “elephant” flows that cause all kinds of nasty hacks to be implemented.

Having a directly connected Ethernet switch allows for many new ways to smooth the network architecture.

The EtherealMind View

The idea of Ethernet connected hard drives seems to be a recurrent theme but the storage folks are notoriously resistant to new ideas so I don’t expect this to happen. That said, its interesting to consider this ideas and wonder how networking could be different. Thanks for following along.

  • Joshua Kugler

    While very exciting, it’s not really anything new. Coraid has been doing this for well over a decade.

    • Hans De Leenheer

      Not entirely correct Joshua. Coraid merely wrote their own SCSI storage protocol (AoE) to use ethernet as a backbone. You were still connecting SCSI disks to an operating system like you would previously have done with SAS/FC/iSCSI.

      The advantage of an ethernet connected drive is that you are using just IP to access a virtually unlimited resource from within the application with no complexity layers other than an ethernet fabric.

      Where Coraid needed their certified OSs and specific Physical Cards with specific drivers you now just need to provide IP to the application.

    • Etherealmind

      As Hans points out, Coraid is simply “ATA over Ethernet” for client-array connectivity. In simple terms, Coraid gets some speed benefits and lower costs through less complexity while sacrificing scalability.

      Ethernet is not scalable networking protocol so VLANs must be kept relatively small for safety (to reduce risk of bridging loops).

  • Ronald Pottol

    I wonder about running a Ceph OSD on the drive? That would be rather handy, I think (though I’d need to research if that would be enough), you run the other parts of Ceph on some real servers, but your bulk storage on drives with ethernet interfaces, and their own OSD.

    • Etherealmind

      I wonder that CEPH would optimised for x86 and would need too much power to be effective on the HDD controller board. I would expect to a see a external board plugged into the SATA interface and some custom code for the controller.

  • Econotarian

    I really want to know how object-based storage is going to work with existing applications, especially those which need to index into large files (like video editors)? Or do you have to “blow up” those applications and completely re-write them for object-based storage?

    • Etherealmind

      Some object storage can present ‘block & file’ interfaces. CEPH is an open source version with commercial support, EMC Atmos has very limited capability although VIPR might be able to present Isilon/Atmos as something more useful. There are number of startup in the market too.

      For indexing large files, there are object storage products that are specifically designed for this. I work with a client on a 7PB storage system that stores scientific data and can read/write to the storage system at more than 1Terabyte/second.

      So it’s about finding the storage product that works for your very specific and niche requirements.

  • Etherealmind

    yes, but the storage experts have not been very positive about it. I can’t tell if it storage industry being backward looking or whether there is a genuine technology limitation.