ZFS Storage Hardware
Above picture shows Dell 2950 with SSD's mounted in CDROM area
For the hardware for the ZFS storage unit used a spare Dell 2950 unit with 32GB RAM and 6x2T SAS drives. This unit has two x5450 CPU's.
I've included 2 Intel 320 SSD's in the build. One is use for the ZFS write-log (SLOG) and one is used for L2ARC. SSD considerations included:
- SLOG - All synchronous writes are written here and to RAM. (Write-only) They are never read unless the RAM is unavailable due to a crash or power failure. Since there is a lot written here, the SSD is considered a consumable part and will fail when the write limits of the device are exceeded. It is best to purchase extended wear Single-Level-Cell (SLC) flash when you can afford it. Another technique is to allocate only a small percentage of the flash to spread the wear out among a larger number of unused memory cells. In later ZFS releases a failure of the SLOG causes the ZIL to move transparently back to the pool, so it is no longer necessary to mirror the SLOG.
- L2ARC read-cache. The larger the better. Multi-level-cell (MLC) devices work fine for the L2ARC.
In this case we have specified (2) Intel 320 SSD's. If/when the SLOG device wears out we can swap the L2ARC device for use as a SLOG until new parts are available.
All RAM beyond the first 1G or is used by ZFS for the Adaptive Replacement Cache (ARC) and helps speed up read operations. In this case we have about 31 G available for ARC, and 80GB of L2ARC in the SSD. This means that about 110G of working set will fit into the ARC + L2ARC combined.
- Dell 2950 (2 x5450 cpu) 32GB RAM 6x2TB SAS ( Approx $2000 on Ebay )
- Dell PERC 6/i disk controller replaced with dell SAS 6i/R JBOD controller ( Approx $25 )
- 2 x Intel 320 MLC SSD review 80GB ( Approx $100 each )
If price is not a concern, then the following substitutions may be desirable:
- Use a 24 bay or larger Supermicro chassis for more disks.
- Add additional RAM
- Add additional SSD's for L2ARC ( as large as your working set )
- Use an 8GB ZeusRam device for the SLOG This is RAM backed up by flash on power failure. Indefinite lifetime, much faster, but much more expensive ( $3k each ).
This unit had an existing Dell PERC 6i disk controller which handles RAID 0,1,5,10,50 but does not support real JBOD. For ZFS this is a problem since ZFS handles RAID in software and we really don't want another RAID controller between ZFS and the disks. I swapped out the PREC 6i for a Dell SAS 6i/R RAID controller, which supports JBOD mode.
POSIX requires that Sync writes be written to permanent storage before they are acknowledged. So instead of waiting for writes to the pool to finish, they are written to RAM and the SLOG SSD.
Since the ZIL is flushed to the pool every 5 seconds, on 1Gb network it can't grow more than 5x125Mb or allowing twice amount this for buffering 2GB is plenty.
Since ESX uses synchronous writes over NFS, using a SLOG SSD will speed up write performance. In addition since the drives are 7200 RPM NL-SAS disks, adding L2ARC read cache SSD would also improve the performance of the box.
I purchased two unused Intel 320 80GB SSD's on Ebay for about $100 each. The Intel 320 drives have internal capacitors that help insure that SSD writes are finished even if power is interrupted during a flash write cycle.
More info about Power Loss protection in Intel 320
Installing the SSDs is a little work since there are no drive bays left in the 2950. I wound up running two 18 inch SATA cables from the MB to the removed CDROM tray unit. This pictures shows the original CDROM SATA cable removed from the MB and the two new cables installed and routed into the shrouding around the edge of the unit.
The new SATA and power cables are routed to the CDTRAY location. A power splitter is require for the 2nd SSD
The two SSD's will just fit stacked into the slot previously used by the CDROM tray.