Monday 7th September 2015: 2.50pm. Link shared: https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/
Came back from a short holiday in Dingle on the Irish west coast to find yet another failed hard drive, this being the third failed drive this year! Thankfully, still no data loss thanks to ZFS, but losing three hard drives in a single year is definitely a record :(
The failing drive this time is the second Western Digital Red 3Tb WD30EFRX to go this year, and the cause is the same:
5 Reallocated_Sector_Ct 0x0033 189 189 140 Pre-fail Always - 323
7 Seek_Error_Rate 0x002e 197 197 000 Old_age Always - 486
9 Power_On_Hours 0x0032 077 077 000 Old_age Always - 17436
At 17k hours powered on, this is the oldest drive in the array after the previous WD Red failed (which at the time was the oldest WD Red drive). I'm getting the sense that these drives tend to quickly die with age, so I checked https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/ and BackBlaze finds an average 7.93% failure rate per three months with little change across quarters (i.e. it's constant). Make that annual, and you will lose about one third of your WD Red 3Tb drives per year.
I have six WD Red 3Tb drives with these powered on hours in order of array position: 6958, 12718; 17437 (failed), 17423; 3745, 3549. That makes their average age 10305, or 1.175 years. That suggests statistically I should have lost 2.35 drives on average, so the fact I have lost two is about right.
WD replace the drives under warranty for three years, after that you are on your own. So in this sense the rate of failure has cost me time and inconvenience and stress, but not much money ... yet. Nevertheless, this rate of failure and the BackBlaze results suggests handing over more of my money to Western Digital for crappy unreliable drives makes no sense despite them being called "NAS ready" drives. In other words, it's time to change my standard drive choice.
Back when I started the ZFS pool three years ago, there was very little empirical information to work with regarding choice of drives, though I knew Seagate drives at the time were appallingly unreliable thanks to very high shop returns from infant mortality (BackBlaze has since found you will lose three out of four of that generation of Seagate drives per year!). Mixing in different models might seem a way of spreading your bets, but ZFS gets upset if the drives in a mirror aren't exactly the same size if you do a swap, so per mirror pair your life is easiest if they are identical which makes spreading your bets difficult as losing a mirror pair means losing everything. One does of course never buy the mirror pair at the same time lest you get two of a bad batch, so I always bought with a three month gap. And due to Seagate drives at the time being known to be awful, I chose the WD Red 3Tb as it was (and still is) the largest three platter drive available for a reasonable price - the NAS anti-vibration sensors etc were a nice value add, but to be honest ZFS makes even the cheapest desktop drives no worse than fancy and expensive server grade drives. The reason for choosing three platter designs was under the assumption of fewer moving parts probably means more reliability ... usually.
The BackBlaze statistics weren't available then, but they are now and there is good year and a half worth of data in there. So now I know that the 3Tb Hitachi Deskstar 5K3000 - launched same time as the WD Red 3Tb - is nearly ten times more reliable. Unfortunately, Hitachi's hard drive division was purchased by WD and they instantly cancelled that model which is no longer available, so the next best low idle power drive not crazy expensive model known to BackBlaze is the Seagate Desktop HDD.15 ST4000DM000 4TB with a 2.87% average rate of failure per quarter, or a one in ten annual rate of failure which is about three times better than the WD Red 3Tb.
That Seagate is still three times more likely to fail than the HGST Megascale 4000.B drive, but it costs at least €280 versus €140 for the Seagate. Which pretty much seals the deal: the ST4000DM000 4Tb drive is going to be my new standard ZFS array drive model choice from now on. Let's just hope I can swap that in before the other WD Red drive in that mirror pair fails!
#nas #zfs #harddrive