Anger...rage...sadness...

ZLTFUL

Just venting.

Yesterday, I was getting my server and amp racks sorted in the basement.
I had everything shut down (media server, storage arrays, etc) and got all my work completed and cable management done.

Started powering everything back on and one of my 16TB (12TB technically as I run RAID 5 for parity) storage arrays went into alarm mode with 2 of the drives showing "red" LEDs.

Got the server up and running and logged into that array's controller to see what the problem was... 2 drives failed S.M.A.R.T. self test. Great...that usually means physical hardware failure.
I pulled the 2 suspect drives and threw them into my diagnostics docks and sure as heck...both drives have massive amounts of bad sectors. And with each subsequent pass, the bad sector count increased which means a read arm is most likely dragging on the platter and ripping grooves into the surface.

Long story short, RAID 5 can survive a single drive failure...but if 2 drives fail in a 4 drive RAID 5 array, the result is data lost.

So total damage:

Drives are under warranty so free replacements, not a big deal...just time to ship them back and for HGST to send me 2 new drives.

Complete loss of entire music library. I have all of the physical media so this just boils down to time...but this is a LOT of time to rip all ~1400 albums (CDs) I have.

Loss of ~400 BD rips. Again, I have all of the physical media but it takes about 30 minutes per BD rip.

So it looks like all of my free time will be spent re-ripping that media once the new drives arrive and I can rebuild the array.

Paradoxex

Were these new HDDs? I have never had a disk fail like that, but I know it is not super uncommon.

I am so sorry for the loss. At least it wasn't something irreplaceable. Well, the time will be lost, but it could have been family pics.

mrbiron

Yuck!
This is my biggest computer fear. Good luck and god speed.

Pay your nephew (if you have one) a $1 a movie and $.25 a CD

ZLTFUL

New drives a year or so ago...it happens. But I went with Hitachi/HGST drives because they are consistently the highest rated reliability drives.

Of my 2 nephews, one I wouldn't trust with a BB gun, let alone computer hardware. And the other is worse...and young...and ADHD...hehe

mrbiron

It's just odd that 2 locked up at the exact same time, isn't it?

ken brydson

Didn't Russ help with the electrical? Coincidence? I think not...

mhardy6647

Literally speaking, what are the odds? I thought the whole deal with RAID arrays was vanishingly low probability of catastrophic failure.

Reading this thread makes me think of how brave the cadre of astronauts who volunteer for the mission to Mars in the present era will be.

Vangard I, this is Houston. Sorry to say you just bluescreened. Good luck and godspeed. Over and out.

<Dang. We shouldn'ta bought them grey market refurb computers from Newegg, Larry>

I do, in all seriousness, feel for the OP, though.

lightman1

ken brydson wrote: »

Didn't Russ help with the electrical? Coincidence? I think not...

Hold on there, sport! I just bring it to the receptacle/outlet. Anything after that is the customers deal. By the way, those computer things scare me.



And.....dammitall Ryan....that bites the big one. Drinks are on me....

Moose68Bash

@ZLTFL,

Sorry to here about your problem and loss of time and effort.

I guess all of us fear this kind of problem after going to a NAS. Yours seems particularly large.

My Synology 713+ has two 4 TB Western Digital Red HDs, one of which is simply a mirror image of the other. I also have a 4 TB external USB HD connected to the NAS for backup of the NAS.

I am not sure whether the "backup" external HD would prevent the loss of data that you experienced, but that is the reason I connected it, even though I also had RAID protection in the configuration of the two drives in the NAS itself.

Willow

[ / quote] . Drinks are on me....

[/quote]

I hope you don't mean literally.....

nspindel

Sorry you got hit so badly, as you say it's fine if 1 drive fails, but if 2 fail at the same time you're dead in the water. That's why I look at Raid as a convenience not as a backup. I've got two 6TB drives in a Raid mirror. That's great if a drive fails, but what if a power surge takes out all the drives? What if there's a fire? I also have all of my data backed up to separate drives that are sitting in a fire safe.

EndersShadow

Ryan:

So sorry to hear.... explains a few things on this end (you know what that means lol).....

I am scared of the same which is why RAID continues to scare me and I've kept up with single drives just added to the library.....

I "might" be able to help save you some time... Email me to discuss further....

ZLTFUL

RAID 5 is one of those exceedingly rare failure things.

But I honestly think it was completely my fault. I shut down the storage array but I did not remove the drives from the enclosure. It did get jostled around a bit while I was relocating it but I didn't think it was a hard enough jolt.

In total, I have around 64TB of storage array capacity and that is split between RAID 5 (Parity) and RAID 1 (mirrored).

As far as backup, only home videos, photos and both my wife and I's devices are actually backed up to our 8TB NAS and the truly irreplaceable data is also backed up to a drive once a month that is stored in our safe deposit box.

Power surges aren't really a concern as the house has a whole home surge protector and the storage arrays are split between 2 APC 3kva UPSes on a pair of dedicated 30A circuits.

Also, Russ didn't do the electrical. I was a sparky in a previous life and only asked Russ to double check my work.

Anyway, no use crying over spilled beer...just need to get more beer to cover it if one does get spilled.

mhardy6647

Be safe and be prepared, kids.

Emergency backups of beer and other fermented and/or distilled beverages is always a best practice.

ZLTFUL

mhardy6647 wrote: »

Be safe and be prepared, kids.

Emergency backups of beer and other fermented and/or distilled beverages is always a best practice.

Beer fridge in the garage, daily drinkers in the kitchen fridge and cellarable beers in the basement "wine cellar"...can't be much more prepared than that...

msg

man, rotten situation, Ryan. feeling for you, as I'm sure is anyone else who's had similar experience.

quick note to everyone reading - Ryan and industry people know this, and do the best they can to do so with resources available, but perhaps not everyone else - RAID is no substitute for backup; you need a backup solution in place.

RAID offers a few different benefits, one of those being a sort of second chance when hardware fails, but you still need a backup in case of catastrophic failure, or something like cryptolocker type viruses. RAID won't help you here. As detailed, Ryan's keeping an off site copy of his most treasured data. 64TB of data is difficult and expensive to cover in even for some businesses, given limited budgets and little consideration for disaster recovery, let alone for home users.

thanks for sharing your experience with this. I feel for the maddening loss of that music collection. cool of you to share as a nudge to others to take a look at how we're protecting our data.

lightman1

ZLTFUL wrote: »

mhardy6647 wrote: »

Be safe and be prepared, kids.

Emergency backups of beer and other fermented and/or distilled beverages is always a best practice.

Beer fridge in the garage, daily drinkers in the kitchen fridge and cellarable beers in the basement "wine cellar"...can't be much more prepared than that...

And I know where each one is and have personally depleted some inventory.

Yeet!

EndersShadow

lightman1 wrote: »

ZLTFUL wrote: »

mhardy6647 wrote: »

Be safe and be prepared, kids.

Emergency backups of beer and other fermented and/or distilled beverages is always a best practice.

Beer fridge in the garage, daily drinkers in the kitchen fridge and cellarable beers in the basement "wine cellar"...can't be much more prepared than that...

And I know where each one is and have personally depleted some inventory.

Yeet!

Speaking of which.. you've got some leftover inventory over here still bro lol...

And @ZLTFUL how much does one of those battery backups run? I'm sure its WAY outside my price range, but curious...

dee1949

... Like having an Audio Brain transplant !!!! Feel for YOU !!

GospelTruth

Another route is to use something like Amazon Glacier services to back up data to the cloud. This solution is not an immediate retrieval of information like Dropbox or similar cloud services, but it is a way cheaper backup plan for things like family photos and even music if you want to do so in the cloud.

For about 1TB of data backup it would cost $4 a month ($.004 per GB per month). Retrieval is where it costs more if you need to get your data back, but it's reasonable at $40 for the 1TB example. Not a bad insurance policy if you lose your NAS in a house fire, flood or what have you. It's a great service for things that are irreplaceable like family photos. In the case of the original poster he mentioned he had 1400 CDs. Even if they are uncompressed at 700MB per disc (.7GB), that amounts to 980 GB of data or roughly 1TB. Paying someone .25 per disc to rip them would still be $350. And if you do it yourself, that's a lot of time. For $40 to retrieve and $4/month as an offsite backup that's not a bad deal.

I know that the Synology NAS systems have Amazon Glacier as a built in application that can be installed and configured so you can do those backups right from your NAS. I have a Synology NAS (model 415+) with 4 4TB drives in mirrored RAID. So I have about 8TB of storage on the NAS. After reading this thread, I think it's time to set up Amazon Glacier to back up some of those things that can never be replaced. Even with RAID, things can still go wrong.

Erik Tracy

When I worked as an engineer for Govt data centers, the biggest fear the crew had was when we had to do power shutdowns for various reasons - usually a facility power upgrade or UPS issue.

Drives, servers, network devices - all subject to being dead on power on.

A lesson to learn is that RAID is NOT a backup strategy. Lots of people believe RAID is a type of backup - no No NO!

It enhances "availability"(eg uninterrupted access w/o downtime) in the event of disk failure (RAID 5 - 1 parity disk, RAID 6 - 2 parity disks), and can enhance performance.

ALWAYS have your data backed up to a separate data 'store' (another RAID, or a single disk), a USB stick, or even tape!

The other lesson to learn is to VERIFY that your backup strategy works! Saying you have backups does not guarantee you can actually restore. Try it out before saying you have backups.

I feel your pain...truly.

tonyb

Tragic man, really sucks.

I'm not a computer nerd, but if your going to rip that much wouldn't it make sense to back everything up to a separate HD then unplug it from the system and just store it on a closet shelf ?

That way even if your system takes a direct hit by a meteor, you still have your stuff.

Erik Tracy

That's what I said.

tonyb

Erik Tracy wrote: »

That's what I said.

Sorry Erik, didn't read all the responses.

Yeah, what he said.....how's that ? lol

Erik Tracy

No biggy...story of my life

EndersShadow

tonyb wrote: »

Tragic man, really sucks.

I'm not a computer nerd, but if your going to rip that much wouldn't it make sense to back everything up to a separate HD then unplug it from the system and just store it on a closet shelf ?

That way even if your system takes a direct hit by a meteor, you still have your stuff.

The cost to do that would be quite pricey given the size of his array.....

Right now each HGST 4TB drive is 150 (and they are on sale).... so he would need 16.... That right there is 2.4k.... and then we are not even talking about the actual equipment needed to create and manage the array....

lightman1

Y'all are a buncha nerds....

ZLTFUL

I agree Russ...but then if someone were to mention LED streetlight modules, you would be pooping yerself for joy that you could geek out along with them. Hehe.


Don't worry guys. The stuff that can't be replaced is backed up, verified and kept off site (the backup). Most of it has 3 different copies of it somewhere...i.e., photos and digital documents live on the NAS, the NAS is mirrored and a backup of the NAS is taken once a month and stored off site in a safe deposit box.

This is more of a frustration than anything as I have all of the physical media and just need to re-rip approximately 1/3 of my BD library and all of my music. But the music will now be backed up to the NAS as well just for convenience sake and since we have more than enough space for it.

The BD library will continue to not be backed up because it just isn't cost effective to do so, so I will continue to run in RAID 5 for those arrays.

I appreciate the thoughts and suggestions but as Dan and others pointed out, it just isn't going to be cost effective to have a duplicate of each storage array. Unless one of the 3 Facebook datacenters down the road from me wants to give me some cloud storage love...hehe.

Erik Tracy

ZLTFUL wrote: »

I agree Russ...but then if someone were to mention LED streetlight modules, you would be pooping yerself for joy that you could geek out along with them. Hehe.


Don't worry guys. The stuff that can't be replaced is backed up, verified and kept off site (the backup). Most of it has 3 different copies of it somewhere...i.e., photos and digital documents live on the NAS, the NAS is mirrored and a backup of the NAS is taken once a month and stored off site in a safe deposit box.

This is more of a frustration than anything as I have all of the physical media and just need to re-rip approximately 1/3 of my BD library and all of my music. But the music will now be backed up to the NAS as well just for convenience sake and since we have more than enough space for it.

The BD library will continue to not be backed up because it just isn't cost effective to do so, so I will continue to run in RAID 5 for those arrays.

I appreciate the thoughts and suggestions but as Dan and others pointed out, it just isn't going to be cost effective to have a duplicate of each storage array. Unless one of the 3 Facebook datacenters down the road from me wants to give me some cloud storage love...hehe.

Well, you are the best judge of what your time is worth.

The risk of mirrored NAS is - you corrupt the primary, you corrupt the mirror as well depending on the sync policy you have setup. You f' up the primary, you f' up the mirror as well, and what good has that done you?

All I can say is, in the "Enterprise" world - the more data you have, the more important it is to have a true backup you can restore from. Cuz restoring from primary data sources is a b#tch. But, that is your time.

tonyb

At least do the music, the movies you'll always have on a disc, right ? The movies are what takes up all the space anyway. Not expensive to do music on a separate HD.

msg

Ryan, how much of that 64TB are you actually using?
Some of this you'll already know, but including for the others who may be following along.

Wondering if you could, indeed, as Tony suggests, get away with backing up to some larger WD 8TB USB 3.0 externals, maybe using a small file sync utility like SyncBackPro or FreeFileSync for the copies?

Example, on my smaller home system, I use Acronis Backup Advanced to make incremental backups of the PC to an always connected external, archives split at 700MB to keep them a manageable size for copies, uploads/downloads. Then, midweek, I use SyncBackPro to mirror the backup archives to a second external that I keep offsite. Using SBP, it copies only the changes, so after the initial backup, updates are fast. SBP also has Cloud Storage functionality built in, which I should probably be using, so you can send up to Amazon, Google, OneDrive, etc. Works the same way as local disk to disk backups - copies only the changes, so updates are fast. Offers scheduling, too, for automation.

I've only just started experimenting with high-res .flac, and don't have movie rips, so my data set is relatively small at the moment, less than 4TB total, so it's pretty easy to manage. Consists of a base backup image and subsequent incrementals taken each night to include the PC installation, photos, software store, and old .mp3 files I collected over the years.

As my data set grows, it will become more challenging to protect. I plan to add a small storage array at some point, but that will also require some method of off site backup as well. Have a few different ideas there. Drobo, Synology and Qnap are some options, and, as noted above, some offer built in replication to cloud storage, or even copy to a locally connected external.

As noted above, for data that doesn't change often, AWS S3 or Glacier can be a good option. I know a small business consultant who uses CrashPlan and likes it; various plans, affordable.

For most people, doing RAID properly can be expensive. I tested a few different "affordable" options in the past using consumer drives. Most were clunky and slow, or ended in corruption of the array, mostly in the DIY versions, taking an old computer, loading it up with drives, and some Linux based storage platform. It's gotten a lot better, I imagine even some of the software based ones if you want to DIY, but as everyone reading understands by now, always have a backup.