My NAS knew the love was gone, so it went and died on me

billbillw

My trusty Zyxel NSA325 gave up the ghost last night. Went into a boot loop/downward spiral. Short of some magic through TTL serial interface, I don't think I will be able to get it up and running again. I swear, the Zyxel knew I was about to retire it. (more info below).
Have you ever had electronics go out on you like that when you were in the process of replacing them? I swear it has happened several times for me.

Back-story...Not too long ago (early December), I had two NAS boxes in the house, the Zyxel, and a Buffalo. Of the two, the Buffalo was somewhat less reliable over the last 4 years. More frequent locked web-access, file server crash, need for reboot, etc. So, sometime in December, I got a bug up my butt and decided to build a real server to run NAS, Plex and a few other programs along with giving much greater storage and speed. I consolidated all the files that were on two NAS (movies on one, music/photos/files on the other) onto the Zyxel, sold the Buffalo, and started my server search/build.

So, I have been plodding along on this new server build. Due to the economics, I decided to use older semi-retired enterprise equipment (they can be super cheap on eBay). I scored a deal on a couple of older Dell servers. I messed around with each of them and found that I didn't really care for either. They were too loud, and not fast enough, and their remote access features were archaic. Also, one had a 2TB limit to drive size. I wanted to use 4TB drives. I sold them both for a pretty handsome profit and bought an IBM x3630 M4 for super cheap earlier this month. I had plans to slowly build it out and try some different configurations with some sort of virtualization, experimenting with Proxmox VE, VMWare, Ubuntu Server, FreeNAS, etc.

So, now all the parts are together. I've put together a fairly quiet IBM 2U server with Dual Xeon E5-2440, 72GB of RAM, RAID/SAS in HBA mode, and I have 4 x 4TB HGST nearline SAS drives to install in the hot-swap bays. I also plan to re-purpose an unused 128GB M.2 SSD as the boot drive. The server has a dedicated internal USB for a hypervisor, but I don't think Proxmox likes being on a USB stick. I will try ZFS RAID-Z which will give a storage pool of about 14TB (10TB usable) to start and see how that works. I know there are a few homelab guys here that will understand some of this. Wish me luck. Looks like I can't take the slow road now.

msg

Looks like you've been busy, there, Bill, some fun tinkering you've got going on. Always enjoy reading about your computing experiments, and I'm curious to see what you do with this beast of a machine.

Virtualization's fun stuff. How's that RAID array work for expansion? Can you add more storage for expansion on the fly, or do you just add another set of disks and create an additional array?

Lots of nice tools to play with. Is this just for experimentation or do you have any practical purpose in mind?

Any plans for disaster recovery/backup with this configuration? Presuming we're alike in system builds, you'll have some serious time logged in configuration and tweaks. Data's a given, but I also like to have a way to back up any OS builds, as well whenever feasible.

mhardy6647

yeah -- they know.

even in the days before smart hardware -- stuff knew.

On the other hand, my experience with vacuum tube electronics is that it often sounds really, exceptionally good just before there is a pop (and/or sizzle) and a quantum of the magic smoke comes out as some passive component (often a capacitor) gives up its life force.

billbillw

msg wrote: »

Looks like you've been busy, there, Bill, some fun tinkering you've got going on. Always enjoy reading about your computing experiments, and I'm curious to see what you do with this beast of a machine.

Virtualization's fun stuff. How's that RAID array work for expansion? Can you add more storage for expansion on the fly, or do you just add another set of disks and create an additional array?

Lots of nice tools to play with. Is this just for experimentation or do you have any practical purpose in mind?

Any plans for disaster recovery/backup with this configuration? Presuming we're alike in system builds, you'll have some serious time logged in configuration and tweaks. Data's a given, but I also like to have a way to back up any OS builds, as well whenever feasible.

Honestly, I'm in over my head on this one. Not sure what direction it will all take or what all I will use this server for. At a minimum, it will be an NAS to host all my movies, music, photos, and documents. Due to the increase in storage, I will probably go ahead and rip all my old DVDs and BluRays (did my HDDVD years ago). It will also be a Plex Server, torrent seedbox, and a few other minor things. I'll probably do something like Crashplan and maybe Owncloud. I know it should be able to handle anything my 'home use' can throw at it. Twin hexacores (with hyperthreading) and a massive amount of ECC memory. Hope it ends up a good learning experience and not a frustration.

BlueFox

That is why I love direct attached USB drives. It dies, then replace it with a replacement drive. Reminds me to order another backup drive. Good to have a backup for the backup.

billbillw

BlueFox wrote: »

That is why I love direct attached USB drives. It dies, then replace it with a replacement drive. Reminds me to order another backup drive. Good to have a backup for the backup.

Except that I stream to at least 8 different devices/Tv's, some wirelessly...usb wouldn't work well for the whole house.

DarqueKnight

BlueFox wrote: »

That is why I love direct attached USB drives. It dies, then replace it with a replacement drive. Reminds me to order another backup drive. Good to have a backup for the backup.

The rules are:

1. At least one onsite backup.
2. At least one offsite backup for the onsite backup.

billbillw

Got the basics up and running over the last few days after several frustrating issues. IBM firmware just doesn't like my using a crossflashed RAID card. I finally got it to work. Proxmox VE on a small SSD. 15GB ZFS pool done and Turnkey File Server container running. Getting Gigabit wire speed transfers from my main PC, well close, 980Mbps. Files will be transferring files all night to repopulate the 4GB or so I had on the NAS boxes. I'll deal with Plex and other stuff later.

tophatjohnny

Mirrioring drives works wonders!!!

billbillw

So, you probably won't believe this, but I'm crazy. I decided the IBM was overkill. Although it wasn't that loud, I found that my tolerance for any fan noise has really gone down over the years. The server was sitting in my spare bedroom/office where I do a lot of computer work. It was irritating (especially when I had a sinus cold and some ear pressure). I also found that I was really never using more than about 10% of the resources on the IBM.

So, I started looking again...I decided I wanted to stick to a tower style server because most of those have much quieter noise profiles. I wanted the same generation of CPU for several reasons (mostly price/performance), so that had me looking at Dell T320/T420 and some IBM x3300 towers. The Dell towers get away with a single 120mm fan in the rear plus power supply fan(s). The IBM used 2 or 3 80mm fans and I couldn't find any discussion of their noise. Published specs had the IBM being at least 20 dB louder though. Scratch the IBM.

Ended up finding a steal on a T420, fixed 550w power supply (quieter fan) setup with a single E5-2430 and 48GB RAM (I may add a 2nd 2430 at some point).

Got it in house about 10 days ago. I was able to simply pull the ZFS pool drives out of the IBM, mount them in the Dell, and put the SSD as a boot drive connected to the board SATA port. Booted up 1st time with no issues and I didn't have to change anything except some of the network configuration. Gotta love Linux! I guess it helps they are same generation Intel chipset/CPU, same SAS card, etc. The Dell is completely silent once booted other than the sound of mechanical drive seeks/writes. Fan noise is below my hearing threshold.

So, after the swap, I finished configuring Plex and PlexPy (both under Proxmox virtual containers) and Windows 10 virtual machine. Next up, migrate my torrent seed client over. With everything running. I still don't see much more than 30% CPU use, but I have seen up to about 70% memory use when the Win10 VM was running and I was doing a lot of file transfers (ZFS can use a bunch of memory).

Overall, I am much happier with the Dell server. Now just need to sell the IBM...

billbillw

@msg
I didn't answer some of your questions before...I'm not sure if I understood all them at the time. It has been a learning experience. So, ZFS RAID does not allow you to grow the pool size by adding drives to an existing vdev. In that respect, hardware RAID is more versatile. You can add new drives under a separate vdev, but I'm not sure if that is recommended. Another option is to replace all the drives with the same number of larger drives, then expand the storage. It will be awhile before I fill up more than 10TB so hopefully that limitation doesn't bite me. (EDIT: I just read that ZFS Expansion is under development...hopefully a stable feature in a year or two?)

As for backing up stuff, Proxmox has a feature that lets you backup each VM/Container. I'm not sure I have it configured in the best way yet, but it is doing something every few days. I would hate to have to reconfigure everything. I also need to figure out a way to image the SSD. There is quite a bit of configuration there with the Proxmox install that I would hate to loose.