hard drive sizes for the core

[drjenk]

Hello,
When buying hard drives for the core, is it best to get a smaller HD for the linuxMCE OS, and reserve the large drives for the media, or does it matter?  I think I may want a raid 5 config eventually for the media only, but I want to just get it up and going first with 2 drives (one os and one media), then expand later. 
Which leads to another question: If my initial configuration is 1 OS drive and 1 media drive, and later I want to expand my media storage to use raid 5, my understanding is I would need to get 2 more identical drives (minimum, 4 more optimal), and format them to use raid 5.  Can linux support a configuration where one drive is non-raid, and another set of drives is configured raid?

Thanks

[seth]

 
First, welcome.

Now as to drive configurations. I set up this way. I have a 500G drive as the core OS/TV drive. I also have 2 external 500G drives attached via USB to the core. These drives are specifically for movies, and music. My reasoning for this is:

1.) Main drive large enough to hold MD images, need a good bit of space for each MD you deploy, and a central location for my TV/Dish recordings from mythtv. Now that the installation DVD asks to if you want to save your "Settings" and "/home" this is perfect situation for upgrading/re-installing

2.) I use the external drives, mounted and maintained by LMCE, for video and music storage. This is great because when you go to rip a dvd or cd, or download a torrent, they can be stored (off site) from the core. This can also be accomplished with an internal RAID set up, just let LMCE control it. When you are actually ripping media, it asks you where you want to store it, I just choose one of my two external drives. And its as simple as that. 

Some notes:

Internal OS/TV drive:
Things to consider here.
1. How many MD's are you going to deploy, you'll need space for them, about 20G each as a safe number.
2. TV recordings.
Here it gets tricky. If you are like me and capture your TV from a Set top box (STB), then figure 2.1G an hour. But I also have 2 Free to AIR (OTA) HD TV cards. These recordings vary in size from 5 to 7.1G an hour. If you choose to use the OS/TV setting (highly recommended) then you will need a drive to support this type of storage capacity, depending on your situation.
The reason I choose to store TV on the core, and everything else on external media is because, the core works best, at least for MythTV, to store the recordings on the core, usually down /home/public/data/videos/tv_shows_1 or whatever number your TV cards are on. For instance if you had tuner cards on moon48, then the last part of the above path would be tv_shows_48. But they are still stored on the core, even if you have a drive in the MD, which is not required.

Finally , in my case, I use Western Digital MyBook's for my external storage solution, and they can be plugged into any MD in my house, I just keep them at the core, to centralize everything. Now whenever, or wherever I am in the house, when I rip media, I point it to be stored on the external drives. And then they are available everywhere there is an MD, and they do not effect my TV recording storage.

Just my particular case. You will find that if you let LMCE manage everything for you, your media will be available anywhere in your particular deployment. And they can be easily managed from there as well.

Just my 2 cents


Regards,

Seth

[skeptic]

Oddly enough, my setup is almost exactly the same as Seth's, and for the same reasons.  500G internal, 1 750G USB MyBook for other media...  Recording from STB to internal drive via MythTV.  I don't have any OTA HD cards though.  The other difference is I rip my media on a desktop and copy it over instead of ripping via LMCE.  I do this because all my music was ripped before I started with LMCE, and I prefer handbrake for movies.  Much better compression using x264 and no annoying previews/legal warnings/menus, just the movies.  I may go back and let LMCE rip my music just to let it get album art and better id3 tags.

[drjenk]

Great info.
But why not put an extra drive in the core, instead of usb? 

I'm thinking my hybrid will be a tower, to allow for extra disks in the future, then maybe after that make it a dedicated core and get another MD.  I want a raid 5 config eventually because I have about 80gb of pictures so far and I would really really hate to lose them.  They are on a DNS323 in RAID1 config right now, along with my music projects, which are quite precious to me as well.  But raid 1 is quite inefficient, so my plan is to buy enough drives eventually to configure raid5, then use my dns323 as extra media storage for linuxmce.  So I'll probably just start off with a single 500gb drive in the core and expand from there, seems like it works ok for you guys.

Thanks for the input.

[chipppy]

RAID 5 is a good idea in the long run but becareful not to get to big a drives, or you could start running into the bit failure rate problem with RAID 5 (search for something like: raid 5 2009.  ZNET has a good article that explains it all.)
Personally I would use 500GB drives and this should help to stay away from this problem.  Personally I back up to a 1TB USB drive that sits in my cupboard every now and then for all my must keep files.  (pitcures of the kids etc)

[alx9r]

I just finished researching then upgrading my system with more storage.  FWIW, here is what I found:
 - There are a few sad stories about media lost from RAIDed drives connected to the core.  These stories were enough to steer me away from a RAID solution built-in to the core.
 - putting more hard drives in my hybrid/core is probably more noise and heat than I ought to have in a case that has to live in my living room
 - any RAID solution raises the following concerns:
    - what if a human, software, or hardware error corrupts, deletes, or otherwise destroys the contents of my files?
    - what if the system hosting my RAID array fails?  how do I recover my data?
 - RAID5 NAS can be economical if you're looking for about 5 TB or more.  If you need less storage than that, RAID1 or a backup scheme is less cost per TB.
 - For my LinuxMCE media, I don't really care about about the benefit of high-uptime that RAID offers.  Rather, I want to make sure that I can always recover all of my media with a reasonable amount of work. (This is the classic backup vs. redundancy discussion.)

Based on the above, I opted for a standalone 2-drive NAS with a scheduled backup from one drive to the other.  I am very happy with it.  A complete description of my setup is here:

http://wiki.linuxmce.org/index.php/Alx9r%27s_Network_Attached_Storage_Setup


Alex

[Zaerc]

I just finished researching then upgrading my system with more storage.  FWIW, here is what I found:
 - There are a few sad stories about media lost from RAIDed drives connected to the core.  These stories were enough to steer me away from a RAID solution built-in to the core.
 - putting more hard drives in my hybrid/core is probably more noise and heat than I ought to have in a case that has to live in my living room
 - any RAID solution raises the following concerns:
    - what if a human, software, or hardware error corrupts, deletes, or otherwise destroys the contents of my files?
    - what if the system hosting my RAID array fails?  how do I recover my data?
 - RAID5 NAS can be economical if you're looking for about 5 TB or more.  If you need less storage than that, RAID1 or a backup scheme is less cost per TB.
 - For my LinuxMCE media, I don't really care about about the benefit of high-uptime that RAID offers.  Rather, I want to make sure that I can always recover all of my media with a reasonable amount of work. (This is the classic backup vs. redundancy discussion.)

Based on the above, I opted for a standalone 2-drive NAS with a scheduled backup from one drive to the other.  I am very happy with it.  A complete description of my setup is here:

http://wiki.linuxmce.org/index.php/Alx9r%27s_Network_Attached_Storage_Setup


Alex

First of all, RAID is never a good substitution for proper backups, try to remember that or you'll find out the hard way sooner or later.  Also, you can find sad stories of data loss on just about any storage system ever used.

And here is another concern, what if (hypothetically speaking of course) your house burned down?  Unless you stored backups offsite or in a fireproof safe... see where I'm going with this?

Now what is more economical depends on the number of drives you will be using, not the size of the array.  As soon as you use more then two drives in your array, RAID5 is more cost-effective.  Here are some examples:

2 drive array
RAID1: total available capacity of 1 disk
RAID5: unavailable

3 drive array
RAID1: unavailable
RAID5: total available capacity of 2 disks

4 drive array
RAID1: total available capacity of 2 disks
RAID5: total available capacity of 3 disks

5 drive array
RAID1: unavailable
RAID5: total available capacity of 4 disks

6 drive array
RAID1: total available capacity of 3 disks
RAID5: total available capacity of 5 disks

And the difference only gets bigger from here...

[indulis]

RAID 5 is a good idea in the long run but becareful not to get to big a drives, or you could start running into the bit failure rate problem with RAID 5 (search for something like: raid 5 2009.  ZNET has a good article that explains it all.)

The "RAID 5 problem" in the article in ZDNET is a beatup and a load of cr@p.

The author does not understand failure modes and disk drives.  The article's premise is that if one drive fails in RAID5, the odds of another failure during rebuild are really high.  What he does not take into account is that you would have to get a 2nd PERMANENT error during data rebuild.

If your RAID array does not do "bit scrubbing" (ie checking all the time for temp/perm errors and relocating the data away) then there is a higher likelikhood of a permanent error during the day or so it takes to rebuild.  If it does bit scrubbing, then the odds of a permanent failure happening on the day that you need to rebuild are v small.

Anyway read the comments at the end of the zdnet article.  You'll find a good explanation why not to believe the fear-mongering in the article.

PS my customers (banks, large retailers) are running hundreds of TB of data on RAID 5 for a number of years and have NEVER seen a problem like this.  Even with 1TB drives and 300GB drives. Cos the controllers do bit scrubbing to avoid temp errors turning into permanent and the drives themselves do relocation of data in firmware.