Storage - where and how to mount it?

[bitmonkey]

Hi,

I have just built an 0810 alpha core, the storage is on another machine and for security reasons I am keeping it encrypted and doing the enc/de-cryption at the mce core where the storage will be mounted. So the file server provides iscsi devices, the core sees them as though they were local drives (/dev/sdb1, sdc1 etc...), then cryptsetup luksOpen is run to mount these crypt disks as /dev/mapper/crypt_audio, crypt_video etc. etc. which are then given an ext3 filesystem and mounted somewhere useful.

Everything works fine in testing - I can mount the drives from the core on test folders in /mnt and read/write data to and from them no problem.

What I want to know is this:

1. Where should I mount the disks? What locations will linuxmce use, or can I just set it up to use my existing mount points under /mnt somehow?

2. Because the mount disks script requires a password to unlock the encryption, it is better that I can ssh in after boot has finished and manually run the script to remount the disks. Can I do it like this, or will something break because the core starts without being able to see the data during boot?

3. If (2) will break something by mounting the disks after boot, how can I mount them earlier remotely? Since I cannot ssh in to the console, how would I go about getting a remote console so that I can access the password prompt during boot remotely?

4. If (3) (remote console during boot) looks difficult, is there some way I could let the server boot as plain kubuntu, ssh in, mount the storage then manually start the linuxmce services? I could pick my way through /etc/rc*.d and see what looks like it's connected to linuxmce, then move it to a script to be run manually, but it would be great if there was an easier way? Any suggestions?

thanks

Paul

[merkur2k]

I doubt you will ever be able to make that setup work. LinuxMCE has an automounter program that mounts and unmounts filesystems on demand, and expects them to be available as normal local drives or remote samba shares. you would make your life alot easier by tossing the tinfoil hat in the trash and only encrypting stuff that really needs it (ie not a/v media files).

[bitmonkey]

I doubt you will ever be able to make that setup work. LinuxMCE has an automounter program that mounts and unmounts filesystems on demand, and expects them to be available as normal local drives or remote samba shares. you would make your life alot easier by tossing the tinfoil hat in the trash and only encrypting stuff that really needs it (ie not a/v media files).

Merkur2k, thanks for the quick and straightforward reply.

I appreciate you have advised me this is a bad idea, but I'd still like to look into it, as the purpose of /dev/mapper is to present something that appears just like a normal local drive to the layers above - surely /dev/mapper/foobar and /dev/foobar look the same to the automounter scripts?

Put it this way, at the least I can present iscsi volumes to the core since when I first ISCSI mounted my remote drives the orbiter on the core popped up messages asking me if I wanted to use them, so that much works fine - I could, therefore, do the encryption and decryption on the file server, present cleartext iscsi volumes to the core, and have it mount and use them. That would definitely work, but I'd prefer to do the encryption and decryption on the mce core as it is a much faster machine.

Going even further with the silliness of multiple layers, I could go fileserver -> iscsi -> mce core -> luksOpen -> iscsi -> mce core
so by iscsi-ing out the decrypted mappings in /dev/mapper I could make them appear to the local host as if they were remote iscsi volumes, then they would pop up in the orbiter just like local drives. It just seems like another unnecessary layer of complication, although I'm sure it would work as I've tested something very similar by mounting the volumes that the fileserver iscsi's out on the fileserver itself. Work or not, it's still messy and I'd sort of rather look at the scripts to see if I can do it in a more elegant way.

If you (or someone else) can tell me where the automounter scripts are, when they are run (and what runs them) etc. I'd like to have a look and see if I can hack something together. At the very least it would be educational. The other thing I'd like to know is at what point during boot this automounting process begins, so I can ensure my volumes are ready in time, and at what point during shutdown the automounting process stops so I can ensure my volumes stay up until after that.

Again, thanks for the reply. I don't wish to appear ungrateful for the help by replying that I'd still like to fiddle with it, I will bear in mind what you've said and if necessary I will revert to a simpler configuration.

Paul

[wierdbeard65]

I'm curious.

Why do you feel the need to encrypt your media?

Or is this just an exercise to see if it can be done?

[tschak909]

bitmonkey,

I am one of the core devs.

I can already tell you there is no way to make your arrangement work, without doing severe amounts of work to the automounter, to the storage configuration scripts, and to the underlying system startup scripts.

This system is designed as an appliance, and as such, simply was not designed with your use case in mind at all. With this said, since you do seem to be very technically inclined to go through the trouble of DOING such an intricate setup, perhaps you could patch the needed parts (There are over 250 scripts in our part of the system alone) to do the needed work?

-Thom

[bitmonkey]

I'm curious.

Why do you feel the need to encrypt your media?

Or is this just an exercise to see if it can be done?

Hi WierdBeard,

I just don't like the idea of anyone who lifts my servers having access to my data, I appreciate the point that it's only media and not documents, but who's to say I won't want to put private pictures, videos or even audio on my mce setup which I wouldn't want accessible to anyone who stole the boxes?

Generally I just encrypt everything I can on principle anyway, it's not that I have much information that I really need encrypted, it's just I have a security mindset so I tend to encrypt by default and only leave things plaintext where there is an overwhelming need to do so, as it seems there might be here.

[bitmonkey]

Hi Thom,

I have seen other posts of yours - thankyou for replying, and I'll take this opportunity to say thanks for all the work you are doing on linuxmce, I've only just installed my first server, so my experience is small as yet, but the system looks very good and I'm very grateful for all the work that's been put into it - I notice you are a very active developer, and I saw your thread about being out of work. I am out of work too at the moment which is why I've been unable to contribute anything via paypal, but I'd like to at least offer my thanks for all your hard work.

I wouldn't call myself technically inclined in the way you are talking about - I am not a bad sysadmin (used to do it for a living) but I'm not much of a developer, and my scripting is fairly basic. So, although I am somewhat technical, I'm not really skilled at development which is why I haven't posted on the 0810 beta "message to the community" thread.

With that in mind, I'm going to do the encryption at the fileserver and let linuxmce see a cleartext iscsi volume so it automounts it in the intended way, that will be sufficient security for my purposes and will still provide OK performance while keeping everything as simple as possible from the point of view of the core.

Paul

[bitmonkey]

Two more questions before I go and change my configuration;

1. I am using LVM on the fileserver and would, for flexibility, prefer to keep my audio, video and images files on 3 separate logical volumes. So from the point of view of the core there would be 3 devices, /dev/hdb1, /dev/hdc1 and /dev/hdd1 - will this work well, and can I tell the core "I want to use this partition for audio, this one for video and this third one for pictures", or is it better to just present one drive /dev/hdb1 and let linuxmce put all the media on there?

One problem I have is that I want to replicate my audio and pictures folders using drbd so I can directly replicate the encrypted block devices, I'm doing this over a WAN link so there's no way I can be replicating blu-ray rips or even DVDs really, so it's much better if they are on separate partitions (LVs) otherwise I have to completely rethink my replication strategy.

I don't mind not being able to add media via linuxmce, as I will do all my ripping separately and add the media directly to the appropriate partition manually then let linuxmce index it, but what about recording from DVB-T, will I be able to tell it to store these recordings on the volume that is not replicated which is set aside for video media?

2. I found a reference in the wiki to a script to nfs share the media storage, if I wanted to use samba instead is there a script for that, or is it best that I just manually configure samba to share the /mnt/devices/{foo,bar,etc} directories?

thanks

Paul

[tschak909]

Dude, if you keep thinking like this, I'm gonna have to thwap you upside the head. YOU'RE DELIBERATELY MAKING THIS DIFFICULT! STOP IT!

Expose the needed shares, use LMCE file structure, let it create the needed directories, put the media in those folders. LMCE will FIND the share, ask about it, and set it up automatically.

This is of course contingent on setting up the network properly for LinuxMCE.

(1) core must have two NICs
(2) core must be DHCP server, gateway, and primary firewall.

No arguments.

-Thom

[bitmonkey]

Thom,

I know you mean well with telling me not to make work for myself, but sometimes when I see you post stuff like this it makes linuxmce seem unnecessarily complicated. What I mean is that things like iscsi are standards based, and are designed to make the high level services like linuxmce and what it depends on see a completely standard block device, it literally looks identical to a locally connected hard drive - surely getting linuxmce to work with that should not be difficult?

I would also think a lot of people will want to replicate part of their media like I do, meaning they will want their audio and images on separate partitions to the stuff like blu-ray rips which are too big to replicate. That this would be difficult to do seems, well, very user-unfriendly actually, but please don't take that the wrong way, it's meant as a constructive comment and not as a jab at what is obviously a big, complex and very cool system which must have taken a very large amount of work to write, so perhaps I am asking too much by saying "this should be simple" when I not only didn't write it, but wouldn't know where to start.

Perhaps I also just don't get the appliance mindset.

In any case, please don't get the impression that these minor issues are going to put me off linuxmce, there's too much else to like, so I will work around the way it needs things set up as necessary.

Thanks for the replies.

Paul

Dude, if you keep thinking like this, I'm gonna have to thwap you upside the head. YOU'RE DELIBERATELY MAKING THIS DIFFICULT! STOP IT!

Expose the needed shares, use LinuxMCE file structure, let it create the needed directories, put the media in those folders. LinuxMCE will FIND the share, ask about it, and set it up automatically.

This is of course contingent on setting up the network properly for LinuxMCE.

(1) core must have two NICs
(2) core must be DHCP server, gateway, and primary firewall.

No arguments.

-Thom

[merkur2k]

It is not the system that is complicated, it is you the user that is making it complicated. More people understand how to use windows shares than iscsi, so that is what it supports. Sure it could support other stuff, and it probably will eventually. But it is not very high priority compared to many other unfinished features.
Media is largely managed by the user. That is you tell it to use linuxmce dir structure which creates dirs for all types of media, but it is up to you where you actually put the media when you copy it to the system. The only exception is TV, which by default will use the drive with the most free space.

[tschak909]

*hmm*

Just use the system as intended, and then make changes as needed. You'll see that we pretty much thought of sane ways to do just about everything, otherwise we can't help you.

-Thom

[tschak909]

The system was designed so it would find:

(1) Network shares
(2) md based RAID slices (md0/md1/mdX)
(3) standard disks attached at standard device names (/dev/sd,/dev/hd)

this was so you could go out, buy a NAS brick, shove it on the network, and it would just work. or

go out and buy an eSATA disk, plug it in, and it just work,etc.

Stop...making....it....complicated.....

-Thom

[wierdbeard65]

The more posts like this I see, the more I understand Thom's frustration and his resulting comments.

Perhaps I also just don't get the appliance mindset.

What's to get? Once it's where it needs to be, plug it in, it works = appliance

I think one of the problems here is that most of the people that end up looking at MCE (at the moment) are techno-heads who don't like to give up control to "the machine".

My advice is the same as I give my students. Nobody with any brains would learn to ski off-piste. You start on the nursery slopes and once you know what the rules are and why they are, then you can break them creatively in the knowledge of what will happen. It's the same here. Begin by setting it all up EXACTLY as recommended. Once it's working, then you can play a bit to see what happens (in the knowledge that if you break it you can put it back!)

Just my 10 Eurocents

[bitmonkey]

Ok, OK - I get it!!!

I'm going to let it see bog standard samba shares, then as suggested if once it's up and working I feel like fiddling I can do so from a known working configuration.

Thanks for the help.

Paul