Bloated MDs

[Armor Gnome]

I think the discussion on this topic has been great.  I just want to remind everyone that I placed this thread in this forum for a reason.  This is clearly a new feature/option and as 10.04 is in beta now, even if I polished off changes today they would need testing, debugging, etc that would have to wait for the real work of bug squashing to wrap up.  Where the implementation of this is pretty straight forward for me, what really remains on the table is the roadmap side of this discussion.

Standardization vs. Customization? 
Three years ago there really didn't seem to be any support for non-typical installs.  I brought up virtual options like VMware and was quickly told "nope, no hardware acceleration" case closed, drop it.  Today if you look over the forums or hop in IRC you hear all kinds of setups.  Virtual Core, virtual MD, virtual core and MD in the same virtualbox... Where a dev trying to support someone used to only ask "Is this your core or an MD?" now has to ask "What OS is your virtual box running in?, You're trying to do what with what?" etc. 

Personally I am a fan of customization options, but I completely understand I am not a typical user.  I actually enjoy time adjusting options, rebooting and tweaking settings and for the most part not using my system.  Non-OCD folks just want it to work.  Buy a Z-Box, Revo, Roku, etc and just plug it in. Want network streaming audio, buy the newest Squeezebox off the shelf and turn it on, etc.  To continue giving the typical user this type of experience I feel the only way to add customization is with very brief timeout/default options at first startup. 

What level of customization is too much?  Everyone knows if it were possible I would have Apple II's running 1/2 an application each all around my house. I would replace my gig switch with a 1920's telephone patch panel and manually route media to different rooms myself.  Tschak909 and Posde want to smack me already so before they do I will just clarify again that I understand that is not what LMCE is all about.

Since I tossed this out there I will be the one to have a first crack at it and will submit what I come up with for everyone to review.  Typically I overthink things but in this case I might be thinking its too easy of a change.  The DVD installer adds some desktop applications on the hybrid/core.  It was odd to me then that the Diskless_Setup by default actually creates more desktop applications and programs.  If there is discussion about the usefulness of a desktop environment lets keep it to media director only.  The core/hybrid has a desktop now and if you want to add little progs or apps then put them there or work on changing the primary installer.

My proposal and the task I believe I am up to involved the Diskless_Setup only, it is something I think I have the ability to change myself with my limited programming ability.  Correct me if I am completely off the wall but this is how I think the process works.

A new device is plugged into the network and looks for a DHCP to point it towards a bootloader.
DCErouter (specifically tftpboot.cfg) creates a few new directories based on its MAC address and points it to initrd and vnlinuz
The new device uses these as kernel= and boot=, and builds its image in root=<core>/?/nfs
Based on the content of vnlinuz and initrd, combined with polled info from various scripts the MD image is created just as it would install itself on a computer with a ubuntu install cd.
-add my changes
Because bootloaders can have options in the menu.1st file (sorry I know grub best, even though lmce I think uses something else because of the APPEND options) I believe I can add options here that will point to alternative initrd and vnlinuz versions.
Default option = just like it is now
Option1 = load initrd and vnlinuz that install different applications
Option2 = load initrd and vnlinuz that install different applications
Option3 = load initrd and vnlinuz that install different applications

There are options in the web-admin and warnings about kernel version on MDs so that part I don't think I have any control over.  I have tinkered though and adjusted the bootloader on the core to point a specific MAC address to a custom folder which booted TinyCore, DSL, 12.04 LTS and openWRT on different machines around my house.  I never got my own custom dsl+squeezebox configuration to boot however and is what led me to the idea of taking LMCE's setup and changing it to my desire.

I believe that I can take all of the source code needed to compile LMCE diskless, change and remove certain components and then compile and make options in the bootloader menu to select that version.  Is that an accurate statement?  Obviously I would need to debug and repeat a lot of effort but because I am only altering working source code, I would assume I am only looking at minor necessary work.  The years of hard work and coding behind current LMCE remains and I am just the jack-wagon that stripped some off.

If some off-the-shelf hardware doesn't quite fit the bill for a full MD, why not have the option to boot as audio only MD?  If an MD is going on the bedroom wall where a keyboard will never touch it why not have the ability to boot it without a desktop?  If I want to build a minimal media MD for use as an intercom outdoors, why can't I load a boot template designed for just that?  These are all just pipe-dreams of mine at this time as I haven't done yet any modification or compiling of lmce code.  I have just booted other pre-compiled kernels using lmce tftpboot. 

Hopefully this novel of a comment clears up what I plan to do and how I plan to do it.  After everyone has a laugh at my ignorance please feel free to correct me.

[phenigma]

Hi, great topic!  Figure I'll put my 2 cents in here as I understand things as I am working on an alternate MD right now.

I don't really think the MDs are bloated.  That will depend on you point of view I suppose but many appreciate the functionality.  My MDs occupy 2.7GB of space each on my core (without Game_Player).  Even with 5 MDs I only consume 22GB of space on my core.  I can't even buy an HD of less than about 30GB (20GB for SDD) these days and can even get an SSD for $50 in that size.  My personal feeling is to leave the stuff in place and don't hit the 'Start KDE' button unless you want to.  Removing KDE entirely would not quite save ~1GB per MD.  I don't see a space concern/savings.  I don't see significant 'bloat'.

Since I tossed this out there I will be the one to have a first crack at it and will submit what I come up with for everyone to review.  Typically I overthink things but in this case I might be thinking its too easy of a change.  The DVD installer adds some desktop applications on the hybrid/core.  It was odd to me then that the Diskless_Setup by default actually creates more desktop applications and programs.  If there is discussion about the usefulness of a desktop environment lets keep it to media director only.  The core/hybrid has a desktop now and if you want to add little progs or apps then put them there or work on changing the primary installer.

The difference in applications is more about where the packages come from than what is intentionally being installed.  The hybrid/core packages are taken directly from a kubuntu live cd, the MD files are built through package installation.  The liveCD has packages removed, by the kubuntu team, to allow for it to live within the small space of a single CD.  The MD diskless image is created and the package 'kubuntu-desktop' is installed from repository, which installs all of kubuntu and all of the packages you see on an MD.  In my recollection of the setup procedures this is the primary difference in the packages available on the core and on an MD.  This also causes some services to be started on MDs that are not run on the core, iirc.

My proposal and the task I believe I am up to involved the Diskless_Setup only, it is something I think I have the ability to change myself with my limited programming ability.  Correct me if I am completely off the wall but this is how I think the process works.

A new device is plugged into the network and looks for a DHCP to point it towards a bootloader.
DCErouter (specifically tftpboot.cfg) creates a few new directories based on its MAC address and points it to initrd and vnlinuz
The new device uses these as kernel= and boot=, and builds its image in root=<core>/?/nfs
Based on the content of vnlinuz and initrd, combined with polled info from various scripts the MD image is created just as it would install itself on a computer with a ubuntu install cd.
-add my changes
Because bootloaders can have options in the menu.1st file (sorry I know grub best, even though lmce I think uses something else because of the APPEND options) I believe I can add options here that will point to alternative initrd and vnlinuz versions.
Default option = just like it is now
Option1 = load initrd and vnlinuz that install different applications
Option2 = load initrd and vnlinuz that install different applications
Option3 = load initrd and vnlinuz that install different applications

There are options in the web-admin and warnings about kernel version on MDs so that part I don't think I have any control over.  I have tinkered though and adjusted the bootloader on the core to point a specific MAC address to a custom folder which booted TinyCore, DSL, 12.04 LTS and openWRT on different machines around my house.  I never got my own custom dsl+squeezebox configuration to boot however and is what led me to the idea of taking LMCE's setup and changing it to my desire.

I believe that I can take all of the source code needed to compile LMCE diskless, change and remove certain components and then compile and make options in the bootloader menu to select that version.  Is that an accurate statement?  Obviously I would need to debug and repeat a lot of effort but because I am only altering working source code, I would assume I am only looking at minor necessary work.  The years of hard work and coding behind current LMCE remains and I am just the jack-wagon that stripped some off.

If some off-the-shelf hardware doesn't quite fit the bill for a full MD, why not have the option to boot as audio only MD?  If an MD is going on the bedroom wall where a keyboard will never touch it why not have the ability to boot it without a desktop?  If I want to build a minimal media MD for use as an intercom outdoors, why can't I load a boot template designed for just that?  These are all just pipe-dreams of mine at this time as I haven't done yet any modification or compiling of lmce code.  I have just booted other pre-compiled kernels using lmce tftpboot. 

Issues/Challenges I see:
There is no 'bootloader' per se.  PXE loads kernel/initrd (common if the MAC is not found, from diskless/XX if the device exists) which bootstraps the running kernel from the diskless/XX directory.  Grub is not used on the MDs as they are PXE booted, which occurs before grub would have a chance to run.  This behaviour cannot be changed without significant user interaction to install grub on the MD before attempting the first boot.  This would significantly detract from the 'plug and play' nature of the system.

Your statements about adding bootloader options and "If an MD is going on the bedroom wall where a keyboard will never touch it why not have the ability to boot it without a desktop?" are contradictory.  Remember that an MD is not intended to have a keyboard and mouse connected and as such there is likely no interaction between the user and the bootloader on an MD.  This is desired behaviour and likely how it should remain.  There is a reason that there is no user interaction with the MD until the AVWizard which allows for many different input devices to manipulate it, including game controllers and IR remotes, which cannot interact with a bootloader.  Adding a bootloader is going to significantly add to the boot time of the media director everytime it boots.  This, to me, is not desireable and would degrade the experience significantly.

What you are trying to achieve IS doable though... The 'right' way to approach this???:
"If some off-the-shelf hardware doesn't quite fit the bill for a full MD, why not have the option to boot as audio only MD?" - This is exactly what device templates are for.  All MDs are currently built on the 'Generic PC as MD' device template.  That doesn't have to be the case.  Device templates could, *should*, be created for different devices that cannot act as a full MD.  This give you the ability to add that device and have an lmce core create the device (and associated diskless folders, etc.) within its' capabilities.  This could be as simple as the device being recognized under the alternate device template on first PXE boot, or if the device is capable of a Full MD but you want less, you could add the 'less of a MD' device template manually and enter the MAC address into the webadmin.  As you have already discovered the nfsboot folders can be altered/changed to accomodate any device.  Those folders can be setup properly (and automatically) through the use of device templates.

Existing functionality of MDs, and the current method of Diskless_Setup, should be left entirely intact as it is desireable for many.  New DTs that create new (different) directory structures under /diskless/XX/ at the moment the DT is added would be my recommendation.  This will require changes to Diskless_Setup as well, but would leave existing MD generation the way it is.  This is what I am attempting with the Raspberry Pi. I haven't made it to the DT/autosetup part yet.

Hopefully this novel of a comment clears up what I plan to do and how I plan to do it.  After everyone has a laugh at my ignorance please feel free to correct me.

No laughing.  I don't intend to sound too critical either, I tend to over-analyze and point out down-sides, it's a back character trait of mine.  But I understand what you are trying to do and think it could be very useful if applied in a way that is consistent with the mechanics of the existing system.

J.

[phenigma]

Well, since I had to touch it anyways...

Here is a /usr/pluto/bin/Diskless_CreateTBZ.sh that I am testing.  I have included a simple option to not install the kubuntu-desktop package.  The resulting MD works and is ~1G smaller.  There is no desktop available.  If you choose Start KDE I don't know what will happen, you're on your own.  This is not intended to be a supported feature, but if you *really* feel strongly about not having kde installed on your MDs then you can try this.

http://pastebin.com/KpEGER17

Copy it into a file on your core (mine is currently /usr/pluto/bin/newDiskless_CreateTBZ.sh) and chmod +x it.  Then run it.  It should work (I've tested it once here) but let me know if you encounter any issues.  Logging is unfortunately disabled as it was causing strangeness I havn't worked out yet.

If it doesn't work then re-run your original /usr/pluto/bin/Diskless_CreateTBZ.sh to recreate your MD tarball and continue on your way.

J.

[Armor Gnome]

Awesome.  I should have posted that this feature request topic I started was back burnered for a few reasons.

Lack of a media director.
Inability.
No large payout (1gig) for the time needed to teach myself coding at that level.
Split opinions on the features benefit.

If you were in there anyways and was curious then awesome!  If you dug around at my request let me know if I owe you for your time.  I will grab your pastebin tommorow as I have a curiosity about dropping other items from create_diskless, such a X.  That is the fail point for 90% of my media director attempts.  Thank you.

[phenigma]

If you were in there anyways and was curious then awesome!  If you dug around at my request let me know if I owe you for your time.  I will grab your pastebin tommorow as I have a curiosity about dropping other items from create_diskless, such a X.  That is the fail point for 90% of my media director attempts.  Thank you.

As I said, I was in there already working on RPi tarball creation.  I have tested it once, it builds a MD tarball that boots to orbiter, that's all I have tested.  This is an experiment and is NOT supported!  !! DO NOT SELECT START KDE, it will hang the MD !!  Again, this is NOT supported. 

Unfortunately X is a requirement for MDs.  There is no removing X.  I have never seen X 'fail' to install on any MD.  Video drivers on the other hand are a headache and can prevent X from running.  L3mce is working on improving video card detection/driver installation.  If you have video card driver problems you should talk to him about your card(s).  You will want to learn about and use the system for a while (at least a year, if not two or more) to really understand the internals before trying to remove something.

J.