LinuxMCE Forums

General => Users => Topic started by: l3mce on June 15, 2012, 04:47:52 pm

Title: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 15, 2012, 04:47:52 pm
So I wrote this a while back, and have made some adjustments to it. I need to do this the right way and use the DB to determine MDs instead of the cheat I am using, but for now it will do. I also need to log... so it is likely to change shortly, but what it does is upgrade your entire system at the same time, upgrading the MDs locally. The steps it takes are:

1. Update/upgrade the core.
2. Copies the packages files to /usr/pluto/deb-cache
3. Creates a new Packages.gz file
4. Backs up the entire /usr/pluto/diskless directory in a tarball in /usr/pluto/backup
5. Updates/upgrades MDs, regardless of their on/off state using NFS link to cores deb-cache (meaning, the upgrades for MDs are local)
6. Cleans up and exits.

So, at the moment, you shouldn't turn an MD off if it is on while it is upgrading. Nor should you try and use this if you have different MD versions (like 810 MD on a 1004 hybrid).

Would appreciate feedback. Just put it somewhere on the core (/usr/pluto/bin would be a good place), and run it as root.
Code: [Select]
svn co http://svn.linuxmce.org/svn/people/l3mce/GlobalUpgrades.shIf you don't have subversion, then
Code: [Select]
wget http://svn.linuxmce.org/svn/people/l3mce/GlobalUpgrades.sh; chmod +x GlobalUpgrades.sh

This is an upgrade. It COULD break things. Remember, there is a backup of the MDs. This occurs before upgrade so you can restore.
This is an upgrade. ALL Machines will need to be restarted. I do not do this automatically in case they are in use.

This is a little bulky atm because I am a bit hesitant to use a mv command with the packages, so I am duplicating them. That WILL change. The backup also takes up some space... and atm this all happens on the core drive... so... you will need a little free space...
There is no more duplication of packages. I have also added some mounting which fixes certain kinds of package upgrades with MD's that are not active.

This will work on any version of LMCE.

Title: Re: Global Upgrades. Need testers.
Post by: l3mce on June 15, 2012, 07:56:31 pm
For the record, I ran the script last night, and it upgraded my core and all 5 of my MDs, some of them on, and some of them off, without incident.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pointman87 on June 18, 2012, 07:58:28 pm
Hi L3mce, your script just finished upgrading my core and 4 MDs with very good result :)
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: Aviator on June 20, 2012, 03:44:26 am
Worked like a charm on my 10.04 core.  Both MD's work fine after.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: JaseP on June 20, 2012, 03:57:19 am
Is this for use with the web installed version too,... or just the snapshots?
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: ardirtbiker on June 21, 2012, 12:56:09 am
No problems experienced here either...  One core/hybrid and 2 MDs

Dennis
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: bushtech on June 21, 2012, 01:03:52 am
For the bandwidth deprived. Rough idea of the size of these downloads? ???
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pointman87 on June 21, 2012, 02:26:46 am
For the bandwidth deprived. Rough idea of the size of these downloads? ???

The download for each upgrade are about 420-500MB
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: bushtech on June 21, 2012, 09:29:07 am
Thanks Daniel
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 21, 2012, 04:24:27 pm
Thank you guys for the feedback!

JaseP this will work on any version of LMCE, and any install method.

Bushtech, it is (as usual) with the bandwidth deprived in mind that I did this. Normally EACH MD has to pull down upwards of 500Mb. This way, ONLY the core pulls down the updated packages, and the MDs leech off of that.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: _if_ on June 21, 2012, 05:05:23 pm
will that script get integrated into the UI under "Advanced" (where we have that unfunctional button already) as well?

IF
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 21, 2012, 05:24:55 pm
No. There are a few reasons why... for now at least...

1) If something breaks upstream, it breaks everything in your system. So, for instance, when KDM upgraded last year, forcing the desktop over LMCE until we could figure out a fix, everyones entire system becomes unusable. This is a real problem for newbs... and would result in a LOT of complaining.

2) People running non-standard systems could seriously hose stuff.

Having to manually run the script, knowing the potential consequences cuts down on support time, should things go awry. I originally wrote it for that purpose though... and I could make the button function as an individual upgrade if someone wanted to make a ticket.

To be honest, this was intended to do dist-upgrades globally, which will be my next round of testing once I get this to a happy place... it was to get around the "NEVER do dist-upgrades to an installed system" rule, fixing all of the problems which that can cause. I still need to do some work now that it seems to be "working" without incident. I need to figure out what to keep/remove from backups. There is no purpose in keeping all of the tarballs a year of upgrading would create. I should probably do some cleanup with regards to multiple old versions in deb-cache... and some other little things. It has led me to want to extend the JobHandler to take script arguments so that things like this can be placed into pending tasks from a script, preventing shutdown (without forcing), and showing progress.

Thanks again.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: bushtech on June 21, 2012, 10:47:00 pm
L3mce
Big, big thanks to you, I really do appreciate it
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: avajon on June 21, 2012, 11:25:04 pm
L3mce

I also want to thank you for the great work you've done so far!
But i also have a question:

My LinuxMCE 0810 runs since more than a year very good on a 2TB drive. I also bought a 60 GB SSD for a new core (1004) because i also have a NAS (unRaid) since 3 months. So, in my opinion i have two options: 1. install from scratch LMCE 1004 on the new SSD with all the work (DVB-S2 on mythtv, MD's, and a lot of other stuff which is running on the server). or 2. use this cool script. But before i can do this i had to clone the content of the drive to the 60 GB SSD - there's only 40GB data on the 2TB drive, so this should work with gparted/clonezilla?

What do you think is the "best" solution?

thanks
Markus
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 21, 2012, 11:40:38 pm
L3mce

I also want to thank you for the great work you've done so far!
But i also have a question:

My LinuxMCE 0810 runs since more than a year very good on a 2TB drive. I also bought a 60 GB SSD for a new core (1004) because i also have a NAS (unRaid) since 3 months. So, in my opinion i have two options: 1. install from scratch LMCE 1004 on the new SSD with all the work (DVB-S2 on mythtv, MD's, and a lot of other stuff which is running on the server). or 2. use this cool script. But before i can do this i had to clone the content of the drive to the 60 GB SSD - there's only 40GB data on the 2TB drive, so this should work with gparted/clonezilla?

What do you think is the "best" solution?

thanks
Markus

Well... clonezilla isn't going to work because of the partition sizes. Clonezilla is gonna puke trying to put a bigger partition onto a smaller one, even though the data would fit... so I see you have put gparted in there, so I assume you want to shrink the size of the partition, and clone it off... which SHOULD work... assuming you also created the swap partition etc, and you would have to install grub2 and update it.

You could also just plug the drive in, boot to a live disk, and dd the stuff over.

I dislike resizing the partition because if anything goes wrong... yuck.
DD is often named disk destroyer because one wrong character can destroy your data, so... yuck.

What I would do, if I were you, is grab the 1004 snapshot, and install fresh on the ssd, and add the 2tb as an extra drive AFTER 1004 is installed on the new... then just cp what you want from the small install over to the new... then use the 2tb for storage.

Of course, having written the dvd creator for 1004 I am a little biased... but it is a much cleaner experience.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: avajon on June 21, 2012, 11:46:12 pm
wow. such a great user support from you! :)

I think you're right, fresh install is much secure. Because what shouldn't happen in any case is that something goes wrong and the data from the big disk is away... WAF would be zero :)

Thanks, so will try the snapshot.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 21, 2012, 11:50:50 pm
Exactly, if something goes terribly wrong, your 810 install is sitting there, untouched, to fall back on.

Keep in mind with the 1004 snapshot that everything is embedded. You do not need to run Diskless_CreateTBZ.sh anymore. Just get your core install finished, and plug MDs in. If you run into any issues, please let me know. I am usually available in irc as well for troubleshooting.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 23, 2012, 11:20:12 pm
Ok... so it is a lot lighter weight now.
I have added some pruning to get rid of old package files that are replaced by newer ones rather than keeping 800 versions of packages.
I have added some logic to remove more than 2 backups of /usr/pluto/diskless removing the oldest, so there will never be more than 3 total.
I have also excluded directories that cause harmless but annoying socket errors from tar.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pedplar on June 24, 2012, 06:33:11 am
I get an error.

brent@dcerouter:~$ sudo ./GlobalUpgrades.sh
./GlobalUpgrades.sh: line 42: syntax error near unexpected token `then:'
./GlobalUpgrades.sh: line 42: ` if [[ "$backup_Number" -gt "1" ]] then:'
brent@dcerouter:~$
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on June 24, 2012, 03:31:40 pm
Sorry for the typo... fixed... thanks for testing.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pigdog on July 10, 2012, 08:40:17 pm
Hi,

I upgraded a core/hybrid with ATI card and VGA/analog audio.

2 MD's - one to a 1080p TV using a Revo 330 nvidia ION and HDMI audio/video.
            - second to a 720p TV using an AOpen DE7000 nvidia and HDMI audio/video.

Everything seems fine.

The DE7000 is a new addition (dumpster diving find) and on initial setup, prior to upgrade, AVwizard didn't want to adjust the screen size smaller.

The screen size is a little big.

I'll check TV settings and if nothing is found try xorg adjustments as per another post.

Great job l3!
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on July 10, 2012, 09:36:19 pm
From core, where XX is your moon number
Code: [Select]
sudo ssh moonXX
rm /etc/X11/xorg.conf
rm /etc/X11/xorg.conf.pluto.avwizard
reboot

Will probably be fine.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pigdog on July 11, 2012, 01:37:14 pm
Hi,

That did it. 

After reboot with AVwizard I could zoom in/out screen size to fit properly.

Thanks l3.

Cheers
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pigdog on July 12, 2012, 04:06:33 am
Hi,

I forgot to mention that I did not have a xorg.conf.pluto,avwizard file - in case that's important.

Just curious but, is there a difference with physically deleting the xorg.conf and rebooting to run the avwizard or just running the avwizard from the advance menu?

Thanks again.

Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on July 12, 2012, 05:30:20 am
Pay no attention to the man behind the curtain... IIIII am the great and powerful OZ!!!!


Yes. I mean both result in the same thing, avwizard runs, but in your case, the existing xorg was causing a problem. It takes your existing xorg.conf and tries to fix it. Sometimes this causes a problem.

There are three conditions which cause avwiz to run.
1. There is no xorg.conf.
2. AVWizardOverride is set to 1
3. AVWizardDone is set to 0
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: pigdog on July 12, 2012, 03:13:04 pm
Thanks.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: m3freak on July 16, 2012, 11:22:52 pm
Keep in mind with the 1004 snapshot that everything is embedded. You do not need to run Diskless_CreateTBZ.sh anymore. Just get your core install finished, and plug MDs in. If you run into any issues, please let me know. I am usually available in irc as well for troubleshooting.

Would one get this "embedded" feature only via a fresh core install from a recent snapshot, or is it also implemented for those who are running the latest 10.04 code through updates/upgrades?
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on July 17, 2012, 01:53:05 am
Would one get this "embedded" feature only via a fresh core install from a recent snapshot, or is it also implemented for those who are running the latest 10.04 code through updates/upgrades?

This is only a feature of the 1004 snapshot.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: jamo on July 17, 2012, 07:50:33 am
This is only a feature of the 1004 snapshot.

That's something I wanted to clarify- If I wanted to add a new MD to my system, presumably it would use the diskless image that was up-to-date as of my snapshot version, even if I'd done a few globalupgrades since then? So to get my new MD up to speed, I have two options-

1. Add it and then run globalupgrades on the whole system to upgrade it
or
2. Run Diskless_CreateTVZ.sh to create a new up-to-date diskless image and then add the MD.

I know option 1 is probably preferred, bandwidth-wise because it doesn't re-do *everything*, only updates the old stuff. But I'm thinking in a scenario where I want to test how cleanly a new MD is detected and setup from scratch, that would be the way to go?
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: l3mce on July 17, 2012, 05:01:29 pm
Using the script is a lot better bandwidth wise... and it does something interesting that nothing else does currently. It moves all of the packages the core knows about to the /usr/pluto/deb-cache directory, and creates two new Packages.gz files for the two directories. This means when mds are created, they draw all packages from the local directory while being setup. The script also cleans up old versions, so you don't end up with 100 different versions of packages.

So lets say you run the upgrade script on the core and remaining MDs. You could create a new MD after that, diskless setup will go much faster, and then running global upgrades again after creation would bring that MD up to speed.

If you do not care about time and bandwidth, you could upgrade the whole system, and then run Diskless_CreateTBZ.sh. This would have future MDs already up to date.

From my perspective, the result should be the same. One takes about half an hour, one can take up to two hours. One downloads a max of 525Mb, one downloads around 1.2Gb. However, running TBZ means you do not have to run a very immature  upgrade script every time you plug an MD in.

So... I guess it comes down to how often you upgrade. If you upgrade a lot... I would use the upgrade script. If you upgrade until you get to a stable image and then hang there... I would run TBZ.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: m3freak on July 17, 2012, 11:07:55 pm
So I ran it on my 10.04 system (mine is not based on a recent snapshot).  I see no problems so far with the core or my lone MD.  I did initially have sound over HDMI issues - it didn't work!  I ran alsamixer and chose the HDMI sound card, which fixed the problem.
Title: Re: Time to update your whole system? Try GlobalUpgrades.sh Need testers.
Post by: Marie.O on July 30, 2012, 09:10:04 pm
Can you tell that l3mce's pipe is severly limited...