FIXED: Make the DVD actually install from the DVD instead of the internet

[phenigma]

EDIT:  As of Aug 18, 2010 (23248), the required changes are now in svn and in the latest snapshot dvds.  The DVDs do still require an internet connection to install as some packages and add-ons cannot be distributed on the DVD and must be downloaded at install time.

I've always noticed that the DVD install doesn't actually use any of the files that are stored on the DVD.  And a statement on the forum (http://forum.linuxmce.org/index.php?topic=10393.msg71401#msg71401) made me think about it for a day.  Even though all the required .debs are available on the local hard drive during the installation none of them were being used.  The DVD installer goes to the internet and re-downloads all the .debs which are on the DVD (which you have already downloaded).

I spent some time investigating and the problem is three-fold: 1) The repository order in sources.list determines fetch-priority and the local repo is added after the internet repos on the dvd installer.  2) Trusted repositories seem to be favored over untrusted repositories with the same priority, the local repository is un-signed and untrusted by apt (go figure... ;-])  3) Priority should be default and identical to internet repos (500) - posde has already done this - yay!

Through testing yesterday and today I've built a script which fixes the first two issues and allows the installer to use all of the .debs from the LinuxMCE install DVD which are on the hard drive.  I have tested this with snapshot dvds 22898 and 23036 and they work great for me.  It should work for any of the snapshot installs.

What it does:  The script moves the local file repository to the top of the sources.list.  Generates a 1024bit RSA key and signs the Release file of the local repository.  Adds the key to apt so the repository is trusted.  Updates your package lists from the repositories in sources.list

Note:  The script works for me, I hope it works for you.  YMMV.

The script should be run as root immediately following the kubuntu install process but before you click on the LinuxMCE install icon on the desktop.

1. Download (http://linuxmce.iptp.org/snapshots/) and burn the dvd
2. Insert the DVD into your Core/Hybrid machine and turn it on
2. Choosing the 'Install LinuxMCE..' method
3. Once the desktop appears open a terminal (K->Applications->System->Terminal) and become root (sudo su -)
4. Download and Run the script (hmm, attaching to the post isn't working, so I've embedded it)
5. Click the LinuxMCE install icon and enter your password, only new packages which are not on the dvd will download.

I will put the patches together to submit this to svn in the proper scripts but...  There is a little more glue that needs to be applied to make /usr/pluto/bin/Diskless_CreateTBZ trust the repo (and therefor all MDs) and and to keep the deb-cache and packages list updated and the Release file signed so apt will continue to trust the local repo as it is updated.  

I am leaving for 2wks at the cottage (no internet) tomorrow so it will have to wait 'till I get back.  Until then... I thought I would share what I have so far.

J.

The script:  trust-local-repo.sh

Code: [Select]

#!/bin/sh
cd ~

echo **Fix the sources.list file order
echo ****Remove the local and LinuxMCE repo
sed -e '/deb-cache/d' -e '/deb.linuxmce.org/d' -i /etc/apt/sources.list
echo ****Re-add the local and LinuxMCE repo to the top of sources.list
sed -e '1ideb file:\/usr\/pluto\/deb-cache .\/\ndeb http:\/\/deb.linuxmce.org\/ubuntu\/ intrepid beta2' -i /etc/apt/sources.list

echo **Begin rsa key creation
echo ****Create the auto gen file
cat >repo-key <<EOF
%echo Generating RSA key for local repo
Key-Type: RSA
Key-Length: 1024
Name-Real: LinuxMCE
Name-Comment: deb-cache repository
Name-Email: me@here.com
Expire-Date: 0
%commit
%echo done
EOF

echo ****Generate the key
gpg --batch --gen-key repo-key

echo ****Export the public key to a file 'local-repo-pub-key'
gpg --output local-repo-pub-key --armor --export $(gpg --list-keys | awk '/LinuxMCE/{print x}; {x=substr($2,7,8)}')

echo ****Add the key to aptitude
apt-key add local-repo-pub-key

echo **Update the Packages File and generate Release and Release.gpg
echo ****Update the Packages
cd /usr/pluto/deb-cache
dpkg-scanpackages . /dev/null > Packages
gzip -9c Packages > Packages.gz

echo ****Generate the 'Release' file
cat >Release <<EOF
Archive: intrepid
Origin: Ubuntu
Label: Local Repository
Architecture: i386
EOF
printf 'MD5Sum: '$(md5sum Packages | cut --delimiter=' ' --fields=1)' %16d Packages\n' \
   $(wc --bytes Packages | cut --delimiter=' ' --fields=1) >> Release
printf ' '$(md5sum Packages.gz | cut --delimiter=' ' --fields=1)' %16d Packages.gz' \
   $(wc --bytes Packages.gz | cut --delimiter=' ' --fields=1) >> Release

echo ****Generate the 'Release.gpg' file
gpg --yes --armor --detach-sign --output Release.gpg Release

echo **Update the package lists
apt-get update
echo **Repository is ready for use.


[tschak909]

Thank you so much, phenigma. You should work with possy to fold this into our install process.

-Thom

[phenigma]

Thank you Thom.

I would like to fold this into the installation procedure sooner rather than later.  I'll try to jump on irc in the morning before I leave and see if I can ask possy a few questions.

J

[Marie.O]

Phenigma,

thanks for your work.

As far as I understand your script, you remove our internet repo from the sources.list, and add a key to the local deb.

This will indeed make sure,  that no package will be taken from our repo. Unfortunatly, we will still need the internet, for things like sqlCVS update, and the installation of files like the Microsoft truetype fonts.

I will modify the install and do a snapshot probably today, to not include the internet repo.

[fastie81]

Phenigma,

thanks for your work.

As far as I understand your script, you remove our internet repo from the sources.list, and add a key to the local deb.

This will indeed make sure,  that no package will be taken from our repo. Unfortunatly, we will still need the internet, for things like sqlCVS update, and the installation of files like the Microsoft truetype fonts.

I will modify the install and do a snapshot probably today, to not include the internet repo.

Hi posde

From what I can make out from the script, it only moves the order of the sources.list. So it look local first before going to the internet.
what we could to is just add a script at the end of the installation to move it back in the order it is now if you are worried that update will not take place.
So it will look locally and install the .deb from local disk then after install it change the source.list back to what it is now and all it good.
I am with Phenigma on this one. Here in NZ we still have to deal with caps from our ISP. So don't want to download a 4 GB iso and then install where the install download it all again just because it does not look locally first for the install files..

Chris

[phenigma]

Don't remove the internet repo that may interfere with mythtv install due to pinning.

The script does not remove the lmce repo, it moves both the local and lmce repo to the top of sources.list.  Prioritizing them over ubuntu repos.  Then the script signs the local repo.  Internet repo access is still completely available and following the installation everything still works as it has been...  nothing needs to be 'undone' following the install.  

DisklessCreate_TBZ.sh will still download all its' packages from the net and not use the local repository for building the md images until the local key is added to apt during the chroot process of building the initial md.

J

[Marie.O]

Don't remove the internet repo that may interfere with mythtv install due to pinning.

The script does not remove the lmce repo, it moves both the local and lmce repo to the top of sources.list.  Prioritizing them over ubuntu repos.  Then the script signs the local repo.  Internet repo access is still completely available and following the installation everything still works as it has been...  nothing needs to be 'undone' following the install. 

I was not aware that the order of sources.list is relevant. The last few snapshots have had the local repo pinned at 600 so it should already have been taken over the internet repo.

[phenigma]

I read that here: http://kudos.berlios.de/kf/kf1.html#netrepo so I implemented it being safe.  And... the man page for sources.list, 2nd sentence, 2nd paragraph 'The file lists one source per line, with the most preferred source listed first.'

Pinning the local repo at 600 is functionally no different than 500 if the local repo is 'untrusted'.  The snapshots I'm using have the local repo pinned at 600 and it still downloads everything from the net.  My testing shows that the local repo doesn't overtake the internet repo until it is pinned above 990.  The problem then is that updates on the internet repo are ignored in favour of outdated versions on hte local repo.  Not desireable.  It works very much as described here: http://linux.die.net/man/5/apt_preferences, under 'How Apt Interprets Priorities'.  Fixing the sources.list order and making the local repo trusted fixes these issues completely.

I have tested these changes with the local repo pinned at 600 and 500 and in my tests it works either way when the local repo is trusted and sources.list is ordered correctly.

J.

[phenigma]

Okay, so after a little more testing, a little more specific.

By default the local repo is actually being prioritized over deb.linuxmce.org because of their order in sources.list and neither is trusted and this works regardless of pinning at 500 or 600 for the local repo (I removed the local repo pinning from preferences and the local repo is still prioritized over deb.linuxmce.org, because of the sources.list order.)

Unfortunately the ubuntu internet repos are prioritized over the local repo becuase they are 'trusted', and the local repo is not, regardless of their order in sources.list.  Pinning the local repo >990 will overtake the ubuntu net repos but will also prevent ubuntu repo upgrades to newer versions of packages that have older versions in the local repo.

To overcome this: leave the pinning at 500 (or remove it since 500 is default anyways), order sources.list by preferred source and sign the local repository.  Everything 'just works'... fast.  :-)  I've tested this pretty extensively from lots of different angles now.

My script makes changes that should probably be implemented in:
changes to: pre-install-from-DVD.sh, pre-install-from-ISOs.sh pre-install-from-repo.sh for sources.list order
changes to: makedvd.sh for local repo pin-priority (it's still 600 and doesn't need to be)
changes to: mce-install.sh/mce-install-common.sh to generate a key/import to apt and sign the local repo

Unfortunately I'm leaving in 5 hours for 2wks so I don't have time to set up a builder and attempt to create a proper patch before then.

J.

[Marie.O]

phenigma,

thanks for digging a bit deeper.

The DVD will be modified accordingly, i.e. remove the pinning for the local stuff, sign the local repo, and put the local repo line first.

Thanks again for the digging, it is greatly appreciated.

[phenigma]

Your very welcome.  Thank you!  I'll dig into more when I return.

J.

[phenigma]

Okay, since I'b been back I've been playing a little... and although I believe that signing the repo is the ultimately the proper way to go, it needs some additional glue to keep everything synchronized.

But... I've had great success with removing the pinning, re-ordering the sources.list (placing the local repo above all others) and doing an apt-get update.  No signing required.  I'm going to test this a few more times and when I'm happy that it's not a co-incidence I'll create, and submit, a proper patch.

J.

[Marie.O]

Thanks phenigma. Looking forward to the patch ticket.

[phenigma]

For anyone following along the required changes were submitted to svn and are in the Aug 18, 2010 (23248) snapshot.  No repository signing was required.  If you have signed your local repository using the script you should still have no problems updating your system.

J.

[Marie.O]

Did anyone test the latest snapshot? Feedback is appreciated.