Author Topic: Backup System Improvements (0810 Critical)  (Read 10012 times)

jondecker76

  • Alumni
  • wants to work for LinuxMCE
  • *
  • Posts: 763
    • View Profile
Backup System Improvements (0810 Critical)
« on: January 06, 2009, 10:04:58 am »
Another project of mine for the weekend (I finally get a few days off work after months of forced overtime) is to work on the Backup System (PlutoRestorePoint.sh).
During an upgrade, this same restore system creates and restores a backup. The problem is, especially during an upgrade, that Pluto_Media nor Mythconverge is not taken into account. Because of this, a lot of things are lost, including:
Bookmarks
Playlists
Scheduled recordings will again start recording duplicates as this data isn't restored
etc.

I propose to include both Pluto_Media and MythConverge in the backup - this is essential to any upgrade!

And as far as doing manual backups this way, I see no problem with it either..  Some logic will just have to be in place to do the restore safely as not to get mixed up with media files added after the backup was made.

Or perhaps just make a flag so that media/mythconverge is only included during the backup/restore done during upgrade.

Any comments?

indulis

  • Veteran
  • ***
  • Posts: 147
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #1 on: January 13, 2009, 03:13:46 am »
Jon,

If you are talking about media added since the last backup point, then something which would allow a "roll forward" of as much data as possible could be a background task that gets all associated information that relates to a media file, and creates a "bundle" that can be used to restore one piece of media. 

This would also be v useful for the case where someone wants to do a complete new install but not lose all their data.

I have had this in mind for a while since using Mythtv originally- that there should be a way to restore or move a single recorded program to another system- same process.

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #2 on: January 13, 2009, 04:33:13 am »
indulis

I don't think jon is talking about some kind of "incremental" backup system because he is talking about backing up the database data not the actual media files themselves. Because media file's metadata is also stored in either an .id3 file or embedded id3 tag, there is no real call for backing up that part of the metadata - on complete reinstalls, all this metadata is automatically recovered by default and injected back into the fresh pluto_media database.

The issues that jon points out are for non-media file type things like playlists, recordings, book marks, etc. As these don't have a physical presence on disk as "media files" there is no id3 storage of the metadata they contain. Thus these are lost during a rebuild. Jon has already done work on a playlist import/exporter, so I guess this is a logical extension of that.. good work jon!

chriss

  • Veteran
  • ***
  • Posts: 140
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #3 on: January 13, 2009, 05:20:00 am »
I already thought about extending UpdateMedia based on Jon's code to automatically sync playlists to a directory, e.g., /home/public/data/playlists/. As a side effect, this would allow to keep the playlists over new installs and db failures like the metadata for media files is kept right now.
Maybe this approach can be used for other data, too, since I think it is quite nice... (had metadata wiped twice within the last few weeks... luckly I didn't lose any information because of UpdateMedia :))

just my 2c
/chriss

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #4 on: January 13, 2009, 07:37:11 am »
chriss - that would be fantastic! (and in fact is the intent of a mantis/trac ticket that I logged a long time ago, still there I think)

Marie.O

  • Administrator
  • LinuxMCE God
  • *****
  • Posts: 3676
  • Wastes Life On LinuxMCE Since 2007
    • View Profile
    • My Home
Re: Backup System Improvements (0810 Critical)
« Reply #5 on: January 13, 2009, 10:31:41 am »
[..]Because media file's metadata is also stored in either an .id3 file or embedded id3 tag, there is no real call for backing up that part of the metadata - on complete reinstalls, all this metadata is automatically recovered by default and injected back into the fresh pluto_media database.[..]

are you sure about this? Did you verify this with video files and with flac encoded audio files?

rgds
Oliver

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #6 on: January 13, 2009, 10:39:02 am »
Yes, definitely - I think you may not have read my post completely. Note, I said either as an embedded id3 tag or a .id3 file

The id3 files (not embedded tags) were introduced for those file formats that were not compatible (or supported in LMCE) with embedded id3 tags, such as flac and avi's etc. That is their entire purpose. UpdateMedia looks for these as well, esp on the first scan - if they already exist, then it simply imports the metadata from them rather than creating new ones (in fact it never creates new id3 files for this type of media on the first scan - as there is no metadata in the database already for the new media file, and there is no id3 file, then there is no metadata so it doesn't bother to create an id3 file)

jondecker76

  • Alumni
  • wants to work for LinuxMCE
  • *
  • Posts: 763
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #7 on: January 13, 2009, 10:53:21 am »
the .id3 file usage of LMCE is nice, however it is only for media meta data. The problem is, that playlists, bookmarks and the like all reference primary keys of the file, therefore just letting LMCE re-detect the metadata from ID3 files after a reinstall will not work if you want to keep playlists and bookmarks etc. because the primary keys will very likely be different. The only way to preserve this kind of stuff would be to backup pluto_media as well.
Like others, I have lost all playlists, book marks and any other feature that references primary keys of media with each upgrade. It would be nice to change this before 0810 is released.

Marie.O

  • Administrator
  • LinuxMCE God
  • *****
  • Posts: 3676
  • Wastes Life On LinuxMCE Since 2007
    • View Profile
    • My Home
Re: Backup System Improvements (0810 Critical)
« Reply #8 on: January 13, 2009, 10:59:25 am »
Yes, definitely - I think you may not have read my post completely. Note, I said either as an embedded id3 tag or a .id3 file

The id3 files (not embedded tags) were introduced for those file formats that were not compatible (or supported in LMCE) with embedded id3 tags, such as flac and avi's etc. That is their entire purpose. UpdateMedia looks for these as well, esp on the first scan - if they already exist, then it simply imports the metadata from them rather than creating new ones (in fact it never creates new id3 files for this type of media on the first scan - as there is no metadata in the database already for the new media file, and there is no id3 file, then there is no metadata so it doesn't bother to create an id3 file)

Colin,

I have a fairly large collection of MP3 files. I went thru the web admin to tag them properly, and things showed up nicely in lmce. Unfortunately, upon a reinstall, none of the change stuck.

rgds
Oliver

bulek

  • Administrator
  • wants to work for LinuxMCE
  • *****
  • Posts: 909
  • Living with LMCE
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #9 on: January 13, 2009, 08:59:33 pm »
Hi,

if I may add some wishes for backup it would be also nice if created scenarios and event handling could get into. In this way one could build own set of frequently used stuff. It would be also nice if one could click and select, what info will go into backup.

For instance I'd like to replicate only scenarios on another machine and all everything else...

Thanks in advance for effort,

regards,

Bulek.
Thanks in advance,

regards,

Bulek.

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #10 on: January 13, 2009, 10:08:02 pm »
Yes, definitely - I think you may not have read my post completely. Note, I said either as an embedded id3 tag or a .id3 file

The id3 files (not embedded tags) were introduced for those file formats that were not compatible (or supported in LMCE) with embedded id3 tags, such as flac and avi's etc. That is their entire purpose. UpdateMedia looks for these as well, esp on the first scan - if they already exist, then it simply imports the metadata from them rather than creating new ones (in fact it never creates new id3 files for this type of media on the first scan - as there is no metadata in the database already for the new media file, and there is no id3 file, then there is no metadata so it doesn't bother to create an id3 file)

Colin,

I have a fairly large collection of MP3 files. I went thru the web admin to tag them properly, and things showed up nicely in LinuxMCE. Unfortunately, upon a reinstall, none of the change stuck.

rgds
Oliver

possy - hmm, interesting, I don't think I have ever tried that. The vast majority of my mp3 files have tags already embedded from the original rip. These get imported by UpdateMedia into the database. Upon rebuild, of course, they get reimported again so appear to be persistent. I have only edited a small number of mp3 tags on the LMCE side, so if these reverted I probably wouldn't have noticed.

I have to say though, I have noticed instances where I have edited mp3 tags in LMCE, and the changes did not appear to be reflected in the embedded id3 tags in the mp3 file when viewed in Windows. But I assumed that was more a Windows problem (particularly to do with NTFS "streams")... perhaps not

But there are definitely instances where the mp3 tags have been updated visibly in Windows... so under some circumstances it does work. Perhaps UpdateMedia can only export to an embedded id3 tag if one already exists? Or perhaps only if one doesn't exist? Either way, if UpdateMedia exports correctly to the id3 embedded tag, then your metadata is definitely persistent, as on rebuild it will import again from this. I really don't know how Windows decides whether to use a "stream" or an embedded tag or both (most likely), and how it coordinates any changes to this, but this could be implicated as well. For mp3s I would recommend editing the embedded tag directly using something like mp3tag as this is reliably imported into LMCE

Note - I did read somewhere on the forums recently that UpdateMedia only imports from embedded tags the "first time". Not sure what was meant by that, but on the face of it, this isn't true. I do single and bulk changes to mp3 tags, and can see updatemedia scanning them in on the next pass without having to do a manual scan. I believe updatemedia is just looking for date stamp changes (or possibly md5?) and rescans fully each file that is different.

But for other media like video, id3 files are immediately created as soon as you tag them. These are always scanned back in on rebuild. And there is less uncertainty on the Windows side because NTFS streams do not apply in this case, of course.

Marie.O

  • Administrator
  • LinuxMCE God
  • *****
  • Posts: 3676
  • Wastes Life On LinuxMCE Since 2007
    • View Profile
    • My Home
Re: Backup System Improvements (0810 Critical)
« Reply #11 on: January 13, 2009, 11:57:55 pm »
Colin,

my assumption is, that it works on tags that are only relevant for a single file. But I really don't know for sure. It is on my list of things to make sure, that every media tagging that happened, is going to flow from one side to the other, no matter, where the tagging took place.

rgds
Oliver

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #12 on: January 14, 2009, 01:05:15 am »
That'd be great, possy! Let me know if you need any help with testing/research. Improving the ability to identify moved/renamed files (with interfaces other than the web admin) would be good too - perhaps add a LMCE field that corelates with PK_file and the installation number so that UpdateMedia can identify a file that has simply been moved or renamed and deal with it correctly (by modifying the existing record rather than creating a new one). Also, expanding beyond embedded id3 tags in mp3 files would be good too...

bulek

  • Administrator
  • wants to work for LinuxMCE
  • *****
  • Posts: 909
  • Living with LMCE
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #13 on: January 14, 2009, 11:37:48 am »
I also have a fairly big collection of mp3s...

And I decided to do embedded mp3 tagging with some other utility - cause in this way mp3s are usable also in other systems...

But then I discovered that LMCE corrupted some of embedded tags in files - so now I keep separate copy of whole library, do mp3 tagging in there and then only transfer files in one way to LMCE.... No more messes after that....

I think it is not so good idea to encourage users to tag their media under LMCE, cause there are some potential bugs that could ruin tagging work.,.. But this is just my personal opinion and I could be wrong...

HTH,

regards,

Bulek.
 
Thanks in advance,

regards,

Bulek.

chriss

  • Veteran
  • ***
  • Posts: 140
    • View Profile
Re: Backup System Improvements (0810 Critical)
« Reply #14 on: January 14, 2009, 12:35:09 pm »
I would suggest to log all bugs in Trac and give a description as detailed as possible... this way the bugs won't get lost.