Auto tagging folder - just a thought

[colinjones]

Was just thinking this over - for those of us who acquire lots of video media, such as TV shows via, say, torrents - tagging manually all the time can be a pain (esp compared with ripping DVDs and mp3 which usually get tagged automatically)

How about if you could mark certain folders under videos as auto-tagging folders. Such that you might have a "movies" folder, that automatically tags any media placed in there as of Movie type, same for Music Videos, etc. Even subfolders that might tag Genres, perhaps Years, or Show, etc. True, it wouldn't be practical to set Titles and so on in this way. But the advantage would be that many people already have a folder structure like this, and merely by directing their BT client to download to the correct folder, you could set 3 or 4 of the general attribute types automatically.

This would assist in increasing the usage of the various other Sort modes other than Filename. I know its almost heresy here, but I have to confess that I do use Filename mode very regularly for this reason! The system could be generic so that there isn't a defined folder structure, you can simply mark the folder you want to have the autotagging feature and then list the attributes you want to set, and their values (perhaps this could be extended at some point to include some level of macro/regular exp)

This could either be integrated into UpdateMedia or be a separate function completely that accesses the database directly to stamp the attributes. The only things I can think of that would need to be addressed would be whether it overrides existing tags, or whether you would have to manually go to the admin console and tell it to override; whether it would look at existing files in a folder just configured for autotagging or only new files....

[samuelmukoti]

Hi,

I've also been looking at the same thing.  I've moved from XBMC to LMCE recently, and its cetainly one think I miss  TV Shows and Music Videos autotagging. 

Downloaded the source of XBMC and Entertainer (another MediaCenter) and am looking a stripping some of the tagging code and adapting it to LMCE.  Maybe I'll be able to come up with a script that we can hook into UpdateMedia so we can get auto tagging done. 

My experience with XBMC tagging is very positive, 90% it gets it right - and also passes .nfo files, which is also nice. We could go a step further and even support libmusicbrainz too for audio tagging - the sky is the limit. 

Lastly been able to manage this from the onscreen orbiter would also be a bonus, as my wife would not be interested  bringing the WebAdmin - as it is, it intimidates her, and she's not interest in going in there when ever she upload her media from her laptop to the NAS drive.

Will post my further progress on this thread.. I'm really excited about this.. and if someone has worked on this before i'd love to chat in IRC and put our heads together to get the best possible experience in LMCE

Was just thinking this over - for those of us who acquire lots of video media, such as TV shows via, say, torrents - tagging manually all the time can be a pain (esp compared with ripping DVDs and mp3 which usually get tagged automatically)

How about if you could mark certain folders under videos as auto-tagging folders. Such that you might have a "movies" folder, that automatically tags any media placed in there as of Movie type, same for Music Videos, etc. Even subfolders that might tag Genres, perhaps Years, or Show, etc. True, it wouldn't be practical to set Titles and so on in this way. But the advantage would be that many people already have a folder structure like this, and merely by directing their BT client to download to the correct folder, you could set 3 or 4 of the general attribute types automatically.

This would assist in increasing the usage of the various other Sort modes other than Filename. I know its almost heresy here, but I have to confess that I do use Filename mode very regularly for this reason! The system could be generic so that there isn't a defined folder structure, you can simply mark the folder you want to have the autotagging feature and then list the attributes you want to set, and their values (perhaps this could be extended at some point to include some level of macro/regular exp)

This could either be integrated into UpdateMedia or be a separate function completely that accesses the database directly to stamp the attributes. The only things I can think of that would need to be addressed would be whether it overrides existing tags, or whether you would have to manually go to the admin console and tell it to override; whether it would look at existing files in a folder just configured for autotagging or only new files....

best regards,

Sam

[bulek]

Hi,

I'm also missing autotagging feature. I think this should be available as an option for each directory (or all directories).
I think what we need is:
- to get more info about current id3 format (it's binary)
- see if it can be written by scripting language
- write import script that will autofill id3 regarding various standard ways of tagging files (like folder.jpg being screenshot of folder, like same name.jpg means screenshot for the file name.xxx, etc...)

- it would be also nice feature (Freevo has this and it's quite functional) if you could change certain parameters for each directory (for instance reverse time order of display for surveillance videos, filter for displaying only certain files, etc...). I used Freevo some time ago and a lot of nice things comes out of that....

Regards,

Bulek.

[colinjones]

Bulek - I don't think you need to go to the extent to writing id3 files - if you insert the metadata into the database, then UpdateMedia should just create the id3 file for you on the next path. If a trigger is needed then I'm sure that the code for this is in the web php code for the admin site when you manually tag a media file. This writes to the database and UpdateMedia on the next pass creates the id3 file, so the logic has to be there somewhere.

[jthodges]

Yes, an auto-tagging feature would be great!  I wrote a small app to help me with my tagging needs, and it's working pretty well so far, but it would be great to have something more integrated.  I'm probably not doing things the right way, but its a standalone app that reads from the media database with an optional filtering clause and lets you walk through and do some basic tagging.  It groups files together based on a common prefix (which would include but not be limited to directories), which helps with (for example) TV shows that might be in a root folder with other files, but share a common prefix.  Avoiding the web interface was also a goal of mine, so I tried to stick with only using big buttons easily hit quickly with the gyro remote.

I only bring this up because I think that I had similar motivations to what you guys are discussing.  Something rule-based that was more automated than what I'm currently using would be fantastic.  I think there probably would still be a place for something in the orbiter that let you walk through and easily tag on a more fine-grained basis, though.

If anyone is interested in this app in the meantime, I'd be happy to share the source.  It still needs some extensions, like offering the other media and file types. 

[golgoj4]

lol, i should really get back to finishing what i started before. Seems like there are more people wanting this so is there a common language we can collaborate in?

Im ok with php/mysql
still learning the javascript
abandoned the adobe flash route.

i think we need to really push and get this done.

i will track down and strip out the code I have later and maybe see how to get this going once and for all. Where I left off was at the point of being somewhat overwhelmed by all of the things I wanted to add while at the same time realizing adobe flash was making the job way harder than it should have been (like smarter folks than me told me already  )

i figure i we combine all these attempts in to one master one, we can get something done that meets our goals.

-auto tagging would be nice
-extended metadata grabber via imdb lookup or other method. there are a few api's out there for this.
-drag and drop tagging for media

anyone else have any ideas?

[jthodges]

I've been using the auto-tagging part of it as an opportunity to get into development.  I have a 7.10 development environment set up and have been looking into adding the auto-tagging code to UpdateMedia.  My plan was to have rules attached to each folder, and update the UpdateMedia::ScanFiles method to load rules for each folder, and apply them to files that are detected as not yet in the database.  These rules could be recursive or not, and example rules would be:

• Apply media sub type XYZ recursively
• Apply title based on regular expression (e.g., "24 - Season X Episode Y" based on filename)

Version 1 was to just have a few simple rule types (short and long attributes), loaded from a text file.  After that, a front end within the admin site (though I have zero php experience, so I wasn't looking forward to that part).  I was thinking nice to have features would be allowing custom rule types, and these rules should probably live in the database instead of the text file.

Does this sound in-line with what you were considering?  I'd definitely be interested in combining efforts ... the only reason I didn't mention that I was working on this was because getting the dev environment set up seemed like such a daunting task, and I didn't want to commit and then bail.

[chriss]

Hi guys,

I've been thinking about starting to redo the media management part in JavaEE, e.g., using the Seam framework for a web application. This would give us the opportunity to have a nicer GUI while sticking to the approach of being able to access your system via a simple web browser.

What do you think?

/chriss

ps I implemented a grabber for IMDB attributes for the current web-admin, check the svn...

[golgoj4]

Hi guys,

I've been thinking about starting to redo the media management part in JavaEE, e.g., using the Seam framework for a web application. This would give us the opportunity to have a nicer GUI while sticking to the approach of being able to access your system via a simple web browser.

What do you think?

/chriss

ps I implemented a grabber for IMDB attributes for the current web-admin, check the svn...

im all for working together on this. I guess to get this seriously started we need:
-a wiki page laying out what we want to do
-volunteers to pick off a part and go to work on it
-....
-profit!!!

My main issue has been interpreting the ADODB that is used. I never tied anything into the webadmin because i could barely figure out what as going on. That should be a major focus so that we dont create problems down the line.

[colinjones]

jthodges - I definitely think it should be recursive at least as an option (default?) The thought was that you could potentially set up a folder structure like genre->series->episode or genre->artist->album, and then apply the folder attributes so that the genre flows all the way down, the artist only flows down the albums and tracks that it applies to and album slows to all is tracks. Should it have a "once off" vs "dynamic" radio button that tells UpdateMedia to restamp either only new media files or even media files that get moved from one location to another to aid management? If the latter or only a "new media" option then my other point of being able to track renamed/moved files better (perhaps by stamping the installation number and PK_File key) becomes more important as currently UpdateMedia considers a renamed or moved file to be a new one...

I think a simple "regular expression" concept would be good, but perhaps a true Regular Expression might be a bit complex (and unnecessarily comprehensive) for the average user?

golgoj - I thought your effort was focused on playlists? Would definitely like to see that expanded on jon's work (which is great). But using the same logic to do the manual tagging side of things would be great, too!

[golgoj4]

jthodges - I definitely think it should be recursive at least as an option (default?) The thought was that you could potentially set up a folder structure like genre->series->episode or genre->artist->album, and then apply the folder attributes so that the genre flows all the way down, the artist only flows down the albums and tracks that it applies to and album slows to all is tracks. Should it have a "once off" vs "dynamic" radio button that tells UpdateMedia to restamp either only new media files or even media files that get moved from one location to another to aid management? If the latter or only a "new media" option then my other point of being able to track renamed/moved files better (perhaps by stamping the installation number and PK_File key) becomes more important as currently UpdateMedia considers a renamed or moved file to be a new one...

I think a simple "regular expression" concept would be good, but perhaps a true Regular Expression might be a bit complex (and unnecessarily comprehensive) for the average user?

golgoj - I thought your effort was focused on playlists? Would definitely like to see that expanded on jon's work (which is great). But using the same logic to do the manual tagging side of things would be great, too!

ive probly done about 3 different versions (only showed the 1st) because I hated all of them I expanded it to media and playlists and gave myself a real headache. I dont have the largest media collection, but at a couple thousand video and media files managing the metadata in a way that was quick and intuitive became a road block for me.

anyways, back to teh drywall

http://langstonball.com/images/stories/LinuxMCE/howto/videopanellayout3.png
this is what the video attributes page last looked like. Attributes on the left with the file list in the middle bottom and the actual file data in the top middle. The empty spaces are where I intended the episode of movie description to land. Its a lot of info and I suppose its hard to decide whats important and whats not. And for some reason im to stupid to embed it properly atm.

[colinjones]

funny, from what I remember it looked really cool! BTW, that link doesn't seem to work (404)

[jthodges]

jthodges - I definitely think it should be recursive at least as an option (default?) The thought was that you could potentially set up a folder structure like genre->series->episode or genre->artist->album, and then apply the folder attributes so that the genre flows all the way down, the artist only flows down the albums and tracks that it applies to and album slows to all is tracks.

Yeah, I agree recursive is possibly the only useful mode.  The management UI may want to hide the option to turn it off altogether to keep it simple.

Should it have a "once off" vs "dynamic" radio button that tells UpdateMedia to restamp either only new media files or even media files that get moved from one location to another to aid management? If the latter or only a "new media" option then my other point of being able to track renamed/moved files better (perhaps by stamping the installation number and PK_File key) becomes more important as currently UpdateMedia considers a renamed or moved file to be a new one...

Hmm, could you clarify this one?  I was under the impression that moved/renamed files were recognized and handled by UpdateMedia, and some (very minimal) testing showed the same.  I was considering the need for a "Run Now" for the rules to allow you to catch up your existing media, but it sounds like you are getting at something different.

Either way, moved files is definitely something to consider.  I think they would keep their tags (which I think would happen automatically), except as overwritten by any autotagging rules on the new directory. 

I think a simple "regular expression" concept would be good, but perhaps a true Regular Expression might be a bit complex (and unnecessarily comprehensive) for the average user?

I agree.  What I really want to accomplish is automatic titling based on standard file names (such as S01E06 type naming used for a lot of tv shows).  Perhaps the answer is to put some stock, well thought out regular expressions behind the scenes and just allow the user to write something like "24 - [Season].[Episode]".  Any other ideas?

[colinjones]

jt

I was told in the past, as well, that UM can handle moves/renames. It simply isn't true - or at least I have never seen an instance where it has worked. Theoretically, there are several triggers it could use as the db stores an MD5 hash (not sure how it gets this, whether it calculates it which seems unlikely due the the network load, or relies on the remote fs to supply it) and the inode number (presumably of the first block, but not sure how this works for non unix-like fs's but certainly all my media is on an NTFS share and these attributes are filled out fine!)

I was told it uses the inode. But in practice, if you rename a file or move it, UM finds it as a new file in the new location (you can see this in the UM log) and in the admin site, if you click the option to show missing files, you will see the old file in the old location. This not only prevents LMCE from treating a file as a continuous, discrete object in its logic, but also ensures that the database gets messy unless you scrupulously clean up or only use the admin site to do the moves/renames. That's a thought - you didn't think I mean via the admin site did you? Of course that works, it would be pointless otherwise. I'm talking about using any other method to move/rename files, LMCE terminal, Windows Explorer, NAS management tool. It is blinkered to say that file management should only ever be done within the admin console - there are many instances where this simply isn't practical or even possible

On the same subject, duplicating files is another difficult one. If you use only the MD5 hash to uniquely identify a file, then duplicating will screw up the logic as both files will match. This seems to indicate that a combination of MD5 hash and 'inode' need to be combined in the logic to complete it.

If a "new" file is encountered that has the same MD5 and inode as an "existing" file -> check does the "existing" file still exist? If no, then action was a move or rename -> update existing db record; else error?
If a "new" file is encountered that has the same MD5 but different inode as an "existing" file -> check does the "existing" file still exist? If yes, then action was a copy -> create new db record, and copy old db record's attributes over; else error? (although this could potentially still be a move/rename, just between different filesystems)

?

[jthodges]

As I said, my test was simple and so didn't cover a network drive - that may be the source of the confusion.  I shutdown LMCE, moved my file and deleted the id3, and started LMCE back up.  The id3 reappeared, and the tags were still correct in the orbiter.  Maybe I'll try a network drive later on, but I can see what you mean about why it would possibly be different.

You are correct about the inode and MD5 use.  The relevant logic is in PlutoMediaFile::HandleFileNotInDatabase in PlutoMediaFile.cpp.  It looks for a File row with a matching inode, and then checks the MD5 of that row against the new file.  If they match, it is recovered.  Duplicate files should not be an assue, because the inode will not match.

Do you know if there's already a bug in the tracker for this?  My thoughts are that if this is not working correctly, that should be addressed directly in the existing UpdateMedia code rather than worked around with autotagging.  I'm not sure though, to be honest I'm still just getting my bearings.  It will be a little while before I get the basic functionality down, so we can kind of mull the options over in the meantime.