Finally: Development environment up and running - some testing to be done

[JoakimL]

So I finally got the right combination of time, hardware and disk space to be able to set up a working dev environment. I've tested to compile some modules and everything seems to be OK. First PoC will be to create a work-around for a very annoying bug from UpdateMedia that I'm sure all Scandinavian users will enjoy (and German, Spanish - or whoever have to use a media tagged with anything outside of the small English alphabet). Some news on that to follow next year!

/Joakim

[Gangsingen]

Oh yeah!! Thanks alot Joakim. A working updatemedia would make life som much easier!

[bongowongo]

So I finally got the right combination of time, hardware and disk space to be able to set up a working dev environment. I've tested to compile some modules and everything seems to be OK. First PoC will be to create a work-around for a very annoying bug from UpdateMedia that I'm sure all Scandinavian users will enjoy (and German, Spanish - or whoever have to use a media tagged with anything outside of the small English alphabet). Some news on that to follow next year!

/Joakim

Good news, hear from you next year.

[JoakimL]

OK, some progress. I now have a working version that doesn't trash the tags in FLAC and MP3 files (and also OGG should work, I did not test it).
But - the data is still wrong in the lmce database. There's some type conversion going wrong, I can see the data from the tags being read correctly but before it's time to write back they're truncated at first non-std latin character.
I have an issue with debugging, but at least I got logging to work so I can dump whatever I want to the log.

Just a little question: There's an issue with non-latin characters (like åäöü) in the database also outside of UpdateMedia. If I add a file from the web admin with such a character in the title it's stored wrong. However, if I edit the field and re-enter exactly the same characters, it's stored OK. Is there anybody that have any thoughts on this?


/Joakim

[sambuca]

It might be that the connection used does not handle UTF-8 in all cases (I think there is a connection parameter for that..). For instance that the UpdateMedia connection does not use UTF-8 while the web-admin connection does?

br,
sambuca

[JoakimL]

Some further digging show the data in the program to be correct until it's saved to the db, it's corrupted when written to the database. Since the database is UTF-8 in all aspects, I assume you're right about the connection being setup wrong. I tried with setting [client] default-character-set=utf8 in my.conf with no effect.
A general question: Is there any known C++ program in the system that successfully use UTF-8? I think I've tracked down how mysql is accessed, but I fail to see where I could specify UTF-8 as character set in the connection string.

/Joakim

[JoakimL]

OK, one step further. I now have a version that not only does not trash the media files, it also updates the database correctly. Proof:
   
   Album   Äntligen!   Edit Remove
   Performer   Peps Blodsband   Edit Remove
   Title   Alldeles Lagom   Edit Remove
   Track   11   Edit Remove

Basic problem - media file coded with latin1 and database with utf8. My current code is hard coded to do a database connect with latin1 and I also modified the Attributes table to store latin1 data.
There are two options to solve this:
1. New property where you can state which character set to use. For every Attribute.name I can convert to utf8 and then store the data. Connection string always utf8.
2. New property where you can state which character set to use. SQL script to alter the Attributes table and a connection string accordingly.

Do you have any design inputs on this? What will work best with the PHP code driving web admin?


/Joakim

[Marie.O]

I am not deeply into this, but is it possible, that the ADODB lib (which is used by web admin for database access) is defaulting to latin1 instead of utf8.

[JoakimL]

I haven't looked at the PHP code, but I believe it's utf8. I tested to add some Swedish characters (åäö ÅÄÖ) to the standard 10.04 installation and it both displays and updates these correctly. With my modified table it's displayed correctly but when I change data it's corrupted.
With some more thinking, I lean towards option 1) - it would mean all installations have the same configuration in the database, no special handling would be needed except for UpdateMedia.

/Joakim

[sambuca]

Be sure to look up the doc on every metadata format to see if it says anything about default encoding. I would think that ogg and flac metadata has a default encoding(utf8 maybe?). For mp3/id3 on the other hand, I don't think the standard specifies anything, but I would recommend you to double-check that.

I've also seen some mp3-tags where there is an encoding string at the beginning. Don't know how common that is, but maybe it's something to examine further?

I would use UTF-8 as much as possible, and convert to (and from) utf8 only at the input (and output). The database should already be utf8, so from what I understand, option 1 seems the best. Maybe even default to latin1 for metadata formats that doesn't have a standard encoding?

br,
sambuca

[JoakimL]

Good advice, I'll look at the various media file specifications.

/Joakim

PS. A peek into what's running at my desktop now:

[JoakimL]

Just another status update: my local LMCE is now doing correct latin1 to UTF-8 encoding for MP3 files. And best of all - the tags in the MP3 are _not_ trashed any-more. FLAC is not working, seems to be encoded in a different way.
So what's left before I release this for alpha-testing?
- FLAC support
- Read config from db (now hard coded)
- I need to look at startup scripts and the UpdateMedia daemon, it cannot start before character encoding is set (otherwise it will trash the tags as before).
- I also need to alter the Media Browser to include some extra characters in the left panel.

/Joakim

[Gangsingen]

Any progress here?