LinuxMCE Forums

General => Feature requests & roadmap => Topic started by: erasmix on April 24, 2008, 08:02:09 pm

Title: New Media Navigation Facility
Post by: erasmix on April 24, 2008, 08:02:09 pm
Hello,

I have all my media files organized on a directory tree stored on a separate box. I have ~1200 video files, not to mention my MP3s  (+10,000), nor my pictures. I use a MacMini with FrontRow to  watch those files on one TV, and Linux MCE on my other TV. The only think I like about the Mac (and the reason I spent $500 on it) is the FrontRow application which offers an incredibly simple and smooth way to navigate thru my media files.

On the other hand LinuxMCE has EVERYTHING else that the mac is missing, including stability (the mac often crashes playing DivX videos probably because I installed an unsuported codec). The only thing is missing is a navigation facility as good as Apple's FrontRow. Currently I use the search facility, which works OK, but it takes me a while to locate the video I'm looking for.

I think a Media Navigator capable of browsing thru a directory tree would be a fantastic improvement for LinuxMCE. Is this planned?
Title: Re: New Media Navigation Facility
Post by: golgoj4 on April 24, 2008, 08:30:44 pm
i use browse by file to locate my media all the time. maybe im missing something in your post. can you be more specific?

Which UI are you using?

under UI-1, on the right hand side (when looking at the video files or audio listing) there is an 'options' button. The option to change the sort is on the screen that comes up when you tap 'options'.

under ui-2 its on of the options in the lower left corner. you can sort by file, performer, genre, etc.

hth

-golgo
Title: Re: New Media Navigation Facility
Post by: erasmix on April 24, 2008, 10:50:02 pm
I think I use the UI-2 (is that the nice one with blended buttons on top?), because I have the Sort button on the left hand lower corner. When I select sort by Filename, I only see:

tv_shows_0
tv_shows_1

Only when I sort by Title, I see all my files, but its quite unmanageable. So I use the "Search" option under "More", but as I said in the original thread its kinda tedious.

Is it possible that sorting by Filename doesn't work for me, because the devices (they're SAMBA shares) are READONLY? 

Let me know your thoughts.

Gracias,

Erasmo
Title: Re: New Media Navigation Facility
Post by: royw on April 25, 2008, 12:55:29 am
What version of LMCE are you using? Are you using CIFS, SAMBA, or NFS to access the remote drive? 

BTW, I've noticed on 0710 beta 4 that not all automount drives are showing up under the filename browse.  For example, it seems to only find one of my NFS shares.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: colinjones on April 25, 2008, 02:42:18 am
I think I use the UI-2 (is that the nice one with blended buttons on top?), because I have the Sort button on the left hand lower corner. When I select sort by Filename, I only see:

tv_shows_0
tv_shows_1

Only when I sort by Title, I see all my files, but its quite unmanageable. So I use the "Search" option under "More", but as I said in the original thread its kinda tedious.

Is it possible that sorting by Filename doesn't work for me, because the devices (they're SAMBA shares) are READONLY? 

Let me know your thoughts.

Gracias,

Erasmo

then you definitely have not got your media shares working yet. In Filename sort mode, you can navigate through your file/folder structure as you would in a file server. If they were not showing up on any sort I would say you have a protocol/online/offline detection issue, but you say you see them in the Title view, so they are definitely online.

Most likely suggestions:

1 - you screwed the system up when you tinkered with the DHCP, need to be very au fait with LMCE to get back from this point - easiest to reinstall, or

2 - when it auto detected the shares (and as I think you may have added them manually, this could be the issue) it asks if you want to use the LMCE folder structure, you need to have answered yes to this....

If the latter, did you add the shares manually? If so, look in the web admin - I think there is a property there for the share device that says to use LMCE structure - I have NO idea whether you can just change that and expect it to fix it up, or whether that was something that had to be done when the share gets auto detected.

The test - look in the root of your share, if they don't exist already it will create a "public", "user_1" & "user_2" folder and a "data" folder under each of these, then video/audio/etc folders under those. If so, you are in business - move all your media to the appropriate folder, and LMCE will work properly from then ... don't worry about it too much, the inconvenience is extremely minor and transient (your Mac will be able to handle it!) but the pay off is huge, don't agonise over it, just do it!

NB: with this, as your media DB has already been populated, it will likely end up with lots of duplicates - probably best to delete everything and rescan

If not, you are going to need to delete your shares, clean up the database, re-enable DHCP and allow it to autodiscover them - and say "yes" when it asks you to use LMCE folder structure, done...

Title: Re: New Media Navigation Facility
Post by: erasmix on April 25, 2008, 05:06:03 am
Fair enough. I should have known better  :) I will reinstall everything again this weekend. If I don't post on this thread again, it means you were right and everything worked fine after the reinstall.

Thanks again,

Erasmo.
Title: Re: New Media Navigation Facility
Post by: royw on April 25, 2008, 06:58:11 am
If the latter, did you add the shares manually? If so, look in the web admin - I think there is a property there for the share device that says to use LMCE structure - I have NO idea whether you can just change that and expect it to fix it up, or whether that was something that had to be done when the share gets auto detected.

It's the PK_Users item that let's you choose between "Public", "Pluto's structure", and any user names you have.  Not the most obvious as the default is "Public".

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 25, 2008, 07:13:28 am
Yes, I added it manually. There is a PK_USERS and it's set to public. Other values are my user name or "Pluto's Structure". If I change it to Pluto, will it alter the structure of share directory? The share is readonly.
Title: Re: New Media Navigation Facility
Post by: royw on April 25, 2008, 09:38:53 am
If PK_Users is set to "Public", then there should be a link to the share under /home/public/data/other.

If PK_Users is set to "Pluto's structure", then it expects the share to have the following structure:
Code: [Select]
public
public/data
public/data/audio
public/data/documents
public/data/pictures
public/data/videos
user_1
user_1/data
user_1/data/audio
user_1/data/documents
user_1/data/pictures
user_1/data/videos
If you have multiple users, then there should be a user_N directory for each user.

For NFS shares, LMCE expects ownership to be root:public (actually 0:1002).  I don't know ownership/permission requirements for samba shares as I don't use them.

If you create a writable share manually, then LMCE will create this structure on the next LMCE boot (at least I didn't find a way to trigger the creation from the web admin pages).

If your share is read only, then you should manually create the structure on your share.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: colinjones on April 25, 2008, 10:06:55 pm
You will have alsorts of problems if you leave the share read-only.

As you have already guessed LMCE will not be able to create the folder structure for you (which is really the best way of proving that everything is working correctly), but in operation, UpdateMedia will not be able to write out the id3 embedded or file-based tags for all your media. These are pretty fundamental in maintaining consistency between the database and the media shares. Best not to second guess it - just make the share read/write...
Title: Re: New Media Navigation Facility
Post by: erasmix on April 26, 2008, 12:35:49 am
I mount my shares readonly because I don't want any extra data written to them, However if you're telling me that linuxmce will only write ID tags to those files, I'm OK with it. However I don't want additional files/directories created on the shares.
Title: Re: New Media Navigation Facility
Post by: royw on April 26, 2008, 01:51:31 am
Each video will get an extra MovieName.id3 file for the meta-data.

Each picture file (at least *.jpg) may get an .id3 and a .tnj (thumbnail)

I'll let someone else answer on audio files as I don't really use LMCE for audio files (I store the files on a NFS share under public/data/audio but then just download to ipod for listening).

Have fun,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 26, 2008, 09:47:01 am
I created an NFS share. I made it writeable. I see its mounted (rw) by runing the linux command mount from a linuxmce terminal session. I see s bunch of:

Apr 26 00:32:00 neo mountd[2453]: authenticated mount request from 192.168.1.2:690 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:05 neo mountd[2453]: authenticated mount request from 192.168.1.2:841 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:11 neo mountd[2453]: authenticated mount request from 192.168.1.2:992 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:17 neo mountd[2453]: authenticated mount request from 192.168.1.2:730 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:23 neo mountd[2453]: authenticated mount request from 192.168.1.2:845 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:29 neo mountd[2453]: authenticated mount request from 192.168.1.2:961 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 00:32:35 neo mountd[2453]: authenticated mount request from 192.168.1.2:657 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)

in the /var/log/messages file from the server. What is it doing? I hope is creating metadata files, however find /usr/fatboy/CleanTorrents | grep -i id3 on the NFS server doesn't reveal anything. Furthermore, when I go on the web interface for the share, I see Readonly checked. I have created the share twice with the same result. I'm willing to let it finish, but I don't see it creating any metadata. Any ideas?
Title: Re: New Media Navigation Facility
Post by: colinjones on April 26, 2008, 11:23:17 am
If you told it to use the share automatically for storing things like recorded TV (when the share was auto detected) then it will also potentially use the share for that if you are recording TV. If not, then it doesn't matter - alternatively you can configure it not to use it for this purpose....

Audio files (eg mp3) will usually have an id3 tag embedded in the file itself, so you will not see the .id3 file for these. If there is already id3 tag meta-data in the audio files, then UpdateMedia will read this metadata in and write it to the database. If you add/change metadata in the database, UpdateMedia will sync this back to the id3 embedded tags in the audio files.

The most common video container formats don't support embedded metadata - there are ones that do, but because not all of them allow it, I assumed the devs thought it was better to be consistent and create a new file of the same name.id3 to hold the metadata. Same for pics, but there is also the .tnj file which holds the thumbnail for the GUI.

So for pics, it should always create the .tnj file. For pics and video, it should only create the .id3 file if there is some metadata to contain - which will either come from you entering it manually into the database through the web admin, or if it finds a match against something in Amazon for DVD rips.

So if you only have audio, very unusual DVD rips, or video files that you haven't manually created any metadata for, then it is entirely possible that you will not see any .id3 files. Try adding some attributes to a video file, and you will see one appear. Or look for .tnj if you have any pics....
Title: Re: New Media Navigation Facility
Post by: erasmix on April 26, 2008, 05:56:32 pm
Yes, I told it to use the NFS share automatically. I followed the instructions on this tutorial to create the share: http://wiki.linuxmce.org/index.php/Howto:_Using_Network_Shares (http://wiki.linuxmce.org/index.php/Howto:_Using_Network_Shares)

I managed to make the share stay r/w permanently (linuxmce is no longer marking it readonly). I still see the same endless chain of messages in the /var/log/messages on the file server:

Apr 26 08:41:45 neo mountd[2453]: authenticated mount request from 192.168.1.2:727 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:41:51 neo mountd[2453]: authenticated mount request from 192.168.1.2:627 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:41:57 neo mountd[2453]: authenticated mount request from 192.168.1.2:739 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:03 neo mountd[2453]: authenticated mount request from 192.168.1.2:1002 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:06 neo mountd[2453]: authenticated unmount request from 192.168.1.2:654 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:09 neo mountd[2453]: authenticated mount request from 192.168.1.2:732 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:15 neo mountd[2453]: authenticated mount request from 192.168.1.2:883 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:21 neo mountd[2453]: authenticated mount request from 192.168.1.2:998 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:26 neo mountd[2453]: authenticated mount request from 192.168.1.2:695 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
Apr 26 08:42:32 neo mountd[2453]: authenticated mount request from 192.168.1.2:811 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)

And I still don't see any id3 tags. I still cant sort my media by other than "title". I only have videos on this share. All files and directories are writeable everybody.

I have reinstalled my server with 2 network cards, and I haven't messed with the DHCP settings. I get a dynamic IP address from the main server on my network (which I configured to be always the same), and pluto's dhcp runs on the second card (the installer did this automatically). Can someone tell me what am I doing wrong?
Title: Re: New Media Navigation Facility
Post by: royw on April 26, 2008, 09:20:40 pm
There was a bug in 0710 beta 3 where update media was not happening on remote shares.  Sorry, I haven't fully tested this yet on beta 4 or RC1.  On beta 4, update media did add all my previous videos with meta-data from the remote share.  On beta 4, update media has not added any meta data to the few videos without pre-existing meta-data.  New videos ripped with beta 4 have .id3 files on NFS shares.

On the log messages, I'm seeing the same type of log messages on my NFS server.  One message thread I found on google suggested increasing the mount's timeout value to something like 10 minutes from a default of 60 seconds.  Currently it looks like /etc/auto.PlutoStorageDevices sets the timeout to 7 seconds.  I haven't tried changing the timeout and don't know what the ramifications would be under LMCE.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: colinjones on April 27, 2008, 12:38:39 am
on the messages, I'm assuming this is just automounting - LMCE mounts shares and unmounts them as it needs them rather than leaving them mounted all the time, using autofs (I think). Its completely transparent to the user - unless you are sitting there reading the messages log!

As for id3 files - as I said in my previous post, if you haven't added any metadata to your video files manually, then it won't create an id3 file for it - that is how it is supposed to work. No metadata, no need for an id3 file. It sounds like your video files are downloaded torrents, so they won't have any metadata to begin with. Sounds like Roy is talking about ripped DVDs (he actually says "rip") - when LMCE does the rip, it also tries to find some metadata online from Amazon for your DVD. If it finds this, it writes the metadata into the database, and thus UpdateMedia creates an id3 file on the share to contain that metadata as well (ie the metadata is stored in 2 locations).

So he will see id3 files, but you will not. Try my suggestion above - login to the admin site, find one of your video files, add a few attributes to it, hit the sync button, then check your share and you will find an id3 file.

Have you selected Filename in the Sort menu for video media browser? Do you see your folder structure and files in there? If yes, then everything is working. "Sort"ing by other attributes isn't going to work very well if you haven't got any attributes in the database! If all you have are downloaded torrents, then you won't have any attributes for it to sort by - it can't just magically know things like "performer" et al, they aren't included in the torrent so you have to enter those manually. Try ripping some common DVDs and you will see that they have attributes (from Amazon) and will display in the other "Sort"s. Or rip some CDs as their attributes are downloaded from the Internet as well. MANY mp3s you download as torrents DO have attributes already embedded (in fact I would say most), so that is a good test as well.
Title: Re: New Media Navigation Facility
Post by: erasmix on April 27, 2008, 06:49:33 am
Yes I have selected Filename in the SORT menu, but it goes from showing me all my files to just showing tv_shows_1. Nothing else. And I'm not going to put metadata on 1100+ videos.

Going back to my first topic on this email. A better media browsing facility is necessary. That is the main reason so many people bought a mac mini to use as a media player, because its damn easy to browse your media and play. You dont need any metadata. You dont need a complicated share structure. You just mount the share on the mac and navigate through the tree with FrontRow and a 6 button remote. Of course it required installing non-Apple codecs to play DivX files and this makes it crash some times. Not to mention that the SMB and NFS client libraries provided with MAC/OS sux big time, but that's another story :)

I have watched other threads on this forum and there is people asking for this. I think you should consider it. This is the most important feature LMCE is lacking so far.
Title: Re: New Media Navigation Facility
Post by: colinjones on April 27, 2008, 07:42:59 am
I'm not going to argue, I have tried to help and you haven't listened on several occasions.

- I never said you had to apply attributes to 1100+ videos, I just suggested one video so that you can see how it works.

- I never said that you had to apply ANY attributes to get the folder/file navigation working, you don't, it does it automatically

- that you have tv_shows and no folders means you have a problem in your system somewhere

- to state it one final time... browsing by folders and files is there already, out of the box, no further configuration or metadata required. It is extremely easy to use, and I can navigate around many thousands of video/audio files in seconds. It is completely intuitive to use as it is exactly what you will be used to in other systems, ie a simple heirarchy of files and folders reflecting what you have on your media share.

Using the metadata sorts is much more sophisticated and intelligent, but to be perfectly honest, I almost always use the file/folder heirarchy sort - always for video. Although I sometimes use metadata for music when I want to pick all from a particular genre and play them. So I'm sorry, you are wrong it isn't a feature that LMCE is missing, it has always been there, it just isn't working on your system for some reason.

Its been so long since I had the same problem (tv_shows_0 and 1) that I can't remember what causes it. Will leave that for someone else to suggest solutions as I have typed enough already! But rest assured a working system has this, and I imagine that most LMCE users use it all the time like I do.
Title: Re: New Media Navigation Facility
Post by: royw on April 27, 2008, 08:48:11 am
If you are pre-0710 beta 4, then upgrade to RC1.  Beta 4 introduced some NFS automount fixes.  BTW, I just did a new install of RC1, added my two file servers each with an NFS partition, rebooted, then ripped one movie to an NFS share, then watched it.  RC1 is so far working nicely.

As stated earlier in the thread, make sure PK_Users is "Pluto's structure".  If you change this, then reboot your core/hybrid.

Open up a terminal on your core and run:
Code: [Select]
find /home/public/data -name "NFS*" -print

You should see something like:
Code: [Select]
linuxmce@dcerouter:~$ find /home/public/data -name "NFS*" -print
/home/public/data/documents/NFS Share [39]
/home/public/data/documents/NFS Share [41]
/home/public/data/pictures/NFS Share [39]
/home/public/data/pictures/NFS Share [41]
/home/public/data/videos/NFS Share [39]
/home/public/data/videos/NFS Share [41]
/home/public/data/audio/NFS Share [39]
/home/public/data/audio/NFS Share [41]

In the above example, I have two NFS shares, devices 39 and 41, using Pluto's structure.  If your NFS share's PK_Users is set to "Public", then you would see the share under /home/public/data/other/NFS Share [XX].

The "NFS Share [XX]" directories are just links to the automount device as you can see by using ls -l
Code: [Select]
linuxmce@dcerouter:~$ ls -l "/home/public/data/documents/NFS Share [39]"
lrwxrwxrwx 1 root public 36 2008-04-26 18:05 /home/public/data/documents/NFS Share [39] -> /mnt/device/39/public/data/documents

If you look in the /mnt/device directory, the device will only show up if they are currently mounted:

Code: [Select]
linuxmce@dcerouter:~$ ls /mnt/device/
30  41

Notice the device 39 is not currently mounted in the above example.  This is cool.  To mount it just access it.

Code: [Select]
linuxmce@dcerouter:~$ ls -l /mnt/device/39
total 28
drwx------ 2 root root   16384 2008-02-18 01:23 lost+found
drwxrwsr-x 3 root public  4096 2008-02-02 23:22 public
drwxrwsr-x 3 root public  4096 2008-02-02 23:22 user_1
drwxrwsr-x 3 root public  4096 2008-02-02 23:22 user_2

OK, that pretty much shows how an NFS share should look from the core/hybrid.  The ownership and perms are critical.

On the NFS server, the exported directory needs root:public (actually 0:1002) ownership and owner+group rwx permission.  Here's my exported directory /var/media:
Code: [Select]
royw-gentoo var # ls -l /var | grep media
drwxrwsr-x  6 root public 4096 2008-04-27 01:26 media

I did edit the http://wiki.linuxmce.org/index.php/Howto:_Using_Network_Shares page the other day to include how to change the group id on the server.

Please let me know what you discover.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 27, 2008, 10:09:04 am
I running RC1. I used http://wiki.linuxmce.org/index.php/Howto:_Using_Network_Shares to create my NFS shares:

linuxmce@dcerouter:~$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                65386088   6207180  55857472  11% /
udev                   1618568        60   1618508   1% /dev
/dev/disk/by-uuid/05062c83-35b1-4ccc-b69d-0dcc8acbf99c
                      65386088   6207180  55857472  11% /
/dev/disk/by-uuid/05062c83-35b1-4ccc-b69d-0dcc8acbf99c
                      65386088   6207180  55857472  11% /dev/.static/dev
tmpfs                  1618568       272   1618296   1% /var/run
tmpfs                  1618568         0   1618568   0% /var/lock
tmpfs                  1618568     37924   1580644   3% /lib/modules/2.6.22-14-generic/volatile
tmpfs                  1618568         0   1618568   0% /dev/shm
tmpfs                  1618568       272   1618296   1% /var/run
tmpfs                  1618568         0   1618568   0% /var/lock
/dev/sda6              9608664   4674224   4446340  52% /mnt/recovery
192.168.1.3:/usr/fatboy/CleanTorrents
                     1153741944 964749496 130385744  89% /mnt/device/40
linuxmce@dcerouter:~$ ls -l /home/public/data/other/
total 0
lrwxrwxrwx 1 root public 15 2008-04-27 00:41 CleanTorrents [40] -> /mnt/device/40/

On the server, CleanTorrents looks like this:
drwxrwxrwx 27 erasmo erasmo 4096 2008-04-27 00:51 CleanTorrents/

My /etc/exports file:
/usr/fatboy                    *(rw,insecure)
/usr/fatboy/CleanTorrents                    *(rw,insecure)
/                               *(rw)



Here are screenshots of my device and my share definitions:
http://download.erasmix.net/linuxmce/Screenshot-1.png (http://download.erasmix.net/linuxmce/Screenshot-1.png)
http://download.erasmix.net/linuxmce/Screenshot-2.png (http://download.erasmix.net/linuxmce/Screenshot-2.png)


I have reinstalled RC1 at least 4 times since thursday. I've been forced to delete a share and re add it (all manually) after a simple reboot. I'm sure this is not normal. I'm at a complete loss. I appreciate any ideas. I still cannot see my videos when I sort on Filename.

One more thing. The server that has the NFS share, is also the gateway (192.168.1.3) of my network. My LinuxMCE box has 2 cards. One gets an IP Address using DHCP. The other is set by the installer to 192.168.80.1 and it runs DHCP on that interface: http://download.erasmix.net/linuxmce/Screenshot-3.png (http://download.erasmix.net/linuxmce/Screenshot-3.png) Could this be the root cause of the problem I'm experiencing?
Title: Re: New Media Navigation Facility
Post by: royw on April 27, 2008, 11:55:00 am
One more thing. The server that has the NFS share, is also the gateway (192.168.1.3) of my network. My LinuxMCE box has 2 cards. One gets an IP Address using DHCP. The other is set by the installer to 192.168.80.1 and it runs DHCP on that interface: http://download.erasmix.net/linuxmce/Screenshot-3.png (http://download.erasmix.net/linuxmce/Screenshot-3.png) Could this be the root cause of the problem I'm experiencing?

Most probably the problem.  I found this on the wiki:

http://wiki.linuxmce.org/index.php/I_can%27t_access_my_media_files_over_the_network (http://wiki.linuxmce.org/index.php/I_can%27t_access_my_media_files_over_the_network)

As a quick test, you might try turning off the core's firewall and see if that works (should be ok in your setup if you are not using the core as your home firewall).

Here's some info on nfs ports:

http://gentoo-wiki.com/index.php?title=HOWTO_Share_Directories_via_NFS#Setting_Up_Firewall_.28Server_Side.29 (http://gentoo-wiki.com/index.php?title=HOWTO_Share_Directories_via_NFS#Setting_Up_Firewall_.28Server_Side.29)

In a nutshell, it looks like you will need to limit NFS ports on the server to a known set, then enable pass through of this set of ports on the core. 

You can search the forum for more explicit instructions on configuring the firewall.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 27, 2008, 07:27:37 pm
Roy, I already disable my firewall on the dcerouter, because I use the web admin from a different box. The gateway server (192.168.1.3) also has two network cards and the firewall filters the external network only, so I have no need to firewall the nfs ports.

I'm also aware that, because the file server is on a different network than the dcerouter, it wont be discovered until I added manually thru the web admin interface.

Do I still need to limit the NFS ports on the server and enable pass thru if nothing in the internal network is being filtered by a firewall?

I know this is not an easy challenge, so I sincerely appreciate you guys helping. I'm willing to try everything.
Title: Re: New Media Navigation Facility
Post by: royw on April 27, 2008, 08:09:36 pm
You should be fine with the dcerouter firewall disabled, no need to mess with individual ports.

On looking over an earlier post of yours, I see:
My /etc/exports file:
/usr/fatboy                    *(rw,insecure)
/usr/fatboy/CleanTorrents                    *(rw,insecure)
/                               *(rw)


Try changing your exports to:
Code: [Select]
/usr/fatboy/CleanTorrents 192.168.0.0/16(async,no_subtree_check,rw,no_root_squash)

First, exporting root on an nfs is pretty dangerous, so please eliminate it unless you really need it.  shudder

Second, let's only allow internal IP's to connect with 192.168.0.0/16.  That should allow the external interface on the dcerouter and any future MDs on the internal interface to connect.

Third, the permissions really need to be what I've specified (they took a couple of days to figure out).  You probably could substitute sync for async as it is just a performance option.  The no_root_squash is critical.  Basically the dcerouter needs any share to look exactly like it is an internal disk with regards to ownership/permissions.  Further the dcerouter requires the ownership be root:public as viewed from the dcerouter.  What this means is the ownership on the files in the share must numerically match the dcerouter's ownership.  Currently root:public is 0:1002 on the dcerouter.  root squashing changes root (0) to nobody (65534).  I really hate this as it is a major security hole, but this is the only way I could figure to get it to work.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 27, 2008, 09:09:50 pm
Dont worry about the / share. Its just me on my network at home. I'll remove since I no longer need it.

I saw the (async,no_subtree_check,rw,no_root_squash) in the article I used to setup my network share, but I thought that (rw,insecure) would be a superset. Anyways I just changed to /usr/fatboy/CleanTorrents 192.168.0.0/16(async,no_subtree_check,rw,no_root_squash), I dropped the share, recreated it, and quick reloaded the router.

Sorry, it made no difference. Sorting by Filename only shows tv_shows_1

would it be possible to setup a reproduction? Is there a way for me to collect diagnose information?
Title: Re: New Media Navigation Facility
Post by: royw on April 28, 2008, 06:20:26 am
insecure just means nfs can respond to originating requests from ports > 1024.

You can view all the options with "man exports".

Let's recap what you have, please run the following on the dcerouter and post the responses, substitute your NFS Share device number for XX:
Code: [Select]
find /home/public/data -name "NFS*" -print
ls -n /mnt/device/XX
What the above will tell me is whether or not automount is working.

Now we are going to peak into the database, again substitute your NFS Share device number for XX:
Code: [Select]
mysql -u root -D pluto_main -E -e "select * from Device where PK_Device='XX'"
mysql -u root -D pluto_main -E -e "select * from Device where IPaddress='192.168.1.3'"
The first command dumps the device attributes for the NFS Share, the second for your file server.  This will let me compare to mine.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 28, 2008, 07:31:23 am
Roy,

I noticed that, some times, after rebooting, my NFS share just doesn't show any files at all. It takes about 2-3 OS reboots for this to happens. I also noticed that, when this happens, I can't see any files, but when I sort on Filename, I see the name of the share below tv_shows_1, but when I select it, its empty. At this point I'm forced to delete the share and re add it again. It just happened again when I turned on th machine to collect the info you asked for. Therefore I decided to collect two sets of data, one with the shared gone bad, and with the new share showing files in it.

This is when the share went bad:

linuxmce@dcerouter:~$ find /home/public/data -name "NFS*" -print
linuxmce@dcerouter:~$ find /home/public/data -name "nfs*" -print
/home/public/data/videos/tv_shows_1/nfslockfile.lock
linuxmce@dcerouter:~$ ls -n /mnt/device/
total 0
linuxmce@dcerouter:~$
linuxmce@dcerouter:~$ mysql -u root -D pluto_main -E -e "select * from Device where PK_Device='41'"
*************************** 1. row ***************************
              PK_Device: 41
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: CleanTorrents
      FK_DeviceTemplate: 1769
FK_Device_ControlledVia: 37
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 0
                  State:
                 Status: **RUN_CONFIG**
               PingTest: 0
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-27 11:47:07
           psc_restrict: NULL
linuxmce@dcerouter:~$ mysql -u root -D pluto_main -E -e "select * from Device where IPaddress='192.168.1.3'"
*************************** 1. row ***************************
              PK_Device: 37
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: neo
      FK_DeviceTemplate: 1837
FK_Device_ControlledVia: 1
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 0
                  State:
                 Status:
               PingTest: NULL
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-26 23:48:09
           psc_restrict: NULL
*************************** 2. row ***************************
              PK_Device: 41
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: CleanTorrents
      FK_DeviceTemplate: 1769
FK_Device_ControlledVia: 37
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 0
                  State:
                 Status: **RUN_CONFIG**
               PingTest: 0
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-27 11:47:07
           psc_restrict: NULL


==========================================================================================================================


This is after dropping and re-adding the share:

linuxmce@dcerouter:~$ find /home/public/data -name "NFS*" -print
linuxmce@dcerouter:~$ find /home/public/data -name "nfs*" -print
/home/public/data/videos/tv_shows_1/nfslockfile.lock
linuxmce@dcerouter:~$ ls -n /mnt/device/42   # I clipped the output for obvious reasons, but it contains all my files :)
total 112
...
drwxr-xr-x  3 65534 65534  4096 2008-04-26 10:58 public
...
drwxr-xr-x  3 65534 65534  4096 2008-04-26 10:58 user_1
...
linuxmce@dcerouter:~$ mysql -u root -D pluto_main -E -e "select * from Device where PK_Device='42'"
*************************** 1. row ***************************
              PK_Device: 42
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: CleanTorrents
      FK_DeviceTemplate: 1769
FK_Device_ControlledVia: 37
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 1
                  State:
                 Status: **RUN_CONFIG**
               PingTest: 0
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-27 22:16:07
           psc_restrict: NULL
linuxmce@dcerouter:~$ mysql -u root -D pluto_main -E -e "select * from Device where IPaddress='192.168.1.3'"
*************************** 1. row ***************************
              PK_Device: 37
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: neo
      FK_DeviceTemplate: 1837
FK_Device_ControlledVia: 1
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 0
                  State:
                 Status:
               PingTest: NULL
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-26 23:48:09
           psc_restrict: NULL
*************************** 2. row ***************************
              PK_Device: 42
               Disabled: 0
                FK_Room: NULL
        FK_Installation: 72635
           FK_DesignObj: NULL
            Description: CleanTorrents
      FK_DeviceTemplate: 1769
FK_Device_ControlledVia: 37
              IPaddress: 192.168.1.3
             MACaddress: 00:30:48:80:2E:7E
            IgnoreOnOff: 0
      FK_Device_RouteTo: NULL
          NeedConfigure: 1
                  State:
                 Status: **RUN_CONFIG**
               PingTest: 0
      EK_LicensedDevice: NULL
            IsNewDevice: 1
    ManuallyConfigureEA: 0
             Registered: 0
                 psc_id: NULL
              psc_batch: NULL
               psc_user: NULL
             psc_frozen: 0
                psc_mod: 2008-04-27 22:16:07
           psc_restrict: NULL
Title: Re: New Media Navigation Facility
Post by: royw on April 28, 2008, 10:02:57 am
It looks like your file server device config is good, but just to be sure, when you create the file server device you:

Device Info #YY section:
* Device Template: select File Server #1837
* Description: "neo"
* Room: select the room your core/hybrid is in
* IP Address: "192.168.1.3"
* leave the rest of this section with the defaults
press Save

Leave the rest of the page with the defaults.

That's it on the file server device.

I'm going to ask you to delete then recreate the NFS share, but this time just set the absolute minimum config options.

After selecting the NFS Share device template #1769:

Device Info #XX section:
* Description:  leave as "NFS Share"
* Room: select the room your core/hybrid is in
* leave the rest of this section with the defaults
press Save

Device Pipes Used section:
* don't touch

Member Of Group section:
* don't touch

Related Devices section:
* don't touch

Device Data section:
* PK_Users: select "Pluto's structure"
* Share Name: "/usr/fatboy/CleanTorrents"
* Use Automatically: selected
press Save

Now on to the flakiness.  I had a similar problem when I was first getting NFS going.  It turned out to be a bad ethernet switch between my NFS server and my dcerouter.  If you can, try eliminating any hardware between neo and the dcerouter.
FYI, I'm using D-Link's DGS-2208 gigabit switches now a days.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on April 30, 2008, 07:06:03 am
Roy,

I did yet another fresh install (no keeping settings at all). I added the file server as you said below, but that was the original way I set it up. On the share, I notice you didn't ask me to specify NFS on "Filesystem", so I didn't. I got exactly the same result. On the NFS "flakiness", after I rebooted my system (with the remote, thru the advanced menu), it happened again! I be willing to connect the server to the dcerouter directly if I wasn't sure I don't have a network router problem. By the way, I can see the device mounted:

192.168.1.3:/usr/fatboy/CleanTorrents
                     1153741944 964777432 130357808  89% /mnt/device/38

Besides this didn't happened with Beta4. I'm actually tempted to go back to B4 and give it a try.

Finally I wanted to ask you. On a previous post you asked me to find /home/public/data -name "NFS*" -print. Well that didn't return anything (see my previous posting). I just ls -l /home/public/data and this is what I got:

linuxmce@dcerouter:~$ ls -l /home/public/data
total 32
drwxrwsr-x 2 root public 4096 2008-04-29 21:53 audio
drwxrwsr-x 2 root public 4096 2008-04-21 13:00 avwizard-sounds
drwxrwsr-x 2 root public 4096 2008-04-29 21:53 documents
drwxrwsr-x 2 root public 4096 2008-04-29 21:45 other
drwxrwsr-x 2 root public 4096 2008-04-29 21:53 pictures
drwxrwsr-x 4 root public 4096 2008-04-21 12:47 samples
drwxr-sr-x 2 root public 4096 2008-04-29 21:53 tts
drwxrwsr-x 3 root public 4096 2008-04-29 21:53 videos

Is that what you expected?

Thanks,

Erasmo.
Title: Re: New Media Navigation Facility
Post by: royw on April 30, 2008, 08:43:53 am
I was wanting to see the links to the share under /home/public/data.  I didn't realize that you had changed the description from the default of "NFS Share" to "CleanTorrents".

For example, if the NFS share's description is the default of "NFS Share", then the links would be "NFS Share [XX]" so I just picked a wild card match of "NFS*".  In your case, if your NFS share's description is "CleanTorrents", you could specify the name match pattern of "CleanTorrents*".

Basically what I was wanting to confirm was the mapping of the links using pluto's structure.  Here's what I was hoping to see something like:
Code: [Select]
linuxmce@dcerouter:~$ find /home/public/data -name "NFS Share \[39\]" -print
/home/public/data/documents/NFS Share [39]
/home/public/data/pictures/NFS Share [39]
/home/public/data/videos/NFS Share [39]
/home/public/data/audio/NFS Share [39]

If there are no found links, then I'd recommend rebooting the dcerouter and trying again.  If still none, then we are seeing a problem.

If the link found was something like /home/public/data/other/NFS Share [39], then we would have a config problem as it is using "Public" instead of "Pluto's structure" for the nfs share's PK_Users field.

If the links are as expected, then we first confirm that we can see the nfs share via the link using something like:
Code: [Select]
ls -n /home/public/data/videos/NFS Share [39]
at this point we can be pretty certain that the nfs share is correct and it's time to revisit the orbiter side of the issue.

On the network flakiness, please read my reply here: http://forum.linuxmce.org/index.php?topic=4260.msg24549#msg24549
I was lucky in that I had a third box so I could verify the NFS server was working separately from LMCE, which led to the isolation of the faulty switch.

HTH,
Roy
Title: Re: New Media Navigation Facility
Post by: erasmix on May 24, 2008, 06:18:50 pm
I just installed 0710B4 and I see the same behavior as RC1. I tried both NFS shares and Windows, and I see the same behavior when I add a share. I see only tv_shows_0 & _1 when I try to sort by Filename. Also, the share is gone after I reboot linuxMCE (I do it nicely by using the remote control). 

However, if I add the share, Windows, or NFS, and mark it Readonly, all the "flakiness" goes away. I reboot any number of times and the share always reappears. I haven't tested back on RC1, since it requires wiping everything off, but I'm willing to bet the behavior will be the same. And this doesn't allow me to sort on Filename either.

At this point, after all the testing we've done, I dare say this is a bug. I suspect that anybody accessing a Network Share outside the LMCE network would experience the same problem. Would it be possible for somebody to test this? On my setup, the file server where the share is, is also the network default gateway from which linuxMCE gets its external IP (192.168.1.2). My linuxMCE box has 2 cards. The internal one  is192.168.80.2 as expected and I haven't tampered with the DHCP settings on the router.

One last thing. Its not a problem, but its a bit obnoxious. I see all these never stopping messages:

May 24 08:53:39 neo mountd[5619]: authenticated mount request from 192.168.1.2:892 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
May 24 08:53:44 neo mountd[5619]: authenticated mount request from 192.168.1.2:990 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
May 24 08:53:50 neo mountd[5619]: authenticated mount request from 192.168.1.2:663 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)
May 24 08:53:55 neo mountd[5619]: authenticated mount request from 192.168.1.2:761 for /usr/fatboy/CleanTorrents (/usr/fatboy/CleanTorrents)

if the share is NFS, or:

[2008/05/24 08:41:03, 1] smbd/service.c:make_connection_snum(1033)
  192.168.1.2 (192.168.1.2) connect to service CleanTorrents initially as user nobody (uid=99, gid=99) (pid 27885)
[2008/05/24 08:41:03, 1] smbd/service.c:close_cnum(1230)
  192.168.1.2 (192.168.1.2) closed connection to service CleanTorrents
[2008/05/24 08:41:09, 1] smbd/service.c:make_connection_snum(1033)
  192.168.1.2 (192.168.1.2) connect to service CleanTorrents initially as user nobody (uid=99, gid=99) (pid 27889)

if it is a Windows (using samba) share. It would be nice to stop them, but I can live with them.
Title: Re: New Media Navigation Facility
Post by: caiman on May 24, 2008, 09:24:58 pm
For what it's worth, I had a similar problem: I could access /mnt/device/40/public but the media was not seen in the orbiter.
I found out that LMCE was setting the flag "Online" to off when it couldn't reach the server, and never switched it back on.
So going to the web admin, to the NFS share and enabling "online" again fixed it for me (after doing all the permission verifications explained earlier in this thread). Not sure if you have the same problem, but it's worth checking.
Title: Re: New Media Navigation Facility
Post by: erasmix on May 24, 2008, 09:59:27 pm
I have checked. The device is never set offline. However it happened once, but it stopped happening when I made the top directory fully writable so LMCE could create files and directories under the share. But that was months ago. I strongly suspect this happens if you mount a network share from outside the LMCE subnet.
Title: Re: New Media Navigation Facility
Post by: totallymaxed on May 25, 2008, 12:16:53 pm
Each video will get an extra MovieName.id3 file for the meta-data.

Each picture file (at least *.jpg) may get an .id3 and a .tnj (thumbnail)

I'll let someone else answer on audio files as I don't really use LMCE for audio files (I store the files on a NFS share under public/data/audio but then just download to ipod for listening).

Have fun,
Roy

For audio files the id3 data is written back to the files themselves so that other applications eg iTunes or Amarok etc can take advantage of it.

Andrew