Help with Orbiter Crashing

[dothedog]

All, I need some help troubleshooting this issue. I have a core/hybrid with 6 mds (all intel mini-itx boards). The core has a 3TB raid5 array that contains all of my media. The problem I have been having is that anytime I start an orbiter on a separate machine, first it always wants to regenerate the orbiters, even though I haven't made any changes. Then once it has regenerated, I can select some music to play and it will start playing. However, after 1-3 songs, the orbiter will crash, sometimes it will regenerate, sometimes not, in the end the music stops playing as well and I usually have a hung orbiter generation screen. This happens on linux boxes running the orbiter in wine as well as on windows xp boxes. This also happens no matter what MD I am hitting. 

Last weekend, I  tried doing an apt-get update upgrade, on the core and all of the MDs and that didn't seem to help either.  I have been looking in the logs in /var/log/pluto but nothing is jumping out at me. Any help on troubleshooting this? What am I missing?

Dothedog

[golgoj4]

This is anecdotal, but make sure your core is healthy. I.E. no extremly high i/o wait (%wa in top) and the use iostat (part of the sysstat package) to check hard drives. I had a similar problem and it seemed it was my main linuxmce os drive was dying.

-hth
golgoj4

[dothedog]

golgoj4,
Thanks for the response. I was wondering about something similar. The %wa in top goes between about 20%-30% most of the time. That does seem high. I have also in the past had problems with apt-cacher-ng and Start_NewMD_interactor.sh taking huge amounts of proc. Although that seems to have been fixed when I did the last apt-get upgrade. It seems like it loses connection sometimes.

The other thing I noticed that may or may not be relevant is in pluto.log, "MythBackend_Restart (server) MythBackend not found running; restarting it" is running every minute. I am not using MythTV in linuxMCE. Is there a way to stop it from trying to restart all the time?

DoTheDog

[dothedog]

Ok, I have been tracking the logs in /var/log/pluto when I play something. I have noticed a couple of things:

died__DCERouter shows DCERouter is dying about every 15 minutes..

Code: [Select]

Sat Apr 2 04:07:59 MDT 2011 died
Sat Apr 2 04:22:21 MDT 2011 died
Sat Apr 2 04:36:44 MDT 2011 died
Sat Apr 2 04:51:05 MDT 2011 died
Sat Apr 2 05:05:25 MDT 2011 died
Sat Apr 2 05:19:18 MDT 2011 died
Sat Apr 2 05:33:13 MDT 2011 died
Sat Apr 2 05:47:34 MDT 2011 died
Sat Apr 2 06:01:59 MDT 2011 died
Sat Apr 2 06:15:55 MDT 2011 died
Sat Apr 2 06:30:19 MDT 2011 died
Sat Apr 2 06:44:12 MDT 2011 died
Sat Apr 2 06:58:08 MDT 2011 died
Sat Apr 2 07:12:30 MDT 2011 died
Sat Apr 2 07:26:50 MDT 2011 died
Sat Apr 2 07:41:17 MDT 2011 died
Sat Apr 2 07:55:42 MDT 2011 died

Also 26_Mplayer_Player.log shows this:

Code: [Select]

Sat Apr  2 07:55:19 MDT 2011 Restart
========== NEW LOG SECTION ==========
1 04/02/11 07:55:27 26 (spawning-device) Starting... 1
1 04/02/11 07:55:27 26 (spawning-device) Found ./MPlayer_Player
05 04/02/11 7:55:28.678 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:29.678 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:30.678 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:31.678 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
Snip...
05 04/02/11 7:55:45.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:46.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:47.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:48.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:49.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>
05 04/02/11 7:55:50.694 Connect() failed, Error Code 111 (Connection refused)) <0xb72366c0>

Also it looks like this is why I need to regenerate Orbiters every time I start them:

Code: [Select]

-r--------  1 root     root            0 2011-04-02 07:56 Spawn_storagedevices_2196.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_25763.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_26318.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_29453.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_32337.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_32556.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_3523.log
-r--------  1 root     root          199 2011-04-02 07:56 Spawn_storagedevices_3612.log

they all say:

Code: [Select]

sudo cat Spawn_storagedevices_32556.log
error: libhal_device_get_property_type: org.freedesktop.Hal.NoSuchProperty: No property storage.serial on device with id /org/freedesktop/Hal/devices/volume_uuid_54823dd8_bc56_489e_9e07_8db56614f2ef

Finally in UpdateEntArea.log I get this:

Code: [Select]

05 04/02/11 7:55:49.324 Resetting db_wrapper connection <0xb664f6c0>
01 04/02/11 7:55:49.324 Table_CommandGroup::Commit Cannot perform update query [update CommandGroup set `PK_CommandGroup`=2, `FK_Array`=26, `FK_Installation`=1001870, `Description`="Sunset", `Hint`="", `CanTurnOff`=0, `AlwaysShow`=0, `CanBeHidden`=0, `FK_Criteria_Orbiter`=NULL, `FK_DesignObj`=NULL, `FK_Template`=31, `AltID`=NULL, `FK_Icon`=NULL, `AutoGeneratedDate`="2009-09-19 22:53:19", `Disabled`=0, `TemplateParm1`=0, `TemplateParm2`=0, `FK_Text`=NULL, `psc_id`=NULL, `psc_batch`=NULL, `psc_user`=0, `psc_frozen`=0, `psc_restrict`=0 where `PK_CommandGroup`=2] Unknown column 'FK_Text' in 'field list' reconnect: 1 result2: 1 <0xb664f6c0>
05 04/02/11 7:55:49.324 Resetting db_wrapper connection <0xb664f6c0>
01 04/02/11 7:55:49.325 Table_CommandGroup_EntertainArea::Commit Cannot perform query [insert into CommandGroup_EntertainArea (`FK_CommandGroup`, `FK_EntertainArea`, `Sort`, `psc_id`, `psc_batch`, `psc_user`, `psc_frozen`, `psc_restrict`) values (0, 2, 1, NULL, NULL, NULL, 0, NULL)] Duplicate entry '0-2' for key 1 reconnect: 1 result2: 1 <0xb664f6c0>
05 04/02/11 7:55:49.344 Resetting db_wrapper connection <0xb664f6c0>
01 04/02/11 7:55:49.344 Table_CommandGroup::Commit Cannot perform query [insert into CommandGroup (`PK_CommandGroup`, `FK_Array`, `FK_Installation`, `Description`, `Hint`, `CanTurnOff`, `AlwaysShow`, `CanBeHidden`, `FK_Criteria_Orbiter`, `FK_DesignObj`, `FK_Template`, `AltID`, `FK_Icon`, `AutoGeneratedDate`, `Disabled`, `TemplateParm1`, `TemplateParm2`, `FK_Text`, `psc_id`, `psc_batch`, `psc_user`, `psc_frozen`, `psc_restrict`) values (0, 5, 1001870, "TV", "JinShinOffice", 0, 0, 0, NULL, NULL, 33, NULL, 11, "2011-04-02 07:55:48", 0, 161, 1, NULL, NULL, NULL, NULL, 0, NULL)] Unknown column 'FK_Text' in 'field list' reconnect: 1 result2: 1 <0xb664f6c0>
05 04/02/11 7:55:49.345 Resetting db_wrapper connection <0xb664f6c0>
01 04/02/11 7:55:49.345 Table_CommandGroup::Commit Cannot perform update query [update CommandGroup set `PK_CommandGroup`=2, `FK_Array`=26, `FK_Installation`=1001870, `Description`="Sunset", `Hint`="", `CanTurnOff`=0, `AlwaysShow`=0, `CanBeHidden`=0, `FK_Criteria_Orbiter`=NULL, `FK_DesignObj`=NULL, `FK_Template`=31, `AltID`=NULL, `FK_Icon`=NULL, `AutoGeneratedDate`="2009-09-19 22:53:19", `Disabled`=0, `TemplateParm1`=0, `TemplateParm2`=0, `FK_Text`=NULL, `psc_id`=NULL, `psc_batch`=NULL, `psc_user`=0, `psc_frozen`=0, `psc_restrict`=0 where `PK_CommandGroup`=2] Unknown column 'FK_Text' in 'field list' reconnect: 1 result2: 1 <0xb664f6c0>
05 04/02/11 7:55:49.352 Resetting db_wrapper connection <0xb664f6c0>
01 04/02/11 7:55:49.352 Table_CommandGroup_EntertainArea::Commit Cannot perform query [insert into CommandGroup_EntertainArea (`FK_CommandGroup`, `FK_EntertainArea`, `Sort`, `psc_id`, `psc_batch`, `psc_user`, `psc_frozen`, `psc_restrict`) values (0, 2, 1, NULL, NULL, NULL, 0, NULL)] Duplicate entry '0-2' for key 1 reconnect: 1 result2: 1 <0xb664f6c0>

Any Ideas on how I can knock some of these things down?

DotheDog

[dothedog]

Not sure if this ranks as a "Solution", but I fixed it by doing a complete reinstall. A bit of a pain but I have most of my configurations for Squid, Dansguardian, etc. saved out, so its only a few hours to rebuild. But the orbiters are no longer crashing.

Dothedog