Author Topic: CIFS errors causing ugly Lag and eventually crashing!!  (Read 7038 times)

seth

  • Guru
  • ****
  • Posts: 485
  • A day w/o LinuxMCE is like a day w/o sunshine!
    • View Profile
CIFS errors causing ugly Lag and eventually crashing!!
« on: December 19, 2008, 01:35:37 pm »
Ok,

So my new setup has been running for almost 3 months now, and in that time I only had to reboot my core for hardware changes. However just the other day, we had a power outage, and the system was in full swing, watching TV in the living room, girls watching videos in the family room, me playing audio in the basement. After the power returned, and running the mysql repair mythconverg for the crashed db, it seemed to work fine.

Now as of yesterday, the system is locking up. However it is only doing so when I boot my Living room MD.
The family room MD runs great, no errors are reported.
The living room gets about all the way booted up, and then I am seeing this on the core syslog:

Code: [Select]
Dec 18 20:46:01 dcerouter kernel: [ 8704.519323]  CIFS VFS: cifs_mount
failed w/return code = -11
Dec 18 20:46:01 dcerouter automount[32082]: >> mount error 11 = Resource
temporarily unavailable
Dec 18 20:46:01 dcerouter automount[32082]: >> Refer to the mount.cifs(8)
manual page (e.g.man mount.cifs)
Dec 18 21:20:01 dcerouter kernel: [  445.598719]  CIFS VFS: server not
responding
Dec 18 21:20:01 dcerouter kernel: [  445.598726]  CIFS VFS: server not
responding
Dec 18 21:20:01 dcerouter kernel: [  445.598733]  CIFS VFS: No response
for cmd 50 mid 870
Dec 18 21:20:01 dcerouter kernel: [  445.598742]  CIFS VFS: No response
for cmd 50 mid 871
Dec 18 21:20:01 dcerouter kernel: [  445.598754]  CIFS VFS: No response
for cmd 50 mid 872
Dec 18 21:20:16 dcerouter kernel: [  460.564915]  CIFS VFS: No response
for cmd 50 mid 873
Dec 18 21:22:05 dcerouter kernel: [  569.376720]  CIFS VFS: Error
connecting to IPv4 socket. Aborting operation
Dec 18 21:22:05 dcerouter kernel: [  569.376735]  CIFS VFS: cifs_mount
failed w/return code = -111
Dec 18 21:22:05 dcerouter automount[20499]: >> mount error 111 =
Connection refused
Dec 18 21:22:05 dcerouter automount[20499]: >> Refer to the mount.cifs(8)
manual page (e.g.man mount.cifs)
Dec 18 21:29:16 dcerouter kernel: [  999.343290]  CIFS VFS: server not
responding
Dec 18 21:29:16 dcerouter kernel: [  999.343304]  CIFS VFS: No response to
cmd 115 mid 875
Dec 18 21:29:16 dcerouter kernel: [  999.343312]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.343512]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.343655]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.343793]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.343930]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.344087]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 21:29:16 dcerouter kernel: [  999.344245]  CIFS VFS: Send error in
SessSetup = -11
Dec 18 22:23:04 dcerouter kernel: [  582.019663]  CIFS VFS: server not
responding
Dec 18 22:23:04 dcerouter kernel: [  582.019670]  CIFS VFS: server not
responding
Dec 18 22:23:04 dcerouter kernel: [  582.019677]  CIFS VFS: No response
for cmd 50 mid 595
Dec 18 22:23:04 dcerouter kernel: [  582.019686]  CIFS VFS: No response
for cmd 50 mid 596
Dec 18 22:23:04 dcerouter kernel: [  582.019704]  CIFS VFS: No response
for cmd 50 mid 598
Dec 18 22:23:04 dcerouter kernel: [  582.019851]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /user_4
Dec 18 22:23:05 dcerouter kernel: [  582.793912]  CIFS VFS: No response
for cmd 50 mid 597
Dec 18 22:23:34 dcerouter kernel: [  611.952082]  CIFS VFS: No response
for cmd 117 mid 603
Dec 18 22:23:34 dcerouter kernel: [  611.952093]  CIFS VFS: No response
for cmd 50 mid 602
Dec 18 22:23:34 dcerouter kernel: [  611.952099]  CIFS VFS: Send error in
QFSUnixInfo = -11
Dec 18 22:23:44 dcerouter kernel: [  621.933493]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /public/data
Dec 18 22:23:54 dcerouter kernel: [  631.912364]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /public/data
Dec 18 22:23:54 dcerouter kernel: [  631.929642]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /public
Dec 18 22:23:57 dcerouter automount[3115]: mount(generic): warning:
/mnt/device/341 is already mounted
Dec 18 22:23:57 dcerouter automount[3198]: lookup(program): lookup for
341/public failed
Dec 18 22:23:57 dcerouter automount[3198]: failed to mount
/mnt/device/341/public
Dec 18 22:24:01 dcerouter /USR/SBIN/CRON[3532]: (root) CMD (bash -c
'/usr/pluto/bin/DiskSpaceMonitor.sh &>/dev/null')
Dec 18 22:24:04 dcerouter kernel: [  642.013539] BUG: unable to handle
kernel NULL pointer dereference at virtual address 00000046
Dec 18 22:24:04 dcerouter kernel: [  642.013547]  printing eip:
Dec 18 22:24:04 dcerouter kernel: [  642.013549] f8d5263d
Dec 18 22:24:04 dcerouter kernel: [  642.013551] *pde = 00000000
Dec 18 22:24:04 dcerouter kernel: [  642.013555] Oops: 0000 [#1]
Dec 18 22:24:04 dcerouter kernel: [  642.013557] SMP
Dec 18 22:24:04 dcerouter kernel: [  642.013561] Modules linked in:
nls_cp437 cifs ztdummy zaptel crc_ccitt rfcomm l2cap bluetooth xt_TCPMSS
ipt_MASQUERADE ipt_TTL iptable_mangle xt_tcpudp xt_mark nfsd exportfs
lockd xt_state sunrpc iptable_nat iptable_filter ip_tables x_tables ppdev
autofs4 speedstep_lib cpufreq_conservative cpufreq_ondemand
cpufreq_userspace cpufreq_stats freq_table cpufreq_powersave container
video button sbs ac dock battery nf_nat_irc nf_nat_ftp nf_nat
nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nfnetlink
lp serio_raw parport_pc parport pcspkr psmouse af_packet i2c_piix4
i2c_core sworks_agp agpgart ipv6 joydev evdev ext3 jbd mbcache sg sd_mod
usbhid hid ata_generic ohci_hcd usbcore floppy pata_serverworks libata
scsi_mod tg3 raid10 raid456 xor raid1 raid0 multipath linear md_mod
thermal processor fan fuse apparmor commoncap r8168
Dec 18 22:24:04 dcerouter kernel: [  642.013662] CPU:    1
Dec 18 22:24:04 dcerouter kernel: [  642.013662] EIP:    0060:[<f8d5263d>]
   Not tainted VLI
Dec 18 22:24:04 dcerouter kernel: [  642.013664] EFLAGS: 00010246 
(2.6.22-14-generic #1)
Dec 18 22:24:04 dcerouter kernel: [  642.013678] EIP is at
cifs_show_options+0x8d/0x140 [cifs]
Dec 18 22:24:04 dcerouter kernel: [  642.013681] eax: 00000000   ebx:
f4e7fcc0   ecx: f6102f28   edx: 00000005
Dec 18 22:24:04 dcerouter kernel: [  642.013684] esi: f3e27900   edi:
f3e27900   ebp: 00000093   esp: f3d57f14
Dec 18 22:24:04 dcerouter kernel: [  642.013687] ds: 007b   es: 007b   fs:
00d8  gs: 0033  ss: 0068
Dec 18 22:24:04 dcerouter kernel: [  642.013690] Process stat (pid: 3762,
ti=f3d56000 task=f19ac530 task.ti=f3d56000)
Dec 18 22:24:04 dcerouter kernel: [  642.013692] Stack: f3e27900 f8d824a7
f6102f28 00000000 f3c02a00 c01984d5 c037aef4 c03b6bb4
Dec 18 22:24:04 dcerouter kernel: [  642.013702]        f3e27900 f3c02a00
c019b446 000003b0 b7fae050 c2715000 f3e27920 00000050
Dec 18 22:24:04 dcerouter kernel: [  642.013711]        00000000 00000014
00000000 00000013 00000000 c2715000 b7fae000 f3d57fa0
Dec 18 22:24:04 dcerouter kernel: [  642.013719] Call Trace:
Dec 18 22:24:04 dcerouter kernel: [  642.013733]  [show_vfsmnt+293/368]
show_vfsmnt+0x125/0x170
Dec 18 22:24:04 dcerouter kernel: [  642.013750]  [seq_read+470/672]
seq_read+0x1d6/0x2a0
Dec 18 22:24:04 dcerouter kernel: [  642.013785]  [vfs_read+188/352]
vfs_read+0xbc/0x160
Dec 18 22:24:04 dcerouter kernel: [  642.013795]  [seq_read+0/672]
seq_read+0x0/0x2a0
Dec 18 22:24:04 dcerouter kernel: [  642.013807]  [sys_read+65/112]
sys_read+0x41/0x70
Dec 18 22:24:04 dcerouter kernel: [  642.013824]
[sysenter_past_esp+107/169] sysenter_past_esp+0x6b/0xa9
Dec 18 22:24:04 dcerouter kernel: [  642.013866]  =======================
Dec 18 22:24:04 dcerouter kernel: [  642.013868] Code: 74 14 89 44 24 08
c7 44 24 04 bc 24 d8 f8 89 34 24 e8 48 87 44 c7 f6 43 24 40 0f 85 9e 00 00
00 f6 43 25 04 75 58 8b 03 8b 40 24 <f6> 40 46 80 74 4d f6 43 25 08 75 67
8b 03 8b 40 24 f6 40 46 80
Dec 18 22:24:04 dcerouter kernel: [  642.013920] EIP: [<f8d5263d>]
cifs_show_options+0x8d/0x140 [cifs] SS:ESP 0068:f3d57f14
Dec 18 22:24:04 dcerouter kernel: [  642.016996] BUG: unable to handle
kernel NULL pointer dereference at virtual address 00000046
Dec 18 22:24:04 dcerouter kernel: [  642.017007]  printing eip:
Dec 18 22:24:04 dcerouter kernel: [  642.017010] f8d5263d
Dec 18 22:24:04 dcerouter kernel: [  642.017013] *pde = 00000000
Dec 18 22:24:04 dcerouter kernel: [  642.017019] Oops: 0000 [#2]
Dec 18 22:24:04 dcerouter kernel: [  642.017021] SMP
Dec 18 22:24:04 dcerouter kernel: [  642.017027] Modules linked in:
nls_cp437 cifs ztdummy zaptel crc_ccitt rfcomm l2cap bluetooth xt_TCPMSS
ipt_MASQUERADE ipt_TTL iptable_mangle xt_tcpudp xt_mark nfsd exportfs
lockd xt_state sunrpc iptable_nat iptable_filter ip_tables x_tables ppdev
autofs4 speedstep_lib cpufreq_conservative cpufreq_ondemand
cpufreq_userspace cpufreq_stats freq_table cpufreq_powersave container
video button sbs ac dock battery nf_nat_irc nf_nat_ftp nf_nat
nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nfnetlink
lp serio_raw parport_pc parport pcspkr psmouse af_packet i2c_piix4
i2c_core sworks_agp agpgart ipv6 joydev evdev ext3 jbd mbcache sg sd_mod
usbhid hid ata_generic ohci_hcd usbcore floppy pata_serverworks libata
scsi_mod tg3 raid10 raid456 xor raid1 raid0 multipath linear md_mod
thermal processor fan fuse apparmor commoncap r8168
Dec 18 22:24:04 dcerouter kernel: [  642.017173] CPU:    1
Dec 18 22:24:04 dcerouter kernel: [  642.017175] EIP:    0060:[<f8d5263d>]
   Not tainted VLI
Dec 18 22:24:04 dcerouter kernel: [  642.017178] EFLAGS: 00010246 
(2.6.22-14-generic #1)
Dec 18 22:24:04 dcerouter kernel: [  642.017193] EIP is at
cifs_show_options+0x8d/0x140 [cifs]
Dec 18 22:24:04 dcerouter kernel: [  642.017197] eax: 00000000   ebx:
f4e7fcc0   ecx: f6102f28   edx: 00000005
Dec 18 22:24:04 dcerouter kernel: [  642.017201] esi: f3e27d00   edi:
f3e27d00   ebp: 00000093   esp: f7be5f14
Dec 18 22:24:04 dcerouter kernel: [  642.017206] ds: 007b   es: 007b   fs:
00d8  gs: 0033  ss: 0068
Dec 18 22:24:04 dcerouter kernel: [  642.017211] Process stat (pid: 3763,
ti=f7be4000 task=dfa32530 task.ti=f7be4000)
Dec 18 22:24:04 dcerouter kernel: [  642.017214] Stack: f3e27d00 f8d824a7
f6102f28 00000000 f3c02a00 c01984d5 c037aef4 c03b6bb4
Dec 18 22:24:04 dcerouter kernel: [  642.017229]        f3e27d00 f3c02a00
c019b446 000003b0 b7f84050 c2715900 f3e27d20 00000050
Dec 18 22:24:04 dcerouter kernel: [  642.017242]        00000000 00000014
00000000 00000013 00000000 c2715900 b7f84000 f7be5fa0
Dec 18 22:24:04 dcerouter kernel: [  642.017256] Call Trace:
Dec 18 22:24:04 dcerouter kernel: [  642.017275]  [show_vfsmnt+293/368]
show_vfsmnt+0x125/0x170
Dec 18 22:24:04 dcerouter kernel: [  642.017296]  [seq_read+470/672]
seq_read+0x1d6/0x2a0
Dec 18 22:24:04 dcerouter kernel: [  642.017343]  [vfs_read+188/352]
vfs_read+0xbc/0x160
Dec 18 22:24:04 dcerouter kernel: [  642.017355]  [seq_read+0/672]
seq_read+0x0/0x2a0
Dec 18 22:24:04 dcerouter kernel: [  642.017373]  [sys_read+65/112]
sys_read+0x41/0x70
Dec 18 22:24:04 dcerouter kernel: [  642.017396]
[sysenter_past_esp+107/169] sysenter_past_esp+0x6b/0xa9
Dec 18 22:24:04 dcerouter kernel: [  642.017454]  =======================
Dec 18 22:24:04 dcerouter kernel: [  642.017458] Code: 74 14 89 44 24 08
c7 44 24 04 bc 24 d8 f8 89 34 24 e8 48 87 44 c7 f6 43 24 40 0f 85 9e 00 00
00 f6 43 25 04 75 58 8b 03 8b 40 24 <f6> 40 46 80 74 4d f6 43 25 08 75 67
8b 03 8b 40 24 f6 40 46 80
Dec 18 22:24:04 dcerouter kernel: [  642.017533] EIP: [<f8d5263d>]
cifs_show_options+0x8d/0x140 [cifs] SS:ESP 0068:f7be5f14
Dec 18 22:24:13 dcerouter kernel: [  650.828244]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /.check_mounted
Dec 18 22:24:13 dcerouter kernel: [  650.828330]  CIFS VFS: Error
0xffffffed on cifs_get_inode_info in lookup of /.check_mounted
Dec 18 22:24:14 dcerouter kernel: [  651.881887]  CIFS VFS: Error
0xffffffed on cifs_get_inode_info in lookup of /public
Dec 18 22:24:14 dcerouter kernel: [  651.905803]  CIFS VFS: Error
0xffffff90 on cifs_get_inode_info in lookup of /public/data
Dec 18 22:24:14 dcerouter kernel: [  651.905837]  CIFS VFS: Error
0xffffffed on cifs_get_inode_info in lookup of /public/data
Dec 18 22:24:14 dcerouter kernel: [  651.908859]  CIFS VFS: Error
0xffffffed on cifs_get_inode_info in lookup of /public/data
Dec 18 22:24:43 dcerouter ddclient[5458]: WARNING:  file
/etc/ddclient.conf, line 16: Invalid Value for keyword 'login' = ''
Eventually the MD gets booted up, although painfully slowly, and then after it is up there is noticeable hanging by the entire box. So last night I rebuilt the MD, deleted the old one, then added a new one. It gets up as far as running through the sarah bits, and then it starts to hang again. This MD has 3 storage drives on it, 2 internal SATA disks, and 1 External Mybook.

I have 2 questions:

1. Does the core use CIFS to attach to these drives that are local to the MD.

2. Is there a way to re-install the CIFS/Samba server bits on the core, to see if it somehow got corrupted?

And the end all option is to re-install the core, which I do not really want to do, but will if necessary.

It seems that CIFS issues should be easily trouobleshot and resolved, however my googling has not payed off yet.

I ask for guidance in this issue, or experiences from other users out there.

Thanks, and Regards,

Seth
".....Because Once you've LinuxMCE'd....."
System stats located at my user page:

http://wiki.linuxmce.org/index.php/User:Seth

colinjones

  • Alumni
  • LinuxMCE God
  • *
  • Posts: 3003
    • View Profile
Re: CIFS errors causing ugly Lag and eventually crashing!!
« Reply #1 on: December 20, 2008, 04:47:05 pm »
Seth - maybe you could start by determining whether it is definitely a CIFS issue root cause or if that is just another symptom, by switching to SMBFS instead? Definitely doesn't look healthy, its oops'ing...

change "cifs" to "smbfs" in all your file share devices and reboot everything.

hari

  • Administrator
  • LinuxMCE God
  • *****
  • Posts: 2428
    • View Profile
    • ago control
Re: CIFS errors causing ugly Lag and eventually crashing!!
« Reply #2 on: December 21, 2008, 01:50:52 am »
remind me to look at show_vfsmnt+0x125/0x170, iirc i've seen that elsewhere already..
rock your home - http://www.agocontrol.com home automation

uplink

  • Administrator
  • Guru
  • *****
  • Posts: 192
  • Linux and LinuxMCE witchdoctor
    • View Profile
Re: CIFS errors causing ugly Lag and eventually crashing!!
« Reply #3 on: November 17, 2009, 06:07:36 pm »
remind me to look at show_vfsmnt+0x125/0x170, iirc i've seen that elsewhere already..

Is it alright if I remind you now? Or is this fixed in 0810? :)