We have been discussing this problem along with my RAID problem here:
http://forum.linuxmce.org/index.php?topic=5892.0. Both problems have become so severe that I decided to separate my network/stability issues into this thread.
For a few weeks now, it appeared that my core was locking up on a nightly basis, without fail. (couldn't SSH in, network would be down, etc..) After more poking around, the core was not actually locking up, but killing the network on Eth:0 (onboard NIC of my Asus M2nPV-VM, a well tested and supported MB) every single night.
Please note that I have been using this same setup for over 6 months with absolutely no problems!
The only changes have been the addition of a new switch (Netgear GS524) and Access Point (Netgear WG302). I've recently spoke with Netgear Tech Support to ensure it is all set up properly.
Here are some different entries from syslog - not sure if any of them are related to my problem:
Aug 6 05:39:20 dcerouter kernel: [ 690.972000] rtc: lost 27 interrupts
Aug 6 05:39:22 dcerouter kernel: [ 693.024000] rtc: lost 28 interrupts
Aug 6 05:39:24 dcerouter kernel: [ 695.076000] rtc: lost 28 interrupts
Aug 6 05:39:30 dcerouter kernel: [ 701.232000] rtc: lost 28 interrupts
Aug 6 05:39:32 dcerouter kernel: [ 703.288000] rtc: lost 27 interrupts
I get the above all the time in my syslog. Not sure if it has always been like this or not, since these problems started before the 6 days worth of archived logs kept on the core.
Aug 6 05:40:42 dcerouter kernel: [ 772.760000] eth0: too many iterations (6) in nv_nic_irq.
Aug 6 05:40:43 dcerouter kernel: [ 773.752000] eth0: too many iterations (6) in nv_nic_irq.
Aug 6 05:40:44 dcerouter kernel: [ 774.752000] eth0: too many iterations (6) in nv_nic_irq.
Aug 6 05:40:46 dcerouter kernel: [ 776.756000] eth0: too many iterations (6) in nv_nic_irq.
I see the above pretty often in syslog as well. Not sure exactly what it means though...
Aug 6 04:00:15 dcerouter kernel: [81443.716000] printk: 1 messages suppressed.
Aug 6 04:00:15 dcerouter kernel: [81443.716000] rtc: lost 27 interrupts
Aug 6 04:00:19 dcerouter kernel: [81447.820000] printk: 1 messages suppressed.
Aug 6 04:00:19 dcerouter kernel: [81447.820000] rtc: lost 28 interrupts
Aug 6 04:00:23 dcerouter kernel: [81451.924000] printk: 1 messages suppressed.
Aug 6 04:00:23 dcerouter kernel: [81451.924000] rtc: lost 28 interrupts
Aug 6 04:00:23 dcerouter kernel: [81452.000000] NETDEV WATCHDOG: eth0: transmit timed out
Aug 6 04:00:23 dcerouter kernel: [81452.000000] eth0: Got tx_timeout. irq: 00000036
Aug 6 04:00:23 dcerouter kernel: [81452.000000] eth0: Ring at 1fe0e000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] eth0: Dumping tx registers
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 0: 00000036 000000ff 00000003 00df03ca 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 20: 00000000 f0000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 40: 0420e20e 0000a455 00002e20 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 60: 00000000 00000000 00000000 0000ffff 0000ffff 0000ffff 0000ffff 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 80: 003b0f3c 00000001 00000000 007f0028 0000061c 00000001 00200000 00007fc9
Aug 6 04:00:23 dcerouter kernel: [81452.000000] a0: 0014050f 00000016 45f31800 00002e3b 00000001 00000000 8000cccd 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] c0: 10000002 00000001 00000001 00000001 00000001 00000001 00000001 00000001
Aug 6 04:00:23 dcerouter kernel: [81452.000000] e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 100: 1fe0e800 1fe0e000 007f00ff 00008000 00010032 00000000 00000017 1fe0ecd0
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 120: 1fe0e050 1c104840 a000ffeb 00000000 00000000 1fe0ecdc 1fe0e05c 0fe08000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 140: 00304120 80c02200 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 180: 00000016 00000008 0194796d 00008103 0000002a 00007800 0194796d 0000f903
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 1a0: 00000016 00000008 0194796d 00008103 0000002a 00007800 0194796d 0000f903
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 1c0: 00000016 00000008 0194796d 00008103 0000002a 00007800 0194796d 0000f903
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 1e0: 00000016 00000008 0194796d 00008103 0000002a 00007800 0194796d 0000f903
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 260: 00000000 00000000 fe020001 00000100 00000000 00000000 fe020001 00000100
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000001
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 2e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 300: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 340: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 380: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 3e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 600: 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 6 04:00:23 dcerouter kernel: [81452.000000] eth0: Dumping tx ring
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 000: 00000000 10394002 20000040 // 00000000 10394202 20000040 // 00000000 10394402 20000040 // 00000000 10394602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 004: 00000000 10394802 20000040 // 00000000 10394a02 20000040 // 00000000 10394c02 20000040 // 00000000 10394e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 008: 00000000 18e68002 20000040 // 00000000 18e68202 20000040 // 00000000 18e68402 20000040 // 00000000 18e68602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 00c: 00000000 18e68802 20000040 // 00000000 18e68a02 20000040 // 00000000 18e68c02 20000040 // 00000000 18e68e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 010: 00000000 19473002 20000040 // 00000000 19473202 20000040 // 00000000 19473402 20000040 // 00000000 19473602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 014: 00000000 19473802 20000040 // 00000000 19473a02 20000040 // 00000000 19473c02 20000040 // 00000000 19473e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 018: 00000000 19472002 20000040 // 00000000 19472202 20000040 // 00000000 19472402 20000040 // 00000000 19472602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 01c: 00000000 19472802 20000040 // 00000000 19472a02 20000040 // 00000000 19472c02 20000040 // 00000000 19472e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 020: 00000000 19471002 20000040 // 00000000 19471202 20000040 // 00000000 19471402 20000040 // 00000000 19471602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 024: 00000000 19471802 20000040 // 00000000 19471a02 20000040 // 00000000 19471c02 20000040 // 00000000 19471e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 028: 00000000 19470002 20000040 // 00000000 19470202 20000040 // 00000000 19470402 20000040 // 00000000 19470602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 02c: 00000000 19470802 20000040 // 00000000 19470a02 20000040 // 00000000 19470c02 20000040 // 00000000 19470e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 030: 00000000 1c107002 20000040 // 00000000 1c107202 20000040 // 00000000 1c107402 20000040 // 00000000 1c107602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 034: 00000000 1c107802 20000040 // 00000000 1c107a02 20000040 // 00000000 1c107c02 20000040 // 00000000 1c107e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 038: 00000000 1c106002 20000040 // 00000000 1c106202 20000040 // 00000000 1c106402 20000040 // 00000000 1c106602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 03c: 00000000 1c106802 20000040 // 00000000 1c106a02 20000040 // 00000000 1c106c02 20000040 // 00000000 1c106e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 040: 00000000 1c105002 20000040 // 00000000 1c105202 20000040 // 00000000 1c105402 20000040 // 00000000 1c105602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 044: 00000000 1c105802 20000040 // 00000000 1c105a02 20000040 // 00000000 1c105c02 20000040 // 00000000 1c105e02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 048: 00000000 1c104002 20000040 // 00000000 1c104202 20000040 // 00000000 1c104402 20000040 // 00000000 1c104602 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 04c: 00000000 1c104802 20000040 // 00000000 20cfb002 20000040 // 00000000 20cfb202 20000040 // 00000000 20cfb402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 050: 00000000 20cfb602 20000040 // 00000000 20cfb802 20000040 // 00000000 20cfba02 20000040 // 00000000 20cfbc02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 054: 00000000 20cfbe02 20000040 // 00000000 0f50d002 20000040 // 00000000 0f50d202 20000040 // 00000000 0f50d402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 058: 00000000 0f50d602 20000040 // 00000000 0f50d802 20000040 // 00000000 0f50da02 20000040 // 00000000 0f50dc02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 05c: 00000000 0f50de02 20000040 // 00000000 0e454002 20000040 // 00000000 0e454202 20000040 // 00000000 0e454402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 060: 00000000 0e454602 20000040 // 00000000 0e454802 20000040 // 00000000 0e454a02 20000040 // 00000000 0e454c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 064: 00000000 0e454e02 20000040 // 00000000 1cd67002 20000040 // 00000000 1cd67202 20000040 // 00000000 1cd67402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 068: 00000000 1cd67602 20000040 // 00000000 1cd67802 20000040 // 00000000 1cd67a02 20000040 // 00000000 1cd67c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 06c: 00000000 1cd67e02 20000040 // 00000000 0d83f002 20000040 // 00000000 0d83f202 20000040 // 00000000 0d83f402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 070: 00000000 0d83f602 20000040 // 00000000 0d83f802 20000040 // 00000000 0d83fa02 20000040 // 00000000 0d83fc02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 074: 00000000 0d83fe02 20000040 // 00000000 105f0002 20000040 // 00000000 105f0202 20000040 // 00000000 105f0402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 078: 00000000 105f0602 20000040 // 00000000 105f0802 20000040 // 00000000 105f0a02 20000040 // 00000000 105f0c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 07c: 00000000 105f0e02 20000040 // 00000000 2740b002 20000040 // 00000000 2740b202 20000040 // 00000000 2740b402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 080: 00000000 2740b602 20000040 // 00000000 2740b802 20000040 // 00000000 2740ba02 20000040 // 00000000 2740bc02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 084: 00000000 2740be02 20000040 // 00000000 1bc58002 20000040 // 00000000 1bc58202 20000040 // 00000000 1bc58402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 088: 00000000 1bc58602 20000040 // 00000000 1bc58802 20000040 // 00000000 1bc58a02 20000040 // 00000000 1bc58c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 08c: 00000000 1bc58e02 20000040 // 00000000 208e7002 20000040 // 00000000 208e7202 20000040 // 00000000 208e7402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 090: 00000000 208e7602 20000040 // 00000000 208e7802 20000040 // 00000000 208e7a02 20000040 // 00000000 208e7c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 094: 00000000 208e7e02 20000040 // 00000000 36193002 20000040 // 00000000 36193202 20000040 // 00000000 36193402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 098: 00000000 36193602 20000040 // 00000000 36193802 20000040 // 00000000 36193a02 20000040 // 00000000 36193c02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 09c: 00000000 36193e02 20000040 // 00000000 113dc002 20000040 // 00000000 113dc202 20000040 // 00000000 113dc402 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 0a0: 00000000 113dc602 20000040 // 00000000 113dc802 20000040 // 00000000 113dca02 20000040 // 00000000 113dcc02 20000040
Aug 6 04:00:23 dcerouter kernel: [81452.000000] 0a4: 00000000 113dce02 20000040 // 00000000 113dd002 20000040 // 00000000 113dd202 20000040 // 00000000 113dd402 20000040
...
...
truncated to stay under the 20000 character post limit
I have a slew (meaning thousands) of these prior to my reboot this morning. The first one appeared in the log at 0358 in the morning, and they continued at least a few times a minute for the remainder of the night until my reboot.
Ok, so what now? I can't fathom why out of nowhere I am having these problems after 6 months of near perfect stability?