Core crashing... routes but little else...

[colinjones]

In recent days (with a clean build) I have been noticing in the mornings sometimes the core isn't quiet working.

Usually, my PC on the internal network cannot browse the Internet. I can route through the core no problem, but it seems to have stopped forwarding DNS requests. And i cannot ssh in.

This morning I may have caught it just before it got into this state as DNS wasn't being forwarded but I was able to ssh in. However, when I did this the shell started behaving strangely and then started generating Seg Fault errors for every command I issued, then died.

When I looked in messages there was obviously some significant problem last night that effectively time bombed the system for when I started using it this morning.

Code: [Select]

Oct 31 19:12:46 dcerouter -- MARK --
Oct 31 19:17:14 dcerouter kernel: [122696.256000]  [bad_page+99/160] bad_page+0x63/0xa0
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [get_page_from_freelist+835/928] get_page_from_freelist+0x343/0x3a0
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [__alloc_pages+79/832] __alloc_pages+0x4f/0x340
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [net_rx_action+200/528] net_rx_action+0xc8/0x210
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [anon_vma_prepare+29/224] anon_vma_prepare+0x1d/0xe0
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [__pagevec_lru_add_active+182/208] __pagevec_lru_add_active+0xb6/0xd0
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [__handle_mm_fault+2247/2816] __handle_mm_fault+0x8c7/0xb00
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [rtc_interrupt+158/256] rtc_interrupt+0x9e/0x100
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [do_page_fault+294/1680] do_page_fault+0x126/0x690
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [handle_IRQ_event+48/96] handle_IRQ_event+0x30/0x60
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [do_page_fault+0/1680] do_page_fault+0x0/0x690
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  [error_code+114/128] error_code+0x72/0x80
Oct 31 19:17:14 dcerouter kernel: [122696.260000]  =======================
Oct 31 19:32:46 dcerouter -- MARK --
Oct 31 19:52:46 dcerouter -- MARK --
Oct 31 19:55:50 dcerouter kernel: [125011.700000] c017cf9a
Oct 31 19:55:50 dcerouter kernel: [125011.700000] Modules linked in: nls_cp437 cifs ide_generic nvidia(P) agpgart smbfs ztdummy zaptel crc_ccitt rfcomm l2cap bluetooth xt_TCPMSS ipt_MASQUERADE ipt_TTL iptable_mangle xt_tcpudp nfsd exportfs xt_mark lockd sunrpc xt_state iptable_nat iptable_filter ip_tables x_tables ppdev autofs4 powernow_k8 cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats freq_table sbs ac button video dock container battery nf_nat_irc nf_nat_ftp nf_nat nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nfnetlink parport_pc lp parport snd_hda_intel snd_pcm_oss snd_mixer_oss mt2060 snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi xpad snd_seq_midi_event dvb_usb_dib0700 dib7000m dib7000p dvb_usb snd_seq dvb_core snd_timer snd_seq_device lirc_imon lirc_dev dvb_pll af_packet dib3000mc dibx000_common pl2303 ftdi_sio usbserial snd i2c_core k8temp pcspkr soundcore shpchp pci_hotplug ipv6 joydev evdev ext3 jbd mbcache s
Oct 31 19:55:50 dcerouter kernel:  sd_mod ide_cd cdrom ahci usbhid hid floppy uhci_hcd e1000 ohci1394 ieee1394 forcedeth ata_generic libata scsi_mod amd74xx ide_core ehci_hcd ohci_hcd usbcore raid10 raid456 xor raid1 raid0 multipath linear md_mod thermal processor fan fuse apparmor commoncap
Oct 31 19:55:50 dcerouter kernel: [125011.708000] c017cf9a
Oct 31 19:55:50 dcerouter kernel: [125011.708000] Modules linked in: nls_cp437 cifs ide_generic nvidia(P) agpgart smbfs ztdummy zaptel crc_ccitt rfcomm l2cap bluetooth xt_TCPMSS ipt_MASQUERADE ipt_TTL iptable_mangle xt_tcpudp nfsd exportfs xt_mark lockd sunrpc xt_state iptable_nat iptable_filter ip_tables x_tables ppdev autofs4 powernow_k8 cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats freq_table sbs ac button video dock container battery nf_nat_irc nf_nat_ftp nf_nat nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nfnetlink parport_pc lp parport snd_hda_intel snd_pcm_oss snd_mixer_oss mt2060 snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi xpad snd_seq_midi_event dvb_usb_dib0700 dib7000m dib7000p dvb_usb snd_seq dvb_core snd_timer snd_seq_device lirc_imon lirc_dev dvb_pll af_packet dib3000mc dibx000_common pl2303 ftdi_sio usbserial snd i2c_core k8temp pcspkr soundcore shpchp pci_hotplug ipv6 joydev evdev ext3 jbd mbcache s
Oct 31 19:55:50 dcerouter kernel:  sd_mod ide_cd cdrom ahci usbhid hid floppy uhci_hcd e1000 ohci1394 ieee1394 forcedeth ata_generic libata scsi_mod amd74xx ide_core ehci_hcd ohci_hcd usbcore raid10 raid456 xor raid1 raid0 multipath linear md_mod thermal processor fan fuse apparmor commoncap
Oct 31 19:55:50 dcerouter kernel: [125011.708000] rtc: lost 1 interrupts
Oct 31 19:55:50 dcerouter kernel: [125011.884000] c017cf9a
Oct 31 19:55:50 dcerouter kernel: [125011.884000] Modules linked in: nls_cp437 cifs ide_generic nvidia(P) agpgart smbfs ztdummy zaptel crc_ccitt rfcomm l2cap bluetooth xt_TCPMSS ipt_MASQUERADE ipt_TTL iptable_mangle xt_tcpudp nfsd exportfs xt_mark lockd sunrpc xt_state iptable_nat iptable_filter ip_tables x_tables ppdev autofs4 powernow_k8 cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats freq_table sbs ac button video dock container battery nf_nat_irc nf_nat_ftp nf_nat nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nfnetlink parport_pc lp parport snd_hda_intel snd_pcm_oss snd_mixer_oss mt2060 snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi xpad snd_seq_midi_event dvb_usb_dib0700 dib7000m dib7000p dvb_usb snd_seq dvb_core snd_timer snd_seq_device lirc_imon lirc_dev dvb_pll af_packet dib3000mc dibx000_common pl2303 ftdi_sio usbserial snd i2c_core k8temp pcspkr soundcore shpchp pci_hotplug ipv6 joydev evdev ext3 jbd mbcache s
Oct 31 19:55:50 dcerouter kernel:  sd_mod ide_cd cdrom ahci usbhid hid floppy uhci_hcd e1000 ohci1394 ieee1394 forcedeth ata_generic libata scsi_mod amd74xx ide_core ehci_hcd ohci_hcd usbcore raid10 raid456 xor raid1 raid0 multipath linear md_mod thermal processor fan fuse apparmor commoncap
Oct 31 19:55:50 dcerouter kernel: [125011.888000] rtc: lost 2 interrupts
Nov  1 07:00:28 dcerouter syslogd 1.4.1#21ubuntu3: restart.
Nov  1 07:00:28 dcerouter kernel: Inspecting /boot/System.map-2.6.22-14-generic
Any thoughts on what might be causing this?

[merkur2k]

The kernel is certainly unhappy about something. I would suspect hardware, most likely memory. Take the machine apart and give it a good cleaning if its been awhile, then start removing or swapping out nonessential hardware (extra stick of ram, storage drives, etc). Make sure the cpu heatsink/fan is still working properly (fan running slow, problem with thermal paste).