Frustrating new Network issue

[PeteK]

I'm having an issue with my 0810 setup that makes no sense whatsover and is yet again driving me crazy.
I installed 0810  successfully in the network configuration  cable modem->router->core->internal network.  In this configuration, everything seems to be working OK.

However, I tried removing the router to give me: cable modem->core->internal network.  In this configuration, I can access the net from the core without a problem, but I can't access http at all from any PCs on the internal network.  I can ping lmce.org without a problem, however.  Switching back to the 1st configuration fixes things consistently. Switching back to the second configuration breaks things consistently.

I captured some data with wireshark.  This is a request for linuxmce.org from firefox on a windows PC on the internal network

looking at the external NIC (IP: 192.168.1.108):

No.     Time        Source                Destination           Protocol Info
      1 0.000000    192.168.1.108         193.200.113.133       TCP      49281 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1456 WS=2
      2 0.175891    193.200.113.133       192.168.1.108         TCP      http > 49281 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=7
      3 0.176709    192.168.1.108         193.200.113.133       TCP      49281 > http [ACK] Seq=1 Ack=1 Win=66976 Len=0

now the http get call

      4 0.177002    192.168.1.108         193.200.113.133       HTTP     GET / HTTP/1.1

the ack

      5 0.356041    193.200.113.133       192.168.1.108         TCP      http > 49281 [ACK] Seq=1 Ack=420 Win=6912 Len=0

and data...
 
     6 0.369530    193.200.113.133       192.168.1.108         TCP      [TCP segment of a reassembled PDU]
     7 0.369780    193.200.113.133       192.168.1.108         TCP      [TCP segment of a reassembled PDU]
....

I see this correct behavior when I use wireshark on eth1 (internal NIC) as well.

But when I connect the core directly to the modem, I get a new IP address of  76.83.118.XX
same request:   (XX is the last byte of my IP address)

No.     Time        Source                Destination           Protocol Info
      1 0.000000    76.83.118.XX          193.200.113.133       TCP      49173 > http [FIN, ACK] Seq=1 Ack=1 Win=16744 Len=0
      2 0.066986    76.83.118.XX          193.200.113.133       TCP      49176 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1456 WS=2
      3 0.215705    193.200.113.133       76.83.118.XX          TCP      http > 49173 [ACK] Seq=2913 Ack=2 Win=54 Len=0
      4 0.244939    193.200.113.133       76.83.118.XX          TCP      http > 49176 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 WS=7
      5 0.245589    76.83.118.XX          193.200.113.133       TCP      49176 > http [ACK] Seq=1 Ack=1 Win=66976 Len=0

now the http get

     6 0.246035    76.83.118.XX          193.200.113.133       HTTP     GET / HTTP/1.1

and the ack

     7 0.428837    193.200.113.133       76.83.118.XX          TCP      http > 49176 [ACK] Seq=1 Ack=369 Win=6912 Len=0

...and then nothing, no data.  Firefox is stuck at 'Waiting for reply.'  This happens with any website I try.  Does anyone have any idea what might be going on?  Could this be an issue with my provider?  Any help would be greatly appreciated as the WAF is quickly decreasing and I have a lot of work on this that I want to do.

[colinjones]

Pete

Packets 1 and 3 in the second trace are from a previous session, so ignore those.

It looks like your cable modem is operating in bridge mode - does your model have a router mode? (worthwhile googling it because often they do and just need a small hack to enable it) Its just that router mode can make things much easier to follow and get working than a bridge.

Can you try doing the same browsing actually on the core rather than a machine in the internal network? Just to eliminate any routeability issues. Also, can you do a full ifconfig on eth0 so we can check the default gateway, subnet masks, etc? Finally, print out the routing table.

When you say you can ping linuxmce.org, do you mean from the same PC that is having the browsing issues? If so, have you tried any other protocols?

Could be a routing issue, could be a NAT issue, could be filtering at the ISP...

[nosebreaker]

Like colinjones says, it might be the ISP.  I've had issues with Comcast and RCN blocking things if you change the device/computer that is plugged into the cablemodem in the past.  Give them a call and ask, especially if it is only HTTP that is being blocked (that indicates routing is fine so probably not the LMCE box).

[PeteK]

Thanks for the info, gang.

colinjones:
Http access works fine from the core, even in both configurations (internal network working, internal network not working), which makes me a bit worried that maybe it's not an ISP issue.  I can print out DNS information and routing tables when I get back in front of the PC, but he DNS information looked good for both the core and the clients in that DNS and gateway information seemed valid.  The only protocols I've tried so far are http and ping, though it looks like whatever protocol Yahoo IM is using is also affected.
 
I appreciate your advice.

[hari]

i've not looked at it in detail but it could be a fragmentation issue related to icmp filtering (fragmentation needed).
I'd try to use an interface mtu of 1400 on the external nic for a test.

br, Hari

[nosebreaker]

Duh, yes hari is probably on to it.  ICMP is very small, the http get request is small, but the http reply is large.  I'd set the MTU to a super-low number to test, like 900 or something to be sure.  If that works, then increase it bit by bit.

[colinjones]

Doesn't really explain why it works fine from the core though - unless the issue is fragmentation between the modem and the core (seems unlikely). Also, the MTU sets the size for outbound packets, these are inbound - the MTU isn't communicated to the other side... still thinking through this one... pete, can you get those pieces of info? (wasn't asking for DNS, not really interested in that)

hmmm. if not MTU, certainly something like it that only effects small packets like the others suggested. The only other thing that springs readily to mind is a NAT issue. (the core fw NATs through to your internal network) but the evidence doesn't support that. It would work from the core but not the internal network (as you have) but not even the inital SYN/ACKs would work, which isn't what you have....

[hari]

ah, i wasn't seeing that this request was from a client. Of course you have to apply the mtu setting there for the test.

best regards,
Hari

[PeteK]

colinjones--

Here's the information you requested.  This issue doesn't seem to be going away, so any advice would definitely be appreciated, though I just don't see anything unusual in this data.

I'm not quite sure I understand the MTU suggestions.  Should I decrease MTU size on eth1 (internal) or eth0 (external)?

Code: [Select]

linuxmce@dcerouter:~$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:17:22:3f:8c
          inet addr:76.88.197.XXX  Bcast:255.255.255.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:576  Metric:1
          RX packets:128870 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:15284361 (15.2 MB)  TX bytes:3252505 (3.2 MB)
          Memory:c2820000-c2840000

eth1      Link encap:Ethernet  HWaddr 00:15:17:22:3f:8d
          inet addr:192.168.80.1  Bcast:192.168.80.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe22:3f8d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16570 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104093 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2085853 (2.0 MB)  TX bytes:4663577 (4.6 MB)
          Memory:c2800000-c2820000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:83145 errors:0 dropped:0 overruns:0 frame:0
          TX packets:83145 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10946905 (10.9 MB)  TX bytes:10946905 (10.9 MB)

note: Last Byte of IP address edited for posting

linuxmce@dcerouter:~$ cat /etc/resolv.conf
domain bak.rr.com
search bak.rr.com
nameserver 76.85.229.110
nameserver 76.85.229.111
linuxmce@dcerouter:~$


linuxmce@dcerouter:~$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.80.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
76.88.192.0     0.0.0.0         255.255.248.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
0.0.0.0         76.88.192.1     0.0.0.0         UG    100    0        0 eth0

linuxmce@dcerouter:~$ netstat -ree
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface    MSS   Window irtt
192.168.80.0    *               255.255.255.0   U     0      0        0 eth1     0     0      0
76.88.192.0     *               255.255.248.0   U     0      0        0 eth0     0     0      0
link-local      *               255.255.0.0     U     1000   0        0 eth0     0     0      0
default         cpe-76-88-192-1 0.0.0.0         UG    100    0        0 eth0     0     0      0

Thanks a lot for the help so far, gang.

[PeteK]

Another interesting piece of data is that when I reconfigure to the modem->router->core->internal network (i.e. working) state, I call dhclient -r to release followed by dhclient to renew dhcp info.  After that, I can access the internet from the core, but I still have the same problem with PCs on the internal network.  They only start working after a full reboot of the core.

[hari]

hmm, your mtu on the ext is pretty small (576):
eth0      Link encap:Ethernet  HWaddr 00:15:17:22:3f:8c
          inet addr:76.88.197.XXX  Bcast:255.255.255.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:576  Metric:1

i'd bet this is the direction where the problem comes from. For a quick test set an mtu of 500 on the windows client. You don't need to touch the core for this test.

best regards,
Hari

[PeteK]

Hari, Colinjones--

Apparently, setting the MTU on windows machines (especially vista) is rather a pain to do.  Instead I followed your previous advice and upped the eth0 (external) MTU to 1500.  Things seem to be working after that.  I'm not sure why the initial settings were different, as both adapters are onboard devices on the core, but this is a new phenomenon on 0810.  The tx queue length is also different, inexplicably.

Thanks a lot for the help guys, it's truly appreciated and I think I can get back to real work now.

Thanks a million

[hari]

Hari, Colinjones--

Apparently, setting the MTU on windows machines (especially vista) is rather a pain to do.  Instead I followed your previous advice and upped the eth0 (external) MTU to 1500.  Things seem to be working after that.  I'm not sure why the initial settings were different, as both adapters are onboard devices on the core, but this is a new phenomenon on 0810.  The tx queue length is also different, inexplicably.

is this surely new on 0810? We maybe get that MTU via DHCP. You may want to check that and verify the proper MTU with your provider.
You may want to try a txqlen of 1000.

Thanks a lot for the help guys, it's truly appreciated and I think I can get back to real work now.

Thanks a million

you're welcome!

br, Hari

[colinjones]

Good news! btw 576 is the "correct" MTU for the Internet, I have never seen any options in DHCP to set this, but IP generally has MTU/MSS minimum protocols for determining things like this. It does it for a path rather than an interface, so I'm not sure how it applies... can't remember the detail now. But either way, that value must be coming from the cable modem/provider

[hari]

Good news! btw 576 is the "correct" MTU for the Internet, I have never seen any options in DHCP to set this, but IP generally has MTU/MSS minimum protocols for determining things like this. It does it for a path rather than an interface, so I'm not sure how it applies... can't remember the detail now. But either way, that value must be coming from the cable modem/provider

The Internet has no MTU, but your interfaces have. So I really don't understand what you mean with "correct for the Internet" (btw it was not working with 576). The BOOTP/DHCP option is 26, called "Interface MTU".
The MSS is the maximum segment size for tcp connections. That one is calculated for the "path". There are options to tweak that on routers (e.g. "adjust-mss" on cisco routers), but that only works for tcp. So better use proper MTUs everywhere and allow ICMP fragmentation needed to avoid such kind of problems.

best regards,
Hari