Improving network proformance

[tux-box1]

Hello all.
I was hoping for some help on this, I'm currently experiencing poor network performance on my hybrid Kb a sec transfers.
I'm using cat6 cable and a gigabit switch, Gigabit nic's on both systems.

The lay out of the network..
Cable modem feeds-> router feeds-> Gigabit switch
Gigabit switch feeds ->
Lmce
PC

I'm getting Kb a sec here not near what I would expect getting.
The drives I'm copying to are 5400 rpm and 130.28Mb/sec using hdparm on average.
I have achieved 10Mib a sec transfers on that system in the past before Linuxmce.

Suggestions??

[maverick0815]

Why don't you try this: cablemodem to lmce to gigabitswitch to pcs you don't need a router this way, thus eliminating a potential troublemaker.

[JoakimL]

What's the layout of your internal network - do you have a Gbps switch there too?
You should be experiencing very good performance both from PC to Hybrid and from PC to Internet; lmce is not adding any overhead more than any normal Linux server. Did you check the logs for any misbehaviour on your NICs? Are they set up to run at Gigabit speed?

Try to isolate the problem  - first investigate throughput speed from a PC connected to the outer network to Internet. Then from The Hybrid to Internet and last from a PC on the internal network to Internet. The results should be close to each other, if not you will see where the problem is introduced.
A second investigation would be to connect a PC to the outer network, disable the firewall in the Hybrid and test throughput towards the Hybrid; then connect the same PC to the internal network and redo the test. Any differences here?


/Joakim

[brononius]

First of all, try to understand (draw) your connection. What's your WAN interface, what's your LAN interface (fe in my case is LAN: eth0, WAN: eth1). This way you know where you're looking. A picture is saying a lot more then some words.

I'll put some basic test here that can point you to the right place where to look...


Connect
Can you connect with ssh to your core? I'll assume you're using a linux client.

Code: [Select]

ssh username@192.168.80.1

Network interfaces
With ifconfig, you'll see a lot more about your network cards.

Code: [Select]

ifconfig-> do you see any errors on the interfaces (collisions)? In fact, you should only see RX/TX packets here...
-> What the subnetmask of you interfaces (the same as for your routing table)?

fe

eth0      Link encap:Ethernet  HWaddr 00:21:85:5d:d6:05  
          inet addr:192.168.80.1  Bcast:192.168.80.255  Mask:255.255.255.0
          inet6 addr: fe80::221:85ff:fe5d:d605/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:4338750 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2156345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1817464938 (1.8 GB)  TX bytes:161459655 (161.4 MB)
          Interrupt:219 Base address:0xa000

eth1      Link encap:Ethernet  HWaddr 00:04:76:97:7b:98  
          inet addr:192.168.0.184  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::204:76ff:fe97:7b98/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:49496 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32634 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:26781428 (26.7 MB)  TX bytes:7483598 (7.4 MB)
          Interrupt:21 Base address:0xac00

For more information, you can always do following:

Code: [Select]

sudo ethtool -S eth0This should give you more information:

NIC statistics:
     tx_packets: 2190314
     rx_packets: 4367767
    tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 1166
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 4350464
     broadcast: 11762
     multicast: 5541
     tx_aborted: 0
     tx_underrun: 0

Routing

Code: [Select]

netstat -rn-> should give you a clear routing table

fe

192.168.0.0     0.0.0.0         255.255.255.0  U         0 0          0 eth1 <<<Your WAN network>>>
192.168.80.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0 <<<Your LAN network>>>
0.0.0.0        192.168.0.1    0.0.0.0         UG        0 0          0 eth1 <<<All the rest to your internetprovider>>>

Traceroute
With a traceroute you'll see the connection path. This is useful to see where your connections goes, and if any devices are taking a long time to respond.
This is best done from you client machine on (i don't have direct access to a client now, so i can't show you my result, sorry).

Code: [Select]

traceroute linuxmce.org
fe (this is done from another network, so it won't be the same for you). The most important thing in here are the first lines, with your private ip's, and the first public ip (your internetrouter):

traceroute to linuxmce.org (193.200.113.133), 30 hops max, 60 byte packets
1  192.168.80.1 (192.168.80.1)  3.575 ms  3.778 ms  4.355 ms <<< Your internal gateway >>>
 2  192.168.0.1 (172.31.31.1)  7.338 ms  7.716 ms  7.932 ms <<< Your internet gateway >>>
 3  3.123-123-109.adsl-dyn.isp.belgacom.be (109.123.123.3)  14.416 ms  16.522 ms  19.280 ms  <<< Your public ip >>>
 4  160.241-183-91.adsl-static.isp.belgacom.be (91.183.241.160)  21.939 ms  24.110 ms  26.778 ms
 5  ae-15-1000.ibrstr1.isp.belgacom.be (91.183.246.107)  52.974 ms  31.339 ms  53.332 ms
 6  bru-11-r6-t9-4.car.belbone.be (80.84.20.18)  36.676 ms bru-11-r6-t8-4.car.belbone.be (80.84.20.78)  13.732 ms bru-11-r6-t7-4.car.belbone.be (80.84.21.34)  14.288 ms
 7  94.102.162.50 (94.102.162.50)  17.156 ms  19.748 ms  22.358 ms
 8  neotelecoms.bnix.net (194.53.172.79)  31.315 ms  33.714 ms  36.135 ms
 9  xe0-0-0.tcr1.gs.par.as8218.eu (83.167.55.20)  58.078 ms  58.461 ms  58.842 ms
10  83.167.56.155 (83.167.56.155)  62.023 ms  64.715 ms  67.321 ms
11  xe0-1-1.r06.uni.vie.at.nextlayer.net (212.69.191.150)  67.712 ms  69.814 ms  35.678 ms
12  xe4-1-1.r05.rdh.vie.at.nextlayer.net (92.60.3.144)  38.981 ms xe2-1-0.r05.rdh.vie.at.nextlayer.net (92.60.2.144)  39.179 ms xe4-1-1.r05.rdh.vie.at.nextlayer.net (92.60.3.144)  40.699 ms
13  vl21.r02.rdh.vie.at.nextlayer.net (81.16.147.192)  43.634 ms  45.582 ms  50.315 ms
14  abaton.rdh.vie.at.nextlayer.net (81.16.148.34)  51.541 ms  37.767 ms  39.880 ms
15  193.200.112.37 (193.200.112.37)  41.829 ms  43.772 ms  45.978 ms
16  94.247.144.120 (94.247.144.120)  48.424 ms  50.878 ms  53.792 ms
17  193.200.113.133 (193.200.113.133)  55.457 ms  35.938 ms  38.298 ms

Send us the results of this, from there on we can maybe start troubleshoot a bit more ...

Of course, it's best that you do all this tests from all the machines (or at least all the machines that are in the loop). This is the reason why i wrote that's always best to have a small drawing beside... This way, you can isolate the problem to  a certain point/connection...

[klovell]

The network and it's components aren't always the root cause of poor network performance.  Cat6 and gig nics only mean you have the potential for high speed data transfer but other factors can  and usually are the bottle neck.  To be completely thorough we'll also need to know:

The system specs of your core?
Are you using IDE drives for some reason?
Are you using raid and if so what type? Is your controller on board or a add-on card?
How much Ram do you have and what is your bus speeds? What's the processor speed?
Are you having speed issues from multiple clients on the network or just one? 
Are they connecting over wireless? 
What's your wireless AP speed? What are the client speeds? If you answer wireless N to the speed, are there any devices on your network that is not wireless N? How many?
What encryption are you using?
What are the client specs? Quick over view.   
How many client are accessing the shares at any given time?
How long is the cabling between each of your network devices? Are you using pre-made cables or did you make them yourself? How old are the cables? 

Most importantly:
What is the current transfer speed?
How are you getting these number?
What's the transfer size?
Are you testing with a large file or a bunch of small files that equal a larger number?

Any single one of these areas I asked about can diminish network performance.  The last 4 goes to the simple fact that sometimes people are not realistic about the performance they're going to get.  I'm not trying to insult you but it is what it is, just have to make sure.

[DragonK]

Are you testing with a large file or a bunch of small files that equal a larger number?

4000 x 1MB files take longer to transfer then 1 x 4GB file .........

[klovell]

4000 x 1MB files take longer to transfer then 1 x 4GB file .........

You hit the nail on the head my friend.  It's funny how many people don't know that. 

We havn't even dove into the configurations of his router, switch and nics.  Jombo frames, auto speed vs locking in a speed, QOS, etc.  All the fun stuff!

[tux-box1]

Thank you all for your posts. I'm not in front of my network at the moment so the testing and tweeking will have to wait.

I thought that I had mention in my original post that I have had 10Mb second transfers on the exact setup in the past(3 days ago) only difference between those speeds and now is the install of linuxmce. So I believe its safe to say its not a hardware problem. 

[klovell]

Thank you all for your posts. I'm not in front of my network at the moment so the testing and tweeking will have to wait.

I thought that I had mention in my original post that I have had 10Mb second transfers on the exact setup in the past(3 days ago) only difference between those speeds and now is the install of linuxmce. So I believe its safe to say its not a hardware problem. 

I noticed the comment about 10mbs transfer speed but I ignored it. 

Here's the deal 10 mbs is far from acceptable from what you have described thus far.  For example, my old network setup was all gigabit, but I was using cat5e cabling and a dinky little $55 8 port unmanaged dlink switch.  I would see 200-250mbs transfer speed from client to client, from Nas to client it was around 300-350mbs.  It really didn't matter to much what I was coping but how I copied it made the world of difference.  Now I have a $400 48 port power connect and I upgraded the cabling to cat6 (which doesn't actually offer much improve over cat5e but why not use it anyway?), I am maxing out my gigabit connections from client to client and nas to client.  Again how I copy the files make the difference but either way I've never notice less than half the pipe filled.  Except for rsync, which I'm sure is by design, but even rsync is using 200mbs to backup my nas.  I'm not even using any advance features of the managed switch like lagg groups, jumbo frames, class of service, etc.  So lets say you're using a cheaper consumer grade switch you still wasn't where you should have been before linuxmce.  I'm not saying it's absolutely a hardware issue but I don't think you've provided sufficient proof to rule it out.   

My questions and every other question and test suggested before mine needs to addressed before any useful, efficient troubleshooting can be done.

[klovell]

After re-reading the OP, is it possible you're talking about internet speeds?  That kind of makes sense to me if you're will to accept 10mbs when you're using gigabit gear.

[tux-box1]

OK... I don't know what was going on but after a reinstall form scratch of LinuxMCE I'm getting much better speeds. I ran 5 quick tests of two types. Meaning I ran each test type 5 times.
First type, a folder of HD pictures ranging form 44 KB to 605KB ~ 500 pictures in all average speed 10Mb a sec.
Second type, A dvd ISO. ~4.15GB average 75MB a sec.

Again no hardware changes, only software changes on one system and that's Linuxmce.
I'll run some IPERF test's this weekend when I have more time.
Until then this is much better and workable.

[brononius]

Start first with some local tests. Check your collisions and so. This can help you to find out if you've got a cable issue, poor networkcard...
See my original post above...

Always work the layers up!!!
It's a very good and proven troubleshooting path...

[tux-box1]

All tests ran on fresh install, after I was getting the "better" speeds. The old install was getting KB second transfers on all files. The new install, well see my last post. I have tested connected to the eth0 port but got slower speeds on the eth1 port. So one thing at a time.
First the results of the test's you wanted. I'll do some hardware testing this weekend. I have a cable tester I can use, as for testing the nics them self's I'll use IPERF..

First of all, try to understand (draw) your connection. What's your WAN interface, what's your LAN interface (fe in my case is LAN: eth0, WAN: eth1). This way you know where you're looking. A picture is saying a lot more then some words.

Been their done that, networking 101 I have a quick and dirty google doc if you can send me an email. I'm not a complete noob just no expert.

I'll put some basic test here that can point you to the right place where to look...


Connect
Can you connect with ssh to your core? I'll assume you're using a linux client.

Code: [Select]

ssh username@192.168.80.1

Network interfaces
With ifconfig, you'll see a lot more about your network cards.

Code: [Select]

ifconfig-> do you see any errors on the interfaces (collisions)? In fact, you should only see RX/TX packets here...
-> What the subnetmask of you interfaces (the same as for your routing table)?

Code: [Select]

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 20:cf:30:c0:ff:46
          inet addr:192.168.3.80  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::22cf:30ff:fec0:ff46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:643754728 errors:0 dropped:3991875718 overruns:0 frame:0
          TX packets:156841564 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1054057917 (1.0 GB)  TX bytes:1290091923 (1.2 GB)
          Interrupt:221I noticed the dropped packets this many raises a flag

fe
For more information, you can always do following:

Code: [Select]

sudo ethtool -S eth0This should give you more information:

Code: [Select]

$ sudo ethtool -S eth0
NIC statistics:
     tx_packets: 156841598
     rx_packets: 643754748
     tx_errors: 0
     rx_errors: 0
     rx_missed: 137
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 643740548
     broadcast: 4967
     multicast: 14200
     tx_aborted: 0
     tx_underrun: 0 Not perfect but nothing big here.

Routing

Code: [Select]

netstat -rn-> should give you a clear routing table

Code: [Select]

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.80.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
239.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth1
0.0.0.0         192.168.3.1     0.0.0.0         UG        0 0          0 eth0

fe


Traceroute
With a traceroute you'll see the connection path. This is useful to see where your connections goes, and if any devices are taking a long time to respond.
This is best done from you client machine on (i don't have direct access to a client now, so i can't show you my result, sorry).

Code: [Select]

traceroute linuxmce.org
fe (this is done from another network, so it won't be the same for you). The most important thing in here are the first lines, with your private ip's, and the first public ip (your internetrouter):

Code: [Select]

$ traceroute linuxmce.org
traceroute to linuxmce.org (193.200.113.133), 30 hops max, 40 byte packets
 1  192.168.3.1 (192.168.3.1)  0.668 ms  1.804 ms  2.200 ms
 2  X.X.X.X (X.X.X.X)  8.378 ms  12.214 ms  13.201 ms
 3  XX.XX.XX.XXX.com (XX.XX.XX.XX)  13.566 ms  13.753 ms  13.921 ms
 4  XXX.XX.XX.XX.com (XX.XX.XX.XX)  19.750 ms  20.698 ms  23.063 ms I marked out the IPs with XXs, why your having me test the out side network is beyond me.

Send us the results of this, from there on we can maybe start troubleshoot a bit more ...

Of course, it's best that you do all this tests from all the machines (or at least all the machines that are in the loop). This is the reason why i wrote that's always best to have a small drawing beside... This way, you can isolate the problem to  a certain point/connection...

[brononius]

eth0   Link encap:Ethernet  HWaddr 20:cf:30:c0:ff:46
          inet addr:192.168.3.80  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::22cf:30ff:fec0:ff46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:643754728 errors:0 dropped:3991875718 overruns:0 frame:0
          TX packets:156841564 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1054057917 (1.0 GB)  TX bytes:1290091923 (1.2 GB)
          Interrupt:221

I'm a bit worried about your dropped recieved packages. What is Eth0? Seen the amount of packages, i would say the LAN side?
This is mostly due a bad driver or lack of networkmemory... Bad cables mostly give collisions...

Maybe a quick test is to change the networkcard?
Try maybe to swap your LAN / WAN side?

[brononius]

I marked out the IPs with XXs, why your having me test the out side network is beyond me.

Just for your info:
I meant to do a traceroute from you client machine (behind your linuxmce box). This can bring forward who gives you latency (your network gateway, internet gateway...). Maybe not directly useful for you GB copy problem, since I guess it's something local (nice to have the drawing, remember).
These are always basic/first test i'll do to troubleshoot network issues.
Another very useful command in this line is the command MTR. Give it a try: 'sudo mtr www.linuxmce.org'

But i would focus on the dropped packets in your case...