How to achieve maximum performance on a high speed ethernet link

Setup and tune your post-production network for maximum efficiency

The wide adoption of 10 to 100Gbit Ethernet technology has supported the amazing development of the media and entertainment industry and has allowed for workflows that would otherwise have been extremely hard to pull off. But in order to achieve the best possible performance from your high speed ethernet setup, efforts in installation and maintenance are required. In this blog, we will go over the things that you should keep in mind in order to achieve maximum performance with a 10Gbit Ethernet connection.

Do a sanity check

1. Check the workstations and servers network cards.

2. Check all switches / used switch ports, bonds, teams ISLs, M-LAGs etc.

3. Check all patches.

4. Check all wall sockets.

Take care of your cabling

Fibre optic cabling

Maximum reach at 40Gb/s and 100Gb/s:
OM3 50µm — 100m (850nm)
OM4 50µm — 125m (850nm)

2. Avoid bending fibre optic cables. Bending the cable can cause damage to the fibres and can result in refraction. Every cable has an established minimum bending radius that will be printed on the cable packing.

3. Any patches should be new or in near new conditions, handle with care.

4. Upon unpacking or re-packing the cables, make sure that the cable connectors have dust caps on. New cables, in some cases, have a connection check report or an inspection sheet in the packing.

5. Inspect and clean fibre cables before use by using compressed air or isopropyl alcohol.

6. Make sure to use only the connector body for connecting and disconnecting the cable. Never pull at the cable directly.

7. Make sure that the standard cable length of patches and installed lines are not longer than standards recommend.

Inspection sheet of a fibre optic cable

Copper cabling

2. Avoid crimping cable ends on your own as this can lead to performance drop.

3. Do not lay the cables near other electrical installations. This can lead to interference and induce errors into the transmitted signal. When the quality of the cable is in doubt, it can be tested with a tool like the Fluke DSP-2000 CableAnalyzer.

4. Avoid long distances. Don’t exceed 100m cable length with Cat.7 cables. When it comes to 40Gbit cabling, they are best used for interconnectivity of the storage devices. Direct attached copper cables are limited to 7m.
Maximum reach 10GBASE-T Copper, Twisted Pair:
Cat. 5e — 50m
Cat. 6a/7–100m

5. Cables without connector clips should not be used. Even if you fasten the connector with glue or tape, they will eventually come loose.

Use the right PCIe slot

2. Read the manual of the board, sometimes the PCIe slots are not fully loaded with lanes — This means that a physical PCI-e x16 slot might only be electrically equipped with 4 or 8 lanes. In that case, the slot will not provide the expected throughput.

3. Thunderbolt is a mystery for several reasons: In Apples marketing, we see the receiver and transceiver speeds added together; when daisy-chaining devices, the speed must be divided by the number of devices; even some separated ports are bound to the same thunderbolt chip and thereby act as if they were daisy chained.

Check general network settings

At Layer 2, not all gigabit switches support jumbo frames. Those that don’t will drop the frames with no warning. For a jumbo packet to pass through a router, both the ingress and egress interfaces must support the larger packet size. Sometimes, if the MTU isn’t supported, the data will have to be fragmented to a supported size and retransmitted, thereby reducing throughput.

Check general client settings

MacOS

2. To set Jumbo frames on macOS: access the System Preferences -> Network and select the 10G interface. Click the „Advanced…“ button in the lower right. In the Advanced window select the „Hardware“ tab and set „Configure:“ to „Manually“. Set MTU to „Custom“ and enter „9000“.

3. Set the mode to full-duplex, which allows the adapter to send and receive packages at the same time. To set full-duplex on macOS: go into the System Preferences > Network, and select the 10G interfaces. Click the „Advanced…“ button in the lower right. In the Advanced window select the „Hardware“ tab and set „Configure:“ to „Manually“. Set duplex to „full-duplex“.

Windows 10

2. To set full-duplex on Windows: navigate to the Device Manager. Open Properties menu of the adapter you would like to configure. Open the Link Speed tab. In the Speed and Duplex drop down menu select the appropriate speed and duplex.

3. Check that the Flow-control is enabled on both RX and TX.

4. Disable Windows Defender and/or other antivirus tools as well as the Firewall on Windows 10 Machines if possible. You can use the following registry key patch for this task.

Avoid network congestions

In very big switching environments Flow Control might cause other issues. In this case it is recommended that a plan is created that shows the expected data traffic and that other mechanisms are used to avoid congestions.

Metering tools

One direction: Server to client with a TCP window size of 1024 kB:

Server side: iperf -w 1024k -s 
Client Side: iperf -w 1024k -c "servers IP Address"

Second direction: Client to server with a TCP window size of 1024 kB:

Client side: iperf -w 1024k -s 
Server Side: iperf -w 1024k -c "clients IP Address"

After executing the test run, turn on the enhancements and redo both tests, one in each direction. It is usually a good idea to do just one enhancement setting at a time and repeat the test before moving on to the next setting.

2. Iftop — A tool that lets you see traffic flows on a network. It is recommended to run it on a dedicated network interface and with an upper bandwidth limit.
Example iftop command for a 10G Interface:

iftop -i "NIC Name" -m 10G

General suggestions

human-centered media storage