Troubleshooting out of order packets impacting the PCoIP protocol?

Problem

The PCoIP session performance is poor. The logs indicate high packet loss and the error messages related to out of order packets.

Note:The default PCoIP server logs will indicate when out of order mode is activated and deactivated. To see frequency and how many out of order packets are detected, logging must be set to verbosity level 3.

Cause

Network transmission that results in out of order packets will negatively impact real-time protocols such as remote desktop delivery using PCoIP Protocol. Packet re-ordering should be avoided/eliminated on networks using the PCoIP Protocol.

Packet reordering can occur for many reasons. The resolutions below suggest some of the more common reasons.

Resolution

Common possible causes

There are a number of possible causes of packet re-ordering in a LAN or WAN network including:

Inefficient routes

• Different network segments should have routes directly to each other and not via a 3rd router. For example, WAN, LAN and firewall converge on the one network segment. The WAN and LAN should route directly to each other and not via the firewall. Routing via the firewall may result in ICMP redirect packets and duplicate packets.

Different network paths

• Packets that traverse the network across different network routes may arrive at the end-point out-of-order

Router and Switch QoS

• Network devices may be configured to give higher priority to small packets which can cause re-ordering

Load Balancing

• Network load balancing devices can cause packet re-ordering if they are not configured to ensure that packets with the same source and destination address follow the same path. 
• MPLS load balancing, if used, must be configured to per-destination.  Per packet load balancing will result in packet re-ordering

WAN Optimization 

• WAN optimization packet processing can result in some packets being processed before others resulting in re-ordering.  WAN Optimizers must be set to bypass PCoIP packets.

Packet Fragmentation

• Network devices may fragment packets if the packet maximum transmission unit (MTU) is not properly configured across a network path.  Packet fragments may take different paths across a network and arrive out-of-order.  Ensure the MTU in network devices is greater than the PCoIP packet MTU.  If possible, increase the network MTU prior to reducing the PCoIP packet MTU. Packet fragmentation is not supported by PCoIP

Intrusion prevention and firewalls

• When any system scans packets they must be held briefly while they are scanned. If the application uses multiple scan threads and doesn't process the same data stream on the same thread, packets may become out of order due to the speed of each scan thread.

In depth diagnostics

The following section provides some brief guidance as to how to troubleshoot in more detail.

Packets can get out of order at multiple points in the system. Using wireshark to perform packet captures can help track down what network device, network segment or software appliance is contributing to the issue.

When performing a wireshark capture of the issue it is suggested that the following be done.

1. Perform an initial capture on or next to each PCoIP end point. This serves as a reference point to ensure the packets are indeed arriving out of order and are leaving in order. Analysis of the packet capture should prove this.
2. After the issue is confirmed, perform additional captures at various points throughout the network where you believe the packets could potentially be put out of order. Analysis of each capture should show if the issue occurs before or after the point of capture.
3. Repeat the captures until the network device, network segment for software applicance has been isolated.
4. Reconfigure the device or consult the OEM's technical support.