VPP Root Cause Analysis 2n-skx-xxv710 2t1c pdr

Legend to Table:
YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, “1” for 2019-08-21, “2” for 2020-03-27).
Diff1 - difference between environment versions “2” and “1” for the same VPP version, 1908.1-2 vs. 1908.1-1 (if data for the latter not present, 1908.0-1 is used).
Diff2 - difference between the current and the previous VPP version in the current environment version “2”, 2001.0-2 vs. 1908.1-2.
RCA1 - root cause analysis reference for Diff1.
RCA2 - root cause analysis reference for Diff2.

RCA1:
[1] DONE, Impact of upgrades: i) Skx ucode from 0x2000043 to 0x2000065, ii) Linux kernel from 4.15.0-60 to 4.15.0-72 and iii) SuperMicro motherboard BIOS from 3.0c to 3.2.
[2] DONE, Applied fix of FVL NIC firmware 6.0.1 for increasing TRex Mpps rate from 27 Mpps to 37 Mpps, [CSIT-1503], [TRex-519].
[3] DONE, Applied VPP PAPI fix to enable memif zero-copy, [CSIT-1592], [VPP-1764].
[4] OPEN, Higher than before StDev of PDR throughput for VPP vhost-user with VPP-inside-VM, under investigation, [CSIT-1699], [CSIT-1704].
RCA2:
[5] DONE, caused by setting MTU [VPP-1876], [CSIT-1705].
[6] DONE, ip4base-nat44 regression, ge-23963 (https://gerrit.fd.io/r/c/vpp/+/23963#message-044278e6_752c3327).
[7] POSTPONED, avf-ip4scale regression, CANDIDATE(S) before ge-22699, [CSIT-1699], [CSIT-1706].
[8] OPEN, VPP vhost-user with VPP-inside-VM higher than before stdev of PDR throughput, under investigation, [CSIT-1699], [CSIT-1704].
[9] DONE, caused by setting MTU [VPP-1876], [CSIT-1707].
[10] POSTPONED, avf-ip4base regression, CANDIDATE(S) range ge-18361..ge-24505, [CSIT-1699], [CSIT-1708].
[11] DONE, memif regression, CANDIDATE(S) confirmed ge-23801.