Release Notes

Changes in CSIT-2005

  1. VPP PERFORMANCE TESTS

    • CSIT test environment is versioned, see Environment Versioning.

    • To identify performance changes due to VPP code changes from v20.01.0 to v20.05.0, both have been tested in CSIT environment ver. 4 and compared against each other. All substantial progressions has been marked up with RCA analysis. See Current vs Previous Release and Known Issues.

  2. TEST FRAMEWORK

    • CSIT PAPI support: Due to issues with PAPI performance, VAT is still used in CSIT for all VPP scale tests. See known issues below.

    • General Code Housekeeping: Ongoing RF keywords optimizations, removal of redundant RF keywords and aligning of suite/test setup/teardowns.

  3. PRESENTATION AND ANALYTICS LAYER

    • Graphs layout improvements: Improved performance graphs layout for better readibility and maintenance: test grouping, axis labels, descriptions, other informative decoration.

Known Issues

List of known issues in CSIT-2005 for VPP performance tests:

#

JiraID

Issue Description

1

CSIT-570

Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.

2

VPP-662

9000B packets not supported by NICs VIC1227 and VIC1387.

3

VPP-1677

9000B ip4 nat44: VPP crash + coredump. VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B).

4

CSIT-1591

All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much longer to program VPP). This needs to be addressed on the PAPI side.

VPP-1763

5

VPP-1675

IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded. Reason: chained buffers are not supported.

6

CSIT-1593

IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx.

7

CSIT-1679

2n-clx VPP ip4 tests with xxv710 and avf driver are sporadically failing.

Root Cause Analysis for Performance Changes

List of RCAs in CSIT-2005 for VPP performance changes:

#

JiraID

Issue Description

1

CSIT-1731

Confirm the cause for ip4scale -rnd regressions.

2

CSIT-1735

Identify cause of ethip6-ip6scale2m progression.

3

CSIT-1738

Identify cause for vppl2xc CSIT progressions.

4

CSIT-1739

Identify cause of ACL regressions.

5

CSIT-1740

Identify cause for avf-eth-l2xcbase CSIT progression.

6

CSIT-1741

Identify cause for vppl2xc VPP regressions.

7

CSIT-1742

Identify cause of ipsec CSIT regression.

8

CSIT-1744

Identify cause of memif VPP progression.

9

CSIT-1745

Verify cause of l2bdscale10kmaclrn VPP progression.

10

CSIT-1746

Identify cause for avf-dot1q-ip6base VPP progression.

11

CSIT-1747

Identify cause of ip4base-nat44 VPP progression.

12

CSIT-1748

Identify cause of aes128cbc-hmac512sha VPP progression.

13

CSIT-1749

Identify cause for l2bdbasemaclrn VPP progression in tests with dpdk app in VM.