Test Environment

Environment Versioning

CSIT test environment versioning has been introduced to track modifications of the test environment.

Any benchmark anomalies (progressions, regressions) between releases of a DUT application (e.g. VPP, DPDK), are determined by testing it in the same test environment, to avoid test environment changes clouding the picture. To beter distinguish impact of test environment changes, we also execute tests without any SUT (just with TRex TG sending packets over a link looping back to TG).

A mirror approach is introduced to determine benchmarking anomalies due to the test environment change. This is achieved by testing the same DUT application version between releases of CSIT test system. This works under the assumption that the behaviour of the DUT is deterministic under the test conditions.

CSIT test environment versioning scheme ensures integrity of all the test system components, including their HW revisions, compiled SW code versions and SW source code, within a specific CSIT version. Components included in the CSIT environment versioning include:

  • HW Server hardware firmware and BIOS (motherboard, processsor, NIC(s), accelerator card(s)), tracked in CSIT branch in ./docs/lab/<server_platform_name>_hw_bios_cfg.md, e.g. Xeon Skylake servers.

  • Linux Server Linux OS version and configuration, tracked in CSIT Reports in SUT Settings and Pre-Test Server Calibration.

  • TRex TRex Traffic Generator version, drivers and configuration tracked in TG Settings.

  • CSIT CSIT framework code tracked in CSIT release branches.

Following is the list of CSIT versions to date:

To identify performance changes due to TRex code development between previous and current TRex version, both have been tested in CSIT environment of latest version and compared against each other. All substantial progressions and regressions have been marked up with RCA analysis. See Known Issues.

Physical Testbeds

FD.io CSIT performance tests are executed in physical testbeds hosted by LF for FD.io project. Physical testbed topology used:

  • 1-Node Topology: Consisting of TG with 1 NIC with 2 ports connected together - loopback connection.

SUT Settings - TRex

TG Settings - TRex

TG Version

TRex v2.88

DPDK Version

DPDK v21.02

TG Installation

T-Rex installation is managed via Ansible role.

TG Startup Configuration

$ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml
- version: 2
  c: 8
  limit_memory: 8192
  interfaces: ["${pci1}","${pci2}"]
  port_info:
      - dest_mac: [${dest_mac1}]
        src_mac: [${src_mac1}]
      - dest_mac: [${dest_mac2}]
        src_mac: [${src_mac2}]
  platform :
      master_thread_id: 0
      latency_thread_id: 9
      dual_if:
          - socket: 0
            threads: [1, 2, 3, 4, 5, 6, 7, 8]
EOF'

TG Startup Command (Stateless Mode)

$ sudo -E -S sh -c "cd '${trex_install_dir}/scripts/' && \
  nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \
  --mbuf-factor 32 > /tmp/trex.log 2>&1 &" > /dev/null

Also, Python client is now starting traffic with:

core_mask=STLClient.CORE_MASK_PIN

TG Startup Command (Stateful Mode)

$ sudo -E -S sh -c "cd '${trex_install_dir}/scripts/' && \
  nohup ./t-rex-64 -i --prefix $(hostname) --astf --hdrh --no-scapy-server \
  --mbuf-factor 32 > /tmp/trex.log 2>&1 &" > /dev/null

TG API Driver

TRex driver