Host stack test framework ========================= Overview -------- The goal of the Host stack test framework (**hs-test**) is to ease writing and running end-to-end tests for VPP's Host Stack. End-to-end tests often want multiple VPP instances, network namespaces, different types of interfaces and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient. For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology. `Ginkgo`_ forms the base framework upon which the *hs-test* is built and run. All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require. This allows us to run those mentioned groups in parallel, but not individual tests in parallel. Anatomy of a test case ---------------------- **Prerequisites**: * Install hs-test dependencies with ``make install-deps`` * Root privileges are required to run tests as it uses Linux ``ip`` command for configuring topology * Set the core pattern so it doesn't contain a pipe. Ubuntu uses ``Apport`` by default, which isn't supported, and therefore hs-test will not detect core dumps. (e.g. ``echo "/var/crash/core-%e-%p-%t" | sudo tee /proc/sys/kernel/core_pattern``) * Tests use *hs-test*'s own docker image, they are rebuilt automatically when needed, you can run ``make build[-debug]`` to do so or use ``FORCE_BUILD=true`` make parameter **Action flow when running a test case**: #. It starts with running ``make test``. Optional arguments are ``V`` (verbose), ``PERSIST`` (topology configuration isn't cleaned up after test run, use ``make cleanup-hst`` to clean up), TEST= to run a specific test and PARALLEL=[n-cpus]. If you want to run multiple specific tests, separate their names with a comma. Other arguments can be viewed by running ``make help``. #. ``make list-tests`` shows all tests. #. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary. #. The Ginkgo test framework runs each function that was registered manually using ``Register[SuiteName]Test()``. Each of these functions correspond to a suite. #. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following: #. Ginkgo compiles the spec, builds a spec tree #. ``Describe`` container nodes in suite\_\*.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus]) #. Suite is initialized. The topology is loaded and configured in this step #. Registered tests are run in generated ``It`` subject nodes #. Execute tear-down functions, which currently consists of stopping running containers and clean-up of test topology Adding a test case ------------------ This describes adding a new test case to an existing suite. For adding a new suite, please see `Modifying the framework`_ below. #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists #. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``infra/suite_*.go``) #. Implement test behaviour inside the test method. This typically includes the following: #. Import ``. "fd.io/hs-test/infra"`` and ``. "fd.io/hs-test/infra/infra_common"`` #. Retrieve a running container in which to run some action. Each suite has a struct called ``Containers`` #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``Vppctl`` method to access debug CLI #. Run arbitrary commands inside the containers with ``Exec`` method #. Run other external tool with one of the preexisting functions in the ``infra/utils.go`` or ``infra/infra_common/utils_common.go`` file. For example, use ``wget`` with ``StartWget`` function #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else #. Verify results of your tests using ``Assert`` methods provided by the test suite. #. Create an ``init()`` function and register the test using ``Register[SuiteName]Tests(testCaseFunction)`` **Example test case** Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other. This can be put in file ``test/hs-test/my_test.go`` and run with command ``make test TEST=MyTest``. To add a multi-worker test, register it to a multi-worker suite. The suite *cannot* have ``s.SetupTest()`` in the ``BeforeEach`` block. Set your desired core counts using ``s.CpusPerContainer`` and/or ``s.CpusPerVppContainer`` and run ``s.SetupTest()`` at the beginning of your test. :: package main import ( . "fd.io/hs-test/infra" ) func init(){ RegisterMySuiteTests(MyTest) } func MyMTTest(s *MySuite){ MyTest(s) } func MyTest(s *MySuite) { clientVpp := s.Containers.ClientVpp.VppInstance serverVethAddress := s.Interfaces.Server.Ip4AddressString() result := clientVpp.Vppctl("ping " + serverVethAddress) AssertNotNil(result) Log(result) } Filtering test cases -------------------- The framework allows us to filter test cases in a few different ways, using ``make test TEST=xyz SKIP=xyz``: * Suite name * File name * Test name * All of the above as long as they are ordered properly, e.g. ``make test TEST=VethsSuite.http_test.go.HeaderServerTest`` * Multiple tests/suites: ``make test TEST=HttpClient,LdpSuite`` All of the above also applies to ``SKIP`` **Names are case sensitive!** Names don't have to be complete, as long as they are last: This is valid and will run all tests in every ``http`` file (if there is more than one): * ``make test TEST=VethsSuite.http`` This is not valid: * ``make test TEST=Veths.http`` They can also be left out: * ``make test TEST=http_test.go`` will run every test in ``http_test.go`` * ``make test TEST=Nginx`` will run everything that has 'Nginx' in its name - suites, files and tests. * ``make test TEST=HeaderServerTest`` will only run the header server test Tests can also be filtered using labels. In our case, operators have to be escaped with ``\``: * ``make test LABEL=http\&\&staticserver`` * ``make test LABEL=\!veth`` See ``_ for more information. Parallel multi-worker test execution ------------------------------------ Multi-worker (MW) suites run serially by default. To opt in to NUMA-aware MW parallelism, use ``MW_PARALLEL=true`` together with ``PARALLEL=auto``:: make test PARALLEL=auto MW_PARALLEL=true For a full run, non-MW tests keep the process count selected by ``PARALLEL=auto``. MW tests then run in separate NUMA-aware phases: * The ``MWWide`` phase uses one Ginkgo worker per NUMA node. Add this label to CPU-heavy MW suites or tests that need the full per-worker CPU budget. * The narrow phase runs MW tests without the ``MWWide`` label. It may use multiple Ginkgo workers per NUMA node when enough CPUs are available. ``MW_WORKERS_PER_NUMA`` controls the narrow-phase worker count. Its default, ``auto``, derives a safe value from the smallest NUMA node and caps it at two. An explicit positive integer can be used for experiments. The option is valid only when ``MW_PARALLEL=true``. For non-MASQUE suites, Ginkgo labels select the phase. An MW test without the ``MWWide`` label runs in the narrow phase. If every MW test in a suite needs a full NUMA worker, pass ``MWWideLabels(...)`` to ``DescribeMWSuite``. If only some tests need it, build those tests' label lists with ``MWWideLabels(...)``. This helper adds ``MWWideLabel`` only when MW parallelism is enabled. MASQUE placement is selected by the registration function. Register the test with the function matching its protocol, for example:: RegisterMasqueMWTcpTests(MyConnectTcpMWTest) ``RegisterMasqueMWTcpTests``, ``RegisterMasqueMWUdpTests``, ``RegisterMasqueMWHttp3TcpTests``, and ``RegisterMasqueMWControlTests`` create independent wide Ordered suites. No separate test-name map needs to be updated. ``REPEAT=N`` runs the selected sequence once and then repeats it ``N`` more times, stopping at the first failed phase. A full run repeats the complete non-MW/wide/narrow sequence; ``LABEL=MW`` or ``TEST=MW`` repeats the wide/narrow sequence. For these split MW runs, hs-test writes every completed phase and attempt to ``summary/report.json`` and prints aggregate selected, executed, passed, failed, skipped, and elapsed-time statistics. With ``MW_PARALLEL=false``, repeat remains native Ginkgo behavior and its JSON report contains the final attempt. Modifying the framework ----------------------- **Adding a test suite** .. _test-convention: #. To add a new suite, create a new file in the ``infra/`` folder. Naming convention for the suite files is ``suite_[name].go``. #. Make a ``struct``, in the suite file, with at least ``HstSuite``, ``Interfaces`` and ``Containers`` structs as its members. HstSuite provides functionality that can be shared for all suites, like starting containers. ``Interfaces`` and ``Containers`` structs are used to provide simpler access to interfaces and containers respectively. ``s.GetInterfaceByName([name])`` or ``s.GetContainerByName([name])`` should only be used to initialize interface and container struct fields within ``SetupSuite``. #. Create a new map that will contain a file name where a test is located and test functions with a pointer to the suite's struct: ``var myTests = map[string][]func(s *MySuite){}`` :: var myTests = map[string][]func(s *MySuite){} type MySuite struct { HstSuite Interfaces struct { Server *NetInterface Client *NetInterface ... } Containers struct { ServerVpp *Container ClientVpp *Container ... } } #. Then create a new function that will add tests to that map: :: func RegisterMyTests(tests ...func(s *MySuite)) { myTests[getTestFilename()] = tests } #. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests. It's important here to call ``ConfigureNetworkTopology()`` method, pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder without the extension. In this example, *myTopology* corresponds to file ``test/hs-test/topo-network/myTopology.yaml`` This will ensure network topology, such as network interfaces and namespaces, will be created. Another important method to call is ``LoadContainerTopology()`` which will load containers and shared volumes used by the suite. This time the name passed to method corresponds to file in ``test/hs-test/topo-containers`` folder. Lastly, initialize ``Interfaces`` and ``Containers`` struct fields using ``s.GetInterfaceByName("interfaceName")`` and ``s.GetContainerByName("containerName")``. Use the names that are defined in ``.yaml`` files :: func (s *MySuite) SetupSuite() { s.HstSuite.SetupSuite() // Add custom setup code here s.ConfigureNetworkTopology("myNetworkTopology") s.LoadContainerTopology("myContainerTopology") s.Interfaces.Server = s.GetInterfaceByName("interfaceName") s.Containers.ServerVpp = s.GetContainerByName("containerName") ... } #. In suite file, implement ``SetupTest`` method which gets executed before each test. VPP configuration and startup is usually placed here. Containers are started by ``s.HstSuite.SetupTest()`` and CPUs are allocated during container creation. :: func (s *MySuite) SetupTest() { s.HstSuite.SetupTest() vpp, _ := s.Containers.Vpp.newVppInstance(s.Containers.ServerVpp.AllocatedCpus, sessionConfig) AssertNil(vpp.Start()) AssertNil(vpp.CreateTap(s.Interfaces.Server, false, 1), "failed to create tap interface") // or if using a veth interface: idx, err = vpp.createAfPacket(s.Interfaces.Server, false, WithNumRxQueues(numWorkers), WithNumTxQueues(numCpus)) AssertNil(err, fmt.Sprint(err)) AssertNotEqual(0, idx) } #. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node. Place them at the end of the suite file * Declare a suite struct variable before anything else * To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered`` * Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped. To override this behavior, decorate the container node with ``ContinueOnFailure`` :: var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() { var s MySuite BeforeAll(func() { s.SetupSuite() }) BeforeEach(func() { s.SetupTest() }) AfterAll(func() { s.TeardownSuite() }) AfterEach(func() { s.TeardownTest() }) for filename, tests := range myTests { for _, test := range tests { test := test pc := reflect.ValueOf(test).Pointer() funcValue := runtime.FuncForPC(pc) testName := filename + "/" + strings.Split(funcValue.Name(), ".")[2] It(testName, func(ctx SpecContext) { Log("[* TEST BEGIN]: " + testName) test(&s) }, SpecTimeout(TestTimeout)) } } }) #. Notice the loop - it will generate multiple ``It`` nodes, each running a different test. ``test := test`` is necessary, otherwise only the last test in a suite will run. For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\. #. ``testName`` contains the test name in the following format: ``[name]_test.go/MyTest``. #. To run certain tests solo, create a register function and a map that will only contain tests that have to run solo. Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node: :: var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() { ... It(testName, Label("SOLO"), func(ctx SpecContext) { Log("[* TEST BEGIN]: " + testName) test(&s) }, SpecTimeout(TestTimeout)) }) #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above **Adding a topology element** Topology configuration exists as ``yaml`` files in the ``test/hs-test/topo-network`` and ``test/hs-test/topo-containers`` folders. Processing of a network topology file for a particular test suite is started by the ``ConfigureNetworkTopology()`` method depending on which file's name is passed to it. Specified file is loaded and converted into internal data structures which represent various elements of the topology. Interfaces are then created and configured in the ``ginkgo`` container, if ``- configured-by-host: true`` is in the yaml topology file. Otherwise, VPP handles creating interfaces in ``SetupTest()``. Topologies with ``veth`` interfaces must always be configured in Linux in the ``ginkgo`` container - configuration process can be viewed in ``infra/netconfig.go - configure()``. These are currently supported types of network elements. * ``netns`` - network namespace * ``veth`` - veth network interface, optionally with target network namespace or IPv4 address * ``bridge`` - ethernet bridge to connect created interfaces, optionally with target network namespace * ``tap`` - tap network interface with IP address Similarly, container topology is started by ``LoadContainerTopology()``, configuration file is processed so that test suite retains map of defined containers and uses that to start them at the beginning of each test case and stop containers after the test finishes. Container configuration can specify also volumes which allow to share data between containers or between host system and containers. Supporting a new type of topology element requires adding code to recognize the new element type during loading. And adding code to set up the element in the host system with some Linux tool, such as *ip*. This should be implemented in ``netconfig.go`` for network and in ``container.go`` for containers and volumes. **Communicating between containers** When two VPP instances or other applications, each in its own Docker container, want to communicate there are typically two ways this can be done within *hs-test*. * Network interfaces. Containers are being created with ``--network=ginkgo`` options, so they are connected with interfaces created in the ``ginkgo`` container * Shared folders. Containers are being created with ``-v`` option to create shared `volumes`_ between host system and containers or just between containers Host system connects to VPP instances running in containers using a shared folder where binary API socket is accessible by both sides. **Adding an external tool** If an external program should be executed as part of a test case, it might be useful to wrap its execution in its own function. These types of functions are placed in the ``utils.go`` file. If the external program is not available by default in Docker image, add its installation to ``test/hs-test/Dockerfile.vpp`` in ``apt-get install`` command. Alternatively copy the executable from host system to the Docker image, similarly how the VPP executables and libraries are being copied. **Skipping tests** ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as: ``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``. However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test. **External dependencies** * Standalone programs ``wget``, ``iperf3`` - since these are downloaded when Docker image is made, they are reasonably up-to-date automatically. Full list of dependencies can be viewed in ``docker/Dockerfile.base`` * Programs in Docker images - ``envoyproxy/envoy-contrib`` and ``nginx`` * Non-standard Go libraries - see ``test/hs-test/go.mod`` Generally, these will be updated on a per-need basis, for example when a bug is discovered or a new version incompatibility issue occurs. Debugging a test ---------------- DRYRUN ^^^^^^ ``make test TEST=[name] DRYRUN=true`` will setup and start most of the containers, but won't run any tests or start VPP. VPP and interfaces will be configured automatically once you start VPP with the generated startup.conf file. GDB ^^^ It is possible to debug VPP by attaching ``gdb`` before test execution by adding ``DEBUG=true``: :: $ make test TEST=LDPreloadIperfVppTest DEBUG=true ... run following command in different terminal: docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)" Afterwards press CTRL+\ to continue If a test consists of more VPP instances then this is done for each of them. Utility methods ^^^^^^^^^^^^^^^ **Packet Capture** It is possible to use VPP pcap trace to capture received and sent packets. You just need to add ``EnablePcapTrace`` to ``SetupTest`` method in test suite and ``CollectPcapTrace`` to ``TeardownTest``. This way pcap trace is enabled on all interfaces and to capture maximum 10000 packets. Your pcap file will be located in the test execution directory. **Event Logger** ``clib_warning`` is a handy way to add debugging output, but in some cases it's not appropriate for per-packet use in data plane code. In this case VPP event logger is better option, for example you can enable it for TCP or session layer in build time. To collect traces when test ends you just need to add ``CollectEventLogs`` method to ``TeardownTest`` in the test suite. Your event logger file will be located in the test execution directory. To view events you can use :ref:`G2 graphical event viewer ` or ``convert_evt`` tool, located in ``src/scripts/host-stack/``, which convert event logs to human readable text. Memory leak testing ^^^^^^^^^^^^^^^^^^^ It is possible to use VPP memory traces to diagnose if and where memory leaks happen by comparing of two traces at different point in time. You can do it by test like following: :: func MemLeakTest(s *NoTopoSuite) { s.SkipUnlessLeakCheck() // test is excluded from usual test run vpp := s.Containers.Vpp.VppInstance /* do your configuration here */ vpp.Disconnect() // no goVPP less noise vpp.EnableMemoryTrace() // enable memory traces traces1, err := vpp.GetMemoryTrace() // get first sample AssertNil(err, fmt.Sprint(err)) vpp.Vppctl("test mem-leak") // execute some action traces2, err := vpp.GetMemoryTrace() // get second sample AssertNil(err, fmt.Sprint(err)) vpp.MemLeakCheck(traces1, traces2) // compare samples and generate report } To get your memory leak report run following command: :: $ make test-leak TEST=MemLeakTest ... NoTopoSuiteSolo mem_leak_test.go/MemLeakTest [SOLO] /home/matus/vpp/test/hs-test/infra/suite_no_topo.go:113 Report Entries >> SUMMARY: 112 byte(s) leaked in 1 allocation(s) - /home/matus/vpp/test/hs-test/infra/vppinstance.go:624 @ 07/19/24 15:53:33.539 leak of 112 byte(s) in 1 allocation(s) from: #0 clib_mem_heap_alloc_aligned + 0x31 #1 _vec_alloc_internal + 0x113 #2 _vec_validate + 0x81 #3 leak_memory_fn + 0x4f #4 0x7fc167815ac3 #5 0x7fc1678a7850 << Report Entries ------------------------------ .. _ginkgo: https://onsi.github.io/ginkgo/ .. _volumes: https://docs.docker.com/storage/volumes/