Using the trace command

Skills to be Learned

  1. Setup a ‘trace’

  2. View a ‘trace’

  3. Clear a ‘trace’

  4. Verify using ping from host

  5. Ping from vpp

  6. Examine ARP table (neighbors)

  7. Examine ip fib

Basic Trace Commands

Show trace buffer [max COUNT].

vpp# show trace

Clear trace buffer and free memory.

vpp# clear trace

filter trace output - include NODE COUNT | exclude NODE COUNT | none.

vpp# trace filter <include NODE COUNT | exclude NODE COUNT | none>

Add Trace

vpp# trace add af-packet-input 10

The following list of nodes are supported under trace add:

  • af-packet-input

  • avf-input

  • bond-process

  • dpdk-crypto-input

  • dpdk-input

  • handoff-trace

  • ixge-input

  • memif-input

  • mrvl-pp2-input

  • netmap-input

  • p2p-ethernet-input

  • pg-input

  • punt-socket-rx

  • rdma-input

  • session-queue

  • tuntap-rx

  • vhost-user-input

  • virtio-input

  • vmxnet3-input

Ping from Host to VPP

vpp# q
$ ping -c 1 10.10.1.2
PING 10.10.1.2 (10.10.1.2) 56(84) bytes of data.
64 bytes from 10.10.1.2: icmp_seq=1 ttl=64 time=0.283 ms

--- 10.10.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.283/0.283/0.283/0.000 ms

Examine Trace of ping from host to VPP

$ sudo vppctl -s /run/vpp/cli-vpp1.sock
vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
Packet 1

00:17:04:099260: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e370 nsec 0x3af2736f vlan 0 vlan_tpid 0
00:17:04:099269: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:17:04:099285: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099290: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099296: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3f7c
    fragment id 0xe516, flags DONT_FRAGMENT
  ICMP echo_request checksum 0xc043
00:17:04:099300: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099301: ip4-icmp-echo-request
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099303: ip4-load-balance
fib 0 dpo-idx 13 flow hash: 0x00000000
ICMP: 10.10.1.2 -> 10.10.1.1
  tos 0x00, ttl 64, length 84, checksum 0x4437
  fragment id 0xe05b, flags DONT_FRAGMENT
ICMP echo_reply checksum 0xc843
00:17:04:099305: ip4-rewrite
tx_sw_if_index 1 dpo-idx 1 : ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800 flow hash: 0x00000000
00000000: e20f1e59ecf702fed975d5b4080045000054e05b4000400144370a0a01020a0a
00000020: 01010000c8437c92000170e3605b000000001c170f00000000001011
00:17:04:099307: host-vpp1out-output
host-vpp1out
IP4: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
ICMP: 10.10.1.2 -> 10.10.1.1
  tos 0x00, ttl 64, length 84, checksum 0x4437
  fragment id 0xe05b, flags DONT_FRAGMENT
ICMP echo_reply checksum 0xc843

Clear trace buffer

vpp# clear trace

Ping from VPP to Host

vpp# ping 10.10.1.1
64 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=.0789 ms
64 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=.0619 ms
64 bytes from 10.10.1.1: icmp_seq=3 ttl=64 time=.0519 ms
64 bytes from 10.10.1.1: icmp_seq=4 ttl=64 time=.0514 ms
64 bytes from 10.10.1.1: icmp_seq=5 ttl=64 time=.0526 ms

Statistics: 5 sent, 5 received, 0% packet loss

Examine Trace of ping from VPP to host

The output will demonstrate FD.io VPP’s trace of ping for all packets.

vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
Packet 1

00:17:04:099260: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e370 nsec 0x3af2736f vlan 0 vlan_tpid 0
00:17:04:099269: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:17:04:099285: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099290: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099296: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3f7c
    fragment id 0xe516, flags DONT_FRAGMENT
  ICMP echo_request checksum 0xc043
00:17:04:099300: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099301: ip4-icmp-echo-request
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f7c
  fragment id 0xe516, flags DONT_FRAGMENT
ICMP echo_request checksum 0xc043
00:17:04:099303: ip4-load-balance
fib 0 dpo-idx 13 flow hash: 0x00000000
ICMP: 10.10.1.2 -> 10.10.1.1
  tos 0x00, ttl 64, length 84, checksum 0x4437
  fragment id 0xe05b, flags DONT_FRAGMENT
ICMP echo_reply checksum 0xc843
00:17:04:099305: ip4-rewrite
tx_sw_if_index 1 dpo-idx 1 : ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800 flow hash: 0x00000000
00000000: e20f1e59ecf702fed975d5b4080045000054e05b4000400144370a0a01020a0a
00000020: 01010000c8437c92000170e3605b000000001c170f00000000001011
00:17:04:099307: host-vpp1out-output
host-vpp1out
IP4: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
ICMP: 10.10.1.2 -> 10.10.1.1
  tos 0x00, ttl 64, length 84, checksum 0x4437
  fragment id 0xe05b, flags DONT_FRAGMENT
ICMP echo_reply checksum 0xc843

Packet 2

00:17:09:113964: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 42 snaplen 42 mac 66 net 80
    sec 0x5b60e375 nsec 0x3b3bd57d vlan 0 vlan_tpid 0
00:17:09:113974: ethernet-input
ARP: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:17:09:113986: arp-input
request, type ethernet/IP4, address size 6/4
e2:0f:1e:59:ec:f7/10.10.1.1 -> 00:00:00:00:00:00/10.10.1.2
00:17:09:114003: host-vpp1out-output
host-vpp1out
ARP: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
reply, type ethernet/IP4, address size 6/4
02:fe:d9:75:d5:b4/10.10.1.2 -> e2:0f:1e:59:ec:f7/10.10.1.1

Packet 3

00:18:16:407079: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e3b9 nsec 0x90b7566 vlan 0 vlan_tpid 0
00:18:16:407085: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:16:407090: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3fe8
  fragment id 0x24ab
ICMP echo_reply checksum 0x37eb
00:18:16:407094: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3fe8
  fragment id 0x24ab
ICMP echo_reply checksum 0x37eb
00:18:16:407097: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3fe8
    fragment id 0x24ab
  ICMP echo_reply checksum 0x37eb
00:18:16:407101: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3fe8
  fragment id 0x24ab
ICMP echo_reply checksum 0x37eb
00:18:16:407104: ip4-icmp-echo-reply
ICMP echo id 7531 seq 1
00:18:16:407108: ip4-drop
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3fe8
    fragment id 0x24ab
  ICMP echo_reply checksum 0x37eb
00:18:16:407111: error-drop
ip4-icmp-input: unknown type

Packet 4

00:18:17:409084: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e3ba nsec 0x90b539f vlan 0 vlan_tpid 0
00:18:17:409088: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:17:409092: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f40
  fragment id 0x2553
ICMP echo_reply checksum 0xcc6d
00:18:17:409095: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f40
  fragment id 0x2553
ICMP echo_reply checksum 0xcc6d
00:18:17:409097: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3f40
    fragment id 0x2553
  ICMP echo_reply checksum 0xcc6d
00:18:17:409099: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3f40
  fragment id 0x2553
ICMP echo_reply checksum 0xcc6d
00:18:17:409101: ip4-icmp-echo-reply
ICMP echo id 7531 seq 2
00:18:17:409104: ip4-drop
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3f40
    fragment id 0x2553
  ICMP echo_reply checksum 0xcc6d
00:18:17:409104: error-drop
ip4-icmp-input: unknown type

Packet 5

00:18:18:409082: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e3bb nsec 0x8ecad24 vlan 0 vlan_tpid 0
00:18:18:409087: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:18:409091: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e66
  fragment id 0x262d
ICMP echo_reply checksum 0x8e59
00:18:18:409093: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e66
  fragment id 0x262d
ICMP echo_reply checksum 0x8e59
00:18:18:409096: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e66
    fragment id 0x262d
  ICMP echo_reply checksum 0x8e59
00:18:18:409098: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e66
  fragment id 0x262d
ICMP echo_reply checksum 0x8e59
00:18:18:409099: ip4-icmp-echo-reply
ICMP echo id 7531 seq 3
00:18:18:409102: ip4-drop
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e66
    fragment id 0x262d
  ICMP echo_reply checksum 0x8e59
00:18:18:409102: error-drop
ip4-icmp-input: unknown type

Packet 6

00:18:19:414750: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e3bc nsec 0x92450f2 vlan 0 vlan_tpid 0
00:18:19:414754: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:19:414757: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e52
  fragment id 0x2641
ICMP echo_reply checksum 0x9888
00:18:19:414760: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e52
  fragment id 0x2641
ICMP echo_reply checksum 0x9888
00:18:19:414762: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e52
    fragment id 0x2641
  ICMP echo_reply checksum 0x9888
00:18:19:414764: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e52
  fragment id 0x2641
ICMP echo_reply checksum 0x9888
00:18:19:414765: ip4-icmp-echo-reply
ICMP echo id 7531 seq 4
00:18:19:414768: ip4-drop
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e52
    fragment id 0x2641
  ICMP echo_reply checksum 0x9888
00:18:19:414769: error-drop
ip4-icmp-input: unknown type

Packet 7

00:18:20:418038: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 98 snaplen 98 mac 66 net 80
    sec 0x5b60e3bd nsec 0x937bcc2 vlan 0 vlan_tpid 0
00:18:20:418042: ethernet-input
IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:20:418045: ip4-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e47
  fragment id 0x264c
ICMP echo_reply checksum 0xc0e8
00:18:20:418048: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e47
  fragment id 0x264c
ICMP echo_reply checksum 0xc0e8
00:18:20:418049: ip4-local
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e47
    fragment id 0x264c
  ICMP echo_reply checksum 0xc0e8
00:18:20:418054: ip4-icmp-input
ICMP: 10.10.1.1 -> 10.10.1.2
  tos 0x00, ttl 64, length 84, checksum 0x3e47
  fragment id 0x264c
ICMP echo_reply checksum 0xc0e8
00:18:20:418054: ip4-icmp-echo-reply
ICMP echo id 7531 seq 5
00:18:20:418057: ip4-drop
  ICMP: 10.10.1.1 -> 10.10.1.2
    tos 0x00, ttl 64, length 84, checksum 0x3e47
    fragment id 0x264c
  ICMP echo_reply checksum 0xc0e8
00:18:20:418058: error-drop
ip4-icmp-input: unknown type

Packet 8

00:18:21:419208: af-packet-input
af_packet: hw_if_index 1 next-index 4
  tpacket2_hdr:
    status 0x20000001 len 42 snaplen 42 mac 66 net 80
    sec 0x5b60e3be nsec 0x92a9429 vlan 0 vlan_tpid 0
00:18:21:419876: ethernet-input
ARP: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
00:18:21:419881: arp-input
request, type ethernet/IP4, address size 6/4
e2:0f:1e:59:ec:f7/10.10.1.1 -> 00:00:00:00:00:00/10.10.1.2
00:18:21:419896: host-vpp1out-output
host-vpp1out
ARP: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
reply, type ethernet/IP4, address size 6/4
02:fe:d9:75:d5:b4/10.10.1.2 -> e2:0f:1e:59:ec:f7/10.10.1.1

After examining the trace, clear it again using vpp# clear trace.

Examine ARP table (neighbors)

vpp# show ip neighbors
Time           IP4       Flags      Ethernet              Interface
1101.5636    10.10.1.1      D    e2:0f:1e:59:ec:f7 host-vpp1out

Examine routing tables

vpp# show ip fib
  ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:[src:plugin-hi:2, src:adjacency:1, src:default-route:1, ]
0.0.0.0/0
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]]
  [0] [@0]: dpo-drop ip4
0.0.0.0/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
  [0] [@0]: dpo-drop ip4
10.10.1.0/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]
  [0] [@0]: dpo-drop ip4
10.10.1.1/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[5:420] via:[2:168]]
  [0] [@5]: ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800
10.10.1.0/24
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[0:0]]
  [0] [@4]: ipv4-glean: host-vpp1out: mtu:9000 ffffffffffff02fed975d5b40806
10.10.1.2/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[7:588]]
  [0] [@2]: dpo-receive: 10.10.1.2 on host-vpp1out
10.10.1.255/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]
  [0] [@0]: dpo-drop ip4
224.0.0.0/4
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
  [0] [@0]: dpo-drop ip4
240.0.0.0/4
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
  [0] [@0]: dpo-drop ip4
255.255.255.255/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
  [0] [@0]: dpo-drop ip4