FD.io VPP  v20.09-64-g4f7b92f0a
Vector Packet Processing
src/vlibapi/api_doc.md
Go to the documentation of this file.
1 # Binary API support {#api_doc}
2 
3 VPP provides a binary API scheme to allow a wide variety of client codes to
4 program data-plane tables. As of this writing, there are hundreds of binary
5 APIs.
6 
7 Messages are defined in `*.api` files. Today, there are about 50 api files,
8 with more arriving as folks add programmable features. The API file compiler
9 sources reside in @ref src/tools/vppapigen.
10 
11 From @ref src/vnet/interface.api, here's a typical request/response message
12 definition:
13 
14 ```{.c}
15  autoreply define sw_interface_set_flags
16  {
17  u32 client_index;
18  u32 context;
19  u32 sw_if_index;
20  /* 1 = up, 0 = down */
21  u8 admin_up_down;
22  };
23 ```
24 
25 To a first approximation, the API compiler renders this definition into
26 `build-root/.../vpp/include/vnet/interface.api.h` as follows:
27 
28 ```{.c}
29  /****** Message ID / handler enum ******/
30  #ifdef vl_msg_id
31  vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS, vl_api_sw_interface_set_flags_t_handler)
32  vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, vl_api_sw_interface_set_flags_reply_t_handler)
33  #endif
34 
35  /****** Message names ******/
36  #ifdef vl_msg_name
37  vl_msg_name(vl_api_sw_interface_set_flags_t, 1)
38  vl_msg_name(vl_api_sw_interface_set_flags_reply_t, 1)
39  #endif
40 
41  /****** Message name, crc list ******/
42  #ifdef vl_msg_name_crc_list
43  #define foreach_vl_msg_name_crc_interface \
44  _(VL_API_SW_INTERFACE_SET_FLAGS, sw_interface_set_flags, f890584a) \
45  _(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply, dfbf3afa) \
46  #endif
47 
48  /****** Typedefs *****/
49  #ifdef vl_typedefs
50  typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags {
51  u16 _vl_msg_id;
52  u32 client_index;
53  u32 context;
54  u32 sw_if_index;
55  u8 admin_up_down;
56  }) vl_api_sw_interface_set_flags_t;
57 
58  typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags_reply {
59  u16 _vl_msg_id;
60  u32 context;
61  i32 retval;
62  }) vl_api_sw_interface_set_flags_reply_t;
63 
64  ...
65  #endif /* vl_typedefs */
66 ```
67 
68 To change the admin state of an interface, a binary api client sends a
69 @ref vl_api_sw_interface_set_flags_t to VPP, which will respond with a
70 @ref vl_api_sw_interface_set_flags_reply_t message.
71 
72 Multiple layers of software, transport types, and shared libraries
73 implement a variety of features:
74 
75 * API message allocation, tracing, pretty-printing, and replay.
76 * Message transport via global shared memory, pairwise/private shared
77  memory, and sockets.
78 * Barrier synchronization of worker threads across thread-unsafe
79  message handlers.
80 
81 Correctly-coded message handlers know nothing about the transport used to
82 deliver messages to/from VPP. It's reasonably straighforward to use multiple
83 API message transport types simultaneously.
84 
85 For historical reasons, binary api messages are (putatively) sent in network
86 byte order. As of this writing, we're seriously considering whether that
87 choice makes sense.
88 
89 
90 ## Message Allocation
91 
92 Since binary API messages are always processed in order, we allocate messages
93 using a ring allocator whenever possible. This scheme is extremely fast when
94 compared with a traditional memory allocator, and doesn't cause heap
95 fragmentation. See
96 @ref src/vlibmemory/memory_shared.c @ref vl_msg_api_alloc_internal().
97 
98 Regardless of transport, binary api messages always follow a @ref msgbuf_t
99 header:
100 
101 ```{.c}
102  typedef struct msgbuf_
103  {
104  unix_shared_memory_queue_t *q;
105  u32 data_len;
106  u32 gc_mark_timestamp;
107  u8 data[0];
108  } msgbuf_t;
109 ```
110 
111 This structure makes it easy to trace messages without having to
112 decode them - simply save data_len bytes - and allows
113 @ref vl_msg_api_free() to rapidly dispose of message buffers:
114 
115 ```{.c}
116  void
117  vl_msg_api_free (void *a)
118  {
119  msgbuf_t *rv;
120  api_main_t *am = &api_main;
121 
122  rv = (msgbuf_t *) (((u8 *) a) - offsetof (msgbuf_t, data));
123 
124  /*
125  * Here's the beauty of the scheme. Only one proc/thread has
126  * control of a given message buffer. To free a buffer, we just
127  * clear the queue field, and leave. No locks, no hits, no errors...
128  */
129  if (rv->q)
130  {
131  rv->q = 0;
132  rv->gc_mark_timestamp = 0;
133  return;
134  }
135  <snip>
136  }
137 ```
138 
139 ## Message Tracing and Replay
140 
141 It's extremely important that VPP can capture and replay sizeable binary API
142 traces. System-level issues involving hundreds of thousands of API
143 transactions can be re-run in a second or less. Partial replay allows one to
144 binary-search for the point where the wheels fall off. One can add scaffolding
145 to the data plane, to trigger when complex conditions obtain.
146 
147 With binary API trace, print, and replay, system-level bug reports of the form
148 "after 300,000 API transactions, the VPP data-plane stopped forwarding
149 traffic, FIX IT!" can be solved offline.
150 
151 More often than not, one discovers that a control-plane client
152 misprograms the data plane after a long time or under complex
153 circumstances. Without direct evidence, "it's a data-plane problem!"
154 
155 See @ref src/vlibmemory/memory_vlib.c @ref vl_msg_api_process_file(),
156 and @ref src/vlibapi/api_shared.c. See also the debug CLI command "api trace"
157 
158 ## Client connection details
159 
160 Establishing a binary API connection to VPP from a C-language client
161 is easy:
162 
163 ```{.c}
164  int
165  connect_to_vpe (char *client_name, int client_message_queue_length)
166  {
167  vat_main_t *vam = &vat_main;
168  api_main_t *am = &api_main;
169 
170  if (vl_client_connect_to_vlib ("/vpe-api", client_name,
171  client_message_queue_length) < 0)
172  return -1;
173 
174  /* Memorize vpp's binary API message input queue address */
175  vam->vl_input_queue = am->shmem_hdr->vl_input_queue;
176  /* And our client index */
177  vam->my_client_index = am->my_client_index;
178  return 0;
179  }
180 ```
181 
182 32 is a typical value for client_message_queue_length. VPP cannot
183 block when it needs to send an API message to a binary API client, and
184 the VPP-side binary API message handlers are very fast. When sending
185 asynchronous messages, make sure to scrape the binary API rx ring with
186 some enthusiasm.
187 
188 ### binary API message RX pthread
189 
190 Calling @ref vl_client_connect_to_vlib spins up a binary API message RX
191 pthread:
192 
193 ```{.c}
194  static void *
195  rx_thread_fn (void *arg)
196  {
197  unix_shared_memory_queue_t *q;
198  memory_client_main_t *mm = &memory_client_main;
199  api_main_t *am = &api_main;
200 
201  q = am->vl_input_queue;
202 
203  /* So we can make the rx thread terminate cleanly */
204  if (setjmp (mm->rx_thread_jmpbuf) == 0)
205  {
206  mm->rx_thread_jmpbuf_valid = 1;
207  while (1)
208  {
209  vl_msg_api_queue_handler (q);
210  }
211  }
212  pthread_exit (0);
213  }
214 ```
215 
216 To handle the binary API message queue yourself, use
217 @ref vl_client_connect_to_vlib_no_rx_pthread.
218 
219 In turn, vl_msg_api_queue_handler(...) uses mutex/condvar signalling
220 to wake up, process VPP -> client traffic, then sleep. VPP supplies a
221 condvar broadcast when the VPP -> client API message queue transitions
222 from empty to nonempty.
223 
224 VPP checks its own binary API input queue at a very high rate. VPP
225 invokes message handlers in "process" context [aka cooperative
226 multitasking thread context] at a variable rate, depending on
227 data-plane packet processing requirements.
228 
229 ## Client disconnection details
230 
231 To disconnect from VPP, call @ref vl_client_disconnect_from_vlib.
232 Please arrange to call this function if the client application
233 terminates abnormally. VPP makes every effort to hold a decent funeral
234 for dead clients, but VPP can't guarantee to free leaked memory in the
235 shared binary API segment.
236 
237 ## Sending binary API messages to VPP
238 
239 The point of the exercise is to send binary API messages to VPP, and
240 to receive replies from VPP. Many VPP binary APIs comprise a client
241 request message, and a simple status reply. For example, to
242 set the admin status of an interface, one codes:
243 
244 ```{.c}
245  vl_api_sw_interface_set_flags_t *mp;
246 
247  mp = vl_msg_api_alloc (sizeof (*mp));
248  memset (mp, 0, sizeof (*mp));
249  mp->_vl_msg_id = clib_host_to_net_u16 (VL_API_SW_INTERFACE_SET_FLAGS);
250  mp->client_index = api_main.my_client_index;
251  mp->sw_if_index = clib_host_to_net_u32 (<interface-sw-if-index>);
252  vl_msg_api_send (api_main.shmem_hdr->vl_input_queue, (u8 *)mp);
253 ```
254 
255 Key points:
256 
257 * Use @ref vl_msg_api_alloc to allocate message buffers
258 
259 * Allocated message buffers are not initialized, and must be presumed
260  to contain trash.
261 
262 * Don't forget to set the _vl_msg_id field!
263 
264 * As of this writing, binary API message IDs and data are sent in
265  network byte order
266 
267 * The client-library global data structure @ref api_main keeps track
268  of sufficient pointers and handles used to communicate with VPP
269 
270 ## Receiving binary API messages from VPP
271 
272 Unless you've made other arrangements (see @ref
273 vl_client_connect_to_vlib_no_rx_pthread), *messages are received on a
274 separate rx pthread*. Synchronization with the client application main
275 thread is the responsibility of the application!
276 
277 Set up message handlers about as follows:
278 
279 ```{.c}
280  #define vl_typedefs /* define message structures */
281  #include <vpp/api/vpe_all_api_h.h>
282  #undef vl_typedefs
283 
284  /* declare message handlers for each api */
285 
286  #define vl_endianfun /* define message structures */
287  #include <vpp/api/vpe_all_api_h.h>
288  #undef vl_endianfun
289 
290  /* instantiate all the print functions we know about */
291  #define vl_print(handle, ...)
292  #define vl_printfun
293  #include <vpp/api/vpe_all_api_h.h>
294  #undef vl_printfun
295 
296  /* Define a list of all message that the client handles */
297  #define foreach_vpe_api_reply_msg \
298  _(SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply)
299 
300  static clib_error_t *
301  my_api_hookup (vlib_main_t * vm)
302  {
303  api_main_t *am = &api_main;
304 
305  #define _(N,n) \
306  vl_msg_api_set_handlers(VL_API_##N, #n, \
307  vl_api_##n##_t_handler, \
308  vl_noop_handler, \
309  vl_api_##n##_t_endian, \
310  vl_api_##n##_t_print, \
311  sizeof(vl_api_##n##_t), 1);
312  foreach_vpe_api_msg;
313  #undef _
314 
315  return 0;
316  }
317 ```
318 
319 The key API used to establish message handlers is @ref
320 vl_msg_api_set_handlers , which sets values in multiple parallel
321 vectors in the @ref api_main_t structure. As of this writing: not all
322 vector element values can be set through the API. You'll see sporadic
323 API message registrations followed by minor adjustments of this form:
324 
325 ```{.c}
326  /*
327  * Thread-safe API messages
328  */
329  am->is_mp_safe[VL_API_IP_ADD_DEL_ROUTE] = 1;
330  am->is_mp_safe[VL_API_GET_NODE_GRAPH] = 1;
331 ```
332 
333 
334 
335 
336 
337 
338 
339 
340 
341 
342 
343 
344 
345 
346 
347 
348 
349 
350 
351 
352