在系統(tǒng)領域中，最具挑戰(zhàn)性的問題通常是組件之間的邊界定位。其中，virtio-net前后端的定界尤為困難。當網(wǎng)絡報文從內(nèi)核發(fā)送到virtio-net后端，或者從virtio-net后端發(fā)送到內(nèi)核時，這一路徑難以進行觀測。一些復雜的網(wǎng)絡抖動問題很可能是由于網(wǎng)卡隊列不正常工作引起的。為了解決這類問題，我們基于eBPF技術(shù)擴展了網(wǎng)卡隊列的可觀測能力，使得virtio網(wǎng)卡前后端的定界問題不再困擾。

virtio-net 前后端驅(qū)動簡介

virtio-net （后面稱為 virtio 網(wǎng)卡）通常由兩個組件組成：virtio driver（也稱為virtio前端）和virtio device（也稱為virtio后端）。virtio前端運行在客戶機的內(nèi)核中，而virtio后端可以由宿主機的內(nèi)核承擔。virtio網(wǎng)卡通常支持多隊列，包括發(fā)送隊列和接收隊列。每個隊列通過三個 ring 來實現(xiàn)，即avail ring、used ring和desc ring?，F(xiàn)在我們將重點介紹 virtio 網(wǎng)卡前端的報文發(fā)送和接收流程，以更好地理解整個工作流程。

virtio 網(wǎng)卡前端發(fā)送報文

virto網(wǎng)卡前端發(fā)送報文主要流程包括：

a.start_xmit：virtio網(wǎng)卡驅(qū)動的報文發(fā)送入口函數(shù)會首先清理已發(fā)送的報文，即通過調(diào)用free_old_xmit_skbs函數(shù)來釋放描述符中的報文，直到avail->idx等于used->idx為止；

b.xmit_skb：主要是為報文添加vnet_hdr頭部信息，并將skb以scatter-gather形式顯示，以記錄報文數(shù)據(jù)的地址和長度信息；

c.virtqueue_add_outbuf：進行DMA映射，將scatter-gather記錄的報文數(shù)據(jù)地址和長度信息添加到desc環(huán)中，并增加avail->idx的值；

d.virtqueue_notify：當發(fā)送隊列存在數(shù)據(jù)，則通知后端。

virtio 網(wǎng)卡前端接收報文

virito網(wǎng)阿卡前端接收報文主要流程包括：

a.網(wǎng)卡硬中斷：硬中斷會將napi加入到CPU的處理隊列，并啟用中斷抑制，以及觸發(fā)軟中斷；

b.net_rx_action：網(wǎng)絡軟中斷入口函數(shù)；

c.virtnet_poll：這個函數(shù)是virtio網(wǎng)卡的NAPI poll的回調(diào)函數(shù)。如果當前隊列是發(fā)送隊列，它將清理發(fā)送隊列，也就是執(zhí)行virtnet_poll_cleantx函數(shù)。如果當前隊列是接收隊列，它將進行報文的接收；

d.virtnet_receive：根據(jù)used->idx的值，從描述符環(huán)中讀取報文數(shù)據(jù)，并更新last_used_idx。內(nèi)核會為報文數(shù)據(jù)分配skb，并進入GRO流程，進行報文的合并；e.try_fill_recv：要給desc環(huán)添加空的內(nèi)存區(qū)域，并增加avail->idx的值，以確保接收隊列始終有可用的內(nèi)存；

f.virtqueue_napi_complete：當接收的報文數(shù)量少于預定的budget（一般為64）時，表示沒有更多的數(shù)據(jù)可以接收。這時，調(diào)用virtqueue_napi_complete來表示單次napi處理完畢。同時，通過virtqueue_enable_cb_prepare來關(guān)閉中斷抑制。

網(wǎng)卡隊列可觀測

經(jīng)過前面的分析，我們了解到virtio網(wǎng)卡隊列中的幾個重要參數(shù)，即avail->idx、used->idx和last_used_idx。使用這些參數(shù)，我們可以清晰地了解網(wǎng)卡隊列當前包含的報文數(shù)量，并進一步得到以下可觀測指標：

a.發(fā)送隊列報文數(shù)：表示尚未被virtio網(wǎng)卡后端發(fā)送的報文數(shù)量。計算方法是avail->idx - used->idx；

b.接收隊列報文數(shù)：表示尚未被virtio網(wǎng)卡前端接收的報文數(shù)量。計算方法是used->idx - last_used_idx；

c.網(wǎng)卡隊列的last_used_idx：表示virtio網(wǎng)卡后端處理報文的進度；

d.隊列飽和度：表示當前網(wǎng)卡隊列使用量，計算方法是隊列報文數(shù)/隊列長度。

工作原理

我們將可觀測的代碼集成在了rtrace的工具里，rtrace是龍蜥社區(qū)推出的系統(tǒng)工具集SysAK的一個網(wǎng)絡診斷分析工具，關(guān)于rtrace的具體原理，我們將在下回分析，eBPF 具體代碼請參考代碼：

https://gitee.com/anolis/sysak/blob/opensource_branch_sync/source/tools/detect/net/rtrace/src/bpf/virtio.bpf.c

virtio 網(wǎng)卡隊列指標采集的主要流程如下：

a.rtrace掛載eBPF采集程序到內(nèi)核dev_id_show和dev_port_show函數(shù)；

b.rtrace周期性讀取/sys/class/net/[interface]/dev_id和/sys/class/net/[interface]/dev_port兩個文件，其中dev_id文件用來表示采集發(fā)送隊列信息，dev_port文件用來表示采集接收隊列信息；

c.當讀取文件時，會觸發(fā)內(nèi)核執(zhí)行dev_id_show和dev_port_show兩個函數(shù)。由于已經(jīng)掛載了eBPF采集程序，內(nèi)核會先執(zhí)行eBPF采集程序；

d.eBPF采集程序通過解析dev_id_show和dev_port_show入?yún)truct net_device獲取網(wǎng)卡隊列vring，然后從vring中解析出avail idx、used idx、隊列長度和last_used_idx；

e.將數(shù)據(jù)發(fā)送給rtrace做進一步處理。

故障檢測

下面是rtrace采集的網(wǎng)卡隊列信息輸出。

我們可以看到0926的1號發(fā)送隊列的飽和度和last_used_idx分別是0.05%/3593，0928的1號發(fā)送隊列的飽和度和last_used_idx分別是0.07%/3593，可以看到發(fā)送隊列的飽和度在增加，但是last_used_idx在多個采集周期內(nèi)保持不變。因此，可以確定1號發(fā)送隊列出現(xiàn)了故障。

隨后我們修復了1號發(fā)送隊列故障，可以看見在0906的1號發(fā)送隊列飽和度和last_used_idx分別是0.00%/3599，隊列里面不再有駐留的報文，恢復了正常。

0924
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24430.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/880.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3010.00%/275
0925
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/890.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3030.00%/275
0926
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/910.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3050.00%/275
0927
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/930.00%/850.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3070.00%/275
0928
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/960.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3090.00%/275
0929
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/980.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3110.00%/275
0930
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/1000.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3130.00%/275
//...省略
0906
SendQueue0.00%/35990.00%/8560.00%/45110.00%/16020.00%/4650.00%/5100.00%/31400.00%/13520.00%/3860.00%/4200.00%/17160.00%/17660.00%/16190.00%/4480.00%/35780.00%/24510.00%/460.00%/940.00%/2120.00%/2310.00%/1480.00%/1490.00%/2260.00%/640.00%/1090.00%/850.00%/870.00%/560.00%/870.00%/1010.00%/1030.00%/52
RecvQueue0.00%/28070.00%/132990.00%/4770.00%/3690.00%/123780.00%/1400.00%/2230.00%/111200.00%/3550.00%/30320.00%/1420.00%/1800.00%/129800.00%/103630.00%/28250.00%/6520.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1700.00%/10570.00%/98200.00%/95860.00%/33740.00%/2300.00%/14140.00%/88000.00%/1180.00%/3270.00%/275

總結(jié)

在virtio網(wǎng)卡中，前端和后端之間通過共享的網(wǎng)卡隊列進行通信。為了更好地理解和觀測網(wǎng)卡隊列的狀態(tài)和性能指標，通過觀測avail idx、used idx、last_used_idx等指標，我們可以對virtio網(wǎng)卡的性能進行評估和優(yōu)化。同時，這些指標也為我們提供了對網(wǎng)卡隊列狀態(tài)的深入理解，有助于進行故障排查和性能調(diào)優(yōu)。