When a XDP program is loaded in SKB mode, I am unable to remove any maps and some programs from code. I can't remove these same BPF entities using bpftool either. Here's my code for cleaning up the entities:
type BpfObjects struct {
CollectIpsProg *ebpf.Program `ebpf:"collect_ips_prog"`
HttpProg *ebpf.Program `ebpf:"http_prog"`
TlsProg *ebpf.Program `ebpf:"tls_prog"`
FlowRingBuffer *ebpf.Map `ebpf:"flow_ring_buf"`
JumpTable *ebpf.Map `ebpf:"jump_table"`
}
var objects BpfObjects
...
// None of these calls return an error
netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 2)
...
netlink.LinkSetXdpFd(link, -1)
objects.FlowRingBuffer.Unpin()
objects.JumpTable.Unpin()
objects.TlsProg.Unpin()
objects.HttpProg.Unpin()
objects.CollectIpsProg.Unpin()
objects.FlowRingBuffer.Close()
objects.JumpTable.Close()
objects.TlsProg.Close()
objects.HttpProg.Close()
objects.CollectIpsProg.Close()
On my VM that I use for development, I only need call do netlink.LinkSetXdpFd(link, -1) to clean up everything. However, when I run on my target server, I observe that program collect_ips_prog is still loaded and none of the maps are removed:
# bpftool prog
117: xdp name collect_ips_pro tag 6f70925233787bae gpl
loaded_at 2022-02-09T19:40:06+0000 uid 0
xlated 1112B jited 785B memlock 4096B map_ids 148,149,150
btf_id 90
# bpftool map
148: percpu_array name bpf_context_tab flags 0x0
key 4B value 276B max_entries 1 memlock 24576B
149: prog_array name jump_table flags 0x0
key 4B value 4B max_entries 2 memlock 4096B
owner_prog_type xdp owner jited
150: ringbuf name flow_ring_buf flags 0x0
key 0B value 0B max_entries 16777216 memlock 0B
collect_ips_prog is the only program I load to the interface using netlink.LinkSetXdpFdWithFlags. I populate the other two programs in jump_table in code. After the program exits, the jump_table map is empty:
# bpftool map dump name jump_table
Found 0 elements
The program is still there if I do bpftool net detach xdp dev eth0. There is no error/error code.
I do not have this issue when I load the XDP program in driver mode.
The server is running CentOS 7, and I replaced the kernel with 5.16.7-1.el7.elrepo.x86_64. My NIC is a Mellanox MT28800 Family [ConnectX-5 Ex].