Freeing struct sk_buff (not freeing behavior)

user_4688800 0
Viewed 74

Writing ndo_start_xmit() NIC driver function assume that after data transmission I must call dev_kfree_skb() of consumed struct sk_buff and return NETDEV_TX_OK.

What should I expect if consumed struct skb will not be freed up during module lifetime?

Can Linux kernel free struct sk_buff after some timeout or in another way?

Are there difference of behavior between various versions of Linux kernel?

Linux kernel Documentation mentioned in the topic:

An ndo_start_xmit method must not modify the shared parts of a cloned SKB.

Do not forget that once you return NETDEV_TX_OK from your ndo_start_xmit method, it is your driver’s responsibility to free up the SKB and in some finite amount of time.

For example, this means that it is not allowed for your TX mitigation scheme to let TX packets “hang out” in the TX ring unreclaimed forever if no new TX packets are sent. This error can deadlock sockets waiting for send buffer room to be freed up.

If you return NETDEV_TX_BUSY from the ndo_start_xmit method, you must not keep any reference to that SKB and you must not attempt to free it up.

linux-device-driver linux-kernel
1 Answers

In my experience, drivers will typically follow some variant of this flow:

  1. In .ndo_start_xmit:
    1. Setup a DMA:able mapping to the skbs data
    2. Fill in a descriptor with a pointer to that data
    3. Let the hardware know that new descriptors are available for transmission
    4. Store a reference to the skb-to-be-transmitted in an internal list
  2. Some time later - either from a napi poll callback or in an IRQ handler, walk through all descriptors that have been marked as transmitted, and for each one:
    1. Fetch the corresponding skb from the internal list and free it.
    2. Tear down the DMA mapping.

In other words, the kernel does not provide any guardrails here. The responsibility lays solely on you. This is because the kernel has no way of knowing when your hardware is done with it.

To properly handle the case of module unloading, your driver must ensure that all tx rings are cleaned when the netdev is unregistered (i.e. performing step 2 above).

This basic division of responsibilities has not changed at all for as long as I've been doing Linux network drivers (10+ years).

My interpretation of the docs you've cited is:

Dear driver writer: You may be tempted to only enable a Tx IRQ on every 256:th descriptor as a performance optimization to batch your cleanup work - don't do that! Because if you do, I (the kernel) may incorrectly determine that I need to apply backpressure on the associated sockets if no other transmissions are taking place.

I.e. you may think "I don't mind having 255 outstanding skbs - I have plenty of RAM to spare". Well, consider this case:

  1. You're on a box with a single open socket with a send buffer size (wmem) of 128k
  2. The socket receives 128 1k-sized buffers and queues them for transmission
  3. The socket can't send any more data until the underlying skbs have been freed - but they won't be freed until another 128 are sent to the NIC
  4. Deadlock
user_7153630
Related