How to handle device removal in Linux Kernel Driver?

user_1080389 0
Viewed 298

You have done it a thousand times: you unplug some USB equipment and any device associated with that USB equipment is removed by the driver. Any program that uses some previously opened file handle will get an error. Somehow most Linux drivers take care of that.

I am currently struggling to implement the same in a simple driver. My driver creates a character device. When the device is opened, I set the private_data member of struct file to the address of some management data that exists once per character device. That management data also includes a mutex that I use to synchronize operations like read, write, and ioctl.

The problem now arises when the USB equipment is unplugged. I must not free the memory where that management data lives. First, any currently running read, write, or ioctl should finish. Any such running call will likely also hold a lock on the mutex and will attempt to unlock it. So the memory where the mutex lives will be accessed.

Any read, write, or ioctl call subsequent to unplugging the equipment should fail, so every such call must read some variable telling whether the USB equipment is still plugged in or not. Again, that variable must live somewhere and the memory where it lives must stay allocated as long as there are open file handles.

Long story short, it seems to me that I must do some sort reference counting: The management data must stay allocated until all file handles have been closed. I could implement that myself, but I have the feeling that I would reinvent the wheel. Such a thing must already exist, I'm not the first to have this problem.

Does Linux internally keep track of the number of open file handles? Can I define a callback that is called when all file handles have been closed? Is that even a viable thing? What is the proper way to remove a character device from the system?

Global variables shall not be avoided, since any number of USB devices can be attached.

usb linux-device-driver linux-kernel
1 Answers

As suggested by 0andriy, I have used reference counting to track the number of open file handles. Per character device, I have added a new struct containing a mutex, a kref, the cdev struct, and a pointer to further data. I use kref_get to increase the reference counter when a new file handle is opened. Accordingly, I use kref_put when file handles are released or the USB equipment is unplugged.

Since all I/O operations (read, write, ioctl, etc.) use the mutex, I can safely change the pointer to NULL when the USB equipment is unplugged. The I/O operations then start reporting errors.

The kref reference counter only returns to zero when the USB equipment is unplugged and all file handles are closed. So then I can also free the memory of the new struct.

This seems to work. It was a surprise that the cdev struct is referenced by file handles. It seems to be needed as long as there are open file handles.

But I'm still surprised that I had to go down this route. It seems redundant to implement that in every driver.

UPDATE: As it turns out, doing reference counting yourself is dangerous. In particular counting the number of open file handles is wrong. There is a whole reference-count-based garbage collection system in Linux and you need to use it.

For example, Linux calls cdev_get when a process opens a character device and cdev_put when the process exists and the file handle is still open. Unfortunately, the call to cdev_put would occur after the file handle's release function. As we would free the memory after the last file handle has been released, we would end up freeing the underlying memory of the cdev and the mutex before cdev_put is called.

The solution is to assign a parent to the cdev via cdev_set_parent. The parent kobject will have its reference counter increased whenever cdev_get is called and decreased after cdev_put is called. The kobject then can have its own release function which frees any memory required by the cdev. Basically I replaced the kref in my struct with a kobject.

Lesson learned: don't do reference counting on your own. Use the existing hierarchy of kobjects, which is the kernel's way of doing garbage collection.

UPDATE2: It turns out, we reinvented the wheel again. The device struct includes a release hook, which is called when the internal kobject reached a reference count of zero. This is the mechanism typically used in the kernel to release resources associated with an device - including the device struct itself.

I was looking for a release hook in the cdev struct. There was none. But it turns out I should have looked one step up in the hierarchy for such a hook.

Long story short: Use cdev_device_add in combination with the device release hook. cdev_device_add will internally call cdev_set_parent, so the device becomes the parent of the cdev. Those are the mechanism other kernel drivers (e.g. evdev) use to release their resources.

user_10803890
Related