concurrent I/O - buffers corruption, block device driver

Admin 1
Viewed 225

I developing block layered device driver. So, I intercept WRITE request and encrypt data, and decrypt data in the end_bio() routine (during processing and READ request). So all works fine in single stream. But I getting buffers content corruption if have tried to performs I/O from two and more processes simultaneously. I have not any local storage for buffers.

Do I'm need to count a BIO merging in my driver?

Is the Linux I/O subsystem have some requirements related to the a number of concurrent I/O request?

Is there some tips and tricks related stack using or compilation?

This is under kernel 4.15.

At the time I use next constriction to run over disk sectors:

    /*
     * A portion of the bio_copy_data() ...
     */
    for (vcnt = 0, src_iter = src->bi_iter; ; vcnt++)
        {
        if ( !src_iter.bi_size)
            {
            if ( !(src = src->bi_next) )
                break;

            src_iter = src->bi_iter;
            }

        src_bv = bio_iter_iovec(src, src_iter);

        src_p = bv_page = kmap_atomic(src_bv.bv_page);
        src_p += src_bv.bv_offset;

        nlbn    = src_bv.bv_len512;
        for ( ; nlbn--; lbn++ , src_p += 512 )
                {
                {
                /* Simulate a processing of data in the I/O buffer */
               char *srcp = src_p, *dstp = src_p;
               int  count = DUDRV$K_SECTORSZ;

               while ( count--)
                {
                *(dstp++) = ~ (*(srcp++));
                }

                }
                }
        kunmap_atomic(bv_page);
        **bio_advance_iter**(src, &src_iter, src_bv.bv_len);
        }

Is this correct ? Or I'm need to use something like **bio_for_each_segment(bvl, bio, iter) ** ?

linux-kernel linux c
2 Answers

Have you considered the use of vmap with global synchronization, instead?

The use of kmap_atomic has some restrictions:

Since the mapping is restricted to the CPU that issued it, it performs well, but the issuing task is therefore required to stay on that CPU until it has finished, lest some other task displace its mappings.

kmap_atomic() may also be used by interrupt contexts, since it is does not sleep and the caller may not sleep until after kunmap_atomic() is called.

Reference: https://www.kernel.org/doc/Documentation/vm/highmem.txt

user_101695300

The root of the problem is a "feature" of the Block I/O methods. In particularly (see description at Linex site reference )

** Biovecs can be shared between multiple bios - a bvec iter can represent an arbitrary range of an existing biovec, both starting and ending midway through biovecs. This is what enables efficient splitting of arbitrary bios. Note that this means we only use bi_size to determine when we've reached the end of a bio, not bi_vcnt - and the bio_iovec() macro takes bi_size into account when constructing biovecs.*

So, in my case this is a cause of the buffer with disk sector overrun.

The trick is set REQ_NOMERGE_FLAGS in the .bi_opf before sending BIO to backed device driver.

The second reason is the non-actual .bi_iter is returned by backed device driver. So, we need to save it (before submiting BIO request to backend) and restore it in the our "bio_endio()" routine.

user_40182320
Related