Efficient way to swap bytes in python

user_3479125 0
Viewed 16293

I have some bytearray with length of 2*n:

a1 a2 b1 b2 c1 c2

I need to switch bytes endian in each 2-byte word, and make:

a2 a1 b2 b1 c2 c1

Now I use next approach but it is very slow for my task:

converted = bytearray([])
for i in range(int(len(chunk)/2)):
   converted += bytearray([ chunk[i*2+1], chunk[i*2] ])

Is it possible to switch endian of bytearray by calling some system/libc function?

Ok, thanks to all, I timed some suggestions:

import timeit

test = [
"""
converted = bytearray([])
for i in range(int(len(chunk)/2)):
   converted += bytearray([ chunk[i*2+1], chunk[i*2] ])
""",
"""
for i in range(0, len(chunk), 2):
    chunk[i], chunk[i+1] = chunk[i+1], chunk[i]
""",
"""
byteswapped = bytearray([0]) * len(chunk)
byteswapped[0::2] = chunk[1::2]
byteswapped[1::2] = chunk[0::2]
""",
"""
chunk[0::2], chunk[1::2] = chunk[1::2], chunk[0::2]
"""
]

for t in test:
    print(timeit.timeit(t, setup='chunk = bytearray([1]*10)'))

and result is:

$ python ti.py
11.6219761372
2.61883187294
3.47194099426
1.66421198845

So in-pace slice assignment with a step of 2 now is fastest. Also thanks to Mr. F for detailed explaining but I not yet tried it because of numpy

swap endianness bytearray python
5 Answers

From some answers in How to byte-swap a 32-bit integer in python?

import struct

def htonl_slice(data):
    byteswapped = bytearray(4)
    byteswapped[0::4] = data[3::4]
    byteswapped[1::4] = data[2::4]
    byteswapped[2::4] = data[1::4]
    byteswapped[3::4] = data[0::4]
    return byteswapped

def htonl_slice_2(data):
    byteswapped = bytearray(len(data))
    byteswapped[0::4] = data[3::4]
    byteswapped[1::4] = data[2::4]
    byteswapped[2::4] = data[1::4]
    byteswapped[3::4] = data[0::4]
    return byteswapped

def htonl_struct(data):
    return struct.pack(">L", struct.unpack("<L", data)[0])

def swap32(data):
    return [struct.unpack("<I", struct.pack(">I", i))[0] for i in data]

def htonl_shift(x):
    return (((x << 24) & 0xFF000000) |
            ((x <<  8) & 0x00FF0000) |
            ((x >>  8) & 0x0000FF00) |
            ((x >> 24) & 0x000000FF))    

def test(self):

    data = [struct.pack('<L', i) for i in range(1000000)]

    start = time.time()
    for d in data:
        x = htonl_slice(d)
    end = time.time()
    print("htonl_slice %f" % (end - start))
    
    start = time.time()
    for d in data:
        x = htonl_struct(d)
    end = time.time()
    print("htonl_struct %f" % (end - start))

    data = [i for i in range(1000000)]
    start = time.time()
    for d in data:
        x = htonl_shift(d)
    end = time.time()
    print("htonl_shift %f" % (end - start))

    start = time.time()
    x = swap32(data)
    end = time.time()
    print("swap32 %f" % (end - start))

    data = bytearray()
    for i in range(1000000):
        data += struct.pack('<L', i)

    start = time.time()
    x = htonl_slice_2(data)
    end = time.time()
    print("htonl_slice_2 %f" % (end - start))

And the results are:

htonl_slice   3.041000
htonl_struct  0.626000
htonl_shift   0.864000
swap32        0.533000
htonl_slice_2 0.025000

I understand that htonl_shift works with ints instead of bytearrays.

It is interesting to see that swap32 is pretty fast, but for a 4 MB array htonl_slice_2 is by far the fastest.

user_62718890

I can repeat the result of in-place slice method from @user2357112 supports Monica. But if data size is ten time as big, it is twice slower compared with array.byteswap():

>>> timeit.timeit('chunk[0::2], chunk[1::2] = chunk[1::2], chunk[0::2]', 'chunk=bytearray(10000)')
12.54664899999625

>>> timeit.timeit('a=array.array("H",chunk);a.byteswap();chunk=a.tobytes()', 'import array;chunk=bytearray(10000)')
6.398380300001008
user_134721240
Related