Is it possible to do byte rotations using AVX2/AVX-512 instructions in less than 5 instructions? Looking for answers in assembly code as I'm not familiar enough with intrinsics.
In AVX-512, direct rotations can be performed with a single instruction! Unfortunately WORD and BYTE data types are not covered. Shifts can be made down to WORD sizes, so that is a minimum of 3 instructions (2 shifts and 1 add/or/xor). The best I have come up with so far is a minimum of 5 instructions as listed below.
zmm0 is loaded with 64 different byte values
SL3 = memory location of 64 continuous bytes of f8h value (for masking)
SR5 = memory location of 64 continuous bytes of 07h value (for masking)
vpslld zmm1, zmm0, 3 ; shift DWORDS《 by 3 bits
vpandd zmm1, zmm1, SL3 ; clear low 3 bits, all bytes
vpsrl zmm2, zmm0, 5 ; shift DWORDS 》by 5 bits
vpandd zmm2, zmm2, SR5 ; clear high 5 bits, all bytes
vpord zmm1, zmm1, zmm2 ; combine left/right shifts for rotation result
Above is either a rotation left of 3 bits or a rotation right of 5 bits. And all 64 bytes yield correct results.
As I said, using WORD size (16 bits) a rotation is a minimum of 3 instructions. But due to needing 5 instructions for byte rotation in a zmm register, I only achieved a 50% performance increase going from 32 byte AVX2 implementation, to a 64 byte AVX-512 algorithm.