Reversing algebraic equation with bitwise-XOR

user_8217448 0
Viewed 75

I'm trying to reverse an encryption scheme, but I seem to have fallen into a pit when it comes to reversal using algebra.

The encryption scheme is as follows for a single char (using registers and constants):

encrypted_char= (original_char XOR dl) + al
where:
eax = eax.previous * c1 +c2
edx = (eax >> c3)
eax.0 is a known seeded constant.

I want to solve this equation algebraically for original_char, but I'm running into a few problems, namely with order of operations for getting original char on it's own. Thinking about wraparound for negative numbers is also giving me a headache.

If anyone had any pointers for how to solve for the original_char, it would be appreciated. My first thoughts are to just subtract al and then xor with dl, but I'm starting to feel confused at this point.

reverse-engineering assembly encryption
1 Answers

I played with a toy example before posting and my answer is as follows:

bitwise xor has the same precedence as multiplication, I just flip it over. I already knew that XOR was the inverse of XOR, but I thought I should state it here. The resulting formula is as follows: (encrypted_char - al) XOR dl = al What goes into the larger registers doesn't need to be toyed with to arrive at the correct solution.

I will solve the wraparound using the modulus operation with the correct size for my variables.

Using the above methods I was able to reverse the code.

user_82174480
Related