Beginner Assembly Language C - Subtracting to find what CPU added

call rel32 is relative to the end of the call instruction.

And the little-endian rel32 is a 2's complement integer, so 0xFFFFFFA7 it's a small negative number. 0xFFFFFFA7 - 2^32 = -89 (decimal), i.e. jump 89 bytes backwards.

You correctly decoded the 4-byte little-endian rel32 displacement into a binary integer, but didn't reinterpret it as signed 2's complement. (It's not subtracted, it's added. That's why it's negative).

e8 a7 ff ff ff callq rel32 ends at address 0x400546 (the start of the next instruction), so that will be RIP during its execution. New RIP after executing it will be
0x400546 - 89 = 0x4004ed, same as objdump -d printed.
objdump of course calculated that address the same way I did.

(Although objdump probably sign-extended the displacement to 64-bit before adding to a 64-bit code address. Working out that bit-pattern 0xFFFFFFA7 means -89 decimal as a 2's complement integer is basically like reading those 4 bytes into an int32_t and adding it to a uint64_t. (Intel's manual for https://www.felixcloutier.com/x86/call also describes the process as sign-extending the rel32 for binary addition, but that's just another way to express the same math in a more machine-friendly way. Other than the sign-extension, all of this works identically in any mode for direct relative call and jmp instructions. jmp rel8 uses an 8-bit 2's complement branch displacement.)

Semi-related: How does $ work in NASM, exactly? has an example of manually encoding a call to a given target address.

At a glance [and loosely] ...

You're using an x86 in 64 bit mode.

64 bit mode has a special addressing mode known as "RIP relative" addressing.

Edit: From Peter, the addressing mode is actually call rel32, rather than RIP relative, although the offset calculations will be the same.

The %rip register is the program counter. It changes on each instruction.

So, when using this mode, the offset is how far away the target address (e.g. apples) is from the address of the current instruction (from the address in %rip for the instruction).

Since you have two callq instructions (from your description, but not shown in the code), they each have a different address, so the offset to apples will be different.

This allows for "position independent code". It also allows an offset to be used, which is usually smaller than a full 64 bit absolute address. That's why the callq instruction (opcode + offset/address) is only 5 bytes (vs. 9 bytes) because the offset is a signed 32 bit quantity.

UPDATE:

I thought rip may be involved. In this particular instance, can you help me decipher how to find the rip% or kind of walk through this specific problem?

You could do: objdump --disassemble myprogram to get a disassembly and look at the disassembly. Or, you could do this with the debugger (e.g. gdb) using the disassemble command.

From your listing, the address of the callq is 0x400541 and [you mentioned that] apples is at 0x4004ed.

So, the offset from the start of the callq instruction is:

-84 FFFFFFFFFFFFFFAC

But, the instruction has an offset of:

0xFFFFFFFA7

(Remember that the disassembly just puts out the bytes, so we have to manually reverse the bytes because the offset is little-endian).

So, this means that the %rip value used is not the start of the instruction, but, rather the end of the instruction.

So, we have to adjust the offset by the length of the instruction [which is 5] to get 0xFFFFFFA7. That is, the %rip value [used by] the callq instruction is the address of the instruction + 5. In pseudo code, the calculation is:

offset = apples - (&callq + 5)