GNU assembler on Windows: WriteFile returns ERROR_INVALID_HANDLE

user_5821910 0
Viewed 167

Until now, I've always written Assembly-code in GAS Syntax on Linux, so I wanted to try out how it'd be on Windows. The first goal was to print a single character onto stdout - which didn't work

Here's my code:

.intel_syntax noprefix

.extern GetStdHandle
.extern GetLastError
.extern WriteFile
.extern ExitProcess

.section .rodata
        .Lchar: .ascii "F"

.section .data
    .Lbytes_written: .long 0

.section .text

.global main

main:
        mov rcx, -11 //-11 = stdout
        call GetStdHandle
        mov rcx,  rax
        lea rdx, [rip + .Lchar]
        mov  r8, 1
        lea r9, [rip + .Lbytes_written]
        push 0
        call WriteFile
        call GetLastError //After this call, rax=0x6

        xor rcx, rcx
        call ExitProcess

After compiling it using gcc -g -o example.exe ./example.S (I use mingw64), nothing is printed. When stepping through the code with a debugger, I noticed that GetStdHandle doesn't fail (calling GetLastError directly after it returned 0), but WriteFile fails with 0x6, which is ERROR_INVALID_HANDLE.

So my question is: What could be the issue here? This question is probably very stupid to some, so I apologize in advance. Thank you!

Edit: This has gotten weirder than I thought. Take these two blocks of code

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0

.section .text

.global main

main:
    sub rsp, 8
    mov rcx, -11
    call GetStdHandle
    mov rcx, rax
    lea rdx, [rip + .Lchar]
    mov r8, 1
    lea r9, [rip + .Lbytes_written]
    push 0
    push 0
    call WriteFile
    call GetLastError
    add rsp, 16
    add rsp, 8
    ret

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0

.section .text

.global main

main:
    sub rsp, 24
    mov rcx, -11
    call GetStdHandle
    mov rcx, rax
    lea rdx, [rip + .Lchar]
    mov r8, 1
    lea r9, [rip + .Lbytes_written]
    push 0
    push 0
    call WriteFile
    call GetLastError
    add rsp, 16
    add rsp, 24
    ret

The only difference is that the second code block allocates 16B more on the Stack, which shouldn't be a problem, as all calls are still 16B-aligned. Yet for the second code block, the call to WriteFile does not work. Funnily enough, both function calls to GetStdHandle succeed and return the same value (which is 84 in my case) What could be the source of that?

x86-64 gnu-assembler assembly winapi windows
2 Answers

The main issue with my code was that I violated Windows' ABI, which included

  • not aligning the stack properly before a function call
  • not adding a 32B padding (shadow space) onto the stack, meaning that the 5th parameter was incorrectly provided

Here's a working version of this code:

.intel_syntax noprefix

.extern GetStdHandle
.extern GetLastError
.extern WriteFile

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0      # surprisingly just needs to be a dword, not qword in win64

.section .text

.global main

main:
    sub   rsp, 40                      # allocate shadow space + re-align the stack to RSP%16 == 0
    mov   rcx, -11                     # Magic number for stdout
    call  GetStdHandle
    mov   rcx, rax                     # hFile = return value
    lea   rdx, [rip + .Lchar]          # lpBuffer
    mov   r8, 1                        # 1 byte to write
    lea   r9, [rip + .Lbytes_written]  # output arg pointer
    mov   QWORD PTR [rsp + 32], 0      # lpOverlapped=NULL  in the stack space that was padding for the first call
    call  WriteFile
    add   rsp, 40
    ret

Thank you to Peter Cordes, Joshua and RbMm for helping me on this one!

user_58219100 
main:
    sub rsp, 8
    mov rcx, -11
    call GetStdHandle

Memory trashed at this point. The top four slots (32 bytes) in the stack need to be free. The correct line seems to be

    sub rsp, 40

Moving right along:

    call WriteFile
    call GetLastError

Mistake here. Check rax for error code. If no error, GetLastError returns the previous error whatever it was.

If something seems absolutely nuts, check code further along. As far as I can determine from trying to make this stuff work, Windows disassembles your code in some places and code that isn't reached can still crash if it would misalign the stack. I had a case where the crash was due to the difference between not commenting code and commenting code that was unconditionally jumped around.

user_147680
Related