Variable Byte Encoding method does not work in Swift

I am trying to read a mapsforge .map file as detailed here, but I can't seem to make the variable byte encoding part work. My implementation works as long as there isn't a second byte (so the number is small), but when the number is encoded into 2 or more bytes, my function cannot decode it properly. I was wondering if anyone could help me out what I am doing wrong.

Class implementation:

final class MapFileInputSerializer {
    private let _0x7full: UInt64 = 0x7f
    private let _0x3full: UInt64 = 0x3f
    private let _0x3fll: Int64 = 0x3f
    private let _0xffl: Int32 = 0xff
    
    private var fileHandle: FileHandle

    init(fileUrl: URL) throws {
        let fh = try! FileHandle(forReadingFrom: fileUrl)
        fileHandle = fh
    }

    [...]

This is when I know the number is an unsigned Integer:

func readVarUInt64() throws -> UInt64 {
    var value: UInt64 = 0
    var shift: UInt = 0

    while true {
        guard let byteData = try? fileHandle.read(upToCount: 1) else { fatalError() }
        let byteValue = UInt8(bigEndian: byteData.withUnsafeBytes { $0.load(as: UInt8.self) })

        value |= (UInt64(byteValue) & _0x7full) << shift
        shift += 7

        if byteValue & 0x80 == 0 { break }
        if shift > 63 { throw Error.tooLongUInt64 }
    }

    return value
}

And here is my func for signed integers:

func readVarInt64() -> Int64 {
    var value: Int64 = 0
    var shift: UInt = 0
    var byteValue: UInt8 = UInt8()

    while true {
        guard let byteData = try? fileHandle.read(upToCount: 1) else { fatalError() }
        byteValue = UInt8(bigEndian: byteData.withUnsafeBytes { $0.load(as: UInt8.self) })

        if byteValue & 0x80 == 0 { break } //0 at 128 means this is the last byte
        value |= Int64((UInt64(byteValue) & _0x7full) << shift)
        shift += 7
    }

    if byteValue & 0x40 != 0 { //0 at 64 means the number is positive, otherwise negative
        value = -(value | Int64(((UInt64(byteValue) & _0x3full) << shift)))
    } else {
        value |= Int64((UInt64(byteValue) & _0x3full) << shift)
    }

    return value
}

I cannot really determine where the error is because I am decoding a huge file and I just notice that suddenly I get an overflow error when I try to cast a UInt64 to Int64. Debugging suggestions can also help. :)