Function result type derived from param without casting

In the following code:

function fourtyTwo<T extends boolean>(num: T): T extends true ? number : string {
    if (num) {
        return 42;
    } else {
        return 'fourty-two';
    }
}

I get 2 instances of the same error:

1. Type '42' is not assignable to type 'T extends true ? number : string'.ts(2322)
2. Type '"fourty-two"' is not assignable to type 'T extends true ? number : string'.ts(2322)

Is there a way to get this working without casting the return type?