Update as of 2021/11/06

It looks like Apple has been changing how the default focus system works and my previous solution is no longer working or required.

UIKeyCommand has a new wantsPriorityOverSystemBehavior: Bool property which needs to be set to true in order for our subclasses to receive certain types of commands, including the arrow key commands.

As of at least Xcode 13.1 and macOS 11.6, maybe eariler, we can now simply add the following to a UITableViewController subclass to replace the default focus behavior with custom keyboard navigation handing:

class TableViewController: UITableViewController {
    override var keyCommands: [UIKeyCommand]? {
        let upArrowCommand = UIKeyCommand(
            input: UIKeyCommand.inputUpArrow,
            modifierFlags: [],
            action: #selector(handleUpArrowKeyPress)
        )
        upArrowCommand.wantsPriorityOverSystemBehavior = true
        
        let downArrowCommand = UIKeyCommand(
            input: UIKeyCommand.inputDownArrow,
            modifierFlags: [],
            action: #selector(handleDownArrowKeyPress)
        )
        downArrowCommand.wantsPriorityOverSystemBehavior = true
        
        return  [
            upArrowCommand,
            downArrowCommand
        ]
    }

    @objc
    func handleUpArrowKeyPress () {
    }

    @objc
    func handleDownArrowKeyPress () {
    }
}

Previous answer (no longer working or required)

Catalyst automatically assigns UIKeyCommands for the up/down arrows to UITableView instances. This does not happen on iOS. You can see this in action by setting a break point in viewDidLoad() of a UITableViewController and inspecting tableView.keyCommands.

So I created a very simple UITableView subclass and disabled the default keyCommmands by returning nil:

class KeyCommandDisabledTableView: UITableView {
    override var keyCommands: [UIKeyCommand]? {
        return nil
    }
}

I then updated my UITableViewController subclass to use the new KeyCommandDisabledTableView subclass:

class MyTableViewController: UITableViewController {
    override func loadView() {
        self.view = KeyCommandDisabledTableView(
            frame: .zero, 
            style: .plain // or .grouped
        )
    }
}

Et voilà! The default arrow key handling is gone and my app's custom arrow key handling is now being called.

Here's another solution I received from Apple DTS. Just add this to the table view delegate:

func tableView(_ tableView: UITableView, canFocusRowAt indexPath: IndexPath) -> Bool {
    return false
}

This works in macOS 11.6 and 12.0. I don't have a 10.15 or 11.5 Mac to test with, so I'll keep my earlier resignFirstResponder solution, too.

I further noticed that the default arrow key navigation only begins after clicking a row in a table, so I guessed the table must be assuming the first responder role. I added this to my table's delegate class:

- (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath {
    #if TARGET_OS_MACCATALYST
    [tableView performSelector:@selector(resignFirstResponder) withObject:nil afterDelay:0.1];
    #endif
}

That fixed it! Now the default keyboard navigation turns off as soon as it turns on, and doesn't interfere with my app's custom keyboard navigation.

(It didn't work without the delay.)

iOS 14 / macOS 11 makes it much easier to disable this behavior thanks to UITableView and UICollectionView's selectionFollowsFocus property:

tableView.selectionFollowsFocus = false