Can no longer git clone large repos via HTTPS since installing Big Sur

This should be a comment (as it's not an answer) but I need some formatting and a lot of space here. The short version is that you need to find out why git index-pack is misbehaving or failing. (Fetching over a smart protocol normally retrieves a so-called thin pack, which git fetch needs to "fatten" using git index-pack --fix-thin.)

The "invalid index-pack output" error occurs if the output from git index-pack does not match what git fetch-pack expects. Here's the code involved:

char *index_pack_lockfile(int ip_out, int *is_well_formed)
{
    char packname[GIT_MAX_HEXSZ + 6];
    const int len = the_hash_algo->hexsz + 6;

    /*
     * The first thing we expect from index-pack's output
     * is "pack\t%40s\n" or "keep\t%40s\n" (46 bytes) where
     * %40s is the newly created pack SHA1 name.  In the "keep"
     * case, we need it to remove the corresponding .keep file
     * later on.  If we don't get that then tough luck with it.
     */
    if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') {
        const char *name;

        if (is_well_formed)
            *is_well_formed = 1;
        packname[len-1] = 0;
        if (skip_prefix(packname, "keep\t", &name))
            return xstrfmt("%s/pack/pack-%s.keep",
                       get_object_directory(), name);
        return NULL;
    }
    if (is_well_formed)
        *is_well_formed = 0;
    return NULL;
}

This is run from fetch-pack.c's get_pack function, which runs git index-pack with arguments based on a lot of variables. If you run your git clone with environment variable GIT_TRACE set to 1, you can observe Git running git index-pack. The call to index_pack_lockfile here only happens if do_keep is set, which is based on args->keep_pack initially, but can become set if the pack header's hdr_entries value equals or exceeds unpack_limit (see around line 859).

You can control the unpack_limit value using fetch.unpackLimit and/or transfer.unpackLimit. The default value is 100. You might be able to use these to work around some problem with index-pack, maybe—but index-pack should not be failing in whatever way it is failing. Note that if you want to force git fetch to use git unpack-objects instead, you must also disable object checking (fsck_objects).

It could be interesting to run git index-pack directly, too, on the data retrieved by git fetch. (Consider installing a shell script in place of the normal git index-pack, where the script prints its arguments, then uses kill -STOP on its own process group, so that you can inspect the temporary files.)

Solved!

TL;DR: ulimit was limiting max file size on my filesystem. Why this was never an issue in Catalina, who knows.

I had this in my shell startup script:

ulimit -n 65536 65536

That second parameter was the bad one. It was limiting max file size to 32MB. Strangely, I have this exact same setting on Catalina and it imposes no limitation whatsoever. Someone must've provided this setting and I just copied it without understanding the implications.

I ran ulimit -n unlimited unlimited and now all is well!

Interesting notes on ulimit changes from Catalina to Big Sur:

I ran this on my Catalina install:

$ ulimit -a                                                   
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              65536
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       11136
-n: file descriptors                65536
$ mkfile 500m whoa                                             
$ 

Look at that! No problem creating a 500MB file, clearly ignoring ulimit's 65536-block file size limit.

Yet on Big Sur, with the same ulimit settings:

$ mkfile 500m whoa                                                                                                                               
[1]    22267 file size limit exceeded  mkfile 500m whoa