Typescript conditional types not able to correctly narrow type for nested properties

I encountered a strange issue. I'm wondering if this is genuine bug or just the lacklusterness of typescript's conditional types.

Consider this interface-

type EnumObj = { enm?: readonly (string | number)[] };

interface ITest {
    [key: string]: string | EnumObj;
}

Each property in ITest is either a regular string or an object with an optional enm property of type readonly array of strings/numbers. Emphasis on optional.

My goal is essentially this- For a given key in ITest, I need to check if the type of the value at said key is an EnumObj - if it is, I need to check if the type of the enm property within is not undefined (non nullable) and then I've to do some type deduction specifically using the value type of the enm property.

There's 2 levels of checks there, one to narrow the union, and one to check if enm is actually present.

To do this, I thought I could do-

// Nested conditional checking
type Foo<T extends ITest, K extends keyof T> = T[K] extends EnumObj
    ? T[K]['enm'] extends NonNullable<EnumObj['enm']>
        ? T[K]['enm'][number]
        // ^ type `number` cannot be used to index `T[K]['enm']`
        : never
    : never;

That's weird, it looks like the type isn't being narrowed properly. So I tried to separate this out. Instead of 2 conditional checks to narrow down the exact type - I'll just narrow down the enm property, given an EnumObj

// Directly checking T['enm'] - no nesting, works fine
type Bar<T extends EnumObj> = T['enm'] extends NonNullable<EnumObj['enm']> ? T['enm'][number] : never;

And this works fine, as expected. So I essentially have to call Bar inside Foo to do the full type deduction. But why does the conditional types freak out for nested narrowing?

Here's the whole thing on typescript playground

Another weird part is, even though Foo gives error. I can actually use it Foo<{ f: { enm: [1, 2, 3] } }> and it will get inferred correctly.

Note that I'm not looking for a solution per se. I'm guessing just calling Bar inside Foo is more or less the way to go here. Note that this is a dummy example but my real goal involves nested type checking in much the same way. I'm just curious as to why this is happening?