In the signature ({state: S, props: P, nextJsx}) => JSX.Element, S and P are not types, they are identifiers. In this context, the colon : does not mean a type annotation, it means rebinding the name of a property, like in the following example:
const obj = {foo: 23};
// bar is not a type annotation here
const {foo: bar} = obj;
// 23
console.log(bar);
I'm not sure what the usefulness of parameter rebinding would be in a type declaration, but for better or worse, that's what it means. So the error is because Merge<K, S> is not a valid identifier for rebinding; but in fact both your jsx2 and your jsx3 declarations have problems, it's just that you didn't get an error for the other one.
The solution is to put the types in a place where, contextually, they actually are types; i.e. in the parameter's type annotation.
type Test<P, S=any, K=any> = {
jsx1?: (state: Merge<K, S>, props: P, nextJsx?) => JSX.Element;
jsx2?: (params: {state: S, props: P, nextJsx}) => JSX.Element;
jsx3?: (params: {state: Merge<K, S>, props: P, nextJsx}) => JSX.Element;
}
Note the parameter is a single object named params, and its type now occurs in a type annotation context so it is not parsed as rebinding identifiers. Also, although there is a single named parameter params in the type declaration, it is still perfectly possible to do parameter destructuring (and rebinding, if you want to) when you actually implement these functions:
const test: Test<number, number, number> = {
jsx2: ({state, props, nextJsx}) => console.log(state, props, nextJsx),
jsx3: ({state, props, nextJsx}) => console.log(state, props, nextJsx),
};
Playground Link