The signing key's algorithm 'SHA256withECDSA' does not equal a valid HmacSHA* algorithm name and cannot be used with HS512

TLDR: private key and secret key are different

There are two radically different kinds of digital cryptography used in the last half-century: traditional or symmetric, also called conventional, and public-key or asymmetric. In Java crypto, and thus in jjwt, the SecretKey[Spec] classes and their subclasses are used only for symmetric algorithms, including HMAC, and the PrivateKey[Spec] classes and their subclasses are used only for asymmetric algorithms, including ECDSA. See https://github.com/jwtk/jjwt#signing-key -- you must use a SecretKey of suitable size for HSnnn but a PrivateKey of both the correct algorithm (RSA or EC) and suitable size for RSnnn PSnnn or ESnnn. Because you gave jjwt a SecretKeySpec it tries to use the only signing algorithm it knows that is usable with a secret key, namely HMAC, but the type you specified is invalid for a secret key and thus for HMAC.

And your key is not PKCS8.

Your private key is in the OpenSSL-specific 'traditional' or 'legacy' format, which is the one first defined by SECG in SEC1 and later officialized (if that's a word) by RFC5915. Java crypto does not directly support the (several) OpenSSL traditional formats, only the industry-standard PKCS8 format published as RFC5208 and updated under a new name as RFC5958.

You say your filename has extension .p8 which suggests someone thinks it is, or should be, PKCS8, but in fact it's not. The command you asked about, openssl ec, will not convert to PKCS8, in fact the reverse: if you give it a PKCS8 format input it will produce traditional format output, exactly what you don't want. If you are able to fix the key before using it, there are two choices:

 openssl pkey -in trad_pem_file -out pkcs8_pem_file
 # only in 1.0.0 up, but that's been 11 years and pretty much everyone has it now

This produces PKCS8 format, but in PEM 'armor' which Java doesn't directly support, so to read it you must: remove the BEGIN and END lines; decode the base64 part to binary (byte[] in Java) ignoring the newlines, which can be done by removing them before decoding or by using j8+ java.util.Base64.getMimeDecoder() which ignores them; and finally put the result through a suitable KeyFactory:

 PrivateKey key = KeyFactory.getInstance("EC").generatePrivateKey(new PKCS8EncodedKeySpec(decodedbase64));
 # note KeyFactory is for asymmetric algorithms (and private and public keys) only
 # while SecretKeyFactory is for symmetric algorithms and secret keys,
 # but KeyGenerator is for symmetric algorithms and secret keys
 # while KeyPairGenerator is for asymmetric algorithsm and private/public keys
 # I think this is to keep you on your toes

The other (older) conversion in OpenSSL is:

 openssl pkcs8 -topk8 -nocrypt -in trad_pem_file -out pkcs8_file [-outform pem|der]

With default outform 'pem' this is the same as pkey above; with 'der' it does the PEM-decode steps for you and creates a binary file which could be read and used directly by Java:

 byte[] p8der = Files.readAllBytes(Paths.of("pkcs8_der_file"));
 PrivateKey key = KeyFactory.getInstance("EC").generatePrivate(new PKCS8EncodedKeySpec(p8der));

or you could embed it in your source by various means.

If you can't convert the key to pkcs8 and need your program to handle traditional format, that is possible but more complicated -- and has been asked and answered before, quite a few times. I will dig up some of them later when I have a little more time.

Thanks for dave_thompson_085 patient,I am solving the problem using this code:

public static String generateToken(AppleJwtHeader jwtHeader, AppleJwtPayload jwtPayload ) throws NoSuchAlgorithmException, InvalidKeySpecException, IOException {
        byte[] p8der = Files.readAllBytes(Path.of("/opt/apps/dolphin-post/AuthKey_YDKL424AF9.p8"));
        PKCS8EncodedKeySpec priPKCS8 = new PKCS8EncodedKeySpec(new org.apache.commons.codec.binary.Base64().decode(p8der));

        PrivateKey appleKey = KeyFactory.getInstance("EC").generatePrivate(priPKCS8);
        ObjectMapper objectMapper = new ObjectMapper();
        Map claims = objectMapper.convertValue(jwtPayload, Map.class);
        Map header = objectMapper.convertValue(jwtHeader,Map.class);
        String accessToken = Jwts.builder()
                .setClaims(claims)
                .setHeader(header)
                .signWith(appleKey)
                .compact();
        return accessToken;
    }

hopes help for others.