For the ca parameter for https.createServer(), what file formats are allowed to be passed in?

From NodeJS tls.createSecureContext:

Any string or Buffer can contain multiple PEM CAs concatenated together

Though, in general NodeJS uses PEM format.

How do I check that a CA bundle is actually being used?

You can use a certificate not signed by your CA, e.g. a self-signed certificate.

One point, possibly more subtle than you wanted: nodejs tls.createSecureContext internally calls OpenSSL PEM_read_bio_X509_AUX which actually accepts three PEM formats (or any sequence of those three formats, since nodejs loops). For two of them the base64/64cpl blob contains (exactly) an X.509 certificate, as respecified in rfc7468 sec 5, with either preferred label "CERTIFICATE" or deprecated label "X509 CERTIFICATE". In addition OpenSSL accepts a format of its own with label "TRUSTED CERTIFICATE" where the blob contains an X.509 certificate plus additional (ASN.1) data defined by OpenSSL; see e.g. the man page for d2i_X509_AUX online here. OpenSSL doesn't use this additional data for much, and of course nothing else uses it at all, so it's rare.

And to avoid confusion it might be worth noting that all OpenSSL PEM_read_ routines, including this one, skip any 'comment' data while searching for the PEM data, so actually a file/buffer that contains garbage, then a PEM cert, then more garbage, then another PEM cert, etc. will work the same as if it contained only the PEM certs.