How to properly validate OpenIddict JWT access_token in API?

I'm trying to implement an identity server based on OpenIddict. The use case we have is one large javascript application that needs to authenticate users to multiple back-end APIs. The javascript application gets a token from a dedicated OpenIddict server using the password flow. The token is then validated by the various APIs that are called by the front end.

I've implemented the server using ASP.NET Identity and EF, and can successfully retrieve a valid token. Our APIs are running in AWS Lambda, so we can't (or don't want to) use the standard .AddDataProtection methods. We use a self-signed certificate stored in S3 to generate the tokens and validate them.

Then problem is that when I send the access_token to the back-end APIs, they are not able to validate the token and provide access. I know the tokens are valid because I can manually decrypt them in Linqpad using a JwtSecurityTokenHandler and the self-signed certificate.

Here is my server configuration:

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();

            services.AddIdentity<ApplicationUser, IdentityRole>(options =>
            {
                options.ClaimsIdentity.UserNameClaimType = Claims.Name;
                options.ClaimsIdentity.UserIdClaimType = Claims.Subject;
                options.ClaimsIdentity.RoleClaimType = Claims.Role;
            })
            .AddEntityFrameworkStores<ApplicationDbContext>()
            .AddDefaultTokenProviders();

            services.AddOpenIddict()
            .AddCore(options =>
            {
                options.UseEntityFrameworkCore().UseDbContext<ApplicationDbContext>();
            })
            .AddServer(options =>
            {
                options.SetTokenEndpointUris("/connect/token");
                options.AllowPasswordFlow();
                options.AcceptAnonymousClients();
                options.AddEncryptionCertificate(MyCustomCertificate);
                options.AddSigningCertificate(MyOtherCustomCertificate);
                options.UseAspNetCore().EnableTokenEndpointPassthrough();
            });

            services.AddDbContext<ApplicationDbContext>(options =>
            {
                options.UseSqlServer(connectionString);
                options.UseOpenIddict();
            });

        }

I've tried two different confiugrations in the API, and neither of them work. Option 1, preferred, using the built-in OpenIddict token validation:

        public void ConfigureServices(IServiceCollection services)
        {

            services.AddControllers();

            services.AddOpenIddict().AddValidation(
                options =>
                {
                    options.SetTokenValidationParameters(config =>
                    {
                        config.ValidateAudience = false; //just to make sure it's not a typo causing the problem
                        config.ValidateIssuer = false; //just to make sure it's not a typo causing the problem
                        config.TokenDecryptionKey = new X509SecurityKey(MyCustomCertificate);
                        config.IssuerSigningKey = new X509SecurityKey(MyOtherCustomCertificate);
                    });

                    options.UseAspNetCore();
                });

        }

This invariably results in {"error":"invalid_token","error_description":"The specified token is not valid."}, even though I can manually decrypt and validate the token. I've tried setting the default log level to Trace, but nothing else shows up in the log that might explain where the problem is.

Option 2, using built-in .NET token validation:

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                .AddJwtBearer(options =>
                {
                    options.Authority = "https://openiddictserver";
                    options.Audience = "clientid"; 
                    options.RequireHttpsMetadata = false;
                    options.IncludeErrorDetails = true; 
                    options.TokenValidationParameters = new TokenValidationParameters()
                    {
                        NameClaimType = OpenIdConnectConstants.Claims.Subject,
                        RoleClaimType = OpenIdConnectConstants.Claims.Role,
                        ValidateAudience = false, //just to make sure it's not a typo problem
                        ValidateIssuer = false, //just to make sure it's not a typo problem
                        TokenDecryptionKey = new X509SecurityKey(MyCustomCertificate),
                        IssuerSigningKey = new X509SecurityKey(MyOtherCustomCertificate)
                    };
                });

Option 2 just results in a 401 with no helpful info whatsoever.

Some more context:

• I can validate the signed id_token just fine. It's only when trying to use the encrypted access_token that I run into trouble
• I'm calling app.UseAuthentication() in the correct place and the [Authorize(AuthenticationScheme="something")] attributes are correct.
• I'm using the OpenIddict.AspNetCore package version 3.0.0 from the dedicated openiddict nuget feed.

I'm fairly certain it just comes down to not being able to decrypt the access_token in the proper way in the API.