My function can index documents in single and bulk to my AWS Elasticsearch from a local Jupyter notebook, but when I deploy to Lambda it keeps returning this error:
"errorMessage": "AuthorizationException(403, 'security_exception', 'no permissions for
[indices:data/write/bulk] and User [name=arn:aws:iam::xxxxxxxxxxxx:role/MyLambdaRole,
backend_roles=[arn:aws:iam::xxxxxxxxxxxx:role/MyLambdaRole], requestedTenant=null]')"
My Elasticsearch domain (v7.7) is configured as such:
Fine-grained access control: Enabled
Master user type: Internal user database
SAML authentication: Disabled
Amazon Cognito for authentication: Disabled
Require HTTPS: Enabled
Encryption at rest: Enabled
KMS master keyarn:aws:kms:us-east-1:xxxxxxxxxxxxx:key/<aws/es key>
Node-to-node encryption: Enabled
The domain's access policy contains:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "*"
}
]
}
The IAM policy for MyLambdaRole contains:
...
{
"Action": [
"es:*"
],
"Resource": [
"*"
],
"Effect": "Allow"
}
]
In Kibana I have mapped both my AWS admin IAM user and MyLambdaRole under Security -> Role Mappings -> all_access. I have tried different combinations of adding them to Backend roles and also adding them to security_manager.
The Lambda uses AWS Signature v4 authentication and the elasticsearch client is version 7.7.0:
import boto3
from elasticsearch import Elasticsearch, RequestsHttpConnection, helpers
from requests_aws4auth import AWS4Auth
session = boto3.Session()
credentials = session.get_credentials().get_frozen_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, \
session.region_name, 'es', session_token=credentials.token)
host = 'search-es-domain.us-east-1.es.amazonaws.com'
es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
# Single indexing call
document = { my data }
es.index(index="my_index", doc_type="_doc", id=doc_id, body=document)
# Bulk indexing call
k = ({ my data })
helpers.bulk(es, k)
If I replace http_auth = awsauth with my Kibana credentials http_auth = (kibana_username, kibana_password) it returns status 200 but then no new documents are created in the index which is weird.
I would like to know what I could be missing or where my configuration could be off.