Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

Questions : Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

373

I am trying to load streaming Data into in4codes_aws-lambda Amazon ES from Amazon Kinesis Data Streams in4codes_aws-lambda as given in the tutorial: in4codes_aws-lambda https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-kinesis

As given in the tutorial, my lambda function in4codes_aws-lambda is:

import base64 import boto3 import json import requests from requests_aws4auth import AWS4Auth region = 'us-east-1' service = 'es' credentials = boto3.Session().get_credentials() awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token) host = '' # the ES domain has been specified here index = 'lambda-kine-index' type = 'lambda-kine-type' url = host + '/' + index + '/' + type + '/' headers = { "Content-Type": "application/json" } def handler(event, context): count = 0 for record in event['Records']: id = record['eventID'] timestamp = record['kinesis']['approximateArrivalTimestamp'] # Kinesis data is base64-encoded, so decode here message = base64.b64decode(record['kinesis']['data']) # Create the JSON document document = { "id": id, "timestamp": timestamp, "message": message } # Index the document r = requests.put(url + id, auth=awsauth, json=document, headers=headers) count += 1 return 'Processed ' + str(count) + ' items.' 

Also, as given in the tutorial, the IAM Role in4codes_aws-lambda is:

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "es:ESHttpPost", "es:ESHttpPut", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:DescribeStream", "kinesis:ListStreams" ], "Resource": "*" } ] } 

and the Trust Relationship is:

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } 

After doing this, the response I get when I in4codes_aws-lambda run the lambda is:

<Response [403]>

Any help in resolving this is appreciated.

Total Answers 3
32

Answers 1 : of Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

Credentials would only be applicable if in4codes_aws-elasticsearch you use an IAM user which isn’t the case in4codes_aws-elasticsearch here as this is a Lambda function and it in4codes_aws-elasticsearch requires an IAM role.

What you might have is fine-grained in4codes_aws-elasticsearch access control enabled which doesn’t in4codes_aws-elasticsearch work well with domain policies.

Read more here and notice the in4codes_aws-elasticsearch highlighted section re-user / IAM mixing in4codes_aws-elasticsearch and not working correctly.

0

5

Answers 2 : of Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

For those of you who are getting 403’s in4codes_aws-elasticsearch and the above solution doesn’t apply…

If you are using granular permissions, in4codes_aws-elasticsearch you need to add your lambda execution in4codes_aws-elasticsearch role as a Backend role (configured in in4codes_aws-elasticsearch kibana).

In Kibana -> Security / Roles

  • Add your role to “all_access” (or whatever role makes sense for your use case)

0

4

Answers 3 : of Unable to connect AWS Lambda to Elastic Search. Getting a 403 error

Make sure that your credentials are in4codes_aws-elasticsearch working. You can validate that using in4codes_aws-elasticsearch aws-cli. Refer the documentation here.

0