Authentication of AWS MFA with Apache Airflow

AWS offers a managed airflow service that is introduced in this blog post: https://aws.amazon.com/blogs/aws/introducing-amazon-managed-workflows-for-apache-airflow-mwaa/. Regarding a certain issue, the user is seeking help to find the Airflow REST API URL in order to trigger a DAG from an AWS Lambda Function. The DAG script is created within a docker container, and the necessary file is also located within it. Since the container is within the AWS boundaries, no credentials need to be specified for resolving requests.

Question:

Utilizing
helm chart
, I have been
running Airflow
. The purpose of implementing airflow is to trigger a
AWS Batch
job within the DAGs, as shown below.

batch = AWSBatchOperator(
    task_id='batch',
    job_name='my-job',
    job_definition='arn:aws:batch:my-job-def',
    job_queue='arn:aws:batch:my-job-queue',
    overrides={
        'command': ["echo", "hello"]
    },
    array_properties={},
    parameters={},
    region_name='my-region',
    dag=dag
)

Upon executing this command, the task does not succeed and the following log is generated.

[2020-12-01 08:30:11,642] {awsbatch_operator.py:150} INFO - AWS Batch Job has failed executed
[2020-12-01 08:30:11,660] {taskinstance.py:1150} ERROR - Unable to locate credentials

Our AWS policy mandates users to authenticate with MFA in order to perform any action within the AWS. This appears to be a sensible measure.

In which location should I define the login details (e.g. aws_access_key, secret, etc.) for the above environment? Would it be more appropriate to define them in the values.yaml of the helm chart or within the AirFlow UI?


Solution:

If you are using
airflow on aws
, it is possible to assign an IAM role to either the instance (EC2), tasks (ECS), or pod (EKS). This way, the IAM role will be utilized instead of credentials.

In addition, AWS offers a service for managing airflow called
amazon-managed-workflows
, which can be used for
apache-airflow
and can be found at https://aws.amazon.com/blogs/aws/introducing-
amazon-managed-workflows
-for-
apache-airflow
-mwaa/.

Frequently Asked Questions