AWS Lambda does not seem to get an private IP address

Question

I am testing the Vault Lambda Extension which allows to reads secrets from Hashicorp Vault without using tokens.

I therefore created a demo lambda function which use the layer "Vault Lambda Extension". The lambda crashes before the python code is run.

Example of the execution result:

foo

Response
{
  "errorType": "Extension.Crash",
  "errorMessage": "RequestId: b16468f9-009c-4b1a-91ca-8b0899e01b7c Error: exit status 1"
}

Function Logs
2023-02-15T18:15:10.999Z [INFO]  vault-lambda-extension: Initialising
2023-02-15T18:15:11.006Z [ERROR] Fatal error, exiting: error="error logging in to Vault: read tcp 169.254.76.1:39510->10.16.1.55:8200: read: connection reset by peer"
EXTENSION   Name: vault-lambda-extension    State: Started  Events: []
2023-02-15T18:15:12.712Z [INFO]  vault-lambda-extension: Initialising
2023-02-15T18:15:12.773Z [ERROR] Fatal error, exiting: error="error logging in to Vault: read tcp 169.254.76.1:39514->10.16.1.55:8200: read: connection reset by peer"
EXTENSION   Name: vault-lambda-extension    State: Started  Events: []
START RequestId: b16468f9-009c-4b1a-91ca-8b0899e01b7c Version: $LATEST
RequestId: b16468f9-009c-4b1a-91ca-8b0899e01b7c Error: exit status 1
Extension.Crash
END RequestId: b16468f9-009c-4b1a-91ca-8b0899e01b7c
REPORT RequestId: b16468f9-009c-4b1a-91ca-8b0899e01b7c  Duration: 1694.95 ms    Billed Duration: 1695 ms    Memory Size: 128 MB Max Memory Used: 24 MB

As you can see the address IP of the lambda is 169.254.76.1 while it has been set to be in private subnet in my Dev VPC.

The role (modified) set for this lambda is

{
  "partial": false,
  "policies": [
    {
      "arn": "arn:aws:iam::123456789012:policy/aws_iam_policy_lambda_vault_lab",
      "document": {
        "Statement": [
          {
            "Action": [
              "ec2:DescribeNetworkInterfaces",
              "ec2:CreateNetworkInterface",
              "ec2:DeleteNetworkInterface",
              "ec2:DescribeInstances",
              "ec2:AttachNetworkInterface",
              "ec2:DescribeSecurityGroups",
              "ec2:DescribeSubnets",
              "ec2:DescribeVpcs",
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:DescribeLogGroups",
              "logs:DescribeLogStreams",
              "logs:PutLogEvents",
              "logs:GetLogEvents",
              "logs:FilterLogEvents"
            ],
            "Effect": "Allow",
            "Resource": "*"
          }
        ],
        "Version": "2012-10-17"
      },
      "id": "ANPATRANPATRANPATRANPAT",
      "name": "aws_iam_policy_lambda_vault_lab",
      "type": "managed"
    }
  ],
  "resources": {
    "ec2": {
      "service": {
        "icon": "data:image/svg+xml;base64,PHN2ZyB2a...CAgPC9nPgogIDwvZz4KPC9zdmc+Cg==",
        "name": "Amazon EC2"
      },
      "statements": [
        {
          "action": "ec2:DescribeNetworkInterfaces",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:CreateNetworkInterface",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:DeleteNetworkInterface",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:DescribeInstances",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:AttachNetworkInterface",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:DescribeSecurityGroups",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:DescribeSubnets",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "ec2:DescribeVpcs",
          "effect": "Allow",
          "resource": "*",
          "service": "ec2",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        }
      ]
    },
    "logs": {
      "service": {
        "icon": "data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIwIDAg...z4KICA8L2c+Cjwvc3ZnPgo=",
        "name": "Amazon CloudWatch Logs"
      },
      "statements": [
        {
          "action": "logs:CreateLogGroup",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:CreateLogStream",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:DescribeLogGroups",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:DescribeLogStreams",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:PutLogEvents",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:GetLogEvents",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        },
        {
          "action": "logs:FilterLogEvents",
          "effect": "Allow",
          "resource": "*",
          "service": "logs",
          "source": {
            "index": "0",
            "policyName": "aws_iam_policy_lambda_vault_lab",
            "policyType": "managed"
          }
        }
      ]
    }
  },
  "roleName": "lambda_role_vault_lab",
  "trustedEntities": [
    "lambda.amazonaws.com"
  ]
}

I don't include the python code has the lambda crashs before starting executing it. For info, I tried a few changes in the policy without any effects. Has someone an idea where I did go wrong?

Elevate Your Expertise with Microservices Certification!

Comments

Appears to be valid

Answer 1

The IP address 169.254.76.1 in the logs suggests that the Lambda function is running in an environment where the IP address is assigned using link-local addressing (also called Automatic Private IP Addressing, or APIPA), rather than being assigned a private IP address within the VPC subnet.

When a Lambda function is configured to run within a VPC, it should be assigned a private IP address from the subnet that it is associated with. This is necessary to enable the Lambda function to communicate with resources within the VPC. It's possible that your Lambda function is not associated with the correct subnet or security group, which is causing the issue.

Here are a few things you can check to troubleshoot this issue:

1. Check that the Lambda function is associated with the correct VPC and subnet. You can do this by looking at the function's configuration in the AWS Management Console or by using the AWS CLI.

2. Make sure that the subnet associated with the Lambda function has available IP addresses. If the subnet has run out of available IP addresses, you may need to expand the subnet's CIDR range or create a new subnet with a larger CIDR range.

3. Verify that the security group assigned to the Lambda function allows traffic to and from the resources that the function needs to communicate with (in this case, Vault). You may need to modify the security group rules to allow the necessary traffic.

4. If your Lambda function is associated with a VPC that has a Network Address Translation (NAT) gateway configured, ensure that the NAT gateway is correctly configured and that the Lambda function's security group is allowed to access it.

5. Verify that your Lambda function has the necessary permissions to access resources within your VPC. This includes permissions to describe and create network interfaces, as well as permissions to interact with other VPC resources.

If you're still having trouble, you may want to consult the AWS Lambda documentation or open a support ticket with AWS.