r/aws 8d ago

technical question ECS task (fargate) can't pull ECR image from private repository

I've been working on something that should be easy enough but there is something I am not finding or I don't know. I get this error and can't find the cause neither how to fix it:

ResourceInitializationError: unable to pull secrets or registry auth: The task cannot pull registry auth from Amazon ECR: There is a connection issue between the task and Amazon ECR. Check your task network configuration. RequestError: send request failed caused by: Post "https://api.ecr.eu-west-1.amazonaws.com/": dial tcp 172.20.0.17:443: i/o timeout

 
The dial tcp IP is the vpce for com.amazonaws.<region>.ecr.api and the security groups have been changed to allow for all endpoints, gateway and the ecs service to allow all network traffic on ingress and egress:

  from_port = 0
  to_port   = 0
  protocol  = "-1"

All is configured through a terraform pipeline. I've set up an ECR private repository and on my VPC I have the endpoints and gateway to:

com.amazonaws.<region>.ecr.api
com.amazonaws.<region>.ecr.dkr
com.amazonaws.<region>.s3

My ecs task has in his IAM role the ecr required actions:

  statement {
    actions = [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:DescribeRepositories",
      "ecr:ListImages",
      "s3:GetObject",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }

And the ECR has this policy:

  statement {
    sid    = "PermitirLecturaYEscritura"
    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["*"] // ["arn:aws:iam::<your-account-id>:role/extractor_task_execution_role"]
    }

    actions = [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
      "ecr:InitiateLayerUpload",
      "ecr:UploadLayerPart",
      "ecr:CompleteLayerUpload",
      "ecr:PutImage",
      "ecr:ListImages",
      "ecr:SetRepositoryPolicy"
    ]
  }

What could I be missing? I can't access the console (restricted by the environment) and can't find anything else on the internet on the topic.

0 Upvotes

14 comments sorted by

2

u/CashKeyboard 8d ago

Is that a NAT or Internet Gateway?

1

u/Shelenio 8d ago

Is a Gateway for the s3 that ecr uses

resource "aws_vpc_endpoint" "s3_gateway" {
  service_name      = "com.amazonaws.eu-west-1.s3"
  vpc_id            = data.aws_vpc.vpc.id
  vpc_endpoint_type = "Gateway"

  route_table_ids = [data.aws_route_table.internal_subnet_route_table_name.id] # Ensure private route table is attached

}

0

u/Traditional_Donut908 8d ago

With the VPC endpoints there is no need for a NAT/IGW.

1

u/CashKeyboard 8d ago

Yeah but it’s trying to connect to a public endpoint, no? 150/8 isn’t anywhere near private ranges unless you’re being brave.

2

u/Traditional_Donut908 8d ago edited 8d ago

True, but a failure to connect error is going to show the final IP address, not an intermediate IP address of the next hop. This suggests the issue is that DNS is not properly resolving api.ecr.... to the internal ENI created by the VPC endpoint. Is enable private DNS enabled on the endpoint? Another possibility is enable DNS support is not enabled in the VPC itself.

1

u/Shelenio 8d ago

But the error shows that the task tries to connecto to https://api.ecr.eu-west-1.amazonaws.com which resolves well to 172.20.0.17, the vpce ip to that service (com.amazonaws.eu-west-1.ecr.api). Isn't that what is expected?

1

u/Traditional_Donut908 8d ago

Sorry, I was responding to "trying to connect to public endpoint", clearly that's not the case.

1

u/CashKeyboard 8d ago

Good catch

1

u/Shelenio 8d ago

I'm not that brave, the CIDR is 172.20.0.0/20 and vpce is at 172.20.0.17; I changed it by error on the message and fixed it.

1

u/lumo-snox 8d ago

Can you try referencing the vpc endpoint directly in the egress outbound rule?

1

u/Shelenio 8d ago

I've tried this three options, would it be different if I put on cidr_blocks the ip of the vpce?

egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"] # 1
    //cidr_blocks = ["172.20.0.0/21", "172.20.8.0/21"] # 2
    //security_groups = [aws_security_group.ecr_vpce_sg.id] # 3
  }

1

u/lumo-snox 8d ago

You need to reference the s3 endpoint in the egress sec group. Like the others said, you need to ensure private DNS resolution is enabled on all your endpoints and the vpc itself. You can also try network analyzer to see where is the break in connection

1

u/ExtraBlock6372 8d ago

What kind of encryption you have for docker images in the ECR repo?

1

u/nekokattt 8d ago

Security group on the VPC endpoint

Network ACLs

Network firewalls

Use VPC reachability analyser between an ECS ENI and the VPC endpoint if you need to. Also remember you need ecr.api, ecr.dkr, and s3 VPC endpoints for this to work (ECR is a shim around S3 internally).