Golang REST API Deployment on AWS EKS Fails with CrashLoopBackOff

Issue

I’m trying to deploy a simple REST API written in Golang to AWS EKS.

I created an EKS cluster on AWS using Terraform and applied the AWS load balancer controller Helm chart to it.

All resources in the cluster look like:

NAMESPACE     NAME                                                READY   STATUS    RESTARTS   AGE
kube-system   pod/aws-load-balancer-controller-5947f7c854-fgwk2   1/1     Running   0          75m
kube-system   pod/aws-load-balancer-controller-5947f7c854-gkttb   1/1     Running   0          75m
kube-system   pod/aws-node-dfc7r                                  1/1     Running   0          120m
kube-system   pod/aws-node-hpn4z                                  1/1     Running   0          120m
kube-system   pod/aws-node-s6mng                                  1/1     Running   0          120m
kube-system   pod/coredns-66cb55d4f4-5l7vm                        1/1     Running   0          127m
kube-system   pod/coredns-66cb55d4f4-frk6p                        1/1     Running   0          127m
kube-system   pod/kube-proxy-6ndf5                                1/1     Running   0          120m
kube-system   pod/kube-proxy-s95qk                                1/1     Running   0          120m
kube-system   pod/kube-proxy-vdrdd                                1/1     Running   0          120m

NAMESPACE     NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes                          ClusterIP   10.100.0.1      <none>        443/TCP         127m
kube-system   service/aws-load-balancer-webhook-service   ClusterIP   10.100.202.90   <none>        443/TCP         75m
kube-system   service/kube-dns                            ClusterIP   10.100.0.10     <none>        53/UDP,53/TCP   127m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/aws-node     3         3         3       3            3           <none>          127m
kube-system   daemonset.apps/kube-proxy   3         3         3       3            3           <none>          127m

NAMESPACE     NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/aws-load-balancer-controller   2/2     2            2           75m
kube-system   deployment.apps/coredns                        2/2     2            2           127m

NAMESPACE     NAME                                                      DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/aws-load-balancer-controller-5947f7c854   2         2         2       75m
kube-system   replicaset.apps/coredns-66cb55d4f4                        2         2         2       127m

I can run the application locally with Go and with Docker. But releasing this on AWS EKS always throws CrashLoopBackOff.

Running kubectl describe pod PODNAME shows:

Name:         go-api-55d74b9546-dkk9g
Namespace:    default
Priority:     0
Node:         ip-172-16-1-191.ec2.internal/172.16.1.191
Start Time:   Tue, 15 Mar 2022 07:04:08 -0700
Labels:       app=go-api
              pod-template-hash=55d74b9546
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           172.16.1.195
IPs:
  IP:           172.16.1.195
Controlled By:  ReplicaSet/go-api-55d74b9546
Containers:
  go-api:
    Container ID:   docker://a4bc07b60c85fd308157d967d2d0d688d8eeccfe4c829102eb929ca82fb25595
    Image:          saurabhmish/golang-hello:latest
    Image ID:       docker-pullable://saurabhmish/golang-hello@sha256:f79a495ad17710b569136f611ae3c8191173400e2cbb9cfe416e75e2af6f7874
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 15 Mar 2022 07:09:50 -0700
      Finished:     Tue, 15 Mar 2022 07:09:50 -0700
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jt4gp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-jt4gp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m31s                   default-scheduler  Successfully assigned default/go-api-55d74b9546-dkk9g to ip-172-16-1-191.ec2.internal
  Normal   Pulled     7m17s                   kubelet            Successfully pulled image "saurabhmish/golang-hello:latest" in 12.77458991s
  Normal   Pulled     7m16s                   kubelet            Successfully pulled image "saurabhmish/golang-hello:latest" in 110.127771ms
  Normal   Pulled     7m3s                    kubelet            Successfully pulled image "saurabhmish/golang-hello:latest" in 109.617419ms
  Normal   Created    6m37s (x4 over 7m17s)   kubelet            Created container go-api
  Normal   Started    6m37s (x4 over 7m17s)   kubelet            Started container go-api
  Normal   Pulled     6m37s                   kubelet            Successfully pulled image "saurabhmish/golang-hello:latest" in 218.952336ms
  Normal   Pulling    5m56s (x5 over 7m30s)   kubelet            Pulling image "saurabhmish/golang-hello:latest"
  Normal   Pulled     5m56s                   kubelet            Successfully pulled image "saurabhmish/golang-hello:latest" in 108.105083ms
  Warning  BackOff    2m28s (x24 over 7m15s)  kubelet            Back-off restarting failed container

Running kubectl logs PODNAME and kubectl logs PODNAME -c go-api shows standard_init_linux.go:228: exec user process caused: exec format error

Manifests:

go-deploy.yaml ( This is the Docker Hub Image with documentation )

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-api
  labels:
    app: go-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-api
  strategy: {}
  template:
    metadata:
      labels:
        app: go-api
    spec:
      containers:
      - name: go-api
        image: saurabhmish/golang-hello:latest
        ports:
          - containerPort: 3000
        resources: {}

go-service.yaml

---
kind: Service
apiVersion: v1
metadata:
  name: go-api
spec:
  selector:
    app: go-api
  type: NodePort
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000

How can I fix this error ?

Solution

Posting this as Community wiki for better visibility.
Feel free to expand it.


Thanks to @David Maze, who pointed to the solution. There is an article ‘Build Intel64-compatible Docker images from Mac M1 (ARM)’ (by Beppe Catanese) here.
This article describes the underlying problem well.

You are developing/building on the ARM architecture (Mac M1), but you deploy the docker image to a x86-64 architecture based Kubernetes cluster.

Solution:

Option A: use buildx

Buildx is a Docker plugin that allows, amongst other features, to build images for various target platforms.

$ docker buildx build --platform linux/amd64 -t myapp .

Option B: set DOCKER_DEFAULT_PLATFORM

The DOCKER_DEFAULT_PLATFORM environment variable permits to set the default platform for the commands that take the –platform flag.

export DOCKER_DEFAULT_PLATFORM=linux/amd64

Answered By – mozello

Answer Checked By – Dawn Plyler (GoLangFix Volunteer)

Leave a Reply

Your email address will not be published.