Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client


I’m trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.

My setup:

  • ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record:
  • I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
  • I’m trying to connect to ETCD database from the controller, using Go client SDK library for ETCD

Here’s my affected Go code:

cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{""},
    DialTimeout: 5 * time.Second,
    Username:    username,
    Password:    password,

if err != nil {
    return nil, fmt.Errorf("opening ETCD client failed: %v", err)

And here’s an error I’m getting when clientv3.New(...) gets executed:

{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}*Controller).processNextWorkItem

The same error happens when I’m passing some dummy, invalid credentials.

However, when I tried to access the database in a HTTP API manner:

postBody, _ := json.Marshal(map[string]string{
    "name":     username,
    "password": password,
responseBody := bytes.NewBuffer(postBody)

resp, err := http.Post("", "application/json", responseBody)
if err != nil {
    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()

…I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn’t work when following the client SDK approach.

When I’m using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I’d really appreciate any suggestions.

I’ve also added a simple pod to try accessing my etcd db via etcdctl:

apiVersion: v1
kind: Pod
  name: test-pod
  namespace: my-controller-namespace
  - name: etcdctl
    image: bitnami/etcd
    - sleep
    - infinity

When logged into the container via kubectl exec, I was able to access my db:

$ etcdctl --user="user" --password="password" put foo bob

I guess the problem is somewhere in the SDK?


Turned out to be version mismatch – my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (

enter image description here

Answered By – magnes

Answer Checked By – David Goodson (GoLangFix Volunteer)

Leave a Reply

Your email address will not be published.