Performing reset, "Failed to decode etcd member-list output"

[luksi1]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.15.0-157-generic k0sproject/k0s#167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.34.1+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 3.8 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 9.0 GiB (pass)
Relative disk space available for /var/lib/k0s: 48% (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.0-157-generic (pass)
  Max. file descriptors per process: current: 1048575 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

While resetting a 1.34 cluster, I get the following error, which is fatal:

level=fatal msg="reset failed - log file saved to /root/.cache/k0sctl/k0sctl.log: failed to decode etcd member-list output: invalid character 'W' looking for beginning of value"

The exact same configuration, using 1.32, does not produce this error.

Steps to reproduce

1. Install a Cluster with k0sctl, using the following manifest. For instance, k0sctl apply --config my-config.yaml.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: acceptance-test
  user: admin
spec:
  k0s:
    version: v1.34.1+k0s.0
  hosts:
    - ssh:
        address: <my IP>
        user: root
        port: 22
        keyPath: null
      role: controller
    - ssh:
        address: <my IP>
        user: root
        port: 22
        keyPath: null
      role: worker
  options:
    wait:
      enabled: true
    drain:
      enabled: true
      gracePeriod: 2m0s
      timeout: 5m0s
      force: true
      ignoreDaemonSets: true
      deleteEmptyDirData: true
      podSelector: ""
      skipWaitForDeleteTimeout: 0s
    concurrency:
      limit: 30
      workerDisruptionPercent: 10
      uploads: 5
    evictTaint:
      enabled: false
      taint: k0sctl.k0sproject.io/evict=true
      effect: NoExecute
      controllerWorkers: false

2. Perform a reset with k0sctl reset --force --config my-config.yaml.
3. You should get the following error: level=fatal msg="reset failed - log file saved to /root/.cache/k0sctl/k0sctl.log: failed to decode etcd member-list output: invalid character 'W' looking for beginning of value"
4. Downgrade to v1.32.8+k0s.0 and this fatal error should no longer exist.

Expected behavior

v1.34.1+k0s.0 should be able to be reset without throwing any fatal errors.

Actual behavior

Resetting v1.34.1+k0s.0 throws a fatal error:

level=fatal msg="reset failed - log file saved to /root/.cache/k0sctl/k0sctl.log: failed to decode etcd member-list output: invalid character 'W' looking for beginning of value"

Screenshots and logs

No response

Additional context

No response

[hedge-sparrow]

Looks like

k0sctl/phase/gather_k0s_facts.go

Line 157 in e5383cc

if err := json.Unmarshal([]byte(output), &result); err != nil {

is tripping up on the output:

root@debian:~# k0s etcd member-list
W1013 13:54:33.744755    3565 logging.go:55] [core] [Channel k0sproject/k0s#1 SubChannel k0sproject/k0s#2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
{"members":{"debian":"https://10.0.2.15:2380"}}

not 100% sure what's causing this first error out of the box on a clean k0s install, but the response does include a valid member list 🤔

[jannispl]

Same issue here trying to upgrade from 1.33.2 to 1.34.1.

# k0s etcd member-list
W1013 14:59:55.918347 1242811 logging.go:55] [core] [Channel k0sproject/k0s#1 SubChannel k0sproject/k0s#2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
{"members":{"sv01165.soptim.net":"https://10.18.107.240:2380"}}

[twz123]

Seems to be #938. I've tripped over it once during testing. Thought this would be a super seldom edge case. But maybe something changed recently that provokes this?

[rtgiskard]

for my case, it's:

# k0s etcd member-list
W1014 09:40:02.644541  292878 logging.go:55] [core] [Channel k0sproject/k0s#1 SubChannel k0sproject/k0s#2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
{"members":{"m0s":"https://192.168.3.2:2380"}}

update from k0s 1.33.4 with config:

$ cat k0sctl.yaml 
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: dev.oo
  user: admin
spec:
  hosts:
  - role: controller+worker
    noTaints: true
    uploadBinary: true
    k0sBinaryPath: files/k0s-v1.34.1+k0s.0-amd64
    installFlags:
      - --cri-socket=remote:unix:///run/containerd/containerd.sock
      - --kube-controller-manager-extra-args="--allocate-node-cidrs=false"
      - --disable-components=coredns
      - --iptables-mode=nft
    ssh:
      address: 192.168.3.2
      user: root
      port: 22
      keyPath: ...
  k0s:
    version: v1.34.1+k0s.0
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          port: 8042
          k0sApiPort: 8043
          extraArgs:
            oidc-issuer-url: https://auth.dev.oo
            oidc-client-id: ....
            oidc-username-claim: email
            oidc-groups-claim: groups
        konnectivity:
          adminPort: 8048
          agentPort: 8049
        network:
          serviceCIDR: 10.48.0.0/12
          # hint for kube only, actually managed by cilium
          podCIDR: 10.64.0.0/16
          # use cilium
          provider: custom
          kubeProxy:
            disabled: true
            mode: nftables
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        storage:
          type: etcd
        telemetry:
          enabled: false
  options:
    wait:
      enabled: true
    drain:
      enabled: true
      gracePeriod: 2m0s
      timeout: 5m0s
      force: true
      ignoreDaemonSets: true
      deleteEmptyDirData: true
      podSelector: ""
      skipWaitForDeleteTimeout: 0s
    concurrency:
      limit: 30
      workerDisruptionPercent: 10
      uploads: 5
    evictTaint:
      enabled: false
      taint: k0sctl.k0sproject.io/evict=true
      effect: NoExecute
      controllerWorkers: false

[jannispl]

I think there's something wrong still with k0s etcd member-list that causes it to throw that warning.

[twz123]

I think there's something wrong still with k0s etcd member-list that causes it to throw that warning.

For me the warning looks like an internal retry of the member-list request inside the GRPC client. The fact that k0s is issuing the correct member list later on without erroring out itself makes me believe that the GRPC client returns without an error, too, and hence the retry succeeded eventually. "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled" might indicate that there's a connection timeout that was exceeded. Maybe some defaults changed within newer ercd client versions or new versions of grpc-go?

[kke]

Or the CLI's exit cancels a context, which is detected by some background task. Log goes to stderr, message goes to stdout, chances are that terminal displays (or ssh client reads) one before the other. The error may have actually happened after the output was already printed.

[kke]

I think the best solution in any case is to only parse output from stdout like in #938

Temporary workaround MAY be to set GRPC_GO_LOG_SEVERITY_LEVEL=error in the host's environment via something like:

hosts:
  - role: controller
    env:
      GRPC_GO_LOG_SEVERITY_LEVEL: error

[shad0wcrawl3r]

hosts:

• role: controller
  env:
  GRPC_GO_LOG_SEVERITY_LEVEL: error

This does not seem to work.

  hosts:
    - localhost:
        enabled: true
      noTaints: true
      environment:
        GRPC_GO_LOG_SEVERITY_LEVEL: error
      role: controller+worker

The same issue is shown.

[shad0wcrawl3r]

To add some context to this, it only happens when the version of k0s being used is v1.34.1+k0s.0. The previous release v1.32.8+k0s.0 seems to work fine.

[kke]

Yeah we can easily roll out a release with #938 still today which will fix the problem from k0sctl's perspective. Getting rid of unnecessary warnings on k0s side can be done separately if desired.

[shad0wcrawl3r]

 ashwin@wrath ~  sudo k0s etcd member-list 2>/dev/null
{"members":{"wrath":"https://10.10.5.2:2380"}}
 ashwin@wrath ~  sudo k0s etcd member-list 1>/dev/null
W1014 15:52:14.915097 1268281 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
 ashwin@wrath ~  sudo k0s etcd member-list
W1014 15:52:20.048875 1268341 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
{"members":{"wrath":"https://10.10.5.2:2380"}}
 ashwin@wrath ~  echo $?
0

Since the k0s process is exiting with an okay status code. I assume #938 should be able to solve this kind of issues in the future.