Hi @WereCatf

In the logs of kubelet (journalctl -u snap.microk8s.daemon-kubelet) there are the following errors:

syys 16 03:16:10 ubuvm microk8s.daemon-kubelet[20094]: E0916 03:16:10.011947   20094 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": failed to get device for dir "/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": could not find device with major: 0, minor: 28 in cached partitions map.

and

syys 16 03:16:10 ubuvm microk8s.daemon-kubelet[20094]: F0916 03:16:10.141628   20094 kubelet.go:1296] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/snap/microk8s/common/var/lib/kubelet": could not find device with major: 0, minor: 28 in cached partitions map

I assume you are on btrfs and kubelet cannot query the stats of the underling filesystem. I wonder what our options are here? Is moving to another file system an option or do we need to further digg into this? I am not familiar with btrfs but this is a possibly related issue: [1]

[1] kubernetes/kubernetes#65204 (comment)

@ktsakalozos Wait, are you...are you serious? Using Btrfs breaks Microk8s?? Jesus Christ, that's definitely something I'd classify as a major issue.

Exactly the same problem. / uses btrfs and I get the same errors in the kubelet logs. Ubuntu 18.04 LTS based system.

Hi @GavinB-hpe

Could you try one of the two suggestions recommended in #1587 (comment)

Thanks @ktsakalozos

--feature-gates="LocalStorageCapacityIsolation=false"

Fixed things for me.

Do you think this is something it should be automated? Can we reliably detect the cases where this is needed?

@ktsakalozos Reliably detecting where this workaround is specifically absolutely needed, I dunno, but detecting cases where it might be needed could be as simple as: if [ $(df --output=fstype "/path/to/directory"|tail -n 1) = btrfs ]; then applyworkaround; fi

Personally, I don't think automating such a workaround should be done at all and instead the issue should be fixed properly at the source, but I don't really have a grasp of where it happens or who is in charge of the code and if they even care.

We could detect btrfs in microk8s inspect command and propose a fix [1]. We could also have a docs section in [2] to suggest such configurations.

[1] https://github.com/ubuntu/microk8s/blob/master/scripts/inspect.sh#L106
[2] https://microk8s.io/docs/troubleshooting#heading--common-issues

Setting: --feature-gates="LocalStorageCapacityIsolation=false" does not fix it for me on a btrfs root filesystem.
Is there any workaround?

To answer my own question: k8s 1.19 will not work, even with the feature-gates flag.
Downgrading to 1.18 solved the issue for me and my containers are starting with the feature-gates flag.