Kubernetes

1053 readers

6 users here now

founded 2 years ago

MODERATORS

erlingur

Daemon

NodeNotReady despite no pressure and an available network kubernetes (lemmy.zip)

submitted 2 days ago* (last edited 2 days ago) by [email protected] to c/kubernetes

4 comments fedilink hide all child comments

My cluster has been showing my raspberrypi node as "Ready" but according to the description of the node, the last log was "NodeNotReady" all debug guides say look for any pressure, like disk, pid, or so on, but there is no pressure, no absence of network. Here is the logs of my pi and pi status: https://pastebin.com/UULz6Hcy My pods are stuck in unknown (except jellyfin which is awaiting another node to come on): https://pastebin.com/vw2masAC A description of one of my pods if that helps: https://pastebin.com/s5W03s0E

also i already tried re-installing k3s

top 4 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 2 days ago (1 children)

These two jumped out at me:

Warning  FreeDiskSpaceFailed  35m (x286 over 24h)  kubelet  Failed to garbage collect required amount of images. Attempted to free 12087084646 bytes, but only found 253243 bytes eligible to free.

and


Warning  InvalidDiskCapacity  21m/14m  kubelet  invalid capacity 0 on image filesystem

Sounds like maybe a disk space issue on your nodes?

[–] [email protected] 1 points 2 days ago (2 children)

Thats weird, because I clearly have free space:

spiderunderurbed@raspberrypi:~ $ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  235G  184G   39G  83% /
spiderunderurbed@raspberrypi:~ $ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  235G  184G   39G  83% /home
spiderunderurbed@raspberrypi:~ $

any idea of things I can try to fix/debug this?

[–] [email protected] 1 points 2 days ago

/dev/mmcblk0p2

Is that a memory card?

Back in the days when I was using memory cards with my raspi, they had a habit of going to hell, every few months, or sometimes a year.

Now I have a M.2 SSD in one of these beautiful (and painfully expensive) external cases, and no problem anymore.

[–] [email protected] 1 points 2 days ago

Sorry, I don’t have as much experience with node management and config, so I’m not sure past that. Aside from focusing on where and how your images are being stored? That’s probably where I’d start first.