Debian operating system

2773 readers

1 users here now

Debian is a free operating system (OS) for your computer. An operating system is the set of basic programs and utilities that make your computer run. Debian provides more than a pure OS: it comes with over 59000 packages, precompiled software bundled up in a nice format for easy installation on your machine.

founded 5 years ago

MODERATORS

[email protected]

Systemd timed out waiting on device, after replacing mdadm drives (mander.xyz)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

13 comments fedilink hide all child comments

I have an annoying problem on my server and google has been of no help. I have two drives mirrored for the OS through mdadm, and I recently replaced them with larger versions through the normal process of replacing one at a time and letting the new drive re-sync, then growing the raids in place. Everything is working as expected, with the exception of systemd... It is filling my logs with messages of timing out while trying to locate both of the old drives that no longer exist. Mdadm itself is perfectly happy with the new storage space and has reported no issues, and since this is a server I can't just blindly reboot it to get systemd to shut the hell up.

So what's the solution here? What can I do to make this error message go away? Thanks.

[Update] Thanks to everyone who made suggestions below, it looks like I finally found the solution in systemctl daemon-reload however there is a lot of other great info provided to help with troubleshooting. I'm still trying to learn the systemd stuff so this has all been greatly appreciated!

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 2 points 1 year ago (1 children)

systemctl disable --now olddisk.mount?

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

Sounds interesting, any chance you can tell me what it does? Google doesn't even seem to have any hits on "olddisk.mount" and I want to make sure this won't break anything else as it could be months before the system is intentionally rebooted again.

Also of note - I don't see anything with a name similar to olddisk.mount in the systemd folder. Is this command unique to a particular distro? For reference, I'm running Debian.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago) (2 children)

I think olddisk refers to the name of your device. Try systemctl status or just systemctl and see if it's in the output. Or find the name in the journal.

[–] [email protected] 3 points 1 year ago

Status reports "State: degraded" but then doesn't say WHAT is degraded and shows no other errors (and /proc/mdstat shows no errors). Trying systemctl by itself does show an error from logrotated but that seems unrelated?

I do see the drive errors again in journalctl but I don't see anything helpful here... maybe you'll see something? These errors get repeated for both of the old drives about every 30 minutes, and I believe the UUIDs are for the old drives since they don't match any existing drive.

Oct 11 07:10:40 Juno systemd[1]: Timed out waiting for device ST500LM021-1KJ152 5.

Oct 11 07:10:40 Juno systemd[1]: Dependency failed for /dev/disk/by-uuid/286e26b0-603a-43b2-bc0f-30853998d5ab.

Oct 11 07:10:40 Juno systemd[1]: dev-disk-by\x2duuid-286e26b0\x2d603a\x2d43b2\x2dbc0f\x2d30853998d5ab.swap: Job dev-disk-by\x2duuid-286e26b0\x2d603a\x2d43b2\x2dbc0f\x2d30853998d5ab.swap/start failed with result 'dependency'.

Oct 11 07:10:40 Juno systemd[1]: dev-disk-by\x2duuid-286e26b0\x2d603a\x2d43b2\x2dbc0f\x2d30853998d5ab.device: Job dev-disk-by\x2duuid-286e26b0\x2d603a\x2d43b2\x2dbc0f\x2d30853998d5ab.device/start failed with result 'timeout'.

Oct 11 07:10:40 Juno systemd[1]: dev-disk-by\x2duuid-96b0277b\x2dcf9d\x2d4360\x2dbf90\x2d691166cff52b.device: Job dev-disk-by\x2duuid-96b0277b\x2dcf9d\x2d4360\x2dbf90\x2d691166cff52b.device/start timed out.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago) (2 children)

Right. systemctl list-automounts
to find the name, maybe? I've never had exactly this problem though.

Looks like list-automounts is relatively new, try systemctl status --full --all -t mount for all mounts and look for your old disks in the info.
-t automount might work but mine is empty, which makes me think this might not be related to the automount unit type.
Hopefully this will point us in the right direction though.

[–] [email protected] 3 points 1 year ago (1 children)

That appears to be a success! Thanks for the pointers, I'm still trying to figure out the systemd stuff since I rarely have to touch it.

[–] [email protected] 2 points 1 year ago (1 children)

Sweet, no problem. Good luck.

[–] [email protected] 3 points 1 year ago (1 children)

Still no new errors in the logs. It wasn't hurting anything, it was just annoying and I didn't want to reboot a server just because of a logging issue! 😆

[–] [email protected] 2 points 1 year ago

Also it was just going to keep trying forever.

[–] [email protected] 3 points 1 year ago

Ah cool... the 'full' command actually advised running systemctl daemon-reload which appears to have cleared the errors listed. Based on previous errors in the log it will likely be another 20 minutes before another error would be generated, so I'm waiting to see what happens now.

[–] Still 1 points 1 year ago

systemd generates .mount for all your disks you have in fstab or via mdadm disabling it will just stop the error appearing