this post was submitted on 29 Aug 2023

98 points (96.2% liked)

Linux

48149 readers

597 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
No misinformation
No NSFW content
No hate speech, bigotry, etc

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago

MODERATORS

[email protected]

I am desperate. Linux is not booting (lemmy.sdf.org)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

59 comments fedilink hide all child comments

SOLUTION BELOW

The actual bug

I have never been in a more confusing situation regarding Linux.

I have a Dell XPS 15 9560, which had a dual boot Windows 10 / EndeavourOS setup. It was running fine for months. 10 days ago I updated Linux and after restart it couldn't boot anymore. It got stuck at "A start job is running for /dev/disk/by-uuid/..." (which is the root partition).

First, with the help of a friend of mine who is quite knowledgeable about Linux (he runs vanilla Arch, etc), we spent 5 hours trying to fix it but had no luck.

Then I decided to back up everything and do a fresh install. Aaaand the same error happened again on the first boot. Then I though "ok, probably some problem with Arch, lets try Fedora". Nope. Some similar error about not finding the root partition. (Here I must say that the kernel which was shipped with the ISO was working fine, but after updating to the latest one, it failed.) Here I thought "ok, then it might be a problem with the latest kernel, let's install EndeavourOS with the LTS kernel." Nope, LTS kernel also didn't boot. Then I tried Ubuntu and it worked, but that's not solving the problem. Then I decided to put another nvme drive in the laptop and try there. The same error again.

Now the greatest part: If I put the nvme drive into an external usb case, EndeavourOS installs, updates, boots without any problem, no sign of the error.

So now I don't know how to proceed... Maybe there is something wrong with the pcie port in my laptop, but except for the booting problem, windows is working, I can also mount and access every partition in the ssd through a live usb. So no other signs of problem with the port whatsoever.

I would be grateful for any advice as I've lost several days trying to solve this and I am out of ideas...

Solution: The last working kernels are from 11. August 2023 (both linux and linux-lts) linux-6.4.10.arch1-1 and linux-lts-6.1.45-1. You can download them from here: linux / linux-lts and install them with

sudo pacman -U the_path_to_the_package

Thank you all for the help!

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 11 points 1 year ago (1 children)

What is the output of blkid and the content of /etc/fstab of the installed-but-not-booting system? You will need a live usb to get the those, probably.

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago) (14 children)

[root@EndeavourOS /]# blkid
/dev/nvme0n1p1: UUID="6903-7FA3" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="86a0a714-815f-45db-ad0e-9eba16861903"
/dev/nvme0n1p2: LABEL="endeavouros" UUID="9ae3c50f-be08-4594-ac30-2d094375868d" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="endeavouros" PARTUUID="9b52162c-a4bf-4e5e-8096-bb7f144c481a"
/dev/loop0: BLOCK_SIZE="1048576" TYPE="squashfs"
/dev/sda2: SEC_TYPE="msdos" LABEL_FATBOOT="ARCHISO_EFI" LABEL="ARCHISO_EFI" UUID="6121-B369" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="1bdb3cc0-02"
/dev/sda1: BLOCK_SIZE="2048" UUID="2023-08-05-13-57-43-00" LABEL="EOS_202308" TYPE="iso9660" PARTUUID="1bdb3cc0-01"
[root@EndeavourOS /]# sudo cat /etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
#                      
UUID=6903-7FA3                            /efi           vfat    defaults,noatime 0 2
UUID=9ae3c50f-be08-4594-ac30-2d094375868d /              ext4    defaults,noatime 0 1
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0

[–] [email protected] 4 points 1 year ago

Please use tripe backticks to create codeblocks - much easier to read. Like this:

```

some text

```

results in

some text

[–] [email protected] 2 points 1 year ago (1 children)

I'm also curious about the specific UUID it's looking for. Any chance you could take a photo of the screen and upload it somewhere?

[–] [email protected] 2 points 1 year ago

https://postimg.cc/cgvdVrgR

load more comments (12 replies)

[–] [email protected] 9 points 1 year ago* (last edited 1 year ago) (1 children)

So this occurs after an update. Is it not possible to boot into the prior kernel?

If possible to boot into the prior kernel, can you inspect logs or the journal to see where your error is cropping up?

This issue sounds like a regression of sorts with a driver, but log/debug would help confirm. This would be one worth reporting to upstream if you can rescue some logs (I gather you can if you can boot the disk from another enclosure).

If you can boot into the machine, investigate note from the journal:

journalctl --list-boots
journalctl -b -1,
- where -1 was the prior boot, -2, the one before that, etc

-- If you are booting into a live environment or are otherwise mounting the disk:

journalctl -D /var/log/journal/ID_GOES_HERE
example path: /var/log/journal/2dff8304d5114c44bfb1311357a3cd87

-- Keep us posted.

If truly a driver regression, but you can boot from the prior kernel (if you don't have it, install it via livecd or so), definitely report this one and remain on the prior kernel until resolved. Bleeding edge things.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

I have already wiped everything, so no logs... The only way to have it booting is to install EndeavourOS using the offline installer, which is using kernel 6.4.8. There is an option to install the LTS kernel alongside. So the system is booting with 6.4.8, but after updating, neither the new 6.4.12, or the LTS, which is 6.2, doesn't boot. I haven't tried booting with the LTS kernel before updating, to see if the same kernel is working before and after or not. I will try to reinstall it using the offline installer and then try to gather some logs after updating.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago) (1 children)

If you do go this route - try to update the kernel only, not the rest of the system. Yes, it's bad under normal circumstances, but I think we're way past those.

sudo pacman -Syy linux

This would rule out some other part of system getting in the way - like systemd.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

I updated only the kernel and I got the following:

:: kernel-install installing kernel 6.4.12-arch1-1
dracut: Executing: /usr/bin/dracut --no-hostonly --force /efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback 6.4.12-arch1-1
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoemon' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoeadm' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nbd' depends on 'network', which can't be installed
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: crypt ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: nvdimm ***
dracut: *** Including module: qemu ***
dracut: *** Including module: qemu-net ***
dracut: *** Including module: lunmask ***
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: virtiofs ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:                     real
dracut: Method:                   memcmp
dracut: Files:                    1987
dracut: Linked:                   8 files
dracut: Compared:                 0 xattrs
dracut: Compared:                 449 files
dracut: Saved:                    1.42 MiB
dracut: Duration:                 0.022128 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback' ***
dracut: *** Creating initramfs image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback' done ***
dracut: Executing: /usr/bin/dracut --hostonly --no-hostonly-cmdline -f /efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd 6.4.12-arch1-1

(Continues in the second comment)

[–] [email protected] 1 points 1 year ago (1 children)

dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: Skipping udev rule: 70-persistent-net.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:                     real
dracut: Method:                   memcmp
dracut: Files:                    780
dracut: Linked:                   2 files
dracut: Compared:                 0 xattrs
dracut: Compared:                 38 files
dracut: Saved:                    356.65 KiB
dracut: Duration:                 0.010006 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd' ***
dracut: *** Creating initramfs image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd' done ***
(4/5) Check if user should be informed about rebooting after certain system package upgrades.
(5/5) Checking which packages need to be rebuilt

[–] [email protected] 0 points 1 year ago (1 children)

Did it boot or did it get stuck at the same spot again?

[–] [email protected] 0 points 1 year ago (5 children)

It got stuck at the same spot.

[–] [email protected] 3 points 1 year ago

Implies it is the kernel. Mark it on hold and stick to the working kernel for the time being.

load more comments (4 replies)

[–] [email protected] 7 points 1 year ago

According to your logs when installing the kernel, your nvme modules are nowhere to be found, and basic system tools are unavailable. Your core system seems severely borked... can you do a memtest (broken ram can corrupt your storage and your attemps at finding the problem) and check your smartctl --all of that ssd?

[–] [email protected] 4 points 1 year ago (1 children)

Try adding "PCIE_ASPM=off" to the kernel boot commandline.

[–] [email protected] 1 points 1 year ago

This didn't help :/

[–] [email protected] 4 points 1 year ago

Use a live USB to back up your home directory and a list of your packages, then reinstall. I don't think it's worth the trouble.

[–] [email protected] 4 points 1 year ago

when it gets stuck on something like that, it's because of a hardware related change; did you update or modify your bios recently or change hardware drivers?

[–] [email protected] 4 points 1 year ago (1 children)

AHAHAHA that is a proper insane bug. One PCIe device shouldn't be able to slap others off of the "bus", "we're not on a bus all you did was mess up your own personal lanes mate".

[–] [email protected] 3 points 1 year ago

I lost 3 days of my life + now I need reinstall everything 🥲

[–] [email protected] 3 points 1 year ago (1 children)

That's a weird issue. Do you have encryption on by any chance? I had a similar error pop-up when I didn't have the correct systems hooks for the kernel so that after install update of kernel would make the system not boot.

[–] [email protected] 4 points 1 year ago (1 children)

No, I have never used encryption

[–] [email protected] 1 points 1 year ago

Still might be worth it to check that you're not missing any kernel modules.

[–] [email protected] 2 points 1 year ago (1 children)

Share the errors you are experiencing, please

[–] [email protected] 0 points 1 year ago (1 children)

Could you please take a look at the comments from abrer and Illecors, I have shares a lot of info there.

[–] [email protected] 1 points 1 year ago (1 children)

Could you link to them or quote them, please?

[–] [email protected] 1 points 1 year ago

The problem is already solved (At least I found the bug and a temporary solution) At the beginning of the post there is a link to the bug report.

[–] [email protected] 1 points 1 year ago (1 children)

Shot in the dark: Did SecureBoot get re-enabled somehow? Could you check the UEFI?

[–] [email protected] 10 points 1 year ago (1 children)

Hard to not sound condescending in text, so forgive me.

If systemd is trying to mount the drives, we're at a point where the (most likely) ramdisk has already booted the kernel - secure boot would've prevented that.

[–] [email protected] 6 points 1 year ago

I won't take it as condescending then! Thank you for the input :)

[–] [email protected] 1 points 1 year ago

I had se similar issue but with bluetooth, worked on windows not on linux. Fixed it by resetting the BIOSwith the CMOS battery thing

load more comments