this post was submitted on 06 Aug 2023
42 points (97.7% liked)

Linux

48234 readers
619 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
42
Unable to run apt update | Lock held by apt-get (lemmy.ml)
submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]
34 comments fedilink hide all child comments
 

I'm still a fairly new Linux-user (on Tuxedo OS), and I just ran into an issue that is new to me. If I try to update my system, either via command line or Discover, the apt update command fails. This is the output:

E: Could not get lock /var/lib/apt/lists/lock. It is held by process 1635 (apt-get)
N: Be aware that removing the lock file is not a solution and may break your system.
E: Unable to lock directory /var/lib/apt/lists/

Process 1635 is apt-get update run by root, and persists through restart. I am tempted to try to kill it (kill 1635), but I'm not sure if anything could break from that, so I thought I'd try to ask for help first before I do something stupid.

EDIT:

I have managed to update my system by killing the process, which releases the lock, and then going on to do normal sudo apt update and sudo apt upgrade. For the sake of troubleshooting, I tried to add back my third-party repos one by one, and none of them caused any problem. However, when rebooting the same issue as described above happens again. Software updates is set to "Manually" in the System settings.

In addition, everytime I ran sudo apt upgrade, at the end some update related to initramfs fails. My disk is encrypted using cryptsetup, and as I’ve come to understand, I should be very careful doing anything related to initramfs when that is the case. Here is the output:

Processing triggers for initramfs-tools (0.140ubuntu13.2) ...
update-initramfs: Generating /boot/initrd.img-6.2.0-10018-tuxedo
I: The initramfs will attempt to resume from /dev/dm-2
I: (/dev/mapper/system-swap)
I: Set the RESUME variable to override this.
zstd: error 25 : Write error : No space left on device (cannot write compressed block) 
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-6.2.0-10018-tuxedo with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

EDIT 2:

The issue seems to have been narrowed down to a failure of Tuxedo's driver configuration service that runs at boot. It is this process that calls apt-get (and something I should've seen earlier...), and systemctl status reveals some errors:

aug. 08 15:33:56 laptop systemd[1]: Starting Tomte-daemon, finishes tasks that could not be accomplished before...
aug. 08 15:34:06 laptop tuxedo-tomte[1393]: no network found!! some fixes might not be applied correctly
aug. 08 15:34:06 laptop tuxedo-tomte[1393]: systemctlCmd: systemd-run --on-active="30sec" tuxedo-tomte configure all >/dev/null 2>&1

I really appreciate the help from everyone so far. It's a good experience asking for help here, and I've learned a lot from your answers. Makes being a Linux newbie a lot easier. So thank you :)

Since this seems to be a very specific issue related to Tuxedo's own services, I will contact their support to get their input on what to do next.

top 34 comments
sorted by: hot top controversial new old
[–] [email protected] 22 points 1 year ago (1 children)

My first guess is unattended-upgrades is running, especially if this is shortly after booting. As others have said, ps aux | grep apt will tell you what’s running. If it’s holding up all the time there might be something wrong with apt causing the update to hang.

[–] [email protected] 6 points 1 year ago (1 children)

Yeah, it happens straight after boot, but it never resolves itself. And the process exists after reboot as well, so it seems that it starts running apt-get update and gets stuck / hangs.

[–] [email protected] 5 points 1 year ago* (last edited 1 year ago) (1 children)

Definitely sounds like auto-update if it's respawning itself on every boot. The fact that it never exits is weird though; have you added any third party repos? What's in your apt sources.list file(s)?

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

My sources.list file is pretty clean, with 3 https-sources for the Tuxedo OS mirror repos of the original Ubuntu ones. In the sources.list.d directory, there are some that have been added by me, such as for Signal, Librewolf and VS Codium. In total there are 11 files in here, each with one additional source. All except one is https, and the last one is mirror+file. In the process tree for apt-get, there are 13 subprocesses, while there are 14 sources in total (11+3). Could it be that it hangs on the last one here?

EDIT: Would this be a viable way to troubleshoot? I backup the sources and just replace them with a blank sources.list file and an empty sources.list.d directory. If that works, I add the repos back one at the time and see which one that fails. Or could I run into unintended trouble if I remove the main repos, even for a short time? I would think that it just wouldn't find anything and just be happy there are no updates.

[–] [email protected] 2 points 1 year ago (2 children)

I don't think that's the problem, and I wouldn't really anything related to sources.list or anything in sources.list.d/.(*)

It does seem like an auto update thing, so got some quick suggestions.

Since you mentioned Discover, I'm guessing you're on KDE Plasma, which is great because it's what I use and know immediately where to go.

Go into the main settings and way at the bottom left after all the other sections, there should be a section labeled "Software Update". The only thing you need to care about is the first option, which will be "Update software: Manually/Automatically".

If "Automatically" is checked, try disabling it and then reboot. If that doesn't, work, there's a more useful GUI tool, but for some reason it doesn't show up when searching the application menu (or at least doesn't for me).

Quick command line thing here:

sudo software-properties-kde

This has a bunch of options which I won't get into, but in the "Updates" tab, you'll also see "Automatic updates" with a few more options than the one in Settings. I'd enable "Only notify about available updates" if everything else is enabled.

You can disable "Check for updates" entirely, but I wouldn't recommend it. You can change that to "Weekly" or something, so that it isn't constantly checking, but I don't know if it starts up on every boot, so might be useless to change that specifically.

After rebooting, take a look at what the file /etc/apt/apt.conf.d/20auto-upgrades says.

It shouldn't be too long, but depending on what the values might be for Download-Upgradable-Packages and Unattended-Upgrades, that could be the source of the problem.

Everything in that file should now have a value of "0" except for the first line (Update-Package-Lists which will have a value of "1").

(*) Theoretically, your idea about sources lists could maybe be an issue, but I'm not sure. I don't know if it's possible that apt-get could be in some kind of loop if there's a mismatch or duplicates with your repo sources. I'd assume it would error out, but maybe it's just being weird.

For now, I'd leave that all alone. If none of the above works, I can give some instructions on dealing with third party repos, but I don't want to make this comment any longer than it is already.

[–] [email protected] 4 points 1 year ago (1 children)

Auto update itself isn't the root problem. The problem is that apt update is hanging and never finishing. It just happens to be getting called automatically as part of an auto update system, but the root issue would still persist even if OP disables auto updates.

When apt update fails to complete, it's almost always because of a broken repo somewhere; hence my question about sources.list.

[–] [email protected] 3 points 1 year ago

You may be right.

I'm wondering why apt-get isn't erroring out, though. Usually after enough tries, it just quits, at least in my experience.

Probably best bet, then, is to remove and re-add the third party repos.

[–] [email protected] 1 points 1 year ago (1 children)

Ok, so now I have managed to update my system by killing the process, which releases the lock, and then going on to do normal sudo apt updateand sudo apt upgrade. For the sake of troubleshooting, I tried to add back my third-party repos one by one, and none of them caused any problem.

However, when rebooting, the same problem happens again. In system settings, auto update was already set to "Manually" and offline updates is unchecked. I have not made any modifications to this. I did not have software-properties-kde installed, and it was also not available by running sudo apt install software-properties-kde. It suggested only software-properties-qt instead. So I could not check those settings, but in the 20auto-upgrades file, Update-Package-Lists and Unattended-Upgrades are the only lines present, and both are set to 1.

In addition, everytime I ran sudo apt upgrade, at the end some update related to initramfs fails. My disk is encrypted using cryptsetup, and as I've come to understand, I should be very careful doing anything related to initramfs when that is the case. Here is the output:

Processing triggers for initramfs-tools (0.140ubuntu13.2) ...
update-initramfs: Generating /boot/initrd.img-6.2.0-10018-tuxedo
I: The initramfs will attempt to resume from /dev/dm-2
I: (/dev/mapper/system-swap)
I: Set the RESUME variable to override this.
zstd: error 25 : Write error : No space left on device (cannot write compressed block) 
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-6.2.0-10018-tuxedo with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

Since this is buried quite a long way down into a thread, I will now also update the main post to include this update.

[–] [email protected] 1 points 1 year ago (1 children)

zstd: error 25 : Write error : No space left on device (cannot write compressed block)

What's the output of df -H?

Also, this sounds like it's installing initramfs, which is normally only done when first installing the OS; can we get a list of the packages it's trying to install/upgrade?

[–] [email protected] 1 points 1 year ago (1 children)

This shows up when running sudo apt update after having killed the process that is holding the lock. Where would I find out which packages it is trying to install or upgrade? It does say update-initramfs in the second line there, so I think it's trying to update?

Relevant output from df -H:

/dev/nvme0n1p1           1,1G  945M   24M  98% /boot
/dev/nvme0n1p2           536M  9,3M  527M   2% /boot/efi
[–] [email protected] 1 points 1 year ago

Ok, there's the problem. Your boot partition is pretty much full. You're using partitions instead of lvm, so expanding the partition will be next to impossible; so start looking through /boot for stuff that's safe to delete. It's weird that you have so much stuff in there, I don't think I've ever seen my boot partition go above 250mb used.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago) (1 children)

I'd leave the main sources.list alone, but temporarily move all of the files out of sources.list.d and see if that fixes it.

[–] [email protected] 1 points 1 year ago

As added as an edit in the original post, I managed to update my system by killing the process, did an upgrade with only the original sources.list and then added back the third-party repos one-by-one. None of the third party repos caused any problem when adding them back. Problem persists upon reboot. There seems to be an issue with updating initramfs at the end of every sudo apt upgrade, but that shouldn't run on boot, so I don't think that can be the cause?

[–] [email protected] 11 points 1 year ago (1 children)

I haven't used Tuxedo, but on apt-based distros it's pretty common for an auto-update daemon of some kind to run in the background on startup to either download updates, or at least download package metadata so some UI component can start nagging you to install the updates that are available.

If you wait a few minutes, the download should complete and you can do what you want. You can probably get away with killing it, especially if you use a gentle signal like HUP. I wouldn't risk it though... if you corrupt your package metadata or worse... and actual important package... it can be a significant hassle to clean up the mess. And the cost of waiting 30s-5m and trying again is so low it's hard to beat that as an approach.

If its happening a ton you can probably find and disable the auto-update thing but I don't know what it would be on Tuxedo.

[–] [email protected] 1 points 1 year ago

It persists even after several hours. Never happened before, and I've been using the system for about 6 months now. I posted the output from the process tree below, but I'm not sure if that makes anything any clearer. It's as if it is not receiving any data and just waiting for the servers to respond?

[–] [email protected] 7 points 1 year ago (2 children)

Check if the process is actually running

‘ps aux | grep 1635’

(You should see a line output)

More likely its not running and crashed without cleaning up the lock file, you might want to figure out why from logs and whatnot but its ok also to remove that file and try again

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago)

(You should see a line output)

Just a reminder that the line you see shouldn't be the grep line. You can use 'ps aux | grep 1635 | grep -v grep'

[–] [email protected] 1 points 1 year ago

It's showing up in the list with a bunch of sub-processes that looks like https-requests (I posted the output below). Which logs could I check to learn more about what's happened?

When you say remove the file, do you mean the lock file?

[–] [email protected] 5 points 1 year ago (1 children)

killing the apt update shouldn't break anything since its not installing any packages it's just updating your repos. i would have just killed the pid 1635...also not sure if it would break something but i doubt it...did you dmesg to see what else info you might be able to find out?

[–] [email protected] 1 points 1 year ago

Would there not be a risk of corrupting some of the repo files and dependencies lists by just killing it?

I have checked dmesg after your suggestion, but I did not see anything that tipped me off to what might be wrong. Is there anything in particular I should be looking for?

[–] [email protected] 2 points 1 year ago (1 children)

I would recommend trying out the process tree

Use ps auxf | less

And then / to search for the process. See what child processes are running from that aptget. I wouldn't expect much but that should give you an idea about the safety of killing it

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (1 children)

Here is the output from running that command:

root        1635  0.0  0.0  22208  8928 ?        S    aug.06   0:00  \_ apt-get update
_apt        1654  0.0  0.0  27528  9600 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1655  0.0  0.0  27528  9600 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1656  0.0  0.0  27528  9600 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1657  0.0  0.0  27528  9760 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1658  0.0  0.0  27528  9760 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1659  0.0  0.0  27528  9760 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1660  0.0  0.0  27528  9760 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1661  0.0  0.0  27528  9600 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1662  0.0  0.0  20908  6880 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/mirror+file
_apt        1663  0.0  0.0  27528  9760 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1664  0.0  0.0  27528  9920 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1667  0.0  0.0  27532 10080 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/https
_apt        1669  0.0  0.0  20864  7680 ?        S    aug.06   0:00      \_ /usr/lib/apt/methods/file
[–] [email protected] 0 points 1 year ago (1 children)

Nice, so here's your next step. Do a Google search and see what exactly that file is. Find out it's purpose and that'll give you an idea what's going on, which then allows you to determine if it's safe to kill or not.

I already looked it up, so I want you to do the same and let me know what you find out and what you think they are being used for.

[–] [email protected] 1 points 1 year ago (1 children)

As far as I can tell, these are the methods apt uses to get information from the repositories that is listed within sources.list and within the sources.list.d directory. The number of subprocesses almost matches the number of sources there - in reality there are 14 listed, not 13 as is seen in the ps output. I can find one entry that starts with mirror+file, but otherwise there are 13 https entries. So that last line I am not sure what is doing.

Anyways, it seems to me that it gets stuck somewhere updating the repositories list. Right now, I'm stuck with three questions:

  1. I'm still unsure as to whether it would be safe to kill the process, as I could imagine that having a corrupted depencies files could be really bad?
  2. Also, would killing the process automatically release the lock, or would I need to remove that myself after?
  3. Is there any reason to believe that this would even work, seeing as this happens everytime on boot. I imagine that if I kill the process, delete the lock and try to run sudo apt update I just end up the same place again.
[–] [email protected] 3 points 1 year ago (1 children)

So my theory on what's going on is this.

I suspect that this is an automatically running update to try to help keep your packages up to date, and I think it's getting stuck on a source configured in your /etc/apt/sources.list I'm willing to bet it's likely a source configured to pull from a "CD" which is used during installation and they forgot to disable that one.

You should be able to stop it, it'll still be locked but you'll need find the lock file (I forgot where it's configured) and just remove it with a simple rm, you'll probably need to sudo the rm though.

So my order of operations would be, kill the process, try to rerun the apt update and see if that tells you which repo it's getting stuck on.

[–] [email protected] 1 points 1 year ago (2 children)

I managed to kill the process, and the lock was released with the operation, allowing me to upgrade my system. After this, I moved all external sources out of sources.list.d and upgraded my system, and continued to add them back one by one to see if any of them triggered an error. None of them did.

When I reboot the system, the problem is back. I've added some more details in the original post.

[–] [email protected] 2 points 1 year ago (1 children)

I'm guessing Tuxedo's distro includes a service to automatically update the system or your repos on boot. It seems like the service is failing. I would take a look through your enabled services to see if you can find something related to updating, then you can check the status of the service for logs. You can also try asking Tuxedo support, who may have more insight into this problem.

[–] [email protected] 1 points 1 year ago (1 children)

Aha, that is exactly it! apt-get is called by their driver update tool, Tomte (https://www.tuxedocomputers.com/en/Infos/Help-Support/Frequently-asked-questions/What-is-TUXEDO-Tomte-.tuxedo). I actually would've seen this from the ps auxf | less command above if I had used my eyes a little better, as the top line in the output I pasted above is a sub-process running under Tomte.

Checking systemctl status tuxedo-tomte.service yields the following output:

aug. 08 15:33:56 laptop systemd[1]: Starting Tomte-daemon, finishes tasks that could not be accomplished before...
aug. 08 15:34:06 laptop tuxedo-tomte[1393]: no network found!! some fixes might not be applied correctly
aug. 08 15:34:06 laptop tuxedo-tomte[1393]: systemctlCmd: systemd-run --on-active="30sec" tuxedo-tomte configure all >/dev/null 2>&1

I guess contacting their support directly is the best way to solve this at this point.

[–] [email protected] 1 points 1 year ago (1 children)

Could you send the contents of the service file? It seems like the problem is that the service is running before the network is up and running. It might be resolved by setting the service to run after network-online.service

[–] [email protected] 1 points 1 year ago 
[Unit]
Description = Tomte-daemon, finishes tasks that could not be accomplished before
After = network.target auditd.target
Before = oem-config.service

[Service]
Type = oneshot
ExecStart = /usr/bin/tuxedo-tomte configure all
Restart = no
[–] [email protected] 1 points 1 year ago (1 children)

Oh fuck. That's actually bad. Just to confirm, when you installed the system you did not manually set the size of any of your partitions, correct?

[–] [email protected] 1 points 1 year ago (1 children)

Oh boy, don't like that sound of that...

That is correct. I used WebFAI to install, the automatic installer provided by Tuxedo Computers (https://www.tuxedocomputers.com/en/TUXEDO-WebFAI.tuxedo).

There are three partition on my primary SSD. One that is mounted to /boot (1G), one mounted to /boot/efi (512M) and my main partition that is encrypted. I'm guessing it is one of the two boot partitions that don't have enough space to update initramfs-tools?

[–] [email protected] 1 points 1 year ago (1 children)

Yeah that's what I suspect. Not sure which though.

[–] [email protected] 2 points 1 year ago

/boot is 98% filled, so I am guessing it is that one. I've submitted a ticket with Tuxedo, and hopefully they are able to point me in the right direction.