Linux

source ~/.bash_history

Perhaps not the same definition of "broken" that you're looking for, but when I first started using Linux, I was using Kubuntu as my first distro have some brief experimenting with Manjaro.

Anyway, back then, I for some reason had the Skype snap installed. Can't recall why I had it to begin with, but I decided later on that ofc I didn't need Skype, and of course uninstalled the snap.

A few days later, I was met with some storage issues, where I had a limited amount of storage left on my SSD. I'm sitting there a little confused since I swore I was using less storage, but I did a thorough cleaning of my computer by deleting files I didn't necessarily need, and uninstalling any programs that I hardly ever used. That seemed to do the job, even if it was less storage space...

Until the next day, when the storage was full again. After getting some help from someone, I found that Skype, despite being uninstalled, was still running in the background, and found that there were residual files. The residual stuff running in the background was trying to communicate with what I had uninstalled, and logged multiple errors per second in a plaintext file that ended up being 176GB.

Whether I did something wrong or if there was something up with the snap, I still don't know as this was over a year ago and I was still learning the ropes of Linux at the time.

Many many years ago I wanted to clean up my freshly installed Slackware system by removing old files.

find / -mtime +30 -exec rm -f {};

Bad idea.

Not me, but one I saw... dude used chmod to lock down permissions across the board... including root... including the chmod command.

"What do I do?"

🤔

"Re-install?"

Tried to convert Ubuntu to Debian by replacing the repos in sources.list and apt dist-upgrading. 💣 Teenagers...

One that I can remember many years ago, classic trying to do something on a flash drive and dd my main hdd instead.

Funny thing, since this was a 5400rpm and noticed relatively quick (say 1-2 minutes), I could ctrl-c the dd, make a backup of most of my personal files (being very careful not to reboot) and after that I could safely reformat and reinstall.

To this day it amazes me how linux managed to not crash with a half broken root file system (I mean, sure, things were crashing right and left, but given the situation, having enough to back up most things was like magic)

First, the classical typo in a bash script:

set FOLDER=/some/folder

rm -rf ${FODLER}/

which is why I like to add a set -u at the begining of a script.

The second one is not with a Linux box but a mainframe running AIX:

If on Linux killall java kills all java processes, on AIX it just ignore the arguments and kill all processes that the user can kill. Adios the CICS region 😬 (on the test env. thankfully)

sudo rm -f /lib /usr/share/backup/blah blah.tar.gz

Note the space.

I thoroughly backup up my slow nvme before installing a new faster one. I actually didn't even want to reuse the installation, just the files at /home.

So I mounted it at /mnt/backupnvme0n1, 2, etc and rsynced

The first few dry runs showed a lot of data was redundant, so I geniously thought "wow I should delete some of these". And that's when I did a classic sudo rm -rf in the /mnt root folder instead of /mnt/dirthathadthoseredundantfiles

I can't even remember how I did this, but overwriting the partition table on the main production server at our small startup (back when "the server" would usually live on the premises of the startup). I remember my boss starting to hyperventilate from panic while I reconstructed it from memory / notes, and all the filesystems came back and he calmed down.

Same job, they gave me a little embedded-systems unit for me to use to build a prototype on. I hooked it up, nothing worked. I brought it back to them.

Hey, this one doesn't work.

Huh... that's weird, it was working before. Did you break it?

I don't think so. Can I have one that works?

They literally told me, as they were handing me the second one: Okay, here's another one. Don't break it.

I figured it out literally seconds after breaking the second one... I was hooking it up to 12 volts of power when it needed 5. Second dead computer. Explaining that and that I needed a third one now was fun.

Not quite catastrophic but:

I'm in the process of switching my main server over from windows to Linux

I went with Deb 12 and it all works smoothly but I don't have enough room to back up data to change the drive formats so they're still NTFS. I was looking at my main media HDD and thought "oh, I'll at least delete those windows partitions and leave the main partition intact."

I found out the hard way that NTFS partitions can't just reclaim space like that. It shuffles all the data when you change the partition. It's currently 23 hours into the job and it's 33% done.

I did this to reclaim 30 MB of space on a 14 TB drive.

CTRL-C-ing apt because it looked stuck for more than 10 minutes. I don't recommend doing it.

Tinkering is all fun and games, until it’s 4 am, your vision is blurry, and thinking straight becomes a non-option, or perhaps you just get overly confident, type something and press enter before considering the consequences of the command you’re about to execute… And then all you have is a kernel panic and one thought bouncing in your head: “damn, what did I expect to happen?”.

Nah, that's when the fun really starts! ;)

The package refused to either work or install complaining that the version of glibc was incorrect… So, I installed glibc from Debian’s repos.

:D That one is a classic. Most distributions don't include packagers from other distros because 99% of the time it's a bad idea. But with Arch you can do whatever you want, of course

My two things:

• I've heard about some new coreutils (rm, cp, cat... this time the name really fits the contents :D) and I decided to test it out. Of course it was conflicting with my current coreutils package and I couldn't just replace it because deleting the old package would break requirements. So without thinking I forced the package manager to delete it "I'll install a new one in just a second". Turns out it's hard to install a package without cp, etc :D
• I don't remember what I was doing but I overwrote the first bytes of hdd. Meaning my partition table disappeared. Nothing could be mounted, no partitions found. Seemingly a brick.
  Turns out, if you run a rescue iso, ask it to try and recognize partitions and recreate the table without formatting, Linux will come back to life as if nothing happened

First time trying Linux I went with an arch install because I Googled "best version of Linux" and went with arch. Followed a guide to the point of drive formatting and I decided to go with a setup with drive encryption. I didn't understand what I was doing, ended up locking myself out of my hard drives and couldn't get windows to reinstall on them. I used a MacBook for a week until I installed Ubuntu and managed to wipe and reset my drives and reinstalled. Needless to say I am going to read up a little more before I try that again.

Before installing Arch on a USB flash drive, I disabled ext4 journaling in order to reduce disk reads and writes, being fully aware of the implications (file corruption after unexpected power loss). I was confident that I would never have to pull the plug or the drive without issuing a normal shutdown first. Unfortunately, there was one possibility I hadn't considered: sometimes, there's that one service preventing your PC from turning off, and at that stage there's no way to kill it (besides waiting for systemd to time out, but I was impatient).

So I pulled the plug. The system booted fine, but was missing some binaries. Unfortunately, I couldn't use pacman to restore them because some of the files it relied on were also destroyed.

This was not the last time I went through this. Luckily I've learned my lesson by now

Found out the hard way that if you edit /etc/sudoers with anything other than visudo you best be absolutely sure the syntax is correct, otherwise sudo will refuse to read it and you'll be locked out.

Also learned to add -rf to the rm command at the end, after I re-read it to make sure it does what it should do. Something like rm /path -rf instead of rm -fr /path. That protects you from your fat fingers hitting the enter key half way through.

fstab bind mount for /home that I misspelled, so I couldn't login as myself.

fstab external hdd mount that didn't have ignore flag so PC would pop if I booted while unplugged

Accidentally booting windows after a year and it overwrite my EFI boot entry.

The best I've see however was an acquaintance who accidentally set perms to own user on /usr/bin

So everything went from root:root to user:user which removed all the SUID/SGID bits as well so a bunch of bins broke lol.

Believe it or not, it was actually fairly easy to fix with chmod and chown

stupid was when I wanted to test Linux Mint on an external SSD, and didn't check that the bootloader wasn't going to overwrite my internal drive's.

So anyway I'm running Linux Mint now.

I uninstalled Python.

I was playing around with Pygame of all things, and it wasn't behaving as the (apparently out of date) documentation was saying it should, so I figured I'd just uninstall and reinstall Python.

EVERYTHING borked. APT wouldn't even work.

I've done my plenty of stupid stuff, from dd disks I was using to forcefully uninstall dependencies of the package manager. But the one that takes the cake for me happened back in 2012, I was working at a research lab in the university and was sharing a computer with another intern. That other intern used Gentoo and so we agreed that the machine should be Gentoo, I've installed it at my house on my PC and got comfortable with it before we shared that computer. One thing that I learnt when installing Gentoo is that the /dev folder is created on boot, you don't populate it when installing, instead you mount the one from the host system you're using to install.

The computer had an issue with a device, can't even remember what it was, so I thought I'll run rm -rf /dev that should take care of the issue and after a reboot it will be repopulated... It might have worked, but what I actually ran was rm -rf /etc.

Years ago I was dual-booting with Ubuntu just to try out whatever this Linux thing was that all the nerds were talking about. Liked it and played around with it, but for whatever reason I wanted to go back to just Windows, I needed the space I had partitioned off or something, can’t remember why. So I just uninstalled or deleted the bootloader somehow (maybe I just deleted the Linux partition and expected the space to clear up like normal).

Go to restart the computer… oh shit. Ohshotohshitohshitohshit.

Deleted my entire efi partition while trying to install some grub themes.

And then my backup didn't work when I tried to restore it.

I have pretty colours now though, so it was all worth it :)

Just straight up overwriting boot sector and superblock of my hard drive thinking it's the USB drive.

Udev tried to warn me, saying there's no permission, and I just typed sudo without thinking.

Then after a second I remembered USB block devices are usually writable by users, but it was too late.

I don't know if that counts, but on a fresh default Debian Stable system, my cat walked across the keyboard and the DE crashed.
I could still switch to another TTY and reboot via command line.
After the reboot, I was greeted by a blinking cursor and nothing else. Had to reinstall.

My own classic was fiddling with the nvidia PRIME config to try and get rid of some very mildly irritating screen tearing. No graphics output at all. Now this is fixable of course, but it's a pig.

And I'd decided to do this 2 hours before an incredibly important progress review meeting for my PhD.

Got it back with about 10 mins to spare and decided just to leave the driver config alone after that.

Bonus round

Also a friend managed to bork his ubuntu 16 laptop by trying to switch from unity to gnome and ending up with sort of neither. That was reinstall territory right there.

I wanted to use fio to benchmark my root drive. I had seen a tutorial saying that the file= parameter should point to the device file, so I pointed it at /dev/sda. As you might expect, the write test didn't go so well.

Learned about the importance of trailing slashes in rsync by using the -delete flag.

I have a stupid one, but far from funny.. I've been using and building computers for a very long time so I'm far from a noob, but I'm still quite cautious, bordering on paranoid, so I like to unplug all other drives when re/installing an OS just to avoid stupid mistakes. I go through the installer on the livecd, there's only one drive to choose from so I don't think much about it, select that it should erase everything, I set up the new partition structure, and start the process. After about a minute I begin wondering "why is it taking so long?", and "what is that ticking noise? SSDs shouldn't be making any sounds when written to", when I realize that I had unplugged the wrong drives and that I was currently overwriting my main storage drive. Of course I had backups of the most important things like photos and code, though not really synced for a couple of months so I lost some stuff permanently.

Somehow convinced a person to run sudo chmod -x /usr/bin/*

I don't remember the exact command so it could be a bit different but it did the job. It was a fun evening.

Backed up the whole disk image to an external drive because I didn't have time for a proper backup but knew I would need some of those files later.

Installed a fresh new OS on the same disk, used it for a couple of months.

Needed to make some space on the external drive I had the backup on so I'll just delete the backed up system files from it.

cd /mnt/external_drive

rm - r /usr /boot ...

As you can probably see, a fresh new install was happening again

This was pre-linux for me but something you can still do in most distros so I think it's a valid story.

In 1999 I was using Napster on computer running MS-DOS. I was 12 years old and an aspiring open media enthusiast/stupid script kiddie. I was using the file explorer interface in Napster and accidentally gave access to my entire C drive. I also had opened ports to share certain media and to fuck with my friends using daemon tools (back then you could do stupid stuff like control a friend's desktop with certain versions of daemon tools). Immediately I started receiving packages called things like "sleep.tight.tiny.mite" and I knew I was fucked so I clicked in the Napster interface and clicked "delete" and deleted my entire active drive.

I panicked and installed the only operating system we had which was a random copy of Red Hat. When my dad came home I pretended like it had always had Linux on it. I do think he was more impressed than mad.

I've broken systems far too many times in the last 24 years, since Mandrake 6.x, to count:

• I've dd a disk or more
• I've rm *
• I've chmod
• I've brought down the network, with every intention tar it would come back - on a remote box
• I've failed to RTFM far too many times