this post was submitted on 15 Mar 2024
1 points (66.7% liked)

The Lemmington Post

20 readers
1 users here now

founded 6 months ago
MODERATORS
 

I have several file systems mounted on /mnt/hdd0, /mnt/hdd1, /mnt/hdd2, etc. on my Manjaro Linux system. Let's call these file systems fs0, fs1, fs2, and so on. Currently, all of these file systems have the same directory structure and contain files.

My goal is to reorganize and redistribute all the files across the drives based on the alphabetical ordering of their relative file paths, following these rules:

  1. The files with the earliest alphabetical relative paths should be moved to the first mounted file system in alphabetical order (e.g. /mnt/hdd0) which contains the appropriate folder structure.
  2. The files with the latest alphabetical relative paths should be moved to the last mounted file system in alphabetical order (e.g. /mnt/hddN) which contains the appropriate folder structure.
  3. The remaining files should be distributed across the remaining mounted file systems (/mnt/hdd1, /mnt/hdd2, etc.) in alphabetical order based on their relative paths.

The process should fill up each file system in alphabetical order (/mnt/hdd0, /mnt/hdd1, etc.) until the given free space threshold is reached before moving on. While reorganizing, I want to ensure each file system maintains a specified amount of free space remaining. If a file system does not have enough free space for the files being moved to it, I want to move enough existing files from that file system to another file system with the most available space to make room.

I need step-by-step instructions, including commands, scripts, or tools in Manjaro Linux, to automate this entire process of reorganizing and redistributing the files while following the rules mentioned above.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] [email protected] 1 points 6 months ago* (last edited 6 months ago) (1 children)

To accomplish this task, we can use a combination of Bash scripting and various Linux utilities such as find, mv, du, and sort. Here's a step-by-step approach:

  1. Create a bash script file, let's call it reorganize.sh.
#!/bin/bash

# Define the mount points and the desired free space threshold
declare -a MOUNT_POINTS=("/mnt/hdd0" "/mnt/hdd1" "/mnt/hdd2" ...)
FREE_SPACE_THRESHOLD=20  # In gigabytes

# Function to get the available space on a file system
get_available_space() {
    local mount_point="$1"
    df --output=avail "$mount_point" | tail -n 1 | awk '{print $1}'
}

# Create an associative array to keep track of the available space on each file system
declare -A AVAILABLE_SPACE
for mount_point in "${MOUNT_POINTS[@]}"; do
    AVAILABLE_SPACE["$mount_point"]=$(get_available_space "$mount_point")
done

# Function to find the mount point with the most available space
find_mount_point_with_most_space() {
    local max_space=0
    local max_mount_point=""

    for mount_point in "${!AVAILABLE_SPACE[@]}"; do
        if [ "${AVAILABLE_SPACE[$mount_point]}" -gt "$max_space" ]; then
            max_space="${AVAILABLE_SPACE[$mount_point]}"
            max_mount_point="$mount_point"
        fi
    done

    echo "$max_mount_point"
}

# Function to redistribute files based on alphabetical order
redistribute_files() {
    local source_dir="$1"
    local target_dir="$2"

    # Get a list of files sorted alphabetically by relative path
    local files=$(find "$source_dir" -type f | sort)

    # Move files to the target directory
    while read -r file; do
        local relative_path=$(realpath --relative-to="$source_dir" "$file")
        local file_size=$(du -k "$file" | cut -f 1)

        # Update available space on source and target directories
        AVAILABLE_SPACE["$source_dir"]=$((AVAILABLE_SPACE["$source_dir"] + file_size))
        AVAILABLE_SPACE["$target_dir"]=$((AVAILABLE_SPACE["$target_dir"] - file_size))

        mkdir -p "$target_dir/$(dirname "$relative_path")"
        mv "$file" "$target_dir/$relative_path"
    done <<< "$files"
}

# Function to move files from one file system to another if the free space is below the threshold
move_files_if_needed() {
    local source_dir="$1"
    local temp_dir=$(mktemp -d --tmpdir="$(find_mount_point_with_most_space)")

    if [ "${AVAILABLE_SPACE[$source_dir]}" -lt "$((FREE_SPACE_THRESHOLD * 1024 * 1024))" ]; then
        # If the available space on the source directory is below the threshold,
        # move the files to a temporary directory on the mount point with the most available space
        redistribute_files "$source_dir" "$temp_dir"
        rm -rf "$source_dir"/*
        # Move the files back to the source directory, preserving the alphabetical order
        redistribute_files "$temp_dir" "$source_dir"
        rmdir "$temp_dir"
    fi
}

# Main script
for ((i=0; i<${#MOUNT_POINTS[@]}; i++)); do
    local mount_point="${MOUNT_POINTS[$i]}"
    # Check if the current mount point needs to be cleared to meet the free space threshold
    move_files_if_needed "$mount_point"
    # Redistribute files from the current mount point to the remaining mount points in alphabetical order
    for ((j=i+1; j<${#MOUNT_POINTS[@]}; j++)); do
        redistribute_files "$mount_point" "${MOUNT_POINTS[$j]}"
    done
done
  1. Save the script file and make it executable:
chmod +x reorganize.sh
  1. Run the script:
./reorganize.sh
[โ€“] [email protected] 1 points 6 months ago* (last edited 6 months ago)

This wouldn't work because of the way it deletes everything when making space on a filesystem. It should just move enough files to make space for the ones it's moving there. Even if this script worked, I would need a lot of testing before I were confident using it to reorganize my actual data.

Also, to copy the directory structure I use:

rsync -av -f"+ */" -f"- *" "/mnt/hdd1" "/mnt/hdd2" &