cross-posted from: https://lemmy.daqfx.com/post/24701
I'm hosting my own Lemmy instance and trying to figure out how to optimize PSQL to reduce disk IO at the expense of memory.
I accept increased risk this introduces, but need to figure out parameters that will allow a server with a ton of RAM and reliable power to operate without constantly sitting with 20% iowait.
Current settings:
# DB Version: 15 # OS Type: linux # DB Type: web # Total Memory (RAM): 32 GB # CPUs num: 8 # Data Storage: hdd max_connections = 200 shared_buffers = 8GB effective_cache_size = 24GB maintenance_work_mem = 2GB checkpoint_completion_target = 0.9 wal_buffers = 16MB default_statistics_target = 100 random_page_cost = 4 effective_io_concurrency = 2 work_mem = 10485kB min_wal_size = 1GB max_wal_size = 4GB max_worker_processes = 8 max_parallel_workers_per_gather = 4 max_parallel_workers = 8 max_parallel_maintenance_workers = 4 fsync = off synchronous_commit = off wal_writer_delay = 800 wal_buffers = 64MBMost load comes from LCS script seeding content and not actual users.
Solution: My issue turned out to be really banal - Lemmy's PostgreSQL container was pointing at default location for config file (/var/lib/postgresql/data/postgresql.conf) and not at the location where I actually mounted custom config file for the server (/etc/postgresql.conf). Everything is working as expected after I updated docker-compose.yaml file to point PostgreSQL to correct config file. Thanks @[email protected] for pointing me in the right direction!
For this purpose I would search for Linux specific thing - a RAM based storage that receives data writes and even flushes/fsycs and then lazily writes them to HDD/SSD. There would be problem in case of power loss, but the gained performance... Unfortunately, I don't know any such tool.
I guess I could just use rsync to periodically sync RAM drive to disk or just rely on backups to restore running state in case of failure or just a restart. On a mostly idle server with few users this could probably work, but I don't think I'm quite ready for such a risky setup. Server is still perfectly usable at 15% iowait - i was just hoping I could reduce it with mechanisms built into PSQL. Appreciate the suggestion though.
Edit: I just want to say this would be an awesome feature for Docker/Podman to have an in-memory volume that syncs to disk either periodically or on container termination as an option.