cross-posted from: https://fanaticus.social/post/154152
Hello everyone! I'm back from vacation (was gone the past two weeks) and am ready to dive back into work on the game bots.
I just wanted to give everyone a quick update. You can check out my previous update thread here.
Status
- All 30 teams have their own game bots configured and activated
- 29 of the game bots run on the same server as fanaticus' lemmy code (big beefy box) with Cerevant running the phillies bot.
- The post limit has been increased in 0.18.1 to 50k from 10k so we no longer have a truncation issue
- The sidebar bot has been ported to lemmy but unfortunately is a bit more limited that we are used to (it only updates the standings of a single division) -- check out what it looks like here
- Cerevant has [identified an issue](Reported: https://github.com/LemmyNet/lemmy/issues/3544) where pinned posts are not being federated correctly on foreign instances
- There is an issue with the game bots that has occasionally taken down the fanaticus servers (I'll discuss this more below)
Current Issue
Game Bots Out of Control
There is an issue with the game bots that has occasionally taken down the fanaticus servers
Game bot will periodically hit a 503 while updating one of its game threads and enter a loop where it continuously attempts to update the post and fail. A bot that enters this state almost immediately spikes one of the server's four CPUs to 100%, then very quickly fills up the shared disk with the same log (GBs of logs). The GBs of logs produced by these rogue bots fill up the server and because the game bots share the same server as fanaticus, they cause the lemmy server to crash.
While I'm not entirely sure what causes this issue, my current hunch is that it's related to a rate limit either with our lemmy or nginx configuration.
When we first started bumping up to the max CPU & memory on our server I upgraded it to a beefy dedicated machine with a quad core and 8GB of RAM. I did not upgrade the shared disk space at that time. The server size was total overkill but I wanted to run the game bots on the same machine.
I've gotten a lot of questions regarding why the game bots were turned off last week and this is the reason. I had to basically babysit them and restart any that ran off the rails and because I was on vacation, I couldn't investigate the bug to fix it or babysit the bots.
Possible Fixes
These are some options I'm tossing around to fix this issue. They're not necessarily going to happen but I'm writing them here publicly so you all know that I'm working on it and to get some public comment.
- Fix the bug, obviously :)
- Move the game bots to another server, separate from fanaticus^1^
- Create some sort of system that detects the bots are in a runaway state, alerts me, and/or programmatically restarts the bad bots
This is a top priority for me now that I'm back from vacation and I will be working on this starting today.
Future Work
- Add more features to the sidebar bot to achieve feature parity with the former Reddit game bots (team/league leaders, multiple division & playoff standings, live league scoreboards)
- Customizable periodic threads (e.g. a daily "Around the Horn")
- Closer to NBA/NFL/NHL season, we'll port those game bots to lemmy
Contributing
First off, huge shout-out to Cerevant who has been working with me porting the bot to lemmy. He's done amazing work, added new features, fixed multiple bugs, and has been overall a great person to work with! Thank you!!
Another shout out to todrob99 for creating the original redball bots. He's been very responsive and encouraging as Cerevant and I work on the bots.
For any of you who want to contribute, whether it's coding, creating features requests, reporting bugs, or reviewing PRs, you can find my fork of the redball app here.
We are also discussing the work in the #lemmy channel of todrobb's Discord channel here.
Finally, another way to contribute is to take over your community's bot and run it yourself! The code is open source and I can help you set it up on your own server. That would make it easier on me so I don't have to run all the bots and you can configure it to your community's liking.
TL;DR bots are live, a little buggy, but are being improved daily!
^1^ This is definitely the correct thing to do but I'm hesitant to do this right away. I'm not a server admin by trade and this means 1) Extra overhead ($$) and maintenance 2) That I have to take time away from improving and extending the bots' functionality to do server admin work