Some technical decisions that a lot of people seem to miss:
- It is impossible to make TCP reliable, even if you encrypt everything. Random middleware will spam you with RSTs and break your client-server connections for no reason, and while you can tell your server firewall to ignore them, you can't expect clients to.
- The internet backbone is designed mostly around TCP. Most UDP-based protocols do not gracefully degrade. Thus, your only reasonable choice is the one they have to deal with: QUIC/HTTP3. Use an existing library, don't roll your own.
- Make sure your low-level UDP library is using
sendmmsg
/recvmmsg
, notsendmsg
/recvmsg
. Without that you often get performance worse than TCP. A lot of libraries are lacking here! Some OSes need other methods, or might not provide any way to get good performance; I'm only familiar with the Linux solution. - Define your (application-layer) packet layout using data (fed to some kind of code generator), not manually-written code. Make sure your basic packet structuring code doesn't (i.e. make every packet declare its size, even if it's logically fixed size and you're absolutely sure that will never change). If you need multiple variable-sized fields, stick them all at the end and use indirection. To ensure compatibility, always ignore extra data and pad short packets with zeros (flexibility only at the end suffices). There are a lot of terrible serialization formats out there, even the ones used by major companies.
- Write for the IPv6 stack only, and turn the
IPV6_V6ONLY
sockopt off to transparently handle IPv4 as if it were IPv6 (although it defaults to off, the local administrator can change that, so always take care of it in your code). - Never have clients connect to multiple servers during the course of a connection (you can separate real HTTP server for ahead-of-time downloads). You should write a proxy-like server that forwards packets appropriate to the lobby, the actual game server, etc. and only let clients even know about that proxy.
- Note that internet routing is bad in parts of the world, so forward-only proxies are sometimes useful too!