Have you tried Syncthing? It's not the same as Google Photos, but I think it's one of the best file synchronization software yet. I don't use it to sync my photos, though, I use it for syncing my markdown notes.
Selfhosting
All things selfhosting and homelab related Resources: - https://github.com/awesome-selfhosted/awesome-selfhosted - https://github.com/awesome-foss/awesome-sysadmin
I use Synology Photos, has worked quite well, but that's obviously only an option if you have Synology NAS.
Syncthing, nextcloud and photosync have apps you can use to sync. I'm using photosync right now for viewing and face recognition
I use OneDrive for the auto-upload feature. I know it's not selfhosted, but I want an off-site copy of my pictures anyway, and the phone app works really well for auto-uploading. I had previously tried Nextcloud's auto-upload, but many times the uploading was stuck and I had to fix it manually.
I then have nightly rclone jobs to download the photos from OneDrive and store them locally. To view them locally, I use Photoview, which works really well with an existing directory structure.