I think Kodi is a good choice, but not really sufficient for everything you might want to do IMO.
You could also look into KDE Plasma Bigscreen. It's still pretty rough around the edges, but I think it aims to do what you want.
I am actually thinking about building something similar with different tech, as I'm not satisfied with any of the existing options. I really want something that's primarily controlled via a mobile Web interface like the Kore app for Kodi.