The reason Photon and the Unity headless server would have to communicate via network is because they would be hosted on different physical machines. That way, the processing of the "Unity only game logic" can be distributed and I can simply add machines for the Unity game server when there's more players. At least as long as the Photon-server is not becoming the bottleneck.(*)
I think a direct connection between clients and game-server would probably complicate matters for me (even though it would probably be desirable in order to avoid the overhead that having Photon as a "proxy" in between will create - but I really want to avoid having much of the "networking information distribution stuff" done on those game servers). That's why I'd move as much logic as possible from the Unity game server over to Photon. So, for most things Photon would be authoritative and just let the server know what happened, like it lets the clients know. Only for logic which is significantly easier to implement directly in Unity, I'd fall back to the Unity game server. I'm mainly thinking of collision detection and movement which involves things like vehicles riding around on moving platforms ... which I just don't feel like implementing from scratch.
Of course, this is all done on the clients anyways - but I just can't trust those clients to do it "right"
I think the greater part of the communication with the Unity game server would also be just one-way from the Unity game server to Photon to the clients: Most of the time, the game server would just send position updates to Photon which Photon then would distribute to the clients so that the main state is in sync for everyone (as the positions are strictly deterministic almost all of the time, I don't really need the positions from the clients except for comparatively rare events). Every now and then, collision events would happen on the game server and go the same route as the position updates (of course, the collision events happen on the clients as well - but the clients wait for confirmation from the server before letting those collision events have any significant consequences).
The only thing I can think of right now, that requires the full path in both directions (client -> Photon -> Unity game server -> Photon -> clients) are turns initiated by the players (left/right ... and some more directions with advanced vehicles
). Those, unfortunately are a rather complex beast because the Unity game server needs to check if a turn was still valid when the message arrives at the server after the vehicle theoretically has already crashed into a wall. Another unfortunate thing about those turns is that players can choose to do many of them in a very short time (usually, they won't - but sometimes, they will).
The good thing is that even in the case of "game server intervention", *most* can be settled between the clients and Photon (client sends "player turned" to Photon, Photon checks if this kind of turn is permitted and distributes the info to everyone else - including the server). Only if after the turn (when the info finally gets to the game server), the server discovers that the player doing the turn must have cheated, it'll let everyone know that this client exploded, using it's full authoritative power
I'm expecting to have to do a rather significant amount of refactoring - this is really where it becomes obvious how convenient Unity's networking is to use. It's nice that I can at least keep the logic that I need inside Photon in C# - but I'll obviously have to pull a lot of existing logic out of MonoBehaviours into "plain classes" that can be used both by Photon (for cheat-prevention and persistence), the clients (for immediate user-feedback) and the game server (for movement and collision "authority"
). For the RPCs I'll probably have to look at each and every single one and decide how to treat them, reducing the number by combining some of them sounds like a very good idea (and should work in many cases). Well ... in the end it'll probably be a much cleaner software-design, so I guess it'll only hurt because of the time this will consume. But if I end up with something that works reliably and scales well, I'll be all happy
(*)Speaking of Photon becoming the bottle-neck: I'm currently thinking of supporting up to 500 concurrent players, for which I would hope to be able to use 3 Unity game servers (so that would be 4 physical machines altogether ... plus db-server and Web-Server, of course ... "aber die stehen auf einem anderen Blatt"
I know this is very hard to say - but would you consider this realistic (purely from Photon's point of view)? I know that you support many more players than that - but keep in mind this is an action-oriented game, not a typical MMO.
Some details I'd consider relevant to this question: Most game-logic checks that would be done in the Photon-server are just simple state-checks, nothing too complex, no AI or anything like that. Any action related persistence is just pushing certain events into the database (asynchronously, in a separate thread with a queue); actual database queries are quite rare.
So I think the "stuff" where it gets interesting will be the position updates: My position updates consist of a Vector3 (= 3 floats) and a bool. They are triggered by the game server and then sent to each player in the given "game group" (as players in one game group could usually see all other players in the same game group when they use they right camera angle, there's no point in trying to optimize by visibility - it simply won't work for this game).
I'd like to do 10 position updates per second - but 5 would still be okay (with too few updates/second, jumps tend to get jerky - but I could easily increase the update-rate for this specific case). With 500 concurrent players, in the worst case 20 positions of vehicles in the game group need to be distributed to 20 clients in the game group, with 25 game groups - with 10 updates/second that would be 100.000 position updates per second. That does sound like a lot to me. But of course, this is a theoretical case that would only happen when everyone on the server is in the largest possible game group and starts a game session at the same time ... nevertheless it would be nice to know that - at least in theory - Photon wouldn't go down in such a case (on "standard" server hardware as you get it with usual root-server hosting packages).
In a more realistic case, I'd have game groups with around 10 players each, and usually not all players are "active" and producing position changes (but everyone needs to receive all position changes - players that have "crashed" and therefore are inactive can follow other players that are still playing). So in that case it would be more like maybe 5 positions to 10 clients with 50 game groups (but with 10 updates/second, that would still be 25.000 position updates per second). As the game design favors collaboration over competition, and some levels require that all players "survive" until the end, it may actually be more realistic to think of 10 positions to 10 clients (so we're back up at 50.000).
In a very realistic case, a few players will be completely idle, some players will be just chatting, others will be joining / leaving game groups and teams, and others playing in different sizes of game groups, some players might even spend their time just following other players (so they need position updates but never create any of their own). But that case is very difficult to put into numbers