Honestly as far as NA server goes I just think you guys are seeing more opposing ships that have similar clip shot protection as your own and it is ramping up the graphics load when light sources from explosions and turret fire interact with all those polys.
Vertex shading may also be an issue but Im relatively uninformed on the games back end.
There are many ways to fix this problem, but we need more data.
First we need to know if the damage is server authoritative. This is, only the server can state which block got hit.
If client data is used in some way, things gets much worse. Due to high velocity of the weapon and frameskip produced by overload, there is the bullet thru paper efect.
Still, we could counter that scenario. When a block is hit, if the block is surrounded by other blocks then using the entry angle find out the nearest block in the opposite direction that is open to space (no contiguous blocks), then correct the hit position to that block.
Hm…this would work if the shot is registered within a thick group of block. Still doesn’t solve the bullet thru paper problem. more to come…
Ok try #2.
When shot hits, somehow you get the coordinate of the entry point of this shot in the ship’s bounding box. Then raycast from there towards the coordinate of the registered shot, translate the damage to the first block hit by the ray.
Hm… ok those were some quick replies from work…
We need to know how the hit detection is done and then we could do some tests and find the cause and solution.
Hit Detection is very obviously client-side. Everything is almost done clientside, otherwise you would see far less of the lag-shots completely missing outer-layer blocks and detonating against inside-blocks.
Client A says “i shot this gun at this time.” and the server sees this and does calculations.
Client B says “I see you shot this weapon at THIS time”, however you’re waiting on the server to send you information from client A regarding where the shot is supposed to go to… if there is even brief, 1-5ms discrepencies, the receiving client of said shot can read the bullet is “passing through” as it has a period of time where a block should have been hit, and the projectile passing through a surface onto the next (hence the anti-clipping armor)
Someone correct me if i’m wrong, but that’s my understanding of client-side and Unity.
Doctor Discord and myself from NeR will attend this
Short and sweet. Gives a really good understanding of the issue.
1 Like
Bearing in mind that I haven’t seen the Empyrion source code and am theorizing solely on the behavior of the game and my experience as a Unity dev myself… take all of this with the possibility it’s wrong. This is basically educated guessing. Also, fair warning, this is going to get pretty technical and I’m not going to explain everything in total detail.If you aren’t too familiar with unity or game physics, it may not make much sense. Also, it’s probably a good idea to watch Xanif’s video first, he did a great job illustrating what I’m going to delve into.
There’s a few factors probably at work here. The game, intelligently, removes block faces where it can to reduce poly counts. This is a basically good thing, because even modern graphics cards can only take so much. It ALSO performs these culls on collision meshes-- that much is clear if you watch Xanif’s video when he’s shooting inside the hull blocks. If collision wasn’t being pruned the same way the render tris are, the shots would impact on empty space.
This actually isn’t the core issue, probably-- BUT it’s important to know later. The core issue is most likely that the game uses the default collision mode settings, which is for discrete collision checks. What this means is that the engine only checks collisions at a given object’s current position each frame. This is (relatively) fast and usually accurate enough for most objects.
Normally, this is fine-- even shots tend to be collision-checked with the kind of frequency where it’ll for the most part work as expected, and you’ll almost never see this kind of lagging-shot thing happen in single-player, because the travel of a bullet between frames isn’t very long.
However. bullets are a notable edge case, because they tend to be small collision shapes and travel very quickly. So there can be large gaps between frames along which no collision will be detected. The solution to this issue is Continuous Collision Checking. CCC checks several locations between the objects current and last-frame positions (making it notably slower), so for high-speed objects, it prevents missed collisions for the most part. It’s generally held to be good practice to keep fast weapon shots either hitscan (raytraced) or continuous-checked. It’s important to note that the collision errors are entirely dependant on the distance an object travels between frames, which is a function of both the object speed and the time between frames. For an online game, positions can lag, so shots and whatnot can jump around (if you’ve watched ships flying on HWS you’ll often see them “jumping” along, this is basically what I mean here.
So, for those following along so far-- what happens in practice is the server gets laggy in a PvP fight. Bullets start jumping long distances between frames because of the latency. A bullet will be outside the ship in one frame, and then inside the ship’s armor in the next frame. If there’s no overlap with the outer polys of the ship, it doesn’t collide with that first block, and continues on. Because the game smartly combines blocks to avoid rendering tris it can’t possibly see (and does the same for collision meshes to reduce the complexity of collisions), there’s nothing inside that first layer to collide with, and it rolls on straight to the core or other components, or otherwise the inside of the ship. Boom.
The only way the players have to deal with this is by essentially breaking the system (sort of). Certain blocks, because they don’t run right up to the edge of the bounding cube that defines a block, don’t get all of their faces culled the way a cube does. So if you fill in the ship’s armor with these kinds of blocks, You end up with internal polygons, and therefore internal collision. This does not always prevent a shot from lagging through to the core (An LBPer was flying one of my new designs as a test recently in a very large fight and it was lag-shot cored almost instantly through seven layers of armor) But it drastically increases the likelihood that a shot that would otherwise skip through armor impacts the armor instead, and therefore protects the vital areas of the ship. The issue here is that if you don’t do this, then in anything but a 1v1 or 2v2 or so you’re probably going to get cored in a salvo or two due to shots jumping through the armor collision.
This could be done using actual gaps in the ship, BUT that’s very space-inefficient compared with using what we call “whitespace blocks”. The technique is essentially the same.
One problem is, of course, that you have to be very choosy about your block choice. It’s a practical tradeoff between coverage and protection for your ship and increasing poly counts affecting performance, and also ease of building. Some blocks are better for poly count or coverage but are harder to use effectively. This is why though 4-way round connectors are very good protection, some more experienced builders avoid them-- the connectors have a very large poly count compared with other options that are almost, but not quite, as good, and with experience, you can get similar results from other blocks (my personal favorite is slopes angled in various ways, but I digress).
Okay, so. Solutions. Personally, if it’s me, and if Continuous collision checks aren’t turned on for weapons fire (it’s an option on the Rigidbody component), I’d turn that on for all weapon bullet objects and see how it goes. While continuous does involve a larger amount of processing (because it checks each bullet more than once), the overhead to do so is probably offset by the reduced complexity of the collision mesh when you don’t need to introduce whitespace to break up each armor block. You’d have to try it and see how it performs, though. My gut says reducing the polys you have to check against on a large ship by a hundred thousand tris or so during the narrow-phase FAR outweighs the extra time spent doing the collision sweeps, and broad-phase shouldn’t be affected much. Speaking of the collision sweeps for continuous mode, Bullet collision shapes should all be simple spheres if they aren’t already-- you don’t need more complexity than a sphere and sphere collision checks are much faster than basically anything else. It doesn’t matter what the shots actually look like.
If that works, we can all go back to solid armor, which should, in theory, speed a lot of other stuff up since the face-reduction will actually be in effect-- because if collisions are more reliable, “whitespacing” armor will actually be sub-optimal due to reduced coverage compared with a full cube.
Side-note: I took one of our ship designs (not a PvP ship) and went through it because I knew over the months it’s had compartments walled off but left hollow and unused. I filled in all those hollow gaps with solid cubes (removing whitespace, essentially). With the 5.1 size calcs, that ship when from Class 6 at 160K-ish tris to a class 4. It just had a lot of little unused pockets of empty space because that happens sometimes when you keep repurposing an old ship. The point to this part is the poly saving can be pretty substantial.
Whew. Okay. That was a lot to go through. But it’s just about everything I personally know about this phenomenon. We’ve spent a lot of effort, resources, and time understanding, researching, and experimenting with it, because building effective ships is such a cornerstone of the PvP game.
Again, I haven’t seen the code. But this is what’s consistent with my experience working in Unity for a long time and general knowledge of game physics, as well as what we’re seeing happen in-game on and off the server. Honestly HWS is probably the only server with enough load to get this to happen like this-- we’ve tried doing weapons and armor testing on our own server, but we never get enough latency there to get results consistent with HWS.
EDIT: This all supposes the game isn’t using continuous checks. If it is… something is seriously messed up in some other fashion. Could be conflicting gamestates among clients or something, hard to say not knowing exactly what the network setup is in term of authority.
7 Likes
“So, for those following along so far-- what happens in practice is the server gets laggy in a PvP fight. Bullets start jumping long distances between frames because of the latency. A bullet will be outside the ship in one frame, and then inside the ship’s armor in the next frame. If there’s no overlap with the outer polys of the ship, it doesn’t collide with that first block, and continues on. Because the game smartly combines blocks to avoid rendering tris it can’t possibly see (and does the same for collision meshes to reduce the complexity of collisions), there’s nothing inside that first layer to collide with, and it rolls on straight to the core or other components, or otherwise the inside of the ship. Boom.”
This right here. This is all you need to read.
This is everything.
This right here. This is all you need to read.
This is everything.
For most, yes. The rest is supporting detail for the most part, more important for devs. 
[quote=“Zizi, post:50, topic:3692”]
Continuous Collision Checking
[/quote]You mean CCD (Continuous Collision Detection). I kinda assumed they use that. Mostly because of the FPS issues most of the time. Without CCD this would be horror.
Here comes a bit of theorycrafting (we don’t have source code):
One of the biggest problems (as I mentioned already somewhere else) is that calculating so many IMPORTANT things client-side causes a lot of dupes, exploits, hacks (EAC prevents this to some degree) and whatnot. And could also cause this. The problem is that doing this server-side would require much bigger servers but would also dramatically increase stability and security. (We still have dupe exploits from 4.0 that the devs can’t fix so far)
Also there is lag. What if the CCD checks that it’s new position is just fine? And then suddenly you get synched again with the server and the enemy ship warps 80m forward (not uncommon) and is now put exactly on the spot where one of your old projectiles used to be… It will now hit the ship wherever that projectile is right?
Unless the devs added special code to prevent such cases (I doubt that).
So not only do you need CCD. Because so much is client-side the projectile must calculate and take the history of its’ trajectory and such into account as well. Because what if another ship flew through the projectile after a synch but the projectile is already passed that point? That’s no good either.
The devs CAN NOT RELY on UNITY CODE or CCD or other basic algorithms to failsafe this all and must code it all by hand. Unless they go for server-side code but I think we all already found out that a lot is client side.
Also in 4.0 I noticed that some projectiles did 0 damage on hit for some reason (as if the projectiles did not even exist). Not sure about 5.0. I have also seen projectiles going straight through the planet and other weird stuff without having bad FPS/lag.
What I think PVP needs (aside from balancing):
- More optimized networking code
- Various failsafes
- Big FPS improvements (not just a limit on polygons for ships, that is just a dirty temporary workaround)
- Proper turret targeting (core-targeting is also a temporary workaround but okay better than 4.0 but still exploitable like I mentioned in 4.0 already). Yes this requires a lot of optimization and thinking about what ‘proper’ is. This is not easy.
With imo the 3 hardest parts yet to come:
- Networking
- Round planets (right now ship control on planets even seem to have temporary code as I proved in 4.0)
- FPS
Thank you very much!
I just skyped with the lead dev regarding this and it helped a lot!
He painted me the logic how it works now and Zizi gave him some things to (re)think about.
Just quick: CCC is not used but the rigidbody component is something to test with. If all would work then the first example of Xanifs video would be the best armor at all. But this is the problem now for him to reproduce.
“Latency” is not something easy to reproduce and not so cool for him to work with.
But in a separate branch he will test 2 things regarding CCC and RC. If he can reproduce some usecases then it should fix a lot of problems (thanks @Mordgier for the ships he will test it with them).
However 5.1 coming maybe next week will be a stable thing without this for now. Changing collision stuff could also break a lot of stuff. In a separate branch after 5.1 he just activates the stuff so we can test it.
So this is the best time right now to get a great PvP experience in Empyrion and you all helped a lot. Thanks again!
But your combat tests today are still very appreciated. Especially with all of this now in mind.
This on the other side will mean that we have to think about the class limits etc. for 5.1.
Am I wrong to think that it is a proportional algorithm?
- Less whitespace ships = less lags = less lagshots ?
- More whitespace ships = more lags = more lagshots ?
So keeping class 3 and 20k blocks or maybe let’s go to 30k should be good?!
Everyone who has now a class 6 ship (before class 3) was theoretically responsible for the lags and have to redesign their ships - legit.
1 Like
20k blocks 3 size class 
Now my ship 3k blocks have 3 class and it without any strange blocks. Even light cruiser 7.5 blocks have 4 class. 19k blocks SW Destroyer have 250k triangles (and it dont have any inner other blocks for anti lagshot armor) So go lower that 250k triangles it is no sens. Or all server will fly on starter paper ship or just leave this game. And that we have after 5 alpha )) Devs make more and more restrictions but they don’t want fix the game or maybe they hand stuck in some place.
This is lag ship?
I dont think so this is normal ship or all of us need make coffin boxes with turrets?
I have great idea - remove warp drive, rcs and others modules and all turrets! Turrets have giant triangles count.
2 Likes
I can not log in the experimental server, my game is giving execution error and closes and back to the desktop, I’m leaving the experimental to see if it solves, someone with the same problem?
Maybe - but what is also true is:
more ships = more lagshots = more need for whitespace.
Plus with enough white space it doesn’t really matter to you if the shots go through the first layer of collision detection as long as they are caught but one of the many other layers.
This mechanic is precisely why pulse lasers tend to be the worst about it due to their projectile speed as well while missiles never go through blocks.
To all who will participate today - please make sure you have Discord and are on the HWS Discord server - we will coordinate via the Nova voice channel.
Please keep in mind that none of the fights are to the death etc - we’re just trying to see what makes lag worse or better and will only engage in combat as long as needed for a consensus to be reached.
At the end I would like to gather the following from those who participated:
- Computer specifications.
- Frequency and lengths of freezes
- Number of total disconnects if any.
- Number of desyncs if any.
- Log files
1 Like
@RexXxuS @Zizi @Xanif I think there is some confusion about what causes the problem.
The problem should not be cause by lag (connection lag). The weapon shots are fast and short-lived. I highly doubt than they syncronise any weapon shot after it was fired. So one of the cause of the problem may be an overloaded cpu that can’t keep a low physic timestep so shots skip a greater distance and pass thru stuff (explained in the doc @Serious_Squirrel linked).
The second problem may be ship position sync, where the client teleports a ship to another place to correct deviations, and you place the ship over a passing shot. YES this one may grow worse with lag.
For both cases if you intercept the colision event and find the correct block for damaging, the problem is ‘patched’ (examples in my previous post). Also you can make the correction math very simple at first and then get it more precise if the game does not choke.
I don’t have a 2nd monitor, but for those who do, it would be nice to open task manager or an app that can measure GPU load and network usage so you can see what happens when you game freezes. (most ppl think is network bottleneck, I think it may be cpu/gpu overload. Maybe we have both?).
we have NOTHING. At all. Just game “freeze”, and thats all…
As I read it… this is not an accurate assessment, as it is comparing different issues…
Less white space (by using all square blocks) = less lag TRUE, less poly, less CPU load, less lag in general. Using connector blocks results in much higher poly count, but still forces a surface render in all directions of the block and stops the real killer, which i don’t call a “Lag Shot”
The confusion i think comes in when people make statements like … “i have 20 blocks of solid combat steel in all directions and my core was taken out” thus i must have been “Lag Shot’d”! THAT issue is better explained as a missed collision detection against the single surface of a square block (likely due to lag itself) and then the projectile having not further “surfaces” to check against and go straight through the remaining blocks to the core.
There needs to be a distinction between a “lag shot” (missed collision) and the bigger problem of …lets call it… “non-rendered surface"lag shot”. As mentioned above, the CCC or CCD being turned on might help with that, but im no Unity coder/dev so…
Only reason i mention this is this needs a bit more clear and concise definition not to confuse the two issues at hand.
1 Like