MantisBT - Zandronum
View Issue Details
0002859Zandronum[All Projects] Bugpublic2016-10-06 05:392019-12-04 06:55
unknownna 
Leonard 
urgentmajoralways
needs testingopen 
2.1 
3.1 
0002859: Gametic-based unlagged seemingly goes out of sync often compared to ping-based unlagged
I did some tests earlier and did some more tests now, and it seems that the gametic-based unlagged causes the client to gradually desync the positions of the opponents until the positions "snap back" in place. In some cases, the positions are seemingly almost permanently desynced until the opponents stop moving. This happens without any ping emulation. It's worse after changemap map changes. When changing to ping-based unlagged, none of this happens.

Is this a bug, or some strange side-effect of gametic-based unlagged?
1. zandronum -host -iwad doom2.wad -file unlaggedactor.wad +sv_unlagged_debugactors 1 +sv_nomonsters 1 +map map01
2. zandronum -iwad doom2.wad -file unlaggedactor.wad -connect localhost
3. zandronum -iwad doom2.wad -file unlaggedactor.wad -connect localhost
4. Join the game with client A, making sure that it uses gametic-based unlagged.
5. Join the game with client B.
6. "changemap map01" in the server console.
7. Move around with client B soon as the map has loaded. Notice the apparent desync.
8. Set unlagged to ping-based on client A and repeat step 6 and 7 and notice how it doesn't desync.
I can always reproduce the gametic desync after changemap map changes. Let me know if you want/need a demo.
No tags attached.
related to 0002712resolved Leonard Teleport + "On Projectile Inpact" is bugged online 
child of 0003334needs testing Leonard Tickrate discrepancies between clients/servers 
child of 0003317needs testing Leonard Gametic-based unlagged does not respect cl_ticsperupdate 
child of 0003314needs testing Leonard The ticbuffer isn't completely processed in some cases 
child of 0003316needs testing Leonard Unlagged is off by one tic every time 
? unlaggedactor.wad (1,244) 2016-10-06 05:39
https://zandronum.com/tracker/file_download.php?file_id=1908&type=bug
png GameticVSPing.png (97,642) 2016-10-06 05:42
https://zandronum.com/tracker/file_download.php?file_id=1909&type=bug
png

? unlagged_debug.wad (7,309) 2017-10-27 21:36
https://zandronum.com/tracker/file_download.php?file_id=2225&type=bug
Issue History
2016-10-06 05:39unknownnaNew Issue
2016-10-06 05:39unknownnaFile Added: unlaggedactor.wad
2016-10-06 05:42unknownnaFile Added: GameticVSPing.png
2016-10-06 05:48unknownnaSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=9590#r9590
2016-10-08 04:31unknownnaNote Added: 0015790
2016-10-08 04:31unknownnaPriorityhigh => urgent
2016-10-08 10:39unknownnaNote Edited: 0015790bug_revision_view_page.php?bugnote_id=15790#r9615
2016-10-11 20:12unknownnaNote Added: 0015907
2016-10-11 20:17unknownnaNote Edited: 0015907bug_revision_view_page.php?bugnote_id=15907#r9674
2016-10-11 20:26unknownnaNote Edited: 0015907bug_revision_view_page.php?bugnote_id=15907#r9675
2016-10-11 20:53unknownnaNote Edited: 0015907bug_revision_view_page.php?bugnote_id=15907#r9676
2016-10-11 21:02unknownnaNote Edited: 0015907bug_revision_view_page.php?bugnote_id=15907#r9677
2016-10-12 05:39unknownnaNote Edited: 0015907bug_revision_view_page.php?bugnote_id=15907#r9685
2016-10-12 06:17AlexMaxNote Added: 0015915
2016-10-12 16:27Ru5tK1ngNote Added: 0015924
2016-10-14 13:29unknownnaRelationship addedrelated to 0002712
2016-10-14 13:47unknownnaNote Added: 0015988
2016-10-14 13:52unknownnaNote Edited: 0015988bug_revision_view_page.php?bugnote_id=15988#r9736
2016-10-14 13:52unknownnaNote Edited: 0015988bug_revision_view_page.php?bugnote_id=15988#r9737
2016-10-27 22:18JKist3Note Added: 0016097
2016-11-06 11:31Torr SamahoNote Added: 0016148
2017-02-11 19:06StrikerMan780Note Added: 0016837
2017-02-13 18:01StrikerMan780Note Edited: 0016837bug_revision_view_page.php?bugnote_id=16837#r10171
2017-02-13 18:59Torr SamahoNote Added: 0016861
2017-02-13 18:59Torr SamahoTarget Version => 3.1
2017-02-14 23:29StrikerMan780Note Added: 0016865
2017-02-14 23:30StrikerMan780Note Edited: 0016865bug_revision_view_page.php?bugnote_id=16865#r10180
2017-09-12 11:48JKist3Note Added: 0018289
2017-09-12 11:49JKist3Note Edited: 0018289bug_revision_view_page.php?bugnote_id=18289#r10938
2017-10-22 13:58LeonardNote Added: 0018586
2017-10-22 13:58LeonardStatusnew => confirmed
2017-10-22 20:05JKist3Note Added: 0018597
2017-10-23 14:50LeonardAssigned To => Leonard
2017-10-23 14:50LeonardStatusconfirmed => assigned
2017-10-27 21:33LeonardNote Added: 0018627
2017-10-27 21:36LeonardFile Added: unlagged_debug.wad
2017-10-27 21:39LeonardNote Edited: 0018627bug_revision_view_page.php?bugnote_id=18627#r11198
2017-11-08 00:59LeonardRelationship addedchild of 0003334
2017-11-08 00:59LeonardRelationship addedchild of 0003317
2017-11-08 00:59LeonardRelationship addedrelated to 0003314
2017-11-08 01:00LeonardRelationship replacedchild of 0003314
2017-11-08 01:01LeonardRelationship addedchild of 0003316
2017-11-10 16:29LeonardNote Edited: 0018627bug_revision_view_page.php?bugnote_id=18627#r11322
2017-11-10 16:31LeonardNote Edited: 0018627bug_revision_view_page.php?bugnote_id=18627#r11323
2018-04-30 15:54LeonardNote Added: 0019179
2018-04-30 15:54LeonardStatusassigned => needs testing
2018-05-07 20:17StrikerMan780Note Added: 0019220
2018-05-08 09:55LeonardNote Added: 0019224
2018-08-25 15:57unknownnaNote Added: 0019397
2018-08-25 16:16unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11687
2018-08-25 16:23unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11688
2018-08-25 16:24unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11689
2018-08-26 16:48unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11694
2018-08-26 17:00unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11695
2018-08-26 17:10unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11696
2018-08-26 17:25unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11697
2018-08-28 05:14unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11724
2018-08-28 05:50unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11725
2018-08-28 05:50unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11726
2018-08-28 06:16unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11727
2018-09-09 07:23unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11771
2018-09-17 01:22unknownnaNote Edited: 0019397bug_revision_view_page.php?bugnote_id=19397#r11803
2019-12-04 06:55StrikerMan780Note Added: 0021077

Notes
(0015790)
unknownna   
2016-10-08 04:31   
(edited on: 2016-10-08 10:39)
You can also reproduce this by simply joining the game with the observing client, no need for a changemap map change.

I did some more tests and can at the moment conclude with that at any given ping, the red debug actor is further behind the player when using gametic-based unlagged.

(0015907)
unknownna   
2016-10-11 20:12   
(edited on: 2016-10-12 05:39)
If this will not be looked into and changed/fixed before 3.0 is released (which I assume is too late now), then I insist that we temporarily reintroduce ping-based unlagged as an option for users. A proper gametic-based unlagged would perhaps be ideal, but in its current state it's simply not good enough to fully replace the tried and true ping-based one. You're going to have to fix the desyncs first before fully replacing it. The fact that it desyncs with a local connection suggests that it's fundamentally bugged.

Won't the over-reconciliation technically allow you to hit other players in your own client's past?

I did some more tests and can confirm that the gametic-based unlagged also desyncs a little consistently every 24-25 seconds. In addition to this, I now just discovered that it also desyncs whenever you respawn. And not too surprisingly, it also seems to desync after map resets. It also seems to desync with cl_ticsperupdate 2/3, causing the client to seemingly over-reconcile the other players permanently, though I'm not sure whether this is a bug or some visual side-effect. In any case, the debug actor is synced perfectly with the player position when using ping-based unlagged, whereas it's always flickering behind the player position when using gametic-based unlagged.

To summarize my findings so far:

* Desyncs after changemap map changes.
* Desyncs when you join the game.
* Desyncs a little consistently every 24-25 seconds.
* Desyncs when respawning.
* Desyncs after map resets.
* Seems to permanently over-reconcile other players with cl_ticsperupdate 2/3.
* For any given ping, even with ping spikes, the position of the debug actor is further behind the player compared to its position when using ping-based unlagged.

Edit:

I noticed that disabling the tic buffer seems to improve things. It then stops desyncing after changemap map changes, map resets and when respawning. Though this makes me wonder if this whole issue is nothing but a visual side-effect of the tic buffer and a misunderstanding on my part. Let me know if this is the case.

The ping-based unlagged also seems to undergo a minor readjustment every 25-30 seconds as well, provided that the client has some ping. It doesn't seem to do it with a local connection like the gametic-based one does.

However, after some extensive tests with actually firing at a moving client, it seems to me that the hitbox is actually more often than not further ahead with the ping-based unlagged, as the player seems to have a bigger "dead-zone" behind where no pellets hit, and a bigger "hit-zone" at the front where pellets hit. This would mean that the ping-based unlagged is the more inaccurate one? Interesting.

So in conclusion, the gametic-based unlagged seems to actually be more accurate, but it still seemingly has some quirks left, particularly with how it starts to "desync" when having a local connection to the server.

(0015915)
AlexMax   
2016-10-12 06:17   
Here is the original thread where this was discussed. For some reason, all of the links to bitbucket are gone, but hopefully it should point you in the right direction to the source of when the patch made it into Zandronum:

https://zandronum.com/tracker/view.php?id=1705 [^]

I find your described behavior incredibly perplexing. At least when I was working on the patch, there was no big redesign of unlagged. Instead, it simply tried to measure the difference in tics between the client and the server more accurately, by simply comparing two tic counts instead of relying on ping, which is a much more mercurial and out-of-date value, averaged over several samples. To wit, the original code that picked the gametic to go back to:

> int unlaggedGametic = ( gametic - ( player->ulPing * TICRATE / 1000 ) );

compared with what happens now:

> int unlaggedGametic = pClient->lLastServerGametic + 1;

Looking around in the source, you can see that in sv_main.cpp, lLastServerGametic is read directly off the wire from the client's movement command. In cl_main.cpp, you can see that CLIENT_SetLatestServerGametic() is updated when connecting, and also when the client gets MoveLocalPlayer commands.

However, if you check post 8277 on that ticket I linked, you will notice that I did notice a discrepancy at higher pings between the two methods and couldn't explain all of it. Most importantly, I could not figure out if the discrepancy between the two was because gametic was more accurate, or because I was making a bad assumption somewhere.

This is where my involvement with the patch ended and Water took over, and it didn't look like anybody followed up on investigating this discrepancy - Water assured the ticket that he had tested it, and I suppose players considered it better than the status quo and wanted it in 2.0.

Hopefully some of this rambling is helpful to you. And yeah, I definitely think that Torr should get involved to make sure that our assumptions are all sound.
(0015924)
Ru5tK1ng   
2016-10-12 16:27   
Recovered the full commit link.https://bitbucket.org/Torr_Samaho/zandronum-old/commits/8e43651ce72ae4b0a66ebc0e1c011b04c883ac74 [^]
(0015988)
unknownna   
2016-10-14 13:47   
(edited on: 2016-10-14 13:52)
Thanks for the response, AlexMax. Hopefully we can sort this issue out.

0002712 confirms some sort of desync caused by the gametic-unlagged working with the tic buffer.

However, even with the tic buffer off it still desyncs a lot if cl_ticsperudate is 2/3.

(0016097)
JKist3   
2016-10-27 22:18   
noob question: how do I change between tic based and ping based unlag? I can't seem to find the variable/command. Also just so I completely understand, what exactly does unlaggedactor.wad do/show?
(0016148)
Torr Samaho   
2016-11-06 11:31   
As unknownna suggested, I restored the option to use ping based unlagged for now.

Quote from JKist3

noob question: how do I change between tic based and ping based unlag?

The CVAR to toggle this is cl_ping_unlagged.

Quote from JKist3

Also just so I completely understand, what exactly does unlaggedactor.wad do/show?

For every client, the server spawns debug actors that show where the other players are moved for determining hits of attacks from that client with unlagged. Of course, you are seeing the unlagged information "from the past" with this, but it should give you an idea by how much players are moved for determining hits.
(0016837)
StrikerMan780   
2017-02-11 19:06   
(edited on: 2017-02-13 18:01)
Yeah, it's definitely best to keep in mind the debug actors are going to lag behind due to latency. Some people back when unlagged was being tested didn't realize this.

Seems a lot of bugs were introduced by the tic buffer, because before it became a thing, Gametic unlagged was pretty much perfect.

By the way, shouldn't this be on the roadmap?

(0016861)
Torr Samaho   
2017-02-13 18:59   
Quote from StrikerMan780

Seems a lot of bugs were introduced by the tic buffer, because before it became a thing, Gametic unlagged was pretty much perfect.

The way unlagged is perceived seems to be very different from player to player. Although, the memory of how well / bad unlagged worked in the past compared to now does not seem to be very objective. In any case, the debug builds allow to turn off the tic buffer with "sv_useticbuffer 0". So, for a fair comparison to check whether something is caused by the tic buffer, you should compare the situation with the same 3.0 build using "sv_useticbuffer 0" and "sv_useticbuffer 1".

Quote from StrikerMan780

By the way, shouldn't this be on the roadmap?

Yes, it should. Since it's not a 3.0 regression, I'll add 3.1 as target.
(0016865)
StrikerMan780   
2017-02-14 23:29   
(edited on: 2017-02-14 23:30)
I dunno, Ping unlagged pretty often missed direct shots in instagib where Gametic didn't for the majority of players in testing. (Ping also hit when it should have missed, like shots too far ahead of target's direction of movement) I'd say that's fairly objective.

(0018289)
JKist3   
2017-09-12 11:48   
(edited on: 2017-09-12 11:49)
unknownna: can you test this again on zan 3.0? I just did and was unable to get gametic unlagged to desync at all on a local server.

(0018586)
Leonard   
2017-10-22 13:58   
I can reproduce this in 3.0.
On top of that, switching cl_ping_unlagged on and off clearly shows the desync only happens with gametic unlagged.
(0018597)
JKist3   
2017-10-22 20:05   
How are you testing this Leonard? are you using a local server? I will try to reproduce your steps if you tell me how you are doing it.
(0018627)
Leonard   
2017-10-27 21:33   
(edited on: 2017-11-10 16:31)
This was a nightmare to debug.
There were actually multiple issues getting in the way of this, both from the ticbuffer and unlagged.
After finally fixing everything, I will now create separate tickets and make pull requests for them one by one to keep things simple.

I will address this ticket once enough has been fixed and we can clearly see what happens.

It's worth noting that while debugging/testing each issue I used this commit which gave me very useful information and also a test wad that goes with it.
In the test wad there is a very useful map that allows easy stress-testing of unlagged.
This map along with the commit will allow us to gather empirical evidence of what works properly or not.
I will refer to this note in the coming tickets whenever it's needed to test the issues.

EDIT: Changed the commit's link (it's in a closed branch now so it doesn't show up but you should still be able to pull it directly if you want).

(0019179)
Leonard   
2018-04-30 15:54   
The relevant issues should be fixed now.
(0019220)
StrikerMan780   
2018-05-07 20:17   
I tried some instagib with a few people, and I didn't get any dropped shots or obvious cases of missing or hitting when I shouldn't have. But, shall I try again when the interpolation is fixed?
(0019224)
Leonard   
2018-05-08 09:55   
If you wish but keep in mind this will have to be tested on a bigger scale anyways.
(0019397)
unknownna   
2018-08-25 15:57   
(edited on: 2018-09-17 01:22)
It seems to have fixed most of the issues, but I noticed that the gametic-unlagged now sometimes take a longer time to readjust itself with ping spikes compared to before, for instance when dropping from 300 ping to 30. It takes around 4-5 seconds before the debug actor "catches up" to the lower ping.
Though it seems to be mostly visual, as the unlagged itself seems unaffected, meaning that the target doesn't shift positions during the readjustment.

And once it's desynced, it can at worst break it permanently, as the debug actor can then start to flicker behind the player instead of being solid. This only happened once so far for me, and I wasn't able to reproduce it yet.

Another thing I noticed is that the debug actors will then still have different distances between the ping- and gametic-unlagged. The debug actor is then further behind again with gametic-unlagged.
Actually, this also happens all the time with low latency (13 ping).
But, after testing with firing at the target, the ping-unlagged is actually wrong here, as I'm able to fire considerably far ahead of the target in thin air and still hit the player.

That's what I found so far. Great work by the way, amazing work! The unlagged definitely feels a lot better and "snappier" after testing with some bots on a server. But the ping-unlagged seems to be broken though at low latencies.

Edit:

Quote from unknownna
* Desyncs after changemap map changes.
* Desyncs when you join the game.
* Desyncs a little consistently every 24-25 seconds.
* Desyncs when respawning.
* Desyncs after map resets.

I still seem to be getting some form of these issues now for some reason when using gametic-unlagged. The gametic-unlagged breaks for one of the clients permanently and all the debug actors flicker behind every single other player. Will test further.
One client or a bot can cause the unlagged to desync or act up for all the other clients. It seemingly goes away when the offending client/bot is removed from the server, and returns when said client/bot returns to the server.

* It can either break by chance(?) for a local client and other clients by connecting or adding bots to the server. Removing the offending client/bot temporarily fixes the issue until said client/bot returns to the server. This seems to be a bug and is seemingly permanent.
* It can break by connecting to the server through a proxy without any ping emulation aside from the hardcoded one (about 2 ms according to Gamer's Proxy). I saw some very bad desync here where both clients disagreed where they were located on the map with unlagged_debug.wad. But this also happens in 3.0, from a quick test.
* The ping unlagged seems to either be broken or non-existent at low latencies sometimes, allowing you to hit up to a half player body distance ahead of the player. This however also happens in 3.0, but I wasn't able to reproduce it consistently yet.
* Other players are sometimes moving incredibly jittery between 10 and 17 ping specifically, but after testing it again for a second time trying to reproduce it reliably it works fine. Weird. Managed to get it again and this time the other player was jittery between 0 and 7 ping. So the clients become up to 7-8 ms behind/ahead of the server once it breaks?
* When it sometimes breaks, it can delay every command from your client by approx. 800-1000 ms on the server. For instance, you have 160 ping, but your projectiles, puffs, item pickups and player movement to other players appear to take effect only 800-1000 ms later instead of 160. I've observed this particular desync a few times by now. It seems to happen after ping spikes, map changes and map resets in particular. It can correct itself after a few minutes, but not always, and the delay is not necessarily always 800-1000 ms, it can sometimes approx. be 300-600 too. Slightly less, but still higher than your real ping.
* The client tends to get stuck in "Connection interrupted!" states after map changes, though it was already reported here: 0003430: [3.1 beta] Connection Interrupted after a few map changes
* The unlagged seems to act odd when firing immediately upon passing and bumping into another player close by, allowing you to seemingly hit behind the player in thin air and still hit. But IIRC, it's always been this way since Skulltag.
* Other players seem to move less smooth in general if playing with cl_capfps 0 (probably due to the tickrate now being forced to 35hz?).

There's something that causes it to break sometimes for sure, but I haven't found out exactly what, but it might have to do something with connecting to the server with very small and random ping fluctuations or something. There's something really weird going on here.

(0021077)
StrikerMan780   
2019-12-04 06:55   
Keep in mind the debug actor isn't the most accurate representation of where unlagged will put someone at a specific given time. It's always going to be behind a bit for the client. It's mostly there to make sure it isn't behaving erratically.