| View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||||||
| ID | Project | Category | View Status | Date Submitted | Last Update | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0003334 | Zandronum | [All Projects] Bug | public | 2017-11-08 00:52 | 2026-08-11 02:56 | ||||||||
| Reporter | Leonard | ||||||||||||
| Assigned To | Kaminsky | ||||||||||||
| Priority | high | Severity | major | Reproducibility | always | ||||||||
| Status | assigned | Resolution | open | ||||||||||
| Product Version | 3.0 | ||||||||||||
| Target Version | 3.3 | Fixed in Version | |||||||||||
| Summary | 0003334: Tickrate discrepancies between clients/servers | ||||||||||||
| Description | This issue was first described by unknownna here: 0002859:0015907. In this summary, all issues are addressed by 0003314, 0003317 and 0003316. All but one issue: [quote=unknownna]* Desyncs a little consistently every 24-25 seconds.[/quote] Turns out this is a little off though, the exact period is 28 seconds. When clients start on both Linux/Windows, a timing method is selected: either a timing event fired by the OS can be created and Zandronum will use it or time polling is used instead. Unfortunately, as described in [url=https://forum.zdoom.org/viewtopic.php?f=3&t=30348#p576033]this thread[/url] from the ZDoom forums, Windows timing events are only precise down to the milliseconds and as such only a delay of 28ms can be achieved due to rounding. Zandronum servers on the other hand always use time polling and, as a way of dealing with this issue, use a tickrate fixed at 35.75hz. [code]lNewTics = static_cast<LONG> ( lNowTime / (( 1.0 / (double)35.75 ) * 1000.0 ) );[/code] Can you spot the problem here? As described in the linked thread, a time step of exactly 28ms would theoritically result in a frequency of (1000 / 28) = 35.714285714...hz. Using a tickrate of 35.75 instead results in a time step of (1000 / 35.75) = 27.972027972ms. Calculating the difference and dividing the client time step by it (i.e. magnifying that difference until it equals one time step) and we get, you guessed it, 28 seconds. (28 / (28 - (1000 / 35.75))) = 1001 tics or 28 seconds. This means that after 28 seconds, the server will essentially be ahead of us by 1 tic. The prediction will account for this and as such clients won't notice a thing but outside spectators sure will. Attached is a demo of the desync occuring by using cl_predict_players false (28ms.cld). By making the server loop use a proper time step of 28ms, the issue is fixed: [code]lNewTics = static_cast<LONG> ( lNowTime / 28.0 );[/code] However a problem still remains: the windows timing method is the ONLY one that has a time step of 28ms. If for some reason the creation of a timer failed at start, Windows will use time polling which will obviously be a proper 35hz. Linux timer events are precise down to the microseconds as reported in the thread and will have proper 35hz both for polling and timing events. As such, even with a fixed time step of 28ms on the server, clients using a proper 35hz tickrate will experience a desync too and a much worse one at that. Calculating the difference again, etc...: ((1000 / 35) / ((1000 / 35) - 28)) = 50 tics. Only 50 tics and these clients will appear to jitter to outside observers. I will attach a demo of this with cl_predict_players false as well (35hz.cld). The solution is obviously to enforce a consistent tickrate in every case but which one? Should we force the time step to be 28ms globally or should we simply disable Windows timer events and use 35hz on the servers and be done with it? In my opinion, we should simply disable Windows timer events and here are my arguments on why: * The clients never slept to begin with if cl_capfps is off. * As per 0001633, the timer events are [i]already[/i] disabled on linux for clients due to being faulty. * Proper 35hz. I'm not sure but doesn't the fact that the tickrate is not a proper 35hz mean that mods that would rely on it being correct experience time drift? For example, a mod uses a timer in ACS that's supposed to count time at 35hz, does it keep track of the time correctly? I haven't really checked this so I'm not sure but that could be a thing. * If we want to look at some sort of standard, Quake 3 uses time polling for both its clients and servers and, just like Zandronum, only the servers sleep. | ||||||||||||
| Steps To Reproduce | Using the test map referenced in 0002859:0018627: -host a server on MAP01 -join a client with cl_predict_players 0 -wait at least 28 seconds -you will notice stuttering for about a second or so, this is what outside observers actually see You may also use the debug output given by the linked commit, it will show the desync happening in real time (the server messages arriving at different times and the prediction suddenly having to do 2 ticks even though this is a local connection). | ||||||||||||
| Additional Information | The demos were both recorded on the same test map described in the steps to reproduce. | ||||||||||||
| Attached Files |
| ||||||||||||
 Relationships |
||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||
 Relationships |
| Notes |
|
Edward-san (reporter) 2017-11-08 10:54 |
Just for curiosity: did you check also if GZDoom multiplayer has this desync problem? AFAIR some years ago, when I, on Ubuntu 64, tested with a Windows user, there weren't issues like that. |
Leonard (reporter) 2017-11-08 17:26 |
No, I didn't check GZDoom. [quote=Edward-san]there weren't issues like that.[/quote] I'm assuming you're also talking about GZDoom here? My guess is that given the nature of GZDoom's multiplayer it literally doesn't matter: everyone connected experience the worst latency so even on a lan a different tickrate would just mean a slight latency (0.5ms) for the Windows user who is ticking faster. If you're talking about Zandronum then bear in mind the client does not see this for itself and even if you hosted on Linux, the server still uses the same loop so you would still only notice the other user's jitter every 28 seconds. |
|
Edward-san (reporter) 2017-11-08 20:25 |
[quote]I'm assuming you're also talking about GZDoom here?[/quote] Correct. |
Ru5tK1ng (updater) 2017-11-10 02:26 |
It seems to make more sense and consistency to disable Windows timer events. I would say just begin working to implement 35hz. |
|
Leonard (reporter) 2017-11-10 18:33 |
[url=https://bitbucket.org/Torr_Samaho/zandronum-stable/pull-requests/65/changed-enforce-a-consistent-ticrate/diff]PR[/url]. This is for proper 35hz. If for some reason we end up going for 28ms though I could always edit it later. |
Blzut3 (administrator) 2017-11-10 20:17 Last edited: 2017-11-10 21:11 |
I'm confused on "The clients never slept to begin with, why bother with a timer?" The difference between the polled and event timer should be whether Zandronum uses 100% CPU or not with cl_capfps on (I think vid_vsync would be affected too but not certain). If disabling the event timer does indeed cause 100% CPU usage expect a lot of angry laptop users. On a related note, I believe 3.0 got the updated Linux timer and should be fine now modulo the issue you're trying to solve. Edit: Regarding the ACS timing point. It has been too long but the last time I looked at the 35.75Hz issue I did notice some mods were correcting for the round off. So some mods are going to be wrong with either choice, but it really doesn't matter unless your hobby is watching timers and looking for drift. :P |
|
Leonard (reporter) 2017-11-11 01:19 |
Oh you're right, I didn't check for the I_WaitForTic functions. Then I guess we loose sleeping with cl_capfps on for Windows. |
|
Ru5tK1ng (updater) 2018-01-23 01:47 |
Is the loss of sleep for windows the reason the patch wasn't pulled? |
|
Leonard (reporter) 2018-01-23 20:04 |
The loss of sleep issue was solved, the reason this wasn't addressed yet is because having consistent ticrates (both 35hz and 28ms) revealed a much bigger problem that needs to be addressed first. |
|
Ru5tK1ng (updater) 2018-01-24 01:40 Last edited: 2018-01-24 01:40 |
For those following this ticket or interested parties, what bigger issue was revealed? I'm assuming 1633 is part of it. |
|
Leonard (reporter) 2018-02-11 19:46 |
Sorry for the extremely late reply, I wanted to respond once I had the basic implementation up and running and time played against me again and I kind of forgot to reply. [quote=Ru5tK1ng]I'm assuming 1633 is part of it.[/quote] No. [quote=Ru5tK1ng]what bigger issue was revealed?[/quote] I will try my best to explain this. In this ticket it was found out that the ticrates between clients/servers differ. This means that inevitably over time one is going to end up behind the other: after 28 seconds the windows clients are so far behind they loose one tic and at the very moment this occurs, the server's ticking gets closer and closer and that's what causes the "desync". At this point, when the client and the server's ticking happen almost at the same time, the server seems to alternate between receiving the client's movement commands before and after ticking. This produces what Alex called the "jackhammering effect" where in the worst case the server consistently processes 2 movement commands (the previous one that was received AFTER ticking and the current one) and then waits the next tic. Obviously this makes the player extremely jittery. The problem is that once the ticrates were actually fixed and made consistent across clients/servers, on rare occasions a connecting client would get the jackhammering effect except this time permanently. This is presumably due to the fact that the clients/servers do not lag behind relative to each other anymore which implies that a client that would tick almost exactly at the same time as its server by "chance" would do so permanently. I'm not sure what causes it to happen in the first place and the fact that searching for such a problem doesn't seem to give anything useful and that apparently this problem never occurred to any other game engines made me doubt what was even happening. Even then, I tried to debug further but couldn't find anything useful anymore on this issue. After that I decided to work on smoothing the ticbuffer which would not only fix (or kludge if it's not something that is supposed to happen) the jackhammering problem but also benefit online play in general. In particular, I expect this will come close to completely solving 0002491. The way this works is simple: detect if movement command "drops" occur frequently and if so simply apply a "safe latency" to the processing in such a way that the movement will end up being completely smooth. I think this is the most logical way to solve such a thing: the jittery player shouldn't even notice a thing as both prediction and unlagged should account for any extra latency except now, instead of being very hard to hit to outside observers, will be much, much smoother. I attached a demo (jack.cld) in which a player experiences the jackhammering effect followed by the new ticbuffer "regulator" being turned on. This means that players experiencing this problem will be completely smooth to outside observers regardless but at the expense of having one extra tic of latency. I also attached a much worse example using gamer's proxy to simulate a jittery player (jitter.cld). |
|
Leonard (reporter) 2018-03-25 15:19 |
Since the ticbuffer smoothing is ready to be reviewed, this needs to be reviewed as well. |
|
Leonard (reporter) 2018-04-30 15:47 |
This was pulled. |
StrikerMan780 (reporter) 2018-05-07 02:07 Last edited: 2018-05-07 05:58 |
There seems to be an issue with this in the latest 3.1 build. Player and actor movement is no longer interpolated, it looks like it's running at 35fps, even with uncapped framerate. Both online and offline. Everything else seems fine however. |
|
Leonard (reporter) 2018-05-07 09:46 |
That was reported in 0003418. |
|
StrikerMan780 (reporter) 2018-06-03 22:39 Last edited: 2018-06-03 22:39 |
https://zandronum.com/tracker/view.php?id=3314#c19255 Linking to this comment, since it may have more to do with this than the ticket I posted in. Not sure though. |
unknownna (updater) 2024-03-06 12:27 Last edited: 2024-03-12 11:49 |
[quote=Leonard]on rare occasions a connecting client would get the jackhammering effect except this time permanently.[/quote] Hey, was this ever added to 3.1, or looked into any further? This issue is still present in 3.1. It's especially noticeable if you connect with some latency, for instance with 300 ping emulated. If you're unlucky and connect to the server at the wrong tick, all the other players will permanently jitter/jackhammer around needlessly. Their player body movements aren't as smooth when you circle-strafe around each other while fighting. It's actually rather bad when it happens. When the bug occurs, the impression from your POV is that the jittering players move slightly faster all the time due to all the micro-skipping they do. Additionally, other clients see you as skipping as well. Glitched clients affect both themselves and others on the server. The only way to fix it is to quit and restart the program and hope that you get better luck upon the next instance of starting the program and joining the server. To sum it up so far: * Clients connected with a bad tick sync will visually jitter/jackhammer around to everyone else, and they in turn will perceive everyone else as jittering as well. * Clients connected with a good tick sync will see other clients with good tick syncs moving smoothly. * To fix it, you have to quit and restart the program and hope for the best upon connecting to the server again. * Because of this desync, you effectively have 2 different positions of the other clients flickering back and forth that you can hit successfully with a hitscan weapon. And to add to the final point, because of this permanent desync and flickering of 2 different positions rapidly, it could explain this issue: [quote=unknownna]The unlagged seems to act odd when firing immediately upon passing and bumping into another player close by, allowing you to seemingly hit behind the player in thin air and still hit.[/quote] Another effect of this is that shots with a wide spread like the SSG aimed at the center of your target are more likely to miss some pellets the further away the target is, because the target is rapidly bouncing back and forth between 2 points. Players are thus potentially harder to hit the further away they are from you in Zandronum. I updated Leonard's example wad to easily reproduce the issue and also gave it Quake 3 hit sounds for target checks and made the hitbox sprites more intuitive. There are 2 pistols, one fast like the chaingun, and one slow like the shotgun. Connect 2 clients to the server and use an emulated ping of 300 on one client to easily see the issue when it occurs. You can 100% hit the flickering position that is incorrect, and you even have a dead zone in the middle of the 2 positions where no pellets hit. It's most definitely bugged. I tested various timings to use when joining the server immediately after starting the server exe. Joining after 6-7 seconds seems to not be enough to trigger the issue. But any later than that and it almost seems to have something to do with the time being an even or odd number, like there's some rounding error that causes the player positions to fluctuate between 2 tics permanently. It's 1 tic off when it glitches, I'd say. https://web.archive.org/web/20181107022429/http://www.mine-control.com:80/zack/timesync/timesync.html Maybe this link has some information that can help? I can't really do anything more from here until someone looks into it further. |
Kaminsky (developer) 2024-03-12 15:13 |
It might also be worth backporting this commit from Q-Zandronum: https://github.com/IgeNiaI/Q-Zandronum/commit/d024154462267f10f65e4c6f78bf836764295e01 to help fix the permanent time desync issue. |
|
unknownna (updater) 2024-03-13 14:08 Last edited: 2024-04-03 10:01 |
Thanks for that! I tested the latest Q-Zandronum 1.4.11, which I assume has the fix, and it's much worse there, and the unlagged is extremely broken. The client-side movement prediction of your own player also jitters a lot. Whatever it's doing is not the correct way at the moment. Back to regular Zandronum, turning the tic buffer off doesn't improve the issue, so it seems to be something more fundamental going on. With 2 clients standing still on the scrolling floor, with one client using 300 emulated ping, older versions of Zandronum (1.0, 1.1, 1.3, 1.2.2, 2.1.2 and 3.0) seem to be very smooth until 16 seconds pass, at which it starts to jitter for 13-14 seconds before correcting itself and normalizing again, and it loops like that over and over again. This time loop is 100% consistent. The permanent time desync jitter starts to appear in 3.1 after the fixes. So it's a time drift issue after all, but Q-Zandronum's fix is not the correct one since it glitches it out even further. [quote=unknownna]older versions of Zandronum (1.0, 1.1, 1.3, 1.2.2, 2.1.2 and 3.0) seem to be very smooth until 16 seconds pass, at which it starts to jitter for 13-14 seconds before correcting itself and normalizing again, and it loops like that over and over again. This time loop is 100% consistent.[/quote] After testing this further, it means that prior to 3.1, the unlagged position of your target would be shifted forward every 16 seconds, and flicker rapidly between 2 points for 13-14 seconds until it normalized. Because of this, You could hit even further ahead of the player than before when these 16 seconds passed. And due to it already being 1 tic off because of a separate issue, it meant aiming extra far ahead to land a hit. The unlagged would break every 16 seconds in older versions of Zandronum. The difference is that it's now permanent, and it's very likely to break often for incoming clients. It's not too fun to play the game in its current state, just saying. |
|
Kaminsky (developer) 2024-04-06 00:22 |
Thanks a lot for your detailed analysis, and for testing the aforementioned commit from Q-Zandronum! I was hoping that Q-Zandronum's approach was good enough and we could easily backport it. I started working on adding time synchronization using the information in the link you posted in https://zandronum.com/tracker/view.php?id=3334#c23304 recently. It's not 100% complete, and I haven't had an opportunity to thoroughly test it to see if it fixes the issues you mentioned. I'm afraid that I won't have the fix ready for the next 3.2 beta that might come in a few days, but I'll be happy to share test builds with you afterwards. |
|
unknownna (updater) 2024-04-06 10:00 |
Incredible, looking forward to it! |
|
Kaminsky (developer) 2024-06-08 15:43 |
Hello, I'm finally back with an update: After spending a lot of time trying to sync the times between the server and client (via https://web.archive.org/web/20181107022429/http://www.mine-control.com:80/zack/timesync/timesync.html), this didn't help fix the stuttering for me. At best, it might've helped reduce it a bit (but could break when the client's ping fluctuated), and at worst, it made the stuttering more severe than when the times weren't synced. So, it seems that syncing the client's clock to the server's isn't a good solution to fixing this issue. This brings me to another solution: In 3.1-alpha, I worked on the skip correction, a server-sided feature that extrapolated the movement of players when their commands didn't arrive on the server's end normally (i.e. during packet loss or ping spikes). The idea was to predict where the player was moving using their last processed move command. When the late move commands finally arrive, the server "backtracked" the player's movement, meaning that it restored the player's position, velocity, and orientation to what they were before extrapolation, then processed all late move commands instantly. This meant that the player could move smoothly for others, even when they're lagging, without their ticbuffer overflowing with late commands which caused them to skip. While the feature itself already exists in Zandronum, there were lots of unwanted side effects that caused problems for players who tested the 3.1-alpha builds. I disabled the feature entirely in the stable release, so you can only use it in test builds. I intended to fix these side effects and improve the skip correction for 3.2 so that the feature can finally be used as it should be. Looking at it now, I also think it would be nice if the skip correction wasn't only server-sided, and clients could use it for themselves in case the stuttering only happens for them. To enable the skip correction for yourself, host a 3.2-alpha server and set "sv_smoothplayers" anywhere between 1-3 (the value represents the number of ticks the server will extrapolate the player for). When using the example WAD and following the steps outlined in https://zandronum.com/tracker/view.php?id=3334#c23304, enabling the skip correction did fix the stuttering for me (maybe not perfectly, but certainly better than without it). Let me know how it works for you. |
|
unknownna (updater) 2024-06-08 18:01 Last edited: 2024-07-03 04:17 |
Hey, and thanks for the update. Appreciate it. At first it made no difference to me, but after some more thorough testing, here's my findings: * sv_smoothplayers 1 fixes the issue for the most part when you observe a client that jitters between 2 ticks, but only if you connect with a good tick sync on top of that. However, there still seems to be some timing issue here, and if your tick sync is slightly off, it'll still flicker a bit permanently. If you restart the .exe and get a good tick sync however, it's incredibly good and like a dream. * sv_smoothplayers 1 works partially when observing a client that jitters between 3 ticks. Without it enabled, you have 3 different positions the client flickers back and forth between that you can hit. Setting it to 1 reduces these 3 positions to 2. Setting it to to 2 or 3 makes no difference from 1 and is no improvement. This is the glitch and flickering that absolutely can't be fixed unless the .exe is restarted. These were the only 2 issues I noticed. Sometimes you for some reason get a client that jitters between 3 ticks instead of 2. And the only way to get a better tick sync is to restart the .exe. Nothing else will do, no matter how many times you try to disconnect and reconnect from the console. I think there's still room for some improvement here. If we can somehow get the gameplay to always be like "sv_smoothplayers 1 + a good tick sync" it would play like a dream. [b]Edit:[/b] What all this tells me is that, since you can get lucky and get a perfect sync without using the tic buffer, is that every time a client .exe starts, it "hits" the server anywhere between 3 ticks upon connecting: 1st tick: Completely smooth if it lands right. Unlagged is perfect since there's no jitter and flickering between different ticks. 2nd tick: Clients flicker between 2 ticks, sv_smoothplayers 1 can fix this if your client hits exactly at 2nd tick. However, if the offset is slightly between the 1st and 3rd tick, it still flickers, although much less. 3rd tick: Clients flicker between 3 ticks, sv_smoothplayers 1 can reduce the 3 ticks to 2, but can't fix it completely at the moment. And I think this 3rd tick issue can maybe only happen if the other client also is synced badly to the server. And for some reason I couldn't fix it with sv_smoothplayers 1 when testing it again today, until I closed the other client and restarted the .exe on it. It's such a weird issue. It's as if each client has its own sync to the server, and because they're synced at different tick offsets, the settings that work in one way won't work when the tick offsets are different the next time. One problem with sv_smoothplayers 1 however is that it causes client prediction errors locally. [b]Edit2:[/b] Another observation: A client's cl_capfps value makes a big difference. A client with cl_capfps set to 1 will not experience the same amount of jitter compared to a client with uncapped FPS. The unlagged can for instance appear as being perfect with cl_capfps set to 1 and you don't have multiple targets that flicker, but when you turn it off, the other client you observe suddently starts to jitter between 2 points that you can hit compared to before. The unlagged behavior breaks even worse with uncapped FPS. I personally always play with cl_capfps set to 1, so I didn't notice this until now. And when you get a bad sync, your client seems to affect the server, which means that bots see you flickering back and forth as well, and thus their aim is very shaky when fighting them. And you can use cl_predict_players 0 combined with cl_capfps 0 and the chasecam to see yourself jitter. For some reason the ping makes a difference. Using an emulated ping of 160, if I connect with a lucky sync that works for 160 ping, changing the ping lower or higher makes it jitter worse. When you get a good lucky sync, it's pretty much as smooth as offline play, especially with cl_capfps 1. You can for instance bump into players and not jitter like you normally would. In addition, clients playing with cl_capfps set to 0 will actually jitter more to other players observing them. [b]Edit3:[/b] [quote=unknownna]For some reason the ping makes a difference. Using an emulated ping of 160, if I connect with a lucky sync that works for 160 ping, changing the ping lower or higher makes it jitter worse.[/quote] The jitter seems to harmonize with every 50-60ms of ping added or subtracted, give or take. For a sync that works for 160 ping, these pings will be just as smooth for the most part: 40: Good and smooth. 100: Good and smooth. 210: Good and smooth. 270: Good and smooth. 325: Good and smooth. 380: Good and smooth. 440: Good and smooth. More specifically, after testing more, I think the exact value might be 56ms for adding and 57ms for subtracting. So for a good lucky sync at 160 ping, these pings seem to be as smooth: 46: Good and smooth. -57ms 103: Good and smooth. -57ms 160: Baseline. 216: Good and smooth. +56ms 272: Good and smooth. +56ms 328: Good and smooth. +56ms 384: Good and smooth. +56ms 440: Good and smooth. +56ms It must be the Windows timer of 28ms x 2 = 56ms, with 1 extra ms for subtracting. It seems that with the way things are working right now, the Windows timer isn't playing nice with the server timer from the looks of it, and there might be some relativity problem on top of it, where an incremental unit of time used for syncing initially, doesn't occur at the same time between the client and server, though I'm not sure about the latter. I think I've tested and mapped out the issue as good as I can for now. I'd like to hear your thoughts from here. [quote=Leonard]I attached a demo (jack.cld) in which a player experiences the jackhammering effect followed by the new ticbuffer "regulator" being turned on. This means that players experiencing this problem will be completely smooth to outside observers regardless but at the expense of having one extra tic of latency.[/quote] Is this the way things are working at the moment? Not only does the intended behavior not seem work at all from my testing and is actually worse than 3.0, but this forced latency should be unacceptable, as 1 tic of difference at Doom's level of speed is the difference between your player successfully moving and hiding behind a wall or getting hit by a SSG shot. Another observation: Going into fullscreen mode with Alt+Enter after having gotten a good tick sync at 160 ping in windowed mode with cl_predict_players 0 and cl_capfps 0 causes your player and other clients you observe to start to jitter. For some reason the tick sync gets worse when entering fullscreen mode from windowed mode. And it's a bad and nasty jitter that can't be dialed out by altering the ping, it's absolutely horrendous. Oversizing the window without going into fullscreen also introduces additional minor jitter, but not nearly as much as when entering fullscreen mode. This means that fullscreen players are at a disadvantage against windowed players since they will jitter more permanently. After looking around for a while, I found some excellent links with discussions on the tick offset issue, and they also talk about the clock sync algorithm found in the link I posted earlier: https://gamedev.stackexchange.com/questions/206851/how-to-sync-ticksframes-between-two-peers https://gamedev.stackexchange.com/questions/18766/network-client-server-message-exchange-and-clock-synchronization-help?rq=1 https://gamedev.stackexchange.com/questions/164367/clock-synchronization?rq=1 https://gamedev.stackexchange.com/questions/55508/start-timer-in-two-clients-the-same-time/55515#55515 I hope it could help you. [b]Edit4:[/b] [quote=Kaminsky]So, it seems that syncing the client's clock to the server's isn't a good solution to fixing this issue.[/quote] If syncing the clocks initially by hopefully making them tick at the same offset wasn't enough to fix the issue, then we must perhaps offset the client's gametic by "x" amount of milliseconds, where x is how many milliseconds the client actually started to tick behind the server in real-life time. Since the jitter harmonizes with 56ms, and never seems to be off by more than 3 ticks, then we can maybe assume that each tick is 28ms long in duration, and multiply it by 3 to get 84ms. This is how far off the client's ticking can be at worst. This means that the client's ticking offset in relation to the server is off by anywhere between 0ms (perfect) to 84ms (lots of jitter) depending on when the ticker synchronizes with the server by chance when starting the .exe. The question is, how do we make the client and server aware of how far off it is, even though they think they are aligned? How can the client or server detect the flicker/jitter and attempt to correct and re-sync it? From my current perspective, this problem has nothing to do with ping spikes, nothing to do with dropped packets, nothing to do with extrapolating/interpolating other players, and nothing to do with the tic buffer. It has everything to do with each client's ticking offset being off by some milliseconds compared to the server's. I think that trying to fix this by extrapolating is maybe something of a hack and will also not work since each client has a different gametic or clock offset to the server, which means that one setting will only work by chance on some offsets and not on others. If each client could somehow correct and re-sync their own ticking offset towards the server's, then I believe the problem would go away naturally. As long as each client knows they're synced towards a certain offset, and other clients on their own ends are also synced towards the same offset, they should all naturally appear smooth and centered to one another. Really, from my perspective it seems to boil down to these 3 points: * Every client's gametic or clock tick offset is off by 0ms-84ms in real-life time, even if they tick "at the same time" in the code. Being able to experience and enjoy smooth gameplay is a matter of chance currently. * Fullscreen and oversized windowed mode compared to regular windowed mode introduces additional permanent jitter unrelated to the above. * The tic buffer "regulator" is not doing what it's supposed to do. It might not even be needed if the fundamental issue is fixed first. In fact, I also believe that a lot of the complaints of a "weak SSG" in Skulltag throughout the years can actually be traced back to this exact issue. Because of the tick offset desyncs between all clients and the server, they all flicker/jitter and effectively rapidly teleport between 2 points, making shots much harder to hit the further away a player is moving sideways from you. Fixing this would "magically" make the SSG feel a lot stronger as more pellets would actually hit the now-centered targets instead of missing, trust me. Some additional links for reading: https://gamedev.net/forums/topic/696756-command-frames-and-tick-synchronization/5378847/ https://gamedev.net/forums/topic/652377-network-tick-rates/5126249/ https://gamedev.net/forums/topic/706887-clientserver-sync-reasoning-and-specific-techniques/5426971/ https://www.gamedev.net/forums/topic/704579-need-help-understanding-tick-sync-tick-offset/5416883/ https://y4pp.wordpress.com/2014/06/04/online-multiplayer-proof-of-concept/#part2 https://gamedev.net/forums/topic/707830-clientserver-clock-sync-issue-confirmation-and-solutions/5430541/ https://devforum.roblox.com/t/do-not-use-high-precision-clock-syncing-tech-between-clients-and-server-with-accuracy-of-1ms/769346 https://devforum.roblox.com/t/how-do-you-sync-server-tick-and-client-tick/440508/3 https://devforum.roblox.com/t/could-someone-please-explain-the-timesyncmanager-module-made-by-quenty/426057/3 |
|
unknownna (updater) 2024-09-24 22:19 Last edited: 2024-10-22 05:44 |
Something interesting was observed with the new "connection strength" feature on the scoreboard in 3.2. If you get a good tick sync to the server, you will always have a good 4/4 connection strength to the server. If you get a bad tick offset sync however, the client will appear as having a terrible connection of 1/4 strength. So knowing that, if you quit and restart the client .exe until you get a good tick sync that syncs good at 160 ping, all the pings listed in my note above will always show a 4/4 connection strength, whereas any ping subtracted by 28ms will yield a dramatically diminished connection strength of 1/4. But every 56ms apart yields a strong connection of 4/4. There's a clear reproducible pattern at play here, we just need to figure out what causes it. It's very important that you check this with "cl_capfps 0" "chase" and "cl_predict_players 0" all combined so you can observe your own client jitter and skip around while moving. I recommend binding them to keys for easy and quick toggling. Kaminsky, when you attempted to sync the clocks, did you only do it a few times at start upon connecting, or did it you do it all the time? From what I've read in all the links posted in my note above, the time syncing should only occur a few times at start and no more after that. It's just important to line up the client and server ticking so they don't hit at the wrong offset relative to one another. I also suspect that the tick offset desync could be causing packet-loss issues for certain users, such as: https://zandronum.com/tracker/view.php?id=1469#c22933 https://zandronum.com/tracker/view.php?id=4415 if it's caused by some asymmetric-like connection caused by the tick offset being off between the client and server, leading to one side of the offset always dropping off further apart the center than the other, causing more missed packets on one side of the offset. [b]Edit:[/b] I did another experiment, to see which pings I would land on that "synced" good to the server when a ping of 160 didn't sync. At first I set the ping emulator to 160 and attempted to tweak and match it to the server by incrementing it by 1 and checking the client movement afterwards. When I found the ping that made the player move as smooth as possible, I left the ping emulator at that value before quitting and restarting the .exe. The goal was to get a good sync at 160 ping. The way to test this quickly is to add the ping emulator port as a custom Zandronum server in the server browser so you can connect to the server through it over and over again when it doesn't sync properly. Leave "cl_capfps" "cl_predict_players" both at 0 and quickly flip into chasecam mode when connected to observe your own client's movement. If the player movement is smooth, it's a "good sync". * 1st attempt: 156 or 212, smooth every 56ms apart. * 2nd attempt: Impossible to get a good sync, all pings jittered. Closest was 181 ping only, but still some jitter. * 3rd attempt: 184 ping or 240, smooth every 56ms apart. * 4th attempt: 156 or 212 again. * 5th attempt: Seemingly impossible to get a good sync again. Closest was 182-186 range, but still jitter. * 6th attempt: Impossible to get a good sync again. Closest seemed to be 155 or 211 only, but still some jitter. * 7th attempt: 178-179 or 234-235, very little jitter sporadically. * 8th attempt: Impossible again, 181-183 range closest to smooth, but still jitter. * 9th attempt: 180-181 or or 236-237, but still some sporadic jitter. * 10th attempt: Impossible again, 182-184 range smoothest, but still lots of jitter. * 11th attempt: Impossible again, 181 smoothest, still jitter. * 12th attempt: Impossible again. And then I became frustrated and suspicious as to why I never got a sync at 160 ping, and attempted to revert the ping emulator back to 160 everytime instead of leaving it at the former setting before restarting the .exe and connecting to the server again. This seemed to change it up a little and made it work for what I intended. * 1st attempt: Synced badly at 160, was smooth at 180. Revert ping emulator back to 160. * 2nd attempt: Synced badly at 160, was smooth at 155. Revert ping emulator back to 160. * 3rd attempt: Synced badly at 160, was smoothest at 153, but still some jitter. Revert ping emulator back to 160. * 4th attempt: Synced badly at 160, was smooth at 180. Revert ping emulator back to 160. * 5th attempt: Finally synced great at 160. What this makes me wonder is, does the server assume the client connects with a certain ping, and assumes that the client will never deviate from it somehow? Maybe my assumption about the whole issue is completely wrong, I just hope I'm not sending you on a wild goose chase with no end in sight with the time and tick syncing. Maybe it has nothing to do with time at all, and has to do with the server assuming something wrong at some point about the client's ping? Sometimes you can seemingly never get a good "sync" to the ticking, or whatever the heck the real problem is. It almost feels like there's always 2 instances of the player running around, and if one of them is out of sync with the other, it jitters. It almost feels like one of these 2 instances land "between" ticks, where it's never quite smooth no matter how much you try to tweak your ping to it. When you get the jitter and the client stops sending CLC_CLIENTMOVE commands, the client will many times keep displaying the running animation while standing still after you release the move buttons. This is made much worse by having cl_ticsperupdate set to 2 or 3. Why on earth are the movement commands dropped, but not the +attack and +use commands, or any other commands? All the other buttons seem to be instantly responsive, it's just the movement commands that are dropped. You never for instance fire any NULL rockets or anything like that, so maybe it's not a tick sync issue? Maybe there's just some fundamental bug with the way the player movement is handled on the server? The server assumes something wrong about the client movment for any given ping? There's almost got to be something wrong with the way the server handles the movement commands in relation to the client's ping upon connecting to the server, and the client's ping after being connected, like the way the movement commands drop often if it's 1 tick (28ms) off, but smooth again if 2 ticks (56ms) off. [b]Edit2:[/b] I repeated the same 2nd experiment with different pings, to see which latencies the sync would land onto when connecting to the server. Connect with a ping of 140, see which pings sync best, revert back to 140 every time before restarting .exe: * 1st: 122, smooth every 56ms apart. * 2nd: Impossible, lots of jitter, client landed between ticks. * 3rd: 127 * 4th: 128 * 5th: 127 * 6th: 156 * 7th: 127 * 8th: 158 * 9th: 158 * 10th: 122 It seemed to land on different pings compared to what I usually get when leaving the ping emulator at 160. The good syncs seemed to hit much lower in general when connecting with a ping of 140, in the 120s mostly. And in retrospect after having done the other experiments below, it seems to have landed less between ticks. Connect with a ping of 200, see which pings sync best, revert back to 200 before restarting .exe: * 1st: 160 * 2nd: 126 * 3rd: Closest was 153, still jitter, client landed between ticks. * 4th: Closest was 157-158, still minor jitter, client landed between ticks. * 5th: Closest was 127, jitter, client landed between ticks. * 6th: 157 * 7th: 157-158 * 8th: Closest was 123-126, still jitter, client landed between ticks. * 9th: Impossible, much jitter at any ping, client landed between ticks. * 10th: Closest was 156, but still sporadic jitter, client landed between ticks. The client seemed to land between ticks more often when using a ping of 200 to connect to the server, where I couldn't tweak the ping to match the sync towards the server's ticking. Connect with a ping of 0, see which pings sync best, revert to 0 before restarting .exe: * 1st: 156 * 2nd: Closest was 181-183, but still jitter, client landed between ticks. * 3rd: 125 * 4th: 156 * 5th: Closest was 155, but still jitter, client landed between ticks. * 6th: 126 * 7th: Impossible, jitter at all pings, client landed between ticks. * 8th: 156 * 9th: Closest was 155, but still jitter, client landed between ticks. * 10th: 157-158 This seemed to land a bit lower as well, and never at 160, but at 156-158 instead. And then I attempted the same thing, but this time with "cl_startasspectator 0", just to test it. Connect with a ping of 0, see which pings sync best, revert to 0 before restarting .exe: * 1st: 159 * 2nd: Closest was 155, still jitter, client landed between ticks. * 3rd: Closest was 154-156, still sporadic jitter, client landed between ticks. * 4th: 158-159 * 5th: Got a rare, never-before sync that lasted from 126-130 with minor jitter. * 6th: Impossible, much jitter, client landed between ticks. * 7th: Impossible, much jitter, client landed between ticks. * 8th: Got the same rare sync at 126-130 with minor sporadic jitter. * 9th: Closest was 123-124, but still lots of jitter, client landed between ticks. * 10th: Got a rare sync all the way from 120-135 with very minor jitter. Using "cl_startasspectator 0" seemed to change up the sync a little as well compared to the other testing conditions. I got some rare syncs that lasted for longer distances between ticks. Ignoring the "fullscreen jitter" issue I found earlier, there seems to be 2 main issues, the 1st issue being where the client might land on a wrong tick offset (between ticks) compared to the server's, and thus can never become properly in sync with it no matter what. The 2nd issue is where the server can only guarantee smooth movement for the client every other tick (56ms) apart from the ticking. Any duration between 1-28ms away from every other tick will cause the jitter to appear and lots of movement commands to be dropped. |
|
Kaminsky (developer) 2024-11-10 21:39 Last edited: 2024-11-10 21:44 |
I finally have an update for this ticket. First and foremost, thank you so much for your detailed analysis and investigation into this issue! Sorry for the very late response. I will say that I've been tackling this issue for about a month now, and after lots of debugging and testing, I concluded that the SNTP approach mentioned and linked above wasn't going to work to fix the stuttering. Simply put, there's too much variance (e.g. latency) and factors involved that made the approach ineffective. Furthermore, suddenly offsetting the client's clock by a huge amount in an attempt to synchronize it to the server's created many problems, and possibly more side effects than I was aware of. I'll list two major reasons why the permanent stuttering was happening: 1. Networking isn't perfect, so sometimes a client or server's packets arrive at their destination a bit too soon, or too late. 2. Unlike the server, whose ticks are generally consistent (28-29 ms in length), the client's ticks tend to fluctuate a lot more, depending on their current frame rate (e.g. 25-32 ms in length). Rendering the game contributes significantly to those fluctuations. Some ticks are inevitably shorter or longer than others, and this affects when the client sends out commands to the server. With these in mind, I realized that for all intents and purposes, we shouldn't be concerned with trying to synchronize the client's clock to the server's clock. In fact, doing this actually made the stuttering worse than it was before, even for clients with very low pings. What we should be doing instead is adjusting the client's clock just enough so that their commands arrive roughly in the middle of a server's tick (i.e. not too soon after the tick started, and not too late before the next tick starts). This way, even if their commands arrive sooner or later than expected because of the two reasons mentioned above, then at least they'll still arrive more consistently on the server's end. These adjustments to the client's clock are only a few milliseconds and are relatively safe to do. Thus, I put this theory to the test and pushed a new topic to the repository: https://foss.heptapod.net/zandronum/zandronum-stable/-/tree/topic/default/client-stutter-fix Here's a detailed step-by-step description of what's happening now: 1. When the client connects, instead of the server telling the client to authenticate their map right away, it tells them to begin adjusting their clock. It sends a SVCC_BEGINCLOCKADJUSTMENT command so that the client changes its connection state to CTS_ADJUSTINGCLOCK. 2. For each tick while the client's connection state is CTS_ADJUSTINGCLOCK, it will send the server a CLCC_SENDCLOCKUPDATE command to the server. 3A. When the server receives a CLCC_SENDCLOCKUPDATE command from the client, it saves the time it processed this command in CLIENT_s::lastClockUpdateTime. 3B. The server also saves the times it called G_Ticker in the current and previous ticks, and uses these epochs to determine how soon and late within a tick did the CLCC_SENDCLOCKUPDATE command arrive at. 4A. After receiving 10 of these commands, the server has enough samples to reasonably determine if the client's commands are arriving somewhere in the middle. It checks for any outliers (i.e. any commands that arrived less than 7 ms after or before the previous/current G_Ticker calls). 4B. If any outliers were found, the server will instruct the client to offset their clock accordingly, then repeat steps 2-4. 4C. If no outliers were found (the clock adjustment completed successfully), or if after 20 attempts at telling the client to offset their clock there's still outliers (at which point we just abort the clock adjustment), the server tells the client to authenticate their map, like normal. The caveat here is that it might take longer for clients to fully connect to a server. However, this should only take up to a few seconds, usually no more than 10 seconds in extreme cases. I'd say that this extra time is worth it, if the client's packets arrive more consistently now. From the testing I did so far, the extra time it took to connect was often around 1-3 seconds if my ping was no greater than 120. For 250+ ping, it took maybe around 5-10 seconds. I tried testing this thoroughly, both on a LAN server with a ping emulator (and using all kinds of different pings) and a remote server, and every single time I tested it, the client's commands arrived consistently on the server's end and they were no longer stuttering, assuming that they weren't also suffering from severe ping spikes or packet loss. While all of this looks promising, I still haven't had an opportunity to get it tested with, or by, someone else. This still needs a lot of testing before I can conclude if it works well enough, which is why I haven't created a merge request for it yet. I'd really appreciate it if you can download the the auto-generated build(s) on Heptapod and test it thoroughly yourself. Feel free to provide any feedback, if possible. If everything seems to be working properly now, I might move ahead with creating that merge request. Thanks again for your dedication and effort! |
|
unknownna (updater) 2024-11-10 23:55 |
Hey, thanks for the update and the test build, much appreciated! I tested it a bit and made some notes below: * The client lands on a wrong offset upon starting the .exe. Although this couldn't be fixed without restarting the .exe before, I noticed in the test build that it could be improved somewhat when using 2 clients to observe each other. One client appeared as smooth and centered to the other client after both reconnected to the server after several attempts. But I was unable to reproduce the improvement in another subsequent session. I also couldn't improve the client's own sync to the server by reconnecting. The offset seems to be latched to some time offset set maybe deeper in the Windows OS? I did some searching and found out that Windows changed some timer stuff starting from Windows 10, build 1809. Maybe this is part of the reason why it glitches when starting the .exe? https://www.overclock.net/threads/best-timer-resolution.1811992/?post_id=29362766#post-29362766 https://community.pcgamingwiki.com/files/file/2563-global-timer-resolution-requests-windows-11/ https://answers.microsoft.com/en-us/windows/forum/all/queryperformancefrequency-returns-10mhz-on-windows/44946807-5355-4b36-ba3e-43aa86ce30c0 * It still starts to lag between every 56ms. Every 56ms apart has the same smoothness, but when the ping drifts up/down closer to 28ms apart it causes the client to drop commands and jitter a lot. This is also observed by other clients, and can be observed through the connection strength feature on the scoreboard. The server isn't accounting for movement commands that are supposed to arrive in that one missing tick between 56ms. * The fullscreen / oversized window jitter also remains. It's map-dependent (for example MAP07 in idl201x_a.wad), maps with lots of scrolling textures and whatnot are affected. This means that it will start to jitter even if the client has a good sync towards the server. This jitter is also observed by other clients looking at your client. So, because you lag locally with "cl_capfps 0", other clients, even with "cl_capfps 1" on their end, will see you skip around. * The way the client perceives its own sync towards the server WILL AFFECT OTHER CLIENTS observing it. Why can't the client improve its own sync towards the server by reconnecting? Why does it still start to jitter when the ping moves away from every other tick, i.e., between 56ms? Would it be possible to allow the client to sync the clock in the background while in the game? It takes a bit long indeed to connect to the server now. Maybe it could also be synced during intermission screens if needed, though I'm not sure if this is needed yet. I think it still needs to be improved upon a bit before it's good enough, it doesn't feel like it fixes the issues yet, but there's a glimmer of hope here. Permanent desyncs still occur after connecting to the server. I feel like there's maybe some deeper issue at play here besides the time syncing, like there's some Windows OS timing issue. Does changing the server tickrate to 28ms from 35Hz fix or improve the issue for Windows users? I appreciate you looking into this, and hope we can figure something out. It would be so incredible and amazing if we could fix this. |
|
Kaminsky (developer) 2024-11-11 16:45 Last edited: 2024-11-11 17:13 |
Hi, thanks a lot for your response and feedback. [quote="unknownna"]The offset seems to be latched to some time offset set maybe deeper in the Windows OS? I did some searching and found out that Windows changed some timer stuff starting from Windows 10, build 1809. Maybe this is part of the reason why it glitches when starting the .exe?[/quote] According to the links that you provided, that change in the timer since Windows 10 build 1809 would affect applications that didn't explicitly set a timer resolution via timeStartPeriod. However, GZDoom and Zandronum already do this when the application is started up: // Set the timer to be as accurate as possible if (timeGetDevCaps (&tc, sizeof(tc)) != TIMERR_NOERROR) TimerPeriod = 1; // Assume minimum resolution of 1 ms else TimerPeriod = tc.wPeriodMin; timeBeginPeriod (TimerPeriod); This is setting the resolution as low as possible (which will depend on your OS and hardware). So, as far as I can tell, the timer change in the aforementioned Windows build shouldn't be an issue. Is your PC, by chance, using a power-saving mode or some other setting that would make the minimum resolution higher than 1 ms? I can confirm that on my PC, the resolution is 1 ms. [quote="unknownna"]Why can't the client improve its own sync towards the server by reconnecting?[/quote] Are you really sure that reconnecting to the server didn't improve anything, or am I missing something? Every time the client leaves a network game, it resets its clock offset back to zero, and their clock gets re-adjusted the next time they connect to the server. So, no matter what ping you connect with, it should always adjust your clock enough to make it smooth. At least from my own experience, if I drifted the client's ping by 28 ms while in-game to cause bad stuttering, then reconnected to the server with the same ping, it fixed the stuttering. [quote="unknownna"]Why does it still start to jitter when the ping moves away from every other tick, i.e., between 56ms?[/quote] Remember that the objective of the fix is to make sure that the client's commands arrive as close to the middle of a server tick (usually 28-29 ms) as possible. This is still dependent on the client's own ping. This means that during connection, the client's clock is adjusted according to whatever ping they have during that period. If you increase or decrease the client's RTT (round-trip time) by 28 ms while in-game, then their commands will now arrive 14 ms sooner or later on the server's end because that is half of their RTT. This means that instead of arriving within the middle of the server's tick (which is what we want), they're now arriving around the same time as the server ticks (which is what we don't want). Due to fluctuations in ping and tick periods on the client's end that I mentioned earlier, the client's commands start arriving inconsistently (again), causing the stuttering to relapse. However, if their RTT increases or decreases by 56 ms, then their commands will arrive 28 ms on the server's end. Even though their ping changed, the commands still arrive around the middle of the server's tick, which is what we still want. Thus, they don't stutter. To be fair, I designed this fix with the assumption that the client's ping, on average, remains stable during connection and while they're in the game. They might occassionally experience ping fluctuations or spikes that cause stuttering, but these are only temporary occurrences and their ping should return back to normal. I could try re-syncing their clock in the background while they're in the game, but this adds a whole new layer of complexity that I'm not sure is worth doing. Not to mention, adjusting the clock is best done before the client fully connects to the server when rendering isn't involved, and the traffic between the client and server is pretty low. How realistic is it that the client's ping might drastically change permanently than what it was during connection? [quote="unknownna"]The fullscreen / oversized window jitter also remains. It's map-dependent (for example MAP07 in idl201x_a.wad), maps with lots of scrolling textures and whatnot are affected. This means that it will start to jitter even if the client has a good sync towards the server. This jitter is also observed by other clients looking at your client. So, because you lag locally with "cl_capfps 0", other clients, even with "cl_capfps 1" on their end, will see you skip around.[/quote] The client's frame rate can affect the rate at which they send out commands to the server. If it's low enough, they might not be sending out commands every 28-29 ms as they should. Unfortunately, this is going to be tough to address. I don't know yet if handling this on a separate thread might improve it, or if it won't work. [quote="unknownna"]The way the client perceives its own sync towards the server WILL AFFECT OTHER CLIENTS observing it.[/quote] I apologize, but can you elaborate a bit more on this statement? I don't think I quite understand what you mean by the client "perceiving its own sync towards the server". Thanks again for the feedback! |
|
unknownna (updater) 2024-11-11 17:39 Last edited: 2024-11-12 00:15 |
[quote=Kaminsky]Are you really sure that reconnecting to the server didn't improve anything, or am I missing something? Every time the client leaves a network game, it resets its clock offset back to zero, and their clock gets re-adjusted the next time they connect to the server. So, no matter what ping you connect with, it should adjust their clock accordingly.[/quote] I find that it can in rare cases improve the sync marginally, but still not seem to fix it. It feels like it doesn't shift the offset enough or something. [quote=Kaminsky]Is your PC, by chance, using a power-saving mode or some other setting that would make the minimum resolution higher than 1 ms? I can confirm that on my PC, the resolution is 1 ms.[/quote] I'm using an older laptop that's plugged into a power source. The power settings on Windows are set to "best performance", and the battery is set to "recommended". Flipping the battery setting to "high performance" doesn't seem to change it up. [quote=Kaminsky]The client's frame rate can affect the rate at which they send out commands to the server. If it's low enough, they might not be sending out commands every 28-29 ms as they should. Unfortunately, this is going to be tough to address. I don't know yet if handling this on a separate thread might improve it, or if it won't work.[/quote] What I find strange is that this only happens online. Why does the client stutter on certain maps online only? It's not like the specs on my laptop can't handle the novel map. It feels like some bug caused by undefined behavior, similar to issue 0000459 (and also the same map even). I find it strange that you don't seem to get any issues whereas I seem to get them all the time for the most part. The sync is still terrible for PvP modes in its current state, and I think that's why it died off. Players get bad or good syncs by chance, and on top of that, if their ping happens to land on a spot that's between ticks, it stutters like there's no tomorrow. It was actually better pre-3.1 because the "blind spot" only lasted for 13-14 seconds (28ms / 2 = 14) until it normalized, and then you could hit the player properly again for at least 16-17 seconds (35Hz / 2 = 17.5) until it cycled to the same "blind spot" again and looped. [quote=Kaminsky]If you increase or decrease the client's RTT (round-trip time) by 28 ms while in-game, then their commands will now arrive 14 ms sooner or later on the server's end because that is half of their RTT. This means that instead of arriving within the middle of the server's tick (which is what we want), they're now arriving around the same time as the server ticks (which is what we don't want). Due to fluctuations in ping and tick periods on the client's end that I mentioned earlier, the client's commands start arriving inconsistently (again), causing the stuttering to relapse.[/quote] I'm sorry, but I think we're talking about different things here and misunderstanding each other. To me, it seems that the 35Hz tickrate is causing the server to expect commands from the client when the client can't deliver them, because the tickrates are different between them. There's a "blind spot" where it will lag no matter what. IF the client by some unlucky chance lands on that "blind spot" with its ping, it will jitter horribly to everyone on the server. The client prediction masks this for the client itself, but turning it off reveals the severe stutter. The offending client will in turn perceive everyone else as stuttering too. I uploaded a crude illustration of how I perceive the issue. If possible, I'd like to request a test build with the following: * Commands to manually offset the time upwards or downwards, to see if it helps with shifting the client's ticking offset when it lands between the server's ticking. * An option to flick the tickrate between 28ms and 35Hz to see if it improves the issues for me. It might fix the "blind spot between every 56ms" issue. Can you absolutely confirm to me that you never get any stuttering and jitter when playing with "cl_capfps 0", "cl_predict_players 0" and "chase" all combined at the same time? Does your player move completely smooth always, or does it jitter and stutter around? I almost never get a good sync, and have to restart the .exe until I do. What does the "gametic-lowtic" error mean? I got it once when reconnecting. [quote=Kaminsky]I apologize, but can you elaborate a bit more on this statement? I don't think I quite understand what you mean by the client "perceiving its own sync towards the server"[/quote] If you play with "cl_capfps 0", "cl_predict_players 0" and "chase" to observe how good your client's sync is, you will appear as stuttering to other smooth clients if you stutter locally. Does that make any sense? [quote=unknownna]I find that it can in rare cases improve the sync marginally, but still not seem to fix it. It feels like it doesn't shift the offset enough or something.[/quote] I can confirm that it can actually almost fix the issue if you're lucky after reconnecting, but it happens very rarely. It seems that shifting the client's time by x amount of milliseconds is the way to go to align it towards the server's ticking, but it doesn't fix or account for the "blind spot between every 56ms" issue. It might just need to be a bit more aggressive when aligning the time. |
|
unknownna (updater) 2024-11-15 04:01 Last edited: 2024-11-15 05:02 |
Hey, I noticed you made some new updates to this and went a head and tested it early. Wow, now this is incredible! I'm almost speechless. It seems that you did the impossible, congratulations! Amazing work! I tested it for a bit now and made some notes below: * I find that I sometimes had to increase or decrease the time offset by 1-2 to make it sync better in unlagged_debug_03.wad on the fast moving floor, the player would jitter every now and then after a few seconds if it wasn't perfect. If it's not too much to ask, could there also be a command to set the complete offset (setclockoffset) instead of having to manually increment up and down by 1 always? Entering it without any value would print the current offset. It would be interesting to test and see the differences to get a good feel for how the changes affect the sync. Did you also make sure that clients are only allowed to use these commands in debug builds so they can't abuse them? Subsequent testing reveals that it sometimes has to be shifted more manually to make it land right. I wonder if it needs to be more aggressive as suggested earlier and maybe offset a bit more. Will test more. * The fullscreen / oversized window issue seems to have been resolved with the new updates. Amazing! * The client jitter/stutter on certain maps (MAP02, idl201x_a.wad) hasn't been eliminated quite yet, which means that certain maps with lots of scrolling textures and floors moving up and down repeatedly will cause players to jitter slightly. Players playing with uncapped FPS and observing other clients also playing with uncapped FPS will notice it mostly, but it's still a massive improvement over the horrible jitter that was prior. Using the "idmypos" cheat and checking with "cl_predict_players 0" confirms that the client's x/y coordinates shift when it stutters, and the camera also glitches locally when it jitters, especially with "cl_capfps 1". There seems to be some undefined behavior causing this bug and should be looked into. It's especially noticeable when using +jump while changing directions. Is the player's z height sent at full precision like x/y? Do the rapid changes in textures and floor offsets collide with the rate of client-server packets? * The client spawns temporarily at the wrong place when joining the game after "changemap" map changes. It can also lead to a "connection interrupted" message that lasts for a while. I think the client erroneously inherits its position from the former map. * I find that it still stutters a very tiny bit sometimes in general even with the movebuffer on. It's very minor and trivial though and not really noticeable to others unless the player changes directions during the stutter. Does the stutter occur due to the server deciding to process 2 movement commands in a single tick? I think it happens because the underlying clock offset didn't land right. Adjusting the clock offset seems to make it better and nearly perfect. * The connection strength feature still shows 1/4 connection very often, which leads me to think it's due to the "blind spot" the client ping lands on due to the server using a timestep of 35Hz, whereas the Windows client can only have a timestep of 28ms. Maybe this makes it land somewhere outside the server tick. Like Leonard stated in the OP: [quote=Leonard]Windows timing events are only precise down to the milliseconds and as such only a delay of 28ms can be achieved due to rounding.[/quote] * There's a visual desync for respawning players. The player temporarily appears on the same spot it died on, alive for a few tics before moving to the new spawnspot. * A minor suggestion to make the wait less boring when syncing the clock, how about printing dots/periods after the message to indicate some form of progress with the syncing? I find that it sometimes takes 5 seconds or more. This is mostly for those who have to wait longer and don't want to think they're stuck. Or maybe just add some extra dots like the "authenticating level..." and "requesting snapshot..." messages. What's the main difference between sv_useticbuffer and cl_usemovebuffer now, for future reference? They seem to interact with each other, turning the ticbuffer off makes it all very stuttery now compared to before. How many milliseconds is a single offset? It's a bit confusing when the numbers sometimes appear high and other times low. What would happen if the server timestep was switched to 28ms? Would Linux clients suffer, even with the new movebuffer in place? Man, with the new near-perfect sync, even medium difficulty bots are a challenge on the test server when using high ping, unlike before when their shots hit thin air due to the rapid teleportation of my player between 2 points. The hitscan weapons hit a lot harder now indeed, and the SSG finally feels as powerful as it always should've been. Pellets hit the centered targets now. As for StrikerMan780's issues way back, I also noticed them myself at the time, but it has since been fixed for many years now. And special thanks to Leonard's Herculean efforts in fixing and mapping out the issue early on. Dude, you've unleashed a new era of online Dooming with this. The sync and unlagged has never been this good. The Megaman community will probably appreciate it a ton as well. Amazing work as always! Found another issue: * The client will be frozen for long after "changemap" map changes if the intermission lasts for the whole duration. |
|
Kaminsky (developer) 2024-11-15 17:51 |
Hey! Thanks a lot for testing the changes I pushed out early! I haven't had a chance to respond to this ticket earlier, but I appreciate you looking into it anyways. It sounds like the changes are working better than before, and I'm happy to hear that. I'm away from home at the moment, so I can't make any changes right now, but I can try answering a few things here. [quote="unknownna"]If it's not too much to ask, could there also be a command to set the complete offset (setclockoffset) instead of having to manually increment up and down by 1 always? Entering it without any value would print the current offset. It would be interesting to test and see the differences to get a good feel for how the changes affect the sync. Did you also make sure that clients are only allowed to use these commands in debug builds so they can't abuse them?[/quote] Admittedly, I only tentatively added the "incrementclockoffset" and "decrementclockoffset" CCMDs as a way to manually offset the clock up or down, so you had a way to adjust it as you saw fit, after reading your previous message. I wasn't planning on keeping these permanently, but if you think they could be useful longterm, I don't mind consolidating them into the "setclockoffset" CCMD that you proposed, and add additional checks so that it only works in debug builds. [quote="unknownna"]Subsequent testing reveals that it sometimes has to be shifted more manually to make it land right. I wonder if it needs to be more aggressive as suggested earlier and maybe offset a bit more. Will test more.[/quote] I suspect that the server is still fairly lenient when it comes to how much the client adjusts their clock. Right now, as long as 10 consecutive commands from the client arrive no less than 7 ms before/after the current/previous G_Ticker calls during the clock adjustment, the server accepts the results and proceeds with requesting the client to authenticate their map. In a server tick that's 28 ms long, this means that there's still 14 ms of wiggleroom for the client's commands to arrive in. Perhaps this tolerance needs to be shrunken down a bit? It might take longer for clients to fully connect if this is more aggressive (e.g. commands can't arrive no less than 10 ms before/after the G_Ticker calls), and in some cases, the server abort the clock adjustment if it takes too long without success. However, this can surely be fine-tuned a bit. Ideally, the client's commands should, on average, be arriving as close to the middle of the server's tick as possible. [quote="unknownna"]The fullscreen / oversized window issue seems to have been resolved with the new updates. Amazing![/quote] Awesome, thanks for checking! [quote="unknownna"]The client jitter/stutter on certain maps (MAP02, idl201x_a.wad) hasn't been eliminated quite yet, which means that certain maps with lots of scrolling textures and floors moving up and down repeatedly will cause players to jitter slightly. Players playing with uncapped FPS and observing other clients also playing with uncapped FPS will notice it mostly, but it's still a massive improvement over the horrible jitter that was prior. Using the "idmypos" cheat and checking with "cl_predict_players 0" confirms that the client's x/y coordinates shift when it stutters, and the camera also glitches locally when it jitters, especially with "cl_capfps 1". There seems to be some undefined behavior causing this bug and should be looked into. It's especially noticeable when using +jump while changing directions. Is the player's z height sent at full precision like x/y? Do the rapid changes in textures and floor offsets collide with the rate of client-server packets?[/quote] Could these be related to other issues with the clientside prediction? I recall there were a bunch of tickets opened regarding the clientside prediction causing some abnormal effects, including scrolling floors and lifts. Also, to answer your question, when the client receives commands for the movement of other players, their z-position and velocity aren't sent at full precision to save on bandwidth (the server will inform them if the player is standing on a moving lift so they don't appear jittery though). However, the server sends the client's own position/velocity at full precision. [quote="unknownna"]The client spawns temporarily at the wrong place when joining the game after "changemap" map changes. It can also lead to a "connection interrupted" message that lasts for a while. I think the client erroneously inherits its position from the former map.[/quote] [quote="unknownna"]There's a visual desync for respawning players. The player temporarily appears on the same spot it died on, alive for a few tics before moving to the new spawnspot.[/quote] I actually noticed these issues too during my own testing, and I've also seen it happen in team-based game modes when a spectator joins the game. I'll need to take a look into it and see what exactly is happening. [quote="unknownna"]I find that it still stutters a very tiny bit sometimes in general even with the movebuffer on. It's very minor and trivial though and not really noticeable to others unless the player changes directions during the stutter. Does the stutter occur due to the server deciding to process 2 movement commands in a single tick? I think it happens because the underlying clock offset didn't land right. Adjusting the clock offset seems to make it better and nearly perfect.[/quote] It could be that the server processing two of client's move commands in one tick in an attempt to empty the overfilling buffer. Right now, for EVERY tick that the client's tick buffer has THREE or MORE move commands, the server will process two commands (similar to Zandronum 3.0) to empty it faster, but I could slow this back down to FOUR commands per THREE ticks (like in 3.1) if it helps make this stutter less noticeable. Like you said before, a more aggressive clock adjustment could improve this enough too. [quote="unknownna"]The connection strength feature still shows 1/4 connection very often, which leads me to think it's due to the "blind spot" the client ping lands on due to the server using a timestep of 35Hz, whereas the Windows client can only have a timestep of 28ms. Maybe this makes it land somewhere outside the server tick.[/quote] I will probably need to adjust the connection strength feature to account for ACTUAL packet loss, instead of including CLC_CLIENTMOVE commands that arrive a tick later because of those "blind spots". Technically speaking, the client's commands are still arriving normally, just not at perfect times, but that shouldn't necessarily be an indication that the client's connection is weaker than it actually is. [quote="unknownna"]A minor suggestion to make the wait less boring when syncing the clock, how about printing dots/periods after the message to indicate some form of progress with the syncing? I find that it sometimes takes 5 seconds or more. This is mostly for those who have to wait longer and don't want to think they're stuck. Or maybe just add some extra dots like the "authenticating level..." and "requesting snapshot..." messages.[/quote] This isn't a bad idea. I'll try touching up the messages to make it look like there's some progress being made in the background better. [quote="unknownna"]What's the main difference between sv_useticbuffer and cl_usemovebuffer now, for future reference?[/quote] In principle, cl_usemovebuffer is exactly the same as sv_useticbuffer. The difference is now that the client buffers incoming player movement commands on its end, including its own, and executes only one per player, per tick (unless the buffers get to full and two commands need to be executed). Thanks again for testing this thoroughly! Indeed, this is a huge improvement. There's only one thing that concerns me: because I needed to slow down the rate at which the server empties the clients' buffers, I wonder if this might introduce a small artificial delay in the client's commands that could be undesirable. If there is an extra delay now (I'm still not 100% sure if it's actually there), then it shouldn't be more than a single tick (28 ms). While that doesn't sound too bad, will more competitive players with a keen eye notice the difference? I'm worried that this is going back to the same issues brought by the movement regulator in 3.1-alpha that many people complained about for the reason that it delayed their commands by so much. Did you notice any delays in your inputs while you tested the changes? I think the only way to be sure is to test this in an actual online game with more players and see what they have to say. |
|
unknownna (updater) 2024-11-15 20:04 Last edited: 2024-11-26 05:14 |
Thanks for the quick reply! [quote=Kaminsky]I wasn't planning on keeping these permanently, but if you think they could be useful longterm, I don't mind consolidating them into the "setclockoffset" CCMD that you proposed, and add additional checks so that it only works in debug builds.[/quote] Yeah, I'd appreciate it as a tester. It helps with checking whether the algorithm is robust. [quote=Kaminsky]It might take longer for clients to fully connect if this is more aggressive (e.g. commands can't arrive no less than 10 ms before/after the G_Ticker calls), and in some cases, the server abort the clock adjustment if it takes too long without success. However, this can surely be fine-tuned a bit. Ideally, the client's commands should, on average, be arriving as close to the middle of the server's tick as possible.[/quote] It might be worth experimenting with this indeed to see if we can make the sync even sharper. [quote=Kaminsky]Could these be related to other issues with the clientside prediction? I recall there were a bunch of tickets opened regarding the clientside prediction causing some abnormal effects, including scrolling floors and lifts. Also, to answer your question, when the client receives commands for the movement of other players, their z-position and velocity aren't sent at full precision to save on bandwidth (the server will inform them if the player is standing on a moving lift so they don't appear jittery though). However, the server sends the client's own position/velocity at full precision.[/quote] My instincts from knowing the engine rather well by now tell me it's not caused by client prediction, since it also happens when the prediction is turned off with cl_predict_players. I suspect there's some weird behavior with the processing of many shifting numbers and animation frames, causing undefined behavior or some disruption of packet flow to the client. A similar issue also causes floors to not be reset properly after map resets, see 0001971. From testing some more, I can confirm that the combination of "cl_capfps 1", "cl_predict_players 0" and "cl_usemovebuffer 1" causes it to glitch and stutter when moving across different heights or something. It feels like some conflict in the order of processing things. It's made worse on the offending maps. [quote=Kaminsky]There's only one thing that concerns me: because I needed to slow down the rate at which the server empties the clients' buffers, I wonder if this might introduce a small artificial delay in the client's commands that could be undesirable.[/quote] When you say artificial delay, do you mean movement input, like mouse and keyboard response, or do you mean a delay in actions, causing desyncs? I tested it again today and got a "missing rocket" desync when switching from the SSG to the rocket launcher twice, but couldn't reproduce it again for now, and I'm not sure yet if the desync was actually caused by the "changemap" desync. Yesterday I got some strange issue with the mouse jittering a lot with uncapped FPS, but I wasn't able to reproduce it again yet. I will say that after playing with some bots on a TeamLMS server, sometimes the jitter causes every player to be shifted dramatically to different places for a few ticks before snapping back into place. It only happens every few minutes, and I'm not sure if it's the move buffer or clock sync that causes it. I think it's the movebuffer due to the issue with the stuttering maps above. [quote=Kaminsky]While that doesn't sound too bad, will more competitive players with a keen eye notice the difference?[/quote] Do every player have the same delay, or is it dependent on each client's connection and buffer status? Personally, I find the difference to be worlds apart in terms of improvement over the former behavior. At least that's my current impression so far. [quote=Kaminsky]I'm worried that this is going back to the same issues brought by the movement regulator in 3.1-alpha that many people complained about for the reason that it delayed their commands by so much.[/quote] During much of 3.1's development I was away doing other things, so I missed out on whatever issues it might have caused for users at the time. [quote=Kaminsky]In a server tick that's 28 ms long[/quote] Are you truly sure that a server tick lasts for 28ms in real-life time instead of 35Hz? Doesn't the resulting "blind spot" between the client and server mean that one tick actually lasts for different durations between the client and server? If the server truly has a tickrate/timestep of 35Hz, doesn't that mean that one tick is effectively longer on the server than on the client? That's why I'm so curious as to whether changing the timestep to 28ms on the server fixes this blind spot between the two. * Other players you observe getting pushed by rocket explosions will jitter a little with "cl_usemovebuffer 1". It's mostly noticed when playing with capped FPS and "compat_explosionthrust 1", that uses the old ZDoom blast physics with a very strong thrust. [quote=Kaminsky]There's only one thing that concerns me: because I needed to slow down the rate at which the server empties the clients' buffers, I wonder if this might introduce a small artificial delay in the client's commands that could be undesirable.[/quote] After testing it again some more tonight and comparing back and forth by toggling the buffer on/off fast, I think I can confirm that there's unfortunately some extra mouse delay when playing with "cl_usemovebuffer 1". There might even be movement delay too. It's more noticeable when playing with uncapped FPS. And I also got the issue with the mouse being jittery again yesterday with uncapped FPS, though I haven't figured out yet what causes it. I figured out what makes the mouse break with uncapped FPS: It's by simply decrementing the clock offset. Incrementing it doesn't break it for some reason. This breaks the mouse permanently until the .exe is restarted. You can't fix it by reconnecting. |
|
Kaminsky (developer) 2025-03-18 18:57 |
I'll look into this again after 3.2's released. |
|
unknownna (updater) 2026-03-29 10:54 Last edited: 2026-03-29 11:00 |
Kaminsky, long time no see! I finally got around to test your new fixes and have the following thoughts: * The new buffer behavior mitigates the fundamental issues, but it doesn't solve it unfortunately. * The client still has to be lucky to land at the right tick offset upon connecting to the server. * Since there's no clock adjustment in the latest fixes, the client jitters a bit more than before. I think the issue boils down to: * The client and server tick at different rates in the sense that one tick lasts for different durations due to the way computers work, like Leonard said. 35.75Hz vs 28ms. This causes the client to drift away from the server until it loops around and catches up again. This is the old, pre-3.1 behavior. * To combat this, Leonard made the client and server tick at the same rate, i.e., 35Hz, and made them locked together. However, since the Windows client apparently HAS to round it down to 28ms due to the way computers work, a single tick still lasts for different durations between the client and server. What seems to happen here is that the server exe itself is able to simulate 35Hz, but a game client is not. I think this is the main cause of the "blind spot" between the two, where the client very often is forced to land between 2 server ticks, leading to what what AlexMax described as a permanent "jackhammering" effect, where the client jitters back and forth between 2 ticks indefinitely. I might be wrong about this particular point, but this is what I observe at the moment. * On top of this, due to relativistic differences in time in real-life, a unit of time doesn't occur at the same time in real-life, meaning that a client will actually land a bit behind in general compared to the server upon connecting and latching on to it. This relativistic difference seems to be somewhere between 0-84ms (28ms x 3), i.e., anywhere between 0 to 3 ticks , with 3 ticks off being the worst sync possible. This is why the 1st proposed fix of extrapolating players (sv_smoothplayers) only fixed minor desyncs at best, and could only mitigate the worst desyncs a little, and would only work by chance depening on where the client tick offset landed relative to the server's. * Because of the points above, the client, when connecting to the server, might land at an offset that's between 0 and 3 ticks off, with 0 being a perfect sync without any jitter, and 3 being the worst offender, causing the client to perceive itself and others as laggy, and also affects the way other clients observe the client. To summarize, the client affects both itself and others due to these issues. If it wouldn't be too much to ask, would it be possible to have a build made where the tickrate is forced to be 28ms instead of 35Hz? It would be interesting to see if that would actually fix the tick duration "blind spot" between the client and server. And I suppose the clock adjustment got removed due to the mouse issue discovered earlier? The one where decrementing the clock introduced mouse jitter? Anyway, there's almost nothing more I'd love than having this fundamental issue fixed, but the fixes so far still don't seem to account for some certain fundamental issue, mainly the issue of real-life time relativity and the client and server seemingly ticking at different rates of time. So here's what I think. Since the client in its current state is actually able to land on a perfect sync by chance, I firmly believe that with the right fixes and adjustments taking this chance occurrence into account, the client could be made to always land on a perfect sync, leading to a much more enjoyable gaming session, and would in a sense revolutionize the state of the game. If nothing else, it would be a good intellectual exercise to have it fixed. I recorded 2 short demos with no special wads using the latest 3.3 alpha build (3.3-alpha-260112-1855) with an emulated ping of 160 that very clearly demonstrate and display the differences in what syncs are attainable. The 1st demo showcases the client landing on a perfect spot, where there's no jitter. The 2nd demo shows a typical sync, where the client lands between ticks and jitters/jackhammers permanently. Is there anything in the metadata of the demos that can indicate what the differences between the syncs are? Note that to verify for yourself that you have a proper working fix for this issue when making fixes, you MUST always observe your own player with the following conditions: * cl_predict_players set to 0 * cl_capfps set to 0 * cl_buffercommands set to 0 * chasecam set to 1 (enabled) If the client moves around smoothly without jittering at all under these conditions, the sync is perfectly solid. If the client jerks around however, then the sync is off. |
|
Kaminsky (developer) 2026-03-29 14:11 |
Welcome back, unknownna! [quote="unknownna"]However, since the Windows client apparently HAS to round it down to 28ms due to the way computers work, a single tick still lasts for different durations between the client and server. What seems to happen here is that the server exe itself is able to simulate 35Hz, but a game client is not. I think this is the main cause of the "blind spot" between the two, where the client very often is forced to land between 2 server ticks, leading to what what AlexMax described as a permanent "jackhammering" effect, where the client jitters back and forth between 2 ticks indefinitely. I might be wrong about this particular point, but this is what I observe at the moment.[/quote] Both the client and server tick 35 times a second, but the reason that a server is able to simulate 35 Hz almost perfectly, with every tick having the same duration (give or take a millisecond), is because the server doesn't need to worry about rendering; it only needs to handle game logic and networking, which can usually be done with little to no fluctuations in the duration of every tick. In contrast, clients must render their game on top of everything else, and depending on what their frame rate currently is (whether it's capped or limited by their hardware), every tick is going to have varying durations (e.g. one tick could be 25 ms, another could be 30 ms) to ensure that 35 ticks pass in exactly one second. [quote="unknownna"]If it wouldn't be too much to ask, would it be possible to have a build made where the tickrate is forced to be 28ms instead of 35Hz? It would be interesting to see if that would actually fix the tick duration "blind spot" between the client and server.[/quote] The problem is that the duration of a tick is tied to the client's frame rate, and each frame's duration must be consistent to achieve smoothness, at the cost of making some ticks shorter or longer than others. To force every tick to 28 ms means that each frame's duration must keep changing, which would only make stuttering more apparent than before. Likewise, we can't force every tick to be 28 ms while keeping frame durations consistent, or else the clients won't tick at 35 Hz anymore, thereby returning us to the original problem of this ticket years ago. [quote="unknownna"]And I suppose the clock adjustment got removed due to the mouse issue discovered earlier? The one where decrementing the clock introduced mouse jitter?[/quote] Not only that, but also from experience, trying to adjust with the client's clock to achieve a "perfect" sync is actually far more complicated and inefficient than it sounds. I'll provide a few reasons: 1. Incrementing or decrementing a client's clock by one millisecond doesn't necessarily mean they'll send or receive commands from the server one millisecond sooner or later. From my experience, a lot of guesswork is involved with trying to find the "perfect" sync by incrementing or decrementing the client's clock, which is far from ideal. In fact, it's possible that a good sync might not even be attainable by adjusting the client's clock. 2. Adjusting the client's clock to achieve a good sync only works one way, not both ways. What that means is that a good sync from client to server can be achieved by adjusting the client's clock, but the sync from server to client will still be bad. This isn't ideal either. 3. If the client's frame rate or connection changes during the course of the game, that throws the clock sync off, so it must then be re-synced over and over. Doing this constantly will be noticeable on the client's end. 4. Messing with the client's clock in general, even on a small scale, causes all kinds of trouble that simply makes it not worth the hassle. The mouse jitter was only one issue that got discovered, and it's a question of how many more side effects this can cause, which might not get noticed until later. Moreover, I can't seem to find real examples of other FPS games that use this approach to achieve a perfect sync. Most games, as far as I know, buffer commands to achieve smoothness, which is what the server/client command buffers already do. Of course, it helps when a game has a higher tickrate such as 60 Hz, unlike Doom's which is 35 Hz and pretty slow by today's standards. Nonetheless, buffering commands does a more effective job at smoothing the movement of players and isn't complicated to do. [quote="unknownna"]So here's what I think. Since the client in its current state is actually able to land on a perfect sync by chance, I firmly believe that with the right fixes and adjustments taking this chance occurrence into account, the client could be made to always land on a perfect sync, leading to a much more enjoyable gaming session, and would in a sense revolutionize the state of the game. If nothing else, it would be a good intellectual exercise to have it fixed.[/quote] Isn't making everyone move smoothly more important to a "much more enjoyable gaming session" than trying so hard to "land on a perfect sync" every time? Isn't that what the buffers already accomplish, or am I missing something? So far from testing that 3.3 has received, I haven't heard anyone reporting that their experience is worse with the buffers now in use. [quote="unknownna"]I recorded 2 short demos with no special wads using the latest 3.3 alpha build (3.3-alpha-260112-1855) with an emulated ping of 160 that very clearly demonstrate and display the differences in what syncs are attainable. The 1st demo showcases the client landing on a perfect spot, where there's no jitter. The 2nd demo shows a typical sync, where the client lands between ticks and jitters/jackhammers permanently. Is there anything in the metadata of the demos that can indicate what the differences between the syncs are?[/quote] I had a look at your two demos. The jackhammering effect in "tick_offset_wrong.cld" was only noticeable when "cl_predict_players" is disabled, which I'd argue is a setting that clients are typically not going to want to disable because it introduces delays and choppy movement (it's not in the menus either). When that CVar's enabled, the player moves smoothly. "cl_capfps" or "cl_buffercommands" seem to have no effect on this. Can you confirm this? Unfortunately, there's no way to tell what the differences between the syncs are. Ultimately, what I'm arguing is that trying to adjust the client's clock to sync perfectly with the server is not as feasible as it seems, and trying to do this requires an overly complicated solution that brings with it all kinds of side effects that make it not worth the effort. |
|
unknownna (updater) 2026-03-30 10:00 Last edited: 2026-03-30 10:02 |
Thank you for your comments and insights. [quote=Kaminsky]The problem is that the duration of a tick is tied to the client's frame rate, and each frame's duration must be consistent to achieve smoothness, at the cost of making some ticks shorter or longer than others. To force every tick to 28 ms means that each frame's duration must keep changing, which would only make stuttering more apparent than before. Likewise, we can't force every tick to be 28 ms while keeping frame durations consistent, or else the clients won't tick at 35 Hz anymore, thereby returning us to the original problem of this ticket years ago.[/quote] Regarding the request for a test build with a server tickrate of 28ms, I'm curious regarding Leonard's earlier comment in the OP, where he states that it fixes the issue for Windows clients, but makes it worse for Linux clients: [quote=Leonard]By making the server loop use a proper time step of 28ms, the issue is fixed: lNewTics = static_cast<LONG> ( lNowTime / 28.0 );[/quote] I'd be interested in testing whether this removes the underlying jackhammer effect where the client constantly fluctuates between 2 ticks. If it indeed fixes the issue, maybe it would confirm the difference in a tick's duration between the client and server. Maybe it would confirm that the client is forced to land between ticks because the start- and end-points of a server tick and client tick differ in length, so they can't agree on which tick the client belongs to. Anyway, I can't know anything for sure unless I'm able to test it for myself. [quote=Kaminsky]Isn't making everyone move smoothly more important to a "much more enjoyable gaming session" than trying so hard to "land on a perfect sync" every time? Isn't that what the buffers already accomplish, or am I missing something? So far from testing that 3.3 has received, I haven't heard anyone reporting that their experience is worse with the buffers now in use.[/quote] Forgive me, of course the buffer makes the current experience very enjoyable and works well for all intents and purposes. The current behavior is like a dream compared to the former behavior, and I'm very grateful for your hard work. Under normal circumstances, it seems to do the job of removing most of the jitter, and the clients perceive each other as being smooth. I am however observing the fundamental issue that the buffer is masking, so I don't quite consider the issue completely fixed yet on a technical level. I sincerely believe it should be possible to improve it even further if one could only control, harness and make consistent how the clients align themselves to the server, which currently only happens by chance. It would indeed probably require a more surgical approach for somebody willing to undertake one day. [quote=Kaminsky]I had a look at your two demos. The jackhammering effect in "tick_offset_wrong.cld" was only noticeable when "cl_predict_players" is disabled, which I'd argue is a setting that clients are typically not going to want to disable because it introduces delays and choppy movement (it's not in the menus either). When that CVar's enabled, the player moves smoothly. "cl_capfps" or "cl_buffercommands" seem to have no effect on this. Can you confirm this?[/quote] A client's own cl_capfps value affects the way other clients observe it, with uncapped FPS introducing observable jitter to other clients, whereas capped FPS removes it. "cl_buffercommands 1" almost fixes issue completely, but one can still see that clients with a bad underlying sync playing with "cl_capfps 0" jitters a bit every now and then to other observing clients. If the offending client then enables capped FPS with "cl_capfps 1", it will stop jittering to other observing clients. Uncapped clients will in general jitter more to other clients. I observe this with 2 clients using an emulated ping of 160 in unlagged_debug_03.wad. It might sound a bit confusing and is indeed confusing to test, but clients affect other observing clients more if they play with uncapped FPS. On the observers' end, their cl_capfps value also let's them see a bit more additional jitter if it's uncapped. [quote=Kaminsky]The jackhammering effect in "tick_offset_wrong.cld" was only noticeable when "cl_predict_players" is disabled, which I'd argue is a setting that clients are typically not going to want to disable because it introduces delays and choppy movement (it's not in the menus either).[/quote] The prediction and buffer settings being off is to observe how the raw underlying sync is from my understanding without needing to connect a 2nd client to observe your own client. When the raw sync is good, the whole gameplay session is pristine, PROVIDED that the ping doesn't deviate by 28ms backwards or forwards. This also sounds confusing, but it's what I observed earlier a year ago or so, where the ping you're synced to is smooth every 56ms apart, but every 28ms is very laggy and jittery. You should be able to confirm this for yourself if you want to by connecting to a server with an emulated ping of 160 and hope that you land on a good tick offset, like in tick_offset_correct.cld. [quote=unknownna]So for a good lucky sync at 160 ping, these pings seem to be as smooth: 46: Good and smooth. -57ms 103: Good and smooth. -57ms 160: Baseline. 216: Good and smooth. +56ms 272: Good and smooth. +56ms 328: Good and smooth. +56ms 384: Good and smooth. +56ms 440: Good and smooth. +56ms[/quote] If you get a good sync at 160 ping, the pings listed above will retain the same smoothness, but any ping between will introduce jitter, with most jitter being present at 28ms between the pings listed. It sounds confusing, and maybe it's a whole separate sync issue I'm not able to convey properly. I'm curious as to what you think of this. On a separate note, I found a bug with the unlagged when testing with unlagged_debug_03.wad: If you run into the player in the same direction of the scrolling floor and aim away from the player in the opposite direction, you can still hit the other player even though the player is behind you. It's maybe related to the bug here: https://zandronum.com/tracker/view.php?id=4202 |
|
unknownna (updater) 2026-04-25 15:08 |
I found another reproducible pattern using the earlier clock sync build, and wanted to make a note of it here after thinking about it. Forgive me if I sound like a broken record at this point, but here goes: [b]1.[/b] The server exe allegedly uses "time polling" at 35.75Hz. This frequency causes the timestep to be rounded [i]down[/i] to 28ms, making the game tick slightly faster. Windows game clients use "Windows OS time events", and are obligated to be at 28ms due to integer rounding issues. This frequency causes the timestep to possibly be rounded [i]up[/i] to 35.75ms, making the game tick slightly slower. With the mismatched tickrate/timestep frequencies, the server will, for any given tick, always want to be slightly ahead of the client. 35.714285714ms - 27.972027972ms = 7.742257742ms If my assumption is correct, a server tick is always 7.7ms shorter in length on average compared to a client tick. This should be the "blind spot" I'm always talking about, where if the client lands here, the client and server will disagree on which tick the client belongs to, causing jackhammering. This is also why the jackhammer effect is additionally seen at 28ms between good syncs: Because the client drifts between server ticks here. Since the server's timestep is effectively shorter (35.75Hz = 28ms), it's already ahead at this point, and wants the client to be "on the next tick", which is actually the server's present tick. The client's timestep (28ms = 35.71428ms) makes it lag behind the server here, and it wants to be "on the former tick" instead, which is actually the client's own present tick. Both the client and server probably think they're on the same tick, but due to the mismatched frequencies between them, it causes them to disagree on which tick the client is on. The bottom line is, if the frequencies and their amplitudes don't match, it will always jackhammer due to latency drift, as the individual ticks between the client and server don't have the same fundamental lengths in their start- and end-points. [b]2.[/b] On top of this, it's probably a relativistic clock sync issue. I found out in my past experiments that the client could at worst be jackhammering between 3 ticks, suggesting that it's not only a frequency mismatch, but also a relativistic one. This is where the clock sync works great in theory, however, due to the mismatched tickrate/timestep frquencies still being in effect between the client and server, all the pings listed below will always cause the clock sync to fail 100%, as the client is [i]guaranteed[/i] to land between server ticks at these particular latencies. You can confirm this with the builds that contain the clock adjustment: 28 56 +28ms 84 +28ms 112 +28ms 140 +28ms 168 +28ms 196 +28ms 224 +28ms etc. It will always fail to sync the clock properly and will cause jackhammering when connecting to the server with these pings. You can fix it manually after connecting by adjusting the ping by 28ms backwards or forwards, but the fundamental issue still remains: The time frequencies don't match, so the client will land between server ticks because of this. Conversely, the following pings will always yield a proper clock sync upon connecting to the server, because the client is guaranteed to not land between server ticks at these latencies: 14 (half of 28) 42 +28ms 70 +28ms 98 +28ms 126 +28ms 154 +28ms 182 +28ms 210 +28ms etc. However, this time it's the opposite issue; adjusting the ping by 28ms backwards or forwards after connecting makes it start jackhammering, because now you force the client to land between server ticks again. I'm probably mixing up "tick" with "tic", adding to the confusion. My apologies. What I mean is that a basic unit of time in the engine, i.e., a tic, differ in length fundamentally between the client and server, leading to the issue that this ticket cannot seemingly resolve. I'd say that, by matching the tickrate/timestep frequencies properly between the client and server, and then possibly locking them in with a basic clock sync alignment in case the client lands way off, it should fix this issue 100% for Windows clients at least. Then the new buffer will smooth and mask out any tiny jitter and make the overall sync razor sharp under stable conditions. There's really nothing more one could do after that. Maybe you won't even need any clock sync if the tickrate/timestep frequencies are matched properly. Anyway, hope the new pattern found helps. There's a clear recurring pattern of having a good sync, and then it breaking if the latency drifts 28ms apart from it. Somehow the tickrate/timestep frequencies aren't matched, or the server is only processesing client ticks properly every other tick (56ms). In any case, there's a glaring blind spot there still remaining. * Mismatched time frequencies. * Minor relativistic clock sync issue. * The server only processing client ticks properly every other tick, i.e., every 56ms. Maybe caused by mismatched time frequencies. |
|
unknownna (updater) 2026-06-16 16:40 Last edited: 2026-06-16 17:21 |
[quote=unknownna]* The server only processing client ticks properly every other tick, i.e., every 56ms. Maybe caused by mismatched time frequencies.[/quote] After thinking about this further, I think my last assumption was wrong. Although the pattern found was correct and reproducible, it was probably not caused by mismatched ticking frequencies alone. I now believe that even if you match the tickrate frequencies 100% and combine it with a clock sync, that it still won't be able to fix the issue completely. I'll try to explain it better below. What I think is happening is that the server is constantly alternating between sending packets one tick, and then receiving and processing packets the other tick. The loop goes like this: [code]1st tick: 2nd tick: send packet -> receive packet, process it, and update server positions -> loop[/code] It spends the 1st tick sending packets, and then spends the 2nd tick processing received packets and updating positions of actors on the server. This is easy to understand. The problem here, is that whenever a connecting client's latency causes it to land and start ticking for itself and send updates within a 2nd server tick, where the server is processing packets and updating positions of actors on the server, that it causes the server to become "blind" to the client, because the client is now sending updates at the same time in parallel to the server processing packets for itself. The root of the matter is that the client can't be sending updates at the same timeframe in realtime as the server when the server is updating positions of actors for itself. It causes a causality error and a phase cancellation effect. This phase cancellation effect manifests as a client jackhammering between multiple ticks. You can think of it as an erroneous and exaggerated form of jitter. To illustrate the issue better, I'll show 2 examples where it assumes that you're using the clock sync build to connect: [b]When you connect with a ping of 28ms and jackhammer upon connecting, it goes like this:[/b] [code]Server: 1st tick: 2nd tick: send packet -> receive packet, process it, and update server positions -> loop Client: 1st tick: 2nd tick connect -> send packet -> receive packet -> loop[/code] As you can see above, after connecting, the client starts to tick and send updates parallelly to the 2nd server tick, where the server is updating positions of actors for itself on the server. Since the client connected with a ping of 28ms, it connected and latched to the server during the 2nd server tick. The client and server are effectively occupying the same timespace in realtime when the client sends updates and the server updates positions on the server, causing a cancellation effect on the client player body location on the server. [b]When you connect with a ping of 14ms and have a near-perfect sync, it goes like this:[/b] [code]Server: 1st tick: 2nd tick: send packet -> receive packet, process it, and update server positions -> loop Client: 1st tick: 2nd tick connect -> send packet -> receive packet -> loop[/code] As you can see above, the 1st ticks of the client and server now line up and match, so they don't cancel each other out. The server is able to see the client updates properly since the client isn't occupying the same timespace in realtime as the server when the client is sending updates and the server is updating actor positions on the server. So thinking of a possible fix for this, does the server behavior have to be changed so the server is always at least 1 tick ahead of every client when processing packets, so that client updates are guaranteed to never collide with server updates as illustrated above? Maybe the server should never be processing packets parallelly within the same time of a client sending them. So to fix it, the server has to be made to always be 1 tick ahead in time to account for this so it doesn't run parallel with client updates when updating actor positions on the server. To summarize, a client update always has to be guaranteed to be in the past compared to the server when the server is updating positions, otherwise they'll occupy the same space in realtime and cancel each other out. Ideally, we wish to guarantee perfect smoothness of a client for every 28ms (every tick) valley/peak of a server tick the client's latency and clock offset lands close to instead of only for every 56ms (every other tick). By letting the server update the positions of actors always 1 tick ahead of any client update currently occurring, the server should then be able to see the client updates properly and thus not cancel them out between every tick. What do you think? Does this sound reasonable, and could it be made possible? * Lower the server tickrate and Linux client timesteps to match the Windows clients at 28ms. Windows clients are the "weakest" of the bunch and aren't able match the higher tickrate of the server and Linux clients. This should assure that a unit of time doesn't differ in length between any client on any server, preventing desyncs. * Implement the clock sync. This is to align the client's ticking offset towards the server's to remove any jitter and to prevent the client from landing between server ticks. Any natural latency drift beyond that should be considered a natural form of jitter that the new buffer will take care of. * Make the server update the positions of actors on the server always 1 tick ahead of any client at all times so that client updates don't collide with server updates and cancel each other out. This should prevent the server from being "blind" to client updates between every 56ms valley/peak. If I'm still wrong about this even after all this time, then all I can think of is some potential duty cycle mismatch between the client and server, but I don't think it's likely. I think it's more likely that the server has to be ahead in time so it doesn't cancel out client updates occurring parallel to it when updating actor positions on the server. [quote=Kaminsky]If the client's frame rate or connection changes during the course of the game, that throws the clock sync off, so it must then be re-synced over and over. Doing this constantly will be noticeable on the client's end.[/quote] I see what you mean, but this shouldn't be necessary, because one can assume that clocks advance at roughly the same rate between computers, so they should only have to be aligned once to get the client and server ticking aligned towards one another. One then has to assume in good faith, that any latency drift beyond the initial sync is an acceptable form of jitter that the new buffer will take care of. We just have to line up the client ticking towards the server to prevent any unnecessary jitter and to prevent the client from landing between server ticks. If you keep re-syncing the clock for any latency change, it will create jitter, because you then shift the client back and forth in time again. And then it still takes additional time to send that to the server and to other clients again, causing even more jitter. It should be good to just sync it once upon connecting, and then let the buffer take care of the rest caused by any natural latency drift. At most, I'd only re-sync the clock if the client somehow knows that it drifted too far away for too long from its initial ping. Without any clock sync present, there's guaranteed to be jitter present depending on the client's ticking offset towards the server. And for what it's worth, I don't think your clock sync algorithm is broken, Kaminsky, though it might be interacting with external ping emulators slightly: * With a ping emulator set to 0ms, you still have to manually either increment the clock offset by 8, or decrement it by 16, specifically, to create a proper sync of the ticking offset towards the server. |
|
unknownna (updater) 2026-08-11 02:56 |
Kaminsky, there's a chance my last assumption above could potentially be somewhat wrong. I found another reproducible pattern, where a client will start to jackhammer more if another client disconnects, and jackhammer less if the same client reconnects, and the reverse. This is using the clock sync build, and is harder to reproduce reliably in regular builds without any clock sync. If the 1st client connects with a ping to deliberately jackhammer, and then lowers/raises its ping by 28ms to remove the jackhammering, the 2nd client connecting with the same ping will start to jackhammer a lot more when the 1st client subsequently disconnects. Conversely, if the 1st client connects with a ping to prevent jackhammering, and then lowers/raises its ping by 28ms to start jackhammering, the 2nd client connecting with the same ping will stop jackhammering when the 1st client subsequently disconnects. To confirm the patterns, "cl_predict_players" + "cl_buffercommands" / "cl_usemovebuffer" + "cl_capfps" should all be set to 0, combined with enabling the chasecam with "chase". [b]To make the new client jackhammer more when old client disconnects:[/b] 1. Connect client A to the server with a ping of 56ms and join the game. It will be jackhammering. 2. Lower the ping by 28ms to 28ms to stop jackhammering. 3. Connect client B to the server with the ping above of 28ms and join the game. It may be jackhammering, but a lot less than client A did initially. 4. Disconnect client A from the server and notice how client B starts to jackhammer a lot more. 5. Reconnect client A to the server and notice how client B starts to jackhammer less. [b]To make the new client stop jackhammering when old client disconnects:[/b] 1. Connect client A to the server with a ping of 70ms and join the game. It will be smooth. 2. Lower the ping by 28ms to 42ms to start jackhammering. 3. Connect client B to the server with the ping above of 42ms and join the game. It will be jackhammering. 4. Disconnect client A from the server and notice how client B stops jackhammering. 5. Reconnect client A to the server and notice how client B starts to jackhammer a lot. Why are the clients affecting each other's tick/clock offsets upon connecting and disconnecting? It's like a separate bug on top of the issues I tried to explain in my former posts above. And why would 2 clients sharing the same ping be able to have different tick offsets, where one client can be jackhammering, whereas the other client is completely smooth, provided that it raised/lowered its ping by 28ms before the other client connected and joined? I can understand if it's not currently possible to implement any clock sync due to the sheer difficulty in doing so in addition to the side-effects it might bring, but it would be nice if we could at least somehow fix the issue of the server being blind to client updates between every 56ms valley/peak of latency on the server, and fix this issue of clients affecting each other's tick/clock offsets upon connecting/disconnecting from the server. |
| Notes |
| Issue History |
|||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-11-08 00:52 | Leonard | New Issue | |
| 2017-11-08 00:52 | Leonard | Status | new => assigned |
| 2017-11-08 00:52 | Leonard | Assigned To | => Leonard |
| 2017-11-08 00:52 | Leonard | File Added: 28ms.cld | |
| 2017-11-08 00:53 | Leonard | Description Updated | View Revisions |
| 2017-11-08 00:57 | Leonard | File Added: 35hz.cld | |
| 2017-11-08 00:59 | Leonard | Relationship added | parent of 0002859 |
| 2017-11-08 10:54 | Edward-san | Note Added: 0018816 | |
| 2017-11-08 17:26 | Leonard | Note Added: 0018818 | |
| 2017-11-08 20:25 | Edward-san | Note Added: 0018820 | |
| 2017-11-10 02:26 | Ru5tK1ng | Note Added: 0018850 | |
| 2017-11-10 18:33 | Leonard | Note Added: 0018851 | |
| 2017-11-10 18:33 | Leonard | Status | assigned => needs review |
| 2017-11-10 20:17 | Blzut3 | Note Added: 0018852 | |
| 2017-11-10 21:11 | Blzut3 | Note Edited: 0018852 | View Revisions |
| 2017-11-11 01:19 | Leonard | Note Added: 0018853 | |
| 2017-11-11 01:19 | Leonard | Description Updated | View Revisions |
| 2017-11-13 09:16 | Leonard | Relationship added | related to 0001633 |
| 2017-11-13 14:08 | Leonard | Status | needs review => assigned |
| 2018-01-23 01:47 | Ru5tK1ng | Note Added: 0019006 | |
| 2018-01-23 20:04 | Leonard | Note Added: 0019011 | |
| 2018-01-24 01:40 | Ru5tK1ng | Note Added: 0019012 | |
| 2018-01-24 01:40 | Ru5tK1ng | Note Edited: 0019012 | View Revisions |
| 2018-02-11 19:45 | Leonard | File Added: jack.cld | |
| 2018-02-11 19:45 | Leonard | File Added: jitter.cld | |
| 2018-02-11 19:46 | Leonard | Note Added: 0019031 | |
| 2018-03-25 15:19 | Leonard | Note Added: 0019158 | |
| 2018-03-25 15:19 | Leonard | Status | assigned => needs review |
| 2018-03-25 15:22 | Leonard | Relationship added | related to 0002491 |
| 2018-04-30 15:47 | Leonard | Note Added: 0019177 | |
| 2018-04-30 15:47 | Leonard | Status | needs review => needs testing |
| 2018-05-02 16:05 | Leonard | Relationship added | related to 0003418 |
| 2018-05-07 02:07 | StrikerMan780 | Note Added: 0019211 | |
| 2018-05-07 02:08 | StrikerMan780 | Note Edited: 0019211 | View Revisions |
| 2018-05-07 03:09 | StrikerMan780 | Note Edited: 0019211 | View Revisions |
| 2018-05-07 05:58 | StrikerMan780 | Note Edited: 0019211 | View Revisions |
| 2018-05-07 05:58 | StrikerMan780 | Note Edited: 0019211 | View Revisions |
| 2018-05-07 09:46 | Leonard | Note Added: 0019215 | |
| 2018-06-03 22:39 | StrikerMan780 | Note Added: 0019267 | |
| 2018-06-03 22:39 | StrikerMan780 | Note Edited: 0019267 | View Revisions |
| 2024-03-06 12:27 | unknownna | Note Added: 0023304 | |
| 2024-03-06 12:58 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-06 16:05 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-07 09:40 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-07 22:38 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-07 22:39 | unknownna | Status | needs testing => feedback |
| 2024-03-09 02:49 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-09 02:49 | unknownna | File Added: unlagged_debug_03.wad | |
| 2024-03-12 05:17 | unknownna | Priority | normal => high |
| 2024-03-12 11:49 | unknownna | Note Edited: 0023304 | View Revisions |
| 2024-03-12 15:13 | Kaminsky | Note Added: 0023377 | |
| 2024-03-13 14:08 | unknownna | Note Added: 0023381 | |
| 2024-03-14 06:22 | unknownna | Note Edited: 0023381 | View Revisions |
| 2024-03-14 08:19 | unknownna | Note Edited: 0023381 | View Revisions |
| 2024-03-14 08:24 | unknownna | Note Edited: 0023381 | View Revisions |
| 2024-03-14 08:26 | unknownna | Note Edited: 0023381 | View Revisions |
| 2024-04-03 10:01 | unknownna | Note Edited: 0023381 | View Revisions |
| 2024-04-06 00:22 | Kaminsky | Note Added: 0023521 | |
| 2024-04-06 00:22 | Kaminsky | Assigned To | Leonard => Kaminsky |
| 2024-04-06 00:22 | Kaminsky | Status | feedback => assigned |
| 2024-04-06 00:22 | Kaminsky | Target Version | 3.1 => 3.2 |
| 2024-04-06 10:00 | unknownna | Note Added: 0023523 | |
| 2024-06-08 15:43 | Kaminsky | Note Added: 0023756 | |
| 2024-06-08 18:01 | unknownna | Note Added: 0023757 | |
| 2024-06-08 18:01 | unknownna | Status | assigned => feedback |
| 2024-06-09 08:31 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-09 08:42 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-09 08:49 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-09 08:58 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-09 18:21 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-12 14:13 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 13:50 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 14:23 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 14:31 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 14:40 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 14:41 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-13 20:41 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-16 14:31 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-16 17:21 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-16 19:31 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-17 14:22 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-24 18:46 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-24 19:28 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-24 21:46 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-26 17:21 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-29 22:49 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-29 22:55 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-06-29 23:14 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-07-03 04:17 | unknownna | Note Edited: 0023757 | View Revisions |
| 2024-09-24 22:19 | unknownna | Note Added: 0024050 | |
| 2024-10-19 23:14 | unknownna | Note Edited: 0024050 | View Revisions |
| 2024-10-22 05:44 | unknownna | Note Edited: 0024050 | View Revisions |
| 2024-11-10 21:39 | Kaminsky | Note Added: 0024130 | |
| 2024-11-10 21:39 | Kaminsky | Status | feedback => needs review |
| 2024-11-10 21:44 | Kaminsky | Note Edited: 0024130 | View Revisions |
| 2024-11-10 23:55 | unknownna | Note Added: 0024135 | |
| 2024-11-11 16:45 | Kaminsky | Note Added: 0024136 | |
| 2024-11-11 17:13 | Kaminsky | Note Edited: 0024136 | View Revisions |
| 2024-11-11 17:39 | unknownna | Note Added: 0024137 | |
| 2024-11-11 17:41 | unknownna | File Added: tickissue.png | |
| 2024-11-11 23:58 | unknownna | Note Edited: 0024137 | View Revisions |
| 2024-11-12 00:15 | unknownna | Note Edited: 0024137 | View Revisions |
| 2024-11-12 18:16 | unknownna | File Added: tickissue_better.png | |
| 2024-11-15 04:01 | unknownna | Note Added: 0024138 | |
| 2024-11-15 05:02 | unknownna | Note Edited: 0024138 | View Revisions |
| 2024-11-15 17:51 | Kaminsky | Note Added: 0024144 | |
| 2024-11-15 20:04 | unknownna | Note Added: 0024147 | |
| 2024-11-17 08:17 | unknownna | Note Edited: 0024147 | View Revisions |
| 2024-11-23 23:07 | unknownna | Note Edited: 0024147 | View Revisions |
| 2024-11-26 05:14 | unknownna | Note Edited: 0024147 | View Revisions |
| 2024-11-27 09:15 | unknownna | Relationship added | related to 0001971 |
| 2025-02-15 23:07 | unknownna | Relationship added | related to 0002294 |
| 2025-03-18 18:57 | Kaminsky | Note Added: 0024256 | |
| 2025-03-18 18:57 | Kaminsky | Status | needs review => assigned |
| 2025-03-18 18:57 | Kaminsky | Target Version | 3.2 => |
| 2025-08-08 23:00 | Kaminsky | Target Version | => 3.3 |
| 2026-03-29 10:53 | unknownna | File Added: tick_offset_correct.cld | |
| 2026-03-29 10:53 | unknownna | File Added: tick_offset_wrong.cld | |
| 2026-03-29 10:54 | unknownna | Note Added: 0024687 | |
| 2026-03-29 11:00 | unknownna | Note Edited: 0024687 | View Revisions |
| 2026-03-29 14:11 | Kaminsky | Note Added: 0024688 | |
| 2026-03-30 10:00 | unknownna | Note Added: 0024692 | |
| 2026-03-30 10:02 | unknownna | Note Edited: 0024692 | View Revisions |
| 2026-03-30 10:09 | unknownna | Relationship added | related to 0004595 |
| 2026-04-01 16:32 | unknownna | Relationship deleted | related to 0004595 |
| 2026-04-25 15:08 | unknownna | Note Added: 0024697 | |
| 2026-05-18 02:09 | unknownna | Relationship added | related to 0001384 |
| 2026-06-16 16:40 | unknownna | Note Added: 0024767 | |
| 2026-06-16 16:41 | unknownna | Note Edited: 0024767 | View Revisions |
| 2026-06-16 16:57 | unknownna | Note Edited: 0024767 | View Revisions |
| 2026-06-16 17:00 | unknownna | Note Edited: 0024767 | View Revisions |
| 2026-06-16 17:03 | unknownna | Note Edited: 0024767 | View Revisions |
| 2026-06-16 17:21 | unknownna | Note Edited: 0024767 | View Revisions |
| 2026-08-11 02:56 | unknownna | Note Added: 0024772 | |
| Issue History |
