0002330: Additional Latency/Jitter with 2.1

Notes
(0012786) Torr Samaho (administrator) 2015-06-25 05:42	On a LAN server, both 2.0 and 2.1 have a ping of 1 for me. Can anybody confirm the latency problems with 2.1?

(0012790) Untitled (reporter) 2015-06-25 13:02	Yeah, I've been experiencing this too, I don't know what's up. I tend to ping 10-20 ms of lag on BestEver, but I was getting spikes of up to 80-100 ping in a quantity that would be considered common. In addition to that, there were (rare) spikes of about 180-220 ping. Considering I normally ping 10-20 ms of lag on BE, that's not a terribly good sign. Plus I found my movement to be somewhat jitterly, just on the whole. I don't really have anything useful aside from a witness account, sadly.

(0012791) Jenova (reporter) 2015-06-25 16:19	I would just like to add thats its entirely possible that it was just a lag spike on BE. Can anyone confirm this issue on multiple servers? I played a game of CTF and two rounds of TDM last night and didn't notice any issues.

(0012792) Razgriz (reporter) 2015-06-25 17:09	Dastan and others were talking about that in NJ scrim server, I know Ivan was having a lot more problems he never had on 2.0 compared to 2.1.

(0012794) Konar6 (reporter) 2015-06-25 18:16	Couldn't it be just connection problem?

(0012798) Untitled (reporter) 2015-06-25 18:46	I don't think it would be a connection problem on my end, if only because this wasn't happening in Zandronum 2.0, which I was playing literally that morning of Zandronum 2.1, which was noticeably skippier - unless something changed with the netcode, there's nothing I can see that I did that would've induced extra lag. Then again, I don't really know too much about how ANYTHING works, so it could well be on me - I don't know. I really don't.

(0012804) AlexMax (developer) 2015-06-25 23:06	I have upgrade most of my FUNCRUSHER servers to 2.1, but have left the ZDA duel server on 2.0. If anybody can demonstrate latency or jitter in 2.1 that does not occur when you play on the 2.0 server (please test the two relatively close together), please update this ticket.

(0012805) Ivan (reporter) 2015-06-25 23:32	I'm completely positive there're no connection problems, my net is fiber and there have been no instances of any packetloss or slowdowns. I also know there are no routing problems currently and when I played on 2.1 on NJ.

(0012806) Watermelon (developer) 2015-06-26 14:03 edited on: 2015-06-26 14:13	I am having this problem, my ping for some reason is also in the 60's on NJ when it's normally 29-40ms. I also get random weird fast warping around the map. It actually feels like the ticbuffer is somehow doing extra prediction on me and sending that data to my client. Is it possible some commands get duplicated in the buffer? Its weird because I will be running along normally and get some random speed boost that feels a lot like the ticbuffer. It's always a speed boost, not like I freeze and then I'm caught up. Of course not saying it is the ticbuffer, but even if NJ was going to hell... I shouldn't be getting extra speed boosts but rather be pulled back, or not notice anything since clientside prediction wouldn't notice any differences when the official movement comes back to the client.

(0012810) Torr Samaho (administrator) 2015-06-27 12:20	Quote from AlexMax If anybody can demonstrate latency or jitter in 2.1 that does not occur when you play on the 2.0 server (please test the two relatively close together), please update this ticket. This. Please do so.

(0012814) Frits (reporter) 2015-06-27 20:08 edited on: 2015-06-27 21:17	Yes, this is a huge problem in 2.1 on higher ping servers for me such as BE.i'm constantly jittering around. Fighting monsters, players and dodging projectiles becomes quite annoying like this. It's quite horrible to play at higher then 50 ping atm.

(0012815) Edward-san (developer) 2015-06-27 21:05	I wonder if the issue happens because of certain client or server settings. I asked Frits and BE on IRC to check their configurations to nail the issue down.

(0012817) Watermelon (developer) 2015-06-28 02:55	I'm getting this on multiple servers. Is this possibly related to changing the ticbuffer to process 1 at a time instead of 2?

(0012818) Torr Samaho (administrator) 2015-06-28 06:19	The ticbuffer change (previously it processed 2 commands at the end of a tic, now it is supposed to processes the first command immediately when receiving it and the second one at the end of the tic) may have an effect on how the player movement feels, but it shouldn't change the displayed ping in any way. I'm still waiting for someone who tested 2.0 and 2.1 on FUNCRUSHER one after another (preferably going back and forth between the versions) to confirm this.

(0012819) Watermelon (developer) 2015-06-28 16:54	I'm asking Alex to set up a CTF 2.0 server. As soon as he does, I'll let you know the results. As a note: I was able to get the jittering happening in 2.1 by myself in a CTF server on NJ. Now to wait for 2.0...

(0012820) Torr Samaho (administrator) 2015-06-28 17:05	According to 0002330:0012804, the FUNCRUSHER ZDA duel server is still be on 2.0. If so, shouldn't this be enough to test? Or is that not the case anymore?

(0012821) Watermelon (developer) 2015-06-28 18:39 edited on: 2015-06-28 18:39	Sorry I put this on IRC but not here, let me paste: <Water`> AlexMax it doesnt appear to happen frequently in duels <Water`> for some reason, CTF is a lot more painful, when i was playing duels on 2.1, it was okay for the most part <Water`> for the duel server, i couldn't tell the difference between random packet loss and there being a bug, but in CTF it's a lot more ridiculous For me, it appears to be happening in CTF servers. I didn't notice much (or anything) last night when doing duels.

(0012823) Watermelon (developer) 2015-06-29 00:11	Tested 2.0, it was fine

(0012824) Torr Samaho (administrator) 2015-06-29 06:14	So you compared 2.0 and 2.1 back and forth on the same host? Was the ping different or the same? Does the problem only happen in CTF? Does it matter how many players are in the server?

(0012825) Watermelon (developer) 2015-06-29 13:44 edited on: 2015-06-29 13:45	Yes, tested 2.0 and 2.1 on the same host. My ping would actually raise randomly to 60-70 (usual: 30) on Funcrusher when this happened. Sometimes my ping doesn't change at all. The player I tested with had no complaints on 2.0. It appears to happen way more in CTF. It also appears to happen when more players are in the server. IMO I'd like to play on Best Ever using CTF in 2.1. I still cannot say I'm fully convinced it is not due to NJ being really bad as of late. If you can give me until the weekend (tons of midterms this week), I'll have a much clearer picture.

(0012830) unknownna (updater) 2015-07-06 20:42 edited on: 2015-07-08 23:12	I've been away for a while so I didn't get to test 2.1 properly on real servers until today. I can most definitely confirm that there's a lot more jitter present in 2.1 compared to 2.0. Like Watermelon, I think the problems once again could be caused by the tic buffer. It doesn't happen all the time, but I notice it relatively often, and it seems to happen when there's a lot of things going on. But I'm unable to reproduce the jitter reliably on my own server. I also noticed that other clients seem to skip a lot more in 2.1 compared to 2.0.

(0012832) Torr Samaho (administrator) 2015-07-07 06:04	Quote from unknownna I've been away for a while so I didn't get to test 2.1 properly on real servers until today. I can most definitely confirm that there's a lot more jitter present in 2.1 compared to 2.0. On which host did you see the problems? By now we started to wonder whether it's a problem of the host and not of Zandronum itself. Quote from unknownna I also noticed that other clients seem to skip a lot more in 2.1 compared to 2.0. Well, the "smoothing" done by the tic buffer caused the weapon sync issues, so I had to partially revert it to fix weapon sync. To confirm whether the altered tic buffer causes these issues I'd say we should again create a new testing binary that allows to turn the buffer on an off.

(0012837) unknownna (updater) 2015-07-07 17:34	So far I observed the jitter on both the Best Ever and the FUNCRUSHER servers.

(0012839) Torr Samaho (administrator) 2015-07-07 18:03	Can you elaborate what you mean with jitter? From what I can gather there seem to be two different effects reported: 1. increased latency / lag 2. less smooth player movement The tic buffer changes shouldn't cause #1, but are likely to cause 0000002.

(0012841) unknownna (updater) 2015-07-07 19:30 edited on: 2015-07-07 20:14	I personally didn't notice any increase in ping. The jitter can best be described as having less smooth player movement. You shake/warp/teleport around very small distances for a few seconds or so. But I must say that playing on a Grandvoid AOW server seemed to work relatively fine. Perhaps Best Ever and FUNCRUSHER suffer from packet issues. It feels like the packets are congested before passing through, thus causing this jitter to occur.

(0012843) AlexMax (developer) 2015-07-07 21:52	I have shut off my CS:GO server to see if perhaps the jitter could be an indirect result of that. However, all this talk about jitter makes me curious - can "jitter" be quantified somehow? Ping can, as can packetloss (which I believe we should have an indicator for as well), but is it feasible that the server could detect when a connection is jittery and do something to indicate this on the scoreboard? What measurement would be used to indicate this? This document gives clues that it might not be as simple as taking a standard deviation of a sample... 'http://zeromq.org/whitepapers:measuring-jitter [^]'

(0012844) unknownna (updater) 2015-07-08 00:02 edited on: 2015-07-08 00:05	AlexMax, I had a few CTF games on your servers after you posted this and while it indeed works a bit better now, it's still not good enough. I have a ping of 40-50 on your german CTF server. There's no way that I'd ever jitter around like this pre-2.1. In general, the player movement should only jitter when packets are lost or when there's an obvious client-side misprediction taking place, like for instance here. I'd say that what seems to be a potential underlying packet issue is currently manifesting itself as jitter through the new tic buffer behavior. I could be wrong though.

(0012845) Zalewa (developer) 2015-07-08 21:41 edited on: 2015-07-08 21:41	Here's a video showcasing this effect: 'http://youtu.be/YDK7RLTWYd0 [^]' In the video, the observed player (yellow) is playing remotely with average ping of 25. The observer plays locally. Please don't forget that we have tools that can emulate network conditions: 'http://sourceforge.net/projects/gamersproxy/ [^]' Perhaps this could help reproduce the issue more reliably?

(0012846) unknownna (updater) 2015-07-08 22:41 edited on: 2015-07-08 22:52	Ok, thanks to Zalewa's newer ping emulator, I think I found the cause: Ping spikes. For some reason 2.1 handles ping spikes badly, and since they happen very frequently on Best Ever and FUNCRUSHER servers in general, the client jitters around all the time. Edit: I can confirm that it's the tic buffer that's causing the jitter to occur.

(0012862) Torr Samaho (administrator) 2015-07-12 07:43	Can you check whether the jitter is any different in this binary? The binary also allows to turn off the ticbuffer with "sv_useticbuffer 0". Can you confirm that the jitter doesn't occur when the tic buffer is turned off?

(0012863) unknownna (updater) 2015-07-12 07:57	The jitter is still present in that binary. I can confirm that the jitter doesn't occur when the tic buffer is turned off.

(0012864) Torr Samaho (administrator) 2015-07-12 07:59	Can you make a short demo of the jitter with that binary? If you are using Gamer's Proxy to get the jitter, please also post all proxy settings.

(0012865) unknownna (updater) 2015-07-12 08:08 edited on: 2015-07-12 08:14	Sure. * The ping is combined at 40 ms. * Ping spike chance is 50.00%. * Max ping spike strength is 400.00%. If you increase this, it jitters even more. zandronum -iwad doom2.wad -playdemo ticbuffer_jitter.cld

(0012867) Torr Samaho (administrator) 2015-07-12 08:14	I see, this is a completely different effect from what Zalewa was showing in 0002330:0012845. I'll investigate.

(0012868) Torr Samaho (administrator) 2015-07-12 08:30 edited on: 2015-07-12 08:30	Ok, how is this binary? EDIT: If it fixes the jitter, can you check that it doesn't break the weapon sync again?

(0012869) unknownna (updater) 2015-07-12 08:50 edited on: 2015-07-12 08:59	That seems to have taken care of the jitter, great job! Regarding the weapon sync, I managed to desync the weapons three times with an emulated ping of 251 when switching back and forth between the rocket launcher and the BFG while rapidly pressing +attack. On a side note, I noticed that it's very easy to desync the weapons when using the proxy settings posted in my former note. This doesn't happen when the tic buffer is disabled.

(0012870) Torr Samaho (administrator) 2015-07-12 09:16	Is the weapon sync with the testing binary at least still better than in 2.0? Quote from unknownna On a side note, I noticed that it's very easy to desync the weapons when using the proxy settings posted in my former note. This doesn't happen when the tic buffer is disabled. With these settings, the movement commands often arrive in wrong order at the server, this is bound to have some side effects. If disabling the tic buffer completely fixes the weapon sync issues though, we'll have to check if the sorting does more harm than good.

(0012871) unknownna (updater) 2015-07-12 09:34	Yes, it's most definitely still better than 2.0. The weapon sync was absolutely horrendous in 1.3/2.0 compared to now. Disabling the tic buffer seems to completely fix the weapon sync issues (of course not those that we didn't fix yet). I'd personally rather have rock-solid movement prediction and weapon synchronization instead of having slightly smoother opponents, but that's just my opinion.

(0012872) Torr Samaho (administrator) 2015-07-12 10:02	Quote from unknownna Disabling the tic buffer seems to completely fix the weapon sync issues (of course not those that we didn't fix yet). Can you check how the weapon sync and the jitter is in this binary? It doesn't sort the movement commands, but still has the tic buffer. Quote from unknownna I'd personally rather have rock-solid movement prediction and weapon synchronization instead of having slightly smoother opponents, but that's just my opinion. I don't like having to make a compromise like this either, but some people were constantly complaining that opponents with a bad connection are constantly teleporting instead of moving and thus very difficult to hit.

(0012873) Zalewa (developer) 2015-07-12 10:22 edited on: 2015-07-12 10:23	Quote from "Torr Samaho" Some people were constantly complaining that opponents with a bad connection are constantly teleporting instead of moving and thus very difficult to hit. When I play on that private server I have a constant ping of 25 with nearly perfectly reliable connection and I can't hear the end of how much I'm warping... We'll test your new binaries and inform you whether they fix the issue.

(0012874) unknownna (updater) 2015-07-12 10:23 edited on: 2015-07-12 10:27	Quote from Torr Samaho Can you check how the weapon sync and the jitter is in this binary? It doesn't sort the movement commands, but still has the tic buffer. That binary works marginally better, but it's still very bad under those circumstances. It desyncs very quickly. You should just keep the sorting if that's all there is to gain from not sorting the commands. Quote from Torr Samaho I don't like having to make a compromise like this either, but some people were constantly complaining that opponents with a bad connection are constantly teleporting instead of moving and thus very difficult to hit. I can understand the complaints as I've been observing this behavior many times as well. But I still observe this behavior in 2.1, which is probably what Zalewa's linked YouTube video showcases. I'd say that the weapon sync works very well under stable conditions, but when ping spikes occur, it's very prone to desyncing, and since the ping spikes happen very frequently on certain server hosts, it could potentially desync rather often. On the other hand, since I haven't personally noticed any major weapon sync problems when playing on real servers in 2.1 so far, I'd say that we should keep the tic buffer for now.

(0012876) Torr Samaho (administrator) 2015-07-12 11:00	Quote from Zalewa When I play on that private server I have a constant ping of 25 with nearly perfectly reliable connection and I can't hear the end of how much I'm warping... I wasn't referring to these issues in 2.1, but to the effects caused by large ping spikes on unstable connections. These were the reason why I introduced the tic buffer and from the feedback we got on 1.3 and 2.0, I'd say that the tic buffer was an improvement in this regard. 2.1 on the other hand definitely has a bug in the tic buffer (that is hopefully fixed now) causing jitter. It just took a while to figure out what's going on because a large chunk of the initial reports were misleading (the tic buffer doesn't increase latency). Quote from unknownna I can understand the complaints as I've been observing this behavior many times as well. But I still observe this behavior in 2.1, which is probably what Zalewa's linked YouTube video showcases. From what I understood, this effect is supposedly worse in 2.1 than it was in 2.0. Hopefully, the fix for the jitter of the local player also restores the behavior of 2.0 regarding Zalewa's report. Quote from unknownna On the other hand, since I haven't personally noticed any major weapon sync problems when playing on real servers in 2.1 so far, I'd say that we should keep the tic buffer for now. I'm very glad to hear that.

(0012880) unknownna (updater) 2015-07-12 11:41 edited on: 2015-07-12 11:50	Quote from Torr Samaho From what I understood, this effect is supposedly worse in 2.1 than it was in 2.0. Hopefully, the fix for the jitter of the local player also restores the behavior of 2.0 regarding Zalewa's report. Yes, it's worse in 2.1, but even with your recent fixes, other unstable clients appear as teleporting/skipping around.

(0012881) Torr Samaho (administrator) 2015-07-12 12:12	So even with the fixes it's worse than in 2.0? Does the tic buffer at least improve the situation, if you compare by turning off the tic buffer in the binary with the recent fixes?

(0012882) unknownna (updater) 2015-07-12 12:22	Yes, it's worse than in 2.0 with the fixes. It's a bit worse if you turn off the tic buffer. It's very difficult to tell whether or not it has improved from 2.1.

(0012883) Torr Samaho (administrator) 2015-07-12 13:02	Can you make short demos, one in 2.0 and one in 2.1.1 that allow to compare the effect in both version? And can elaborate everything that is necessary to reproduce the situation shown in your demos?

(0012884) unknownna (updater) 2015-07-12 13:18 edited on: 2015-07-12 13:19	Done. zandronum -iwad doom2.wad -playdemo ticbuffer_jitter_2.0.cld zandronum -iwad doom2.wad -playdemo ticbuffer_jitter_2.1.1.cld The unstable client uses these proxy settings with Gamer's Proxy: * The ping is combined at 40 ms. * Ping spike chance is 50.00%. * Max ping spike strength is 400.00%.

(0012885) Torr Samaho (administrator) 2015-07-12 13:28	Thanks! The recording player has a direct LAN connection to the server? What's the cl_ticsperupdate setting of the recording player?

(0012886) unknownna (updater) 2015-07-12 13:30 edited on: 2015-07-12 13:31	No problem. I have cl_ticsperupdate set to 1. Yes, the recording player has a direct LAN connection to the server.

(0012887) Torr Samaho (administrator) 2015-07-12 14:03	To me it looks as if both demos show the same effect: The observed player moves in one direction, then jumps a little backwards and continues its movement in the original direction. For some reason, it happens more often in 2.1.1 though. Note: This is on a much smaller scale than the teleporting I was referring to earlier. Does this effect only happen if the observed player regularly changes its direction (like in your video) or also while the player moves in the same direction for a longer period of time?

(0012888) unknownna (updater) 2015-07-12 14:20	Quote from Torr Samaho Does this effect only happen if the observed player regularly changes its direction (like in your video) or also while the player moves in the same direction for a longer period of time? It also happens when the player moves in the same direction for a longer period of time. It's noticeably worse in 2.1.1 compared to 2.0 though, which makes it considerably more difficult to hit the unstable client properly, particularly in close combat. IMHO, it kind of defeats the purpose of having a tic buffer when unstable clients skip around.

(0012889) Torr Samaho (administrator) 2015-07-12 14:39	Quote from unknownna It also happens when the player moves in the same direction for a longer period of time. Then, I have no idea right now why the backwards jumps happen. I suspected that the movement commands arrive at the server in wrong order and are processed in wrong order, which would make a player jump backwards in case a movement command originating from before a direction change is processed after the direction change. This can't happen in the "same direction movement" case though. Quote from unknownna It's noticeably worse in 2.1.1 compared to 2.0 though, which makes it considerably more difficult to hit the unstable client properly, particularly in close combat. IMHO, it kind of defeats the purpose of having a tic buffer when unstable clients skip around. The main purpose of the tic buffer is to prevent large jumps, which it should still do since at most two movement commands are processed in a single tic. It cannot completely hide the effects of a bad connection, but I don't understand yet why 2.1.1 is so much worse than 2.0 in this regard.

(0012890) Torr Samaho (administrator) 2015-07-12 15:10	I remembered one thing that could cause the backwards jumps, but it should affect 2.0 and 2.1.1 in the same manner: The server sends player position updates to the clients using an unreliable buffer. One of the effects is that if the updates arrive in wrong order at the client, the client will show the player at the position contained in the last update it received, not based on the gametic associated to the position (since the server simply didn't send this information). This binary should take care of this. Can you check if this has any effect?

(0012891) unknownna (updater) 2015-07-12 15:20 edited on: 2015-07-12 15:25	That binary seems to be just as choppy as 2.1/2.1.1 compared to 2.0. It feels even choppier than 2.1/2.1.1, but I'm not quite sure about this.

(0012892) Torr Samaho (administrator) 2015-07-12 15:27	Then the "server -> client" part of the player position update seems to be not the cause of the issue. Can you check that it has nothing to do with unlagged by using "sv_nounlagged 1"?

(0012893) unknownna (updater) 2015-07-12 15:30	It's still choppy with sv_nounlagged set to 1.

(0012894) Torr Samaho (administrator) 2015-07-12 15:48	Here is an attempt to get a better compromise between the weapon sync fix in 2.1 and the movement smoothness of 2.0. Untested so far.

(0012896) unknownna (updater) 2015-07-12 16:22	Yeah, this works a lot better. Nice! It's smoother than 2.1.

(0012897) Torr Samaho (administrator) 2015-07-12 16:26	Great! And the weapon sync? Still better than in 2.0?

(0012898) unknownna (updater) 2015-07-12 16:32 edited on: 2015-07-12 16:38	Yeah, the weapon sync seems to be fine. I managed to desync it 3 times so far with an emulated ping of 251, but the 3rd time it seemingly corrected itself. I managed to desync it once more. It didn't correct itself this time. And managed to desync it 2 more times. Hmm, let's hope that this works well.

(0012899) Torr Samaho (administrator) 2015-07-12 16:45	As long as the sync is still better than in 2.0 I think it's fine. As I said earlier, the tic buffer inevitably is a compromise.

(0012900) unknownna (updater) 2015-07-12 16:50	One more question: Is it possible to make it a client option as well? I'd probably want to disable it in certain situations if possible, i.e. in situations where I need 100% weapon sync.

(0012901) Torr Samaho (administrator) 2015-07-12 17:02	Well, the tic buffer affects the actual position of a player on the server, it's not a purely visual change on the client end. Technically, you could allow a client to decide whether its player is handled by the tic buffer or not, but this would defeat at least part of the purpose of the tic buffer: IIRC there were CTF players with a very unstable connection who were very difficult to hit and thus had an unfair advantage when carrying the flag. Those players would surely opt-out of the tic buffer and we get the problem of unhittable players again.

(0012902) unknownna (updater) 2015-07-12 17:09	I see. But then you simply need to add another server variable that decides whether or not clients should be allowed to disable the tic buffer, right?

(0012904) AlexMax (developer) 2015-07-12 17:16 edited on: 2015-07-12 17:30	As a server administrator, I would rather a compromise be struck than potentially having to put up alternative tic buffer/no tic buffer servers. By the way, if you need an easy way to test for weapon desync, BFGSHOOT in Duel40 is a really easy way to make said desyncs appear. Play it against a bot, and you can make the BFG ball desync from the weapon animation really easily over the course of normal play.

(0012905) Torr Samaho (administrator) 2015-07-12 17:33	Quote from unknownna But then you simply need to add another server variable that decides whether or not clients should be allowed to disable the tic buffer, right? Yes, that would work, but makes things quite complicated. Are cases where one could convince the server admin that players should be able to opt out of the tic buffer but "sv_useticbuffer 0" is not an option likely to occur? Quote from AlexMax By the way, if you need an easy way to test for weapon desync, BFGSHOOT in Duel40 is a really easy way to make said desyncs appear. So weapon sync problems still happen in 2.1 over the course of normal play on that map?

(0012906) unknownna (updater) 2015-07-12 17:38	Quote from AlexMax As a server administrator, I would rather a compromise be struck than potentially having to put up alternative tic buffer/no tic buffer servers. As a player who takes his skill seriously, I'd rather have 100% response if given the choice. Why should I suffer because some random person has an unstable connection to the server? I'd hate to die in duel for instance because I happened to have a ping spike at the moment I picked up a BFG and fired it with nothing coming out of it. Quote from Torr Samaho Yes, that would work, but makes things quite complicated. Are cases where one could convince the server admin that players should be able to opt out of the tic buffer but "sv_useticbuffer 0" is not an option likely to occur? Why would we even need sv_useticbuffer if clients would be allowed to disable the tic buffer at will? Quote from Torr Samaho So weapon sync problems still happen in 2.1 over the course of normal play on that map? I reproduced it in the latest build a few times with an emulated ping of 251. But it could also be caused by this issue since it happens at spawn.

(0012907) Torr Samaho (administrator) 2015-07-12 17:59	Quote from unknownna Why would we even need sv_useticbuffer if clients would be allowed to disable the tic buffer at will? Actually, I only added sv_useticbuffer for debugging purposes. I hoped the compromise made by the tic buffer to be good enough so that it doesn't need to be turned off. That's also why 1.3 and 2.0 didn't have an option to turn it off. I fully agree though that weapon sync issues are a serious problem. Perhaps we can further improve the compromise.

(0012910) unknownna (updater) 2015-07-13 04:16	May I ask why you insist on having it on at all times when it's fundamentally flawed to begin with in terms of sync? Quote from Torr Samaho IIRC there were CTF players with a very unstable connection who were very difficult to hit and thus had an unfair advantage when carrying the flag. Those players would surely opt-out of the tic buffer and we get the problem of unhittable players again. Using the tic buffer as an "anti-cheat" device sounds really weird to me. That's why there should be at least 3 options: * sv_ticbuffer (Odamex has this.) * sv_allowticbuffertoggle (This is for those who use the tic buffer as an "anti-cheat" device.) * cl_ticbuffer This is the only compromise that I can see work. This way, hosts can control whether clients should be allowed to disable the tic buffer. Sure thing, by allowing clients to bypass the buffer you make room for the ping-spike-movement-teleport-exploit, but it always worked like this until 1.3 since Skulltag's birth anyway. Quote from Torr Samaho That's also why 1.3 and 2.0 didn't have an option to turn it off. Which is kind of funny considering how horrible those versions felt and played in terms of the weapon sync. Pardon my aggression here, it's just that I'm very passionate about sync issues in general, particularly the weapon sync issues. I strongly believe that the sync should be 100% since this is a FPS first and foremost.

(0012911) Frits (reporter) 2015-07-13 09:13 edited on: 2015-07-13 13:38	I agree with unkownna here. An FPS should be very responsive and you should at least strive to be able to maintain perfect control over your own actions at all times. Zan 2.1 has been a step back in this regard. High spikers are more hitable now, but what good is that if I keep jittering around in the map and can't even reliably move and shoot at all? EDIT: I ment having either the (self)jittery movement or weapon desyncs is equally horrible unknownna. I'd rather go back to teleporting players.

(0012913) unknownna (updater) 2015-07-13 12:06	Frits, the jitter has already been taken care of, but the compromise is that by having smoother players (other players that you see running around), you end up with less than perfect weapon sync, which means that you could for instance switch from the SSG to the rocket launcher and fire with nothing coming out of the rocket launcher due to a desync caused by fluctuating ping. By disabling the tic buffer you won't get the weapon desync, but unstable players will then skip around instead. That's why I'd like to be able to toggle it at will. If I'd play with stable clients I'd turn it off, but if I'd see someone who skipped around a lot and also happened to be my opponent, I'd turn it on.

(0012915) Ivan (reporter) 2015-07-13 13:31	Is there no way to fix both the client problems to a minimum and the weapon desyncs? They are both equally annoying and frustrating.

(0012917) Torr Samaho (administrator) 2015-07-13 17:11	Quote from unknownna That's why I'd like to be able to toggle it at will. If I'd play with stable clients I'd turn it off, but if I'd see someone who skipped around a lot and also happened to be my opponent, I'd turn it on. I think you misunderstood what the client could decide. The only thing a client could decide is whether the tic buffer is applied to its own player body. That means if you turn it off, the tic buffer related sync problems will not affect your weapons anymore, but this doesn't have any effect on the other player bodies. So if you see somebody else skipping around, there is nothing you can do (except for asking the player to turn on its tic buffer) since only the corresponding client can decide whether the tic buffer is applied to its body. Quote from Ivan Is there no way to fix both the client problems to a minimum and the weapon desyncs? They are both equally annoying and frustrating. We didn't understand the cause of the sync problems in all its details yet, so I can't answer this yet. We need to investigate further.

(0012918) Torr Samaho (administrator) 2015-07-13 17:22	Can you check if this has any effect on the weapon sync problems related to the tic buffer? Chances are not very high, but it won't hurt to give it a try.

(0012921) unknownna (updater) 2015-07-14 02:25 edited on: 2015-07-14 03:54	Quote from Torr Samaho I think you misunderstood what the client could decide. [...] I see. Indeed, I got it wrong. But still, I'd still like to be able to turn it off if I noticed that my weapons were desyncing. Too bad that it would affect other's view of my client as well. Quote from Torr Samaho Can you check if this has any effect on the weapon sync problems related to the tic buffer? Chances are not very high, but it won't hurt to give it a try. It still desyncs very quickly when ping spikes occur, as expected, but under stable conditions (using an emulated ping of 251) it seems to work nearly as good as 2.1. I managed to make it desync 6 times so far, but it seemed to correct itself 3 times. It's still not as good as 2.1 though. Could you tell us what you did in order to improve the sync? Weapon sync aside, unstable clients still seem to be moving relatively smoothly on stable clients. BTW, does this fix also include this? Quote from Torr Samaho I remembered one thing that could cause the backwards jumps, but it should affect 2.0 and 2.1.1 in the same manner: The server sends player position updates to the clients using an unreliable buffer. One of the effects is that if the updates arrive in wrong order at the client, the client will show the player at the position contained in the last update it received, not based on the gametic associated to the position (since the server simply didn't send this information). I think this could hypothetically improve player-to-player collision and reduce the jitter that occurs when bumping into other players if done right. Ok, you win, but I'd appreciate it if you at least kept sv_useticbuffer. It'd also make it a lot easier to track potential weapon desyncs unrelated to the tic buffer. Speaking of which, it seems that the sv_useticbuffer value is archived. Edit: I noticed that it desyncs very badly after "changemap" map changes in DM. In particular, it breaks badly when using the PK3s found in this ticket. This does not happen in 2.1 or when sv_useticbuffer is set to 0.

(0012923) Zalewa (developer) 2015-07-14 06:02	We made another video showcasing the problem that still persists in version 2.1.1: 'https://www.youtube.com/watch?v=fZQGKCon8ZE [^]' This was recorded over net play with ping slightly increased with my proxy program. As you can see, player sometimes moves smoothly, and sometimes jitter is clearly visible. I'm gonna do some more experiments in my LAN network with my laptop, but I don't doubt I'm gonna get similar results.

(0012924) Torr Samaho (administrator) 2015-07-14 06:12	Quote from unknownna It still desyncs very quickly when ping spikes occur, as expected, but under stable conditions (using an emulated ping of 251) it seems to work nearly as good as 2.1. I managed to make it desync 6 times so far, but it seemed to correct itself 3 times. It's still not as good as 2.1 though. So this version is worse regarding weapon sync than 2.1 (but better than 2.0), but less jittery than 2.1? Is it better than what we plan to release as 2.1.1? Quote from unknownna Could you tell us what you did in order to improve the sync? When processing a tic from the buffer the server now thinks (at least to a certain degree) it is processing a network packet from the corresponding client. That's just some internal change to check if the code makes any assumptions that are not fulfilled when the movement command comes from the buffer instead of being processed directly when parsing a network packet. Quote from unknownna BTW, does this fix also include this? No, since this didn't seem to improve the situation but does increase net traffic, I removed it again. In case, you want to make more experiments with it, I can add it again though. Quote from unknownna Ok, you win, but I'd appreciate it if you at least kept sv_useticbuffer. It'd also make it a lot easier to track potential weapon desyncs unrelated to the tic buffer. Speaking of which, it seems that the sv_useticbuffer value is archived. We can definitely keep sv_useticbuffer. And you are right, sv_useticbuffer has the archive flag. I think I'll just adjust the history entry accordingly. Looks like we still need to get a better understanding of why the weapon sync problems occur. I'll think about this and possibly make a testing binary with a lot of debug output.

(0012925) Torr Samaho (administrator) 2015-07-14 06:35	Quote from Zalewa We made another video showcasing the problem that still persists in version 2.1.1: Can you make a short client side demo showing the same situation under 2.0 and 2.1.1?

(0012926) unknownna (updater) 2015-07-14 07:06 edited on: 2015-07-14 08:58	Quote from Torr Samaho So this version is worse regarding weapon sync than 2.1 (but better than 2.0), but less jittery than 2.1? Is it better than what we plan to release as 2.1.1? The build seems to be marginally smoother than what you plan to release as 2.1.1, but it's difficult to say for sure. I could hardly make ZandroDev2.1.1-TicBufferExperiments6.7z desync when testing it this time compared to zandronum2.1.1-win32-base.zip. However, ZandroDev2.1.1-TicBufferExperiments6.7z and zandronum2.1.1-win32-base.zip both suffer from what potentially seems to be a serious showstopper bug: Quote from unknownna I noticed that it desyncs very badly after "changemap" map changes in DM. In particular, it breaks badly when using the PK3s found in this ticket. This does not happen in 2.1 or when sv_useticbuffer is set to 0. What happens is that the client and server completely desync when it comes to switching weapons. For instance, puffs could be spawned when you're selecting/deselecting weapons. This also happens with a direct LAN connection to the server. One thing I also observed is that clients will display the lag icon above their sprites for longer periods of time very often if the tic buffer is enabled, even when they aren't lagging anymore. I managed to reproduce this reliably as well by clicking and holding the Zandronum window for a few seconds to induce the fake lag. This doesn't happen in 2.1. Regarding the weapon desync, for some reason the weapons fire twice as fast on the server-end after "changemap" map changes on certain maps. This is really bad.

(0012932) Zalewa (developer) 2015-07-14 17:27	@Torr_Samaho I've added "zalewa_warpdemo_zandronum.zip" attachment with two demos you requested. To record this demo I used my desktop PC and my laptop, which were connected through 1GB cable LAN. You can see a lot of jitter in 2.1.1 demo and there's also some in 2.0, although much less.

(0012933) Torr Samaho (administrator) 2015-07-14 17:33	Quote from unknownna Regarding the weapon desync, for some reason the weapons fire twice as fast on the server-end after "changemap" map changes on certain maps. This is really bad. That sounds pretty bad. Is it any different in 2.0? Regarding the lag icon: Is this any better?

(0012934) Hypnotoad (reporter) 2015-07-14 17:48	Just thought I'd say, I've been testing a 2.1.1 server put up on TSPG-Exciter, where I ping 180 to, and have experienced no issues with jitter yet personally (although I've been playing on my own in the server).

(0012935) unknownna (updater) 2015-07-14 18:20	Quote from Torr Samaho That sounds pretty bad. Is it any different in 2.0? It's a bit better in 2.0, but the latest build improved it for some reason. It still desyncs though. But 2.0 and 2.1.1 are probably the same now after "changemap" map changes. 2.1 is still better than both though. Quote from Torr Samaho Regarding the lag icon: Is this any better? That took care of it. For some reason it seemed to improve the "changemap" map change desync as well. The weapons no longer fire twice as fast.

(0012936) Torr Samaho (administrator) 2015-07-14 18:55	I just had an idea what could be causing the weapon sync problems: The tic buffer only handles movement commands, it does not handle any other commands (e.g. weapon switching, inventory using, etc.). So if you turn the buffer on, movement commands (which include attack button presses) will be potentially deferred, but weapon change commands are always executed immediately. Thus, the order of attack and weapon change commands can be switched by the buffer. Fixing this is not trivial, I'll have to look into this when I have some more time.

(0012937) unknownna (updater) 2015-07-14 19:10 edited on: 2015-07-14 19:13	Regarding Zalewa's issue: If the observing client is unstable (ping spikes), other stable clients skip around as well.

(0012938) Zalewa (developer) 2015-07-14 19:30	@unknownna I assure you, there's no instability in any of my cases.

(0012946) Torr Samaho (administrator) 2015-07-15 19:43	I redesigned the tic buffer so that it now handles both movement and weapon select commands (implementation still very experimental and nearly untested). Please check if this works better now.

(0012948) Zalewa (developer) 2015-07-15 20:24	Movement jitter seems to be much less visible right now in my tests.

(0012950) unknownna (updater) 2015-07-16 01:27 edited on: 2015-07-16 03:44	Yeah, this is incredible! The weapon sync is now better than 2.1 even. And unstable clients seem to be just as smooth as 2.0, or perhaps even smoother at times. Great job! Weapons also don't desync with ping spikes anymore, which is awesome! This is like having the best of both worlds here. I didn't even manage to make it desync even once so far, with or without ping spikes. One thing I noticed is that clients will display the lag icon above their sprites if sv_useticbuffer is set to 0 and eventually time out from the server. Ok, it seems that the gameplay is actually worse now when the tic buffer is disabled. Weapons now fire twice as fast on the server-end. Quote from unknownna And unstable clients seem to be just as smooth as 2.0, or perhaps even smoother at times. In 2.0, stable clients (bots) seem to be moving a bit smoother on unstable clients. Actually, it's very hard to say for sure. Sometimes 2.0 seems worse. Quote from unknownna I didn't even manage to make it desync even once so far, with or without ping spikes. I managed to desync the BFG very reliably in BFGSHOOT from duel40a.pk3 when picking up ammo immediately after running out of ammo. Perhaps caused by this issue. I'll need to have the "sv_useticbuffer 0" issues fixed before I can pinpoint this any further. Ok, I managed to reproduce it in 1.2.2 (pre-tic buffer) so it's probably caused by this issue.

(0012953) Torr Samaho (administrator) 2015-07-16 06:11	Quote from unknownna Yeah, this is incredible! The weapon sync is now better than 2.1 even. And unstable clients seem to be just as smooth as 2.0, or perhaps even smoother at times. Great job! Weapons also don't desync with ping spikes anymore, which is awesome! This is like having the best of both worlds here. Wonderful! Seems like we are getting there :). Quote from unknownna One thing I noticed is that clients will display the lag icon above their sprites if sv_useticbuffer is set to 0 and eventually time out from the server. There was a bug in the "sv_useticbuffer 0" case, this should fix it. Possibly this also fixes the "twice as fast" problem.

(0012954) unknownna (updater) 2015-07-16 06:50 edited on: 2015-07-17 06:19	Quote from Torr Samaho There was a bug in the "sv_useticbuffer 0" case, this should fix it. Possibly this also fixes the "twice as fast" problem. That took care of both issues indeed. On a side note, I noticed that projectiles come out irregulary from the weapons if the ping is fluctuating on the client. This isn't caused by the tic buffer though since I was able to reproduce it in 1.2.2 as well. I'm not so sure whether it's a bug at all. Regarding Zalewa's issue: When playing against bots I find it easier to aim at them when sv_useticbuffer is set to 0, especially when cl_ticsperupdate is set to 2 or 3. They seem to move slightly smoother with the tic buffer off. Well, they're not smoother, but it feels a little less "jumpy" for some reason. And it seems that your ping might have an effect here. With an emulated ping of 251 they seem to be more "jumpy". I could be wrong though. I suppose it's more of an illusion than anything. Edit: It's mostly just with cl_ticsperupdate set to 3 that it feels a bit "jumpy" with the tic buffer on.

(0012957) Combinebobnt (reporter) 2015-07-16 20:07	Played two real 50 frag duels on judas and king1, neither player spotted rockets or bfgballs not coming out sometimes as per the zand 2.1 usual. (both players had sv_useticbuffer 1) Fix seems to be doing a good job

(0012965) cobalt (updater) 2015-07-18 09:43	Issue addressed by commit 7e094c43c1ad: Fixed the remaining jitter and weapon sync problems caused by the client movement buffer (fixes 2330). Committed by Benjamin Berkels [Torr Samaho] on Saturday 18 July 2015 11:29:00 Changes in files: docs/zandronum-history.txt \| 6 + src/sv_main.cpp \| 171 +++++++++++++++++++++++++++---------------- src/sv_main.h \| 16 +++- 3 files changed, 129 insertions(+), 64 deletions(-)

This issue is already marked as resolved. If you feel that is not the case, please reopen it and explain why.
Supporters:	Combinebobnt Ivan Watermelon Mobius unknownna
Opponents:	No one explicitly opposes this issue yet.

Issue History
Date Modified	Username	Field	Change
2015-06-25 04:34	AlexMax	New Issue
2015-06-25 05:42	Torr Samaho	Note Added: 0012786
2015-06-25 13:02	Untitled	Note Added: 0012790
2015-06-25 16:19	Jenova	Note Added: 0012791
2015-06-25 17:09	Razgriz	Note Added: 0012792
2015-06-25 18:16	Konar6	Note Added: 0012794
2015-06-25 18:46	Untitled	Note Added: 0012798
2015-06-25 23:06	AlexMax	Note Added: 0012804
2015-06-25 23:32	Ivan	Note Added: 0012805
2015-06-26 14:03	Watermelon	Note Added: 0012806
2015-06-26 14:04	Watermelon	Note Edited: 0012806	View Revisions
2015-06-26 14:04	Watermelon	Note Edited: 0012806	View Revisions
2015-06-26 14:13	Watermelon	Note Edited: 0012806	View Revisions
2015-06-27 12:20	Torr Samaho	Note Added: 0012810
2015-06-27 20:08	Frits	Note Added: 0012814
2015-06-27 21:05	Edward-san	Note Added: 0012815
2015-06-27 21:17	Frits	Note Edited: 0012814	View Revisions
2015-06-28 02:55	Watermelon	Note Added: 0012817
2015-06-28 06:19	Torr Samaho	Note Added: 0012818
2015-06-28 16:54	Watermelon	Note Added: 0012819
2015-06-28 17:05	Torr Samaho	Note Added: 0012820
2015-06-28 18:39	Watermelon	Note Added: 0012821
2015-06-28 18:39	Watermelon	Note Edited: 0012821	View Revisions
2015-06-28 18:39	Watermelon	Note Edited: 0012821	View Revisions
2015-06-29 00:11	Watermelon	Note Added: 0012823
2015-06-29 06:14	Torr Samaho	Note Added: 0012824
2015-06-29 06:14	Torr Samaho	Assigned To	=> Torr Samaho
2015-06-29 06:14	Torr Samaho	Status	new => feedback
2015-06-29 13:44	Watermelon	Note Added: 0012825
2015-06-29 13:44	Watermelon	Note Edited: 0012825	View Revisions
2015-06-29 13:45	Watermelon	Note Edited: 0012825	View Revisions
2015-07-06 20:42	unknownna	Note Added: 0012830
2015-07-06 21:04	unknownna	Note Edited: 0012830	View Revisions
2015-07-07 06:04	Torr Samaho	Note Added: 0012832
2015-07-07 17:34	unknownna	Note Added: 0012837
2015-07-07 18:03	Torr Samaho	Note Added: 0012839
2015-07-07 19:30	unknownna	Note Added: 0012841
2015-07-07 20:14	unknownna	Note Edited: 0012841	View Revisions
2015-07-07 21:52	AlexMax	Note Added: 0012843
2015-07-07 21:52	AlexMax	Status	feedback => assigned
2015-07-08 00:02	unknownna	Note Added: 0012844
2015-07-08 00:05	unknownna	Note Edited: 0012844	View Revisions
2015-07-08 21:41	Zalewa	Note Added: 0012845
2015-07-08 21:41	Zalewa	Note Edited: 0012845	View Revisions
2015-07-08 22:41	unknownna	Note Added: 0012846
2015-07-08 22:45	unknownna	Reproducibility	sometimes => always
2015-07-08 22:45	unknownna	Status	assigned => confirmed
2015-07-08 22:52	unknownna	Note Edited: 0012846	View Revisions
2015-07-08 23:12	unknownna	Relationship added	related to 0002255
2015-07-08 23:12	unknownna	Note Edited: 0012830	View Revisions
2015-07-12 07:43	Torr Samaho	Note Added: 0012862
2015-07-12 07:57	unknownna	Note Added: 0012863
2015-07-12 07:59	Torr Samaho	Note Added: 0012864
2015-07-12 08:08	unknownna	Note Added: 0012865
2015-07-12 08:08	unknownna	File Added: ticbuffer_jitter.cld
2015-07-12 08:14	Torr Samaho	Note Added: 0012867
2015-07-12 08:14	unknownna	Note Edited: 0012865	View Revisions
2015-07-12 08:30	Torr Samaho	Note Added: 0012868
2015-07-12 08:30	Torr Samaho	Note Edited: 0012868	View Revisions
2015-07-12 08:50	unknownna	Note Added: 0012869
2015-07-12 08:59	unknownna	Note Edited: 0012869	View Revisions
2015-07-12 09:16	Torr Samaho	Note Added: 0012870
2015-07-12 09:34	unknownna	Note Added: 0012871
2015-07-12 10:02	Torr Samaho	Note Added: 0012872
2015-07-12 10:22	Zalewa	Note Added: 0012873
2015-07-12 10:23	Zalewa	Note Edited: 0012873	View Revisions
2015-07-12 10:23	unknownna	Note Added: 0012874
2015-07-12 10:27	unknownna	Note Edited: 0012874	View Revisions
2015-07-12 11:00	Torr Samaho	Note Added: 0012876
2015-07-12 11:41	unknownna	Note Added: 0012880
2015-07-12 11:50	unknownna	Note Edited: 0012880	View Revisions
2015-07-12 12:12	Torr Samaho	Note Added: 0012881
2015-07-12 12:22	unknownna	Note Added: 0012882
2015-07-12 13:02	Torr Samaho	Note Added: 0012883
2015-07-12 13:14	unknownna	File Added: ticbuffer_jitter_2.0.cld
2015-07-12 13:15	unknownna	File Added: ticbuffer_jitter_2.1.1.cld
2015-07-12 13:18	unknownna	Note Added: 0012884
2015-07-12 13:19	unknownna	Note Edited: 0012884	View Revisions
2015-07-12 13:28	Torr Samaho	Note Added: 0012885
2015-07-12 13:30	unknownna	Note Added: 0012886
2015-07-12 13:31	unknownna	Note Edited: 0012886	View Revisions
2015-07-12 14:03	Torr Samaho	Note Added: 0012887
2015-07-12 14:20	unknownna	Note Added: 0012888
2015-07-12 14:39	Torr Samaho	Note Added: 0012889
2015-07-12 15:10	Torr Samaho	Note Added: 0012890
2015-07-12 15:20	unknownna	Note Added: 0012891
2015-07-12 15:25	unknownna	Note Edited: 0012891	View Revisions
2015-07-12 15:27	Torr Samaho	Note Added: 0012892
2015-07-12 15:30	unknownna	Note Added: 0012893
2015-07-12 15:48	Torr Samaho	Note Added: 0012894
2015-07-12 16:22	unknownna	Note Added: 0012896
2015-07-12 16:26	Torr Samaho	Note Added: 0012897
2015-07-12 16:32	unknownna	Note Added: 0012898
2015-07-12 16:35	unknownna	Note Edited: 0012898	View Revisions
2015-07-12 16:38	unknownna	Note Edited: 0012898	View Revisions
2015-07-12 16:45	Torr Samaho	Note Added: 0012899
2015-07-12 16:50	unknownna	Note Added: 0012900
2015-07-12 17:02	Torr Samaho	Note Added: 0012901
2015-07-12 17:09	unknownna	Note Added: 0012902
2015-07-12 17:16	AlexMax	Note Added: 0012904
2015-07-12 17:25	AlexMax	Note Edited: 0012904	View Revisions
2015-07-12 17:30	AlexMax	Note Edited: 0012904	View Revisions
2015-07-12 17:33	Torr Samaho	Note Added: 0012905
2015-07-12 17:38	unknownna	Note Added: 0012906
2015-07-12 17:59	Torr Samaho	Note Added: 0012907
2015-07-13 04:16	unknownna	Note Added: 0012910
2015-07-13 09:13	Frits	Note Added: 0012911
2015-07-13 12:06	unknownna	Note Added: 0012913
2015-07-13 13:31	Ivan	Note Added: 0012915
2015-07-13 13:38	Frits	Note Edited: 0012911	View Revisions
2015-07-13 17:11	Torr Samaho	Note Added: 0012917
2015-07-13 17:22	Torr Samaho	Note Added: 0012918
2015-07-14 02:25	unknownna	Note Added: 0012921
2015-07-14 02:45	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 02:46	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 03:07	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 03:07	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 03:15	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 03:54	unknownna	Note Edited: 0012921	View Revisions
2015-07-14 06:02	Zalewa	Note Added: 0012923
2015-07-14 06:12	Torr Samaho	Note Added: 0012924
2015-07-14 06:35	Torr Samaho	Note Added: 0012925
2015-07-14 07:06	unknownna	Note Added: 0012926
2015-07-14 07:07	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 07:07	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 07:26	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 07:44	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 07:46	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 07:53	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 08:20	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 08:28	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 08:58	unknownna	Note Edited: 0012926	View Revisions
2015-07-14 17:24	Zalewa	File Added: zalewa_warpdemo_zandronum.zip
2015-07-14 17:27	Zalewa	Note Added: 0012932
2015-07-14 17:33	Torr Samaho	Note Added: 0012933
2015-07-14 17:48	Hypnotoad	Note Added: 0012934
2015-07-14 18:20	unknownna	Note Added: 0012935
2015-07-14 18:55	Torr Samaho	Note Added: 0012936
2015-07-14 19:10	unknownna	Note Added: 0012937
2015-07-14 19:13	unknownna	Note Edited: 0012937	View Revisions
2015-07-14 19:30	Zalewa	Note Added: 0012938
2015-07-15 19:43	Torr Samaho	Note Added: 0012946
2015-07-15 20:24	Zalewa	Note Added: 0012948
2015-07-16 01:27	unknownna	Note Added: 0012950
2015-07-16 01:28	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 02:02	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 02:03	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 02:04	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 02:08	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:08	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:16	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:37	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:38	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:43	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 03:44	unknownna	Note Edited: 0012950	View Revisions
2015-07-16 06:11	Torr Samaho	Note Added: 0012953
2015-07-16 06:50	unknownna	Note Added: 0012954
2015-07-16 06:53	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 12:28	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 12:29	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 13:02	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 13:03	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 13:10	unknownna	Note Edited: 0012954	View Revisions
2015-07-16 20:07	Combinebobnt	Note Added: 0012957
2015-07-17 06:19	unknownna	Note Edited: 0012954	View Revisions
2015-07-18 06:04	Zalewa	Note Added: 0012964
2015-07-18 08:26	Zalewa	Note Deleted: 0012964
2015-07-18 09:43	cobalt	Status	confirmed => needs testing
2015-07-18 09:43	cobalt	Target Version	=> 2.1
2015-07-18 09:43	cobalt	Note Added: 0012965
2015-09-13 12:41	Dusk	Status	needs testing => resolved
2015-09-13 12:41	Dusk	Fixed in Version	=> 2.1
2015-09-13 12:41	Dusk	Resolution	open => fixed
2018-09-30 21:37	Blzut3	Status	resolved => closed