0001705: Gametic-based unlagged

Notes
(0008191) AlexMax (developer) 2014-02-13 02:36 edited on: 2014-02-13 02:43	The pull request is located here: 'https://bitbucket.org/Torr_Samaho/zandronum/pull-request/32/gametic-based-unlagged/diff [^]' Also note that my pull request has a serverside debug flag that toggles between the old ping-based unlagged and new gametic-based unlagged. Before 2.0 is released, we should remove that flag, so server owners don't have yet another poorly-understood toggle to deal with. The patch _has_ been tested with players, but not extensively so.

(0008192) AlexMax (developer) 2014-02-13 04:53 edited on: 2014-02-13 04:56	Some observations from more testing: - I did a few blind tests, and in blind tests players could either not tell the difference between ping-based and gametic-based unlagged or they preferred the gametic-based unlagged. - Players observed that other players with high ping or unstable connections were MUCH easier to hit with gametic-based unlagged compared to ping-based unlagged. - With gametic-based unlagged, players sometimes observed that they could kill people 'around corners' in some situations (in that their perception was that they JUST missed someone going around a corner, but the server registered a hit anyway). Perhaps it might be a good idea to experiment with blanketly offsetting unlagged tics by some amount to see if that makes things better, but that also runs the risk of making accurate shots miss more often. The complaints weren't bad in any case. - Also, there were slightly more occurrences of players saying that their point-blank shots were missing with gametic-based unlagged, but only slightly, and only by one or two players.

(0008195) ZzZombo (reporter) 2014-02-13 12:49 edited on: 2014-02-13 12:51	Why not the other way around? The client sends its own gametic, that on arrive will be used by server to calculate the difference with currently simulating gametic while processing the command thus knowing how far move players back. Look at'https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking#Lag_compensation. [^]'

(0008196) ibm5155 (reporter) 2014-02-13 16:36	hmm so with that a mod like "flappy lost soul" where players control a projectile with their movements would get a smooth projectile on the right position?

(0008200) AlexMax (developer) 2014-02-13 22:45	Zombo: To my knowledge, the client's gametic and servers gametic are completely disjointed. However the client DOES use its own gametic to predict the server's gametic when sending movement commands. The formula looks something like Predicted Server Gametic (sent to the server by the client) = (Current local gametic - Last local gametic that we got a server gametic) + Last server gametic we got. Pretty simple.

(0008201) ZzZombo (reporter) 2014-02-13 22:54	It can be solved by synchronizing them upon client connection can't it?

(0008202) AlexMax (developer) 2014-02-13 23:29	I already tell the client what the server's gametic is on full update (which you get on connect), and a truncated single-byte representation is sent to the client with every opponent movement command to ensure the client always has the latest server gametic - this would be necessary in any case to ensure the client and server do not drift out of sync.

(0008267) Torr Samaho (administrator) 2014-02-22 16:42	I finally had a chance to look at the patch in more detail. I think it could be better to do what SERVERCOMMANDS_ServerGametic does in the existing SERVERCOMMANDS_MoveLocalPlayer function instead. I'm not even sure if it's necessary to send the server tic at all. Could the number of tics the client has to predict be exactly the delay you are looking for (see ulPredictionTicks in CLIENT_PREDICT_PlayerPredict)? If that's the case, you'd just need to send this number instead of the predicted server tic in CLIENTCOMMANDS_ClientMove and get rid of all code that submits the server tic to the client.

(0008270) ZzZombo (reporter) 2014-02-23 05:49	Also about that shoot-behind-corner-and-hit: isn't it normal in the sense it represents what attacker saw when he fired the weapon?

(0008271) AlexMax (developer) 2014-02-23 06:20	Thanks for giving me the pointer. I took a gander at it, and my conclusions are as follows: In the prediction amount calculation, the client compares the current client gametic with the last client gametic it got from the server. Importantly, this offset is calculated and used almost immediately thereafter in order to predict. Likewise, with my unlagged implementation, the server compares the current server gametic with the last server gametic it got from the given client, and this offset is also calculated and used almost immediately thereafter. Sending the clientside offset to the server is not necessarily wrong, but by the time the offset gets to the server network conditions might have changed and the offset might not be correct, and the backwards reconciliation might mispredict. Likewise, if we ripped out the clientside tic prediction out entirely and sent the serverside offset, clientside movement predictions would mispredict more often because the offset would again be stale. By limiting ourselves to only one 'offset' prediction on one side of the connection, this means that any prediction or backwards reconciliation on the other side of the connection is not using the absolute most up-to-date latency data possible. Therefore, I must conclude that offset calculations done independently on either side of the connection is the better option. Precisely, it is better by the length of time an offset calculation would take to go from one side to the other. However, I only conclude that it would be best if latency offsets were calculated on either side on the connection using some form of time taken on the other side. I should emphasize that I do not claim that my proposed implementation is the best possible, or even particularly good. :)

(0008276) AlexMax (developer) 2014-02-23 20:46	After our conversation in IRC, I will move serverside gametic updates to SERVERCOMMANDS_MoveLocalPlayer. I also believe that I need to ensure that the 'partial gametic' is not out of date if the player waits more than 255 tics to join the server.

(0008277) AlexMax (developer) 2014-02-24 04:38 edited on: 2014-02-24 05:30	I put all of my changes into one commit. 'https://bitbucket.org/alexmax2742/zandronum/commits/6ed4ec0b085007eb09cff8a5a9ba5b7d78d2a85b [^]' However, tonight I've been testing my implementation against the ping-based implementation on a stable connection with latency using a lag proxy. With low pings, there is a 1 tic discrepancy between the ping measurement and the gametic measurement, and with high pings there can be 2 or sometimes even 3 tics discrepancy. The gametic-based measurement is always the 'lesser' of the two, which means that the gametic-based measurement grabs 1-2 more tics of backwards reconciliation than the ping-based measurement. It doesn't sound like a lot, but 2 tics is 57ms difference. The problem is that I'm not entirely sure which one is 'correct'. Using the gamer proxy, the "ping" that Zandronum seems to generate looks correct, so at least on paper it seems like I'm going back in time "too much" with gametic-based unlagged. With the ping-based unlagged, a loopback server looks like it uses the current gametic, while there's always at least one tic of unlagged used with gametic-based unlagged. But let's think about this. If the client is getting updates from the server about the servers gametic and the positions of players, and all the client is doing is sending that, plus accounting for any additional time that elapsed on the client since it last got a server update...wouldn't that make the gametic-generated unlagged tic more accurate to what the player was actually seeing at the time, regardless of if it matches up with the ping or not? More thoughts. In the loopback situation above, the server first reads tics off the wire with everybody's packets, and then does gameloop logic and sends out messages. If a client has tics that apply to that gamestate, by definition it can't arrive on that tic, it has to arrive on the next tic. Which means perhaps that the original ping-based logic: unlaggedGametic = ( gametic - ( player->ulPing * TICRATE / 1000 ) ) was _always_ at least one tic off because it would always round down, instead of always rounding up. I don't know..that only accounts for one of the tics, it still doesn't explain why there's sometimes a 2-3 tic difference with higher pings. Maybe my way is more accurate to what the player is actually seeing, or maybe we're losing a tic somewhere. Any ideas on how to verify that my measurement is accurate? Am I totally off base somewhere?

(0008281) Xsnake (reporter) 2014-02-25 13:11	I've played ZDaemon for many years and I could never get used to Zandronum's unlagged, having the feeling that, even though it was on, I had to slightly aim at where the player was going to be rather than where they actually were in order to perform a successful hit. Correct me if I'm wrong, but ZDaemon has a tick-based unlagged implementation (I'm not sure where I heard that, though). That one tick difference you're referring to could well explain the difference I've always felt between these two ports. So, I'm tempted to say your implementation is more accurate than the ping-based one (regardless of that 2-3 tick difference with high pings). But once again, this is just me speaking with my own experience of the game. Those who do not come from ZDaemon probably think the current ping-based unlagged is just fine.

(0008286) AlexMax (developer) 2014-02-26 23:06	Actually, I have changed my mind on this. Here's why: Normal Doom game loop goes like this: - Handle Events (Keyboard, Mouse, etc.) - Do game logic - Render frame - Goto first step Let's say rendering the frame shows an opponent who is ready to kill you. You press +attack, but because event handling happens at the beginning of the gametic, any reaction you have to the opponent must happen on the NEXT gametic. Now, with the way I have things set up, the client movement command sends the server gametic that it thinks it is SEEING. However, the server then takes this gametic and uses that as the server gametic to backwards reconcile to. This violates the laws of causality, because if you see something on a gametic, your reaction to it cannot be taken into account until gametic + 1. Thus, on my local repository, I add one to the predicted serverside gametic when reconciling. int unlaggedGametic = player->lPredictedServerTick + 1; This accounts for one of the tics of difference. However, it does not account for the others. I will let you know if I have anything concrete to add to that issue.

(0008564) Torr Samaho (administrator) 2014-04-13 15:46	I thought about your implementation a bit more and think I found one inaccuracy: It doesn't handle the effect of cl_ticsperupdate: The client does not predict the movement of the other players, so on a client the other players are always at the last position the server told. A client does predict its own position, so the following part of your patch // [AM] Also send the predicted server gametic we think we're on. const LONG lPredictedServerGametic = g_lLastServerGametic + ( gametic - g_lTimeOfLastServerGametic ); Printf("CLIENT_SendCmd: I predict we're on gametic %d\n", lPredictedServerGametic); CLIENTCOMMANDS_ClientMove( lPredictedServerGametic ); will properly handle the position of the moving client, put will look for the position of the other players at a wrong tic.

(0010276) StrikerMan780 (reporter) 2014-09-19 14:47 edited on: 2014-09-19 16:27	Any headway with this? This has to be one of the things I'm looking most forward to in Zan 2.0.

(0010410) Watermelon (developer) 2014-10-09 03:35	StrikerMan I'm going to be working on this and pushing it through to Zan 2.0

(0010440) Watermelon (developer) 2014-10-09 22:06	I implemented the gametic module the way Alex did it with integration into USERINFO so players can select what kind of unlagged they want (Ping or Gametic, both are functional and I don't intend on removing one that has stood the test of time). The gametic unlagged is always 1, sometimes 2 or even 3 tics ahead. I'll try to test this again with people I know.

(0010450) Watermelon (developer) 2014-10-10 14:07 edited on: 2014-10-10 14:21	This should be operational: 'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/b085844c4e773364710939920064e14654c491dd [^]' 'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/46a2efa181781947b6b730e09ec7fa63400d3304 [^]' 'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/2fdbc74c84bbb8b3f3f38860d99d6d618ced908b [^]' 'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/2ab22941c2b533147a56629c725d33631650afe9 [^]' 'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/aa0fed27f18a104a44f59bf6c0a21585c014ab08 [^]' NOTES: - Takes into account cl_ticsperupdate The user can change their ticsperupdate at any time and it will work just as it should. This is due to it being bundled with the movelocalplayer command, the client gets the server gametic as needed. - Gametic bundled with SVC_MOVELOCALPLAYER This packet is sent to the client always as the most up-to-date thing. If this is lost, the client will be in trouble regardless, so putting it here saves on another header, and allows it to work with cl_ticsperupdate flawlessly. - Sanity checks prevent exploitation The full gametic is sent, so the server checks to see if it's either > than it's gametic (indicating a corrupt client), or if its less than the last received update. If it's less, we're getting stale data which we don't care about and will ignore. This keeps the client in an honest range, if they decide to send an old gametic to attempt to exploit this, they will struggle because of the constraints put on it. It'd be pretty obvious too since the only way such an exploit could be successful is if they rigged the client to not send any updates and freeze everything in place. By the time they'd line up the shot, the gametic it sends might already be stale due to clamping done in unlagged.cpp. Plus they'd lag out every time they tried to shoot, so it'd be pretty obvious (easy ban). - Gametic base sent in authentication This way, the clients will never have an erroneous value. - Clients accidentally sending the same gametic won't change anything The worst that will happen is they will be reconciled to that gametic. Therefore an honest client won't get screwed over in any shape or form if they happen to lag or send the same gametic for any reason. I have tested all the above as well with high ping, different cl_ticsperupdate values...etc, and it all was working good.

(0010464) StrikerMan780 (reporter) 2014-10-11 06:00 edited on: 2014-10-11 06:01	Is the new unlagged the default setting? If so, awesome. Epic either way though... got tired of how unreliable the ping-based unlagged can be. (Shooting ethereal pellets that do nothing, woo!)

(0010477) Torr Samaho (administrator) 2014-10-11 11:48	Can you collect all necessary changes in a single changeset? Quote from Watermelon The gametic unlagged is always 1, sometimes 2 or even 3 tics ahead. You mean compared to the ping based unlagged? Did you check that it's not overcompensating? For clients with a LAN connection, the unlagged shouldn't do anything. In particular, it's important to take into account 0001705:0008286. Quote from Watermelon The full gametic is sent, so the server checks to see if it's either > than it's gametic (indicating a corrupt client), or if its less than the last received update. If it's less, we're getting stale data which we don't care about and will ignore. It's very likely that client movement commands sometimes will arrive in the wrong order. I don't think that dropping those is a good idea. If you are walking in a straight line for instance, the old handling will work perfectly, but if you drop some of the packets the movement will get choppy.

(0010483) Watermelon (developer) 2014-10-11 13:30 edited on: 2014-10-11 13:32	Quote You mean compared to the ping based unlagged? Did you check that it's not overcompensating? For clients with a LAN connection, the unlagged shouldn't do anything. In particular, it's important to take into account 0001705:0008286. Thinking on this further, it would make sense to do Alex's suggestion since I believe the server does (with my own reasoning): - SERVER_GetPackets() (at least once) then - G_Ticker() // Send out gametic X then - gametic++ // Now X + 1 then start again - SERVER_GetPackets() (at least once) // Getting X when it's X + 1 Therefore any packets received have to be at least 1 gametic old. Does this make sense? Quote It's very likely that client movement commands sometimes will arrive in the wrong order. I don't think that dropping those is a good idea. If you are walking in a straight line for instance, the old handling will work perfectly, but if you drop some of the packets the movement will get choppy. My wording was bad: Packets aren't actually dropped, it just only uses the latest server-gametic because it can't guarantee the previous ones have been spoofed or not. The downside is out of order packets might use a later gametic if it arrives after a later one. Is this okay? I can make it so it blindly trusts the client on this, but that's when we get into the possibility of some nasty hacks. If all is good after the comment you give back, I'll compress the changeset.

(0010490) Torr Samaho (administrator) 2014-10-11 15:42 edited on: 2014-10-11 15:42	Quote from Watermelon Therefore any packets received have to be at least 1 gametic old. Does this make sense? I'm not sure what this has to do with 0001705:0008286. Also keep in mind that SERVER_GetPackets() is called more than once per tic to allow for ping measurements that are not a multiple of the ticrate. The essence of 0001705:0008286 is: Movement commands generated at tic X are a reaction to the gamestate at tic X and need to be processed at tic X + 1, not any earlier. Quote from Watermelon The downside is out of order packets might use a later gametic if it arrives after a later one. Is this okay? That should be fine.

(0010493) Watermelon (developer) 2014-10-11 15:56	Quote I'm not sure what this has to do with 0001705:0008286. Also keep in mind that SERVER_GetPackets() is called more than once per tic to allow for ping measurements that are not a multiple of the ticrate. The essence of 0001705:0008286 is: Movement commands generated at tic X are a reaction to the gamestate at tic X and need to be processed at tic X + 1, not any earlier. Misunderstanding on my part (for some reason I thought it'd cache the packets and process them all at once in the main loop, which is wrong). Therefore, if changing the code to this (with the +1): Quote unlaggedGametic = pClient->lLastServerGametic + 1; Is fine with you, I will compile the changeset.

(0010498) Torr Samaho (administrator) 2014-10-11 16:35	Sounds good to get started! Without seeing the full patch, I can't say this for sure though.

(0010585) Watermelon (developer) 2014-10-14 18:48	Pull request sent. I expect there to be changes you may request, but I sent a pull request anyways so you can code review/comment there. I tested this online and under ridiculous situations (300 ping w/ insane jitter) it performed slightly nicer for me than the ping based unlagged did. So far I am happy with it.

(0010795) Watermelon (developer) 2014-11-02 18:47	'https://bitbucket.org/Torr_Samaho/zandronum-stable/pull-request/95/added-in-new-gametic-unlagged-to-replace/diff [^]' I need discussion at the dev meeting about this.

(0010852) Watermelon (developer) 2014-11-10 16:06	Tested on LAN, worked just fine as expected (no gametic ahead of the server). Pull req updated.

(0010937) Torr Samaho (administrator) 2014-11-23 11:10	You may have overlooked my question in the pull request: Quote from Torr Samaho Hmm, you are already so confident in this implementation that everybody will prefer it in its current form over the old ping based version?

(0010948) Watermelon (developer) 2014-11-24 12:30	We discussed on IRC about having a flag to enable/disable it. Did you want it to be a serverside flag? Or a clientside one? I was thinking of adding in a serverside boolean that could allow them to switch between ping/gametic when releasing, so we can get testing done. Thoughts?

(0010953) StrikerMan780 (reporter) 2014-11-24 15:30 edited on: 2014-11-24 15:31	A serverside bool would be good. Last thing we need is to have a testing session, and some people going "This sucks!" or "What's the Difference?" only to realize after the session they had Ping-based unlagged enabled, and not Gametic-based.

(0010960) Torr Samaho (administrator) 2014-11-24 19:47	I would strongly prefer a client side flag. Your original patches had this and that's one of the reasons why we introduced cl_clientflags :P. This way everybody can compare the difference between the two methods. Quote from StrikerMan780 Last thing we need is to have a testing session, and some people going "This sucks!" or "What's the Difference?" only to realize after the session they had Ping-based unlagged enabled, and not Gametic-based. People just need to be informed what exactly is supposed to be tested and how to do so. Those who are actually interested in this kind of difference should have no problem toggling this on their own.

(0010962) Watermelon (developer) 2014-11-24 21:33	Ok I'll update to clientflags.

(0010966) Watermelon (developer) 2014-11-24 23:14	Update complete

(0011047) Watermelon (developer) 2014-12-08 15:23	Updates ready.

(0011365) Torr Samaho (administrator) 2015-01-11 21:32	I changed how the sorting of the movement buffer is done and merged everything with the stable head. Currently, a testing session is run to get an idea how well the latest implementation works.

(0011406) cobalt (updater) 2015-01-17 13:14	Issue addressed by commit 8e43651ce72a: [Water, AlexMax, Torr Samaho] Added gametic unlagged which should perform better under varying net conditions compared to ping-based unlagged (addresses 1705). Committed by Benjamin Berkels [Torr Samaho] on Saturday 17 January 2015 12:12:48 Changes in files: docs/zandronum-history.txt \| 1 + src/cl_commands.cpp \| 2 ++ src/cl_main.cpp \| 34 ++++++++++++++++++++++++++++++++++ src/cl_main.h \| 2 ++ src/d_netinf.h \| 1 + src/d_netinfo.cpp \| 2 ++ src/m_options.cpp \| 8 +++++++- src/sv_commands.cpp \| 3 +++ src/sv_main.cpp \| 22 ++++++++++++++++++++++ src/sv_main.h \| 4 ++++ src/unlagged.cpp \| 24 +++++++++++++++++++++++- 11 files changed, 101 insertions(+), 2 deletions(-)

(0011409) Dusk (developer) 2015-01-17 16:45	This was tested rather extensively so I think we can close this now.

This issue is already marked as resolved. If you feel that is not the case, please reopen it and explain why.
Supporters:	ZzZombo Combinebobnt Xsnake WaTaKiD Soul rosking swordgrunt StrikerMan780 Leonard Karakurt
Opponents:	No one explicitly opposes this issue yet.

Issue History
Date Modified	Username	Field	Change
2014-02-13 02:34	AlexMax	New Issue
2014-02-13 02:36	AlexMax	Note Added: 0008191
2014-02-13 02:39	AlexMax	Note Edited: 0008191	View Revisions
2014-02-13 02:43	AlexMax	Note Edited: 0008191	View Revisions
2014-02-13 03:30	Dusk	Status	new => needs review
2014-02-13 04:53	AlexMax	Note Added: 0008192
2014-02-13 04:56	AlexMax	Note Edited: 0008192	View Revisions
2014-02-13 12:49	ZzZombo	Note Added: 0008195
2014-02-13 12:51	ZzZombo	Note Edited: 0008195	View Revisions
2014-02-13 16:36	ibm5155	Note Added: 0008196
2014-02-13 22:45	AlexMax	Note Added: 0008200
2014-02-13 22:54	ZzZombo	Note Added: 0008201
2014-02-13 23:29	AlexMax	Note Added: 0008202
2014-02-18 15:03	Dusk	Relationship added	related to 0001638
2014-02-22 16:42	Torr Samaho	Note Added: 0008267
2014-02-23 05:49	ZzZombo	Note Added: 0008270
2014-02-23 06:20	AlexMax	Note Added: 0008271
2014-02-23 20:46	AlexMax	Note Added: 0008276
2014-02-24 04:38	AlexMax	Note Added: 0008277
2014-02-24 05:30	AlexMax	Note Edited: 0008277	View Revisions
2014-02-25 13:11	Xsnake	Note Added: 0008281
2014-02-26 23:06	AlexMax	Note Added: 0008286
2014-04-13 15:46	Torr Samaho	Note Added: 0008564
2014-04-13 15:47	Torr Samaho	Assigned To	=> Torr Samaho
2014-04-13 15:47	Torr Samaho	Status	needs review => feedback
2014-05-05 22:23	Watermelon	Relationship added	related to 0001442
2014-09-19 14:47	StrikerMan780	Note Added: 0010276
2014-09-19 16:27	StrikerMan780	Note Edited: 0010276	View Revisions
2014-10-09 03:34	Watermelon	Assigned To	Torr Samaho => Watermelon
2014-10-09 03:34	Watermelon	Status	feedback => assigned
2014-10-09 03:35	Watermelon	Note Added: 0010410
2014-10-09 22:06	Watermelon	Note Added: 0010440
2014-10-10 14:07	Watermelon	Note Added: 0010450
2014-10-10 14:07	Watermelon	Assigned To	Watermelon =>
2014-10-10 14:07	Watermelon	Status	assigned => needs review
2014-10-10 14:08	Watermelon	Note Edited: 0010450	View Revisions
2014-10-10 14:09	Watermelon	Note Edited: 0010450	View Revisions
2014-10-10 14:10	Watermelon	Assigned To	=> Watermelon
2014-10-10 14:10	Watermelon	Status	needs review => assigned
2014-10-10 14:10	Watermelon	Status	assigned => needs review
2014-10-10 14:21	Watermelon	Note Edited: 0010450	View Revisions
2014-10-11 06:00	StrikerMan780	Note Added: 0010464
2014-10-11 06:01	StrikerMan780	Note Edited: 0010464	View Revisions
2014-10-11 11:48	Torr Samaho	Note Added: 0010477
2014-10-11 11:50	Torr Samaho	Status	needs review => feedback
2014-10-11 13:30	Watermelon	Note Added: 0010483
2014-10-11 13:30	Watermelon	Status	feedback => needs review
2014-10-11 13:30	Watermelon	Note Edited: 0010483	View Revisions
2014-10-11 13:31	Watermelon	Note Edited: 0010483	View Revisions
2014-10-11 13:32	Watermelon	Note Edited: 0010483	View Revisions
2014-10-11 15:42	Torr Samaho	Note Added: 0010490
2014-10-11 15:42	Torr Samaho	Note Edited: 0010490	View Revisions
2014-10-11 15:43	Torr Samaho	Status	needs review => feedback
2014-10-11 15:56	Watermelon	Note Added: 0010493
2014-10-11 15:56	Watermelon	Status	feedback => needs review
2014-10-11 16:35	Torr Samaho	Note Added: 0010498
2014-10-11 16:35	Torr Samaho	Status	needs review => feedback
2014-10-12 00:57	Watermelon	Status	feedback => assigned
2014-10-14 18:48	Watermelon	Note Added: 0010585
2014-10-14 18:49	Watermelon	Status	assigned => needs review
2014-10-25 03:00	Watermelon	Status	needs review => assigned
2014-11-02 18:47	Watermelon	Note Added: 0010795
2014-11-02 18:47	Watermelon	Status	assigned => needs review
2014-11-09 20:18	Watermelon	Status	needs review => assigned
2014-11-10 16:06	Watermelon	Note Added: 0010852
2014-11-10 16:06	Watermelon	Status	assigned => needs review
2014-11-23 11:10	Torr Samaho	Note Added: 0010937
2014-11-23 11:10	Torr Samaho	Status	needs review => feedback
2014-11-24 12:30	Watermelon	Note Added: 0010948
2014-11-24 12:30	Watermelon	Status	feedback => needs review
2014-11-24 15:30	StrikerMan780	Note Added: 0010953
2014-11-24 15:31	StrikerMan780	Note Edited: 0010953	View Revisions
2014-11-24 19:47	Torr Samaho	Note Added: 0010960
2014-11-24 19:47	Torr Samaho	Status	needs review => feedback
2014-11-24 21:33	Watermelon	Note Added: 0010962
2014-11-24 21:33	Watermelon	Status	feedback => assigned
2014-11-24 23:14	Watermelon	Note Added: 0010966
2014-12-01 13:26	Watermelon	Product Version	2.0-beta => 1.4
2014-12-01 13:26	Watermelon	Target Version	=> 1.4
2014-12-08 15:23	Watermelon	Note Added: 0011047
2014-12-08 15:23	Watermelon	Status	assigned => needs review
2015-01-06 09:49	Dusk	Target Version	1.4 =>
2015-01-11 21:32	Torr Samaho	Note Added: 0011365
2015-01-11 21:33	Torr Samaho	Status	needs review => needs testing
2015-01-11 21:34	Torr Samaho	Product Version	1.4 => 1.3
2015-01-17 13:14	cobalt	Target Version	=> 1.4
2015-01-17 13:14	cobalt	Description Updated	View Revisions
2015-01-17 13:14	cobalt	Note Added: 0011406
2015-01-17 16:45	Dusk	Note Added: 0011409
2015-01-17 16:45	Dusk	Status	needs testing => resolved
2015-01-17 16:45	Dusk	Fixed in Version	=> 1.4
2015-01-17 16:45	Dusk	Resolution	open => fixed
2017-10-28 02:07	Leonard	Relationship added	parent of 0003317
2018-09-30 23:26	Blzut3	Status	resolved => closed