MantisBT - Zandronum
View Issue Details
0000757Zandronum[All Projects] Suggestionpublic2012-04-08 02:422016-04-10 19:27
TIHan 
 
normalfeatureN/A
newopen 
 
 
0000757: Monster Bandwidth Mitigation
I've been thinking about this for a while. Currently, it is absolutely impossible to play most of the sunder maps without a disconnect, even from the localhost. The cause is due to the constant movement updates from monsters.

We need a way to mitigate the situation. It's impossible to get it perfect; but we can remedy it in a way where we will not disconnect the clients. :)
No tags attached.
related to 0000803closed TIHan Reducing Monster Bandwidth Patch 
related to 0001268new  Huffman optimization 
related to 0001954new  Changing sound sending from String to logical index to save bandwidth 
related to 0001469needs testing Torr Samaho Problem connecting to slaughtermaps 
related to 0001030closed  Clients lag out more when connecting to a server with a high active monstercount 
Issue History
2012-04-08 02:42TIHanNew Issue
2012-04-08 06:19TIHanNote Added: 0003180
2012-04-08 06:20TIHanNote Edited: 0003180bug_revision_view_page.php?bugnote_id=3180#r1668
2012-04-11 04:36WatermelonNote Added: 0003203
2012-04-11 17:07bluewizardNote Added: 0003208
2012-04-11 18:24TIHanNote Added: 0003209
2012-04-11 18:25TIHanNote Edited: 0003209bug_revision_view_page.php?bugnote_id=3209#r1704
2012-04-11 23:16Torr SamahoNote Added: 0003214
2012-04-11 23:17Torr SamahoNote Edited: 0003214bug_revision_view_page.php?bugnote_id=3214#r1707
2012-04-11 23:17Torr SamahoNote Revision Dropped: 3214: 0001706
2012-04-11 23:23TIHanNote Added: 0003215
2012-04-11 23:27TIHanNote Edited: 0003215bug_revision_view_page.php?bugnote_id=3215#r1709
2012-04-29 02:33TIHanRelationship addedrelated to 0000803
2012-04-29 02:44Torr SamahoNote Added: 0003486
2012-04-29 02:48TIHanNote Added: 0003488
2013-01-12 06:56ZzZomboNote Added: 0005737
2013-01-13 10:01Torr SamahoNote Added: 0005750
2013-02-07 00:18WatermelonNote Added: 0005953
2013-06-08 05:35WatermelonNote Added: 0006381
2014-06-13 18:05WatermelonStatusnew => acknowledged
2014-06-13 18:05WatermelonRelationship addedrelated to 0001268
2014-10-09 00:07WatermelonAssigned To => Watermelon
2014-10-09 00:07WatermelonStatusacknowledged => assigned
2014-10-09 00:22WatermelonNote Added: 0010400
2014-10-09 00:24WatermelonNote Edited: 0010400bug_revision_view_page.php?bugnote_id=10400#r5622
2014-10-09 03:13ZzZomboNote Added: 0010403
2014-10-09 03:32WatermelonNote Added: 0010408
2014-10-09 11:53ZzZomboNote Added: 0010419
2014-10-10 16:00WatermelonRelationship addedrelated to 0001469
2014-10-10 16:00WatermelonRelationship addedrelated to 0001030
2014-10-10 16:01WatermelonRelationship addedrelated to 0001954
2014-11-11 03:05WatermelonNote Added: 0010856
2014-12-01 13:27WatermelonTarget Version => 1.4
2015-01-06 08:15DuskTarget Version1.4 =>
2015-04-01 02:54WatermelonNote Added: 0011959
2016-04-10 19:25DuskStatusassigned => new
2016-04-10 19:27DuskAssigned ToWatermelon =>

Notes
(0003180)
TIHan   
2012-04-08 06:19   
(edited on: 2012-04-08 06:20)
If anyone is interested, I have a binary that will try to tackle this problem.
'http://www.2shared.com/file/9bUWyZX1/skulltag_monster_mitigation_te.html [^]'

There are two cvars.
sv_monstermitigation - default value: true
sv_monstermitigation_amount - default value: 100

What this will do is actually throttle down the number of SERVERCOMMANDS_MoveThing/MoveThingExact that can be called in one tick by a non-dormant monster. In this case by default value, only 100 calls of updating monsters' position can be sent to the client in one tick. If there are any more, they will not be sent.

There are a few exceptions though:
1. A monster can be indefinitely updated if it took any damage.
2. A monster can be updated if the monster sees its target or the target sees the monster.

Testing this, players can play maps with enormous amounts of enemies 3000+ without disconnecting. The players will at times notice monsters glitching to their updated locations; but which is better? Disconnections or some glitching?

PS:
This is just a WIP and an idea.

(0003203)
Watermelon   
2012-04-11 04:36   
This would help ZDoomWars tremendously! The game tends to crash with 800+ monsters. That game always sends so much data... up to 1 meg/second.
(0003208)
bluewizard   
2012-04-11 17:07   
Recently I've been trying to play sunder and DVII with some friends, this would help a lot!
(0003209)
TIHan   
2012-04-11 18:24   
(edited on: 2012-04-11 18:25)
Could both of you test the binary to see how well it works for you?

(0003214)
Torr Samaho   
2012-04-11 23:16   
(edited on: 2012-04-11 23:17)
> Disconnections or some glitching?

IMHO that's a though question (certainly depends on which kind of glitching is possible). I think if something like this ever finds its way into Skulltag, the client should be able to decide this (like unlagged can be selected with cl_unlagged).

(0003215)
TIHan   
2012-04-11 23:23   
(edited on: 2012-04-11 23:27)
> I think if something like this ever finds its way into Skulltag, the client should be able to decide this (like unlagged can be selected with cl_unlagged).

So you are saying the client can choose if he wants to kill the bandwidth of the server?

Edit:
> (certainly depends on which kind of glitching is possible)
The glitching you will notice is that a lot of the monsters will not update their positions when they change directions. However, there are checks in place to update the position if it can see the player or it has taken damage.

(0003486)
Torr Samaho   
2012-04-29 02:44   
> So you are saying the client can choose if he wants to kill the bandwidth of the server?

I'm saying that the client has to be able to forbid the server to kill the client connection by flooding it with too much data. Apparently we are looking at this from two diametrically opposed perspectives.
(0003488)
TIHan   
2012-04-29 02:48   
We have discussed something very similar to this on another topic. I agree the client should be able to control how much data he wants to receive. I also agree that the server should have its own limit as well.
(0005737)
ZzZombo   
2013-01-12 06:56   
To add, there must be a check if non-updated actors trigger any clientside functions mainly ACS scripts, but also spawning of clientside actors otherwise mods could be broken.
(0005750)
Torr Samaho   
2013-01-13 10:01   
Well, as soon as you start not sending information the client needs things will break inevitably. For instance, demo_spectatefreely will most likely be the first victim no matter which information you hide from the client. The main question is whether a player prefers glitches or a disconnect if there is not enough bandwidth to transfer the necessary information. If it's the former, you can only try to introduce as few glitches as possible, but surely there will be some.
(0005953)
Watermelon   
2013-02-07 00:18   
I had an idea about using REJECT tables or GL_PVS for openGL in determination of mitigating bandwidth. This raises many issues as well that have to be ironed out should such an idea be gone through with, but it could lead to revolutionary bandwidth reduction.
(0006381)
Watermelon   
2013-06-08 05:35   
Current discussion would allow a player to stay connected even if they are losing too many packets (+1024), are we interested in adding another cl_ cvar that syncs with the server for this?

cl_notimeout true/false ?
(0010400)
Watermelon   
2014-10-09 00:22   
(edited on: 2014-10-09 00:24)
As discussed with other developers, the goal is to send less updates and push through a Frame/XYZ update as needed.


For some simple heuristics, priority is given based on ranges. The client could select how much they want. For example, we could go with "within X units" the priority drops off, or anything in the LOS gets priority and the ones outside don't, or even if in LOS but far enough away we ignore...etc


I'm unsure how to handle spectators since I don't recall if they're position is sent to the server (I'm pretty sure it isn't). Since people can disconnect very easily, one option is to use their bandwidth to determine the frequency to send. The main issue is, if the player joins a server with an insane amount running around... they can easily lose all the packets before even loading the level.

One solution is to send the x/y/z from the spectator to the server... but that might break the idea of 'spectators are separate from the server'. I'm also unsure about F12'ing and what that sends.

In game players are a lot easier to deal with since they have defined LOS and positions.


-------------------

Furthermore in a slightly related area, I'm hoping to do a data dump of all the packets sent and see if theres any other areas that we can 'trim the fat' on before going straight to this implementation (as suggested I believe by Blzut).

(0010403)
ZzZombo   
2014-10-09 03:13   
I had an idea about collapsing packets. Consider A_Recoil(10) and ThrustThingZ(...) used on a thing. Both functions will ultimately lead to two sync commands being sent to clients. We could record the end value of all coordinate changes and send only one command with those data.
(0010408)
Watermelon   
2014-10-09 03:32   
Quote
I had an idea about collapsing packets. Consider A_Recoil(10) and ThrustThingZ(...) used on a thing. Both functions will ultimately lead to two sync commands being sent to clients. We could record the end value of all coordinate changes and send only one command with those data.


This is a great idea, if you have any more let me know, I will check and see if Zan does or doesn't already do this.
(0010419)
ZzZombo   
2014-10-09 11:53   
Well, there are plenty to go. Pitch, angle, velocities, basically any data clients are notified about should use that. If you meant a different idea, then maybe after I will go to bed and finally sleep :P
(0010856)
Watermelon   
2014-11-11 03:05   
I was thinking more about your ideas and it should probably go in another ticket. I'm probably going to be using that idea for some other stuff though.
(0011959)
Watermelon   
2015-04-01 02:54   
Ok so for an update:

I tried a new method where I made a larger (2^x where x is either 9 or 10) blockmap for iterating over monsters closest to the player. For example, if the player was in block (5, 5), it would do:
- monsters in (5, 5)
- then the top row from (4, 6) -> (6, 6)
- the right row w/ no overlap
- the bottom row w/ no overlap
- the left row w/ no overlap
... and repeat moving outwards a single layer at at time. The 'monster blockmap' hookup was added along with LinkToWorld() to piggy back along with it.

At each node, there was a linked list. Since monsters could flux in and out of the box quickly, the linked list was converted to my own custom rolled hashmap which increased performance a lot.

Sadly it wasn't enough, when you had 4000+ monsters like in Chillax, eventually going over 10+ players and only sending updates was too taxing. The only solution here is to use threading. Since this is a slippery slope, I decided not to go with this.



Therefore, a custom blockmap iteration method was too expensive (until processors get better), and threading -- while a definite option that would make this functional -- is not something I want to introduce due to how insane they can be to debug.



I have a new method I'm going to try which will use a smarter version of my original prototype that checks based on distance. The idea I had from the blockmap is not wasted, because I can exploit the blockmap to create an estimated density function by iterating over every block once, and then using the maximum here to make some statistical estimation on the upper bound of how far of a distance is safe to go.


I also need to investigate into how stable bandwidth is and to what percent of saturation should be allowed from the user. For example, if a user has 10 megs up/down, what is a safe amount of this value to allow zandronum to consume. 80%? Not sure yet, I'm going to do some tests.