MantisBT - Zandronum
View Issue Details
0001954Zandronum[All Projects] Suggestionpublic2014-10-10 15:282016-04-10 19:27
Watermelon 
 
normaltweakN/A
newsuspended 
1.3 
 
0001954: Changing sound sending from String to logical index to save bandwidth
Right now all the sound functions communicate via String. Therefore (excluding compression) we'd be sending stuff similar to 'myactor/attacksound123' over the wire rather than it's index (like 155).

I noticed while playing ZDoomWars, when I edited out the sounds, the bandwidth had a significant decrease if they were played a lot. After all, 23 bytes vs 2 is a big chunk when sent a lot.

I figure sending the logical index as a unsigned short and letting the client look it up should be good enough. Another option for unknown sounds could be to send 65535 and then followed by the custom string if it's unknown, or to not send the sound if it's not indexed as a logical sound.
No tags attached.
related to 0001473closed Dusk "Optional Wads" for servers 
related to 0000757new  Monster Bandwidth Mitigation 
Issue History
2014-10-10 15:28WatermelonNew Issue
2014-10-10 15:28WatermelonStatusnew => assigned
2014-10-10 15:28WatermelonAssigned To => Watermelon
2014-10-10 16:01WatermelonRelationship addedrelated to 0000757
2014-10-10 17:13WatermelonNote Added: 0010456
2014-10-10 17:13WatermelonStatusassigned => needs review
2014-10-11 01:19Blzut3Note Added: 0010461
2014-10-11 03:15WatermelonNote Added: 0010462
2014-10-11 03:21WatermelonRelationship addedrelated to 0001473
2014-10-11 07:58Torr SamahoNote Added: 0010467
2014-10-11 07:58Torr SamahoStatusneeds review => new
2014-10-11 07:58Torr SamahoStatusnew => feedback
2014-10-11 08:47Edward-sanNote Added: 0010468
2014-10-11 08:51Torr SamahoNote Added: 0010470
2014-10-11 12:44WatermelonStatusfeedback => assigned
2014-10-11 12:45WatermelonNote Added: 0010478
2014-10-11 12:45WatermelonNote Edited: 0010478bug_revision_view_page.php?bugnote_id=10478#r5680
2014-10-30 22:58WatermelonNote Added: 0010743
2014-10-31 03:21WatermelonNote Edited: 0010743bug_revision_view_page.php?bugnote_id=10743#r5898
2014-10-31 23:36WatermelonNote Added: 0010753
2014-10-31 23:36WatermelonStatusassigned => needs review
2014-11-01 05:03Blzut3Note Added: 0010756
2014-11-01 11:29Torr SamahoNote Added: 0010759
2014-11-01 11:29Torr SamahoStatusneeds review => feedback
2014-11-01 11:59DuskNote Added: 0010762
2014-11-01 12:17Torr SamahoNote Added: 0010763
2014-11-01 14:32WatermelonNote Added: 0010765
2014-11-01 14:32WatermelonStatusfeedback => assigned
2014-11-01 15:53Torr SamahoNote Added: 0010766
2014-11-01 17:20WatermelonNote Added: 0010768
2014-11-01 17:34WatermelonNote Added: 0010769
2014-11-01 17:34WatermelonNote Edited: 0010769bug_revision_view_page.php?bugnote_id=10769#r5903
2014-11-01 17:37Torr SamahoNote Added: 0010771
2014-11-01 19:25Edward-sanNote Added: 0010780
2014-11-01 19:25Edward-sanNote Edited: 0010780bug_revision_view_page.php?bugnote_id=10780#r5914
2014-11-01 19:49WatermelonNote Added: 0010781
2014-11-02 20:32WatermelonNote Added: 0010798
2014-11-02 20:32WatermelonStatusassigned => needs review
2014-11-03 19:52Torr SamahoNote Added: 0010818
2014-11-03 19:53Torr SamahoStatusneeds review => feedback
2014-11-10 03:24WatermelonNote Added: 0010850
2014-11-10 03:24WatermelonStatusfeedback => assigned
2014-11-10 03:24WatermelonStatusassigned => needs review
2014-11-23 11:07Torr SamahoNote Added: 0010936
2014-11-23 11:07Torr SamahoStatusneeds review => feedback
2014-11-23 11:10Torr SamahoNote Edited: 0010936bug_revision_view_page.php?bugnote_id=10936#r6007
2014-11-24 12:35WatermelonNote Added: 0010949
2014-11-24 12:35WatermelonStatusfeedback => assigned
2014-11-24 12:35WatermelonStatusassigned => needs review
2014-11-24 20:51Torr SamahoNote Added: 0010961
2014-11-24 20:53Torr SamahoStatusneeds review => feedback
2014-11-24 21:40WatermelonNote Added: 0010963
2014-11-24 21:40WatermelonStatusfeedback => assigned
2014-11-24 21:40WatermelonStatusassigned => needs review
2014-11-24 21:41WatermelonNote Edited: 0010963bug_revision_view_page.php?bugnote_id=10963#r6020
2014-11-24 21:42WatermelonNote Edited: 0010963bug_revision_view_page.php?bugnote_id=10963#r6021
2014-11-29 07:40Torr SamahoNote Added: 0010995
2014-11-29 07:41Torr SamahoStatusneeds review => feedback
2014-12-01 12:59WatermelonStatusfeedback => needs review
2014-12-08 15:22WatermelonNote Added: 0011046
2014-12-26 09:56cobaltStatusneeds review => needs testing
2014-12-26 09:56cobaltDescription Updatedbug_revision_view_page.php?rev_id=6156#r6156
2014-12-26 09:56cobaltNote Added: 0011091
2015-01-03 19:14Torr SamahoProduct Version1.4 => 1.3
2015-01-03 19:14Torr SamahoDescription Updatedbug_revision_view_page.php?rev_id=6293#r6293
2015-01-03 19:16Torr SamahoNote Added: 0011246
2015-01-03 19:16Torr SamahoStatusneeds testing => feedback
2015-01-04 12:47Torr SamahoNote Added: 0011271
2015-01-04 12:48Torr SamahoTarget Version1.4 =>
2015-01-05 20:09DuskStatusfeedback => acknowledged
2015-01-05 20:09DuskResolutionopen => suspended
2015-02-22 18:13WatermelonStatusacknowledged => assigned
2016-04-10 19:25DuskStatusassigned => new
2016-04-10 19:27DuskAssigned ToWatermelon =>

Notes
(0010456)
Watermelon   
2014-10-10 17:13   
'https://bitbucket.org/ChrisKOmg/zandronum-stable/commits/f0303bc828a6f067b7047ed2186c6da0be597dff [^]'


I tested this and the client functions just as normal, but with much less bandwidth when there's tons of weapons being fired.

I could extend it to others, but these were the main culprits of bandwidth hogging.

Can someone code review this to make sure I'm not missing anything?
(0010461)
Blzut3   
2014-10-11 01:19   
I don't believe we make any guarantee that the sound indexes are the same (would require SNDINFO to be protected, which in turn would make skins problematic). That said, I think that only raises a concern if anything extra is loaded before the mods loaded by the server. Or if for some reason someone modified their zandronum.pk3.

Could have some implications for 0001473 as optional wads would need to be loaded by the server following those required.

Requiring proper load order could be a good thing as it could potentially also imply that the FName table is safe as well.
(0010462)
Watermelon   
2014-10-11 03:15   
One solution to this if it fails I'm guessing would be to make a custom array for required sounds and send from that.

I'm going to investigate how skins handle sound overriding.
(0010467)
Torr Samaho   
2014-10-11 07:58   
Yeah, I'm rather sure that with the current possibilities of the skins directory you cannot assume that the sound indices are equal on the server and the clients. The files from the skins dir are loaded before the files specified with "-file".
(0010468)
Edward-san   
2014-10-11 08:47   
I wonder if you can do a reverse indexing: last has index 1, first without skins dir has last index.
(0010470)
Torr Samaho   
2014-10-11 08:51   
This wouldn't help: While skins are loaded before "-file" stuff, they are loaded after zandronum.pk3 and the IWAD.
(0010478)
Watermelon   
2014-10-11 12:45   
I'm going to have to create an indexed list that doesn't take into account the skins folder.

My plan is to not write any SNDINFO definitions from files that aren't required (this way it can be expanded to ignore optional wads too when added).

(0010743)
Watermelon   
2014-10-30 22:58   
(edited on: 2014-10-31 03:21)
My idea right now is as follows (using Dusk's idea):

1) Index all the sounds in a TArray based on their index
2) Send the index and sound over to the client
3) Mark in a boolean list which have been sent (or some kind of hashmap)
4) Only send the sound ID when it has been sent once
5) Clear on client disconnecting

This way we get around the optional wad and sound order problem.

(0010753)
Watermelon   
2014-10-31 23:36   
Requesting review:

'https://bitbucket.org/ChrisKOmg/zandronum_stable/commits/6b8e9408c28d8dc012620e8895743e5728c39161 [^]'

'https://bitbucket.org/ChrisKOmg/zandronum_stable/commits/49a37ba6efe65f4b6a0b9263fdca8decc0a37ce7?at=sound_bandwidth_compression [^]'
(0010756)
Blzut3   
2014-11-01 05:03   
Wouldn't it be possible to do the sound name lookup before storing into g_sndinfoLookupTable? That way it just translates a server index to a client index.
(0010759)
Torr Samaho   
2014-11-01 11:29   
First, there is lots of copy and paste in here, which is always bad :P. Second, trying to manually maintain a list of used sounds for each client is quite cumbersome. Can't you just send the full table when a client connects? Or is the table too big for this?
(0010762)
Dusk   
2014-11-01 11:59   
Quote
Can't you just send the full table when a client connects?

I'm a bit worried this might be a rather large bandwidth sink.
(0010763)
Torr Samaho   
2014-11-01 12:17   
Did you check how big the table is? You may be overestimating its size.
(0010765)
Watermelon   
2014-11-01 14:32   
Quote
First, there is lots of copy and paste in here, which is always bad :P.


Are we okay with macros? That'd cut down the annoying code duplication.
(0010766)
Torr Samaho   
2014-11-01 15:53   
I'd say macros are a last ditch effort to reduce code duplications, where functions and classes simply can't do it. But I don't see why you can't put

    // [CK] Now read the short and bitmask.
    sSoundID = NETWORK_ReadShort( pByteStream );
    
    // [CK] If the server is sending us a new sound, read the string, otherwise
    // we will get it from our indexed sound map.
    if ( sSoundID & SOUNDID_CONTAINS_STRING_MASK )
    {
        pszSound = NETWORK_ReadString( pByteStream );
        CLIENT_AddToSndinfoTable( sSoundID & SOUNDID_NETWORK_ID_MASK , pszSound );
    }
    else
    {
        pszSound = CLIENT_GetSoundStringFromSndinfoTable( sSoundID );
    }

into a function that gets pByteStream and returns pszSound.

But the code fragment above is not necessary anyway, if the whole table is transferred on connect. So, before just reducing the code duplication, how big is the table?
(0010768)
Watermelon   
2014-11-01 17:20   
So far on IRC we estimated it to be around 20 kb at worst, on average most mods are 10-12 kb.
(0010769)
Watermelon   
2014-11-01 17:34   
On further thought, I'm going to try out the "send the full table" idea since in addition to what you said, it will integrate *way* easier with Zan 2.0's netcommandification (and then I don't have to do these annoying copy paste things)

(0010771)
Torr Samaho   
2014-11-01 17:37   
Quote from Watermelon

So far on IRC we estimated it to be around 20 kb at worst, on average most mods are 10-12 kb.

That amount shouldn't be any problem if it's only sent when initially connecting to the server.
(0010780)
Edward-san   
2014-11-01 19:25   
Is is there a hardcoded limit for the number of sounds per zdoom game? It's in order to get the number of bytes to assign to the index.

(0010781)
Watermelon   
2014-11-01 19:49   
So far the idea is to allow 32767 sounds, which no wad should exceed hopefully ever.


If they ever did, then we can always tweak it to make 0x7FFF represent a raw string.
(0010798)
Watermelon   
2014-11-02 20:32   
Pull request with a table sent on connection done.
(0010818)
Torr Samaho   
2014-11-03 19:52   
I left some comments in the pull request.
(0010850)
Watermelon   
2014-11-10 03:24   
Submitted updated pull request.
(0010936)
Torr Samaho   
2014-11-23 11:07   
(edited on: 2014-11-23 11:10)
You may have overlooked my question in the pull request:
Quote from Torr Samaho

You didn't post the pull request for the NetCommand rewrite of the SERVERCOMMANDS_* functions in question, did you?


(0010949)
Watermelon   
2014-11-24 12:35   
The commit is in the zandronum branch here:
'https://bitbucket.org/Torr_Samaho/zandronum/commits/cd2f520a94dac386ca7952fa6243b57a64cd095a [^]'

Am I supposed to pull this into the sandbox, and then get you to pull this? Wouldn't that create a possible merge headache for you when merging into zandronum branch?
(0010961)
Torr Samaho   
2014-11-24 20:51   
No, you misunderstood me. I'm not talking about upgrading all SERVERCOMMANDS_* functions in 1.4. Let me dig out my old comment from the pull request that is hidden by now:
Quote from Torr Samaho

The easiest thing would be if you first send a pull request for the stable repository that just upgrades the SERVERCOMMANDS_* functions necessary for the sound table changes to their 2.0 versions. Then I can pull the changes and merge them into the main repository. Finally, you can send a new pull request with the actual sound info changes.
(0010963)
Watermelon   
2014-11-24 21:40   
(edited on: 2014-11-24 21:42)
I'm sorry if this isn't making sense. Can I get some further clarification? Based on what I read, if I interpreted this right:



Quote
The easiest thing would be if you first send a pull request for the stable repository that just upgrades the SERVERCOMMANDS_* functions necessary for the sound table changes to their 2.0 versions


If I read this right, send a pull request from the sandbox-zandronum-stable repository to the 2.0 repository that has the new SERVERCOMMANDS update. Then somehow select from that pull only the lines that we want that will be applied.

Then you pull the changes into zandronum-stable from the sandbox repo from a pull request I make based off of this. After that I then send just the sound info pull request based on the SERVERCOMMANDS update.



I think I'm getting confused by your first line.
If I get this wrong, could you restate it (maybe something like: sandbox pulls from 2.0 branch <something>, then you do <something>, then sandbox makes a pull request for stable.

(0010995)
Torr Samaho   
2014-11-29 07:40   
Sorry, for the unclear wording. Let me elaborate. First you need to manually copy the NetCommand based implementation of the SERVERCOMMANDS_* functions necessary for the sound table changes from 2.0 to your stable head (replacing their old versions in the stable head) and commit this. Then send a pull request for this changeset only to the stable repo. I'll then pull this changeset to stable and merge it with 2.0 (which should actually mean no changes and no conflicts during the merge, because the changes originate from 2.0). Then you can make a pull request for the traffic optimization based on the stable version that already has the necessary upgraded SERVERCOMMANDS_* functions.
(0011046)
Watermelon   
2014-12-08 15:22   
Updates done.
(0011091)
cobalt   
2014-12-26 09:56   
Issue addressed by commit a15cb89a8f98: Changed sound transmission from a string to a lookup table short, thus saving a Significant amount of bandwidth (fixes 1954).
Committed by ChrisKOmg [Watermelon] on Monday 08 December 2014 23:59:21

Changes in files:
 docs/zandronum-history.txt | 1 +
 src/cl_main.cpp | 61 ++++++++++++++++++++++++++++++++++++++++++++++++-------------
 src/network_enums.h | 1 +
 src/sv_commands.cpp | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
 src/sv_commands.h | 1 +
 src/sv_main.cpp | 4 ++++
 6 files changed, 122 insertions(+), 19 deletions(-)
(0011246)
Torr Samaho   
2015-01-03 19:16   
Apparently, the current approach of sending the table is infeasible, see 0002045. Either we need to fix this or completely remove it.
(0011271)
Torr Samaho   
2015-01-04 12:47   
To not stall the release of 2.0, we decided to remove this for the time being. A proper way to fix this, is to introduce output packet buffers that allow us to control how many packets per time are send to an individual client. This will also solve other problems. Once we have those, we can re-introduce the sound table optimization.