How can I figure out how this DoS on my server works?

[t3hplayer]

Hey guys. There's been a player connecting to my Shotgun Frenzy server and somehow causing the server to run extremely slow (extreme CPU usage, causing very poor framerates for clients) for the remainder of the current level. I've seen it twice, and I'm fairly certain it's intentional as I've gotten various reports of trolling on the server.

All I know about how it works is that the player always rapidly jumps between spectating / joining before it happens. The last time, the player 11 times joined the game, and 10 times went spec, all in a very short period of time, and then immediately disconnected. For the duration of the current level, the server ran like shit with high CPU usage.

I did some experiments, and I think there's more to it than just the hopping between spec / in-game, because I was not able to replicate by doing that myself. I set up a test server on the LAN and had a couple clients connect, and a third client jump rapidly between spec / in-game, and I didn't notice any much elevated CPU / memory usage in top, or any noticeable slowdown on the test server.

So my question then is...how can I figure out what the client is doing? Is there some way to enable like a verbose logging or similar, so that I can get more details about what the client is going when switching between spec / in-game? Additionally, I'm not sure this if this is an SF bug or a Zandronum bug. Is there any way I can deduce that without being able to replicate? Like, can I have Zandronum to show me where the CPU cycles are going? (i.e., is it spinning its wheels running ACS code, or is it some other code that's eating up time?)

I looked through the RESPAWN, DEATH, etc scripts in SF to see if anything jumped out, and I don't see anything that seems like it should be locking up CPU cycles. I did notice, though, that the DEATH script in SF doesn't set TID to 0 like recommended in the ZDoom wiki...

Edit: A few more details. I've seen it happen on SF02 and another map (can't remember which right now, I'll have to check the logs). When I saw it happen on SF02, the logs indicate there was originally one player soloing, another client tried to connect but got a "Protected lump authentication failed", and another client did successfully connect. These two players were playing without issue for a short while. Then, the troublemaker connected, switched between spec and in-game (11 times joining, 10 times going spec before leaving), and then the server ran like shit until the level ended. There aren't any suicides or deaths from the troublemaker indicated in the server log. The only messages are the connect, the spec / in-game switches, and the disconnect.

[nax]

Hey t3hplayer,

I believe it would more likely be a sf bug vs Zandronum. There are plenty of servers such trolls would use that bug on if it were client wide. More specifically, I'm not sure what this could be, but I'll see if I can reproduce it myself.

[t3hplayer]

Hey t3hplayer,

I believe it would more likely be a sf bug vs Zandronum. There are plenty of servers such trolls would use that bug on if it were client wide. More specifically, I'm not sure what this could be, but I'll see if I can reproduce it myself.

Hey nax,

It's good to see you're still around. I added a few more details which might help with any replication efforts you or anyone else might try. I've tried a few myself (as mentioned in the OP), but thus far haven't found anything. I gotta figure he's invoking some kinda command, which doesn't cause a message in the log, which is somehow locking up cycles. I just haven't figured out what the command is. Alternately, maybe there's an issue if you go spec / come in from spec when located in a certain part of the level, for some reason? Like, maybe there's a hidden corpse or something which is eating up CPU time running some needless ACS?

I'll poke around in the code some more later today.

Also, the person doing it is probably a known troll, because there's a pretty similar IP (same ISP, different subnet) in the banlist.

[Ænima]

When you start your server, type “SV_MeasureOutBoundTraffic” and then change the map.

Then after the troll takes a shit on your server, type “DumpTrafficMeasure”. If there’s any bulky runaway scripts, you should be able to see them this way, IIRC.

[t3hplayer]

Ænima,

Thanks, I'll give that a shot next time it comes up. One source of frustration, though, is that when the server runs slow, the server console becomes sluggish as well. I tried to use the acsprofile function earlier to see what was going on during a slowdown (I just recently found out that function exists), but it took like 5+ minutes before the results showed up in the console. Possibly it gave me some useful info, though. I'll find out later.

[t3hplayer]

Quick update on this.

The player doing this also has a botnet (or possibly just a VPN, though I checked the IPs and none of them detected as VPNs according to the sites I tried). The player commands additional clients to connect from various IP addresses (so far in Japan and Korea) and vote to kick other players for no reason, and proceeds to have their other clients vote "Yes" and kick people by outnumbering them.

For now, I disabled voting.

Regarding the slowdown, part of it is that people are joining, buying and placing turrets, going spec, rejoining, buying and placing more turrets, etc. Though I'm not sure this is how all of it works, because it really seemed like one of the players was doing it too quickly for this to be what was happening.

Anyway, I'm working on an update that uses proper TIDs to destroy turrets when a player disconnects / specs. I'll also add some code to prevent the commander from adding too many turrents.

[nax]

Quick update on this.

The player doing this also has a botnet (or possibly just a VPN, though I checked the IPs and none of them detected as VPNs according to the sites I tried). The player commands additional clients to connect from various IP addresses (so far in Japan and Korea) and vote to kick other players for no reason, and proceeds to have their other clients vote "Yes" and kick people by outnumbering them.

For now, I disabled voting.

Regarding the slowdown, part of it is that people are joining, buying and placing turrets, going spec, rejoining, buying and placing more turrets, etc. Though I'm not sure this is how all of it works, because it really seemed like one of the players was doing it too quickly for this to be what was happening.

Anyway, I'm working on an update that uses proper TIDs to destroy turrets when a player disconnects / specs. I'll also add some code to prevent the commander from adding too many turrents.

Please PM me the IP you encountered.