Random Heavy Desync to ALL clients - getting kicked by BattlEye: Client not Responding

[GolovaRaoul 221]

Yo all,

I'm having a very serious problem on our Exile Chernarus server. At peak hours we have 65 people onboard the server, which is really nice. But our issue is: sometimes, all clients are getting desync'ed like HELL, with 100000 desync, and 2/3 of the people on the server are getting kicked by BattlEye: Client not Responding. We've had this problem yestherday again ( 3 times in a row! ), and I tried a manual restart. After restart the same thing happend again. I'm really not sure what's causing that. At that moment, we had a server FPS of 10 ( which is not really good, but it should do ). I've checked all connected clients: they are all having a LOW PING and HIGH bandwidth. This issue appears to be happening randomly. Sometimes it happens with "only" 40 client connected, sometimes with 65.

We've rented a dedicated machine, with Xeon E3 1245v2, 32GB RAM, SSD and very high Upload/download, so I doubt hardware is an issue here.
I really don't know how to fix this. If we want to fix this, we'll have to know the root cause first. I'm not sure what information would be relevant on this subjects, so I'll post everything in spoiles, in the hope someone can help me find this problem and solve it. Any help is appreciated!

Basic.CFG
 

Spoiler

//
// Exile Mod basic.cfg Settings
//
// This file has been tuned for a box running only one Arma 3 server on a 
// 1 GBit/s pipe with 100 slots
//
// See https://community.bistudio.com/wiki/basic.cfg
//

///////////////////////////////////////////////////////////////////////////////
// Default Options
///////////////////////////////////////////////////////////////////////////////
language = "English";
adapter = -1;
3D_Performance = 1.000000;
Resolution_W = 800;
Resolution_H = 600;
Resolution_Bpp = 32;

///////////////////////////////////////////////////////////////////////////////
// Bandwidth Tuning 
//
// Be sure to adjust this to your server settings!
// This example is for 1GBit/s pipe
///////////////////////////////////////////////////////////////////////////////
// 100MB * 1024 * 1024 = 104857600
MinBandwidth = 104857600;    

// Do NOT set this too high or your Arma server will simulate ego-ddos
// 600MB * 1024 * 1024 = 629145600
MaxBandwidth = 629145600;        

///////////////////////////////////////////////////////////////////////////////
// Network Tuning 
// 
// Usually something we developers set. Only change these if you really know
// what you are doing. Do something wrong here and your sever will burn in hell 
// and cause massive desych.
///////////////////////////////////////////////////////////////////////////////
MaxMsgSend = 256;
MaxSizeGuaranteed = 512;        
MaxSizeNonguaranteed = 256;    
MinErrorToSend = 0.004;    
MinErrorToSendNear = 0.03;        
MaxCustomFileSize = 0;            

Config.CFG
 

Spoiler

hostname                            = "PNG Exile Chernarus - 50k Start|Garage|Militarised|ZCP|Missions";
password                             = "";
passwordAdmin                          = "DELETED";
serverCommandPassword                = "DELETED";
logFile                                = "server.log";
verifySignatures                       = 1; // Do NOT use verifySignatures=2! It costs too many FPS
BattlEye                            = 1;
requiredBuild                         = 138497; // 1.62 stable
allowedLoadFileExtensions[]         = {:};
allowedPreprocessFileExtensions[]     = {"sqf"};
allowedHTMLLoadExtensions[]         = {"html"};
motdInterval                         = 5;               
maxPlayers                             = 65;        
voteMissionPlayers                     = 200;   
voteThreshold                         = 200;
disableVoN                            = 0;        
vonCodecQuality                     = 30;   
persistent                            = 1;       
kickDuplicate                         = 1;
equalModRequired                     = 0;
kickClientsOnSlowNetwork            = 0;
timeStampFormat                     = "short";
onUserConnected                     = "";
onUserDisconnected                     = "";
doubleIdDetected                     = "";
onUnsignedData                         = "kick (_this select 0)";    
onHackedData                         = "kick (_this select 0)";
onDifferentData                     = "kick (_this select 0)";
motd[]                                 = {"Welcome to PixelNationGaming!"};
maxping                                = 499;
maxdesync                           = 4999;
maxpacketloss                       = 49;

class Missions
{
    class Exile
    {
        template = Exile.Chernarus;  // Exile.Namalsk, Exile.Altis
        difficulty = "ExileRegular";
    }; 
};

RPT while server is having problems:

RPT Link

These are my Arma3server startup parameters:

Spoiler

start "Arma3" /min /high /wait arma3server.exe -mod=@CBA_A3;@CUP_Core;@CUP_Maps;@CUP_Units;@CUP_Vehicles;@CUP_Weapons;@Exile;Kart,Mark,Heli; -enableHT -servermod=@exileserver;@marma; -config=C:\Arma\Chernarus\@ExileServer\config.cfg -profiles=SC -cfg=C:\Arma\Chernarus\@ExileServer\basic.cfg -name=SC -autoinit -maxMem=4096 -loadMissionToMemory -noSound 
 

Thanks for responding! Any help is welcome here!
GolovaRaoul

Edited October 7, 2016 by GolovaRaoul

[redbutterbox 3]

Lower your maxping a fair bit perhaps?
We are running with maxping 250, which is fine for 95% of the players on the same continent as the server. YMMV

Side question: does anyone know usefull values/examples for maxdesync/maxpacketloss?

Edited October 7, 2016 by redbutterbox

[BetterDeadThanZed 1006]

How much AI do you have on the server? Missions, roaming AI, zombies... too much will really bog down your server.

[GolovaRaoul 221]

We have DMS missions, Capture Points and Occupation. But occupation is not used when server FPS < 42. And it will also downscale after 10 players.
We have +-20-25 Ai around when all missions are not cleared.

[GolovaRaoul 221]

1 hour ago, redbutterbox said:

Lower your maxping a fair bit perhaps?
We are running with maxping 250, which is fine for 95% of the players on the same continent as the server. YMMV

Side question: does anyone know usefull values/examples for maxdesync/maxpacketloss?

Thanks for your input
I've put it down, but it does not seem to have any effect. Do I need to have kickClientsOnSlowNetwork            = 0; on 1 perhaps?

[1Man 100]

Maxping isn't available anymore from what I read a few arma updates ago, even tells you this when you start your server. If you have infistar use show server information to see all objects before and after desync.

[GolovaRaoul 221]

31 minutes ago, 1Man said:

Maxping isn't available anymore from what I read a few arma updates ago, even tells you this when you start your server. If you have infistar use show server information to see all objects before and after desync.

I think you can still use it using server.cfg. Source: https://forums.bistudio.com/topic/160288-arma-3-stable-server-164-performance-binary-feedback/page-76#entry3035423 

Edit: just saw this: new server admin #commands in profiling (performance) binaries 1.60.136560 , so not sure if that will work in stable

Edited October 7, 2016 by GolovaRaoul

[redbutterbox 3]

 

1 hour ago, GolovaRaoul said:

Thanks for your input
I've put it down, but it does not seem to have any effect. Do I need to have kickClientsOnSlowNetwork            = 0; on 1 perhaps?

Maxping works well in your server config. Came a few updates ago in arma. (Battleeyes version is still out to lunch.)

You need kickClientsOnSlowNetwork = 1;  ..otherwise the laggies just stay :-)

Kicks the odd player from time to time, but doesnt react to spikes(which is good.)
Our laggy overseas players have stopped coming, which is actually good for the rest of the player base :-)

 

Ps!
Running 15-35+ players with DMS and zombies (approx. 30-60 AI).
Ye old shameless plug:

rgds Rusler/redbutterbox

https://www.gametracker.com/server_info/89.34.96.145:2472/

Had a player in from Thailand(server is in UK) with ping 350-800(!), complained that our server was laggy. Knew I had to do something then. lol

Edited October 7, 2016 by redbutterbox

[redbutterbox 3]

53 minutes ago, 1Man said:

Maxping isn't available anymore from what I read a few arma updates ago, even tells you this when you start your server. If you have infistar use show server information to see all objects before and after desync.

That's battleye, not the server commands in arma. Works wonders on our server.

[GolovaRaoul 221]

3 hours ago, redbutterbox said:

 

Maxping works well in your server config. Came a few updates ago in arma. (Battleeyes version is still out to lunch.)

You need kickClientsOnSlowNetwork = 1;  ..otherwise the laggies just stay :-)

Kicks the odd player from time to time, but doesnt react to spikes(which is good.)
Our laggy overseas players have stopped coming, which is actually good for the rest of the player base :-)

 

Ps!
Running 15-35+ players with DMS and zombies (approx. 30-60 AI).
Ye old shameless plug:

rgds Rusler/redbutterbox

https://www.gametracker.com/server_info/89.34.96.145:2472/

Had a player in from Thailand(server is in UK) with ping 350-800(!), complained that our server was laggy. Knew I had to do something then. lol

So I implented it and it's kicking some players with bad ping
Now I'm hoping this will fix our issue. I'll report back. But thanks allready

[GolovaRaoul 221]

Issue fixed. Using KickClientsonSlowNetwork = 1; + MaxPing = 350; is working very well! This can be set in Config.Cfg .

So the issue appeared to be 1 guy connecting which has a terrible connection, and the arma server is like oh yeah let's just wait till he's good and all and let all other players down. Thanks arma ^^

Can be marked as solved