Connection issues on private server.

Discussion about topics that don't fit elsewhere.
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Connection issues on private server.

Post by r7st »

Hello, I have set up a private server on a Linux VPS. I have run into some problems both with the VPS and local.

I added BIND_NAME and BIND_IP in config.h and compiled tomenet.server. I then copy the binary back and run the sever directly, or use ./runserv{1,2,3}. }What happens is, everything works as expected for a while. Then the server stops accepting connections. The server has not crashed (usually), the network is fine, ports are still open, iptables is happy. I also tried changing ports in case digitalocean was getting mad for whatever reason. The is no core dumped or error message in the server. I did get this once:

EXPORT_PLAYER_STORE_OFFERS: o_list export completed.
07 Sep (Fri) 23:00:06_CRON1H_23:0:6
07 Sep (Fri) 23:00:06 EVENT_CREATE: #7 of type 5 parms='>'
07 Sep (Fri) 23:30:06 EVENT_NOPLAYERS: 7 (Dungeon Keeper) has only 0/1 participants.
07 Sep (Fri) 23:30:06 EVENT_END: 7 - 'Dungeon Keeper'.
free(): invalid next size (fast)
Received signal 6.
corrupted size vs. prev_size
Aborted (core dumped)

But I was unable to find a core file. I also added a fuser -k <port>/tcp for the game/console ports into runserv to clean them out after any crashes. Everything works perfectly for a few hours or so, then no connections accepted. The client says to check on the hostname. I have noticed that the server does mention the "Dungeon Keeper" event before it stops accepting connections, in case that is related.

tomenet.eu is apparently a Debian server. I am running Fedora 28 on the VPS and locally. I have not tried running on a Debian machine/chroot. I will try this next to see if that could be related, although, I think it is more likely I am missing something with my config.h. The only changes I made were to remove the #if 0/#endif around BIND_IP and BIND_NAME and to set them to the ip and domain of the server.

Thanks in advance for any help. I and several other people are excited to try tomenet, but would like to play with some custom rules privately before we jump on the main server as well.
mikaelh
Developer
Posts: 216
Joined: Sun Dec 13, 2009 3:18 pm

Re: Connection issues on private server.

Post by mikaelh »

I don't remember seeing an issue like this before so it seems to be something new. Are you running the latest release which is 4.7.1a? Have you changed anything besides config.h? BIND_NAME and BIND_IP from config.h are not actually used anymore so that's probably not the cause.

Core dumps are very useful if you can find it. Especially the backtrace produced by GDB really helps to pinpoint the problem. There's some information about core dump handling in Fedora here: https://ask.fedoraproject.org/en/questi ... p-located/

Fedora tends to be a bleeding edge type of distribution. So that means they make releases with the latest versions of glibc and GCC. Those can cause old bugs to behave in new ways. So trying a different distribution can produce different results if it has different versions of glibc and GCC. The actual game server is running Gentoo and not Debian but the exact distribution doesn't really matter.
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Re: Connection issues on private server.

Post by r7st »

I changed tomenet.cfg some, but not too much. I found where the coredump was located, but unfortunately it was not kept for very long. I have changed the appropriate config to keep them around longer! Thank you for the link.

I got the same results on Debian 10, but with the same configuration. I went ahead and compiled a fresh build with no changes. So far it is running fine; if it does not crash in a few hours I will diff the config files to see what may have caused the problem. If it does, well, I should have the core dump.
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Re: Connection issues on private server.

Post by r7st »

I am getting the same issue. The server has not crashed, so I don't get a core dump. It just simply stops accepting connections. I am not sure of the best way to debug this.
mikaelh
Developer
Posts: 216
Joined: Sun Dec 13, 2009 3:18 pm

Re: Connection issues on private server.

Post by mikaelh »

If the process is still running, you can try attaching gdb to it. Just run "gdb -p PID" in a console (where PID is the ID of the process). Then type "bt" to get a backtrace. Copy and paste the backtrace here. Then just type "quit" to get out of gdb and kill the process after that.
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Re: Connection issues on private server.

Post by r7st »

__lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
95 ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
t(gdb) bt
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1 0x00007f14457a646d in _IO_vfprintf_internal (s=0x558011252e80, format=0x55800f2c99a7 "%s - %03d %s\n", ap=0x7ffd83e9cd78) at vfprintf.c:1325
#2 0xfd8576fa01a59600 in ?? ()
#3 0x00007f1445914780 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ffd83e9ce80 in ?? ()
#5 0x0000000000000010 in ?? ()
#6 0x000055800f34d3c0 in month_names ()
#7 0x000055800f34d3e0 in ?? ()
#8 0x000055800f352e00 in ?? ()
#9 0x0000000000000000 in ?? ()
mikaelh
Developer
Posts: 216
Joined: Sun Dec 13, 2009 3:18 pm

Re: Connection issues on private server.

Post by mikaelh »

Well, looks like it's stuck waiting for a lock. The stack trace is a bit weird because it's not complete. Does gdb complain about any missing debugging symbols? It seems to have found at least some symbols since it resolved the first two stack frames. It could also be a corrupt stack which is harder to debug. I may need to spin up Fedora in a virtual machine to have closer look myself. Did you do anything special in tomenet.cfg?
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Re: Connection issues on private server.

Post by r7st »

No, gdb didn't complain. That was from a debian 10 chroot on a fedora 28 vps. I did not alter tomenet.cfg. Thank you for looking into this!
mikaelh
Developer
Posts: 216
Joined: Sun Dec 13, 2009 3:18 pm

Re: Connection issues on private server.

Post by mikaelh »

I found the reason why a fresh new server crashes. Apparently it crashes when it's trying to dump the contents of player stores but there are no player stores. So a simple workaround is to go and make a player store. So just inscribe some item with @S and put in a house.
r7st
Posts: 9
Joined: Sat Sep 15, 2018 2:23 pm

Re: Connection issues on private server.

Post by r7st »

That is strange. How did you find it? I will try making a store and see how it goes.
Post Reply