Issue with lemon tool in Zandronum 3.0.1

killall -9

Moderator: Developers

Post Reply
voidpointer
New User
Posts: 6
Joined: Sun Feb 17, 2019 1:33 am

Issue with lemon tool in Zandronum 3.0.1

#1

Post by voidpointer » Mon Sep 21, 2020 2:06 pm

For the developers: The lemon tool in 3.0.1 has a segfault when compiled & run on ARM64 systems. Thanks to the TSPG folks, I was made aware this is apparently a known issue. As a test, I generated a patch between 3.0.1 and tip (as of yesterday). Once applying that patch over tools/lemon, the build succeeds. I think at this point I'd like to explore one of two possibilities:

1. Setup CMake to look for lemon as a third party tool on the system first, and if not found, build & use the source version. This should keep things backward compatible but allow the latest tool to be used regardless of which Zandronum tag I'm building. IMHO, this is the most reasonable and desirable solution.

2. Backport the lemon upgrade to 3.0.1 as a hotfix, if you have support in your release process for this (maybe a 3.0.2?). Part of what will make this tricky is that there's no public/observable changes to Zandronum itself here AFAIK; it's just a build process/tooling hotfix. But I leave this up to you guys; I'm honestly not familiar with your build & release process.

Thanks in advance for any feedback.

User avatar
Torr Samaho
Lead Developer
Posts: 1543
Joined: Fri May 25, 2012 6:03 pm
Location: Germany

Re: Issue with lemon tool in Zandronum 3.0.1

#2

Post by Torr Samaho » Sun Sep 27, 2020 11:45 am

voidpointer wrote:
Mon Sep 21, 2020 2:06 pm
For the developers: The lemon tool in 3.0.1 has a segfault when compiled & run on ARM64 systems. Thanks to the TSPG folks, I was made aware this is apparently a known issue. As a test, I generated a patch between 3.0.1 and tip (as of yesterday). Once applying that patch over tools/lemon, the build succeeds. I think at this point I'd like to explore one of two possibilities:
We know that the lemon code from 3.0.1 can't be compiled any more under Debian buster on the Raspi and upgraded it in 3.1 to fix that. I wasn't aware of segfaults under ARM64, but it's very possible that the lemon upgrade also fixes those.
voidpointer wrote:
Mon Sep 21, 2020 2:06 pm
1. Setup CMake to look for lemon as a third party tool on the system first, and if not found, build & use the source version. This should keep things backward compatible but allow the latest tool to be used regardless of which Zandronum tag I'm building. IMHO, this is the most reasonable and desirable solution.
Actually, I'm not sure whether the integrated lemon we inherited from ZDoom behaves identical to the original lemon. ZDoom usually used installed libraries (e.g. libjpeg) before using its own copies of the source of that libraries. I suspect that there is a reason why ZDoom never used the system's lemon.
voidpointer wrote:
Mon Sep 21, 2020 2:06 pm
2. Backport the lemon upgrade to 3.0.1 as a hotfix, if you have support in your release process for this (maybe a 3.0.2?). Part of what will make this tricky is that there's no public/observable changes to Zandronum itself here AFAIK; it's just a build process/tooling hotfix. But I leave this up to you guys; I'm honestly not familiar with your build & release process.
The problem with this is that I don't know whether the lemon upgrade has any side effects. We included it in 3.1, so it can get tested, but I don't think 3.1 got enough testing (likely due to the problems with the movement regulator) to know for sure that it works as intended.

Just apply https://osdn.net/projects/zandronum/scm ... 52798ebc37 to 3.0.1 and you have the upgraded lemon code. If you do (or anybody else does) sufficient testing to confirm that it has not side effects, we can talk about releasing 3.0.2 with the upgrade lemon code.

voidpointer
New User
Posts: 6
Joined: Sun Feb 17, 2019 1:33 am

Re: Issue with lemon tool in Zandronum 3.0.1

#3

Post by voidpointer » Sun Sep 27, 2020 4:06 pm

What kind of testing would you like to see happen? I've been running servers for the past week or so with this build of lemon and it has no issues that I've observed.

Regarding external lemon used by CMake, you can of course move your specialized fork of lemon to its own repository and have that built and installed separately from the zandronum code. Then it can still be searched for using CMake. Again this would just be an optional feature in CMake and it can continue to default to the behavior today. I'm happy to contribute this, but I want to first make sure you are willing to accept it.

User avatar
Torr Samaho
Lead Developer
Posts: 1543
Joined: Fri May 25, 2012 6:03 pm
Location: Germany

Re: Issue with lemon tool in Zandronum 3.0.1

#4

Post by Torr Samaho » Sun Oct 04, 2020 7:28 pm

voidpointer wrote:
Sun Sep 27, 2020 4:06 pm
What kind of testing would you like to see happen? I've been running servers for the past week or so with this build of lemon and it has no issues that I've observed.
I think it should be tested with a good mixture of maps and mods. Anything that goes through lemon, so we can make sure the behavior doesn't change.
voidpointer wrote:
Sun Sep 27, 2020 4:06 pm
Regarding external lemon used by CMake, you can of course move your specialized fork of lemon to its own repository and have that built and installed separately from the zandronum code. Then it can still be searched for using CMake. Again this would just be an optional feature in CMake and it can continue to default to the behavior today. I'm happy to contribute this, but I want to first make sure you are willing to accept it.
Our lemon code is a straight copy of the lemon code from GZDoom (in 3.1 we just updated to GZDoom latest lemon code) and I'd like to keep it that way (unless we find a good reason why GZDoom's approach on this should not be followed).

voidpointer
New User
Posts: 6
Joined: Sun Feb 17, 2019 1:33 am

Re: Issue with lemon tool in Zandronum 3.0.1

#5

Post by voidpointer » Sun Oct 04, 2020 9:12 pm

Torr Samaho wrote:
Sun Oct 04, 2020 7:28 pm
I think it should be tested with a good mixture of maps and mods. Anything that goes through lemon, so we can make sure the behavior doesn't change.
I have zero experience with Zandronum development. I don't even know what lemon is. I think clearer, more specific direction here would go a long way to allowing me to help you out.
Torr Samaho wrote:
Sun Oct 04, 2020 7:28 pm
Our lemon code is a straight copy of the lemon code from GZDoom (in 3.1 we just updated to GZDoom latest lemon code) and I'd like to keep it that way (unless we find a good reason why GZDoom's approach on this should not be followed).
I don't think we're on the same page. I'm not advocating that you change anything about your lemon distro right now. I only suggested that you separate your current lemon code from your main repository so that it may be versioned (and used) independently of Zandronum itself. Remember, the main issue here is that lemon is too tightly coupled to the Zandronum release cycle. Splitting that code into its own repository is one way of solving that problem.

Furthermore, once it is separated, the Zandronum CMake scripts can be coded to look for a lemon executable on PATH somewhere, and in your case that lemon executable will happen to be the GZDoom forked version (again, nothing different about the code itself).

I apologize if I'm not being clear about this.

User avatar
Torr Samaho
Lead Developer
Posts: 1543
Joined: Fri May 25, 2012 6:03 pm
Location: Germany

Re: Issue with lemon tool in Zandronum 3.0.1

#6

Post by Torr Samaho » Sun Oct 11, 2020 7:25 pm

voidpointer wrote:
Sun Oct 04, 2020 9:12 pm
I have zero experience with Zandronum development. I don't even know what lemon is. I think clearer, more specific direction here would go a long way to allowing me to help you out.
AFAIK the only thing GZDoom (and thus Zandronum) uses lemon for is to compile xlat_parser.y. Here are some more details on what xlat is doing: https://zdoom.org/wiki/Map_translator
voidpointer wrote:
Sun Oct 04, 2020 9:12 pm
I don't think we're on the same page. I'm not advocating that you change anything about your lemon distro right now. I only suggested that you separate your current lemon code from your main repository so that it may be versioned (and used) independently of Zandronum itself. Remember, the main issue here is that lemon is too tightly coupled to the Zandronum release cycle. Splitting that code into its own repository is one way of solving that problem.
As I said, we are handling this exactly like GZDoom is. I don't know it GZDoom keeps custom lemon changes compared to the normal lemon and whether older GZDoom versions could be safely mixed with newer GZDoom lemon versions.

voidpointer
New User
Posts: 6
Joined: Sun Feb 17, 2019 1:33 am

Re: Issue with lemon tool in Zandronum 3.0.1

#7

Post by voidpointer » Sun Oct 11, 2020 10:24 pm

Torr Samaho wrote:
Sun Oct 11, 2020 7:25 pm
As I said, we are handling this exactly like GZDoom is. I don't know it GZDoom keeps custom lemon changes compared to the normal lemon and whether older GZDoom versions could be safely mixed with newer GZDoom lemon versions.
I think my frustration here is that you aren't addressing what I'm saying directly, you just say "this is how GZDoom does it". Maybe I'm missing your point here.

Are you suggesting that you aren't willing to make any changes (or maybe don't understand much about it at all) because it's decided by GZDoom? Does Zandronum have a dependency upstream on GZDoom? I feel like that is what you are implying here.

If that's the case, I'd rather you had said I should contact the GZDoom folks for this. I made (an incorrect) assumption that this was controlled 100% by the Zandronum project. Again, my inexperience with your code base shows here. Please understand that I'm just a simple maintainer of a Docker container for Zandronum. I happen to build the source code as part of this process, so I can communicate the problems but I rely on you and the other developers for direction. I'm happy to contribute in whatever way I can, assuming I'm enabled to do so.

Another possibility is that you are suggesting you have no idea about this specific part of the code base. I'm honestly not sure. I feel like I'm having a lot of difficulty getting some progress here.

I apologize for my frustration. I hope we can somehow get on the same page about this. I'm interested in solving the problem properly. I can work around the issue with the patch (for now) but a long term solution is in order. I feel my idea perfectly addresses the issue for the long term. If I can't do this in Zandronum, let me know what I need to do.

Thanks so far for your time.

User avatar
Torr Samaho
Lead Developer
Posts: 1543
Joined: Fri May 25, 2012 6:03 pm
Location: Germany

Re: Issue with lemon tool in Zandronum 3.0.1

#8

Post by Torr Samaho » Sun Oct 18, 2020 7:59 pm

voidpointer wrote:
Sun Oct 11, 2020 10:24 pm
I think my frustration here is that you aren't addressing what I'm saying directly, you just say "this is how GZDoom does it". Maybe I'm missing your point here.
Ah, I see. Sorry for the confusion. The fact that Zandronum is essentially GZDoom (albeit an outdated version by now) + C/S code is so well known in the community that I just assumed you were aware of that. Without that knowledge, my comments don't make much sense. So yeah, you can consider Zandronum to be a fork of GZDoom, that replaces GZDoom's P2P netcode with C/S netcode. For things not related to the C/S netcode or multiplayer (and lemon has nothing to do with these), we try to stick to GZDoom as closely as possible and only deviate when there is a good reason to do so. Moreover, I never had to touch any of the lemon code myself, so I don't even know if GZDoom's lemon differs from the standard lemon and what could happen if we replace the GZDoom's lemon with the standard lemon. I also don't know why GZDoom doesn't use the standard lemon. So, if you'd like the lemon handling to be changed, GZDoom is where to ask.

BTW: Sorry, for the delayed responses, but recently I only have time to check the forums on Sunday.

Post Reply