Fix "executable stack" errors

[mxpph]

Purpose

Fixes #1079

Approach

Adds the compiler and linker flags -Wa,--execstack and -Wl,-z,execstack respectively, as proposed by @anzz1 in the discussion on that issue.

I have tested these fixes and no longer crash due to the "executable stack" error on both Debian 13 and Fedora 42 (both with glibc 2.41).

[anzz1]

Not including the steam_api.dll and libsteam_api.so, so they have to come from the legacy branch when downloading hlds is actually a good feature. That if the user doesn't realize to change to the legacy branch, they will crash with "procedure entry SteamApps not found", "undefined symbol: SteamGameServer_Init" or other such error. It is a good thing, as if you include the steam_api and libsteam_api in the rehlds archive, the user might accidentally download the anniversary branch and mix two engine version together, which is a big no-no.

I've explained the problem in more detail in these two comments:
#1149 (comment)
#1149 (comment)

Otherwise i don't see a problem, except a similar gcc version check should be added like you can already see with the --fcf-protection, that you only add the execstack to a gcc version that understands it

# GCC >= 8.3
if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 8.0)
	set(COMPILE_FLAGS "${COMPILE_FLAGS} -fcf-protection=none")
endif()

Or is it supported by older GCC versions too, so such guard isn't needed? I've not done the research just asking whether you've confirmed this.

[anzz1]

Something like this should probably work

include(CheckCXXCompilerFlag)

# Glibc >=2.41 fix
check_cxx_compiler_flag("-Wa,--noexecstack" HAS_NOEXECSTACK_ASM)
check_cxx_compiler_flag("-Wl,-z,noexecstack" HAS_NOEXECSTACK_LINK)
if(HAS_NOEXECSTACK_ASM)
	set(COMPILE_FLAGS "${COMPILE_FLAGS} -Wa,--noexecstack")
endif()
if(HAS_NOEXECSTACK_LINK)
	set(LINK_FLAGS "${LINK_FLAGS} -Wl,-z,noexecstack")
endif()

I already checked that this CheckCXXCompilerFlag function exist in 3.1 cmake which is the current minimum version, so it should work without further changes. i didn't test to compile though.

[mxpph]

Not including the steam_api.dll and libsteam_api.so, so they have to come from the legacy branch when downloading hlds is actually a good feature. That if the user doesn't realize to change to the legacy branch, they will crash with "procedure entry SteamApps not found", "undefined symbol: SteamGameServer_Init" or other such error. It is a good thing, as if you include the steam_api and libsteam_api in the rehlds archive, the user might accidentally download the anniversary branch and mix two engine version together, which is a big no-no.

Understood. I'll incorporate your suggestions.

[mxpph]

Thanks for your snippet, I tested and it builds and runs just fine.

[anzz1]

Adding to this, my understanding is that the problem comes from trying to load shared objects with an executable stack while the main executable doesn't have it set. Meaning that dynamically loaded libraries (shared objects) cannot change the state of the executable stack, not that they aren't allowed per-se.

Where I'm getting at here is mods and their gamedlls, whatever libraries they might load, plus metamod and its modules, could run into the same issue where they're marked as requiring executable stack and fail to load.

Therefore the option of simply setting the stack executable to retain original functionality and compatibility would probably be the right way forward. That is instead of having the shared objects declared as not having executable stack, declare the main hlds binary having it, therefore allowing all the loaded modules to either have or not have it.

So instead of changing the makefiles for the modules, change it for the hlds_linux and hltv executables, and instead of noexecstack do the opposite: -Wa,--execstackand -Wl,-z,execstack

I don't have a machine with glibc 2.41 to test, could you test if that works?

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}

[mxpph]

It compiles and runs, but worth noting that I get this warning:

/usr/bin/ld: warning: cputype32.o: missing .note.GNU-stack section implies executable stack
/usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker

This is because the object files in rehlds/lib/linux32/libacof32.a were assembled from raw assembly .S files that don't contain the .note.GNU-stack section. They can be found here. https://github.com/lukego/asmlib/tree/master