Unable to build esp32 port

[Tobinator32]

Hello,
im currently unable to build your lib along with micropython. They are in the same directory. The selected port is for the esp32.
Just to get things right im supposed to build all of this from the micropython-wrap folder right?
Doing:
make usercmodule MICROPYTHON_PORT_DIR=../micropython/ports/esp32
results in:
CMake Error at /project/libs/micropython/py/usermod.cmake:46 (include):
include requested file is a directory:
/project/libs/micropython-wrap
which makes sense to me. By fixing that line to point to module.cpp for example this wont build too resulting in.
CMake Error at /project/libs/micropython-wrap/tests/module.cpp:23:
Parse error. Expected "(", got identifier with text "namespace".
Call Stack (most recent call first):
/project/libs/micropython/py/usermod.cmake:46 (include)
esp32_common.cmake:22 (include)
main_esp32/CMakeLists.txt:11 (include)

Im guessing im doing something fundamentally wrong?
Can you point me in the right direction.
Kind Regards Tobi

[stinos]

I have no experience with building micropython or modules with CMake, so micropython-wrap test builds are currently Makefile-based only. However: note that those are test builds, so if your goal is using micropython-wrap you don't need actually need those, and you should (apart from toolchain-specific issues, not sure if anyonre used this on ESP32 already) be able to just include the header file in your own code to use it.

[Tobinator32]

Thanks for the reply. I just reverted to using cmake for the build and it seems to be somehow more successful.
Now im gettings lots of other errors:
/project/libs/micropython-wrap/tests/../detail/micropython.h:68:19: note: conversion of argument 2 would be ill-formed: /project/libs/micropython-wrap/tests/../detail/micropython.h:98:97: error: invalid conversion from 'void* (*)(long unsigned int, void* const*)' to 'void* (*)(size_t, void* const*, mp_map_t*)' {aka 'void* (*)(unsigned int, void* const*, _mp_map_t*)'} [-fpermissive] 98 | static mp_obj_t Create( BuiltinFixedT, BuiltinVarT call ) { return MakeFunction( NumArgs, call ); } | ^~~~ | | | void* (*)(long unsigned int, void* const*)
Any idea how this can be resolved? Btw im building with idf5.2 in docker and latest micropython. Currently using the tests folder from up-wrap as a starting point.
Kind Regards Tobi

[stinos]

invalid conversion from 'void* ()(long unsigned int, void const*)' to 'void* ()(size_t, void const*, mp_map_t*)' {aka 'void* ()(unsigned int, void const*, _mp_map_t*)

Aha. Could it be that on ESP32 size_t isn't the same as mp_uint_t? Does casting like (mp_uint_t) NumArgs fix this?

building with idf5.2 in docker and latest micropython

If you can elaborate how to do that exactly I can have a go at it.

[Tobinator32]

yes, it seems the types are different here. I tried changing the types but broke more along the way.
Here are the steps to reproduce.
Clone latest micropython and wrap into same directory (along each other like in your docu).
In up examples/usercmodule modify the cmake file to point to the next cmake file inside up-wrap i.e.
include(/project/micropython-wrap/micropython.cmake) and uncomment the rest.
create a cmake in up-wrap:

Create an INTERFACE library for our CPP module.
add_library(usermod_cppexample INTERFACE)

 Add our source files to the library.
target_sources(usermod_cppexample INTERFACE
    ${CMAKE_CURRENT_LIST_DIR}/tests/cmodule.c
    ${CMAKE_CURRENT_LIST_DIR}/tests/module.cpp
)

 Add the current directory as an include directory.
target_include_directories(usermod_cppexample INTERFACE
    ${CMAKE_CURRENT_LIST_DIR}
)

target_compile_features(usermod INTERFACE cxx_std_17)
target_compile_options(usermod INTERFACE -Wno-missing-field-initializers -Wno-cast-function-type -Wextra -frtti -fexceptions)

 Link our INTERFACE library to the usermod target.
target_link_libraries(usermod INTERFACE usermod_cppexample)

launch docker inside the folder where up and upwrap are located:
docker run --rm -v ${PWD}:/project -w /project -it espressif/idf:v5.2
cd to micropython/ports/esp32
then build with make USER_C_MODULES=../../../examples/usercmodule/micropython.cmake

[stinos]

Thanks. I started with a stripped down version to add things one by one, but even just

target_compile_options(upywraptest INTERFACE -frtti)

leads to that option being used for every micropython source file:

cc1: warning: command-line option '-frtti' is valid for C++/D/ObjC++ but not for C

which doesn't seem right. Isn't there there a way to specify these options only for the user module? Like CXXFLAGS_USERMOD for Makefile builds? Or at least only for the C++ sources? I'm afraid that like this it's going to cause issues later on.

Anyway there are indeed some locations with type mismatches; all the mp_uint_t in the MakeFunction definitions should actually be size_t for instance. I'll add a 32bit build to the CI to spot/fix these. Also there's some strange long vs int thing going on but only for ESP32 it seems.

[Tobinator32]

Im slowly refining my makefile in the meantime also because of the other issue we are discussing.
Essentially you can add $<$<COMPILE_LANGUAGE:CXX>:-frtti> to only use this for c++ files.
Kind Regards Tobi

[stinos]

Essentially you can add $<$<COMPILE_LANGUAGE:CXX>:-frtti> to only use this for c++ files.

Thanks, that works. I also found that it's possible to specify things per file, but then I wonder what happens if other C++ objects would be built with conflicting options like -fno-rtti -fno-exceptions. Linker errors probably, so: might actually be better this way.

I'll check to remove need for rtti and exceptions at one point though, both are not strictly needed.

FYI this branch https://github.com/stinos/micropython-wrap/tree/32bitlong already has all (well, all the ones I could find) 32bit / long type conversions fixed. So now it's a matter of figuring out those duplicate symbols. Could be an upstream micropython issue: after all we're just saying "here is a user module" but then it creates 2 objects and seems to link them as well:

find . -iname *module.c*obj
./build-ESP32_GENERIC/esp-idf/esp_hw_support/CMakeFiles/__idf_esp_hw_support.dir/rtc_module.c.obj
./build-ESP32_GENERIC/esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/projects/micropython/py/objmodule.c.obj
./build-ESP32_GENERIC/esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/projects/micropython-wrap/tests/module.cpp.obj
./build-ESP32_GENERIC/esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/projects/micropython-wrap/tests/cmodule.c.obj
./build-ESP32_GENERIC/CMakeFiles/micropython.elf.dir/projects/micropython-wrap/tests/module.cpp.obj
./build-ESP32_GENERIC/CMakeFiles/micropython.elf.dir/projects/micropython-wrap/tests/cmodule.c.obj

That's just not right. Also note that it doesn't do it for micropython's nor esp's own files

[Tobinator32]

I will try the new branch within the next days hopefully. Thanks for fixing that.
Yeah i found the duplications too. Whats more curious is that the object files differ in size. Especially the c++ object which is sometimes double the size of the other. No idea why it behaves like that.
I played with static and inline extern etc keywords but as soon as i want to call/use the bogus functions everything it came in contact with is duplicated. So that seems like a deadend.

[stinos]

I have the impression the linking issue is actually because -fexceptions so next I'll work on making everything build (again) without RTTI and exceptions available.

[Tobinator32]

Can confirm it seems like my example cpp project without up-wrap and no exceptions and rtti compiles fine. I will play a little more if rtti or exceptions are the issue. Templates and such are involved and a lot of classes etc.
PS:
Tested a few configurations. Im rather sure that exceptions are a problem. Enabling them causes lots of multiple definitions.
RTTI on the other hand does nothing although not extensively used in my example.
Also the compiled binary runs and the c++ functions are accesible in micropython.

[Tobinator32]

Did some more testing. First of all, exceptions need to be turned off otherwise it wont build at all even without the lib. rtti makes no difference. When introducing the Classwrapper everything behaves like exceptions have been turned on (lots of duplicates during linking).
upywrap::ClassWrapper< TestClass > wrapNewSimple( "TestClass", mod );
this line is enough to trigger it.

[stinos]

this line is enough to trigger it.

Yeah I'm aware, was hoping you found a more minimal (as in: not containing any micropython-wrap code) case.

I've fixed all build issues in the the 32bitlong branch so that everything builds as-is on ESP32 now, and added a sample .cmake file for the user module.

Next I'm hoping to get some time to look into the CMake issue.

[Tobinator32]

Hey, yes ive found several cases without your library where the exact same thing happens. Formulating a clear workaround is very difficult. It seems to me that after a certain amount of function or method calls involving templates and variants (not sure about the "ingredients") the linker issues that all symbols in the file are duplicated. Dissection the functions where the fault originates from into smaller functions sometimes yields more function calls before the error reappears. Also using static functions seems to postpone the magic limit here.
My workaround of choice at the moment is splitting it into multiple files when applicable. This results in horrible code structure but at least it builds that way. Of course when it fails the within a single line of code this doesnt help at all, so its still an issue.
I have not the slightest clue whats the error here. Im more and more convinced that its a fundamental toolchain bug

[stinos]

Im more and more convinced that its a fundamental toolchain bug

Positive. It makes no sense to me the user C module files are built twice, and with different flags, whereas everything else isn't. Also very strange this does not always lead to duplicate symbols.