xmlsec version issue

[tmorrell]

Package version (if known): 1.0.0

Describe the bug

I get an errorlxml & xmlsec libxml2 library version mismatch when I use xmlsec==1.3.14 on ubuntu. Switching to xmlsec==1.3.13 resolves the issue. There is discussion here xmlsec/python-xmlsec#316, but I'm not sure on the best resolution.

[utnapischtim]

is doing pip install --no-binary=xmlsec --no-binary=lxml lxml xmlsec an option for you?

I do that in my Dockerfile.

I am not sure but switching back to a older version was not an option for me because i wanted to use python3.12 and i think the newest version was necessary for that

[tmorrell]

I like configuring the python dependencies in my pipfile, and the no binary option doesn't look easy to implement in an automated fashion pypa/pipenv#5362 https://stackoverflow.com/questions/55730350/is-it-possible-to-specify-no-binary-all-in-the-pipfile.

So hopefully there is a better option?

[utnapischtim]

i am sorry, i don't see another option at the moment. i had the hope that python xmlsec will release a version which works out of the box, but it looks like that's not going to happen any time soon

[panta-123]

Tried pip install --no-binary=xmlsec --no-binary=lxml lxml xmlsec
But getting error:

STEP 5/14: RUN pip install --no-binary=xmlsec --no-binary=lxml lxml xmlsec
Collecting lxml
  Downloading lxml-5.2.2.tar.gz (3.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 25.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting xmlsec
  Downloading xmlsec-1.3.14.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.9/68.9 kB 41.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: lxml, xmlsec
  Building wheel for lxml (pyproject.toml): started
  Building wheel for lxml (pyproject.toml): still running...
  Building wheel for lxml (pyproject.toml): finished with status 'done'
  Created wheel for lxml: filename=lxml-5.2.2-cp39-cp39-linux_x86_64.whl size=6798686 sha256=b735c64a0b30a2453bbcf466d8a88a4931b66ab1a70083124622634abaf25b89
  Stored in directory: /root/.cache/pip/wheels/11/4b/af/f27a8f1fdd59c25b8ae674826f17f459743ac0a46934bbd3cd
  Building wheel for xmlsec (pyproject.toml): started
  Building wheel for xmlsec (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for xmlsec (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [43 lines of output]
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      Traceback (most recent call last):
        File "/tmp/pip-build-env-s3lgah33/overlay/lib/python3.9/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject
          section = defn.get("tool", {})[tool_name]
      KeyError: 'setuptools_scm'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-39
      creating build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/__init__.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/constants.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/template.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/tree.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/py.typed -> build/lib.linux-x86_64-cpython-39/xmlsec
      running build_ext
      building 'xmlsec' extension
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/tmp
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src
      gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_MD5=1 -DXMLSEC_NO_SHA1=1 -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes/__pycache__ -I/usr/include/python3.9 -c /tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/constants.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/constants.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_MD5=1 -DXMLSEC_NO_SHA1=1 -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes/__pycache__ -I/usr/include/python3.9 -c /tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/ds.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/ds.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_MD5=1 -DXMLSEC_NO_SHA1=1 -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml -I/tmp/pip-build-env-s3lgah33/overlay/lib64/python3.9/site-packages/lxml/includes/__pycache__ -I/usr/include/python3.9 -c /tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/enc.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/enc.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      /tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/enc.c: In function ‘PyXmlSec_ClearReplacedNodes’:
      /tmp/pip-install-4x5bqc8q/xmlsec_d06496842dda4b2a94a2eea3f81552be/src/enc.c:207:14: warning: assignment to ‘struct LxmlElement **’ from incompatible pointer type ‘PyXmlSec_LxmlElementPtr’ {aka ‘struct LxmlElement *’} [-Wincompatible-pointer-types]
        207 |         elem = PyXmlSec_elementFactory(doc, n);
            | 

OS is alma9 linux and has following installed:

RUN dnf groupinstall -y "Development Tools" && \
    dnf install -y \
        pip \
        python3-devel \
        cairo-devel \
        dejavu-sans-fonts \
        libffi-devel \
        libpq-devel \
        libxml2-devel \
        libxslt-devel \
        ImageMagick \
        openssl-devel \
        bzip2-devel \
        xz-devel \
        sqlite-devel \
        xmlsec1-devel

[utnapischtim]

i tried to build the almalinux docker image locally but it failed with the same error. what i observed:

• xmlsec1 is missing
• the installed version of xmlsec1-devel is 1.2.29
• i have in my alpine based docker image xmlsec 1.3.4 installed.

since the error error: ‘xmlSecKeyDataFormatEngine’ undeclared (first use in this function) says something about missing function declaration i think that the first step should be to checkout how to update to the newest xmlsec version.

i have no idea how to install the newest version of xmlsec and a quick search didn't gave me any insights how to do it.

sorry for not helping you more with it.

[panta-123]

Thank you for confirming.
Somehow pining the xmlsec==1.3.14 is also not working for me. Getting same sort of error.

@panta-123 it is 13 not 14 that you need to pin: xmlsec==1.3.13
@tmorrell thanks for already documenting the issue - pinning xmlsec to 1.3.13 worked for us as well.

[m6121]

Hi all, is there a better alternative than pinning xmlsec to 1.3.13? The build fails at us due to the same reasons that have been reported already. Many thanks in advance.

[utnapischtim]

@m6121 i do a RUN uv pip install --no-binary=xmlsec --no-binary=lxml lxml xmlsec in my Dockerfile. the uv is optional. it works the same with pip

the only downside is, that it takes longer to create the docker image. i created a base-image for my build step where i put that in.

[m6121]

Thanks a lot. This failed at my current base image as the xmlsec1-devel package is only 1.2.29 which is too old for the python package requiring libxmlsec1 >= 1.2.33. So I'll have to upgrade the base library first.

[utnapischtim]

@m6121 you could have a look to our base images:
https://github.com/tu-graz-library/docker-invenio-base

[m6121]

Thank you very much. That helps a lot!

[tmorrell]

xmlsec 1.3.15 is currently missing linux wheels xmlsec/python-xmlsec#344. I've set the max version to 1.3.14 in https://github.com/inveniosoftware/invenio-saml/pull/51/files