New configuration system and DAQ recording

[christoph2]

So guys, this (v. 0.22) is a really comprehensive update... 🍾 🍾 🍾

The two most important changes are:

• On multiple requests there is now native support for DAQ-lists!
• New traitlets based configuration system.

Configuration System

To get a first impression, run

xcp-profile create | less

Note

Using xcp-profile is not really required, every pyXCP based program/script can handle profiles.

To write a configuration file, use option -o (-d works the same way). Note: extension .py is required in any case.

It is recommended but not necessary to use pyxcp_conf.py as file name,
because it is the file pyXCP looks for, if there is no -c option.

Simply run

python xcphello.py

instead of

python xcphello.py -c pyxcp_conf.py

Upgrading an existing configuration

Convert a legacy configuration file as follows:

xcp-profile convert -c conf_can_user.toml -o conf_can_user.py

If there's no output option (-d or -o), the result is written to stdout.

---

• The infamous issue 130 is now fixed.

This means, the new configuration system now interprets the parameters

c.Transport.Can.can_id_master = 47
c.Transport.Can.can_id_slave = 11

correctly.

For the sake of compatibility, nothing is changed when using old .JSON/.TOML files, but the IDs are crossed while converting your configuration files.
For the utterly confused, the logger dumps out some information on what's going on, e.g.:

2024-08-06 16:25:54 INFO     XCPonCAN - Interface-Type: 'kvaser' Parameters: [('channel', '0'), ('fd', False), ('bitrate', 500000),                 can.py:333
                             ('receive_own_messages', False), ('sjw', 2), ('tseg1', 5), ('tseg2', 2)]
                    INFO     XCPonCAN - Master-ID (Tx): 0x00000300S -- Slave-ID (Rx): 0x00000301S                                                   can.py:334
                    INFO     XCPonCAN - Using Interface: 'Kvaser U100, S/N 1093742 (#1)'                                                            can.py:256
2024-08-06 16:25:55 INFO     XCPonCAN - Filters used: [{'can_id': 769, 'can_mask': 2047, 'extended': False}]                                        can.py:257
                    INFO     XCPonCAN - State: BusState.ACTIVE

---

Python as configuration language opens completely new posibilities, e.g.: a Python function instead of a .DLL can be used for seed-and-key unlocking:

def SeedKeyXCP(resource: int, seed: bytes):
    temp0 = seed[0]
    temp1 = seed[1]
    temp2 = seed[2]
    temp3 = seed[3]

    temp = (temp3 << 24) | (temp2 << 16) | (temp1 << 8) | temp0
    temp = ((temp >> 5)) | (temp << 23)
    temp = (temp * 7)
    temp = temp ^ 0x26031961

    key = bytearray(9)
    key[0] = ((temp >> 0)   & 0x00FF)
    key[1] = ((temp >> 8)   & 0x00FF)
    key[2] = ((temp >> 16)  & 0x00FF)
    key[3] = ((temp >> 24)  & 0x00FF)
    return bytes(key)

#c.General.seed_n_key_dll = 'SeedNKeyXcp.dll'
c.General.seed_n_key_function = SeedKeyXCP

Tested with Vector's XCPsim (CANape 20.0)

DAQ lists

First of all, there's an example run_daq demonstrating how-to DAQ -- Please read on before trying to run!

run_daq was tested with CANape 20.0 and XCPlite;
The CANape addresses may work, the XCPlite ones will certainly NOT!

DAQ measurements can be done in two ways:

• Online, collect and or process data while DAQ measurement is running.
• Offline, data is recorded to a .xmraw file and then processed afterwards.

This basic descision looks like this:

# daq_parser = DaqToCsv(DAQ_LISTS)  # Record to CSV file(s).
daq_parser = DaqRecorder(DAQ_LISTS, "run_daq", 2)  # Record to ".xmraw" file.

Note: DaqToCsv could be used as as startingpoint for own live processing.

---

If you look at the signature of the callback function

def on_daq_list(self, daq_list: int, ts0: int, ts1: int, payload: list):
	...

or a finished recording

timestamp0,timestamp1,byteCounter,sbyteCounter,wordCounter,dwordCounter
1724389279036840900,1970172536,141, -115, 4749, 70285
1724389279067043600,1985243936,142, -114, 4750, 70286
1724389279128505800,2046627736,146, -110, 4754, 70290
1724389279128540700,2061804936,147, -109, 4755, 70291
1724389279144103000,2077478736,148, -108, 4756, 70292

you notice two timestamps, timestamp0 and timestamp1.
timestamp0 is created by pyXCP, while timestamp1 is the DAQ timestamp from your ECU or simulator (contained in the first ODT).

Setup

There are basically only two data-structures the user has to care about:

from pyxcp.daq_stim import DaqList

A DaqList correspondeds to a XCP DAQ-list and looks conceptually like this:

class DaqList:
	meas_name: str
	event_num: int 
	stim:  bool
	enable_timestamps:  bool
	measurements: list[Measurement]

• meas_name is important when it comes to storage later on, it can be used for example as a name of a .csv file, a SQL table, a MDF channel-group, and so on.
• event_num the number of the event the DAQ-list is attached to, depending of the implementation, events may only used once.
• stim is the direction of the DAQ-list (DAQ/STIM), and must be Falsefor now.
• enable_timestamps timestamp-generation may enabled or disabled or may fixed, this is completely implementation-defined.
• measurements is a list of tuples with the following content:

  • name in most cases the name of the A2L MEASUREMENT element.

  • address

  • address_extension

  • datatype datatype mnemonic, to get an overview run:

     python -c "from pyxcp.recorder import DATA_TYPES; print(DATA_TYPES)"

    Note

    16-bit floating-point variables are fully supported, but the availability is compiler-dependent;
    F16 (5-bit exponent and a 11-bit mantissa), BF16 (8-bit exponent and a 8-bit mantissa, "brain-float")
    for details s. cppreference, and here

The allocation and optimization of ODTs is done automatically by pyXCP.

After a recording is done, you can now start processing; the basic pattern is as follows:

from pyxcp.recorder import XcpLogFileDecoder

class Decoder(XcpLogFileDecoder):

    def __init__(self, recording_file_name: str) -> None:
		"""
		Regular Python constructor, gets the file name of the recording.
		At this point you may set an output file name, e.g.
		
		self.my_file_name = Path(recording_file_name).with_suffix(".txt")		
		"""
		
	def initialize(self) -> None:
		"""
		Here you may create files, objects, ...
		"""
		
	def finalize(self) -> None:
		"""
		Close files, and the like at this stage.
		"""
		
	def on_daq_list(self, daq_list_num: int, timestamp0: int, timestamp1: int, measurements: list) -> None:
		"""
		As you may guess, this is called on every finished DAQ-list.
		"""
		
decoder = Decoder("my_recording.xmraw")
res = decoder.run()	

The are three examples for common use-cases (currently no Excel, sorry):

• ex_arrow create Parquet files.
• ex_mdf create MDF file.
• ex_sqlite create SQLite3 database.

Miscellaneous / Minor changes

• The build system is now poetry -- setup.py is history!
  As an alternative to python setup.py develop one should use pip install -e .

• Timestamping was somewhat chaotic (time.time(), time.perf_counter(), ...) -- now handled by a C++ extension, timestamps are now UTC values (64-bit, with nano-seconds resolution). The startup timestamp (incl. timezone, UTC-offset, and DST-offset) is accessible:

with ap.run() as x:
    print("Start DT: ", x.start_datetime)	

• If you already have a communication link to your bus/network, you may re-use the connection. (s. parallel access to one device on windows #162), e.g.:

import can

from pyxcp.cmdline import ArgumentParser

can_if = can.Bus(interface="kvaser", channel="0", fd=False, bitrate=500000)
ap = ArgumentParser(description="external interface test")

with ap.run(transport_layer_interface=can_if) as x:
	x.connect()	
	x.disconnect()

can_if.shutdown()	# Ownership is (of cause) not transfered to pyXCP.

The type of the interface is currently not deduced, so the interface and the transport-layer configuration must match, e.g.:

c.Transport.layer = 'CAN'

• rich as a terminal-IO library, more comprehensible exceptions and colorful logging.
• STIM is not implemented yet, but there's some infrastructure.
• Also work-in-progress: PTP / IEEE1588 hardware-timestamping support (C++ extension).
• Building on RasbPIs and Beagles is problematic, due to memory restrictions -- Docker to the rescue, but I'll open a separate discussion on this topic.
• Minimum supported Python version is 3.8.

---

[SaxElectronics]

Hi @christoph2 ,
I am trying to run
xcp-profile create | less
and getting the error
ImportError: DLL load failed while importing cpp_ext: The specified module could not be found.
as reported in the issue here: #188

What i did is to clone the repo and execute "pip install -e . " - the cpp exstensions are building fine (there are warnings but all gets build )

-- Up-to-date: ./pyxcp/pyxcp/recorder/rekorder.cp312-win_amd64.pyd
-- Up-to-date: ./pyxcp/pyxcp/cpp_ext/cpp_ext.cp312-win_amd64.pyd
-- Up-to-date: ./pyxcp/pyxcp/daq_stim/stim.cp312-win_amd64.pyd

I have the feeling that installing from pip or a a precompiled wheel works fine, but the custom way does not work. By custom way I mean to download the code and execute "pip install -e .", perhaps something else is still missing to execute?

[SaxElectronics]

OK now xcp-profile is working, i had to copy manually the mingw dlls to cpp_ext folder, do you know why this is required? (although MinGW was on the path)

Copy-Item "C:\MinGW\mingw64\bin\libstdc++-6.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"
Copy-Item "C:\MinGW\mingw64\bin\libgcc_s_seh-1.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"
Copy-Item "C:\MinGW\mingw64\bin\libwinpthread-1.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"

then it generates a large text outcome. Not sure how exactly to use it.

Update: OK, starting to understand it, do you plan to keep the old way (yaml) also and the converter or do you want to remove it at some point. I don't know the "traitlets" what benefits does it offer vs. yaml? I see it is python code but still don't get the practical benefits.

I am thinking now how to integrate this to my own (high level) class. Either to keep the yaml as it is now and use the converter or to think of an way to integrate directly pyxcp_conf.py. I am not sure what is the best approach.

How can I pass my c configuration file I have created pyxcp_conf.py to the application?
I tried:
application = create_application(["--config", config_file_path])
and it seems not to work.

I tried also:

os.environ["PYXCP_CONFIG"] = "D:/Project/protocols/XCP/pyxcp_conf.py"
config_file_path = "D:/Project/protocols/XCP/pyxcp_conf.py"

# Create the application
application = create_application()

# Set the path to your configuration file
config_file_path = "D:/Project/protocols/XCP/pyxcp_conf.py"

# Read the configuration
application._read_configuration(application.config_file)

master = Master(application.transport.layer, config=application)

It is failing with:

    raise FileNotFoundError(f"Configuration file {file_name!r} does not exist.")
FileNotFoundError: Configuration file 'pyxcp_conf.py' does not exist.

Uh I must be honest with you, it is a struggle. :) I still cannot get it runnig although more than couple of hours/days spent and although my prev code with old configuration was working fine. Couple of more examples without using the from ArgumentParser and cmdline tool would be good. I mean putting all configuration work "behind the scenes" is in general good, but not for every use case. When you have to integrate the pyxcp-lib in a much bigger project it is inevitable to understand at least how to interface the library correctly without having to debug each class. For example how one can pass a configuration and in which order to initialize whic class etc..

For example this works:

sys.argv = [sys.argv[0], f"--c={config_path}"]
application = create_application()

master = Master(application.transport.layer, config=application)

But without using sys.arg it is not finding the configuration file.

Now I am getting the next error:

self.can_interface = PythonCanWrapper(self, self.interface_name, config.timeout, **parameters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: keywords must be strings
And I am clueless, because the configuration seems fine to me.

not sure what is wrong with my configuration file:

# CAN interface supported by python-can
c.Transport.Can.interface = "slcan"

# Enable self-reception of sent messages.
c.Transport.Can.use_default_listener = True

# Channel identification. Expected type and value is backend dependent.
c.Transport.Can.channel = "COM5"

# CAN bitrate in bits/s (arbitration phase, if CAN FD).
c.Transport.Can.bitrate = 500000

# CAN-ID master -> slave (Bit31= 1: extended identifier)
c.Transport.Can.can_id_master = 4

# CAN-ID slave -> master (Bit31= 1: extended identifier)
c.Transport.Can.can_id_slave = 3

# Auto detection CAN-ID (Bit31= 1: extended identifier)
c.Transport.Can.can_id_broadcast = 256

# Master to slave frames always to have DLC = MAX_DLC = 8
c.Transport.Can.max_dlc_required = False

# One CAN identifier per DAQ-list.
c.Transport.Can.daq_identifier = [0x5, 0x6, 0x7]

# Bus timing value tseg1 (arbitration, TSEG1 if CAN classic).
c.Transport.Can.tseg1_abr = 8

# Bus timing value tseg2 (arbitration, TSEG2, if CAN classic)
c.Transport.Can.tseg2_abr = 4

# Bus timing value sample jump width (arbitration, SJW if CAN classic).
c.Transport.Can.sjw_abr = 4

Do you have any idea of what is missing in the transport configuration?

OK the configuration is fine, but seems the parsing of the pyxcp_conf.py failed (for some unkown reason)

[christoph2]

@SaxElectronics

• The old configuration system was really half-baked, had many flaws and was not suitable for more advanced scenarios; traitlets on the other hand is used by large projects like IPython and Jupyter. And no, there are currently no plans to remove support for the old format,

• The name of configuration file is passed as command line argument:

  python xcphello.py -c some_config.py

  But may also hardcoded in your script, if you add at the beginning:

   import sys
   sys.argv.extend(["-c", "some_config.py"])

  btw: The old files can be used at this point, without conversion.

• The main-switch for the transport-layer in the configuration file is:

  c.Transport.layer="CAN"  # ['CAN', 'ETH', 'SXI', 'USB']

• The interface c.Transport.Can.interface = "slcan" is then automatically instantiated.

• Follow in any case the offical way to start pyxcp:

  from pyxcp.cmdline import ArgumentParser
  
  ap = ArgumentParser(description="")
  with ap.run(policy=daq_parser) as x:
     x.connect()
     x.disconnect()

  Especially the with context-manager is crucial for the correct start and shutdown of the (multi-threaded) transport-layer machinery.
  ArgumentParser is probably as misnomer, it is more an app-launcher that also does stuff like command-line arguments.

• I'm currently working hard on the big picture (asamint project), this will then a larger "example" of how-to integrate pyxcp.

[SaxElectronics]

thanks for your support! I will try it. You did really great work with all this efforts in bringing an open-source solution for xcp.

Perhaps one side note here, if you allow me. It would be great if you keep the other use case of pyxcp also in mind while doing future improvements.

"What would be the most easy way to integrate pyxcp into a bigger (custom) project?"

I this case pyxcp acts as a lib and has to interface other classes and libraries etc, other threads.... (One Example: GUI + backend processing + uds/xcp protocols). Some standartisation of the inetrface to the rest of the world in such a case would be beneficial. In terms of configuration passing but also initialisation and shutting down.

I think this way pyxcp can really scale and would be widely used by many projects (I can imagine).

Perhaps you are already taking into account this in the asamint project.

[SaxElectronics]

For example I am not sure how this way of initialisation would fit into my project, but will try it out.

from pyxcp.cmdline import ArgumentParser

ap = ArgumentParser(description="")
with ap.run(policy=daq_parser) as x:
   x.connect()
   x.disconnect()

I might have to come up with some way of patching to make it work.

What do you mean "old way files of configuration can be used"

can_auto_detector = CANAutoDetector(protocol="XCP")
detected_config = can_auto_detector.get_can_config_by_feature("inv_general")
    
config = readConfiguration(detected_config)
print(config)
# Create an instance of the Master class with the CAN transport
master = Master("can", config)

where detected configuration is a default config:

DEFAULT_XCP_CONFIG = {
        "TRANSPORT": "CAN",
        "CAN_DRIVER": "SlCan",
        "CAN_USE_DEFAULT_LISTENER": True,
        "CHANNEL": "COM5",
        "ACCEPT_VIRTUAL": False,
        "BAUDRATE_PRESET": True,
        "CAN_ID_MASTER": 4,
        "CAN_ID_SLAVE": 3,
        "CAN_ID_DAQ0": 5,
        "CAN_ID_DAQ1": 6,
        "CAN_ID_DAQ2": 7,
        "CAN_ID_BROADCAST": 256,
        "MAX_DLC_REQUIRED": False,
        "BITRATE": 500000,
        "BTL_CYCLES": 20,
        "SAMPLE_RATE": 1,
        "SAMPLE_POINT": 87.5,
        "SJW": 4,
        "TSEG1": 8,
        "TSEG2": 4,
        "CREATE_DAQ_TIMESTAMPS": False,
        "LOGLEVEL": "DEBUG",
        "ALIGNMENT": 8,
        "TIMEOUT": 3.0,
        "DISCONNECT_RESPONSE_OPTIONAL": False,
        "DAQ_IDENTIFIER": [0x5, 0x6, 0x7],
    }

Do you expect this to work in the very same way?

[SaxElectronics]

OK now I have tried with the official approach and I am getting the very same issue, I added some print statements, for some reason master and slave can ids, cannot be parsed with the configuration parser.

DEBUG:root:pyxcp version: 0.22.28
<pyxcp.config.PyXCP object at 0x0000018B1CAB6120>
DEBUG:can.interface.detect_available_configs:interface "slcan" does not support detection of available configurations
[PyXCP] WARNING | XCPonCAN - 'slcan' has no support for parameter 'fd'.
[PyXCP] WARNING | XCPonCAN - 'slcan' has no support for parameter 'receive_own_messages'.
Interface name: 'slcan'
Timeout: 3.0
Parameters type: <class 'dict'>
Parameters keys type check: [('channel', <class 'str'>), ('bitrate', <class 'str'>), (None, <class 'NoneType'>)]
Parameters full content: {'channel': 'COM5', 'bitrate': 500000, None: 4}

Here None: 4 should be actually 'can_id_master'

# CAN-ID master -> slave (Bit31= 1: extended identifier)
c.Transport.Can.can_id_master = 4

# CAN-ID slave -> master (Bit31= 1: extended identifier)
c.Transport.Can.can_id_slave = 3

Do you have any idea why these two are not properly parsed?

i just tried converting my old configuration to the new one and there are

 xcp-profile convert -c conf_can_slcan.json -o conf_can_slcan.py
[PyXCP] WARNING | Unknown keyword 'BTL_CYCLES' in config file
[PyXCP] WARNING | Unknown keyword 'SAMPLE_RATE' in config file
[PyXCP] WARNING | Unknown keyword 'SAMPLE_POINT' in config file
[PyXCP] WARNING | Unknown keyword 'DAQ_IDENTIFIER' in config file

Why it is not recognizing the daq identifiers?

I think I am giving up for today, will try further tomorrow ;) Will appreciate any help to get it up and running. The issue is definitive in how the configuration is parsed.

[SaxElectronics]

Hi @christoph2,

There is an issue i have found in how the configuration is parsed for SlCan. I created here this issue: #217

The problem is that some parameters are replaced with "None" for SlCan class and then PythonCanWrapper is failing. I created a patch in my case, but I guess the best would be if you look into the issue and find the most appropriate place to fix it.

[christoph2]

@SaxElectronics
Works for me, both Kvaser U-100 and a cheap serial interface.

c.Transport.Can.can_id_master = 768
c.Transport.Can.can_id_slave = 769

xcp-info.exe -c .\conf_can.py

2025-05-09 11:40:55 INFO     XCPonCAN - Interface-Type: 'kvaser' Parameters: [('channel', '0'), ('fd', False), ('bitrate', 500000),                       can.py:333
                             ('receive_own_messages', False), ('sjw', 2), ('tseg1', 5), ('tseg2', 2)]
                    INFO     XCPonCAN - Master-ID (Tx): 0x00000300S -- Slave-ID (Rx): 0x00000301S                                                         can.py:334
                    INFO     XCPonCAN - Using Interface: 'Kvaser U100, S/N 1093742 (#1)'                                                                  can.py:256
                    INFO     XCPonCAN - Filters used: [{'can_id': 769, 'can_mask': 2047, 'extended': False}]                                              can.py:257
                    INFO     XCPonCAN - State: BusState.ACTIVE    

[SaxElectronics]

Now connecting and disconnecting works fine! :)

@christoph2

regarding

ImportError: DLL load failed while importing cpp_ext: The specified module could not be found.
as reported in the issue here: https://github.com/christoph2/pyxcp/issues/188

I could only solve it by copying these DLLs manually:

Copy-Item "C:\MinGW\mingw64\bin\libstdc++-6.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"
Copy-Item "C:\MinGW\mingw64\bin\libgcc_s_seh-1.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"
Copy-Item "C:\MinGW\mingw64\bin\libwinpthread-1.dll" -Destination ".\pyxcp\pyxcp\cpp_ext\"

Any idea why how to avoid this manual copy step?

[christoph2]

@SaxElectronics
If the *.dlls are on PATH it should work -- but maybe there are 32bit versions found earlier? I don't know.
Have you tryied to compile and run a simple C/C++ program?

[SaxElectronics]

yes, i compiled fully your cpp code for daq list handling. Seems that the DLLs are not on the path in a standard MinGW installation. I have to double check this, manual copy is not so nice.

Finally, my previous script is running with latest pyxcp code 🎉🎉🎉!

[SaxElectronics]

@christoph2 One more question I have, If I cannot use the ArgumentParser, would this be a legitime way of instantiating the classes and reading a configuration file?

from pyxcp.master import Master
from pyxcp.config import PyXCP

# Create a PyXCP application instance
app = PyXCP()

# Initialize the app (sets up basic properties)
app.initialize([])  # Empty list indicates no command-line args to parse

# Read configuration - use _read_configuration as it sets up all necessary instances
config_file_path = "your_config.py"  # or .json or .toml
app._read_configuration(config_file_path)

# Create the Master instance
master = Master(
    app.transport.layer,
    config=app
)

Update: I am having difficulties bypassing the "official" way of initializing pyxcp and the above approach seems to not work. While the sys.arg might be a valid approach in simple scripts, it is much more troublesome when integrating in a more bigger project. Just as an example I have not only XCP but also UDS as a lower level protocol communication. Any help on how to bypass "create_application" and "sys.arg" is appriciated ( it will save me hours of frustrated debugging ). In my scenario a more detailed level of control and initialization of the library is required. Also a of inititializing the BUS and shutting down the bus separetly from the protocol inititialization would be very beneficial.

Questions:

• What would be the correct sequence in initializing PyXCP and passing after a configuration?
• How to initialize PyXCP and pass an active BUS communication (decoupling protocol from transport layer)?

I think the answers to these questions can be useful also to others using pyxcp.