CANBus Logging and DBC Decoding

[lawsonampere]

Using this library, I have created a canbus logger application. For post processing, we apply DBCs to the canbus log MF4 file in order to decode the CAN signals. I am having issues getting the time channel data in the correct format for my post-processing tools to decode the CAN data properly. If I resample my time channel using the asammdf tools, then it is decoded correctly by the asammdf dbc application tools.

Once the DBC is applied, I get errors that the CAN signal Channel Group's acquisition source address must point to the same block as the master CG.

The result of this is that tools such as asammdf GUI split each of the CAN signals into unique channel groups without a common master time.

I have compared this to the output of processed MF4 can log files from CSS Electronic's canedge.

If I use the asammdf python library tools to re-base the time channel from zero, the problem is fixed and the DBC decode tools work as expected. Am I missing something in the way I record my timestamps using this library? The can messages right now are read from a message buffer queue and then written to the MDF file using the SaveCanMessage function.

channelGroup_ = writer_->Header()->LastDataGroup()->GetChannelGroup("CAN_DataFrame"); writer_->SaveCanMessage(*channelGroup_, queueEntry.second, *queueEntry.first);

Sample Data:
2025_03_11_time_acquisition_debug.zip

[lawsonampere]

Some further troubleshooting seems to point towards a difference between these files being in MDF 4.1 and MDF 4.2. It's possible this is an issue with the asammdf tools rather than my MDF file. When it's converted to MDF V4.1 before applying the dbc, everything is ok.

[ihedvall]

@lawsonampere
I will look into this but I just need some basic information. I assume the below data flow.

1. You use the mdflib writer object to create an MF4 CAN bus log file. I assume that the CAN messages are received on-line.
2. The MDF file and its companion DBC file is processed and converted into a sorted MDF (one DG/CG per CAN message) .
3. The resulting sorted MDF file is analyzed with some external tool (ASAMMDF?).

When calling StartMeasurement(start time), the start time sets the HD block creation time. All timestamps in the MDF file is now relative to this absolute time. When some later calling the SaveSample(sample time + CG reference), the sample is temporary stored in an internal queue. This internal queue is periodically (10s) flushed onto the MDF file. The sample time is now converted to a relative time = sample time - start time.

Creating a sorted MDF file is little bit difficult as it each CAN message is not equal to one CG sample. There exist so-called multiplexed signals meaning that a CAM message may have a new signal value. To solve this, the converter normally use the last reported multiplexed signal value.

A worse problem is signals (typical Diagnostic signals) that is larger than 8-bytes. To make it worse, it depends on the transport protocol on top of the CAN bus. If you are lucky, this is defined in the DBC file but you are normally not lucky.

There is something wrong with the sorted file. Its output configuration looks like it uses a remote (common) time master. This is a very complicated way of handling the time channel. I need to investigate further why the ASAMMDF library behave like this.

There is a weakness in the mdflib CAN configuration. It doesn't add the SI block. I will fix this issue later on (issue #120) but for the time being, you may add this block manually, after the CreateCANConfiguration() call.

I will try to reproduce the problem. I might need the DBC and the MDF bus logger file.

[ihedvall]

@lawsonampere
Analyzing the files.

1. canedge_example_dbc. This sorted file looks OK. Most viewers should handle it.
2. mdflib_application_example. This unsorted file looks OK but no DBC parsed signals.
3. mdflib_application_example_dbc. Something went wrong here. It uses the MDF 4.2 DV block instead of the ordinary DT block. The sorting application (asammdf), splits the bus log file wrong.
4. mdflib_application_retime. This unsorted file looks almost identical to [2].
5. mdflib_application_retime_dbc. This sorted file looks as [1]. It uses the MDF 4.1 DT blocks instead of DV blocks.

My guess is that [2] used the mdflib bus writer. The "only" difference between [2] and [4], is the MDF version number. I think that the asammdf is using the version number in the ID block and do some "new" type of sorting conversion.

I don't have the DBC file, so check if you can "tell" the asammdf, to not do the 4.2 conversion. I will try to see if it's possible to set the mdflib version to 4.11 instead of 4.2.

[lawsonampere]

Thank you very much for taking a look at this! I was able to use the asammdf tools to just save the mdflib-generated file as a V4.1 file. This resolved the issues with how it is interpreted in the asammdf tools which indicates the problem appears to be with the conversion of the V4.2 files as you suggested.

There is a related issue on the asammdf repo. Link for reference: Link. I've added to this and it may be something they wait to address for when the CSS loggers begin using the V4.2 format.

mdflib_application_example_MDF4.1.mf4.zip

[ihedvall]

@lawsonampere
If you add some lines that sets the minor version (4.xx) in the ID block, after you initialized the MDF file. This generates an 4.11 MDF file [2].
If we are lucky, this would solve the above problem.

  auto writer = MdfFactory::CreateMdfWriter(MdfWriterType::MdfBusLogger);
  writer->Init(mdf_file.string());
  if (auto* md4_file = writer->GetFile();
      md4_file != nullptr ) {
    md4_file->MinorVersion(11);
  }

[lawsonampere]

I confirmed this solution works great as well.

[ihedvall]

@lawsonampere
Hint 1:
A new feature in the mdflib is that it's possible to log with multiple active DG blocks at the same time. In principle, you can produce the sorted MDF directly without the unsorted middle MDF file.

Hint 2:
There is a mdf2csv executable that is a replacement for the CSS converter. Tell me if you want a "similar" converter doing the above. The CSS converters only handle CAN Edge log files. That's the reason for the mdf2csv application so other log files also works.

[lawsonampere]

I saw this development recently, I just have not quite wrapped my head around implementing the direct sorted writer. We log multiple CANbus channel simultaneously, so I believe we would need one DG for each channel?

The CSS tools do it a little bit differently when you use their tools to sort and finalize the MDF produced by their logger. They create two DG for each CAN channel, one for standard IDs and one for extended IDs but I believe this isn't required for the file to be considered sorted.

It would certainly be ideal to produce a pre-sorted file as it will help us with the local processing of the data.

[ihedvall]

@lawsonampere
There are 3 standard configurations in the ASAM MDF Bus logging.

1. All CAN channels and message into one DG block. This is the current solution and use by most loggers.
2. One DG per CAN channel. Only messages to a specific channel.
3. One DG per channel and CAN message ID.

Just to be evil, the CG naming is different for each configuration. Note that issue #120 includes [2] and [3].

The last configuration is of interest as it is now possible to add DBC signal values to the CG block. You need something like the dbclib so you can update both the CAN message bytes as well as the signal values, on the fly. Note that you need one DBC file per CAN channel, so it's a little bit of a setup issue.

[lawsonampere]

Ok, so as I understand it:

1. Currently my mdflib application/the standard mdf bus logging function in the library is doing [1] since it writes everything to one DG block.
2. The CSS electronics finalizer/sorter application used with the canedge devices creates configuration [2], but also breaks it out by extended and standard IDs. This could be done using the new sorted mdflib writer.
3. The asammdf DBC signal extractor creates [3], which could be done by the mdflib using the new sorted writer in combination with a DBC parser such as the dbclib.

[ihedvall]

@lawsonampere
There is several variations on [3] with or without DBC signals. If you are after faster read of signal values, the MDF isn't a good choice. MDF is optimized for long term storage and it a so-called sample append-able file. Column-oriented files as Parquet files are much faster to read..

A typical design is to divide the measurement data system into 2 sub-system. The first system shall long-time store and validate the measurement data files (CAN log files as above). The second sub-system create Parquet files and stores them in some optimized way for data processing. The second sub-system is a so-called Data Lake system.

Often a small index database is added for keeping track of the MDF/DBC files. The Data Lake system normally only stores a sub-set of the last measurements i.e not for long-time storage.