Support for CAN protocol

[HighPriest]

I am currently researching possible solutions similar to this project, but to monitor data packed into CAN packets.

Does this project support such a data source?
Otherwise, if I am able to translate the CAN packets into C data types, would it be difficult/worth it to extend this project?

[Foalyy]

I have a similar question regarding data sources : my data is encoded in the CBOR format (similar to JSON but more compact) and comes from a USB device (using standard USB requests, not some kind of emulated serial port). I wanted to develop a similar software to visualize this data in real-time, but Serial-Studio looks like it could fit very well if it could read it.

Being able to read from different data sources and with different formats could be very interesting, but it may quickly get complicated. However, what do you think about adding an option to open a listening socket and reading serial data from there instead of from a serial port ? I see three advantages :

• sockets and serial ports have very similar raw read/write interfaces so this should be fairly easy to implement; adding the option should not require a lot of modification to the GUI as well
• users would be able to write simple scripts that get data from any source, convert the format, and send it into Serial-Studio via the socket
• this would also allow remote viewing by transferring data over any network

[alex-spataru]

I am currently researching possible solutions similar to this project, but to monitor data packed into CAN packets.
Does this project support such a data source?

For the moment no. However, Qt does have support for CAN. It could be possible to integrate Serial Studio with it in some way.

Being able to read from different data sources and with different formats could be very interesting, but it may quickly get complicated.

I agree with you, it could be very useful. But documenting it, making tutorials & maintaining could get complicated very fast.

However, what do you think about adding an option to open a listening socket and reading serial data from there instead of from a serial port ?

I am currently working in making an application that connects & manages several RFID reader devices, these can work both through serial & and ethernet connection.

For obvious reasons, I need to let the client use their readers from both serial & TCP/IP connections. And I am currently working on having multiple data sources for the same comm. protocol / logic.

This is done relatively easy, because Qt network sockets and serial ports are both based in the QIODevice class. In other words, the module/class that interprets incoming data could work with multiple data sources without any major problems.

Regarding Serial Studio, the current software architecture is similar to this (ignore the "Plugins" block, I am still reading about developing application plugins with Qt 🤣):

By making some changes, we could get to something similar to:

With this approach, any future changes on the JSON protocol, UI, etc would not depend on the data source. Any ideas/suggestions regarding the architecture are welcome!

[alex-spataru]

EDIT: I wrote:

I agree with you, it could be very useful. But documenting it, making tutorials & maintaining could get complicated very fast.

By this I mean supporting other communication formats. I think that the best approach for the near future is to let the JSON file loaded by Serial Studio define how to interpret the incoming data, and concentrate on extending what can be defined/done with this JSON file. In other words, the JSON file could be represented as a "Serial Studio project file" specific for each MCU project.

This gives us the advantage that we do not need to use any JSON library in the microcontroller (and we can complicate the JSON file as much a normal computer allows, which is a lot). The downside is that we need to find a way to support reading binary data, which could probably be done using a combination of JSON flags & base64 in the JSON Generator module.

[Foalyy]

That looks good to me.

I don't know about Qt-specific "application plugins", but one thing that would be cool would be being able to dynamically load plugins (like some SOs/DLLs) that would provide the "manager" layer on your schema (second column from the left) independently of the main app. Those plugins would have a simple interface with the app, by providing some functions such as init(), read() and write(). You could ship the software with at least two default data source plugins, Serial and IP, as well as any user-contributed plugin that you see fit, and users could easily develop their own plugins by simply writing and compiling a simple shared object.

This would be similar to the idea in my previous post (Python scripts retrieving and converting the data), but more integrated and easier to use for the end user. This would also solve the problem you mention in your last sentence, because the "data manager" plugins could convert data from a low-level, device-specific format (such as CAN dataframes) into a serialized format that could be described with JSON. Following this example, the doc of the CAN plugin would specify that that binary field of the dataframe is mapped to that field in the JSON output with that name and that format, and the JSON description file could be written accordingly.

By wanting to be able to get data from different sources we are trying to solve two problems : actually getting the data (interfacing with the right source) and decoding it. I think being able to describe any possible binary encoding in a JSON description file (the "project" file) would be pretty difficult, but since these two problems are actually linked (you will only decode a CAN dataframe from a CAN adapter), I think it makes sense to put the logic of both (retrieving and decoding/converting) in the same place : the plugin.

In my case, I would be interested in writing a plugin to get data from my USB device and feed it to the app. With this, I could make Serial Studio the standard data-viewing interface for this project with easy setup for the users, without having to ask users to mess up with Python adapter scripts and loopback TCP interfaces.

A few other ideas/subjects of discussion :

• You mention avoiding using a JSON library in a microcontroller, I couldn't agree more. This is why I use CBOR (more specifically, the robust and open-source tinycbor implementation developed by Intel), which is similar to JSON but lower-level (binary-encoded) and way more compact. Once on the computer the data can be converted to JSON, there are a few caveats but data types and structures generally match 1:1. For this reason, it may be relatively easy to specify in the description file if the data described is JSON- or CBOR- encoded, and therefore supporting CBOR data directly without converting it inside the data source plugin. Users could encode data in CBOR instead of JSON in their microcontrollers, which would lower considerably the resources required. This is especially interesting when the serial data stream passes through some kind of wireless serial bridge with low bitrate and reliability.

• I you put a high priority on users being able to easily interface custom data sources, you could for example allow using Python scripts as plugins. There are C++ to Python bindings to call Python functions from C++ code. There would be some details to think about though (for example : this probably means embedding a Python runtime in the main app, but I don't know how you would install Python libraries into this runtime, which most plugins would probably need... I've never done this.)

• Being able to get data from multiple sources at a time could also be useful...

• Completely unrelated, but another interesting feature would be to be able to output the JSON data in realtime to send it to another Serial-Studio instance somewhere else. This would be useful for telemetry applications.

Of course, those are only ideas, and as always one should be careful with feature creep with a system such as this. Simplicity is also a virtue. Let me know what you think.

[Foalyy]

Note regarding my first bullet above : after looking more carefully at the documentation, I realize I got things mixed up a bit. Only format declaration files are in JSON, the data is a simple CSV frame that can easily be generated by a microcontroller, so my point doesn't hold as such. Sorry for the confusion.

[alex-spataru]

Update regarding this issue, I just added support for network communications in the latest commit. I do not have any MCU with network support for the moment.

However, Serial Studio seems to handle TELNET connections pretty well:

I'll work on the GUI & address resolution/validation and notify you about any updates.

[jpnorair]

Thanks for the network socket feature. It enables me to hook-up target hardware to a Pi-Zero-W (or whatever) with a serial-to-socket forwarder, and then access serial studio on a tablet anywhere in the lab (or, even, the entire office).

[alex-spataru]

As for the latest commit, I have begun working on a "plugin" system based on network sockets. Serial Studio listens for any TCP connection at port 7777 and establishes a TCP connection with the "client" plugin.

Benefits of this approach:

• You can write your plugins in the programming language & framework that you want.
• We don't need to deal with loading DLLs during runtime, which can get pretty complicated if we want to support all platforms.
• Plugins can be run from other computers in the LAN network (just specify the IP address of the machine that runs Serial Studio).

After the connection is established, Serial Studio can send two types of JSON packets:

• Data packets encode the received device data from the device in Base-64 format.
• Frame packets provide a JSON array with the received & processed JSON information.

Example of frame packet:

{
   "frames":[
      {
         "data":{
            "g":[
               {
                  "d":[
                     {
                        "t":"Team number",
                        "v":"1714"
                     },
                     {
                        "t":"Runtime",
                        "v":"00:00:11"
                     },
                     {
                        "t":"Packet count",
                        "v":"11"
                     },
                     {
                        "t":"Channel",
                        "v":"C"
                     },
                     {
                        "t":"Mode",
                        "v":"SIM"
                     }
                  ],
                  "t":"Container Status"
               },
               {
                  "d":[
                     {
                        "t":"SP1 Released",
                        "v":"0"
                     },
                     {
                        "t":"SP2 Released",
                        "v":"0"
                     }
                  ],
                  "t":"Payload Status"
               },
               {
                  "d":[
                     {
                        "g":true,
                        "max":1000,
                        "min":0,
                        "t":"Altitude",
                        "u":"m",
                        "v":"442.34",
                        "w":"bar"
                     },
                     {
                        "g":true,
                        "max":40,
                        "min":0,
                        "t":"Temperature",
                        "u":"C",
                        "v":"21.2",
                        "w":"bar"
                     },
                     {
                        "g":true,
                        "max":12,
                        "min":0,
                        "t":"Voltage",
                        "u":"V",
                        "v":"0",
                        "w":"bar"
                     },
                     {
                        "t":"Compass",
                        "u":"DEG",
                        "v":"10",
                        "w":"compass"
                     }
                  ],
                  "t":"Readings"
               },
               {
                  "d":[
                     {
                        "g":true,
                        "t":"X",
                        "u":"m/s^2",
                        "v":"1.24",
                        "w":"x"
                     },
                     {
                        "g":true,
                        "t":"Y",
                        "u":"m/s^2",
                        "v":"0.52",
                        "w":"y"
                     },
                     {
                        "g":true,
                        "t":"Z",
                        "u":"m/s^2",
                        "v":"-9.81",
                        "w":"z"
                     }
                  ],
                  "t":"Accelerometer",
                  "w":"accelerometer"
               },
               {
                  "d":[
                     {
                        "t":"Latitude",
                        "v":"20.835996656",
                        "w":"lat"
                     },
                     {
                        "t":"Longitude",
                        "v":"-99.85082993",
                        "w":"lon"
                     }
                  ],
                  "t":"GPS",
                  "w":"map"
               },
               {
                  "d":[
                     {
                        "t":"Software state",
                        "v":"0"
                     },
                     {
                        "t":"SP1 Count",
                        "v":"0"
                     },
                     {
                        "t":"SP2 Count",
                        "v":"0"
                     },
                     {
                        "t":"CMD Echo",
                        "v":"NC"
                     }
                  ],
                  "t":"Mission Status"
               }
            ],
            "t":"CanSat"
         },
         "id":"11",
         "timestamp":"Sat Mar 13 01:03:48 2021"
      }
   ]
}

As you can see, each "Frame" contains the following info:

• JSON data
• Frame ID number
• RX timestamp

Example of data packet:

{"data":"IzE3MTQsMDA6MDA6MTEsMTEsQyxTSU0sMCwwLDQ0Mi4="}
{"data":"MzQsMjEuMjAsMC4wMCwxLjI0LDAuNTIsLTkuODEsMjA="}
{"data":"LjgzNTk5NjY1NiwtOTkuODUwODI5OTMsMCwwLDAsTkM="}
{"data":"LDEwOwo="}

Decoding the base-64, we get...

{"data":"#1714,00:00:11,11,C,SIM,0,0,442."}
{"data":"34,21.20,0.00,1.24,0.52,-9.81,20"}
{"data":".835996656,-99.85082993,0,0,0,NC"}
{"data":",10;"}

I am currently working on adding support for client applications to "write" data directly to the connected serial/network device through the plugin system. I'll keep you updated.

Of course, any ideas or suggestions are welcome!

[alex-spataru]

Update: I just released a new version that supports the TCP plugins feature. Any data written by client applications to the 7777 port shall be sent to the connected device. Here is an example of such plugin/application.