Ceedling road map

[dermot-murphy]

I use unit testing by using CMake and Unity, and have it integrated into the CI workflow for my work projects.

I would like to be able to use Ceedling to create the mocks and test runners, but ...

• It's very slow - a CI build of the source on my project takes 3 minutes, while Ceedling takes around 20 minutes - they need to be equivalent times
• Ceedling is difficult to configure - I don't want to learn Ruby - I want to use it as a tool
• Ceedling fails for functions called via a macro - eg an API layer function calling a temperature sensor which can be implemented in a number of different sensor modules. Calling via a function macro is clean at the API layer, and a config H file defines the macro to be the driver layer function, and includes the driver layer H file, allowing a build to use the temperature sensor used on the PCB for that build
• Ceedling fails if a function is declared in an H file but not implemented

I would suggest:

• to add an option to allow Ceedling to run, generate the mocks and test runners only if any H file has changed, then generate CMake files, and exit. The CI could then build and run the tests faster by just running the CMake files, in the very usual case where H files had not changed.

• convert Ceedling to use YML files for configuration.

• a roadmap to indicate what changes are in the pipeline - to allow me to evaluate whether or not to switch to FFF

[Letme]

You can generate mocks and runners directly - using CMock's scripts. You do not need Ceedling for that. What Ceedling is great, is generating header files for that exact test file (so making "presonalized dependencies").

Not sure why API layer function like macros are something you would even want to unit test. You want to unit test sensor's main functions. You need to know which sensor has what expects/returns, so this is certainly not the part you would want to generalize. Do you maybe want to just mock API layer? Then create a fake proper function header file, which Ceedling will easily mock for tests. Or wrap around your function like macro magic with `#ifndef TEST #define .... #else void proper_function #endif and it should also work just fine.

Ceedling already generates full YML file for you and guides you through the process. Can you specify what should be improved/missing from https://throwtheswitch.github.io/Ceedling/latest/getting-started/installation/#getting-started-after-installation ? You wrote it is hard to configure, but it seems really easy with the dedicated commands like new, module:create and other commands. This is something that makes Ceedling way more easier than any CMake or even Makefile alternative - as long as you do not want to extend outside of the Ceedling's scope (and even then you have options). You could use for example Ceedling to build and run tests, while building release binary is done via CMake/Makefile or whatever else. I am not saying Ceedling does not do good job at building a release, but some people want some special stuff to happen for the release - and that might be difficult to inject by Ceedling.

Roadmap is something in head of maintainer and shared with us based on the issues, but since project is open-source you are free to open a feature request and contribute to it. Or pay someone to contribute, since you do not want to learn Ruby. It is a lot of effort to create a Roadmap (and maintain it), especially since there is just a small team which has no use of such overview - or even time to perform prioritysation. Mark does regularly merge the Pull requests, so there is that option.

[dermot-murphy]

Thank you for your speedy and comprehensive answer.

I have not explained the issue with macros well:

If I write a module api_temp.c that #includes api_temp_cfg.h
and in the api_temp.c write a function which calls API_TEMP_STATUS_GET(),
and in api_temp_cfg.h write
#ifdef PCB_IS_V2
#include drv_tmp144.h
#define API_TEMP_STATUS_GET() DRV_Tmp144_Status_Get()
#endif
#ifdef PCB_IS_V3
#include drv_sht30.h
#define API_TEMP_STATUS_GET() DRV_Sht30_Status_Get()
#endif
etc

For the unit test of api_temp, I would mock drv_tmp144 and drv_sht30 for different PCBs. For example, if I create a Ceedling project for V2 PCB. However, Ceedling/CMock has issues with API_TEMP_STATUS_GET. It seems to see API_TEMP_STATUS_GET as a macro and then not mock drv_tmp144.c functions.

I will try your suggestion of using just CMock rather than Ceedling to generate the mocks and runners. I think that would just leave the CMake files to be generated semi-manually.

[Letme]

So what prevents you to have that same cfg macro decision inside the utest file and then mock directly those files? You want to run those tests twice anyway it seems (I assume you looked at partials). We are diverging from the roadmap topic, so maybe open a new issue with nice codeblocked example so that we can give you advice quicker. As for roadmap, I assume maintainer has similar opinion, as I am not a member of the maintaining team :)

[mkarlesky]

@dermot-murphy Thank you for submitting all this. There's quite a lot to respond to.

• There is work we will be doing to speed up build times. A casualty of the massive amount of work that went into the 1.0.0 release removed delta builds (only building what has changed). This will return again. That said, Ceedling can only be so fast. The feature set and limitations of the C language require quite a bit of text scanning, preprocessing, and file generation.
• I am quite confused about your request to configure Ceedling with YAML. In fact, a Ceedling configuration file is YAML already. That is Ceedling’s claim to fame—one (often short) YAML file to configure a test and/or release build. Only if you are intending to implement advanced features or customizations do you need to touch Ruby.
• Generating CMake files might perhaps appear in the future. You are not the first to suggest this. That said, it's a huge undertaking and not presently envisioned to be core to Ceedling.
• You do not need to evaluate FFF vs. Ceedling. Ceedling already supports FFF. If you would like to use Ceedling and FFF, you may do so today. An FFF plugin exists that entirely replaces CMock to generate and use fakes instead of mocks.
• Yes, a test executable build will fail if a function is not defined. If that function is called, it must have a definition. There's multiple ways to manage this. You have a couple mechanisms to pull in the source definitions or you can trigger a mock of that function declaration (see documentation links below).
• I'll need to digest the macro issue you discussed. I'm quite sure your need is testable with Ceedling.
• Your request to generate only mocks and/or test runners has come up before. I think we're in a good spot to implement this. I will try to squeeze it into the upcoming 1.1.0 release.

As for a roadmap, we would love to be able to produce such a thing. Please understand that today Ceedling is almost entirely volunteer contributions. A proper roadmap, a variety of features, proper support, etc. are simply out of reach. That said, we are working towards building a business around Ceedling and generating income to properly sponsor such efforts. We are early on in this effort. Ceedling Assist engagements have begun to sponsor work. A large European company sponsored the headlining feature, Partials, for the upcoming 1.1.0 release. If you or your company are in such a position, please also consider becoming a Github Sponsor for Ceedling.

I will get back to you on some of these items. In the meantime, I will point you to the (not quite finished) new documentation site for Ceedling. Styling and layout are in a bit of flux right now. It begins with version 1.1.0 which is not released yet, but some of the issues you have raised are addressed quite comprehensively in the documentation.

• Getting Started
• Configuration (YAML)
• Testing Guide
• FFF plugin

Please feel free to offer feedback on the new documentation site that supersedes the previous monolithic user manual. You probably represent exactly the sort of user of the documentation we want to support.