Additional test source files not being included if they are nested in a different folder

[rolandsfr]

Description:
Necessary source file is not being linked (and compiled for that matter) for a test file whose source file use other source file which is nested deeper in the project tree. To illustrate this, I have provided a minimal example.

Project stcture:

├── compile_commands.json
├── include
│   └── parent.h
├── project.yml
├── src
│   └── parent
│       ├── child
│       │   ├── child.c
│       │   ├── child.h
│       │   └── child.test.c
│       ├── parent.c
│       └── parent.test.c
└── tests_build
    ├── artifacts
    │   └── test
    ├── logs
    ├── test
    │   ├── cache
    │   ├── dependencies
    │   │   ├── child_runner.d
    │   │   ├── child.d
    │   │   ├── child.test_runner.d
    │   │   ├── child.test.d
    │   │   ├── parent_runner.d
    │   │   ├── parent.d
    │   │   ├── parent.test_runner.d
    │   │   ├── parent.test.d
    │   │   └── unity.d
    │   ├── mocks
    │   │   ├── child
    │   │   ├── child.test
    │   │   ├── parent
    │   │   └── parent.test
    │   ├── out
    │   │   ├── child
    │   │   │   ├── child_runner.o
    │   │   │   ├── child.o
    │   │   │   ├── child.out
    │   │   │   └── unity.o
    │   │   ├── child.test
    │   │   │   ├── child.o
    │   │   │   ├── child.test_runner.o
    │   │   │   ├── child.test.o
    │   │   │   ├── child.test.out
    │   │   │   └── unity.o
    │   │   ├── parent
    │   │   │   ├── parent_runner.o
    │   │   │   ├── parent.o
    │   │   │   └── unity.o
    │   │   └── parent.test
    │   │       ├── child.o
    │   │       ├── parent.o
    │   │       ├── parent.test_runner.o
    │   │       ├── parent.test.o
    │   │       ├── parent.test.out
    │   │       └── unity.o
    │   ├── results
    │   └── runners
    │       ├── child_runner.c
    │       ├── child.test_runner.c
    │       ├── parent_runner.c
    │       └── parent.test_runner.c
    └── vendor
        ├── cmock
        │   └── src
        │       ├── cmock_internals.h
        │       ├── cmock.c
        │       ├── cmock.h
        │       └── meson.build
        └── unity
            └── src
                ├── meson.build
                ├── unity_internals.h
                ├── unity.c
                └── unity.h

child.c:

int mul(int a, int b) {
    return a * b;
}

child.test.c

#include "unity.h"
#include "child.h"

void test_mul(void)
{
    TEST_ASSERT_EQUAL_INT(6, mul(3,2));
}

child.h

int mul(int a, int b);

parent.c

T

int sum(int a, int b) {
    return a + b;
}

int square(int n) {
    return mul(n,2);
}

parent.test.c

#include "unity.h"
#include "parent.h"

TEST_SOURCE_FILE("child/child.c");

void test_sum(void)
{
    TEST_ASSERT_EQUAL_INT(5, sum(3,2));
}

include/parent.h

int sum(int a, int b);
int square(int n);

Output from running ceedling test in root:

🚧 Loaded project configuration from working directory.
 > Using: <dir>/project.yml
 > Working directory: <dir>

Ceedling set up completed in 56 milliseconds

👟 Preparing Build Paths...

👟 Collecting Test Context
--------------------------
Parsing parent.c for build directive macros, #includes, and test case names...
Parsing parent.test.c for build directive macros, #includes, and test case names...
Parsing child.c for build directive macros, #includes, and test case names...
Parsing child.test.c for build directive macros, #includes, and test case names...

👟 Ingesting Test Configurations
--------------------------------
Collecting search paths, flags, and defines parent.c...
Collecting search paths, flags, and defines parent.test.c...
Collecting search paths, flags, and defines child.c...
Collecting search paths, flags, and defines child.test.c...

👟 Determining Files to be Generated...

👟 Mocking
----------

👟 Test Runners
---------------
Generating runner for parent.c...
Generating runner for parent.test.c...
Generating runner for child.c...
Generating runner for child.test.c...

👟 Determining Artifacts to Be Built...

👟 Building Objects
-------------------
Compiling parent.c...
Compiling parent::parent_runner.c...
Compiling parent::unity.c...
Compiling parent.test.c...
Compiling parent.test::child.c...
Compiling parent.test::parent.c...
Compiling parent.test::parent.test_runner.c...
Compiling parent.test::unity.c...
Compiling child.c...
Compiling child::child_runner.c...
Compiling child::unity.c...
Compiling child.test.c...
Compiling child.test::child.c...
Compiling child.test::child.test_runner.c...
Compiling child.test::unity.c...

👟 Building Test Executables
----------------------------
Linking parent.out...
ℹ️ NOTICE: If the linker reports missing symbols, the following may be to blame:
  1. This test lacks #include statements corresponding to needed source files (see note below).
  2. Project file paths omit source files corresponding to #include statements in this test.
  3. Complex macros, #ifdefs, etc. have obscured correct #include statements in this test.
  4. Your project is attempting to mix C++ and C file extensions (not supported).
  5. This test does not #include needed mocks (that triggers their generation).

NOTE: A test file directs the build of a test executable with #include statemetns:
  * By convention, Ceedling assumes header filenames correspond to source filenames.
  * Which code files to compile and link are determined by #include statements.
  * An #include statement convention directs the generation of mocks from header files.

OPTIONS:
  1. Doublecheck this test's #include statements.
  2. Simplify complex macros or fully specify symbols for this test in :project ↳ :defines.
  3. If no header file corresponds to the needed source file, use the TEST_SOURCE_FILE()
     build diective macro in this test to inject a source file into the build.

See the docs on conventions, paths, preprocessing, compilation symbols, and build directive macros.

🧨 EXCEPTION: 'Default Test Linker' (gcc) terminated with exit code [1] and output >>
Undefined symbols for architecture arm64:
  "_mul", referenced from:
      _square in parent.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
🌱 Ceedling could not complete operations because of errors

As you can see, parent.test.c file misses source from child.c

Please notice that I do not expect solutions in form "change your project structure" because I desire flexibility and freedom in my projects.
Theoretically TEST_SOURCE_FILE("child/child.c") should have done the thing but it didnt

project.yml:

:defines:
  :common: &common_defines []
  :test:
    - *common_defines
    - TEST
    - UNITY_INCLUDE_DOUBLE # Add it here without D prefix

:paths:
  :source:
    - src/**
  :include:
    - include
    - src
  :test:
    - src/** # unit tests live next to code

:flags:
  :preprocess:
    :compile: []

:project:
  # how to use ceedling. If you're not sure, leave this as `gem` and `?`

  :ceedling_version: 1.0.1

  # optional features. If you don't need them, keep them turned off for performance
  :use_mocks: TRUE
  :use_decorators: :auto # decorate Ceedling's output text. options are :auto, :all, or :none

  # tweak the way ceedling handles automatic tasks
  :build_root: tests_build
  :test_file_prefix: ""
  :test_file_suffix: ".test.c"
  :default_tasks:
    - test:all

  :release_build: FALSE

:plugins:
  :load_paths: []
  :enabled:
    - module_generator # handy for quickly creating source, header, and test templates
    - report_tests_pretty_stdout

:extension:
  :executable: .out

---
# Configuration Options specific to CMock. See CMock docs for details
# ====
# ====
# ====

:cmock:
  :mock_path: test/support/mocks
  :stub_path: test/support/stubs
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    :enabled:
      - :ignore
      - :callback
      - stdout_pretty_tests_report
      - module_generator
  :treat_as:
    uint8: HEX8
    uint16: HEX16
    uint32: UINT32
    int8: INT8
    bool: UINT8

################################################################
# PLUGIN CONFIGURATION
################################################################

#clang Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :summaries: TRUE # Enable simple coverage summaries to console after tests
  :report_task: FALSE # Disabled dedicated report generation task (this enables automatic report generation)
  :utilities:
    - gcovr # Use gcovr to create the specified reports (default).
    #- ReportGenerator # Use ReportGenerator to create the specified reports.
  :reports: # Specify one or more reports to generate.
    # Make an HTML summary report.
    - HtmlBasic
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90

[rolandsfr]

Did more experimentation and generally put, this issue is reproducable whenever the header file (in this case child.h specifically) is not in include/ but rather anywhere else

[Letme]

I would be more guessing your test file needs mock_child.h among includes? Because if parent.c has include of child.h then you want to mock child.h in that case when you are testing the implementation of parent. But if you do not, I would still throw in the include in parent.c (could be a typo)

[matrdr]

If I understand the problem correctly, there would be two possible solutions:

The first one would be to include child.h in the test for parent.h . Even though it is not needed from a C point of view, it tells Ceedling to also link the it with child.o, as this is required for parent.o.

But, as you only call sum, which does not need mul from child.o, you could also resolve that using a linker setting. For that I add the compiler flags ffunction-sections -fdata-sections and the linker flag -Wl,--gc-sections in my project.yml. This works with GCC, you seem to use Clang for ARM, I cannot speak for that, but you could try it.

[mkarlesky]

@rolandsfr This is a curious failure. Sorry about this. One of your test variations you provided should have run exactly as you expect.

Ultimately, Ceedling needs to know what to compile and link together. The one place where it needs help from you the test author is in which source modules are in that list. There's two ways to do this:

1. The original method of doing so by convention still works. That convention is that any .h file #included in a test file that has a corresponding .c file in the list of project source files will be compiled and linked in the resulting test executable. So, in your case, you simply need both #include "child.h" and #include "parent.h". From your project structure, this should work fine.
2. In the more recent history of Ceedling, we added the TEST_SOURCE_FILE() macro. The main purpose of this is to support situations where there is no correspondence of a header file and source file. It also serves as a bit of self-documentation. It's fine to use it in place of the original convention. It will work. It's even fine to double up. No problem.

In one of your test examples, you're using TEST_SOURCE_FILE(), and it should work. I'm guessing perhaps there's a subtle bug in Ceedling's path handling.

Would you be willing to try the option as in (1) and report back? If that works, then there's almost certainly a problem in TEST_SOURCE_FILE() handling. If it doesn't work, then something is up with project path handling or some such.

p.s. Testing terminology is crazy-pants with so many differing terms and definitions. We refer to the sort of test you're setting up here as an "integration test.” You're integrating multiple source modules in your test. A "unit test" isolates a single source module. Generally speaking, unit tests are far more common than integration tests. To be clear, there's absolutely nothing wrong with what you're doing, and Ceedling should be able to support it easily. I just wanted to throw in some background for sake of future discussion.

[rolandsfr]

@mkarlesky The issue was that I assumed that ceedling would autmatically resolve dependecies in files that are used in some other file which is being tested. The reality is that ceedling expects headers for all outside references and does not automatically resolve dependencies, otherwise linking step fails.

The problem that i stumbled upon specifically in my project that, even while having quite modular files, they still depend on other functions. This project of mine is a tree-walk interpreter and for parsing I use recursive descent which in its nature if recursive for all functions that each corespond to some rule from formal grammar, thay defines a language. As a result I needed to either start testing in white-box paradigm and mocking internals or start including all underlying parsing functions if I needed theur functionality.

Where it becomes even worse is kind of integration tests. This is evident in the following file:
https://github.com/rolandsfr/immutant/blob/main/core/tests/integration/interpret.test.c
There, end-to-end evaluation expects nearly all core functionality to be present. As you can see these includes get real nasty and slows down writing tests drastically as code base continues to grow.

Perhaps this is not the right testing tool for my project or I am doing something fundamentally wrong. Coud you please check some of the tests in the given repo and tell me if there is any better way to handle testing for such project? Perhaps there is some ceedling configuration that I am not aware of that would help me.

[matrdr]

I'm afraid this is the only way to use Ceedling. In your case it is a bit unfortunate as you have a huge amount of dependencies. I usually try to just follow the linker warnings until I get my test compiling. As soon as you have the test setup compiling once, it is usually fine and you can go on writing your tests without new linker errors.

In many cases it is an advantage that Ceedling does indeed not auto resolve dependencies. This allows you to create your own fake versions of modules that are only linked when testing instead of their production counterparts.

[mkarlesky]

@rolandsfr What a great application you've got and a great stress test for something like Ceedling/Unity/CMock.

I haven't looked at your files yet to offer an informed perspective. Generally, though, the options for testing any sort of project regardless of details are pretty much the same. The art is in selecting and implementing the techniques in a layering and combination that works best for your needs. For a project like yours, you might want to implement all of the following techniques:

1. Pure state-based unit testing. This is reserved for functions at the “edges” of your project where said functions do not call other functions (or do so in limited ways within the same module). As you've discovered, attempting to test functions inside your project leads to a cascade of dependencies.
2. Interaction-based unit testing. This is where mocking or stubbing comes in. Functions under test are isolated from the functions they depend on with those mocks and stubs. Here the test cases exercise the interaction of functions. This is what prevents the cascade of dependencies.
3. Integration tests. We discussed these previously. I'd recommend against this for your entire project. Certain combinations of modules might be especially useful to test together. You may need to break dependencies with mocks just so that you can isolate the functions under test. (On occasion, mocks are solely used simply to break code dependencies on other modules rather than being incorporated fully into the test cases.)
4. End-to-end system tests. This exercises your full application.

(1)–(3) can all be done with Ceedling/Unity/CMock. You may need to do some refactoring to better support testing. Generally speaking, that refactoring almost always leads to a better project structure overall. It's a magical side effect of design for testability.

System testing (4) is another beast. There are few general-purpose system testing frameworks. I tend to start with considering Robot Framework for such needs.

Incidentally, your desire for Ceedling to discover the dependencies and such is well placed. Unfortunately, it's wildly difficult to do this in a perfectly reliable and cross-platform way.