Submitting some exercises results in Success no matter the code

[Rune-Christensen]

I tried the Hello World and Collatz Conjecture exercises.
And both gives me succesful runs, without making the code that is needed for the tests to pass.
I then tried the Yacht exercise, and found that it works correctly.

So something is wrong in the setup of some of the exercises.

[0xE282B0]

Did you use a STOP RUN.statement?

[Rune-Christensen]

Hi @0xE282B0
Of course I do.
I know in simple programs, you do not necessarily need it, but when working with more complex programs, you would need it.
Also, the framework should be able to distinguish between, no test results returned, and succes.
I can also confirm, that goback and exit section also doesn't work with the framework as is.
Regards,
Rune

[0xE282B0]

That's a known issue with cobol-check, the test returns with zero return code even if not all tests are executed.

[Rune-Christensen]

But cobol-check also returns with zero return code, if some tests failed.
There must be a check after cobol-check runs, to determine if all tests passed or failed.

[0xE282B0]

Right, cobol-check itself always returns with zero RC. There is a potential enhancement issue openmainframeproject/cobol-check#224.
As workaround, cobol-check only builds the test and we run it manually https://github.com/exercism/cobol/blob/main/exercises/practice/hello-world/test.sh#L16-L20.

There is no additional logic layer.

[Rune-Christensen]

Ok, then I do not understand how you determine succes of the testsuite.
Just stating that it should not be possible to pass a test, by just adding the stop run statement to the program.

[0xE282B0]

When a test fails the test program returns non-zero and the website shows that the exercise was not successful. But when you terminate the program during the test execution with a clean exit you pass the test.

Do you have any ideas on how to determine that all tests have been executed?

[Rune-Christensen]

I have a couple of ideas.
The produced Cobol-check program could at runtime return 1 for a run without fails, and 4 for a run with fails. Then a zero return code would indicate that something is wrong.
You could look at the second to last line of the output, and determine if the line contains the string:
0 FAILED
If you use Cobol-check to run the test, it can examine the output, and return the result in other formats, and I believe that would fail, if the result is malformed. You could also easily parse the XML returned in those formats, to determine if there are failed tests.

[0xE282B0]

A return code other than zero should always indicate an error.

I wasn't aware that cobol-check can output the results as XML. I tried to set test.results.format = xml and ran the tests with cobol-check but the testResults.xml is an empty file.
@Rune-Christensen do you have an example of how to get the results as XML output?

[Rune-Christensen]

@0xE282B0 if you want to get the result as xml, you need to use cobol-check for compiling and running the test.
XML and HTML outputs are generated through postprocessing of the text output that cobol writes.

Perhaps Cobol check should be changed, so the generated cobol begins with setting return code 8. When the run is complete, the generated cobol should then set rc 4 if there are failed tests, and zero if no tests failed.
In that way, if the program exits before the run is complete, a return code of 8 is in effect.