Test Results Not Recorded

[applebyter]

There appears to be a problem with running the tests or recording the results of the test run. The test results are not being stored in the JSON string. Instead an IOException is stored in the string. An example string is provided below.

{
   "Assignment":"Assignment 2",
   "Current Grade":"Needs Grading",
   "Submission Field":"There is no student submission text data for this assignment.",
   "test":{"error":"IOException occurred"},
   "Comments":"There are no student comments for this assignment.",
   "root":".",
   "Date Submitted":"Friday, 20 September 2019 6:23:46 PM AEST",
   "files":{"a2.py":"Assignment 2_s344713_attempt_2019-09-20-18-23-46_a2.py"},
   "Name":"Mark Bayley (s344713)",
   "sid":"s344713",
   "timestamp":1576212692
}

[BraeWebb]

Going to hit you with the classic "it works on my machine".

I've added some lines to output some extra information when this situation arises. The situation is that something has gone wrong trying to spawn a child process for the test runner. It will now print out which of the child processes the issue is occurring in (json mode or not). It will also dump the stack trace for what actually when wrong. If you want to post that output I can fix it up.

I've also made a test for an automated system test of this particular assignment using some anonymised student submissions. So running gradle test -i will provide you with similar information far quicker than running the whole thing.

[applebyter]

Here is the output from running gradle test -i.
Chalkbox_Test_Results.zip

Note: I have pushed a fix to FieldAssignmentTest.java so it is platform independent.

[applebyter]

Running Chalkbox on the assignment two BB zip file still results in the same behaviour as described in the original issue, with the same JSON output.

[BraeWebb]

Thanks Richard, #26 should have fixed all the issues in that test output (if Paths.get works the way I think it does)

Most of the test failures appear to be from platform-specific issues. I've added a workflow to automatically test on a windows platform now.

If you want to try and run the tests on master again it should now give us some clues as to the original issue.

[BraeWebb]

When running it outside of the test environment on the original blackboard zips you should now get some output to stderr. Likely starting with an error message saying "Error occurred trying to spawn the test runner process".

The error described must be occurred because of an IOException when spawning the python test process for which I've added some debugging output now.

[applebyter]

When running chalkbox from the command line on the a2 example files I now get the message:
Unable to find file: test_a2.py

The file test_a2.py is in the directory .\ass2\queens
The test.box file contains the lines:
runner=test_a2.py
included=ass2\queens

I've tried replacing the file separator for the path ass2\queens to be either \ or /. I get the same result with both file separators.

[applebyter]

When running gradle test -i I get the exceptions shown in this text file:
gradle_output.txt

and I get the following output from JUnit, saying all tests pass:
Chalkbox_Test_Results.zip