Using a Bash Script To Automate Testing of A Prolog File

Question

I am in the process of grading student assignments, and wish to automatically run queries on the students database, and print the results of those queries to a file. The issue that I am running into is that I wish to run the multiple queries, but cannot find a way to just add all the queries I wish as a command line argument.

Relevant Files

Student Submission of Prolog file:

I cannot show you this code, but the students had to solve 4 problems using Prolog, and I have access to those files.

hw4_autograde.sh NOTE: The compiler was changed to SWI-prolog, not gprolog.

testcases.txt: These are the test cases I wish to run. They consist of various calls to the 4 rules/'functions' the students solved.

I am able to create and append to the output files, but I need to get my bash script working first, then can add the required code to do that. Any assistance or a point in the right direction would be most helpful.

I have tried to scrutinize this post, but am struggling to get this working:

https://lists.gnu.org/archive/html/users-prolog/2013-12/msg00004.html

Thank you.

EDIT:

The solution described by Paulo Moura in the comments solves this problem, however, in using the logtalk tester, I am coming across issues of using multiple prolog files in subdirectories from within submissions. I used to get output with the student files working, but now all the tests say crashed, when I used to get passed or failed: Here is a screen shot of what I am currently getting when running logtalk_tester -p swi -t 60:

Below is a screen shot of what is within a student's subdirectory:

Here is tests.lgt:

:- object(tests,
    extends(lgtunit)).

    %Tests for shuffle
    %--------------------------------------------------------
    test(shuff_working) :- 
        {shuffle([a,b,c],[d,e,f],[a,d,b,e,c,f])}.

    test(shuff_wrong, false) :-
        {shuffle([a,b,d],[d,e,f],[a,d,b,e,c,f])}.
    
    test(shuff_wrong_2, false) :-
        {shuffle([d,e,f],[a,b,d],[a,d,b,e,c,f])}.
    
    test(shuff_wrong_3, false) :-
        {shuffle([a,b,c,e],[d,e,f],[a,d,b,e,c,f])}.

    test(shuff_var) :-
        {shuffle(X,Y,[1,2,3,4,5,6])}.

    %--------------------------------------------------------
    %Tests for double

    test(doub_working) :-
        {double([a,b,c],[a,a,b,b,c,c])}.

    test(doub_wrong, false) :-
        {double([a,b,c],[a,b,c,a,b,c])}.

    test(doub_var_1) :-
        {double(X,[a,a,b,b,c,c])}.

    test(doub_var_2) :-
        {double([a,b,c],X)}.


    %--------------------------------------------------------

    %Part 2:  Sudoku Solver
    test(sudoku, false) :- 
        {test0}.
    
:- end_object.

Here is tester.lgt

:- initialization((
    % minimize output to the essential
    set_logtalk_flag(report, warnings),
    % load the student submissions
    logtalk_load(['hw4.pl','sudoku.pl']),
    % load the testing tool
    logtalk_load(lgtunit(loader)),
    % load the tests and run them
    logtalk_load(tests, [hook(lgtunit)]),
    tests::run
)).

user_634629 · Answer

You can use Logtalk's testing automation to help performing the grading. First, define the queries as tests. As an example, assume that the students are expected to define foo/1 and bar/2 predicates then when called should succeed with the bindings foo(42) and bar(a,d). We can then define a tests.lgt file with those two tests:

:- object(tests,
    extends(lgtunit)).

    test(foo, true(N == 42)) :-
        % call foo/1 in "user"
        {foo(N)}.

    test(bar, true(X-Y == a-d)) :-
        % call bar/2 in "user"
        {bar(X,Y)}.

:- end_object.

Assume the students submit their code in foo.pl and bar.pl files, saved in directories named after the student id. For example:

submissions
  id1
    foo.pl
    bar.pl
  id2
    foo.pl
    bar.pl

You will also need a tester.lgt driver file:

:- initialization((
    % minimize output to the essential
    set_logtalk_flag(report, warnings),
    % load the student submissions
    logtalk_load(['foo.pl', 'bar.pl']),
    % load the testing tool
    logtalk_load(lgtunit(loader)),
    % load the tests and run them
    logtalk_load(tests, [hook(lgtunit)]),
    tests::run
)).

After copying the tests.lgt and tester.lgt files into each of the student directories, run the logtalk_tester shell script from the submissions directory containing the students directories:

$ cd submissions
$ logtalk_tester -p gnu -t 60

You will get a full report of all failed tests and any broken or timed out student submissions. Post-process the report to what you want. Any failed tests will be prefixed by the student id. Also need some stats report?

$ cd submissions
$ logtalk_tester -p gnu -t 60 -f xunit
$ logtalk_allure_report
$ allure open

See the lgtunit tool documentation for details.

Note that this solution shouldn't require any changes to the student submissions. For other examples of how to test plain Prolog code, see e.g. the tests/prolog directory on the Logtalk distribution. This solution will work with all Logtalk supported backend Prolog compilers, including GNU Prolog.

Using a Bash Script To Automate Testing of A Prolog File

1 Answers