Installing and configuring Judge

Judge system reads user submissions and test submissions and grades. It can be run with different configurations (called "environments") depending on the situations, e.g., during the exam, the judge probably grades with example test cases but while grading it grades with different sets of test cases. This multiple configurations can be done with multiple "environments" where configured so that the judge finds different sets of test data.

Basic directory structure

/[judge-root]
  /ev                     (containing grading information for each problem)
    /problem_1
     ...
    /problem_n
    /test_request         (containing TEST interface template for each problem)
      /problem_1
      ...
      /problem_n
  /result                 (containing grading results)
    /user_1
      /problem_1
        /submission_1     
    ...
    /user_n
  /scripts                (where all script are)
    /config               (where all config files are)
    /lib         
    /std-script           (grading scripts)
    /templates            (used for importing scripts)
    /test
  /log

Judge environments

Currently there are 3 environments: exam, grading, and test. Only the first two are relevant to normal usage. The test environment is only used when the system runs its unit test scripts.

The main different between the exam environment and grading environment, other than having different locations for test cases, is on how the outputs from the grading are shown to the user. In the exam environment, the system is configured so that it only reports 'passed' or 'failed', but in grading environment, each result for each test case is shown.

How to install and start using it

Easy as it is: check out, edit config files, put in test data, and go!

Edit config files

Config files are in (judge-home)/scripts/config. In that, you will find sample config files (under *.SAMPLE).

• First you have to copy and edit environment.rb.
  • RAILS_ROOT --- The judge accesses submissions through Rails ActiveRecord; therefore, it has to run Rails' environment. You should set RAILS_ROOT to point to where the root of Rails application for the web interface is. (There is a drawback for this design: you have to install and configure the web interface even when you just want to run the judge system.)
  • GRADER_ROOT --- This is the directory where the scripts are. It should be (judge-home)/scripts/config. (Notes: This should actually read JUDGE_SCRIPT_ROOT, will fix it later ---Jittat 17:35, 16 มีนาคม 2008 (ICT))

• For each environment, you'll have to edit its configuration. The configuration file for environment (ENV) is env_(ENV).rb. Most configuration should work as it is, (except that current both grading environment and exam environment are configured to share the same ev directory). You configure the system by modifying Ruby commands running inside a Grader::Initializer.run do |config| ... end block. For each configuration parameter, you set the attribute of the config variable.
  • Basic attributes
    • config.problems_dir --- This is where test data are. Usually it is (judge-home)/ev, but you may want to configure this differently for exam and grading environments.
    • config.user_result_dir --- This is where the grading results are stored. Again, as in problem_dir, you may want to set it differently for different environments.
  • Other attributes (shall be documented later --- Jittat 18:03, 16 มีนาคม 2008 (ICT))
    • Locations
      • config.problems_dir
      • config.user_result_dir
      • config.test_request_input_base_dir
      • config.test_request_output_base_dir
      • config.test_request_problem_templates_dir
    • Logging and reporting status of the judge
      • config.talkative
      • config.logging
      • config.log_dir
      • config.report_grader
    • Reporting result
      • config.comment_report_style

Test data

The judge system keeps grading information for each problem under a directory with the same name inside config.problem_dir (usually (judge-home)/ev or (judge-home)/ev-exam). This article documents how each directory would look like.

Importing test data

In normal usage where every test case has the same constraint and each corresponds to a unique test run, you can use a script to import the problem test data in a simpler format: each test case i has two files i.in and i.sol, all stored in some directory. For example you may have 1.in, 1.sol, 2.in, 2.sol, ..., and so on.

Then you invoke the script call import_problem in the test data directory (e.g., (judge-home)/ev). The following is from the script usage:

using: import_problem name dir num check [options]
   where: name = problem_name [
          dir = importing testcase directory
          num = number of testcases
          check = check script, which can be 
                   'integer', 'text' (for standard script), 
                   path_to_your_script, or
                   'wrapper:(path_to_your_wrapped_script)'
   options: -t time-limit (in seconds)
            -m memory-limit (in megabytes)
What it does:
  * creates a directory for a problem in the current directory,
  * copies testdata in the old format and create standard testcase config file
  * copies a check script for grading
  * creates a test_request template in the current directory + '/test_request'
For wrapped checked script see comment in templates/check_wrapper for information.

Note that you have to specify how the submission's output will be graded by specifying check script (See this for specification). Standard checkers are available: integer and text. To decouple the development of the check script and the grading system, one can develop the checker with no direct access to problem configuration and then wrap it using wrapper check script. The wrapping check script would call the real script with the following arguments

 (real-script) <lang> <test-num> <in-file> <out-file> <ans-file> <full-score>

and the wrapped script should report to the standard output in the format specified in the spec above.

Go

You can start the judging system by call grader in (judge-home)/scripts.

Its usage page (call grader --help to see) reads:

Grader.
using: (1) grader 
       (2) grader environment [mode]
       (3) grader stop [all|pids-list]
       (4) grader --help
(1) call grader with environment = 'exam', mode = 'queue'
(2) possible modes are: 'queue', 'prob', 'test_request'
(3) create stop-file to stop running grader in queue mode
(4) You are here.

Running judge. Normal usage would be to call grader specifying environment (which tells the judge where to look for test data and how to report) and mode (which tells judge how it should work).

Descriptions for modes are as follow:

• queue mode -- the judge looks for oldest ungraded task from the queue, and grades it.
• prob mode -- you have to also specify which task you want to grade, then for each user, the judge takes the most recent submission for that task and grades it.
• test_request mode -- the judge looks for the oldest ungraded test submission, and grades it.

The judge can be run from any machines that can access the database. See article running judges from many machines for details on how to have more than one judge grading in the same system.

Stoping. A judge running on queue mode waits for tasks. To stop it, you can either kill it (not safe?), or you can call grader script to stop it. If you want to stop a specific grader process, call grader stop (pids) specifying a list of pid. You can stop all grader processes sharing the same grader root directory by calling grader stop all.

When you call stop, one or more stop files are created. For stopping a specific process, a fill call stop.(pid) is created; if the judge with that pid sees the file, it removes the file and stops. If called with all option, a stop file named stop.all is created; and all judge processes, when seeing this file, stop. Note that in this case, the stop file remains in the directory and you should delete the file manually when all processes stop. (This is because it is difficult to know when to delete the file.)

How to grade

It is good that you have used it this far. Now, it is the fun part!

Here are the steps. 1. Make sure that you have stopped all other running grader on the grading environment. If you know the pid of judge, call

grader stop (pid)

or you can stop all running graders by calling

grader stop all

but you have to delete the stop file stop.all by yourself, otherwise you won't be able to start the grader.

2. Make sure that you have your test data ready. (You should!) Assume that you have it in grading environment.

3. Then for each problem, say (problem-name), call

grader grading prob (problem-name)

and iterate through your list of problems.

Have fun!