Installing and configuring Judge

Judge system reads user submissions and test submissions and grades. It can be run with different configurations (called "environments") depending on the situations, e.g., during the exam, the judge probably grades with example test cases but while grading it grades with different sets of test cases. This multiple configurations can be done with multiple "environments" where configured so that the judge finds different sets of test data.

Basic directory structure

/[judge-root]
  /ev                     (containing grading information for each problem)
    /problem_1
     ...
    /problem_n
    /test_request         (containing TEST interface template for each problem)
      /problem_1
      ...
      /problem_n
  /result                 (containing grading results)
    /user_1
      /problem_1
        /submission_1     
    ...
    /user_n
  /scripts                (where all script are)
    /config               (where all config files are)
    /lib         
    /std-script           (grading scripts)
    /templates            (used for importing scripts)
    /test
  /log

Judge environments

Currently there are 3 environments: exam, grading, and test. Only the first two are relevant to normal usage. The test environment is only used when the system runs its unit test scripts.

The main different between the exam environment and grading environment, other than having different locations for test cases, is on how the outputs from the grading are shown to the user. In the exam environment, the system is configured so that it only reports 'passed' or 'failed', but in grading environment, each result for each test case is shown.

How to install and use

Easy as it is: check out, edit config files, put in test data, and go!

Edit config files

Config files are in (judge-home)/scripts/config. In that, you will find sample config files (under *.SAMPLE).

• First you have to copy and edit environment.rb.
  • RAILS_ROOT --- The judge accesses submissions through Rails ActiveRecord; therefore, it has to run Rails' environment. You should set RAILS_ROOT to point to where the root of Rails application for the web interface is. (There is a drawback for this design: you have to install and configure the web interface even when you just want to run the judge system.)
  • GRADER_ROOT --- This is the directory where the scripts are. It should be (judge-home)/scripts/config. (Notes: This should actually read JUDGE_SCRIPT_ROOT, will fix it later ---Jittat 17:35, 16 มีนาคม 2008 (ICT))

• For each environment, you'll have to edit its configuration. The configuration file for environment (ENV) is env_(ENV).rb. Most configuration should work as it is, (except that current both grading environment and exam environment are configured to share the same ev directory). You configure the system by modifying Ruby commands running inside a Grader::Initializer.run do |config| ... end block. For each configuration parameter, you set the attribute of the config variable.
  • Basic attributes
    • config.problems_dir --- This is where test data are. Usually it is (judge-home)/ev, but you may want to configure this differently for exam and grading environments.
    • config.user_result_dir --- This is where the grading results are stored. Again, as in problem_dir, you may want to set it differently for different environments.
  • Other attributes (shall be documented later --- Jittat 18:03, 16 มีนาคม 2008 (ICT))
    • Locations
      • config.problems_dir
      • config.user_result_dir
      • config.test_request_input_base_dir
      • config.test_request_output_base_dir
      • config.test_request_problem_templates_dir
    • Logging and reporting status of the judge
      • config.talkative
      • config.logging
      • config.log_dir
      • config.report_grader
    • Reporting result
      • config.comment_report_style

Test data

The judge system keeps grading information for each problem under a directory with the same name inside config.problem_dir (usually (judge-home)/ev or (judge-home)/ev-exam). This article documents how each directory would look like.

In normal usage where every test case has the same constraint and each corresponds to a unique test run, you can use a script to import the problem test date in a simpler format.