AutoGrader

Note: AutoGrader only works on Mac OS X computers at this time.
Below is an early version of C++ source code. I was planning to clean it up, but because of the issues at the Graduate Student 90 Second Thesis Competition, I wanted to be as transparent as possible. The code is exactly what was run for that competition on February 21, 2013. AutoGrader Template Source AutoGrader Source

Installation

1. Download the AutoGrader files

AutoGrader Files
Unzip the files.

2. Install ImageMagick and GhostScript

Download and install the ImageMagick and GhostScript applications from the first and third links on this page: http://cactuslab.com/imagemagick/

3. Install OpenCV

(Unfortunately, this step is long, although straightforward.)
  1. Follow the three steps (copied below) to install MacPorts: http://www.macports.org/install.php
    1. Install Xcode (free) from the Mac OS X App Store
    2. Open Terminal and type in "sudo xcodebuild -license
    3. Install MacPorts.
  2. Open Terminal, and type in "sudo port install opencv"

Preparation

1. Create the Scoring Sheet

Use the included Word template (or any other method) to generate your own scoring sheets. Scoring sheets should consist of a row of empty bubbles for each individual's score, a comment box for each individual, and red circle markers for computerized image alignment. Take care to follow the following conditions regarding placement.

Conditions:

  1. Each row must have ten bubbles.
  2. Bubbles must be vertically aligned on each page.
  3. Comment boxes (optional) should be to the right of the bubbles.
  4. Each page needs a solid red circle in the upper-right corner and in the lower-right corner.
  5. Each page needs solid red circles in the lower-left corner indicating the page number. These red circles should be horizontally aligned to the red circle in the lower-right corner
    (I.e., One red circle in the lower-left corner of Page 1, two red circles on Page 2, etc.)

Example Page 1 | Example Page 2

2. Generate the Template

This step will annotate the scoring sheet so that the computer understands where the bubbles and comment boxes are located.

Convert your scoring sheet into .png files.
For example, on a Mac, you could use Preview to do this:

  • "Print to PDF"
  • Open the PDF in the Preview application
  • Export each page as a .png file.
  • Name then Template1.png, Template2.png, etc.

Mark bubbles on each page.
Do the following for each page .png file.

  1. Open the page file using Photoshop, GIMP, or another image editor.
  2. Use the editor's fill bucket to mark the first row of bubbles blue.
  3. Mark the first column of bubbles blue.
  4. Use the editor's pencil tool to place a green mark in all corners of each comment box.
  5. Save the .png file.

Example Annotation

3. Run the Template Generator

To pre-process the templates, you must run the template generator.
  1. Create a new folder, called "Template".
  2. Put your Template#.png files in them.
  3. Right click on the AutoGraderTemplateGen and select "Open". Allow it to run.
  4. A folder chooser will open. Choose the Template folder you created.
  5. It should generate .txt files in the Template folder. These are your preparation files.

4. Provide a Name List

Use TextEdit or another plain-text editor to create a single list of names, one name per line. These should correspond with the order in the scoring sheets, although the text need not match precisely. Save the file as "names.txt" and place it in the Template folder.

5. Prepare AutoGrader

  1. In addition to the Template folder from Step 3, create two more folders, one for Output and one for Input.

Judging

On the day of the event, you will want to instruct the judges to do the following:
  1. Use BLACK ink, BLUE ink, or pencil.
  2. Make sure marks go through the middle of the circles. Circles do not have to be fully bubbled, SAT-style, but checkmarks, X's, etc. need to pass through the eyes of the appropriate bubbles.
  3. Remind judges that comment boxes will be aggregated and fed to the participants after the event.

Scoring

After you have collected the judging sheets, you should scan them with a rapid office scanner and run the scoring software. You will want to test the scanner settings beforehand to make sure the resolution is high enough that the software works robustly, yet low enough that the scanning occurs expediently. Also make sure to scan in color.

Scoring steps:
  1. Scan the judge forms to PDF.
  2. Copy the PDF to your Input folder and rename it input.pdf (all lowercase)
  3. Run the AutoGrader application.
  4. As prompted, select the Template folder, the Input folder, and the Output folder.
This should generate two .txt files in the Output folder.
  • tally.txt: The numerical scores in sheet order.
  • sorted.txt: The names of all of the participants (with their scores), ranked in scoring order.

To accelerate fancy computations, such as declaring "Best-in-Department" or other awards, you may wish to create an Excel spreadsheet with formulas set for score computation. Then you can complete the calculations by simply copying the contents of tally.txt into the appropriate column.

AutoGrader will also generate one raw_out .csv file for each page indicating each judge's score for each participant. Each row is a unique judge; each column is a participant, in sheet order.