Feature request: customize report on timeout

[wilbowma]

Currently, the default report doesn't give the student access to anything when their container times out. The tests for my course still output a lot of useful information before timing out, and I'd like to be able to give that to students.

[andrewstec]

@wilbowma

There may be room for this feature, but it would also need to be refined a bit further with your expectations. How do you envision this feature? Do students get access to just the leftover files, or do they get custom reports built with the data once a timeout threshold is triggered? The former option should already be possible. Were you having any issues with students accessing artifacts from prior grading runs?

I have a potential workaround for you. You could ensure that your grading logic times-out internally if a grading-run is taking too long. For example, if a test takes longer than 2000ms, then abort. If a test-suite takes longer than 1 minute to run, then create a result record with a custom timeout message to supplement the container artifacts. This timeout scenario would not be the container timing out, but it would be the internal grading logic timing out. This may not work with infinite loops, however.

Any additional thoughts would be welcome.

[wilbowma]

I was imagining a custom built report for timeout, but I could live with just giving them the the same report and the log file I'm normally generating from their test suite.

Your proposed work-around makes sense. I think the Racket test harness supports sandboxing tests and timing out.