Feature Request: Make model timeouts for unloading from memory configurable

[chpiatt]

Summary

Allow developers to set different timeouts when loading models for when the model is purged from memory.

Motivation

Many model use cases are latency-sensitive and potentially long-running. For those tasks that have more than a 5 minute gap between requests, the only way to avoid a cold start is to implement a separate polling script which is inefficient.

Proposed Implementation

Change the timeout from a static 5 minutes to a configurable time period - maybe from 1 minute (or even 1 request) to 24 hours?
Add an option to keep the model running without an automatic purge from memory for advanced users (only manually purging the model or replacing with another model)

Technical Considerations

Should probably couple with some feedback mechanism and/or limits set to prevent users from overusing system resources
Should gracefully handle bad shutdowns

Questions for Maintainers

What is the right time range to allow for maximum flexibility?

[ericcurtin]

@chpiatt just curious we are putting in effort to make contributing to docker model-runner more contributor friendly, would you be interested in working on this?