How to respond to memory pressure events for LocalLLM?

[bryan1anderson]

What Stanford Spezi module is your challenge related to?

Spezi

Description

I get them pretty regularly on iOS. I need a way to respond to them, deinit everything in cache and then load them back up when appropriate

Reproduction

                                let schema = LLMLocalSchema(
                                    model: llmLocalModel,
                                    parameters: .init(
                                        systemPrompt:
                                            "system prompt",
                                        displayEveryNTokens: 1
                                    ),
                                    samplingParameters: .init(temperature: 0.1),
                                    injectIntoContext: true
                                )

                                let session = runner(with: schema)

session.setup()

Expected behavior

The ability to clear memory to avoid getting killed

Additional context

No response

Code of Conduct

• I agree to follow this projects's Code of Conduct and Contributing Guidelines

[philippzagar]

Thanks @bryan1anderson for the great post, this is definitely a required feature of SpeziLLM in real-world use scenarios! 🚀

Formalized as a feature request in StanfordSpezi/SpeziLLM#90