No Evidence for LLMs Being Useful in Problem Reframing

*** Project page under constrution ***

• In this project, we studied the benefits and drawbacks of using LLMs in problem reframing.
• We tested three approaches to use LLMs in problem reframing compared to a not using LLMs:
• Direct approach: Prompting an LLM directly to generate alternative problem frames.
• Free-form approach: Conversing with an LLM as needed in reframing problems.
• Structured approach: Prompting an LLM to generate the entire content throughout Kees Dorst's nine-step reframing process.
• In this repository, use can test the Direct and Structured approach (we tested the Free-form approach using ChatGPT).
• This project is published at CHI'25 (paper) (project page)

Setup

First, you need an OpenAI API key. If you do not have one, follow Developer quickstart from OpenAI to create your own.

1. Download our repository.
2. Install libraries (we tested with Python 3.10.2).

pip install -r requirement.txt

3. Create config.env in the root folder to set your OpenAI API key. In the file, add your key as shown below.

OPENAI_API_KEY=your-OpenAI-API-Key

4. All set. Follow the usage below to try our workflows.

Usage

We provide example design problems. Update the file, if you wish to use your own design problems.

Direct approach

Run direct_approach.py

Structured approach

Run structured_approach.py

Bibtext

@inproceedings{shin:2025:problemReframingLLM,
    title={No Evidence for LLMs Being Useful in Problem Reframing},
    author = {Shin, Joongi and Polyanskaya, Anna and Lucero, Andrés and Oulasvirta, Antti},
    publisher = {Association for computing Machinery},
    booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
    year={2025},
    url={https://doi.org/10.1145/3706598.3713273},
    doi={10.1145/3706598.3713273}
}