PRISM is a research-driven toolset designed to combat media polarization. It leverages Large Language Models (LLMs) to synthesize multiple news sources into comparative summaries, helping users access diverse perspectives. This repository contains the code for my thesis, featuring two interfaces (Web/Streamlit & Reddit) and a benchmarking suite for three prompting methodologies.
The core of this project compares three LLM architectures to determine which best highlights contrasting narratives in news media:
- Zero-Shot Learning (ZSL): Direct summary generation.
- Chain of Thought (CoT): Step-by-step reasoning for deep analysis.
- Multiagent Group Chat (MGC): Collaborative agents debating different viewpoints.
| File / Folder | Description |
|---|---|
autograder.py |
The LLM-as-a-Judge framework used for automated evaluation. |
compare_multichat.py |
Implementation of the Multiagent Group (MGC) approach. |
compare_openai_3.py |
Implementation of Chain of Thought (CoT) and Zero-Shot logic. |
evaluation_scores.csv |
Raw data from the LLM-based autograder and manual evaluation. |
summaries_pairs.csv |
Dataset containing the human written pairs of news summaries for comparison. |
📰_Manual_Compare.py |
Another page on the Streamlit Web App allowing for manually entering the news links to be compared |
To determine the most effective summarization technique, I utilized a dual-layer evaluation strategy:
- LLM-as-a-Judge (Autograder): A programmatic evaluation using
autograder.pyto score summaries. - Manual Evaluation: Human-in-the-loop scoring (recorded in
evaluation_scores.csv) to validate the autograder's findings and ensure nuanced qualitative analysis.
Key Finding: The Chain of Thought (CoT) method consistently outperformed other approaches in creating high-quality, balanced comparative news summaries.
- Clone the repository:
git clone https://github.com/AnanyaGrover1/prism.git
- Install dependencies:
pip install -r requirements.txt
Follow the streamlit link: prismbot.streamlit.app/
Note that you may need to add API keys (paid) for the News API and OpenAI API for the application to work as intended.