Agent

[OJ-Pulp]

🤖 Task: Agent Integration for Analyst Question Answering via LLM

🎯 Objective

Develop an Agent system that leverages either a local or API-accessed Large Language Model (LLM) to answer analyst-level supply chain questions. The Agent should be capable of operating over both the dense and sparse datasets, using the available graph tools to generate answers.

The system will also log the tools used, output the answers, and compare them to known correct answers to evaluate the Agent’s performance.

---

📝 Task Breakdown

1. Agent Capabilities

• Accept analyst-level questions as input (predefined or new).
• Determine which tools to use from the available set (via tool descriptions/config).
• Execute the tools against both the dense and sparse datasets.
• Produce and output answers for both datasets.

2. LLM Integration

• Support calling either:
  • A local LLM (e.g., via Hugging Face, Ollama, LM Studio)
  • A remote API (e.g., OpenAI, Anthropic)
• Use the LLM to:
  • Interpret the question
  • Plan tool usage
  • Generate explanations alongside the answer

3. Output and Evaluation

• Log:
  • The question
  • The dataset type used (dense or sparse)
  • The toolchain used (in order)
  • The intermittent steps taken
  • The final answer
  • The confidence level (if sparse)
• Compare LLM-derived answers to ground-truth answers from the dense dataset tools.
• Record accuracy, tool consistency, and divergence.

---

📁 Deliverables

• Agent integration above the manager.
• Logged outputs mentioned above.
• Summary report on LLM-agent performance across question set.