Merge branch 'main' into main

Author: PriyanJindal

README.md

@@ -117,8 +117,18 @@ prompt-learning/
 
 ### Installation
 
+Install the `prompt-learning` package via pip:
+
 ```bash
-pip install -r requirements.txt
+pip install prompt-learning
+```
+
+Or install from source for development:
+
+```bash
+git clone https://github.com/priyanjindal/prompt-learning.git
+cd prompt-learning
+pip install -e .
 ```
 
 ### Environment Setup
@@ -131,29 +141,142 @@ export OPENAI_API_KEY="your-api-key-here"
 
 ```python
 import pandas as pd
-from optimizer_sdk.prompt_learning_optimizer import PromptLearningOptimizer
+from prompt_learning import PromptLearningOptimizer
 
 # Create dataset with English feedback
 dataset = pd.DataFrame({
-    'input': ["Generate a tech company's career page"],
-    'output': ["{incorrect JSON output}"],
-    'feedback': ["The generated JSON breaks several rules: missing 'updatedAt' field, top-level key should be 'page'"]
+    'query': [
+        "I can't log in to my account anymore",
+        "My password reset email never arrived",
+        "I was charged twice for the same order",
+    ],
+    'output': [
+        "Login Issues",
+        "Password Reset",
+        "Billing Inquiry",
+    ],
+    'feedback': [
+        "correct",
+        "correct",
+        "correct",
+    ]
 })
 
+# Define your prompt with template variables
+prompt = """You are a customer support classifier.
+Classify the query into a category.
+
+Query: {query}
+
+Category:"""
+
 # Initialize optimizer
 optimizer = PromptLearningOptimizer(
-    prompt="You are an expert in JSON webpage creation. Generate: {input}",
-    model_choice="gpt-4"
+    prompt=prompt,
+    model_choice="gpt-4o"
 )
 
-# Optimize the prompt using English feedback
+# Optimize the prompt using feedback
 optimized_prompt = optimizer.optimize(
     dataset=dataset,
     output_column='output',
     feedback_columns=['feedback']
 )
+
+print(optimized_prompt)
+```
+
+### Advanced Usage
+
+#### Using Custom Evaluators
+
+You can run evaluators on your dataset before optimization:
+
+```python
+from prompt_learning import PromptLearningOptimizer
+
+optimizer = PromptLearningOptimizer(
+    prompt="Your prompt with {variables}",
+    model_choice="gpt-4o"
+)
+
+# Run evaluators first
+dataset, feedback_columns = optimizer.run_evaluators(
+    dataset=dataset,
+    evaluators=[your_custom_evaluator],
+    feedback_columns=["existing_feedback"]
+)
+
+# Then optimize
+optimized_prompt = optimizer.optimize(
+    dataset=dataset,
+    output_column='output',
+    feedback_columns=feedback_columns
+)
 ```
 
+#### Using Annotations
+
+Generate detailed annotations to guide optimization:
+
+```python
+annotations = optimizer.create_annotation(
+    prompt=prompt,
+    template_variables=["query"],
+    dataset=dataset,
+    feedback_columns=["feedback"],
+    annotator_prompts=["Analyze why the model made errors and suggest improvements."],
+    output_column="output"
+)
+
+optimized_prompt = optimizer.optimize(
+    dataset=dataset,
+    output_column='output',
+    feedback_columns=['feedback'],
+    annotations=annotations
+)
+```
+
+#### Optimizing Rulesets
+
+For coding agents or complex systems, optimize dynamic rulesets instead of the full prompt:
+
+```python
+optimized_ruleset = optimizer.optimize(
+    dataset=dataset,
+    output_column='output',
+    feedback_columns=['feedback'],
+    ruleset="- Rule 1: Always check for edge cases\n- Rule 2: Validate inputs"
+)
+```
+
+### API Reference
+
+#### `PromptLearningOptimizer`
+
+**Constructor:**
+```python
+PromptLearningOptimizer(
+    prompt: Union[PromptVersion, str, List[Dict[str, str]]],
+    model_choice: str = "gpt-4",
+    openai_api_key: Optional[str] = None,
+    meta_prompt: Optional[str] = None,
+    rules_meta_prompt: Optional[str] = None,
+)
+```
+
+- `prompt`: The prompt to optimize. Can be a string, list of messages, or Phoenix PromptVersion.
+- `model_choice`: OpenAI model to use (default: "gpt-4")
+- `openai_api_key`: API key (or set via `OPENAI_API_KEY` env var)
+- `meta_prompt`: Custom meta-prompt template (optional)
+- `rules_meta_prompt`: Custom meta-prompt for ruleset optimization (optional)
+
+**Methods:**
+
+- `optimize(dataset, output_column, feedback_columns, ...)`: Optimize the prompt using feedback data
+- `run_evaluators(dataset, evaluators, feedback_columns)`: Run evaluators on the dataset
+- `create_annotation(...)`: Generate annotations for optimization guidance
+
 ## Contributing
 
 You can contribute to the optimizer sdk itself within the optimizer_sdk notebook. You can also add notebooks, datasets, or other additional material. 

src/prompt_learning/__init__.py

@@ -0,0 +1,20 @@
+"""
+prompt-learning: A prompt optimization SDK using meta-prompt approaches.
+
+This package provides tools for optimizing LLM prompts using feedback
+and evaluation data.
+"""
+
+from .prompt_learning_optimizer import PromptLearningOptimizer
+from .annotator import Annotator
+from .meta_prompt import MetaPrompt
+from .tiktoken_splitter import TiktokenSplitter
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "PromptLearningOptimizer",
+    "Annotator",
+    "MetaPrompt",
+    "TiktokenSplitter",
+]

src/prompt_learning/annotator.py

@@ -0,0 +1,90 @@
+from typing import List, Tuple
+import pandas as pd
+import openai
+import os
+
+from .constants import END_DELIM, START_DELIM
+
+
+class Annotator:
+    def __init__(self, annotations_prompt_template: str):
+        self.annotations_prompt_template = annotations_prompt_template
+
+    def construct_content(
+        self,
+        batch_df: pd.DataFrame,
+        baseline_prompt: str,
+        template_variables: List[str],
+        feedback_columns: List[str],
+        output_column: str,
+        ground_truth_column: str = None,
+    ) -> str:
+        """
+        Generate annotations based on the evaluation results.
+        
+        Args:
+            batch_df: DataFrame containing the evaluation data
+            baseline_prompt: The original prompt that was evaluated
+            template_variables: List of template variable names
+            feedback_columns: List of feedback column names
+            output_column: Name of the output column
+            
+        Returns:
+            Formatted prompt string for annotation generation
+        """
+        content = self.annotations_prompt_template
+        content = content.replace("{baseline prompt}", baseline_prompt)
+        
+        examples = ""
+        # Iterate over the batch of data and populate the template with actual values
+        for ind, row in batch_df.iterrows():
+            row_dict = row.to_dict()
+            output_value = row_dict[output_column]
+            if output_value is not None and isinstance(output_value, str):
+                output_value = output_value.replace(START_DELIM, " ").replace(END_DELIM, " ")
+            else:
+                output_value = "None"
+            if ground_truth_column is not None:
+                ground_truth_value = row_dict[ground_truth_column]
+            else:
+                ground_truth_value = "N/A"
+            current_example = f"""\n
+                Example {str(ind)}
+
+                Input: {[row_dict[temp_var] for temp_var in template_variables]}
+
+                Output: {output_value}
+
+                Ground Truth: {row_dict.get('ground_truth', 'N/A')}
+
+                Feedback:
+            """
+
+            for feedback_column in feedback_columns:
+                feedback_value = row_dict[feedback_column]
+                if feedback_value is not None:
+                    # Cast to string to handle integers and other types
+                    feedback_value = str(feedback_value)
+                    feedback_value = feedback_value.replace(START_DELIM, " ").replace(END_DELIM, " ")
+                else:
+                    feedback_value = "None"
+                current_example += f"\n{feedback_column}: {feedback_value}"
+            examples += current_example
+            
+        content = content.replace("{examples}", examples)
+        return content 
+    
+    def generate_annotation(
+        self,
+        prompt: str,
+    ) -> str:
+        client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+        response = client.chat.completions.create(
+            model="gpt-4o",
+            messages=[
+                {"role": "user", "content": prompt},
+            ]
+        )
+        return response.choices[0].message.content
+
+        

src/prompt_learning/constants.py

@@ -0,0 +1,113 @@
+# Constants for the prompt-learning-sdk module.
+
+
+# Delimiters for template variables
+START_DELIM = "{"
+END_DELIM = "}"
+
+SUPPORTED_MODELS = [
+    "o1",
+    "o3",
+    "gpt-4o",
+    "gpt-4",
+    "gpt-3.5-turbo",
+    "gpt-3.5",
+]
+
+# Meta prompt template sections
+META_PROMPT_TEMPLATE = """
+You are an expert in prompt optimization. Given the original baseline prompt and the following associated metadata (such as model inputs, outputs, evaluation labels and explanations),
+generate a revised version of the original prompt that would likely improve results with respect to the evaluation labels.
+Your goal is to align the prompt with the feedback and evaluation criteria.
+
+BELOW IS THE ORIGINAL BASELINE PROMPT
+************* start prompt *************
+
+
+{baseline_prompt}
+************* end prompt *************
+
+BELOW ARE THE EXAMPLES USING THE ABOVE PROMPT
+************* start example data *************
+
+
+{examples}
+************* end example data *************
+
+HERE ARE SOME ANNOTATIONS THAT MAY BE HELPFUL:
+{annotations}
+
+FINAL INSTRUCTIONS
+Iterate on the original prompt (above) with a new prompt that will improve the results, based on the examples and feedback above.
+
+A common best practice in prompt optimization is to add guidelines and the most helpful few shot examples.
+
+Note: Make sure to include the variables from the original prompt, which are wrapped in either single brackets or double brackets (e.g.
+{var}). If you fail to include these variables, the LLM will not be able to access the required data.
+Do not add any single or double brackets around anything other than the variables from the original prompt. The only curly brackets that should be used are the ones that wrap the variables from the original prompt.
+Make sure to copy paste the exact return instructions from the original prompt. Do not add any brackets here. 
+
+YOUR NEW PROMPT:
+"""
+
+CODING_AGENT_META_PROMPT_TEMPLATE = """
+You are an expert in coding agent prompt optimization.  
+Your goal is to improve the dynamic ruleset that guides the coding agent.  
+
+Process:
+1. Carefully review the baseline prompt, the current dynamic ruleset, examples, and annotations.  
+2. Identify high-level issues in the baseline prompt and dynamic ruleset — focus on missing guidance, vague constraints, or areas where rules could be made more robust.  
+3. Revise the dynamic ruleset so it is stronger, more reliable, and generalizes well beyond the provided examples.  
+
+BELOW IS THE ORIGINAL BASELINE PROMPT WITH STATIC RULESET
+************* start prompt *************  
+
+{baseline_prompt}  
+************* end prompt *************  
+
+BELOW IS THE CURRENT DYNAMIC RULESET (CHANGE THESE OR ADD NEW RULES)
+************* start ruleset *************  
+
+{ruleset}  
+************* end ruleset *************  
+
+Now you will be given data examples that use the above prompt and ruleset. Each example consists of:
+- problem_statement: the problem statement
+- coding agent patch: a patch generated by the coding agent, which is supposed to fix the problem.
+- ground truth patch: a ground truth solution/patch to the problem
+- test patch: a test patch that the coding agent's output should pass, which directly addresses the issue in the problem statement
+- pass_or_fail: either "pass" or "fail" indicating whether the coding agent's code changes passed the unit tests (indicates whether the coding agent's output is correct or incorrect)
+- explanation: explanation of your reasoning: why/why not the coding agent's output is correct, why the coding agent may have taken that approach, and general improvement suggestions for the coding agent to improve its output.
+
+BELOW ARE THE EXAMPLES USING THE ABOVE PROMPT AND RULESET
+************* start example data *************  
+
+{examples}  
+************* end example data *************  
+
+FINAL INSTRUCTIONS  
+Iterate on the **dynamic ruleset only**. You may:  
+- Add new rules  
+- Edit or strengthen existing rules  
+
+Important constraints:
+- Do **not** modify the static rules in the baseline prompt.  
+- Do **not** add rules that request user input, confirmations, or follow-up questions (e.g., `ask_followup_question`). The coding agent should always act autonomously.  
+- Keep the ruleset concise and relevant — avoid unnecessary rules that don't match the general types of problems that the coding agent is likely to encounter or overly specific rules that only patch the given examples.  
+- Remember that you are writing GENERAL rules. They should not be specific to the repositories or problems that you are given. They should be general rules that would improve the overall ability of the coding agent. 
+Output format:
+- Return only the final, revised dynamic ruleset as a bullet-point list.  
+- Do not include any extra commentary, explanations, or text outside the ruleset.  
+
+New ruleset:
+"""
+
+
+# Template placeholders
+EXAMPLES_PLACEHOLDER = "{examples}"
+
+# Example formatting constants
+EXAMPLE_HEADER = "Example {index}"
+ORIGINAL_TEMPLATE_LABEL = "Original Template With Variables from the Baseline Prompt Populated:"
+OUTPUT_LABEL = "Output from the LLM using the template above:"
+FEEDBACK_LABEL = "Feedback from the evaluator using the template above and the output above:"