Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
b234051
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
c0ffaff
Revert previous docs update
promptless[bot] Dec 20, 2024
7d7247a
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
f188d52
Revert previous docs update
promptless[bot] Dec 20, 2024
d7047b0
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
6f617ea
Revert previous docs update
promptless[bot] Dec 20, 2024
5bfe971
Revert previous docs update
promptless[bot] Dec 20, 2024
5c2ad5c
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
69fbc94
Revert previous docs update
promptless[bot] Dec 20, 2024
8d95f72
Revert previous docs update
promptless[bot] Dec 20, 2024
997bb16
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
af86b16
Revert previous docs update
promptless[bot] Dec 20, 2024
64950ff
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
d4d73c7
Revert previous docs update
promptless[bot] Dec 20, 2024
5bb2ef4
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
0846133
Revert previous docs update
promptless[bot] Dec 20, 2024
f3371f9
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
e6a7afc
Revert previous docs update
promptless[bot] Dec 20, 2024
6cb12ee
Docs update (c2c1439)
promptless[bot] Dec 20, 2024
2e54b78
Revert previous docs update
promptless[bot] Dec 21, 2024
80dfe3e
Docs update (c2c1439)
promptless[bot] Dec 21, 2024
71b5484
Revert previous docs update
promptless[bot] Dec 21, 2024
dff36c5
Revert previous docs update
promptless[bot] Dec 21, 2024
bbf5b52
Docs update (c2c1439)
promptless[bot] Dec 21, 2024
fbaedcc
Revert previous docs update
promptless[bot] Dec 21, 2024
e22fe0f
Docs update (c2c1439)
promptless[bot] Dec 21, 2024
12d408e
Revert previous docs update
promptless[bot] Dec 21, 2024
1a62d26
Docs update (c2c1439)
promptless[bot] Dec 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/features/experiments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ import QuestionsSection from "/snippets/questions-section.mdx";

<Info>
**Who can use this feature**: Anyone on Pro or Enterprise
[plan](https://www.helicone.ai/pricing), this feature will be charged at cost.
[plan](https://www.helicone.ai/pricing), this feature will be charged at cost. Experiments supports multiple LLM providers, including OpenAI, Google, Mistralai, X-AI, and Meta-Llama, enabling a broader range of AI capabilities.
</Info>

## What are Experiments?

Helicone's Experiments is designed to help you tune your LLM prompt, test it on production data, and verify your iterations with quantifiable data. In Helicone, you can define and run LLM-as-a-judge or custom evaluators, then compare the results between prompts side-by-side to determine the best one for production.
Helicone's Experiments is designed to help you tune your LLM prompt, test it on production data, and verify your iterations with quantifiable data. This feature also supports a wide range of LLM providers, including Google, Mistralai, X-AI, and Meta-Llama. In Helicone, you can define and run LLM-as-a-judge or custom evaluators, then compare the results between prompts side-by-side to determine the best one for production.

<Frame>
<img
Expand All @@ -28,6 +28,7 @@ Helicone's Experiments is designed to help you tune your LLM prompt, test it on
- **Prevent regression:** Prompt engineering is iterative. Engineers want to prevent regression with each prompt change.
- **Model update:** LLMs are sensitive to prompt changes. As new models with advanced capability come out, prompts that worked previously can become less effective.
- **Quick feedback loop:** Instead of waiting for user feedback, get feedback immediately and adjust before pushing to production.
- **Model variety:** The experiment feature supports a wide range of models from multiple providers, including Google, Mistralai, X-AI, and Meta-Llama. This ensures coverage of a wide range of AI capabilities and more robust prompt optimization.

## What is the prompt iteration workflow?

Expand All @@ -41,7 +42,7 @@ Helicone's Experiments is designed to help you tune your LLM prompt, test it on

### 1. Create a new experiment

Head to the `Experiment` tab. If you have a prompt with Helicone, click on `start from a prompt`, then choose the desired prompt version. Otherwise, click on `start from scratch`.
Head to the `Experiment` tab. If you have a prompt with Helicone, click on `start from a prompt`, then choose the desired prompt version (including new models from Google, Mistralai, X-AI, and Meta-Llama). Otherwise, click on `start from scratch`.

<Frame>
<img
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/integration-method/custom.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Custom Model Integration"
sidebarTitle: "Custom Model"
description: "Integrate any custom LLM, including open-source models like Llama and GPT-Neo, with Helicone. Step-by-step guide for both NodeJS and Curl implementations to connect your proprietary or open-source models."
description: "Integrate any custom LLM, including open-source or third-party models like Llama, GPT-Neo, Google PaLM, Mistralai, X-AI, or Meta-Llama, with Helicone. Step-by-step guide for both NodeJS and Curl implementations to connect your proprietary or open-source models."
"twitter:title": "Custom Model Integration - Helicone OSS LLM Observability"
---

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/integration-method/hyperbolic.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ icon: "cloud"
iconType: "solid"
---

You can seamlessly integrate Helicone with the OpenAI compatible models that are deployed on Hyperbolic.
You can seamlessly integrate Helicone with the OpenAI compatible models that are deployed on Hyperbolic. In addition to the example below using a Meta-Llama model, you can also specify other open-source LLMs from Google, Mistralai, or X-AI.

The integration process closely mirrors the [proxy approach](/integrations/openai/javascript). The only distinction lies in the modification of the base_url to point to the dedicated Hyperbolic endpoint `https://hyperbolic.helicone.ai/v1`.

Expand Down
3 changes: 3 additions & 0 deletions docs/getting-started/integration-method/openllmetry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,9 @@ response = client.chat.completions.create(
- [x] Cohere
- [x] Bedrock
- [x] Google AI Platform
- [x] Mistralai
- [x] X-AI
- [x] Meta-Llama

# Other Integrations

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/integration-method/vercelai.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Vercel AI SDK Integration"
sidebarTitle: "Vercel AI"
description: "Use Vercel AI SDK to integrate with Helicone to log your OpenAI usage."
description: "Use Vercel AI SDK to integrate with Helicone to log your LLM usage (OpenAI, Anthropic, Google, Mistralai, X-AI, Meta-Llama, etc.)."
"twitter:title": "Vercel AI SDK Integration - Helicone OSS LLM Observability"
icon: "triangle"
iconType: "solid"
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cookbooks/debugging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ We are currently developing dedicated error filters to further enhance your debu

# Debugging Prompts with Playground

<Info>Currently, only ChatGPT is supported</Info>
<Info>The Playground supports multiple LLM providers, including ChatGPT, Google, Mistralai, X-AI, and Meta-Llama.</Info>

Helicone's 'Playground' feature offers a platform for debugging your 'prompt'. This tool enables you to test your prompt and swiftly observe the model's output for minor adjustments within the Helicone environment. Here's a step-by-step guide on how to use it:

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cookbooks/experiments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ To start an experiment, first, go to the [Prompts](https://www.helicone.ai/promp
</Frame>
</Step>
<Step title="Configure your experiment">
Select the dataset, model and provider keys.
Select the dataset, model and provider keys. Helicone now supports a wide range of providers, including Google, Mistralai, X-AI, and Meta-Llama.
<Tip>
To run your experiment on a random dataset, click `Generate random
dataset`. We will pick up to 10 random data from your existing
Expand Down
6 changes: 3 additions & 3 deletions docs/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ The optimal method to integrate with Helicone is through our Gateway.

</Card>
<Card title="OpenRouter" href="/getting-started/integration-method/openrouter">

</Card>
<Card title="LiteLLM" href="/getting-started/integration-method/litellm">
Supports new providers from Google, Mistralai, X-AI, and Meta-Llama
</Card>
<Card title="LiteLLM" href="/getting-started/integration-method/litellm">
</Card>
</CardGroup>

Expand Down
2 changes: 1 addition & 1 deletion docs/quick-start.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ iconType: "solid"

<CardGroup cols={4}>
<Card
title="OpenAI"
title="OpenAI, Google, Mistralai, X-AI, and Meta-Llama"
href="/getting-started/providers/openai"
icon={
<svg
Expand Down