Name: Helicone
Uploaded: 2024-12-09T09:04:12.267Z
Duration: 2 min 33 s
Description: An open-source platform providing observability for developers working with Large Language Models (LLMs), offering tools for logging, monitoring, and debugging.

helicone

An open-source platform providing observability for developers working with Large Language Models (LLMs), offering tools for logging, monitoring, and debugging.

Helicone

Hey I'm Cole co-founder of Helicone. I'm extremely excited to share with you an open-source llm observability platform built for every developer. As a developer myself who has built many generative AI applications, I felt the challenges of trying to monitor, debug, and improve my application with existing solutions. I felt the pains of trying to trace an AI agent throughout its entire workflow, trying to figure out exactly where it went wrong or manage, improve, and iterate on my prompts without accidentally taking down production. 

This is why we wanted to build helicone. We wanted to build the solution that we wanted to use ourselves, something that just worked out of the box and was easy to get started with. Because of this, we have such a strong

Focus on the developer experience. In just a single line of code you can get fully integrated with helicone and all features are fully configurable via headers. There's no SDK required. After launching 18 months ago we have thousands of startups using us logging a total of 1.5 billion llm requests. That's equal to 16 terab of data all publicly available anonymized at hon. 

Aopen Stats great. Now let's hop into a tour of Helicone. You instantly get a rich dashboard just like one of our power users KPTI. Within this dashboard, you get all of the advanced metrics that you need such as what are your cost over time, how many unique users are using your application, what are your latency quantiles, and then also how many security threats were detected in your application. 

We support billions of requests, all queriable within milliseconds thanks to ClickHouse. Now let's start from scratch. Imagine you're building an AI powered course generator. I'm going to show you how to use Helicone to monitor, improve, and debug our application. First, I'm going to enable our built-in demo. This is also available to all users at Hon. 

A/d demo I'm going to then click on our course generator app and begin gener generating a new course on hon.

Best practices as the course is generating. I can click the refresh button on the dashboard and view all the new requests and metrics that are flowing in. The course just finished generating. Let's head over to our request page to see all of the requests at a quick glance. 

Diving into a specific request shows you additional metadata about that request. You can see the token count, the latency, or scroll down to view the pure messages sent and received from the AI. Scrolling back up, I'm going to add this request to a data set that I can eventually use to egress to one of our fine-tuning partners. 

Custom properties are used for segmentation by any key value pair. This is used for very complex filtering. Scores can be manually added to or.

Automatically through one of our evaluation partners. The request view is great and gives me a lot to work with when trying to debug my course generator app. The problem is that many use cases, including the course generator, includes multiple requests in a single session. This is why we built the sessions feature. 

Let's head over there now. Clicking on the latest generated course, I can view all the requests from start to finish. You can see the overview, the section titles, the content, and quizzes for each section. This is extremely important for most use cases, especially AI agents reviewing this session. 

I don't like the content I generated for setting up your environment. It wasn't specific enough. Let's jump into the playground and change that. 

The playground is meant to quickly test and iterate on your prompts. For example, I'm going to tell the AI to be more specific with its instructions. I'm going to select my model of choice, increase the max tokens, and then rerun the requests. The response looks a lot better now. 

I'm ready to regression test this new prompt change with previously ran production data. To do this, I'm going to copy the new change and head over to the prompt section. The Prompt that inputs tab shows different prompt versions as well as all the production data run on that prompt version. Here you can see all the different variables for this prompt. Clicking on a specific input will input those variables and show you the response. 

Going to select the base version prompt. I'm going to add my change we copied earlier. You can select the data set you created already or you can generate a random data set on the production data we saw earlier. Select your desired model and specify your provider key and then continue. Lastly, I will confirm back on the prompt page the experiment is currently running.

Taking a look at an experiment that's already run, we can see an overview of all the changes between the original prompt and the experiment prompt. Text quality is an example of an evaluation that you can run on the original and experiment prompt and then you can compare the differences. Scrolling down, I can compare the responses for each input set that we defined in our data set.

If you're happy with the results from your experiment now it's time to add this change to your prompt which lives in your code base. Helicone will automatically detect and increment the prompt version within our platform. This means you in full control of your prompts. If our database is down your application will still work. 

That concludes the tour of hone. There are many other features to explore and some new amazing features coming very soon. I appreciate all the support on our product hunt launch and I'm excited for you to join some of the amazing companies building on our platform such as QA wolf, file mint, lefi, and many more. 

Note, that there may be slight differences in text and video. Watch video for the original content.

Helicone Video