langwatch

An LLM Optimization Studio that empowers AI teams to ship faster with quality assurance at every step, offering tools for monitoring, evaluating, and optimizing large language model applications.

LangWatch

Hi there. I'm ANUK, one of the co-founders of LangWatch. I'd love to present LangWatch today, the quality control and analytics platform for Genera VI Solutions. We allow you to stress test your llm applications to feel confident before launching, safeguarding from misbehaving users and understand how it's going to perform with actual users.

Before diving into this analytics section, you first want to understand what input and output has been going through your app in order to understand the product performance. Here you have.

Two ways to see everything that goes in and goes out. Seeing per message various important metrics such as the quality and safety of the evaluations. But also in a more general overview seeing all messages in this table card. Allowing you to click through the conversation understand what has been happening. If certain evaluators have been stopping the product. But also seeing hey this Rec has been used this context has been provided for the input of this message. 

Going back to the table overview it will give you a.

Complete overview of what the performance of your of the output has been but also allowing you to filtering on certain ra regas evaluators we've added to our system allowing you to filter on a certain score and dive deep into the messages. As an AI developer we allow you to select these messages which you just have been filtered on and add them to a certain data set. 

This data set you can use to find Junior models and make your AI product much much better. But what are these evaluators I just talked about for this?

I'm going to this section. Uh our guard was an evaluations. We've divided that in certain categories. Safy that's more about personal information detected or jailbreaking user trying to De Break Your solution policy. Uh is there being talked about a certain competitor. Uh or are the conversations of topic. Then our quality evaluators are the ones mostly used at the moment. Uh where we use our ragas. Uh open source evaluator for which is checking the faithfulness and the answer relevancy of the output of your product. Then.

Lastly you can put our llm on top of your your llm product to evaluate the actual quality of your product. But be as creative as possible to detect certain things happening in your product. Now you understand what is been happening in your product and you know how to set up these guard RS and evaluators. 

You're now going to prepare for when 10 thousands of messages are running through your app. With that we go to our analytics uh section where we provide you with Crystal Clear insights into how users are engaging with. 

Your application empowering you to fine-tune and elevate user experience to new heights. Seeing topics, um, user analytics, um, very in-depth input sentiment, thumbs up, thumbs down, where you can also add your own conversion metric in there. Talking about various topics, very in-depth topic details detection. H, but also seeing what documents have been mostly used and is that actually comparable with the top topics where has been talked about. Well, the general LLM metrics such as the cost and the number of tokens. And then we talked. 

About the evaluations you can see that per message. But now you can also see that actually a trend on what's happening and seeing hey it's the trend is going down so I should move and act on it and improve my product there. 

And then lastly you can build your own report with all the metrics available in our platform and all the data we have. You can build your own reports. This is also available per OPI so you can build it into your LLM app as well showing to your customers for example. 

And then lastly uh inter in LangWatch.

Will only cost you a couple of lines of code done within a few minutes with LangWatch. You're not just launching an AI application. You're launching a success story. I'd love to tell you a little bit more about it. Please get in contact and thanks for watching this. 

Share the AI Agent Store with colleagues.

Note, that there may be slight differences in text and video. Watch video for the original content.

LangWatch Video