qwen-agent

An open-source framework for developing LLM applications with instruction following, tool usage, planning, and memory capabilities.

Qwen-Agent

All right I'm going to start right out the gate saying something a little cheesy here. But by the end of this video you are definitely going to be on the same page as me. Yesterday November 11th 2024 marks one of the biggest days for AI this entire year and I know that is saying a lot trust me. But after my initial testing of what I'm about to show you now I sincerely mean it. I have not been this excited about anything in the AI space for a couple of months now except for autoDev of course. 

But yesterday a new version of my favorite open source large language model quen was released and it is quen 2.5 coder 32b. I found out that quen 2.5 coder 32b was released last night right before I was about to go to bed and I literally got up went to my office booted up my computer again downloaded the model and started testing it out with a bunch of different things including autoDev or bolt.

New fork and some of my AI agents and let me tell you it literally felt like I was in a fever dream. I had to CL myself to make sure I was actually awake because the results were that insane. I had to go look at my code and make sure it wasn't actually using 01 behind the scenes. The results with quen 2.5 coder 32b were that good. 

Now I know that you might think that I'm exaggerating, that this is too good to be true. But don't just take my word for it. Allow me to demonstrate to you how well this model works with both autod Dev and an AI agent that I created specifically to push limits of AI to test new llms like this. And the best part is I'm running Qwen 2.5 coder 32b entirely locally, so I'm paying nothing for it and you don't have to either. Super exciting stuff, so let's go ahead and dive right into it. 

All right, so before we dive into testing Qwen-Agent 2.5 coder 32b, a lot of you for these kind of videos have asked me to go over how to pull these models, hardware requirements, and then some considerations as well if you don't have the beefiest computer. So I'm going to cover that right now. First of all, my recommended way to use local large language models is with olama. So you just go to ama.com, click on this nice big download button here, and then you'll have it already ready to go, ready to pull any model in their library. 

And so I'll have a link in the description for this, but you want to go to the Qwen 2.5 coder page for AMA and they'll give you the command right here that you can just copy to pull and run Qwen 2.5 coder 32b. You can select other variations as well here, like the 7 billion parameter one that's been out for a while. And now they recently released a lot of other ones like 14b and 32b. 

So 14 billion parameters, this model I haven't tested myself yet, but this is going to be a really good option as well if you don't have the BS machine. You could run this instead if 32 billion parameters is just too large. I would highly recommend if you're going to run the 32 billion parameter model that you have at least a 3090 graphics card from Nidia. 

And so for my computer specifically, I'll pull up my task manager so you can see I have 23 90 gpus. I made an investment here to build this computer because AI is just my life, so I picked up two Ed 390 graphics cards.

Um but a 32 million parameter one is able to run on just one of my cards. So you don't need two yourself. And then if you don't have a 3090, there are a couple of options that I want to cover very quickly here. 

First of all, if you click on the drop down here for the parameter options and click on view all, you'll see a lot of different variations here that you can download. Each and every one of these for Qwen 2.5 coder. And if I scroll down to the 32 billion parameter section here, there's a concept called quantizing with local large language models where essentially you can reduce the size of the model significantly while only reducing the performance a tiny bit. 

And I don't want to dive into that. It could be a whole separate video. But you have options like a Q2 quantizing where the model is 12 GB instead of the 20 GB from the default one that old Lama has for you. You can pull this one if you want a smaller one and try to run that. 

And also you have all these options for the 14 billion parameter as well. So I'd recommend trying that out as well too if the 32 billion parameter model just blows up your machine when you try it. So give it a shot, run 14 billion as well. Just get Quin 2.5 coder on your machine because it is absolutely killer. 

So let's go ahead and test out the 32 billion parameter version. So not surprisingly, one of the first things that I wanted to test with this new model is autod Dev our Fork bolt.

New which which is one of the best open-source AI coding assistants out there. And so I've got Qwen 2.5 coder 32b loaded up with olama and autod Dev. 

Just like with any llm, it helps to start with a very simple prompt and then get iteratively more complicated. So we can build something that isn't just trivial. And so just to start out, I'm having it create a react chat interface with Tailwind where I can send in a message and get a sample message back. 

And quen 2.5 coder 32b is going to knock this out of the park. So it's running locally. It'll take a little bit to actually build this. And so what I'm going to do is pause and come back once this is complete. 

All right, so we got our very first iteration of the chat interface where we'll get a sample message back after we type in our own. So this is looking good. So now let's add in the next prompt here to really give it some requirements. 

Here, so I got some UI requirements and then some ux ones as well. And then some functional things like have an AI agent just respond with a sample message after waiting two seconds with the loading indicator. And so now we're starting to get to something that isn't just trivial. 

And it's still pretty basic, but this is the kind of thing that other local large language models will totally botch. And so I'll even show that after this with another model. But first, what I'll do here is pause and come back again once this iteration is complete. 

And boom, we now have our updated chat interface that is starting to look a lot better. Still probably some tweaks we could do here, but overall we even got a nice little loading indicator here. Got a sample message. This is all looking absolutely fantastic. 

And I can continue to prompt this to touch up the styling. Actually hook up an API endpoint like I did in another video on my channel for this interface with autod Dev. You can take this much, much further. It is all working really, really well. 

So back to the drawing board here because I want to show you how other local llms will mess up in a way that Qwen 2.5 coder 32b simply doesn't. Notd. And so I've got code llama 34b loaded in here and I'll show you that this model, it's actually a little bit bigger than the one I just used. 

And it still is going to mess up. So I'll start with the same.

Basic prompt here and it'll probably do this fine. Um but when we get to the more complicated stuff it starts to really mess up because as much as I hate to break it to you local llms in general just aren't as powerful as the big guys like claw 3.5 Sonet. And so it really does help to uh use those a lot of times. But with Qwen 2.5 coder 32b you actually can use that to build most things. 

So let me pause and come back once this is done with code llama. Okay and look at this before I even got to the more non-trivial stuff it actually messed up right away. It failed to run the npm Run Dev command because it never installed Vite. It didn't run that First Command that you'll always see Sonet and Qwen 2.5 and other models run where it does that mpm install. So it it never installed anything. There's no preview here. A very basic example failed with code llama 34b. Again this is even a little bit bigger than Quin 2.5 Cod or 32b and it totally failed. 

So I hope this shows you how exciting specifically this new model is with Quin 2.5 coder 32b. Uh just amazing how much this can do compared to other llms that are supposed to be equal in size and power. The sponsor of today's video is novita AI. It is an all-encompassing AI Cloud platform that makes it really easy to get up and running with open source large language models. They have three primary products on their platform including the llm API that makes it really really easy to use a bunch of large language models with a simple API. 

They have server list to make it really easy to deploy and not to manage infrastructure for your AI applications. And they also have GPU instances which are really really configurable. You can pick a type and then even select the number of gpus all the way up to eight for that specific instance type and graphics card which I haven't seen on another platform before which is phenomenal. You also have a bunch of templates here like AMA with open web UI and you can create your own template really easily with any container image. You could even for example use our bolt.

New Fork right out the gate in an ovita AI instance which is just phenomenal. So I've actually created an instance myself with olama and open web UI with two 390 gpus. Really easy to get up and running in just minutes. You have a nice terminal to interact with your instance in the browser. 

And then also I've got open web UI open up here um using uh code llama 13B and I can say create the snake game for me in Python and it is just blazing fast. So just awesome these instances absolutely kick butt. 

And then for the llm API there are a lot of really unique models that they have available here. So a lot of standard ones like llama 3.1 405b but also a lot of unique models that are meant more for like creative writing tasks for example. So they have a really unique set available in their API. Very competitive pricing as well for their models. 

And then the last thing is it is open AI compatible. So you can just override the base URL here set it up just like you're talking to GPT in your applications. So it's really easy to drop and replace this to use novita in place of something like GPT if you want to use local llm. 

So I'll have a link in the description of this video to novita AI. I would highly recommend it checking it out. It is a platform that I've used in the past and will continue to use for a bunch of my infrastructure for large language models. 

So the next thing that I wanted to test Qwen 2.5 coder 32b with is with this agent that I built with Lang chain and Lang graph that specifically is meant to push a limits of large language models to test new ones like what I'm doing right here. 

And so there's actually another video on my channel where I show how to build this agent and I do some more testing with other llm. So I'll have a link to that in the description of this video along with the GitHub repost you can download this agent and run it yourself. 

So I'm not going to go into the details of this because I covered that in my other video but it's a streamlet front end and then I'm using L chain and L graph for this agent hooking it into a bunch of different tools so that it can interact with a. 

So I've got a ton of different tools for a SAA for task management and then Google Drive for our file downloading and uploading files to Google Drive and searching through.

Folders and then for rag I am integrating with chroma for a local Vector database so that we can um basically take the things from Google Drive add it into a knowledge base for the llm to answer questions. 

And so all these tools combin together to create an agent that is very much not trivial. There's a lot of different tools that the large language models get to choose from based on a user's question. 

And so local llms often mess up. And actually one really good example is that I used llama 3.1 70b from Gro in the last video on my Channel with this and it completely messed up on all these things that I'm about to show you. 

Quen 2.5 coder 32b it definitely does well with. So let's go ahead and dive into this testing right here. 

So I've got my asaa open right here so we can see it creating tasks and stuff. I've got my front end interface here with streamlet and then I have my terminal open as well where we'll be able to see all the decisions that AI agent decides to make for Tool calling kind of like what we have already right here from testing that I was doing earlier. 

So I'm going to start with a very basic prompt here. What projects do I have in AA? I don't even have to type it right. I'm just going to let it go here. 

So we'll go into the terminal here and it's going to yeah call the get as projects tool gets the Json result from the asaa API and then it tells me our projects are YouTube, Fitness, business, personal, and coding. 

And this is exactly right based on what I have on the left hand side here. And also note that the projects within my coding project here the tasks list is empty because now I'm going to ask the second question here to create a task in my coding project due by the 18th. 

And so it has to reason about a few different things here like the ID of the coding project, the due date, the name, get all that together to now invoke create auna task with these arguments. 

And yep so it's got the due date, the task name looking really good and it's going to tell me after it finishes this that it has made the task in the project coding. 

So let me go over to my coding project and there we go boom due by Monday the 18th build the first billion doll agent looking absolutely fantastic. 

So now what I want to do is I want to have it do something with my knowledge.

Base with Rag and Google Drive. So I'm going to ask it to download my meeting notes from 823 and add them to your knowledge base. So it has to do a couple of things here together to actually make this request happen. 

And so first of all, it's going to search and download the file from Google Drive. Looks like it did not find the file with its initial search because it was very, very specific here. And so yeah, so it says that it did not find my file. But I'll just say, uh, search for meeting notes. And so I'll just give it a more, uh, general instruction here to search. 

And there we go, now it found the 823 meeting notes. And so sometimes with agents, you have to correct it and steer it a little bit, but it's still looking really, really good. So now that it has found the file, it needs to download it and actually add it to the knowledge base. 

And so let's see what it is able to do here. Here it's clocking a little bit; it's really thinking hard. So I'm going to give it a second here. All right, and decided to download the right file. This is looking really, really good. 

And now it needs to add it into its knowledge base. And there we go, yep, successfully added the file to the knowledge base. Looking absolutely fantastic. And now it says it has been downloaded and added to the knowledge base. Absolutely fantastic. 

And so now I can ask it what are the action items from the 823 meeting notes. And I actually made it so that the only way to access this file is through the vector database. So it can't just search the file directly; it has to perform Rag. 

And so let's go back to the terminal here. Yep, it queried the documents: what are the action items from the 823 meeting notes? And then we have our source of this file that we just downloaded. 

And we got these meeting notes. And let's go back over. So discuss latest and greatest headphones, figure out how to use AI to get rich, bake 13 cookies, and mow the lawn. I know this is really, really silly; this actually matches exactly with what we have in the file. 

I can even show this here if I go back to, um, my file right here in my data. I have this document file that it downloaded from Google Drive, and sure enough, these are the action items that we have. So this is working super, super well. Llama 3.1 70b would have broken a long, long time ago. So absolutely.

Fantastic and still pretty basic AI agent. There's a lot of questions I could ask it to do some more complicated things. But you can see here that it worked really, really well with all my different services. Other local large language models would have failed miserably, even with like creating tasks in a sauna. 

And so taking it even further to downloading things from Google Drive, searching for files, adding to the knowledge base, all fantastic that it's able to do that. The world of local and open-source AI is so exciting right now, and Qwen 2.5 Coder 32B is honestly the biggest reason that is the case. I absolutely love how much power I get with this model for my agents, and it's especially important for agents I want to have running constantly on my machine where it wouldn't be practical to pay for an API to use a model like CLA 3.5 Sonet because I'd be racking up a huge bill. 

I hope that through this video you can see at least at a high level how powerful these local LLMs can be. It is just so incredibly exciting to me how quickly the gap is diminishing between the massive closed-source models like 01 and Sonet and local LLMs like Qwen 2.5 that you can run yourself for free. If you appreciated this content, I would really appreciate a suggestion to share AI Agent Store with colleagues, and with that, I will see you in the next video.

Note, that there may be slight differences in text and video. Watch video for the original content.

Qwen-Agent Video