Name: HuggingGPT
Uploaded: 2025-02-14T06:38:36.812Z
Duration: 2 min 33 s
Description: LLM-powered agent connecting various AI models to solve complex tasks across multiple domains.

hugginggpt

LLM-powered agent connecting various AI models to solve complex tasks across multiple domains.

HuggingGPT

Hey everyone. My name is Vedemin. In this video, we're going to have a look at Hugging GPT, a paper by Microsoft Research that describes an idea that you can use a watch language model such as Charge GPT or GPT-4 in order to control some specific AI models, for example, image classification, object detection, etc., from the Hugging Face repository. 

Then, carried out much more complex tasks compared to what just a large language model can do. There is also an open-source system called Jarvis by Microsoft, which is available on GitHub, which is essentially an implementation of what the paper is describing. We're going to have a brief overview over that as well. 

So the authors claim that this is a step towards advanced artificial intelligence. Is this true? Let's find out. The paper is available on archive, and I'm going to link it into the description down below. Let's have a look. 

So the Hugging GPT paper, or the title of it, is "Hugging GPT: Solving a Task with Charge GPT and Its Friends in Hugging Face." The authors are already hinting in what this should be. So they're talking about solving AI tasks with one model, which is in this case Charge GPT, and then other models. 

The first sentence within the abstract is also quite revealing: "Solving complicated tasks with different domains and modalities is a key step toward advanced artificial intelligence." So what they're hinting at or describing is that the most important part of AI is to solve some specific tasks within different domains. They're talking also about ROMs and GPT, etc., but they're also providing a

Description right here on what this model might look like or what the main idea behind it is. So we have a prompt by a human problem. Can you describe what this picture depicts and count how many objects in the picture? Then this is given to uh llm or one large language model such as charge GPT for example. 

And then this controller is doing a couple of things. Select the model, execute the model right here, and then generate response. So the watch language model as you is used as the brains of the operation or the brains of the answer. The heavy lifting or the parts that are exclusive to this task are done by some models within the hugging phase. 

So in this case we have to describe this picture and how come and count how many objects do we have within it. So probably this is done with resnet from Facebook in order to detect the objects. And then after the object detection is done we can count the objects. 

And then we can do gpt2 image captioning and then give the image to this model and then use it in order to get the text of the model. And in this case a text that can describe the given image. I heard of giraffes and zebras grazing in a fields. 

Okay so this looks pretty interesting. And in order to complete this task essentially charge GPT was given some models and then it took the model predictions and synthesized this response. Let's have a deeper look into how this works. Very interesting. 

So hugging GPT the workflow of hanging GPT consists of four stages: task pointing, model selection, task execution, and response generation. So these are the steps that we look.

Through and here we have a much more general or better image of what is happening under the hood. So again we have a prompt with a specific image and this time we have please generate an image where a girl is reading a book and her pose is the same as the boy in the image and some images given. Then please describe the new image with your voice.

Okay so given this actually what the charge GPT in this case is doing is to do some task pointing. You can see that actually task pointing looks like a lot of like function or something like a tool from the line chain Library which we've looked at briefly in the previous video. So here we have a couple of tasks pause control image classification image to text pause to image object detection and then text to speech. 

So all of these tasks are going to be used by charge GPT but the specific tasks are going to be carried out by some particular models. So in this case for pulse control object detection and image classification etc. For object detection we are using again Facebook debtor resnet1101. So in this case charge GPT is using this model. 

Then this is given to execute the task with the resnet model and then we are creating bounding boxes with probabilities etc. for the object detection part of this task. Then we have all of the tasks with the particular images and then the responses. 

All of this is again given to charge GPT or other model and it says that the image you gave me of is a boy. The first thing that I did was to use opencv open constant open pose control model to analyze the pose of the boy.

In the image so here you can see that this is the response from the pose of the boy. I fed that image specifically located Etc into the model to get an output image of the pose. So this is the pose image. Then I take that post and generate a new image located at image 3 using control net open Pulse. 

Okay so it is using the pose as an input and then using another model to generate something a girl that is reading as the input to this next model. Next are an object detection and image generated in the previous step to get predictions of the objects present in the image including the Box location. 

Okay so it's running object detection on the newly generated image. These predictions will run through the better resnet 101 model. So it is doing object detection on this with the resonant or debtor model. The box was drawn on the image. 

Okay so it is given the bounding boxes of duplicated objects. Then use the predictions of an image classification model and an image captioning model to generate the caption of newly generated image. It generated the text agaro sitting on a bed reading a book. 

And finally it did with fast speech from Facebook convert the generated text which was this caption into text audio and it says where the audio should be read through. So this is pretty mind-blowing actually. 

Yeah so charge GPT is our other ROMs can essentially provide the brains behind all of this execution which is mind-blowing. And then here is a list of the possible tasks that are provided using the particular models from hugging face tax classification token classification.

Summarization Etc and note that those are carried out by specific models from the hanging face models selection. We have image related ones, text relation related ones, audio and then video tasks. Very very very cool. 

Okay, the authors are also providing other examples right here. Those are very interesting. You might go and look through those. Uh, here is another one that I really liked. Given a collection of images, so you're giving a lot of a couple of images, three images. Please tell me how many zebras are in these pictures to determine this. So it says that there are four zebras. 

Okay, this model called Etc image captioning. Okay, from the caption I was able to determine that image A contained the cat and image B contained two zebras. Okay, the model generated images with predicted boxes for each zebra. Okay, the model generated images Etc which is zebra and the images can be found in the following locations. So it does the object detection on those and finally it analyzed the captions from the model output and then it took a confirmation that there are indeed four zebras. 

Wow, okay another a lot of cool examples right here. So look at this image. What is the red topping? Sure, the red topping on the pizza in the image is tomato and it's telling you which model has been used. It is giving actually the prompt to the model. What is the red topping? The answer with high score is tomato and is giving you the confidence of the model. That's amazing. 

All right, so the authors are describing a couple of limitations of this approach and here there are hugging GPT.

Inevitably suffers from some limitations. One of the limitations were most concerned about efficiency. So the inference time of this virtual language model can be quite large. So for each round of user requests in GPT requires at least one interaction with the watch language model during the task planning, model selection, and response generation stages. Yeah, but you can think about probably that in somewhere in the near future I guess those models are going to be sped up quite a lot compared to today's efficiency. So this might not be an issue in the future. Of course, we need to work on that.

The second is the limitation of maximum context land. Okay, so if you recall that watch language models are actually bounded by the number of tokens that they can input and output. So this is a limitation that hugging GPT is also having. But yeah, GPT 4 has 32k tokens that can use in a single input and output, which is quite large actually. So probably in the future this might not be such a big problem after all. But at least for now it is. We have used the conversion conversation window and only track the conversation context in the task pawning stage to alleviate it. So there are basically trimming the conversation. Yeah, but if you go through complete conversations it might be quite a large context that you need to carry around.

And then the third is system stability, which includes two aspects. One is the Rebellion that occurs during the inference of large language models. Yeah, sometimes the responses of those types of models GPT or gpt4 are quite unexpected. The second is.

Uncontrollable state of the expert model hosted on how you face Empress. So first the output format might not be what you expect. So you might ask for something and then the model might provide something slightly different. And if you're trying to pass through the result and you're giving it a specific parser, your parser might fail. 

And the next one is the model. So heavy phase might be affected by network latency or service state. So the authors of the paper hugging GPT, as I've already told you, are actually Microsoft researchers. So in Microsoft Jarvis, you can see that they're trying to implement actually the Jarvis system, which is essentially what hugging GPT is already providing as a concept. 

And then here you have the implementation. So language serves as an interface for LLMs to connect numerous AI models for solving complicated AI tasks. So exactly the same that what we look through the paper. And they have the image from the paper right here and they're saying that they introduce a cooperative system that consists of an LLM or large language model as the controller and numerous expert models. 

And these are the four steps that we look through when the model is generating the response based on the prompt query. So here they're also giving us a OpenAI a quick start of everything that you need to do in order to run this model. And yeah, it is looking that it is working. Let me know if you want to have a deeper look into this and probably try out some examples. 

In this video, we had a look through the paper hanging GPT. The main idea behind the

Paper is that you can use a watch language model to orchestrate simpler models that are more specific to some particular tasks. In our case, we had a look through how you can maybe use charge GPT or GPT-4 in order to do object detection and then counting of objects, image captioning, etc. In order to provide the responses to much complex queries or prompts by the user. The paper is really interesting idea and the authors from Microsoft are already have already published some code on the Jarvis system, which is available in GitHub. You might have to go and look around. Let me know if you want to run through this system and maybe do some demo with it. 

Let's see the capabilities of that open source project. Thanks for watching. Please share the AI Agent Store with colleagues. Bye.

Note, that there may be slight differences in text and video. Watch video for the original content.

HuggingGPT Video