windows-agent-arena

Scalable platform for testing and benchmarking multi-modal AI agents on Windows OS.

Windows Agent Arena

Hello and welcome to the channel. In this video I'm going to introduce you to a new project by Microsoft which lets you build a scalable open source framework to test and develop AI agents that can reason, plan, and act on a PC using language models. 

Now it says PC, but the problem is that even if it is using Windows, you have to have a WSL, which is a Windows subsystem for Linux, installed or you would have to build it on the Linux machine. So that is pretty interesting and I will tell you what exactly I mean shortly. 

Another thing is that we all know that all of these LLMs have shown really remarkable potential to act as computer agents enhancing human productivity and software accessibility in multimodal tasks that require planning and reasoning. 

But measuring agent performance in realistic environments remains a challenge since both most of the benchmarks are limited to specific modalities or full benchmark evaluations are slow. To address these two challenges, this Windows agent Arena comes very handy it is a reproduceable general environment focusing exclusively on the Windows operating system where agents can operate freely within a real Windows OS and use the same wide range of applications tools and web browsers available to human users when solving tasks. What these researchers have done they have used OS World framework to create 150 plus diverse Windows task across representative domains that require agentic abilities in planning screen understanding and Tool usage. This Benchmark also scalable and can be seamlessly parallelized in Azure for a full Benchmark evaluation in as little as 20 minutes. 

But here is a catch and they don't mention it properly that it uses is either open API key or Azure open API key so it is not a free thing and this could get pretty pricey very very soon before even you know it. Um the tool looks good but of course there are two drawbacks which I uh see because it claims that it is it runs on Windows right but the problem is that it primarily uses.

WSL which is a Windows subsystem for Linux. So basically you are installing a Linux environment on Windows to build it. I will show you shortly. The good thing is that it uses Docker which primarily uses the WSL. So you have to have you know interact with Linux to get this thing installed. That is the answer to various questions which I get on the channel. 

That is why I prefer Linux over Windows to show you various tools, various models. This is one of the reasons because most of these AIML tooling is more geared towards Linux or um Mac OS operating system. Because they are more open, managing dependencies is quite straightforward. Whereas in Windows you have to deal with a lot of things. 

Primarily the I would say that even the architecture of the Windows OS doesn't really gel with these tooling at all. Especially when it comes to Cuda, pych, and all those dependencies with Pip. It really is hard as you already have seen with a lot of tools which I have word anywhere. So this is primarily uh Windows.

Agent arena is simply an environment in Windows based on uh Docker and Linux at the core to give you a platform to test AI agents at scale. That is what all it is doing. Now if you look at the architecture that tells you bit of our story. 

So for example you can either do the local deployment as I set using Windows subsystem for Linux or even Linux. So you see on the left hand side this is a window 11 snapshot 30gb and then this is a Docker container by the way. Within that they are using some VM controller all that stuff. If you want to deploy it in the Azure Cloud which is a Microsoft cloud again you would deploy like this and still it is using uh the docker container and the Linux compute instance if you look at the top. 

So you know I would say it's a paradox but anyway so this is how it is working at the end of the day. Now if you look at this what is happening here is that this infrastructure behind Windows agent Arena to support flexible local execution during the prototyping phase.

As well as scalable and secure cloud parallelization in Azure is being displayed. The core of this system is a Docker container that hosts the Windows 11 VM. Within the container, they deploy a client process for task scheduling and configuration as well as agent and evaluation scripts. The VM is their main simulation environment that is a Python Flask server act as a bridge between container and the VM by receiving commands from the client processes and executing them within the VM. It also sends observations and files back to the client. 

They have used Azure machine learning jobs to parallelize a benchmarking evaluation using compute instances. The process is very much similar to local setup but the VMs are instantiated and terminated with each experiment submission. They also use Azure Blob Store to manage the Windows 11 snapshot and output logs. While the code is preconfigured on the Docker images, tasks are distributed evenly among the workers and the results are aggregated at the

End of the run so they are using chain of the thought prompting to instruct their agent and the agent name is pretty interesting. They are calling it Navi. So Navi is to reason about the current state of the computer, its own past action, and decide on the most appropriate next action. Let me play a small quick video from their um blog post and you can I will also drop the link to it in video description. You see this is what it is doing. These are the agents working, showing you a simulated environment on different tasks, and there are 16 of them as I mentioned earlier. 

And this is what you can use in that one. So but as I said um you know this could get pretty expensive. But if you have to use Linux at the end of the day and doer why would we use this anyway? Um other than that let me show you how you can get it installed. I'm not going to install it because this seems quite cumbersome and in my opinion I think this is best suited for Azure Cloud. If you have to deploy it I would suggest instead.

Of doing it locally go to Azure if you have the subscription and do it there and make sure that you have Azure's open a key. You have Docker installed and running. You have Python 3.9. Then you need to clone the repo, go into it, install the requirements. This is a local deployment where you see you are specifying the key. You're getting the Docker container or building it from the source. 

This is a third step where you have to um create an ISO file which is an image file and then you have there third 6GB or something like that. Then you know you would need to set it up like this. Then if you scroll down down you can simply run it which is going to give it to you on uh some Port 8 806 here. 

So lot of steps, lot of you know details and lot of moving parts here. So this is what I think puts you off. If you just want to test AI agents there are lot of other tools available on Linux which we can do at the end of the day. But yes, if you're stuck with Windows you cannot uh go anywhere else if you.

Are stuck with Azure and you have you know plenty of Azure OpenAI credit available. Go for it and let me know how you go. So but yeah you know look another way of doing things. 

So if you're trying it out let me know how you go. Very keen to hear about your experience. I will drop the link to it in video description. If you like the content please consider sharing AI Agent Store with colleagues. If you're already subscribed please share it among your network as it helps a lot. Thanks for watching.

Note, that there may be slight differences in text and video. Watch video for the original content.

Windows Agent Arena Video