Name: Azure AI Vision
Uploaded: 2025-01-22T12:30:52.258Z
Duration: 2 min 33 s
Description: A cloud-based service providing advanced algorithms for processing images and videos to extract rich information.

azure-ai-vision

A cloud-based service providing advanced algorithms for processing images and videos to extract rich information.

Azure AI Vision

Here's how to use Azure AI vision and analyze images in Vision studio. In this video, we're going to look at what computer vision is and how images become data. Then we're going to look at how we can teach computers to recognize and understand images. At the end, I'm going to walk you through a guided demo to see how Azure AI Vision makes it accessible for all of us to create solutions where we can analyze images. 

But before, make sure that you share AI Agent Store with colleagues. Also, this video is the first in a full series on Azure AI engineering. So if you're interested in this topic, drop a comment below and let me know what you'd like me to cover. 

Computer vision is a field of AI that enables machines to interpret and understand visual information from the world around us. But how does a computer actually see an image? To a computer, an image is simply an array. It's a list of pixels where each of them has a numeric value. 

Picture a layer with a grid where each square is filled with a number from zero, which is the color black, to 255, which is white, and everything in between being represented in shades of gray. Now the world around us is colorful, so color images are in fact multi-dimensional, meaning that they consist of not just one but three layers that represent the red, green, and blue color channels. 

So we can represent a color image by defining three channels of pixel values. All you have to do is to mix these values, and then you can just create any color that you can think of. Now we know how images are represented in data, but how can we

Teach computers to recognize and understand these images. This is where machine learning comes in and specifically convolutional neural networks or CNN. These neural networks use layers of filters to extract specific features from images and then they use those features to make predictions like for example identifying an animal or fruit or anything else. This technology has been used for a very long time but nowadays it's getting used less and less. 

This shift actually started with the huge advances in Transformers because their success in natural language led actual researchers to think about whether they can use the same approach for image data. So this natural language revolution led to the multimodal models that we're actually using today because multimodal models are trained using a large volume of captioned images with no fixed labels. An image encoder extracts features from images based on pixel values. Then it combines them with the text embeddings that are created by a language encoder. 

So the model actually consists of relationships between natural language embeddings and image features. Now the scope of this video is not to get too deep into the specifics of multimodal models but I really hope that this helped to get a better sense of how much computer vision actually evolved in the last years. 

So how can we start building these AI solutions using computer vision? This is where Azure AI Vision comes in. It's a cloud service that provides both pre-built and customizable computer vision models. This makes it easy for everybody to create powerful AI vision.

Applications with Azure AI Vision you can perform many tasks. Think about tasks like optical character recognition to extract text from images or you can generate captions and detect objects. And you can also summarize the content of an image. 

So let's use Azure Vision Studio to analyze images using these built-in features. Let's consider this scenario, right? Let's say you manage a store and you decided to transform it into a smart store. To do this, you're going to need to use an AI service to first monitor the store and then to identify the customer that might need assistance. This way, the store employees could help customers a lot faster, right? 

By using Azure AI Vision, all these images that are taken by the cameras in your store, they can be analyzed to describe what's in them. So let's do this and get both what's in the images and also what actions are shown in those images. 

And the first thing that we need to do is to create an Azure AI Services multiservice account. For this demo, we need to make sure that we create this resource in one of these seven regions. Usually, East US is a safe bet, so we're going to create an Azure AI Services resource here. 

We're going to search for AI services and we're going to create the resource. Select a resource group or create a new one. Now give it a name. These need to be unique, so choose one that's available. For the pricing tier, select standard. Don't worry here as you're not going to be billed by just deploying this resource. You only pay for what you actually use, so deploying a service or leaving it just idle won't really cost you.

Anything now check the terms and conditions box and now you're good to go. Click review and create and once the validation passes you can just just click on Create and the resource will be deployed. You got to give it a minute or so until the deployment will be complete and then you can just start using it. 

Now we're going to go to Vision Studio. You can access the studio by going to this link and we can just get started. Click on view all resources and now click on the three dots and click refresh. Sometimes you need to refresh if you just deployed an AI Services resource because this won't show up immediately. The thing is that it's not really obvious that you can refresh from here but there you go. 

Now select it and select as default resource and now can exit the screen on the main dashboard. If we scroll down we can see all the services that we can try out. We can extract text from images, we can detect objects in images, and we can also add captions to images. We're going to select this and in this screen you can see the default resource is AI Services 3. 

In order to try out this service we need to check this box that says that we acknowledge that trying it out will cost us some money. By running this with a couple of images is going to cost you about 0.01 to 0.02 so it's not that big of a deal but still every time that you use this service it's going to cost you some money. So you need to be aware of this because it's not really free. 

Now we're going to use some images from this image analysis data set. Here's the link that you can use to download the zip file there.

Are four images and we're going to drag this first one to try out the service. Now let's drag and drop it and wait to see what attributes it detects. Now remember that this only works if you check the box that you understand that it will incur these costs. After a couple of seconds it detects that a woman and a girl are in a store. You can also view the Json response as well because you're going to use this service using the rest API. Here we're going to use the UI for demo purposes, but you're going to do this through APIs when you're going to be running your app. This is just a great way to test out some images, see what information that you can get, and then you can figure out if it's enough so that you can build your use case. 

Here the most important keys are the text and the confidence score. So your logic can say that you only take into consideration the captions that the system is very confident in. The captions to images tool only creates a sentence that describes the content in an image. It doesn't really dive deeper than that, and for our use case we want to direct our employees to people that are performing some actions. So actions that are performed or at least to get a better sense of the object in an image and what those people are actually doing or holding or wearing. 

So only to do something if we see that somebody's shopping or standing looking at something or, uh, or just holding a product. This way we can infer they might actually need some help. So the captions to images tool is not really optimal for what we want to do. So now let's look at how we can detect.

All the object in an image and afterwards we can look at the actions that people perform to see important objects in an image. We need to use Den captions so we can go to the image analysis Tab and select add dense captions to images. Now let's drag the same image here and analyze it. 

The dense captions feature provides multiple captions for an image, one describing the image's content and the others each covering essential objects that were detected in an image. Each detected object has a bounding box that defines the pixel coordinates of the object within that image. The first caption is like the captions feature; it relates to the whole image, but the next ones are specific to some areas within the image, like the woman standing and holding a phone or row of jars on the Shelf. 

Now all of this information can be useful for your use case since you want to know what people are doing and holding so that you can actually check if they need some help. For example, you can check images at various timestamps and if the person in the image is doing the same thing or if they're holding the same object, they might need some help from a story employee. So the dense captions feature is suitable for our use case. 

The next image analysis tool that I want to cover is the one that helps us with extracting actions from these images. So extracting tags that refer to objects, to people, and to actions, and these actions being performed in those images. Again under image analysis, this feature is extract common Tex from images. We're going to drag another image here and we're going to test it out.

We wait a bit to get the detected attributes. Now we can see that the attributes are not sentences but words that are referring to objects, to people, attributes of people, locations, but also actions like, for example, standing, selling, and shopping. This feature is again useful for us because now we have words, words that we can use to trigger some logic in our applications. If we see a person shopping next to an area in our store, then we can perform some actions.

The next and the final feature that we're going to look at is the detect common objects in images. This is a simple feature that other tools cover as well, but it's still worth mentioning. It detects and extracts bounding boxes that are based on recognizable objects, people, and animals. It's similar to dense captions, just that this one actually returns words that represent the detected objects. 

We're going to drag and drop another image here, and it detects a person with a confidence score of 84%. It also detects a supermarket with a confidence score of only 51%. Now the reason why the confidence score is low here for the actual supermarket is because the background is a bit blurred, but still, it recognized it. Now you can play around with the threshold if you want to exclude objects that the system is not very confident in.

We just scratch the surface of what's possible with Azure AI vision, and these features are great because they can definitely help you ideate on so many business ideas and many solutions that you can propose to your clients. Also, if you're taking the AA fundamentals exam, these features.

Are enough if you learn them well cuz they fully cover the image analysis part. Speaking of this, at getdbash.com we offer practice exams to help you prepare for cloud certification exams. We currently have both Azure and Databricks practice exams, and we're adding more every week. Definitely check it out as if you're looking for a way to support this channel, this is a way you both support Decision Forest and you support yourself by learning a new skill. 

But realistically, you're going to need to explore more, and you need to experiment yourself with AI Vision. Practice tests and guided demos help, but nothing beats your own experience with these tools cuz getting this experience is going to make you a better AI engineer, and you're going to create better AI solutions.

Note, that there may be slight differences in text and video. Watch video for the original content.

Azure AI Vision Video