black-forest-labs

A pioneering AI startup specializing in state-of-the-art generative models for image and video synthesis.

Black Forest Labs

Even though the progress of Open Source AI seems to just be accelerating faster and faster every week, I think we still don't talk about the progress made in generative AI enough. It always seems like when we get updates in generative stuff, whether it's just text to image or text to video, it seems to always come kind of all at once or not at all. 

This past week, although there was a lot of other impressive news with large language models, something that flew kind of under the radar and that I think is incredibly interesting is a project called Flux from Black Forest Labs, which is a brand new team that was spawned from the original engineers who worked on Stable Diffusion. 

Now, obviously, Stable Diffusion and Stability AI are basically no longer being actively developed. Stability AI is basically just being purchased for parts, and this new team basically wanted to set out and continue to make state-of-the-art open source models. 

It's really interesting to think back just a year ago when the state-of-the-art was basically Mid Journey, DALL-E 2, and very quickly, seemingly out of nowhere, base models from Stability AI as Stable Diffusion got better and better over the last year. Now this is kind of the next step, and of course now it's ironic that it's actually rare to get impressed by text to image. 

Sure, there have been small updates like adding text and things like that, but what's really cool...

Is what potentially can be done with these models and flux and the series of models that's been initially released by bfl is nothing short of that. And what they want to do next I think will be very interesting to a lot of you. So welcome to AI flux. Let's get into it. 

So what is Black Forest Labs and what is flux? The first model they've released. So Black Forest Labs again is an offshoot that's been built by some of the original Engineers who were working on stable diffusion. They don't have a ton of information on their Twitter, but one thing that they highlighted in their launch on August first was that they were excited to announce the launch of Black Forest Labs. Their mission being to develop Advanced state-of-the-art generative deep learning models for media and to push the boundaries of creativity efficiency and much more. But other than that there really wasn't that much information and all they released at the time was what they're calling flux.

One or their Suite of models that they call is the next generation of pushing the frontiers of text to image synthesis forward. They gave us some information about this but not really a lot. Most of what they were saying here is that their incentives are very similar to that of stability AI in terms of making really powerful text image or just generative models uh open to everyone and people without tons of compute that was also still fast. Their funding is pretty impressive coming from Anderson Horwitz, Gary Tan, and a number of big names in the Bay Area. 

What's cool is their current model in terms of Elo score is really impressive. Uh flux one Dev is better than stable diffusion 3 Ultra and a number of other really impressive models like mid Journey V6. There are a few different models. There's Pro Dev and schel basically schel being the fastest and this model is really impressive. 

Although I want to jump into kind of what you can do with this model, that's not really what I want to focus on in this video. What I want to focus on is what they actually want to do with this model next. So obviously text to video is the next sort of big Frontier in terms of generative AI. It's the thing that we used to think was basically impossible outside of Sora and the Chinese copy of Sora was also incredibly impressive. We were guessing that it was probably just trained on tons of YouTube data but we'll get to.

That in just a bit so what Black Forest Labs is promising next and what I think is their true purpose and their most exciting project is what they're calling state-of-the-art text a video for all. Now we don't really have any preview other than this kind of a interesting stylized bit on their website but it looks pretty impressive and I wonder what they use to actually train it. One of the things that I thought was interesting was with the latest release of stable diffusion 3 there just wasn't a lot there when it came to video. We heard that there would be 3D options. We heard that there would be multimodal options and they almost delivered but really the video was still not that great and clear.

Black Forest Labs knows what they're doing because they understand that in order to set out to build a state-of-the-art text to video model first you have to have text to image models that are really fast because you're generating 30 still frames per 1 second of video generally. And to have all of the particle effects make sense a lot of has to actually fall into place. So we haven't seen an official demo yet other than this footage you're seeing right here but I think it's really interesting. 

The reason that I think this is more interesting and more exciting than just another AI paper or another team from another company um like Open AI or whatnot is they want this to be open in theory supposedly based.

On some threads on Reddit that the team has shared, they want this to be able to run on a single 3090 or 4090. Which again is crazy since the 3090, although I think it provides the best value in terms of AI compute, it's crazy that's a 6-year-old GPU. So they want it to be available to everyone, be as powerful or as impactful as Sora, and maybe we'll see this in the next few months. They don't really give too much of a timeline for this, but I think we're going to see exciting things from Black Forest Labs just given the team history, given the people funding this, and I think it's going to be something special.

And what's interesting is they're not the only ones who have been working on this model that have maybe gotten some surprising press this week. The other people who got press were Nvidia. So you might have heard about Nvidia this week because they found some issues with their Blackwell B200 GPUs and potentially the 5090. But 404 Media found something really interesting, which were sets of emails and Slack messages that pointed to Nvidia using some open source tools to download YouTube videos. That I actually use on a day-to-day basis to download a ton of YouTube videos and train a multimodal text to video model called Cosmo that they were using predominantly for their Omniverse 3D world generator, self-driving car systems, and a number of other projects that they would basically describe as being.

Used to unlock various downstream applications critical to Nvidia. Which obviously if you followed Nvidia at all this is a massive part of their playbook. So what's interesting is this all came from a former Nvidia employee and it's pretty interesting. The funniest thing for me reading through this is that they used a project called YouTube DLP, which is a source of a really famous project that's been around for about a decade called YouTube DL or ytdl. 

And basically this is a really popular open source python project that supports downloading videos from hundreds of different websites. And what's funny is they claim here that using 20 to 30 virtual machines on Amazon AWS they were able to download about 80 years of videos per day, which is like terabytes near petabytes of video data. And then use that to actually train a lot of video. Um, the funny thing is they also used some clever tricks that I've used in the past and that I developed in college to refresh IP addresses to avoid being blocked by YouTube. 

And what's clever about this or what's impressive about this is generally all of the IP space or all the IP addresses used by Amazon are blocked by YouTube. So it was probably a pretty massive feat to actually even have this work. Obviously there's the question of should they have been using that data. Um, they also mentioned using video from Netflix but I can assure you that YT the YouTube DLP doesn't.

Download Netflix videos and it's basically impossible. So I'll link to the 404 media article below. But the cool thing here is that it shows that Nvidia is also incredibly focused on these models. 

Nvidia's focus is a little bit different though because Nvidia, when it comes to making these um state-of-the-art texted video models and engines, a lot of the reason that Sora from Open AI looked so realistic had more to do with physics than it had to do recreating graphics. Obviously, Nvidia has already pivoted in major ways in terms of like moving away from rate tracing to just using AI to infer how light um will fall. 

Inferring physics is the next big step here and that's really where more value for Nvidia is in this model. Like obviously the Black Forest Labs project will be mostly visual, but when Nvidia wants to use this to simulate things, they're basically inferring physics simulations and seeing that that has more value than um using something like fizx that they've been working on for years. 

Which again kind of was their previous idea of why don't we perfectly recreate and model everything and then go from there. And we're now realizing that with diffusion models we can actually just infer what it's going to be, have it happen faster, and basically be accurate enough that you can consider it useful enough to use in the real world. 

So I'm hoping this Nvidia model is real but they're basically describing.

This as is as a a new foundational um opsource U video model which would be incredibly cool whether or not you can run this on a single GPU. Um I would have lots of doubts um about that but hopefully you know we see them fix their um Blackwell issues in the next 6 to 8 months. We hope that there're no issues with Taiwan and tsmc and then maybe we'll see this model. 

But this is some really cool reporting from 404 media and it's entirely believable given the tooling that was described. As an engineer I can endorse this is probably real and it'll be really cool to see um how good this model is based on some previous work coming out of Nvidia. 

So do you think this is real? Do you think Nvidia was in the wrong for using this data to train their model? Are you excited to use um this new work from Black Forest labs? Let me know if you want me to actually test this model in depth. 

But um yeah so as always I hope you learned something. Um sorry this video came out a bit late the last time I recorded it the audio was completely screwed up but now it's good. And um yeah if you share the AI Agent Store with colleagues, we'll see you in the next one.

Note, that there may be slight differences in text and video. Watch video for the original content.

Black Forest Labs Video