11 AI Breakthroughs You Didn’t Know Existed

0:00So every week I like to make an end of week AI news breakdown video where I share everything that happened in the world of AI for the week. Usually those videos are very focused on some of the bigger labs. They're the ones that get most of the news. OpenAI, Anthropic, Microsoft, Google, Nvidia, companies like that. However, there's often some really cool news that flies under the radar. There's a lot of research going on and smaller tools that not as many people, including myself, are talking about right now. So in this video, I want to share some of the cooler things that I've come across recently that

0:35haven't made the bigger end of week news videos. Let me know in the comments if you like this style of video and I'll probably make them more often and share some stuff that's not from the big labs. But let's start here with Fish Audio. Fish Audio released a model called Drama 3 and it's a text-to-speech model where you direct the performance in plain English instead of picking from like a menu of emotional presets. So like most voice tools, they'll give you a drop-down for happy, sad, angry, things like that. Drama 3, however, it lets you write the direction the way you'd like explain it to an actor. You put it in

1:11square brackets right in the script. So for example, here's some spoken text with no direction. I'm fine, really. Go on without me. And now here's one where there's some direction written in, barely holding it together, forcing a smile. I'm fine, really. Go on without me. And you can even pick specific words. Like right here, they specifically tell

1:33it to whisper go on without me. I'm fine, really. Go on without me. So you can like sculpt an effect to specific words. You can drop in sound effects, things like a small laugh, shaky breath, and you can put them exactly where you want them. I'm fine, really. Go on without me. You can even have multiple speakers with

1:55multiple voices. You were there that night. I was home, asleep. Then why is your coat still wet? It rained. They even licensed Uncle Roger's voice for the demo page, which is kind of hilarious. Hiya, nephew. Why you put ketchup in egg fried rice? Fried rice need soy sauce, need MSG, need wok hei, not tomato sauce. Uncle

2:20Roger cannot watch this. If you've ever watched like any of the Uncle Roger videos, it's actually pretty spot on. If we take a quick peek at artificial analysis here, we can actually see that Fish Audio Drama 3 here scores the highest on pronunciation. So, it's really good at pronouncing words, even scoring higher than 11 Labs and SpaceX AI's text-to-speech model, which is what that I actually demoed in last week's news video. However, when looking at the arena leaderboard preference, which you know, sort of did a blind taste test where people listen to a bunch of different models and picked their favorites without knowing what model they were picking. Well, it didn't quite

2:59do as well. So, you know, it reads hard words better than almost anything else, but a lot of people still prefer other voices. It also appears to be open weight available on Hugging Face. It is a 36 billion parameter model, so you're going to need some pretty decent hardware in order to run it. People have reported that you need roughly a 46 GB VRAM card, or you can get away with 32 GB with a community quantized 4-bit version, but neither of those are going to run on like most laptops. A couple other quick notes that I found about this one. So, the license is free for

3:36commercial use, but only if you're making under half a million a year, which is probably a non-issue for most people. The API is about $10 per million characters right now, but that is a launch discount that jumps to $50 after November 12th. And Fish audio takes do vary, so plan on generating a few and keeping the best one. And if you are going to use the API, double check that you set the model to drama three because it often will quietly fall back to older models and you'll still get billed even if you're using the older models. So, make sure you check that. But if you

4:10head on over to fish.audio/dramat3, click try drama three for free, you can play around with it right now. And if you make audiobooks or game dialogue or anything with characters, probably worth trying out. All right, while we're talking about voices, once you have that voice performance, you might want like a face to go with it. Well, ByteDance recently released Omni Human 2.0. Now, this model will take a single image and an audio track and then it'll generate a video of the person talking or singing or acting it out. Now, this is the 2.0 version and the biggest difference is that the previous version topped out at

4:48just 60 seconds. This one will allow you to go up to five minutes in one continuous shot. So, you don't have to like stitch a bunch of short clips together and hope that you know, you don't see the seams or hear the seams. So, here's some examples. The kingdoms called it a legend. Tonight,

5:09I make it real. And it even works with singing. Now, double [music] time, follow the flow. Every expression ready to go. Voice in the feeling, feeling emotion. Wait, did you catch that? You can also add in text prompts to control camera moves or add things that weren't in the original photo. Now, it generates in 720p resolution and it doesn't just have to be humans, it can work with like cartoon characters and

5:33animals and things like that as well. Now, if I'm being totally honest, I still think these lip syncing models have a long way to go. When I watch a video with like an AI lip sync attached to it, it still just like isn't working for me. We're getting closer and closer. I mean, like we're way better than we were a couple years ago, but it still has a lot to be desired. I don't know, what do you think? Let me know in the comments. As far as pricing goes, the price on this one dropped from 12 cents a second in the last model down to 7

6:03cents per second in this model. So, if you are making a full 5-minute video, it's still going to cost you about 21 bucks. Now, a few other quick things to note about this one. Everything I'm showing you is ByteDance's own demo reel, and some early viewers did notice like background shifting around in their clip. Again, it's only 720p. You're not

6:25going to get 1080p or 4K out of this. And because this is only one photo plus any audio, there's some pretty obvious potential deepfake concerns with stuff like this. Although, if you saw what I just saw, I don't think we have to worry about too many people being fooled just yet. This one is available now through Byte Plus and through their API. All right, now let's talk about one of my more favorite topics recently. I've been absolutely loving these. Let's talk about world models. So, these are AI models that generate a video world that you can move around in. And Tencent's Arc Lab released one called World

7:05Crafter recently. And this one starts from just a single image or a single line of text. Now, one of the big issues that we've had with world models is that if you're inside a world model and you look away from something and then turn back, a lot of times it's like gone or changed. The model only remembers the last few seconds of what it showed you, and if you move around the room, everything kind of goes wonky. Now, World Crafter's fix is a memory that you look up by camera position. So, when you point the camera somewhere, it pulls the past views that match where you're

7:38looking and feeds those back to the generator. So, when you turn back to the original restaurant, it remembers what it saw the first time. So, they prove it with this little trick here. They take the video the model generated and rebuild a 3D point cloud from it. If the world were inconsistent, that 3D reconstruction would come out as just like mush. Instead, you get clean geometry and the camera path lines back up when it revisits a spot. It handles moving things, too, like a character going out of frame and then later actually coming back into the frame, like we can see with this horse video here, where the camera takes the horse

8:18out of frame, and then when the camera pivots back around to the right, you can see that the horse is still there even though it was moving. Now, on their own benchmark of 145 uh different scenes, they say that it beats the other models they tested on revisit consistency and camera control. But, just keep in mind that that is their benchmark. So, you

8:38know, take that with a grain of salt. The code for this is available on GitHub right now, and there's even a version on Hugging Face that you can go and play around with right now. In my own testing, though, telling it what to generate, it's kind of not the fastest thing in the world, but eventually it gets there. ByteDance also showed off their own version of a world model recently at an event in Beijing. My buddy Tim from Theoretically Media, he posted some footage that somebody sort of took during the event. So, it's like this bootlegged camera version. Now, the wildest demo they shared was they took a

9:15scene from Stephen Chow's movie King of Comedy and generated a brand new camera path through it. It's like reshooting a shot that was filmed decades ago. You can now get new angles on that shot. You can also type a description to get a world spawn a giant spoon and then climb it and walk through a portal from a room into outer space. Now, there is a catch, and that's that we can't use it yet. It doesn't even have a public name yet, and there's no release date, and the only footage I've actually seen of it was from people who recorded it at this event, as, you know, Tim from

9:51Theoretically Media shared here. So, yeah, we'll keep an eye on that one, but it looks pretty impressive. Real quick, before we get back to the video, I wanted to show you guys how I've been using this new feature, Dots from OpenAI. Now, I've talked about this in a recent AI news video, and since then, I've dug into it even more. Basically, Dots are these always-on AI agents that you can assign ongoing responsibilities to, and they keep on working on them even when you're not using ChatGPT anymore. So, for example, I've been using my Dot to help me with my YouTube analytics and updating them automatically every time I post. I

10:24connected it to my YouTube channel, and it pulls views and engagement data to find patterns across my titles and topics and angles and post times. And I also have a shared space with my team, where my Dot keeps all this organized and creates new subpages with weekly reports and video ideas based on what's working. What's cool is that each Dot has its own cloud computer and can use ChatGPT work and Codex to get things done. It's powered by GPT-6 Astra and learns my preferences, so it can proactively suggest things without me having to ask. So, check it out in the link in the description, and thanks so

10:57much to OpenAI for sponsoring this portion of today's video. All right, so we already saw two world models trying to fix the memory problem. Well, Sky Works Matrix Game 4.0 takes a completely different approach. This one stops asking the AI to remember the world at all. With a normal world model, the AI is doing everything. It's drawing the frame, but it's also the physics engine, the game rules, and the memory. When something leaves the screen, the only record it ever existed is in the pixels the model already drew, and that's kind of why things tend to vanish. Matrix Game splits that job into two. An ordinary game engine runs the game. It

11:36knows where every enemy is, what happens when you shoot something, what's behind you. It draws a cheap, blocky, low-poly version of that scene. Then the video model's only job is to repaint that blocky frame so it looks finished. So we can see in some of these models down here, on the right is what the game engine actually draws. On the left is what you see. It's the same frame in the same moment, but the engine handles the gameplay and the AI handles the look that's skinned over the gameplay. Since this game engine is keeping track of everything, the world's not going to forget a pyramid behind you is still in

12:16the engine's map whether the AI is drawing it back in or not. Here's another example of a car game. You can see the blocky, early rendered version and then AI putting its own graphics over the top. So the game engine only has to remember this and the AI does that. There's also another piece. If you ask for something the game rules don't cover, a coding agent writes new rules into the engine and updates the prompt the video model uses. So the world can change mid-game without the AI having to invent new physics on the fly. Down here we can see three games, a medieval castle shooter, an ancient Egypt

12:53shooter, and city racing. These long play sessions can run up to about a minute and a half without any sort of cuts. Now there are some catches. Each game has its own model and each one is about 5 billion parameters each, trained just for that scene. So it's not really a general make me any game model and the output is 480 by 864. So it's not anywhere near HD. Now the project page does talk about running on a single RTX 5090, but the model card does say there's no guaranteed frame rate, consumer GPU memory fit, or exact scene reconstruction. So, it may not totally fit into your GPU's memory, and fast

13:35camera moves and long sessions still can cause some drift, and the actual training code isn't included. But, you can get the weights and the inference code under Apache 2.0 license, so you can use them commercially. Now, there is a hugging face space listed here for hugging apps Matrix game 4.0. So, you should be able to play around with it here. They do have some examples that are pre-cached in if you want to check out some of those examples. Now, personally, I think this engine plus like AI paint job or AI reskin is a pretty realistic path to AI games, at least more so than like asking a model

14:14to do everything. Turns out Nvidia thinks so, too. Next up, let's talk about something that came out of Nvidia called Stream Render H3. This is actually a pretty similar concept to what we just looked at. It uses kind of that same split. Code runs the world, and then a video model draws it in real time. The difference with this one is how little the game actually hands the AI. Matrix game gives the model a blocky 3D render. Nvidia gives it a color-coded map. Every pixel is a flat color that means road or grass or barrier or rival

14:49car. There's no lighting, no textures. The video model actually draws everything you see. So, here's the game engine's own view, the color map that it actually sends, and then the final frame that, you know, the user sees. Now, because the map carries no like looks at all, you can actually restyle the whole game by swapping one reference image. We can see here that this is all the exact same drive, but when we fast forward a little bit, they can swap out the scene for anywhere they want and put the exact same drive in any different scene. It's the same exact input what we see in the

15:23middle just rendered out in different locations and different settings during the day and things like that. So, one thing to notice here, this is kind of one of the hard parts for this model, is that the color map paints all of the rival cars the same exact color. So, the model has to actually remember on its own which car is which and make sure to keep the same paint job across the various cars. So, a fun fact about this model is they actually handed Claude code the job of building a faster video decoder for this. They gave it a speed target and then they had a working

15:56version running in the pipeline in about 3 hours. Now, this does actually require some serious hardware. Each rendering step takes about 218 milliseconds. However, that's on eight GB200 GPUs. These are like the chips that they put into data centers from Nvidia. So, this isn't something you're going to be running on your gaming PC, at least not for a while. Also, it's only been trained on racing so far. That's why all of the demos are like race cars. So, you can put this one kind of next to Matrix game which we just looked at and you get

16:31a decent picture of the trade-off. Matrix game aims for one consumer GPU at like lower resolution where Nvidia's going for photoreal but out of rack of data center hardware. Okay, this one's a little weird but I want to show it off cuz it's also kind of cool. So, Odyssey, the world model company that launched Odyssey 3 not too long ago posted a clip saying they put a fruit fly brain inside their world model and now it flies around imagined worlds by itself. So, this fly brain wasn't actually created by Odyssey. It's a simulation built by a company called Eon systems from a complete map of a fruit fly's brain,

17:11about 140,000 neurons and 50 million connections. Odyssey swapped out the virtual body that brain was hooked up to and connected it to a camera moving through an Odyssey 3 world. I don't know. I thought this one was fun, but there's like no paper, no code, no explanation of how the two are connected. And people in the comments did note that the fly never actually tries to land anywhere, which, you know,

17:35if it was a real fly, it probably would. But it's brought up a lot of questions about, you know, simulation theory. Like, does this fly believe it's real and in a real world, but it's actually in a simulation? And if so, what are the implications of route simulation theory for, you know, humans and stuff? We don't need to go down that rabbit hole, though. While you can't actually play with this fly flying around inside of Odyssey 3, you can actually mess around with Odyssey 3 itself. Just head over to experience.odyssey.systems, give it a prompt of what you want it to create, and you can actually interact in

18:11this world. I am steering this fox right now. I'm telling it to jump. I'm telling it which direction to go. I'm changing camera directions. All of this is me actually controlling this. Pretty cool. All right, moving into 3D. Preferred Networks, which is based in Japan, just released Telescope, which turns a set of photos of a place into a detailed 3D mesh. Like, from a single object all the way up to city blocks. Now, the usual way to get a high-quality mesh from photos is to spend a long time fitting a

18:42model to that one specific scene. Telescope actually skips all that. It's a generative model, so it's learned what surfaces generally look like, and it fills in gaps where you didn't get many photos. It also handles shiny and glass surfaces, which normally wreck photo-based 3D. So, the name is actually the method itself. It builds the whole scene at low detail first, then zooms in on areas that need more detail, and refines them, like a telescope. The colors here show which level each surface came from. So, if we look here, we have a Gaussian wrapping, which is sort of the standard method. And if you look at this, it looks pretty good, but

19:22it's lacking a lot of detail. Now, this is telescope. And there's so much more detail here. So, here's the original Gaussian wrapping, and here is telescope. Like, it's a pretty big difference. Here's another example of Gaussian wrapping. And when I move this over, you can see their version and all of the details pop into focus. Now, there is a little bit of a catch. You do

19:45need a lot of photos to achieve this. You can see this one took 256. This one also took 256 right here. This one actually only took 56, which isn't too bad. And it can only rebuild what's inside the area your camera actually moved around. This is one we don't have any sort of real access to. It does say the code is coming soon, but it looks like a pretty promising leap from what we got out of Gaussian splatting. Here's another pretty cool one called Gaussi Animate. And this one's actually used for animating 3D characters. It's designed for rigging up characters that are soft on the outside, but rigid on

20:22the inside. So, people in loose clothes, animals, or garments just on their own. So, a normal animation skeleton gives you control, but a loose shirt or like a dog's belly doesn't move like bones. So, this adds on a whole bunch of other floating nodes on top. These additional floating nodes sort of capture all the jiggle and flow and the additional physics that come along with the, you know, the cloth or the material on top of the bones. So, we can see here that it finds the sort of shape, but then once it finds the shape, it actually puts a skeleton in the middle to figure out which parts need to actually be

20:59rigid. Once it figures out the points that need to be rigid, then it figures out what points need to sort of flow outside of the bones. So, here's some other examples like a clothed human. You have the skeleton, you've got the bones, you've got the skinning, and then everything combined where you can see the the dress sort of flow. Now, you do need a captured motion sequence of the character first. It's not going to just work from like a single image, but the code for this one is available on GitHub and available for research use right now. All right. Now, this is the last one and it's actually not AI, but I just

21:35thought it was really cool, so I wanted to share it anyway. This one's called World Gacha. I think I'm pronouncing that right. And it's actually a free browser tool from a Japanese developer that actually creates random fictional regions that has mountains and rivers and coasts and towns and railways and rice paddies and and almost anything you can imagine and it draws it as like this manga style line art. Well, this one's actually not in that manga style line art. This is a randomly generated city, but you can see if I add it with the lines, you kind of start to get that look. You could also put it on lines and

22:09tone and you get like a black and white version. You could put it on this like clay mode and this is again a randomly generated seed. I could re-roll the dice and it's going to build me a whole new town or city or whatever. So, now I can zoom in on this city that it just created. We can also change it to downtown. It's going to build us a new city that looks like a downtown area or skyscrapers. It'll build us a new city and these are again all randomly generated every single time and you can zoom in to different views. So, let's say we want like a top view, we can look

22:43straight down on this city like this. Maybe you want an eye level view of Central Street and it actually brings us into the city where we can look around and wander around in this city. Now, imagine using something like this and then re-skinning it with like one of the AI models we were looking at earlier. You can build a pretty primitive looking city and then use AI image or video models to reskin it into something really realistic and maybe even build scenes for like films and animations and stuff. Again, this is not actually an AI model. There's no outside data. It's just random numbers and math generating

23:17all of this. And if you do like a specific scene, you can actually export it as a PNG or an SVG or even a 3D object file. So, you could theoretically create a world and then pull it into Unreal Engine and build a game inside of it or something like that. I know I usually focus on AI stuff, but really, you know, the theme of my channel is just cool stuff that I come across online that I want to share and sometimes it's not always AI. But that's what I got for you today. Again, I make a news video every single Friday where I break down all of the news that came out

23:48in the AI world each week, but it's usually the biggest stories with the most impact on the most amount of people. I'm sort of trying to filter out the signal from the noise of everything that's going on and help you understand what might actually change what you do on a day-to-day basis. And in doing that, sometimes I miss some of these smaller research projects that don't get a ton of love and not a lot of people talk about. And I find all of this stuff really cool and really fascinating. And if you go back like 3 years on my channel, this is the type of stuff I

24:18used to talk about a lot more frequently until all of the big labs and big companies started having new launches every single week and I kind of shifted more focus to that. But I want to get back to my roots a little bit and make these videos a little bit more often. So, if you liked videos like this where I sort of dig and find some more underground things that are happening in the world of like AI and just cool tech research, let me know. I'll make sure to make more videos like this. I'm planning on making at least one of these a month because it's just really fun for me to

24:49dig in and find some of this stuff. But again, that's what I got for you. If you like videos like this and you also want to be super looped in and informed on all the latest AI news. I'm going to keep up with all of that for you. I put out a video every Friday with all of the news. I also put out a newsletter twice a week. You can subscribe for free over

25:06at futuretools.io. But, thanks so much for hanging out with me, nerding out with me today. There's a lot to nerd out on. I really, really appreciate you, and hopefully I'll see you in the next one. Bye-bye.