My agent rebuilt Slack

0:00So I've been obsessed with Cursor lately, and it's actually not for the reason you'd expect. And so today I'm going to walk through how I've been using Cursor, and I'm going to give you a tactical example so you can take this pattern and bring it into your own development practices. And it's actually not about building software,

0:16though what I'm going to do is very similar. And then I'm going to talk about why giving an agent a computer is one of the most powerful things you can do in an AI agent era. So in front of me, I have a Cursor chat, and I was actually building some motion video. Now what you see on the right is a Linux VM.

0:33Every Cursor cloud agent kind of comes with this remote workspace. And the nice thing about this remote workspace is that Cursor has computer use over that entire environment. So it can do things like maybe open Chrome, or I can take control here. I'll do that so I can open Chrome, show you what Cursor had been doing.

0:50So I had Cursor running Slack in my browser. That's a hint for what I'm going to show you all today. So Cursor can do all of this really cool stuff with computer use, and it can manage this desktop for me. And what I was thinking is, well, how do I create something like a video? Now, I create a lot of video, maybe you've seen some of it,

1:10and often we create video assets with code. And so the idea of video as code, of kind of everything being code, is that everything sort of becomes like software. And a video output, if we're treating it as code, the sort of conditions for it being correct, quote unquote, is does it render, does it compile,

1:31and does it look the way I want it to? Now, how do I check those things? If I try to render a video, I might check if it looks the way I'd like it to by visually inspecting it. Now, historically, this is something that agents and AI models were very bad at, right? Today they're much better. And what that means is that we can integrate these feedback loops

1:52into the video that we create as code. So what I want to show you today, actually, are the videos that I created with my agents to replicate the Slack workspace. And so I'm going to show you some renders, we're going to walk through these, of Slack that I built with my agents. And these are actually pretty outstanding. And the important thing to mention is that I didn't build any of these assets by,

2:22like, decompiling Slack. All I did was I told an agent what I wanted, I gave it a video reference, and then I opened the Slack workspace on the agent computer, and I told it to use those visuals as inspirations for what we want. And the agent was basically able to extract all of those visuals, copy a lot of the motion elements,

2:46and then implement the same sort of easing, motion properties, motion patterns, etc. And so all of these things were not possible at all a year ago, but what they highlight is actually a deeper thing. They highlight patterns for building with AI. And so that's what I want to talk about next. And then we'll go back to some more examples. So really, what I'm noticing

3:14is that there's a single shape for most of these demos. That's, first, I'm interacting with an agent via a chat, and that agent has access to its own computer where it has full control over that environment. It can take screenshots, it can interact with elements on the screen, and that can be running in the cloud or it can be running locally. Now, I find things running in the cloud to be awesome because I can have them

3:39run for a really long time without worrying about my laptop going to sleep. However, I've also been experimenting with my Mac mini and having agents run there. Maybe that's a topic for another video. But these things have the same shape. And effectively, what we're doing is we're building an inner loop and an outer loop. Now, I've used these terms before when I talk about certain types of software,

4:02but in this instance, it's slightly different. And so in the inner loop, the agent is checking its work. It's doing something, it's checking against something real, and then when it fails, notice there is a failure criteria, there is an evaluation criteria or some goal, it fixes that thing, and then it continues. Now, eventually it gets to a point where it's done,

4:23and at that point it hands the work off to me to check or to validate. And that's kind of where the outer loop kind of comes in. We're providing inputs such as references, preferences, and access, right? Ideally, we want to enable this inner loop to do whatever we can. In the example of Slack, right, I had to create a Slack sandbox.

4:44I had to give it some credentials. I had to log into that computer. That may or may not be against the terms of service, so don't tell anybody at Slack HQ. But we're providing inputs to this inner loop by which the agent's performing all this stuff, and then it's providing outputs to us. So there's really this outer loop and an inner loop. And what we can try to do as practitioners

5:06is basically automate as much of this as possible. And so what we want to do is do this thing once. So I showed you that agent that was doing all this work, but what I didn't show you is that I built this thing myself by just vibe coding and chatting back and forth with an agent.

5:22And then I thought, okay, how do I turn this into a skill? So we do it once, we codify that workflow into a skill. And then, you know, we can hand it off to an agent to automate it, or we can actually try to generalize it. So why do this thing just for Slack?

5:39Why not do it for X? Why not do it for different applications that I want to create visualizations for, or mirror, or create B-roll or assets for my videos, right? Then we generalize that workflow. So how do I teach an agent to do this for any product? And finally, we automate it in the outer loop,

5:54which is really what you're seeing here. This is the outer loop of automation, where now I have this feedback loop, this workflow that I've built just like a piece of software. And when I have that piece of software, now I can really just dump inputs into that inner loop and expect high quality outputs that I can then use. So it really is about these feedback loops, and creating virtuous cycles where then we can just like shovel information,

6:25essentially shovel context into the system and get outputs that we really want. And so this is not perfect, and it will break. But when it breaks, we have to ask why, we have to step back and reflect, and then we have to basically be engineers, right? So if your lamp breaks, I've used this analogy before,

6:48and it's plugged in number one, and it's turned on, and you check the bulb, right? If you check all of those things, then the problem's somewhere between the outlet and that light bulb, and your job as an engineer is to figure out why it broke. Our jobs as engineers are now to figure out why the agent failed. And when the agent fails, we have to ask why, we have to figure it out,

7:09we have to do some debugging, and then we have to encode that why as a skill. And when we do that, every future agent inherits it, and we effectively eliminate that class of problem in the future. And so this is sort of the process that I've started to learn in building with these agents. And I think this is going to be very key to the future of like software development, but also media creation, like every function, because essentially every function is kind of collapsing

7:37into interacting with agents and building software. And so it's this meta loop that we're building, and the core skills to that meta loop are debugging. And so we can take a look at our agent, right? And we have a bunch of assets that we just exported. So I can take a look here. And what I've told my agent to do, right, is as it's building with Slack, as it's updating these things,

8:03I want you to build a skill that allows you to do the same thing. And so we're taking the lessons that we learned in creating this asset and generalizing those so we can do them moving forward. Now, this is crazy to me because, you know, I'm very grateful for what I do. I get to come out and make videos with you like this,

8:23but my entire life has led up to this form of development. Let me explain. Everything that I do, I'm thinking, how do I do it a little bit better? And the way you do things better typically are feedback loops. So if you wake up, you know, you can abstract this to anything, right? Like I have plants.

8:39If a plant dies, you say, why did the plant die? Well, maybe I didn't water it enough. Maybe I watered it too much. Maybe it wasn't in the right environment. If I can figure out why, I fix that, and then the next time I have a plant, it doesn't die for the same way. And that continuous form of improvement, that feedback loop, or like sort of problem-solving, conscientiousness, encoding,

9:04is what makes you a good botanist. That feedback loop is how you get good at anything, whether that's strength training, playing the guitar, any skill you want to get good at. And then the other way you do it is by being around people that are good at this thing as well. And those same skills actually translate to software engineering

9:21if you understand the tools, and they always have. But what's different now is that they translate to agents. And what it means to be a good manager or a good person who works with people or a good practitioner, it is exactly the same, but it's magnified when you work with agents, and it really is about conscientiousness, attention to detail,

9:45and ultimately like habits in a weird way. And so, not entirely sure about these ideas. I'm kind of workshopping them a little bit and playing with how I present them. But what I try to do is record these things for you all and get them out to you so that you understand how I've been building.

10:02So hopefully this video has been helpful. And it actually shows you a practical workflow for how you can create motion assets like the ones that I showed you, like the ones that are playing right now inside Slack. And it basically starts with replicating that product, creating an environment for agents to replicate that product, check themselves,

10:24and then share feedback with you through this like inner loop and outer loop cycle. But that was a lot. What I talked about today was how to build visual assets like a software engineer, how to think like a software engineer with Cursor, cloud agents, and remote desktops, Linux VMs. I'm Matt with SpaceXSI. Until next time, peace.