Grok Bot on Construction Drawing Takeoff (Does It Compete?)

0:00If you've been keeping up with the latest developments of XAI and Grok bot, you'd have seen updates are coming thick and fast. And a major change recently, driven by an admission that they aren't quite at the frontier with the top models of Anthropic and Open AI, is that they now allow Grok bot in the back end to select models, including Claude models, namely Opus 5.5, meanwhile continuing to develop their top model Grok 4.7 until it's at a point where it's comparable with these models. This is clearly a smart move while trying to drive users to Grok bot and serve them the capability of the world-leading models. So far, I haven't tested any of

0:31XAI's capability on my drawing takeoff benchmark. So today, I want to put that to the test. Firstly, inside the regular harness using Grok 4.7. Secondly, directly inside the Grok bot harness to see how it performs against our drawing takeoff benchmark. I'm particularly interested in the Grok bot version because it doesn't actually tell you when it delegates to other models or more powerful models like Opus 5.5 as opposed to using Grok 4.7. And in addition, what I've been finding is the actual harness, i.e., the application built around the models, also has a big factor of how well these drawing takeoff benchmark perform. Let's get into it. So

1:03if you're familiar with the previous videos on this, you'll see I take a full construction information architectural drawing pack of a residential building, a 42-page pack including extensive detail of the layout. We have the measurements here such as the gross internal floor areas. Including, for example, the door schedule. So we've taken the measurements, we've counted the elements, and then we've produced a redacted pack which doesn't include any of the gross internal floor areas, for example. Basically, run this redacted version against various models inside the harness without providing additional skills or tools to the models. We benchmark how they perform in terms of the gross internal floor area, the

1:33counts of the windows, and various measurement metrics. And then we rank them against each other. Just a quick rundown of the results. We have GPT-6 Astra leading the way at the top of the list there. However, that one chose not to measure five of the apartments because it couldn't give the accuracy that it thought was needed. The ones it did, it got incredibly accurate. So

1:50that's why it's at the top of the tree. We'll have a rundown there of other models. And then Opus 5.5 not performing as well as perhaps expected. So we're going to run it through the same test and then we're going to see how it performs. So, first up a quick run down of Grok 4.7. So, it's XAI's most capable model, but it's not at the same level as the leading models now. For example, Opus 5.5 isn't even on here. That's up pretty much in line or above Fable. We see the different performances at different levels here. So, low, medium, high. We've been running these tests at

2:16high. So, we can see at high it performs around Opus 5. So, not expecting big results from that, but I'm interested to see how it performs inside the Grok Bot harness. Quick run down of the prompt. So, essentially saying you're a quantity surveyor commercial manager doing drawing takeoff from an architectural drawing pack. Can you provide the window count, windows per level, gross internal floor area, list ID? So, we can measure each outcome against each other and we use the prompt consistently across all of the different runs we've done. In terms of model, we're going to click expert. So, here this is the essentially equivalent of hard mode. Things harder

2:45to Grok 4.7. We've been running each model inside the harness using the hard setting where available. And in here, I've dragged in that drawing pack, the redacted version inside the chat window in the prompt. So, we're going to go ahead with that one. And now I'm going to do the same prompt inside my Grok Bot. Now, I've given it instructions basically to say, "Do not use any of the skills it has available to it or any of the team that we've been running in terms of the Grok Bot project." You may have seen some of my other videos on that. So, we can drive that comparable

3:08like-for-like analysis. We've got the same prompt in here and the same drawing pack. We're going to hit go on that and see what comes back. Run down here just in terms of which AI models Grok Bot powers. They're not very transparent in terms of how that is used, but what it says here that Grok Bot chooses the model behind each request and since it's not tied to a single model, each task it will review the task required and then it will select a model. So, you can't see which model it uses, but we see here it uses Opus 5.5. Now, my guess is it would use Opus 5.5 for this, which is a

3:34fairly big task for it to do. However, we're not going to know whether it's done that or not. So, I'm interested to see what the actual outcome is and we can maybe infer which model it may have used based on the comparison between 4.7 inside the harness and then now this by Grok Bot itself. It has one for each discipline, but for hard work it's Opus 5.5. For I think image generation. It's mid-journey. There's one for music and so on and so forth. A quick one while we wait for those models to run. If you want to know more about AI in construction, feel free to head across

4:00to my community construction accelerator where you can find foundational courses on AI, more advanced courses on AI use in construction, a pre-built library of skills and workflows. We're building out skills and workflows across each discipline in the construction sector and you can download and use them today. We also have unlimited one-on-one text support. So if that interests you, feel

4:16free to head across via the link below. Okay, so Grok 4.7 is back. It's provided measurements for all windows by the looks of it and it's had to go all apartments. So really interested to see how that one performs against the benchmark. Now it did that in six or so minutes which is much quicker than these models typically take to do this. Okay, so this one took about 20 minutes and you can see here there's loads more reasoning, loads more chat back and forth that you'd probably expect if it was to be using over 5.5, checking the detail, checking the logic. It can work out the areas it needs to just reason

4:42its way through it and then it's come back with all the measures. And here we have it. So we have Grok bot app pretty much identical to Opus. So clearly Opus in the back end. I'm amazed to see it that close to be honest because I think even running these tests on the same model twice you'd expect a bit of a variance, basically bang on. Then we've got Grok 4.7 via the harness down here performing pretty poorly. We could sense that by the speed in which it came out

5:05with the outcome in sort of six minutes. And interestingly, we looked at the benchmark house around Opus 5 and that just validates the approach XAI have taken which is we're not going to be able to compete with that frontier intelligence at the moment. So let's take their models and put them inside Grok bot. So that's a great example in my opinion of how that plan is coming together for them. For me that was a really interesting use case to see that

5:25strategy coming together from XAI. Clearly they know they can't compete on intelligence with the frontier models from Anthropic or OpenAI and they put that Opus inside their harness. You can see with the difference in those two comparisons why they've done that and how that's working well for them. If you found that interesting, want to see more content like this, feel free to hit like and subscribe. Really helps me out. And that's all for now. See you on the next

5:44one.

5:51Mhm.