Sub Club Podcast: News Corp's Data Strategy Can Teach Small Businesses

0:00hello I'm your host David Bernard and with me today Revenue cat CEO Jacob iding Our Guest today is Taylor Wells director of data products at News Corp having worked on data projects at deoe Disney and Business Insider Taylor now works on data strategy product insights and identity Frameworks across news corp's many digital products on the podcast we talk with Taylor about how to make better decisions with data the many pitfalls of collecting and interpreting data and why the best executive dashboard is probably a handwritten weekly email Hey Taylor thanks so much for joining us on the podcast today thanks for having me excited to be here

0:41so you've had a ton of experience working on data at deoy Business Insider Disney plus and now News Corp and I wanted to kick things off just talking about like what data science is why data I would almost separate that into two different things why I choose to beat myself up getting into a career in data or uh or why data matters Honestly though on the why data matters part I would almost start with instead of looking at it as what data can do for you it's more just like how badly can data be misused frankly these days I think that you see that across most

1:17companies so everyone when they think of how data and analytics and data science teams work if they're not in those spaces but are in corporate settings they often think of the problems that they've observed in the past broken reports or or problems trying to get to data or a lot of time effort and money spent on data related initiatives or things like that that ultimately don't show a lot of the value that they think that it's going to show so I think it's very valid for people to be hesitant or reluctant about the return on investment in data itself but I think that people also realize that data as a concept and

1:52as a raw set is essentially some of the most valuable assets or products on earth when used well so it's almost like atomic energy or something like that right like um it can be used for great things but you remember Fukushima and things like that instead so everyone knows what bad data looks like and the challenges that you have like they start a new initiative where they're going to build out a new data warehouse or move something over and then that turns into a slog there's now changes that were unexpected or it's even harder to do something takes longer than they thought or costs are crazy out of control and

2:27then you also have things like companies that go out hire a lot of data scientists or data analysts but they don't really know what they should be focused on and it's more an afterthought of like I'm going to hire great talented smart people in those spaces because I don't understand them and they'll figure it out as we're going and that can work but it so often doesn't I think that when data is driven by a business strategy that informs a data strategy then things are incredibly effective and you're able to really extract a lot of value and learnings out of data especially in data science in

3:03understanding consumer interaction and engagement in ways that aren't privacy violating or like invasive but are used to better that product and that experience for the customers no one wants a lot of ads on the page but they may need to generate that ad Revenue in order to support things but if you're actually making the experience better in order to have those ads on the page that are less invasive or the customer is not having to cover as much of that cost for the company because you've built things really effectively and intuitively internally then it can ultimately add value across the board that's how I see data and data science

3:42adding value generally so then what would you say is the foundation of a good data practice like data is just bunch of raw numbers right well one thing I would say is that more often than not companies start by saying we should just be collecting everything and then we'll figure out what we need right so there's a lot of challenges around that but there's also the challenge of even if you have a honed focus on what you want to collect say I don't need to record a viewport of the user but I at least need to know when they click on a tile or whether or not the video plays

4:18whenever they click on the play button without having structure around how you expect that to change over time to grow other use cases that are likely to come you inevitably fall into the Trap of building something that will have to be reversed or modified or adjusted and each step of that exponentially increases your cost catching something early on could be 1X of your cost and then 10x if you're fixing it by clean up Downstream because now you've got new data coming in that you'd have to continually add new iterative processes to fix and then lastly at The Last Mile maybe 100x more cost if you have bad

4:56reporting that are actually telling the wrong things because you all this framewor yeah make some wrong business move it's basically like when you look at data data should be looked at in the same way that you're engineering a building you're going through the process of permits and agreements and Alignment across the board on what things are going to be called are we going to call this a wall or are we calling like the wires within the wall like the conduit or are we calling it this so that if someone says that term later we all know exactly what that is referring to so it's a lot of getting

5:25that alignment and then thoughtfully laying out what you respect both the current version and the potential modularity or general generic nature that it needs to be in in order for you to continue to scale that out or use it for different use cases and then lastly you need to really have a deep understanding from the data team side of what the business is trying to do even further than what they're asking for today but also what the product does itself and then what your output looks like for the user you often see people that'll build dashboards but they don't NE necessarily understand how the dashboard is even going to be consumed

6:02they just know the generic requirements of the dash and therefore the dash May technically meet the requirements but is a challenge to use or misses the mark on what you're trying to uncover everybody looks at it right and on the flip side of that you may have data Engineers or data scientists who are trying to understand Behavior based on how that's been described to them in the problem statements or the requirements the tickets whatever it is but they aren't regular users of the application and therefore don't understand how that works and it doesn't necessarily even mean that just heavy usage will fix that like on Disney plus if you're a heavy

6:33user but you only have an Apple TV and you're saying this is the behavior that I understand which is very different than a low-end Roku device or a phone and how they're engaging with a phone like a swipe all the way down to the last row versus having to use the remote to do that then you still may not fully understand how the product is being consumed right so that deep understanding of what the Upstream is what needs to come out of this and what are the business teams within it trying to accomplish both now and in the future yeah I think you we see similar problems

7:04all the way from like Indie developers with just the basic fire-based analytics just collecting a ton of data mislabeling that data and then making bad decisions based on data they don't fully understand but then that propagates up at every stage like you move from Firebase to amplitude with the same mistakes and you move from amplitude to like a custom data stack make the same mistakes and you have a whole data team propagating the same data mistakes what are some like concrete things you've seen in kind of what data to collect and how to think about that data collection I'm curious on that spectrum of collect everything

7:40versus things that are semantically important obviously like I think both extremes don't work but collecting everything has cost implications right and you're just drowning your data team in noise typically so like how do you navigate that Spectrum right and that frankly I think one of the more hidden problems too is that when you have all that data it's simp similar to saying what are the lunch options and giving them two options to choose from or three at a meeting versus bringing in an entire catering staff that can make any food now you've opened the door to like well I didn't actually want you to order an omelet we don't have the equipment

8:14for an omelet but now you've ordered it because I've said that you could do that right as well as whether meaningful non-harmful But ultimately like violating data or willfully violating data privacy and usage you know if you're collecting all data and listing it for example on the EU by their gdpr constraints as functionally necessary because it's an internal tool that you also use to make sure that the site is up that the pay wall is working that consent is being captured Etc and you're flagging those so that they can't be used Downstream but you're not gating that to where it technically can't be used Downstream as well or masking it or

8:49protecting it or omitting it when not needed then you now open the door for a potential risk now that data may be used by marketing team that is uninformed and how it can be used to now Target somebody based on geolocation or IP address or something so I think that the balance is to go back to the original question of like where is the Right Mix if there was a clear answer to that for an app versus a website versus a cell phone or whatever you're manufacturing then it would be more standardized across the industry but the fact is it really goes back to what is the company

9:25trying to accomplish I think that even in our earlier talks we spoke about how data in itself is not a value add to the business collecting it storing it having that is not a value to the business it's whether or not you're actually using it and turning it into meaningful insights that impact the business not even insights just for the sake of insights which is also a problem area where you get kind of that 90% of the way but it's not actually tied to the decision makers or the right levers aren't available for those people so I think it is literally a process of not trial and error but a

10:03full understanding across data and that's why to kind of your example that you gave earlier of how it just continues to work the problem up in different ways if you're starting without a foundation each time that you restart and wondering why it's not working it's because you have to slow down and set that Foundation have a full understanding across those teams and then everything else becomes exponentially faster and easier but so often companies do not do that and frankly you know I get to brag or hang my hat on the fact that Disney plus was not only such a great success which a lot of that's a mixture of things

10:37happening covid and things like that but it also was a great luxury to start from scratch from a data perspective on an app that we had the full support of the company both financially and backing it and pushing it as this is what our future will look like this will be a key factor as much as Parks or anything else but also that it was they gave us the runway to launch the app a year and a half at now you see these streaming apps come out after like two months of being announced because they can kind of burn through that but that opens the door for

11:13a lot of problems too or they try and bolt several together and then they're building off of Legacy systems with Disney plus we were starting from scratch ESPN plus Disney plus we got to really go in and say what do we need to collect how is this going to work across all the devices what are the limitations of each device that we need to factor in so that we can normalize some of this information and ensure that we aren't tracking something in one way on one and not that's what I mean when I say semantics like when you're coming up with a tracking plan it's like thinking

11:42about the semantics and how can you compare that because like to some degree you don't want to compare an action on a phone from an action on an Ott device but in some cases you do probably like a watch should mean the same thing yes right to some degree you made the point of like some amount of trial and error and guess work and that's kind of what I've found historically is like for Indies and whatever like when you're thinking about that first iteration of what do I care about just be logical go through and use your app and think about the things you do that you think would

12:10be interesting to know and put those 10 or 20 actions in and that's a good place to start and then I've always found it's usually if you miss something it doesn't take long to go back and add you know you might lose some historical data but also you keep making the point that data as collected is in some ways valueless right it's like crude oil or some other raw asset like until you're refining it and actually actioning on it it actually has zero value so like don't worry too much about hoarding everything because I've seen people do that too and like it can be used but as you said like there's

12:43implications for data regulation and privacy there's also implications for costs like running a data if you have your own data warehouse or even just running through all of these events platforms they usually charge on events some of them charge on users which is nice because then like if you have a user you can track zero or 100 events which is nice but they're not all like that and so yeah just being somewhat logical it's a good start yeah to that point I think that there are ways to structure how you're approaching it in a generic form where you can ultimately not then hinder changes going forward

13:15where it has changed like what you're trying to collect by having that framework that actually allows for modularity or scale so one example would be if you were planning to track how many people clicked a download button in a streaming app to see how many people actually use the download feature how many downloads are completing you might build an event that is download clicked download started download paused download completed deleted whatever but instead if you're focusing generically you can say I'm going to build tracking for button clicked or click action and then give keys to those buttons ensure that there's structure in how that's built so there's a bottom button that

13:56has to sit within a container that sits within a set p and I've those hierarchal keys built up so that now when comes and says well now we need to know when they hit download and then pause and then share and then download again oh that's a new function well now I can just say that either way those were clicks of those actions and I've mapped that behind the scenes to what those keys and tags represent and that's kind of like how you ultimately can say we don't necessarily need to know everything up front but we haven't backed ourselves into a corner where we were tracking

14:28something is down click before and now it's button click and I've got to merge that data and clean it and normalize it for pretty easily I mean let still a pretty far way down the semantic meaning stack versus like tap on x value yv value you know what I mean which like you can Theory could collect what does every single tap event on the device and that's probably pretty useless but you know hooking into your like generic button class and making sure you capture all those I think about some of the tracking plan for Revenue cat is like we did have semantic stuff like created an

14:59app created a project which just creates you know going to the conversation about Downstream consumers if you have an event that is like project created that sets yourself up for much better self-serv use cases because most people understand what that means and they can probably pull a thing together and it's most likely going to be correct but we still also have a page loaded event right so like we know when a page loads and we have the path broken down and in some cases we've been able to retroactively without going in and re-engineering a semantically tracked whatever we're able to go in and retrofit that in which is good and

15:27that's still like you know there not that many page loads right there's not that many button clicks right so and honestly like as things mature over time you really figure out what you need to be focused on for example like do I need to know that all the icons of the home menu were shown when they clicked the word home I can assume fairly safely with QA testing and things like that of the app that all nine Tabs are showing I don't need to confirm that nine were there every time but to your point some of that can also be done within the stream of data and you can say that this

16:00button with this type that's being clicked has a key that represents that and as that data is being pulled in I'm enriching it to then classify it as this is project started or app created and send that to the teams that it's relevant to while still being able to reverse engineer the raw data that you collected before but we're companies I think most people think of GA Google analytics and others when they think of these things is it collect especially Adobe each event is collecting a hundred pieces of metadata across them yet they're almost all identical for that session or that user or whatever and in

16:36Broadways like for a lot of the interactions offloading some of that to essentially just capture when any of those States change and then being able to apply logic Downstream to how frequently they're resetting a password instead of just saying I'm capturing password field every event don't recommend that one don't capture your password Fields official security advice so to start collecting data one of the things you and I talked about before the podcast was how data is really just a symbolic representation but that information is what we derive from the data so tell me a little bit more about what you meant about that yeah so I

17:17think everyone also knows that's had a data set anywhere that exacts will often hone in on something and then there's not a lot of questions asked as to whether or not still needed is it tracking the right thing are we looking at the right thing and are we making any decisions off of it I can go into a theoretical company every week and give a weekly status that shows how many subscriber signups there were across the app or how many page views or whatever but the question is rarely asked are we doing something with this information each time and if so what are we retroactively doing it do we do it every

17:50week would we like to do it more frequently are we able to do it more frequently and so that's kind of the value that ultimately eludes companies around data very often what do you think the bottleneck is there cuz like you have ostensibly smart people consuming this but like there's failure to act is that because there aren't things to do which I think is maybe even a valid path what's your theory for when that fails yeah I think it depends on the company's size frankly so at a larger company what you often get is people not asking questions because a senior asked for it and they don't want to push back and so

18:23you're building layers in between to the point that the people that are actually executing on building a report or surfacing a report or interpreting a report can't even necessarily get some tells me it's a stupid question like just that would save me so much time cuz like that's actually probably what the Crux I'm trying to get to I have a curiosity about something and so I ask oh can we pull this together if somebody who knows better than me of why this data is trash could tell me before I spend all that time with or knows what you'll ask after you get there and just gets you what

18:56you knows the why of what I'm trying to but you're right and I'm sure it depends on the culture you have as a company as to like how much push back you can give and stuff like that right and again I think a lot of these are rooted in some foundational things like do the employees of the company across the board fully understand what the business is trying to do I've worked at too many companies where it is unclear to all how all of it comes together what is adte and marketing doing what are the business development teams doing what is strategy focusing on who are our core

19:28customers even at a prior publication I found it incredible to see that when asked what Their audience was who are your readers who are the people that you're trying to reach but like who are your readers today that they would have strong opinions of who those are but they were based on the assumptions of the executives right like maybe it's even the CEO our readers are techsavvy Northeastern or West Coast they don't want old Legacy Media blah blah blah you know like whatever the definition is yet you go back and say what is this based on are these based on personas that you've created from data science is this

20:04customer interviews and like working sessions for how the product is actually consumed have you done polls do you send out newsletters and you see what they're responding to and then you're profiling it it's often no it's just like that's what I built it for originally and I'm assuming that that's who my audience is cuz no one said otherwise so it's really just like breaking that down of like if no one's pushing back to say we can't Define our customer based on nothing based on just our assumptions then things continue to only actually like exponentially do that who do you think like for data teams to understand what

20:37the business should be focusing on and like how do you think the best way for those folks to get that information where should that come from absolutely it should be a coupling with data product frankly data product is that weird space right it is not a product team in the traditional sense and I like to think of myself as a product person can you define data product a little bit yeah exactly uh but data product kind of lives in between I've seen it live under product and Engineering I've seen it live solely under product I've seen it live under its own umbrella under data Like Chief data officer potentially and

21:07then I've seen it live in some hybrid forms or embedded models like a hub and spoke where you have different people from the data product side working closer with the businesses while other product teams also are involved so think you're developing a new feature like 3D video for a streamer you may have that team that's conceptualizing the 3D video that they've picked pitched and sold Executives on you have a data product person or team that are working now with the engineers alongside that to understand what that product team envisions for that how they actually expect it to happen how Engineers actually expect that it can be built and

21:42delivered in order to inform how they should collect and capture that data and how it's going to impact all the stuff that they actually sit on top of Downstream and then they have to take that message and understanding back to the data science data engineering and data analytics teams to explain that sell the vision of what we're trying to track get feedback on how it could be better done or how they Envision that it's most efficiently done what are some pitfalls Etc and then lay out that road map of features you know like in a Jirus sense you know like the work that needs to actually happen so you're saying like

22:17the folks in the company who are responsible for integrating data into the tip of the spear product features and stuff like that they're learning through osmosis or direct interaction with the producted leaders engineering team building who should ideally have a better understanding of why they're doing and that often would not be effective to try and ask them to understand the data part while going on that in fact it often never works if you were to go to a product team and say put data first in your mind when you're developing a new product idea or a new feature they will still wind up even if they're coming up with kpis and metrics

22:51that they're going to track to measure Roi and success down the road they still wind up going oh I forgot about that cuz I was so focused on seeing feature come together in wireframes tests blah blah blah that now it's live and we don't have any data for it and I've actually seen that like even at a publication doing a like a podcast or a streaming audio where they'll partner with a third party company build this thing or this idea we all know data is important and that we'll need to understand the data about it but at the last second you're like now this is actually already been

23:20pushed out live and we're not collecting any data on it oh no well what do we do how do we do a little bit before that usually I do my Data Tracking like in the last day right which honestly kind of fine cuz like you don't know instrumentation these days is at least for the like Indie case you're using like a tool or something like that can be done pretty quickly but yeah I also you said teams coming up with kpis it's interesting I've watched us build sub features and things like this that we design very bespoke and interesting kpis for and a kpi's usefulness goes down proportional

23:51to the amount of time you spent thinking about it like absolutely like the best ones are just like how many people use how many dollars through feature and I almost wonder if going back to interpretability it's like if a kpi you need PhD to design it you probably need at least a masters to understand it and like let's be frankly most people don't understand that stuff so don't overthink it you know like in fact it might be yes don't don't overthink it as it starts to grow um a lot of my examples too here are companies at large scale that have a lot to lose and a lot of Investments up

24:27front in terms of like have a data product person who's bringing stuff back and like most companies it's either going to be well startups it's going to be the founder maybe like the lead engineer maybe if you're a little bit big you've got somebody who's like data literate but they're going to be going across the stack they're probably responsible for the technical stack but then also probably the instrumentation and like all of that stuff and I've been that guy before where I've had to be like jumping up and down being like hey like you know we don't have to do a lot but like let's make sure we collect the

24:53basics here the beauty is I think today there's enough tools that you can get 80% of what you need with some of this offthe shelf stuff but yeah I mean even at Revenue cat like David and I were talking about build versus Buy on the way over here in terms of like even just like an event tracking platform and we like yeah I don't know why anybody would ever build turn I was like well actually today we're almost kind of building our own internally like because like at some point you begin to rely on like usually an internal bi system and you probably have a data warehouse and at that point

25:22it's like well tracking event streams is kind of trivial at that point right and there's some customization advantages you can get yeah I'm really curious your thoughts on that cuz you and I were talking about your time at Disney plus where it was Zer era there were high hopes for Disney plus and there was almost a blank check for what you're able to build inside Disney plus and I'm curious you know it sounds like it was ultimately very successful being able to ground up but very few companies have the luxury I think you told me at one point there was like 300 engineers and multiple teams across the company all

25:58working on essentially building an entire analytics company inside Disney and that was really not even outside of Disney direct to Consumer that's not taking into account Parks cruises yeah so so yeah so then what are your thoughts on the build versus buy and blending the two and when it actually makes sense to start building out your own data product versus relying more on off theh shelf what are the benefits and drawbacks of each you know then after hundreds of Engineers building this now you still need tens of Engineers to maintain it over time very few companies especially in our space and subscription apps have the luxury of dedicating teams to that

26:42and then if they do maybe those 10 Engineers would even maybe be for a dollar R&D spend like it's almost doesn't really make sense right yeah like like their time would be better spent on consumer facing features versus like building infrastructure so I've yeah really teed up a long question no no no I think this is actually going to be one of the one of the more succinct ones for me I think it does depend but I would generally lean towards use off-the-shelf use open source things like that early on if you're not expecting immediate massive scale unless you just think that you've like struck the motherload in terms of an idea for

27:19an application or a website and you could there is a 10% chance that you could suddenly have a 100 million users or something and now you backed yourself into a vendor corner from a volume perspective or pricing then start with something more basic but with a clear evaluation structure around when you cut over to other things and a culture that both acknowledges and accepts that it's okay that things may not be congruent over time when you switch one of the biggest pitfalls of every company that's existed for more than a couple years that I've been at is literally that they say well we've already started to

28:03collect this and if we cut over to something new it's not going to match the old reports or I can't see if we start geological layers of like I mean imagine the Panic that's been going across the industry in publishing where it's not Tech focused for the most part about the switch from ga30 to ga4 ga360 lived on for years and years past when it should have when Google said that they would kill it because customers were just so concerned about my bosses are saying that this is a completely new way to do it and they're not ready to start looking at things from a session

28:37perspective versus an event perspective or a device perspective it's a see change in the way that we track it and we know that it's better but it means that now I can't show year-over-year comparisons that actually have the charts that we've relied on for years are now like IM balid they just don't make sense in this new paradigm yeah it's very painful organizational change right yes absolutely even think about something like for Disney plus the idea of the bundle was introduced very late in the leadup to the launch we had an idea again we I'm kind of just referring more to like non-executive levels that something like that would happen but we

29:10also thought that it was more likely that it would just become subsumed in some way in either direction like maybe it's Disney plus merging into Hulu or vice versa but that you end up with one app and I still think that'll ultimately happen but at the time the idea of needing to synchronize user access across who and Disney when someone changes to a bundle option that was not present the idea of ads were not present so like you didn't really have the ability to figure out how that was then going to look on a report and it kind of like as long as the company culture was

29:40setting the expectation of like I don't actually care if it translates over time then you can win but also again startups have the luxury of saying we don't have any historical data so as long as we're moving quickly and we end up in a good spot in the first couple of years then you can usually establish a good practice going forward and when those businesses are coming back and trying to challenge like that but I want it this way just pushing back on what value do you get out of being able to translate that if they have all historical clicks of an article for the journal or Business Insider going back

30:14for 10 years so that they can show a line graph of 10year month over Monon growth down to like a section but the sections have changed over time and now they're frustrated that the reports aren't going to be congruent say what value are you getting and comparing something that happened 10 years ago when the market was absolutely different going up it's just nice to look at charts ex yeah yeah what's wrong with that but they're not pushing is that not a valid they're often not pushing back on the they're often not pushing back on the boss to say actually like what are we going to do if we see a big change you know it's

30:47like a photo album I just like to be able to flip through and be like Oh remember when we were at this much but you know it's a really good point I mean I think about our transition from off the-shelf amplitude to like you know when did add an ETL or like when did we add a phase where we could like transform some of the like evented data into something more semantically meaningful and yeah it kind of happened when we started to break definitions and not just break them through like foolishness but through like okay the app is different now like I think one big example for us was like we sort of

31:17refactored what it meant to be a platform versus like different platform anyway big structural change that we did stuff and suddenly like X did not mean X anymore right and like to capture that sometimes in a third party tool like amplitude it can be very difficult because like the operator of building charts and stuff has to have that information constantly loaded in their memory or they'll pull up stuff that doesn't make sense it's non- congruent right and that's where for us it made sense to introduce some sort of pipelining Step because then what we can do is encode that change into a pipeline and be like okay like in this month of

31:50this year we change this definition and create some sort of like normalized super definition that takes some of the sharp edges off right for for data consumers but again if you're just launching your app like you don't need that like that is a luxury of a problem to have and to your point early you're not going to launch that app that has 100 million users in the first year and anyway if you do and you owe amplitude a bunch of money you don't care you're like great like one fire I don't have to worry about right now they'll cut you a deal probably right absolutely to that point though of like

32:22at scale where it can be problematic if you do have success is you know when we built glimps internally and Glimpse is Glimpse is the internal event structure of clickstream data within Disney plus which is still there today at least as of the last time that I'd looked so those events were that generic structure it was named Glimpse it's an acronym for Gathering live interactions via multi-purpose surfacing of events were in that meeting somebody felt really proud that was me congratulations and I had come up with about 40 of them like one one night of like what are we going to call this and by the way there's a

32:57reason that I did it it's because my boss had been calling it uat for so long user activity tracking and I was saying uat means user acceptance testing we can't confuse it you got to call it something else so I finally said I got to come up with an acronym you came in pitching like a director like pitching a new movie you're like storebox at the time the the head of data Laura Evans she's fantastic she just said I don't care Glimpse is like sent a big list of them oh you know they care that's impressive preg GPT for to be able to come up with such a fancy

33:35acronym oh it would have saved me countless nights while my two-year-olds were asleep and should have been getting a bedtime story countless nights of me coming up with ridiculous AC I mean we we're joking about it but naming and interpretability of this data really matters like it really matter I mean even a cute name for a system helps like it helps people talk about a thing and like we I'm battling this now in some cases in terms of data a lot of times in terms of like how we explain the product and it's not always like yeah at a high level it's easy to explain our product

34:06but like a specific aspect of our product like you need a surprising number of names for things and the consequences of choosing those names is a lot higher than you think right I mean I think design UI product marketing all of that are so underrated frankly these days like just because it's AI is the focus and people want to cut cost and that's often an area that they feel like they can cut cost because now every app has been built like in several different ways but there aren't many like new ways that people are coming up with things they think oh I could just use like

34:37standard practices and such and not have to worry about it but branding is incredibly important think about Apple intelligence and how much we'll be saying that term for the next 5 to 10 years right if it's successful versus if they just stuck with we added AI in here because no one's going to continue to call Google's services with Gemini it'll more for whatever or they'll just never say it they'll just say it's Google with AI right but to brand that in a way that Windows did with co-pilot right now you see that ecosystem expanding to now their Hardware is as a co-pilot driven laptop so I think that it's so important

35:12but it also then like you said allows people internally to discuss things quickly without having to go back to like where is this so when people talk about clickstream events here they have no necessary standard source of Truth in their mind what that means is it GA is it parsley is it Adobe Etc or is it our internal events but at Disney plus it was the Glimpse events which everyone then knew to mean these activity tracking events that are used for product analytics business decisions Etc and then over here is dust which is like our service monitoring events and I don't know that one predated me from the

35:49BAM Tech days but Glimpse was an important branding for that and what I was saying on the cost piece is when we built glimps the reason was that we wanted to only have one integration within the app we didn't want to rely on third parties then it became kind of sticky in terms of gdpr and consent and things like that we'd rather just say the data that we're collecting within the app is only going to Disney and then we use that internally in the ways that we Define but the moment that you add Adobe or GA or anything like that you've now opened up things like competition in

36:19terms of them being able to understand how much your volume is increasing people to monitor things and look to find insights about something versus if you're just ENC just like even to add a third party processor like you have to go through the legal rig roll internally as a data team you have to advocate for that like and all that stuff you have to maintain the DPA and make sure all that stuff's up to date like gdpr really I mean I actually think the requirement for disclosure of third party data processing was good I think it was one of the things that came out of GPR that

36:46was really pretty well done and like maybe could have been more simplified I feel like there's a lot of DPA theater and this is like very much like analytics company problems but like people just need a DPA it's like all right okay you could also just tell your customers you're sending data to us and what you're sending and everybody will just trust everybody but I guess we'll have to do this little dance I get it now and like the nice thing too is like the tools for Rolling some of this stuff yourself are becoming you know between the data you know just talking about AWS like the ways for you to like store bulk

37:15data in AWS the tooling on processing over large flat files it's gotten so much better tools like DBT have made it much better to like manage like SQL pipelines and then the database options have gotten so much better too like when we when snowflake yeah yeah snowflake has changed everything even red shift has come a long way from where it was when we started using it a decade ago click house I've not used personally but there are like options now right and so I still think like when you're at day Zero do not do any of that stuff unless you're an expert and you've done it and

37:46you know exactly what you want and you know how to set it up very well quickly and cheaply but I think the time to it making sense to actually use this stuff is actually much earlier than it it's ever been in that case it was built because of that but it was also built because of a concern that cost would exponentially increase in a really successful scenario even though the bar I think was set very low early on I think the original Target was 50 million to 60 million users 40 to 60 in 3 years and we hit 10 million the first day and 100 million in the first quarter but to

38:18that point they also had Adobe as the backup plan right so let's build it internally but we've never done this before in this sense at Disney so let's also in the back have a contract with Adobe and be integrating it lightly but able to pull it out later right um in the first couple of weeks we were able to show that the Glimpse events that we' built were running us around $1,000 a day in total like end to end cost and I think the Adobe was costing us about 33,000 a day and so we were able to quickly say let's rip this out because the data is reliable it's working like

38:52we're conf you have it under control and if you have the right people who understand it to respond to like enhancement requests and stuff like that that I think it can be very superior but like on the cost point it's like if your product generates any Revenue like marginally per user usually the like tracking and analytic stuff is rounding air I mean when it's rounding air you know and when you're talking about like billions of events a day for example versus like we were in like another level on that but like I think that if Disney plus had launched incredibly slowly or or anything yeah like NN plus

39:26I don't think that we even had Adobe integrated there maybe we did but like in that case it would have been fine to run that and not have to argue that like oh but we should move that internally immediately the volume wasn't there at first and but even 33,000 versus 1,000 isn't the actual cost because you have what 10 Engineers all in cost of you know 400k I mean if you're paying if you're paying your uh your resources on your team and things like that $32,000 a day combined to make those two even then I'd love to work there 32 $333,000 a day adds up really quick and then you really

39:59can't argue that you're getting any value out of the data because what on Earth in terms of insights would you be producing off of that data that would actually be worth that 336 Millions right it was a lot of money if you had just continued with that and by the way that was our early estimates we added triple quadruple the number that's I think one of the one of the you know locking into a tool stack that has constraints like that I think it forces you to not do things that might be helpful because because of the cost and that stinks like cuz sometimes if you

40:30really want to do something and you're like oh I can't because mix panel or whoever is going to charge me an arm and a leg then like you know which you also run into with your own stack sometimes but again it's like sometimes you can just bring events into the first stage of your processing and leave them there like you're like we might even not pull these into snowflake yet but we're just going to let them Stack Up In Cold Storage and like if we ever need them we have them you can make those decisions right and control costs quite effectively it goes back to the original

40:56kind of question and State which is that having those checkpoints or evaluation points understood across the business and the teams and then adhering to them can actually prevent a lot of what seems like a risk right you can say we're going to have Adobe in here but if we're moderately successful we need to evaluate within three months whether or not the continued Cost needs to start being offset whether or not it's that we're scaling back some of the events or whether or not we are adding additional tracking at a certain rate or you could say if we're wildly successful we need to evaluate within a month how quickly

41:31we can start to deploy an internal solution that can offset that how much of an overlap time that we need of like a month for comparison of the metrics or something to ensure that things you put in worked and then turn the old one off but as long as you have those then the process is actually pretty easy it's just that they don't often plan those and they'll say we're going to buy instead of build and then it becomes well we bought so you know like we didn't think about the scenario oh well we're here now it's just going to be an inherent buil-in cost that I have to now

42:00raise my price or do something different to offset and find it elsewhere but in reality you could have just said no we already had a plan for when we' check this and now we checked it and it didn't pass so let's pivot I uh made me think of an example where a company I worked at where we did the opposite where we started off with like a very Bare Bones dumping logs into a a SQL database and we actually moved everything back to I think it was mix panel in this case and the reason was is it was too hard to play with like you couldn't play with

42:27yeah at the time like Tableau wasn't where it is looker didn't exist like there wasn't really a world where you could easily layer a bi tool at least in the startup Mark like something a small startup would pay for like you could layer on and that was prohibitive and you know cuz like we just weren't asking questions cuz like my data process that's a big part of how I'll use data is like I like to just explore like just play around just like if you have like good based data and like the semantics are mostly like understandable and if 75% of the answers you come to are valid

42:55right like once in a while you'll pull a query like well it doesn't make sense cuz you know whatever I find that to be an incredibly powerful tool and I don't know I might be an exception in the exact world of somebody who's like very you know you are from a you are but you're one of the exceptions that's actually on the right thing yeah but I feel like it's got to be on the rise right it's like more it's more at least at least not what I've seen uh think think about it this way like as the sea Suite becomes more and more that they

43:23started their careers when SQL existed yeah like it's likely that data literacy becomes a more common thing in the boardroom I mean I would say you're right in that this example would be Michael Paul was the president of Disney Plus at the time he was one of the types of people who would say dashboards are meaningless if it's just showing me the same stuff I'm now losing the thread I don't know what I should be focused on and I'm continually adding to that stack of that if you're thinking about like printing them out I'm adding to a stack of what I have to go through each time

43:55but it all looks fairly the same as what it did the last time that I looked at it so what exactly is it telling me and it got to the point where near the end of when I was working on data products there was I was tasked with writing an email each day by noon that just had three bullets and up to two sub bullets within each of the bullets just an email that just said here are the three things that you should be focused on that happened the day before here is the change and even if that's here is an interesting Insight that you should be

44:26Focus on that's based on something where I'm looking over time or something but I'm abstracting that and they have the trust to understand and believe that you're surfacing them the right things then suddenly dashboards become absolutely irrelevant frankly to me like I love a good dashboard it's a data person I build dashboards on my own personal data you know like I just I I love playing around with it but if I needed to get value out of it before saying that like is it worth a time and investment then I'd probably say no it's not worth me me charting how many iMessage emojis I send to each person in

45:00a list or something like you know some nonsense but companies will continue to track that and say which are the most used emojis right instead of just saying does it actually matter does it cost anything to do that and they're handing that report every month every day or whatever so once you actually can raise that and it starts at the top of I don't actually want to look at dashboards anymore I don't want to look at reports and I don't care about month over month if it's not actionable I'd much rather you tell me that we're actually seeing a weird Trend where people that log in

45:31after 1 p.m. on weekends wind up being our longest tenured customers over time or people that sign up for the bundle annually wind up using it 40% more than other consumers then that's what I actually want to know your dashboard right right because then I can I can aim business teams at those problems hey we need to go promote more this annual bundle we need to go aim at add campaigns that run in the evening because we're missing that market Chek after him even made a comment at one conference and said one of the craziest things that he found from the data at Disney plus was that more than 50% of

46:09the audience was single adults I think even broadly adult male but the Assumption coming in was that this is going to be a kids app right I mean that was like what consumers kind of interpreted initially and then they saw that it would be more than that but it also was never planning to have R-rated content FX content Etc what are you going to do with the fox stuff so he was surprised to see that this Disney branded family oriented thing was surprisingly used by people that didn't have kids and were adults but that's the kind of stuff that you can unlock from those insides that you find early on and

46:44then you can adjust but if I'm just showing you a report that you asked that was saying show me XYZ and it wasn't focused on how much has this thing grown into a tool that's primarily used by what we think are adult individuals that don't have children then yeah it could be 6 months before you actually realize it and you've missed out maybe that's when another competitor came out and ate your Lune you maybe think of like a neural activation right like the dashboards are raw input but like there needs to be like a filter you would think that AI would get there but like I'm not sure we're anywhere close to

47:15that because it's like I have a bunch of Dash we have our reporting for like board decks and all that stuff it's a little bit more formal but then I also have my own crazy dashboards that are like not super interpretable but I know what they mean they give me signal but again it's like I don't really care how many API connects we had yesterday I care about that piece of data in context with everything else on the dashboard in context what I'm seeing in slack not budgets people you know like all of it but yeah but it it's like if a human at this stage still is not there to like

47:47synthesize that into what you're talking about those three bullet points which like I don't Sy anybody because they're for me sometimes I'll send them to the team and I'll be like look look at this the value is questionable and it's interesting to think like this is just speculative but like will we be able to in data land go I think that is like the holy everybody says they want to like generate insights but I'm not sure we're anywhere close to that I was going to say I think my biggest takeaway from this whole conversation is that the absolute best bi tool that you can build is a really smart human with access to

48:16good data who writes three bullet points a day in an email and who isn't being asked to do things that don't add value right like who is challenged to actually get out of the normal practicing data because I mean some of the bullets that we would send are there to give that context right like it could be hey we had a 10% increase in signups in the EU that may be on a main Dash but drilling into that may be and it was primarily in the Netherlands yesterday and that's because we launched a new promo in the Netherlands overnight for 3 months free but we spent with a target of 20%

48:54overnight so we actually performed poor but you would lose that at a typical dashboard level by saying um hey look things are going well but in reality you lost the money what other anomaly like and how these questions morph over time someone may want to know how a Taylor Swift folklore exclusive service that launches at midnight does for the next 24 hours and then going forward they may say now I want to see how Beyonce's does when we launch that and then I want to Stack it against how well movies are doing versus those exclusive things and ultimately like what was the budget how much much promotion did we do what did

49:28we initially expect and did that one actually outperform but it had 3x higher budget and expectations and it actually underperformed right like I need something that can kind of synthesize that and I think that I may have a counter viw of that and that I do think that with good data and well-informed AI that you actually can reach that it's that you would have to have the patience and commitment of teams to both get out of their comfort zone know that things can be wrong sometimes and just accept that you have to double check a bit but that ultimately it will get you to that point I've even dumped credit card data

50:06Bank data and things like that into GPT and run with its new data analytics model things like look for insights of like what jumps out to you graph these by categories Etc what's changing and it can do a fairly good job of saying hey you're spending more on restaurants late in the month or in the summer your restaurant costs double but your factoring a budget of X across the year as a standard thing so you should have a seasonal budget for these things because you're going out more for conferences or what you know like it's just nice out it does a decent job of not only finding

50:37those but then having the memory to add that information later and continue where it left off and say yeah maybe I'll be wrong maybe I'll be wrong maybe I'll be wrong I definitely would not bet anything on it currently but I think that it's actually surprised me in the last couple of months versus where it started about a year ago there's a blindness there where you're always assuming assuming that the thing you do will never be replaced by AI yeah exactly once I saw a like C3 AI demo that's a Anderson Horwitz one it's solely geared at AI for analytics within Enterprise corporations synthesizing all information citing it sources and being

51:13able to look across all these things without people having to like do the ETL and all that I was like this is amazing oh no where do I go it's good to a beach somewhere yes so no one has safe all right we've gone super deep into all the theory and practice and I did want to end with a few more concrete examples cuz you shared some really cool ones with me before this so let's just go through a few and we'll wrap up but one of the examples you shared was Bluey and understanding it as content in ways that you wouldn't expect right the short of

51:48that was that early on we had assumptions about what could be popular we had assumptions about what would be better algorithmically driven versus curated sets of content you know do we want to create one called puppies and kittens or do we want something that's going to figure out whether or not you like puppies but hate kittens and don't want to see a row with mix and so early on we had to make decisions on which rows would be curated and where those would be placed and then also what content would be in the curated rows and what order those fell and a lot of times that was based on without getting into

52:20like specifics maybe it's priority based on budget of the content itself like we want to self-promote some of these Blockbusters hire maybe it was well these probably won't be popular or oh this is what Disney has found on its linear channels in the past are successful in terms of audience ratings from neelon or whatever so these are probably going to be our more popular shows one of those sets was Disney Jr on the adult section and they thought this can go essentially this is my assumption of what they were thinking was well this would just go kind of at the end because otherwise it'd use a kid profile right

52:54so we'll just put Disney Jr at the bottom one of the newer shows on that was Bluey which is a BBC and ABC Australian Broadcasting Company collab show that's based out of Australia on an Australian cartoon family of dogs it is by far the best show that I almost would say like it's worth watching even if you're not a parent or planning to be aent cartoon that's made me cry exactly I've cried more times than I'd like to that uh frankly maybe more than I've cried over things that my children have actually done in total it's such a moving show and it's so well formatt seven minute sections it's not this long

53:31form content so it can be something to get them tired or to distract them for a moment or whatever but they squeeze such a fantastic message in it but it's not on the radar of Disney Jr for example historical numbers it had just launched on cable channels so there weren't really metrics on how many people were watching it or how it was used but on Disney plus we were seeing fantastic numbers from it if you were just looking at like what content gets watched all the way through like a 99 98% completion rating each episode right or how frequently do they finish the whole season or how frequently do they rewatch

54:06The Season or something like that but it was still a limited number of people that were actually doing that and as a data person I wanted to use that one as an example not only because I love the show and I just intuitively thought that it would be successful if it had more of a platform but why should it be a hero row like how am I going to justify that it's a big Mast head image at the top of an adult profile so then let's look at the child profile well we're not really promoting the child profile most people may not even know that it exists they

54:34log into Disney and if they skip the profile setup then your just name is blank it's a Mickey head and you never think about it again right unless somebody prompts you to reset that up so my whole Disney plus is a child profile right and also like as a parent when you're launching this thing you probably just bought it cuz the kid's screaming in the corner and you're like purchase honey I have to do more forms say the word blue in there pull it up hit play and then We're Off to the Races but now you've got an adult profile using it so to me it was already inherently flawed

55:05because most people were not setting up child profiles and I didn't think that it was because they didn't have kids it was just because they were watching younger age content but how discoverable was Bluey it's on the last row in one of the last tiles so even on like an Apple TV with a quickly scrolling wheel you'd have to still scroll through 25 rows and then scroll over 15 20 tiles before you got to that content but if you completed the content it now was not in your continue watching and you'd have to go find that each time and at the time there was only one season so you could

55:35burn through it in a couple of hours right it became so frustrating because the kids would want to watch it so I had the luxury of at least having a captive audience that I could test on and a real use case but it was a good example of that it could have been any content and it could have been anything how hard is it for them to get to it and what shows up when I start to type Bluey like if I type BL and it's Blues Clues or something like that and down here buried is Bluey because we chose to do it alphabetically or because we chose to do

56:03it based on number views how hard is it for that parent to ultimately get back to that and in the child profile the search was non-existent you couldn't search so you had to literally find it within these groupings that didn't have names cuz your expectation as a 5-year-old's using it so I would use that one to say you have this content that is being consumed heavily by a subset of users of the app and they are religious about it they're sharing links to it Etc but you've made it nearly impossible for them to pin that as a rewatch once they've completed it or for them to find any of that other similar

56:38kid content but your root problem is actually because of a broken process with how the profiles were set up and how they would find that content in the other piece so that problem is a representation of something that you can what's what you can do right this second move Bluey up higher move that row up higher or make something taggable but what's something you could do long term think about thoughtful ways to promote or nudge people that do have kids at home to set up that child profile and not so that you can just know that they have a kid or know that they have three

57:06kids because they set up three profiles but what value do they get out of it by doing this their Bluey stuff is always right there and you're recommended for you row is not getting muddied with child content as well now I know the shows that you want to watch and I can show you the right thing so that's your longterm but in the short term I can also profit and for Disney's case how that then turns into a massive business is if this thing is successful and you move it up and you're seeing that exponential growth you're seeing it in real time versus neelon scores or a

57:34movie in a theater or something like that where you're waiting for box office and runup and all that how can we quickly say here's where we need to do a Broadway show we need to ramp up merch manufacturing because maybe that needs a two-month lead time of design and Engineering of what all these toys or environments are going to look like how do we get shipping out for that how do we promote the show more on Disney Jr you know do we need to reh shuffle a lot of what we've already put there do we need to promote it in the Parks and have a ride or an experience have blue

58:02walking around whatever it is you see how that has that KnockOn effect of like not only that one Insight of like this is a good piece of content buried down here but you miss that if you're not understanding the data and you're asking for structured things that are expected and you're not saying what's the thing I should be looking at instead and that could cause you to miss out on maybe a small bump but maybe an entire new business model and in this case what has turned into like a phenomenon of a show with Broadway might be the killer app for Disney plus I think like we maybe

58:33don't watch that many movies but bles is one of these things we watch over and over and over again because it's infinitely repeatable it's a good example of just not carrying in too many assumptions about how you're users will use the thing and just listen to your users like they will tell you what they care about and have it maybe tune your data stack yes so that you can detect and understand and promote like how your users are actually using if your app is revenue can't or whatever that's kind of classic product Market fit finding right it's like don't let your assumptions carry you too far and I think that's one

59:02of the problems of building in big Orcs too like you have so many ways people think it should work because their org and their thing and whatever and you know it's all very well-intentioned but at the end of the day you have to look at the data and the presumption yeah the presumptions that they lead in with and that's kind of like to that last point of how it can impact is coming in with the expectation of what we think that these are going to be bucketed by like Marvel and Star Wars super fans combine or Disney animated Fanatics that might be what your executives are telling but

59:32if you're not giving the leeway for data scientists and data analytic teams to explore what the true definitions may be or whether or not they can reinforce those definitions then you can miss out and in that case that's where we were able to reorient and say hey the data science teams the data analytics teams they're finding that these buckets actually make more sense in terms of what expected Behavior will happen or what they'll resonate with based on trying is like parents with kids parents with young kids parents with older kids adult couples single adults teens whatever those groupings are actually have more patterns in common than users

1:00:07when you try to bucket based on a Star Wars Super Fan because a third of them may love Star Wars but also like animated movies a third of them may love Star Wars and Marvel a ton a third May hate Marvel and love Star Wars you know what I mean so if you're going in with those presumptions instead of being curious and allowing those teams the leeway to explore that you can miss out and then don't shoot down based on personal biases what's right in front of you there are times that I you know have intuition on something and I chase it down and I see early on that like hey

1:00:39that's actually going different than I thought and there is that urge internally maybe you've gone through those executive meetings and said I'm really confident that this is actually a good hypothesis but you get so much farther in your career by being ruthless with yourself about what you're seeing Co data things like that you can apply this almost anywhere but also that it goes so far in companies with how they will trust you with larger and larger things in the future if you're able to own up with the fact that that actually turned out to not be I know that I thought that that was and I told you

1:01:12that and we're half bought in on it but we got to Pivot and this is just a Miss here's a good thing that I found of that as well like some positive but yeah it kind of stinks that we missed the mark they now know Miss the Mark is better than like chasing and they now know that when I'm thinking about making a VP or you know like other decision to launch a new thing that you're not going to buy into that and potentially ride into the ground and risk something because you don't want it to go against what you had previously kind of stated so one bias

1:01:40there I think often times I feel like people have a bias to believe that everything should be like causally explainable right like you'll have some some truth in the data but like the world's chaotic and there may not be a like causitive explanation for this other than just random fluctuations and I I find that to be a really hard thing where people get hung up on is they're like if a number goes up there must be a reason why and it's like yes this is doing that this is leading to that yeah it's but like sometimes also the dice get rolled a certain way and a number

1:02:11goes and how quickly the answer back to them of what would you do to prove that those are interrelated would it be that you change for 10% of the users that they don't see the thing that you think is driving this action and then that you're measuring whether or not that actually declined and if it didn't then it actually design an experiment cool okay then let's how quickly can we do that right so like even if somebody wants to try and correlate just challenge them to okay trust and verify forces them to think about the production process of the data right and then you go well like I guess the

1:02:40production process is inherently chaotic and so likely there isn't actually data here well I think that's a great place for that up yeah we could go on for like four more hours I told you it could turn into a book if you if you let it so but thanks so much for join us this was a really fun conversation I think a lot of insightful things for folks to think about and I know if any Indie developers made it all the way through to this point a lot of this is not directly applicable except that you need to think through a lot of this even as an indie

1:03:09like thinking about your little the best the most successful Indies I know are the most data oriented ones yeah your little Firebase instance like it matters because that's how you're going to improve your product and having good data in there making good decisions based on that data is key to building a great product and I think that like thinking of it with a data oriented mindset as an engineer and a developer on frontend applications is super important in thinking what happens if I change this later and how that's going to impact all the stuff Downstream that relies on that data whereas that's usually an afterthought in an engineer's

1:03:43mind unless data is pushing for it right so yeah that's a big takeaway too yeah all right well thanks so much for joining us yeah thank you guys it's been a wonderful conversation I always I can always dive in stuff like this for hours so thank you [Music] thanks so much for listening if you have a minute please leave a review in your favorite podcast player you can also stop by chat. sub club.com to join our private community [Music]