Building Mowie: How a Concierge Service Became an AI Marketing Platform

Welcome to Just Now Possible with Teresa Torres. My name's Chris O'Connor. I'm the CEO of Mowi. We're an AI marketing platform. I'm here with my co-founder, Jessica. We like to say that I'm the hacker behind Mowi, in charge of all of the product details, the technical vision, drivers, and then Jessica is our hustler in charge of all things customer related. My name's Jessica Valenzuela, and I'm the co-founder of Mowi AI. I am in that role, I handle customer experiences as well as, I'm the commercial side of the business. As Chris has mentioned, I'm the hustler. So I deal primarily with marketing, sales, and partnerships, and partnerships meaning our platform integrations as well as customer relationships. So tell me a little bit about, tell me about your target customer and what do you do for them? So our target customers are primarily small, medium-sized businesses, and we're focusing on two to three main categories, and that would be the restaurant space, the retail space, as well as e-commerce, so DTC. The reason why we started with these categories within the SMBs is because of our domain expertise of eight years with our prior company, really hands-on working with these categories, building SaaS platforms, SaaS solutions that's very focused on marketing. We had been working together at a startup that was focused on customer data management for roughly those same types of customers. We had seen how the average business owner is using marketing tools, marketing platforms, not really aware of what they're spending, how much they're spending, how effective it was. So they would constantly ask us because we were gathering data for them, if we could also try and analyze how effective their marketing strategies were, how effective Instagram, social media, their e-mails were, and then help drive those workflows, like actually create some of their content for them. Over time, we saw that we got to 45 customers, and we saw that out of 45, only two were really using their marketing platforms that they paid for. And this is even basic customer segmentation, e-mail marketing, sending out e-mails, period. There was just widespread purchasing of those tools, but always zero adoption. And the fun angle was that Jess, in her role, was the closest that a lot of the business owners had to some kind of marketing expertise. And so they'd frequently ask Jess, even though it was completely out of scope and outside of our focus, they'd ask Jess for advice, and it got down to asking Jess to actually use the tools, and perform some of their marketing, comm, and their marketing messaging for them. Yeah, just to give you a deeper view of the SMB space when it comes to their marketing, usually we're working with one-person, two-person, three-person teams. And these are really small teams when you think about the types of businesses that they handle. So a one-person marketing team handling eight different concepts, a restaurant concept. Each concept would usually have three locations to 10 locations. And so it's no longer just eight different brands within multiple markets. You're looking at eight different brands within 10 locations, within even, it just becomes, it's not a small business, really, it's an enterprise. And so when you look at the time that they have to really be able to do marketing well, and all the solutions that they're currently using, whether it's gathering data, understanding that data, being able to send personalized email campaigns, or personalized SMS campaigns, the time to be able to do that, and do that really becomes, it becomes an impossible challenge for our ICPs. Yeah, okay, I can see this complexity. So you had this experience at your previous company. Yes. And so then tell me a little bit about what inspired you to go out on your own as founders. What, give me a picture of, let's say I own a, I'm a marketing manager for a restaurant group. We've got a number of restaurants. What are you doing for me? Yeah, so the evolution of Mowi, let's get into that, is as Chris described, we would be setting up all their segmentations, all their automations, and I would be manually doing this. I was like the physical Mowi, prior to Mowi becoming Mowi AI. And we would be creating segmentation, we would be creating automations, we would even be reading their data in terms of insights around campaigns that they should be sending. So it's no longer looking at generic campaign calendars in terms of today it's pizza day, or today's ice cream day. Ice cream day doesn't really apply to a pizza restaurant, right? And being able to really look at what types of products and what types of incentives does each of the persona really value within that restaurant location is what we were helping, what we would help a restaurant do manually. And part of those problems are what we've helped, what we've built into Mowi and turned Mowi as a solution to make this more a seamless process. Okay, so if I work, let's just run with this example. I work at a restaurant group, I'm in charge of marketing, I sign up for Mowi, what is Mowi doing for me? At a high level, we're helping that marketing agency or mark their produce social media digital content. And we're then giving them visibility into what content is effective based on their natural customer segments, both from a marketing perspective that someone or a business focuses on families versus stay-at-home moms or sports enthusiasts, right? They're searching the natural way that a business feels that their customers are segmented. And on top of that, we look at the transactional segments based on a rich experience with the underlying point of sale systems and the commerce systems that the businesses use. So we look at the natural customer segmentation and then speak in terms of what a business owner understands in terms of churn and their best customers and the customers that are lagging in either visits or spend so that we're able to come up with a marketing strategy to send Instagram content, send email content, offer promotions that are targeted to those different customer groups and to do it all automatically without the business owner needing to think about that because they have an intuitive sense that they have these customer segments and they even have a personal belief as to what those customer segments are. But we help them to understand not only what they feel their segments are but what the data shows that their segments are. And that data both comes from their underlying sales system and then also from their customers themselves. So we go through and look at what customers are saying about the business, both through reviews and through other online forums and allow that to also influence or kind of validate what the business feels that they know about themselves and their customer segment. Yeah, so it sounds like a lot of your platform is for someone in your target customer segment, you're looking at, based on the data that they provide, based on what people are saying about that business, here's what we think your segments are and then you're helping them develop content based on those segments to use on social media, maybe. Okay, so let's get into how this works. Tell me, I can see the clear need for this. I know for a lot of these types of businesses, their teams are small, they definitely don't have full head count on these areas or if they do, it's a very small number of people. Jessica, like you described, just this idea of it could be a big enterprise but still a small company. Tell me, what does your target customer have to have in place to be able to start to work with you? So Chris, you already mentioned like a point of sale system, what are the inputs into this whole process? The input is literally their name in a form, at least how we built it. It's a form and then a URL, their website URL. And then from there, while we then ingest any publicly available data across the internet, I think we have about 80 artifacts that we take a look at to be able to create what we call a brand and business dossier and an audience dossier around that particular company. And maybe another question is like, what makes a good customer and what types of information does Moe like to use to provide a richer experience and to provide more targeted content? So we really look for some kind of web presence, obviously some way that the customer describes their own business, a product or services catalog, how they're defining the products that they're offering or bring to market. And then some type of customer presence that can be as light as third-party articles, reviews, magazines, that sort of thing that give a sense of what, like it gives like third-party validation as to where this customer strikes by. And then a best case, they also have some form of customer reviews. So some form of Google, Yelp, et cetera reviews that allow us to get the voice of the customer. Okay, so it sounds like you're trying to make this as easy as possible for a restaurant, a retailer, an e-commerce company. You just come in, you put in your URL. We're gonna go do a bunch of research. And by we, Moe, the AI is gonna go do a bunch of research and basically build out like a dossier about your company, who your target segments are. It's gonna pull in some voice of the customer information. And I imagine, Jess, since you had so much experience with this, doing this hands-on at your last company, it's clear that you bring a lot of domain knowledge to this space. I'm curious. So you knew it was a customer problem. You were already doing it from like a concierge standpoint. I'm curious in the early days, like how did you evaluate whether or not AI could even do this? How did you prototype some of these ideas or what did the beginning look like? So really backed into the solution. So originally, like a year and a half, two years ago, Jess was, I was encouraging Jess to look at some of the newer AI tools and capabilities that she could use to help make her workflow a bit more efficient. So help with a little bit of content creation, help with the wordsmithing, some of the language descriptions that she was normally spending a lot of time, putting a lot of time and energy into. And then also look at some of the gen AI tools for image generation to make it easier to reuse assets that some of our customers had to support different campaigns and things like that. As she started to use those tools piecemeal, it became very difficult and challenging, even on her side, to keep track of everything and to look at new tools that were coming to market. I had started to automate kind of parts of her workflow, but had seen through a variety of experiments and I was pushing the envelope how well we could support an overall workflow. And we had some spectacular failures that were really fun. At the time we were just seeing visual creation, visual creation using AI technologies isn't really there yet. And you were seeing that there was just a lot of flaws. And so even trying to create a scene or in replacing a cupcake or a turkey for Thanksgiving was just not, you just see it and it's just not something that I would want my customers to be using or I would want this restaurant to be utilizing. We had a lot of firsthand experience with eight finger turkeys and cupcakes floating in the air and in front of the Eiffel Tower and things like that, that were more us looking at what's the current state of the art. Yeah, so that was the portion where we're actually testing our own generative AI image creation tool sets. And can we make it super easy for our customer to just say, I would like to have this cupcake photograph that I've been using across all my different channels, including my product catalog so I can recreate it for a different season without having to hire a new photographer, without having to do a production, a whole production for just for Thanksgiving or just for a birthday scene and magically make that all happen. And so we did a lot of testing and we did a lot of plating, experimenting with different plates, experimenting with different angles until we got to the point where, where Mowi is actually able to create different angles automatically based on the product shot that a customer provides. I wanna understand the origin of this. You mentioned like it was a year or two you were starting to play with generative AI in your own workflows, Jess, and Chris, you were encouraging her to do more of that. And troubleshooting, I was her tech support. Yeah, was that at Mowi? Did Mowi start as like you were just doing marketing for your customers? Yes, Mowi started with us doing marketing for our customers. And after six months of doing this manually, that's when we saw the opportunity and as AI technologies were expanding and really getting better with the LLMs, we saw how this could potentially be a product that we would like to develop and spend our times on. Also, we specifically did a test that Jess had introduced a framework. It was like golden. Oh yes, Simon Sinek's Golden Circle Framework. This is how someone who's trying to focus understand their customers and understand their marketing needs. This is a framework that tends to resonate with them. Could you infer parts of this using an LLM, using AI tools? And so that was our early test was going out and saying, well, how complete of a Golden Circle dossier could you potentially build or infer based on basic customer information? And if we showed that to customers, would they find this to be interesting, useful, or completely erroneous results? And so that's where Jess had put together a one-pager together with some sample dossiers that we're generating using the Golden Circle Framework and then wiped out a little bit of a mini roadshow with some of our early customers to see how enthusiastic they were. Let's just talk about this from the customer's point of view. I sign up for Mowi, I put in my URL, I put in my company name, you're gonna go out and build this dossier. What happens next? What happens over the next couple of weeks? What is Mowi doing for me at a high level? We are designing a content calendar for the business that consists of what we call their marketing pillars, which is how do they interpret educational content, promotional content, general branding content? How does that apply to their calendar? And then in that calendar, we give them a schedule of the posts and the content behind the posts. And so that's where our customers are living on a day-to-day or week-to-week basis. They're getting our view of how frequently they should be posting across multiple channels, that mix of content, and then the actual content itself. And then once they trust Mowi and once they've been using us for a few weeks, they're managing that by exception. We give them email alerts. Hey, here's your upcoming week. You can go into the tool to override what we think you should be doing. But over time, and this can get into a rebel conversation, but over time, we're looking at what are they accepting without edits? What are they editing? And what are they regenerating in terms of their calendar and the content? And then over time, getting into just a cadence of accept this upcoming weeks of posts, accept this, your overall calendar, that sort of thing. So your onboarding is really simple. Here's my company. You're gonna go crawl the web and find stuff about my company. Are you helping me identify what my pillars should be as part of that? Like you're identifying segments. Tell me what's the deliverable that comes out of that first step. After they've onboarded, the brand dossier will basically present the customer with a business profile section, a customer intelligence section, a competitive analysis section, customer reviews, their catalog intelligence, the industry trends within their local markets, their sales and marketing intelligence, and then the marketing pillars. From a technical perspective too, that doesn't always go perfectly, right? You might have a cigar vendor that is called Joe's Cigars. And Joe's Cigars may have five different locations in the Dallas area, but it turns out there's another Joe's Cigar in the East Coast. And so sometimes we do have some ambiguity in how we're resolving, like what is actually your location? And so in general, we have a happy case where we're gathering and inferring all this information, building a dossier, but we're also popping alerts to our customers as part of the onboarding process. Hey, it looks like we found multiple locations that are different than how you describe yourselves. Please help us to resolve this. The same thing with the product catalog and services catalog. Or the same with a consistent problem that comes up is like Yelp reviews. So your Google reviews and Yelp reviews. And oftentimes when you're like a 30 location brand, most of the brands would assign a different Yelp review page for each of the locations. And so sometimes they would close a location, but that location, that Yelp link is still live. And so we would get errors and say, asking, is that location still live or not? And so it also gives the customer a way to clean up their own data. So you're doing this onboarding step where you're gonna go out and you're gonna collect a bunch of stuff about the business. You're showing it to the customer so they can correct things. And there's some ambiguity that might come up and you're reviewing all that. And then from that point, it sounds like on a weekly basis, you're creating a content calendar and like publishing for them. Help me understand what channels are we doing, email and social media and all of the above? So today it's email and social media, yeah. So one thing you mentioned earlier that I wanna come back to is that a business has like its own calendar. You mentioned like ice cream day and pizza day. Is that something that, like I can see how a restaurant had their seasons and maybe they're doing seasonal food. I know like Starbucks and pumpkin spice lattes is like a whole thing. I can very much imagine each business has its own calendar of meaningful events. Is that something the business has to enter? Is that something you're able to grab from their website? That seems like a very important part of running campaigns for a business. Where does that come from? Yeah, so there are three mini calendars that we synthesize. So there's the like public events or well-known events that could be regional, local or national. This is how most barbecue restaurants will focus on the Superbowl, right? Or Wingsplaces will focus on the Superbowl. And so we build out depending on the type of business, the generic events that business based on their business category would focus on. We look at the different events that the business has published themselves through things like calendars of their own. We have a wine bar that has like regular Friday night like blues open mic type of things that they wanna promote. And then we recommend longer term quarterly campaigns that are associated with their particular business and their customer segments. What I really like about your problem space is I can see how like if we had perfect data, marketing would be easy. If I'm a restaurant near a college campus, my perfect data, I know when every football game is, I run a lunch promotion before the game, a dinner promotion after the game. I know when your birthday is. And if I know your birthday, you get an extra bonus at dinner after the game. And right, like marketing is all about the right message to the right person at the right time. And I feel like a lot of that is, do you have the data? And I love that you're already starting to talk about this as there's national data, there's regional data, there's data that's specific to that business. So let's talk a little bit. To me, it seems like your business, your product has two key moments. There's this onboarding moment of how do we understand this business and start to collect the right data and build out our products, mental model of your business. And then there's this sort of weekly cadence of we're building a content calendar, we're running your marketing campaigns. How do we keep a human in the loop? Is that a fair way to think about it? Yes, the content creation and the content calendar is really, has weekly and a quarterly context, but a decent component of the content and marketing relies on understanding data that's coming in on a daily basis. So we haven't really talked about it, but even things like weather or like inventory control, right? On Thursdays, I have a lot of beef brisket, that's a slow day of the week. And so we wanna push the beef brisket or it's cold out. And so like a warmer temperature drinks, that sort of thing should be pushed. That's our, right now we're focusing on a daily information gathering cadence and then more of a weekly planning cadence, but we're trying to push that as much as possible to more of a alerting when there are opportunities to have more of a daily or hourly or lunch versus dinner cadence, yeah. Okay, awesome. So I wanna dig in. Let's look at both of these in different components. So for this first onboarding piece, building out a business, I'm curious, is this a one-time thing? What happens from an ongoing standpoint? Are you constantly looking at evolving segments? So like we can talk about first what happens in that onboarding stage and then what do you do to maintain that from an ongoing standpoint? Architecturally, you can think of the kind of customer research as a hierarchy of research documents. And each of those are inferred. Each of those have a refresh cycle. By default, most of them are on a weekly or monthly basis. And we're then detecting whether or not the information that document or the inputs into that documents have changed enough that we should regenerate that document. And if a document is higher up in the hierarchy, we then determine what depends on that information and then go through a new regeneration cycle. And then, yeah. So this is exactly what I was talking about, data layers. So I love this idea of a hierarchy. If this document up here changes, it means it's gonna, the changes need to trickle down to everything. And I think using like a real example, you could probably imagine what happens if a business adds a location, right? And so they're at a high level, the way that the business operate doesn't change, but the new location has been added that has new location-based information. It has possibly new location competitors. It may have slightly different customer segments that are leaving different feedback about that location. And so you can imagine like a portion of that tree will need to regenerate itself. At the end of the day, all of that content feeds into that calendar and then into the individual posts or the individual advertising artifacts that we're making. And it's depending on when it gets regen. We then offer suggestions as to whether or not this has a major calendar impact or a minor impact and whether or not they should review the proposed calendar. technical interest to spread that information differently. We try and make sure that it's always grounded in, would a customer, would one of our businesses understand why this document is important and would they be able to trace back its impact onto their advertising schedule, their calendar and their publication of it? Okay, amazing. And this is happening during that onboarding step. They see you creating these documents, they can review them. And then you mentioned something that I think makes a lot of sense too is they have a refresh rate. So you're looking at, are you basically repeating that, those elements of the onboarding process on a cyclical basis to see? Yeah, exactly. Okay, I can imagine with reviews changing, customer segments might change what people like or dislike about the restaurant or what's differentiating might change. Exactly, and then we're fingerprinting that document or the results to see how much it changes as we regenerate. And if it changes over, and this isn't consistent across all documents, but you can imagine if 95% of it hasn't changed, we're going to determine that doesn't generate a refresh or we'll alert the business owner, hey, you're getting a little bit more customer segmentation data, do you want to perform a refresh? Versus, hey, we see a brand new customer segment, right? Like for whatever reason, I don't know, Ted Burns are coming into your location, right, and they weren't part of your original customer segmentation, that seems important enough that it should force a downstream regen. So it should regen your calendar, it should regen your approach towards the channels you're targeting, et cetera. Okay, so you kick off onboarding with we're going to create this document hierarchy. Yes. We're going to capture what we know. Ideally, the customer is in there reviewing it, making sure it looks great. They can do that at any time. And then it sounds like there's calendar of events that are like maybe across all businesses. What are some of the other data inputs that start to inform what is going to later end up in this content calendar? So based on the type of business that you have, we have a advertising mix. So we're suggesting that you target on 30% educational or 50% promotional or 10% brand awareness. And depending on the type of business you have and the information that we've gathered, we'll offer suggestions on how you should deviate from best practices, the best mix. Yeah. So part of that document hierarchy is a content strategy that you're bringing everything in and saying, here's what we think your content strategy should be. Yes. Including in that document hierarchy. So again, the customer can still review it, give feedback, maybe even modify it. Yes. I think we're talking over each other because there's a mismatch between the tech perspective where in the tech world, everything is a document that is processed through a series of inferences with input and output data. But from a customer perspective, what I see on the front end is just marketing pillars and then it's divided into core pillars, the brand strategy, content strategy, then what the implementation should look like. And so when you, you know, so the brand dossier is really foundational within a brand's organization. When they look at their marketing pillars and they look at the content calendar and the channel mix that Moe is recommending, those will all tie back to what the marketing pillars say. And then if they want to expand that view just beyond their marketing pillars and they're like, I want to compare what I'm doing with my competitors, they can look at their competitive intelligence, which is also part of the brand dossier. And it seems like in order to build these documents, you're dealing with a lot of unstructured data. Yes. A lot of it is go crawl the web and see what you can learn about this business. Yes. Tell me a little bit about, I cannot imagine that was easy. I've been involved with a number of web scraping, searching kind of information gathering projects. Tell me a little bit about, you've already surfaced like sometimes a business name will be ambiguous. You need the customer to help clarify. What are some of the other challenges that come up in that step? I think we're fortunate. Now there are a number of LLM friendly services, right? Both for like scraping and the delivery of so many structured data, if it's customer reviews. And API integration. And API integration as well, especially on the point of sale or the commerce systems. But that's now pretty, it's not LLM friendly, but it's easy for us to understand and easy to analyze and to convert into a format that's meaningful or useful. I'm just curious about the challenges of trying to, I can picture this very clean, structured document hierarchy. And I can also picture this very open web of customer reviews and review articles and magazines. And how do you go from A to B? So our approach has evolved over time. So I've been an engineer or an architect for a long time. Like the dot-com world isn't unknown to me. Not to age myself too much. But my original approach was to try to define structure at every level of that kind of document hierarchy that failed spectacularly in terms of the quality of the results and being able to guarantee that the sort of data structure schema from each document was sliding incorrectly and reliably. So nowadays we break things up into more of a loosely structured content, which in our world usually translates to markdown as our sort of intermediary. And then a few of those artifacts that have very precise output so that we can make sure that the calendar is being generated against the advertising or product mix that people expect. So really, I don't think I would have used a lot of structured data for any of those other inference steps if we didn't have a need to let our customers interact and offer edits to edit the content. But really, aside from the API data that we're getting in, most of the intermediate steps that they do, intermediate documents are rather loosely structured. Yeah. And is an LLM generating those documents? Are you sending LLMs off to search the web and generate those documents? Yeah, in some instances. And then we're using third-party APIs in other instances. So a lot of the review data comes from services that excel at cataloging and summarizing reviews from other sources. And is that pretty fixed? If you have a business, if I give you a business name, are you running the same set of searches to grab data and like the same API calls independent of the business? Or is there- Yeah, the same API calls for sure. So like the Google reviews are coming from a service that we're using. And now we'll call that the same way. We're looking for the last 500 reviews for location. And then we're running that through the same set of inferences to try and categorize them to surface up some customer segmentation details. For some of the information, like competitive information or location-based information, we're using the same like query templates, but it varies based on the type of business and the location of that business. So you can imagine using that like sports analogy we had earlier. If you're in an area that's very football-focused versus hockey-focused, we're going to still search for like sports events near you, but then we're going to drill down into football events, not only at the national level, but at the university level or the same thing with like hockey in Canada. This seems like you could, this seems like this could be a product in and of itself. Like this idea of I'm going to enter a business name and I'm going to figure out all the right stuff to grab to build out your documentary. And so- Yes. I know that you have more to your product than just this, but I can see like I built small AI products and I have an appreciation for it doesn't always do the right thing and you got to have evals at lots of different steps. And- Yes. I can imagine based on what you just described, like this could be your whole product where you're just our product is we're generating a brand strategy. And so I'm curious how you like, one phrase that keeps coming up on the podcast, our very first guest talked about, she really encouraged her teams to take one bite of the apple at a time because this is a big event with a big vision with a big footprint for just this onboarding step, like just this creating this initial set of documents about a business. How did you take one bite of the apple at a time? Yeah, I'm just laughing because- He had to rebuild it. Yeah, I took many bites. He took many bites. I'm sorry. Specifically around our document management approach. So our first fight was a rather simplified more traditional workflow. So it wasn't using this kind of hierarchy of documents, but it was more, again, based on the golden circle. It was like, hey, can you infer the types of kind of core business research that a marketing person would be interested in? So it was like, what's the how, the why, the what, and the- The why, what, and how? Yeah, yeah, yeah. So it was more of a, hey, could we go out and both research on information, pull information from third-party sources and from the underlying point of sale systems? And could we answer those like high-level questions? That was the first bite. From there, we built a mini sort of document hierarchy that we felt would allow us to create basic social media posts. Basically like use that as input into the state-of-the-art tools at the time, just about a year ago, that would allow Jess to post content. And from there, the architect Demetri took over and I tried to look at a convenient abstraction, a way of abstracting that data in a way that would also tie into sort of natural evals, like natural understood research that a business person could give us feedback on. Okay, these cut for segments seem like they're appropriate or not. Okay, so Chris, if I understood you correctly, like your first version was really maybe just a workflow where you broke up parts of what you were trying to collect. So like- And generated a PDF. Yeah, like I could imagine it, you're telling it one LLM, go find the why, another go find the what, another go find how, and then you're bringing it back and aggregating it. What happened that made you outgrow that, that you started to move towards this document hierarchy? It was the feedback we were getting. So original hypothesis or original ideas, we can help streamline the sort of text content and image content that marketers were generating for individual channels, right? That at the end of the day, they would want to go through and select, I want to advertise this product or this promotion or basically have a web interface with dropdowns that they would use to automatically create posts. Okay, so you wanted to be able to let them mix and match based on what you knew about them. Yes. Like we want to run a campaign for this day that does this thing, that does that. Yeah, exactly. And you could probably visualize, it was a series of web forms, a workflow format that would try and gently lead them through. But the feedback we were getting was, why are you, I'm using it, I'm finding some value out of it, but really I don't have the time or energy to translate any of these marketing goals. Well, I know that I need to be posting on LinkedIn and Instagram and Facebook. Why can't you just tell me not only what text should be in these posts, but also what the posts should be, what my strategy should be. And then we could work its way up. Like what campaigns I should be running, what my calendar should be like, what my cadence should be like. And naturally Jess and I come from a place where we enjoy when software, it seems like it's magical. Like from the end user perspective, right? It just works and it aligns with like their mental model for how they want to be managing their content. And so we challenged ourselves to take more bites out of going upstream. And at the same time, as we did that, we realized we could enrich the data we're collecting about the business beyond the golden circle basic data so that we could inform ourselves on, we could be informed on the best sort of product, sorry, best product calendar to make. And then what are the types of events and competitive events and local events that could drive that like end-to-end content creation process. I see. So I can imagine this started out as a very simple, here's some what posts, here's some why posts, here's some how posts. Yes. And then it evolved into, okay, we'll help you. And like, we'll do a little more, we'll do a little more until eventually it became, here's your calendar for the week. And wouldn't it be nice if weekly, you were just as a business owner, verifying that you want to go with this instead of going into a tool to craft content and to pick products and yeah. That's what's amazing. And this is how it works today. Like today your customers get emailed or they log in and they see a full blown content calendar and they can either, do they have to approve it? Or is it going to go by default and they can edit it? We require some level of approval. So every week we send out an email, summarize what's coming up in the current week. The quickest process is for the customer to, sorry, for our business to accept, click yes, write in that email. And then we treat that as approval. We also are, from a tech perspective, we're queuing up posts ahead of time and we can always roll back. And so if we had a customer that approved something and later on realized that there was something that changed within the business and they didn't want to post one of these content, they can always go and manually look in their calendar and decline to put, or stop the posting of that content. Okay, so I can imagine, I do use like a social media scheduler and I can imagine like I have a very clear visual in my head, I'm going to lock it, log in. I see my content calendar. It's populated with what's going out when on what channels. I got an email to tell me it's ready. And I had to say, it looks good. And then from that point, do you do all the posting? Everything's just done. And they, so if I'm a marketing manager, I'm reading an email saying it looks good and then I'm done. Or you can override the entire posts or the content within the posts, either the visuals or the actual. So if I just want to be a lazy marketer, this isn't a dig, it feels magical. I can just check my email once a week and my social media and my email campaigns are just going to go. Accepted. Okay, let's talk about how this happens. So we've got a document hierarchy about the business. How are you creating a content calendar? Yeah, so again, our calendars are reviewed on a quarterly basis. So we have quarterly, longer general campaigns. We have campaigns that are around a specific event, regular events that tend to span multiple types of business or focus on your business. So again, like the Super Bowl back to school, 4th of July, Halloween, the pumpkin spice lattes. You're reviewing that mix on a quarterly basis and telling us that this is an acceptable input. These are the things you care about. We tried through our UI, try to limit our businesses to only pick three of these kind of main calendar campaigns and then pick three to five of the events that they want to target over the next three months. And then from there, we look at their marketing pillars. They have a certain mix of educational content. This is what user-generated content means to them. And then also recommend the marketing mix across the different channels. This sounds more complicated than it is. We basically take it as a first pass. We're rather conservative. We just, we throw out the idea that you're not interested in a full blown, like fully loaded Instagram and Facebook campaign, but instead you're a typical business that wants to do a total of 10 content pieces a week. And it's a little bit focused on Instagram, Facebook with one or two emails. And then we generate like a skeleton calendar for them over the next quarter. And if they approve that, or when they approve that, or just that, we then start to fill out the details behind those events. There's also part of the marketing pillars within the dossier where we present the different content pillars. And then within those content pillars, there is a rating that actually came from existing, whether customer reviews or how they engage with previous campaigns, where there's a rating in terms of this particular content pillar or content theme has a higher engagement. It's a great opportunity for your brand to do this and to act on this. And then there are content themes within that pillar that also says, this is a medium area of content interest for your audience and demographics. So you don't need to exert as much efforts. And so the calendar itself is weighted based on, you know, high opportunities and high engagement and low engagement. And so that's already all fully automated for the customer. They can easily override it by adding or testing, creating a campaign and testing against what Mowi recommends or they can just say, okay, this is something that we would go and utilize. I think the missing piece also that we haven't really discussed is that some of our early feedback from our first set of customers when we were golden circling things was that they really wanted to see transparency and traceability. When we're coming up with a recommendation, a very specific example, a number of customers have some kind of a new year, new you campaign. You can look at it as like a get fit, or you can look at it as like a coming in to meet more wings, depending on the business. It might be very different campaigns, but that new year, new you, they can look at our recommendation that this is a new year, new you campaign is our number one or number two general marketing campaign that we feel you should run. And they can trace it back to, all right, here are the top customer reviews that went into determining that this is, seems like it's important. Here's the data from your sales and your sales cadence that's showing that, I don't wanna use chicken wings as the example all the time, but that you should be eating more chicken wings. And they can go back and trace that through the hierarchy of documents to see how or why Moe arrived at this recommendation. And if they feel it's an error, they can inform us that through evaluative checking that they don't necessarily understand, and we can then rerun or assess the inferences that we're making at the customer level or the feedback level or. Okay, I, you know what, there's something that you just said that was like a little bit of an unlock for me. I almost think about it as your document hierarchy is the context you're providing to an LLM. Yes. With the goal of your job LLM is to output a content calendar. And here's all the context you're gonna use to create that calendar. Yes. And then you're, when you present the content calendar to the customer so that they can trust it, you're giving them almost back to the context that generated it. Yes, exactly, exactly. Okay, I do love this. And this idea of trust has been a big theme as well. As if we, it's rare that we can just do something for a customer. I want to know how do I trust that it's right? How do I have confidence it's gonna be good? Yes. And so I love that there's this connection back to those context files. So we've got, we have a whole, all these processes around how do we build context files? How do we get the human to give us feedback on those context files? Yes, exactly. And then we're using that to output a calendar and we're keeping those links. So this is, this generated this. Yes. But I think what's hard about that last piece, this generated this, is can you reliably identify that? Is the LLM good at providing that this piece of context is what is leading me to recommend this campaign or this post on this day? So it's good at providing insight into the different targeted context pieces that went into that recommendation. It's not good at identifying relative importance from different context pieces, right? So if it's like competitive information versus like customer context, that's leading to recommendation for a particular campaign, we present that entire kind of context. Those are the context inputs. Yeah, yeah. And it's still, the customer still has to use some judgment about I can see what's in that context that led it down this path I don't like. Okay. So tell me about the process. Like what, tell me about the process to generate your weekly calendars or your quarterly calendars. What's, is there an agent in there? Is it just a prompt? What's happening? I imagine there's a ton of context to manage. So I'm curious about what that content calendar generation looks like. Yeah, so on our side, we are simplifying things somewhat because we're not a, these calendars aren't being generated real time. So we do have a sort of a quarterly cadence for the overall calendar and then a weekly cadence in terms of creating the content itself. And then a nightly cadence in terms of determining what needs to be published tomorrow and then dealing with some of the transactionality of it. Like we're actually pushing this to a queue that will go out into Instagram tomorrow, that sort of thing. It's primarily, you could say it's workload based with the established prompts that have been templated out both with context, the type of pillar that we're generating and then the channel that we're focusing on. So it's three main pieces of context. The pillar and the channel is structured and structured with inputs from, so we have general context information that's coming into the post. And then we actually have fields that are being populated with the channel and the pillar. So we're generating an educational post for Instagram. We asked for what an educational post looks like for Instagram. At least from a prompt template perspective, yeah. And then I can, okay, I can see the scaffolding of how this works. I still am curious about, is it, is the LLM just creating content based on the context of the business? Is the business marketing team providing content? Let's just take pumpkin spice lattes, like has their pumpkin spice latte. If I just let the LLM say, what should this business's pumpkin spice latte be? It's gonna have hits and misses. So what, I can see how you can have templating around all of this, but I imagine the business marketer has to give some input on the actual content of the campaign. So we have three like override slots, you could think in terms of the templates. That's usually in terms of, so it depends on the marketing pillar, but it's overriding the product that we have analyzed and chosen, or in the case of like educational content, let's say that there is services who's having a beer week. And so we have a campaign that's associated with the beer week. We're gonna recommend educating the audience about their brewmasters bring approach or technique, right? And that's gonna, by default, that'll go out with an override, but the marketing person can provide input into those slots. There's a more general tone override slot as well. And then there's a kind of an open-ended, do you have any other, you can think of that as a bit more agentic. Do you have a general input into the overall content creation prompt that will change the way that- Okay, this is a great example. I wanna run with like beer week. Let's do SF beer week. I'm a restaurant. You're running my marketing for me. It's SF beer week. I'm assuming SF beer week is coming from original calendar that you know about, independent of my business. I'm a restaurant that sells beer. What if I don't, as a business owner, I have a preference that I don't wanna do promotions around alcohol? So in the quarterly calendar review, so there are two answers. If you don't, if you're not, I guess it's between marketing alcohol versus having a customer that enjoys alcohol, right? A customer that enjoys alcohol. Moe's recommendation of your campaigns and so low on the list there will be the best matches Okay, and then if it did I could still say I don't want that one. Yeah. Okay. Yes now Let's say that I like beer. I want to do a SF beer week campaign and Like I could see for one restaurant the right campaign could be around like let's learn about the different types of beer But for a different restaurant the right campaign might be let's learn about How to drink beer responsibly and those are very different campaigns is Matt is Maui Moe making Recommendations is the business higher is the hierarchy document like the document hierarchy Enough context that Moe can look at those two businesses and just say for this one Let's focus on types of beer and for this one focus on responsible drinking naturally It has to be aligned with Context that's coming from somewhere, right? So yeah, you as a business owner have just decided that you want to These aren't great examples, but if you want to promote the idea that You shouldn't participate in beer week events on a Sunday or whatever right if that didn't come from if there was no way of understanding that context as input into our alone then you're going to have to specify that as an override, but if we understand through your current educational content or Articles that were written about you or customer reviews that not serving alcohol on Sundays is important to you then Yes then that does become part of the recommended campaigns when we get to the educational pillar that the surfaces or if it's Not surfacing as your top three educational content sources. You can override that you can always Here's what I really like about this like Okay, so I'm a small business I run marketing campaigns and I like have nerded out on marketing So I like this why I know about the Simon Sinek stuff and I know about doing event-based marketing and whatever But I also think about if I had to sit down and design quarterly campaigns. This is very overwhelming there's a lot of content to create to do it really well as the synthesis of all this data and to really think about how it aligns with my brand and the events that matter to me and my customer segments and like you Highlighted just like nobody does this. It's just maybe marketing agencies do this because that's their Yeah, like yeah real people at real businesses, like even if they're a marketing person They're spread in a million different directions and we don't do this I think is maybe good or like maybe once a year we do this. Well They know so just to be clear like small businesses and the marketing teams within these businesses Know and understand the importance of things It's more like you said that the time and their availability to do it Really well to dig into the data to be able to but when the benefit that Mowi brings is that the data is there whether it's just looking at the marketing Content pillars and then applying that or just validating what Mowi is recommending at the content calendar level and to which segments and then looking at their content pillars Understanding their customers becomes clear to them It's basically something that they've thought of and something that they are probably aware of But they didn't have time to dig into the data with the data and the insights are already there for them. It makes it Magical it makes it exactly where I was going. Is that yeah to do marketing We're looking at least especially data-driven marketing. We're looking at a lot of permutations We're looking at customer segments times events times channels times pillars times the Different content I could pick for that pillar and as a human thinking about all those Permutations and all the possibilities and how would I even manage this very quickly overwhelming? but if I have an AI that can say I can run through all those permutations and I have a data structure that supports all These permutations. Yes, then I can first rank which ones are most relevant to your business pick a handful give those handful to a LLM that can run through some possibilities for the content and like step-by-step we took these Thousands if not tens of thousands of permutations and made them very manageable and in a way where we could present to a human These are your three campaigns for the quarter. Yes. Yes. Yes So, let's talk about how do you know, it's good I know you've mentioned evals a few times Give me a high level for your eval strategy and then we'll dig into the details. I Can describe how Moe Helps that customer understand whether a campaign is working or not And for that particular channel or content or campaigns working out for that particular channel Chris can dig into like how that happens in the background. And so on the front end we did a content calendar. They have Let's say a campaign for this pumpkin spice latte for a given week. They have a for Instagram Let's just use Instagram. They have three they're presented three different options And they choose one option They push it out or Moe decides to push it out or they approve that if it's not doing too well after 48 hours then the The campaign can either auto I'd say no longer would no longer be push us an ad if it's an ad and Then What we will recommend a new piece of content or campaign because that first one is not working in terms of Let me pause you and ask about that So yeah, it sounds like performance is built into the tool it Moe can see Are people engaging with this post? Are there people clicking if it's an ad are people clicking on it? And then that can feed back into whether to keep it live or not Yeah, and we're doing some this is Dusty space. Yes. Yeah, but the bite that we're taking is also looking at the underlying sales data. And so we from our past lives We've worked with most point of sale systems and most kind of smb e-commerce systems. And so we're looking at again It's not an exact attribution A link between the posts and sales, but can we at least come up with a statistically meaningful? inference on how this Marketing this marketing is impacting product sales or sales within a customer segment And we provide that as feedback to okay so this is great because I think one of the best evals ever is Did it work like yeah, you have an ultimate measure of did it work? Yeah, did it work? So it sounds like performance of the post is a feedback loop that you have it is that you're starting to integrate I imagine also a really strong feedback loop you have is that your your customer Approves the content calendar exactly. That's what's going to say. Yeah. Okay. Yeah, so it's almost like we get automatic eb testing if a customer Does it the book? They approve the calendar and then they approve the posts themselves, right? So that weekly email that says this is our weekly content that's going out. They will sometimes decline to post piece of content And we lead them through a workflow that allows them if they're interested to Regenerate the content to either change one of those three input slots or to choose to use a different pillar Yeah, and so we're gathering those metrics, right? What's working at the end of the day in terms of like usage metrics either dwell time on On the instagram post or jesse does for the ads. Yeah, it's more around They did generate a click-through that actually generated So if it's on shopify or if it's in a toast menu, did it generate a purchase? And so i've clicked through to the purchase down to the actual purchase pattern What type yeah, but then there's the traditional like never opens never clicks like time spent which I don't personally get as excited about I'm more excited about the actual yeah more Because and not because not that those statistics are not important. They definitely are important But what where we get asked about often is like, how did it make me money? Did it perform did it actually? Did actually work and and for us that kpi is quite important and valuable and that's why we're We're so focused on that entire journey and measuring moe's output of the content the channel output In terms of like how did that impact that kpi for our customers? Okay, so I can see on the content calendar side You have a lot of really good pieces in place from an eval standpoint First of all, you have a human in the loop that's approving everything on some level You're able to track what they edit what they regenerate what gets published Once it's been published you can track performance metrics If you're plugged into the point of sale system, you can even look at how does it affect sales all of that sounds amazing It also all seems like it depends entirely on this process around building your document hierarchy And i'm curious about how you do evals there How do you know you've retrieved enough? How do you know you've retrieved the right things? I know you have a human in the loop and that the business person can go and review those documents but I imagine You have to decide when you've found enough you have to decide if you found enough relevant stuff What's what are you doing on that first onboarding piece? so there was the like initial journey to creating moe where Jessica was a human in the loop for her not only for our the businesses that were starting to pilot this but actually each and every step of the eval for all of our documents nowadays, we're looking at how often does it a business view any of those documents Is providing us with a little bit of input into the whether or not we should review the quality of the output We're also looking at when they make edits. So for any of these documents We're allowing them to hit an edit button to actually like change the Segments that customer segmentation that we've defined we don't allow them to change the input of the reviews But the final inferences that we're making if they want to override that they can't reviews Customers don't they tend to trust they don't tend to to touch that the same with general like product or services catalog types of artifacts But for some of the more the inference heavy steps We're looking at how often their customer is editing that if at all And then later on when a customer is choosing to override One of the layers that we're recommending the content calendar or the content itself again We're presenting them with the context going into that step and we're asking them if they feel that any of those contact steps needs It's not great review, but this indicate what you think is working well, or what might need help We have more of a thumbs up thumbs down Customers do that work like I love that you're providing that transparency because I feel like that's a big part of the trust here But I also feel like it's a little bit like telling your customers to eat broccoli Like this post looks wrong and they want to just regenerate it. Do they take the time? To we get traced through and look at the context files and realize this is the problem there are different personalities of business owners and there is definitely a subset Who really enjoys information and data and understanding and so we do get good Results from I don't want to say that it's an owner who's naturally an engineer in a past life But the same people who like spreading pulling up multiple spreadsheets to understand like day-over-day sales from one location versus another Do provide good feedback? Most of our inferences our feedback inferences are based on whether or not they're accepting the content So it's just at a high level are they kicking content back and offering us or asking us to regenerate? Or are they kicking back a calendar recommendation that we made and asking us and then is that enough for you to start to identify? Where there might be a context problem? With a person in the middle's expertise like jess. Yes. I still tend to look. It's just me. No. Yeah Yeah, I tend to look at the calendars that are being produced as just to double check Across the organizations of our customers just to see okay. Is it on point? It also helps us validate internally, okay are is are the document hierarchies enough is the system working Are we getting new information that we didn't see last week? So there's us internally also making those checks in addition to what our customers are doing at the marketing Or at the business owner level I do really love this idea of the customer being the ultimate eval as a discovery person this is like the sweet spot for me of like How do we know our ai product is good when our customer tells us it's good but I think the challenge with that is When it's not good, like when we're talking about doing error analysis, like it's incredibly difficult To get the average customer to give us the data We need this is why bug reports are terrible, right? Like it's really hard to get the average person to provide enough detail to be meaningful enough to act on But this is the area i'm like so fascinated in exploring And it sounds like you're already you've built in the product these checks like human in the loop checks along the way That's like allowing you to get some of this structured data But I think this is the area we're going to see a lot of evolution more and more teams of Experiment with what's the right way to do this and the lightest customer touch exactly, right? And we're still that's still an area that we're definitely exploring too. Once I go now We rolled out just a simple like per ui Give us any feedback you want not just on the ui elements for the data That's being presented and it is a challenge so We have a thumbs up thumb sound approach even for the page and then they can highlight like circle What they think is working or what is causing a problem? But but even that it is a balance between trying to give the customers tools to give them very lightweight easy address but usable feedback, but not Walk work them through a vibe like a bug. Yeah workflow You know with tickets and numbers Yeah, it's not tickets and numbers it's more of like when they see when they encounter like a piece of content That's not in the context of what they would like to communicate They're immediately able within the application to be able to send us that feedback back to us. So it's pretty immediate and Same for the most part our customers have been really good about it Because they see it's either this or i'm creating this entire content from scratch It is funny how a lot of jobs are becoming Managing context instead of it used to be marketing people put together content calendars And now their job is becoming manage the context that generates your content calendar. Yeah, which is a lot of fun to see Okay, tell me what's What is next for moe what's next what's the next bite of the apple? We as we mentioned attribution is a big core of a big part of Completing the circle and so being able to really apply the different attribution points that we need to be able to really Deliver that clear picture to a customer in terms of like where they're spending the money in terms of advertising which channel And it was messaging is really Most effective depending on the segment and the persona Is a big part of our next step the other pieces. Yeah, how Far we extend beyond Sort of digital social content into more omni-channel Yeah, so that's being driven more by like customer use cases where they're saying hey like just mentioned at the beginning We do have a couple of customers who are starting to use the content that we're creating for social on their website not as social content, but as operational operational refreshing of the look and feel of their website we've had we I guess being a startup. We're we we're being discovered in various ways by people on google And so we also have folks who are in marketing more on the like digital out of the home or out of the house space Where they're trying to figure out how they measure The ads that are being displayed on like their terminals both in the stores that terminals the screens Yeah in the store as well as like outside of the their building outside of the venue Because traditionally these types of channels are very focused on just brand awareness I just spoke and it's very small short span of time But then if you're a local business and you have a digital out of the house on-prem billboard that you you have and all you say is vacancies or Like it's very just your brand name. Yeah or brand name Yeah, or just your brand name, but if you can actually utilize that out of the house channel for more That's more targeted to your segments. That's more targeted to a product that's aligned to that segment and it's Automated and rotating as it is Then it would probably Be more valuable to that business As a as right beyond being a sign. It could be their own digital billboard That's sitting right outside of their retail store. It's updating based on the day of the week or the time of day It's promoting. I could run a campaign like it's pumpkin spice Yeah Yeah, exactly, yeah So then this has been really amazing. I really appreciate you taking the time. Of course. Thank you so much It's been delightful to dive into your product Yeah, thank you. It's been a sure. Yeah. Thank you If you enjoyed this conversation, please subscribe in your favorite podcast app and give us a rating as it helps others find the show Thanks. I appreciate it You