In this episode, Audrow Nash speaks with Melonee Wise, former CEO of Fetch Robotics and current VP of Robotics Automation at Zebra Technologies. Melonee speaks about the origin of Fetch Robotics, her experience at Willow Garage, her experience being acquired by Zebra Technologies, challenges in the warehouse setting, on autonomous cars, and on the future of robotics.

Hi, everyone, welcome to the sense think act podcast. I'm your host Audrow Nash. Today I speak with Melonee Wise, former CEO of Fetch Robotics and now VP of Robotics Automation at Zebra Technologies. We talked about her experience with Fetch robotics, the differences and similarities between mobile route robots and a factory setting, and autonomous cars, and her outlook on the future of robotics. I really enjoyed this interview. First, it was really nice getting to know Melonee. And also I found it very interesting to hear about the different stages that FX Fetch robotics went through before being acquired. Specifically, they started as a hardware company building the robots, which only lasted five months or so. And then they were a software company. And now, Melanie is describing them as a data company is very interesting. With that, I hope you enjoy. Hi, Melanie, would you introduce yourself?

Yeah, I'm Melonee Wise. I'm the former CEO of Fetch Robotics now the VP of Automation, Robotics Automation at Zebra,

So that's really exciting. Would you tell me a bit how Fetch and Zebra are related?

Yeah, so I started Fetch robotics about seven years ago. And this week, we actually closed on an acquisition of zebra acquiring Fetch robotics. And so Zebra Technologies acquired Fetch to create a new robotics automation division. and Fetch makes autonomous mobile robots for the logistics and manufacturing industries.

So going into Fetch a bit more congrats on being acquired. That's so exciting. And you said it was ahead of your timeline, your 10 year, whatever being your highlight, it was actually acquired in seven years. Would you tell me a bit more about Fetch. So it's for warehouse related robots, but just I guess, background information on the company.

Yeah. And to be clear, when I said I had a 10 year vision, What I meant is when I, when they first started talking about it, Fetch robotics to investors, I kind of I kind of made sure that they were in it for the long haul, because robotics is such a hard problem. So I was like, Hey, you realize that this is a 10 year problem, it's not like you're gonna get an acquisition, like you would in a software company and, you know, pump a couple dollars into flip into an IPO or an acquisition a couple years. And so, so that's there wasn't really like, I was like, Oh, I'm gonna sell the company in 10 years, it was more like, you know, when you go and try and convince people that robotics is something interesting to invest in, you need to help temper their expectations. Because, you know, if you look at the car industry, we've been saying, since, I don't know, 20, or 27, like 2007, that we're gonna have autonomous cars next year. And we've been saying that every year for the last, you know, 14 years.

So I so seven years ago, I started Fetch and I started the company with three other co founders, Derek King, Eric Dyer, and Michael Ferguson and we all worked at Willow Garage together. We were part of the robot development team had at Willow Garage. It's a team that that basically for about a two year period built a lot of different robotic prototypes to look at different industries that that were of interest to Willow Garage because but two years before it ended, the founder that started Willow kind of said, well, what's next? Congratulations, you did open source, although that's kind of a funny statement to make. But he kind of said Congratulations, you did some open source. Now what you know and, and Steve cousins at the time said, Hey, we need to go look at you know, the home and then industrial applications. And it's really funny when you look at you know, the people that worked at Willow Garage and, and the companies that were then formed by the former employees of Willow Garage, they all kind of reflect the different areas that we investigated in the last two years. And we investigated everything from like grocery retail to manufacturing to hotels, for example, Steve went and started savvy oak and did hotels and, and the guys in AI that started Fetch robotics. We we looked at at the manufacturing and logistic verticals and we saw it as a an interesting problem to solve. And, and when you look at that the reason that was interesting to us is we thought it was attractive, tractable problem, like most warehouses and most manufacturing facilities are what you might call semi structured. But there's also a large, like, dynamic happening in the industry where finding new employees is very hard. And even before the pandemic, there was a large labor shortage.

Yeah. For factories and warehouses. Before the pandemic, there was about a 600,000 person shortage. So there were 600,000 open jobs,

I don't know that statistic off the top my head, but I do know, before the pandemic there were, there were about 6 million jobs available. And the job openings in the manufacturing and logistics industry represented about 10% of all open jobs in the United States. Yeah, crazy. Yeah. And for a long time, the United States was running at a deficit for labor, right? So we had, for a long time, we only had about five or 6 million people unemployed, and we only had about five, we had about five or 6 million jobs available. And so it was a close thing. There were actually we were running what you might call a negative employment rate, where there just weren't enough jobs, there weren't enough people for all the jobs.

So that's interesting. I mean, you hear about robots taking jobs and things like this. In this case, it's the complete opposite. It's there are not people for that. But we need more people to fill out the roles so that the companies can be more efficient. That's very interesting.

And I think that's one of the key reasons why over the last couple of decades, there's been this strong push for automation. And why you're seeing in a lot of industrialized nations, you're seeing a lot of push to get more and more automation so that they can stay competitive so that they can grow their GDP, etc, etc, etc. And so that led us to be interested in creating Fetch and how know the

so that that is the case now, just before COVID. Fetch is seven years old, or so. Was that the case then, too? And how has that changed over time?

Yeah, so that that trend was very much true before. before when we started Fetch, there was just a huge labor gap in the industry. And it what's even worse is it's coupled with a turnover rate of about 40%. So super

high, basically. What kinds of jobs are these? Like? What are the what is a person and these jobs do further day at work?

Yeah, so they, they can do anything from like packing out a box to walking around in 110 degree warehouse picking boxes off the shelf? Yeah,

they can be assembling small parts or machinery they can be. I mean, our robots are doing the delivery task of someone's job and that and that's the other thing that conflate some of the problems is if you look at it, when they're looking at automation to offset labor, it's not typically like this robot is going to play replace this person 100% there's just there really isn't the possibility to do that in most cases today. And so we're why exactly um because if you look at it, there's there's a lot of tasks that make up your job. So if you look at say, me think a pickers job, if we look at what a picker does,

are those a person that picks things out of bins and then puts them into boxes, like an Amazon person?

Yeah, that's that's the overall they fulfill orders from customers. And so there's, there's a lot of parts of their job. So there's the part of the job where they first prepare all the boxes, and then they put them on a cart. And then they push that cart to a location in a warehouse where there's a shelf and they pull something out of a box, and they put it into a box on that cart. And they're like scanning each of those locations to make sure that that thing that they're they're picking is correct, etc. And they do that through an entire warehouse and then they come back and they typically offload the boxes from that cart on to an area where either separate person or that same person packs it out. So they take it from those boxes, confirm that all the the items in it are correct, and then get it ready for shipping. And that means putting like paperwork in it and basically shoving it down a line. So someone can put a label on it and tape it. But if you look at it, the picking task is very hard. It's not like there's a lot of robots today, they can just wander up to a shelf and pull an item out of a box, especially when they're, let's say they're their t shirts. And they're, I don't know, baby onesies. And they're packed in a box, you know, that is probably only meant to hold 50. But there's 500 of them in there. And so it's a pretty complicated task from a robotic standpoint, whether it's manipulation or vision. And so we today can't do all of the task. And so there's some things that we replaced completely, like we don't have them walk all that distance, they'll meet the robot or something like that, or, or we change the roles like the now if you have a robot that where the person used to configure the cart and put all the boxes on the cart, maybe one person now is just responsible for doing that for every robot, as opposed to that person individually beer sponsor responsible for their cart. And so the the jobs change what the roles are change, and and really, it's about taking out the part of their job or the set of tasks that they do that can be done by a robot. And I, I think that there's good analogues in kind of the technical world, you know, if you if you look at, if you look at maybe writers, or journalists, they previously had researchers, and they had secretaries, and they had all these, all these other tools or resources available to them. And now, many journalists, their entire job has been reduced down to doing all those things themselves. But now the tool they have instead of a person. is Google, or, or some search tool. Yeah. Yeah. And, and so I think there there's plenty of analogues for kind of this transition from people doing tasks or part of our jobs for us to us now using automated tools.

Yep. So if I understand correctly, basically, the shipping process has a whole bunch of different components that must get done on the way. And what you do is you say, oh, a robot might be very good at this one spot. And so by adding robots into the system, you free up people from doing that one task. But it's kind of it's not like saying a robot replaces a person. Because the there are other things that the people can focus more on, when the robot frees up that one, that one area with this kind of thing. So that's why it's like apples to oranges to say one robot in is not one human out, kind of

gigantic shortage, so you can just shift the people elsewhere? Probably.

Yeah, that's the other thing is, is when you have such a labor shortage, you you're not displacing anyone, you're just trying to fill in the gaps.

Mm hmm. Yeah. And make the few people that are there as efficient as possible. So that they can you can get as much work done as possible, socially? How did you guys find the So you mentioned kind of the detailed process of finding or of what occurs in sending out pick and place this kind of thing? or picking products out? How did you find the exact part of that where you thought a robot would be a good fit?

Yeah, I think that you know, if you look at it, some of this stuff that that we did, it will Oh, it became clear that that when you look at manufacturing and logistics, whether it's picking or it's it's assembling something, there's always a process in which material has to come from some obnoxious Lee faraway part of the facility to another part. And it became very clear that there was a high transportation cost. And there's been plenty of studies done for like labor analysis, looking at different different jobs, like the fulfillment job that I just described, or a manufacturing job. And typically walking time or transport represents upwards of 50% of the person's total time. And so that's a that's a very compelling statistics when statistic when you're looking at Should I make a mobile transport robot

It seems like to me that seems like almost a simple way of identifying low hanging fruit that can be solved with robotics things. I'm thinking of like nurses spend a large period of their time commuting from patient to patient, and then company, hello t robotics, where did an internship previously, they were saying how a huge they would have physical therapy or caretakers come to people's homes in Norway, and they spent like 60% of their time commuting from home to home. And these seems like it seemed like really good robotics applications to me if you can just put a robot in there and have it do the commuting wabbit do away with their commuting in some way. Okay, so you looked at this, and you saw this number? 660? He said, 50% of their time walking? Yeah. Which is crazy. So you started there? And

then we started with building mobile robots? Uh huh. And then and then it came down to now you have a mobile robot, you have a base now what do you have to put on it to make it productive for the application. And if you look at Fetch today, we offer three major platforms, and have different payload capacity. So 100 kilograms, 500 kilograms and 1500 kilograms. And then we put tops on it to basically attack the, the different problems in the warehouse. So we have, for our small and big robots, we have cart Connect, which allows us to move around carts, we have roller top accessory, which allows us to interface with conveyor, we have standard shelving, it's kind of boring, but you know, like the shelf with the touchscreen on it. Stuff like that. I mean, it really comes down to the accessories for the hardware. I mean, if you look at that, and like, the vision and and what we were trying to build from, from day one, you know, we started as a hardware company, and as all robotics companies go, you quickly become software companies. I mean, we spent like a hot five seconds as a hardware company. And I mean, you work in Open Robotics, and like the whole business, their software. Yeah. And so we spent a large part of our middle years as a software company in over the last couple of years, we've been transitioning more towards being a data company as well.

So Wow, that's the next level. Yeah. And so building

out our data platforms, building out our data capabilities, things like that. But that's still an ongoing process, because you know, you have to build a pretty large data like to do anything.

So I imagine a lot of like interest in cloud robotics at the moment with this is what you mean with the data? Yeah,

also. So if you look at it, if you if you look at it, the middle years, when we were becoming a software company, that's when we became we built our cloud robotics platform. So all of our robots are actually controlled from the cloud, they have their autonomous units. But if you want to tell them to like, go from point

A to decision making, the lower level control is done on the robot problem

was safety, navigation stuff is done on the robot. But the task level commands like yeah, go here, do this come from the cloud. And we did that, because it makes getting set up on site a lot easier. If you, if you look at like many of these facilities, they're kind of out in the boonies, they don't have good IT support, stuff like that. And if you come along with a very complex thing, like a robot, and you're like it Oh, by the way, where do we install the server? It kind of adds a lot. So we went off to the cloud. But one of the advantages of that is, we also save the robot data. And we're building capabilities on top of that data. So forklift tection, we can tell when forklifts are speeding, we can tell when, when they get too close to people, things like that. And so those are some of the capabilities that we're starting to build on top of our data platform. But the important thing is, is you need a lot of training data, you need a lot of real world data to do anything. And I think it's it's one of the reasons why when people talk about autonomous cars, I don't think they understand the scale and the scope of what we're talking about when they when they talk about like, Oh, yeah, it'll just machine learn it. It's like Yeah. Yeah. Or they they'll say like, the car has the AI if I if I hear AI one more time, right now

I know. It's true for so I want to go back and talk about the hardware company but just tying in this for the data company and talking about kind of this of the challenge for for autonomous cars. So you mentioned warehouses, semi structured environments is probably your data is a lot nicer than what you get. Would you just talk a bit about the difference between the data you're getting and the type of data needed for like an autonomous car? And how much how difficult that will be?

Probably. So I think I think the thing is, is is, I wouldn't say that the problem we're solving is any less difficult than that of autonomous cars. I would say they have unique idiosyncrasies,

So we're basically building a ton of indoor cars for the warehouse. The differences is we don't have weather, they have weather, but they typically their sensor package costs about $200,000. Our sensor package costs about 18 $100. So there's order of magnitude difference in the cost and capability of the sensors that we use.

So that becomes a massive difference your sensors, is it like a LIDAR and a few cameras? Or

Okay, is that 1800? And it's like, okay, that's one LIDAR and through?

Well, that's, the bigger robot has to LIDAR and eight cameras. But yeah, I mean, it's still not a $70,000 velodyne, you know, and so so there's, there's that there's one, the the kind of sensing tools that we have, are very different. And so although although they have somewhat of a harder problem to solve from weather and some other aspects, they have better budgets, and sensing tools and fidelity, to deal with some of those environments that we we don't. The other thing that's very unique about our environment, that's not true for cars is our environments are far more dynamic in some ways. While while roads can be very dynamic, and, and things like that, it's, yeah, they have lots of rules of the road. And pedestrians don't just end up in major highways on a regular basis. Whereas if you look at warehouses, there's like main highways, but people walk into them all the time. The other the other, the other thing that you have to understand is that our vehicles typically go around 1.5 to two meters per second. So we're talking like three to four miles an hour. But some of the vehicles we have to operate around like forklifts can go up to eight or nine miles an hour, or even faster. And so it's very, it's a very difficult problem when you are in a scenario in which vehicles around you are going much faster than you. And then there is the forklift problem.

cars don't have to deal with this notion of of a piece of equipment being around them, that has tines that are very thin, very

hard to detect the like probes at the top of

the fork of the tines before, and they can be at any height at any time. And

so they're super hard to sense probably to Yeah, they're

super hard to sense. They can be at any height. And there's a lot of things like we don't, cars don't have to deal with a lot of overhanging things. Roads are built such that nothing is going to be hanging in the middle of the road. It's not like that in manufacturing. So

what are you just taking a plane or whatever. So that makes it super difficult to do a lot with computer vision? Probably. Yeah. And that's challenging, because you have to recognize all these different circumstances.

Yeah. And so I'm not saying that that the autonomous car problem isn't hard. I'm saying it's very hard. But I think sometimes people tend to look at the problem we're looking at, it's like, well, it's got to be easier than autonomous cars. And it's like, well, it's it's not it. It's different. And, and there's things that constrain our problem solving, and things like that. And then the other thing that is is very true, is our industry is heavily regulated from a safety perspective. And autonomous cars are not yet. They're not heavily regulated yet. I mean, there's some regulation, but there's no, there's no you have to do this test to prove that your autonomous car is safe, etc, etc. Sorry, I actually need to stop and take this call really quick. Okay.

So we were talking about the differences between the data and that basically the problem of autonomous cars versus for manufacturing, logistics. Can you tell me a bit more about the data differences From these perspectives,

yeah, I will say that from a, you know, data perspective, you know, autonomous cars are using a lot more data, that's what makes us a little bit more open to, to being able to port and move all of our data to the cloud. And you do like algorithms and some of these things in the cloud, which a car may not have some of the luxury of doing. I mean, that's, I think one of the big things that's going to happen in the next decade or two is we realize what it's going to take to make the practical deployment of autonomous cars real is the sheer amount of data, you need to just store, the map of the local area drive there that you drive on a daily basis is, is massive, and what data networks are going to support just that capability, you know, everyone's going to have to have, you know, fiber to their house, practically, to be able to like overnight, download the local area updates and things like that. And then, if you want to go on a long trip, you know, say, from San Jose to Tahoe, I can only imagine what the strategies are going to be for, like a four hour drive. So for four and a half hour drive, the the data strategy is gonna have to be to do that the nice thing for warehousing and in our robots is it's a million square feet, that's about as big as the problem gets. And so we definitely, we we have, we're running into the multiple source problem. And I think right now, in the autonomous car space, they're still struggling with the single source problem of how do you get just data out of one car, and we're dealing with, okay, the data that we have to move isn't ginormous, but we have 3050 100 500 robots that we're, we're transmitting data from. And that's a lot of data. And so it's a pipe problem.

Yep. So with autonomous cars, they're just now thinking how to locally do it. So how can I make this one car do drive from one place to another autonomously, and you're going, we have these robots driving around this entire environment. They're all doing some sort of mapping of this environment. Like what maybe there's an obstacle that has occurred. And so then they all coordinate based on this, they go, Oh, that one path is blocked, I'm gonna have to go around, and then it reroutes

that, yes, as part of our coordination, we also use peer to peer Bluetooth. But I was just speaking more from the sheer amount of like, loading the data, and then post processing. If you if you will get today I think most companies Google way Mo, all those guys use, quote, unquote, sneakernet, which is putting a hard drive in a mailbox and sending it in the mail.

Oh, yes. That's crazy. The heart? What do you mean exactly? Where are they doing that? So like, well,

so you say, company has 30 autonomous cars that their field testing?

Oh, rather than actually putting it on the cloud and sending it and uploading it, they actually just physically mailed the device because of bandwidth constraints.

They take out the hard drives and mail them. That's crazy. sneakernet

Yeah, I heard I heard some companies are uploading more than 40 or 50 terabytes a week? Wow. Yeah. So a lot of data.

So you, so um, what it sounds like, to me, you have a good bit of faith in these machine learning approaches in the long game. So like 1020 years from now, for? Is that what you think? Or because I mean, data is not terribly important if you're going to do classical approaches for things, but you're finding in the warehouse setting that it is important to have a lot of data. Just Can you talk a little bit about this?

Yeah, I think that there are are some types of semantic behaviors that we care about, that we need models for. So we need to know what the thing is. And the problem is, is that there's a lot of variation in the world. And so if you tried to build a classically trained model for every object in your, you know, I guess traversable space, it would be very hard.

And by classically trained, would you talk about about what that means?

You have 1000 images and data examples of say a Volkswagen bug from 1976. And you put that all into an algorithm and it spits out a model and And it's really good that one detector is really good at processing Volkswagen bugs. And so if you were to look at like the most, I guess you might call it asinine way of approaching the problem is you could say, I'm going to build a specific model for every car on the road every year because they change year to year. And I'm then going to spin up parallel detectors for all of these things. And, and maybe want to know, the car model, maybe you don't, but maybe there is some material reason why you want to know this. Right now, I think they reduce it down to car, truck or other. But But say you want to do this this thing, then the problem is, is you're probably going to have some set of detectors that all say that they found the thing, right, so if you have 1000 detectors, and 15 of them, say I found the thing, then you need to put it through another set of detectors and voting system to decide which one of the things that says they found the thing is right,

or some approach to figure out exactly what is the one you listen to? Or what's most likely. Okay.

And so the the challenge with that is it takes massive amounts of data. And that's why people are very old,

like 1000 detectors or something like this, because it's that one problem. But there's a lot more and you have to make distinctions between that or is it the first model, but just a bunch of times? Because you could probably do that too, and then just have a confidence and wait to confidence?

Yeah, those those models aren't as good because so

you want it all in one model, basically, yeah. So it learns the discrimination between I think

the problem is, is if you look at it like a good a good way to think about, like, our way that we've been doing learning for a long time, is, for a very long time, we built models, and systems where we said, this is the thing you care about, this is the thing you care about, we kept showing it, the thing they cared about from all angles. But if you look at the way people learn, it's a little bit different. We we learn, this is the thing that you care about. But we also see on a regular basis 1000s of examples of this is not the thing you care about. Hmm, interesting. Yeah. And so but there's a problem of trying to prove the negative when the space of things that you're trying to prove the negative on is, is infinite, right. And, and so this is one of the problems with, like machine learning in general. And, and even if you want to show, this is the thing that you care about, say we wanted to show a phone, right, this is the thing you care about, and specifically this phone. So it's blue, it has some yellow buttons. You know, a person probably only has the tolerance to take 50 pictures 100 pictures of this phone, ad nauseum. Yeah. And one of the reasons that convolution neural nets are so interesting is is it It allows us to take the 100 images and make it into like 10,000 images. Because that's what it is, the purpose of it is it basically allows us to take the pieces of all these things, split them up, put them back together and use them to train to train systems. And that's why so it's a way of artificially creating more data. And it's found that that artificial data that we create results in high, high confidence confidently trained models, typically, now, okay, there's a lot of tuning and a lot of mumbo jumbo. Yeah, it's kind of a flasher, but definitely the same time. But a lot of those systems, again, are only trained on positive examples. I'm positive. And and this is this is in general, a bigger problem, theoretically, with all of robotics is because the negative examples that can be very bad, they can have very bad outcomes, like how do you teach a robot not to hit someone when hitting them could kill a person, right? So, so not it? Whether or not training with only positive reinforcement is good. Sometimes you can't produce negative reinforcement data, although I have seen work coming out more and more on how you build simulation models to create negative reinforcement data, for example. But

probably that got us all the way there, because you can overfit the simulator pretty heavily.

Yeah, yeah. And so that's the problem. As far as all sorts of problems overfitting weird edge cases, you know, things like

and so how this relates. So, if I understand quite correctly, how this relates in the autonomous car problem or in the way Your house problem, you are using some sort as you get all of this data, and you train different models on it. And you use that effectively for perception to help the robot understand the world and what's going on around it.

So I didn't know say that a person is a person. And so then the robot can exhibit different behaviors appropriate for that person. So, for example, the one thing that's the most mature in our navigation stack is robot, the robot detection. So we know when something is another robot. And that allows us to implement certain behaviors within the robot that make them more effective. They they will, one give priority to other robots based on what those robots are doing. They will, you know,

really cool. So it's going, Oh, it's a robot. But it's also going, Oh, I can understand from the context of what I can sense what this robot is doing. And that's a higher priority than me, kind of thing. That's interesting.

And until then, that extends into things like a forklift. One of the things we were talking about earlier, it's forklifts are very difficult. Well, one of the more difficult scenarios with a forklift is the tines can be all the way up in the air, they can be 15 feet up in the air well above the robot, and the robot can't see it. Of course, it can't, it's very high up near. Now, if you know what's a forklift, you can know not to drive into that area. Even if you can't see what's there. And so that's one of the examples in which, if you can know what things are, if you submit to glean know that this is a person

you can inform your policies of don't drive right in the spot where the tongs made come down, right, this kind of thing.

Yeah. And that, that is that is what the majority of applications for machine learning autonomous cars are also for is semantic understanding. This is a ramp, this is a person, this is a crosswalk, this is a stop sign.

Now there's one thing that I've seen, which is interesting to me, is ontologies and ontologies. It's like a map that shows how all of the different parts that you might see in an environment relate to each other. So if I was an autonomous car, it could be like, this is a road, this is a sidewalk, the sidewalk might have a person or a dog on it, or something like this. So then it'd be like road road has a sidewalk, sidewalk has a person, this kind of thing. It's a relationship between all these, do you? And this is kind of a manual way to specify all the different components. Yeah. Do you guys use ontologies? Or? So is it interesting?

If you look at that, and topologies in our environment, our district and this is one of the problems I was saying is like, there's a shelf, there's a ioway next to it, but people and everything else can be in a box can be in a trash can be in it. person can be in it. So it's a very flat ontology. This is a shelf. This is a floor. Yeah, so i think i think i agree that ontologies can be very, very helpful for creating kind of semantic maps and relationships. But they they don't always apply well in our environments.

Unfortunately, it's very complex. Yeah, yeah. Gotcha. Okay, going back a little bit. No, it's cool. This is all very interesting. Going back to being a hardware company. Yeah. Can you tell me a bit about when you guys started when you were a hardware company?

Oh, yeah. Yeah. I mean, we were hard. We were a hardware company for about five months.

In the seven years that have been so fun, yes. Crazy. Because

that's, that's when we didn't have any robots. So we had to make the robot. Yeah, we had to make the robots. The good thing is, is we had a really stellar team of co founders who, who all knew a lot about hardware. And it was really funny. I think one of the more kind of interesting experiences for me is is and I think for a lot of the the guys they started competing with is, you know, we while we were will abroad, we probably build five or six different types of robots together. And, you know, we learned a lot of lessons sometimes they didn't power on sometimes they have problems.

Yeah, yeah, all the things. But when we, when we started Fetch, we had so much experience like when we built the hardware, while we design the hardware, we sent it to his job. It came back from the shop we put together turned on it worked fine. And what was really funny is there were some employees who had been in other robotics companies who had been at other hardware companies. And like, we're like, what do you Like, this timeline is insane. All of this assumes that the hardware is going to turn on and work and not catch on fire. And we're like, Well, first, it will never catch on fire. We're not that bad. But second, second, we'll be fine. And it was, it was an interesting thing, because I think one of the things that I had developed is a somewhat like, what you might call robotics amnesia, where it kind of started forgetting that, that hardware could ever really be kind of shitty. And so and so like,

give us a nice position to be in when you really are like, you're like, Oh, I'm surprised that this didn't work. Yes, everything has just been working.

Yeah. And so it was, it was definitely an interesting juxtaposition. As, as the four co founders were kind of like, Oh, you know, we got this, this isn't going to be a problem. And then some of the new employees that we hired early on, were kinda like, this rodeo before. And never, it's never this move. So it was, it was cool. We were a hardware company for a very short amount of time early on me then eventually cycled back and we built the bigger robots, we became a hardware company briefly again. But the part of the company became a hardware company. It wasn't like, everyone there full time was just doing fucked.

Yeah. So then, how did you go about, like, the manufacturing process and everything for like, so once you worked out the designs, how do you go into actually making lots of the robots? Like I assume that that's quite a big thing?

It's actually relatively simple. Because it's not like we're bending metal, someone else does all of that stuff. Like they build the sheet metal. So you do the all the CAD designs, you do all the drawings, you send it out to a shop, they fabricate it. And really what you're doing at a robotics company, if you're doing assembly is you're doing assembly and test. And not to say that we're not manufacturing, but we're not bending metal. And so yep, the parts are putting together Yeah, we're putting them together, and then upgrading them, and then making sure that they turn on. And so it's, it's actually relatively easy to build or assemble, let's call it, robot hardware, especially in the in the case of the smaller robots, the larger robots get a lot more complicated because of the safety systems and things like that, that are required. But for smaller robots, almost all the parts in in smaller remotes are custom. And so we have a lot of control over it and comes in, we put together and worked.

Gotcha. That's interesting. I've heard some companies that I've talked to I've had a lot of difficulties, like finding suppliers and working with it. And I've heard things like the CEO will fly every rec to or some some person in the company will fly every three months to say, the Schengen area of China, and work with them on the manufacturing. Did you guys have to do these kinds of things? Or?

No, we predominantly made everything in the United States. And anything that wasn't made in the United States, we had us partners that handle all

that. Interesting. Can you speak a bit about that the manufacturing in the US because I'm not honestly terribly aware of many robotics. things being made in the US? Yeah, sure. new info to me.

Yeah. So if you if you look at it, if you look at the freight 100 or 100 kilogram platform, it has, it has about 300 total parts about 100 unique parts. And the the things that are made not in the United States are things like the laser scanner that's made in Germany by the, the skins, those are injection molded in in China, but they're designed in the United States, actually, the molds are designed in the United States. And then they're built in China, but and then our wheel motors are based off of a hub motor that you see in like scooters and stuff like that. And that was clever. Yeah. And we did the we do the engineering design in the United States with a company that we use their their Sai motors, they make custom motors, and then they own parts of factories in China. So in terms of the extent of the componentry, that's like fabricated or made not in the United States outside of the computer, let's call it it's all made in the United States. So we designed our own PCBs, though. prefabricated at a PCB fabrication place in actually California. All the sheet metal is bent in, in many, I mean, the bay area has a lot of manufacturing capabilities. And all of the sheet metal is I mean, one of the shemale vendors is literally blocked from our office. They they kind of scooted down the road. And so all of all that in the paint, you know, the any of the other things like powder coating, stuff like that, it's all done locally. So if you if you look at it, unfortunately, the the a US commerce defines defined in the USA by cost of goods. So because we have a laser scanner that's made in Germany, it we can't call the robot made in the USA, we can say design in the United States, but but if you look at it on a per

like because the laser scanner is such a large component of the whole robots cost. Yes. Yeah. Can you can you tell how much one of the robots costs? No. Okay.

Sorry. That's a closely guarded secret. Okay. Okay. Yeah. But if you look at it from a total part count, sense, the majority of the parts are made in the United States. And so most of the labor and things like that, and the jobs are being created in the United States. But there's this technical way of judging. You know, if

Melonee Wise's website

Fetch Robotics

Zebra Technologies

Fetch from startup to exit

In this episode, Audrow Nash speaks to Charles Brian Quinn (aka, CBQ), CEO and a Co-Founder of Greenzie. Greenzie make an autonomous driving system for commercial lawn mowers. We talk about Greenzie's autonomous mowing system, how Greenzie has worked with manufacturers to up-fit their system into comercial mowers, how Greenzie does dog-fooding, safety and standards, and about CBQ's experiences bootstrapping and with venture capital.

Start

Introducing CBQ and Greenzie

Why go by initials?

Describing the Greenzie autonomous system

Working with mower manufacturers to up-fit their system

Software on autonomous system

Mapping the lawn

Handling poor GPS signal

Perception pipeline

Having a human use their mowers

Standards in the off-road space

The value of standards bodies

Depth cameras in the autonomous system

LIDAR and being venture backed

Replacing depth cameras with LIDAR? + Tesla and computer vision

Ruggedized computer

Microcontrollers, testing, and visualizing data

Deploying software updates

Safety

Testing for deployment + dogfooding

Training people to work with their autonomous system

Greenzie’s business model

Bootstrapping a startup versus venture capital

What CBQ learned from bootstrapping

Future of Greenzie

Advice to someone starting their career

Links and social media

This is a conversation with Charles Brian Quinn, also known as CB Q, who is the CEO and co founder of green Z. Greens. He makes an autonomous driving system for commercial lawn mowers. In this interview, we talk about greens these autonomous mowing system, how they've worked with manufacturers to upfit their system into commercial mowers, how greens he does dogfooding and about CPQ his experience with bootstrapping, and raising venture capital in startups. This is the SENS think act Podcast. I'm Audrow Nash. Thank you to our founding sponsor, Open Robotics. And now, here is my conversation with CPQ. Hi, CPQ Would you introduce yourself?

Hey, Audrow Good to be here on the podcast. I am CPQ I go by my initials Charles Brian Quinn, and then the co founder and CEO of green Z. Greens he had our our big mission is to free humans from repetitive outdoor labor. And we're starting in the green industry. We have a product that helps create self driving commercial zero turn lawn mowers. So we help big landscaping crews that maintain the largest crop in the United States by square footage to help them reduce their cost of labor,

the cost of living grass by largest crop that is,

if you think about it, you know, I know you've had great people on your podcast that are doing farming and but if you look at square footage in terms I mean, grass is a crop that the we maintain across the United States in terms of watering, cutting, maintaining, cultivating, in terms of square footage about about 3x corn, which would be the next and so there's people who do that that industry, the green industry we call it is pretty big industry and not a lot of people want to sit stand or push a mower for eight hours in the hot sun, especially here in Atlanta, Georgia, where we're based. And all over the US there's a shortage of labor. And that translates as you know, in supply and demand to higher sort of demands on that and and again, when when it doesn't people don't want to do it. And it's also dangerous, dirty, repetitive, monotonous, you know, things that we we all know our robots are very good and capable of and don't care about, then it is probably a good place for us to help supplement and add robotic workers to these crews, and our software that's based on ROS does that.

Awesome. See, you brought up a lot of things I want to start. So first, you kind of unrelated to your mower. You mentioned there's a story behind CPQ and why you prefer that

out there. Yeah, I've just always gone by my initials starting from high school. And so it's even become my little moniker, you know, on every sort of side seq, which is CPQ I used to even have a license plate See viki.com And since 99, so I've been gone for a while. So yeah, and it's funny and professionally to a lot of people were like, hey, you know, CPQ and they're like, who? And it's like Charles and they're like, oh, yeah, CBK it's kind of a moniker that both professionally and friendly. I go by. So Andre Audrow were close enough. Now you can call me CPQ.

Funny. Okay, so now tell me about the greens, the mower or your setup?

Yeah, absolutely. So we it's important to note that what we do is we are an automated driving sort of system, right? We are basically a software package that's enabled by a set of, we call them off the shelf sensors. As you know, Robotics has gotten very far in the past, Gosh, 20 3050 years, even more recently with the introduction of the Robotic Operating System. And so with the standardization these a lot of the reps and things like you know, data structures, like point clouds and ROS enhancement proposals are like being standards for the ROS community. An easy one would be like, hey, use meters as opposed to, you know, inches, right? Like, that's a very good rep, one of the three I believe, says, you know, there's some good ones out there. And so with all that, what we've found is that the time is now to take some of these sensors, like the real sense camera, and use that to basically do a function or task and so green z, we build software. That software is enabled by a set of sensors and we certify those sensors, we call it being a robotic ready, that's part of our robotic Ready program. So we help these mowers become robotic ready. And once they have the sensors, they can then run our software and our software adds self driving. So, we we are trying to like I said reduce the cost of labor. So when you think about autonomous and the behaviors and things that make something autonomous. It is sort of self directed at x You know, again, it's sensing thinking, perceiving and acting on the world. And so what we do is we have a sensor stack that does that for a very particular part of the process that landscapers do, and that these mowers do, we call it sometimes we call it cutting out the middle. And that's kind of a unique term that landscapers kind of kind of resonates with. Because when you get to a job site, you know, there's a certain set of tasks to do to make, you know, the property look beautiful or safe. If you're, you know, mowing a hospital facility that has a helipad, you know, or right of way for safety, or on roadways, or industrial, you know, a lot, a lot of safety reasons why you want to keep that, that maintain that property maintained, as well as make it look beautiful, then there's a tougher task you do. And so our software helps automate some of that. And, again, by doing that we help reduce the cost of labor for those landscapers, which is why they like our robotic workers, but a lot of people assume we have mowers and we don't, they're some of our competitors, who are wonderful, and happy to mention them here. Companies like Saif do are building a automated mower, and that we unfortunately do not we actually build a a kit, basically. And we do it at the factory. So we help manufacturers and OEMs produce self driving equipment that's comes from the factory. Both strategies, I think are noble. And I think that it's tough to compete. You know, especially talk about John Deere, who I know that one has been on your podcast, it's tough to compete with them, they produce lawn equipment, I've been doing so for hundreds of years. So they've got a lot of learning ahead of them and distribution and all that stuff. But so what we do is, again, add sensors, and a sensor suite package and our software that goes on these existing commercial mowers to add autonomy. Gotcha. Yeah, it's it seems like a clever approach, especially for a startup to work with a company that actually makes them ours, and then kind of retrofit that with your autonomous system. Yeah. And the word like a lower barrier than it is for sure. You know, in talking about lower barrier, when we first started, we did try and like retrofit, like posts sort of sale and so we go get these mowers and, and put all these things on them. And I'll never forget the first time we did it, everything broke. And then we're like, well, well, let's just make it more robust. Right? And so we tried, and what do you mean, everything broke out, bro. You know, like, when you think about how these are used, I remember the first demo, I saw ED, right, who's the CEO, right manufacturing, who is one of our current manufacturing partners that were partner with to bring out our first product. I remember seeing him demo, and he's nonchalantly talking as he's demoing. And he drives off the curb, and then like drives back on it and drives back on it and off of it, and the mower is just bouncing off the curb, repeatedly. Yeah. And he puts down to the deck and the deck, like hits it and bumps up. And I was like, oh my goodness, like the amount of vibration ruggedization required to keep what we had originally had, we just cert put these servers with an eMag clutch to basically actuate the the the left wheel and the right wheel. And it worked. But you know, it was very fragile and delicate. Absolutely. And you know, and it's funny, like, it's not that they're our end user customers are rough on the equipment, it's just that they have a job to do, and not a lot of time to do it. And so the you know, they're they move fast, they roll over the curb, they run it back into the truck and are on to the next site. So but retrofitting was very difficult. And so we immediately moved it from sort of this post retrofit to at the factory, that way we can use the design principles, design it in with the mower and get that support.

How does that work? It's like, so do you have to go to the company and say, Hey, we needed at this stage? And just tell me a bit about that. Because that's interesting.

Yeah. So this is part of my consulting background. You know, in a previous life, I, we I helped some of the big companies like GE Lighting, and some other companies do software that went with hardware. And one of the ways you do it is you go to the factory, where that thing is produced as close as possible to the line. They call the assembly line or manufacturing line. And by doing the upfitting is what we call it. You call it retrofitting, but

it's up fitting that way. You think about it, like when you buy a guitar go has it a little different than retrofitting, because you're not getting it all the way in. Like you're not getting it from the retailer where it's entirely set up for the user. You're getting it part of the way and then you're adding things and changing things. Yeah,

we call it a good way to look at it. If you think about it. If you go to the Let's assume you went to the Ford dealer to go get a truck. When you go they have some base models. You know the F 150 which is the most popular are selling truck and we have a couple of Ford Yep. And if you go and you say I want this one, they're like, great. And you say, You know what would be awesome, those, I would love one with a V eight, like turbo diesel, and they go great, we don't have any, but we could get that upgrade package like we can do that. And we get that from the factory, like we'll put in that order that it'll come down the line, we'll put it on there for you. And you can get even more custom than that. But so that's the idea is that this is a feature, we call it a autonomous mowing that we built that you could up fit or upgrade and order. And then it comes from the factory. So what we've done is work with, right. And I'm working with a couple others right now, that where we go and we do that, that outfitting as close to as I said the factory as possible so that they come off the line. And then when they go to the you know, because manufacturers are people who do this, the end user customers buy their equipment from outdoor power equipment, so much like a dealer. It's kind of like a dealership, but for outdoor power equipment. They sell these things to do this kind of activities. They have those mowers already there. And then they activate them just like I guess OnStar or Sirius XM Radio, if you've ever done that process that's similar to us, except you're doing software that does autonomy, autonomous driving, that's cool. Okay, is there.

So when you approached the company that makes the mower and you want to add these custom features, I assume they don't have like an autonomous mower package. So I assume you have to work with them to do that.

Yeah. You know, when we started the company, we were sort of like people were like, you know, we already have robot mowers, right. And so if you look at the idea of a robot mower, those are those little Roomba like machines that and they've been around since the 1970s. That patent is has been there, I think since Yeah. 70. And so you put a boundary wire in the yard. And so the big commercial mowers were like, Hey, what is this? You know, and how is this different? And, you know, we had to do a little bit of convincing to tell them no, no, this is self driving assisted sort of like almost like Lane Assist. And when we start using those terms, they're like, oh, like my new truck? has you know that where it keeps them in the lane? Probably? Yeah, it definitely helped. And so we said, hey, this is self driving, but for these big commercial mowers, where it takes over on part of the operation, and they said, well, that's not possible now is it? And we said, well, look over here, and we built prototypes, and we did the retrofitting. And we did all these things. And you know, and I'll get back to that. But that's actually the one of the thesis of the business is that we have been iterating since we started and the term iterate is very important to us. It means do learn, improve, and repeat. And the process of iteration has helped us tremendously. So I think it's, it's interesting, as an entrepreneur, I look back and say, Oh, it's always been this way. But I can guarantee you that, you know, we've been through several iterations where we probably thought certain things, and then we learned and did and now the new reality is that that is what we do. So, you know, every entrepreneur will tell you, Oh, yeah, I knew everything to do with the business. Yeah, they don't actually what they do say is they say, Well, we learned a lot. So a lot of them talk about, like learning through failing, right, failing fast, right? So we were doing a lot of that in the beginning. And so that's how we got there. To answer your question, yes, when we go to manufacture, that is, you know, the, the now what we do is we pitch and we tell them about this robotic ready, which is now been proven and commercialized with a couple of partners already. So there are there are a few systems out there, I would say that are competitive. X mark is a big manufacturer who makes Toro are sorry, Toro, who owns X mark and, and so they have one that they've been working on for a while. And then they even made an acquisition in our space, which was amazing. And I'm so thankful for that, you know, normally when, when, when, when a competitor is acquired by a big manufacturer, it's actually an incredible opportunity. You know, like a lot of companies are like, Oh, that's scary, because the big guy like has it now. They slow down. No, isn't that I mean, it's honestly that. I mean, I live in a world of abundance. And so yeah, investors and everybody, and there's always enough for everybody to do. And so when that happened precedent, well, yeah, that any legitimizes you and it also helps every other manufacturer who doesn't have that, like will pick up the phone and say, Hey, I see that you do this, right. If you know, there are 46 outdoor power manufacturers, and you may actually be hearing some one of them on the background. Apologies. That's in our test lab. But if one of those 46 Like starts to think that they get ahead, then the other 45 are gonna be like, hey, and there may be another one, you know, a new one comes online, you know, 4648, you know, so you're saying new companies, yeah, new manufacturers or other sort of entrants to the market. But I think competition is wonderful. It's definitely legitimising and it's a rising tide lifts all boats. I believe that and I believe that, you know, my entire life.

Yeah, it also seems like it makes it It easy the space grows. Yes, there is a larger events in it so educate, it's easier to harness all that. Yeah, yeah. And parts may be become more easily procured and

things like this. And then you get stuff like, you know, like, maybe standards are created, right that are there to help you too, as well.

Uh huh. So what is your autonomous driving system? What's it composed of? You mentioned a lot of sensors. And it probably has a big computer with it. And it has some things to interact with existing mower hardware, but tell me a bit about it.

Sure. So we're big fans of ROS. And they're not like operating systems. So and we're big fans of open source. So while there are parts of our stack, a few nodes that we probably won't open source, just because they're not going to be useful to the overall community. specific use case exactly. I see. But you know, if you look at like, say, robot localization, which we use, you know, it's funny, we were out there, I guess, couple years ago, two years ago, and in the field, and we were mowing for a customer. And I'll remind me to tell you about that becoming an actual becoming the customer and doing it ourselves. We still do that. Now. We learned a lot. We call that eating around dog food. But I'll come back. Yeah. But we were talking about robot localization, we were using this package and won one field, we would be mowing, and all of a sudden, we'd get to this one area, and the mower would shoot and say, Hey, I've got to head 50 miles that way, for my waypoint, and we were like, that is a bug. And we're like, that is really weird. And whenever flow error, it has to do with UTM boundaries. Are you familiar with, you know? So in,

I don't even know what is UTM stands for? Yeah, so I

think it's a universal time Merkin teal, and we can look it up. I apologize. But what it has to do with is, latitude and longitude are all straight lines. So you know, if you're going on Google Maps, you'll maybe notice that you can get Google maps of like, you know, it's a 33 or 90, and it's all in degrees, right? So you can do any location on the planet, you can have a certain it is a sphere band. So you got so right, so why can't really wrap a sphere in a rectangle, right? There's some area that's adjusting for that. Correct. So we, we we most of the lot of problem to run into, yeah, mostly libraries take a area and they'll do it in bounding boxes. And so you'll be in a UTM quadrant. And so once you're in that quadrant, the the space goes from sort of, you know, it stays in that sort of area, right. And then once you move outside that boundary, that UTM boundary, and there are like, obviously 50 or so of those, you know, as you talked about the latitude longitude and the US, I think there's like five or six that you go across. Okay, don't correct me if I'm wrong on that. But there's quite a few. And we just happened to be on it, that one site happened to be on a UTM boundary. So when it crossed that boundary, it reset to the other UTM, which was 50 miles away. So anyway, we found that bug, and it was great, we worked with more to push up a PR and, you know, and push it back to the community. And now we're about localization can go across UTM boundaries. So big fans and open source been doing in my entire life. And we'd love to keep doing that as long as as long as it exists, which I hope is forever. And and so you'd ask us about our stack big fans of robot localization. So we use a lot of the base base stuff, we're not trying to reinvent the wheel. So standard cost maps, we're big fans of ROS control. We have hardware interfaces, which really help us with, you know, left wheel right wheel, differential drive. What else do we use? We use a lot of the standard data. You know, I

guess they're missing depth by using ROS 1 or ROS 2.

We are currently our current generation stack is on ROS 1. And I think we're on melodic, and I think we have a noetic running around downstairs, that works just perfectly fine. And so we may upgrade our entire fleet to that. And then ROS T we've evaluated some of the packages we're using aren't quite there yet. And we've been working closely with some of you guys at Open Robotics, the Michael Carroll who said, close we're close. As soon as we are we would love to upgrade and we can even do that in field. So but we're enjoying the ROS 1 ecosystem. But yeah, so you asked what's our sensor stack and sort of package look like but autonomous mowing is a suite of ROS nodes that do a function and that function is autonomous mounting. So if you can imagine our system feels like an appliance. So it is a ruggedized computer and I think you need to go to our site. We're pretty open with like everything we use. It's an x86. I actually just gave a talk at ROS con. This last one on how to boot to ROS and seven seconds where I took this machine and really minimize the colonel and use the ubuntu LTS and kind of minimize that install with the precede to make it very minimize it boots up very quickly. Super fast. And I and just go straight into ROS, using System D, which we're big fans of, we like a lot of the Unix stuff. And so where there's a lot of custom stuff we don't, we're sort of not big fans of it. So yeah, we use a lot of like sort of trusted stuff like system D networking and open VPN and some of the sort of standard standard stuff that's been around for a while. It's very modular, extremely big fans of node based, you know, single responsibility, tiny things that do one thing very, very well, liberal and the input that they accept, very specific in the output that they do. And so a lot of design principles for software that we like about ROS. And our stack does that very, very well. It also interfaces with a web app that we've built, that our customers love. And that's built in Ruby on Rails, which we love, and is probably the best for doing rapid web application development. And, you know, there's a bunch of stuff behind the scenes there. But I won't go into all the details. But all that does a very specific function, and that is landscapers by our product, they take the motor out, they press map, they mow a area of the area, they want it to sort of finish the job for them, they press go on a little remote, and it basically just finishes and Stripe still fills in everything. So they do the border. And from there, you fill it in autonomously, exactly.

Okay. Let's see. So for all of the what kind of sensors are you using for this whole? So when you map it? Is that are using GPS? Or? Yeah, so what kind of thing are using?

Yeah, so I'll start with control and get all the way sort of there. It starts with control. And in order to be, you know, I have a theory about robotics that a lot of people when they envision robots, they think of the Willow Garage, sort of PR to like, has to be humanoid. But in reality, you know, robots are in robotics as a way to I think automate and customize what would normally be considered a human level task. And so there's a lot of ways you can do it. When you think about self driving cars, they are in in terms, sort of a robotics. So the first thing you have to do is get good control. And so what we do is with let we need control of left wheel, right wheel, because these are differential drive, and a lot of them have caster wheels on the front. So they while they have bias, they don't really matter and are controlled. And so we start with left wheel and right wheel control. And we've been able to do that with a drive by wire system. And so that starts with with ROS control and having a canvas adapter. And so we use can which is on you know, very

I think it's already gonna do you know it right. What is it? It's the automotive protocol. Right? Yeah,

I know, it is the automotive protocol. I don't know

what exactly stands for. But it is a protocol, we could probably look up the acronym. But it is a protocol that a lot of cars use. And it's very small, it's very high speed. And good for redundancy. Yes, it's built in. And so we have that can interface layer that allows us to left wheel right wheel. So once we have that we then need to read. So we use encoders. This standard Hall Effect sensor encoders. And on our electric platform, we can read that straight from the wheel itself, they have encoders built in. And the reason you'll need to know that is no velocity. And so when you're able

to say hey, you do your own mapping with this, you're not you're not necessarily using GPS, you're using odometry, large, right, and then maybe sensors and things.

So you're exactly right. So we're slowing down for

just a second just to speak about a hall effect sensor. So it's a magnetic sensor, and you can count the ticks as a wheel spins. So maybe you have one magnets, or you could have a bunch of magnets and you can count do you do? What's it called where they're 90 degrees out of phase? If you have to, let's say that you can figure out the directionality. Yes, that's exactly correct. Because one way it's traveling 90 degrees over to get the next bump. And one way it's traveling 270 degrees or three fourths of a rotation or three fourths of a phase? I guess

that's correct. And you can get some telephones have the gear backwards. Exactly. So they have gear teeth that they're reading. And so then if you know, the tick count, you know, and you could just do the math to say how many wheels and velocity? Yeah, so yeah, exactly.

And you were saying sensor fusion? Because if you're using raw odometry, like just seeing how many times your wheels spin, there's slip and all sorts of things that all accumulate as errors. So you're dealing with that through some sort of sensor fusion?

Yeah, so you're exactly right. When we first started, if you just use our DOM, or odometry, on a very nice platform that doesn't have a lot of slip, you go out, you turn left, you turn, right, you turn again, you get that square and in you know what'll end up happening is you'll be like this right? And it won't be distorting. And what I'm saying is you can't for those who can't see, it's like a you know, unconnected square right? Because of the slip. And so what we do is we fuse that with an IMU. And so we have a high grade IMU we've evaluated many, many many IMU use small ones we liked I happen to like some of the vector NAB stuff because it's nocturnally calibrated alias that for the gyroscope. So they don't even notice all of the sensor data from it. But yes, we do fuse a lot of the components from it. Yeah, the gyroscope, which tells it which direction you're facing? Yep. Yeah. So I did something like this in grad school, it was like some course where you use the sensor fusion of the inertial measurement unit, and the odometry. And it gets much more accurate, which is pretty amazing. So you're doing now we fuse that along with GPS, and we do high accuracy GPS, we have the ability to run on standard GPS, but we prefer and for some of our customers for the straight lines and tracking that they want to do. And a lot of the fields that we are we have RTK GPS, and are just gonna have real time kinematic. And what that means is, is if you think about normal EPA accurate, yes. And I think you've had people in your podcast talk about this before. But

you may not have this podcast, previous one, I think so yeah, you get crazy.

You get correction data. So you can get a very accurate centimeter level. So we fuse all that now I can tell you that our sensor fusion is pretty heavily biased towards the RTK GPS, because the quality of it is just super accurate. Those are

so if I remember correctly, and maybe the price has come down, those are like $20,000, or something to get a good RTK GPS isn't correct

or and we first started down, it has come down quite a bit. And when we first started exciting, I remember we were using an RTK setup that had a base station. And you're right, it was 1000s of dollars for this one and 1000s for the one on the machine. And gosh, just recently, I saw that. And a lot of it's been enabled by a company called ublox. And essentially, they have a chipset called the F nine and F nine P and that has taken the cost down to in saying I think we just got another sensor for I'll admit it, I think we just got our most recent RTK for like 129 bucks, and I'm like, Oh my God, that's crazy. And so the real the real kicker, though is that the data, Correctional Service really matters. And that's where you're seeing a lot of the innovation. So there's a service out there called SEF. There's service out there called Point perfect, that provides what we call n trip, which is I forget what that stands for, I think it's a network transport protocol. And so instead of having a base station, what they do is they're basically like a cellphone network and every 50 kilometers from the, you know, from certain high density areas, they put their own base station, they calibrate it with other sort of known areas. And then they can send out that correctional data over the Internet. And so you don't have to have a base station. That's nominal. And you already have that data. It's gotten really good, low latency, you know, super high availability. And with that correctional data, it's been really good and very good for our business.

Do you ever? I don't know with RTK GPS, but do you ever run into situations where you're mowing a field and it doesn't have good GPS coverage? Absolutely.

So we've had to design a lot around that. And what happens is in localization, at least in the way we've done it, which is the best practices, you know, by you know, roboticist, you've done this before, you have what's called continuous and you have your discrete, and I think I'm getting that right. But in continuous, you've got things like, that are just going continuously. And so that's

odometry. And that's the wheels spinning, okay? And knowing how far they spend, right? In

your discrete, you can have jumps, right? And so, in a good localization system, what will happen is, when you have when your discrete jumps, like, let's say your GPS shifts, and you know, it runs out or it's there's no connectivity, or whatever, and it says, oh, you know, I really thought you were here and you're moving along and boom, you're over here. A good localization system will go, I don't think so I, I had all this momentum in my ETF sort of saying bias that I was saying I was going five meters a second this way. Yet. Now when when you say me, I'm over here, I'm gonna ignore that and and yet back in.

So you use probabilistic methods like an extended Kalman filter and ukef, as you were saying, exactly. And so what that does is you have a pretty good idea, like you have a probability distribution that says I'm most likely here. And if you get some information that says, I'm way over here, it's like, well, it's probably not very likely it takes that information into account. And then it adjusts its belief accordingly, but if you tune it in a way, it probably won't listen to that very hard, or very significantly.

And what ends up happening the thing that's worse actually is not GPS going out, per se, it's the thing that will mess you up. Is it is it changing and you should between a high accuracy and a low accuracy. So we've actually designed our system such that when it starts, if you get a high accuracy fix, like we go, and we're like, great. And then if we lose it, which has happened, you go down, you know, big hill and go down, and there's no coverage down here. And it just, it's starting to go out. If the system loses it, the mower will pause, our software will pause and say, hey, something's up. seconds ago was the last time I got a high accuracy. And so the mower pauses, and it'll kind of wait, maybe it's just like fluid, maybe it's a fluke or something. And so sometimes the motor will just kick back and start going, it'll get that fixed back. But if it can't, you know, there is a chance for the operator, at least with the mobile interface that we've built to say, you know, what, go ahead and turn off the low accuracy or high accuracy, and there may be a small shift in the map. And so it's not enough to make a huge difference based on what we've learned. But it is enough that you could say, hey, just turn it off. Because what what's, what stinks is how, again, going in between them, and it's constantly shifting is what will mess you up. But again, yeah, sensor fusion, what we do think about it, like we basically use our coordinate system in ROS, right? You have 00, right, which is where a robot starts, and then we record right. And so we're using all this data to determine where we think, you know, the next point is, you know, on the map and coordinate space, and we're using all these other systems. So the robot doesn't think it's at, you know, lat launch. It's using lat lodge as a data point,

internal coordinate system. Yeah. Yeah, you transform it into your primary reference system.

And as all robotics is, right, it's a transform series of sensors all connected with a universal descriptive robot format. You RDF you. And that's, we love you RDF. We use this a lot as well, because it's all just transforms all the way down.

Yeah. Funny. So if you so for your robot, where you're driving it, and you haven't heard it, and 50 seconds from the GPS, which is your really good source of information for driving it around. Do you have so if you're using a probabilistic way of reasoning about where it is, does that uncertainty just it keeps, so it received one good information? And then it drives using odometry? And yeah, using odometry. And then, so the uncertainty continues to grow as it goes. And at some point, you say, This is too uncertain, I'm gonna wait for a better GPS, or then you trigger this has have the operators switch the GPS, from high to low, or whatever it might be.

Generally, I don't know the, the correct answer to this. But what's interesting is, I think, when you think about cars, and how they drive so far, you know, that that where that bias could really go, and you talked about even going through UTM boundaries and things like that, that's where a lot of that will grow. In the areas that we're doing even like, like the the large acreage, even, it's enough to do a full day of mowing, and you're totally okay, running on that same localization and, and being able to cover that field. And we're good enough, then we haven't run into really any of those. And what's interesting is, at that, that's great, you reset, you turn off, you go to the next site you turn back on. And so since it's not long running, like the second truck, or a truck going across the US and really accumulating so much bias, we've been okay. Knock, yeah, so the error doesn't stack up that significantly and so sign I'm aware of.

Awesome. Okay, so then, what? What other sensors do you have? So you have odometry, you have some depth cameras and things like this? What else? So or how is the how are the depth cameras use

your perception pipeline we're really proud of we are. I'm a big fan of AI and ML, and I think it's gonna happen. But I'm actually not, we're not using it, I believe it's gonna happen. I believe that some of the AI and ML modeling and some of the training and the networks that are being developed out there are going to revolutionize a lot of the tasks that we have to do algorithmically or programmatically. And I truly believe that. However, I think that right now, it's a bit of a black box approach. And you know, that the the amount of data required to really train it up for what we would consider to be good enough, prohibitive. It is and,

and so we need like a million examples of someone walking in front of it or something

like that. And everybody corrects me says, Oh, we've got synthetic data. And I'm like, Well, you know, synthetic data is synthetic data, like you can only like change the color of a, you know, seven times and it doesn't really add that much.

In distribution mismatches between simulated data and real data are huge a lot of times,

I mean, I know because we've been mowing in the field and you'd be surprised that the stuff we really ever Yeah. So what we've done is we've taken a really, I guess, a pretty unique and this goes back to our design principles approach of really keeping it simple. And so what we have is a lightning fast C++ pipeline that does, you know, ransack, which is stands for I don't, I can't remember random sampling. Yes. And what it does is it tries to consensus, right? So it tries to find the ground plane. And, and then anything above the ground plane is an obstacle. So, this is not like we're trying to classify and say You said, so I, the way I described our customers is they're like, visit detect, like kids. And I'm like, you know, it does not make a distinction between a garden gnome and a kid. If it sees something above the ground plane, it's going to stop

it'll success to go back slightly for ransack and talk about a little bit about that. So you, you have your depth camera, your depth camera points over the ground. And then what you're going to do is check a bunch of, it's going to give you points at different depths, kind of. So from this, you're going to try to fit with a math equation, a plane on that. So a 2d surface, and the one that matches the majority of the points is the one you're gonna select. And that's it's a fast algorithm. Yeah, so the thing that's on top of that are deviating from it, you're like, This is something thing. Yeah, is what I understand.

Yeah, and not to get into all the details, but we have some special fun sauce, we do around line fit, and some of the other stuff on stuff to make it really fast and accurate. And, and then again, all that's translated into just a standard cost map. And the cost map would be, hey, you know, a, when they see something that is a, you know, tree or vertical plane or anything above that, that that is not part of that, that that surface, then it says, Hey, I can't go there. And so it creates a very large infinite cost map, you know, to say, like, kind of out there. Yeah, exactly.

Okay, how do you deal with like? So I imagine like the way you want to handle a kid, in close distance to the mower, and the way you want to handle a pole would be very different. How do you, like if I, if there's poles in my lawn, or in my field here,

you want to go around them? Whereas the other kids want to stop? Right? Yeah, what's interesting is when you think about, and this is natural as as humans to sort of say, like, let's solve the entire domain, right, like, let's do it just like a human would. And what's interesting is, having developed this company, you know, from iterative and from these basis is sort of like, how do I solve the first immediate problem, which is a job to be done, and get as far as you can get, as far as

that iterating, you're speaking about exactly,

is that when you get when you actually talk to customers? They're like, Do you know what they're like, I'm good at going around things. Like, you know, by the way, like, I can go right around a tree, I know where the the brushes, or the pine straw or whatever it is, or the roots there like I can do that. What I need help with is just the sort of big valueless. Exactly. And so what we've done is we've, we've built our system is such that it is made for those open areas. And so when a lot of people do that, they're like, Well, I probably can't use it. Because you know, I, you know, I, you know, I do this one that just has trees everywhere. And we're like, great. The good news is the mower still works like you normally do. Yeah, you know, like, I don't turn on cruise control, the moment I get out of my driveway, I hope you don't either. I only do it when it starts to get really repetitive and boring. And I'm like, I'm on the highway, let me turn on cruise control. And so it's the same functionality that we've developed, right? Yeah,

cruise control is a very good metaphor for those. Yeah, see, because you use it when you have a spot that's incredibly monotonous. Like, it's just this big, open thing that's going to take a bunch of passes. And thus, a big open field, that's going to take a lot of back and forth. And so that's when you use it. But for the highly custom stuff, like if you had a bunch of poles or trees to go around, in which case you don't engage cruise control, you keep the robot. And so your systems are made so that a human operator can just jump on it. Like you're not modifying the spot where the human works,

right. And that's been very critical. Even when when we first developed it, I remember a customer early on telling me, you know, Hey, your system may be great and may never break, but this is my livelihood. So there always needs to be a way to disconnect. And so that's one of the features we design and the robotic ready. If you look at our website, there's a switch from manual to autonomous and, you know, it's funny, you and I both know that it's probably not needed in the future at one point. But it is a physical safety blanket type thing when you do this, the entire system is cut off, that could have autonomous control. And so there's no way for the system to take over and sort of issue commands to the wheels and that's a very fun thing for our customers that are relieving, you know a safety blanket that they like, and it's also a productivity thing like they can always sort of say, and when you say jump on to I want to be careful to say that that when the motor is going autonomous because we have sort of 360 cameras, if you run up to it, even from the back, it'll stop. If you even somehow were able to get up and dodge it or whatever, be a ninja. And it wasn't meaning this. But yeah, but it's fun to talk about, like, if you were to jump on the back, a lot of the mowers have a thing, you stand on a platform operator presence, if you get on that, like we used to allow people to ride on it while it was autonomous, and now we don't, because it's not a safety thing. But it was fun, you can ride on it while the matter was going to be pretty cool. But now if somebody is on there, it won't allow that. Right? So you know, we've had our customers try and put their safety people on it. And there's videos of them, like running up to it, jumping on it, and just stopping immediately. We're like, don't do that. And they're like, Well, you gotta test it. I was like, okay, funny.

So it almost to me, it seems like a clever approach to do this, where you have the ability to toggle between the two, because it seems like if you're making a fully autonomous mower, and you're building the entire mower for the fully autonomous case, so you're not making a spot for the human to sit and work on it, for the drive it, it makes it so you have to solve every hard problem. And I think that that's probably like, there's a lot of them, I'm sure you can speak to like, how do you know if I don't know, you can go over these routes or go around all the things. But then having the human do these hard ones, that seems very

clever too good example would be like high grass, right? You know, so like, there's a big, you know, brush of grass. And we ran into this, right? Like, you know, one of our customers got a little behind, because they don't have enough labor. And they came out, they're like, Hey, listen, like this field is like knee high grass, like I was supposed to hit it last three weeks before, and I didn't, it's like way up here. And we're like, instead of the mower was like, like, can't go here can't go here. Like, even we have a feature for avoidance, which you know, and feel they can just have it go around. And so it's just constantly trying to go around everything. And he's like, it looks like it's drunk. And we look at the cameras. And so we look through the RGB. And we're like, whoa. And so we did design a feature for relaxed obstacle detection. And so we raised the ground plane of the ransack. And so we do let him operate with a higher, but but when we do that, again, this is all about jobs to be done. So we trust our operators, they're trained, they're certified to do it. And we, you know, train them on this. And we said, hey, listen, when you press this button, it'll relax it, you know, you better be sure that there's nothing in there. And when he did, he was like, works great. And we rolled that out, you know, two weeks later, and he was like, This is wonderful. Thank you. Can you can you add this feature? And we're like, Yep, sure can. So we keep going with them. And that's kind of how we run. Which you're right. It's, it's, it's the iterative approach, right? And you You hit the nail on the head, which is, how do we augment human capabilities, you know, especially were in these highly repetitive, dangerous, dull, dirty, monotonous things that robots are very good at and likened to, and then how do we keep the person able to do the creative, you know, unique, beautiful, and things that humans will always be good at and should be good. And, you know, that to me is the future. It's, it's what I love, like, why I do this, and why I get up every day.

It's also the So for all those are great reasons. It's also pragmatic as a startup to, because you're not doing the hardest problem. First, you're not doing every hard problem all at once. Yeah, you're doing just the minimum to provide value in a market, which is fantastic for I mean, like, I mean, I'm thinking about autonomous cars now. And you see huge companies like, say, like Cruz, who has like, I don't know exactly, but like 1000s of engineers working on this problem, to fully automate driving. And a lot of really great things have come out of that. But as a startup, it's hard to do because you don't have 2000 engineers. So solving an easier problem that still provides a lot of value, kind of like what Tesla's doing, in a sense.

Yeah. And again, we're just we're sort of copying a lot of this crazy has been amazing. Our product manager that was just in San Francisco and holidays and was like a solid couple around around he's like, it's incredibly out there everywhere. It's crazy. And you that way Mo Yeah, yep. So it's the only choice here it's just not evenly distributed. And what I tend to think about the off road capabilities, what we're doing in the autonomous space where you know, it's interesting for us is I always tell people is like, hey, when we don't know what to do, we stop. Now, I think as a self driving car, I think when you don't know what to do, like, you can't just stop in the freeway, right? Yeah. Oh, gosh. There has to be a lot more and there's people involved and plenty for us who are in controlled environments like we can in some of the center's we're working with are sort of industrial Right. So there's a controlled environment, you know, you don't have people running around. Hell, it's it's interesting, because people were like, Well, what about schools and kids and it's like, hey, why don't you ask us landscaper, what they do? And have you ever seen a landscaper milling around kids know if the kids are out, they don't know, period and the story like manually. So, you know, so So the automation is capable of doing now and I think that the offered space will continue to have leaps and bounds. I?

So when you think there's when I think about autonomy, there's a lot of standards right now, if you look at some of the bodies that are certifying on road autonomy through some of the standards, like the ISO 2626262. Yeah. Or what I think it is, it's one of those I just to say, just say this is until it repeats, but you have the

right Ryan therapy from clear path you mentioned next Tuesday, or something like this, often, okay.

And so those are the those are the autonomy levels, right? Level one up to level five were like, level five is just, you know, and level five, kind of interesting, because I've heard people say that level five would be like, You tell your car, like, hey, take me to the beach, and it takes you to work instead. Because it's like, Hey, you actually need to go to work. Like it's truly self directed, right? It's like, No, you don't need to compete. But um, the, the i digress a little bit, but we talked about on river software, there's the standards and in off road, we don't have that, like there isn't a body learning is exactly that that's, that's determining these, we do have is a lot of some of the ones for industrial robotics. And so their standards like ar 1508, which is a robotics for automated machinery, and vehicles. And those are the ones that we're going for those standards. And we've been really lucky, we've working with some external consultants for risk analysis, as well as we've got just hired some validation engineer from from Rivia. And so we have a lot of firepower on that. And I think that are, and we're working with some of the industry as well, some of the outer power industry people directly to help make sure that those are safe standards, but we're doing that in conjunction with our customer, who also like has a job that they want to get done. You know, I tell my team, we have six core values and safety first is first. And number six is customer success. And they're not in real order. You know, they're all important. But if you think about it, like if safety is like the most critical, then the safest thing to do would be to just sit in a insulated bubble and tell our customers you can No, because mowings dangerous, right? That'd be the safest it is. But in the end, there's customer success, and they've got a job to do. And while it is dangerous, we accept those risks. And we the reason we accept this risk is because we've done enough risk analysis to know that like, hey, if I do this safely, and I follow the procedures and the guidelines, the risk is low enough, that I'm safe to do this. And so where we fit is trying to bridge that gap between I've got this job to be done. And I still want to do it safely and with the least amount of risk possible. And I frankly think that when we look back in the in the future, we're gonna look back and say, Wow, can you believe that we had humans sitting standing riding on these things that were prone to rollovers and a huge spinning blade? Yeah. And we'll be like, oh, yeah, we used to used to drive around in cars and smash into each other. Oh,

no, I'm just gonna say, yeah, yeah, the cars are so dangerous, but, and it's funny, because, like, they're still relatively safe, but they are one of the most dangerous, dangerous things in modern society. So I think, like 50 years from now, it'll be like, our parents that we were driving around, like the risk, but

it's true. Yeah, unless you walk or bicycle and bicycling is much more dangerous. And city it is.

Speaking of bicycling, you got to get, you've got to get Brandon from Luxan us on your podcast, the homework that he's been doing with depth a AI and the depth AI platform. These are our cameras that basically run AI ml on them. And they have great ROS drivers. And we're helping a little bit with that open sourcing a lot, but phenomenal and vegan advice. And he started this company because he had some friends who and I won't mess with his story. But they had a some accidents. And he was like, how do we prevent this? And he's like, we got to get a camera that faces behind. And so he tried to prototype one and was like, you know, how do we do this and, and with AI and ML where it is right now he was able to do it. And so he open sourced the platform so that other people are able to do it. And we're actually excited to start implementing that on our next gen, which we've already started. And so in terms of, you know, think about like the real sense and the depth. Cloud is a great protocol for algorithmic but like I said that AI stuff is coming fast. I mean, the stuff that they can even do, they can even output a Deaf cloud using you know, something. Exactly, exactly. And the quality better than stereo and these kinds of things and I've been super impressed and they fast. So we're definitely got to get him on. Because he's, I mean, in terms of robots, they're gonna revolutionize it, I think.

Awesome. I'll look into having him on the podcast. Thank you for the recommendation. One, one thing that I'm wondering, like, where do you see standards bodies being helpful in these? Is it because they establish well defined protocols that are well thought out? And thus they reduce risk? Or where's the value in establishing these standards? Yeah, I probably

not the best person to ask this as an entrepreneur who thinks that all rules are there for somebody else. I'm kidding. The rules are meant to be broken. I don't believe that, but I'll say this, I mean, I benefit from them every single day. You know, like, from, from the headphone jack, I just plugged into to the USB that I had to plug in, and then turn around and then put back and then do it that way. Right. Those are all standards, right? Web Standards. You know, ROS being a standard, the rep, like we talked about, I'm a huge fan of those. And so what I think where, where I think the best things get built, is in a company is leading, and really driving that. And then it isn't all driven by consensus and committee. Now, I'm a big fan of committees, we're on a few. And but where I think that you can get in trouble is trying to design by consensus, I have a rule in our company, and my team will will kill me for saying this. But there's three types of decisions that are made at Green sea. And this is all companies by the way. There's command control. There's consensus. And then with there's consultative, right, so let's go through the three. So in command is when I come come in the room, and I'm like, Hey, guys, or hey, people, Hey, everybody, Hey, person's everybody. We're doing this. And everybody goes, Okay, let's do it. consultative is, His command is pretty good. And sort of wartime and scenarios when you need somebody just come and say, Hey, this is what we're doing. We make decisions. Yeah. consultative is very good. And consultative is very critical, and probably the one you should be doing in business. And the most important, that's the one where, where someone comes in and says, Hey, there's a decision that I'm in charge of making, because I'm responsible for this, you seek expertise, correct. And you said, I'm seeking your consultation on it. But ultimately, I have the call. And that's how all decisions in business should be made. The only decision in business that should be done consensus is where do we go for lunch? Yeah, because I just think that the that watering down of sort of, like, you know, we've got to get all everybody to buy in and fully understand it. You know, if you had to ask everybody, just nothing would get done. And so I do appreciate committees that are the appropriate size, that are in that actually have stakeholders who are committed and not just in it to sort of Express power or sort of lobbying or interest that aren't their own, then I think you have very good standards, and I'm all for that, you know, so again, you get real world, you know, and that's what I like about the ones we're on there. They're not from, you know, people who are just trying to stamp this out or competitive. They are people who are actually like, nd user, you know, who benefit from it. And where I think there's benefits is that, you know, we've seen this in the automotive industry, you know, there are standards bodies that say, like, Hey, here's what we think true sort of autonomy would look like. And these are some of the things we do you know, it's funny, there's a simple one that you know, I got a new car. And when you go to the back, there's a liftgate and there's a button, and all the iconic, iconic RV, is all sort of standardized, right? And so, you know, it seems funny, like it's an SUV, but the the Connor icon or Greg, I forget it, the icon itself is that looks like a card or something, right. And it's a hood closing, but it's I know, when I see that button, I click that, and I know that the hood will close. And that's because of standards help us all. So we all don't have to read the manual every time we do something. They really help with usability safety, and I'm a big fan. I do do sense, though, that there are some that are probably gotten too big or don't have appropriate stakeholders that matter anymore where I think a lot of damage can be done.

How do you determine if it's too big? Oh, or how do you figure out the one size?

Yeah, I'm not the one to determine it. I think it's all context dependent, but I'll quote Yeah, he says, because he seems to be the smartest man on the planet. He says, What is it the pizza rule? Right? So if you can feed a pizza, that group is fine. It's the fine size but anything bigger than that? He's like, get rid of it. So everything is done in feeding. I think it's I think it's the pizza rule and maybe two pizzas. I'll have to look it up.

Something some small number of pizzas. Anyways. Interesting. Okay, going back to your autonomous system a bit more. So you have many depth cameras, they're pointed. So you use odometry, odometry. And GPS. You're doing some sensor fusion there and is fusion. Correct. So we local and global, right? Yep, go ahead. And then you have cameras, depth cameras. So you were mentioning the Intel real sense, which is structured lighting, I believe, or maybe it sends out a laser or something

it does in the outdoor, it probably ignores a lot of the pattern project that happens the IR projection because of just using stereo, or what doesn't it works pretty well said is is stereo depth, right? Yeah. And no, in between them. Yep. And so

you can infer how far away everything is by the disparity in the images taken by both of the cameras and go, Okay, so you have those, those are facing all you have, like, one of them on each corner, or they're all around

the 360. But what's interesting is that there are directions that a mower cannot go just like a car, right? You know, you can't see a car can't suddenly move laterally diagonal back. Yeah. Perfect. Yeah, exactly. Exactly. Yeah, exactly. Not morphic. And so having said that, we have pretty close to 360. But we actually have a bias towards the front being more important, because that's the primary direction that it goes. And so that that sensor is actually a little bit duplicated. If you look at our current one, we have a little bit of overlap. And those are calibrated, like I said, to make sure that we can see. But yeah, so we have very good coverage around the mower, and then in the back. But there's I mean, there are definitely sort of areas, you know, sort of here where it does not move, but the

left and back right kind of thing. Yeah. Yeah. Okay, cuz you're not as worried about someone approaching in the mower, run running them over

2 posts with the autonomous driving tag