In this episode, Audrow Nash speaks to Charles Brian Quinn (aka, CBQ), CEO and a Co-Founder of Greenzie. Greenzie make an autonomous driving system for commercial lawn mowers. We talk about Greenzie's autonomous mowing system, how Greenzie has worked with manufacturers to up-fit their system into comercial mowers, how Greenzie does dog-fooding, safety and standards, and about CBQ's experiences bootstrapping and with venture capital.

Start

Introducing CBQ and Greenzie

Why go by initials?

Describing the Greenzie autonomous system

Working with mower manufacturers to up-fit their system

Software on autonomous system

Mapping the lawn

Handling poor GPS signal

Perception pipeline

Having a human use their mowers

Standards in the off-road space

The value of standards bodies

Depth cameras in the autonomous system

LIDAR and being venture backed

Replacing depth cameras with LIDAR? + Tesla and computer vision

Ruggedized computer

Microcontrollers, testing, and visualizing data

Deploying software updates

Safety

Testing for deployment + dogfooding

Training people to work with their autonomous system

Greenzie’s business model

Bootstrapping a startup versus venture capital

What CBQ learned from bootstrapping

Future of Greenzie

Advice to someone starting their career

Links and social media

This is a conversation with Charles Brian Quinn, also known as CB Q, who is the CEO and co founder of green Z. Greens. He makes an autonomous driving system for commercial lawn mowers. In this interview, we talk about greens these autonomous mowing system, how they've worked with manufacturers to upfit their system into commercial mowers, how greens he does dogfooding and about CPQ his experience with bootstrapping, and raising venture capital in startups. This is the SENS think act Podcast. I'm Audrow Nash. Thank you to our founding sponsor, Open Robotics. And now, here is my conversation with CPQ. Hi, CPQ Would you introduce yourself?

Hey, Audrow Good to be here on the podcast. I am CPQ I go by my initials Charles Brian Quinn, and then the co founder and CEO of green Z. Greens he had our our big mission is to free humans from repetitive outdoor labor. And we're starting in the green industry. We have a product that helps create self driving commercial zero turn lawn mowers. So we help big landscaping crews that maintain the largest crop in the United States by square footage to help them reduce their cost of labor,

the cost of living grass by largest crop that is,

if you think about it, you know, I know you've had great people on your podcast that are doing farming and but if you look at square footage in terms I mean, grass is a crop that the we maintain across the United States in terms of watering, cutting, maintaining, cultivating, in terms of square footage about about 3x corn, which would be the next and so there's people who do that that industry, the green industry we call it is pretty big industry and not a lot of people want to sit stand or push a mower for eight hours in the hot sun, especially here in Atlanta, Georgia, where we're based. And all over the US there's a shortage of labor. And that translates as you know, in supply and demand to higher sort of demands on that and and again, when when it doesn't people don't want to do it. And it's also dangerous, dirty, repetitive, monotonous, you know, things that we we all know our robots are very good and capable of and don't care about, then it is probably a good place for us to help supplement and add robotic workers to these crews, and our software that's based on ROS does that.

Awesome. See, you brought up a lot of things I want to start. So first, you kind of unrelated to your mower. You mentioned there's a story behind CPQ and why you prefer that

out there. Yeah, I've just always gone by my initials starting from high school. And so it's even become my little moniker, you know, on every sort of side seq, which is CPQ I used to even have a license plate See viki.com And since 99, so I've been gone for a while. So yeah, and it's funny and professionally to a lot of people were like, hey, you know, CPQ and they're like, who? And it's like Charles and they're like, oh, yeah, CBK it's kind of a moniker that both professionally and friendly. I go by. So Andre Audrow were close enough. Now you can call me CPQ.

Funny. Okay, so now tell me about the greens, the mower or your setup?

Yeah, absolutely. So we it's important to note that what we do is we are an automated driving sort of system, right? We are basically a software package that's enabled by a set of, we call them off the shelf sensors. As you know, Robotics has gotten very far in the past, Gosh, 20 3050 years, even more recently with the introduction of the Robotic Operating System. And so with the standardization these a lot of the reps and things like you know, data structures, like point clouds and ROS enhancement proposals are like being standards for the ROS community. An easy one would be like, hey, use meters as opposed to, you know, inches, right? Like, that's a very good rep, one of the three I believe, says, you know, there's some good ones out there. And so with all that, what we've found is that the time is now to take some of these sensors, like the real sense camera, and use that to basically do a function or task and so green z, we build software. That software is enabled by a set of sensors and we certify those sensors, we call it being a robotic ready, that's part of our robotic Ready program. So we help these mowers become robotic ready. And once they have the sensors, they can then run our software and our software adds self driving. So, we we are trying to like I said reduce the cost of labor. So when you think about autonomous and the behaviors and things that make something autonomous. It is sort of self directed at x You know, again, it's sensing thinking, perceiving and acting on the world. And so what we do is we have a sensor stack that does that for a very particular part of the process that landscapers do, and that these mowers do, we call it sometimes we call it cutting out the middle. And that's kind of a unique term that landscapers kind of kind of resonates with. Because when you get to a job site, you know, there's a certain set of tasks to do to make, you know, the property look beautiful or safe. If you're, you know, mowing a hospital facility that has a helipad, you know, or right of way for safety, or on roadways, or industrial, you know, a lot, a lot of safety reasons why you want to keep that, that maintain that property maintained, as well as make it look beautiful, then there's a tougher task you do. And so our software helps automate some of that. And, again, by doing that we help reduce the cost of labor for those landscapers, which is why they like our robotic workers, but a lot of people assume we have mowers and we don't, they're some of our competitors, who are wonderful, and happy to mention them here. Companies like Saif do are building a automated mower, and that we unfortunately do not we actually build a a kit, basically. And we do it at the factory. So we help manufacturers and OEMs produce self driving equipment that's comes from the factory. Both strategies, I think are noble. And I think that it's tough to compete. You know, especially talk about John Deere, who I know that one has been on your podcast, it's tough to compete with them, they produce lawn equipment, I've been doing so for hundreds of years. So they've got a lot of learning ahead of them and distribution and all that stuff. But so what we do is, again, add sensors, and a sensor suite package and our software that goes on these existing commercial mowers to add autonomy. Gotcha. Yeah, it's it seems like a clever approach, especially for a startup to work with a company that actually makes them ours, and then kind of retrofit that with your autonomous system. Yeah. And the word like a lower barrier than it is for sure. You know, in talking about lower barrier, when we first started, we did try and like retrofit, like posts sort of sale and so we go get these mowers and, and put all these things on them. And I'll never forget the first time we did it, everything broke. And then we're like, well, well, let's just make it more robust. Right? And so we tried, and what do you mean, everything broke out, bro. You know, like, when you think about how these are used, I remember the first demo, I saw ED, right, who's the CEO, right manufacturing, who is one of our current manufacturing partners that were partner with to bring out our first product. I remember seeing him demo, and he's nonchalantly talking as he's demoing. And he drives off the curb, and then like drives back on it and drives back on it and off of it, and the mower is just bouncing off the curb, repeatedly. Yeah. And he puts down to the deck and the deck, like hits it and bumps up. And I was like, oh my goodness, like the amount of vibration ruggedization required to keep what we had originally had, we just cert put these servers with an eMag clutch to basically actuate the the the left wheel and the right wheel. And it worked. But you know, it was very fragile and delicate. Absolutely. And you know, and it's funny, like, it's not that they're our end user customers are rough on the equipment, it's just that they have a job to do, and not a lot of time to do it. And so the you know, they're they move fast, they roll over the curb, they run it back into the truck and are on to the next site. So but retrofitting was very difficult. And so we immediately moved it from sort of this post retrofit to at the factory, that way we can use the design principles, design it in with the mower and get that support.

How does that work? It's like, so do you have to go to the company and say, Hey, we needed at this stage? And just tell me a bit about that. Because that's interesting.

Yeah. So this is part of my consulting background. You know, in a previous life, I, we I helped some of the big companies like GE Lighting, and some other companies do software that went with hardware. And one of the ways you do it is you go to the factory, where that thing is produced as close as possible to the line. They call the assembly line or manufacturing line. And by doing the upfitting is what we call it. You call it retrofitting, but

it's up fitting that way. You think about it, like when you buy a guitar go has it a little different than retrofitting, because you're not getting it all the way in. Like you're not getting it from the retailer where it's entirely set up for the user. You're getting it part of the way and then you're adding things and changing things. Yeah,

we call it a good way to look at it. If you think about it. If you go to the Let's assume you went to the Ford dealer to go get a truck. When you go they have some base models. You know the F 150 which is the most popular are selling truck and we have a couple of Ford Yep. And if you go and you say I want this one, they're like, great. And you say, You know what would be awesome, those, I would love one with a V eight, like turbo diesel, and they go great, we don't have any, but we could get that upgrade package like we can do that. And we get that from the factory, like we'll put in that order that it'll come down the line, we'll put it on there for you. And you can get even more custom than that. But so that's the idea is that this is a feature, we call it a autonomous mowing that we built that you could up fit or upgrade and order. And then it comes from the factory. So what we've done is work with, right. And I'm working with a couple others right now, that where we go and we do that, that outfitting as close to as I said the factory as possible so that they come off the line. And then when they go to the you know, because manufacturers are people who do this, the end user customers buy their equipment from outdoor power equipment, so much like a dealer. It's kind of like a dealership, but for outdoor power equipment. They sell these things to do this kind of activities. They have those mowers already there. And then they activate them just like I guess OnStar or Sirius XM Radio, if you've ever done that process that's similar to us, except you're doing software that does autonomy, autonomous driving, that's cool. Okay, is there.

So when you approached the company that makes the mower and you want to add these custom features, I assume they don't have like an autonomous mower package. So I assume you have to work with them to do that.

Yeah. You know, when we started the company, we were sort of like people were like, you know, we already have robot mowers, right. And so if you look at the idea of a robot mower, those are those little Roomba like machines that and they've been around since the 1970s. That patent is has been there, I think since Yeah. 70. And so you put a boundary wire in the yard. And so the big commercial mowers were like, Hey, what is this? You know, and how is this different? And, you know, we had to do a little bit of convincing to tell them no, no, this is self driving assisted sort of like almost like Lane Assist. And when we start using those terms, they're like, oh, like my new truck? has you know that where it keeps them in the lane? Probably? Yeah, it definitely helped. And so we said, hey, this is self driving, but for these big commercial mowers, where it takes over on part of the operation, and they said, well, that's not possible now is it? And we said, well, look over here, and we built prototypes, and we did the retrofitting. And we did all these things. And you know, and I'll get back to that. But that's actually the one of the thesis of the business is that we have been iterating since we started and the term iterate is very important to us. It means do learn, improve, and repeat. And the process of iteration has helped us tremendously. So I think it's, it's interesting, as an entrepreneur, I look back and say, Oh, it's always been this way. But I can guarantee you that, you know, we've been through several iterations where we probably thought certain things, and then we learned and did and now the new reality is that that is what we do. So, you know, every entrepreneur will tell you, Oh, yeah, I knew everything to do with the business. Yeah, they don't actually what they do say is they say, Well, we learned a lot. So a lot of them talk about, like learning through failing, right, failing fast, right? So we were doing a lot of that in the beginning. And so that's how we got there. To answer your question, yes, when we go to manufacture, that is, you know, the, the now what we do is we pitch and we tell them about this robotic ready, which is now been proven and commercialized with a couple of partners already. So there are there are a few systems out there, I would say that are competitive. X mark is a big manufacturer who makes Toro are sorry, Toro, who owns X mark and, and so they have one that they've been working on for a while. And then they even made an acquisition in our space, which was amazing. And I'm so thankful for that, you know, normally when, when, when, when a competitor is acquired by a big manufacturer, it's actually an incredible opportunity. You know, like a lot of companies are like, Oh, that's scary, because the big guy like has it now. They slow down. No, isn't that I mean, it's honestly that. I mean, I live in a world of abundance. And so yeah, investors and everybody, and there's always enough for everybody to do. And so when that happened precedent, well, yeah, that any legitimizes you and it also helps every other manufacturer who doesn't have that, like will pick up the phone and say, Hey, I see that you do this, right. If you know, there are 46 outdoor power manufacturers, and you may actually be hearing some one of them on the background. Apologies. That's in our test lab. But if one of those 46 Like starts to think that they get ahead, then the other 45 are gonna be like, hey, and there may be another one, you know, a new one comes online, you know, 4648, you know, so you're saying new companies, yeah, new manufacturers or other sort of entrants to the market. But I think competition is wonderful. It's definitely legitimising and it's a rising tide lifts all boats. I believe that and I believe that, you know, my entire life.

Yeah, it also seems like it makes it It easy the space grows. Yes, there is a larger events in it so educate, it's easier to harness all that. Yeah, yeah. And parts may be become more easily procured and

things like this. And then you get stuff like, you know, like, maybe standards are created, right that are there to help you too, as well.

Uh huh. So what is your autonomous driving system? What's it composed of? You mentioned a lot of sensors. And it probably has a big computer with it. And it has some things to interact with existing mower hardware, but tell me a bit about it.

Sure. So we're big fans of ROS. And they're not like operating systems. So and we're big fans of open source. So while there are parts of our stack, a few nodes that we probably won't open source, just because they're not going to be useful to the overall community. specific use case exactly. I see. But you know, if you look at like, say, robot localization, which we use, you know, it's funny, we were out there, I guess, couple years ago, two years ago, and in the field, and we were mowing for a customer. And I'll remind me to tell you about that becoming an actual becoming the customer and doing it ourselves. We still do that. Now. We learned a lot. We call that eating around dog food. But I'll come back. Yeah. But we were talking about robot localization, we were using this package and won one field, we would be mowing, and all of a sudden, we'd get to this one area, and the mower would shoot and say, Hey, I've got to head 50 miles that way, for my waypoint, and we were like, that is a bug. And we're like, that is really weird. And whenever flow error, it has to do with UTM boundaries. Are you familiar with, you know? So in,

I don't even know what is UTM stands for? Yeah, so I

think it's a universal time Merkin teal, and we can look it up. I apologize. But what it has to do with is, latitude and longitude are all straight lines. So you know, if you're going on Google Maps, you'll maybe notice that you can get Google maps of like, you know, it's a 33 or 90, and it's all in degrees, right? So you can do any location on the planet, you can have a certain it is a sphere band. So you got so right, so why can't really wrap a sphere in a rectangle, right? There's some area that's adjusting for that. Correct. So we, we we most of the lot of problem to run into, yeah, mostly libraries take a area and they'll do it in bounding boxes. And so you'll be in a UTM quadrant. And so once you're in that quadrant, the the space goes from sort of, you know, it stays in that sort of area, right. And then once you move outside that boundary, that UTM boundary, and there are like, obviously 50 or so of those, you know, as you talked about the latitude longitude and the US, I think there's like five or six that you go across. Okay, don't correct me if I'm wrong on that. But there's quite a few. And we just happened to be on it, that one site happened to be on a UTM boundary. So when it crossed that boundary, it reset to the other UTM, which was 50 miles away. So anyway, we found that bug, and it was great, we worked with more to push up a PR and, you know, and push it back to the community. And now we're about localization can go across UTM boundaries. So big fans and open source been doing in my entire life. And we'd love to keep doing that as long as as long as it exists, which I hope is forever. And and so you'd ask us about our stack big fans of robot localization. So we use a lot of the base base stuff, we're not trying to reinvent the wheel. So standard cost maps, we're big fans of ROS control. We have hardware interfaces, which really help us with, you know, left wheel right wheel, differential drive. What else do we use? We use a lot of the standard data. You know, I

guess they're missing depth by using ROS 1 or ROS 2.

We are currently our current generation stack is on ROS 1. And I think we're on melodic, and I think we have a noetic running around downstairs, that works just perfectly fine. And so we may upgrade our entire fleet to that. And then ROS T we've evaluated some of the packages we're using aren't quite there yet. And we've been working closely with some of you guys at Open Robotics, the Michael Carroll who said, close we're close. As soon as we are we would love to upgrade and we can even do that in field. So but we're enjoying the ROS 1 ecosystem. But yeah, so you asked what's our sensor stack and sort of package look like but autonomous mowing is a suite of ROS nodes that do a function and that function is autonomous mounting. So if you can imagine our system feels like an appliance. So it is a ruggedized computer and I think you need to go to our site. We're pretty open with like everything we use. It's an x86. I actually just gave a talk at ROS con. This last one on how to boot to ROS and seven seconds where I took this machine and really minimize the colonel and use the ubuntu LTS and kind of minimize that install with the precede to make it very minimize it boots up very quickly. Super fast. And I and just go straight into ROS, using System D, which we're big fans of, we like a lot of the Unix stuff. And so where there's a lot of custom stuff we don't, we're sort of not big fans of it. So yeah, we use a lot of like sort of trusted stuff like system D networking and open VPN and some of the sort of standard standard stuff that's been around for a while. It's very modular, extremely big fans of node based, you know, single responsibility, tiny things that do one thing very, very well, liberal and the input that they accept, very specific in the output that they do. And so a lot of design principles for software that we like about ROS. And our stack does that very, very well. It also interfaces with a web app that we've built, that our customers love. And that's built in Ruby on Rails, which we love, and is probably the best for doing rapid web application development. And, you know, there's a bunch of stuff behind the scenes there. But I won't go into all the details. But all that does a very specific function, and that is landscapers by our product, they take the motor out, they press map, they mow a area of the area, they want it to sort of finish the job for them, they press go on a little remote, and it basically just finishes and Stripe still fills in everything. So they do the border. And from there, you fill it in autonomously, exactly.

Okay. Let's see. So for all of the what kind of sensors are you using for this whole? So when you map it? Is that are using GPS? Or? Yeah, so what kind of thing are using?

Yeah, so I'll start with control and get all the way sort of there. It starts with control. And in order to be, you know, I have a theory about robotics that a lot of people when they envision robots, they think of the Willow Garage, sort of PR to like, has to be humanoid. But in reality, you know, robots are in robotics as a way to I think automate and customize what would normally be considered a human level task. And so there's a lot of ways you can do it. When you think about self driving cars, they are in in terms, sort of a robotics. So the first thing you have to do is get good control. And so what we do is with let we need control of left wheel, right wheel, because these are differential drive, and a lot of them have caster wheels on the front. So they while they have bias, they don't really matter and are controlled. And so we start with left wheel and right wheel control. And we've been able to do that with a drive by wire system. And so that starts with with ROS control and having a canvas adapter. And so we use can which is on you know, very

I think it's already gonna do you know it right. What is it? It's the automotive protocol. Right? Yeah,

I know, it is the automotive protocol. I don't know

what exactly stands for. But it is a protocol, we could probably look up the acronym. But it is a protocol that a lot of cars use. And it's very small, it's very high speed. And good for redundancy. Yes, it's built in. And so we have that can interface layer that allows us to left wheel right wheel. So once we have that we then need to read. So we use encoders. This standard Hall Effect sensor encoders. And on our electric platform, we can read that straight from the wheel itself, they have encoders built in. And the reason you'll need to know that is no velocity. And so when you're able

to say hey, you do your own mapping with this, you're not you're not necessarily using GPS, you're using odometry, large, right, and then maybe sensors and things.

So you're exactly right. So we're slowing down for

just a second just to speak about a hall effect sensor. So it's a magnetic sensor, and you can count the ticks as a wheel spins. So maybe you have one magnets, or you could have a bunch of magnets and you can count do you do? What's it called where they're 90 degrees out of phase? If you have to, let's say that you can figure out the directionality. Yes, that's exactly correct. Because one way it's traveling 90 degrees over to get the next bump. And one way it's traveling 270 degrees or three fourths of a rotation or three fourths of a phase? I guess

that's correct. And you can get some telephones have the gear backwards. Exactly. So they have gear teeth that they're reading. And so then if you know, the tick count, you know, and you could just do the math to say how many wheels and velocity? Yeah, so yeah, exactly.

And you were saying sensor fusion? Because if you're using raw odometry, like just seeing how many times your wheels spin, there's slip and all sorts of things that all accumulate as errors. So you're dealing with that through some sort of sensor fusion?

Yeah, so you're exactly right. When we first started, if you just use our DOM, or odometry, on a very nice platform that doesn't have a lot of slip, you go out, you turn left, you turn, right, you turn again, you get that square and in you know what'll end up happening is you'll be like this right? And it won't be distorting. And what I'm saying is you can't for those who can't see, it's like a you know, unconnected square right? Because of the slip. And so what we do is we fuse that with an IMU. And so we have a high grade IMU we've evaluated many, many many IMU use small ones we liked I happen to like some of the vector NAB stuff because it's nocturnally calibrated alias that for the gyroscope. So they don't even notice all of the sensor data from it. But yes, we do fuse a lot of the components from it. Yeah, the gyroscope, which tells it which direction you're facing? Yep. Yeah. So I did something like this in grad school, it was like some course where you use the sensor fusion of the inertial measurement unit, and the odometry. And it gets much more accurate, which is pretty amazing. So you're doing now we fuse that along with GPS, and we do high accuracy GPS, we have the ability to run on standard GPS, but we prefer and for some of our customers for the straight lines and tracking that they want to do. And a lot of the fields that we are we have RTK GPS, and are just gonna have real time kinematic. And what that means is, is if you think about normal EPA accurate, yes. And I think you've had people in your podcast talk about this before. But

you may not have this podcast, previous one, I think so yeah, you get crazy.

You get correction data. So you can get a very accurate centimeter level. So we fuse all that now I can tell you that our sensor fusion is pretty heavily biased towards the RTK GPS, because the quality of it is just super accurate. Those are

so if I remember correctly, and maybe the price has come down, those are like $20,000, or something to get a good RTK GPS isn't correct

or and we first started down, it has come down quite a bit. And when we first started exciting, I remember we were using an RTK setup that had a base station. And you're right, it was 1000s of dollars for this one and 1000s for the one on the machine. And gosh, just recently, I saw that. And a lot of it's been enabled by a company called ublox. And essentially, they have a chipset called the F nine and F nine P and that has taken the cost down to in saying I think we just got another sensor for I'll admit it, I think we just got our most recent RTK for like 129 bucks, and I'm like, Oh my God, that's crazy. And so the real the real kicker, though is that the data, Correctional Service really matters. And that's where you're seeing a lot of the innovation. So there's a service out there called SEF. There's service out there called Point perfect, that provides what we call n trip, which is I forget what that stands for, I think it's a network transport protocol. And so instead of having a base station, what they do is they're basically like a cellphone network and every 50 kilometers from the, you know, from certain high density areas, they put their own base station, they calibrate it with other sort of known areas. And then they can send out that correctional data over the Internet. And so you don't have to have a base station. That's nominal. And you already have that data. It's gotten really good, low latency, you know, super high availability. And with that correctional data, it's been really good and very good for our business.

Do you ever? I don't know with RTK GPS, but do you ever run into situations where you're mowing a field and it doesn't have good GPS coverage? Absolutely.

So we've had to design a lot around that. And what happens is in localization, at least in the way we've done it, which is the best practices, you know, by you know, roboticist, you've done this before, you have what's called continuous and you have your discrete, and I think I'm getting that right. But in continuous, you've got things like, that are just going continuously. And so that's

odometry. And that's the wheels spinning, okay? And knowing how far they spend, right? In

your discrete, you can have jumps, right? And so, in a good localization system, what will happen is, when you have when your discrete jumps, like, let's say your GPS shifts, and you know, it runs out or it's there's no connectivity, or whatever, and it says, oh, you know, I really thought you were here and you're moving along and boom, you're over here. A good localization system will go, I don't think so I, I had all this momentum in my ETF sort of saying bias that I was saying I was going five meters a second this way. Yet. Now when when you say me, I'm over here, I'm gonna ignore that and and yet back in.

So you use probabilistic methods like an extended Kalman filter and ukef, as you were saying, exactly. And so what that does is you have a pretty good idea, like you have a probability distribution that says I'm most likely here. And if you get some information that says, I'm way over here, it's like, well, it's probably not very likely it takes that information into account. And then it adjusts its belief accordingly, but if you tune it in a way, it probably won't listen to that very hard, or very significantly.

And what ends up happening the thing that's worse actually is not GPS going out, per se, it's the thing that will mess you up. Is it is it changing and you should between a high accuracy and a low accuracy. So we've actually designed our system such that when it starts, if you get a high accuracy fix, like we go, and we're like, great. And then if we lose it, which has happened, you go down, you know, big hill and go down, and there's no coverage down here. And it just, it's starting to go out. If the system loses it, the mower will pause, our software will pause and say, hey, something's up. seconds ago was the last time I got a high accuracy. And so the mower pauses, and it'll kind of wait, maybe it's just like fluid, maybe it's a fluke or something. And so sometimes the motor will just kick back and start going, it'll get that fixed back. But if it can't, you know, there is a chance for the operator, at least with the mobile interface that we've built to say, you know, what, go ahead and turn off the low accuracy or high accuracy, and there may be a small shift in the map. And so it's not enough to make a huge difference based on what we've learned. But it is enough that you could say, hey, just turn it off. Because what what's, what stinks is how, again, going in between them, and it's constantly shifting is what will mess you up. But again, yeah, sensor fusion, what we do think about it, like we basically use our coordinate system in ROS, right? You have 00, right, which is where a robot starts, and then we record right. And so we're using all this data to determine where we think, you know, the next point is, you know, on the map and coordinate space, and we're using all these other systems. So the robot doesn't think it's at, you know, lat launch. It's using lat lodge as a data point,

internal coordinate system. Yeah. Yeah, you transform it into your primary reference system.

And as all robotics is, right, it's a transform series of sensors all connected with a universal descriptive robot format. You RDF you. And that's, we love you RDF. We use this a lot as well, because it's all just transforms all the way down.

Yeah. Funny. So if you so for your robot, where you're driving it, and you haven't heard it, and 50 seconds from the GPS, which is your really good source of information for driving it around. Do you have so if you're using a probabilistic way of reasoning about where it is, does that uncertainty just it keeps, so it received one good information? And then it drives using odometry? And yeah, using odometry. And then, so the uncertainty continues to grow as it goes. And at some point, you say, This is too uncertain, I'm gonna wait for a better GPS, or then you trigger this has have the operators switch the GPS, from high to low, or whatever it might be.

Generally, I don't know the, the correct answer to this. But what's interesting is, I think, when you think about cars, and how they drive so far, you know, that that where that bias could really go, and you talked about even going through UTM boundaries and things like that, that's where a lot of that will grow. In the areas that we're doing even like, like the the large acreage, even, it's enough to do a full day of mowing, and you're totally okay, running on that same localization and, and being able to cover that field. And we're good enough, then we haven't run into really any of those. And what's interesting is, at that, that's great, you reset, you turn off, you go to the next site you turn back on. And so since it's not long running, like the second truck, or a truck going across the US and really accumulating so much bias, we've been okay. Knock, yeah, so the error doesn't stack up that significantly and so sign I'm aware of.

Awesome. Okay, so then, what? What other sensors do you have? So you have odometry, you have some depth cameras and things like this? What else? So or how is the how are the depth cameras use

your perception pipeline we're really proud of we are. I'm a big fan of AI and ML, and I think it's gonna happen. But I'm actually not, we're not using it, I believe it's gonna happen. I believe that some of the AI and ML modeling and some of the training and the networks that are being developed out there are going to revolutionize a lot of the tasks that we have to do algorithmically or programmatically. And I truly believe that. However, I think that right now, it's a bit of a black box approach. And you know, that the the amount of data required to really train it up for what we would consider to be good enough, prohibitive. It is and,

and so we need like a million examples of someone walking in front of it or something

like that. And everybody corrects me says, Oh, we've got synthetic data. And I'm like, Well, you know, synthetic data is synthetic data, like you can only like change the color of a, you know, seven times and it doesn't really add that much.

In distribution mismatches between simulated data and real data are huge a lot of times,

I mean, I know because we've been mowing in the field and you'd be surprised that the stuff we really ever Yeah. So what we've done is we've taken a really, I guess, a pretty unique and this goes back to our design principles approach of really keeping it simple. And so what we have is a lightning fast C++ pipeline that does, you know, ransack, which is stands for I don't, I can't remember random sampling. Yes. And what it does is it tries to consensus, right? So it tries to find the ground plane. And, and then anything above the ground plane is an obstacle. So, this is not like we're trying to classify and say You said, so I, the way I described our customers is they're like, visit detect, like kids. And I'm like, you know, it does not make a distinction between a garden gnome and a kid. If it sees something above the ground plane, it's going to stop

it'll success to go back slightly for ransack and talk about a little bit about that. So you, you have your depth camera, your depth camera points over the ground. And then what you're going to do is check a bunch of, it's going to give you points at different depths, kind of. So from this, you're going to try to fit with a math equation, a plane on that. So a 2d surface, and the one that matches the majority of the points is the one you're gonna select. And that's it's a fast algorithm. Yeah, so the thing that's on top of that are deviating from it, you're like, This is something thing. Yeah, is what I understand.

Yeah, and not to get into all the details, but we have some special fun sauce, we do around line fit, and some of the other stuff on stuff to make it really fast and accurate. And, and then again, all that's translated into just a standard cost map. And the cost map would be, hey, you know, a, when they see something that is a, you know, tree or vertical plane or anything above that, that that is not part of that, that that surface, then it says, Hey, I can't go there. And so it creates a very large infinite cost map, you know, to say, like, kind of out there. Yeah, exactly.

Okay, how do you deal with like? So I imagine like the way you want to handle a kid, in close distance to the mower, and the way you want to handle a pole would be very different. How do you, like if I, if there's poles in my lawn, or in my field here,

you want to go around them? Whereas the other kids want to stop? Right? Yeah, what's interesting is when you think about, and this is natural as as humans to sort of say, like, let's solve the entire domain, right, like, let's do it just like a human would. And what's interesting is, having developed this company, you know, from iterative and from these basis is sort of like, how do I solve the first immediate problem, which is a job to be done, and get as far as you can get, as far as

that iterating, you're speaking about exactly,

is that when you get when you actually talk to customers? They're like, Do you know what they're like, I'm good at going around things. Like, you know, by the way, like, I can go right around a tree, I know where the the brushes, or the pine straw or whatever it is, or the roots there like I can do that. What I need help with is just the sort of big valueless. Exactly. And so what we've done is we've, we've built our system is such that it is made for those open areas. And so when a lot of people do that, they're like, Well, I probably can't use it. Because you know, I, you know, I, you know, I do this one that just has trees everywhere. And we're like, great. The good news is the mower still works like you normally do. Yeah, you know, like, I don't turn on cruise control, the moment I get out of my driveway, I hope you don't either. I only do it when it starts to get really repetitive and boring. And I'm like, I'm on the highway, let me turn on cruise control. And so it's the same functionality that we've developed, right? Yeah,

cruise control is a very good metaphor for those. Yeah, see, because you use it when you have a spot that's incredibly monotonous. Like, it's just this big, open thing that's going to take a bunch of passes. And thus, a big open field, that's going to take a lot of back and forth. And so that's when you use it. But for the highly custom stuff, like if you had a bunch of poles or trees to go around, in which case you don't engage cruise control, you keep the robot. And so your systems are made so that a human operator can just jump on it. Like you're not modifying the spot where the human works,

right. And that's been very critical. Even when when we first developed it, I remember a customer early on telling me, you know, Hey, your system may be great and may never break, but this is my livelihood. So there always needs to be a way to disconnect. And so that's one of the features we design and the robotic ready. If you look at our website, there's a switch from manual to autonomous and, you know, it's funny, you and I both know that it's probably not needed in the future at one point. But it is a physical safety blanket type thing when you do this, the entire system is cut off, that could have autonomous control. And so there's no way for the system to take over and sort of issue commands to the wheels and that's a very fun thing for our customers that are relieving, you know a safety blanket that they like, and it's also a productivity thing like they can always sort of say, and when you say jump on to I want to be careful to say that that when the motor is going autonomous because we have sort of 360 cameras, if you run up to it, even from the back, it'll stop. If you even somehow were able to get up and dodge it or whatever, be a ninja. And it wasn't meaning this. But yeah, but it's fun to talk about, like, if you were to jump on the back, a lot of the mowers have a thing, you stand on a platform operator presence, if you get on that, like we used to allow people to ride on it while it was autonomous, and now we don't, because it's not a safety thing. But it was fun, you can ride on it while the matter was going to be pretty cool. But now if somebody is on there, it won't allow that. Right? So you know, we've had our customers try and put their safety people on it. And there's videos of them, like running up to it, jumping on it, and just stopping immediately. We're like, don't do that. And they're like, Well, you gotta test it. I was like, okay, funny.

So it almost to me, it seems like a clever approach to do this, where you have the ability to toggle between the two, because it seems like if you're making a fully autonomous mower, and you're building the entire mower for the fully autonomous case, so you're not making a spot for the human to sit and work on it, for the drive it, it makes it so you have to solve every hard problem. And I think that that's probably like, there's a lot of them, I'm sure you can speak to like, how do you know if I don't know, you can go over these routes or go around all the things. But then having the human do these hard ones, that seems very

clever too good example would be like high grass, right? You know, so like, there's a big, you know, brush of grass. And we ran into this, right? Like, you know, one of our customers got a little behind, because they don't have enough labor. And they came out, they're like, Hey, listen, like this field is like knee high grass, like I was supposed to hit it last three weeks before, and I didn't, it's like way up here. And we're like, instead of the mower was like, like, can't go here can't go here. Like, even we have a feature for avoidance, which you know, and feel they can just have it go around. And so it's just constantly trying to go around everything. And he's like, it looks like it's drunk. And we look at the cameras. And so we look through the RGB. And we're like, whoa. And so we did design a feature for relaxed obstacle detection. And so we raised the ground plane of the ransack. And so we do let him operate with a higher, but but when we do that, again, this is all about jobs to be done. So we trust our operators, they're trained, they're certified to do it. And we, you know, train them on this. And we said, hey, listen, when you press this button, it'll relax it, you know, you better be sure that there's nothing in there. And when he did, he was like, works great. And we rolled that out, you know, two weeks later, and he was like, This is wonderful. Thank you. Can you can you add this feature? And we're like, Yep, sure can. So we keep going with them. And that's kind of how we run. Which you're right. It's, it's, it's the iterative approach, right? And you You hit the nail on the head, which is, how do we augment human capabilities, you know, especially were in these highly repetitive, dangerous, dull, dirty, monotonous things that robots are very good at and likened to, and then how do we keep the person able to do the creative, you know, unique, beautiful, and things that humans will always be good at and should be good. And, you know, that to me is the future. It's, it's what I love, like, why I do this, and why I get up every day.

It's also the So for all those are great reasons. It's also pragmatic as a startup to, because you're not doing the hardest problem. First, you're not doing every hard problem all at once. Yeah, you're doing just the minimum to provide value in a market, which is fantastic for I mean, like, I mean, I'm thinking about autonomous cars now. And you see huge companies like, say, like Cruz, who has like, I don't know exactly, but like 1000s of engineers working on this problem, to fully automate driving. And a lot of really great things have come out of that. But as a startup, it's hard to do because you don't have 2000 engineers. So solving an easier problem that still provides a lot of value, kind of like what Tesla's doing, in a sense.

Yeah. And again, we're just we're sort of copying a lot of this crazy has been amazing. Our product manager that was just in San Francisco and holidays and was like a solid couple around around he's like, it's incredibly out there everywhere. It's crazy. And you that way Mo Yeah, yep. So it's the only choice here it's just not evenly distributed. And what I tend to think about the off road capabilities, what we're doing in the autonomous space where you know, it's interesting for us is I always tell people is like, hey, when we don't know what to do, we stop. Now, I think as a self driving car, I think when you don't know what to do, like, you can't just stop in the freeway, right? Yeah. Oh, gosh. There has to be a lot more and there's people involved and plenty for us who are in controlled environments like we can in some of the center's we're working with are sort of industrial Right. So there's a controlled environment, you know, you don't have people running around. Hell, it's it's interesting, because people were like, Well, what about schools and kids and it's like, hey, why don't you ask us landscaper, what they do? And have you ever seen a landscaper milling around kids know if the kids are out, they don't know, period and the story like manually. So, you know, so So the automation is capable of doing now and I think that the offered space will continue to have leaps and bounds. I?

So when you think there's when I think about autonomy, there's a lot of standards right now, if you look at some of the bodies that are certifying on road autonomy through some of the standards, like the ISO 2626262. Yeah. Or what I think it is, it's one of those I just to say, just say this is until it repeats, but you have the

right Ryan therapy from clear path you mentioned next Tuesday, or something like this, often, okay.

And so those are the those are the autonomy levels, right? Level one up to level five were like, level five is just, you know, and level five, kind of interesting, because I've heard people say that level five would be like, You tell your car, like, hey, take me to the beach, and it takes you to work instead. Because it's like, Hey, you actually need to go to work. Like it's truly self directed, right? It's like, No, you don't need to compete. But um, the, the i digress a little bit, but we talked about on river software, there's the standards and in off road, we don't have that, like there isn't a body learning is exactly that that's, that's determining these, we do have is a lot of some of the ones for industrial robotics. And so their standards like ar 1508, which is a robotics for automated machinery, and vehicles. And those are the ones that we're going for those standards. And we've been really lucky, we've working with some external consultants for risk analysis, as well as we've got just hired some validation engineer from from Rivia. And so we have a lot of firepower on that. And I think that are, and we're working with some of the industry as well, some of the outer power industry people directly to help make sure that those are safe standards, but we're doing that in conjunction with our customer, who also like has a job that they want to get done. You know, I tell my team, we have six core values and safety first is first. And number six is customer success. And they're not in real order. You know, they're all important. But if you think about it, like if safety is like the most critical, then the safest thing to do would be to just sit in a insulated bubble and tell our customers you can No, because mowings dangerous, right? That'd be the safest it is. But in the end, there's customer success, and they've got a job to do. And while it is dangerous, we accept those risks. And we the reason we accept this risk is because we've done enough risk analysis to know that like, hey, if I do this safely, and I follow the procedures and the guidelines, the risk is low enough, that I'm safe to do this. And so where we fit is trying to bridge that gap between I've got this job to be done. And I still want to do it safely and with the least amount of risk possible. And I frankly think that when we look back in the in the future, we're gonna look back and say, Wow, can you believe that we had humans sitting standing riding on these things that were prone to rollovers and a huge spinning blade? Yeah. And we'll be like, oh, yeah, we used to used to drive around in cars and smash into each other. Oh,

no, I'm just gonna say, yeah, yeah, the cars are so dangerous, but, and it's funny, because, like, they're still relatively safe, but they are one of the most dangerous, dangerous things in modern society. So I think, like 50 years from now, it'll be like, our parents that we were driving around, like the risk, but

it's true. Yeah, unless you walk or bicycle and bicycling is much more dangerous. And city it is.

Speaking of bicycling, you got to get, you've got to get Brandon from Luxan us on your podcast, the homework that he's been doing with depth a AI and the depth AI platform. These are our cameras that basically run AI ml on them. And they have great ROS drivers. And we're helping a little bit with that open sourcing a lot, but phenomenal and vegan advice. And he started this company because he had some friends who and I won't mess with his story. But they had a some accidents. And he was like, how do we prevent this? And he's like, we got to get a camera that faces behind. And so he tried to prototype one and was like, you know, how do we do this and, and with AI and ML where it is right now he was able to do it. And so he open sourced the platform so that other people are able to do it. And we're actually excited to start implementing that on our next gen, which we've already started. And so in terms of, you know, think about like the real sense and the depth. Cloud is a great protocol for algorithmic but like I said that AI stuff is coming fast. I mean, the stuff that they can even do, they can even output a Deaf cloud using you know, something. Exactly, exactly. And the quality better than stereo and these kinds of things and I've been super impressed and they fast. So we're definitely got to get him on. Because he's, I mean, in terms of robots, they're gonna revolutionize it, I think.

Awesome. I'll look into having him on the podcast. Thank you for the recommendation. One, one thing that I'm wondering, like, where do you see standards bodies being helpful in these? Is it because they establish well defined protocols that are well thought out? And thus they reduce risk? Or where's the value in establishing these standards? Yeah, I probably

not the best person to ask this as an entrepreneur who thinks that all rules are there for somebody else. I'm kidding. The rules are meant to be broken. I don't believe that, but I'll say this, I mean, I benefit from them every single day. You know, like, from, from the headphone jack, I just plugged into to the USB that I had to plug in, and then turn around and then put back and then do it that way. Right. Those are all standards, right? Web Standards. You know, ROS being a standard, the rep, like we talked about, I'm a huge fan of those. And so what I think where, where I think the best things get built, is in a company is leading, and really driving that. And then it isn't all driven by consensus and committee. Now, I'm a big fan of committees, we're on a few. And but where I think that you can get in trouble is trying to design by consensus, I have a rule in our company, and my team will will kill me for saying this. But there's three types of decisions that are made at Green sea. And this is all companies by the way. There's command control. There's consensus. And then with there's consultative, right, so let's go through the three. So in command is when I come come in the room, and I'm like, Hey, guys, or hey, people, Hey, everybody, Hey, person's everybody. We're doing this. And everybody goes, Okay, let's do it. consultative is, His command is pretty good. And sort of wartime and scenarios when you need somebody just come and say, Hey, this is what we're doing. We make decisions. Yeah. consultative is very good. And consultative is very critical, and probably the one you should be doing in business. And the most important, that's the one where, where someone comes in and says, Hey, there's a decision that I'm in charge of making, because I'm responsible for this, you seek expertise, correct. And you said, I'm seeking your consultation on it. But ultimately, I have the call. And that's how all decisions in business should be made. The only decision in business that should be done consensus is where do we go for lunch? Yeah, because I just think that the that watering down of sort of, like, you know, we've got to get all everybody to buy in and fully understand it. You know, if you had to ask everybody, just nothing would get done. And so I do appreciate committees that are the appropriate size, that are in that actually have stakeholders who are committed and not just in it to sort of Express power or sort of lobbying or interest that aren't their own, then I think you have very good standards, and I'm all for that, you know, so again, you get real world, you know, and that's what I like about the ones we're on there. They're not from, you know, people who are just trying to stamp this out or competitive. They are people who are actually like, nd user, you know, who benefit from it. And where I think there's benefits is that, you know, we've seen this in the automotive industry, you know, there are standards bodies that say, like, Hey, here's what we think true sort of autonomy would look like. And these are some of the things we do you know, it's funny, there's a simple one that you know, I got a new car. And when you go to the back, there's a liftgate and there's a button, and all the iconic, iconic RV, is all sort of standardized, right? And so, you know, it seems funny, like it's an SUV, but the the Connor icon or Greg, I forget it, the icon itself is that looks like a card or something, right. And it's a hood closing, but it's I know, when I see that button, I click that, and I know that the hood will close. And that's because of standards help us all. So we all don't have to read the manual every time we do something. They really help with usability safety, and I'm a big fan. I do do sense, though, that there are some that are probably gotten too big or don't have appropriate stakeholders that matter anymore where I think a lot of damage can be done.

How do you determine if it's too big? Oh, or how do you figure out the one size?

Yeah, I'm not the one to determine it. I think it's all context dependent, but I'll quote Yeah, he says, because he seems to be the smartest man on the planet. He says, What is it the pizza rule? Right? So if you can feed a pizza, that group is fine. It's the fine size but anything bigger than that? He's like, get rid of it. So everything is done in feeding. I think it's I think it's the pizza rule and maybe two pizzas. I'll have to look it up.

Something some small number of pizzas. Anyways. Interesting. Okay, going back to your autonomous system a bit more. So you have many depth cameras, they're pointed. So you use odometry, odometry. And GPS. You're doing some sensor fusion there and is fusion. Correct. So we local and global, right? Yep, go ahead. And then you have cameras, depth cameras. So you were mentioning the Intel real sense, which is structured lighting, I believe, or maybe it sends out a laser or something

it does in the outdoor, it probably ignores a lot of the pattern project that happens the IR projection because of just using stereo, or what doesn't it works pretty well said is is stereo depth, right? Yeah. And no, in between them. Yep. And so

you can infer how far away everything is by the disparity in the images taken by both of the cameras and go, Okay, so you have those, those are facing all you have, like, one of them on each corner, or they're all around

the 360. But what's interesting is that there are directions that a mower cannot go just like a car, right? You know, you can't see a car can't suddenly move laterally diagonal back. Yeah. Perfect. Yeah, exactly. Exactly. Yeah, exactly. Not morphic. And so having said that, we have pretty close to 360. But we actually have a bias towards the front being more important, because that's the primary direction that it goes. And so that that sensor is actually a little bit duplicated. If you look at our current one, we have a little bit of overlap. And those are calibrated, like I said, to make sure that we can see. But yeah, so we have very good coverage around the mower, and then in the back. But there's I mean, there are definitely sort of areas, you know, sort of here where it does not move, but the

left and back right kind of thing. Yeah. Yeah. Okay, cuz you're not as worried about someone approaching in the mower, run running them over

for an even go that way. Exactly. And even the turns that it does, as soon as it turns it sees you and he's like, Well, I can't finish that turn, right. I see.

And yeah, these are you said, it's like two wheel drive two, and you can turn in place because the front wheels or casters.

That's right, he's got all that zero turn or turn on a dime. Right. And so what's interesting is when I first we first got the the mower, you know, we're I used to be able to say I was new to landscaping. Now I can't write having been in a while. And so when we would do, we would go out and we'd be like, Oh, cool. Let's just turn and we would just zero turn right there. And when you do it, landscapers will go like, Oh, you just rotted the grass, because it's true. Can you imagine? If you have one wheel that stays stationary, and you pivot the other you especially good? Yeah, you actually tear up the grass. And so our first turns were terrible. You don't say don't put one wheel forward one wheel in reverse. You do now. And that's what our algorithm does, right? So that's what we do is we call it a perfect, they call it the Y turn or the you know, and so you can watch our machine do it, but it goes forward. And then it actually uses its own an inertial I guess. I don't know what you call it, but it's sort of kinematically so that it backs up, and then goes straight, and it lines up those casters perfect. And so it executes what is a very good turn. And when when landscapers see it, they're like, Whoa, it looks like an excuse. That's right. And it doesn't run the grass. And of course it does it every time, which is kind of fun. I mean, if you're in a hurry, you can always just you know, get out of there. Yeah. Well, what's great about a robot, right, is that a robot is one time to do it. And it's like, I'm happy to do that exact same turn every single time.

Yeah. Okay. So do you have any other sensors?

On your outside? Let's see. So I think we are. Yeah, so we we've had ultrasonic video on it. Ultra problem with ultrasonics is and that was shortly that some of our machines still have it, but we're not using those. And I'm happy to disclose why that at the speeds inside times we were going it was only turning on for a very minimal amount of the thing because, you know, we're going I would say pretty fast. And we have been insanely happy with the point clouds. And we've actually our newest generation has a lidar that has added a secondary form of redundancy. We're actually using the ouster, and we're super happy with that. And we actually on our current gen, we use the live ox LIDAR as well. And with that, and the depth sensing cameras, we've got our redundancy and it's much better than the the ultrasonics were really fun. Yeah, they're very noisy and at the speed and vibration we were doing they just weren't adding a lot of value. There was no they were not making the system safer, which was the whole point. They were adding complexity to the BOM cost bill of material and so we just we decided to cancel them I would be I'm impressed with some of the new stuff that's coming out. We're always evaluating sensors in our lab downstairs. We have some fun stuff that I probably can't talk about, but there is some unique stuff out there that are in the ultrasonic and sort of i What would you would call closer to that almost audio type spectrum than visible light, right, that I think will be capable. But the neat thing about what we're doing is, again, if it's a structured data and we can use it, then we will incorporate it as part of our standard and spec. But as of right now, those ultrasonics are tremendous. A lot of it. Yeah.

Yeah. And so for LiDAR, are you using? Is it a planar lighter? Or is it one that's sending out a few strips, or

the live box is a unique? One that I think is I am probably not the best to talk about, we have a whole systems team that evaluates a hardware, and I saw a couple of them come in, and I definitely saw the pricing of some of them coming in. We have stuff in our lab. You know, I think we, so we're venture backed. And so what's interesting to me is, and I've never I've been always done bootstrap companies. But in venture, we're trading a capital for speed and growth and, and have making our customers happy. And so what our team does now is and it's real fun to be on the green team, because they don't come to me. They don't come to me period anymore. But they don't come to me and say, you know, hey, there's a $35 IMU and $1,000 IMU, which you know, they buy, they buy two of each one, and we evaluate it, we choose the best one, right and, and we FedEx Ship it like overnight if we can, right because time is of the essence. So yeah, they do that now. And we know that. So I think the one i The ones I've seen are the ouster OSC, Ro 128, which is the highest grade one. And the reason is because, you know, we all were software heads, too. So we're like, let's buy the 128. Because if we, if we can software, make it in software to a 64. Or the lower Rez like we can do that in software? And we can figure out if, you know, yeah, right. So we can always you can always go down lines in data and destroy data, but you can't recreate it. So I've seen some of the you and I've seen some of the initial work on that. And it's been phenomenal, like the amount of you know it with our depth cloud, you can tell what things are like you can see humans and you know, because it's hard to visualize, right? Like as humans were not good at visualizing a depth cloud, even though there are some things to do it with color and things like that, like red is close. And blue is far whatever it may be. But with the the LiDAR, like the amount of data in line scanning that we get with both of those units has been phenomenal. In terms of range.

Interesting. Do you think the LIDAR will ever replace the depth cameras?

a prevalent probabilistic guy, right? You you, you nail that ahead of time. And so I think that the world is full of dirty sensor data. And you should take as much of that dirty sensory data as you can apply probability to do the task at hand. And whatever comes out needs to be the one I don't think that there is an end all be all. Yeah, it's funny. I mean, you look at like the Tesla. And I know that they're all super deep on vision. But yeah, we know I have some friends in the industry have been doing this well, and they're like, hey, like, fog. You know, like, they're like fog. And they're like, so this is like a known thing. So I would not be surprised if, if, if the future is better than than humans. And it should be. It could be. Oh, yeah, totally using sensors and things that we don't have. We just, we happen to see in RGB. So we send that tend to sort of go for that. But there's no reason why the world of a robot shouldn't have be like an insect that can totally see, you know, completely different spectrum than us. Right. Yeah. Infrared seems very useful for a lot of things. Like all sorts of other things.

Okay. That's really cool. The let's see. So, these are your sensors. And then do you have a you have a computer onboard? You mentioned it's an x86 architecture. What is it? Is it like a super powerful computer is the now

here's, there are a couple other things. I mentioned that so the computer is not very powerful, but I would say it probably is. The main thing though, is heat. So they're ruggedized industrial din, rated for fanless for heat and vibration, and those are critical. And by the way, we have hit the limit on those in Texas several times. It do not mess around in Texas. They do not mess around with mowing. They have practice fields on practice fields.

Wow. That's crazy. That field for mowing,

you think it's practice fields for football, but and, and all this, but it has just for mowing? Yeah. Well, we have those. We have those but Yes, but But yeah, and so we'd be out there and it was 105 Fahrenheit, which is I don't know what that is in Celsius. I should because I'm 30 pharmacists close to. Yeah. And so our computers are rated for I think they may be up to 70 or something like that. which is nuts 7070 Celsius, which is like 140. So I may be off and maybe 40 and 140 because I'm not very good at the conversions. Because I have Siri that just tells me. But anyway, the point of the story is that we hit the limit. And our USB bus died and fell off. And so all the cameras were gone. And of course, the mowing is like, stop. But what's funny is we were like, We were apologizing their customer customers, like I'm done to me. They're like in terms of human like, we shouldn't be out here either. Like at when it's 105 humid, and our you know, yeah. So they're like, it's, you know, you're good. You're good. buy our books. So but what is cool is we did fix some of that, you know, when it made it more robust? Yeah, we've had heat shielding. And some of the, you know, exhausts i We got one of those flot FLIR Thermal guns to flir measurements. Yeah. And I was surprised how hot do you think it does get on the at the exhaust point of the engine on a mower in Texas? Like, what's your guess? I don't know. 180 730 coming off, the exhaust says, That's pretty well, I guess. Seven? Yeah, it's very hot. And so we originally had our sort of exhaust sort of blowing towards the back of the computer, and there's this area. And so ours would just heat up to where it stopped. So now we vented it differently. We've added some extra shielding with air, which is another reason why we work we love working with the manufacturers, because they're like, we've been dealing with heat shielding forever. They're like, yeah, what you do. And they just added three things. And they're like, done, and we're like, oh, cool. Check, check.

I don't think we're done yet. But there's that now we do. Also, we love microcontrollers. So we're a big fan of, I think we use Arduinos, we use Teensies. I think TNCs are the current generation. And the more we can do our we have another theory about probabilistic robotics. But another theory is that the more we can do at the low level hardware, the safer the system will be. And so anything we can do that that becomes sort of rote and is not in C++ or something high, like at a high level, I would call C++ a high level language, even though you can go low, you know, we can move that down to the microcontroller and have a bonafide you know, thing that when it turns on, that's the only thing it does. And it's dedicated to the really single purpose. So it's the same reason you like nodes. In a sense, you're exactly right. Yep. Okay. Well, I'm super excited about some of the ROS 2 you know, stuff that you can run, you know, oh, yeah, a device where it just outputs a topic. Yeah, embedded, whatever, right. Yeah, that's gonna be really cool.

CBQ's website

Greenzie's website

Greenzie's YouTube channel

Commercial-grade Autonomous Mowers, Safety, and Dogfooding

Does it Detect Sprinkler Heads?

Making Perfect Turns

Seasonal Workers

Leap of Faith

Differences Between Kids and Poles

In this episode, Audrow Nash interviews Erik Schluntz, co-founder and CTO of Cobalt Robotics, which makes a security guard robot. Erik speaks about how their robot handles elevators, how they have humans-in-the-loop to help their robot make decisions, robot body language, and gives advice for entrepreneurs.

Having their robot take elevators

How the sensors are used (aka, fulfilling IoT promises)

Training the robot as a security guard

Adding a taser on your robot? + What it means to be a security guard

On being a technology and security company

Stopping a thief with their robot

How fleets of security robots are controlled

Understanding the human operators

Tech stack

Experience with ROS

Extending ROS packages to work with elevators

On open source software

Growing the team

Manufacturing the Cobalt robot + battle-testing their robots

Sourcing components

Making people feel safe around the robot

Robot body language

“Dog-fooding” to improve robot body language

Working with the research community

Having the Cobalt robot behave like a receptionist during the day

Impact on security guard jobs

Human-in-the-loop with robotics applications + levels of autonomy

How are you using machine learning?

Erik’s background in startups

Advice on picking a problem

Cobalt’s funding

Advice for founders looking for investors

Advice on avoiding bad investment

Advice for a younger you

Getting in touch

Hi, everyone. Welcome to the Sense Think Act Podcast. I'm the host Audrow Nash. And in this episode, I'm talking to Erik Schluntz, co founder and CTO of Cobalt Robotics, which makes indoor security guard robots. Erik and I talked about having the Cobalt robot take elevators, how Cobalt has humans help their robots make decisions, robot body language, advice for entrepreneurs and several other things. It was great to talk to Erik, I enjoyed learning about how cobalt uses humans and robots in their systems, which each play to their strengths. It was also a great seeing how cobalt robotics has navigated getting investors and funding. As always a big thank you to our founding sponsor, Open Robotics, and I hope you enjoy. All right. Hi, Eric, would you introduce yourself?

Hello, I'm Eric Schluntz. I'm the co founder and CTO of Cobalt Robotics.

Absolutely. So Cobalt builds indoor security guard robots. So these are robots that will patrol around an office building, looking for anything out of the ordinary. We work with human operators to help support the robots. So if the robot comes across anything, it doesn't know how to respond to it switches over controls are humans. And then our humans can escalate exactly like a human guard on site would, whether that's calling the customer or calling police, or having a friendly video chat, instead of asking how someone's doing just like a security guard.

Nice. And so for the people listening? Would you describe what your robot looks like? And how do we refer to it? What do you just Is it the cobalt robot

or just a cobalt robot, we actually we decided we didn't want a separate product name from company name. That can be hard to to brand both of us. So it's just a cobalt robot. And I'll describe this actually, while playing a video for people that are watching. Sure, we'll get this going here. So the cobalt robot is about five feet tall. And it'll control around it has wheels, two main drive wheels. And then for supporting casters, it's designed to navigate around through an interoffice space, looking for anything ordinary, that can include doors or windows that have been left open, the body is actually coated in fabric. So it's a little bit soft and squishy. And this was actually a big part of our industrial design was that we wanted the robot to be friendly. The job of a security guard is to keep people and employees feeling safe and comfortable. And so if the robot is scary, you know, it really hasn't done its job, right. There's also a touchscreen on the front of the robot that we use for being able to interact with anyone around the robot, where one of our human operators can video call and talk to a person on site. We also have a lot of sensors, cameras that are 360 degrees around so no one can sneak up on the robot. And then a ton of sensors that you find in a self driving car like LIDAR and depth cameras, and I am Yes.

So how I'm thinking of it. It's like a telepresence robot that has a lot of sensors that makes it more sophisticated. And then you have the whole like person in the loop thing, which makes it a good bit more sophisticated. But is that a like a squishy, a basically a telepresence robot with a kind of cylindrical fabric body?

Yeah, yeah. And I'd say that's what our design is basically that it's telepresence five or 10% of the time. And I'll tell you this 90% of the time.

Okay. Now, and the one behind you has a little arm.

Yes, yeah. So this is one of our add ons for some of our customers. For our security patrols, we want to be able to patrol through multiple different floors. And so to do that, we need to ride elevators. Now in the past, we've done elevator integrations that are wireless. Those take a long time, elevator companies are often very slow. And so actually what we do now at a lot of customer sites is just physically press the button. So we added a one degree of freedom actuator, swing up and press elevator buttons. And we use the cameras and computer vision to detect the buttons know where we should press. And we actually have this rolled out at a bunch of customer sites now. And it is a lot faster to roll out in a wireless integration and it works pretty well. So yeah, I'll show a video of that for people at all kind of narrate that. So the robot first starts and presses the button on the outside of an elevator. waits for the door to open, navigates inside. elevators are very technically challenging thing to do. We find tons of things like elevator thresholds that are actually out of spec that would cause a lot of problems for people in wheelchairs.

What do you mean? Is that like a big bump getting into it?

Exactly, you know, we find elevators where it's a two inch gap, you know, between the elevator car and the floor or you know, one inch vertical, but once the robot gets inside, it uses computer vision to see the paddle and know which button it should press, flex the correct button drives forward, depress it. And as I mentioned earlier, this arm that we add is only a single degree of freedom. So we actually use the robots drive train to steer the robot left and right. And to move forward and backwards, the arm only moves up and down. Yep. And we use the robot for the rest of the degrees of freedom.

You may I think you mentioned so it has to the cobalt robot has differential drive or something so you can turn it's not Omni drive, so it doesn't move side to side.

It's it's differential drives, we have two main drive wheels and four support casters.

Okay, is that tricky to get lined up with the button?

Yes, so we definitely put a lot of work into sort of that motion control and planning to be able to nicely line up and press a button. Especially because elevators often have very tight space, we don't have a lot of a lot of room just sort of back up by 10 feet and approach it nicely. It's, yeah, we have to kind of operate in these very tight spaces.

Gotcha. I wonder, do you have ever the situation where the elevator is so tight that you have to raise the arm and then kind of swing into it?

We haven't hit that yet. And luckily, there are rules around elevators, for people with wheelchairs. So we can actually just follow a lot of those same design specs to know what the robot hardware needs to be capable of gotcha.

And is it difficult when you go into a new elevator? To find the buttons? Like I imagine that this is fairly custom, depending on the use case, depending on how you're detecting things?

Yeah, so we have a sort of a proprietary system that we've set up where one of our sales engineers can go on to new customer site with the robot, and very quickly calibrate the robot for that site. So we don't need a ton of training data or anything new like that. We can kind of just bring the robot through the elevator once, and then after that, it's able to use the elevator by itself.

That's awesome. You kind of... Can I guess that what it is?

So do you use the depth camera at all in this for figuring out how far away you are. And

Yes, we mainly use the LIDAR for that. One of the other interesting challenges is that elevators are very reflective surfaces, a lot of times you have sort of a polished metal. And time of flight depth, the cameras can do very poorly with that, because

they're shooting out infrared and trying to see that but then it bounces off. So you don't get any good depth information, maybe very noisy information,

you get very noisy, but the LIDAR at the bottom of the robot is pretty good. And we know that in an elevator, it's a fairly structured environment where we see the walls is usually in line. Why is LIDAR

more reliable? intuitively, that doesn't make much sense to me. So for the depth cameras, are they structured lighting, where they're projecting something out? And then trying to observe what that is, with the pattern and the deformation in it? Or?

That's a great question. So the one we use is not structured light, it's time of flight. Oh, and I don't think there's any inherent reason that a time of flight depth camera is is worse than a LIDAR, which typically, what we see is that LIDAR is are a lot more powerful, and just put a lot more into those be able to get a much better signal response.

Gotcha. And the LIDAR of this robot is at the it's like, on the base, right?

Yes, at the very bottom of the robot, and you can see around 270 degrees around the robot. So that gives us kind of a good sort of footprint of what the the room that said, Yeah, and then we use the optical camera on the front of the robot, and can get very good sort of optical readings, write up the buttons and see where they are.

Yep. And then how do you control the press for this, I'm imagining like, you draw, you basically raise that to the correct height that you imagined the thing is, and then you just drive forward until you hit some sort of drive

forward. And we have a set Oh, very tip here to know whether we've made contact cool little forest, what are the eggs? Initially, we had a force sensor, and then for reliability, we just switched it to a, you know, a switch. That's, that's calibrated. Yeah, one of the interesting things is sort of, you can we can tell that we've hit something, but you don't necessarily know whether you engage a button or the or that you know, the rim of the button and maybe didn't engage it. So there's some interesting things we can do to you know, to sense whether we'd hit that one of the first things we thought of is oh, you know, we can tell whether the light, the light turned on in the button, but you'll actually find plenty of elevators that don't light up. And so actually what we do now, is we have an accelerometer of the acceleration of the elevator.

Exactly that's kind of you know, the sort of that's like the the ultimate end goal is we feel like the elevators not moving. We know that we've correctly told the elevator what to do.

Now is there ever the case where the robot is in the elevator pushes the button, but Miss This so that it didn't actually click the button and then someone calls the elevator from a different floor, and it sees it.

Absolutely. And so I think what you brought up is that one of the really interesting challenges of this is that when we are in an elevator, in the worst case, we're inside a Faraday cage. And so this has to be often this has to be 100%. I'll

speak about it in a cage for just a little bit. Yeah,

absolutely. So a Faraday cage is a closed metal structure that no radio waves or electromagnetic waves can get in or out of. And so that means that for some elevators, when we're in them, we don't have Wi Fi or cellular. And that means that we can't rely on our human operators that generally we use for ourselves.

That's hard, because it's like one of the more difficult situations. Exactly, exactly. They're in a Faraday box. That's crazy.

Yeah, so it's very funny but And generally, what we say cobalt is that one of our sort of really fun advantages for our engineering team over say, working at a self driving car company, is that we just have a lot of inherent safety and stability that we move slowly through indoor spaces. If anything happens, we can just stop and call a human operator, you can't do that on a freeway. And so that means that we can basically solve for the 99% case, and we don't need to worry about solving the last percent bring anything in the elevator, except in the elevator. That's where we need to get 100% You mean safety

instability? For this yet safety and stability and stability?

Yeah, you know, we're not trying to balance on two wheels, it is always a safe situation for the robot to stop. And so we can take a lot of advantage of that, and basically, handle covering a lot of edge cases with that safety behavior of I stopped the robot and wait for a human operator, except in the elevator, then they realize

3 posts with the safety tag