In this episode, Audrow Nash interviews Brett Aldrich, author of SMACC and CEO of Robosoft AI. Robosoft AI develops and maintains SMACC and SMACC2, which are event-driven, behavior state machine libraries for ROS 1 and ROS 2, respectively. Brett explains SMACC, its origins, other strategies for robot control such as behavior trees, speaks about the challenges of developing software for industry users and hobbists, and gives some advice for new roboticists.

Start

Introducing Brett and SMACC

Events in State Machines

Clients and Client Behaviors

State reactors

Explaining dance bot + hierarchy in states

Recovery states

Origins of SMACC

SMACC and market pull

Robotics domains using SMACC

A problem to push the limits of SMACC

Making ROS packages smaller

SMACC for industry users

Making SMACC easy to use?

Control in many robotics applications

Comparing state machines to behavior trees

Future of SMACC

Advice for those starting out in robotics

Links and getting involved

This is a conversation with Brett Aldrich who is the CEO of a Robosoft AI. Robosoft AI develops and maintains SMAC, which is a library that aims to make it easier for roboticists to control robot behavior. One of the key ideas in SMAC is that a finite state machines, we'll be referring to finite state machines a lot in this interview, so I'll explain what they are. Before we start. Imagine that you have a lawn mowing robot. Imagine that that robot is powered off, and that you turn it on by pushing a button, it boots up and drives to a field to mow. When it gets to the field, it starts mowing. Once it finishes mowing, it drives home and powers off. This robot has states like powered off, booting up driving, and mowing. The robot also has transitions that occur on events, such as when the robot is turned on by you pressing the power button, or the robot arrives at the grass to mow these transitions change the robot from one state to another. For example, the event of arriving at the field to mow could transition the robot from the state of driving to mowing. So summarizing, a finite state machine is something made up of states like driving, and mowing, and transitions when the robot has reached the destination, or when it has finished mowing. This is the sense think act Podcast. I'm Audrow Nash. Thank you to our founding sponsor open robotics. And now here's my conversation with Brett Aldrich. Would you introduce yourself?

My name is Brett Aldrich, and I'm the author of Smak. State Machine library for robotic applications written in C Plus Plus, I'm also the founder of robust soft AI. Robot soft. We're the keepers of the smack tool chain. We're also the creators of the smack to runtime analyzer. A graphical application lets you visualize your state machines both statically and at runtime.

Hmm. And what so can you start at a high level and explain what SMAC is, and then what its motivation is? Sure.

SMACC is a state machine library for robotic applications. I'd like to describe it as dealing with the highest level of task planning and management on a robotic system. It has many unique features. And.

Okay, so it's a, it's a high level state machine library that helps you figure out so you get to control your robotics application with it.

It's a control, it's an architecture for control. And a couple of things about it. You know, when you're writing your state machine, what one it it provides a tight integration with. With ROS, allowing you to generate events for ROS topics, services actions right out of the box. It also offers a reference library of state machines that are guaranteed to compile and run right out of the box.

So by reference library, you mean a bunch of state machines you can already use for specific things. Okay.

Exactly, exactly. With, with simulation included for many of them. Gotcha. It also offers a client library. And this, this is part of the architecture of SMAC. That allows ROS users to use ROS packages like move it to or nav to right out of the box. And it also provides a pathway for other ROS packages. And for the interfaces of those packages to be developed by specialists that are then tied to your state machine it all of these features have the effect of dramatically lowering the bar for new developers to get started in ROS, and then in programming in general controlling their robot. I can go on quite a bit longer about this, but I would say so it's a good high level overview of what SMAC is,

is it kind of so that if I if I'm using smack, I can kind of compose more complex behaviors out of almost like blocks that will affect their behavior and I can, I don't know, like, if I have this sensor, I can run use this block to send out the data? Or is that how it looks kind of like a bunch of blocks that I can compose together?

Yes, yes, you have, you have state you have states, right? That you can let us to divide your problems. You have hierarchy, which does much the same thing. You also have the concepts of orthogonal, which are like boxes that allow you to modularize the physical subsystems on a robot.

Okay, and by that you mean kind of control them independently or use them into basically, right. Can you give me a more an example, I suppose?

Yeah, absolutely. In fact, I'm gonna show a robot that that was recently built using smack as an example. So this robot is from field work robotics, it was built by them along with ourselves in Bosch, India. And as you can see, there's four robot arms and a couple of trays, and navigation based, so looks like a fruit picking robot or exactly. It picks raspberries. Yeah. And it was based on the work of Dr. Sutton's at the University of Plymouth and his soft robotics lab.

So that just to describe it for the audio listeners, it's kind of like a big moving cart with four different arms. That probably it that's on wheels and probably drives through and picks stuff for it. Would you say? That's, that's a good description of it, I think to add Exactly, exactly. Okay, so use this.

So each orthogonal, contains an arm. So there's this orthogonal one with each arm. Another thought for the base, there's probably two more orthogonal is for there was like, like a packaging. Not exactly sure what you call it, almost like a gantry. A small gantry inside for for packaging, the raspberries. You probably also have your cameras with perception, all of these would would go into an orthogonal using a model of let's say, the human body. You might have orthogonal for your eyes, another two orthogonal for each arm. Yet another orthogonal, so is

it? Can you think of it kind of like a degree of freedom? Yes, in a sense, yeah, you can. Yeah, cuz, so I mean, I'm thinking of so orthogonal, which means, like, it has a technical definition, which I think is it's the dot product is zero, or something like this, right? So it basically means perpendicular, but it allows for higher dimensions. Right? And so it's like a new dimension, a new dimension and state space kind of thing.

You know, I, the term that is I use it comes from comes from boost HR. And it was always tied to physical subsystems. But actually from boosted, charted, inherited originally from David Harold's 1987. Paper on state charts. So in 1987, David Harrell came up with state charts. And he developed that formalism that I guess you'd call it an extension of the formalism of state machines while working as a consultant, Israeli Aircraft industries, and in particularly, in particular working on an avionics system. And so in that, in that context of a fighter jet, each orthogonal basically was tied to a physical subsystem. Hmm. Yeah. As opposed to, I think a more like a mathematical interpretation, which is, I think, where you are going more

with yes, that's where I know the word from I haven't heard it. And this in this this state machine sense. Okay. So then, so you have all these different orthogonal, I'm thinking of them kind of like a general degree of freedom, where multiple so if you said a full arm is one, it's kind of like a subsystem. And then maybe you want to do things with it. And maybe that subsystem has several degrees of freedom, like a shoulder and an elbow and a wrist, which is controlling it independently of other parts of the system. Is that kind of the core part? It's an orthogonal is something that is controlled independent of other parts of the system? Yes. Okay. Yes. So I'm thinking generalized degree of freedom. So multiple degrees of freedom

And can you have can you have orthogonal in orthogonal? Like, if I have an orthogonal that's my arm? Can I have that broken down into orthogonal is that are my shoulder, wrist and elbow? Is that kind of this hierarchy? Okay,

so I think the short answer is no. The orthogonal is can be quite complex, and so you deaf We can have multi joint, real, you know, arms in

But one of the things that that that we did we being Pablo and I in terms of the robot, sorry, what? Oh,

sorry, I thought we were still talking about the robot for picking

things in Yeah, not for that robot in particular, but in, in SMAC, as a library and viewing smack in a historical context. One of the things that that we we did was, we took the orthogonal concept that David Harrell and Andreas Huber Dawn, put into, into, you know, instantiated essentially in boost a chart, and we simplify, and because up until that point, so I think the world of Andreas Huber Donny, but he didn't, he never really used the orthogonal and the orthogonal in a, in a, in like a context of a PC really don't make that much sense. And so we took them and began to use them, and we saw shortcomings with them. And so we, we simplified the orthogonal structure and, and then what we did was we added this, this architecture part of clients and client behaviors. So inside of each orthogonal, so visualizing, and orthogonal is a box, right, a box inside an eight box, there are generally two to further boxes inside that have a client and a client behavior. The client is has a lifetime of the entire state machine. So if you if you have something that you have some piece of code that you want to effect that robot in every state, then you just put it in the client. So this would be something like, let's say, you've got a connection to a robot arm. And if at any time you lose that connection, you want some event to be thrown, you put that in the client, the client behavior contrast,

oh, it's like a global state in a sense. Yeah,

there is there was a blow valatie to it. It's a different type of state, though, in that. Typically, in smack, we're talking about states in a control sense. The states that are stored inside clients are more like, like an image state. This gets confusing actually, what it's like when you hear about something like state estimation, it's more like an image in a sense, right? So that part is stored, whereas the control states are always explicitly controlled within smack the state machine back to the talk. So if you've got clients that are scoped to the life of the state machine and client behaviors, they're only scoped to the life of that state. And and you can have more than one client behavior at any one time. And I would say that that's where really the modularity comes in. That's basically how you interact with the orthogonal is through the client behaviors, the client that sets up, you know, would say, all the communication and all the kind of stuff that you want throughout the lifetime of the state machine. But the particular behaviors are in are in these modular client behaviors that

so the client is like, it's the manager of, but so how do you call it so you have the client and the client is in a? What was the word again, orthogonal? That so you haven't or thought so you have like a global state, and then from the global state, you have your orthogonal, your orthogonal is some piece that can be controlled independently of all of the other orthogonal, and then inside that orthogonal, you have a client, and that client is going to control that orthogonal to run several different behaviors given the state. Am I understanding correctly? Not the client

and the client behavior. But okay, what's the sorry? What's the client behavior? The client behavior is different in that its lifetime is scoped only to that state?

And how are we talking about states? And this You were saying something about images? And what is the state in this?

Yeah, we're talking about it thinking from a control structure. So going back from like, a global, yep, the state machine is essentially synonymous with the application layer. Right? So the state machine is your application. And there's one state machine on any one robot I mean, one time, okay. Many libraries have like where you start nesting state machines inside of each other. I don't, we don't do that with you don't like it. Yeah, okay. I think it it creates problems with data with routing the data and other problems. So, so you have a state machine that It is your application. And within that state machine, you have a series of states, and they can be grouped in different levels of or higher of hierarchy. So let's say let's say back to

that vegetable picking robot. Like that would be a nice concrete example, to talk about this. Okay.

Okay, yeah, we'll use that. And also, we'll blend in an example that I heard about it for the DARPA subterranean challenge. So in starting out, or in one of these teams that I've had the pleasure of speaking with people involved, they started out with a four state state machine. Okay, so four states. So in that, in that example, you would have one state machine with four states moving forward, nor hierarchy at this point, I

see, I see one of my points of confusion. So state in robotics, and like a control sense can mean like, the thing that defines where the robot is. So this joint has this degree, this angle, this joint has the next one. So I was trying to understand this, but you're using state as an in a state machine. So is a specific state that the state machine is in. And that's what we're talking about when we're saying stuck in this. Okay. So then we're saying we have a finite number of states, those control basically the behaviors, or we can do different behaviors in different states. And we move through from one state to another as the operation proceeds or something like this based on Okay, exactly, I see an ambiguity, okay, that I was confused by. But it is it's a confusing thing. And especially, yeah, we'll just have to say state for state machines or something and states for controls or something. And that will help keep me clear with it. Or I'll try to make it explicit anyways. Okay, so you have your state.

So this is back to this example, we have four states, right? So, and we'll just name them state, 123, and four. So in each of these states, going back to the example of that field, work robotics robot, we would have, let's say, Okay, we got one orthogonal for each arm, one for a base, one for the two gantries, that's for six, seven, and let's say one for perception eight. Okay, so in each of those four states, you're going to see eight orthogonal 's, that okay, inside you, each one will have a client in every state. And then, depending on what behaviors we want that robot to express, we arranged the client behaviors accordingly. And so it provides like a modular control structure for each subsystem of the robot.

Gotcha. So if you are, yeah, and the state

basically serves is like, this gets more into like the sub state architecture of smack, but those, the client behaviors, and the clients throw events based on what happens outside, you know, so smack is a single node, operate back to this example. Smack is a single process operating at runtime. And you're also going to have, you know, other processes on this system, you're going to probably have nav two, you're going to have move it. One with perception. And so when those other nodes communicate via let's say, ROS topics or services, the the SMAC clients will turn those messages, which you'll specify into events. And then those events are that's cool ordinated by the state, like, like a conductor would, you know, on this event, we transition to this state, at the end of the day states are a lot of people, you might you might incoming to state machines, you might think that states, that's the state machines are about states, and they're really not they're about events. And okay, it's through events that you begin the subject of states almost, it's like, it begs the question, you know, like, when you have an event, you have what was before and what's after, right?

And then can you define an event? Can you define event,

an event is general explanations. It's like some type of discrete.

So in ROS, it would be have I received this message or not sure. Is the event is the message occurring, this kind of thing.

are they action completing or something like that? So if I'm using naff to, and I want to move from one spot to another spot, the action completing of me moving to the To location could be the event. So it says I'm in this location event has occurred, I then occur.

And then when that event is occurred in some in smack in a state Yeah, you specify often where you will like what the next state you'll transition to is.

So are we always transitioning on events? Or should you always go to another state, or you can probably stay in the same state and have several events. I'm just trying to think of more complex behavior. But so are events always used? They're not always used for transitions, but they

not always but and here, we get into the minute you start using events, you start using more. One of the another one of the features we added to I'm saying these are things that I think are interesting, historically insensitive, like what's unique about smack, you know, versus a blue state chart. And what happened before we we brought in the concept of state reactors.

Let us define a few things just because there's a few things that I don't quite understand before we get into state reactors. So I don't quite understand the difference between clients and client behaviors.

Sure, it comes down to the lifetime client, oh, or lifetime to the state machine. And behaviors have a lifetime specific to that event.

Oh, okay. So so, so client behaviors are just for behaviors? Or how does it what why are they named that? Let me just see, I think my smoke alarm is gone, or some smoke alarm?

I do hear a beep beep. We just filter it out in the thing. So long as I don't smell smoke and whatever. Yeah. Okay,

you do get out of there. Yeah, for sure. Just gets into like the systematic architecture of code layout SMAC offers. So let's say you have something that you want that robot to do, like some type of event that you want to respond to in a particular way. But you want that to be throughout the like every state of the state machine, you put that code in the client.

I see. Okay, so and then. So you're kind of you generate these client behaviors, when the event happens. And that will take care of a specific, it's the kind of created and destroyed on a specific event, whereas the other one occurs all the time, I'm thinking of them, like, kind of simple, but for like web pages, or something like you always want to load this one thing, it's going to happen no matter what page you do. So it's like the header or something, that's always going to be the same. And then your client behaviors would be the very specific things that are on a specific route, in this case, yep. Is that a fair way to think about it?

It's not very good, and the only nitpicking thing I would have there is that it's there not so much. The client behaviors aren't created with events, they're created with new states. So one state to another, that's when the client behavior is created at runtime.

Okay, I see. Okay. And then you're gonna so there was another thing that you were going to introduce?

Yeah, state reactors. So this is another type of this is all part of the the sub state architecture. So like, once once we're in a state and looking down at the pieces inside of it, what state reactors do is so

what so in the sub state what composes a sub state? Is it so other states within states or are there so others?

We do have we do have? We do have hierarchy? Yep. But when when we're talking about the the, but to in the hierarchy, one thing that that is unique and smacker? Or maybe, I don't know, what's one feature? You we break down the library into having container states and leaf states and it's the leaf state that has the orthogonal and the full substate architecture, V. Okay. The container states don't by default. You can turn them on. It's smack we follow the philosophy of we do things in a certain implementation that you know, as we see that we think is best but we try not to turn off like other ways of doing things because some of the early clients that we had really challenged my views of how they would interact with the system. And I'm thinking to you, Bob peetha, gray, Orange. They, they did things that I, what, what are you doing? You know, it took me back. But, uh, but it totally valid way of running things. So we try to leave those those sorts of things open anyway, back to the hierarchy. So. So generally, there's always a leaf state that has orthogonal and other things like that a state reactor is is a just another one of these objects. And what it does is it receives events and throws its own event. And so it basically brings in second order events, the first ace, so we came up with it. This is in like, mid 2018. And we just begun the library in like, at the end of 2017. And we were contacted by the company here.com. And they were working on a self driving car at the time that it had a very complicated array of six, like super high end cameras, was for like a self driving car that would map you know, as well, and what, what they wanted it to do was, you know, navigate from one point to another, when it started up, it wanted to make sure all the cameras were ready to go, and then drive, if anything happened, stop, bring all the other cameras back on line and go. And that led me to create the state reactors. And so what we did is we have a state reactive called all events go, and it's a runtime object that resides inside the state. And it receives events from camera one, that camera to event and once it's received it from all then it throws its own event, and then the state transitions. It's become such a popular, I'd almost say it's a state machine design pattern. You know, we call it S T acquire sensors. And I don't know what percentage of state, you know, we have in the reference library, we have certain state machines that are extremely simple. It's just like this one back and forth. So they don't have it. But I think for all of the other state machines that we've like, if there's a simulation video, I'm pretty sure all of them have one of these SD acquire sensor states and very early on in the execution chain.

Gotcha. Okay, so you have those that will kind of take information and they take in all the events, and then when a specific condition is met, they may event they may emit another event, and that can be used to trigger some sort of transition or whatever might occur. Correct. Okay.

They also, one other thing too about him, that's I don't know, that might be interesting for the specialist out there. They they've set the stage for us to move ahead with cogeneration in a new way, one of the things that's always cool state machine libraries,

when you try to play code, or Yeah, well, boilerplate,

but also, like graphical cogeneration. That one of the problems that that has are always complicated, that is callback functions. But with state reactors, you avoid that by with, you know, through these second order events. And so yeah, and that's part of our plans over the over the long run, you know, to build out an entire tool chain along with the the ability to right now we have this Mac to runtime analyzer, but we want to create a smack to writer where you can, you know, graphically align your state machine and have it auto generate the code, or at least generate some of the code, you know, like, like a lattice framework type thing. And, yeah, the only thing about that is that, in my opinion, you have to do that last. Because if you do that too early, and you've got some problems in your, you know, in your architecture, you basically baked it in place. So, yeah, so that's one of the things I'm really hoping to accomplish with SMAC, in this current year is basically getting more and more use cases, to a point where it's like, okay, no, no, we've seen you know, we've got an idea about every which way that the library could conceivably be used. I don't know if we'll get all the way there, but,

but if you get a lot of them, then you have a fairly general solution, and then it can be used for a lot of things. That'll be great. Yeah. So one thing that I don't quite understand, well, I guess I'm having trouble mapping The idea is together. So I understand this architecture with the client behaviors and you have your orthogonal and you have your states and things like this. Can you explain kind of so like a simple application, say say like that video of the Fetch, sorting, the Fetch, in simulation, sorting cubes by color, okay, like, how do we go from these states? And then the orthogonalize in and then how, just like an idea of how all the systems work together to create some reasonably complex behavior? Or can be a simpler one, if that's kind of complex?

Yeah, maybe we'll start with the dance bot example. We'll kind of dive into that one a little bit. All right. Sure. In so for those who, you know, who haven't seen it, the best spot to see the dance bot example I can like it. Yeah, is in is in the the ROS world 2021 presentation on smack to It's the opening video how we link that all together.

And just to describe the video a bit, it's a robot that it's in a sort of maze, and it goes to a location and then it drives until it hits a wall, and then it goes back to its original location. And then it goes out again, but slightly different angle from where it was heading. So it makes kind of like a star pattern. And where it goes, and it bumps each of the walls around it, and then it goes back to its original location, and then it goes to some other location to start doing the same pattern there. And I believe the intention with this is to map out an environment.

Not exactly it's oh, there is there is a mapping that, that, that that does happen. But it's really more about floor coverage. We were modeling, you know, how how a robot might, you know, cover a floor space in in modular ways. In that one, the robot navigates to to global waypoints and begin, proceed to perform dinners of superstate this is, so now we get into hierarchy a little bit. So we've got our leaf state. And then you can have these containers. These are like levels of hierarchy, which we have shorthand for, which I'll go into right now. So the first level of hierarchy we refer to as super states. The next level we refer to as mode states. There's no difference,

per se, it's just because it's a hierarchy. So super states is like high level control. And that's like fundamentally what the robot is doing. It's mapping it's driving to some location.

In your application, I think I think the most common use for for Super states is for loops, for loops, or like, like sequences, you know. So in that application, it moves to the first global waypoint. Okay, it's to execute a super state, that is a radial pattern. And you can set the parameters of that radial pattern in the superstate. So I think in the video, it showed something like 16 rays,

and the 360 divided by 16. And then it just went out exactly what each of those angles,

okay, but you would just go in there, and let's say, put in 32. And so it's parameterised. Exactly, exactly. It does three of those super states in a row, at which point it navigates to a fourth way point, and we haven't execute an F pattern, same kind of thing, although it's like, I think it's a series of seven states that execute this repetitive F pattern. Again, though, designed for floor coverage. Then it goes to a final waypoint where it's more of an S shaped pattern. So five total super states and five global way points, all the super states parameterised. In this case, we're only using two or thought well, let me think there might be more than one orthogonal in there. But in terms of like the crucial orthogonal for that example, there's two, there's the, the the LiDAR, and to cool. Yeah, in that particular example,

and nav to handling the navigation of the robot from one spot to another, okay, and controlling the motors and everything to drive it around. Exactly, right. Okay.

Yep. Yep. In other in other state machines, we might use, you know, something like ROS 2 control the same kind of a similar way that that move it relies on ROS to control the move the robot. And we've had we've built examples and navigation in ROS, one where it uses the ROS 2 or the ROS control draft controller. But yeah, it's you know, you start using like your standard ROS stack to get around. Yeah.

And then so if we're in superstate one we're in the radial star. Okay, one. What does that look like from the mode states, so not the super states, but now we're in Super state one. And so super state one is composed of mode states, as you were saying, with the shorthand, yeah. So how does it look? They're like, what, what's going on down there? And how do we get closer to

the CLI that example? Like, in depth, or just ballpark? Yeah, I believe the way it has is that we've got a mode state called Run, and we have another one called recover. And where I see that architecture evolving, in going, you know, in terms of real world is that one of the things that SMAC offers is very advanced recovery sequences, and capabilities. So I imagined, and I think one of the things that SMAC one of the most important things that SMAC offers and in why it's disruptive, is that it allows for a new order of magnitude in the complexity of robotic missions, particularly age missions. So So for instance, let's let's take an example of a fixed wing aircraft, right. So you'd have one mode for taxing on the runway, another mode or takeoff, another mode for I don't know, you know, the your initial flight, where you're finding your right heading, stuff like that. And for each one of those modes, you're probably going to want to have a separate recovery state, you know, separate recovery mode state, where dependent, so let's say you're in takeoff, right, and something wrong happens, then you switch from that mode.

So the appropriate recovery for that will take

off recovery, and you begin your, your, you know, error recovery procedures there. And as that continues to expand, you just, you end up with like greater, greater needs of hierarchy. It also brings in the question of where your data goes, another unique feature of SMAC is that every state is a class and every container is a class. So you can scope your variables and other things to the lifetime. And that's needed. You know, perhaps you need a variable across every state in in a certain mode. Perfect, you put it in the mode, you can also do you we also have the opportunity, or the option of global variables. But

yeah, okay. So interesting. And so, can you tell me a bit more about the real? So you're saying, I guess, the origins of smack and how it was born, from, I guess, a real application and then how it's evolved since? Yeah,

it's in the oranges of smack it's hard for me to put a finger on it i i would say it started the same. So I graduated from UCLA with a degree in biochemistry. And one of the the, my first job outside of outside of university was under a NASA grant at the UCLA UCLA Medical Center in the department of Head and Neck Surgery under a brilliant researcher named Dr. Larry Hoffman. And I, I ran one of the research labs there. I forget the exact title, but that one of the labs was basically mine. And what we were doing was it was this insane program where we were dissecting the inner ears of we were, we were injecting, like dyes, and then dissecting the inner ears of mammals, mainly chinchillas and mice. And then we were studying the organelles of the inner ear, the you trickle and the sack you will and stuff like this.

I don't know anything about this. Okay. It's just an our little parts of the object. Yeah,

even then. And in dissecting these organelles, we would then visualize them, you know, under fluorescent microscopes, and imagine like, I'm holding up for the listening audience. I'm holding up like my hand, it's like an Organon. It's like they would, they would have these Nerf patterns that would kind of follow through these organelles. And we were studying those in an effort to create better rocket guidance systems, which I have no idea if it ever, you know, resulted in such things, I have a feeling gyroscopes do a pretty good job. But it got me it brought up.

So it was this this was like, fundamental. So like, one thing that might be similar is I think it's rapid random exploring trees, R R T, or something like this. Yeah. So it's, it's almost like an you're trying to identify an algorithm that controls these nerves spreading through them. And then maybe that algorithm would be useful in applications, like, guided missiles, or whatever it might have been.

Yep, that's exactly right. And that's, and so it opened, expose me to to concepts like this. neural nets was talked about. I got to program a little bit of Fortran. Also, decision trees were part of that. And it, it, it began an interest that I had, like, even in terms of office hours, where I just started, I started sketching out these kinds of systems, you know, like, and then, in looking back on it, it was clear to me that I was drawing out state machines, although my early my earliest state machines, it's like, each state was a neuron. And so I was imagining these chains of neurons and how they would process information. And you know, what, where that was just like, yeah. And so anyway, that was, that was where, like, some of my earliest interest in that arose. Later, I went, I went back to school to get a degree in math and theoretical computer science at the University of Irvine, and which I did not complete. But in some of the early courses, you know, I was exposed to set theory automata theory. And so I had

no, any idea what that latter one is. Oh, automata

theory. That's like, that's like the, that's like the math behind things like finite state machines. Okay. Yeah. You get there into like, whether or not state machines are Turing complete stuff, like

proving things about them. Theoretically. Yeah,

exactly. Particularly when you start getting into advanced state machines with event queues. You know, and like, how then how that maps so there was that that part as well. I moving on later, I remember I got there was a when I started believing that robotics was the next wave that it was in I believe this still that robotics is going to be the next boo and probably the biggest one in my lifetime. I feel like that's going to happen to within the next 10 years. And so as I was looking forward to that there was a book called UML of real time systems and I read that book like I I've gotten rid of it since but I don't know I could almost pry tell you on a particular page what's on it and sketching too at that time though, I although I was so interested in them I never would have thought that I could have written the state machine library like that I just I would have thought that well and at this time this is really before ROS you know so like the systems were more closed off I would have thought that that was part of some giant proprietary stack I see Yeah. Anyway my my first project you know that involved ROS was creating what the company is real robotics and we were creating level line cable reels that we're going to provide mobile robots with tethered AC power for which you know more

It's a level wind meaning level wind level meaning that if we've got a real here it goes like a fishing exactly like a fishing line fish yeah, there's like a goes like this. So it winds perfectly every time. Cool. And I've got a picture too of like what, what we had in mind for the system holding it up for the camera right now and it's on top of ClearPath robotics Ridgeback robot so you can see the box that it is the real there's a yellow it's meant to be Like a nine gallon Shopback that goes on top in this application here, I'll show it from another angle.

Okay, so it's, it just looks like. So it looks like one of those small clear paths, or maybe not small, one of those clear path robots that has kind of the wheels underneath. And then there's a big cube on top of it, and it kind of looks like there's so I guess that's the Shopback cylinder that sticks out of the cube.

to get started with this. So the problem was to do level cabling. So you wanted to basically have the robot drive, and it would leave a cable behind it. And then you could wind back on the same path, and it would organize that cable as it retracted it, basically,

which, which required tight coordination with navigation. And, and then also tight coordination with what would be really the state machine of whatever the plan was to basically sell those to, to OEMs. You know, and so we knew

that OEM is original equipment manufacturers. So let's say so that's like,

yeah, go ahead. Well, yeah, somebody like Dyson vacuum or something, you know, more probably more industrial, something like 10 internal fills, but But yeah,

whoever was going to meet or know them, but I see. Yeah.

And so we knew that we were going to so we built, you know, we started with that software stack. And we moved all the way up. And we really did some beautiful work there. Some notable people to contributed on that project, Dave Coleman wrote something Oh, project. Yep. Yeah. He was this was before picnic, he was a graduate student at University of Colorado Boulder, and he wrote some of the original ROS control code. Vince Magyar, also worked on the project. He's the maintainer of ROS 2 control. A mathematician by the name of ROB Hanna from UCLA. And I did some, we mapped out all the mathematics where, imagine that level line system, you know, the goal is to know exactly how much cord is being produced with every encoder tick, but it's changing all the time, depending on how many, you know, layers of cord are on the real position of the level line carriage where the robot is the robot rotating. And so we did all that. I focused on the microcontroller code mainly. And that's kind of how I that that was more my background is I came into smack I was more like an embedded programmer. And that time, I thought that C 99.

That was my belief in anything involving templates. I looked it like, you know, I didn't like that at all. Yeah, Pablo country project too. And there was also a developer, from the former ClearPath developer named Devin Nash, he did some really great work on the project, too.

Anyway, it's cool to see. It's funny to me that it's such a small community, in a sense, as in these names keep popping up for a lot of endeavors. Okay, so you worked on this robot. And so this was with real robotics. And this was the application. Yep. And smack was born from this.

Correct? Yeah. Because we knew that we were going to have to integrate with whatever the OEMs were using for their top level coordination, because you have to coordinate you'd have to coordinate the movements of the reel with the navigation, you know, the navigation of the robot, which we accepted. You know, we knew we were going to have to do custom work like that. But in order to, you know, develop our own demos, the idea of a state machine came in, also early on, I'm going to say so we started this in like 2015 2014 2015 by 2016, maybe even a little earlier, I realized that smash was not up to the task,

Oh, yes. Smash was smashed was a state machine library developed by Jonathan Bullrun, I believe in 2009. And it was a state machine that that powered the PR two examples. Yeah. I think very highly of that library. I think he did a number of very, very innovative things that that inspired SMAC probably the most important was Tying the events to actions, you know, like, like, like, like in the example you were describing to navigation finally gets to a goal that, you know, that then creates an event and you transition to the next state. Exactly. There were things though about smash that never fully. So I want to give credit to Jonathan Boren like for like, it was a, it was a creative and in essence a leap forward. I think in terms of though, like, it was never really suitable for like more of a production level of robotics. The concurrency always had issues, the data porting is another subtle problem. Anytime you see, in my opinion, anytime you see data porting and state machines, run,

Brett Aldrich's LinkedIn

Robosoft.AI's website

SMACC2 on Github

SMACC blog

State Machines for Complex Robot Behavior

In this episode, Audrow Nash speaks to Charles Brian Quinn (aka, CBQ), CEO and a Co-Founder of Greenzie. Greenzie make an autonomous driving system for commercial lawn mowers. We talk about Greenzie's autonomous mowing system, how Greenzie has worked with manufacturers to up-fit their system into comercial mowers, how Greenzie does dog-fooding, safety and standards, and about CBQ's experiences bootstrapping and with venture capital.

Introducing CBQ and Greenzie

Why go by initials?

Describing the Greenzie autonomous system

Working with mower manufacturers to up-fit their system

Software on autonomous system

Mapping the lawn

Handling poor GPS signal

Perception pipeline

Having a human use their mowers

Standards in the off-road space

The value of standards bodies

Depth cameras in the autonomous system

LIDAR and being venture backed

Replacing depth cameras with LIDAR? + Tesla and computer vision

Ruggedized computer

Microcontrollers, testing, and visualizing data

Deploying software updates

Safety

Testing for deployment + dogfooding

Training people to work with their autonomous system

Greenzie’s business model

Bootstrapping a startup versus venture capital

What CBQ learned from bootstrapping

Future of Greenzie

Advice to someone starting their career

Links and social media

This is a conversation with Charles Brian Quinn, also known as CB Q, who is the CEO and co founder of green Z. Greens. He makes an autonomous driving system for commercial lawn mowers. In this interview, we talk about greens these autonomous mowing system, how they've worked with manufacturers to upfit their system into commercial mowers, how greens he does dogfooding and about CPQ his experience with bootstrapping, and raising venture capital in startups. This is the SENS think act Podcast. I'm Audrow Nash. Thank you to our founding sponsor, Open Robotics. And now, here is my conversation with CPQ. Hi, CPQ Would you introduce yourself?

Hey, Audrow Good to be here on the podcast. I am CPQ I go by my initials Charles Brian Quinn, and then the co founder and CEO of green Z. Greens he had our our big mission is to free humans from repetitive outdoor labor. And we're starting in the green industry. We have a product that helps create self driving commercial zero turn lawn mowers. So we help big landscaping crews that maintain the largest crop in the United States by square footage to help them reduce their cost of labor,

the cost of living grass by largest crop that is,

if you think about it, you know, I know you've had great people on your podcast that are doing farming and but if you look at square footage in terms I mean, grass is a crop that the we maintain across the United States in terms of watering, cutting, maintaining, cultivating, in terms of square footage about about 3x corn, which would be the next and so there's people who do that that industry, the green industry we call it is pretty big industry and not a lot of people want to sit stand or push a mower for eight hours in the hot sun, especially here in Atlanta, Georgia, where we're based. And all over the US there's a shortage of labor. And that translates as you know, in supply and demand to higher sort of demands on that and and again, when when it doesn't people don't want to do it. And it's also dangerous, dirty, repetitive, monotonous, you know, things that we we all know our robots are very good and capable of and don't care about, then it is probably a good place for us to help supplement and add robotic workers to these crews, and our software that's based on ROS does that.

Awesome. See, you brought up a lot of things I want to start. So first, you kind of unrelated to your mower. You mentioned there's a story behind CPQ and why you prefer that

out there. Yeah, I've just always gone by my initials starting from high school. And so it's even become my little moniker, you know, on every sort of side seq, which is CPQ I used to even have a license plate See viki.com And since 99, so I've been gone for a while. So yeah, and it's funny and professionally to a lot of people were like, hey, you know, CPQ and they're like, who? And it's like Charles and they're like, oh, yeah, CBK it's kind of a moniker that both professionally and friendly. I go by. So Andre Audrow were close enough. Now you can call me CPQ.

Funny. Okay, so now tell me about the greens, the mower or your setup?

Yeah, absolutely. So we it's important to note that what we do is we are an automated driving sort of system, right? We are basically a software package that's enabled by a set of, we call them off the shelf sensors. As you know, Robotics has gotten very far in the past, Gosh, 20 3050 years, even more recently with the introduction of the Robotic Operating System. And so with the standardization these a lot of the reps and things like you know, data structures, like point clouds and ROS enhancement proposals are like being standards for the ROS community. An easy one would be like, hey, use meters as opposed to, you know, inches, right? Like, that's a very good rep, one of the three I believe, says, you know, there's some good ones out there. And so with all that, what we've found is that the time is now to take some of these sensors, like the real sense camera, and use that to basically do a function or task and so green z, we build software. That software is enabled by a set of sensors and we certify those sensors, we call it being a robotic ready, that's part of our robotic Ready program. So we help these mowers become robotic ready. And once they have the sensors, they can then run our software and our software adds self driving. So, we we are trying to like I said reduce the cost of labor. So when you think about autonomous and the behaviors and things that make something autonomous. It is sort of self directed at x You know, again, it's sensing thinking, perceiving and acting on the world. And so what we do is we have a sensor stack that does that for a very particular part of the process that landscapers do, and that these mowers do, we call it sometimes we call it cutting out the middle. And that's kind of a unique term that landscapers kind of kind of resonates with. Because when you get to a job site, you know, there's a certain set of tasks to do to make, you know, the property look beautiful or safe. If you're, you know, mowing a hospital facility that has a helipad, you know, or right of way for safety, or on roadways, or industrial, you know, a lot, a lot of safety reasons why you want to keep that, that maintain that property maintained, as well as make it look beautiful, then there's a tougher task you do. And so our software helps automate some of that. And, again, by doing that we help reduce the cost of labor for those landscapers, which is why they like our robotic workers, but a lot of people assume we have mowers and we don't, they're some of our competitors, who are wonderful, and happy to mention them here. Companies like Saif do are building a automated mower, and that we unfortunately do not we actually build a a kit, basically. And we do it at the factory. So we help manufacturers and OEMs produce self driving equipment that's comes from the factory. Both strategies, I think are noble. And I think that it's tough to compete. You know, especially talk about John Deere, who I know that one has been on your podcast, it's tough to compete with them, they produce lawn equipment, I've been doing so for hundreds of years. So they've got a lot of learning ahead of them and distribution and all that stuff. But so what we do is, again, add sensors, and a sensor suite package and our software that goes on these existing commercial mowers to add autonomy. Gotcha. Yeah, it's it seems like a clever approach, especially for a startup to work with a company that actually makes them ours, and then kind of retrofit that with your autonomous system. Yeah. And the word like a lower barrier than it is for sure. You know, in talking about lower barrier, when we first started, we did try and like retrofit, like posts sort of sale and so we go get these mowers and, and put all these things on them. And I'll never forget the first time we did it, everything broke. And then we're like, well, well, let's just make it more robust. Right? And so we tried, and what do you mean, everything broke out, bro. You know, like, when you think about how these are used, I remember the first demo, I saw ED, right, who's the CEO, right manufacturing, who is one of our current manufacturing partners that were partner with to bring out our first product. I remember seeing him demo, and he's nonchalantly talking as he's demoing. And he drives off the curb, and then like drives back on it and drives back on it and off of it, and the mower is just bouncing off the curb, repeatedly. Yeah. And he puts down to the deck and the deck, like hits it and bumps up. And I was like, oh my goodness, like the amount of vibration ruggedization required to keep what we had originally had, we just cert put these servers with an eMag clutch to basically actuate the the the left wheel and the right wheel. And it worked. But you know, it was very fragile and delicate. Absolutely. And you know, and it's funny, like, it's not that they're our end user customers are rough on the equipment, it's just that they have a job to do, and not a lot of time to do it. And so the you know, they're they move fast, they roll over the curb, they run it back into the truck and are on to the next site. So but retrofitting was very difficult. And so we immediately moved it from sort of this post retrofit to at the factory, that way we can use the design principles, design it in with the mower and get that support.

How does that work? It's like, so do you have to go to the company and say, Hey, we needed at this stage? And just tell me a bit about that. Because that's interesting.

Yeah. So this is part of my consulting background. You know, in a previous life, I, we I helped some of the big companies like GE Lighting, and some other companies do software that went with hardware. And one of the ways you do it is you go to the factory, where that thing is produced as close as possible to the line. They call the assembly line or manufacturing line. And by doing the upfitting is what we call it. You call it retrofitting, but

it's up fitting that way. You think about it, like when you buy a guitar go has it a little different than retrofitting, because you're not getting it all the way in. Like you're not getting it from the retailer where it's entirely set up for the user. You're getting it part of the way and then you're adding things and changing things. Yeah,

we call it a good way to look at it. If you think about it. If you go to the Let's assume you went to the Ford dealer to go get a truck. When you go they have some base models. You know the F 150 which is the most popular are selling truck and we have a couple of Ford Yep. And if you go and you say I want this one, they're like, great. And you say, You know what would be awesome, those, I would love one with a V eight, like turbo diesel, and they go great, we don't have any, but we could get that upgrade package like we can do that. And we get that from the factory, like we'll put in that order that it'll come down the line, we'll put it on there for you. And you can get even more custom than that. But so that's the idea is that this is a feature, we call it a autonomous mowing that we built that you could up fit or upgrade and order. And then it comes from the factory. So what we've done is work with, right. And I'm working with a couple others right now, that where we go and we do that, that outfitting as close to as I said the factory as possible so that they come off the line. And then when they go to the you know, because manufacturers are people who do this, the end user customers buy their equipment from outdoor power equipment, so much like a dealer. It's kind of like a dealership, but for outdoor power equipment. They sell these things to do this kind of activities. They have those mowers already there. And then they activate them just like I guess OnStar or Sirius XM Radio, if you've ever done that process that's similar to us, except you're doing software that does autonomy, autonomous driving, that's cool. Okay, is there.

So when you approached the company that makes the mower and you want to add these custom features, I assume they don't have like an autonomous mower package. So I assume you have to work with them to do that.

Yeah. You know, when we started the company, we were sort of like people were like, you know, we already have robot mowers, right. And so if you look at the idea of a robot mower, those are those little Roomba like machines that and they've been around since the 1970s. That patent is has been there, I think since Yeah. 70. And so you put a boundary wire in the yard. And so the big commercial mowers were like, Hey, what is this? You know, and how is this different? And, you know, we had to do a little bit of convincing to tell them no, no, this is self driving assisted sort of like almost like Lane Assist. And when we start using those terms, they're like, oh, like my new truck? has you know that where it keeps them in the lane? Probably? Yeah, it definitely helped. And so we said, hey, this is self driving, but for these big commercial mowers, where it takes over on part of the operation, and they said, well, that's not possible now is it? And we said, well, look over here, and we built prototypes, and we did the retrofitting. And we did all these things. And you know, and I'll get back to that. But that's actually the one of the thesis of the business is that we have been iterating since we started and the term iterate is very important to us. It means do learn, improve, and repeat. And the process of iteration has helped us tremendously. So I think it's, it's interesting, as an entrepreneur, I look back and say, Oh, it's always been this way. But I can guarantee you that, you know, we've been through several iterations where we probably thought certain things, and then we learned and did and now the new reality is that that is what we do. So, you know, every entrepreneur will tell you, Oh, yeah, I knew everything to do with the business. Yeah, they don't actually what they do say is they say, Well, we learned a lot. So a lot of them talk about, like learning through failing, right, failing fast, right? So we were doing a lot of that in the beginning. And so that's how we got there. To answer your question, yes, when we go to manufacture, that is, you know, the, the now what we do is we pitch and we tell them about this robotic ready, which is now been proven and commercialized with a couple of partners already. So there are there are a few systems out there, I would say that are competitive. X mark is a big manufacturer who makes Toro are sorry, Toro, who owns X mark and, and so they have one that they've been working on for a while. And then they even made an acquisition in our space, which was amazing. And I'm so thankful for that, you know, normally when, when, when, when a competitor is acquired by a big manufacturer, it's actually an incredible opportunity. You know, like a lot of companies are like, Oh, that's scary, because the big guy like has it now. They slow down. No, isn't that I mean, it's honestly that. I mean, I live in a world of abundance. And so yeah, investors and everybody, and there's always enough for everybody to do. And so when that happened precedent, well, yeah, that any legitimizes you and it also helps every other manufacturer who doesn't have that, like will pick up the phone and say, Hey, I see that you do this, right. If you know, there are 46 outdoor power manufacturers, and you may actually be hearing some one of them on the background. Apologies. That's in our test lab. But if one of those 46 Like starts to think that they get ahead, then the other 45 are gonna be like, hey, and there may be another one, you know, a new one comes online, you know, 4648, you know, so you're saying new companies, yeah, new manufacturers or other sort of entrants to the market. But I think competition is wonderful. It's definitely legitimising and it's a rising tide lifts all boats. I believe that and I believe that, you know, my entire life.

Yeah, it also seems like it makes it It easy the space grows. Yes, there is a larger events in it so educate, it's easier to harness all that. Yeah, yeah. And parts may be become more easily procured and

things like this. And then you get stuff like, you know, like, maybe standards are created, right that are there to help you too, as well.

Uh huh. So what is your autonomous driving system? What's it composed of? You mentioned a lot of sensors. And it probably has a big computer with it. And it has some things to interact with existing mower hardware, but tell me a bit about it.

Sure. So we're big fans of ROS. And they're not like operating systems. So and we're big fans of open source. So while there are parts of our stack, a few nodes that we probably won't open source, just because they're not going to be useful to the overall community. specific use case exactly. I see. But you know, if you look at like, say, robot localization, which we use, you know, it's funny, we were out there, I guess, couple years ago, two years ago, and in the field, and we were mowing for a customer. And I'll remind me to tell you about that becoming an actual becoming the customer and doing it ourselves. We still do that. Now. We learned a lot. We call that eating around dog food. But I'll come back. Yeah. But we were talking about robot localization, we were using this package and won one field, we would be mowing, and all of a sudden, we'd get to this one area, and the mower would shoot and say, Hey, I've got to head 50 miles that way, for my waypoint, and we were like, that is a bug. And we're like, that is really weird. And whenever flow error, it has to do with UTM boundaries. Are you familiar with, you know? So in,

I don't even know what is UTM stands for? Yeah, so I

think it's a universal time Merkin teal, and we can look it up. I apologize. But what it has to do with is, latitude and longitude are all straight lines. So you know, if you're going on Google Maps, you'll maybe notice that you can get Google maps of like, you know, it's a 33 or 90, and it's all in degrees, right? So you can do any location on the planet, you can have a certain it is a sphere band. So you got so right, so why can't really wrap a sphere in a rectangle, right? There's some area that's adjusting for that. Correct. So we, we we most of the lot of problem to run into, yeah, mostly libraries take a area and they'll do it in bounding boxes. And so you'll be in a UTM quadrant. And so once you're in that quadrant, the the space goes from sort of, you know, it stays in that sort of area, right. And then once you move outside that boundary, that UTM boundary, and there are like, obviously 50 or so of those, you know, as you talked about the latitude longitude and the US, I think there's like five or six that you go across. Okay, don't correct me if I'm wrong on that. But there's quite a few. And we just happened to be on it, that one site happened to be on a UTM boundary. So when it crossed that boundary, it reset to the other UTM, which was 50 miles away. So anyway, we found that bug, and it was great, we worked with more to push up a PR and, you know, and push it back to the community. And now we're about localization can go across UTM boundaries. So big fans and open source been doing in my entire life. And we'd love to keep doing that as long as as long as it exists, which I hope is forever. And and so you'd ask us about our stack big fans of robot localization. So we use a lot of the base base stuff, we're not trying to reinvent the wheel. So standard cost maps, we're big fans of ROS control. We have hardware interfaces, which really help us with, you know, left wheel right wheel, differential drive. What else do we use? We use a lot of the standard data. You know, I

guess they're missing depth by using ROS 1 or ROS 2.

We are currently our current generation stack is on ROS 1. And I think we're on melodic, and I think we have a noetic running around downstairs, that works just perfectly fine. And so we may upgrade our entire fleet to that. And then ROS T we've evaluated some of the packages we're using aren't quite there yet. And we've been working closely with some of you guys at Open Robotics, the Michael Carroll who said, close we're close. As soon as we are we would love to upgrade and we can even do that in field. So but we're enjoying the ROS 1 ecosystem. But yeah, so you asked what's our sensor stack and sort of package look like but autonomous mowing is a suite of ROS nodes that do a function and that function is autonomous mounting. So if you can imagine our system feels like an appliance. So it is a ruggedized computer and I think you need to go to our site. We're pretty open with like everything we use. It's an x86. I actually just gave a talk at ROS con. This last one on how to boot to ROS and seven seconds where I took this machine and really minimize the colonel and use the ubuntu LTS and kind of minimize that install with the precede to make it very minimize it boots up very quickly. Super fast. And I and just go straight into ROS, using System D, which we're big fans of, we like a lot of the Unix stuff. And so where there's a lot of custom stuff we don't, we're sort of not big fans of it. So yeah, we use a lot of like sort of trusted stuff like system D networking and open VPN and some of the sort of standard standard stuff that's been around for a while. It's very modular, extremely big fans of node based, you know, single responsibility, tiny things that do one thing very, very well, liberal and the input that they accept, very specific in the output that they do. And so a lot of design principles for software that we like about ROS. And our stack does that very, very well. It also interfaces with a web app that we've built, that our customers love. And that's built in Ruby on Rails, which we love, and is probably the best for doing rapid web application development. And, you know, there's a bunch of stuff behind the scenes there. But I won't go into all the details. But all that does a very specific function, and that is landscapers by our product, they take the motor out, they press map, they mow a area of the area, they want it to sort of finish the job for them, they press go on a little remote, and it basically just finishes and Stripe still fills in everything. So they do the border. And from there, you fill it in autonomously, exactly.

Okay. Let's see. So for all of the what kind of sensors are you using for this whole? So when you map it? Is that are using GPS? Or? Yeah, so what kind of thing are using?

Yeah, so I'll start with control and get all the way sort of there. It starts with control. And in order to be, you know, I have a theory about robotics that a lot of people when they envision robots, they think of the Willow Garage, sort of PR to like, has to be humanoid. But in reality, you know, robots are in robotics as a way to I think automate and customize what would normally be considered a human level task. And so there's a lot of ways you can do it. When you think about self driving cars, they are in in terms, sort of a robotics. So the first thing you have to do is get good control. And so what we do is with let we need control of left wheel, right wheel, because these are differential drive, and a lot of them have caster wheels on the front. So they while they have bias, they don't really matter and are controlled. And so we start with left wheel and right wheel control. And we've been able to do that with a drive by wire system. And so that starts with with ROS control and having a canvas adapter. And so we use can which is on you know, very

I think it's already gonna do you know it right. What is it? It's the automotive protocol. Right? Yeah,

I know, it is the automotive protocol. I don't know

what exactly stands for. But it is a protocol, we could probably look up the acronym. But it is a protocol that a lot of cars use. And it's very small, it's very high speed. And good for redundancy. Yes, it's built in. And so we have that can interface layer that allows us to left wheel right wheel. So once we have that we then need to read. So we use encoders. This standard Hall Effect sensor encoders. And on our electric platform, we can read that straight from the wheel itself, they have encoders built in. And the reason you'll need to know that is no velocity. And so when you're able

to say hey, you do your own mapping with this, you're not you're not necessarily using GPS, you're using odometry, large, right, and then maybe sensors and things.

So you're exactly right. So we're slowing down for

just a second just to speak about a hall effect sensor. So it's a magnetic sensor, and you can count the ticks as a wheel spins. So maybe you have one magnets, or you could have a bunch of magnets and you can count do you do? What's it called where they're 90 degrees out of phase? If you have to, let's say that you can figure out the directionality. Yes, that's exactly correct. Because one way it's traveling 90 degrees over to get the next bump. And one way it's traveling 270 degrees or three fourths of a rotation or three fourths of a phase? I guess

that's correct. And you can get some telephones have the gear backwards. Exactly. So they have gear teeth that they're reading. And so then if you know, the tick count, you know, and you could just do the math to say how many wheels and velocity? Yeah, so yeah, exactly.

And you were saying sensor fusion? Because if you're using raw odometry, like just seeing how many times your wheels spin, there's slip and all sorts of things that all accumulate as errors. So you're dealing with that through some sort of sensor fusion?

Yeah, so you're exactly right. When we first started, if you just use our DOM, or odometry, on a very nice platform that doesn't have a lot of slip, you go out, you turn left, you turn, right, you turn again, you get that square and in you know what'll end up happening is you'll be like this right? And it won't be distorting. And what I'm saying is you can't for those who can't see, it's like a you know, unconnected square right? Because of the slip. And so what we do is we fuse that with an IMU. And so we have a high grade IMU we've evaluated many, many many IMU use small ones we liked I happen to like some of the vector NAB stuff because it's nocturnally calibrated alias that for the gyroscope. So they don't even notice all of the sensor data from it. But yes, we do fuse a lot of the components from it. Yeah, the gyroscope, which tells it which direction you're facing? Yep. Yeah. So I did something like this in grad school, it was like some course where you use the sensor fusion of the inertial measurement unit, and the odometry. And it gets much more accurate, which is pretty amazing. So you're doing now we fuse that along with GPS, and we do high accuracy GPS, we have the ability to run on standard GPS, but we prefer and for some of our customers for the straight lines and tracking that they want to do. And a lot of the fields that we are we have RTK GPS, and are just gonna have real time kinematic. And what that means is, is if you think about normal EPA accurate, yes. And I think you've had people in your podcast talk about this before. But

you may not have this podcast, previous one, I think so yeah, you get crazy.

You get correction data. So you can get a very accurate centimeter level. So we fuse all that now I can tell you that our sensor fusion is pretty heavily biased towards the RTK GPS, because the quality of it is just super accurate. Those are

so if I remember correctly, and maybe the price has come down, those are like $20,000, or something to get a good RTK GPS isn't correct

or and we first started down, it has come down quite a bit. And when we first started exciting, I remember we were using an RTK setup that had a base station. And you're right, it was 1000s of dollars for this one and 1000s for the one on the machine. And gosh, just recently, I saw that. And a lot of it's been enabled by a company called ublox. And essentially, they have a chipset called the F nine and F nine P and that has taken the cost down to in saying I think we just got another sensor for I'll admit it, I think we just got our most recent RTK for like 129 bucks, and I'm like, Oh my God, that's crazy. And so the real the real kicker, though is that the data, Correctional Service really matters. And that's where you're seeing a lot of the innovation. So there's a service out there called SEF. There's service out there called Point perfect, that provides what we call n trip, which is I forget what that stands for, I think it's a network transport protocol. And so instead of having a base station, what they do is they're basically like a cellphone network and every 50 kilometers from the, you know, from certain high density areas, they put their own base station, they calibrate it with other sort of known areas. And then they can send out that correctional data over the Internet. And so you don't have to have a base station. That's nominal. And you already have that data. It's gotten really good, low latency, you know, super high availability. And with that correctional data, it's been really good and very good for our business.

Do you ever? I don't know with RTK GPS, but do you ever run into situations where you're mowing a field and it doesn't have good GPS coverage? Absolutely.

So we've had to design a lot around that. And what happens is in localization, at least in the way we've done it, which is the best practices, you know, by you know, roboticist, you've done this before, you have what's called continuous and you have your discrete, and I think I'm getting that right. But in continuous, you've got things like, that are just going continuously. And so that's

odometry. And that's the wheels spinning, okay? And knowing how far they spend, right? In

your discrete, you can have jumps, right? And so, in a good localization system, what will happen is, when you have when your discrete jumps, like, let's say your GPS shifts, and you know, it runs out or it's there's no connectivity, or whatever, and it says, oh, you know, I really thought you were here and you're moving along and boom, you're over here. A good localization system will go, I don't think so I, I had all this momentum in my ETF sort of saying bias that I was saying I was going five meters a second this way. Yet. Now when when you say me, I'm over here, I'm gonna ignore that and and yet back in.

So you use probabilistic methods like an extended Kalman filter and ukef, as you were saying, exactly. And so what that does is you have a pretty good idea, like you have a probability distribution that says I'm most likely here. And if you get some information that says, I'm way over here, it's like, well, it's probably not very likely it takes that information into account. And then it adjusts its belief accordingly, but if you tune it in a way, it probably won't listen to that very hard, or very significantly.

And what ends up happening the thing that's worse actually is not GPS going out, per se, it's the thing that will mess you up. Is it is it changing and you should between a high accuracy and a low accuracy. So we've actually designed our system such that when it starts, if you get a high accuracy fix, like we go, and we're like, great. And then if we lose it, which has happened, you go down, you know, big hill and go down, and there's no coverage down here. And it just, it's starting to go out. If the system loses it, the mower will pause, our software will pause and say, hey, something's up. seconds ago was the last time I got a high accuracy. And so the mower pauses, and it'll kind of wait, maybe it's just like fluid, maybe it's a fluke or something. And so sometimes the motor will just kick back and start going, it'll get that fixed back. But if it can't, you know, there is a chance for the operator, at least with the mobile interface that we've built to say, you know, what, go ahead and turn off the low accuracy or high accuracy, and there may be a small shift in the map. And so it's not enough to make a huge difference based on what we've learned. But it is enough that you could say, hey, just turn it off. Because what what's, what stinks is how, again, going in between them, and it's constantly shifting is what will mess you up. But again, yeah, sensor fusion, what we do think about it, like we basically use our coordinate system in ROS, right? You have 00, right, which is where a robot starts, and then we record right. And so we're using all this data to determine where we think, you know, the next point is, you know, on the map and coordinate space, and we're using all these other systems. So the robot doesn't think it's at, you know, lat launch. It's using lat lodge as a data point,

internal coordinate system. Yeah. Yeah, you transform it into your primary reference system.

And as all robotics is, right, it's a transform series of sensors all connected with a universal descriptive robot format. You RDF you. And that's, we love you RDF. We use this a lot as well, because it's all just transforms all the way down.

Yeah. Funny. So if you so for your robot, where you're driving it, and you haven't heard it, and 50 seconds from the GPS, which is your really good source of information for driving it around. Do you have so if you're using a probabilistic way of reasoning about where it is, does that uncertainty just it keeps, so it received one good information? And then it drives using odometry? And yeah, using odometry. And then, so the uncertainty continues to grow as it goes. And at some point, you say, This is too uncertain, I'm gonna wait for a better GPS, or then you trigger this has have the operators switch the GPS, from high to low, or whatever it might be.

Generally, I don't know the, the correct answer to this. But what's interesting is, I think, when you think about cars, and how they drive so far, you know, that that where that bias could really go, and you talked about even going through UTM boundaries and things like that, that's where a lot of that will grow. In the areas that we're doing even like, like the the large acreage, even, it's enough to do a full day of mowing, and you're totally okay, running on that same localization and, and being able to cover that field. And we're good enough, then we haven't run into really any of those. And what's interesting is, at that, that's great, you reset, you turn off, you go to the next site you turn back on. And so since it's not long running, like the second truck, or a truck going across the US and really accumulating so much bias, we've been okay. Knock, yeah, so the error doesn't stack up that significantly and so sign I'm aware of.

Awesome. Okay, so then, what? What other sensors do you have? So you have odometry, you have some depth cameras and things like this? What else? So or how is the how are the depth cameras use

your perception pipeline we're really proud of we are. I'm a big fan of AI and ML, and I think it's gonna happen. But I'm actually not, we're not using it, I believe it's gonna happen. I believe that some of the AI and ML modeling and some of the training and the networks that are being developed out there are going to revolutionize a lot of the tasks that we have to do algorithmically or programmatically. And I truly believe that. However, I think that right now, it's a bit of a black box approach. And you know, that the the amount of data required to really train it up for what we would consider to be good enough, prohibitive. It is and,

and so we need like a million examples of someone walking in front of it or something

like that. And everybody corrects me says, Oh, we've got synthetic data. And I'm like, Well, you know, synthetic data is synthetic data, like you can only like change the color of a, you know, seven times and it doesn't really add that much.

In distribution mismatches between simulated data and real data are huge a lot of times,

I mean, I know because we've been mowing in the field and you'd be surprised that the stuff we really ever Yeah. So what we've done is we've taken a really, I guess, a pretty unique and this goes back to our design principles approach of really keeping it simple. And so what we have is a lightning fast C++ pipeline that does, you know, ransack, which is stands for I don't, I can't remember random sampling. Yes. And what it does is it tries to consensus, right? So it tries to find the ground plane. And, and then anything above the ground plane is an obstacle. So, this is not like we're trying to classify and say You said, so I, the way I described our customers is they're like, visit detect, like kids. And I'm like, you know, it does not make a distinction between a garden gnome and a kid. If it sees something above the ground plane, it's going to stop

it'll success to go back slightly for ransack and talk about a little bit about that. So you, you have your depth camera, your depth camera points over the ground. And then what you're going to do is check a bunch of, it's going to give you points at different depths, kind of. So from this, you're going to try to fit with a math equation, a plane on that. So a 2d surface, and the one that matches the majority of the points is the one you're gonna select. And that's it's a fast algorithm. Yeah, so the thing that's on top of that are deviating from it, you're like, This is something thing. Yeah, is what I understand.

Yeah, and not to get into all the details, but we have some special fun sauce, we do around line fit, and some of the other stuff on stuff to make it really fast and accurate. And, and then again, all that's translated into just a standard cost map. And the cost map would be, hey, you know, a, when they see something that is a, you know, tree or vertical plane or anything above that, that that is not part of that, that that surface, then it says, Hey, I can't go there. And so it creates a very large infinite cost map, you know, to say, like, kind of out there. Yeah, exactly.

Okay, how do you deal with like? So I imagine like the way you want to handle a kid, in close distance to the mower, and the way you want to handle a pole would be very different. How do you, like if I, if there's poles in my lawn, or in my field here,

you want to go around them? Whereas the other kids want to stop? Right? Yeah, what's interesting is when you think about, and this is natural as as humans to sort of say, like, let's solve the entire domain, right, like, let's do it just like a human would. And what's interesting is, having developed this company, you know, from iterative and from these basis is sort of like, how do I solve the first immediate problem, which is a job to be done, and get as far as you can get, as far as

3 posts with the software tag