In this episode, Audrow Nash interviews Dave Coleman, Chief Executive Officer at PickNik Robotics. Dave speaks at a high level about what MoveIt is and what problems it helps roboticists solve, they talk about supervised autonomy, including a collaboration with NASA and MoveIt Studio, and Dave talks about MoveIt 3.0.

Hi, everyone, welcome to the sense think act podcast. I'm Audrow Nash, your host and a software engineer at Open Robotics. In today's episode, I speak with Dave Coleman, CEO at PickNik, which does engineering services and motion planning. Motion planning is the problem of moving something from one place to another, such as moving a robot from one location to another, or picking up an object and placing it somewhere else. This can be made more difficult if you try to avoid collisions. For example, if you had something beneath a table and want to lift it up over the table, you have to go around the table. And as you can imagine, this is a fundamental problem in robotics. And Dave has been involved in move it a motion planning framework that has built on top of robot operating system or ROS for a number of years, and has created picnic to do engineering services around motion planning. We talked about his motion planning or the motion planning framework, move it, how PickNik came to be there work with NASA, and a paid service called movement studio that hopes to help the problem of supervised autonomy. And we teased the idea of move it three. I really enjoyed talking to Dave, he is a fun person to interview. And I find that he has a great perspective on robotics and open source, probably in no small part because of the diverse problems that they work on at picnic, and his time that he has been in the robotics community. I hope you enjoy. So Dave, would you start off by introducing yourself.

Yeah. I'm Dave Coleman, CEO of PickNik robotics and a longtime supporter of open source robotics contributor to ROS and MoveIt. Glad to be here today.

Awesome. To start off as a general overview, would you just tell us about move it? What problems does it solve and how long it's been going?

Sure. Move it is a framework developed to do motion planning primarily for robotic arms, but also for the full kinematics of your robot. It's had some success and walking in mobile bases, even by esoteric things like quite like flying robots and submersible so all over the place move it's been applied. And it's the problem of how to motion plan your, your joints through space, so you don't hit things and that you can do useful things like manipulation and grasping. And your second question, it's been around since pic. 2010 was when development begun, I think 2011 was the beta release of move it and this was developed at Willow Garage. Same place that ROS was developed, it was developed with the PRT program. Sasha Cheetah was the program manager at the time of that project. And yeah, Willow Garage eventually closed, but it lives on.

Let's see. And I know we said we talked about kind of you're coming into this a bit later in the interview, but might as well get into it now. How did you get involved? So it started in 2010 ish? at Willow Garage. When did you come along?

I was an intern at Willow Garage. Willow was very clever and having a lot of interns huge program. So I guess I'm not that special. And that the CEO at the time, Steve cousins said that the intern program was a way of spreading ROS. I like like a virus, he wanted us to go back to our research groups and get everyone to use ROS and I think it was overall really effective strategy. And so I was certainly part of that I brought ROS and particularly move it back to our research group at CU Boulder, Colorado. And and so I was developing some of the initial parts of move it together with Johan sukan. And the team. And I kind of decided to base the rest of my research during my graduate program around move it and kind of did a lot of open source contributions to

it. Mm hmm. How did that look like doing a PhD but also contributing largely to the open source part of this head? Like, it seems like the incentives are slightly different for what the PhD would want and what an open source really useful tool with. One How did that go? Do

I think I think you know the answer this question. leading question. I think it worked well for me, because working at open source robotics gives you so much hands on experience. It's like relevant to industry relevant to getting jobs after the fact. So I think it's a great way to do That kind of sharpen your role or experience. However, if your main motive is to publish papers, then it could be distracting. Maybe just I don't have any experience about this possibly could distract you from publishing enough. But on the flip side, if done well, there's been a lot of examples where you do some pretty cool research and you publish the open source code to it. I encourage everyone listening to do that if you're in academia and and you can actually increase your citations because people within convention market and compare it against other methods. And so there's there's definitely advantages to doing both.

Yes, for sure. And then how did you go from PhD student contributing to this to now leading up to kind of taking on the initiative yourself.

I just looked it up. By the way, movie beta was officially released in 2013. And that same year, that will close to actually and there was just kind of a leadership void in the movement. I think the the project was kind of floundering, and there are a lot of pull requests piling up and not being merged in. There was a little bit of support from SRI International after Willow close, but it didn't last for very long. And so I was encouraged by a number of people that Open Robotics and others that were had been at Willow Garage who had been involved in move it that I kind of like try revitalize it. So Michael Ferguson was a big player and encouraging it to be revitalized, I think he would the time was starting, but the precursors to Fetch, and so that he was interested in using Uber for that. So he and I plan will move a day, the first one ever and so that was a big Kickstart to getting a community kind of excited about moving again. And conveying that it is a thing and a big part of that was getting the website updated, so that it was just showing that the latest things happening and with community. And there were two other maintainers at the time, who were still involved, Robert hash and Michael corner. And and so we had our first maintainer meeting, I think it was kind of three or four of us. And we're like, yeah, let's, let's see if we can get this thing flying. Again, I think that was a what Michael said. And sure enough, what we did we you know, to keep an open source project is just responding to pull requests fast enough and reliably. And if you have enough people, so myself, Michael and Robert, responding to pull requests, giving decent reviews, keeping the codebase stable, you'll eventually build a community assuming that the software does anything useful for people. And that's exactly what we did. One of the big issues I first took was to put our first continuous integration in the place which nowadays various Bordeaux belaz. A, but like back at Willow Garage, that was still kind of cutting edge. And we didn't have ci yet. So Jonathan Bourne helped me get Travis working. And then for the first time, we can merge things more confidence not breaking down,

just to be clear, so ci is continuous integration, and that's running your tests that test that the code is working correctly.

Right? Yeah. And he only went on your test that you have tests and and we've over the years, gotten the test, better remove it, but there's still not enough and at the time, there wasn't not much at all. And so not only the test it just test the builds, and that you can recreate the build, the dependencies are also correctly and, and so one of the big goals of adding ci was reducing the burden for the volunteer maintainers. And so you just want to get as many as much automation and bots helping so we, I implemented like a clang format style guide enforcer, and, and those sorts of things so that the burden of code reviews got easier for us. And you know, you don't have to always download the pull request and test it locally. Maybe for complex ones, you still want to do that. But you can just review and say this looks right. The ci says it works. So let's merge it. And that's a way of leveraging automation to

reduce what we have to do. Okay, so you did a lot of kind of this important infrastructure that made it easier for you guys to go and review pull requests quickly so that the community can kind of build it up? Is that really the way to look at it with the community contributing a lot, and you're kind of just like, managing it so that it stays working? Or you guys do your own feature development?

Yeah, yeah. It is the way I kind of look at it. There's different models. So some open source projects, there's just one or two or maybe a small lab group of people who are doing all the actual feature development. And then everyone else is kind of just benefiting and taking and maybe submitting a bug report but not knowing how to fix it. Personally, I've always taken the approach of this project has too many too or too much code. There's too many features like I can't do all this myself. So we're going to leverage the community and there's pluses and minuses to putting your energy towards fixing the problems versus it making easier for others to fix the problems. And you know, it's not as simple as it sounds, adding people in because they have to then spend the time understanding the codebase And like, like dissecting it and having quality reviews. So there's a there's a bit of both, I definitely added a lot of features just as I was working on my PhD, I was like, Oh, I need this feature, I'm going to get this merged in. So

gotcha. Okay. So that's, it's an interesting approach to focus mostly, are focused largely on making it easier for people to contribute. And it's a cool approach, I do see what you mean, when the code base grows to be so large that you need to make sure that like, basically, you just facilitate it. Rather than going deep into adding additional users, you can't do it yourself, because it's too large. So it seems like a good scalable approach to make this kind of thing, especially when you guys have a few reviewers

have we been able to grow the number of reviewers, we call them maintainers over the years, and that that's an important aspect is encouraging newcomers to get more involved and get eventually give them the the access to have merge access, right access, and we actually set up a program, I'd love to share real quick to help with that, to bring in beginners more, we set up a core contributor program. And the idea being that when we see people who are getting really involved, and maybe they don't have quite the in depth skills yet, then we don't fully trust them. But we want to really appreciate what they're doing and set them up to become a maintainer in the future, we'll add them as a core contributor and put them on our website. So they get some credit for that. And, you know, I'm not sure if recently we've been using that enough, but it's been a plus, you can see it on our website on the on the people.

How does it compare to like an internship or something, it's something someone does on their own time, and you try to guide them or mentor them a little bit more, or work.

It's pretty informal. It's not as nearly as structured as an internship, I mean, one, there's no no pay, and most engineering internships, you do get paid. But it's more like if you were a grad student working on motion planning for your research, and you know how to use move it and you're just a rising star, then we'll start off saying, like, you're awesome, be a core contributor. And then after a few months or a year, if they continue to, like really demonstrate that they understand the code base, and then they're capable of maintaining the quality standards and so forth. We would invite them to become, you know, a full ride access maintainer. So is there a mentorship? Like, I think anyone who gets involved in that will get mentorship via pull request reviews, I think this is often undervalued, but the time that people put, so you say I want to change this code. And if there are maintainers, who decide to respond to it, they will spend their time showing you like, why certain code patterns or bugs or memory leaks, like helping you improve your code quality over time. So that's a great way to get mentorship. It actually was one of the ways I learned was just watching my recruiter get reviewed. And also I watched other people's code get reviewed. And I learned a lot that way. Because I'd be like, Oh, I didn't see that issue. Next time. I'll know better to like, look out for that,

huh, yeah, that's been a lot of my experience to that code reviews are it because it's, it's line by line through the thing through the code change that you have. And then they can critique you on the approach or anything. So it's a really good way to get feedback. And it can be done asynchronously. So you can take a while to like, understand the comments, and get lots of different approaches. So I've personally found that very valuable at Open Robotics, code reviews, and even community code reviews, which are always amazing when someone steps in from the community.

Yeah, totally, there's just got to be, yeah, sometimes have a thick skin because there's a lot of code reviewed. And sometimes, I'm sure I've been guilty of this, like you review it really quickly, and you don't like cherry code it so much, you're just like, not listening to me. And then I've encouraged people at the movement maintainer group to still always say thank you for this contribution. It's kind of a summary. Even if we've like, totally red line and torn apart, you know, so you gotta gotta just be aware that we're just trying to help you and keep the code quality up so that future people can improve it

as well. Yeah, at Open Robotics, we talked about how a lot of times in text, it seems to be a lot more harsh, whatever you say. So then kind of being aware of that while giving feedback. But you're right, when there's a lot of code to review. It's like, it's, it's tough to have really, you have to get through it. And so you might make a comment. That's a little rough. That's awesome, though. Are you so you have this program to bring on people who are contributing and give them a little more responsibility within the organization. How has it been to grow from your core group of three to now you guys around 30 employees if I'm correct, or

Yeah, I do want to differentiate the new community project and my company picnic are not they're not the same set.

Okay, let's, let's talk. So let's explain picnic, I guess.

Yeah. But to finish the finish that first thought. So Michael and Robert, for example, they are a professor or a PhD student and individ respectively, in Germany, on affiliated with picnic, and yet they are major contributors to the project, there's a number of other maintainers unaffiliated with picnic like Felix, who does understand a lot of great stuff. So maybe half at this point of the maintainers are from picnic and I, I'm proud of that. And I'm glad that we've been able to hire people and get them excited enough about open source that they're like, major contributors to the project now. like hitting Kaiser has been a big lead for the movie project Tyler. And so yeah, there's, there's a mix, and we want to make sure that the movie project remains something separate from picnic, although there are just plenty of overlap as well.

There's a lot of overlap. So can you tell me a bit about the two? So move it is the open source, motion planning framework? And picnic? is a consulting and research company around this area? Or how would you think of it?

Yeah, we'd be called reasonable consulting. In the early days, I've actually tried to distance myself from that word, because I prefer the word engineering services, it comes off more as like we want to co develop or develop for you robotic software, using movement using ROS using other perception navigation packages. And so you're not just like advising you on how to do it. I mean, that is a small part of what we do. But it's not our main goal. By working with your with other companies and other teams, we do kind of a consult with them, but we do it through working side by side.

Yeah, so it's developing a lot rather than just advising. And that's why you like this engineering services term better than consulting. We're more hands on, hands on. And does it? So if you use the two different terms, does it change the types of clients that come to you? Or what's the or it's how you think of yourself, or why

it changes the mean, words are important and changes the mindset when, when a potential client comes to us. It's not that we're going to get on a meeting with you once a week for an hour and discuss the best way of using moving we do offer feasibility studies. And those have been very popular with our customers. But we hope that after the feasibility study that they would then hire us to actually implement a lot or some of it all of it, you know, some combination of that with their team. And so that's kind of our consulting offering is the feasibility study.

I see. And then so now, a lot of this is it. The business model is something like, if they allow you to open source, whatever you're developing for move it, or for them for move it, then they don't like it's incentivized kind of you charge them less if what you are doing can be open source, or it's a feature you want, or how does a How does it work? we don't we

don't have a model like that. We were pretty clear and contracts. And we have a lot of discussions with with lawyers that when you hire a picnic, some portion of it is going to be open source. And the only question is how much and we've had some projects that everything is open sourced, and we're delighted. But in reality, we don't expect that that's not normal. It's it's typically that like, we're helping them build out their application specific IP, intellectual property. And it's building off of move it or ROS or other components of the whole open source device ecosystem. And we insist that bug fixes or small features, hooks, just little things that like that really shouldn't be anything different shitting them, those get open sourced. And you know, we if it's like a thing that originated in, move it or in ROS, and we're just like slightly improving, it will likely open source that. But if it's like some fancy new algorithm for a particular type of application, then absolutely like, if our customer wants that proprietary, we're not, we're not here to like, be be like do to zealot about it. Huh. Okay. The key is that we don't want to fix the same bug over and over

again. That would be ridiculous. Yeah, for sure. And also, it's nice for the whole community to just benefit from any work you do with these things.

Yeah, it's kind of like giving it back paying it forward type of thing of like, you're benefiting from all this prior work that's saved you tons of r&d hours and like bootstrap, or program you have and and now we just ask that some of that be contributed back to the next person or, you know, like one of the big moments we have about the question of like open sourcing versus like forking open source project, is that you don't want to take on maintenance and ownership. That project in the future, but if you stay with the mainland project, then companies like picnic and Open Robotics and, and other contributors will take care of that for you maintaining the basics of bug fixes and new opera operating systems and patches. Yeah.

And just to be clear with the terminology, so when you're forking a project, you're taking it and you're saying, This is unlike I'm taking it in its current state, and I'm gonna go move it somewhere else. And then I'm going to work on it and maintain so you can hide it, what you actually contribute to, if you're some company, I'm sure that they diverged. Oh, totally sure.

Yeah. So it's nice having it all in the open, so they can just grab the main movement branch or main movement repository and then apply fixes directly to that and benefit everyone. That's it seems like the big argument for open source to me.

Yeah, and it's like, it's not just moving everyone, but it's benefiting the individual companies making these decisions, because they're not taking on the ownership. They say that like, once you like writing a line of code, that the cost of that for like an engineer to do that is only a fraction of the overall cost of that code over its lifetime. Because for every like hour spent coding, you're gonna spend several more hours maintaining it bug fixing in over the lifetime of your product, or your robotics project, or what have you. So it's really reducing your maintenance cost over the long term.

Yep. Yeah. And time and I don't know, energy and everything. There's just so many things, even like running up your continuous integration time, because you have to test more things which make it more difficult, as well as just engineering hours to fix bugs.

It's a good point. And we're talking a lot about like industry users of open source software. But you know, there's a lot of other reasons and motivations why students graduate students would but choose to contribute to open source as well. So there's, there's both worlds, I just want to

want to say, what is what are some of the motivators for people in academia, students to contribute to open source,

it's building up your portfolio, your experience, getting that mentorship, we were talking about, kind of being a part of something bigger, there's, there's this just a sense of like, seeing that you contribute to code that's being used all over the world, and being able to brag about that. And like for many people, maybe they haven't ever contributed to real software before. And so this is like, kind of a good first step in their career. So there's those sorts of things, but really, there's a chance to still learn a lot.

Hmm, totally agree. Yeah, my experience at open robotics. Really, I get a lot of satisfaction from contributing to things that I know that other people are using. So I agree with that. So okay, going back a little bit. picnic, when did picnic come along? in the scheme of things, so you were contributing and your PhD to this? And it was you in a small group of other maintainers? How, where did picnic come from?

It was 2015. And I like yourself. But recently, I was an intern. And I was interning at Google. So pretty lucky place to be at Google robotics to be exact. And after the internship ended, there was a group there who wanted I'd help them use move it for some projects. may actually say too much here. But they wanted more of that they wanted more support. And they asked for another internship. I said, No, I'm not going to. I've had a lot of internships. And so they said, Do you want to consult for Google? And I said, Of course I do. And so that is what bogen picnic. And technically, like we had been so picnic, the term originated from a team at the University that I was attending for the Amazon picking challenge. And so it can challenge picnic. A friend just helped me coined this term. So I asked the team, I was the team lead. I was like, Hey, can I use this term for doing some consulting? And everyone's like, no problem. So that's kind of the origin story.

Gotcha. And so it was just you at the beginning consulting for Google for some project that use move it. And you grew from there to now 30 people or so. Yep, kind of thing. Yeah. That's exciting. So 2015 it started and we're in 2021. Yeah,

that's also I had to finish my PhD. So I graduated, took a little time off to travel. And when I came back, I just kept getting requests from different companies that were using move it for support. And so I'd fly out to different companies and do more consulting but also a lot of coding. And I realized I had too much work and I made the pivotal decision to bring on some co founders and that certainly changed things. Spent a lot of work since then an energy company. But it's been really fun and exciting also.

Hmm. Let's see. So what have been some of the projects that you've worked on with picnic? Like just the nature of it. So the kinds of things you can solve with move it in less abstract form?

Sure, yeah, I will avoid company names because India is get tricky. And we're on a public forum here. But one of so like, logistics was our second project, like warehouse bin picking type of stuff. The next one was robot cooking. And then a really big one for us was cooking. Yeah, you know, like, during things cutting things like home chef.

like, there's a couple a couple companies out there doing this. Yeah, we've actually worked with a lot of them. But that was one of our early projects. And, yeah, we know, all of our bigger projects has been surgical, robotics, and, you know, move. It's been used to do surgery. But, of course, they had to do a lot of rewriting of certain segments to make it FDA certifiable. And, like, move, it's a really great starting point for some applications. But sometimes you have to, like, do a lot of shaping it to be something that's safe enough for human use. So I don't recommend you use move it out of the box to do surgery. Don't mistake me here. But yeah, we work with this company for a long time. It was a really fun project. And and since then, like the application we use it for, it just astonishes me how diverse it's been since then. I mean, we're, it's we've worked for several space applications, several underwater robotics, like doing manipulation with oil and gas. telehealth because all over the place we're doing one of our big projects now is robot farming. So harvesting of fruits and vegetables, you name it, like we've probably been involved.

That's so cool. And it's because motion planning is just such a fundamental robotics problem. And you guys have a bunch of really great tools around there.

Yeah, it turns out, like, even though you don't see robot arms used a lot outside of the factory, I think you're gonna see a lot more of that very soon, because we're working with a lot of early stage r&d groups, whether corporate or startup, to use robots for new applications. But it turns out that robot arms are extremely useful because the whole world is designed for human arms. And that's what we're trying to mimic.

Hmm. Okay, that's really cool. And I actually towards the end of the interview, I want to ask about where you think things are going. Because I assume from all the consulting, that you have an incredible idea about kind of where things are headed, because you're seeing these early applications, which is so cool. But the Okay, so you started this now? Congratulations, you're having larger customers, you mentioned that you have a collaboration with NASA. Would you talk a little bit about this? Sure.

Yeah, thank you for for bringing that up. I just think space is really cool. This is really cool. It seems like there's this big movement, at least in the US. But I think it's pretty international of commercializing space, it's becoming more privatized, there's a lot of startups there was the X PRIZE a few years ago. And some companies came out of that, like that Virgin Galactic flight just recently came out of that. And just in general, now that SpaceX has made flights, getting stuff to space, so much cheaper and more regular, we're seeing a lot of other applications. And so a lot of companies are working on satellite assembly is a big one. And using robot arms for that doing more experiments in the space station or in the future gateway station being worked on.

I don't I don't know very much about space applications, or the different ways that companies are trying to make a profit out of space. So you said satellite assembly that's on Earth, or is that in space? Oh, wow. So you send up the components and then build it out up there? Yeah. Okay.

Yeah, I think one of the big motivations is that things have to currently compress into these little rocket modules. But sometimes you want to build things bigger than that. And so you need to assemble in space.

Hmm. Okay. And that would be used for a bunch of applications. Like taking photos of the earth or maybe sending internet places like starlink or whatever it might be this kind of thing.

Yeah. It I guess I don't know. a ton about like, What? What I mean, yeah, assembling satellites for all the normal cases what you just said,

Yep. Okay, and then what other applications were you saying? first base.

So the doing more things without astronauts in space like on the space station so NASA has had this long standing program to have robots augment the astronauts time astronauts are wildly expensive to send up like every hour of their work day their their billable rate is very very high now because because of just all the training all the fuel Yeah,

infrastructure would you I mean, I bet it's like comically large how expensive they are per hour what would you do about what it was but it's like $10,000 an hour or is it way more than that?

okay so what are what is it that you guys are doing? Are you looking for a number

Yeah, I am looking for a number but i don't i don't see it so they're using move it right now on the this program called the Robonaut which is in the space station. And to like help do some more than maintenance tasks of the station. Like cleaning surfaces or like just flipping switches just like things that like maybe aren't worth the astronauts time

would you describe Bravo not because it's a pretty wild looking thing.

Yeah, it's it's very anthropomorphic, it like to look like a human, for the most part. So it's got fully dexterous fingers. It's got a head that can rotate. Look around, it's got like a chest just like a human. The legs however, are a bit creepier. They're think more like yes, spider legs, there's less. But because you're in zero G, you don't need legs that can walk like on Earth, you just need something to grab on to handrails. And so that part's a little creepier. And I am not the definitive source of the status of this program. But it's, I think it's a little bit winding down. And they're like, revisiting what kind of robots best for space. And, and so for the gateway, which is a new station being launched soon, to orbit the moon, one of the big needs, there is a inner vehicle robot that can take care of the station when it's unmanned, because they aren't planning on having a human there. All times of the year, I think closer to like four months of the year, it'll be manned. Eight months a year, the robots got to take care of the place. And so we're currently on a grant involved with that.

Hmm, so it'll be the robo Robo robot on this down the gateway, which is orbiting the moon. And it will be taking care of a lot of operations by doing what a human might do if they were on it. Is it going to be tele operated? Or is it going to be largely autonomous or?

Yeah, I love this topic. I think teleoperated to me, the definition of that is really drive by wire, you've got a joystick, or a mouse, or these days, a VR headset with two hand controllers. And you're telling the robot what to do like basically one to one. And you know, it's not exactly one to one because oftentimes, the robot arm may have a different kinematic morphology than a human. And so the way it maps out your command might be slightly different. But for the most part, you know, I'm talking about like Cartesian control, which is like effector control. But for the most part, you're just you tell what to do. That's totally up to me. Yeah, so

you move like it would be like moving joysticks around, like the VR things. And the robot would do pretty much the same motion, but it might do slightly different things. Because its joints might be different than humans. Exactly. Okay.

Yeah. And so that breaks down when you're on the moon. And particularly if you're going to Mars, which is the next stepping stone for the space program, as Ilan, lets us all know. Because latency and bandwidth issues, delays in time. So I think to the moon it's like three or four seconds, and to the Mars to Mars, it's like several minutes. Don't quote me on this, I don't memorize numbers, as you can tell. So that becomes really annoying and frustrating to control something with that amount of lag because you send a command and then you have to wait round trip for it to come back. So we're very interested technic, we're developing some technology that is a mix of fully autonomous and tele operated. So what's between those two extremes? And that's, yeah,

that's what you're trying to figure out, or that's what we're proposing is what we're working on right now. So is it like a higher level thing where you can delegate tasks like at a task level, you say, Go select this, go do your thing here and it will kind of get over so if it's flip a switch will, the robot will go over there and position itself and perhaps flick it, but you've directed that it should do that? So it's like a higher level command and I think we're, Ah,

that's, this goes under a number of names. But supervisor autonomy is maybe the most common one where you You are in such

it does. So you tell the robot, hey, here's a high level task. And then you're kind of observing the robot as it goes about doing it with the understanding that maybe the robots not quite smart enough to fully do the task. Because these are just hard problems. They're kind of unsolved so far, especially when you're in unstructured environments. I've had arguments about whether a space station like the ISS is unstructured or not, it's kind of an academic argument. Because I know my mind, these things are highly engineered things where they have exact computer models before they get launched. You look at videos on like YouTube of astronauts, giving you a tour of the Space Station. It's really striking how many bags and wires and new experiments are just floating in the middle of the way. And so actually, it's an extremely complex environment. So for a robot to be able to fully autonomously navigate this ever changing environment to do useful things. It Yeah, it's hard for today's algorithms. And so the AI going back to supervise autonomy, there's a human there to help it along when needed, but really minimizing how much intervention is needed from that from the human.

Okay, that's really cool. Do you think that this will be a larger thing in robotics? Like the supervise autonomy? Do you think before, I mean, we see it with like, like, you can think of like Tesla and its self driving as being somewhat supervised autonomy, you sit there in the seat, and it drives for you until it cannot handle whatever's going on. And then it gives you back control. Do you think that this is going to be a larger trend before robots are like actually capable of exactly those things entirely?

It's exactly what I think. So to Tesla versus whammo versus the other crews, self driving car companies, they all have slightly different approaches to how they're doing autonomous vehicles. But I think it's really interesting that Weibo and a number of the other ones, they've all kind of realized that in order to get to L five, autonomy, actually, you need a remote operator that occasionally can help this, like the robo taxi, get unstuck. So you know, a lot of the prototype or concept vehicles that I think are going to be out soon, they may or may not have a student steering wheel. And so if you think about a taxi, and maybe the seats are facing inward, there's no one there to take over. If the robot just the car really gets stuck. A Tesla has a steering wheel still, and they're not. They're a different player. So I would, they're a different category of autonomy in my opinion. But the idea with like way, Mo is that there's someone in a call center who maybe isn't watching every turn every step of the way. But when it's needed, they can be, you know, called in to give the robot a little more understanding, like, Hey, this is a construction site. We haven't quite seen this problem before. But this is what I want you to do. And the robot can then go, Okay, I'm going to maneuver around this. So that's proving out to be a necessary need in self driving cars. And I think it's also going to be more and more in need as we get robot arms outside of factories,

huh? Yeah, I bet you're right. It's a very interesting thing. Because you can kind of augment where, like robots can do maybe 80% of it, or 70% of it. But then there's some part of it that people just need to be there for their judgment. And so we can buffer.

I think you kind of hinted at it a second ago, like, eventually, we won't need this again, like we're gonna get the machine learning algorithms and the software and all of it so intelligent that the human will be then removed completely. But there's been some interesting research about like, how AI is harder than we think. And we've hit some critical limits of current machine learning techniques. And this is why the self driving car industry has been so delayed. And so I think it's going to be longer than we think like, you kind of have to get general intelligence, General AI in order to completely eliminate the human from it. And I'm not super stoked personally about even achieving that, because then you have the whole super intelligence thing. And what if it gets smarter than us? And

Terminator? Yeah, yeah. All that stuff. Yeah, I don't know. It is all very interesting. And I do from my own, like, foray into machine learning and things I have trouble understanding how the metaphors can get to a more general intelligence at the moment,

which is probably good for humanity. Maybe Yeah.

But Alright, so then your, how long have you guys been on this NASA project?

It's actually just the past year so not not even a year, just under a year. We've been involved in this. But we recently got our next phase of grant money with them and Exactly. So So that's we're doing this for NASA. But we're also starting to do this more and more with our other customers, using open source, move it, but also developing some higher level task planning proprietary aspects of it that we, because we've had some compasses customers asked us to build this kind of one off. And now we're making kind of a general solution for this overall supervisor autonomy problem. So I think that's going to really unlock a lot of applications in the next decade

or two. Yeah, I've been I don't know, I've been thinking that this is one of the big problems on the front of robotics like, and I think you're hitting on the head and actually doing it, which is very cool. Okay, yeah, making it so that you can delegate at a higher level for robots what to do, and then have some sort of thing come in and help babies. It just it seems so unnecessary.

I think there's an interesting, interesting pattern about past robot companies and how they've gotten around this and why maybe there isn't a solution out there for this exactly, is that typically, you choose an application, I'm going to solve like a robot, that was a good example here that picks widgets from a bin, that's just right, this is a big one, right now, a lot of companies going after like, a mixed parcel box. And you can make a bunch of assumptions and fixture stuff, and do just enough autonomy and AI and like computer vision to solve that problem really well and reliably. And those those solutions that keep coming to market. So like in the 70s, maybe the problem was, how do we spray paint a car really reliably? And they came up with exactly the tools for that. And so we keep pushing slowly, the envelope of a robot doing a very fixed thing for harder problems. But and there's there's a lot of benefits, like fully autonomous has a better return on investment in terms of like, you're totally eliminating the job versus like still requiring a human to be kind of assisting and sometimes, yep, I just think that's not going to go all the way to like a robot that can navigate through your your office, your home, like outside, in a wreck and just like assist or space, in particular, for any arbitrary problem you throw at it and have it be able to achieve it. So that's, I think this was me next?

Yes, yeah, even the modern ones, it's like, they're somewhat like the picking the widgets out of the bin. The approaches seem to be somewhat flexible to variation. But it's like you throw a different bin or a new widget in, and then you have to reprogram everything and add new, like hacks or less general solutions to get it to work again.

And maybe yet, maybe machine learning techniques, the difficulty of adding a new widget is gone way down. Maybe I think it's probably really easy to arbitrary object, train it really quickly. Yeah, but he still if you ask him to then, like, paint a car or do a different, totally different type. It's just not going to be Yeah.

Yep. Yeah, that's a better way to phrase it. It's just if you choose a different task, it is unable to cope with the same system, you can run another program, and have that actually work, perhaps, but it has to be a specialized program in itself to do the different tasks. Yeah, I

think the key there is you said you have to run another program. And so who isn't you the US? Yep. So so like really giving the operator a slew of a library of programs that are more general and not the robot do a lot of autonomy. But when the robot like gets stumped, there's a human there to like, push it along, like, Hey, here's a little hint. And then one really cool idea that we are not, you know, pursuing exactly yet. But like, you have all this human interfering or intervening helping the robot, and then that actually creates more training data to generate. So then you have this life cycle of labelled semantic data that you can use for future machine learning models. It's

a virtuous cycle, it is totally more data more to train on, you can get more sophisticated and you get closer to actually solving your problem.

I'm not a machine learning expert, though, so I won't I won't claim any of them

either. Sure. Let's see. So that's really exciting. How do you so witness and I don't know if you can say, how are you doing? Like, do you have a simulation of what it's like to be on the on the gateway that space? The What do they call them? space stations? Yeah, the space station that's going around the moon, do you have a simulation of that? Are you working on like, don't like really specific problems for them? Are you working on general technologies that they're going to apply to that situation or I assume it's a mix

more general technologies at this point, we hope to move towards more specific problems in the future. But move it in has been used a lot with the space station. And so in gazebo, another open source common library and ROS, they groups at NASA have created not full simulation space station, but a full simulation of a particular module, they included the railing that you can grab on to, and maybe some problems and so they're dead there hasn't worked that the Robonaut using the open motion planning library would move through this environment using its creepy spider legs and arms to like, pull a cargo bag and like open the Velcro, like some pretty cool tasks. That that's been worked on. And so some of the guys that are at our company have been involved in that program.

So cool. Is there any heart? So I guess, does this already? Does the gateway already exist? Or is it already up there? That's a thing in the future.

So are there? I mean, I'm just wondering if you're like doing the sinusoidal flight in an airplane with Robo not that's floating for a few seconds and trying to do something or

no, no, I mean, like it we're not, we're not that involved in this yet. Because that would be so cool. That would be

so you could fight against gravity by doing that. It's so cool.

I mean, we don't have gravity, it just makes all the manipulation problems, I think easier, because if they don't have any momentum, they just stay.

It's true. Unless if they would be floating, it would probably be very difficult. And there's no forced precedent like they just move away from you if you put any force that they symmetric into good point. But easier if they're stationary, because you just but Hmm. So I think one thing that seems really exciting about this, to me is open source being used at NASA is amazing. Like that. I feel like that's a huge win for the open source community. Can you talk a bit about that?

Yeah, they've they've been involved with ROS and funding Open Robotics for a while, I believe is my understanding. And I think that they see that this is a great way of tech transfer from what's the latest in academia. And they see it as a way to kind of standardize and hire people who already know, the technology and I ROS just kind of become a standard. And so NASA organizes that on various teams at NASA recognize it.

The really interesting thing to me is when I think NASA I think of like incredibly robot, like they're really concerned about reliability. And it's interesting to me that they're going to open source for things that are important in their application, like controlling a robo not. Yeah, station, like that's really interesting to me.

Yeah, good question. May is not a question. But I have a thought on that is the robot, their strategy, the group at Johnson Space Center, was to make the low level controls really heartened, and like, I'm not sure if I'd call it like flight certified but certified quality and rigorous such that it was guaranteed not to extract to exert the force to like punch into a space station, for example, like, they can't break the station, because that's extremely dangerous. And so if you aren't like if the underlying controllers and like the hardware interface, it like reaches the right certification level of safety and that it can't go beyond a certain velocity and sort of torque, then you can then think about the layer above that kind of the application layer with a more modern like rapid iteration software development techniques, that allows for more powerful compute and more like advanced methodologies. But sometimes it does, maybe sacrifice the rigor of a very simple system that is flight ready.

1 post with the motion planning tag