New directions for the MoveIt project | Sense Think Act Podcast

2. New directions for the MoveIt project, with Dave Coleman

2021-09-06 · 1:02:53

In this episode, Audrow Nash interviews Dave Coleman, Chief Executive Officer at PickNik Robotics. Dave speaks at a high level about what MoveIt is and what problems it helps roboticists solve, they talk about supervised autonomy, including a collaboration with NASA and MoveIt Studio, and Dave talks about MoveIt 3.0.

Comments

Transcript

The transcript is for informational purposes and is not guaranteed to be correct.

(0:00:03) Audrow Nash

Hi, everyone, welcome to the sense think act podcast. I'm Audrow Nash, your host and a software engineer at Open Robotics. In today's episode, I speak with Dave Coleman, CEO at PickNik, which does engineering services and motion planning. Motion planning is the problem of moving something from one place to another, such as moving a robot from one location to another, or picking up an object and placing it somewhere else. This can be made more difficult if you try to avoid collisions. For example, if you had something beneath a table and want to lift it up over the table, you have to go around the table. And as you can imagine, this is a fundamental problem in robotics. And Dave has been involved in move it a motion planning framework that has built on top of robot operating system or ROS for a number of years, and has created picnic to do engineering services around motion planning. We talked about his motion planning or the motion planning framework, move it, how PickNik came to be there work with NASA, and a paid service called movement studio that hopes to help the problem of supervised autonomy. And we teased the idea of move it three. I really enjoyed talking to Dave, he is a fun person to interview. And I find that he has a great perspective on robotics and open source, probably in no small part because of the diverse problems that they work on at picnic, and his time that he has been in the robotics community. I hope you enjoy. So Dave, would you start off by introducing yourself.

(0:01:57) Dave Coleman

Yeah. I'm Dave Coleman, CEO of PickNik robotics and a longtime supporter of open source robotics contributor to ROS and MoveIt. Glad to be here today.

(0:02:11) Audrow Nash

Awesome. To start off as a general overview, would you just tell us about move it? What problems does it solve and how long it's been going?

(0:02:20) Dave Coleman

Sure. Move it is a framework developed to do motion planning primarily for robotic arms, but also for the full kinematics of your robot. It's had some success and walking in mobile bases, even by esoteric things like quite like flying robots and submersible so all over the place move it's been applied. And it's the problem of how to motion plan your, your joints through space, so you don't hit things and that you can do useful things like manipulation and grasping. And your second question, it's been around since pic. 2010 was when development begun, I think 2011 was the beta release of move it and this was developed at Willow Garage. Same place that ROS was developed, it was developed with the PRT program. Sasha Cheetah was the program manager at the time of that project. And yeah, Willow Garage eventually closed, but it lives on.

(0:03:22) Audrow Nash

Let's see. And I know we said we talked about kind of you're coming into this a bit later in the interview, but might as well get into it now. How did you get involved? So it started in 2010 ish? at Willow Garage. When did you come along?

(0:03:35) Dave Coleman

I was an intern at Willow Garage. Willow was very clever and having a lot of interns huge program. So I guess I'm not that special. And that the CEO at the time, Steve cousins said that the intern program was a way of spreading ROS. I like like a virus, he wanted us to go back to our research groups and get everyone to use ROS and I think it was overall really effective strategy. And so I was certainly part of that I brought ROS and particularly move it back to our research group at CU Boulder, Colorado. And and so I was developing some of the initial parts of move it together with Johan sukan. And the team. And I kind of decided to base the rest of my research during my graduate program around move it and kind of did a lot of open source contributions to

(0:04:22) Audrow Nash

it. Mm hmm. How did that look like doing a PhD but also contributing largely to the open source part of this head? Like, it seems like the incentives are slightly different for what the PhD would want and what an open source really useful tool with. One How did that go? Do

(0:04:40) Dave Coleman

I think I think you know the answer this question. leading question. I think it worked well for me, because working at open source robotics gives you so much hands on experience. It's like relevant to industry relevant to getting jobs after the fact. So I think it's a great way to do That kind of sharpen your role or experience. However, if your main motive is to publish papers, then it could be distracting. Maybe just I don't have any experience about this possibly could distract you from publishing enough. But on the flip side, if done well, there's been a lot of examples where you do some pretty cool research and you publish the open source code to it. I encourage everyone listening to do that if you're in academia and and you can actually increase your citations because people within convention market and compare it against other methods. And so there's there's definitely advantages to doing both.

(0:05:36) Audrow Nash

Yes, for sure. And then how did you go from PhD student contributing to this to now leading up to kind of taking on the initiative yourself.

(0:05:49) Dave Coleman

I just looked it up. By the way, movie beta was officially released in 2013. And that same year, that will close to actually and there was just kind of a leadership void in the movement. I think the the project was kind of floundering, and there are a lot of pull requests piling up and not being merged in. There was a little bit of support from SRI International after Willow close, but it didn't last for very long. And so I was encouraged by a number of people that Open Robotics and others that were had been at Willow Garage who had been involved in move it that I kind of like try revitalize it. So Michael Ferguson was a big player and encouraging it to be revitalized, I think he would the time was starting, but the precursors to Fetch, and so that he was interested in using Uber for that. So he and I plan will move a day, the first one ever and so that was a big Kickstart to getting a community kind of excited about moving again. And conveying that it is a thing and a big part of that was getting the website updated, so that it was just showing that the latest things happening and with community. And there were two other maintainers at the time, who were still involved, Robert hash and Michael corner. And and so we had our first maintainer meeting, I think it was kind of three or four of us. And we're like, yeah, let's, let's see if we can get this thing flying. Again, I think that was a what Michael said. And sure enough, what we did we you know, to keep an open source project is just responding to pull requests fast enough and reliably. And if you have enough people, so myself, Michael and Robert, responding to pull requests, giving decent reviews, keeping the codebase stable, you'll eventually build a community assuming that the software does anything useful for people. And that's exactly what we did. One of the big issues I first took was to put our first continuous integration in the place which nowadays various Bordeaux belaz. A, but like back at Willow Garage, that was still kind of cutting edge. And we didn't have ci yet. So Jonathan Bourne helped me get Travis working. And then for the first time, we can merge things more confidence not breaking down,

(0:07:51) Audrow Nash

just to be clear, so ci is continuous integration, and that's running your tests that test that the code is working correctly.

(0:08:00) Dave Coleman

Right? Yeah. And he only went on your test that you have tests and and we've over the years, gotten the test, better remove it, but there's still not enough and at the time, there wasn't not much at all. And so not only the test it just test the builds, and that you can recreate the build, the dependencies are also correctly and, and so one of the big goals of adding ci was reducing the burden for the volunteer maintainers. And so you just want to get as many as much automation and bots helping so we, I implemented like a clang format style guide enforcer, and, and those sorts of things so that the burden of code reviews got easier for us. And you know, you don't have to always download the pull request and test it locally. Maybe for complex ones, you still want to do that. But you can just review and say this looks right. The ci says it works. So let's merge it. And that's a way of leveraging automation to

(0:08:50) Audrow Nash

reduce what we have to do. Okay, so you did a lot of kind of this important infrastructure that made it easier for you guys to go and review pull requests quickly so that the community can kind of build it up? Is that really the way to look at it with the community contributing a lot, and you're kind of just like, managing it so that it stays working? Or you guys do your own feature development?

(0:09:13) Dave Coleman

Yeah, yeah. It is the way I kind of look at it. There's different models. So some open source projects, there's just one or two or maybe a small lab group of people who are doing all the actual feature development. And then everyone else is kind of just benefiting and taking and maybe submitting a bug report but not knowing how to fix it. Personally, I've always taken the approach of this project has too many too or too much code. There's too many features like I can't do all this myself. So we're going to leverage the community and there's pluses and minuses to putting your energy towards fixing the problems versus it making easier for others to fix the problems. And you know, it's not as simple as it sounds, adding people in because they have to then spend the time understanding the codebase And like, like dissecting it and having quality reviews. So there's a there's a bit of both, I definitely added a lot of features just as I was working on my PhD, I was like, Oh, I need this feature, I'm going to get this merged in. So

(0:10:15) Audrow Nash

gotcha. Okay. So that's, it's an interesting approach to focus mostly, are focused largely on making it easier for people to contribute. And it's a cool approach, I do see what you mean, when the code base grows to be so large that you need to make sure that like, basically, you just facilitate it. Rather than going deep into adding additional users, you can't do it yourself, because it's too large. So it seems like a good scalable approach to make this kind of thing, especially when you guys have a few reviewers

(0:10:51) Dave Coleman

have we been able to grow the number of reviewers, we call them maintainers over the years, and that that's an important aspect is encouraging newcomers to get more involved and get eventually give them the the access to have merge access, right access, and we actually set up a program, I'd love to share real quick to help with that, to bring in beginners more, we set up a core contributor program. And the idea being that when we see people who are getting really involved, and maybe they don't have quite the in depth skills yet, then we don't fully trust them. But we want to really appreciate what they're doing and set them up to become a maintainer in the future, we'll add them as a core contributor and put them on our website. So they get some credit for that. And, you know, I'm not sure if recently we've been using that enough, but it's been a plus, you can see it on our website on the on the people.

(0:11:41) Audrow Nash

How does it compare to like an internship or something, it's something someone does on their own time, and you try to guide them or mentor them a little bit more, or work.

(0:11:52) Dave Coleman

It's pretty informal. It's not as nearly as structured as an internship, I mean, one, there's no no pay, and most engineering internships, you do get paid. But it's more like if you were a grad student working on motion planning for your research, and you know how to use move it and you're just a rising star, then we'll start off saying, like, you're awesome, be a core contributor. And then after a few months or a year, if they continue to, like really demonstrate that they understand the code base, and then they're capable of maintaining the quality standards and so forth. We would invite them to become, you know, a full ride access maintainer. So is there a mentorship? Like, I think anyone who gets involved in that will get mentorship via pull request reviews, I think this is often undervalued, but the time that people put, so you say I want to change this code. And if there are maintainers, who decide to respond to it, they will spend their time showing you like, why certain code patterns or bugs or memory leaks, like helping you improve your code quality over time. So that's a great way to get mentorship. It actually was one of the ways I learned was just watching my recruiter get reviewed. And also I watched other people's code get reviewed. And I learned a lot that way. Because I'd be like, Oh, I didn't see that issue. Next time. I'll know better to like, look out for that,

(0:13:07) Audrow Nash

huh, yeah, that's been a lot of my experience to that code reviews are it because it's, it's line by line through the thing through the code change that you have. And then they can critique you on the approach or anything. So it's a really good way to get feedback. And it can be done asynchronously. So you can take a while to like, understand the comments, and get lots of different approaches. So I've personally found that very valuable at Open Robotics, code reviews, and even community code reviews, which are always amazing when someone steps in from the community.

(0:13:40) Dave Coleman

Yeah, totally, there's just got to be, yeah, sometimes have a thick skin because there's a lot of code reviewed. And sometimes, I'm sure I've been guilty of this, like you review it really quickly, and you don't like cherry code it so much, you're just like, not listening to me. And then I've encouraged people at the movement maintainer group to still always say thank you for this contribution. It's kind of a summary. Even if we've like, totally red line and torn apart, you know, so you gotta gotta just be aware that we're just trying to help you and keep the code quality up so that future people can improve it

(0:14:11) Audrow Nash

as well. Yeah, at Open Robotics, we talked about how a lot of times in text, it seems to be a lot more harsh, whatever you say. So then kind of being aware of that while giving feedback. But you're right, when there's a lot of code to review. It's like, it's, it's tough to have really, you have to get through it. And so you might make a comment. That's a little rough. That's awesome, though. Are you so you have this program to bring on people who are contributing and give them a little more responsibility within the organization. How has it been to grow from your core group of three to now you guys around 30 employees if I'm correct, or

(0:14:56) Dave Coleman

Yeah, I do want to differentiate the new community project and my company picnic are not they're not the same set.

(0:15:06) Audrow Nash

Okay, let's, let's talk. So let's explain picnic, I guess.

(0:15:10) Dave Coleman

Yeah. But to finish the finish that first thought. So Michael and Robert, for example, they are a professor or a PhD student and individ respectively, in Germany, on affiliated with picnic, and yet they are major contributors to the project, there's a number of other maintainers unaffiliated with picnic like Felix, who does understand a lot of great stuff. So maybe half at this point of the maintainers are from picnic and I, I'm proud of that. And I'm glad that we've been able to hire people and get them excited enough about open source that they're like, major contributors to the project now. like hitting Kaiser has been a big lead for the movie project Tyler. And so yeah, there's, there's a mix, and we want to make sure that the movie project remains something separate from picnic, although there are just plenty of overlap as well.

(0:16:00) Audrow Nash

There's a lot of overlap. So can you tell me a bit about the two? So move it is the open source, motion planning framework? And picnic? is a consulting and research company around this area? Or how would you think of it?

(0:16:14) Dave Coleman

Yeah, we'd be called reasonable consulting. In the early days, I've actually tried to distance myself from that word, because I prefer the word engineering services, it comes off more as like we want to co develop or develop for you robotic software, using movement using ROS using other perception navigation packages. And so you're not just like advising you on how to do it. I mean, that is a small part of what we do. But it's not our main goal. By working with your with other companies and other teams, we do kind of a consult with them, but we do it through working side by side.

(0:16:52) Audrow Nash

Yeah, so it's developing a lot rather than just advising. And that's why you like this engineering services term better than consulting. We're more hands on, hands on. And does it? So if you use the two different terms, does it change the types of clients that come to you? Or what's the or it's how you think of yourself, or why

(0:17:15) Dave Coleman

it changes the mean, words are important and changes the mindset when, when a potential client comes to us. It's not that we're going to get on a meeting with you once a week for an hour and discuss the best way of using moving we do offer feasibility studies. And those have been very popular with our customers. But we hope that after the feasibility study that they would then hire us to actually implement a lot or some of it all of it, you know, some combination of that with their team. And so that's kind of our consulting offering is the feasibility study.

(0:17:45) Audrow Nash

I see. And then so now, a lot of this is it. The business model is something like, if they allow you to open source, whatever you're developing for move it, or for them for move it, then they don't like it's incentivized kind of you charge them less if what you are doing can be open source, or it's a feature you want, or how does a How does it work? we don't we

(0:18:11) Dave Coleman

don't have a model like that. We were pretty clear and contracts. And we have a lot of discussions with with lawyers that when you hire a picnic, some portion of it is going to be open source. And the only question is how much and we've had some projects that everything is open sourced, and we're delighted. But in reality, we don't expect that that's not normal. It's it's typically that like, we're helping them build out their application specific IP, intellectual property. And it's building off of move it or ROS or other components of the whole open source device ecosystem. And we insist that bug fixes or small features, hooks, just little things that like that really shouldn't be anything different shitting them, those get open sourced. And you know, we if it's like a thing that originated in, move it or in ROS, and we're just like slightly improving, it will likely open source that. But if it's like some fancy new algorithm for a particular type of application, then absolutely like, if our customer wants that proprietary, we're not, we're not here to like, be be like do to zealot about it. Huh. Okay. The key is that we don't want to fix the same bug over and over

(0:19:24) Audrow Nash

again. That would be ridiculous. Yeah, for sure. And also, it's nice for the whole community to just benefit from any work you do with these things.

(0:19:32) Dave Coleman

Yeah, it's kind of like giving it back paying it forward type of thing of like, you're benefiting from all this prior work that's saved you tons of r&d hours and like bootstrap, or program you have and and now we just ask that some of that be contributed back to the next person or, you know, like one of the big moments we have about the question of like open sourcing versus like forking open source project, is that you don't want to take on maintenance and ownership. That project in the future, but if you stay with the mainland project, then companies like picnic and Open Robotics and, and other contributors will take care of that for you maintaining the basics of bug fixes and new opera operating systems and patches. Yeah.

(0:20:15) Audrow Nash

And just to be clear with the terminology, so when you're forking a project, you're taking it and you're saying, This is unlike I'm taking it in its current state, and I'm gonna go move it somewhere else. And then I'm going to work on it and maintain so you can hide it, what you actually contribute to, if you're some company, I'm sure that they diverged. Oh, totally sure.

(0:20:35) Dave Coleman

You really can't see it typically? Nope.

(0:20:39) Audrow Nash

Yeah. So it's nice having it all in the open, so they can just grab the main movement branch or main movement repository and then apply fixes directly to that and benefit everyone. That's it seems like the big argument for open source to me.

(0:20:54) Dave Coleman

Yeah, and it's like, it's not just moving everyone, but it's benefiting the individual companies making these decisions, because they're not taking on the ownership. They say that like, once you like writing a line of code, that the cost of that for like an engineer to do that is only a fraction of the overall cost of that code over its lifetime. Because for every like hour spent coding, you're gonna spend several more hours maintaining it bug fixing in over the lifetime of your product, or your robotics project, or what have you. So it's really reducing your maintenance cost over the long term.

(0:21:25) Audrow Nash

Yep. Yeah. And time and I don't know, energy and everything. There's just so many things, even like running up your continuous integration time, because you have to test more things which make it more difficult, as well as just engineering hours to fix bugs.

(0:21:43) Dave Coleman

It's a good point. And we're talking a lot about like industry users of open source software. But you know, there's a lot of other reasons and motivations why students graduate students would but choose to contribute to open source as well. So there's, there's both worlds, I just want to

(0:22:00) Audrow Nash

want to say, what is what are some of the motivators for people in academia, students to contribute to open source,

(0:22:08) Dave Coleman

it's building up your portfolio, your experience, getting that mentorship, we were talking about, kind of being a part of something bigger, there's, there's this just a sense of like, seeing that you contribute to code that's being used all over the world, and being able to brag about that. And like for many people, maybe they haven't ever contributed to real software before. And so this is like, kind of a good first step in their career. So there's those sorts of things, but really, there's a chance to still learn a lot.

(0:22:36) Audrow Nash

Hmm, totally agree. Yeah, my experience at open robotics. Really, I get a lot of satisfaction from contributing to things that I know that other people are using. So I agree with that. So okay, going back a little bit. picnic, when did picnic come along? in the scheme of things, so you were contributing and your PhD to this? And it was you in a small group of other maintainers? How, where did picnic come from?

(0:23:06) Dave Coleman

It was 2015. And I like yourself. But recently, I was an intern. And I was interning at Google. So pretty lucky place to be at Google robotics to be exact. And after the internship ended, there was a group there who wanted I'd help them use move it for some projects. may actually say too much here. But they wanted more of that they wanted more support. And they asked for another internship. I said, No, I'm not going to. I've had a lot of internships. And so they said, Do you want to consult for Google? And I said, Of course I do. And so that is what bogen picnic. And technically, like we had been so picnic, the term originated from a team at the University that I was attending for the Amazon picking challenge. And so it can challenge picnic. A friend just helped me coined this term. So I asked the team, I was the team lead. I was like, Hey, can I use this term for doing some consulting? And everyone's like, no problem. So that's kind of the origin story.

(0:24:13) Audrow Nash

Gotcha. And so it was just you at the beginning consulting for Google for some project that use move it. And you grew from there to now 30 people or so. Yep, kind of thing. Yeah. That's exciting. So 2015 it started and we're in 2021. Yeah,

(0:24:32) Dave Coleman

that's also I had to finish my PhD. So I graduated, took a little time off to travel. And when I came back, I just kept getting requests from different companies that were using move it for support. And so I'd fly out to different companies and do more consulting but also a lot of coding. And I realized I had too much work and I made the pivotal decision to bring on some co founders and that certainly changed things. Spent a lot of work since then an energy company. But it's been really fun and exciting also.

(0:25:06) Audrow Nash

Hmm. Let's see. So what have been some of the projects that you've worked on with picnic? Like just the nature of it. So the kinds of things you can solve with move it in less abstract form?

(0:25:22) Dave Coleman

Sure, yeah, I will avoid company names because India is get tricky. And we're on a public forum here. But one of so like, logistics was our second project, like warehouse bin picking type of stuff. The next one was robot cooking. And then a really big one for us was cooking. Yeah, you know, like, during things cutting things like home chef.

(0:25:51) Audrow Nash

That's cool. Oh, yeah, there's

(0:25:53) Dave Coleman

like, there's a couple a couple companies out there doing this. Yeah, we've actually worked with a lot of them. But that was one of our early projects. And, yeah, we know, all of our bigger projects has been surgical, robotics, and, you know, move. It's been used to do surgery. But, of course, they had to do a lot of rewriting of certain segments to make it FDA certifiable. And, like, move, it's a really great starting point for some applications. But sometimes you have to, like, do a lot of shaping it to be something that's safe enough for human use. So I don't recommend you use move it out of the box to do surgery. Don't mistake me here. But yeah, we work with this company for a long time. It was a really fun project. And and since then, like the application we use it for, it just astonishes me how diverse it's been since then. I mean, we're, it's we've worked for several space applications, several underwater robotics, like doing manipulation with oil and gas. telehealth because all over the place we're doing one of our big projects now is robot farming. So harvesting of fruits and vegetables, you name it, like we've probably been involved.

(0:27:06) Audrow Nash

That's so cool. And it's because motion planning is just such a fundamental robotics problem. And you guys have a bunch of really great tools around there.

(0:27:15) Dave Coleman

Yeah, it turns out, like, even though you don't see robot arms used a lot outside of the factory, I think you're gonna see a lot more of that very soon, because we're working with a lot of early stage r&d groups, whether corporate or startup, to use robots for new applications. But it turns out that robot arms are extremely useful because the whole world is designed for human arms. And that's what we're trying to mimic.

(0:27:40) Audrow Nash

Hmm. Okay, that's really cool. And I actually towards the end of the interview, I want to ask about where you think things are going. Because I assume from all the consulting, that you have an incredible idea about kind of where things are headed, because you're seeing these early applications, which is so cool. But the Okay, so you started this now? Congratulations, you're having larger customers, you mentioned that you have a collaboration with NASA. Would you talk a little bit about this? Sure.

(0:28:15) Dave Coleman

Yeah, thank you for for bringing that up. I just think space is really cool. This is really cool. It seems like there's this big movement, at least in the US. But I think it's pretty international of commercializing space, it's becoming more privatized, there's a lot of startups there was the X PRIZE a few years ago. And some companies came out of that, like that Virgin Galactic flight just recently came out of that. And just in general, now that SpaceX has made flights, getting stuff to space, so much cheaper and more regular, we're seeing a lot of other applications. And so a lot of companies are working on satellite assembly is a big one. And using robot arms for that doing more experiments in the space station or in the future gateway station being worked on.

(0:29:00) Audrow Nash

I don't I don't know very much about space applications, or the different ways that companies are trying to make a profit out of space. So you said satellite assembly that's on Earth, or is that in space? Oh, wow. So you send up the components and then build it out up there? Yeah. Okay.

(0:29:19) Dave Coleman

Yeah, I think one of the big motivations is that things have to currently compress into these little rocket modules. But sometimes you want to build things bigger than that. And so you need to assemble in space.

(0:29:31) Audrow Nash

Hmm. Okay. And that would be used for a bunch of applications. Like taking photos of the earth or maybe sending internet places like starlink or whatever it might be this kind of thing.

(0:29:44) Dave Coleman

Yeah. It I guess I don't know. a ton about like, What? What I mean, yeah, assembling satellites for all the normal cases what you just said,

(0:29:54) Audrow Nash

Yep. Okay, and then what other applications were you saying? first base.

(0:30:00) Dave Coleman

So the doing more things without astronauts in space like on the space station so NASA has had this long standing program to have robots augment the astronauts time astronauts are wildly expensive to send up like every hour of their work day their their billable rate is very very high now because because of just all the training all the fuel Yeah,

(0:30:24) Audrow Nash

infrastructure would you I mean, I bet it's like comically large how expensive they are per hour what would you do about what it was but it's like $10,000 an hour or is it way more than that?

(0:30:36) Dave Coleman

I wish I had that on my head

(0:30:39) Audrow Nash

okay so what are what is it that you guys are doing? Are you looking for a number

(0:30:46) Dave Coleman

Yeah, I am looking for a number but i don't i don't see it so they're using move it right now on the this program called the Robonaut which is in the space station. And to like help do some more than maintenance tasks of the station. Like cleaning surfaces or like just flipping switches just like things that like maybe aren't worth the astronauts time

(0:31:07) Audrow Nash

would you describe Bravo not because it's a pretty wild looking thing.

(0:31:11) Dave Coleman

Yeah, it's it's very anthropomorphic, it like to look like a human, for the most part. So it's got fully dexterous fingers. It's got a head that can rotate. Look around, it's got like a chest just like a human. The legs however, are a bit creepier. They're think more like yes, spider legs, there's less. But because you're in zero G, you don't need legs that can walk like on Earth, you just need something to grab on to handrails. And so that part's a little creepier. And I am not the definitive source of the status of this program. But it's, I think it's a little bit winding down. And they're like, revisiting what kind of robots best for space. And, and so for the gateway, which is a new station being launched soon, to orbit the moon, one of the big needs, there is a inner vehicle robot that can take care of the station when it's unmanned, because they aren't planning on having a human there. All times of the year, I think closer to like four months of the year, it'll be manned. Eight months a year, the robots got to take care of the place. And so we're currently on a grant involved with that.

(0:32:18) Audrow Nash

Hmm, so it'll be the robo Robo robot on this down the gateway, which is orbiting the moon. And it will be taking care of a lot of operations by doing what a human might do if they were on it. Is it going to be tele operated? Or is it going to be largely autonomous or?

(0:32:38) Dave Coleman

Yeah, I love this topic. I think teleoperated to me, the definition of that is really drive by wire, you've got a joystick, or a mouse, or these days, a VR headset with two hand controllers. And you're telling the robot what to do like basically one to one. And you know, it's not exactly one to one because oftentimes, the robot arm may have a different kinematic morphology than a human. And so the way it maps out your command might be slightly different. But for the most part, you know, I'm talking about like Cartesian control, which is like effector control. But for the most part, you're just you tell what to do. That's totally up to me. Yeah, so

(0:33:17) Audrow Nash

you move like it would be like moving joysticks around, like the VR things. And the robot would do pretty much the same motion, but it might do slightly different things. Because its joints might be different than humans. Exactly. Okay.

(0:33:31) Dave Coleman

Yeah. And so that breaks down when you're on the moon. And particularly if you're going to Mars, which is the next stepping stone for the space program, as Ilan, lets us all know. Because latency and bandwidth issues, delays in time. So I think to the moon it's like three or four seconds, and to the Mars to Mars, it's like several minutes. Don't quote me on this, I don't memorize numbers, as you can tell. So that becomes really annoying and frustrating to control something with that amount of lag because you send a command and then you have to wait round trip for it to come back. So we're very interested technic, we're developing some technology that is a mix of fully autonomous and tele operated. So what's between those two extremes? And that's, yeah,

(0:34:22) Audrow Nash

that's what you're trying to figure out, or that's what we're proposing is what we're working on right now. So is it like a higher level thing where you can delegate tasks like at a task level, you say, Go select this, go do your thing here and it will kind of get over so if it's flip a switch will, the robot will go over there and position itself and perhaps flick it, but you've directed that it should do that? So it's like a higher level command and I think we're, Ah,

(0:34:51) Dave Coleman

that's, this goes under a number of names. But supervisor autonomy is maybe the most common one where you You are in such

(0:35:00) Audrow Nash

an oxymoron. But I get it. Yeah,

(0:35:03) Dave Coleman

it does. So you tell the robot, hey, here's a high level task. And then you're kind of observing the robot as it goes about doing it with the understanding that maybe the robots not quite smart enough to fully do the task. Because these are just hard problems. They're kind of unsolved so far, especially when you're in unstructured environments. I've had arguments about whether a space station like the ISS is unstructured or not, it's kind of an academic argument. Because I know my mind, these things are highly engineered things where they have exact computer models before they get launched. You look at videos on like YouTube of astronauts, giving you a tour of the Space Station. It's really striking how many bags and wires and new experiments are just floating in the middle of the way. And so actually, it's an extremely complex environment. So for a robot to be able to fully autonomously navigate this ever changing environment to do useful things. It Yeah, it's hard for today's algorithms. And so the AI going back to supervise autonomy, there's a human there to help it along when needed, but really minimizing how much intervention is needed from that from the human.

(0:36:14) Audrow Nash

Okay, that's really cool. Do you think that this will be a larger thing in robotics? Like the supervise autonomy? Do you think before, I mean, we see it with like, like, you can think of like Tesla and its self driving as being somewhat supervised autonomy, you sit there in the seat, and it drives for you until it cannot handle whatever's going on. And then it gives you back control. Do you think that this is going to be a larger trend before robots are like actually capable of exactly those things entirely?

(0:36:45) Dave Coleman

It's exactly what I think. So to Tesla versus whammo versus the other crews, self driving car companies, they all have slightly different approaches to how they're doing autonomous vehicles. But I think it's really interesting that Weibo and a number of the other ones, they've all kind of realized that in order to get to L five, autonomy, actually, you need a remote operator that occasionally can help this, like the robo taxi, get unstuck. So you know, a lot of the prototype or concept vehicles that I think are going to be out soon, they may or may not have a student steering wheel. And so if you think about a taxi, and maybe the seats are facing inward, there's no one there to take over. If the robot just the car really gets stuck. A Tesla has a steering wheel still, and they're not. They're a different player. So I would, they're a different category of autonomy in my opinion. But the idea with like way, Mo is that there's someone in a call center who maybe isn't watching every turn every step of the way. But when it's needed, they can be, you know, called in to give the robot a little more understanding, like, Hey, this is a construction site. We haven't quite seen this problem before. But this is what I want you to do. And the robot can then go, Okay, I'm going to maneuver around this. So that's proving out to be a necessary need in self driving cars. And I think it's also going to be more and more in need as we get robot arms outside of factories,

(0:38:08) Audrow Nash

huh? Yeah, I bet you're right. It's a very interesting thing. Because you can kind of augment where, like robots can do maybe 80% of it, or 70% of it. But then there's some part of it that people just need to be there for their judgment. And so we can buffer.

(0:38:29) Dave Coleman

I think you kind of hinted at it a second ago, like, eventually, we won't need this again, like we're gonna get the machine learning algorithms and the software and all of it so intelligent that the human will be then removed completely. But there's been some interesting research about like, how AI is harder than we think. And we've hit some critical limits of current machine learning techniques. And this is why the self driving car industry has been so delayed. And so I think it's going to be longer than we think like, you kind of have to get general intelligence, General AI in order to completely eliminate the human from it. And I'm not super stoked personally about even achieving that, because then you have the whole super intelligence thing. And what if it gets smarter than us? And

(0:39:13) Audrow Nash

Terminator? Yeah, yeah. All that stuff. Yeah, I don't know. It is all very interesting. And I do from my own, like, foray into machine learning and things I have trouble understanding how the metaphors can get to a more general intelligence at the moment,

(0:39:33) Dave Coleman

which is probably good for humanity. Maybe Yeah.

(0:39:37) Audrow Nash

But Alright, so then your, how long have you guys been on this NASA project?

(0:39:45) Dave Coleman

It's actually just the past year so not not even a year, just under a year. We've been involved in this. But we recently got our next phase of grant money with them and Exactly. So So that's we're doing this for NASA. But we're also starting to do this more and more with our other customers, using open source, move it, but also developing some higher level task planning proprietary aspects of it that we, because we've had some compasses customers asked us to build this kind of one off. And now we're making kind of a general solution for this overall supervisor autonomy problem. So I think that's going to really unlock a lot of applications in the next decade

(0:40:29) Audrow Nash

or two. Yeah, I've been I don't know, I've been thinking that this is one of the big problems on the front of robotics like, and I think you're hitting on the head and actually doing it, which is very cool. Okay, yeah, making it so that you can delegate at a higher level for robots what to do, and then have some sort of thing come in and help babies. It just it seems so unnecessary.

(0:40:58) Dave Coleman

I think there's an interesting, interesting pattern about past robot companies and how they've gotten around this and why maybe there isn't a solution out there for this exactly, is that typically, you choose an application, I'm going to solve like a robot, that was a good example here that picks widgets from a bin, that's just right, this is a big one, right now, a lot of companies going after like, a mixed parcel box. And you can make a bunch of assumptions and fixture stuff, and do just enough autonomy and AI and like computer vision to solve that problem really well and reliably. And those those solutions that keep coming to market. So like in the 70s, maybe the problem was, how do we spray paint a car really reliably? And they came up with exactly the tools for that. And so we keep pushing slowly, the envelope of a robot doing a very fixed thing for harder problems. But and there's there's a lot of benefits, like fully autonomous has a better return on investment in terms of like, you're totally eliminating the job versus like still requiring a human to be kind of assisting and sometimes, yep, I just think that's not going to go all the way to like a robot that can navigate through your your office, your home, like outside, in a wreck and just like assist or space, in particular, for any arbitrary problem you throw at it and have it be able to achieve it. So that's, I think this was me next?

(0:42:29) Audrow Nash

Yes, yeah, even the modern ones, it's like, they're somewhat like the picking the widgets out of the bin. The approaches seem to be somewhat flexible to variation. But it's like you throw a different bin or a new widget in, and then you have to reprogram everything and add new, like hacks or less general solutions to get it to work again.

(0:42:50) Dave Coleman

And maybe yet, maybe machine learning techniques, the difficulty of adding a new widget is gone way down. Maybe I think it's probably really easy to arbitrary object, train it really quickly. Yeah, but he still if you ask him to then, like, paint a car or do a different, totally different type. It's just not going to be Yeah.

(0:43:09) Audrow Nash

Yep. Yeah, that's a better way to phrase it. It's just if you choose a different task, it is unable to cope with the same system, you can run another program, and have that actually work, perhaps, but it has to be a specialized program in itself to do the different tasks. Yeah, I

(0:43:26) Dave Coleman

think the key there is you said you have to run another program. And so who isn't you the US? Yep. So so like really giving the operator a slew of a library of programs that are more general and not the robot do a lot of autonomy. But when the robot like gets stumped, there's a human there to like, push it along, like, Hey, here's a little hint. And then one really cool idea that we are not, you know, pursuing exactly yet. But like, you have all this human interfering or intervening helping the robot, and then that actually creates more training data to generate. So then you have this life cycle of labelled semantic data that you can use for future machine learning models. It's

(0:44:07) Audrow Nash

a virtuous cycle, it is totally more data more to train on, you can get more sophisticated and you get closer to actually solving your problem.

(0:44:16) Dave Coleman

I'm not a machine learning expert, though, so I won't I won't claim any of them

(0:44:19) Audrow Nash

either. Sure. Let's see. So that's really exciting. How do you so witness and I don't know if you can say, how are you doing? Like, do you have a simulation of what it's like to be on the on the gateway that space? The What do they call them? space stations? Yeah, the space station that's going around the moon, do you have a simulation of that? Are you working on like, don't like really specific problems for them? Are you working on general technologies that they're going to apply to that situation or I assume it's a mix

(0:44:55) Dave Coleman

more general technologies at this point, we hope to move towards more specific problems in the future. But move it in has been used a lot with the space station. And so in gazebo, another open source common library and ROS, they groups at NASA have created not full simulation space station, but a full simulation of a particular module, they included the railing that you can grab on to, and maybe some problems and so they're dead there hasn't worked that the Robonaut using the open motion planning library would move through this environment using its creepy spider legs and arms to like, pull a cargo bag and like open the Velcro, like some pretty cool tasks. That that's been worked on. And so some of the guys that are at our company have been involved in that program.

(0:45:47) Audrow Nash

So cool. Is there any heart? So I guess, does this already? Does the gateway already exist? Or is it already up there? That's a thing in the future.

(0:45:57) Dave Coleman

I believe it's not launched yet.

(0:46:00) Audrow Nash

So are there? I mean, I'm just wondering if you're like doing the sinusoidal flight in an airplane with Robo not that's floating for a few seconds and trying to do something or

(0:46:11) Dave Coleman

no, no, I mean, like it we're not, we're not that involved in this yet. Because that would be so cool. That would be

(0:46:21) Audrow Nash

so you could fight against gravity by doing that. It's so cool.

(0:46:27) Dave Coleman

I mean, we don't have gravity, it just makes all the manipulation problems, I think easier, because if they don't have any momentum, they just stay.

(0:46:36) Audrow Nash

It's true. Unless if they would be floating, it would probably be very difficult. And there's no forced precedent like they just move away from you if you put any force that they symmetric into good point. But easier if they're stationary, because you just but Hmm. So I think one thing that seems really exciting about this, to me is open source being used at NASA is amazing. Like that. I feel like that's a huge win for the open source community. Can you talk a bit about that?

(0:47:11) Dave Coleman

Yeah, they've they've been involved with ROS and funding Open Robotics for a while, I believe is my understanding. And I think that they see that this is a great way of tech transfer from what's the latest in academia. And they see it as a way to kind of standardize and hire people who already know, the technology and I ROS just kind of become a standard. And so NASA organizes that on various teams at NASA recognize it.

(0:47:42) Audrow Nash

The really interesting thing to me is when I think NASA I think of like incredibly robot, like they're really concerned about reliability. And it's interesting to me that they're going to open source for things that are important in their application, like controlling a robo not. Yeah, station, like that's really interesting to me.

(0:48:06) Dave Coleman

Yeah, good question. May is not a question. But I have a thought on that is the robot, their strategy, the group at Johnson Space Center, was to make the low level controls really heartened, and like, I'm not sure if I'd call it like flight certified but certified quality and rigorous such that it was guaranteed not to extract to exert the force to like punch into a space station, for example, like, they can't break the station, because that's extremely dangerous. And so if you aren't like if the underlying controllers and like the hardware interface, it like reaches the right certification level of safety and that it can't go beyond a certain velocity and sort of torque, then you can then think about the layer above that kind of the application layer with a more modern like rapid iteration software development techniques, that allows for more powerful compute and more like advanced methodologies. But sometimes it does, maybe sacrifice the rigor of a very simple system that is flight ready.

(0:49:10) Audrow Nash

I see what you mean. So it's nice to so we, you can have that base layer that NASA writes that make sure that the robots got not going to break anything. And then after that, you can put your more sophisticated layer on your layer that does higher level control. To do this,

(0:49:26) Dave Coleman

there's often a trade off of like, if you want to have these, like high resolution meshes that in like sensor data, where you're like, able to plan at higher level thinking and that task levels, the software gets very complex. And there's just like a lot of possibilities where sensor noise could cause errors, or just all the interacting interacting systems on different threads and cores, just can cause a little bit more unexpected timing. And so you definitely need to have like, an underlying like, super safe layer on top of all these advanced capabilities.

(0:49:57) Audrow Nash

Yep. So Now I see that we are, like run where I see that we're running out of time. And I know there's a few things that we want to talk about. So could you tell me a bit about premium? So move it premium? And how like, what is that? And what's the model?

(0:50:15) Dave Coleman

No, I just did. So we're calling into the studio. And that studio. Yeah. So I'm glad you made time for this. I'm surprised that we're running out of time this is flown by, but yeah, do it. So we've, we have been doing engineering services for the past four or five years. And it's been a lot of fun. But we think that the next stage of growth at picnic is going to evolve, and not only the next stage of growth, but the way that we can have a bigger impact towards growing the move at open source project is by also offering a premium version that that like larger companies would be really excited to pay for, because they get all these additional features. And, and so we tried to make sure not to come up with a premium feature that conflicts with the core feature set of move it because we don't want to be trying to make a better version of move it but instead, move it studio is the operator human loop supervisors autonomy aspect. That's right. Oh, that's Yeah, so. So like, we're gonna keep improving the underlying grasping library and planning libraries and kinematics and like, the handy calibration tools, improving all that for the overall community. But also, like we've we've made this dividing line. Some might say it's controversial, but there's been, I've studied a lot of like, open source business models of past successes. And it's like, very common, either you come up with a SAS play, where you're selling an internet service on top of your open source library, think WordPress, or you're coming up with a premium version of the software that says Software as a Service software as a service. Thank you. Yeah. Or you have this just like this license offer on top of it. So WordPress actually did that. Also, they have jetpack, like premium plugins, if you know WordPress at all. Or another example is like Apache Spark is a really popular open source library for I think, data analysis. I don't use it. But this company data bricks, made a product on top of it that has allowed Apache Spark to continue to grow also. So that's the the new one of our new strategies on top of our engineering services.

(0:52:25) Audrow Nash

Seems smart. The AI reminds me like another example. There's tailwind, which is a CSS framework, if I understand correctly. And they have tailwind UI, which is kind of it's the same breakdown that you're doing. So they have the open source one that gives you all the functionality. But then they have components. So if you want to make it more useful, and that I guess not exactly the same, but the like the supervised autonomy, like more structured interactions with what you're doing that kind of fit a template, they have literal templates for making HTML and CSS components. That's their point that they charge for.

(0:53:05) Dave Coleman

a pretty good example. I mean, for us, it's like, one way to think about it is that there's like a front end for moving now. But there is a lot more under the hood besides just a front end.

(0:53:16) Audrow Nash

Yep. Awesome. Very interesting. So what's the timeline on that? Like, where are we now in this?

(0:53:26) Dave Coleman

Great question. We have to reference customers currently that are that we've released an alpha two. And we're in talks a number of other early early customers. And so we plan to have the beta operating at ROS con, we should have it should be running at our booth, if I assume our viewers are familiar with the ROS conference in New Orleans this year. So that'll be October 2021. So hopefully, we'll have a pretty cool demo to show.

(0:53:54) Audrow Nash

I can't wait to see Will you guys be we have move at con or anything nearby at around the same time, like yeah, or whatever

(0:54:03) Dave Coleman

it might be. This year, we're not doing a standalone Lubich con event. I think our main motivation was we weren't 100% sure about COVID. And so we thought we would just focus our resources on the main ROS con event. So we are having we are having a workshop the day before the official ROS con event starts, but it's part of the ROS con workshops we're having. And that one is actually totally unrelated to supervise autonomy, we have been adding to open source move it some better features around mobile manipulation, which is the idea that you can synchronize the mobile base of your robot with the arms so that you can expand your workspace for for doing tasks. So like when you think about a human, you're trying to reach something across the table, you could I don't know get up and walk around the table or you could like reach across it but both of those modalities require using more than just your arm. So that that kind of functionality it's always kind of been a move it but we've greatly proved it. And so the workshop will be about how to use that on a Hello robot stretch platform, which is a startup in the Bay Area, making some pretty cool mobile manipulators. And by cool, I mean low cost. It looks like a stick figure, but it's also

(0:55:14) Audrow Nash

really, I'm pretty happy with it. Yeah, yeah, that is awesome.

(0:55:18) Dave Coleman

So they will be there with us. picnic will be kind of doing tutorials on how to use, move it for that application.

(0:55:26) Audrow Nash

Now I know we've said a lot about, like talking about the studio version, and you guys are working with NASA. But what's, what's next for picnic and move it outside of those, like where you guys had? would you think?

(0:55:42) Dave Coleman

Yeah, so mobile innovation is mobile manipulation is one of our big focus. Big like guiding winds is like, we need arms to be able to synchronize better with basis. And so continuing that thread. Another big thread for us is hybrid planning, which is the combination of global planners and local planners, the best of real time planning and global, really smart planning. They both have pros and cons. If you combine them, you get the best of both worlds. And so we've been adding more features on that the past year, we're working with Fraunhofer over in Europe on a demo of this technology. That's that's one of our big threads. I'm trying to think what else is on our roadmap, I can check real quick. Sure. We're still like I guess getting the last finishing touches on move it to I mean Move Move it to is basically a feature complete, we have a blog post recently just talking about

(0:56:41) Audrow Nash

what feature complete you mean up to move it one in the correct abilities,

(0:56:46) Dave Coleman

and move it to his reminder is just move it migrated to ROS 2. Except it's not just that because we've been adding, like all of our new features are not going to move it to so if you're on the move at one, we encourage you to upgrade, it has a lot of hassle, because you have to upgrade your ROS version as well. But I think more and more of the community's making the jump. And so when I said about feature complete parameters and launch files are still a little bit rough in like ROS 2 Foxy I think a lot of ethics and galactic. But now that it's fixed and galactic, we're having to finish up the moobot Setup Assistant, which is simply a user interface that lets you easily set up an arbitrary robot with movement. And we have to just make it work with a new style of launch files and configuration files. So you can do it without the Setup Assistant, it's just more work, we're trying to make it easier to do it.

(0:57:37) Audrow Nash

For sure. Yeah, hopefully that will get even easier with humble. The next one, which will be released, I don't know may of next year 2018. to hopefully, hopefully,

(0:57:48) Dave Coleman

another big one that I think we've momentum on is being able to remove into factors and different into factors, swap them out, live in move. And so this is a deep structural change. But there's a number of people at picnic working on this right now talking about the architecture, we'll see if we get merged in and the company doesn't balk at the changes. But when you're you already have changes it really like it breaks a lot of underlying data structures of how movement thinks about your robot. And so we're trying to restructure that. And that's been a common request from customers for years. So imagine, yeah, I'm excited about if that actually lands in the next version. And that actually, just to change topics Audrow that might end up being what we call move three is about API change of that magnitude we'd probably wrap up

(0:58:35) Audrow Nash

Oh, it's that significant of a change it's a huge change.

(0:58:39) Dave Coleman

And in general move it three is like we it's when we stopped thinking move it one to move it to so right now we're actually using get able to sync the code changes back forth them for import, and we're gonna draw a line in the sand and say, we're not gonna make a lot of huge breaking changes that sinkings over and that's to me kind of what movie three represents.

(0:59:01) Audrow Nash

Gotcha. Any, that's a it's an exciting reveal. Any idea when move it three will be a thing when you actually do that. Will it be like when this when this urdf? Or that the way of switching the robot to use a different gripper? Well that kind of be the catalyst to switch it or I honestly don't know

(0:59:25) Dave Coleman

the date. I'm not sure if the the main move it team at picnic knows the date yet. But it's just gonna be one day that we're like, okay, we're making huge breaking changes. We're now drawing a line here.

(0:59:39) Audrow Nash

Boom, move it three. Yeah, that'll be exciting. Do you think it'll be in the next year?

(0:59:43) Dave Coleman

I hope so. I really Yeah, definitely. I think so. Awesome.

(0:59:49) Audrow Nash

I exciting. I'm hesitant

(0:59:51) Dave Coleman

to make commitments because this is still open source software and funding major feature changes is always difficult. We don't have anyone fun. Doing this right now. So Nick uses a small, like a portion of our profit and like our extra revenue, to go towards these things. But if you know, let's say, changing and effectors live is something that you really want, and you're a company with funding, you know, you can just hire us, we will do it way faster and your timeline. So that's my thought like, we're here to do open source changes for hire.

(1:00:26) Audrow Nash

Now, so wrapping up, how does someone get involved with move it if they want to? Like how do they become a contributor? How should they get started,

(1:00:37) Dave Coleman

you use the word involved. I like that. So the big call to action button on the movement website says get involved. So you click that big blue button at the top right. And we recently revamped these pages to be really pretty and hopefully easy to follow to like guide beginners into the project. It used to be just a bunch of plain text on a web page. And so let's move on to @ROS.org Click the Get involved button. And I'm looking at right now the first two things it says is, you know, get in touch, like saying taps by vamos and discourse and discord, which is unfortunately, similar names. similar

(1:01:14) Audrow Nash

name. Yeah, yeah, that's me up to Yeah,

(1:01:16) Dave Coleman

we decided not to use slack. We thought discord was a better open source,

(1:01:20) Audrow Nash

like discord much more than slack. Yeah, totally agree with that decision? Yeah,

(1:01:24) Dave Coleman

we use slack at our company. But I think discord is great for open source. And then on the other side, it says, you know, start contributing day, you can click on that, it'll tell you how to, like, get involved with fixing issues, improving documentation and contributing good. Awesome,

(1:01:40) Audrow Nash

and how can how can people like any social media or anything that you want to? So if anyone wants to follow up and learn more, how can they?

(1:01:50) Dave Coleman

Yeah, we post every week on LinkedIn and Twitter under picnic robotics handle. So a lot of good content about the latest things and move it and a lot of like white papers, kind of the latest research on how we're thinking about doing hybrid planning or grasping or what have you. So we have a lot of our PhDs published on a frequent basis there. So I encourage you to follow us on either those platforms. That's been Nick robotics, and I have a Twitter account and waffle just panel. I don't post often though, but I always mean to I just don't.

(1:02:26) Audrow Nash

All right. Awesome. Thank you. I really enjoyed this. And Bye, everyone. Thanks. That's all we have for you today. If you liked this, feel free to subscribe, head over to the ROS discourse if you want to comment on the episode. Thank you to our founding sponsor, Open Robotics, and I'll see you in two weeks. Bye, everyone.