14. DARPA Subterranean Challenge, with Tim Chung

2022-02-22 · 1:24:51
challenge
darpa
government
hardware
mobility
networking
simulation
subt

In this episode, Audrow Nash speaks to Tim Chung, Program Manager in the Tactical Technology Office at the Defense Advanced Research Projects Agency (DARPA), on the DARPA Subterranean (SubT) Challenge. The SubT Challenge is a robotics challenge that aims to develop innovative technologies that would augment operations underground. In this conversation, they talk about the motivation of the SubT Challenge, its systems (hardware) and virtual (simulation) challenges, how the resulting technology has been shared with the world, on benchmarking the robotics behaviors in simulation, on the challenge of scoping the SubT Challenge, and on building the final environment for the systems challenge.


Links

Comments

Outline

  • 0:00:00 - Start
  • 0:01:07 - Introducing Tim and the SubT Challenge
  • 0:09:51 - Systems and Virtual competitions
  • 0:13:32 - Team makeup
  • 0:20:39 - Challenges of networking
  • 0:24:37 - Motivation for this challenge
  • 0:29:40 - Four tech areas of focus
  • 0:34:47 - Sharing the results
  • 0:38:47 - Virtual challenge
  • 0:51:37 - Applying benchmarking setup elsewhere
  • 0:54:53 - Final course for SubT System’s challenge
  • 1:06:03 - Scoping the SubT challenge
  • 1:17:50 - Role of DARPA in technology
  • 1:21:01 - What’s next for DARPA?
  • 1:23:19 - Links to share

Transcript

The transcript is for informational purposes and is not guaranteed to be correct.

(0:00:02) Audrow Nash

This is a conversation with Tim Chung, who is a program manager in the Tactical Defense Office at the Defense Advanced Research Projects Agency, also known as DARPA. We speak about the DARPA subterranean challenge or SubT, which is a robotics challenge that aims to develop innovative technologies to augment operations underground. I was excited about this interview, as I have heard a lot about the sub D challenge around the edges here at Open Robotics. But it wasn't one of the projects that I'm working on. So I was happy for this opportunity to learn more. From the interviewer, it seems like the SubT challenge was a great success, and that the challenge stimulated a lot of great work. You'll hear more from Tim during this interview. I also found it fascinating to talk about how DARPA picks their robotics challenges, and very importantly, how they scope the difficulty of these challenges. This is the sense think act Podcast. I'm Audrow. Nash. Thank you to our founding sponsor, Open Robotics. And now, here is my conversation with Tim Chung. I Tim, would you introduce yourself?

(0:01:09) Tim Chung

Sure. Hi. My name is Tim Chung. I'm a program manager at the Defense Advanced Research Projects Agency, or DARPA. And I'm the program manager for a couple of programs that involve robotics and autonomy.

(0:01:22) Audrow Nash

And so this interview is mostly going to be about the subterranean challenge. Would you tell me a bit about that?

(0:01:29) Tim Chung

Sure. DARPA subterranean challenge is all about inspiring folks from all around the world to discover revolutionary technologies for robotics, so that they can help in diverse underground settings for time critical missions.

(0:01:45) Audrow Nash

And so this competition has already occurred, correct? Yeah, what is right, what did it look like? Like, what kind of challenges were involved?

(0:01:57) Tim Chung

Yeah, so I like to describe the DARPA SubT challenge, as we call it, as kind of like this underground scavenger hunt slash triathlon for robots. So the general idea here that we've heard time and time again, from many of the folks like those first responders that need to go into really challenging environments is, you know, how come this one piece of component technology works in the, in the one tunnel or the or the one parking garage that I tested it in, but not in all the others that I have to take this equipment into? And so the SubT challenge was about how do we get robotic technologies to work across many different types of underground settings. And that includes human made tunnels like mines for mine search and rescue, urban underground. That's everything from subway tunnels to infrastructure, like storm drains, to parking garages, and even naturally occurring cave network. So for the cave rescue type settings. How would you get robots to be the best triathlete? Not just the best runner cyclist, but you really want that well rounded, robust, capable set of teams or robots that go do this. And the problem that we posed in the safety challenges, this underground scavenger hunt, DARPA goes into these environments, places, objects of interest, and it's up to these teams of robots to go out and find them and report them back to their human teammates.

(0:03:24) Audrow Nash

What would be an object of interest? I could say, what's an example?

(0:03:28) Tim Chung

Yeah, so we really intentionally set up a whole array of artifacts, as we like to call them. So you might think, for example, we put a survivor artifact. This is a mannequin. So it looks like a survivor. It has that visual appearance, but it also was a thermal mannequin. So it has a thermal signature that's human like, wow. And it even had a voice box. So you could have voice recordings or sound recordings in there. And so those robots going in to look for the survivor artifacts. Of course, most typically carry cameras, that kind of makes sense. But if you wanted to do a little bit better, or use other cues, maybe you're going to have a thermal camera on board, or a set of microphones. And that's just one example of the 10 types of artifacts that we placed in these courses. We had a cell phone in there that on the camera, probably really hard to find, but with its Wi Fi, in Hotspot mode on cool and Bluetooth and discoverable mode. Now you can start listening for RF signatures for signs of life. We even had something we call the gas artifacts. And this gas artifact clearly has no visual signature, but instead, in this case, we're, of course being saved. So we use co2 carbon dioxide. But you can imagine that, that these robots equipped with co2 sensors, gas sensors could be used to figure out where there's good air versus bad air or noxious chemicals that might endanger those responders going into these And then environments. And so this quite wide array of types of artifacts meant that you can't just send in one robot with one type of sensor, but in fact, likely benefit from teams or robots carrying an array of sensors to kind of cross cue and help each other out much like we do as humans, or you

(0:05:18) Audrow Nash

could I mean, so I could imagine one robot that has like every sensor possible, but maybe this isn't the best strategy. So then you said, teams of robots, what does a team of robots look like?

(0:05:31) Tim Chung

Yeah, well, one of the things that we have telegraphed early on in this whole competition, and this is the combination of multiple years of teams coming together to build and ultimately break some of their robots. But we said attrition is not only possible, but likely in these types of really challenging environment, terrain and other things. So for any team that came with that one glorious robot, loaded out with all of its sensors, and that's really putting all your eggs in one basket. And so, you know, thinking through what that means for your strategy for your, your competition, gameplay, if you will, really led many of our teams to think about diversifying their robot portfolio. And that not only means two types of sensors that go on different types of robots, but how they even use them in terms of like the positions on the team or the roles on the team. So all of that coming together, you had all sorts of robot types that can get into crawl into different spaces, maybe fly into high verticals or other open areas, some had legs that could walk upstairs or downstairs or traverse. Adding more rugged terrain, tracks, wheels, you name it. That's what the challenge is all about is to go and highlight that the problem is this, bring your best and brightest solutions to figure out what solves these problems.

(0:06:56) Audrow Nash

And I'd like to just think about the kind of make explicit to like, difficult robotics challenges that are involved in this competition. So like, whether it was like, Can you speak a bit about it, like I mentioned, like mud and reflective surfaces, and like all sorts of other things. But what were some of the challenges that were especially difficult for robots in this situation?

(0:07:20) Tim Chung

Yeah, you name some of them, for sure. I think having been crawled through a number of tunnels and urban settings and caves myself, you know, I can testify, I can testify that these are the types of challenge elements that made it hard for humans, let alone robots to navigate. So some of those you already mentioned, the types of surfaces, mud, moisture, puddles, water, you never know, we might be able to see your tell if the puddle depth is just an inch, and I'm going to be able to step on right through. But the reflectivity of that and the difficulty that robots might have, that makes it a really daunting kind of, you know, survival question of whether or not to step foot or roll into a potential puddle, or, you know, potential hole. Yeah, lighting, lighting is a major issue. So oftentimes he or only relying on the lighting you bring yourself. A lot of times there's dust or fog, or certainly in emergency settings, you can imagine things like smoke, including many of your sensors. So thinking through what kind of paradigms you would use to be able to navigate when you really can't see, again, in the context of cameras, so DARPA used smoke machines to go and learn from sections of these environments, so that we would intentionally stress test some of that. And then you start to learn, you know, things like what types of sensors are transparent to certain types of dust or a if I fly my multi rotors, my flying vehicles, they're going to kick up dust as they're flying through. Yeah, as was the case in these minds. And that can hose your senses as well. So self inflicted something problems as well. So verticality, I'll throw that one out. So the three dimensional nature's a really big player in these underground settings, because you're not just going out, but you're also going down or up. And so elevator shafts to stairwells of varying types to scrambling up older piles to wow, other opportunities to look in these wide open caverns that you might find a real real, you know, like a real diverse set of those kinds of challenge elements. And never knowing what's around that next corner is, is perhaps the most daunting challenge for most of these robots.

(0:09:49) Audrow Nash

Yeah, sure. How did you how were teams structured so I know there was the hardware and the software come or the systems and simulation components of this. But for the hardware one, how did you say a team can have some certain number? How do you organize what robots a team could have? And was there standardization?

(0:10:15) Tim Chung

Sure, well, as you mentioned, there's the systems competition and the virtual competition. And there were our our sister competitions, where one could build and design and break robots. And the other, you would do all of that in the cloud in a simulator. And so in the context of the parameters, which we defined the challenge, it was, quite frankly, quite loosely defined intentionally. In some cases. It's the you know, without constraining the design or solution space, sometimes that's where you find those really creative or solutions, innovative solutions, right. And so this is specifically for the hardware. The system, yeah, version of the competition now. Yeah, that's right. And so we would specify that the type of environment is a coal mine, and highlight that most robots, if you want to reach all parts of the course need to be able to fit through a one by one meter area. So something like a manhole cover, if you elected to bring much larger robots, then yeah, you still might be able to access parts of the course. But you're probably going to be leaving some points on the table, if you can't access some other segments of the course. And then beyond that, quite frankly, DARPA, you know, left it to the competitors imaginations, to identify not only what they might face, but what robot types or technologies would be advantageous for them to bring two to conquer that kind of an environment. So there was no limit on the number of robots, there was very little limit on the style of robots. There were some safety considerations on maybe exotic fuel types or what have you. But other than that, teams were able to use existing available platforms, and then really just keep on sensors or payloads, radios, or what have you on top. Or we did have teams that design robots from scratch, really thinking through the nuances of these underground environments in their build design power. So yeah, what, you know, clean sheet approach to trying to solve a really hard technical problem as a kind of a mechanism here for identifying interesting solutions that, you know, frankly, we might not have seen or thought about. And we too narrowly defined the problem.

(0:12:44) Audrow Nash

Gotcha. Was there like a budget constraint for the teams? Or was there was it like open budget?

(0:12:52) Tim Chung

You can kind of buy by and bring whatever you will for the systems competition? Yeah, it really was kind of open teams could and in fact, many didn't seek out external sponsorships or donated hardware. One of our teams found some repurpose some robots that he found on Craigslist. Wow, that's crazy, was able to turn those into ground robotic suite. Right. And so being really creative with both the the the resources that you have, and the solutions you want to bring to bear. Yeah, that's where a lot of that creativity came.

(0:13:30) Audrow Nash

Gotcha. Well, so what was the what did some of the teams look like? Like? How would you describe some of the robots maybe that show kind of the diversity of teams of robots that were entered?

(0:13:44) Tim Chung

Yeah. Well, what was really exciting from my perspective about this empty challenge is we had constructed it in kind of a iterative development manner. And so we started off, for example, in a coal mine, and we call it our tunnel circuit. So, of course, this human made tunnels, thinking about the mine rescue type scenarios, consulting with mine, rescue responders, and so forth. We went to a coal mine. And then approximately six months after that, we went to an unfinished nuclear power plant, and that was our urban circuit, and then working and exploring the possibility of kind of transforming a cave into a test course. But of course, we did have to cancel that due to COVID. But then, the final events allowed us to bring together all three of those types of environments together. And well, I'm sure we'll have a chance to talk about that here in a bit. But we fabricated this course that allowed teams to test their robots against all three of these environment types all in this magnificent course that that DARPA fabricated. And so, the reason I bring that up is because you saw there was no one form factor for a team, there was this continuous growth or evolution, if you will, who will have both the individual technologies, but also the team compositions. And so we saw a lot of robots that were wheeled, and we tried to see some flying robots in the tunnels circuit in the coal mines. And I can say that the wheeled robots did well, and the flying robots did not. And that was a great learning point. Whereas we pivot to the urban circuit. Now these are expansive think everything from warehouses to elevator shafts to other things that these drones really helped allow robots in Canada teams to go and explore places that just the ground robotically would never have been able to do. And then also saw the showcasing of legged robots, these quadruped Ed robots, so that they could go up and down stairs with a fair bit more ease than some of their wheeled counterparts. And so you kind of see this progression of both tailoring the fleets right who's going to be in your starting lineup might be a little bit shaped by the type of environment you're going to send these fleets of robots into. But by and large, that continued learning process culminated in the final event, where I'd say you saw a pretty healthy mix of air grounds, wheeled and legged. We even had marsupial robots, so ground robots carrying aerial robots on their on their back, piggybacking them, it's so cool to other parts of the course. And really, that, that diversity, the fact that there's not one and have common makeup of a team, I think is a hallmark of kind of the, you know, the the what we're going for in a challenge of being able to tease out all these various ways to attack this problem.

(0:16:56) Audrow Nash

Now, how do you, I guess, so you have these teams, these teams all have the hardware. They go and the competition is about finding the markers, or what we find them the points that artifact artifacts, artifacts, and also it was about what total distance covered or something like this with this, these were the two significant indicators of performance,

(0:17:25) Tim Chung

or Yeah, so work, we yeah, we, you know, we kept it as simple as you have a fixed amount of time, you have an hour. And you have to go and find these artifacts. And what we mean by that is you have to be able to return the location and the type of the artifact that you found, you know, you have to do that location positioning to within five meters. So the idea here is, yeah, so you go into these places, you have no GPS, I should have, you know, maybe stated that upfront. And so you're moving through these environments, and you know, it's pretty easy to get twisted and turned around and get lost. And so being able to report out where you found that survivor artifact, for example, to within five meters of global position. So that if you had to send in humans and would know, that location, it'd be pinpointed within, nominally within arm's reach. That's the kind of notional metrics for firefighters, you want to get them within arm's reach, because it's going to be smoke filled, and they need to be able to reach out, touch the thing that they got sent into to go find. So within five meters, you have to go find that out, and it's not sufficient to have found it. But you have to get to get that information back out. Right? You have to get it correct, of course, what type it is, but then you have to report it out. And if you successfully report out the physician to within that five meters and the correct time, then you get a point. And so the aims that find the most artifacts in that hour, are the ones that take home. Yeah. It to do the victory lap at

(0:19:02) Audrow Nash

the prize. And it was the prize was for the what was the prize for the competition?

(0:19:08) Tim Chung

Yeah, so we had cash prizes. This is one of the hallmarks of these DARPA Grand Challenges. And so for that systems competition, building the robots in such the top place to come $2 million. Wow, to their credit, that's awesome. And Wow, very good. Like publicity and everything to which is wonderful. Yeah. All right. It's the glory of winning the challenge for DARPA.

(0:19:33) Audrow Nash

So the robots going back to finding the artifacts. They would you're not providing the robots a map of the minds or of the environment beforehand. They're building it while they're going through it. Is

(0:19:45) Tim Chung

that correct? Yeah, that's absolutely correct. In fact, that's one of the hallmarks of how we constructed the SubT challenge. Neither robot nor human knew the man on these competitor teams has set foot inside this course and know anything really about And so it really is about exploring the unknown environment and then exploiting it to go and find those artifacts quickly and returning that

(0:20:09) Audrow Nash

information. Gotcha. So the robots are building a map while they're going in it. And this is why you can get twisted up. So if the robots building in an accurate map, say it's like a crew accumulating error as it's building the man, yeah, absolutely, then the map might be inaccurate, inaccurate. And then you could, you could say, I found something, I found it here. But because it found it, and the map is distorted, the location is wrong. And they would get no points because of a distorted map. Gotcha. That's right. Now, one thing that I imagined because it's or that I've seen discussed in watching some of the videos about SubT, one of the major challenges was networking. Can you talk a bit about this?

(0:20:51) Tim Chung

Yeah, sure. I think this is probably one of the understated types of problems, when you go into these underground environments, that no amount of testing in a lab or in even the basement of a university building, will really get you the feel of RF, the radio frequency of underground environments, because it's just so tightly coupled to everything from the thickness of the concrete, whether they're steel girders, or you know, rebar, if there's, you know, metallic, you know, metals in the in the soil, you know, all of those, if it rains that morning, it that plays a role. And so because of those dependencies Sue so hard to really think about communications in a robust way. Now, that was one of the four technology areas that the subsea challenge was deeply interested in. And so we did see teams really get in some cases, both surprised and confounded by how they're calm solutions were working under ground. So we saw a lot of different approaches, we had some teams come up with tethered robots that they would deploy from the outside, and tethers long as they could to maintain connectivity. And that robot, imagine, yeah, it's a long cable. I certainly twisted it will get snagged it would be a tangle hazard. And so there are many, many good reasons why not to use a tether, but some were able to make good use of that. There were others that used breadcrumbs. So now I'm carrying breadcrumb relay nodes on on the robot. And as they traverse either fixed amount of distance, and to like console and Gretel, they were at corner. Drop. Yeah, that's right. So they would drop.

(0:22:51) Audrow Nash

Hansel and Gretel. You're right.

(0:22:52) Tim Chung

Yeah. breadcrumb comms nodes. And that would allow them to relay their communications from robots robot or robots a base station. And then you had other models of comms, which was, you know, I'm gonna live without communications, I'm gonna go, I'm gonna go off the grid for a little while do my searching, scouting, mapping, finding. And then after either a set amount of time, or maybe I have really good data that I don't want to lose, I'm going to come back into columns, and then try to dump it, you know, to my teammate, or, or to the base station. So there are all sorts of columns technologies, but also come strategies that emerged as a way to deal with the difficulties of RF of communicating underground, is there.

(0:23:44) Audrow Nash

So I guess there's a lot of structural things that make it really hard to do radio frequencies in these like tunnels and things like this, like you turn a corner, and then the signal kind of just doesn't bounce down the hall back down back through the corner, and so you can't get it out. And so that's why there were all these difficult. This was this was such a difficulty. Right?

(0:24:07) Tim Chung

Yeah, yeah, exactly. And that's a good rule of thumb. You know, if you turn the corner, your signal goes down dramatically. Yep. And, you know, if you've ever had those kinds of problems, getting your cell service, working in a subway station, or, you know, something like that, you know, that RF just doesn't propagate as well underground and through the earth. And so that's something that the teams really had to wrestle with.

(0:24:37) Audrow Nash

Yeah, I'm curious about. So the motivation for this competition. Part of it, it's first responders and things like this. Can you talk a bit more about this? Sure.

(0:24:53) Tim Chung

Well, I'd say it was highly motivated at many of the robotics technologies We have as well as some of the component technologies, whether it's sensors or radios.

(0:25:05) Audrow Nash

You still there? Yep. Still, yeah, you're saying so many of the robotics technology seems to freeze.

(0:25:12) Tim Chung

Yeah, there's so many other robotics technologies that we had, you know, four years ago. You know, even the component technologies at radios or component sensors, they were, you know, frankly, limited in how robust they were how resilient, they were to that vast, very diverse set of underground settings. And so that's kind of know, from a technology point of view, we viewed this DARPA subterranean challenge as a way to spotlight and accelerate technology development, not just of these individual small component technologies, but in fact, how we think about integrating all of them into resilient systems of technologies. And I think that's kind of the technology motivation. But when we come back to it operationally, you know, if there is this period of time where an emergency response scenario is occurring, and the incident commanders marshaling our forces and trying to figure out what resources he has available, wouldn't it be great in that kind of roughly one to, say, four hour time period, you know, that, that golden hour, if you will, or she knows enough to send in humans, you know, at the at great risk, wouldn't it be great to send it robots, if we could do that, and pinpoint where the survivors are located, or trapped or otherwise, or even if it's what kind of equipment these first responders should be carrying in with them. That was kind of the key takeaway, if you can tell me that I should be bringing in shoring equipment, because the ceiling might collapse. And that will prolong and reduce risk to the responders that have to operate in that environment. It'd be great that the firefighter didn't have to go in there first, figure that out, come all the way back out, and then carry in all of that equipment. Same with re breathing equipments, and things of that nature. So just being able to we like to call it actionable situational awareness. Yeah, it's it's not just not just the map, which is really helpful. And in fact, you know, a lot of folks just kind of would be grateful just to even have a map. Yeah, but DARPA likes to go well beyond what is needed today, and anticipate what would be really beneficial in the future. And this notion of actionable situational awareness, is this idea of being able to provide the incident commander, who's about to send in her teams of humans, the best possible set of information like hazards, where key objects of interest are like those survivors, where you know, where blind corners might be, where she might lose comms, all of those points are very relevant. And that's the kind of information we really want it to be able to develop the technologies to be able to provide or in the long run, so that really served as the motivating operational setting. Yeah. And I think it's been exciting to see how this technology is mature to try to meet that Mark,

(0:28:18) Audrow Nash

for sure. To me, it seems like the two hardest parts of this competition for robotics teams would be first, the networking, and then also just the mobility, getting around in these environments, because I think, like, you can find different sensor pack, and maybe I'm mistaken in this. But I imagine you can find different sensors that are good in different environments. But they may have big, they may be bulky, they may have three power requirements, whatever it might be. But it seems like given the constraint of these tunnels, or these environments, where cons aren't really good, it's like, the big benefits of this challenge would be testing actual robot hardware in these environments to see how well it can do. And maybe like, what approaches might be a bit more robust. And then the second part would be okay, now that we have these robots moving around, how do we get them communicating their information out? Or connecting? So I mean, like, you could have kind of done robots that aren't doing much of the compute other than like, their localization. And whatever you're doing, they're sending information all the way back outside, for processing and kind of keeping a global state of the situation or whatever it would be. It seems like kind of the two areas that are pushed most by this competition.

(0:29:40) Tim Chung

So So you nailed them. And so in fact, we had four tech areas that we predominantly focused on the first. The first ones you've already mentioned, it's the networking. Mobility is the second one. You got to be able to get to these places, of course, but the other two areas that I'd also highlight are the autonomy the decision making of these robots in the face of significant uncertainty. And fourth and final was the perception piece. So how do you tie together? Not just like you said, the sensors, but what you do with that information? How do you fuse different sensing views, right? Those myriad sensors that you might be carrying? How do they provide you with a common picture of what is actually happening in this environment? And so all for autonomy, mobility, networking, and perception, we kind of really need to all come together, for sure. And we found the most successful teams were the ones that really thought about it holistically, and they didn't, again, rely solely on an autonomy solution to solve all of their networking problems, or didn't, you know, invest all of their dollars in the best radios and forego thinking about sensors in perception? So?

(0:30:59) Audrow Nash

Yeah, it's like four dimensions, really? And yeah, to be successful in this company?

(0:31:05) Tim Chung

I think that's a twist. Yeah, that's a true statement.

(0:31:09) Audrow Nash

I think I need to change the name of this podcast to have something about networking. Because something thinking acting are kind of They're analogous to three of those. But then, yeah, working systems, robots, and have fun.

(0:31:21) Tim Chung

And, and that's kind of another insight coming from this competition has been the role of teams, not just, you know, we might think of robots and humans as teammates, and you have robots and robots as teammates. And so just like you said, the name of this podcast, is this additional element where, in addition to what autonomy, perception and mobility I might have, as an individual player, how do I leverage the team so that I can cover all the bases appropriately, so that maybe I don't need, you know, it's like the relay races, I don't need to be the best cyclist. If I know, I'm a good runner, and my teammates are a good cyclist, oh, they can help augment what I might be deficient in. And so that, that kind of level of flexibility, kind of the, the design choices that that just explodes. The design space, really. And, you know, I think that's really one of the fascinating pieces. And so communication, communicating, teaming is that additional angle, additional perspective that the teams really needed to think about and not again, not just on transmitting bits, over wireless signals, but in fact, what information to which teammate when to send this information, stuff like that. And it turned out to be a really interesting dynamic to the way this research progressed.

(0:33:01) Audrow Nash

Yeah. And it's interesting, it kind of catches my attention, because you were saying triathlon earlier. And I didn't quite understand. But now it's like, there are mixed events that each mode of robot may be best at. And so it's like, if one robot say a flying robot is really good at going up elevator shafts, which might be a very important thing in this competition, then that robot is specialized for that, but it's not going to be good for exploring in general, because it might kick up a bunch of dust. And that could be prohibitive to its sensors, so then it can't do its job. Well, this kind of thing. It's a triathlon, because it has many events at which of which different modes of robot may do better. Yes, yeah.

(0:33:44) Tim Chung

And yeah, exactly right. And even calling it a triathlon kind of glosses over the fact that our urban environments, when you think urban, or your listeners are thinking about urban, they might have many different mental pictures of what urban underground might actually be. Some might be thinking about the sewer system, or others might be thinking about the metro rail or others are thinking about where they park their car, in the, you know, in the basement lot. And so that, that even within a single so called event, right, like, like running, it's like running in a very different very courts and trying to figure out how

(0:34:28) Audrow Nash

best suited you're running. You're running. Yeah, I don't know, upstairs, you're running. Like there's a ton of different environments. Exactly. It's like a Tough Mudder. In a sense.

(0:34:38) Tim Chung

That's not that sounds like a future DARPA program, right? Yeah.

(0:34:44) Audrow Nash

Gotcha. How did you? How do you disseminate the results of this? Because I imagine a lot of teams came up with clever solutions. There was probably some push in terms of algorithms or ways of doing things. How do they results get pushed and folded back into the public.

(0:35:05) Tim Chung

Yeah. So that's one of the things I'm probably most proud of from a kind of a community sense is that the subsea challenge competitors, you know, they are competitors, they're all vying for that top prize. But in the end, they were all wreck, you know, recognizing that this is a an opportunity as a community to leap ahead and change that, you know, potential trajectory of, of this research in robotics. And so many of these teams have gone to great lengths to open source their code, to share data extensively. In fact, one of the limitations of trying to, you know, with, with our off the shelf, image classifier, so object recognition, you can get off the shelf classifiers. But what do you need, you need a lot of training data, Well, turns out finding training data in underground environments and various lighting out there for that, that's just like, you know, I left it Yeah. And so, you know, finding that and aggregating that, and even even collecting such data is also very intensive, and time and labor. And so we found teams actually sharing their datasets by collecting so they had this kind of pool of data that they could train, and improve collectively. And I think that was kind of just one small example of how communal and collaborative that this community ended up being. And so all of that to say, even amidst helping each other, you know, whether it's lending, monitors, and screwdrivers all the way to sharing open source code and data, I'd say all of the members of the subsea community has really embraced this idea of, of sharing it, and, and helping this community grow. So you can go to their listing of repositories they've shared, not only in their publications, but also in their online videos. They've shared their research, they've shared their code share their data, many of the researchers have gone on to work on follow on projects that are leveraging the technologies that they developed in sub T's, in some case, working with former competitors, now, actual collaborators on these projects. And so a lot of that has kind of kind of made it out to the wild. And those are just the research product projects. There's also a pretty big effort amongst these teams to kind of take these pardons technologies, those that have been tested in the crucible of the darkness of T challenge, and turn them into commercial products. And so we've seen startup companies spin out from these competitor teams, productization, of even some of these components, technologies, you can in fact, so go and buy these sensor integrated sensor packages, that you can slap on the hood of your car, or carry in your backpack, and get a nominal kind of imaging, mapping capabilities. That's now as a product. And a lot of that really came out from the SubT challenge. And so again, being able to make that impact, not just the research community with code, but in fact really turning us around and impacting the, you know, safety community in the security community in the mining industry, in the construction site industry. And you know, all those kinds of folks that can really benefit from these, even the component technologies have already started to see that impact from from getting the research, but also the technology out into the wild. It's been awesome. For sure. Yeah, that is super cool.

(0:38:45) Audrow Nash

How did the so another part of this that we haven't really talked about? Is the simulation or the virtual competition? Would you tell me a bit about this and how it relates?

(0:38:57) Tim Chung

Yeah, so we designed the virtual competition of the SubT challenge to be an opportunity for us to kind of explore some of these What if cases, and that's one of the incredible values of using virtual environments to study advanced technologies. And so whereas in the system's competitions, we might take over a coal mine or an unfinished nuclear power plant, you only going to get one such course you know, one such environment. And despite having a lot of variability within that, one course, there might be a whole lot of other things we might want to study in the virtual environment gave us those opportunities to do that. To give you an example, you know, if you wanted to, you could use in the virtual competition, we enabled this ability to create your own world, you could procedurally generate your tunnel like environments or procedurally generate your own cave like environment, specify some parameters it would be Have you long and narrow? Or, you know, highly vertical? Or, you know, kind of

(0:40:05) Audrow Nash

parameterised? Yeah.

(0:40:06) Tim Chung

Yeah. And and that gives teams on competing in the virtual competition, this opportunity to test against a wide range of worlds far greater than one would find yet a single test site to kind of avoid overfitting to the problem to avoid studying to that one specific test. And so this virtual competition really kind of embraced that we, instead of testing against one virtual world, in many cases, we're testing against half a dozen upwards of eight worlds, in some cases to see how well a given fixed solution would work against a varying set of environments. So the way this works, we invested heavily in a next gen simulation capability, ignition, Zeebo, our SubT simulator. And that's the simulation

(0:40:59) Audrow Nash

environment made by Open Robotics where I'm working. That's right.

(0:41:03) Tim Chung

Yeah, that's right. So Open Robotics, provided that virtual environment capability and advanced the state of robotics simulation, I'd say. And in that simulator, we're able to run that in the cloud using our cloud based simulation infrastructure that we also developed. And so now teams are able to develop containerized solutions. So bundle their robotic software, if you will, and upload it to the cloud and run their software against our simulation environments, not knowing what the simulated world would look like is, of course, we are managing all that in the cloud.

(0:41:41) Audrow Nash

Yeah. You have your tests, basically, which this is how well does it do in the cloud environment? Okay.

(0:41:48) Tim Chung

Yeah. And so the other fun part of this is we constructed what we call the sub t tech repo. Think of this as like your, you know, your storefront, your web based storefront, you can go and add robots to cart. And what I'm doing here is I'll give each competitor 1000 SubT credits. Yep. And you go to this storefront, this tech repo, and each of these robots that we have virtual models of cost you some number of sub D credits, yeah, up to that kind of salary cap tees of, you know, NAC league analogy here, you get to build, mix and match the types of robots that you want to add to your team subject to this budget of 1000s of t credits. And these different robots are all different sorts, they have different sensor payloads, they might have, again, wheeled or flying or kind of all the different different modalities you might explore. And so it boils down down to these teams, thinking about where they want to spend their 70 credits, and the number of robots they want to add to their team. And then designing their autonomy and perception solutions to best match the virtual robot teams that they're constructing. And so in this way, you really get to kind of explore what are the best composition of teams? What if, given this cost constraint? He asked earlier? If they're, you know, budget limitations for

(0:43:17) Audrow Nash

that? Yeah, cuz I heard about the virtual, the budget. Yeah, the virtual one. Yeah. So I was wondering if there was something similar in the hardware and how that would work, because it's quite complex, if they're building hardware was my thing. Right,

(0:43:31) Tim Chung

right. And so we wanted to abstract away some of the, you know, constraints of having to embody some of the software based side of things, and free up that develop, you know, the development to explore some of these other scenarios. The other really big difference between the systems and the virtual competition, was that in the system's competition, these robots have to report out where these artifacts are located. And they can coordinate with a human supervisor, a single human, to act as a teammate amongst a fleet of robots. And that human supervisor, is typically the one that's forwarding the artifact reports to DARPA to get scored. And so you now have this partnership, you can have the human supervisor, kind of interact with the teams of robots and have the robots go take a closer look at things or maybe show a an image or review and high man's like, go over there probably or whatever. That's right. Yeah. And they did. And that was really great partnership. But in the virtual competition. These robots are fully autonomous. You have uploaded your software to the cloud. It's one set of software that has to get run against all of these different types of virtual environments. And there's no human back and forth and so really being able to explore complete autonomy of The solution where you don't get the chance to, you know, back your robot out manually, even if you do it, you know, that gave us again, studying what this what if scenario in the future where you didn't have the luxury of having a human supervisor, or the levels of autonomy got to the point where you could deploy these robots to good effect in these really diverse environments. So now we have, you know, brackets, you have the system competition, highlighting where we are, we push the state of the art in applied realized technologies. And we also have the the solution space of what can be done in the completely autonomous regime where it's software only. And now we can quantifiably say how far are we from complete autonomy, autonomous solutions, or we can quantify it to some degree, the value of the human teammate. And so the virtual consecration is benchmarking ability. Right now we have comparative analysis between systems and virtual. And we did this all throughout the whole competition, where we had tunnel circuit in both systems, virtual, urban in both systems and virtual in cave, we couldn't do the system's competition. But man did the virtual competition really shine through in the in the face of the global lockdown, where now we had teams flocking to the virtual competition, because they still were thirsty to develop their technologies. And so we had numerous systems, competitors, crossover into the virtual competition, in fact, and carry that through to really good effect. So it was a lot of fun on the virtual side to kind of help us imagine what future technologies could be while also being closely coupled to where the current state of the art of the technology was, as well.

(0:46:54) Audrow Nash

Yeah, I find it to me the most exciting part about that, as you said, with the benchmarking, it's that so like, I'm thinking about, like the advancement of technology and say, like the computer image or computer vision space, like we established image net, and it's a benchmark in a sense, so that people can more fairly compare what they're how their algorithm performs. And so what you see is people keep trying it, and then the performance keeps going up. And robotics, especially on like a behavioral level, has not really anything similar to my knowledge, except for what you're describing here, which is really cool, where you can have some sort of, so you have your challenge. And you have a way that people can try by basically upload their algorithm to it. And it runs it given their robots and and all of this, to see the performance, and you can see the performance over time. That's really, really interesting and very exciting to me.

(0:47:56) Tim Chung

Yeah, totally with you. I think this idea of being able to set those benchmarks allows for us as a community to see how well we're doing right and see that progress. And it's exciting to see that progress. And to be able to do that in near real time with this side by side systems and virtual competition, I think, you know, was really, really cool. And I'll highlight two more things. One is we took systems robots, so robots from the system's competition, and went ahead and scan them, generate 3d models of them. So controller code that were contributed by the competitor teams, and validated it against their sensor spec sheets and battery durations. And you know, quite laboriously. But now, in that SubT tech repo, this is these virtual robot models that are digital twins of their systems, real world robots. Now live in the tech repo. And now virtual competitors, they might not have had the resources to build their own robots, but they can actually go and use a systems embedded robot models in their virtual fleet.

(0:49:10) Audrow Nash

Yeah, and that's very cool. If anyone had an especially good idea for like, how to, like a morphologie. That's really, really good. And now, I can play with that more as a community. Right off. Yeah,

(0:49:22) Tim Chung

exactly. Right. So I'm going to pull the fantasy league analogy. Again, you know, if it's a virtual competition, gave you the flexibility to mix and match across the league. With a systems competition was kind of like the, you know, like the NFL, right? You kind of have to pick your roster at the beginning of the season, you kind of play the players that are on your roster, because it really is too difficult to go back and do clean sheet design after each one of these competition events to start from scratch, so you're kind of trying to do that systems thinking for the systems teams holistically. But man in virtual you get to mix and match. Play around, see, if you want to be an all ground team or an all air team or, you know, beg and borrow from different, try different styles of robots, you can do that. And many of the competitors were able to do that exploration. And there's still, of course, significant exploration left to do. And then also highlight that there's a virtual testbed, the SubT virtual testbed, which encompasses the simulator, and the tech repo, all of that is publicly available and open source. And so, okay, yeah, you can still go, despite the competition being over, we know that the value to the community is so important that DARPA is keeping up the testbed open for a period to come, you can still go to SubT challenge that world and learn how to how to develop a SubT solution and upload it to the cloud and run it, you know, and get free free reps in for Yeah. For your simulate, you know, your simulation solutions. And so I think, yeah, the availability is another testament to, you know, how we really cherish the the community really are interested in fostering the growth of the community, because that's at the end of the day, what's going to turn around and develop those technologies that are, you know, first responders and warfighters and others are going to be able to make use of or precisely the technologies that these people that the community develops and matures?

(0:51:37) Audrow Nash

Yes, for sure. Do you imagine that the the framework that you guys have developed which hosts this the high level testing and benchmarking of this challenge to imagine that this can be applied to other different like, basically, the whole infrastructure that you've built? Do you think it can be applied to other robotics challenges, say robots like moving around in hospital and be inefficient or any, any rope, any tasks that robots might be very good for? To have this provide like a standard benchmark, so you can look at it over time, and compare more fairly, the algorithms and approaches?

(0:52:18) Tim Chung

Yeah, hands down. I'm a, I'm a believer in this type of a model of being able to not only test at scale, but also open up the innovation space, by lowering the barriers to entry. And simulation is a great way, you know, to avoid having to lay out capital expenses to build a fleet of robot and oh, by the way, have access to a hospital or the underground mines or the moon or wherever your setting might be. So absolutely think this back end infrastructure absolutely critical, as a, as a method as a set of tools to, to help out. And in fact, I'm pleased to say that, in fact, we are seeing this precise infrastructure being used in other robotics development settings already. So the impact is already near and dear. And stay tuned to see some of those competition events making use of this kind of infrastructure.

(0:53:22) Audrow Nash

And is that infrastructure? Is that in the SubT tech repo that you've been mentioning? Or is it Yeah, somewhere

(0:53:27) Tim Chung

else? Yeah. Yeah. So the tech repo is the the source code, if you will, of all, the majority of everything we talked about is already available, you can go to get a, I believe, under OSR. F, and see the SubT project there. You know, nearly everything that we've talked about is already located there. So there are a handful of resources under github.com/sub T challenge, and you'll be able to find a number of resources there as well.

(0:53:59) Audrow Nash

Awesome. I didn't know because I haven't. The sub T has never been one of my projects. So I'm not that aware of what where everything lives in this kind of thing. It's awesome to hear that it already is all online and people can access it. And that people are also using this infrastructure to benchmark other tests in simulation. I think that's really, really exciting.

(0:54:19) Tim Chung

Yeah, it's gonna be great when kind of the impact to not just the subterranean robotics community, which admittedly is just one sliver of the community. But being able to open this up, you suggested kind of hospital environments or maritime settings or other types of environments where we really want to explore more broadly, I do think that, you know, this investment will pay significant dividends for many, many years to come for the robot's community.

(0:54:51) Audrow Nash

Thank you, right. Let's see. So we are running out of time, and by that I mean, like 20 more minutes or so and I have a lot of things that I want to talk About in this, but you you mentioned the final events and that we do you'd like to talk about that and kind of how you fabricated this big environment? Would you tell me a little bit more?

(0:55:11) Tim Chung

Yeah. So, you know, I think one of the visions for the subsea challenge from the get go was to have teams of robots, that and, you know, unhindered by the fact that you might face all three of these environments. Now, the challenge, and I've crossed crisscross the country to try to find a place that has tunnels, urban caves, all next to one another, all co located all connected in a meaningful way, if any of your listeners have mine, please, please let us know. So instead, as DARPA intends to do is, you know, because we are always seeking those breakthroughs. We transformed a limestone cavern and built a one of a kind course, kind of think, almost like Hollywood set design with the level of realism that, you know, is, you know, very compelling for both human and robot, to the degree where we're getting inspiration from caves and mines and sites that we've seen, we replicated New York City metro station, all the way to steam tunnels, and old rustic abandoned mines to large caverns that are either show caverns that you might go to as a tourist to the, you know, the raw caverns or the untouched ones that have a very different feel as well. And so we were able to take all of those different sites that we had been to or found to be really compelling, and fabricated, we built that we had warehouse structures connected to a metro station connected to a featureless hallway. And that's just a fraction of the urban setting, we had small places where you had to crawl on your hands and knees often over the different segments and places where I'm certain many dozens of times, if I hadn't had a helmet on, if you walk away with a lot of bumps and bruises, you know, just trying to build a place where we can test and push the boundaries of autonomy, mobility, networking, and perception was a phenomenal experience to, to do that. And in having to build that course, going back to where it all started with all of our stakeholders, all the end users of people who this was going to who we would have to have them believe that these environments were realistic enough, right? So we did go back and say, Hey, what are the pieces that you need to see, to be able to say that you believe this technology is relevant to you. And we were able to do that to good effect. And whether and since we were building it ourselves, we in fact, had different segments of the course take on different personalities. And so in total, even though we had three subdomains, the tunnel urban cave, we in fact had, like 60 different segments, kind of like going on a on almost a Disney ride through different parts of the subterranean world, is what we were able to pose to these robots now facing all of that difficulty, really, really, really allowed for testing the diversity. And we went to great lengths, I'd say to hone that realism. We talked earlier about communications, and networking. But I'll tell you, we went through various types of RF shielding and pains and other things baked into the walls of, of, of this course, to be able to replicate the kind of the RF propagation in underground environments. And so I'm pleased to say that I think we did a good job of trying to replicate not just the look and feel but the RF, the radio frequency, the wireless transmission all the way to you know, the little things like amount of water moistures, slick surfaces, Rubble, gravel, you know, all that kind of stuff. Really, really proud of this one of a kind opportunity to, you know, raise the bar for the robotics community writ large, for sure. Yeah, it

(0:59:49) Audrow Nash

sounds like such a great test environment that's so funny that you like the shielding and things so you could get the RF properties you wanted in the environment. Amazing. So I assume that will be used In future competitions or how so actually How did it go? In the finals? You use this environment, correct? Yeah, that's right. How did it go? How was it?

(1:00:10) Tim Chung

Yeah, I think I think it was really great. I think, you know, one of the benchmarks for me as the DARPA Program Manager here is, was it hard enough, right? Does it push the boundaries, it needs to be hard, so that not everyone gets 100. But we don't want it to be impossibly hard, because then we're not going to know where the boundaries are, where the envelope is. And so being able to design this course, in a way that had the level of difficulty variants, that allowed some pieces of technology to really be showcased, and others, you know, you found where their limitations were, you know, that was really great about the the environment. I'll say, also, from the kind of the government's perspective, the DARPA team, we are in the command post, as you know, this is not something that our competitors get a really good glimpse of, because you're all behind the scenes. But in our command post, we had instrumented this course, with everything from cameras, of course, so we could see where robots might be. But we had things like motion detectors, we had triggers, all of our artifacts were instrumented so we could make sure that they're consistently Halloween house. Yeah. Yeah, that's right. Yeah. And so we could orchestrate kind of like movie style, movies and style, like, and so we had dynamic obstacles. And so these would be triggered when robots entered a particular area, and then section of the course would collapse behind them now blocking their way home, they would need to find a new way they would not be able to follow their that's amazing, around the home. And so that was built into the course as well. actuated, from, you know, from these types of signals that we had instrumented this course with, and so does a Hollywood set up. Yeah, in many ways it was. And it was always done with a technology objective in mind. And that's the cool part about it is and showing this collapse was something that was very relevant to many of our, for example, mine rescue personnel, because that's one of the hard things is debris collapses, or debris, integrity checks of the environment. And so collapsing, part of this was near and dear to them. But for us, we are interested in testing the autonomy, the ability to recognize that your map has changed. And then your way home is no longer viable, you got to find another way home. And so there was a really fun way, a really tight way of coupling, the operational kind of need and insight with the technology that we were trying to push in this one, of course, was explicitly designed, you know, from from CAD drawing all the way to graphic artists painting inside the course. So to be able to deliver that type of tight coupling between operational and technical objectives.

(1:03:20) Audrow Nash

Gosh, that sounds amazing. Do you have is there like a YouTube of like a walkthrough of this course? I haven't seen it.

(1:03:27) Tim Chung

Yeah, there are. So you can go to DARPA TV, that's the YouTube channel. And all of our videos have been placed there. There are walkthroughs of every course that we DARPA has transformed into a competition course. So you can go and do a walkthrough of Hardrock, like a gold mine, coal mine, that nuclear power plant, and then of course, our grand finale, final event course there multiple walkthrough videos. And then this might be a good opportunity to share that DARPA also collected very high resolution, high accuracy datasets of these environments, to the point where we could dance, of course, because we're also scoring the artifact position reports. So we were down at the, you know, millimeter scale of accuracy. And we've released that as a public data set as well. So if you were unable to test in the final event course, well, you can actually go to the SubT tech repo on the virtual side and get a mesh 3d rendering a 3d version of the final event world and, and see what it looks like for your robots as well. And fun fact here, one of the final event, virtual competition worlds was in fact, a 3d virtual world of the system's competition final event world. So kind of bringing everything full circle Now you get this chance to test how the system's competitors were testing in this, again, unique one time world. And at the same time, we're running virtual competitors do the same as that course, having developed this 3d model of the course, with all the, all the pain, the same pain points as the real robots had to face, but not in a virtual domain. And so all of that's been publicly released, we can see the video, but if you want the point cloud, you can download the Point Cloud yourself and work with it, or the mesh, 3d virtual mesh file is available. So you can, you know, do put in your own simulators, if you wish, or loaded up in the SubT simulator and, and drive some robots around.

(1:05:44) Audrow Nash

So cool. What a great public offering. That's so it's so neat, that people have access to all of this. Let's see. So I have a few things that I want to talk about. And we have like, really 10 minutes left or so would it be all right? If we run a little longer is it? Yeah, absolutely. Okay. So one thing that's very interesting to me, and I really, really wanted to talk about this. So I'm very happy. We can run a little long. But can you tell it's like one of the huge challenges, I believe in setting this whole thing up, which we talked about before, is figuring out how to scope these challenges, because you want to make it so it's not like like, as you were just saying, you don't want all the teams to fail, because it's too hard. And then it drives no innovation, because no one can do anything. But you don't want to make it too easy. Also, can you just talk a bit about scoping the entire SubT challenge? Yeah. Yeah, I think process of that.

(1:06:45) Tim Chung

Yeah, you know, I think it boils back down to the really doing our homework and kind of getting, I think that's what DARPA is all about is not only gauging where the current state of technology is, and not only kind of trying to understand the trend lines of where technology might be gradually going, but understanding the levers where if we were to exercise, some, you know, nudge here, hope they're tested that, that we would actually bend the curve of the trajectory, and accelerate where technology is gonna go. I think that's what DARPA is, a foundational mission is to try to create a kind of technological surprise. And so by doing our homework, both on terms of where the technology is, but also what the end users really needed, kind of doing that, that you know, where we want to be, right. And we want to skate to that, to where the puck is going to be, combined with knowing where we were, you know, I think it really was a laborious process to tune and tweak the evolution of the challenge to be able to arrive, as well as we have to this end result of having advanced as much as we had. And so I'll tell a story here, at one of our very early events, we transformed the a gold mine into a test course, and this was not for competition, it was really just a test. It was an integration test, really. So we invited competitors if they wanted to, to come out and test and we had done it up much like a competition course. And this was really to give teams a first look at what DARPA had in store for them. And so when we conducted this, teams went in, and you can safely say they got their butts kicked, right, I think they realize that, you know, it's a little things that we'll get to, in addition to the big things that they were worried about. And one of the things that it became a tradition afterwards. But we took all of our competitors on a walking tour of the entirety of the course that we had laid out that DARPA had laid out. And whereas most teams maybe made it, I'll be generous and say, the best team made it 10% into this first test course. Yeah. When we went on the walking tour, and showed them how far and how expansive this course really was. You could very easily imagine that there are those out there who would say, Man, this is impossible. What is DARPA thinking? They'll never, you know, no one can ever accomplish this. And yes, they did say that to some degree, like this is impossible, man. But no, what really kind of stuck with me is that the prevailing sentiment after that walk about two are in that very first course, was not that this can't be done, but that we didn't do it. But man, somebody believes in us that, you know, DARPA must believe that this is possible because otherwise why would they have crawled up that One ladder in the far reaches of his mind, why would they have kind of probably twisted ankles to climb up to that location, replace that cell phone company, all the, you know, 80 pound man, again, three kilometers into this course, if they didn't think that this was possible and that spark, I think for them of setting the bar really high, you know, and saying yes, we think that in due time you accelerate the technology to get to this point, you will be able to conquer a good portion of this course in the future. And that trajectory, I think, has continued to manifest throughout the challenge. And so by the time they got to their final event, you have many teams on record saying that you asked if you ask them three years ago, or four years ago, that they could have conquered or handled any part of this final event course they'd say, Absolutely not. For the fact that we're here, in truth be told, by the way, you know, I don't think we talked about it. DARPA placed a fixed number of artifacts in these courses. And so for the final event, we placed 40 artifacts in nearly half a mile of underground terrain that we had built. And the highest scoring teams scored 23 points, 23 artifacts. So you might say to him, hey, that's kind of a failing grade, right? That's kind of like, well, 50% is just over 50%. And you're saying you did a great job? Well, yeah, that's a phenomenal job, given the difficulty level that we know is where we want to be right. And so again, it's really been showcasing that the technology still has a lot of headroom to grow. But to be able to quantify the level of impacts, right, we're able to cover this massive environment in about an hour with robots that, frankly, took an order of magnitude, even two orders of magnitude longer with my human team to go gather that high precision data, you know, it took them a fair bit longer to be able to go and do what robots are doing in under an hour under duress, never having seen the courts before. That's a market improvement that we can point to and say, four years ago, we might not have been able to do it, if not, for this high bar that DARPA sets. And I think calibration process is a iterative thing that we've fine tuned, I think, here at DARPA,

(1:12:31) Audrow Nash

I think it would be more worrying if people got all 40 already. That would mean the challenge was not hard enough today, because it's a benchmark. I mean, if you look at, again, going back to like image net, it'd be like 50% level when they were starting, and then it's like a free thing up 70% Now, maybe, or 80, or whatever it is, but it starts pretty low, and then you have plenty of room to improve, because you have a hard challenge and getting all of it, especially in that like limited period of time. Right? Like that's a real challenge. Yeah, it's very interesting.

(1:13:03) Tim Chung

Yeah, wholly agree. And think that there's, you know, really promising work to get to be done and to be a

(1:13:10) Audrow Nash

component, for sure. And you were mentioning, mentioning, DARPA has this very iterative process for scoping, like difficulty of these things. Can you talk a bit about that, because that's really interesting to me.

(1:13:23) Tim Chung

Sure. And so specifically, for the sub D challenge, we had designed it, so that we would have those opportunities to, to to learn as we go. So by virtue of how we constructed the SubT challenge, we held that first event where teams first saw what DARPA had in store, but then, about six months after there was the tunnel circuit, where we conducted our first event focusing on tunnel environments, in this case, a coal mine. And then we broke that up into again, six months after an urban setting. And so the teams knew that if they were going to be competitive at the final event, where they were anticipating that all three of these environments would all get mashed together, then they should be thinking about their designs from the get go, that can survive, quite frankly, tunnel urban cave, all the way to the very ends. And so we saw teams to kind of internalize this, this need for resilience in their early kind of design scoping. And so they went to the tunnel and man, Were there a lot of really hard lessons learned at the tunnel a bit, but what you saw was a learning not just by team, but across teams. An example is that a using a a sensor on the outside to be able to help correct as robots still within line of sight of the mouth of the tunnel. You know, we had a team that was using that and so they could extend how far their accuracy was, you know, subject to drift they could they could extend And how far they were by using this total station sensor to, to correct for any drift. And yeah, that's, that's clever and so even even at the tunnel circuit he saw teams going out and trying to find and they found a local university and borrowed one of these little stations and tried to use it and by the Urban circuit, you had more teams using such a technology. Same goes for legged robots say, legged robots didn't do so well at the tunnel circuit. But at the Urban circuit where there were stairs and curbs and other things, man, those legged robots really shine. And so at the final event, you saw a lot of legged robots coming to the fore. And so that's what I mean, by the iteration, there was multiple opportunities for these teams to come field go through the the rigor of a competition events, but that wasn't the end of it, they would have to go home, lick their wounds, and then come back and do it again. And then, of course, break some more robots and learn what it means to have to operate in the cold, called nuclear power plant or the human limestone cavern or, you know, all of those kinds of things. We baked in that iteration and the kinds of opportunity to learn quickly and often into the stuff the challenge. And I think, as a roboticist, myself, I think that's where you learn the most by thinking and then doing right thinking of doing breaking, learning, and then thinking and doing it again. And Field Robotics, I think is, you know, needs that and sometimes trying to do robotics in the field, especially at the scales that the sub the challenge was interested in really hard, logistically, you know, it takes a lot of time, a lot of people power to get to transport robots and get things set up. But, you know, sub The challenge really gave all these teams, not only the excuse, but the incentive to have to go and test in the wild. And many of the teams, especially the top performing teams, really took that to heart found, you know, and partner with local caver clubs, local gaming, clubs to go and gain access to their caves and practice, or, you know, all of those kinds of opportunities, I think, came about, with many, many lessons learned a lot, a lot of a lot of broken robots, but a whole cadre of field roboticists and the next generation, I'd say, Field Robotics is coming out, having been battle tested in the subsea challenge.

(1:17:49) Audrow Nash

That's really cool. So that makes me curious about kind of the long term role of DARPA in all of this. So you're, you're mentioning like one thing. Very interesting, as you're mentioning, training the next generation of Field Robotics, roboticists, can you just talk a bit more about the role of DARPA? Yeah, as you see kind of on a bigger picture?

(1:18:13) Tim Chung

Sure. Well, you know, at its heart, DARPA is always going to be about bringing about technological surprise, and those breakthrough technologies that will have broad impact overall, and whether that's deep investments that will lead to breakthroughs when we need them. And an example of that is that mRNA vaccine approach that in the time COVID, was investments by DARPA early on to identify that kind of methodology in the time of need, where it would arise. And so identifying, you know, those opportunities where, even if you don't need it today, but you might need it in the future. That's what DARPA is all about. And I think, recognizing that the state of the technology for robotics Field Robotics wasn't where we wanted it to be in the future. That's what DARPA was eager to inspire, and incentivize, and then help shape that future trajectory. As far as building the community. I really love the DARPA Challenge model for innovation, where, you know, we have a couple of different ways that we seek out revolutionary ideas. But the challenge model is just one type of those and, and I really love it, because it also gives us an opportunity to think about solving a problem without having defines the solution. When you do that, and the community hasn't yet informed around that problem area, or the solution space. The DARPA Challenge is fantastic for being able to plant the seed and nurture this community that's now going to go off and do great things right. So the DARPA first started Grand Challenge, self driving cars to the desert or in the urban environments, you know, had planted the seed for a lot of the of the self driving tech investments today. And I imagine that many of the types of technologies that we've discovered and invested in for the SubT challenge will have that long term impact. While also, and I'm excited to say, have a very near term impact for many of our end users as well, given how we structured this this challenge to kind of think about marrying, you know, what the what the what the end user needs tomorrow, not just many years from now. So I think being able to span that has been a real hallmark of the DARPA subterranean challenge of understanding what the near problem is, as well as anticipating and planting the seed for the communities to address the fire problems as well.

(1:20:59) Audrow Nash

Awesome. Let's see. So I think I don't know how it works. But what's next? Yeah, like, what next with this SubT challenge next with maybe another, I suppose Grand Challenges take a long time to think about and I don't know, things can be revealed. But what's next? Yeah,

(1:21:20) Tim Chung

well, I'll say that the SubT challenge has concluded. And while we have all of those resources, all of the Open Source Repositories, all the things that we talked about, available out there, it's my wish that folks, you know, take it and run with it, they can reinvent or recreate or invent the new, different types of approaches to solving some of these types of problems. And so I think that what we see already in the community is kind of we've spun up the firewall, and now it's operating on its own momentum. And it's really invigorating and inspiring to see all these folks already out there. You know, organizing amongst themselves the the ability to have this kind of an impact. And being a resource. As far as DARPA is concerned, I think, you know, if any of your listeners have ideas for Grand Challenges, that's an opportunity here, I think DARPA is always on the lookout for those types of problems, those kind of technology questions that are out there that don't have or would benefit from not having a pre defined solution, kind of a direction to go, you know that there's a breakthrough waiting to happen. But you don't quite have a finger on where that breakthrough is going to come from. And a challenge model is great. And so DARPA, I am certain that the DARPA Challenge model is here to stay. It will, as it at, like all things at DARPA, continued to evolve and tailor itself to the technology of interest. But now, I'm excited to say that, you know, which DARPA Challenge model is demonstrated, at least with sub t, that it's a it's an exciting way to both drive a community and drive technology development.

(1:23:20) Audrow Nash

So wrapping up, do you have any links, websites, anything to share? I know the SubT website. So I'll include that in the post anything, anything else to highlight?

(1:23:31) Tim Chung

Yeah, I think, you know, if you go to one of the repositories, we mentioned earlier that a GitHub repo slash SubT challenge, there, you'll see a project there, just called simply SubT resources. But this is where it's kind of the one stop shop for all of the references, links to software that the competitors themselves have now released. So you can see that the datasets that both DARPA have provided, as well as many that SubT community has developed and curated, are there, links to papers and other materials are there. So you know, I encourage folks to go check that out. And then of course, the DARPA TV YouTube channel has many, many videos that you can go learn about the sub D challenge, its impact and why it matters, as well as learn about all the teams as well. So I think those are a great place to start. And if you're interested in learning more about DARPA, I think darpa.mil is a is a fantastic landing point for any of your listeners.

(1:24:34) Audrow Nash

Oh, yeah. Okay, thank you very much.

(1:24:37) Tim Chung

Cool. It's been a pleasure. Thank you so much.

(1:24:41) Audrow Nash

Thanks for listening to my conversation with Tim Chung. Thank you again to our founding sponsor, open robotics. See you next time.