In this episode, Audrow Nash speaks to Christian Fritz, CEO and founder of Transitive Robotics. Transitive Robotics makes software for building full stack robotics applications. In this conversation, they talk about how Transitive Robotic's software works, their business model, sandboxing for security, creating a marketplace for robotics applications, and web tools, in general.

Episode introduction

Introducing Christian and Transitive Robotics

Fleet management + networking challenges

Problems in fleet management

Business model

Using MQQT for message passing

Developing your own capabilities

Testing capabilities

Sending data: authentication and authorization

Sandboxing for safety

Discussing web technologies

Open source business models

Scaling, finding market fit, and taking investment

Plans in starting a marketplace

Plans to grow the team + remote work

Web and robotics

Web technologies on the backend

Future of robotics

This is a conversation with Christian Fritz, who is the CEO and founder of Transitive Robotics. Transitive Robotics makes software for building full stack robotics applications, meaning that it aims to make connecting your robot to a user interface easier. Some reasons you might want to do this are to remotely monitor the health of robot fleets. See what a robots camera sees, or to tele operate a robot? Transitive Robotics does this while solving the hard networking problem of robots potentially coming into and out of connectivity. I enjoyed talking to Christian. His perspective in forming Transitive Robotics seems to have been very pragmatic. And in my opinion, Transitive Robotics is building a tool that is very needed by the robotics community. It was also interesting to hear his thoughts on growing a robotics app marketplace, which may be in Transitive's future. This is the Sense Think Act Podcast. I'm Audrow Nash. Thank you to our founding sponsor, Open Robotics. And now here's my conversation with Christian. Christian, would you introduce yourself?

Hi. I'm Christian Fritz. I'm the founder of Transitive Robotics. People may know a little bit from my career, where I'm coming from, I used to be the head of software at Savioke. And for those who don't know, Savio, said hope makes a really cute delivery robot for hotels and hospitals. And it was, it was a really fun ride. For me. Being there, I started when there were about five robots in the field. When I left there were over 100. And it was a really an eye opening experience. For me, my background is in robotics, I've done Robocop back in 2001, I've seen all sorts of versions of robotics, including before ROS, which was bearish. And then at Savioke, I learned about not just the robotics piece, but the full stack that went on top of that. So that's really kind of my background, I was leading the application development team there, which basically, I always described it to new candidates or employees as as follows. The application development team gets a robot that knows how to stay localized and not bump into anything, and turns it into value to the company. So what that meant was all of the HDRI was done by our team, anything that involves just a sense of ambition, right, like running a delivery, the notion of delivery, writing elevators, calling phones, notifying users, all of that was done by a team. And it was, it was a really interesting experience. Because I found out that while ROS is really great, there's, there's so much more that you need in order to deploy a successful robotics fleet in the real world. And so that's, that's how I came to start trying to

know, what exactly are you doing at Transitive? Like, how are you solving this deployment problem and the problems that you kind of saw from your work, it's.

So it's heavy, we spend a lot of time building, you know, those those full stack applications, data synchronization between robots and the cloud at the front end, visualizing elements of the robot, such as maps at laser scans, and sending data back and forth and doing all of that DevOps. And it was really interesting for me to find out that other people at other companies like fetch robotics and places that we spoke with, were basically building the exact same thing, right. So we, we felt that, that that's not a good idea. And granted, there are some solutions out there that help try to offer you the kind of a one stop shop for fleet management and DevOps. And some of them are actually done by friends of ours. And they approached us and said, We have to, to see whether we would want to use them. But the problem that we had with them was that they all are sinking, like standalone platforms. So really, they give you the choice of do I build my own fleet management? Or do I outsource that. And the issue that we had with that was that when you outsource that your hands are tied, if the thing is not perfect, you cannot extend? And if there's something that you need, you still have to build it separately. And then you have, well, first of all, some type of work. But also you're duplicating your data, landscape, right? Some data will live in the one side or the other in

your domain in so software you're using for fleet management, and then whatever features you build custom yourself kind of outside of that. For your own. Yeah,

that's exactly right. Yeah, exactly. Right. So you will have separate users on different accounts. You'll have different roles. You'll have complex some data about the robot. Yeah. And so what struck us was that what people really want is the ability to just take individual complaints and plug them into your existing web application. A good example of that is low latency video streaming, almost everybody wants to see what's the ability to see what the robot is currently seeing. And it's a complicated application or capability to implement. Because there's some really great technology like the one that we're using here right now, which is why Partisi, which is running in the background, it's very complex. And a lot of robotics companies don't want to get in the weeds don't want to become experts, they don't want to get into the weeds. So I tried to give were basically developing these things as individual components. That's the one thing that's different about us from from the existing offerings out there. And the other one is that we believe in open source. So we actually following an open core model, with a platform that we're developing will be open source, and anybody can use it, and we hope people will, and they will build cool things on top of that. And this is really kind of a philosophical thing for us. We think that ROS has done such a good job at uniting the robotics community by giving us a common platform that we can build packages on. And so we can share, right? So if you if you build a really awesome navigation functionality, and thinking of your last you can you can share that with the world and the world will be a better flex, right. And I think ROS has done such a tremendous job at establishing that framework or platform on based on which we can share it. And we would really love to do the same now for the for the full stack. If it's open if it's open source, and it's easy to implement new capabilities that we really think that people will start contributing back. And we can finally stop all reinventing the wheel and building the same.

Yeah, for everyone who has a fleet of robots that needs to manage them in some way. So now one thing I'd like to do just kind of going more into detail, what does it mean to do fleet management? And what are some of the challenges with fleet management?

Well, so I'd start perhaps a little bit lower and start with why why sleep management or anything to do with robotics is not just another react app, or something else where you can use common tools out there. And they don't just solve the problem. The big one is really that these robots as much as they seem to behave like servers are not really servers because they run around, they lose network connectivity, they gather data faster than anybody can process them. And so you have some unique requirements in creating a uniform picture of your fleet in the front end. And this is not very unlike what some of the cloud providers offer with their IoT

when we say that you mean like a web browser or some other web browser?

Yeah, it's um, yeah, I guess I should, I should say that I'm asking I'm a no, it's a it's a good point. Because in robotics, and especially in bras, a lot of Qt applications are still being fronted. We think that if you're dealing with a broad set of users who are distributed, have different backgrounds and so forth, using web technologies, really the way to come. Because everybody knows how to use them. Obviously, most of the software being developed these days, runs in some on some form of web technology. And it's just a very lively and flourishing ecosystem. So using web tools, I think it's the best approach for building these front ends. And so at that point, you have to create a link between your your robot gathering data and user sitting in front of a browser.

Okay, so for fleet management, webs, web related tools are really good. Because they let you it's like it's better. There's really good tooling around there. There's a good ecosystem, this kind of thing, and I guess, getting into kind of the fleet. So and then actually, you were saying that a server, like you can't just treat it like a normal sir, a robot like a normal server. And so because of that, you have Transitive robotics, you're trying to make it so that you can, in a sense, treat a robot like a server while addressing some of those challenges.

Yeah, that's exactly right. I think it kind of comes down to the core abilities that the platform gives you as a developer. And one of them. I think that's perhaps the most important one is the data model that it gives you. And what we're doing there is actually not very different from what some of the cloud providers offer in the IoT solutions at all on AWS, they call them IoT shadows. In a jury, they call them desired and reported state of a robot, the idea is always the same that you have some form of data model that tells the robot this is what I want you to be like, whatever that is configuration deployment. So as a robot to get into, basically. So it's, it's saying I want to think of it as you think of it as a state. Yeah. At the state can have all sorts of aspects to it, right? Like which software to run, what users to have, what configuration values to use, which cloud to connect to, and, and then the problem is your robot. Sometimes it's not just not online, right. So you have to make sure that you reliably synchronize that data to the robot, and also get the get the data back. On the coming back side, you have that same issue that you may want to monitor your robot while it is traversing through some areas where it has very poor connectivity. But you don't want the front end experience to be chopped up, right? You want the user to have a smooth experience and still see the robot, even if right now you can't get signal from it. And that's, that's why it is not a server, you can't just run a web server on it and have to reuse it connect to it directly.

Is that is that kind of the biggest challenge in this? Would you say where it's, the robot comes in and out of connectivity? And it makes it so that you can't make a lot of the assumptions that people would make about, like web servers? In general?

I think that's a good way of putting it. Yeah. So

what do you what do you do in the case? How do you how do you make a smooth experience when a robot is popping in and out of connectivity?

I think really, the key here is an intelligent data synchronization where the robot essentially updates the cloud, whenever it has connectivity. And then the cloud serves the serve the users front end, right, so the user always sees data, sometimes data will be a little bit delayed, because the robot hasn't gotten an update through to the cloud. But the cloud will be able to tell, hey, I haven't gotten a heartbeat from this robot in 10 seconds. So what you're seeing maybe maybe a little bit old, but from the user's perspective, pages, load fine. Everything will be there. And you will always see the latest data that you could possibly get independent of when you show data, some people think you could just run a web server on the robot itself, and then connect and then the problem is sometimes you try to build a patient that just doesn't know because your servers down or not connect you.

You mean literally host the website from the robot.

That's, that's a lead. So let's take an alternative. Some people are this into the robot harvest or like a non ROS users is a visualization tool for ROS. And it's it's an awesome tool. But it's, it's not really meant to be used remotely for for several reasons. One, it sends a humongous amount of data because it actually thinks that the connection is very good and fast and sending duplicate data is not a problem. And it also just sends a ton of data. So there's not a lot of filtering typically going on. Which means that remote experience with harvest can be very choppy, and the robot goes in and out of connectivity sometimes. So if you lose that you won't have a lot of disruptions, you're losing interest. So I think some kind of caching of the data in the cloud. That's one, reducing the amount of data that you send from the robot to the cloud is another Oh,

yeah, that's an important point, reducing it so that you don't actually spend that many resources on kind of, I don't know, perhaps you can get by with less. And then you have less requirements of your connectivity to your network.

Yep, that's exactly right. A good example, here is health monitoring. A lot of your your listeners may be familiar with how ROS solves the diagnostics problem where there is a nice hierarchy of message definitions that allow us to share diagnostic information about your robot such as set of values, battery stage, and so forth. But the way that ROS publishes status, here's an update, here's an update, here's an update, even if nothing changes, right. And that's fine with ROS, that's not ROS is designed on the robot. And that's, that's good. It's actually a good approach on the robot, but not for the cloud, because you don't want to repeat messages if they're literally the same as the previous one. So even just duplicating sequences of messages will already save you a ton of data.

Yes. So how I'm understanding what it sounds like you're describing. So if you want some sort of way of monitoring your robot, your robot has information it's driving around, it may move in and out of networks. What you're doing is you're placing a cloud component so some computer somewhere else that's talking to the robot. This is basically it's keeping up the whole time. And it's displaying whatever the most relevant data is on there. And then I suppose, well, actually, and then you have the user who looks at whatever the cloud is doing. So the cloud has its own host, its own website is hosting. And the user can look at that and get to see the state of the robot. And then I assume on the robot, you have something that's saying, this is necessary information to send up to the cloud. So if it if it's overheated message that's being sent, you don't bother sending it again, because of limited bandwidth effectively, to try to send data to the cloud. Like we only want to prioritize and send the most important updates, probably, to the cloud. Yeah,

that's exactly right. That's exactly right. The way that these decisions are made is that we believe in vertically integrated capabilities. So I've described so far is this very fundamental data synchronization layer, but then what you build on top of that, we think in terms of apps, right, so let's take a remote Tailor of capability. A lot of people need that on the robot. So that involves sending video from the robot to the to the front end, displaying the video on the front end, and sending back some control signals from the from the joystick. The Transitive is structured is that we have this concept, very strong concept of packages, similar to ROS. But what's special about Transitive is that inside a package, you don't just have components that run on the robot, or the code or the front end, you have all of them together. And what's great about that, is that you can you have an end to end exchange. And if you want to change the way that things are visualized on the front end, if you need, say, some new data that you didn't have in the previous version, it's fine, because you're going to publish another version of the code that runs on the robot and push us to the cloud as well. So this gives you some really nice. We think of this as cross device dependency management. Yeah, you don't have to worry about oh, well, we just upgraded our front end. So we now also need to upgrade our backend. It's all it's all taken care of.

Yeah. Yeah. Interesting. Okay. So going back. So some of the challenges with fleet management, we have, basically the connectivity issue where it's going in and out of the network, we have, perhaps, the, the so you have the robot, it's sending data back and forth. The robot may or may not be connected, you have health monitoring, and things like this checking the status of the system, and you have updating it, which you've kind of just mentioned, you can ship things as components, and it's updated universally. Are those kind of the other additional challenges to fleet management, and network updates. And the second, but yes, those are there additional ones that are kind of critical and fleet management, health monitoring.

Actually, if you don't mind, I'm going to share a kind of famous Greenbrier course. So this is a list that we've put together off things that we needed at one point or the other, in the past. And it's grouped by categories of kind of apps or capabilities that you might need to run a fleet successfully. So there's the category of DevOps and fleet management software, super modular uploads a video is Iraq, prospect view and health monitoring, and it goes observability is a big one. alerting, right. A lot of people like to integrate with pager duty and so forth. But then there's also things like application logic, right Task Entry to a queuing of tasks, user management, account management, and a lot of those things are really not that much fun. Yeah, so that and they're not really they're not differentiating to a robotics company, right? So we were we validated this with a lot of CEOs and CTOs of robotic startups. And they always said the same thing. There are certain things that we would like to focus on. Often, in the ml or computer vision space, we were talking with an apple picking robotics company, you can probably guess what that was. And they said, well, our our secret sauce is in our ability to identify grasping points from the computer vision, we would never want to outsource that. And that's where we want to differentiate ourselves. But there are so many things on your list like CI CD, like anomaly detection, that's not that's not important to us. We would love to outsource that. And that's how we think about Transitive a, a both a platform to develop those kinds of capabilities and then also a company that will that will offer them

yes to you So where are you guys currently in this because the list of things, there must be like 40 items or some large number of things that all seem really important to shipping a robot application. Do you have all these implemented? Or is it like a set of them? Or we're yet?

Yeah, no, not at all. This is the aspirational slide so to say. And it's also meant to, perhaps give people ideas of what they could develop and contribute. This is really the motivation to do all of that. And the way we're going about it is we want to develop the platform First, open sources, and then start building the ecosystem to develop all of them. Right now, if you go to our website, there's three capabilities that you can try out right away. And they are some of the ones that I've talked about. So far. The low latency video streaming, remote telehealth and, and health monitoring. Everything else?

So TBD? Yeah. So it's, so you have this list, you kind of gathered this list, and I assume maybe it was somewhat iterative in the process. But I assume this really informed the data model that you're using, and the development of the system so that you could fit all these cases under your framework.

So that's exactly how we're thinking about it. And

that's really cool. So the, I'm kind of imagining this is, like, I'm mapping it to like web concepts, it's eventually going to be like the hosting service. For robotics applications, probably you have your specific logic running on your robot. If you're using the ROS ecosystem, you get a lot of things for free, that might be good enough for your applications, like navigation or move it for motion planning of arms, these kinds of things, and then you have this to make it so you can quickly build your web interface to monitor your application for the people that should make sure the robots are running or I don't know, whatever it is that's engaging and engaging with your robots, or whoever it is, yes, whatever they're doing.

We're thinking about this in two different ways. One is to get people started more quickly. So they don't need to build these things. And then sometimes our investors ask, well, aren't you going to lose those people, once they grow to a certain scale? And they will, they will not care? Because they are now they now have 200 employees, and they will build it themselves? And my answer to that is they still not want to. And the reason is, like my counter example is slack. Yeah. Why do people use Slack? It's not impossible to do. In fact, it's pretty trivial. However, are you really going to bake it quite as well as slack? And that's all I do. Where, exactly, it's, it's the value of specialization, right. And in the end, you will use the best solution out there, startups in especially in robotics are actually not as price sensitive as you may think. And that's because the bulk cost of robots all of the operations is not that cheap. So if they have to shell out $20 a month for a capability, there's really well, who cares, they're spending more on the LTD Bill per robot. Right. So I actually think it this is something that I've felt with ROS, too, is that I think ROS would actually benefit from more entities having commercial a business plan behind them and commercial backing, so that the contributions that are being paid by those companies to ROS have have a longer a longer outlook, right? There's, there's some real money behind it, we can be certain that they will be maintained. And there's somebody on the hook for making sure that they that they do that they work well. Yeah, there was this really, I think I wrestled last year somebody presented this paper, I think it was called something like it takes a village, I forget the details, you remember that? I don't think I thought it was really interesting, because they did a study of ROS packages, and how many which are used, like how many how many people use them, and how well maintained they are. And it was really striking that there are some like the core modules that are they're all being used by almost everybody. But then the vast majority of packages, and of course, there's a ton of packages, they are not maintained very well. So that puts users in a difficult spot. And in deciding of can I really rely on this package? Or do I have to fork it if I want to if I want to use it. And I think that the JavaScript community does a very good job at that with their with the NPM package manager where with every package that you can find on their site, they will show you how about how well does it maintained me to how many commits Have there been made lately? How popular is it? How many people use it? And I think some other metric and I think I think that's very helpful quality, some sort of They bought the holiday

trick. I don't, I don't know how to get that, but it's quite cool.

I think it's very helpful. And when I choose libraries to use, that's definitely something that I look at. And Oh, for sure, knowing that there's others who use it is sometimes perhaps the most important factor. Because I know that if I run into issues, I will not be alone, there will be others who will be who are also invested

in it. And you can feel a little more secure in that other people might pick up some of the slack, you can pick up some of the slack, but it's not entirely on you to maintain if the developer just goes silent, and maintaining it. Yep. Gotcha.

And I think if we can stand something like that up here, we're obviously taking the perspective of the full stack. But I think some of those ideas translate into into other ROS packages as well. And one thing that we haven't touched on yet is you could have a rust package be part of one of these apps, right? You can, the way that we actually designed our deployment mechanism. These packages can contain whatever you want. Some people cringe when they hear whatever you want, because that sounds dangerous. And we can talk about that in a second. Yeah. Yeah, so we we've designed this from the bottom up as a framework that

what are the components? So in any one of these packages? Or in any of these components? It has the robot arm? Or what are the parts of it that are involved? And how do they talk? And then sandboxing sounds very interesting to talk about, too.

Okay, so let's, let's dive a little deeper up. Before we before we get into that, I think I should mention that we're using TGT as the underlying communication mechanism for just about everything. I think that will help our conversation here.

I don't know much about that. I don't know anything about that.

Okay, so MQ TT is a Pub Sub mechanism, where you have topics that are hierarchical, separated by slashes, just like a path in Unix,

very similar to ROS topics, it seems like from the description so far, in a sense that it's good topics, and then it's hierarchical, like namespacing, and this kind of thing.

That's right, I think the main difference rough topics is that they don't have this notion of a message definition. Any message in the end is just a buffer, or a string, like a character buffer. And then on the receiving end, where you will interpret

And other than that, I mean, it's a very well developed technology, it's actually underneath all of the IoT, cloud offerings like AWS, GCP, and Azure, all built on MQ TT. So it's sort of battle tested. And it's meant for IoT devices. So for low resource usage, very high throughput, communication. And unlike ROS, it was really meant to be cross device, like the typical IoT application, you have a large fleet of small devices that all send data to the cloud, and from there potentially onwards. So this seems like a very good technology to build this on. And then our packages, they are exactly like you suggested, they actually have three components, there's the code that will get installed on the robot, there's the code that will run in the cloud in a Docker container. And then the third piece is, are any sort of front end components. And there we are using this new technology called Web Components, it's not a very good name, because it's not very unique. But web components is a thing. And it's very nice, because it's front end framework agnostic, you can use it in React or Angular, or even without any of that. So it really just is like a custom tag in HTML. So you have lots you've loaded that component to that component, you can just type, you know, pointy brackets, the name of your component, and it will appear with where you want it to be. That's what we use for the front end. And then underneath, they all communicate through through MQ TT, in a in a namespace that is not just specific to the capability, but actually even to the version of the capability. And this is what allows us to create this cross device dependency management. This is this was a very big pain point that that I've faced at one point in my career where if you update the cloud, you may be forced to also update all of your robots, depending on what you've changed on the cloud. Now, guess what? Your robots if you have more than 100 robots, at no point in time, a lot of your robots be online at the same time. So when do you make the switch? Right? And so there's different ways to deal with it. Some people

get both, and you just have them send their version messages. That's how we are solving it except super cool. And then you kind of delete it once they're all on board with the most recent version.

Nobody has been talking to me be on that version for a month. So just got to shut off that Docker container.

That's super cool. Okay, so you use MQ TT, this gives you that really nice version messaging behavior that makes updates really cool. And I really like the idea of kind of it trickles in as the robots become available. Okay.

So that that tells you how the three components in the, say the three again, just explicitly, the robot, yep, the cloud and the UI. Okay.

And so for these components, one thing, so are they all? You get them to talk? Like, so if I'm writing one of these components? How does it look? Like how can I test all of these? What do you call with the components within the component? Or how do you refer to the the web robots and cloud parts of component?

We haven't, we should get that terminology. We don't have a good name for them yet.

Gotcha. That component makes me think React components, which is kind of it's a, and then that makes me think just the web part of what your thing is. But

we could call them modules. And well, actually, on right now we're referring to this whole thing as a capability capability. Okay. And then the three underneath are packages or modules. But okay.

Gotcha. Okay. I like the word capability better for discussing it just so it's not as overloaded in my head. So how do you write a module? And how do you write a capability and test all these modules? How do they work from a developer's point of view?

Some of this is still being developed right now, we have not yet made the open source release. And one thing that we want to get done well, is the SDK, which is kind of what you're asking about, right? So there will be tools that will allow you to create new capabilities, quickly in the sense that they will, I'm taking a lot of inspiration from again, how things like reactive office where you have like a CLI tool, that's where you might say, Transitive, create new capability, whenever it will, it will set up the directory structure and you know, the package json and the web, web pack, whatever you need, in order to compile all of those. Yeah, we do use I guess, I might have to say that we do use NPM exhaustedly, and for those of you who don't know, NPM is the node package manager. It's a very nice tool. A lot of people think like, it's only for node. But it turns out, you can put anything you want it to that package, it has a lot of advantages over say Debian packages. That was probably too much detail to go into, but I'll be

talking about it just a bit later in the interview. Yeah, good.

So once you have these NPM packages, there's, you know, scripts that you can have for NPM publish to publish those capab those those packages have your capability into a into a private registry or an Artifactory store. And from there, your robots will be able to pull them and and run them the connections that you were asking about for communication, where they will be set up through the further data synchronization. So from the from the developer's point of view, all you're manipulating is actually one big JSON object. And within that JSON object, you can decide which sub paths you want to subscribe to which ones you want to publish. But you will always be dealing with a JSON object as if it was just on your machine, you know,

the thing that's being passed between the modules in the capability correct. So you're sending that JSON object around and that JSON object. So that's just a data structure for viewing everything. JavaScript Object Notation, or whatever it stands for.

That's what it stands for. And data data data synchronization that I mentioned previously, basically just takes care of making sure that what the robot sees from that tracing is exactly the same as what what the capability in the front end sees, and you don't have to worry about it, you just manipulate it on the robot. And you will can see it like that on the on the front end. For those of your listeners who know how, you know, reactive web programming works is this is a reactive data source. So if something changes here, your React component will update. dynamically, you don't have to pull you don't have to worry about it. I simply will take care of the push. And in the moment there's a change on the robot. You will see it on the front end without ever folding.

So what that means is at the second row About sends a new image from the camera is your component updates and shows that new image correct a web component.

I wouldn't necessarily use images or an example here, okay, because that is more complex said for our web RTC and like with good compression. But the example still holds. If you if you could do that you could send an image over that channel. Another example would be like, let's say the battery charged, if your battery charged, that's probably a better example. It will show up right away on the front end.

That's really cool. So how if I if I'm developing this, like if I if I'm writing my own capability, how do I test that all of the three modules are working well, for the capability? Is there any like, I don't know, the thing that makes sure that all of them are working well together, that data is being sent around that I can test my components, my q&a module for the web browser locally, these kinds of things, like a storyboards for the mind, all these things. Component,

there's a lot of tooling, yet, so some of these automated checks that you're alluding to don't yet exist. But there's definitely the developer workflow and the documentation on how to run it locally in a development server. So that you can you can try this out. And I think that's a good way to get started, you can use a simulator to run you know, your, your, your robot, whenever we use turtle sim a lot, which honestly is enough for a lot of work we want to do, and then you open the local web page on your machine, and will you will be able to see whether or not things are working.

Gotcha. That seems super cool. And what is the so in the cloud component? So there's a web, the cloud the robot, what does the cloud component do?

So the cloud component is actually a very module. The Cloud Module, there's, there's obviously what you what you want the functionality to be there can be visualizations, and so forth. But before it can get there, what the what the Transitive wrapping for that, for that module does is twofold. One, and that's very important is authentication and authorization. Because if you want to embed this, this component in a third party web application, you need to get the appropriate permissions to actually show the data of the robot, right? I mean, not, you don't want the camera of your robot to be publicly available on the internet. And obfuscation is not enough, you want some really strong authentication, the way that we solve that is through JSON web tokens, which is a security technology, it's it basically allows you to take a JSON object and sign it with an asymmetric or symmetric key depending on what you choose. And we use that to basically get permission from the back end of your service, whatever you're embedding it in, to show certain things to the user. So you will basically give your user a signed JSON web token that says this user has permission to see this device, this capability for this and that long. And then it will be embedded in the in the webpage that your user goes to that webpage will talk to the transit server, whether you host it yourself or whether you use our transit products hosting. And it will let us know whether or not we are allowed to show the users that that kind of data. So that's our Yeah, what are they authorized to see that and that's that by itself. So it's a big pain point that a lot of people struggle with just this. For sure. It's complex. And we've seen some very successful companies offering authentication as a service like Okta. And for good reason. It's, it's not a lot of fun. And if you make a mistake, a lot of at stake, right, your company, you're sharing too much data than you then you wish you were.

Yeah. And I think that's an area where it's like, oh, you need to go be an expert at to do well. And so it's nice to specialize there and have another company that just does this really well. But so you don't have to worry about doing this for your application. So the cloud component for any of these capabilities you're making is going to handle the authentication and authorization. I imagine that's kind of like baked in to what you're building. So they don't probably have to handle that when writing their application. Does it also handle is it on the robot that it's running? Like, I know that sounds silly, but is it the thing that sends the information up to the cloud? Or is it actually running on the cloud? It's actually,

yeah, the human way the authentication is going to be processed,

or I mean, I guess. So the authentic authentication. That's I guess, I'm just I'm a little confused, because so for that, I would think that that would be something that you would set up very well. And maybe all the code lives in the cloud component. And the person hardly who's the developer hardly touches the cloud one it just like, because I don't think like, if I want to use this to write capability, I don't probably want to do the authentication and authorization side. So I'm imagining that's probably generated or exists in something that's already being used, like my code is using.

Yeah, that's part of the the essence of the framework. And it's actually a feature of the MQ TT brokers that we're using, where you can have custom authentication and authorization, which which we try to offer. So the way it works is that the the web component actually establishes a web a MQ TT connection over web over WebSockets. Okay. And in order to authenticate for that, for that connection, it will send its Simes JSON Web Token. The MQ TT broker will interpret that and say, first of all, yes, you are who you claimed you are, because this is a valid JSON, valid DWT. And also I've decoded that JSON web token. And in it, it says you have access to the following topics. And this can be read or write access. And that's all you need. Right? Because again, this is the beauty of relying on MPT for everything. Now, you know, that the broker knows which messages it is allowed to send to you, and which ones you are allowed to write. And then it kind of goes into the ether and and everybody connected to that. That's it. Yeah, it's, yeah,

that's magic. That's so cool. I feel like it's taking such a painful problem and making it a lot less painful. So just so the cloud, if I'm a developer, using writing my own capability to cloud component, I will probably have the ability to use the default way of doing authentication and authorization. Or I can write a custom one, if I desire. And that's kind of all the cloud component does, or is there anything else?

Well, I mean, the other thing that the cloud component does is, of course, the data synchronization piece, right? So that in the cloud component, you have immediately access to you can subscribe to a subset of the of the topics that you want, and they will enter data will magically appear on your doorstep. And then you can visualize it in whichever way you want. So that you can really focus on on just that, right? How do I, how do I present the data. So you know, I mentioned your little play, I like to think in terms of black boxes, if you're a web component, your black box that you receive, after all of the magic has happened, you receive a JSON object. And what you're responsible for is outputting, a nicely rendered HTML page or whatever or react whatever you want to use. You have visualize that data, right? Health Monitoring is a good example. Again, like you get a JSON object that represents the health of your fleet. And you decide to show that as a table with red and yellow and green and and what have you. Right? That's that's really the that's really what people want to use web components for. They don't want to spend too much time on the on the plumbing, as I call it. Yeah, they just want to visualize it. And then in the reverse the same right, so your, your joystick opponent says the user has been pressing the forward button, now you need to send that to the robot. So you will send the appropriate data into the onto on that topic where you know, the robot will be

listening. Gotcha. That's really cool. What does it look like on the robot? How, like, how do you set it up and what's actually running on your robots when you actually are using this.

So the the way to get started ended depends a little bit on whether you want to use the hosted solution that we offer or whether you want to sell photos. But in the hosted solution, you just go to our website and create an account and you will get a curl script that you can run on your robot, which will install a Transitive robotics agent. This is just a small little piece of software that runs on the robot. And it's primarily responsible for checking whether you are running whether which packages are installed, which packages should be installed, and then installing them and starting them. And then once it is told to install a certain capability, it will fetch that from the repository and install it and run it in a sandbox. The Sandbox is one of the other pieces that is really important to the way that we're thinking about this because we want to enable third parties to develop for you. We think that sandboxing is really important.

Tell me tell me what sandboxing is. I know what it means, but just like describe it.

So the issue is always that when you're buying software from a third party, you you may not want to trust it 100% So you want to reduce basically the damage it can do you want to reduce the permissions that it has on both iOS And Android has a very, very strong sandboxing model. So if you install an app on your phone, it won't be able to read the password from your from your other apps, right, which is kind of important. Yeah. And the same thing here, right. And in the ROS world, people are used to using sudo, to install ROS packages. And that's a very honorable thing to do, as opposed to give that much trust. But if you are dealing with a much larger ecosystem, you may want to be a little bit more careful about the trust that you give and not run everything as root. So a lot of people wouldn't even want to run it in the normal user space, because even the running user could have could have secrets and so forth. That's one thing. The other thing is you want to separate those the capabilities running on your robot from one another so that they cannot see each other's data. Otherwise, the authentication and authorization is kind of goes out of the window, because somebody could install everything. Yeah, and a malicious capability could just listen in on everybody else. So you're solving this. Through a number of technologies that all borrow from Docker, we do not use Docker on the robot, because we don't want to use require that as a as a as a requirement. But we use some of the same technologies, we use the Linux namespaces, using the unshare tool, to to isolate things. This allows us to do pretty cool things, honestly, without requiring sudo on the on the robot, for instance, we hide a lot of folders on the drive, we can hide the home folders. And then we're also using overlay Fs, which this technology exhaustively used by by Docker, to minimize the number of copies of things that you need on the system. So if a capability has a requirement for a ROS package, it will it will first check is that already installed. And if not, it will install it in an overlay on top of what's already there. This way, we don't have to install ROS in each and every component that we have, which would happen if we were running in Docker containers, right, there would be a lot of duplication, which we can, which we can elegantly avoid.

That is really cool. So you see if you have it elsewhere. And then if you have it elsewhere, you use it from that location. If not you install it. That's really cool. So what about handling things? So that that handles? You mentioned a lot of things? And I don't I'm not that familiar with all those technologies. But it can you hide things like the environmental variables that one capability can see versus another one? Or it just like all of the different things you give it access to only a small part of the file system. Yeah. How does all this work? It sounds so cool.

Right? Now we go with a one size fits all type sandbox, you don't actually customize it, we just we just reduce it down to what we think is all you need. If you don't see the environmental variables of your user, you you're either home folders or hidden certain sensitive folders in the in the file system such as var lib, or from you as much as possible, like the data, if you're, if you're running a database, it would live there and he wouldn't be able to see that. And you explicitly

opt in to permissions with this, or is it implicit in that, like, oh, this app needs this? And then we'll give it that? I'm not sure it's explicit. So

so right now we, we think actually, we have the permissions down to what most capabilities will will need we there could certainly be some people have asked us about things like user management on the robot, right? This is clearly something that would need administrative rights. And we don't currently support that. We are currently only supporting things. It's a what one way to think about this, especially for your audiences ROS and up if you a lot of capabilities run a ROS node, they're able to talk to the other ROS notes. And that's honestly all they need to do. Yeah, it's it's none of their business what I have on my on my desk, right? That's that's not what we're here to do. So, all of that accesses is going to be very, very limited. Other than say, you know, what you need for, for for ROS operations, such as the message definitions, right. But most of the file system will be hidden from

good I saw I'm imagining if I had some node that so I have a module on my robot, and that's sending up video. And maybe I want to send some sort of thing down from the cloud, to say, hey, we're a bit bogged down right now. Reduce the video quality you're sending up or something like this. And maybe it's netstat and an environmental variable, like, could you give it access to that? Or maybe there's a better way to go about doing that kind of thing?

I would have to think about it. If by setting an environmental variable, you mean one that would also be written, read by all of the other capabilities, like some sort of shared space? Yes, not currently possible, there is no shared drive space. So another piece of technology that we've taken a good look at is snaps on on Ubuntu. And they, they solve this very elegantly to where they have, of course, you know, containerized, or sandbox capabilities, but they also have some, some shared folders, right, in which you can collaborate if necessary. So that that certainly, that certainly exists. In the example of the video streaming, I would, I would argue that that's the capability itself, that then reduces that.

But yeah, you probably wouldn't need to do it the way I described. Because yeah, the capability would probably just do it. And it'd be setting a variable inside the module, robot module. That would be probably the best way to do it.

And actually, so we do have that video capability. And we use Web RTC, of course, and one of the beauties of web RTC is that it will actually do that automatically, right? That's so Turman, I can't get those frames through, we're going to reduce quality for a little while, and and it will adjust to that, which is very nice. And really one of the reasons also to use Web RTC

for industry. Do you think that this idea of sandboxing will catch on more generally, in robotics? Like I mean, if I'm using a bunch of software from the internet, you never know if something has been maliciously included? So

yep. Yeah, I, I think so. I mean, if you've been following what Ubuntu has been doing with their push for four snaps, for instance, they had a person that was also very active in the Rust community. And he was in charge of ROS and snaps on the canonical side. And he was making some very good points as to why snaps are a good solution for that because they are sandbox, like you say, right, they are snaps for, for those of your listeners who don't know, basically, or you can think of them as Docker containers that run apps on on Ubuntu now, people will yell at me for describing it like that. Because of course, in reality, it's more sophisticated, and it actually doesn't use Docker. But for that, for the event, who knows a little bit of Docker, that's roughly how it works. And so you get all of the beauty of isolation, and, you know, reliable requirements, like dependency, dependencies, and so forth. So I think it's useful. I'm very curious to see how this will evolve, and whether the sandboxing will get into the way of development, whether people will prefer to install their capabilities, quote, unquote, privileged. But I think it's the I think it's a, it's a requirement for more broader cooperation, like you said, if we want to enable third parties to run, and it's our fleet of robots, right. I mean, for it from the perspective of a business owner in robotics, they're very conservative about what gets to run on the robot. Because if you screw up, like, these are implications, there's a lot of implications for on the security side, but also, even just on the operational cost, if for some reason, your robots one morning, don't do it anymore, or, or you know, they dropped from the internet, you can't reach them, somebody is getting on a plane flying somewhere. Very expensive.

Gosh, uh huh. One thing a little bit of a tangent, but one thing that I thought was really interesting, and I am hoping more things start to go this way. There's Dino, which is a TypeScript library or TypeScript runtime environment. And Dino, for us. It's kind of like Node, but better suited for TypeScript also runs JavaScript. But if you explicitly opt in to every permission that you want to use, you can say all if you want, but so you can say this should only be accessing these folders, this should only have access to this environmental variable. And all of the software that you use can have these permissions set, which is quite cool. Because I mean, if you're running software, there's the chance that it's like trying to access something you don't want it to use. And this would be a nice way to bet that hey, there was an error or tried to access some sensitive location. We denied it and the program exited.

That's pretty cool. I had heard about the though I didn't know that they have that feature. That's uh, I'm actually gonna take a look at that. That's that sounds really useful.

Yeah, I'm sponsoring site quite a few you know, projects cuz I really like the direction that they're going But the whole node ecosystem is really, really, it's just so much there, that it's hard to use the you know, for anything that you really want to build, like, I've just been doing it for, like, command line tools that helped make my job at open robotics easier to try them. And you know, but lately node is kind of thing.

Gotcha. So it's not a replacement for Node js at this point.

Not yet. But there are quite a few packages that are like, I mean, I don't know, there's like an express like thing for hosting web servers, and routing. But if you want to do like, really nice date time stuff, or I don't know, there's just a bunch of missing stuff. Or you can go look at the CDN serialization or whatever it is, I don't actually know what that does. But you can go some packages can be translated over to it, but it has a different import setup. So it doesn't work flawlessly with existing node things, which is a bit of a bummer. But the ecosystem is growing. And there seem to be a lot of really smart programmers doing stuff on it. And building tools.

Yeah, I mean, that's really, I think, key to a framework platform. You name a working ecosystem of developers, right. And my favorite example there that the reason why Node js is so popular and works so well is because it's so popular and works so well, right? Like people use us developing more things. And because they develop more thing, it's the network effect, right? And then more things. So you can find a library for just about anything if you don't need to develop it, yourself. There's downsides, of course, and we've seen the news about some, you know, either hostile takeovers of packages, where people have have stopped maintaining it. Whereas sometimes even malicious changes that people made sometimes out of spite or break everything, yeah, yeah. And those issues are, of course, serious, and we need to take care of them, and it'd be worried about them. But it's a very cool technology to build on.

Oh, definitely. Yeah, I find it. So I mean, working at Open Robotics, we use a lot of C++ and Python. And type like years and years of using Python. But I really like TypeScript, because defining the interfaces is really nice. And it helps me I feel like I'm much more productive in it. And so now I'm getting familiar with the node, or NPM, packaging, and just downloading libraries and software from it. And I'm very impressed. And I feel like it's a, it's a lot more capable, in my opinion, than a lot of the Python libraries. And there are obviously exceptions, like, like, pandas is a really nice Python library that I don't think has a good analogue in TypeScript or JavaScript. But just it's an interesting space.

It absolutely is. And I like to play dog to that. Note, J S is actually a bit faster than Python. And that's because JavaScript is compiled just in time, it's not actually interpreted, it's just compiled on the fly, which makes the execution a lot faster. And then I can, you know, say a lot of positive things about NPM. As a package manager, it's, it has a lot of features. And it has a lot of capabilities that other package package managers have within the past leg. So for instance, one big one is having multiple versions of the same package in the repository, you would think that this is a trivial thing. But DVM, for instance, doesn't, doesn't really support that apt, you're not really supposed to have multiple versions of the same package in the in the repository. But it is important

for your application where you're trying to wean robots off of version, when they're when you're updating them. This is, this would be very useful.

Next time, it also for the for strict versioning. I think that strict versioning, of strict, strict semantic versioning is a real benefit. Where you in npm date, there's this concept of a lock file, where you will save I'm going to lock down my versions. And when I commit to get is exactly this, if you execute this, you will get the exact same result, no matter what any of the dependencies have done in the meantime, in terms of updating, you will get the exact same. And and this is really helpful when you're dealing with an ecosystem like NPM, where you have hierarchies of dependencies that can be very, very deep, right? You can super nest 2000 packages being installed as part of your of your somewhat simple application. And that's, that's a benefit, right? You don't want to reinvent all of those 2000. But you do want to be certain that if anything happens to either of them, maybe they introduce a bug in the new version, your system doesn't go down. So really locking down the versions and deciding Yes, I want to update or I don't want to update it. Is, is very powerful. And for that you need to be able to have multiple versions of the same package in the repository. The other one is, of course, all of the books like pre installed post install, pre uninstalled post, uninstall, pre publish all of those kinds of scripts that you may want to run as different types of the software lifecycle, they are very powerful they can really allow you to do I mean, us? They are, they are very powerful in Debian packages as well. But because NPM has them you can do, you can pretty much install anything through NPM packages that you have, because the payload is just a tarball. And then you run these scripts when when it gets installed. And at that point, you can, you could install things out of the tarball that say you Debian packages, or play system files if you have the permissions to do that. So it's very, it's very versatile tool. And I think it's, it's a real good technology to build on and share.

Let's see. So now segwaying, a little bit. Tell me a bit about your open core model at Transitive.

Yeah, this is still still in flux. But I am a big believer of open source businesses, I think that there are some very good examples that we've seen in the past. And so we, we really want to make this open source model work. And from everything that I've learned. There's, there's basically two things that work really well from, from an open source business model point of view, one is testing. So in our case, we are offering a hosted solution where the cloud will be taken care of by us, you can still, like I said, have the components and embed them in your own web applications, if even a web application is running somewhere completely different. But you don't have to worry about it, setting up the cloud, dealing with the maintenance and administration of that storage of data and whatever your capabilities want to do. That's the that's the one model. Some of our customers have asked about self hosted solution. So we are offering that as well. And this is still a work in progress. So that has not been released. And the business model in that case, is premium applications. So it's it's really similar, I guess, to Android, again, right? Where Android is open source, you can install it on any phone you want. But then the capability the apps in Android can also be free, or they can be they can be premium. And I think this is a good model, right? It allows people to first of all get started with it. They it provides developers with a marketplace where they can distribute their their their contributions, their work, they don't have to worry about, you know, advertisement and and billing and and all of that. And on the receiving end, you know, the the companies using those capabilities will have an ecosystem with writings with commercial support behind it that they can that they can rely on. So I think that this is a model that can that can really work.

Gotcha. And so are you experimenting between those two models? Or your you'll think you'll kind of see what capabilities you have? And then possibly add? Actually, it seems like it makes sense to do the Android model for what you're talking about, where it's and then you're doing both in a sense, because you are providing the hosting, but you're also possibly going to have specific capabilities that people can opt into, by purchasing. Yeah, that's,

yeah, that's exactly right. And both are evolving, we'll have to see which one will be more popular and how, for instance, what we are still keen to find out as to what extent will people prefer the hosted solution versus self hosting? There's definitely pros and cons with both. Some people really don't like the idea of relying on a third party, especially if it's a startup, right? Because it multiplies to whatever No. That's why the ability to self host is certainly desirable. Others really don't want to take care of the scaling of that infrastructure and would much rather outsource that. So this is mongos applet service, which I think is fantastic. If you if you just want to set up a database, you don't need to set it up yourself and manage it. But I Yeah, time will tell really. And we're curious to hear from from more people what they think. And also, on the premium app site, there's this additional persona of the third party developer, right. Are people interested in developing for this platform? I think so. I've spoken with a lot of people who have been working on specific capabilities, a lot of them in the ml or computer vision space that they want to market to robotics companies. And they like the idea to not having to do all of the legwork in terms of marketing it and figuring out deployment and so forth,

having more of an app store third approach. Yeah, that would be super cool. I would love if we had that kind of thing in the robotics ecosystem. Just like ROS really solves a lot of problems, making it easier for people to build applications. But it's hard to get it outside of your robot, as you're saying, as we've kind of motivated with the whole first part of this conversation. And so if it was like, you just try to get you would just add different capabilities on through an app store like setting and you can use them elsewhere. That would be super cool. Yeah, that's it. Gotcha. How have you guys been funded so far? And like, Would you talk a bit more about the team and everything,

the team is very small, you're looking at it. God's just, it's, I'm the only one fully committed right now and doing technical development, I have a short, I have a small list of advisors. First and foremost, my previous co founder, Bob Bauer, he was at Willow Garage, and he's still advising me. But he's not a developer. So there's not too much for him to do right now. And funding, it's, it's completely bootstrapped right now, where, where we're profitable in the sense that I'm doing consulting work for customers, helping them both doing custom development, and then helping them figure out how to use Transitive as well, which is a good model, it's, as you can imagine, you don't get rich doing that. But it's, it's, it keeps the lights on. And one thing that I've seen with robotics companies, one, one way that they have failed, was that once they take money, they have to spend it once you once you because your investors want to use that money to hire a bigger team and scale faster, which, which means that you are lacking in a certain certain product, a certain business model, and a certain market fit. And if the market fit is not great, then you're trying to scale a business that was not meant to be, then you're making you're you're digging yourself a hole. And I really want to be certain that we have absolutely nailed market fit that we have gotten adoption that we have validated that this is something that people want before we take on money. And so yeah, I consulting that actually solves two problems here. Because, number one, yes, there's some money coming into the bank. And number two, we are actually getting validation from our customers. So this, this really lends itself to an iterative process that will make sure that we will build the right product for the market. Yeah. And I think we would get into race, no problem. We've had conversations with investors, they, they like the idea that we are basically indexing on the robotics market. And what they mean by that is, rather than so they have the choice of which companies to invest in. And they know that in robotics, the differences are stark, right? One company may be doing apple picking the other one maybe doing drywall construction, another one may be layout printing, and another one is doing delivery on sidewalk, another one on the street, if they don't know what to invest in, they all know that robotics is going to go up. And there's plenty of studies by IFR and all of those API research that tells us it's going to go through the roof. It's it's it's a phenomenal technology. And it's only sort of at the start, but they all want to invest and they don't know where to invest. So what they like about platforms like this is that they know that if robotics will succeed this company,

as well, gotcha. That's really cool. I really like bootstrapping, too, as a model. In general, I get it. Because one of the other things with if you take investment is you kind of commit yourself to a timeline that you will have some sort of liquidation event within some period of years, like typically less than 10, maybe closer to five years that the investor would like to see a return on their investment. And that's tough, because as you said, you have to like really commit to the current fit, and it lends itself to a certain type of product. Yeah, and that being said, like cobalt robotics, they're doing the investment route, and they're doing a phenomenal job as far as I can tell. Which is really interesting. If you position everything well. And it'll be cool to see kind of if you start accepting investment, and how that helps you scale. But once you found that really good market fit, that seems like a very good approach. So me.

Yeah, I'm keen to get into that point. And I do have a concrete plan on how to use that money in the employee to to foster third party development. I think there's, there's there's some really interesting models that you can that you can follow. Yeah, I think that there are a lot of capability. developers out there and I love this idea of, of the marketplace because it has inherent network effect. So more capabilities We have of high quality, the more people would want to use it, the more people who come to our, to our app store to look for capabilities to what people want to develop for it. And so that's, you know, that's the classical network effect that can that can actually justify exponential growth. And investors always want to see exponential growth. And I always ask them, Well, based on what, right, it's not, it doesn't just happen, right? There needs to be some sort of mechanism behind that. And it's not bouncing a QR code on the Super Bowl. In Super Bowl x, right? That's, that's a lot of money. No, you have to have something where the classical Atlas one, right if the ad plus one user has a better experience than all of the end before and makes the experience all of the end users before that better. That's network effect. That's why you have these these conversions effects where everybody will use the same social network, right? If, if you're the owner, if you and I go on to a social and create a new social network, and we're the only users Well, you don't have to have great conversations, but but it's not as useful as if they're more friends of ours on that same thing, right. And so for me, one that we had, it gets better. And I think, you know, marketing marketplaces have that dynamic, inherently built in, they're also hard to pull off. That's where money can be really helpful. But I think first and foremost, you have to make sure that it that it goes about it the right way and openness. And I think it's really the the key to all of that.

Christian's LinkedIn

Transitive Robotics' Website

Transitive Robotics on LinkedIn

Transitive Robotics' Twitter

Sign up for Transitive Robotics' Beta

Contact Transitive Robotics

Full-stack Robotics and Growing an App Marketplace

They Still Wouldn't Want to Make It

Business Models for Open Source Software

Openness and Market Fit

Tent Poles

Web Tools and Robotics

In this episode, Audrow Nash speaks to Youssef Benmokhtar, CEO of GelSight, a Boston-based company that makes high resolution tactile sensors for several industries. They talk about how GelSight's tactile sensors work, GelSight’s new collaboration with Meta AI (formerly, Facebook AI) to manufacture a low cost touch sensor called DIGIT, on the digitization of touch, touch sensing in robotics, how GelSight is investing in community and open source software, and Youssef’s professional path in several industries.

Start

Introducing Youssef and GelSight

Background of GelSight technology

What’s hard about it

Miniaturization

DIGIT sensor + Collaboration with Meta AI (formerly, Facebook AI)

Bringing DIGIT from research to a product

Having a camera in your finger

DIGIT in robotics applications

Meta’s plans with tactile sensing

On the digitization of touch

On the current state of haptics

Haptics to verify products and parts

Growing GelSight

Open source software and datasets

Slipping objects and heavy loads

Youssef’s professional path

Advice

Useful habits

Links and getting in touch

This is a conversation with Youssef Benmokhtar. Yousef is the CEO of GelSight of Boston based company that makes high resolution touch sensors. In this conversation, we talk about how GelSight's touch sensors work, their new collaboration with Meta AI, formerly Facebook AI, to make a new low cost sensor called DIGIT, the digitization of touch, touch sensing and robotic, how GelSight is investing in community and open source software. And we talked about Youssef's professional path in several industries. This is the Sense Think Act Podcast. Thank you to our founding sponsor, Open Robotics. And now here's my conversation with Youssef Benmokhtar. Youssef, would you introduce yourself?

Absolutely. Youssef Benmokhtar, CEO of GelSight and a high tech executive for the last 25 years, and excited about this interview and sharing my thoughts with the audience.

So GelSight is a MIT spin off, we have a very unique technology, we have an imaging based digital tactile sensing technology, which basically allows us and our customers to get a 3d information of anything that sensors in contact with, regardless of the material properties that is in contact with, that's the essence and the core of our technology.

Yeah, and so you have several products that all kind of work this way. Can you tell me at a high level, how they work? Like, how are you looking at? How are you understanding the deformation of the material and inferring touch information? Absolutely. So we have a unique proprietary elasomeric material. That mean, so it basically means it's a material that has kind of elastic properties where it can conform easily to surfaces that is in contact with, but still keeps some rigidity to it and has optical properties to allow basically to be integrated into an imaging system.

I'm picturing like a transparent silicon. Is that something like very similarly to

that? Absolutely, you're right, it's something very, so it's basically a polymer like material that lets lights go through it. So the essence of our technology is we have this material, we we coat it with a reflective surface. And we basically eliminated with RGB light, and place a camera behind it. And now we have the ability to, to take scans with these cameras, using a technique called photometric. 3d reconstruction. And that's basically the essence of gel sight technology.

Okay, so let's see, I've looked at some of the images, and they're quite cool from this technology. So it looks like you have, so you'll have the gel. And it looks like you have the light shining from different sides of the gel so that it gives you different shadows when the gel is pushed into by an object, is that correct? Or how does that work? Like? How do you set up the lights? Basically?

Yes, I think the lights is really more about getting the most uniform light possible. at different wavelengths we use, we use red, green, and blue. And making sure that the surface is uniformly illuminated, what each one of these different LEDs. And then we have a proprietary algorithm that's using the images captured at each one of these wavelengths to reconstruct 3d from that.

Why, what's the significance of different wavelengths? why did why are they necessary? Well, that's the principle of photometric 3d.

So that's something that I'm not an expert on. I think that's something I co founder could better explain, since he's the kind of the inventor and the one who developed the algorithms, but he's basically, you know, having three images done at different wavelengths. And knowing what those wavelengths are basically, being able to get those three different images, he's able to build an algorithm that that builds 3d information from that.

So I'm wondering, like, maybe this is wrong, but this is how I'm thinking of it. Perhaps having different colors lets you separate them in the image space, so you can get more dimensions like you get different views. So you get different views from each of the lights. Because each of them casts a shadow. And so if you have, say you do red, green and blue lights, now you can separate it into red, green and blue in your image. And you can kind of tease those apart. And that might be more useful for analysis and inferring 3d structure. Does that make it kinda like that? Yes,

it's kind of like that. Yeah, absolutely.

Cool. And you mentioned that the sensor is reflective. Is that just on the side that's touching some object? Or? Yes, that's

correct. Yes. Because basically, you need that, that reflect that reflection is what allows you to build that 3d scan, basically,

is it reflective, or just opaque for this guy is actually it is reflective? What, why do you want it reflective?

Because you want to basically, you know, keep the light within within the system, you know, again, in a very uniform way, so any kind of light dissipation outside will make the computation of the 3d very difficult. So Oh, so you want to have actually, you know, I think opaque is actually another way to look at it, but you need to think about it as it's very opaque, like light is really not going through.

Oh, so you mean, you're keeping all of the light within the system, it probably helps. You don't have to normalize it, or something, because you know, how much light is in it. And thus, you can have very accurate calibration, when it corrects

graduate, you avoid having to D noise and other things like that.

Okay. So, what I'm imagining that we have the lights behind a piece of gel, the gel at the front of it, and the part that we'll touch things, has some sort of reflective or opaque like thing that doesn't let light through. And you push into the top of the gel, the part with the reflective part, and it deforms it. So that, and then the gel deforms, and you see kind of like shadows or something within the gel that you use to infer how the 3d how the gel is being deformed in 3d, correct?

Now, let's see what So how long has this method been around? So the company was founded in 2011, I believe the first papers describing the concept were a few years before that. Yeah, so I want to say maybe 2009 is the first time that this kind of idea was described. And then our co founders, Ted Ellison, who is a professor at MIT, and his student, Chemo Johnson, wrote those papers, and got actually a lot of interest, not only from academia, but also from industry. And that gave give them an idea to say, Well, why don't we just, you know, found his company and and, and try to do something with that. Just respond to the, you know, what, what they felt was a strong demand signal.

Hmm. Why? I'm just wondering, what makes like, I'm sure that, like, so I have a little bit of experience in computer vision. And so thinking about like, Okay, we want robots to touch things. Now using some sort of gel and looking at how it deforms, to me, it seems like a, like, kind of an AI solution a lot of people would come to, in terms of how to understand touch. But I think that a lot of companies have tried this kind of approach. And I'm sure there's something hard about it, that has stopped a lot. Like the fact that this technology is taking so long, or is like just coming about now, what's been so hard about this technology.

I think what's been hard, you know, in, especially in the robotics, you know, field is that, you know, sensors were bulky, expensive, hard to calibrate, so hard to get actually repeatable performance over time. And, you know, really just low resolution. And when I mean low resolution, I mean, it's really low resolution. So what we're hoping at GelSight is that we are solving all of those issues, you know, with the, whether it's the DIGIT which is now you know, something that is, you know, completely, you know, mountable on a robotic hand, right. That's what it was designed for. With a price point of $300. That's what we announced with Facebook. It's available for $300

Yes, absolutely, absolutely. Right. And also, we because we're imaging based, right, the resolution of our sensor is actually directly linked to the resolution of the sensor, the cameras that we're using the image sensor we're using. So, you know, in the case of the digit, I believe it's a, it's a, you know, it's a VGA resolution sensor. So you getting, you know,

so for VGA, that's like, that's one of the display ports. And so is that 640 by 480, or 20? By 480. Correct? No, why would know that? But

okay, yeah, yeah. So so, you know, suddenly, you know, you have, you know, a million plus pixels to play with all the other technologies out there will give you in the resolutions in the 10s, or hundreds, you know, pixels for the same surface. And now, suddenly, you have this amazingly high resolution available to you at $300, in something that you can mount. So, you know, now, when you starting to think about applications, I think you have something that is actually even better than the resolution of a of a human finger. So it puts things in perspective, I think AI technology helps to, you know, bridge that gap that existed before. That's why That's why we believe it's exciting. And for gel sight. You know, we had also this reputation of being extremely expensive and big, you know, our standard products on the 20 or $30,000 apiece. But hey, we are, you know, we are a company that's able to do things, you know, for different applications. And in this $300, you know, sensor shows that, you know, we can do things at low cost.

Yes, and I want to talk about DIGIT quite a bit. But I want to still talk, I want to make sure I understand everything pretty well, before going into that. So one of the big things that was really hard with getting this into robotics previously was probably that the sensors were extremely bulky, as you were saying. And so how have you guys made it smaller? Like, how, how have you reduced the bolt?

It's a great question. You know, the, what Wally actually attracted me to jail side is that at the core of the technology, which which I described, and I think you have a good understanding of now, the three pieces, you should think about them as a modular approach, a design approach, so we can really be

modular and that they can each be improved independently,

correct, and then put back together as a system. So for example, into your question about miniaturization, we can design a gel material a polymer material that is specific to an application like robotics, right? And actually make them in any x, y, z dimension that you'd like, you know, it's almost, you know, infinite from end to end design space. And until that we can, you know, put in LED illumination, you know, in a fairly easy way, these are not big, you know, type of electronics. And, and then the challenge becomes more about the image sensor that you use, and most importantly, the lens and the field of view of the lens. And how close can you get it to, you know, to this elastomeric materials that we have, but you can play with that, depending on what the application isn't

work one more time really slowly. It's so Marek, or what is it? Elasto Merrick sorry. Elasto Merrick. Yeah, okay.

Yeah, I try to avoid using it from now on, but that's yeah, I'm just thinking they're so

clear pop deformable polymer, Paul, yeah,

it's a gel. You know, internally, honestly, we call it gel. The easy way that we think about it. But But then it's to your question. So what we did is that we basically found what is the right surface contact surface that would be attractive to robotics applications? And then how small how thin can you make that sensor by playing around with the, with the camera module, basically, and the field of view of that lens? And, and that's how you get something like DIGIT in that kind of size. There are ways actually, that we have developed a gel site to make it even thinner. So it we DIGIT is not the limit of what how small we can make things.

Yeah. Okay, so we keep mentioning digit. Let's, what is digit? Could you tell me a bit.

So DIGIT is a basically a tactile sensor is a digital tactile sensor imaging base. It's basically in a job site technology and in a meta design. Facebook made a I should say, sorry, has, you know, identify the need to really have this small, easy to integrate into robotics hands and fingers sensor to be able to develop, you know, AI algorithms for a bunch of different applications, object manipulation, you know, pose estimation and things like that. And basically, we are their commercial partners to bring bring that sensor to the world, we share the same vision, we believe that digital touch and feel is the next sensor to be digitized. To do that, yeah, that's right. And to do that, you actually just need to make it easy to use and affordable and small. So that's, that's, that's what we're doing together.

Yeah, so DIGIT is something that meta AI or formerly Facebook, AI, research, I don't know, organization or part of Facebook that does research and development for that. So they created DIGIT and they open sourced it. Correct? Correct. So it's freely available online. And then you guys now are taking this and building it at scale for this so that you can buy it, and it's out of kind of the research lab and into the world. What, so it was a research paper with maybe like a demo and this kind of thing. How long ago? How long ago did Facebook or meta release? This? The DIGIT design?

I believe it's over a year ago? I don't know the exact date. Wow. But over a year ago,

yeah. And then how did you guys? How did you begin a collaboration to work on this and produce it at scale? Like, how did that how did it occur, that they produce a paper and now you guys are actually manufacturing lots of these? Well, I think the you know, that's also a great question to ask Mehta, but my understanding is that some of the researchers at MIT, I had experience with GelSight during their academic research. And this is where they've learned about tactile sensing, they've learned about the GSI technology, and you know, kind of, you know, use that, that knowledge and gain expertise in academia, and wanted to actually focus on AI, I think more than, than building hardware. But such a hardware does not did not exist, and, and hence the need to design it.

And you mean, it didn't exist, because you guys have your product line, but it didn't exist in a way that was affordable to like, put 10 of them on to iRobot? Hands, one at each fingers end? Correct.

And in the right form factor as well, you know, we had was way too big, correct?

You know, so did they come to you? And they were like, Hey, we think it'd be really cool if you guys could work on this or, I mean, can I know these details? Or can you disclose them? I'm curious.

You know, the way I would answer this question is, I would say, we have a shared vision. And, you know, it was very easy to align on this partnership because of that shared vision. Yeah, and I think, you know, when two parties basically have the same ultimate goal, which is, you know, let's bring tactile sensor to the masses. And, you know, make it attractive for people to want to research with it. Then, you know, the rest actually became really easy to do on in a straight if part. And I believe, you know, and I hope that this partnership actually is just at its beginning, and more more will come from it.

Yeah. So So what is the nature of the partnership? Exactly.

So at the moment, it is really about GelSight, commercializing the DIGIT design, and that's already available for for people to purchase through our online portal. And it really, that's the extent of, of what we have agreed to do at this point. You know, as I said, you know, more is being talked about, I think now, you know, that we've reached that one.

Nice, what can you hint at what additional steps will occur or what they might look like?

Well, I think, you know, it's going to touch upon a few different things, you know, One of them is, we believe at Dell side that we can have a complimentary offering to the DIGIT sensor that will be attractive to the robot robotics crowd but also to the hobbyist market. And this is basically aligned with our vision that again, that we want tactile sensing to be available for any engineer, any technician, any hobbyist to use in their creative moments, for anything, and I think, to do that, you need to actually think beyond DIGIT and just offer a more universal sensor. So you know, something that could be mounted to robotic hand if you want to, but you can also use it as a digital microscope and a high school class. You know, something like that. Right? So

is that is that the kind of resolution you're getting with this approach? Or Yeah,

so what we're planning to do is actually having a much higher resolution sensor than digital. But a similar or even smaller form factor. And our partnership with DIGIT would be with with meta would be that we will probably want to leverage their open source community, and make our sensor compatible with with their Python community, you know, libraries and everything else like that. So that we have a kickstart, you know, when people want to use our sensor, they already have libraries accessible to them right away. So that this is where I see the next step being. So this is not something that is done today, maybe it's you know, I'm going to be scolded by, by my metal friends that I'm already talking about something like this. But, you know, regardless of that, you know, this is the direction we're taking. And this will happen, because it's a, it's something that we are jealous I believe in.

And I want to talk a lot more about the different libraries or ways of standardizing this. And it's quite cool if you make a sort of universal sensor, or you may maybe like a universal way of visualizing touch data and relating it. But going from actually creating, alright, so you had DIGIT was released as a paper, you guys agreed to start manufacturing them. I assume that since it was coming directly from the research community, that it wasn't built in a way that's exactly the easiest for manufacturing, or the best way for manufacturing. Can you tell me a bit about those challenges? Because I assume you have to take basically what exists? And then like, make it robust and make it reliable and make or whatever, grow it up for?

Correct? Yeah, absolutely correct. You know, it's always it's always over simplified, you know, how difficult it is to take something that looks like you can make 10 of them in a lab. And now suddenly, you need to be ready to make hundreds and 1000s That's exactly why I think gel sites it wasn't the right, you know, the right partner for meta, because we already know how to make products like these, we have an existing business, you know, we know how difficult it is to do in volume and the repeatable way. So yes, we did have to learn on the spot, really quickly about the meta design, and then how to improve, you know, some of the manufacturability issues that we encountered, as we were always rebuilding them, but that that was something that was expected. But this is also where Joel site expertise is, we, you know, we we had some quick cycles of learning that we did in the early phases, when you build the first ones, to make sure that we have a pretty high yield and building this gel material for digit, which mean pretty

2 posts with the computer vision tag