In this episode, Audrow Nash speaks to Youssef Benmokhtar, CEO of GelSight, a Boston-based company that makes high resolution tactile sensors for several industries. They talk about how GelSight's tactile sensors work, GelSight’s new collaboration with Meta AI (formerly, Facebook AI) to manufacture a low cost touch sensor called DIGIT, on the digitization of touch, touch sensing in robotics, how GelSight is investing in community and open source software, and Youssef’s professional path in several industries.

Start

Introducing Youssef and GelSight

Background of GelSight technology

What’s hard about it

Miniaturization

DIGIT sensor + Collaboration with Meta AI (formerly, Facebook AI)

Bringing DIGIT from research to a product

Having a camera in your finger

DIGIT in robotics applications

Meta’s plans with tactile sensing

On the digitization of touch

On the current state of haptics

Haptics to verify products and parts

Growing GelSight

Open source software and datasets

Slipping objects and heavy loads

Youssef’s professional path

Advice

Useful habits

Links and getting in touch

This is a conversation with Youssef Benmokhtar. Yousef is the CEO of GelSight of Boston based company that makes high resolution touch sensors. In this conversation, we talk about how GelSight's touch sensors work, their new collaboration with Meta AI, formerly Facebook AI, to make a new low cost sensor called DIGIT, the digitization of touch, touch sensing and robotic, how GelSight is investing in community and open source software. And we talked about Youssef's professional path in several industries. This is the Sense Think Act Podcast. Thank you to our founding sponsor, Open Robotics. And now here's my conversation with Youssef Benmokhtar. Youssef, would you introduce yourself?

Absolutely. Youssef Benmokhtar, CEO of GelSight and a high tech executive for the last 25 years, and excited about this interview and sharing my thoughts with the audience.

So GelSight is a MIT spin off, we have a very unique technology, we have an imaging based digital tactile sensing technology, which basically allows us and our customers to get a 3d information of anything that sensors in contact with, regardless of the material properties that is in contact with, that's the essence and the core of our technology.

Yeah, and so you have several products that all kind of work this way. Can you tell me at a high level, how they work? Like, how are you looking at? How are you understanding the deformation of the material and inferring touch information? Absolutely. So we have a unique proprietary elasomeric material. That mean, so it basically means it's a material that has kind of elastic properties where it can conform easily to surfaces that is in contact with, but still keeps some rigidity to it and has optical properties to allow basically to be integrated into an imaging system.

I'm picturing like a transparent silicon. Is that something like very similarly to

that? Absolutely, you're right, it's something very, so it's basically a polymer like material that lets lights go through it. So the essence of our technology is we have this material, we we coat it with a reflective surface. And we basically eliminated with RGB light, and place a camera behind it. And now we have the ability to, to take scans with these cameras, using a technique called photometric. 3d reconstruction. And that's basically the essence of gel sight technology.

Okay, so let's see, I've looked at some of the images, and they're quite cool from this technology. So it looks like you have, so you'll have the gel. And it looks like you have the light shining from different sides of the gel so that it gives you different shadows when the gel is pushed into by an object, is that correct? Or how does that work? Like? How do you set up the lights? Basically?

Yes, I think the lights is really more about getting the most uniform light possible. at different wavelengths we use, we use red, green, and blue. And making sure that the surface is uniformly illuminated, what each one of these different LEDs. And then we have a proprietary algorithm that's using the images captured at each one of these wavelengths to reconstruct 3d from that.

Why, what's the significance of different wavelengths? why did why are they necessary? Well, that's the principle of photometric 3d.

So that's something that I'm not an expert on. I think that's something I co founder could better explain, since he's the kind of the inventor and the one who developed the algorithms, but he's basically, you know, having three images done at different wavelengths. And knowing what those wavelengths are basically, being able to get those three different images, he's able to build an algorithm that that builds 3d information from that.

So I'm wondering, like, maybe this is wrong, but this is how I'm thinking of it. Perhaps having different colors lets you separate them in the image space, so you can get more dimensions like you get different views. So you get different views from each of the lights. Because each of them casts a shadow. And so if you have, say you do red, green and blue lights, now you can separate it into red, green and blue in your image. And you can kind of tease those apart. And that might be more useful for analysis and inferring 3d structure. Does that make it kinda like that? Yes,

it's kind of like that. Yeah, absolutely.

Cool. And you mentioned that the sensor is reflective. Is that just on the side that's touching some object? Or? Yes, that's

correct. Yes. Because basically, you need that, that reflect that reflection is what allows you to build that 3d scan, basically,

is it reflective, or just opaque for this guy is actually it is reflective? What, why do you want it reflective?

Because you want to basically, you know, keep the light within within the system, you know, again, in a very uniform way, so any kind of light dissipation outside will make the computation of the 3d very difficult. So Oh, so you want to have actually, you know, I think opaque is actually another way to look at it, but you need to think about it as it's very opaque, like light is really not going through.

Oh, so you mean, you're keeping all of the light within the system, it probably helps. You don't have to normalize it, or something, because you know, how much light is in it. And thus, you can have very accurate calibration, when it corrects

graduate, you avoid having to D noise and other things like that.

Okay. So, what I'm imagining that we have the lights behind a piece of gel, the gel at the front of it, and the part that we'll touch things, has some sort of reflective or opaque like thing that doesn't let light through. And you push into the top of the gel, the part with the reflective part, and it deforms it. So that, and then the gel deforms, and you see kind of like shadows or something within the gel that you use to infer how the 3d how the gel is being deformed in 3d, correct?

Now, let's see what So how long has this method been around? So the company was founded in 2011, I believe the first papers describing the concept were a few years before that. Yeah, so I want to say maybe 2009 is the first time that this kind of idea was described. And then our co founders, Ted Ellison, who is a professor at MIT, and his student, Chemo Johnson, wrote those papers, and got actually a lot of interest, not only from academia, but also from industry. And that gave give them an idea to say, Well, why don't we just, you know, found his company and and, and try to do something with that. Just respond to the, you know, what, what they felt was a strong demand signal.

Hmm. Why? I'm just wondering, what makes like, I'm sure that, like, so I have a little bit of experience in computer vision. And so thinking about like, Okay, we want robots to touch things. Now using some sort of gel and looking at how it deforms, to me, it seems like a, like, kind of an AI solution a lot of people would come to, in terms of how to understand touch. But I think that a lot of companies have tried this kind of approach. And I'm sure there's something hard about it, that has stopped a lot. Like the fact that this technology is taking so long, or is like just coming about now, what's been so hard about this technology.

I think what's been hard, you know, in, especially in the robotics, you know, field is that, you know, sensors were bulky, expensive, hard to calibrate, so hard to get actually repeatable performance over time. And, you know, really just low resolution. And when I mean low resolution, I mean, it's really low resolution. So what we're hoping at GelSight is that we are solving all of those issues, you know, with the, whether it's the DIGIT which is now you know, something that is, you know, completely, you know, mountable on a robotic hand, right. That's what it was designed for. With a price point of $300. That's what we announced with Facebook. It's available for $300

Yes, absolutely, absolutely. Right. And also, we because we're imaging based, right, the resolution of our sensor is actually directly linked to the resolution of the sensor, the cameras that we're using the image sensor we're using. So, you know, in the case of the digit, I believe it's a, it's a, you know, it's a VGA resolution sensor. So you getting, you know,

so for VGA, that's like, that's one of the display ports. And so is that 640 by 480, or 20? By 480. Correct? No, why would know that? But

okay, yeah, yeah. So so, you know, suddenly, you know, you have, you know, a million plus pixels to play with all the other technologies out there will give you in the resolutions in the 10s, or hundreds, you know, pixels for the same surface. And now, suddenly, you have this amazingly high resolution available to you at $300, in something that you can mount. So, you know, now, when you starting to think about applications, I think you have something that is actually even better than the resolution of a of a human finger. So it puts things in perspective, I think AI technology helps to, you know, bridge that gap that existed before. That's why That's why we believe it's exciting. And for gel sight. You know, we had also this reputation of being extremely expensive and big, you know, our standard products on the 20 or $30,000 apiece. But hey, we are, you know, we are a company that's able to do things, you know, for different applications. And in this $300, you know, sensor shows that, you know, we can do things at low cost.

Yes, and I want to talk about DIGIT quite a bit. But I want to still talk, I want to make sure I understand everything pretty well, before going into that. So one of the big things that was really hard with getting this into robotics previously was probably that the sensors were extremely bulky, as you were saying. And so how have you guys made it smaller? Like, how, how have you reduced the bolt?

It's a great question. You know, the, what Wally actually attracted me to jail side is that at the core of the technology, which which I described, and I think you have a good understanding of now, the three pieces, you should think about them as a modular approach, a design approach, so we can really be

modular and that they can each be improved independently,

correct, and then put back together as a system. So for example, into your question about miniaturization, we can design a gel material a polymer material that is specific to an application like robotics, right? And actually make them in any x, y, z dimension that you'd like, you know, it's almost, you know, infinite from end to end design space. And until that we can, you know, put in LED illumination, you know, in a fairly easy way, these are not big, you know, type of electronics. And, and then the challenge becomes more about the image sensor that you use, and most importantly, the lens and the field of view of the lens. And how close can you get it to, you know, to this elastomeric materials that we have, but you can play with that, depending on what the application isn't

work one more time really slowly. It's so Marek, or what is it? Elasto Merrick sorry. Elasto Merrick. Yeah, okay.

Yeah, I try to avoid using it from now on, but that's yeah, I'm just thinking they're so

clear pop deformable polymer, Paul, yeah,

it's a gel. You know, internally, honestly, we call it gel. The easy way that we think about it. But But then it's to your question. So what we did is that we basically found what is the right surface contact surface that would be attractive to robotics applications? And then how small how thin can you make that sensor by playing around with the, with the camera module, basically, and the field of view of that lens? And, and that's how you get something like DIGIT in that kind of size. There are ways actually, that we have developed a gel site to make it even thinner. So it we DIGIT is not the limit of what how small we can make things.

Yeah. Okay, so we keep mentioning digit. Let's, what is digit? Could you tell me a bit.

So DIGIT is a basically a tactile sensor is a digital tactile sensor imaging base. It's basically in a job site technology and in a meta design. Facebook made a I should say, sorry, has, you know, identify the need to really have this small, easy to integrate into robotics hands and fingers sensor to be able to develop, you know, AI algorithms for a bunch of different applications, object manipulation, you know, pose estimation and things like that. And basically, we are their commercial partners to bring bring that sensor to the world, we share the same vision, we believe that digital touch and feel is the next sensor to be digitized. To do that, yeah, that's right. And to do that, you actually just need to make it easy to use and affordable and small. So that's, that's, that's what we're doing together.

Yeah, so DIGIT is something that meta AI or formerly Facebook, AI, research, I don't know, organization or part of Facebook that does research and development for that. So they created DIGIT and they open sourced it. Correct? Correct. So it's freely available online. And then you guys now are taking this and building it at scale for this so that you can buy it, and it's out of kind of the research lab and into the world. What, so it was a research paper with maybe like a demo and this kind of thing. How long ago? How long ago did Facebook or meta release? This? The DIGIT design?

I believe it's over a year ago? I don't know the exact date. Wow. But over a year ago,

yeah. And then how did you guys? How did you begin a collaboration to work on this and produce it at scale? Like, how did that how did it occur, that they produce a paper and now you guys are actually manufacturing lots of these? Well, I think the you know, that's also a great question to ask Mehta, but my understanding is that some of the researchers at MIT, I had experience with GelSight during their academic research. And this is where they've learned about tactile sensing, they've learned about the GSI technology, and you know, kind of, you know, use that, that knowledge and gain expertise in academia, and wanted to actually focus on AI, I think more than, than building hardware. But such a hardware does not did not exist, and, and hence the need to design it.

And you mean, it didn't exist, because you guys have your product line, but it didn't exist in a way that was affordable to like, put 10 of them on to iRobot? Hands, one at each fingers end? Correct.

And in the right form factor as well, you know, we had was way too big, correct?

You know, so did they come to you? And they were like, Hey, we think it'd be really cool if you guys could work on this or, I mean, can I know these details? Or can you disclose them? I'm curious.

You know, the way I would answer this question is, I would say, we have a shared vision. And, you know, it was very easy to align on this partnership because of that shared vision. Yeah, and I think, you know, when two parties basically have the same ultimate goal, which is, you know, let's bring tactile sensor to the masses. And, you know, make it attractive for people to want to research with it. Then, you know, the rest actually became really easy to do on in a straight if part. And I believe, you know, and I hope that this partnership actually is just at its beginning, and more more will come from it.

Yeah. So So what is the nature of the partnership? Exactly.

So at the moment, it is really about GelSight, commercializing the DIGIT design, and that's already available for for people to purchase through our online portal. And it really, that's the extent of, of what we have agreed to do at this point. You know, as I said, you know, more is being talked about, I think now, you know, that we've reached that one.

Nice, what can you hint at what additional steps will occur or what they might look like?

Well, I think, you know, it's going to touch upon a few different things, you know, One of them is, we believe at Dell side that we can have a complimentary offering to the DIGIT sensor that will be attractive to the robot robotics crowd but also to the hobbyist market. And this is basically aligned with our vision that again, that we want tactile sensing to be available for any engineer, any technician, any hobbyist to use in their creative moments, for anything, and I think, to do that, you need to actually think beyond DIGIT and just offer a more universal sensor. So you know, something that could be mounted to robotic hand if you want to, but you can also use it as a digital microscope and a high school class. You know, something like that. Right? So

is that is that the kind of resolution you're getting with this approach? Or Yeah,

so what we're planning to do is actually having a much higher resolution sensor than digital. But a similar or even smaller form factor. And our partnership with DIGIT would be with with meta would be that we will probably want to leverage their open source community, and make our sensor compatible with with their Python community, you know, libraries and everything else like that. So that we have a kickstart, you know, when people want to use our sensor, they already have libraries accessible to them right away. So that this is where I see the next step being. So this is not something that is done today, maybe it's you know, I'm going to be scolded by, by my metal friends that I'm already talking about something like this. But, you know, regardless of that, you know, this is the direction we're taking. And this will happen, because it's a, it's something that we are jealous I believe in.

And I want to talk a lot more about the different libraries or ways of standardizing this. And it's quite cool if you make a sort of universal sensor, or you may maybe like a universal way of visualizing touch data and relating it. But going from actually creating, alright, so you had DIGIT was released as a paper, you guys agreed to start manufacturing them. I assume that since it was coming directly from the research community, that it wasn't built in a way that's exactly the easiest for manufacturing, or the best way for manufacturing. Can you tell me a bit about those challenges? Because I assume you have to take basically what exists? And then like, make it robust and make it reliable and make or whatever, grow it up for?

Correct? Yeah, absolutely correct. You know, it's always it's always over simplified, you know, how difficult it is to take something that looks like you can make 10 of them in a lab. And now suddenly, you need to be ready to make hundreds and 1000s That's exactly why I think gel sites it wasn't the right, you know, the right partner for meta, because we already know how to make products like these, we have an existing business, you know, we know how difficult it is to do in volume and the repeatable way. So yes, we did have to learn on the spot, really quickly about the meta design, and then how to improve, you know, some of the manufacturability issues that we encountered, as we were always rebuilding them, but that that was something that was expected. But this is also where Joel site expertise is, we, you know, we we had some quick cycles of learning that we did in the early phases, when you build the first ones, to make sure that we have a pretty high yield and building this gel material for digit, which mean pretty

high yield and building this Gel it well, like, you have to manufacture gel, and then you can cut it up and you want it to be I don't know, get a lot of it at once or what. And some of it has quality issues or

Well, you know, like like anything like any any part of a product that you that you make, you know, need to pass, you know, some some specifications. And, you know, this gel material, which is kind of the secret sauce, you know, that we have, and we build it for different applications, you have to make sure that it has the right optical density that you want that it's cut and, you know, in the right shape, and that the coatings are, you know, opaque, like we talked about before to make sure that the light uniform uniformity is going to be good that you don't have, you know, an unacceptable level of cosmetic defects that will affect image quality, you know, those kind of things. So, this is what we're, you know, experts in and it took, you know, took us a few cycles Learning to get right so that we can start making hundreds of these are 1000s of these.

Okay, so it's interesting. A would expect that that's the case with the challenges of making a lot of them. Were there any particular things that came up that, you know, you had to change? From the DIGIT design?

You know, I it, candidly, the biggest problem we had were components supply. You know, in these days, you know, like, I'm sure you've been hurt hearing around shortages. And what was the challenge, I think for both, both parties actually was to find replacements compared to the digits original component choices, just because of supply issues. So I would say the biggest challenge we had was honestly, around that is actually, you know, finding alternative chips and things like that, qualifying them and getting our hands on it, honestly, has been the biggest challenge.

What an interesting thing. So you had to like, find it called for a certain part and you had to swap it in, use something else and validate that it did the exact same thing,

correct? Correct, then, you know, and develop the firmware because, you know, the original firmware was meant for another chip, so you have to make some modifications and that kind of stuff. That honestly was the biggest challenge. The rest of us, okay.

Okay, so, just and I haven't I don't think we've said DIGIT is like, how large is DIGIT? And what's it? What's its case? Basically?

Oh, I think it's about, say, three quarters. Yeah, it's a little over an inch deep. I think it's 28 millimeters, if I remember correctly. Just over an inch. Yeah. And I think I wouldn't need to get a specs on that. But I want to say it's maybe half an inch, and little bit over maybe three quarters of an inch in width. Anyways. Yeah. And then in height, or then it's probably another inch in height. Yeah, slightly below that. Probably. So it's

kind of like an extra thick human finger. Correct. In terms of size, and it has a little pad on it. That has the gel.

gel material and the gel material. It's kind of it's like framed by the case material. Correct. So you put something on the gel material. And it will measure the deformation there. There's the edges on the same side as the gel.

Correct? Yeah, there is a there is a bezel I think the bezel is like one and a half or two millimeters. I know fairly thin. Yeah. And the the gel itself is you know, I forgot what it is like three to four millimeters probably high. It kind of a slightly dome, like, you know shape just like your finger would. Basically yeah, yeah.

Yeah, like the tip like the last DIGIT of your finger. Correct kind of thing.

Yeah, you know, what's another thing maybe that would be interesting to you. The way I look at these tactile sensors like DIGIT is almost thinking about having a camera inside your finger. So imagine if you had a camera inside your finger and it would actually be able to see anything that the forms your skin as you're touching things. That's pretty much what DIGIT is. Yeah, you know?

Uh huh. So just so I can picture the internals of it a little bit so you have the gel surface? Yep. And if so, then we have the camera and the camera could be so if the sensors facing down and the gel is at the bottom, the camera is above looking at the gel Correct? Do we have it does the camera touch the gel? Or is it just like a sensor pad that touches the gel or how do we get our

know the camera is is offset from the gel there's a the gel is mounted you know think about it mounted on a like a plastic window if you'd like okay and the camera is behind that I guess small distance you need to have a minimal you know distance between the camera and the

yeah gel. I was wondering if you could like use the gel as a lens or something like this but I imagine because it's not like a pinhole camera you would get a lot of light going and bouncing all over. So be very it'd be like a very blurry image as opposed to a crisp image kind of thing. So you need that separation as opposed to not having a lens or having a sensor pad directly into the gel.

You need to be able to image that area right so you need a little bit of distance to be able to see that surface.

Now for robotics just like wondering, how do you connect to it? How does it? How does it work as a sensor? What exists for libraries? Like all these things?

Yeah, so, you know, so connect connectivity is done through a USBC type, you know, of connectivity. And then, you know, you basically have

high data flow or something through USB C, or was there any reason to choose it other than it's a nice ports, the

most common port, and also, you know, robotics actually does care about real time information. And so you can run it, you know, easily at 30 frames a second, you know, you know, it has even the capabilities, I think, should run a little bit faster than that. But you want to have real time and USB was was, is basically a good way to do that. And then you basically, you know, you know, program, the sensor using all the firmware that meta has made available on PI touch. So that's how basically, you, you connect it to your your computer, have access all the Python libraries, and you're ready to go.

What, so I don't I don't know too much about the available libraries for the what do they let you do? Like, you assume you can get 3d deformation, but is there like, I mean, there's for vision for computer vision, there's like open CV, or open computer vision library, I believe, started by Intel. And it's a big open source library has been around for a long time. And so that lets you do a lot of basic computer vision things. So if you're working with a camera, like almost right out of the box, after you figured out the install, you can like detect faces, or do object tracking or something like this, is there any such things that let you kind of identify what you're touching or identify properties of what you're touching? Yet, and these libraries,

you know, I think the meta team is, is kind of adding things on a regular basis, but it's all the basic things you get are, for example, you know, the ability to just, you know, see an image, you know, so even even a 2d interpretation of the image or 3d representation of the image, see where the sensor has been touched in. So if you look at the surface area of the sensor, which area has been touched, you know, which is an interesting information, you know, in robotic application, that they typically called pose estimation. And from, that's the kind of the very basic information that they give, you know, the purpose of mirrors, as far as I understand it, is to then in turn this into creating datasets of deformations of that gel based on the type of objects that the robotics hand or gripper is, is manipulating. So, you know, you kind of train models to say, you know, let me grab this example, if you say, this is, this is a cable, right? And you basically have your robot kind of holding it in all kinds of different, you know, orientations, and different forces and things like that, and all that is captured in their, you know, in their datasets. And it's used to basically, you know, create AI to basically say, Well, this is how I recognize that this is a cable, you know, touching the sensor? And then can you do things like that, for example, like, you know, reorienting that cable and space, because you want to move it from A to B, you know, those kinds of things. So, I Yeah, you know,

to me, I'm saying it's similar to like, image segmentation, or something like this. So we do all of the, when we had like image net, or some sort of thing that has a bunch of labeled images. So we have cameras, the cameras take pictures, and then we say, Okay, that's a cat, that's a tree, that's what whatever. And then you do that a lot of times, and then maybe one approach is you can have a more complex image, and you can identify the different parts that are in it. And I think now they do that typically with like bounding boxes, that they just slide over the image or something, to try to figure out where in the image might be a cat, but they're getting more and more sophisticated. And I'm definitely not aware of this, but are aware of the like cutting edge of this research. But it seems kind of like that, where the intention is to start with very simple identifications of things. And then perhaps eventually a robot can grab a chunk of stuff from a messy, messy desk and identify that it might have a few cables and whatever. So it could look for things or

I think you're right, I think, you know, the Holy Grail is actually to be able to provide that kind of intelligence to robots the same way that you've you've mentioned about image recognition, you know, forecast And dogs and other things, right? It's kind of the same idea is, you know, if I want to be able to do complex object manipulation in a robotic application, or you know, to do, you know, kidding, right, and other things like that, what to do kidding, for example, if you want to, you know, take, you know, you have a box of a whole bunch of different objects, you don't really know what those objects are, but you want to basically put a certain class of objects and put them in another box, you know, but this time classified by the type of objects. This is something that actually very difficult to do. The word Kidding, kidding. Yes? Like a kit?

You No kidding. So like you're assembling kits, correct thing? Correct. Okay. I'm not heard that before. Yeah. So if you're, if you're grouping objects by type or something, for example, I, yes, go ahead.

You know, the other thing could be if you're trying to just assemble things that today, I really only done by humans, because humans have the kind of dexterity you need to do those kinds of assembly of complex assemblies, you know, handling, you know, cables, or handling, you know, types of fibers and things like that are just, you know, really hard to do for a robot, you need to have that kind of sensitivity feel that humans have. So providing this kind of perception capability to robots may allow them to start doing some of these more complicated tasks.

Gotcha. And I imagine that it's not terribly interesting to be like, Okay, now we can grab this and identify it as much as it is that you can use it in like a sensor fusion kind of context where you say, I think that I'm grabbing this, and I grabbed something. Does it agree with what I expect to feel given this? I think that would lead to more interesting applications of this kind of thing.

You're absolutely right. You know, one thing that's really interesting is that machine vision, and computer vision is extensively used in in robotics today. But one thing that happens is, especially in a robotic hand, for example, is once the hand grabs something, he can't see anything is occluded. It's no idea what it is, right? So I think complementing vision, with tactile information, is something that's going to be extremely useful.

Yeah. Now, so one thing that I'm wondering as a roboticist with this is, so say, I want to like screw in a screw with this. So I want to pick it up with my fingers, and I want to screw it in. So maybe I can pick it up and identify it. But I think that highly like I'm imagining the gel is quite soft for this. And that deaf formation might make it hard to control whatever you're picking up. Are there? Is this kind of it? Would you think that this is correct? And are there strategies for using these sensors with tasks do you like, identify it, and then move a little lower and then to factor or higher, so that you can identify what it is and then pick it up with rigid grippers that you kind of know what you have? Or is it is it more complex and how you grip it.

So, you know, first of all, you know, the gel that we use in robotics application today is actually not that soft, at least not to the human touch is actually it would feel, you know, it will have this kind of elasticity feel to it, but it's still pretty hard to actually, you know, if you're grabbing a, you know, a screw, for example, as you as you mentioned, you would actually not really need to, you know, pass it on to another robotic arm. To do something else, you actually will have a very good understanding of what the screw is how you know, how hard you actually grasp, it will actually give you more information about it's about its shape, we actually have done some experiments that you might find interesting to us are actually tactile sensor as a hardness a relative measurement of hardness, because same similar to, to human skin, right? And human finger, if I'm pressing on something that is really, you know, soft, you know, I really don't have a lot of feedback on my, on my skin, it's actually very, very light. But if it's really a hard material, I my my my skin will get indented really fast. Well, it's the same thing that happens to us with our sensor. So when you're holding an object and actually pressing it, pressing it against another, it actually also gives you gives you that that really interesting information about the potential hardness or softness of the of the object that is touching. So it's really look

at the deformation of both objects, the gel, and the objects are kind of infer the hardness of the object is

it the jail by itself would tell you how hard the object is that that whatever you're holding is in contact with, you know, just just by, again, you know, just imagine we have a video, I think, on our libraries of a gripper holding a fork. And then we have hardness samples. And as you push the fork into the harness sample, you see how the tactile sensor deforms, depending on the hardness of that sample, and the hotter it is and the more deforms, right? So this is actually how you get a relative hardness measurement.

does this require that you have 3d knowledge of whatever the object is that you're doing? Or at least you know, like, say the width of it, or how far it is between the two sensors are? How much of it should be pushed into the sensor?

Um, I think 3d helps, but I don't think it's even necessary.

Oh, it's, I think, intuitively see how it works. So

well, because you know, all I really care about is, you know, for example, is the depth of the indentation. But not necessarily the exact shape of the object, you know, is how deep do you get into the gel their formation, for example. But if you have the shape, then you can also say, oh, now I'm starting to recognize that that's the tip of a, of a fork, for example. Because you can add that to object recognition.

So you think if we're picking up like a screw or something, and we have, say you have the two digits, or whatever would be next after DIGIT with higher resolution, and you have it between the two sensor pads, between the gel on the both of the sensors. Now, you can kind of see how it is pressed into both of the sensors. And you can infer in 3d how the screw is oriented, correct with respect to the sensor, and then from there, you can figure out how it's oriented with respect to the rest of the robot, because you know, how the fingers are positioned with respect to the rest of the robot, this kind of thing?

Absolutely, that that is possible today, we actually have, we're doing that at a job site, we, we build, basically, a 3d point cloud model of an object that the that we want to our grippers to, to manipulate. And based on just the contact area, we actually able to infer the orientation in 3d of the of the 3d Point Cloud, which is, which is a really cool thing to do. Yeah.

So when you say point cloud, you mean, mesh, kind of thing. So it's not you're using your sensor to get the 3d information. And it's creating like a mesh representation of the object that you see. And like a 3d graphics or not, I was thinking LIDAR initially, which you get all the points, and then you can

Yeah, it's not Yeah, I think a mesh is a good, good explanation to it for it.

Then, so, you know, what, meta or formerly Facebook? Like, so I know, they had the one robot that would turn and look at you. And it was used for like video calls and things like this, like an echo with a neck kind of thing. Amazon Echo with a neck? Do you know what their interest is in tactile sensing and find manipulation? Are they going to get more into robotics or any ideas here?

I think it will be a bit of a relation of mine. Well, it's definitely started as research. But I think the the idea of providing this intelligent perception tool to robots is what is compelling to meta, as you imagine, you know, robots being able to do more and more tasks. It became obvious to them that it needed to have additional sensing capability. And that vision and sound was just not enough for robots to be able to, to do the kind of complex tasks that the human does. And, you know, I think, you know, in their very grand vision, I believe that Robotics has a place in their Metaverse story, in the sense that, you know, you have to imagine a

1 post with the haptic tag