ep.352: Robotics Grasping and Manipulation Competition Spotlight, with Yu Sun

Kegan: [00:00:00] Hi, welcome to the Robohub podcast. Would you mind introducing yourself for us please?

Dr. Yu Sun: Sure, certainly. My name is Yu Sun. Uh, I’m a professor in the computer science and engineering departments at the University of South Florida. Um, in 2009, I started a lab called the robot perception and action lab, uh, and, uh, we have been working on, uh, several robotics, uh, research problems since then, uh, mainly related to, uh, robot perception and action, but steady the, um, the type of the lab.

Uh, so recently, uh, we, uh, have mainly working on. Um, domestic robotics and try to really, uh, focusing on cooking problems, uh, in a home kitchen. Um, so, uh, to me essentially, um, robot, if a robot can cook, uh, you know, home a kitchen [00:01:00] and, uh, we think, um, uh, the we’ll be able to do all the remaining tasks because, you know, cooking, you know, home kitchen, uh, is, is probably the most complicated, uh, domestic job.

Kegan: Is that where you see robotics really making a large impact in the near future is in the domestic, application?

Dr. Yu Sun: Yeah. In the, um, I think, uh, so first from my point of view, I think if a robot, um, how to grow, uh, If a robot can grow, and we have to have a lot of, uh, commercialization, um, capability and the future, right? So, uh, someone has to, uh, put a lot of investments into this technology and they certainly want to see, uh, revenue, uh, from this technology.

Um, I think everybody pretty much, uh, can see. Uh, if we can get the robots that can, [00:02:00] so robots, uh, in everyone’s home, uh, that kind of revenue would certainly be able to drive, um, the whole, uh, robotics research forward.

Kegan: Yeah, definitely, and talking of driving robotics forwards, uh, you you’ve worked with robotic competitions and you have a robotic competition that you help run, right.

Yu Sun: Yes, I do. Um, I work, uh, we, um, uh, a few researchers, uh, in the field and, uh, uh, starting, uh, started this competition in 2016. It’s called the robotics grasping and manipulation competitions. And, uh, so this year, um, this is the fifth time we’re going to run this and we’re going to run this at, ICRA and IROS.

Kegan: Cool. What motivated you to start this competition?[00:03:00]

Dr. Yu Sun: Well, it’s pretty interesting story at the very beginning. Um, I was tapping into co-chair, uh, there are competition committee. Uh, these are two of my friends and in 2016 for IROS, uh, at that time, uh, we, uh, tried to solicitate people who propose competitions, uh. At that time, don’t not allow people wanting to organize.

Um, so I’ve been asked to see, okay, is it possible, um, to come up and do some kind of competition? Um, so I, at that time, uh, thought about, uh, you know, there is a lot of research going on. For robotics, grasping and manipulation. And I also do research in that field and, um, and also at that time, uh, industry is a little bit, uh, tempted to apply robots to their applications, but they’re just not sure.

So I, I [00:04:00] found that was a good time to really connect with them both, the academia and industry online side. So what do we can do right over the years? So what do we have? What, what are mature and, uh, and also set that expectation, um, strict, right? So there’s a lot of, um, very fancy demo videos on YouTube and the claim robots can do everything.

And, um, you know, uh, people industry seeing that to really have a, a wrong, um, kind of expectation. So, uh, competition, I think serve, uh, uh, for both, right, for industry people to, uh, get to know what we do, which is mature and all show, uh, what we expect of all academia. Um, people really first demonstrate what they can do.

Um, second, uh, they basically use this as a, uh, as [00:05:00] a way to communicate and with industry, and also at the same time, getting some feedback from the industry to really see what kind of problem they are interested in.

Kegan: Interesting. Is that something you’ve seen in the last, you said that it’s been going for five years?

Dr. Yu Sun: Uh, has there been sort of that back and forth with industry and academia?

Yeah, sure. Suddenly, uh, we were really, uh, fortunate enough to get a lot of sponsors at the very beginning. So over the years, every year we have. Uh, some support from the industry. Um, usually there are multiple companies, um, either provide a price money or the even provide, um, uh, travel support.

Um, so we, we need to get a lot of, uh, support from the industry. Um, and, uh, we also seen a lot, the teams, um, can work their technology and commercialized their technology. And also the, uh, become a startup, right. So I think we [00:06:00] have, uh, at least two teams. I know two or three, I think maybe three teams, um, after the competition, the, a, uh, startup and they, basically, it has been quite successful.

And also I also have seen, uh, a small startup, um, participated in 2016 and they called to be picked up for more, uh, venture capital, uh, and, uh, they grow very fast from there. Um, so yeah, we, we, we do see a lot of progress.

Kegan: Wow. That’s great. I’m sure it, it’s something that can bring together people from a lot of different backgrounds and, and, and, you know, industry and academia and that seems awesome.

Uh, what contributions have you seen from the, the competition or what, what value do you see that the competition is bringing to you personally and maybe more from an academia side?

Dr. Yu Sun: yeah, from academia side, I think a lot of times, uh, you know, [00:07:00] we, each of us, uh, we’ll work on our own things.

Right? So, um, uh, we, we attend conferences. We kinda know what other people are doing. Uh, but we really don’t, um, have a, uh, a lot of ideas of how, and when this can be commercialized or what are the industry needs and really need. Uh, so for me, um, organize, organize this. I basically. Uh, starting to solicitate, uh, research problems from the research industry.

So before, uh, our first competition, I send out the emails to the robotics mailing list and asked people to, um, send us the problem they think are interesting and important and need to be solved. And, uh, we got a lot of replies and, uh, from that, so we come out from it’s about 30 problems, uh, I think exactly 30, uh, 36 problems.

Um, [00:08:00] and we formulate them and, uh, you know, can work into past, uh, pools essentially. We use that task pool for a few years. Um, and also at the same time, I, uh, reach out to, um, some companies and, uh, companies, including manufacturing companies and the resistor companies. And uh, many other companies and I went to their facility and talk to them, uh, for example:

I talked with, um, uh, manufacturing, uh, directors, um, at Samsung, um, they working with a small electronics and to try to make them and they have a lot of industry problems. And a lot of people were not aware of such as putting cables and, uh, uh, inserting, uh, handling flexible cables, uh, those kinds of things.

Um, so we incorporate them [00:09:00] into, uh, our competition tasks and, uh, um, yeah, because that’s the experience. I think I learned a lot and really got to better picture, uh, what the field looks like. Um, mature. And what are still, I think is in infancy.

Kegan: It seems like a great, um, opportunity to learn as well, too, to find a, find a competition, to sort of learn and, and try to reach for something.

Do you see that the people that are, competing in these competitions are the same set of people or, do you have new people coming in and, and, um, um, sort of entering the field through a competition, I guess.

Dr. Yu Sun: Right. Um, we do, we do see both. We see, uh, several teams and basically continue, uh, in the competition, you know, over a year.

And we can see, um, how they grow, and in the beginning of they can do a little. And, uh, um, their standing [00:10:00] in the, in the competition, uh, at very beginning is not as good. And then over the years, the, um, continually improve their technology and we can see their performance has improved dramatically.

Uh, every year we also see new teams and, um, you know, uh, teams coming in. And, uh, so first the, the, the, the, um, kind of, at a certain stage, right? So at a certain stage, the people they are ready to show what they can do. Um, so, um, yeah, so we, we always have a new people, uh, to reach that stage and, uh, and they want to participant.

Kegan: Do you have any recommendations for people that want to get started in competitions either, I guess, starting a competition or joining an existing competition and working on it?

Dr. Yu Sun: Yeah. Well, participating in our competition. Uh, I think, uh, there’s many different ways. Um, first we [00:11:00] have, uh, three different tracks this year and they usually, every year we have, uh, uh, different tracks, uh, this year.

Uh, we have this cloud, um, kind of, uh, uh, track and it’s, uh, it’s called the OCR talk, um, it’s uh, it’s uh, basically we provide a robotic platform for everyone on cloud. Um, so people can, um, program and, uh, create their solutions and submit, and there’ll be people basically can run, uh, the submission, on a standarized platform.

So that lowers the barrier, right? So you don’t even need to have a robotic system to participate. Um, and we all show in that track, we also provide simulator so you can simulate, and then we can, we provide the real robot, a system you can upload, or we can run that for you. Um, then, um, on the other side, if you [00:12:00] have a robotic system and, uh, then you can decide which direction is your passion, right?

So we have, uh, two tracks, uh, on service, uh, robotics and another is on manufacturing. So if you’re interested in, uh, domestic service, uh, so you can participate in that. And if you’re interested in manufacturing problems and we do have that track and Joe Falco take the lead on that side and from a, he is from the NIST.

Um, he, uh, viewed, uh, this nice task board. And then we send everyone to task board to for free, and then they can try those task boards, at their lab and. Um, you know, if they feel they can do whatever well, then they can move forward. Yeah,

Kegan: Why robotic, grasping, and manipulation, why that competition, um, specifically, and you’ve kind of touched on this

Dr. Yu Sun: Right. So grasping [00:13:00] manipulation is a very old problem, right? So we can, you can understand it’s, uh, it’s a fundamental problem at the very beginning of robotics.

People want to do something, right? So you you want to do something how to, uh. How to touch something you, how to change the environment, right? You’d be wanting to change the environment. You have to touch it, but how you use hand and the arm, uh, those kinds of things. And, uh, so that’s why it robotics, uh, grasping on the manipulation is very important.

It’s a fundamental problem. Um, it needs to be solved to me, and that’s the reason I get into this direction, because I think if robotics is wanting to go anywhere. Uh, the grasping and the manipulation, um, needs to be solved.

Kegan: Have you always seen yourself working in this area even more generally, you know, have you always seen yourself going into robotics and computer science or STEM or, or what’s your, what’s your history kind of.

Dr. Yu Sun: Oh, [00:14:00] okay. Yeah, that’s quite interesting. Um, I do have, um, quite a diverse background. Um, so, um, when I was doing my bachelor’s degree, uh, I, I mainly, uh, I majored in, uh, automation, uh, particularly in the direction of, uh, control theory. Uh, I also have a minor, uh, in mathematics. Um, so during my, uh, uh, bachelor’s degree program and, uh, I learned a lot, uh, control theories, obviously.

And, um, uh, mathematics, um, also, uh, like circuit designs and, uh, also sensory designs, uh, mechatronics uh, mechanical designs and, uh, uh, also even, you know, force analysis for. Building the [00:15:00] substructures. And obviously as at the time, um, you know, full controls, we helped to learn computers. So I learned a lot of architecture, uh, programs, and even I learned AI at that time.

Um, then I, uh, went to Japan and, uh, I got my bachelor’s and master’s degrees in China. Then when I went to Japan and to become a software engineer and, uh, started to, uh, do, uh, lab application development for, uh, Uh, for phones at that time, I think it’s was the infancy of a smartphones. They didn’t really have a smart phones.

They only have a very tiny screen, but they want to use a phone to, um, get access to internet and, uh, you know, access to some of the applications. Um, then I, uh, uh, got into, uh, University of Utah, uh, started, uh, [00:16:00] PhD program, um, uh, study and, uh, I in the computer science department and the school of computing, uh, uh, and I worked with, uh, John Hollaback and I started to learn robotics.

I think my diverse background helped me, uh, in robotics because, you know, when we deal with, uh, robots, we, how to deal with robots as a whole. And, uh, there are just so many different components, um, of a robot, right? So, and anything can go wrong. So if you really not want to deal with, uh, electronics, or you’re not want to deal with AI and not want to deal with programming.

It’s not going to go very well, because any part of this system can go wrong. Um, if you don’t really know what’s happening, uh, is going to take a lot of time, uh, to, to really get something working.

Kegan: Robotics is very interdisciplinary and you really have to think [00:17:00] about all of it.

Um, what did you do any competitions while you were a PhD student or, or.

Dr. Yu Sun: Yeah. Yeah. That’s a very good insight, in 2007. Uh, there is a DARPA, uh, urban challenge. So I participated in there, uh, at that time. Uh, I was, uh, I did my last year of, uh, I had my last year of my PhD program. Um, and, uh, Pretty much in charge.

I work with a lot of people, right? So we, we have a big team. I think we have about 10 people. And at that time I think, uh, Tom Henderson and, uh, mark, uh, minor, uh, they, they were the team leaders and the ricotta group of, uh, people from mechanical engineering, computer science, uh, participated in this. And I was mainly working on in the [00:18:00] section uh, vehicle detections and negotiating between vehicles, you know, how to follow the rules, the traffic rules of the intersection. Um, yeah, that’s, what’s quite impressive. I mean, I’m talking about the organization of that challenge. Um, I can also see, uh, from that challenge, the whole autonomous vehicle, uh, field has exploded, right?

So, uh, with that common goal. Uh, everybody, bought that common goal and that the research community, um, really, really spend a lot of effort to that, uh, to that, uh, you know, to try to fulfill that kind of vision.

Kegan: Is that? How you see. The robotic grasping competitions and stuff like that, playing out where that really helps then, you know, bring in this domestic robot that’s in your house, helping you with all these kinds of tasks and stuff.

Um, do you think like competitions are [00:19:00] necessary to kind of spur on that growth?

Dr. Yu Sun: Yeah. So I think after that too, they are many different competitions and even DARPA perhaps another competition that involves some components of, uh, manipulation. Um, uh, obviously the focus, a lot of walking, uh, but also, uh, in the robotics community.

Uh, many different competitions. A lot of them has to do with domestic service. Uh, every year you have, you can see, uh, some number of the competitions. Um, so what we want to do is, uh, we want to do something, um, consistent, right so we want to keep monitoring the progress. Uh, we want to really understand. Uh, what, what are, uh, kind of, uh, uh, what, are the research problems, what are not, what has been solved and, um, uh, yeah, I think, you know, as I mentioned before, uh, [00:20:00] this, I hope this serve as a, as a bridge between the academia and the industry.

And, uh, um, I always think, uh, all the technology we develop it in academia should be used in industry. Right. We should have, um, find a place to be used and, um, yeah, so that’s, that’s probably the driving force to need to try to try to, you know, organize it. It’s kind of like try to really bring what we can do to industry and to bring what’s the research problem, uh, in industry to academia.

And also , you know, you asked me about the whether, you know, DARPA challenge will be the vision. Um, to that I think it’s very difficult for us to really get to that scale. Right. So DARPA has, uh, probably unlimited budget. Uh, [00:21:00] they have about a billion dollars to spend.

Uh, each year we have some support from the industry, but we never will be able to get, um, you know, a significant amount. Right. And we certainly want to, um, if it’s possible to form some kind of alliance with key players in industry, um, uh, who real benefit from the growth of robotics and manipulation.

Um, they, if they want to contribute and I think this is a good place to contribute and, uh, so we can certainly organize those kinds of things, getting people together. And, um, yeah to make the event, uh, better and more inclusive.

Kegan: Yeah! In your past, you’ve worked on medical applications. Is there any overlap between the competition and any sort of medical or hospital application?

Dr. Yu Sun: Well, [00:22:00] uh, not really at this time. Um, so previously I mainly working, um, Uh, in virtual reality for, uh, medical applications? Um, so at that very time, I was mainly tried to, uh, help the surgeons to see better. Um, so the surgeons using endescope, um, for minimum minimum invasive surgery. And, uh, when they, when they use that, um, they.

Have to look at an overhead monitor and the move the, uh, endescope and can look from any directions and that any kind of orientation, so the picture on the overhead monitor, is really, kind of, this display of a certain angle. You don’t really know, right. So you really have this terrible hand-eye correlation and it, uh, makes training of surgeons [00:23:00] very difficult.

So we want to solve that kind of problem. So we basically, develop a kind of, um, uh, transparency display and, uh, um, convert the image from the endoscope cameras, um, uh, into, uh, morph image. And it can be projected on the abdomen to generate that transparent effect to give the natural hand-eye correlation to.

Kegan: Yeah, that’s awesome. To really help support the surgeons doing their work. Do you think we’re anywhere close to having a robotic surgeon, um, to a robot actually doing the physical grasping and manipulation that’s required for surgery?

Dr. Yu Sun: So first I think certainly, uh, in the future, um, I can see a lot of, uh, uh, research progress in many different labs in academia doing a lot of, uh, surgical robotics research.

And also in the [00:24:00] industry Intuitive surgery is the leader in the industry. Um, making a lot of progress, which there are DaVinci system and each year the, they come up with some kind of, uh, uh, automated, uh, procedures. And, uh, it’s always interesting to see, um, uh, how much progress they are making as usually very, very impressive.

So for me, um, I mainly. In the last few years, mainly focusing on robotics cooking. So I mentioned in the beginning, um, uh, because as I said, In the domestic environment, cooking is probably the most challenging task. It’s involves so many different things and, uh, um, you know, a person you can think about when you’re cooking, you have to allocate, recognize objects, allocate them, and really figuring out how [00:25:00] to pick them up, holding them, holding tools.

And you also have to do all kinds of different manipulations, right? So cutting and pouring, a very diverse environment . There was, uh, a number of, uh, objects that you have a different kinds of shapes and tools, right? So God, there’s so many tools, so many kinds of sorts in the kitchen and, uh, um, and also, um, you know, manipulations, right.

So how to handle them properly. Yeah, so it’s a very challenging. So for me, um, we really working on three different aspects of this. The first thing we work on is, uh, knowledge representation and retrieval, so we try to really figure out how to represent this complicated and complex information, when you really do the, do the cooking.

So for us, we understand the recipes. [00:26:00] We even watch a YouTube instructional video. We understand how to do the cooking and how we can, we’re not to information to robot. Right? So how a robot will be able to gain cooking. So that’s one of the things we’ll work on. The second thing we work on is a multi objects grasping.

So not only, um, you know, we, we grasp one object at a time, right? So we do that, but a lot of times, um, uh, at our home kitchen, for example, we want to pick up strawberries from a box. Um, you know, we don’t pick one by one. We pick multiple strawberries from the box at one time. And at the same, for eggs the two or three at the same time, and it’s not only in homein kitchen also industry like logistics, you see people.

Uh, know, stick they pick couple of apples from one bin being put into another [00:27:00] bin. You don’t really see, you know, a worker, a human worker pick apple one by one. You know, if that is our number is for example five, right? So they usually pick two and three and then you get to five, um, and also in manufacturing, right?

So we pick up multiple screws at the same time and put them one by one. Um, so that’s one of the reasons robotics is not as efficient as a human yet because when they do, when people do picking, we do, we pick multiple at the same time, when robot doing the picking at this time, even the best robots, they still can only do pick one by one.

And with that is by default, it’s like, uh, you know, two or three times less efficient. Uh, there are. Um, so that’s, that’s one of the things, um, I’ve worked on. Um, and I called this, um, [00:28:00] cookie jar problem because you know, when you have a kid, okay, you can go to the cookie jar, pick some cookies. You don’t really see a kid just go in and pick one cookie.

Right. So you can have a kid basically go, goes in, pick a couple of cookies out. And, but if you have a robot and say, okay, robot, you can go to pick up cookies. Just pick one cookie at a time. If there is a competition obviously, then the robot will not win. The kid will win that one. Right. So, uh, that’s uh, interesting direction we currently working on and, uh, another direction we’re working on is, uh, motion generation.

And we’re particularly working on pouring, uh, because pouring is the most frequent manipulation, uh, motion in cooking and pouring, not only for liquid, like water, oil, honey, [00:29:00] syrup, those kinds of liquid, but also, you know, rice and beans, flour, uh, those kinds of things, and also even large chunks, right?

You caught something you put into a bowl and you don’t want to pour the whole thing into, uh, into a pan, portion of that, uh, how will you control, how will you do the, do the pouring? Um, so we’ve been working on that. Um, so we had a finished, uh, pouring liquid that we do 30 mile, um, precise, uh, pouring liquid, precisely.

And then all we mainly focusing on pouring, um, big chunks, of objects precisely.

Kegan: Interesting. And you mentioned that pouring precisely I could imagine, a lot of challenges in, in doing, in tackling these problems. What do you see as the largest problems right now?

Dr. Yu Sun: Yeah, there are many different challenges. And, uh, while the things is, um, for some reason, we as [00:30:00] humans understand that that dynamics fairly well. We can kind of, we can predict what’s going to happen. So we basically compensate our emotions, in terms of how everything’s going to fall or how many are going to fall?

We have a pretty accurate prediction. We basically recovered the pouring motion before everything, um, everything basically poured out. And, uh, it is kind of, reversible, right? So if you pour more than you need, and for some of the cases, you will be in trouble, right? So, you can’t, just control the pouring speed that you have something falling back into the cup.

So it is more of, uh, irreversible, uh, kind of motion. So we have to do, pretty good job on predicting what is going to happen. The second challenge we also deal with is generalization because we don’t want to. Um, you know, [00:31:00] learn something that’s only working for this kind of container or this kind of object.

Uh, we want to really learn a skill, a polling skill, follow robots that can basically use any containers and import anything. Um, yeah, so we, we basically, uh, try to use practice, uh, to, to solve this kind of problem because when we, uh, do pour unknown things, we kind of practice a little bit. And then after a couple of times in practice, we will be able to do fairly well.

So we try to use practice to, uh, really modify I’ll, uh, I’ll transfer our models to this new situation, uh, without, um, a lot of failures. Right. So we are going to make sure the practice is also, uh, doesn’t really have failures, but not necessarily to have the best performance from that but at least not have failures.

Kegan: You mentioned that out of this [00:32:00] couple of students have started, uh, companies or, or have gone into industry.

And then I also noticed looking into your background, you have multiple patents. Could you kind of talk about how, um, how, sort of the balance between papers and patents and how patents maybe as a grad student, you know, when to patent or why to patent, or even how to patent your work that you’re working on.

Cause like, if you’re working on these competitions that are related to industry and industry is looking to use it, um, sort of how does patent play a role in there?

Dr. Yu Sun: Right. So. patent is quite important. If you want to, uh, start thinking about, uh, uh, after you graduate, uh, you may want to, uh, form a startup for example, and, um, uh, during your, um, academia year and, uh, on the doing your research, you probably want to think about building your own IP portfolio, right?

So, um, that gave you [00:33:00] a solid foundation. Uh, um, start to do, uh, this, uh, entrepreneurship, try to try to try to form a startup. Uh, so I think from that point of view, um, it’s quite, uh, quite important for everyone who wants to go to a startup route, um, in terms of starting, um, uh, when to, uh, apply for a patent.

I think after the time you have a technology, it works and you know, is, you know, it is innovative and that’s at the time you should, um, think about the patent and usually university has a patent and licensing office. So you should talk with those patent and the licensing office people, and they can help you to apply.

And they usually, um, however, pair, IP attorneys and they will help you to write it up and draft the claims and everything. [00:34:00] Um, yeah. Then they will apply.

Kegan: Yeah. we’re getting close to time here, but I like to finish with this. So what are you most excited about moving forward with your research, with anything? What are you most excited about?

Dr. Yu Sun: So I’m pretty excited about robotics in general. I think this is the right time for robotics and in terms of, uh, making progress in research and also, um, apply the research outcomes into industry and really getting the technology, uh, used.

Right. So, uh, before. Uh, robotics are seen as, uh, kind of, uh, demos, uh, fun and people will get inspired to get into the engineering. Um, we see a lot of the larger companies have tried and got into robotics field and, uh, but we don’t see a lot of follow-up. Um, but now I think we really could reach to that kind of maturity and lots of technology, I [00:35:00] think, um, has found the way of industry like in the logistic industry.

And I believe in very short period of time, a lot of, uh, uh, technology development, for service robotics, domestic robotics will be, uh, picked up by industry will be used to develop, uh, uh, robotic system that can be really used in our home and to help us to do our domestic works.

Kegan: Awesome. Well, that was great. I loved hearing your insight on everything and your experiences and thank you for taking the time to do this chat .

Dr. Yu Sun: Yeah. Yeah. That was my pleasure.

Kegan: Thank you.