Duckietown Competition Spotlight

Abate: [00:00:00] Hello everybody. This is Abate. Next week is ICRA and a core part of this year’s conference is going to be robotics competitions. So we’re going to deep dive into some of the influential robotics competitions out there. with a couple of short spotlights on several different ones this week, we’ll be talking to Dr. Liam Paul, the co-founder of the Duckietown competition.

Hey Liam, welcome to Robohub. Could you give us a little bit of background about yourself?

Dr. Liam Paull: Sure. My name’s Liam Paul. I am a professor at the university of Montreal. I’m also the president of the Duckietown foundation and one of the co-founders of that project.

I did my PhD in in new Brunswick. And then I did a postdoc in MIT, which is where this Duckietown thing started. And now I’ve been a proffer about five years or so.

Abate: Yeah. So today actually we really want to dive into the Duckietown competition. Um, so could you give us a little bit of [00:01:00] information about how you started it, what your motivations were?

Dr. Liam Paull: Yeah. So, I mean, the Duckietown thing is something that’s kind of taken on a life of its own, for sure. It started as a class first and foremost, it was used for educational purposes, but then at some point along the way we thought that it would have also value as, as a scientific benchmark. And so we started to see if we could reformulate and repurpose the platform to host these these competitions.

And the first one was it NeurIPS. And I want to say 2018 and then we’d done at least one at ICRA and a few at NeurIPS and it’s sort of something that’s really really gathered the motivation, I think really is it’s all about trying to rigorously benchmark robot algorithms. And this is a pretty, it’s actually a pretty [00:02:00] hard task.

A lot of robot research is done in some specific lab with a very specific setup and is quite hard to reproduce. And so we wanted to build a very standardized but very accessible platform that people could easily get their hands on, easily, put their algorithms on, and that we could somehow like compare a wide variety of algorithms in some.

Standardized and like fair, fair way.

Abate: Yeah. So what’s the exact challenge that they’re competing for and how does it, how does it look?

Dr. Liam Paull: Yeah, so this is evolved over the years, but the basic premise is, is, is. Mostly the same. So as part of the Duckietown platform, we have the cars, which are these little, little cars that you can build, but then there’s also an environment in which they operate.

And the environment is [00:03:00] made up of like yoga mats and duct tape and signs that we’ve like printed and stuff. Um, but the idea is that it’s very standardized and very reproducible. To you or me, like, it looks like a small city. Like it’s a very simplified view of a city, but it’s something that approximates somehow a small city and the challenges are very in complexity, but mostly involved the robots navigating in this city.

And we can. we can vary the complexity by having different typologies of the city intersections. We can have different obstacles, we can have other vehicles. And so the complexity can really grow. Um, but the most kind of like basic, fundamental, like a PR thing that an agent should be able to do is like drive down the road in the city, avoid obstacles and stay in their lane kind of thing.

Abate: Yeah. So what was the motivation behind the name Duckietown?

Dr. Liam Paull: That’s an [00:04:00] interesting, that’s an interesting one as well, actually. So like the ducky not too many people know this, but the ducky branding, not only does it, it predates the Duckietown project, but it also has an ICRA connection. So the other co-founder of the project his name’s Andrea Censi and now he’s at ETH Zurich.

And I think the year before Duckietown started, he was… I forget exactly what the title was, but it currently this push for everybody to submit videos and they were going to try and stitch all of these videos together to make like a promo video for the, for the conference. And Andrea came up with the idea that every video should have a rubber ducky in it sort of for a number of reasons.

But I think that partially, it was like for scale and also for like some kind of coherence between the different videos. So they could do like fun cuts and stuff in between the videos, but somehow the branding of it just like totally exploded. And then when we started this project, [00:05:00] like before anything else, the one constraint was that it had to have like rubber duckies involved.

I… I don’t know… Just sort of happened that way.

Abate: Yeah,

no, it’s great. Because when you like grounded in something, that’s like a fun concept it makes it much more engaging for people to, to want to do it.

Dr. Liam Paull: Yeah. And there’s also an aspect of I mean, my view is that some, some robotics in particular is kind of portrayed in a certain way.

And I think that like Hollywood has something to do with this. Scary, not like either it’s like Terminator are going to come and kill you, or it’s scary in the sense that it’s going to take your jobs or whatever. And I think, yeah, in the end part of, part of the motivation behind this like kind of fun, playful kind of thing was that we would break this mold a little bit of trying to make something that’s super fast and super scary and super big or whatever that maybe this would appeal to.

Different people who are maybe not [00:06:00] attracted to the, like, let’s build a big, fast, scary thing, but instead, you know, also want to be able to like express themselves somehow through like through their work. And I think yeah, I think that’s also been, been part of it and has been kind of, kind of successful.

Yeah.

Abate: And so the competition now it’s been running for, is it a decade or two?

Dr. Liam Paull: It’s not, no, it’s not that long. I think it’s, I think the first iteration was in 2018. So I think we’re at like, around the five-year mark. Um, but the five-year time. Yeah. The first iteration of the class at MIT would have been something around 2016.

I think. So the project itself has probably been around for six or seven years, but the, the, the competition itself maybe only four. Hm. Yeah.

Abate: So what have been some of the, the real-world benefits that that you’ve seen out of the competition?

Dr. Liam Paull: Yeah, that’s a great question. I mean, I think with Roberta [00:07:00] robotics, I mean, part of our you know, philosophy is that robotics should involve a robot.

And I think especially in more recent past, there’s been this huge trend towards like machine learning and deep learning. Type of algorithms. And I think these algorithms certainly have huge potential, but when you try and put some of these algorithms on robots, you see some of the, some of the kind of nitty-gritty details that you maybe didn’t think about really have a big impact, you know, like how the latency of your system you know, how it’s dealing with.

asynchronous singles versus synchronous signals, like treating time, you know, non-model defects and things like friction and slippage and things like this. And so for a lot of the folks, I think like the real, like the real world benefit has been that, wow, they really have gotten an appreciation for just how, how tough it is [00:08:00] to, to build these systems.

And then when you look at like what, although we’re not all the way to having, you know commercial, autonomous vehicles. I think that you can get some kind of an appreciation for just how remarkable, what has already been achieved. You know, it really is when you consider all the different pieces that have to work together and how robust they all have to be.

Yeah.

Abate: And I can imagine over the years, you know, different technologies have taken more interest in the eyes of roboticists and that the approach that the different people competing has changed quite a bit as well.

Dr. Liam Paull: Oh, for sure. Yeah. At the beginning, I mean, we very much saw quite traditional what I would call like classical.

Not because they’re old, but just because it’s like the way that things used to be done, kind of like stacked that had the very standard abstractions of like, you know, perception and state estimation and planning and control, and now much more we’re seeing competitors [00:09:00] try and solve this. And to end machine learning type of techniques, whether they’re based on more like imitation learning paradigm leveraging data that we make available, or whether they’re using the simulator primarily.

And just trying to do like reinforcement learning stuff. Style approach and then transfer their agents that the real, the real robot, these, I, I still think it’s like remains to be seen at this point at this juncture, like which one is actually better at solving the task. But one thing that’s definitely true is that the students in the competitors seem to be much more they find the, like, I think the machine learning kind of approach is more appealing at this point.

It’s kind of like this hot, hot topic, I guess.

Abate: Oh, that’s interesting. So it’s maybe it’s more appealing, but maybe it’s not necessarily as of right now resulting in a more success for the competitors.

Dr. Liam Paull: Yeah. I mean, the way that I view it, especially like from a say a scientific standpoint is that [00:10:00] especially in this environment, everything’s really well specified a really well engineered solution with very little learning is going to be very hard to be.

you know, the potential benefits of more learning based systems or that they should be able to be more robust to varying conditions, be able to generalize in sort of a more, a smooth, more S more easy way to different environments. And so, yeah, it’s, it’s not, it’s not always easy. It’s not always easy to like we have we have to think carefully about even just what the metrics we’re going to use.

to compare, you know, these different algorithms, like, is that just the one that, you know, drives the fastest? I’m not sure that’s the best, you know, that’s the best metric. Um, there’s all these other components about like robustness and ability to generalize, to different like scenarios and things like that.

And in those cases, the [00:11:00] machine learning solutions maybe do a bit.

Abate: Yeah, no, it’s an interesting point about overfitting your solution to specifically the competition environment, other than like whether or not that’s something that you really want to do as a judge to say whether or not this is a better solution, it might be better in this competition because it was faster… but should the obstacle course change a bit, the topology change, now, maybe it’s not so robust.

Dr. Liam Paull: I think this is actually the central challenge in building robot competitions. It’s very difficult to build a robot competition. That’s like not hackable in some sense that you can’t win by just really overfitting to the specifics of that particular of that particular setup.

And so, yeah, I mean, I think. You hit the nail on the head there it’s this is the big challenge for sure. And [00:12:00] trying to build like really good robot benchmarks.

Abate: Yeah. So as you, as you think about next year’s competitions have you guys ever considered maybe doing a not releasing the map and having it be a bit more of a surprise and have a little more randomness associated?

Dr. Liam Paull: Yeah. So we, we have, we have typically done that. Like, we have a sort of a, like a, a validation set that people get the results and they can see everything. And then what they’re actually evaluated on as like a held out test set that they don’t see. But what we are thinking about doing this year, So typically what we’ve done is we’ve had sort of like maybe two or three main challenges, like the lane following challenge, the lane following with obstacles, challenge, and the lane following with intersections challenge or whatever.

And each one of these challenges is, has its own defined metrics. Like how long you survive for, or how far you’re traveling in a certain amount of time, sort of like standard stuff. What we’re going to do this [00:13:00] year is we’re going to. Have a sequence of levels effectively that are just increasingly complex and increasingly difficult.

And each one of them maybe has like some, some level in terms of the metrics that you have to achieve in order for it to be passed. But what we’re trying to do is actually alleviate the overfitting to any specific kind of like specific task and stage. You’re going to have too much more. B building a general purpose agent that’s able to do reasonably well in a, like a really like varying like environments of varying complexity and increasing complexity.

And so I, this is our, this is our next attempt, actually at kind of trying to alleviate this, like over-fitting to the specifics of the, of the the specific like challenge or whatever.

Abate: Thank you.