On Friday, June 21, 2024, the ITCS, a career fair and tech conference, took place in Hamburg. The Bit-Bots were invited to set up a booth and give a presentation, and we gladly accepted.

Our booth was quickly set up, and the first robot was assembled (we always have to detach the head for transport) and started. It didn’t take long for the first curious visitors to arrive. It quickly became clear that the questions here delved much deeper into the technology. We were all the more pleased to visually demonstrate what our robot can do and to explain the software behind it.

However, we stood out a bit as exhibitors because we are not a company looking for new talents. Nevertheless, we took the opportunity to do some networking ourselves.

Around 2 PM, things got a bit exciting as it was time to prepare the robots for our presentation. At 2:30 PM, we gave our talk on why playing soccer with robots is beneficial. The quick answer: there is more than one good reason.

Here are a few more impressions:

On June 1, 2024, the Deutsches Elektronen-Synchrotron (DESY), a scientific research center in Hamburg, opened its doors to over 15,000 guests. Various research groups showcased their work, and we were there with our robots.

Our booth was located right at one of the entrances, so many visitors passed by us. As a result, we constantly had a crowd of people in front of our booth, including many wide-eyed children who were amazed. After all, you don’t see a soccer-playing robot every day.

One of our robots ran continuously throughout the day and was repeatedly used for demonstrations. We showcased walking, kicking, and cheering. For a second robot, we only ran the vision system, which was then visualized live. Children were frequently allowed to interact with the robot, positioning it in various poses and having their parents take pictures with it.

We also learned a few things about our robots that day: our demo robot ran for nearly 8 hours straight (with several battery changes, of course), which is impressive considering they usually only need to last two 10-minute halves before taking a break. Additionally, our vision system occasionally recognized children as kid-sized robots.

Overall, we had a lot of fun and thoroughly enjoyed interacting with the visitors.

From April 17 to April 20, 2024, the first German Open (GO) since 2019 took place in Kassel. Naturally, we participated with our robots again. We set off for Kassel on April 16. Most of us (Laura, Jörn, Jan-Niklas, Jan, and Lea) took the train, which was about an hour late. Timon and Flo had better luck with a rental car, including the robots and other competition equipment, and welcomed us at our accommodation with a warm meal.

Satisfied, we played a billiard match (as there was a large billiard table in the living room) and then jumped into bed. The next day, we headed to the fair early. First, we set up our workspace, unpacked the robots, and took a tour of the venue. Afterward, we diligently worked on previously discussed tasks related to both software and hardware, which were always visible in the form of labeled notes.

The first day of the GO, also known as Set-Up Day, is primarily for setting up and acclimating the robots to the new environment. Accordingly, we adjusted the “mental model” of our robots regarding the field size. There was much to do, and the day flew by quickly.

On the second day, there were three games in total. In the very first game, we achieved the goal of scoring a goal. Tragically, our robot fell in the middle of the field and, confused by the field’s symmetry, wasn’t sure which side was its own. Unfortunately, it made the wrong decision and scored an own goal. In the second half, however, we managed to score again, ending the game with a 1-1 draw.

The second game of the day was a friendly match against an SPL (Standard Platform League) team. In the SPL, all teams use the same robot model, a NAO. Despite the different rules in the two leagues, it is possible to arrange matches with a few tricks. The game ended in a 0-0 draw.

To protect some material, we played the first two games with only two robots, but for the third game, we added a third robot. This paid off, as we emerged victorious with a 2-0 win. That evening, we cooked for ourselves again and played some billiards.

On the third day, we won all our matches within our league and played another friendly match against the SPL. By now, we had become more coordinated in preparation and execution, partly due to well-defined roles during the game. We also used a new structure for pre- and post-game discussions, which helped us greatly. Earlier, we had noticed a grill in the garden of our accommodation and, with the host’s permission, invited another humanoid team (the WF Wolves) for a joint barbecue in the evening. Besides billiards, we also played brought-along card games and had good conversations.

Unfortunately, our accommodation wasn’t available for the entire time, so we had to pack up and take everything to the fair on the fourth morning. Thanks to good preparation the night before, the morning was very relaxed, and Timon delighted everyone with homemade pancakes. Well-strengthened, we tackled the next games at the fair, which we won again within our league. Between games, there was also a lot of programming to make the games run even more smoothly.

Thanks to good monitoring during the games (each team is allowed to receive data from the robots via a laptop), errors could be quickly located and sometimes even fixed within a game. We spent the evening in a hotel more centrally located in the city.

Saturday was our last and most important day. It began with a friendly match against the SPL. During halftime, the SPL teams switched, allowing us to play against the SPL league champion in the second half. This same team had won against the world champion of the Humanoid League at the World Cup in Bordeaux last summer. Therefore, we didn’t expect to win and were quite satisfied with our defeat.

Around 1:30 PM came the crucial part of the day: the final match of our league. Our robots were starting to show the wear and tear from the previous days’ games, falling more frequently when kicking but managing to get up quickly. In the first half, we managed to take the lead with a goal. In the second half, it took much longer to get the ball in front of the goal, as we fell with each kick and had to chase the ball anew. Finally, our opponent managed to score the ball we had played close to the goal line, resulting in cheers from both teams. The opposing team celebrated their first goal in the competition, and we celebrated increasing our lead. There were no further goals, and we won the final 2-0!

After the final, we had something to eat. Strengthened, we started packing. We took a short break to attend the award ceremony and collect our trophy, applauding the other winners vigorously. It didn’t take long to load everything into the car and head home. Our conclusion of the German Open: We achieved a best performance in our team’s history and are very satisfied with it.

The so named “Schnupperstudium” offers students a one-week glimpse into university life during the holidays. In addition to presenting various research fields, there are discussions with professionals and students, as well as a sample lecture and project work for the students. The projects provide a practical introduction, allowing students to continue working on them every day of the trial study week. During this week, which took place from March 25, 2024, to March 28, 2024, our team accompanied a project group and presented our robots in research demos to other groups.

At the beginning of the week, our project work involved so-called Cozmos, small robots that can be programmed via an app using a language similar to Scratch. These robots handle small cubes that can be lifted or rolled by them. The students were tasked with teaching the robots to play soccer through increasingly challenging tasks. By the beginning of the third day, a small game with three robots on each side was possible.

At the end of the week, the students worked with our large competition robots in a simulation and managed to reach the ball and score goals. Additionally, video recordings were made for the final project presentation.

Both the students and the supervising members of our team had a lot of fun throughout the week. The students reported that they wished they had more time to accomplish more in the project work. Due to Good Friday, the trial study program was shortened by one day. We were also able to help some students discover that they enjoy computer science, with one participant wanting to start studying right away. We will definitely be part of the Schnupperstudium again next year.

Until then, this month we also have the German Open (GO) coming up, where we will compete against other teams in Kassel.

From Friday, March 15th to March 22nd, 2024, we had our Integration Week. The goal of this week was to have as many members as possible in the lab during this time and to see how the individual components of a robot interact with each other. This year, we aimed to improve our walking and work on our hardware. With some snacks, we met in the lab for eight days and accomplished a lot together.

By the end of the week, we were able to walk very reliably, which allowed us to successfully move on the field and dribble the ball towards the goal. Additionally, our “Teaching Mode,” which allows positioning the robot’s motors at the push of a button, was completed. This mode was also used for a small photo session. We improved our localization and extrinsic calibration and resolved some sensor issues with our foot pressure and position sensors, opening up further possibilities for enhancing our walking. We also identified and solved a problem with the time stamps in the camera driver. As a result, we now know exactly where we’re looking, even if the robot is shaken vigorously.

Furthermore, we worked on parts of the hardware that often break. A new case for the robot’s computer was designed, improving its mounting. Moreover, we performed general maintenance on our robots, which, for example, made the walking more stable.

In the end, we all left the Integration Week very satisfied.

In November, the Bit-Bots were invited to the AI Specialist Day at the VHS Stuttgart to give a presentation and lead a workshop. So, we (Flo, Timon, Jan-Niklas, and Lea) set off very early on Wednesday, November 29, 2023, to Stuttgart. At 7 am, we all gathered at Hamburg Central Station. We would have preferred to take a later train (which we had also booked), but the railway had already canceled it in advance (as they did with our return train too).

Due to the early hour, the journey started off quite sleepy and generally quiet. This was probably because we hadn’t noticed during the reservation that it was a quiet car. As time passed, the drowsiness wore off, and the snow, surprisingly still present in Hamburg in the morning, diminished as we traveled south.

With about half an hour of delay, we reached Stuttgart and found ourselves amidst an attraction: Stuttgart 21. Although the new train station is not yet completed, on the approximately ten-minute journey out of the construction site, we were informed about the progress of the project. We had discussed beforehand to first go to the VHS Stuttgart, where the lecture would take place, and then check into the hotel we had booked for the evening.

At the venue, we were warmly greeted and led to our workshop room. Upon arrival, robots were unpacked, prepared for action, and the later presentation was practiced one last time. It didn’t take long before the first interested visitors found our room, which was well-attended at the beginning of the workshop.

In the workshop, we explained how we solve certain tasks, passed around parts of our robot for attendees to touch, and also demonstrated our robot in action. For this, we set up a piece of grass and placed our robots on it. Simultaneously, we showed on a screen what the robot was currently seeing, how it recognized balls and lines, and what it thought about itself. Subsequently, we demonstrated walking and standing-up motions.

The latter was particularly well-received, which is why we started our presentation in the large auditorium with a live demonstration. Afterward, we explained what the RoboCup is and how we use artificial intelligence in our software. The presentation can also be found on YouTube at the following link: https://www.youtube.com/watch?v=8aAUJXZS0xE

After the presentation, we exchanged a few words with our contact person, Monika Kurz. The official part was now over, and we set out in search of something to eat, finding a nice restaurant based on recommendations. Well-nourished, we were ready to head to the hotel, which was not too far away. Upon reaching the room, it was already quite late, so we all went to sleep relatively soon.

The next day, starting with a delicious breakfast, there was some time to explore Stuttgart a bit before heading back in the early afternoon. We strolled through Christmas markets in the city center, discovered a paternoster in the town hall, and walked to a small lake where there were many ducks. Finally, we bid farewell to Monika, who was delighted that our presentation and workshop went so well.

The return journey was relaxed but also productive, making the one-hour delay not as noticeable.

In the quarterfinals on Saturday, we played against the CIT-Brains team from Japan and unfortunately lost. Considering that CIT-Brains were the world champions last year, we had expected this outcome and were not too disappointed. Nevertheless, we managed to prevent a few goals. With that, we were eliminated from the competition, but it was our best result since the inception of the Bit-Bots (excluding the virtual competition during Covid-19). However, that wasn’t the last game for our robots at this RoboCup. The hosts and world champions Rhoban invited us to a friendly match at short notice, which was accompanied and filmed by Arte. We agreed and tried to get everything ready for the game as quickly as possible. During the match, our robots were able to showcase their current abilities, although there were some last-minute issues. The game was then called off after the first half, as Arte had limited time, and both teams wanted to conserve their materials. Now our robots could truly rest and recover from the games.

We still wanted to witness exciting games, so the idea arose to arrange a friendly match between the Humanoid League and the Standard Platform League (SPL). In the SPL, all teams play with the same robot model, a “Nao”. This brings some different rules and a slightly different playing field (for example, they play on relatively short grass with a standardized ball, while we can play on long artificial turf with a variety of balls). However, due to their humanoid appearance and compatible size, it was still possible to have the current world champions of both leagues (Rhoban and B-Human) play against each other. We watched with anticipation as the Naos (B-Human) won on the field of the Humanoid League.

For the remaining time of the RoboCup, we engaged with other teams, leading to an exciting experiment: we equipped our robots with the legs of the NUBots and were actually able to make them walk.

With that, the week came to an end, and so did our time in the exhibition hall. Everything was packed up and taken to our accommodation. The next day, we moved to another part of Bordeaux: the university. Here, the symposium, a scientific conference associated with the RoboCup, took place. In addition to attending different talks, we read posters and also presented one ourselves. In the afternoon, we were invited to visit Rhoban’s lab and saw some intriguing robots. With the symposium, the RoboCup concluded, and the following days were used to explore Bordeaux a bit more and head to the sea.

The RoboCup started for us with successful preparation days on Tuesday and Wednesday. Already in the first hour after entering the event hall, our first robot stood on the field and successfully took its first steps. Since a lot of effort of the AG, especially by our PhD student Marc, went into improving the walking and making it more robust last year, we had to make almost no adjustments to cross the field at top speed of 25cm/s. At the same time, preparations continued in the rest of the software. We configured and stabilized the stand-up, which usually does not work well in new environments because it has to be adapted to the robots and the grass in the field. We also developed a new component for our behavior: Since we currently can’t shoot the ball reliably, we decided to just dribble instead. Of course, this had to be practiced diligently. We also tested our localization. Localization is the part of the robot software that uses the lines recognized by our image recognition system to determine where it is on the playing field. Since that didn’t work reliably yet, we initially decided not to use the localization and just dribble the ball towards the goal.
On Thursday, the games started at nine in the morning. As usual, RoboCup starts with the so-called drop-in games. In these games, five robots compete against five robots, the special feature being that all the robots come from different teams. So the goal is for the robots to try to cooperate with other robots while playing. However, this generally doesn’t work very well and led to a lot of chaos on Thursday.
Things were a bit more structured in the afternoon during the first games of the group phase. This RoboCup has the peculiarity that there are not only one, but two group phases, with the results of the first group phase determining the distribution of groups for the second phase. In our first games, we already had all four robots ready to go and could see them running on the field at the same time. This in itself is a peculiarity in RoboCup, where the robots, which frequently fall and lose parts, need constant repair. Even though we didn’t manage to score a goal yet, we were able to defend our own goal a few times and keep the opponents from scoring. In the evening we played a short friendly match against the multiple world champion and this year’s RoboCup organizer “Rhoban” from Bordeaux. The game was also filmed by the TV station Arte and used for TV footage from RoboCup.
On Friday, after the end of the first group phase, there was a deciding match between us and the other German RoboCup team from Wolfenbüttel. The match was very nerve-wracking, as our two teams have been working together for a long time and know each other well, and of course neither team wanted to lose to the other. The game was finally decided in a penalty shootout, where we won with a final score of 5-3.
However, we and the robots were not allowed much rest, because already twenty minutes later the next game against the Chinese team TH-MOS was scheduled. Directly after that there was another match against the NUBots from Newcastle, Australia, which ended our second group stage. Both games went 0-0 and we eagerly watched the match between our two opponents to see if we would advance to the quarterfinals. The NUBots then lost the game, which means that we and TH-MOS still get to play on. Our next game will then be on Saturday in the quarterfinals against the CIT Brains. The Japanese team is the reigning world champion – so we don’t expect to win against them. Of course, that won’t stop us from doing our best and making more improvements in our software.
Apart from the games, RoboCup also consists of other aspects, especially the exchange with other teams. We like to take the opportunity to have the teams from all over the world in close proximity and exchange ideas about problems and their solutions, look at each other’s robots and fantasize about improvements to ours or analyze the other teams’ games together from the sidelines. The whole thing was even supported on Thursday evening by provided buffet, cheese, wine and French quark balls. We are looking forward to more opportunities to exchange ideas with other teams in the coming days and also at the Symposium, the scientific conference on RoboCup, which will take place on Monday.

For our trip to the RoboCup this year we decided to travel by car. Given the short distance to the south of France, this was the most practical choice as it allowed us to take more equipment with us, such as a 3D printer and monitors.

We picked up the nine-seater on Friday, packed everything and drove off around noon. The journey took about twenty hours in total, but it was manageable thanks to our five drivers. In between we made a small stop in Versailles behind Paris and wanted to take a look at the palace at night. However, restoration work was probably taking place and the entrance gate was covered with wooden panels.

We finally reached our destination on Saturday morning around nine o’clock. We live in a small house on a campsite right next to the Parc des Expositions where the RoboCup takes place. However, we couldn’t move into the house until 5 p.m., so we spent the day having an IKEA breakfast and going swimming. In the evening we were finally able to go into the house and immediately started to set up our workspace.

On Sunday we dedicated ourselves to the preparations for the RoboCup: screwing the robot together, testing the software, fixing errors and so on. On Monday we continued the preparations, but we also had the first contact with other teams: The HULKs, another team from Hamburg, also live at the campsite, we met the HERoEHs from Korea in the laundromat and tonight we’re going out with the Wolves from Wolfenbüttel.

Tomorrow, the exhibition center will open for the teams to begin their setup. Then the final tests are carried out in the real gaming environment and the competition can begin!

As a small side experiment of my PhD thesis, I investigated latencies of ROS message transfers.
I know that I have not been the first person to investigate this and that there are also frameworks for running tests now. I will list here some links if you are interested in what others have already investigated.

Exploring the performance of ROS2
Latency Analysis of ROS2 Multi-Node Systems

Nevertheless, I performed my own experiments because of two reasons. First, I wanted to test it on our specific hardware (the humanoid robot Wolfgang-OP). Second, the previous works did not use the EventExecutor and message frequencies. For reasons on why we need to use the EventExecutor, see our previous post here.

Experimental Setup and Influence of the Idle Function

If you want more details on the robot that I used, you can find it here.

In these experiments, the main computer of the robot was already upgraded to an ASUS PN51-E1 with Ubuntu 22.04. The latencies were just measured for a single transfer between a sender and a receiver (no ping-pong or multi node setups). We each recorded messages for 100s after discarding
the first 100 messages, which often had high outliers as nodes were still starting. We were interested in the latencies for messages that control the robot’s joints (since quick reactions to disturbances are crucial in a bipedal robot), and therefore, we used a small joint command message for our tests. Please keep in mind that our results might not apply for larger messages like images or pointclouds.

Naturally, it is important how the nodes of the sender and the receiver node are executed. ROS 2 allows to run them in two separate processes or in a single process (which we tested both). It is also important how this process(es) are then executed by the operating system. We forced Ubuntu to run these on isolated cores by using “taskset” and “isolcpus” (see our previous post for more infos about this). Additionally, we realized that it is important to deactivate the idle function of the CPU as it has a high influence on the latency (see image below). You can deactivate it in Ubuntu with the following command:

echo 1 | sudo tee /sys/devices/system/cpu/cpu∗/cpuidle/state∗/disable > /dev/null

Executor

As stated above, we need to use the EventExecutor for performance reasons in our system. Still, I was interested how it performs in relation to the other standard executors. For this small sender-receiver setup, the CPU usage of the executor is not important. Therefore, we could test all of them without further issues. We used a frequency of 1000Hz in this experiment. The difference between the executors is small. All of them profit from running the nodes in the same process.

Comparison to ROS 1

While migrating to ROS 2, our team had the impression that there is a large difference between using nodes in Python and using nodes in C++, which was not so large in ROS 1. Therefore, I also compared the performance of the ROS 2 C++ EventExecutor to the ROS 2 Python Executor (no event based implementation exists at the moment for Python), as well as to their counterparts in ROS 1. Naturally, the ROS 1 nodes can not be executed in the same process, therefore, this test case was omitted. The QoS settings for ROS 2 were set to default, as these mimic the ROS 1 behavior best.
The results (see image below) show that the performance of ROS 2 Python nodes is indeed very bad. Using C++, ROS 2 can be faster than ROS 1 but only if the nodes are executed in the same process.

DDS

ROS 2 also allows the usage of different DDS implementations. Naturally, I also tested if this choice might influence the latency. I only tested FastDDS and CyclonDDS, since these are open source.
In our testcase, Cyclone DDS performed better than FastDDS (although the impact of this choice is smaller than, e.g., using the idle function). Previous work (https://ieeexplore.ieee.org/abstract/document/9591166/) indicated that FastDDS has a lower latency. But they only tested frequencies up to 100 Hz.

Integrated Test and Scheduling

After performing these simple sender-receiver experiments, I tested the latencies between two nodes of our motion software stack that were running at 500Hz while the robot was actively walking (and therefore many other nodes were running). Interestingly, the latencies were much higher than in the toy example. After some investigation, it turned out that the Linux scheduler was the problem. In the experiment before, we only had two threads which each had their own core, so there weren’t any issues. Now, in the integrated case, we have many threads running simultaneously. Unfortunately, the used “taskset” command leads to the scheduler to only execute one thread at a time. I solved this issue by setting the Linux scheduler behavior to the round-robin mode.
Interestingly, deactivating the idle function of the CPU still resulted in improved latencies, although the CPU was under high load.

I hope that these insights might help others in reducing the latency of ROS 2 message transfers for high frequency topics.