How hard can it be to design a self-driving boat for inland shipping?

As part of my Master’s, I took a course called Mechatronic Design. There was no curriculum. Instead, we were split into groups and given a real world case, a breath of fresh air after a very academic program.

We were tasked with creating an self-driving boat. An organic farmer outside the city of Ghent wanted to deliver produce to the city centre in an ecological way. Since his farm and the city centre were connected by a river, an electric boat was the perfect and ecological solution. The owner of a boat rental company in the city would come every few weeks to pick up produce and deliver it to restaurants and supermarkets. The boats, or rather, floats, were slow, since they were powered by solar panels and small engines. The average speed was 4km/h to 6km/h.

We were tasked with making these boats sail autonomously. Here, I’ll describe the problems we faced.

I should note that no one in our team knew anything about boats, waterways, or shipping. We were six people, with skills ranging from software engineering to CAM and electronics.

A constraint that weighed heavily on us was the shoestring budget. We were given €150 that would be reimbursed, a laughably small amount of money for such a project. Fortunately, we were able to get access to many things through stakeholders (mainly the boats and engines through a boat rental company and sensors, such as a RealSense, through the university).

Depth

If there’s a road, a car can drive on it. If there’s a river, a boat may get stuck on it because it’s too shallow.

It’s hard to estimate where it’s safe to drive on a river. Most canals within the city have walls, which are easy to detect and avoid. But once we went outside the city, we saw fields gradually passing to rivers, meaning shallow (unsafe) banks. While we were able to detect river banks, we did not find a satisfying solution to estimating river depth. We settled for “stay clear of the river banks by at least 2 metres”.

In perspective, sailing on the ocean is easier. You rarely have to care about ocean banks or other shallow occurrences. On a river, this is more of a problem. Even with a draft of about 40cm, we would need to sail carefully to avoid getting stuck (and we got stuck a lot).

You might think you only need to chart the depth of a river once. Natural processes and garbage change the depth of a river, but human processes do too. For example, we found out that the city of Ghent would lower the canals twice a year for a week, allowing homeowners to perform repairs to houses that were next to the river. The large amount of locks in Belgium further increases the variability in river depth.

There are solutions for getting the current depth under the boat. Getting a depth map of what lies in front of you is harder, or rather, more expensive.

In the end, we did not do depth measurements. We simply detected the shores, and stayed clear of them by 2 metres.

Lessons learned:

  • The depth of a river varies over time, and that can be abrupt
  • The depth of a river is hard to measure

Avoiding other ships

Ships cost an order of magnitude more than cars. We wanted to avoid collisions at all costs, but this turned out to be harder than expected.

First, there’s less traffic rules on the water than on a road. The bad thing is that they are not friendly to the small floating development platform we created. It was categorised as a pleasure craft, meaning that it had to yield to all commercial crafts (and also most other pleasure craft). This meant looking in front, to starboard, to port, and behind, yielding when possible and avoiding dangerous situations. At one point in the proposed route, the boat would be sailing on a river frequented by commercial boats, crossing another busy river. We sidestepped this by requiring manual intervention for such tricky locations.

Second, there’s no such thing as braking. Thanks to Newton’s First, your boat will continue to go forward long after you’ve stopped the engine. The closest thing you have to braking, is going in reverse, which you can do with electric engines. When we tried aggressive braking with an electric engine, the current could easily spike to 60A (we burned many a fuse). We had to let the engines slow down for a bit before going in reverse, further increasing our braking distance.

Third, there’s no such thing as staying put either: thanks to the flow of the river, your boat will drift when you try this. You always have to control the boat, or risk hitting something or getting stuck. For a normal, manned boat this, this is not an problem, but for an experimental platform on which we were ironing the bugs out as we sailed, this was stressful. We had a lot of times where we had to avoid hitting a moored boat by pushing against it with oars.

Fourth, how do you detect a ship? I thought I had the solution, since every boat has a transponder sending its location. If we could capture Inland-AIS signals, we would know what big ships were coming our way. After discussing this with our domain expert, the owner of the boat rental company, he informed us that that would not work. Transponders and location senders are not mandatory, and in fact, a lot of Belgian commercials ships do not have one. (Turns out that a lot of Belgian inland shipping companies are barely scraping by and have no money for such systems).

Detecting ships with a sonar, then? Not in our budget, but it would have been ideal. (We weren’t sure if it would be able to detect small objects accurately, such as a child swimming in the river). A camera and a neural network? Pre-trained neural networks did not perform well enough in testing (I presume because they are not trained with input from the viewpoint of a ship, looking at the hull of a ship at eye-level looks different than a picture of a ship in the sea). Furthermore, machine learning got confused easily due to the reflections in the water. We made some attempts at pre-filtering out the water before feeding to a neural net, but ultimately gave up on this approach. (Hint: FLIR cameras make it easy to see the cold water)

In the end, we settled on a combination of stereoscopic vision and Lidar. It bundled multiple problems into one. If you think about it, avoiding a column in the middle of the river is the same as avoiding a ship. Prediction of where oncoming ships are going (or even if they are sailing towards or away from us) or not very relevant due to the slow sailing speed.

After experimenting with two FLIR cameras, we used an Intel RealSense. The SDK and toolkit were pleasant to work with and we got good results with it. We converted the 3D representation of the world into a 2D map of what lay ahead of the boat, coupled this with Lidar data and shore inference, and then it was a matter of obstacle avoidance, taking care to stay close to the shore as to let other boats pass.

Lessons learned:

  • Traffic rules are different, and yielding for everyone is hard
  • You can’t brake or stay in one place with a boat
  • Have a back-up plan (possibly in the form of oars and people to man them) in case things go wrong
  • Handling reflections (for example, in water) is not something we had stumbled upon before in machine learning. Neural nets get confused easily by them.
  • Stereoscopic vision is great, but hard to implement yourself. RealSense provides great hardware for this, together with a good SDK and toolkit.

Positioning and maps

You would think that, in modern countries, rivers and waterways are stable and mapped, just as roads are. That is not the case. Through someone with GIS experience, we got maps of waterways in Flanders, but they were not accurate enough for our needs.

We eventually abandoned the effort, focusing on a static route to start with. This was just a list of coordinates. These way-points would be about 50m apart. GPS precision was low and unreliable, but enough to get a course when combined with a compass sensor. The boat would then turn towards the way-point and use an array of other sensors to avoid hitting others boats or the shore.

Lessons learned:

  • Waterways are not always stable
  • Waterways are not as well-mapped as roads
  • GPS is not precise enough. Use global positioning to get a course, then use local navigation to avoid obstacles.

Unmanned boats face the same challenges as unmanned cars when it comes to the law. If the driver is responsible for the actions of the vehicle, who’s responsible if the driver is a computer?

We sidestepped the legal issues by always having people on the boat, who could manually take control.

A month after we finished the project, the Flemish government announced that it would allow experimental self-driving boats to deviate from the existing laws to experiment, and that it would investigate new laws to allow unmanned ships. I’m aware of one research group already using this in practice. I believe that lawmakers are finally getting sold on the potential benefits of self-driving cars, and that this also paves the way for self-driving boats.

Lessons learned:

  • Laws and regulations can be frustrating when designing new technology, but can be side-stepped and investigated later
  • Self-driving boats can surf on the same hype wave as self-driving cars

No standardised development platform

Cars are standardised and the data is easy to read out. The CAN bus equivalent for boats is called NMEA 2000/NMEA 0183. But not every boat implements this -m ost pleasure crafts are very simple, with the most complex piece on the boats we used being the charge controller for the solar panel.

Our boats were lacking any interface or sail-by-wire provisions, so we spent a lot of time implementing it ourselves. We decided not to implement the standard, since it would probably be inadequate to our needs anyway. Instead, we set up an MQTT server for all command and data to go through. It proved to be a great choice, since everything was flowing through it, we had a full overview of everything that happened. The only data that did not go over MQTT was the imagery provided by the RealSense. Two 1080p cameras at 30fps would put undue stress on the MQTT server.

We had a 9-DOF chip that provided the orientation and acceleration of the ship. With no money left for a display or GPS controller, we looked for alternatives. We realised we were carrying a GPS and nice display in our pocket. Using an Android app, we sent the location data to the data collector and displayed basics indicators.

Designing the human-machine interface was hard because the domain was new. How do you pilot a self-driving boat? How do you take over control? Our early prototypes reminded me of an amusing Navy story, where a clunky $38K custom system was replaced by an XBox controllers, because they are cheap, intuitive, reliable, ubiquitous, and tested through thousands of hours of gamers playing with them. So we took a page from the Navy’s book and hooked up a PlayStation controller. Sailing the boat with the PlayStation controller was an surprising fun experience and we eagerly wait our turn to try it out.

Lessons learned:

  • A standardised platform, or one that is open to inspection and extension, makes it easier to iterate and scale. A standardised development platform for self-driving boats would speed up advancements, IMHO.
  • Centralised buses may not scale very well, but they often good enough in practice and are great for debugging
  • When designing interaction for a novel platform, consider existing interfaces, in particular gaming controllers
  • A lot of sensors are already implemented in phones. Consider this a cheap way to enrich the data of your system. Phones can provide time, location, acceleration, and often even barometric pressure information. If your system requires a phone as the display anyway, you can make use of this data.

On Machine Learning

We had taken AI and ML courses together, and were looking for a problem to apply this solution to. Our autonomous boat seemed like a good candidate at first, but we did not use any artificial intelligence in the end, opting instead for hand-written algorithms.

We first tried using pre-trained nets for object detection, specifically, other boats. As described in another section, this gave less than stellar results.

Local navigation seemed like a good problem to solve with artificial intelligence. Feed a depth map generated by stereoscopic vision and Lidar data to a neural network, couple it with captured inputs of the captain, and you’ll have a boat that can sail itself. However, the training data is hard to come by because the boats used are slow. If we had the money, we could install sensors on every boat at the boat rental company to capture data of all boats, but this would still be slow. Moreover, it was winter and few boats were rented, and the people they were rented to were inexperienced, often have never sailed a boat themselves. Our self-driving boat would forever be sailing at the level of an amateur.

Reality began to hit when we pondered how to install an AI system on the boat if we were ever to get it working. We would need to install a computer or board on the boat. Apart from splashes and deck flooding, it would need to be energy efficient. The boats were solar powered, with a lead-acid battery. Energy efficiency is not a primary concern when working with ML. We considered embedding an NVidia board, but shelved the idea when we could not get satisfactory results with AI/ML.

Testing was difficult, since the boat was in another city and weather conditions were not optimal. One time, we brought salt to help de-ice the deck after cleared the snow. We had to switch working on the laptop because our hands were freezing and we could only go so long without gloves. This limited our data set.

Lessons learned:

  • Is AI/ML a solution to your problem or just something you want to play with?
  • Pre-trained neural nets may not fit your specialised use case
  • Heavy AI solutions don’t mix well with constrained environments
  • Think about the practicalities of gathering training set when considering an ML solution

Conclusion

We came into this project with no knowledge of boats. We only had limited amount of time and money to produce a working prototype. While we did not produce a completely self-driving, autonomous boat, we got close and delivered work on which others can build. We made the press and got good grades, what else could a college student want?

The future of logistics and last-mile delivery will have a spot for delivery of parcels using boats in cities with a river (which is, most big cities of the pre-railroad era). The city of Ghent was investigating transport of people using its waterways. Autonomous inland shipping over long distances may prove to be very cost-effective. In essence, I believe this is an important and promising industry.