DelivAir

Disclaimer: everything written here are my own personal opinions and do not represent Cambridge Consultants in any way.

 

Where Role Language used
Cambridge Consultants Software Lead Python

 

Outline

This was one of the amazing project I got to work on during my time at Cambridge Consultants as a Computer Science Intern. DelivAir is a proof of concept for an automated drone delivery system. However unlike other proposed drone delivery systems, instead of landing the drone at a certain location, it delivers a package anywhere to the user, without landing. It achieves this using a winch system to lower the package, with the help of the on-board vision system, right into the hand of the user. I worked as Software Lead on this project for 6 weeks, while it went from a few working systems and a concept to a fully working prototype.

Rebuilding from the ground up

DelivAir (internally known as Project Dangled) was originally was a Tech Scholar project (Tech Scholars work at Cambridge Consultants before their first year at University, where while working on project, they undertake a project, where they get to have more responsibilities). I was brought in to the project by the previous (Tech Scholar) software engineer on the project, who was leaving the company. Project Dangled has just kicked-off Phase 2, where the aim was finish the prototype by the end of summer, and produce a video about it. The jobs for Phase 2 were to make the drone lighter, create a cowling for the drone to improve aesthetics, improve flight profile, and finish the drone software.

This is where I came in. However I had to realise, that although most of the mission code was written, it was badly designed, implemented riddled with mistakes, and almost entirely untested. Some of the systems in place were worth keeping, but otherwise it had to be rebuilt from the ground up. But every setback comes with an opportunity. The primary on-board computer was an Nvidia Jetson TK1. I thought the reason for using this board was the dedicated GPU, for the vision system. But I had to realise, that this was not the case, as the vision code was running on the CPU, not the GPU. On top of this, the TK1 was large, bulky, needed extra fan cooling, and only had a single USB port, while 4 different system had to communicate via USB, so next to the board there was a USB hub with its own separate power supply.

Given, that one of the objectives was making the drone lighter, I proposed switching from the TK1 to an overclocked Raspberry Pi 3B, where only there are 4 dedicated USB ports, also we could use the UART port to communicate with the flight computer (a PIXHAWK2). It was also significantly smaller and lighter. This also gave us a chance to switch from a regular webcam to the Pi Camera, which made interfacing with the camera much easier. Luckily I’ve realised that the vision code can be optimised significantly, and with sufficient multi-threading it can work on the Quad-core processor of the Pi.

This was my starting point, from where I started of, rebuilding the control systems, but I have spent some time designing the system, and was testing each addition thoroughly before moving on to the next one.

The state of Project Dangled at the end of Phase 1

Processes, lots of processes

I started working on the code for the Pi, but I had to realise that the speed of things will be a problem. Frame read is about 80-100ms on the Pi Camera, when using OpenCV. Image processing (running the same sort of filters as for the final one, just not yet tuned, for testing) also takes around 100ms. At this point the code is only running at 5 fps, and it isn’t even flying yet. I knew I had to paralellise everything. I started this with using what I knew, threads.

So Python Threats are a bit of a lie. They run on the same core. They are managed by GIL, to run simultaneously but that’s not what you need when you need to get every ounce of power out of a CPU. A Raspberry Pi will start to significantly slow down if it’s running 3-4 threads. Processes to the rescue! They do run on different cores. Great! But cross process communication is difficult and slow. First I had to find a way to communicate between processes, via a non-blocking method. Because the most commonly used way, pipes, are blocking both on the read and the write end, meaning that they bottleneck both processes, making the whole thing pointless. On the other hand Python has a Manager object for processes, that can create a global namespace object, that can be passed to each process, and they can all simultaneously read and write from it, without interfering with each other, even tho read and write is slow. This just meant, that I had to keep all communication to the minimum between processes.

I decided to have three processes. One only responsible for getting the frame from the camera. This than passes the frame to the image processor frame, that had the pattern recognition code as well, that picked up on the target, and just sent the relative location to the main process, handling flying the drone, telemetry, lidar and all other systems.

Vision and Positioning

For obvious reasons I can’t release the code, or talk about the vision system in too much detail. However I can talk about how it worked, and about a few interesting things.

The vision rig part of the drone consisted of a Pi Camera and a Lidar mounted on a gimbal. This was used to position the drone precisely above the user, who pointed their smartphone directly upwards, with the camera LED flashing a certain pattern. The vision system finds the light, recognises the pattern, positions the drone directly above the light, and starts delivery. A specification was, that it should be able to position the drone even if the user is walking.

The Lidar was used as a precise measurement of the distance of the drone and the ground. It was used to determine pixel-meter proportions on the image, and to closed loop control the position of the package to precisely position it to a comfortable height for the user to collect.

What I can say about the vision, that it was tracking the change and position of the bright spots, adding time-stamps where they appear and disappear, accounting for frame time, and than was trying to match it to the unique pattern of the delivery. It was even able to pick up on the pattern, if due to processing times it misses crucial frames. Once it gained some confidence in a point, it will position the drone using a PD control, and once it’s sufficiently confident, and withing a small error value to the user, it starts delivery.

Systems and Integration

There were many, many different systems working as one on this project. Here is a quick rundown of the systems and the communications used:

  • Pi Camera – dedicated parallel connection
  • Lidar – I2C to Arduino
  • Arduino – USB to RasPi
  • Pixhawk 2 flight computer – UART to RasPi
  • Winch speed controller – USB to RasPi
  • Wifi dongle for debug connection – USB to RasPi
  • 3G dongle for server connection – USB to RasPi
  • Battery voltage monitor – I2C to RasPi

Most of these got their own python wrapper and driver, so once each system worked in itself, it shouldn’t interfere with other systems. Obviously all these comes with an asterisk, like how the speed controller needed its own special C++ driver, or how the reason for having an Arduino between the Lidar and the RasPi is needed, because the distance is to great, and interference becomes a problem on the I2C connection. However the most challenging problem from an integration point of view was having both wifi and 3G the same time. I had long discussion with the Project Manager, that we need both the same time, because I didn’t wanted to fly the drone without being able to SSH in, and see live telemetry, which was not possible via 3G due to the internet providers firewall that was in place. Usually this shouldn’t be a problem, but the wifi router that we were using for field testing (because it was made to run of a LiPo battery) was using the same subnetwork IP as the dongle, so routing became a nightmare on the RasPi. Although usually subnetwork masks can be changed on a router, the interface for our was in Chinese. It needed a serious coincidence to have it translated and we were able to change the IP. This was one of the final hurdles after which all systems worked together surprisingly nicely!

Field testing

This is where most nice ideas hit a wall, where theory meets the real world. In some sense this was the case for Project Dangled as well. One of the obvious, not so obvious problems with the project, that it’s a drone. While some systems can be tested in a lab, most of the systems and integration had to be tested in flight. We were field testing 2-4 times a week for 3 weeks, but in this case it’s a literal field, somewhere near Girton. Testing was difficult, fun and stressful the same time. Difficult, because I needed a complete setup where I can be both portable and able to fix problems on the fly. It was quite stressful too. For conditions to be ideal, and have the least amount of wind, we had to fly early morning or late afternoon into the evening. Sometimes both. I had some 14 hour work days, where even tho we made great progress, and all together it was worth it, I was happy to finally get home. It was also stressful, because anything can happen during these tests. From a constant threat from dogs, to the drone just orbiting the target, the package just falling out to the drone coming crashing down, we had to be prepared for everything. (All of these things have happened. Lucky the drone survived the crash) But even with all of this, field testing was incredibly enjoyable. There is always a satisfaction in seeing your work finally working. This is even more so the case when it’s a 10kg flying drone! Seeing all of these systems work together to create something amazing makes all the long testing sessions and extra hours worth it.

Results

The drone worked. Many prototypes like this are just somewhat working, or just smoke and mirrors, this project worked. The only thing we had some problems with was the precision of the GPS on the phone. But all of the drone mission was working and delivering packages reliably. There are two videos with the official press release on the Cambridge Consultants site, I recommend reading it and watching the videos. Some of those videos were faked. Like every video like this. But only the shots near urban areas, since according to UK regulations, we were not allowed to fly the drone near buildings or people. It pains me personally, how some of it had to be faked, but it was needed to create a compelling video.

The press release was picked up by many news sites, tech blogs, like Gizmodo, Engadget, International Business Times, The Verge, and others but my all time favourite coverage was the Daily Mail online. I actually would recommend reading it, it’s a great laugh.

Conclusion

I have learned a lot from this project. I have learned about vision, good coding practices, making many different systems work together, and about good teamwork. Because although it may feel like from what I wrote that I did most of it, this project wouldn’t have been possible without great people working on it. Every engineer and designer working on this project did a phenomenal job, and I’m truly proud to have been a part of it. Finally, here is an awkward blurry picture of me standing next to the drone in the lab where I’ve worked on it.