As engineers, we all celebrate successful projects. But what about those that failed? Should we sweep them under the rug and pretend they never happened? There’s nothing wrong with a failed project. It’s not a shame but just part of the normal process to succeed. As long as we can learn from our mistakes and the lessons that come out of them, it’s a positive result. Today, I would like to share the story of my recent failed pet project – an ESP32-based Tesla dashcam video remote access system written in Rust.

The needs

It’s great being a Tesla driver that the dashcam system keeps recording, so if there’s any situation that happened, you have evidence to prove it. However, the video clips are on a USB drive. When you get home, if you want to pull out the clips, you need to unplug the USB drive, plug it into your computer, and copy the files you want. I always wish to have a USB thumb drive connecting to my home Wifi to make it possible for me to grab video files over the network without the manual process. A meme pops up in my mind to explain why you need a system like that.

There is already a Raspberry PI-based open-source project for this purpose – teslausb. However, I want something that comes with an easier setup process and, ideally, a cheaper price. And potentially, I want to make it a product. The post-covid high price and low availability of Raspberry PI make it a not-so-desirable solution for me.

Here’s how I envision how it should work. To configure it, you simply write a TOML file and drop it into the USB drive with a specific filename. The USB drive will read the file during the bootup, know which wifi to connect to and know which API endpoint to talk to. Here’s an example:

[wifi]
ssid = "my-wifi"
# Could potentially be in an encrypted format making it harder to others to read
password = "my-super-duper-password"

[api]
endpoint = "ws://192.168.100.123:8080/remote-access"

The system

With the idea in mind, the question is which platform to build on top of it. I don’t have much experience with embedded systems. The last time I built anything with an embedded system was over a decade ago. It’s a WinCE-based augmented reality navigation system integrated with a webcam, digital compass, and GPS module. It was a graduation project for my computer science degree. There was no 3D capability in that system, so I had to learn how to render a 3D world on a surface and craft a straightforward software-based 3D rendering pipeline. I still have the demo video rolling the whole system on a trolly cart in the campus:

Pretty fun project 😄

A while back, I cannot recall where, but I learned about the existence of ESP32, probably from some maker YouTuber’s channel. It is a tiny yet versatile SoC platform. I have always wanted to find a project to build something with ESP32. The Tesla dashcam wireless USB drive seems like a perfect project for this platform, so I purchased a couple from Amazon and started working on them. They are dirt cheap!

The project has been on and off, making progress slowly in my free time. Initially, I put it on a breadboard and manually wired an SD card slot module plus a USB-A male connector. Like this:

But later on, I realized that was silly. There’s a development board with everything I needed: an SD card slot plus a USB-A male connector. Like this:

So I purchased this ESP32-S3-USB-OTG instead. I finally found time to reboot this project during the new year. This time, I was learning Rust and discovered a whole community-building embedded system in Rust. ESP32 also has great Rust support. There’s a book, The Rust on ESP Book about this topic. I chose the std library route, and it’s really nice to use the standard Rust library in an embedded environment. I must say the development experience is enjoyable. There are some bugs in it, and I helped reporting and fixing one in the SD card driver, but overall, everything almost works as expected. Being able to write await for a button press interrupt is simply amazing:

let peripherals = Peripherals::take()?;
let mut button = PinDriver::input(peripherals.pins.gpio14)?;
button.set_pull(Pull::Up)?;
button.wait_for_low().await?;

It took me around one week to start learning ESP32 with Rust and finish the project.

And it worked!

Wait, I thought you said the project failed?

Well, yeah, it was too enjoyable, so I enjoyed building it too much, and I forgot one important thing to check first – the speed of USB and SD card writing… 😅

When I built the project, I read a lot of source code from ESP32 and TinyUSB to understand how things work. I kept seeing “High-speed mode” for SD cards and USB drivers. So I wondered, okay, let me finish the development of the core functionalities and think about performance later, as we always did with most software. I can always enable the so-called high-speed mode, which should be way faster.

After finishing the functionalities, I was about to optimize the performance, so I wrote a benchmark. I realized the speed was slow. Writing files from a PC via USB to the SD, it can only get as high as 700KB/s.

I looked at the Tesla dashcam video files, and just one minute’s worth of video files is around 100MB to 150MB. It must sustain at least 2.5MB/s of writing speed to make it. Then, I started looking at that high-speed mode and learning how to enable them. It turned out that ESP32-S3 doesn’t support those due to hardware limits. For SD cards, ESP32-S3 can go only as high as 52MHz for the clock frequency. With DDR (Double Data Rate) mode, 4-bits lane, you can still reach 49.59MB/s in theory. However, the 4-bit data lanes mode for SD cards didn’t work as I expected. It doesn’t provide too much extra speed compared to 1-bit data lane mode. I didn’t jump into the rabbit hole to find out why it works that way. But even if that works, USB also comes with similar limitations. Turns out, this is a dead end.

Potential solution - Genesys Logic GL823K-HCY04 controller

I searched for information about ESP32 USB and SD card speed and found an interesting solution.

Based on the video, some controllers like Genesys Logic GL823K-HCY04 can facilitate USB 2.0 speed accessing read and write to SD cards directly. It’s a smart solution.

So, the idea is to have a switch in the circuit. Before we detect the home Wifi, we can let the GL823K controller take USB write requests into the SD card directly and at high speed. When the home Wifi is used, it will switch to let ESP32 take over control of the SD card and read the content on demand via the Wifi and API server’s command.

This way, the system could work. However, I haven’t considered the Wifi bandwidth of ESP32 and the reading speed from the SD card. Even if this can work, it might not be a great solution. So, I still consider the project failed even though there’s a potential workaround.

Lessons I’ve learned

Even though this project failed, I’ve learned a lot from it. Here are some lessons I would like to share:

Rust has great potential to become the go-to embedded system language

I have heard good things about Rust for a while, but I’ve never really written any Rust until I recently kicked off another project with Rust. Rust never struck me as a programming language for embedded systems before. To my surprise, writing Rust for an embedded environment is very enjoyable.

First is the memory safety part. While many think about memory safety, it’s all about security. The only purpose of memory safety is for security to avoid a buffer overflow security bug like the one I found previously in ZeroMQ. There’s more to it, actually.

In embedded systems, debugging is not as easy and accessible in the computer or server environment. Sometimes, it’s hard to access those deployed devices. Thus, the cost of fixing a bug would be higher. Some bugs, particularly memory bugs, are tricky to debug. For example, the system may not crash immediately because of memory corruption caused by a race condition. The corruption may stay for a while before crashing the system. Sometimes, it could take days just to hunt a bug like that due to the difficulity to reproduce. You want to avoid the possibility of making any bug as much as possible, as early as possible. This is where Rust comes in very helpful; enforcing strict memory-accessing rules makes writing bugs much harder.

Other than memory safety, I also found the toolchain very helpful. In the past, with a C or C++ project, sometimes I would spend hours, or even whole days, to fix a compiling issue caused by introducing a new third-party library or a newer version. I also need to switch the context between CMake, SCons, Meson, and Bazel, you name it. There are tons of different build systems that the library could use.

But with Rust, things are different. The ability to write the building process itself in Rust is also convenient. There’s a component concept in ESP32 SDK. It’s a CMake-based package management solution for the ESP32 ecosystem. With Rust, the community built a custom build process for compiling an ESP32 component for you. You don’t even need to know CMake unless you need to dig into it.

There are also interesting projects like the Embassy, which aims to be the framework for building embedded system applications. The ecosystem is growing. Surely, rough edges exist, such as the embedded system development support not being so great with the IDE I use, RustRover. But I believe over time, it will only get better.

I think Rust is a great language for embedded systems, and it will be the mainstream choice in the long term, given the lower cost and higher speed for development.

ESP32 is still a fantastic platform

Despite the IO speed not meeting my requirements, I still find the developer experience with ESP32 very enjoyable. The SDK provided by the vendor is already very rich out of the box. Do you need a WebSocket client? Sure, you got it. USB driver as an MSC device? No problem. SD/MMC card driver? Sure. FAT file system support? Here you go. There’s a reason why I can build this side project in around a week. The existing libraries save me a ton of time rebuilding the Lego bricks. With the support of the standard Rust library, surely you can find one for WebSocket clients, but for drivers? It would be harder. I can also find many sample projects, even like this one, which is a remote wifi USB drive.

The community is also very big and active. For other vendors to compete with ESP32, the most challenging part would be the ecosystem and the community it has right now. It’s tiny, yet very versatile, yet low cost, making it very popular. It comes with a pre-certificated module, making it more cost-efficient to integrate into a product. I like this platform and enjoy the development process. I can’t wait to build more interesting stuff with it.

Don’t take things for granted in the embedded environment

We are all spoiled by modern computers. Any entry-level PC comes with many CPU and GPU cores and abundant memory. As software engineers, we barely care about constraints nowadays. A simple application can easily take up multi-GB memory. USB2.0? That’s so slow, and we are talking about USB4 now. But in an embedded environment, things are very different.

Adding a third-party library is supposed to be easy. But in an embedded system, even your binary size matters. While Rust will stripe the unused sections in the binary for you, adding a new library means making your image size bigger. I had to increase the size of my partition section multiple times after introducing a third-party library.

Therefore, you cannot carelessly add as many libraries as you want. Forget about your favorite is_even library from nodejs. Other than that, of course, you need to carefully consider your memory usage, CPU usage, and even which core to use matters. Because some tasks are assigned to a particular core, in a real-time system, you should have two critical background tasks scheduled to the same core; otherwise, they may affect each other.

Premature optimization may not be so premature

The biggest mistake I’ve made is to mindlessly follow my experience in the past to avoid premature optimization and think that I can address performance issues later. I should have considered the platform limit earlier. There’s only so much you can optimize by software when the hardware cannot run fast enough. So, it’s not really about optimization. It’s more about learning the limits of the platform. For embedded systems, particularly, datasheets are your friends.

Some limitations may not be that obvious by reading the datasheet. For example, the clock generator limits the SD card read/write speed. Without that knowledge, I didn’t know it doesn’t support high-speed SD read/write, such as UHS-I speed. In that case, running a benchmark and pushing it to its limit upfront is more helpful than just reading the datasheet. And it depends on the vendor; writing an email to ask them could be helpful, provided the vendor is responsive and willing to help.

Brace yourself for bugs in a fairly new platform

When choosing which platform/framework/language/tool to use, a newer one always has more bugs and risks. We need to weigh the upsides of newer technology and its corresponding risks. This project is obviously not mission-critical, so it’s okay to fail. In this case, choosing Rust is more about optimizing the learning opportunity. I knew there would be some rough edge, and indeed, it did. It’s not just the SD card driver bug I found and fixed. I also noticed that somehow, the read method of the file is supposed to return the read bytes, but it always returns zero instead.

let written_size = file.write(&buf[..chunk_size])?;
// Nope! The written size is always zero somehow
assert_eq!(written_size, chunk_size);

I am still finding time to file the bug report, but I probably have no time to go down the rabbit hole of why it is and potentially fix it.

Due to the nature of new technologies, they always come with unknown risks like these. I like to kick-start fail-safe pioneering projects to understand the technology better. If the new technology works out great in the pioneering project, I will bring it to the production projects. I will gain more experience with the same technology in production projects, inspiring me to find a better tool to try out in the next pioneering project or improve how we use it. I stole the name of Intel’s dual CPU development model name, Tik-Tok. The risk in trying out the project is tik; if it works out, it will return to Tok. Back and forth.

This project is a tik to me. While there’s no production ESP32 project for me to bring the experience to, if I kick start the next ESP32 project aiming to get it for production use, I can get into it quickly.

I open-sourced it anyway

When I kicked off this project, I decided to open-source it. Although ESP32’s speed is not fast enough for its original purpose, this project might still be useful in some use cases. Or, it could be an example project for people we are interested in ESP32 with Rust. And, who knows, there will be a new ESP32 chip capable of solving my problems. Therefore, despite it’s a failure to me, I still open-source it regardless. Here’s the repository.

https://github.com/LaunchPlatform/securedash-esp32

I recall when I was a student of computer science, and some students submitted assignment coding projects that could not run or even compile. They called it “body”. Well, even though my project didn’t achieve my original goal, it compiles and runs, so I would say it’s a zombie 😅 There might still be bugs in it since I don’t have the motivation to maintain it right now. Please brace yourself.

Final thoughts

While this sounds like a waste of time, I think it’s a great side project and has taught me a lot. In the end, there’s nothing wasted. What you’ve learned is what is yours. Nobody can steal it from you. My next ESP32 project in Rust will be better, thanks to the experience from this one.

Regardless, in the end, I still want to make it work. And potentially, I want to make it a product. So, I am looking for the next embedded platform for building this project. I am not particularly familiar with microcontrollers and microcomputers. If you’re familiar with this field, I would like to know what platform you would recommend for my requirements:

• High-speed SD card read/write (at least UHS-I)
• High-speed USB connection as device (at least USB 2.0)
• High-speed wifi connections (ideally)
• Low power consumption (ideally)
• Low cost
• High availability
• Rust supported (ideally)
• Great ecosystem and libraries (ideally)

While I know Raspberry Pi can meet most of the requirements, I still want to know if there’s a better option. I guess in the end, I may want just a slightly more powerful ESP32 that can fit my needs. Anyway, hope you find my project somewhat interesting and learn something from it.