What Makes Tesla's FSD Autonomous Driving So Special? You’ve probably heard terms like neural networks, end-to-end models, occupancy networks, and shadow mode. But what do they all mean? Today, I'll break down these core technologies of Tesla's FSD in a simple and easy-to-understand way.
First, let's talk about the history of Tesla's FSD. As early as 2013, Elon Musk envisioned integrating autonomous driving into Tesla vehicles. Initially, Tesla followed the path pioneered by Google in developing autonomous driving. However, due to some issues and safety concerns that arose during testing of Google's semi-autonomous driving system, AutoPilot, introduced in 2013, Google halted the project.
Thus, the baton for exploring autonomous driving was passed to Tesla. Tesla's AutoPilot and Google's FireFlight were both early-stage autonomous driving projects, but the major difference in their technological approaches was that Tesla opted for pure vision instead of LiDAR. In October 2014, Tesla released Hardware 1.0, marking its entry into the automotive industry's autonomous driving arena. This hardware included a forward-facing camera, a millimeter-wave radar, 12 ultrasonic sensors, and a computing platform from Mobileye, the EyeQ3.
In the first generation of Tesla's autonomous driving system, Tesla did not have its own core computing platform but instead partnered with Israel's Mobileye. Now a subsidiary of Intel, Mobileye focuses on the development of both hardware and software for advanced driver assistance systems (ADAS). Mobileye's EyeQ series of vision processing chips and software systems are used in over 125 million vehicles from manufacturers including Audi, BMW, Volkswagen, General Motors, and others.
In early 2016, Tesla officially introduced the concept of Full Self-Driving (FSD) and began developing a fully autonomous driving platform. In October 2016, Tesla released HW2.0, which expanded from the one front-facing and one rear-facing camera in HW1.0 to a total of eight cameras, providing 360-degree vision around the vehicle. Elon Musk also announced that HW2.0 was sufficient to support full autonomous driving, confirming Tesla's commitment to a purely vision-based approach.
In March 2019, HW3.0 began mass production in the Model S and Model X, followed by the Model 3 a month later. On April 22, 2019, during Tesla's AI Day, the company unveiled its Full Self-Driving (FSD) computing platform featuring Tesla's proprietary FSD chip. In August 2020, Tesla's Autopilot team restructured the software's underlying code and deep neural network. They also developed a new training supercomputer called Dojo and introduced the BEV+Transformer architecture. This architecture uses bird's-eye view to upgrade 2D images to 3D, providing a better understanding and processing of the vehicle's surrounding environment, marking Tesla's entry into the era of large models.
On August 26, 2023, Elon Musk live-streamed the release of Tesla's FSD V12.0, which is Tesla's first end-to-end AI autonomous driving system. This version replaced approximately 300,000 lines of C++ code on the backend with neural network decision-making from Tesla Vision AI, marking a significant step forward in Tesla's autonomous driving technology.
Now, let's talk about what end-to-end means. In traditional autonomous driving system design, the perception module is responsible for gathering environmental information through various sensors, such as cameras and LiDAR.
The planning module then uses this information to make path planning and decision-making, while the control module executes specific actions based on the planned results. Traditional autonomous driving systems operate like a funnel, where information is gradually lost layer by layer.
Tesla's end-to-end model streamlines these complex processes by creating a unified neural network architecture. It takes raw input data and directly processes it to output control commands for the vehicle, eliminating the need for separate modules. This approach reduces delays and errors that can accumulate during information transfer between modules, making the autonomous driving system more responsive and accurate.
Decisions are no longer made based on rule-based code but are instead driven by data and computational power. The model is trained by mimicking human thought processes, learning from vast amounts of video data. The more high-quality data and computational power provided, the better the model's performance. This approach can even lead to an emergent phenomenon common in large models, where the AI suddenly grasps complex concepts, similar to a human "aha" moment.
However, end-to-end systems are not without their drawbacks. For instance, these systems often have weaker interpretability, making it relatively more difficult to pinpoint issues. Essentially, this kind of system operates as a black box, meaning even the engineers may not fully understand how its decisions are made. As a result, there has been an increased likelihood of making basic errors during user operation. Despite continuous training, certain cognitive blind spots remain, such as taking unnecessarily long routes or parking on the curb.
End-to-end systems also heavily rely on massive amounts of high-quality data. Without sufficient data collection and supercomputers like Dojo for training, achieving the precision required for autonomous driving is impossible. Consequently, many automotive brands with fewer vehicles on the road will take a long time to accumulate the billions of miles of data that Tesla has. Additionally, without Tesla's supercomputers, training these models will take significantly longer.
What is a neural network?
In 2021, Tesla developed HydroNet, an autonomous driving neural network. This is a pure vision-based neural network architecture designed for multitask learning. The principle behind it is to use a unified neural network model to process various perception tasks in parallel. In autonomous driving scenarios, the vehicle needs to understand the complex surrounding environment, which includes tasks such as object detection (cars and pedestrians), lane detection, drivable area segmentation, and depth estimation. Essentially, the autonomous driving system divides the collected information into multiple task threads, each handling and analyzing different features in parallel, and then aggregates the results.
In simpler terms, it works like our sensory organs—eyes, ears, mouth, and nose—simultaneously gathering information from our daily environment, which is then sent to the brain for unified processing and recognition of the surroundings.
What is an Occupancy Network?
The original name of the Occupancy Network is "Occupancy Network." It works by dividing the space around the vehicle into small cells and identifying whether these cells are occupied. This helps Tesla's autonomous driving system create a detailed three-dimensional map in real-time. This map enables the vehicle to better perceive and understand its surroundings, leading to smarter driving decisions. The rendered 3D models in the latest Model 3 vehicles are also a result of the Occupancy Network algorithm.
Finally, what is Shadow Mode?
Shadow Mode can be understood as a state where, although the system and sensors are running, they do not control the vehicle. Instead, the system's algorithms continuously make simulated decisions for validation. It's like having a co-pilot constantly learning driving techniques. The system compares its algorithm with the driver's actions, and if there is a discrepancy, the scenario is flagged as an edge case. This triggers data feedback to identify potential errors in the neural network algorithm. The system then records the driver’s actions and the surrounding environment, uploading this information to the backend to further optimize the algorithm.
All Tesla models support this feature, meaning every Tesla user acts as a free tester for the company. The more users there are, the more data Tesla can collect. However, it's important to note that in countries like China and some European nations, vehicles cannot freely upload data without government approval. Therefore, to localize, Tesla must establish data centers and data teams within these regions for localized training.
Elon Musk is very strategic. Tesla pre-installs autonomous driving hardware in all its models, but access to these features is locked and requires payment to unlock via software. This pre-installed hardware is essential for enabling Shadow Mode. Although this seems like an added cost, the benefit of having car owners provide free data testing far outweighs the expense. Musk's long-term planning is indeed evident in this approach.
