🏬3D Deep Learning
Few technologies can change the landscape of Computer Science at the pace and impact that Artificial Intelligence does. But one specific aspect is even more profound: 3D Deep Learning.
Why Learn 3D Deep Learning?
The impact of 3D Deep Learning is massive. From extracting information in MRI brain scans to automated 3D asset generation in games (including massive real-world datasets from 3D mapping and 3D reconstruction techniques), its outreach and potential are unrivaled, especially in ensuring that self-driving cars are a reality.
Of course, I want to give you a clear roadmap that best fits your goals for developing 3D Deep Learning Systems along three tracks: the Hobbyist, the Engineer, and the Researcher / Innovator.
Let me demystify and provide something as close to a recipe as possible to unlock hands-on 3D Deep Learning Skills and become an “elite” 3D Innovator.
I approach the topic with five main modules, as illustrated below:
Foundational Knowledge of 3D Deep Learning
The good news is you can spend less than 10,000 hours. You can cut down the time to only the essentials to try early on your id, and the essentials here are on three main pillars: mathematics, artificial intelligence, and 3d data.
Mathematics Knowledge
You want to ensure you perfectly grasp geometry, 3D projections or reprojection, and trigonometry; that is enough at this stage.
Artificial Intelligence / Computer Science Knowledge
At this stage, knowing what a neural network is, what a convolutional neural network is, and how an architecture is designed is good.
3D Data Expertise
Finally, when we talk about 3D Data Expertise, two things will be perfect to have as a foundational layer in 3D data representation. How do you represent 3D data at large and cloud processing? Because this kind of data is canonical, you can attach other ways of representing data directly, which is phenomenal.
Top-tier Architectures for 3D Deep Learning
Implementing a 3D Deep Learning Solution is, in my fair opinion, one of the best ways to truly get into the game. But that usually requires willpower and time, with some coding knowledge (E.g., Python).
If you want to avoid getting on that path right away, grasping top-tier architecture and knowing how to read it is at least a great starting point. And here, I have selected three major ones for you.
Point-based 3D Deep Learning for unstructured datasets: PointNet
The first one is Pointnet. Pointnet was among the first to process point clouds directly as unstructured data.
PointNet examines 3D data, such as point clouds comprised of object- or scene-representing points, in a savvy manner.
Unlike images or videos, point clouds lack a predefined layout or system. To solve this problem, PointNet examines each point individually.
The process involves the extraction of nuanced details, which is then integrated into a mechanism and leveraged to appreciate the general shape and characteristics of the collection of points.
Consequently, PointNet recognizes objects in a scene and extracts individual parts of the point cloud data.
Moving to 3D Convolutions for 3D Deep Learning
The second one is KPConv, which is very interesting to understand how you can do convolutional with convolutional kernel on top of unstructured data — creating a top structured data approach on an unstructured data set.
Envision a typical filter in an image processing application, except it comprises several freely arranged points in 3D space instead of gliding across a grid.
These “kernel points” are employed to inspect a point cloud, weighing the attributes of close-by points against their relationships with the kernel points.
This is the core of the KPConv framework, which enables learning directly from random and scarce 3D data, like point clouds, without a rigid grid pattern.
As a result, it is precious for work like categorizing and sectioning objects in 3D, such as self-driving cars comprehending their environment.
Unsupervised learning for 3D Point Clouds with 3D Deep Learning
The final one for point cloud is GrowSP, which is an unsupervised method, which I like mainly because it resonates with my thinking of having Gestalt’s Theory, which groups a set of elements and reasoning them from this group of visual cues instead of going down at the point level, right?
The GrowSP architectural framework is made up of three key components:
1. Feature harvester: This module picks up signals from individual points in the input point cloud to understand their characteristics. 2. Semi-automated superpoint buildout algorithm: This segmentary increases the populations of key points, into superpoints. 3. Clustered semantic scoring module: This piece groups super points into “recognizable” elements that can be used to generate a final prediction.
In short, the key to the GrowSP framework is a feature harvester that understands the individual characteristics of each key point in the point cloud. This is then used to build up superpoints by slowly grouping them based on those characteristics. Then, these superpoints are used to find the main components of the studied environment.
Last updated