Project 4 - NeRF

0.1 Calibrating the Camera

I calibrated the camera by using ArUco markers. Importantly, the markers have a known geometry. So, we can simplmy set the marker as the origin of our world coordinate frame. Then, we can get point correspondences by using the corners of the marker.

Luckily, OpenCV does much of the heavy lifting for us. It provides a method to detect the markers and get the pixel coordinates of the corners in the image. It also has solvers for various least squares problems allowing us to solve for the camera intrinsics (distortion, focal lengths, etc).

I downscaled all images before calibrating.

0.2 3D Object Scan

Now we want to scan a 3D object by taking many pictures of it from different angles, making sure to keep the same zoom so that the intrinsic parameters of the camera remain the same as the last part.

These images will be used to train a NeRF model; in the next step, we will solve for the extrinsic parameters for each image.

0.3 Estimating Camera Pose

We are given the intrinsic parameters of the camera and can get some correspondences using ArUco markers. We will now use these to solve for the extrinsics in the classic Perspective-n-Point problem.

Again, OpenCV saves us by providing a method, solvePnP, which takes as input 3D-2D point correspondences + intrinsics and spits out the extrinsics.

Here are the viser visualizations of the camera poses:

0.4 Undistorting images and creating a dataset

Now we’ll package everything up into a dataset. First, we want to remove lens distortions from the images - OpenCV has a method for this. Then, we store everything in a nice .npz file, split up into training, validation, and test sets. We store images and c2ws extrinsic matrices for each image.

1. 2D Neural Field

I implemented a simple MLP with 4 layers, starting with a positional encoding input of dimension 2 * (2L + 1). This is passed into blocks of 256-wide Linear + ReLU layers. The final one outputs 3 values and goes through a sigmoid to output RGB.

I trained for 1000 iterations with a batch size of 10000 with the suggested hyperparameters; learning rate of 1e-2.

Here is the training progression for L = 10, from iteration 100 to 1800:

Here is the final output using L = 15 with width = 256:

Here is the final output using L = 10 with width = 128:

Here is the final output using L = 15 with width = 128:

Here are the results on a big cat:

2. 3D Neural Radiance Field

2.1 Create Rays from Cameras

We start with the intrinsic parameters of the camera and 2D pixel coordinates. Using these, we first convert pixels into camera coordinates with pixel_to_camera. Then we transform these camera coordinates into world coordinates using transform and the camera-to-world matrix. Finally, in pixel_to_ray, we compute the ray origins and directions. The ray origins come from the camera position in the world, and the directions are obtained by normalizing the vectors from the camera to each pixel in world space (the difference). This gives us a way to go from image pixels to 3D rays in the scene.

2.2 Sampling

After we have the ray origins and directions, we need to sample points along each ray in 3D space. In sample_along_rays, we first generate evenly spaced points between a near and far bound. To avoid overfitting during training, we can add a small random perturbation to each sample point. Finally, we compute the 3D coordinates by moving along the ray from the origin in the direction of the ray. This gives us a set of 3D points along every ray that we can later use for volume rendering or training the NeRF.

2.3 Dataset Creation

In this part, we create a dataset of rays from multiple images. First, we store the images, camera intrinsics, and extrinsics. We then create a grid of pixel coordinates for all images and add a 0.5 offset to move to the pixel centers. Using pixel_to_ray, we convert these pixels into ray origins and directions in world space. We also flatten the rays and pixel colors so that we can easily sample random rays for training. The sample_rays method randomly picks a batch of rays and their corresponding colors, while sample_rays_single_image lets us sample all rays from a single image, which is useful for validation and visualization. This setup allows us to efficiently feed ray data into a NeRF model.

2.4 MLP for 3D Neural Radiance Field

After sampling points in 3D, we want to predict both the color and density for each point. The MLP class takes as input the 3D world coordinates of the points and the corresponding ray directions. We first apply positional encoding to both coordinates and directions, with more frequencies for the coordinates and fewer for the directions. Positional encodings are important, because they allow the network to learn non-linear relationships between the input and output - specifically in the frequency space. The network is deeper than before to handle the more challenging 3D task. We also inject the encoded coordinates into the middle of the network to help the model remember the input. The network outputs density using ReLU to ensure it is positive, and color using Sigmoid to keep it in the range [0, 1]. The ray direction acts as a condition for color, reflecting the fact that appearance depends on view direction. This setup allows the MLP to represent a continuous 3D radiance field that can be used for volume rendering.

2.5 Volumetric Rendering

Once we have the density (sigma) and color (rgb) for points along each ray, we can perform discrete volume rendering to get the final pixel color and depth. In volrend, we compute the transmittance along the ray, which tells us how much light reaches each point, and then calculate the contribution of each sample to the final color. Similarly, volrend_depth uses the same weights to compute the expected depth along the ray. By discretizing the ray into samples and combining their contributions, we approximate the continuous volume rendering integral in a way that is efficient and differentiable for training NeRF.

I trained for 1000 iterations with a batch size of 10000 with the suggested hyperparameters.

Here is the PSNR curve on the validation set:

Along with the progress visualization across iterations:

Iteration 100

Iteration 200

Iteration 400

Iteration 600

Iteration 800

Iteration 1000

Here is the gif of the 3D reconstruction:

2.6 Training with Own Dataset

For this part, I changed various hyperparameters to train the model. First, I set near = 0.02 and far = 0.5, which gave more sensible points along the rays. I also increased the number of samples per ray to 128.

Here is the plot of the loss during training with these changes:

This was a clear sign of overfitting, as the loss on the validation set is much higher than the loss on the training set. I tried to solve this by adding more regularization to the model and using the AdamW and NAdamW optimizers to no success in the time I had.

I also tried lowering my learning rate to 1e-4 and 2e-5 to try to reduce overfitting.

Regardless, here are the intermediate results to show what the model learned (noise, unsurprisingly):

Iteration 100

Iteration 300

Iteration 500

Iteration 700

Iteration 900

Iteration 1000