Point Cloud Processing

16-825 Learning for 3D: Assignment 5

1. Classification Model (40 points)

Commands to run:

python train.py --task cls
python eval_cls.py --load_checkpoint best_model --i 7,666,778,406,650,883

Results

Visualization	Label	Prediction	Interpretation
	Chair	Chair	Correct prediction
	Vase	Vase	Correct prediction
	Lamp	Lamp	Correct prediction
	Chair	Lamp	It’s a folded chair not a typical chair so the model misclassified it.
	Vase	Lamp	The vase has an ambiguous shape that resembles a lamp. Even human judges might find it difficult to distinguish.
	Lamp	Vase	Similar to the above example, the lamp’s shape is ambiguous and resembles a vase.

Final Accuracy on Test Set: 97.901%

2. Segmentation Model (40 points)

Commands to run:

python train.py --task seg
python eval_seg.py --load_checkpoint best_model --i 26,238,297,471,616

Results

Index 471, 297, 616 are the top 3 well predicted examples while 238 and 26 are the 2 bad predictions.

Accuracy	Ground Truth	Prediction	Interpretation
99.64%			Well predicted
99.56%			Well predicted
99.14%			Well predicted
42.49%			The model fails to segment the object because the boundaries are not very obvious.
45.89%			The sofa is not a typical chair in the training data as it is asymmetrical, so the model fails to segment it properly.

Final Accuracy on Test Set: 91.185%

3. Robustness Analysis (20 points)

Rotate 30 Degrees Around Z Axis

Commands to run:

python pointcloud_transform.py --rotate-deg 30 --input data/cls/data_test.npy --output data/cls/data_test_transformed.npy
python pointcloud_transform.py --rotate-deg 30 --input data/seg/data_test.npy --output data/seg/data_test_transformed.npy
python eval_cls.py --load_checkpoint best_model --test_data ./data/cls/data_test_transformed.npy
python eval_seg.py --load_checkpoint best_model --test_data ./data/seg/data_test_transformed.npy

Classification

Below are some failed predicted examples:

Original	Transformed	Label	Original Prediction	Transformed Prediction
		Chair	Chair	Vase
		Lamp	Lamp	Vase
		Lamp	Lamp	Vase

Final Accuracy on Test Set decreased from 97.901% to 76.705%. As we can see from the examples above, the model is not rotation invariant, and the model tends to misclassify rotated objects.

Segmentation

Below are some predicted examples, accuracies are reported both before and after rotation:

Accuracy	Original GT	Original Pred	Transformed GT	Transformed Pred
97.74% -> 89.06%
95.21% -> 89.14%
55.29% -> 38.90%
95.31% -> 33.55%

The overall accuracy on the test set decreased from 91.185% to 67.956%. From the examples above, we can see that the model tends to segment parts based on their x-y orientation. From the 3rd and 4th examples, we can see that the model produces horizontal segments even after rotation, indicating a lack of rotation invariance.

Reduce Number of Points to 1000

Commands to run:

python eval_cls.py --load_checkpoint best_model --test_data ./data/cls/data_test.npy --num_points 1000
python eval_seg.py --load_checkpoint best_model --test_data ./data/seg/data_test.npy --num_points 1000 --i 585,397,235,351 --exp_name p1000

Classification

Below are some failed predicted examples:

10000 pts	1000 pts	Label	Original Prediction	Prediction
		Chair	Chair	Lamp
		Vase	Vase	Lamp
		Lamp	Lamp	Vase

Final Accuracy on Test Set decreased from 97.901% to 97.166%. We can wee that the overall accuracy only slightly decreases when reducing the number of points from 10000 to 1000, indicating that the model is robust to point cloud density to some extent. As long as the overall shape is preserved, the model can still make correct predictions.

Segmentation

Below are some predicted examples:

Accuracy	10000 pts GT	10000 pts Pred	1000 pts GT	1000 pts Pred
99.43% → 99.20%
99.47% → 99.60%
57.01% → 43.60%
55.29% → 51.60%

The overall accuracy on the test set decreased from 91.185% to 90.819%. Similar to the classification task, the segmentation model is also robust to point cloud density to some extent. The overall shape is more important than the number of points for the model to make correct segmentations.

4. Bonus Question - Locality (20 points)

Not implemented.