CluRender pretrained encoders

DGCNN encoders pretrained with CluRender on ShapeNetCore, from Unsupervised Point Cloud Representation Learning by Clustering and Neural Rendering (IJCV 2024). Code: github.com/gfmei/clurender.

Checkpoints

Folder Clustering Rendering target Decoder Epochs ModelNet40 SVM
dgcnn-rgb-ep100 yes RGB views point-transformer U-Net 100 91.1
dgcnn-dino-ep100 yes DINOv2 feature maps per-point MLP 100 91.7

Each folder holds backbone.pth, the encoder state dict, and config.json, the main_pretrain.py settings (checkpoint_epoch is the epoch of the released weights).

ModelNet40 SVM: a linear SVM (C = 0.002) on frozen global features of 1024 points, each feature standardized with the mean and standard deviation of the training split, fitted on the official training split and scored on the test split (main_svm.py --standardize --c 0.002); overall accuracy in percent.

Usage

import torch
from huggingface_hub import hf_hub_download
from models.dgcnn import DGCNN  # from the CluRender repository

path = hf_hub_download("RyanMei/clurender", "dgcnn-dino-ep100/backbone.pth")
encoder = DGCNN(emb_dims=1024, k=20, num_cls=-1)
encoder.load_state_dict(torch.load(path, map_location="cpu"))
global_features, point_features = encoder(points)  # points: B x 3 x N

The downstream scripts of the repository take the downloaded file as --pretrained.

Training

  • Data. ShapeNetCore v2 meshes, except the ShapeNetPart test objects, rendered with prepare_shapenet.py. These renderings differ from the ones used in the paper, so the scores differ from the paper's.
  • Settings. 1024 points (farthest point sampling), 4 views of 256 Γ— 256 pixels, 64 clusters, sliced Wasserstein image loss with 128 projections, AdamW with learning rate 0.001 decayed by 0.7 every 20 epochs, batch size 32 over 4 GPUs.
  • RGB target. Points are colored by the U-Net decoder and compared with the RGB views. Epochs 45–100 ran on 2 GPUs (same total batch size). From epoch 57, training used 1000 instead of 2000 Sinkhorn iterations, 64 instead of 128 sliced Wasserstein projections, and TF32 matrix multiplications (resumed_at_epoch_57_with in config.json).
  • DINOv2 target. The decoder predicts a 64-dimensional feature per point, which is splatted onto the patch grid and compared with DINOv2 ViT-B/14 patch features of the views, reduced to 64 dimensions by PCA (extract_dinov2.py).

Citation

@article{mei2024clurender,
  title   = {Unsupervised Point Cloud Representation Learning by Clustering and Neural Rendering},
  author  = {Mei, Guofeng and Saltori, Cristiano and Ricci, Elisa and Sebe, Nicu and
             Wu, Qiang and Zhang, Jian and Poiesi, Fabio},
  journal = {International Journal of Computer Vision},
  volume  = {132},
  number  = {8},
  pages   = {3251--3269},
  year    = {2024},
  doi     = {10.1007/s11263-024-02027-5}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train RyanMei/clurender