Add prequantized NF4 backbone for Marigold V2
Summary
Adds a shared, prequantized NF4 backbone that matches the base-model quantization used during Marigold V2 training. Together with the corrected sampling configuration in the companion ComfyUI PRs, this addresses the prediction-quality mismatch between the existing ComfyUI workflows and the original Marigold V2 pipeline.
Changes
- Adds the shared NF4 backbone for use with the separate depth, normals, and albedo LoRAs.
- Enables direct loading of packed NF4 weights without downloading the BF16 backbone or quantizing at runtime.
- Documents loading with the companion native ComfyUI loader.
Existing INT8 checkpoints and modality assets remain unchanged.
Visual comparison
The comparison images below show depth, normals, and albedo predictions from the existing ComfyUI workflows (INT8 backbone, flow sampling) and the corrected workflows (training-matched NF4 backbone, img_to_img_velocity sampling).
These comparisons reflect the combined checkpoint and workflow corrections, not the quantization change alone.
Related PRs
- ComfyUI loader: Comfy-Org/ComfyUI#16495 adds native loading support for this checkpoint.
- Workflow templates: Comfy-Org/workflow_templates#1283 updates all three modality workflows to use the NF4 backbone and corrected sampling mode.
Native ComfyUI inference requires the companion loader changes; this checkpoint is not a drop-in replacement for the INT8 model in existing workflows.
This is great, the corrected one looks good! Just to be clear; we just need to install the 16495 loader, load the NF4 checkpoint in one of the workflows found in 1283?
Thanks for doing this! <3
Thanks! The biggest qualitative improvement actually comes from switching the sampling method from flow to img_to_img_velocity.
To match the Marigold V2 training setup as closely as possible and get the highest-quality reconstruction while also reducing VRAM usage, I also opened the companion PRs to add the prequantized 4-bit NF4 base model.
Since the previous sampling bug had a fairly visible impact on output quality and may have given users the wrong impression of Marigold V2, advertise advertise/showcase Marigold V2 again once these fixes are merged, with updated examples.