DPF-Nutrition (RGB-D) — weights & Core ML

Trained weights for DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion (Han et al., Foods 2023, arXiv:2310.11702), trained and evaluated on Nutrition5k under the official train/test splits, and converted to Core ML for on-device use.

• 🔬 Research repo (training & reproduction): https://github.com/T0MYYY/Nutrition5k
• 📱 iOS deployment app (Core ML): https://github.com/T0MYYY/CalBro
• 🤗 Reproduction & ablation checkpoints: https://huggingface.co/T0MYYY/nutrition5k-experiments

Contents

File	What
dpf_nutrition_food2k.pt	DPF-Nutrition PyTorch checkpoint (Food2K ResNet-101 backbone, RGB-D fusion head)
food2k_resnet101.pth	Food2K ResNet-101 pretrained backbone used by DPF
coreml/DPFNutritionRGBDepth.mlpackage	DPF-Nutrition converted to Core ML (rgb+depth → [cal, mass, fat, carb, protein])
coreml/DepthAnythingV2SmallF16P6.mlpackage	Depth Anything V2 (Small) Core ML, 518×392, depth stage used in CalBro

Pipeline

RGB → Depth Anything V2 (predicted depth) → DPF-Nutrition RGB-D fusion → [calories, mass, fat, carbs, protein]

On LiDAR iPhones the hardware depth can replace the monocular depth stage.

Citation

@article{han2023dpfnutrition,
  title = {DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion},
  author = {Han, Yuzhe and Cheng, Qimin and Wu, Wenjin and Huang, Ziyang},
  journal = {Foods}, volume = {12}, number = {23}, pages = {4293}, year = {2023}, doi = {10.3390/foods12234293}
}
@inproceedings{thames2021nutrition5k,
  title = {Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food},
  author = {Thames, Quin and Karpur, Arjun and Norris, Wade and Xia, Fangting and Panait, Liviu and Weyand, Tobias and Sim, Jack},
  booktitle = {CVPR}, year = {2021}
}