|
I am a Research Scientist at Google DeepMind in San Francisco, USA, where I teach Gemini and Nano Banana multimodal understanding and generation. I received my PhD from ETH Zurich and Max Planck Institute for Intelligent Systems under the supervision of Marc Pollefeys and Andreas Geiger. After the PhD, I worked as a Senior Researcher/Postdoc at ETH Zurich for 6 months. I was a research intern at Google Research with Tom Funkhouser, Meta Reality Labs Research with Michael Zollhoefer, Technical University of Munich with Daniel Cremers, and INRIA with Peter Sturm. I completed an Erasmus Mundus Masters in Computer Vision and Robotics (VIBOT) with distinction, and a Bachelors in Automation at Xi'an Jiaotong University. |
|
|
|
|
|
|
|
 |
 |
News
|
Research ( | ) |
 |
 NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
Weining Ren*, Zihan Zhu*, Boyang Sun, Jiaqi Chen, Marc Pollefeys, Songyou Peng Conference on Computer Vision and Pattern Recognition (CVPR), 2024 (* equal contribution) paperproject pagevideocode We enable robust novel view synthesis from casually captured in-the-wild images. Master thesis project. |
 |
Lei Li, Songyou Peng, Zehao Yu, Shaohui Liu, Rémi Pautrat, Xiaochuan Yin, Marc Pollefeys Conference on Computer Vision and Pattern Recognition (CVPR), 2024 paperproject pagevideocode The straight line belongs to men, the curved one to God.-- Antonio Gaudi Master thesis project. |
|
|
Weiyang Liu*, Zeju Qiu*, Yao Feng**, Yuliang Xiu**, Yuxuan Xue**, Longhui Yu**, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf (*/** equal contribution) International Conference on Learning Representations (ICLR), 2024 paperproject pagecode 
BOFT (Orthogonal Butterfly) is a general finetuning technique that adapts foundation models to different tasks such as Vision, NLP, Math QA, and Controllable Generation.
|
 |
 NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM
Zihan Zhu*, Songyou Peng*, Viktor Larsson, Zhaopeng Cui, Martin R. Oswald, Andreas Geiger, Marc Pollefeys International Conference on 3D Vision (3DV), 2024 (Oral, Best Paper Honorable Mention) (* equal contribution) paperproject pagevideocode RGB-only version of our NICE-SLAM, making it NICER. |
|
|
Lixiang Lin, Songyou Peng, Qijun Gan, Jianke Zhu International Conference on 3D Vision (3DV), 2024 (Spotlight, top 8.2%) paperproject pagecode Shape As Points (SAP) for fast human body reconstruction. |
 |
Songyou Peng PhD Thesis, 2023 ECVA PhD Award, 2024 3DV Outstanding Dissertation Award Honorable Mention, 2026 thesisslides PhD supervisors: Prof. Marc Pollefeys (ETH Zurich), Prof. Andreas Geiger (MPI-IS) External committee: Prof. Leonidas J. Guibas (Stanford), Prof. Vincent Sitzmann (MIT) |
|
|
Shengqu Cai, Eric R. Chan, Songyou Peng, Mohamad Shahbazi, Anton Obukhov, Luc Van Gool, Gordon Wetzstein International Conference on Computer Vision (ICCV), 2023 paperproject page A diffusion-model based unsupervised framework capable of synthesizing novel views depicting a long camera trajectory. |
 |
 OpenScene: 3D Scene Understanding with Open Vocabularies
Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser Conference on Computer Vision and Pattern Recognition (CVPR), 2023 paperproject pagevideocode Zero-shot approach for novel 3D scene understanding tasks with open-vocabulary queries. |
 |
 : A Unified Framework for Surface Reconstruction
Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, Andreas Geiger Open Source Project, 2023 project pagecode We provide a unified framework and benchmark for neural implicit surface reconstruction. |
 |
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, Andreas Geiger Conference on Neural Information Processing Systems (NeurIPS), 2022 paperproject page Monocular depth and normal cues significantly boost the performance of neural implicit surface reconstruction methods. |
 |
 NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
Zihan Zhu*, Songyou Peng*, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys Conference on Computer Vision and Pattern Recognition (CVPR), 2022 (* equal contribution) paperproject pagevideocode A neural implicit-based RGB-D SLAM that can be applied to large-scale scenes. |
 |
Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, Andreas Geiger Conference on Neural Information Processing Systems (NeurIPS), 2021 (Oral, top 0.6%) paperproject pagevideo (6 min)video (12 min)podcastcode An interpretable hybird shape representation that yields HQ watertight meshes at low inference times. |
  |
Michael Oechsle, Songyou Peng, Andreas Geiger International Conference on Computer Vision (ICCV), 2021 (Oral, top 3%) paperproject pagevideoteaser videocode Our method enables to reconstruct accurate surfaces without input masks. |
 |
Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger International Conference on Computer Vision (ICCV), 2021 paperproject pageblogvideoteaser videocode Over 2000x speed-ups for NeRF are possible by utilizing thousands of tiny MLPs. |
 |
Stefan Lionar*, Daniil Emtsev*, Dusan Svilarkovic*, Songyou Peng Winter Conference on Applications of Computer Vision (WACV), 2021 (* equal contribution) papervideocode A student project of 3D Vision course at ETH Zurich where I served as the advisor. |
  |
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, Andreas Geiger European Conference on Computer Vision (ECCV), 2020 (Spotlight, top 5%) paperproject pageblogvideoteaser videocode Most influential ECCV'20 papers #13 A flexible implicit representation for accurate large-scale 3D reconstruction. |
  |
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui Conference on Computer Vision and Pattern Recognition (CVPR), 2020 paperproject pageteaser videopostercode A differentiable renderer for deep implicit signed distance functions. |
  |
Songyou Peng and Peter Sturm International Conference on Computer Vision (ICCV), 2019 (Oral, top 4.6%) papervideopostercode A novel system that interactively guides a user to take optimal calibration images. |
  |
Bjoern Haefner*, Songyou Peng*, Alok Verma*, Yvain Queau, Daniel Cremers IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019 (* equal contribution) paperproject page Recover high-resolution depth maps with fine geometric details using photometric techniques. |
 |
Le Zhang, Songyou Peng, Stefan Winkler IEEE Transactions on Affective Computing (TAFFC), 2019. In press. papercode A journal extension of our ACM MM 2018 paper. |
 
|
Songyou Peng, Le Zhang, Stefan Winkler, Marianne Winslett ACM International Conference on Multimedia (ACM MM), 2018 paperslidescode Technical Demo. A deep Siamese-like network is introduced to predict one's Big-Five personality and arousal-valence emotion from one portrait photo. |
  |
Songyou Peng, Bjoern Haefner, Yvain Queau, Daniel Cremers International Conference on Computer Vision (ICCV) Workshops, 2017 paperslidescode & data A novel depth super-resolution approach for RGB-D sensors is presented. This paper a part of my master thesis, and subsumed by our TPAMI paper. |
 
|
|
Mentored Students and Interns
I am fortunate to (co-)mentor some talented and highly motivated students and interns. I have learnt from and gotten inspired by them:
|
Invited Talks
Teaching
|
Lecturer, GAMES003: 图形视觉科研基本素养 (How To Do Research in CV/CG), Fall 2024
Together with Sida Peng, Jun Gao, and Qianqian Wang. |
|
|
Teaching Assistant (Lead), 3D Vision, Spring 2023
Teaching Assistant, Computer Vision, Fall 2022 Teaching Assistant (Lead), 3D Vision, Spring 2022 Teaching Assistant, Deep Learning for Computer Vision: Seminal Work, Spring 2022 Teaching Assistant, 3D Vision, Spring 2020 Teaching Assistant, Deep Learning for Computer Vision: Seminal Work, Spring 2020 |
|
Teaching Assistant, Deep Learning, Winter 2020/2021
|
Academic Services
|
|
template adapted from this awesome website |