Vierasluento: The 3D Gaussian Splatting Adventure: Past, Present and Future

The talk is on a phenomenon that started on George's 2023 work in view interpolation, a problem that sits between graphics and computer vision, that has received a huge surge of interest in the past few years in the form of Neural Radiance Fields (NeRF). Basically, George et al. showed that one doesn't really need as much of the "magic of neural networks" as careful thinking of the problem combined with modern optimization tools, and that classical approaches carry very far when done right. The work has caused a huge splash, and I personally always find it very satisfying when careful thinking pays off.

George is a towering figure in European graphics and graphics-oriented vision, who just received his second ERC Advanced Grant (right after his first concluded, no less).

The 3D Gaussian Splatting Adventure: Past, Present and Future

Abstract: Neural rendering has advanced at outstanding speed in recent years, with the advent of Neural Radiance Fields (NeRFs), typically based on volumetric ray-marching. Last year, our group developed an alternative approach, 3D Gaussian Splatting, that has better performance for training, display speed and visual quality and has seen widespread adoption both academically and industrially. In this talk, we describe the 20+ year process leading to the development of this method and discuss some future directions. We will start with a short historical perspective of our work on image-based and neural rendering over the years, outlining several developments that guided our thinking over the years. We then discuss a sequence of three point-based rasterization methods for novel view synthesis -- developed in the context of G. Kopanas' Ph.D. and the ERC Advanced Grant FUNGRAPH -- that culminated with 3D Gaussian Splatting. We will emphasize how we progressively overcame the challenges as the research progressed. We first discuss differentiable point splatting and how we extended it in our first approach that enhances points with neural features, optimizing geometry to correct reconstruction errors. We briefly review our second method that handles highly reflective objects, where we use multi-layer perceptrons (MLP), to learn the motion of reflections and to perform the final rendering of captured scenes. We then discuss 3D Gaussian Splatting, that provides the high-quality real-time rendering for novel view synthesis using a novel 3D scene representation based on 3D Gaussians and fast GPU rasterization. We will conclude with a discussion of future directions for 3D Gaussian splatting with examples from recent work.

Bio: George Drettakis graduated in Computer Science from the University of Crete, Greece, and obtained an M.Sc. and a Ph.D., (1994) at the University of Toronto, with E. Fiume. After an ERCIM postdoc in Grenoble, Barcelona and Bonn, he obtained a Inria researcher position in Grenoble in 1995, and his "Habilitation" at the University of Grenoble (1999). He then founded the REVES research group at INRIA Sophia-Antipolis, and now heads the follow-up group GRAPHDECO. He is a INRIA Senior Researcher (full professor equivalent). He received the Eurographics (EG) Outstanding Technical Contributions award in 2007, the EG Distinguished Career Award in 2024 and is an EG fellow. He has received two prestigious ERC Advanced Grants in 2018 and in 2024. He was associate editor for ACM Trans. on Graphics, technical papers chair of SIGGRAPH Asia 2010, co-chair of Eurographics IPC 2002 & 2008, chairs the ACM SIGGRAPH Papers Advisory Group and the EG working group on Rendering (EGSR). He has worked on many different topics in computer graphics, with an emphasis on rendering. He initially concentrated on lighting and shadow computation and subsequently worked on 3D audio, perceptually-driven algorithms, virtual reality and 3D interaction. He has worked on textures, weathering and perception for graphics and in recent years focused on novel-view synthesis, relighting as well as material acquisition often using deep learning methodologies.