"I love Matthew Tancik"

2022-07-06 08:42:57

About meArt/ProjectsPublicationsPublicationsBlock-NeRF: Scalable Large Scene Neural View SynthesisMatthew Tancik,Vincent Casser,Xinchen Yan,Sabeek Pradhan,Ben Mildenhall,Pratul P. Srinivasan,Jonathan T. Barron,Henrik KretzschmarCVPR (2022)arXivProject WebsiteVideoWe present a variant of Neural Radiance Fields that can represent large-scale environments. We build a grid of Block-NeRFs from 2.8 million images to create the largest neural scene representation to date, capable of rendering an entire neighborhood of San Francisco.Plenoxels: Radiance Fields without Neural NetworksAlex Yu*,Sara Fridovich-Keil*,Matthew Tancik,Qinhong Chen,Benjamin Recht,Angjoo KanazawaCVPR (2022)arXivProject WebsiteVideoWe propose a view-dependent sparse voxel model, Plenoxel (plenoptic volume element), that can optimize to the same fidelity as Neural Radiance Fields (NeRFs) without any neural networks. Our typical optimization time is 11 minutes on a single GPU, a speedup of two orders of magnitude compared to NeRF.PlenOctrees for Real-time Rendering of Neural Radiance FieldsAlex Yu,Ruilong Li,Matthew Tancik,Hao Li,Ren Ng,Angjoo KanazawaICCV (2021) OralarXivDemo / Project WebsiteVideoWe introduce a method to render Neural Radiance Fields (NeRFs) in real time without sacrificing quality. Our method preserves the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects.Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance FieldsJonathan T. Barron,Ben Mildenhall,Matthew Tancik,Peter Hedman,Ricardo Martin-Brualla,Pratul P. SrinivasanICCV (2021) Oral - Best Paper Honorable MentionarXivProject WebsiteVideoThe rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. We prefilter the positional encoding function and train NeRF to generate anti-aliased renderings.Putting NeRF on a Diet: Semantically Consistent Few-Shot View SynthesisAjay Jain,Matthew Tancik,Pieter AbbeelICCV (2021)arXivProject WebsiteVideoWe introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses. Our semantic loss allows us to supervise DietNeRF from arbitrary poses. We extract these semantics using a pre-trained visual encoder such as CLIP.Learned Initializations for Optimizing Coordinate-Based Neural RepresentationsMatthew Tancik*,Ben Mildenhall*,Terrance Wang,Divi Schmidt,Pratul P. Srinivasan,Jonathan T. Barron,Ren NgCVPR (2021) OralarXivProject WebsiteCodeVideoWe find that standard meta-learning algorithms for weight initialization can enable faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available.pixelNeRF: Neural Radiance Fields from One or Few ImagesAlex Yu,Vickie Ye,Matthew Tancik,Angjoo KanazawaCVPR (2021)arXivProject WebsiteCodeVideoWe propose a learning framework that predicts a continuous neural scene representation from one or few input images by conditioning on image features encoded by a convolutional neural network.NeRV: Neural Reflectance and Visibility Fields for Relighting and View SynthesisPratul P. Srinivasan,Boyang Deng,Xiuming Zhang,Matthew Tancik,Ben Mildenhall,Jonathan T. Barron,CVPR (2021)arXivProject WebsiteVideoWe recover relightable NeRF-like models using neural approximations of expensive visibility integrals, so we can simulate complex volumetric light transport during training.Fourier Features Let Networks LearnHigh Frequency Functions in Low Dimensional DomainsMatthew Tancik*,Pratul P. Srinivasan*,Ben Mildenhall*,Sara Fridovich-Keil,Nithin Raghavan,Utkarsh Singhal,Ravi Ramamoorthi,Jonathan T. Barron,Ren NgNeurIPS (2020) SpotlightarXivProject WebsiteCodeVideoWe show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. NeRF: Representing Scenes as Neural Radiance Fields for View SynthesisBen Mildenhall*,Pratul P. Srinivasan*,Matthew Tancik*,Jonathan T. Barron,Ravi Ramamoorthi,Ren NgECCV (2020) Oral - Best Paper Honorable MentionarXivProject WebsiteCodeVideoFollow-upsWe propose an algorithm that represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x, y, z) and viewing direction (θ, φ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. With this representation we achieve state-of-the-art results for synthesizing novel views of scenes from a sparse set of input views.StegaStamp: Invisible Hyperlinks in Physical PhotographsMatthew Tancik*,Ben Mildenhall*,Ren NgCVPR (2020)arXivProject WebsiteCodeVideoWe present a deep learning method to hide imperceptible data into printed images that can be recovered after photographing the print. The method is robust to corruptions like shadows, occlusions, noice, and shift in color .Lighthouse: Predicting Lighting Volumesfor Spatially-Coherent IlluminationPratul P. Srinivasan*,Ben Mildenhall*,Matthew Tancik,Jonathan T. Barron,Richard Tucker,Noah SnavelyCVPR (2020)arXivProject WebsiteVideoWe present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair. We propose a model that estimates a 3D volumetric RGBA model of a scene, including content outside the observed field of view, and then uses standard volume rendering to estimate the incident illumination at any 3D location within that volume. TurkEyes: A Web-Based Toolbox for Crowdsourcing Attention DataAnelise Newman,Barry McNamara,Camilo Fosco,Yun Bin Zhang,Pat Sukham,Matthew Tancik,Nam Wook Kim,Zoya BylinskiiCHI (2020)arXivProject WebsiteCodeEye movements provide insight into what parts of an image a viewer finds most salient, interesting, or relevant to the task at hand. Unfortunately, eye tracking data, a commonly-used proxy for attention, is cumbersome to collect. Here we explore an alternative: a comprehensive web-based toolbox for crowdsourcing visual attention.Towards Photography Through Realistic FogGuy Satat,Matthew Tancik,Ramesh RaskarICCP (2018)Project WebsiteLocal CopyVideoMIT NewsWe demonstrate a techniquethat recovers reflectance and depth of a scene obstructed bydense, dynamic, and heterogeneous fog. We use a single photon avalanche diode (SPAD) camera filter our the light that scatters off of the fog in the scene.Flash Photography for Data-Driven Hidden Scene RecoveryMatthew Tancik,Guy Satat,Ramesh RaskararXivVideoWe introduce a method that couples traditional geometric understanding and data-driven techniques to image around corners with consumer cameras. We show that we can recover information in real scenes despite only training our models on synthetically generated data.Photography optics at relativistic speedsBarmak Heshmat,Matthew Tancik,Guy Satat,Ramesh RaskarNature Photonics  (2018)Project WebsiteNature ArticleVideoMIT NewsWe demonstrate that by folding the optical path in time, one can collapsethe conventional photography optics into a compact volume or multiplex variousfunctionalities into a single imaging optics piece without losing spatial or temporalresolution. By using time-folding at different regions of the optical path, we achieve an order of magnitude lenstube compression, ultrafast multi-zoom imaging, and ultrafast multi-spectral imaging. Synthetically Trained Icon Proposals for Parsing and Summarizing InfographicsSpandan Madan*,Zoya Bylinskii*,Matthew Tancik*,Adria Recasens,Kim Zhong,Sami Alsheikh,Hanspeter Pfister,Aude Olivia,Fredo DurandarXivVisually29KCombining icon classification and text extraction, we present a multi-modal summarization application. Our application takes an infographic as input and automatically produces text tags and visual hashtags that are textually and visually representative of the infographic’s topics respectively.Lensless Imaging with Compressive Ultrafast SensingGuy Satat,Matthew Tancik,Ramesh RaskarIEEE Transactions on Computational Imaging (2017)Project WebsiteLocal CopyIEEEMIT NewsWe demonstrate a new imaging method that is lensless and requires only a single pixel. Compared to previous single pixel cameras our system allows significantly faster and more efficient acquisition by using ultrafast time-resolved measurement with compressive sensing.Object Classification through Scattering Mediawith Deep Learning on Time ResolvedMeasurementGuy Satat,Matthew Tancik,Otkrist Gupta,Barmak Heshmat,Ramesh RaskarOptics Express  (2017)Project WebsiteLocal CopyOSAA deep learning method for object classification through scattering media. Our method trains on synthetic data with variations in calibration parameters that allows the network to learn a calibration invariant model.