Modeling large-scale scenes from unconstrained image collections in-the-wild has proven to be a major challenge in computer vision. Existing methods tackling in-the-wild neural rendering operate in a closed-world setting, where knowledge is limited to a scene's captured images within a training set. We propose EvE, which is, to the best of our knowledge, the first method leveraging generative priors to improve in-the-wild scene modeling. We employ pre-trained generative networks to enrich K-Planes representations with extrinsic knowledge. To this end, we define an alternating training procedure to conduct optimization guidance of K-Planes trained on the training set. We carry out extensive experiments and verify the merit of our method on synthetic data as well as real tourism photo collections. EvE enhances rendered scenes with richer details and outperforms the state of the art on the task of novel view synthesis in-the-wild.

Method

We learn a scene through two alternating stages. Scene fitting optimizes our K-Planes representation to reproduce the images in the training set, as traditionally done in neural rendering techniques. Scene enriching finetunes a pre-trained generative prior to this K-Planes representation, and then infers a new one, which will subsequently be corrected by scene fitting.

In-the-wild Renders

Synthetic Renders

Citation

@article{kassab2023eve,
  title={EvE: Exploiting Generative Priors for Radiance Field Enrichment},
  author={Kassab, Karim and Schnepf, Antoine and Franceschi, Jean-Yves and Caraffa, Laurent and Mary, Jeremie and Gouet-Brunet, Val{\'e}rie},
  journal={arXiv preprint arXiv:2312.00639},
  year={2023}
}