Newer
Older
---
layout: default
title: Passive scanning
nav_order: 3
mathjax: true
---
# Passive scanning
{: .no_toc}
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
_Photogrammetry_ is the collection and organization of reliable information about physical objects and the environment through the process of recording, measuring and interpreting photographic images and patterns of electromagnetic radiant imagery and other phenomena.
Photogrammetry was first documented by the Prussian architect [Albrecht Meydenbauer](https://opus4.kobv.de/opus4-btu/files/749/db186714.pdf) in 1867. Since then it has been used for everything from simple measurement or color sampling to recording complex 3D [Motion Fields](https://en.wikipedia.org/wiki/Motion_field)
<img src="images/data_model_photogrammetry.png" alt="Data Model of Photogrammetry">
<img src="images/colorized_point_cloud_manufacturing_facility.jpg" alt="Colorized Point Cloud">
Unlike other scanning methods that require precise orbital plans or specialized equipment, photogrammetry can be achieved simply by flying a drone in a circular pattern and capturing multiple photos. Utilizing the location data from the drone, one can construct detailed models like the example shown here: A typical medium resolution aerial photogrammetry scan of a barn.
With 50-100 images a reasonably accurate model can be produced. Such models are often used in surveying and restoration projects from the scale of hand helf objects to cities. This accessibility makes photogrammetry an attractive option for various applications, with results that can be sufficiently accurate depending on the specific requirements.
However, it's essential to note that photogrammetry lacks inherent scale. Without a reference point or prior knowledge of the camera locations, the resulting model lacks a definitive scale, as cameras inherently lack absolute scale information. Therefore, incorporating at least one reference point is crucial. For example, marking a facade with visual markers or known distances, such as pieces of tape, allows for scaling within a 3D modeling program based on these references.
<p>Stereo matching is also known as "disparity estimation", referring to the process of identifying which pixels in multiscopic views correspond to the same 3D point in a scene.</p>
<p>Early uses in stereophotogrammetry, the estimation of 3d coordinates from measurements taken from two or more images through the identification of common points. This technology was used throughout the early 20th century for generating topographic maps.</p>
<p><img src="images/stereo_plotter.jpg" alt="StereoPlotter"></p>
<p><img src="images/may_alexander_analog_photogrammetry.jpg" alt="Analog Photogrammetry"></p>
Cameras were applied to photogrammetry as early as 1973, such as in
[Imaging characteristics of photogrammetric camera systems](https://www.sciencedirect.com/science/article/abs/pii/0031866373900069)
, an early paper describing groundwork camera properties and standards for USGS (United States Geological Survey) photgrammetry surveys.
When describing photogrammetric camera systems, *intrinsics and extrinsics parameters* are used to characterize systems: <br>
Intrinsic parameters describe the properties of a camera's lens and sensor. <br>
Extrinsic parameters describe the camera's position and orientation in 3D space
[Levenberg–Marquardt algorithm](https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm) or damped least-squares algorithm (dls) are used to minimize the error across 3d coordinates. This procedure is typically called [bundle adjustment](https://en.wikipedia.org/wiki/Bundle_adjustment).
While the analog versions of these techniques have waned in popularity, stereophotogrammetry still has applications for capturing dynamic characteristics of previously difficult to measure systems like running [wind turbines](https://www.spiedigitallibrary.org/conference-proceedings-of-spie/8348/1/Dynamic-characteristics-of-a-wind-turbine-blade-using-3D-digital/10.1117/12.915377.short?SSO=1).
Photogrammetry is useful for outdoors settings, where all you need is a handheld camera and some patience. In this example, note the loss of quality towards the top, as pixel resolution becomes problematic:
<div class="sketchfab-embed-wrapper"> <iframe title="Arc de Triomphe - photogrammetry" frameborder="0" allowfullscreen mozallowfullscreen="true" webkitallowfullscreen="true" allow="autoplay; fullscreen; xr-spatial-tracking" xr-spatial-tracking execution-while-out-of-viewport execution-while-not-rendered web-share width="640" height="480" src="https://sketchfab.com/models/65937fd27de647c0a8ac99ce8275c03e/embed"> </iframe> <p style="font-size: 13px; font-weight: normal; margin: 5px; color: #4A4A4A;"> <a href="https://sketchfab.com/3d-models/arc-de-triomphe-photogrammetry-65937fd27de647c0a8ac99ce8275c03e?utm_medium=embed&utm_campaign=share-popup&utm_content=65937fd27de647c0a8ac99ce8275c03e" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;"> Arc de Triomphe - photogrammetry </a> by <a href="https://sketchfab.com/nicolasdiolez?utm_medium=embed&utm_campaign=share-popup&utm_content=65937fd27de647c0a8ac99ce8275c03e" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;"> Nicolas Diolez </a> on <a href="https://sketchfab.com?utm_medium=embed&utm_campaign=share-popup&utm_content=65937fd27de647c0a8ac99ce8275c03e" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;">Sketchfab</a></p></div>
Depending on the end use application almost any camera will work given there is enough light and your post processing software is robust.
*as models improve. Nasa's [Ingenuity Drone](https://mars.nasa.gov/technology/helicopter/#) relies on photogrammetry-based onboard processing for ground distance estimation, showcasing the efficacy of passive sensing approaches in complex environments.
<p><img src="images/lidar_vs_photogrammetry_drone.jpg" alt="Drone imaging"></p>
Light conditions in the scene are crucial to the quality of the scan. A controlled environment is highly preferred. Precision is improving but can still be completely thrown off by certain light conditions in much the same way LiDar struggles with smooth surfaces.
Increasingly industry pairs vision systems for photogrammetry with laser systems to balance the benefits of both.
<img src="images/full_wetsuit_scan.jpg" alt="Self Assembly Lab">
<small>Photogrammetry scan using Polycam, 2m range, 3 circular passes around the mannequin. ~2 minute process to get the result.
On the left, rendered view, on the right, the untextured mesh.</small>
[AliceVision](https://alicevision.org/) Open-source photogrammetric computer-vision framework<br>
[Meshroom](https://github.com/alicevision/Meshroom) is the 3D Reconstruction Software built on AliceVision
[Autodesk ReCap](https://www.autodesk.com/products/recap/overview?term=1-YEAR)<br>
[Pix4D](https://www.pix4d.com/blog/machine-learning-meets-photogrammetry) - Uses ML for improving accuracy while scraping information on the contents of photogrammetry data sets.
### Apps
iPhone and Android apps for photogrammetry and now LiDAR scanning have multiplied over the last several years:
[Rekrei](https://projectmosul.org/)<br>
[Adobe Aero](https://www.adobe.com/products/aero.html)<br>
[ScandyPro](https://www.scandy.co/apps/scandy-pro) - Real-time meshing <br>
As the apps multiply, scanned objects become more widely available for both art preservation and computer graphics applications:
[Scan the World](https://www.myminifactory.com/scantheworld/)<br>
[Quixel Megascans](https://quixel.com/megascans)
<img src="images/quixel_megascans.jpg" alt="Quixel Megascans Asset Library">
_plenoptic_: Of or relating to all the light, travelling in every direction, in a given space.
<small>Light fields represent an advanced form of passive sensing, aiming to capture full plenoptic content: all possible light rays emanating from a scene in any given direction. This results in a four-dimensional function, as it involves selecting a ray's position and angle. If the ideal plenoptic function was known, any novel viewpoint could be synthesized by placing a virtual camera in this space, and selecting the relevant light rays.</small>
"We begin by asking what can potentially be seen" - Edward H. Adelson & James R. Bergen, Media Lab, Vision & Modeling Group:
[The Plenoptic Function and the Elements of Early Vision](https://persci.mit.edu/pub_pdfs/elements91.pdf), 1991.
Image-Based Rendering (IBR) for view synthesis is a long-standing problem in the field of computer vision and graphics.
Applications in robot navigation, film, and AR/VR.
The affect of this technique was popularized in _The Matrix_, in the famous bullet dodge shot which used [120 still cameras and two film cameras](https://filmschoolrejects.com/the-matrix-bullet-time/).
<video width="500" height="290" controls>
<source src="images/The_matrix_rootfop_bullet_dodge.mp4" type="video/mp4">
</video>
<iframe width="560" height="315" src="https://www.youtube.com/embed/9XM5-CJzrU0?si=eWG2ipj0fqA1Ng13&start=73" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
This is such a resource intensive setup, that it prompts researchers to seek simulation shortcuts to reach this result, such as this paper [using thousands of virtual cameras](https://openaccess.thecvf.com/content/ACCV2022/papers/Li_Neural_Plenoptic_Sampling_Learning_Light-field_fro
m_Thousands_of_Imaginary_Eyes_ACCV_2022_paper.pdf) and neural networks to capture a complete dense plenoptic function.
In practice, we can only sample light rays in discrete locations. There are two popular optical architectures for this:
### Multi-Camera Systems
Simply shoot the scene from several locations using an array of camera (or a single moving one).
Some sophisticated acquisition rigs shown by Google in in a SIGGRAPH 2018 paper:
<iframe width="560" height="315" src="https://www.youtube.com/embed/4uHo5tIiim8?si=7tmQx2MrG5WYh0SK&start=23" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
Lenslets: a single CMOS sensor with an array of lenses in front.
In the lenslet approach, each pixel behind a lenslet provides a unique light ray direction. The collection for all lenses is called a <strong>sub aperture image</strong>, and roughly corresponds to what a shifted camera would capture. The resolution of these images is simply the total number of lenslets, and the number of sub-aperture images available is given by the number of pixels behind a lenslet. For reference, the [Lytro Illum](https://en.wikipedia.org/wiki/Lytro) provides 15x15 sub-aperture images of 541x434 pixels each, which is a total of ~53 Megapixels.
<img src="images/viewpoint7_png+img+margin.gif" alt="LF sub aperture images">
The most efficient layout for lenslets is hexagonal packing, as it wastes the fewest pixel area. Note that some pixels are not fully covered by the lenslet and receive erroneous or darker data. This means some sub aperture images cannot be recovered.
<img src="images/LF.png" alt="LF preview">
### Depth Estimation
Forming an image from these cameras requires sampling one pixel from each micro lens to generate virtual viewpoints. The resulting "sub-aperture images" offer different perspectives with subtle shifts, presenting a challenge for depth estimation due to their minute disparities.
Depth estimation on Light Field data is an active domain. For now, algorithms are commonly tested on ideal, synthetic light fields such as this [dataset](https://lightfield-analysis.uni-konstanz.de/). Here is one example of point cloud obtained from a stereo [matching method](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8478503).
<div class="sketchfab-embed-wrapper"> <iframe title="4D light field - depth estimation" frameborder="0" allowfullscreen mozallowfullscreen="true" webkitallowfullscreen="true" allow="autoplay; fullscreen; xr-spatial-tracking" xr-spatial-tracking execution-while-out-of-viewport execution-while-not-rendered web-share width="640" height="480" src="https://sketchfab.com/models/b9edfdd28c154ecf995da7b8c6590da8/embed"> </iframe> <p style="font-size: 13px; font-weight: normal; margin: 5px; color: #4A4A4A;"> <a href="https://sketchfab.com/3d-models/4d-light-field-depth-estimation-b9edfdd28c154ecf995da7b8c6590da8?utm_medium=embed&utm_campaign=share-popup&utm_content=b9edfdd28c154ecf995da7b8c6590da8" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;"> 4D light field - depth estimation </a> by <a href="https://sketchfab.com/pythonzen?utm_medium=embed&utm_campaign=share-popup&utm_content=b9edfdd28c154ecf995da7b8c6590da8" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;"> pythonzen </a> on <a href="https://sketchfab.com?utm_medium=embed&utm_campaign=share-popup&utm_content=b9edfdd28c154ecf995da7b8c6590da8" target="_blank" rel="nofollow" style="font-weight: bold; color: #1CAAD9;">Sketchfab</a></p></div>