MicMac [4, 21–23] is a FOSS tool that executes photogrammetric workflows. It is highly configurable and allows to create and run different workflows tailored for different applications. After the images have been acquired, the steps to generate a dense point cloud and a true-orthophoto with MicMac usually include: executing the Tapioca command for the tie-points extraction, running the Tapas and the GCPBascule (or Campari) commands to perform the BBA and to georeference the image block, executing the Malt command for the dense image matching, running the Tawny command to generate the true-orthophoto and executing Nuage2Ply to convert the DSM and the true-orthophoto to the colored dense point cloud. Note that in the adopted MicMac pipeline is first necessary to create the true-orthophoto before the colored dense point cloud.
The solutions proposed in this paper have been developed starting from the the steps described above and using the data structures and exchange formats used in MicMac. The implemented algorithms are therefore additional or alternative steps that have been inserted in the workflow.
Tie-points reduction algorithms
The memory requirements for the BBA are in some cases simply unaffordable for large datasets. We propose an elegant solution that consists of running a tie-points reduction before the BBA. The tie-points reduction decreases the number of tie-points effectively decreasing the memory requirements of the BBA. The general idea is that a sufficient number of well distributed tie points in the overlapping images is sufficient to assure a stable and reliable image orientation, as it was commonly done in the traditional analogue Photogrammetry. Two different algorithms of tie-points reduction have been developed. A similar idea of tie-points reduction has been already introduced in a study conducted by Mayer [24]. However, in that case the tie-points reduction was part of the BBA. On the other hand, the proposed solutions are intended as a preliminary step to run before the BBA. In this way, the memory requirements of the BBA are decreased since the beginning preventing the need for large memory use.
In the following subsections the overview of the two different algorithms is given. The first algorithm performs the reduction using only the information provided by the tie-points in image space, while the second approach exploits the information provided by the tie-points both in image and object space to assure a better distribution in both spaces.
For an extended description and a study on the impact in the quality of the produced point cloud using the reduced set of tie-points, the reader can refer to the online report of the project “Improving Open-Source Photogrammetric Workflows for Processing Big Datasets” funded by the Netherlands eScience Center [25].
Tie-points reduction in the image space (RedTieP)
The input of the RedTieP algorithm consists of the set of images and their corresponding sets of tie-points. The relative position between overlapping images can be given as optional input to improve the process. The tie-points reduction is divided in tasks as depicted in Fig. 2, and there are as many tasks as images. The steps performed for each task can be described as follows:
-
The master image, driving the reduction process, is initially determined: the ith task uses as master image the ith image from the list of images.
-
The overlapping (related) images are selected considering the ones that share tie-points with the master image. A minimum number of common tie-points must be available to consider an image as related (10 points is the default value).
-
The tie-points of all the related images are loaded. If the related image was a master in a previous task, then the reduced list of tie-points obtained by this earlier process is considered for that image.
-
The loaded tie-points are topologically merged into multi-tie-points (MTPs). This means that a correspondence is created for the tie-points in different images pairs that relate to the same object point. A MTP keeps track of the number of image pairs in which a related tie-point is visible. This value is called multiplicity.
-
The image space is divided in a regular grid (i.e. Von Gruber regions [26]). The algorithm fills the grids in with the positions of the MTPs. To locate a MTP in the image, the algorithm uses the position of the related tie-point in that image.
-
The reduction is driven by the master image grid. Ideally the algorithm stores a single MTP (the related tie-points in fact) for each grid cell of the master image. However, some checks are done in order to guarantee that the distribution of tie-points in the related images is not damaged. For each grid cell of the master image, the MTPs are sorted according to their G
a
i
n
MTP
value given by formula 1.
$$ Gain_{MTP} = \frac{M_{MTP}}{1 + (K \cdot \frac{A_{MTP}}{A_{median}})^{2}} $$
(1)
where M
MTP
is the multiplicity of the MTP; K is a threshold value defined by the user (the default value is set equal to 0.5 from the performed tests); A
MTP
is the estimated accuracy value of the MTP on the master image and it is computed as the highest residual value achieved by this point in the relative orientation with the other images; and A
median
is the median of relative orientation residuals. Note that if K is zero or the relative orientation is not used, the G
a
i
n
MTP
is directly equal to the multiplicity of the MTP.
The MTP with the highest G
a
i
n
MTP
in each grid cell of the master image is stored and the rest are removed if they meet the following conditions. A MTP can be removed (a) if none of its homologous points in the other images are part of the minimal set of reduced tie-points computed in a previous task with another master image (otherwise it would have been deleted before) and (b) if for all the image pairs another point is present in the same cell of the master image and in the corresponding cells of the overlapping images (in order to assure an homogeneous distribution of the points).
To overcome the cross-dependency of image pairs, the parallel algorithm guarantees that all the tasks running in parallel, at any time, process different image pairs. Noodles [27], a programmable workflow engine implemented in Python, has been adopted for the execution of these tasks in parallel.
Tie-points reduction in image and object space (OriRedTieP)
The second tie-points reduction algorithm is called OriRedTieP. It exploits the initial image orientation between adjacent stereo-pairs. In addition to the images and the tie-points per image pairs, this algorithm also requires the initial image orientation given by the pose initialization approach.
Figure 3 shows an overview of the steps performed by the OriRedTieP algorithm. From the initial pose estimation of the images, the algorithm projects the images in the 3D space to obtain the approximated 2D footprints. The object space is roughly defined by intersecting the tie-points extracted in the previous phase. The algorithm computes a bounding box containing the footprints of all the images and splits the bounding box in tiles. For each tile, the algorithm determines which images overlap it. The following of the process is split in tasks and the tiles are processed independently. For each tile, the algorithm follows a sequence of steps:
-
The image pairs insisting on the tile and the corresponding tie-points are initially loaded. The tie-points with the estimated XY coordinates outside the tile are discarded.
-
The loaded tie-points are topologically merged into 3D multi-tie-points (3DMTPs). This means that a correspondence is created for the tie-points in different images pairs that relate to the same point in the object space. For each 3DMTP the algorithm determines its multiplicity (number of image pairs in which a related tie-point is visible) and its estimated 3D position in the space computed from the initial image orientation.
-
A Quad-Tree with the 2D (XY) coordinates of the 3DMTPs is created.
-
A Heap data structure is then generated to sort the 3DMTPs according to their G
a
i
n
3D
M
T
P
value. The formula 2 is used to compute the G
a
i
n
3D
M
T
P
.
$$ Gain_{3DMTP} = \frac{M_{3DMTP}}{1 + \left(K \cdot \frac{E_{3DMTP}}{E_{median}}\right)^{2}} \cdot \left(0.5 + \frac{n_{3DMTP}}{N_{3DMTP}} \right) $$
(2)
where M
3D
M
T
P
is the multiplicity of the 3DMTP; K is a threshold value configurable by the user (the default value is 0.5); E
3D
M
T
P
is the estimated error when computing the 3D position of the 3DMTP given by the forward intersection; E
median
is the median of estimated residual of all the 3DMTPs; N
3D
M
T
P
is the number of images where the tie-point is visible; n
3D
M
T
P
is the number of images where there are no other points in the neighborhood that have been already selected by the reduction process. Note that the value of n
3D
M
T
P
is dynamic and it changes every time a neighbor 3DMTP is preserved in the reduction process.
-
The 3DMTPs are reduced in an iterative process using the Heap data structure. The 3DMTPs with higher G
a
i
n
3D
M
T
P
are selected to be kept while their neighboring 3DMTPs are deleted only if the homogenous distribution of tie-points in the 3D object space is not compromised. In other words, a 3DMTP is usually deleted if it is visible in the same images of another 3DMTP that has been already preserved in the reduction process.
-
Before storing the reduced set of tie-points, a final validation of the points in the image space is performed. This process is based on the von Gruber point locations [26]. For each deleted tie-point and in each image pair, the algorithm checks which is the closest tie-point which has been preserved. If that tie-point is further than a certain distance, then the deleted tie-point is re-added to the reduced list. The distance threshold is defined by the user.
The tasks for the different tiles can be executed in parallel. When all the tiles have been processed, the algorithm merges the tie-points of the image pairs that are processed in more than one tile.
Distributed computing
In the previous section we proposed two algorithms to tackle the memory requirements of the BBA when dealing with large image sets: the solution is to reduce the amount of processed data to a lower, but sufficient, number of tie-points. For the other challenging components of the photogrammetric workflows, i.e. tie-points extraction, dense image matching and true-orthophoto generation, distributed computing hardware systems can be used since the involved processes can be divided in chunks that can be processed independently (pleasingly parallel). In the next subsections we describe the distributed computing algorithms.
In “Methods” section we mentioned that in MicMac the true-orthophoto is actually generated before the dense colored point cloud. Therefore, both the dense image matching and the true-orthophoto generation are tackled by the distributed computing solution presented.
Tie-points extraction
The current single-machine implementation available in the Tapioca command of MicMac is the following:
-
For each image of the dataset, the set of corresponding overlapping images is first determined. This process can be eased by the image geo-tag, reducing the number of homologous images to a small set. If the geo-tag is not available, then the whole dataset needs to be considered;
-
For each image the features are extracted;
-
For each stereopair, the features are matched to determine the candidate tie-points. The extraction and matching process are performed in parallel exploiting all the cores of the machine. However, this process is insufficient to process very large datasets.
The distributed computing solution is depicted in Fig. 4. The algorithm splits the list of stereopairs (image pairs) in chunks of N stereopairs where N is configured by the user: the first chunk contains the stereopairs from the position 0 to N-1 of the original list, the second chunk contains the stereopairs from the position N to 2N - 1 of the original list and so on. Each chunk can be processed independently and in a different machine. For each chunk, the algorithms works according to the following steps:
-
The list of images in the chunk is determined from the list of image pairs. At this point the images are copied to the machine where the chunk is going to be processed.
-
The feature extraction on the images belonging to the chunk is performed.
-
The feature matching among the image pairs of the chunk is computed.
The tie-points generated by all the chunks are finally combined. Note that since the distribution of the images in chunks is performed by image pairs, it may happen that different chunks use the same images. This means that the feature extraction step is duplicated in those images. This could be avoided by a two-step distributing computing solution where the feature extraction is completely performed in the first step while the second step is devoted to the feature matching among image pairs. However, due to implementation details (separating feature extraction and feature matching in Tapioca is not trivial) the two-step distributed computing solutions for tie-points extraction have not been considered at this stage of the work.
Dense image matching
The current single-machine implementation available in MicMac to generate the dense colored point cloud is the following: (i) the dense image matching is run and a dense depth map, i.e. a Digital Surface Model (DSM), is initially generated; (ii) using the DSM and the image spectral information, a true-orthophoto is then produced; (iii) the true-orthophoto and the DSM are finally used to generate the colored dense point cloud.
The process is generally divided in tiles and each of them is processed independently by a core. The GPU can be also adopted for this parallel process. However, the process can take very long times on a single machine. For a large dataset, a distributed computing solution is therefore required.
The distributed computing dense image matching algorithm is shown in Fig. 5. This algorithm has been specifically conceived for aerial images oriented in a cartographic reference system. The algorithm assumes that the Z component of the reference system does not have a real impact on the definition of the matching area. A 2D (XY) bounding box of the camera positions on the ground is therefore computed using the image orientation parameters. Then, this ground bounding box is split in tiles as shown in Fig. 6.
Considering the 2D (XY) coordinates of the image projection centres, the algorithm determines the images that overlap each tile: these images are initially considered for this tile. Then the images partially overlapping with the tile are added as well: for this purpose, the images neighbouring the center and the vertices of the tile are considered. For each of these points the algorithm determines the n closest images (with n equals to 6, by default) and it adds them to the list of images, as shown in Fig. 7.
After the list of overlapping images is determined for each tile, every independent process can be run in a different machine. For each tile the following tasks are executed:
-
All the images corresponding to the processed tile as well as their orientation are copied to the machine where the tile is processed.
-
The dense image matching algorithm is run considering the tile extension as area of interest. A DSM is therefore generated.
-
The true-orthophoto of the tile is then generated.
-
Using the DSM and the colour information of the true-orthophoto, the point cloud is generated.
-
This point cloud is exported in order to be merged with the other tiles.
Since all the generated point clouds are in the same reference system they can be easily merged together.
Implementation
The tie-points reduction algorithms are implemented as stand-alone commands within the MicMac tool. The names of the commands are RedTieP and OriRedTieP.
The distributed computing algorithms for tie-points extraction and dense image matching are implemented in pymicmac (https://github.com/ImproPhoto/pymicmac). Pymicmac is a Python interface for MicMac workflows execution. It allows to define through a XML file photogrammetric workflows to be executed with MicMac. A sequence of MicMac commands is specified in the XML file. During the execution of each command pymicmac also monitors the usage of CPU, memory and disk. Pymicmac can also define parallel commands via XML, using a set of commands to be executed in parallel. This is what is used for the distributed computing tie-points extraction and the distributed computing dense image matching. The sequential and parallel commands defined in pymicmac can be executed with pycoeman (https://github.com/NLeSC/pycoeman), a Python toolkit for executing command-line commands. Pycoeman has tools to run both sequential and parallel commands locally (in the local computer) as well as parallel commands in a set of remote hosts accessible via SSH as well as in SGE clusters (computer clusters with Sun Grid Engine batch-queuing system).