As per holographic printing system requirements, the pre-processing stage has to generate a series of two-dimensional (2D) images of the object(s) to be imaged on the holographic print, and said images shall fulfill certain geometrical assumptions. There is no limitation for using real or virtual objects or even a combination of these. One possible application of the mentioned system is digital documentation and archiving of objects of cultural heritage. A generation of three-dimensional (3D) digital models of such objects is very popular nowadays and applied widely in practice. Obtained results are usually used as a presentation of a virtual model of the real objects on the computer screen, as passive or interactive objects. There is no way to display or visualize such models without optical aids such as virtual reality goggles. The only way to produce 3D images visible to the naked eye is holographic prints. The standard way of using 3D scanning data for the holographic printing process includes conversion of acquired data (cloud of points) into triangle mesh, texturing, virtual light operation, and finally generating input data for the holoprinter. Disadvantages of such a way are highly noticeable when the objects have a complex shape, which in effect causes errors and/or discontinuities in a resulting model. To eliminate these faults in the final virtual 3D model, a time-consuming manual operation performed by a highly experienced person is needed. Elimination of this stage of manual operation simplifies a pre-processing procedure, but the whole data acquisition process requires a proper scanning system anyway. In the experiment, we used an automated Optographx (OGX) scanner applied by Wilanów Historic Museum in Warsaw (Polska). A cloud of points is generated by this scanner.

Shape measurements of microscopic objects can be done by active or passive systems. Widely known active systems use in most cases structures or patterns projected onto an object under measurements, and then a shape reproduction is calculated. The disadvantage of these methods is the limitation of applicability for diffused objects only. Any type of reflective surfaces can be measured only after application of the scattering layer (which is hard to apply and/or hard to remove). Passive systems do not project any patterns on measured objects, and thus the camera cannot be blinded by light reflected from any part of the object. Consequently, for some object types (i.e., reflective or partially reflective ones), the passive systems are better. On the other hand, many of active and passive methods of shape measurements are completely useless for objects variable in time.

Among popular passive methods used for microscopic objects, confocal microscopy can be noticed, for instance^[1]. This technique is generally based on quite simple rules to acquire multi-focused images (acquired at many different focal planes). In practice, it needs to use precise, quite complicated, and thus expensive equipment. In this Letter, a very simple shape measurement system for microscopic objects with no calibration demand is presented. The system is based on the widely known structure from motion (SfM) technique^[2–5]. Thanks to the applied focus stacking (FS) method^[6,7], the presented system uses very simple hardware and can be applied for any type of microscopic object with practically unlimited depth. The combination of these two techniques in a hologram registration process is a new completely approach.

The shape measurement system presented in this Letter can reconstruct 3D shapes of the microscopic object from multiple camera views. The SfM theorem assumes unknown or partially unknown parameters of the camera, which additionally may be unstable in time. With the use of a set of n images taken from m fixed viewpoints, the SfM method can estimate n projection matrices, and thus an estimated 3D point cloud of the object can be achieved. However, in this method, there is a kind of latitude allowed during measurements (no need for any additional procedures, adjustments, or calibration), and the SfM method hardly charges computer resources. In typical applications, the SfM method uses images in which the number of pixels ranges from 1 to 5 million pixels. We decided to use the collection of input images with approx. 24 million effective pixels to reduce the total number of images to a possible minimum. Higher resolution of input images increases the quality of shape reconstruction in the SfM method very effectively^[8]. Based on the anthropometric measurement, the results can help conclude that the higher the resolution of the input images, the higher the number of points in the cloud (of the reconstructed object) that can be calculated. Consequently, the smallest distance possible to be measured is smaller, and the measurement error also decreases.

Images taken with the use of traditional optical microscopes are characterized by an insufficient (in most cases) depth of field (DoF). However, the optical microscopes nowadays are present practically everywhere; the availability and ease of use is excellent, but the insufficient DoF is a huge disadvantage of it. One of the possible solutions for this problem can be the use of the FS technique, which allows us to increase the depth of focus by use of multiple images. The FS method applied for the mentioned purpose only is relatively simple in use and is effectively utilized by many classic microscopy users [i.e., the extended DoF (EDF) imaging method]^[9].

However, the SfM method of 3D model creation is relatively tolerant of image imperfections (for example, caused by aberrations); it fails when the algorithm does not recognize sufficient similarities (i.e., matching points) in a series of source images of the same object area taken from different perspectives. This will occur even when input images are blurry or have insufficient depth of focus (can also be interpreted as a blur). In this Letter, we propose to link these two techniques for microscopy like it was done for other purposes by, for example, Clini et al.^[10] or Kontogianni et al.^[11]. The goal is to give the possibility of generating full 3D models of real objects in the microscopic scale with the use of very simple and low-cost hardware.

In the case of this work, we decided to use a biological specimen as an object—a prepared imago of the Asian beetle of Catoxantha opulenta family. It was originally glued to a small white base made of paper.

Images were registered with use of a high resolution color digital camera with a total number of effective pixels of 24 million, equipped with a specialized optical set-up, allowing the required magnification of the image to be equal to 1 and larger (Sony alpha 6000 with MD Macro Rokkor f=50mm, magnification β≈0.8-1,f/11). All exposure parameters were adjusted manually to maintain constant exposition of each frame during the registration process. Illumination was homogenous and shadowless in the volume of the scene (diffused light sources) with the daylight spectrum (approx. 5000 K).

The prepared images used later in the FS process were taken as a consecutive series varying by points of view (perspective). The origin position of the camera was a result of changing angles of the x and y axes and (when applicable) magnification. At a defined perspective, a series of images was taken, where each consecutive frame differed with object distance. It is obvious when optical magnification is quite large and thus DoF is relatively small. Having a series of images with a small DoF along the object’s depth, the FS technique can be used to achieve a final sharp image of the object. The scheme for recording is shown in Fig. 1.

The process of registration of input images consists of two consecutive stages: registration of a series of images with a fixed direction of the optical axis, the optical axis aligned with the center of the specimen, and variable object distance for increasing DoF; repeating the series as above with variable camera positions (with variable angles between the optical axis and base stage plane and variable angles of stage rotation around its own axis). All images were recorded as RAW-type files and were processed with use of Photoshop software FS method.

The SfM method implemented in VisualSfM software^[12] consists of two steps, bundle adjustment and final cloud of points calculation. The bundle adjustment algorithm used in the software was developed by Wu et al.^[13]. The dense reconstruction is done with Furukawa’s patch-based multi-view stereo (PMVS)/clustering multi-view stereo (CMVS) tool chain. Proposed by Furukawa and Ponce^[14], the algorithm for multi-view stereopsis produces as a result a dense set of small rectangular patches, which are partly visible in the images. It starts from a sparse set of matched keypoints, and then in a loop the set is by turns expanded and filtered to discard false matches. An important feature of this algorithm is that it automatically detects obstacles and outliers and removes them.

Each resulting image taken after the FS process (with large DoF) was then used as an input image for the SfM method. The SfM algorithm needs to use more than a few images (practically, a minimum of ten) from different viewpoints. Additionally, input images should give the possibility of finding similarities (matches) between neighboring images; thus, they have to overlap, and each of the selected viewpoints cannot differ in distance too much. In the case when the specimen rotates, it means that each rotation should possibly be small—in the experiment, it was maintained to not to exceed approx. 10 deg between consecutive camera positions. A total number of 10 final input images (with increased DoF) showing specimens from varying camera viewpoints was taken into the SfM for 3D data reconstruction. Samples of these images are shown in Fig. 2. In the next step, collected data was imported into 3Ds Max software. In the software, preparation of the series of input images for the holoprinter was performed with use of necessary geometry of the set-up (see Fig. 3). With the use of the prepared images, representing single-parallax multi-views of the object, a final hologram is printed. Geola’s digital holographic printing system is based on three channel optical set-ups for red, green, and blue laser beams, consecutively (see Fig. 4). A final hologram is made of series of hogels, which consists of data calculated from a series of images as a object beam (see Fig. 5).

Final results of a shape reconstruction—a point cloud view—are shown in Fig. 6. As can be noticed, the correct shape of the specimen (insect) is present. Some errors and lack of points in selected areas are caused by imperfections of the dense reconstruction algorithm applied in the SfM, which sometimes has problems with finding matching patches in the analyzed images. The areas where the lack of points in the cloud occur can be relatively easy to fill up in a 3D modeling software such as Autodesk 3Ds Max, Blender, or others. It can be done before or after conversion to the triangle mesh, but this is a tedious and usually time-consuming process (generally needs manual operation).

In Fig. 7, a reconstructed model of the insect is shown as a set of axial slices. It shows a spatial shape of the specimen. It proves that the obtained result can be used for further processing, i.e., converting to the triangular grid production and rendering the final virtual model by real textures. The proposed method is a part of data acquisition in the complete generation of the virtual model of the real objects in the microscopic scale.

The flowchart of the process is given in Fig. 8. The estimated processing time for the selected steps is also given with the use of a personal computer (PC) station based on i7 CPU, medium class advanced micro devices (AMD) graphics processing unit (GPU), and 16 GB random-access memory (RAM).

With use of the data collected and processed and a digital holographic printer described above, a final hologram of the virtual object represented as a cloud of points was printed. The size of the hologram is 200mm×280mm. It is printed on the high resolution silver-halide photomaterial specially developed for this printing process by Geola (Lithuania) and Sfera-S (Russia)^[15] (resolution grain size 8–10nm, 10,000 l/mm). It was developed in a dedicated holographic developer (SM-6), bleached in PBU-Amidol and dried in isopropyl alcohol. A detailed chemical processing procedure is described on Geola’s website^[16]. After processing, the film with the hologram was laminated onto black plastic using 3M optically clear pressure sensitive laminating material.

Proposed fusion of two well-known methods allows us to correctly reconstruct a 3D shape of the small microscopic objects (i.e., small insects). This method can be applied where specialized hardware is not present. All of the experimental work can be performed with the use of very basic equipment: a typical microscope with a camera mount, high resolution digital camera, and simple diffused light source. The simplification on the hardware side is paid by more manual work that is necessary to capture hundreds of images (a dozen or so for FS calculation multiplied by number of different camera viewpoints needed by SfM). The registration of the series of images at different object distances can be automatized, of course, but if so it leads to evidently expanded hardware equipment.

However, the proposed method needs a lot of calculations to be performed in post processing; the satisfactory results can be done without use of specialized software. In this Letter for SfM, reconstruction freeware and open access software were used.

Objects that are relatively complicated (as an insect shown in this experiment) need as many as possible different views for reconstruction. The number of 10 views is absolutely the minimum, and, in future work, it should be greater. In the experimental example shown in this Letter, the significant increase of the number of views (with large DoF) will give better shape reconstruction (i.e., more points in the cloud, less “blank” areas). It is clear that the number of views necessary for good quality shape reconstruction strongly depends on the actual object selection and should be determined individually for actual conditions and used equipment.

As it was proved, the data collected within the presented experiment can be used for further processing in virtual space and/or 3D computer graphics. It allows us to acquire shapes of real objects of a microscopic scale without use of sophisticated hardware.

This method of shape reconstruction does not quite need any special conditions regarding the specimen used, and thus it can be applied in many applications and many types of objects. The only limitation is to have objects stable in time (possibly lifeless or stable in time).

Digital holographic printers developed by Geola Digital for hologram production need parallax-related image sequences. Instead of the well-known method of data recording from real objects by use of HoloCams, said holographic printers also accept images obtained from virtual representations of the object. In this Letter, a range of possible representation of virtual objects is widened by a cloud of points. In comparison to the “standard” method using a complex textured shape of the triangular net, the presented method eliminates one step of data processing. It can be applied in the systems where the conversion from cloud of points to triangular mesh is impossible, complicated, or excessively time consuming.