Progress in imaging and display optical systems exerts significant influences on the development of science and technology. Imaging and display systems intrinsically utilize optical elements (geometric or phase elements) to modulate optical wavefronts and achieve expectational imaging relationships, system specifications, and structure requirements. As the representative elements of geometric and phase elements respectively, freeform optical elements (FOEs) and holographic optical elements (HOEs) have significant advantages in optical system design. FOEs possess high degrees of design freedom, which can greatly enhance the ability to modulate wavefronts and improve imaging performance. Additionally, freeform surfaces can correct the aberrations of optical systems with off-axis nonsymmetric structures. Meanwhile, HOEs can unconventionally deflect rays at large angles due to their unique ability to modulate optical wavefronts. They can dramatically reduce the weight and volume of optical systems due to the lightweight form factor, and realize better optical see-through experiences and full-color display due to unique selectivity and multiplex ability, achieving mass productions owing to relatively simple fabrication methods and low costs. Meanwhile, it is easy to fabricate HOEs with large sizes due to the unique fabrication methods. Considering the above-mentioned advantages, designers may design imaging and display optical systems that combine FOEs and HOEs, significantly improving the degrees of design freedom and the ability to correct aberrations. Additionally, we can achieve advanced system specifications, excellent system performance, compact and lightweight system forms, and unconventional system structures with off-axis nonsymmetry, with further development of optical systems promoted. It is important to summarize the existing design methods of imaging and display systems combining FOEs and HOEs, analyze the problems restricting their further development, and predict the development trends. Meanwhile, it is essential to summarize the existing designs and applications of these systems to better guide and promote the development.

We describe the basic principles, ray-tracing models, advantages, and applications of FOEs and HOEs respectively, summarize the system design methods, review the designs and applications of these systems, and analyze current restrictions and future development trends. The design of these systems can be divided into three types. 1) FOEs and HOEs are simultaneously utilized to correct the aberrations of optical systems. 2) The freeform surface is adopted as the substrate shape of HOEs. 3) During HOE fabrication, FOEs are introduced to modulate the recording waves of HOEs. In practical optical system designs, the design can be a combination of the above three ways. The first way directly builds ray-tracing models of freeform optics and HOEs in the optical system design and then adopts the optimization strategy to achieve expectational requirements. The second way coats the holographic recording medium on the freeform substrate to yield HOEs with freeform substrates. The third way bridges the numerical relationship between freeform optics and recording waves of HOEs to fabricate HOEs with unconventional profiles of holographic phase function or grating vector. The methods for defining HOEs based on ray tracing are described in detail, including the phase functions (direction cosines) of the recording waves, holographic phase function, and holographic grating vector, which guides the basic combined design schemes. We review the ways of fabricating HOEs including the whole-area exposing and sub-area exposing (holographic printing) to provide references for combined design fabrication. The calculation methods of starting points of optical systems based on HOEs are summarized in detail, including point-by-point construction and iteration methods, confocal methods, and simultaneous multiple surface (SMS) methods, which guide the design of the optical system combining FOEs and HOEs. The designs and applications of these systems are summarized based on the classifications of HOEs, including augmented reality (AR) near-eye display systems, head-up display (HUD) systems, and HOE-lens imaging systems. Additionally, combined designs of freeform optics and other types of phase elements are also presented, such as liquid crystal polarization hologram (LCPH) based on freeform exposure, and metasurfaces with freeform substrate, which has certain guidance for the combined design of FOEs and HOEs.

Studies on the system design combining FOEs and HOEs make significant progress in the basic principles, design frameworks, and fabrication methods, which has been employed for developing imaging and display systems with high performance, novel structure, and lightweight form factor. There are also some problems and challenges for the research on the system design combining FOEs and HOEs. They include how to fabricate HOEs with freeform substrates by innovative coating technologies of the holographic recording medium, how to correct chromatic aberrations in the imaging and display system using HOEs, how to reduce the nonuniformity of diffraction efficiency and stray light of systems combining FOEs and HOEs, and how to conduct tolerance analysis of such systems. In summary, the research on the design of imaging and display systems combining FOEs and HOEs will promote the development of next-generation high-performance and compact optical systems.

Color reproduction plays a very important role in textile, printing, telemedicine, and other industries, but affected by the manufacturing process or color rendering mechanism of digital image acquisition equipment, color image transmission between digital devices often has color distortion. Meanwhile, once the distortion appears, the above-mentioned industries will suffer losses or even irreversible damage. During color image acquisition, the most commonly employed acquisition equipment is the digital camera, which is an important method to convert the color image collected by the digital camera into the image seen by the human eye (or the camera characteristic method). Although the existing nonlinear camera characterization methods have the best camera characterization performance at present, these methods have hue distortion. To retain the important properties of the hue-plane preserving and further improve the camera characterization performance, we propose a hue-subregion weighted constrained hue-plane preserving camera characterization (HPPCC-NWCM) method.

The proposed method improves weighted constrained hue-plane preserving camera characterization from the perspective of optimizing the hue-subregion. First, the camera response value RGBs and the colorimetric value XYZs of the training samples are synchronously preprocessed, with the hue angles calculated and hue subregions preliminarily divided. Then, by operating in the hue subregion, the minimum hue angle differences between each training sample and the samples in the hue subregion are employed as the weighted power function, and the pre-calculation camera characterization matrices (pre-calculation matrices) are calculated for each sample respectively. Additionally, the weighted constrained normalized camera characterization matrix in the hue subregion is obtained by weighted averaging of the pre-calculation matrices using the weighted power function. Combined with the characterization results of samples within the hue subregion and all samples, the number and position of the hue subregions are optimized, and those under the best performance are obtained. To verify the performance improvement of this method, we conduct simulation experiments. Firstly, the hue-subregion number selection experiment is carried out by combining three cameras and three groups of object reflectance datasets under the D65 light illuminant. Then, the two cameras from the previous experimental data are compared with existing methods for further experiments and the exposure independence of each method is verified by changing the exposure level. Finally, the SFU dataset is compared with the existing methods repeatedly with 42 cameras under three light illuminants.

Verified by many simulation experiments and real camera experiments, in the simulation experiment of selecting the hue-subregion number, the camera characterization performance of this method is generally enhanced with the increasing hue-subregion number (Fig. 7), tends to stabilize when the number is 6, and yields the best performance when the number is 9. The performance of the subregion number 2 is worse than that of 1, and the analysis is that the small subregion number results in poor universality and low specificity of the characterization matrix in the hue subregion, which affects the characterization performance of the camera. After comparing the simulation experiment with the existing methods, the performance of this method is about 10% to 20% higher than those of the existing hue-plane preserving camera characterization methods, and it is better than or close to the nonlinear method (Table 1). In the variable exposure experiment, the performance of each method is close to that of the fixed exposure experiment, and that of the linear method and the root-polynomial method is close, which can prove the exposure independence. While the polynomial method is obviously worse, exposure independence does not exist (Tables 1 and 2). In the simulation experiments of supplementary light illuminants and cameras, the comparison trend of the results is basically the same as that of the previous experiment, and this method performs better in the supplementary experiment. In addition to being better than the existing camera characterization methods, it can be better than or equal to the nonlinear methods in many environments (Table 3).

By optimizing the hue subregion to improve the weighted constrained hue-plane preserving camera characterization method, the number and position of the hue subregion are optimized to achieve a more accurate camera characterization transformation for different hue subregions. By adopting the theoretical derivation and experimental verification of camera characterization transformation, this method features exposure independence, excellent hue-plane preservation properties, and the combination of the stability of low-order methods and the accuracy of high-order methods. In simulation experiments, it can be better than the existing hue-plane preservation methods, and better than or close to other nonlinear methods. In multi-camera supplementary experiments, the 95 percentile error improvement shows that this method has strong robustness and practical significance.