2.1.

Integral Three-Dimensional Display Using a Single Display Device

First, as a premise, a method for reconstructing an integral 3-D image using a single display device will be described. The differences in how the viewing zone of the integral 3-D image is formed for the case where a direct-view display is used and the case where a projector is used for the display device are explained in Fig. 1. As shown in Fig. 1(a), for the method using a direct-view display, elemental images are displayed on a liquid-crystal display or organic light-emitting diode panel, a lens array is placed in front of the direct-view display, and an integral 3-D image is reconstructed. The light-emitting part of the pixels of the direct-view display and the lens array are arranged such that the distance between them is equal to the focal length of the elemental lens $f_{\mathrm{l}}$. Moreover, a viewing zone is formed by the elemental image and the corresponding elemental lens. However, since the light of the pixels of the display is diffused, the next elemental image will be seen through the elemental lens at a viewing position outside the central viewing zone. The viewing zone at the center is called the primary viewing zone, and the viewing zones at other locations are called the secondary viewing zones. On the other hand, as shown in Fig. 1(b), when reconstructing an integral 3-D image using a projector, the light rays passing through a display element of the projector are emitted from the projection lens of the projector and spread in proportion to the projection distance. By arranging the collimating lens in front of the projector so that the distance between the projection lens of the projector and the collimating lens is the same as the focal length of the collimating lens $f_{\mathrm{p}}$, the projection light is collimated after expansion to the desired projection size. Therefore, the integral 3-D image is reconstructed by passing it through the lens array so that the elemental image and the elemental lens correspond to each other. The focal plane of the projected image should be positioned away from the lens array by a distance equal to $f_{\mathrm{l}}$. Since the projection size is somewhat large, a Fresnel lens, which can be fabricated with a large diameter, is used as the collimating lens.

Fig. 1

Difference in how viewing zones of integral 3-D image are formed when a single display device is used: (a) direct-view display and (b) projector.

In the method using the direct-view panel, since the light rays are diffused as described above, an image at the secondary viewing zones is reconstructed in addition to the primary viewing zone. On the other hand, in the method using the projector, an image at the secondary viewing zones is not formed since only the integral 3-D image, which is formed with the corresponding elemental image and elemental lens, is reconstructed. Therefore, a 3-D image formed by one projector is reconstructed only around the direction of projection, and the secondary viewing zone does not appear.

2.2.

Integral Three-Dimensional Display Using Multiple Projectors

By applying the aforementioned features, an optical system using multiple projectors is constructed as shown in Fig. 2. If multiple projectors project images from different directions and the viewing zones formed by each projector are combined, the viewing zone can be enlarged according to the number of projectors. For example, if two projectors are used in the horizontal direction and two projectors are used in the vertical direction, it is possible to reconstruct a 3-D image having twice the viewing angle in both the horizontal and vertical directions. It is necessary for all of the projectors to have the same image size on the focal plane, so the sizes of the 3-D images reconstructed by each projector are equal.

Fig. 2

Integral 3-D display using multiple projectors.

There are two problems to be solved when constituting an integral 3-D display with a wide viewing zone using multiple projectors. The first problem is the distortion in the projected image. Owing to projection from an oblique direction and the aberration of the collimating lens, the projected image is distorted on the focal plane, and the 3-D image cannot be reconstructed correctly. The second problem is that the 3-D image becomes discontinuous at the parts where the viewing zones are connected, unless elemental images corresponding to each projection angle are input for each projector. A method for solving these two problems and reconstructing a natural wide-view integral 3-D image without distortion will be described in Secs. 3 and 4, respectively.

3.1.

Image Distortion Caused by Projection

The image projected by the projector is distorted by aberration when it passes through the collimating lens. Moreover, when using a projector that does not have a sufficiently large lens shift or has no lens shift function, it is necessary to project the image to the collimating lens from an oblique direction by tilting the projector, resulting in a trapezoidal distortion in the projected image. When the projected image is distorted on the focal plane of the projected image, the correspondence between the lens array and the elemental image is lost, and it is not possible to display the desired integral 3-D image.²¹ Figure 3 shows an example of a reconstructed integral 3-D image when projecting from an oblique direction using one projector, a collimating lens, and a lens array. When viewing an image without correcting the projection distortion, the correspondence between the elemental image and the elemental lens is lost, and the desired integral 3-D image cannot be obtained, as shown in Fig. 3(a). By correcting the projection distortion and matching the shapes and positions of the elemental image and elemental lens, it is possible to reconstruct the desired integral 3-D image, as shown in Fig. 3(b).

Fig. 3

Example of a reconstructed integral 3-D image using one projector: (a) before and (b) after correcting for projection distortion.

3.2.

Distortion Correction Method Using Image Processing

To solve this problem, image processing is applied to the elemental-images that are input to the projector, and the projection distortion is eliminated. As the image-processing method, a distortion correction method combining projective and affine transformations is applied.^19,20 The device setup for projection distortion correction is shown in Fig 4. A diffuser plate is placed on the focal plane of the projected image, and image correction is performed while viewing the image projected onto the diffuser plate. An image of a triangular mesh is input to the projector and projected onto the diffuser plate. A reference sheet of a triangular mesh of the desired screen size is prepared, and the projected image of the triangular mesh is dynamically changed to match that of the reference sheet using image processing. The equation for combining the two transformations is expressed as

Fig. 4

Device setup for projection distortion correction.

A flowchart of the projection distortion correction method for multiple projectors is shown in Fig. 5. Distortion correction is performed for each projector, and a correction matrix for each projector is derived. Consequently, the images projected by all projectors are corrected, so they have the same size on the focal plane. After correction, the diffuser plate is removed, and the integral 3-D image is reconstructed by arranging the lens array away from the focal plane of the projected image at a distance equal to the focal length of the elemental lens.

Fig. 5

Flowchart of the projection distortion correction method for multiple projectors.

4.1.

Discontinuity Between Different Viewing Zones

Viewers can see a 3-D image while moving through a wide angle by enlarging the viewing zone using multiple projectors. Therefore, it is necessary to smoothly combine the images of multiple viewing zones to see the 3-D image naturally. It is necessary for each projector to input elemental images corresponding to each projection angle. Although the projection angle can be roughly calculated from the arrangement of the optical system, a discontinuity due to misalignment of the optical system practically occurs at the parts connecting different viewing zones of the reconstructed 3-D image. For a more precise combination, we propose a method that carries out dynamic correction while changing elemental images by referring to the reconstructed 3-D image.

4.2.

High-Speed Generation of Elemental Images for Adjustment

By making it possible to render elemental images of a CG model in real time and making adjustments while dynamically changing the elemental images, it is possible to derive the projection angles of multiple projectors quickly and precisely. Therefore, a method for rendering elemental images of the CG model at high speed is applied.²² In this method, when creating elemental images using the CG model, the pixels in the same direction are collectively acquired using virtual cameras, so the calculation is performed efficiently. With this method, it is possible to reduce the number of cameras arranged in the 3-D virtual space to approximately the number of pixels of an elemental image, and it is possible to increase the speed at which elemental images are generated. Figure 6 shows collective light-ray acquisition by virtual cameras for the high-speed generation of elemental images corresponding to each projector. By changing the direction of acquisition of the virtual cameras in the 3-D virtual space, it is possible to generate elemental images according to the projection angle $\theta$ of the projector, as shown in Fig. 6. Using this method, elemental images are dynamically generated while minutely changing parameters in the direction of projection, and an adjustment is made so that the 3-D images reconstructed by multiple projectors are continuously combined.

Fig. 6

Collective light-ray acquisition by virtual cameras for the high-speed generation of elemental images corresponding to each projector: projector (a) A and (b) B.

4.3.

Derivation of the Projection Angle According to Each Projector

Figure 7 shows a flowchart for deriving the projection angle. First, with respect to one projector (projector #1), the projection angle is determined from the arrangement of the optical system, and elemental images are created. The 3-D image reconstructed by this projector is used as a reference. Next, for another projector (projector #$i$), the elemental images are created on the basis of the arrangement of the optical system in the same way. The elemental images of projector #$i$ are dynamically changed so that the positions of the 3-D images reconstructed by the two projectors match, and the projection angle ${\theta }_i$ of projector #$i$ is derived. The above procedure is repeated for all projectors with projector #1 as a reference, and the projection angles of all projectors are determined.

Fig. 7

Flowchart of the projection angle derivation method for multiple projectors.

The image displayed in front of the condensing point of light rays is used as the 3-D image to be displayed for adjustment. This is because the amount of displacement is enlarged using the 3-D image distant from the condensing point; thus, adjustment with such an image results in adjustment with a higher precision. Considering the influence of blur in the integral 3-D image,⁹ the 3-D image is reconstructed at the maximum depth position where the maximum spatial frequency is maintained. A square object without depth is used as an example of an integral 3-D image (Fig. 8). Figure 8 was taken from the overlap part of the viewing zones. Adjustment is dynamically performed using the high-speed elemental-image generation method, so the 3-D images displayed using all the projectors overlap at the same position in the parts connecting the viewing zones. Then, the adjustment parameter of this state is derived as the projection angle for each projector. The adjustment was performed with a visual check. Once the projection angle is determined, as long as the optical system is not moved, elemental images can be created on the basis of the information of the projection angle when new elemental images are created, and multiple viewing zones can be continuously combined.

Fig. 8

Example of an integral 3-D image for adjustment when combining multiple viewing zones: (a) before and (b) after adjustment.

5.1.

Experiment for Verifying the Continuous Combination of Multiple Viewing Zones

First, we conducted an experiment to verify the continuous combination of the viewing zones of multiple projectors as described in Sec. 4. Two projectors were arranged in the horizontal direction, and the experiment was conducted to combine their viewing zones. The projector (DLA-PX1) is manufactured by JVCKENWOOD, and images with a resolution equivalent to 4 K are displayed by projection with a half-pixel shift of a high-definition image by time division (e-shift technology). After applying the projection distortion correction method described in Sec. 3, the method for adjusting the continuous combination of multiple viewing zones described in Sec. 4 was applied.

As described in Sec. 4, to adjust the projection angle dynamically, a program for generating elemental images at high speed was coded using OpenGL with the OpenGL Shading Language. A computer equipped with an Intel(R) Xeon(R) E5-2687W v4 central processing unit and an NVIDIA Quadro P5000 graphics card was used for the experiment. A $30\times 30$ array of virtual cameras was placed in the 3-D virtual space to generate elemental images with a resolution of $3840\times 2160\text{pixels}$. The frame rates at which elemental images were generated were 23.4 and 19.8 fps when rendering objects with 4 (a square model) and 34,834 (a bunny model) vertices, respectively. Using this program, elemental images can be generated almost in real time. Accordingly, it was possible to derive the projection angle of each projector and generate the corresponding elemental images quickly, precisely, and efficiently.

Figure 9 shows the integral 3-D images when viewed from the left, at the center, and from the right without or with adjustment of the projection angle when two projectors are used in the horizontal direction. In the parentheses in Fig. 9, the first and second values represent the angles of the viewing directions in the horizontal and vertical directions, respectively. In Fig. 9, the green ring is reconstructed about 32 mm in front of the condensing point, and the background of the chess board image is reconstructed about 16 mm behind the condensing point. Since it is difficult to combine multiple viewing zones without overlap, depending on the lens aberration, optical arrangement, and optical system specifications, we slightly overlapped two viewing zones. As shown in Fig. 9(a), when the adjustment for the continuous combination of viewing zones is not applied, multiple images appear since multiple 3-D images are combined in a state of inconsistency at the central viewpoint. When viewing while moving horizontally, the 3-D image was accompanied by a discontinuity at the part connecting multiple viewing zones. As shown in Fig. 9(b), the continuous and smooth combination of multiple viewing zones was realized by precisely deriving the projection angles of each projector using the proposed adjustment method and applying it to generate elemental images.

Fig. 9

Integral 3-D images viewed from various directions without or with adjustment of the projection angle when two projectors are used in the horizontal direction: (a) without and (b) with adjustment.

5.2.

Experiment Demonstrating an Enhancement in the Viewing Zone Using Four Projectors

Next, we built a prototype to combine four viewing zones using four projectors, a collimating lens, a lens array, and a computer for correcting the projection distortion and deriving the projection angle. Figure 10 shows the appearance of the prototype and the arrangement of four projectors, two horizontally and two vertically. A projector with a resolution equivalent to 4 K, which is the same as the projector used in the experiment presented in Sec. 5.1, was used. Since each projector is projected toward the center of the lens array, each viewing zone is formed in a direction diagonal to the arrangement of the projector. The total viewing zone is enlarged by combining four viewing zones. The specifications of the device are listed in Table 1.

Fig. 10

Appearance of the prototype device.

Table 1

Specifications of the prototype.

First, the method for correcting the projection distortion as described in Sec. 3 was performed by placing a diffuser plate at the focal position of the projector. The control points of the triangular mesh were detected with a visual check. Projected images without distortion were realized for all four projectors by image processing using projective transformation and affine transformation, and their transformation matrices were derived.

Next, the diffuser plate was removed, and the lens array was set away from the focal plane of the projected image at a distance equal to the focal length of the elemental lens. Then, the elemental images were dynamically changed using the high-speed generation method for the elemental images described in Sec. 4, and the desired projection angles for all projectors were derived.

Figure 11 shows an example of an integral 3-D image displayed by the prototype device when viewed from various directions. In Fig. 11, the green ring is reconstructed about 48 mm in front of the condensing point, the red bunny is reconstructed at the condensing point, and the background of the sky image is reconstructed about 48 mm behind the condensing point. It was confirmed that the 3-D image changes according to the observation direction, and multiple viewing zones are combined continuously and smoothly. When only one projector was used, the measured viewing angles were as narrow as 25.4 deg horizontally and 24.5 deg vertically. On the other hand, when four projectors were used, the measured viewing angles were as wide as 49.2 deg horizontally and 45.4 deg vertically. Compared with the case of only one projector, the viewing angle is expanded by a factor of $\sim 2$ in both the horizontal and vertical directions using four projectors. Regarding the 3-D image display performance of the prototype, the resolution of the 3-D image was $264(\mathrm{H})\times 148(\mathrm{V})$, the total number of pixels was 39,072, and the screen size was $320\mathrm{mm}(\mathrm{H})\times 180\mathrm{mm}(\mathrm{V})$.

Fig. 11

Example of an integral 3-D image displayed by the prototype device when viewed from various directions.

As described in Sec. 5.1, we slightly overlapped the parts connecting multiple viewing zones. In the overlapped parts, since light from multiple projectors is multiplexed, the luminance is higher than that in other parts. Therefore, by lowering the luminance value of the pixels of the elemental images corresponding to the connected parts, the luminance throughout the image can be smoothed.

Although we used a Fresnel lens as the collimating lens, it is considered that various types of aberrations appear because the imaging characteristics are not sufficient with only a single Fresnel lens. Using the proposed distortion correction method, the projected image can be transformed precisely into a desired shape, so the distortion due to the aberrations can be sufficiently suppressed. Moreover, by correcting distortion for each of the RGB images, the influence of chromatic aberration can be corrected.