The goal of our research is to develop a real-time free-viewpoint image synthesis system for dynamic scenes using
multi-view video cameras. To this end, depth estimation that is efficient and suitable for dynamic scenes is
indispensable. A promising solution for this is view-dependent depth estimation where per-pixel depth maps are
estimated directly for the target views to synthesize. Such view dependent methods were successfully adopted in
previous works, but their depth estimation quality was limited especially for textureless objects, resulting in low quality
virtual views. This limitation comes from a fact that their depth estimation depended only on passive approaches such
as traditional stereo triangulation. To tackle this problem, we considered to use active methods in addition to the
passive stereo triangulation. Inspired by the success of recent commercial depth cameras, we developed a customized
active illumination using a DLP projector. The projector casts spacially incoherrent patterns to the scene and makes
textureless regions identifiable from the cameras, so that stereo triangulation among the multi-view cameras can be
greatly improved. Moreover, making the illuminations time-varying, we can stabilize depth estimation more by using
spatiotemporal matching across multi-view cameras based on the concept of spacetime stereo method, and also remove
the artificial patterns from the synthesized virtual views by averaging successive time frames. Our system consisting of
16 video cameras synchronized with the DLP projector runs in real-time (about 10 fps) thanks to our sophisticated
GPGPU implementation.
In this paper, we discuss a Free viewpoint synthesis with View + Depth format for Multiview applications such as 3DTV and Free View-point Television(FTV)1.When generating a virtual image, 3D warping is used with view and depth of a reference camera. This process includes the problem that holes appear in the virtual image. In conventional method, the holes were dealt with collectively by median filter. There are some different reasons why holes appear through this process. So, it is improper that they are not distinguished particularly and treated all at once like conventional method. We analyze the factors, and recognize that two ones exist, boundary between foreground and background, and reduction of resolution. In this paper, we propose a new hole filling method considering these factors. In the first step, we classify nearby pixels into boundary or same object area according to the gradient of depth value. For boundary case, we hold them and refer to other two real cameras. For another case of same object area, we set up sub-pixels between nearby pixels and warp them if the depth is gradually changing or virtual viewpoint of the warped image is closer to the object than the original view position because they probably cause holes from reduction of resolution. We implement these methods in the simulation. As a result, we prevent boundary in the virtual image from being ambiguous, and confirm the availability of proposed method.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.