This PDF file contains the front matter associated with SPIE Proceedings Volume 11766, including the Title Page, Copyright information, and Table of Contents.

The efficiency of lossless image coding depends on the pixel predictors, with which unknown pixels are predicted from already-processed pixels. Recent advances in deep learning brought new tools that can be used for pixel prediction, such as deep convolutional neural networks (CNNs). In this paper, we focus on the processing order of the pixels and propose a new pixel predictor constructed using CNNs. Instead of the conventional scanline order, we design a new processing order where the pixels are processed in a progressive, parallelizable manner and the reference pixels are located in all directions with respect to a target pixel. Our pixel predictor is implemented using a CNN architecture that was originally developed for image inpainting, a task of filling in missing pixels from known pixels in an image. We compare the performance of our method against the conventional scanlinebased CNN in terms of the potential coding efficiency and computational cost.

Multispectral pedestrian detection has attracted extensive attention, as paired RGB-thermal images can provide complementary patterns to deal with illumination changes in realistic scenarios. However, most of the existing deep-learning-based multispectral detectors extract features from RGB and thermal inputs separately, and fuse them by a simple concatenation operation. This fusion strategy is suboptimal, as undifferentiated concatenation for each region and feature channel may hamper the optimal selection of complementary features from different modalities. To address this limitation, in this paper, we propose an attention-based cross-modality interaction (ACI) module, which aims to adaptively highlight and aggregate the discriminative regions and channels of the feature maps from RGB and thermal images. The proposed ACI module is deployed into multiple layers of a two-branch-based deep architecture, to capture the cross-modal interactions from diverse semantic levels, for illumination-invariant pedestrian detection. Experimental results on the public KAIST multispectral pedestrian benchmark show that the proposed method achieves state-of-the-art detection performance.

The latest video coding standard, Versatile Video Coding (VVC), has been finalized in July 2020. In traditional codecs, it is not possible to refer to different resolution pictures in inter-picture prediction except for inter layer cases. However, VVC does support Reference Picture Resampling (RPR), which allows pictures to refer to different resolution pictures so that video resolution can be adaptively switched in sequence. This work’s objective is to improve compression efficiency in the case of RPR. In the preliminary experiments, the compression efficiencies of reduced size encoded case where frames are encoded in half size and up-scaled to their original size and original size encoded case are compared. The result shows that most sequences get worse BD-rate when the sequences are encoded/decoded in reduced size. Here, it is noted that up-scaling is done by VVC Test Model, VTM-10.0, which uses a linear interpolation filter. Thus, the frames cannot recover high-frequency components sufficiently and cannot use temporal correlation. Therefore, this paper proposes a multi-frame super-resolution between frames of different resolutions where some frames are encoded in high quality frames in their original size and other frames in half size by using RPR. The proposed method uses the original size frames, which contain a lot of information, to perform super-resolution to the reduced frame. By doing so, it is observed it saves the bitrate by reducing the frame resolution and increases the quality of reduced frames by using multiframe super-resolution, which leads to improving the compression efficiency.

Recently, convolutional neural network-based generative models of image signals have been proposed mainly for the purpose of image generation, restoration and compression. For example, PixelCNN++ approximates probability distribution of the image intensity value as a parametric function pel-by-pel, and can be used for lossless image coding tasks. However, such an approach cannot work well for specific images which have statistical properties different from the image dataset used for the network training. In this paper, we improve the coding efficiency by introducing a few parameters for adjusting the probability model generated by PixelCNN++. These parameters are numerically optimized to minimize coding rates of the given image and then encoded as side-information to enable same adjustment at the decoder side.

We propose CAVRoom, a multiplatform virtual shared space framework for multiplayer sandbox interaction. Participants using various devices such as PCs, Macs, mobile tablets, and mobile phones can interact together in the virtual shared space. CAVRoom aims to provide accessible solution for enhanced remote interactions beyond services offered by existing videoconference solutions. We demonstrate the framework in a virtual classroom setting in which students play a Heart Attack card game.

Video super-resolution (VSR) aims to generate a high-resolution (HR) video by reconstructing each frame from the corresponding low-resolution (LR) frame and its neighboring ones. However, it is challenging to generate a clean HR frame due to displacement between the frames. To alleviate this problem, most existing approaches explicitly aligned neighboring frames to the reference. However, these methods tend to generate noticeable artifacts in the aligned results. In this paper, we propose a detail-structure blending network (DSBN), which structurally aligns the adjacent frames by blending the frames in deep-feature-domain. The DSBN extracts deep features from the reference frame and each neighboring frame separately, and blends the extracted detail and structure information to obtain the well-aligned results. Afterward, a simple reconstruction network generates a final HR frame using the aligned frames. Experimental results on the benchmark datasets demonstrate that our method produces high-quality HR results even from videos with non-rigid motions and occlusions.

We investigate a tool to aid the viewing of PC displays by Unilateral Spatial Neglect (USN) observers by preventing overlook of the left side of the focused window. The basic concept of this tool is using observer's requests to magnify an appropriate part of the display. Based on user-interface trials, three techniques are employed: (1) whole window flashing when opening the magnifying window to realize effective attention, (2) temporal control sequence uses same special key (Ctrl, Alt, etc.) followed by normal key for easy operation by hemiplegia users, and (3) a color combination design to emphasize four sides of the focused window frame to ensure attention is paid to the left side. This tool is also effective for users with presbyopia and/or users who want to add more active window effects for presentations.

In this paper, we propose a novel CycleGAN without checkerboard artifacts for counter-forensics of fake-image detection. Recent rapid advances in image manipulation tools and deep image synthesis techniques, such as Generative Adversarial Networks (GANs) have easily generated fake images, so detecting manipulated images has become an urgent issue. Most state-of-the-art forgery detection methods assume that images include checkerboard artifacts which are generated by using DNNs. Accordingly, we propose a novel CycleGAN without any checkerboard artifacts for counter-forensics of fake-mage detection methods for the first time, as an example of GANs without checkerboard artifacts.

Depth (disparity) estimation from 4D Light Field (LF) images has been a research topic for the last couple of years. Most studies have focused on depth estimation from static 4D LF images while not considering temporal information, i.e., LF videos. This paper proposes an end-to-end neural network architecture for depth estimation from 4D LF videos. This study also constructs a medium-scale synthetic 4D LF video dataset that can be used for training deep learning-based methods. Experimental results using synthetic and real-world 4D LF videos show that temporal information contributes to the improvement of depth estimation accuracy in noisy regions. Our dataset and source codes are available at: https://mediaeng-lfv.github.io/LFV_Disparity_Estimation/.

In this study, we propose a hardware-oriented Gaussian mixture model – multiresolution co-occurrence histograms of oriented gradients (GMM–MRCoHOG) algorithm for efficient human detection by a field-programmable gate array (FPGA). GMM–MRCoHOG is a HOG-based human detection method in which the computation of angles is quantized to 36 directions and 2D Gaussian distribution computation causes a decrease in processing speed and an increase in hardware resource usage. We propose a hardware-oriented algorithm to solve these problems. First, we propose a rough angle computation method of comparison with a tangent table. Second, we propose a bit-shifting-based Gaussian distribution computation method. Experimental results show that the proposed hardware-oriented algorithm does not significantly reduce the detection accuracy of GMM–MRCoHOG. High-level synthesis results of the FPGA implementation show that fast, low-resource processing is possible.

Motion blur is easily observable in an image having moving objects, and the coded exposure photography is one of the well-known solutions to its removal. In this paper, we propose a novel coded exposure photography technique to capture images of moving objects using coded flashes. Noting that the camera acquires a multi-channel image at once by using a color filter array, we modulate each channel image differently by using multi-band coded flashes. By modulating the point spread function of the acquired channel image to have a jointly invertible point spread function, its deconvolution performance is significantly improved compared to the existing single image coded exposure photography. We prove the effectiveness of the proposed method through experiments using synthetic and real data.

This VR system simulates a table tennis game for adoption in training. The virtual environment allows a real player to play with a virtual AI opponent, with an update and display of the score they gained. Technical innovations are made in modelling of the multi-variant collision between the moving bat and ball with adjustable colliders, and determination of the ball flight path. We present the conceptualization of VR techniques, the detailed implementation of the system and comparative advantages.

With the digitization of work production, CG has come to be widely used as a background for movies, game maps, and illustrations. The background CG handled by these is generally produced one by one using CG software. However, when creating background CG with conventional CG software, it is necessary to determine the position of buildings and roads, change the size of the CG model to be placed, and create textures corresponding to each CG model. This requires specialized knowledge and skills, and takes a lot of time and effort. Therefore, we propose an automatic background CG generation system that uses two types of deep learning.

Drawing contour lines is one of the most important techniques for understanding the characteristics of 3D models in nonphotorealistic rendering. Conventional methods draw continuous contour lines with the same thickness. However, contour lines are depicted by set of short lines just like growing grasses in furred animal illustrations. To depict furred animal 3D models in non-photorealistic rendering, we propose a new method that expresses animals’ fur by drawing a set of short and skew lines crossing along the contours.

The visualization of the sound field which is an invisible-physical phenomena is useful for detection of noise sources, understanding the sound propagation, and architectural acoustics. Room Impulse Response (RIR) is widely used in various acoustic applications because RIR characterizes the sound propagation between a source and a measurement point in a room. Thus, measuring the RIRs at multiple points allows us to visualize the sound field in more detail. For example, we can observe the animation of sound wave fronts by using the RIRs at multiple points. However, many microphones and repetitive measurements are required to measure the RIRs with high resolution. In this paper, we propose a method for visualization of the sound field with high spatial resolution based on the estimation of the RIRs around the microphone using a small number of microphones. The RIRs are modeled using sparse equivalent sources and the image source methods. We conducted the evaluation experiment in an anechoic chamber. By animating the RIRs estimated by three microphones, the sound wave fronts including direct sound and primary-reflection can be clearly observed.

This paper describes a support system for tracheal suction skill acquisition based on augmented reality technology. A few tracheal suction skill training tools have already been developed and used, but its operating cost is expensive, such as maintenance, storage, and cleaning/drying/sterilization after use as well as its purchase cost. Besides, the tracheal suction skill is a procedure inside the trachea, it is difficult to observe and evaluate the skill because the state of the procedure is to be invisible directly. Our proposed system solves these existing tools problems using augmented reality technology and compact 3D printing models and sensors. The system validity has been verified by evaluation experiments.

This paper proposes a customer interest estimation method using security camera to meet the demand of the retail industry. In the field of retail industry, it is considered that the understanding of customers’ interests in the real store can be used for various marketing activities such as the product development and the layout of the store. Then, it is important to pay attention to customers’ behavior in the real store. Their behavior is often recorded by the cameras installed in the store for security purposes. A method for estimating their interests from the videos of the security camera is presented in this paper. The novelty of our method is three-fold. Firstly, the experimental data of subjects in our group were taken by using the security camera already installed in the real store. Secondly, we used a pre-trained posture estimation model and treated the results as the features to be trained by a two-layer neural network model. Finally, a professional have annotated the subjects’ interests. The effectiveness of our method was confirmed by comparing with benchmark supervised machine learning models.

We propose a method of synthesizing a dense focal stack and an all-in-focus image from a sparse focal stack that consists of only a few differently-focused images. Our algorithm is implemented using a convolutional neural network (CNN), which is trained to produce a denser focal stack by interpolating the input focal stack, and an all-in-focus image by fusing the dense focal stack. Experimental results show that our method can achieve better image quality than the baseline methods..

Computer-generated hologram runs into difficulty with multi-perspective rendering because it handles threedimensional images. In this study, we propose a hidden surface removal method that enables continuous motion parallax by ray tracing from a point light source placed on a virtual object to the hologram plane. This method allows us to investigate in detail whether light from a point source reaches each pixel of the hologram plane, which in turn enables the removal of hidden surfaces and shading corresponding to continuous viewpoint movement. Although ray tracing is computationally intensive and takes a long time, NVIDIA’s Turing architecture GPUs, which have computation cores specialized for ray tracing, enable fast calculation.

In this paper, we propose a method for reconstructing rough patterns from a texture image while retaining fine texture information using deep learning. The proposed method is based on a deep neural network called pix2pix, which can learn the conversion process between input and output images. In the previous method, which is an improvement on pix2pix, two images were input: an original texture image to be edited and an editing pattern image to be reflected in it. However, these two images alone do not accurately reproduce the color of the original texture image. Therefore, we improve the previous method and improve the reproducibility of the pattern by inputting a pattern image that links the input original texture image to the editing pattern image. The effectiveness of the improved method was verified through several experiments. Future work is to improve the quality of the generated images to improve blur and reproduce accurate input texture information.

This paper describes a method of designing a 2D post filter for reducing coding artifacts caused by lossy image compression. Though Mean Squared Error (MSE) has been typically used in such filter design, it is not necessarily a good quality measure in terms of consistency with subjective perception. In this paper, we employ a more reliable quality measure called Structural SIMilarity (SSIM), and derive filter coefficients that can maximize the SSIM score for each image.

This paper proposes a black box extraction attack model on pre-trained image classifiers to rebuild a functionally equivalent model with high similarity. Common model extraction attacks use a large number of training samples to feed the target classifier which is time-consuming with redundancy. The attack results have a high dependency on the selected training samples and the target model. The extracted model may only get part of crucial features because of inappropriate sample selection. To eliminate these uncertainties, we proposed the VAE-kdtree attack model which eliminates the high dependency between selected training samples and the target model. It can not only save redundant computation, but also extract critical boundaries more accurately in image classification. This VAE-kdtree model has shown to achieve around 90% similarity on MNIST and around 80% similarity on MNIST-Fashion with a target Convolutional Network Model and a target Support Vector Machine Model. The performance of this VAE-kdtree model could be further improved by adopting higher dimension space of the kdtree.

In recent years, research on virtual fitting has been conducted in the fashion field. Many of them have been put to practical use in prepared clothes, and companies are using the information on the shape, size, and fabric of their clothes to provide users with virtual fitting. In the case without known data, there are many methods of estimating the shape and size of the clothes in images. Using these methods, users can try on virtually the clothes they want to wear while fitting the users’ body shape and pose. On the other hand, a method for estimating the fabric of clothes remains to be developed. Because the materials of clothes are related to the softness of clothes in virtual fitting, it is difficult to reproduce the realistic movements and wrinkles of clothes using the conventional virtual fitting system. This study proposes a method for estimating the material of fabric from clothes images, aiming at realistic virtual fitting. A dataset focusing on each fabric’s texture and luster is constructed and estimated using a Convolutional Neural Network (CNN).

This paper presents a method for action detection based on Temporal Cycle Consistency(TCC) Learning. The proposed method realizes the action detection of flexible length segments based on a frame-level action prediction technique. We enable calculation of similarities for spatio-temporal features based on TCC to detect target actions from input videos. Finally, our method determines temporal segments by smoothing the frame-level action detection result. Experimental results show the validity of the proposed method.

It is widely known that deep learning using a deep neural network (DNN) provides high performance in the fields of image recognition and image generation. On the other hand, DNN is composed of a huge number of filters, and the amount of information to express the filter coefficients becomes very large. In this paper, we propose a new method to reduce the amount of information for expressing DNN by adaptively using linear quantization and Lloyd-Max quantization for the weighting coefficients that compose DNN. As a result of the simulation experiments, it was found that the amount of information representing the DNN designed by CIFAR-10 and CIFAR100 can be reduced by up to 1/80 and 1/9, respectively, while maintaining the recognition accuracy.

Action recognition is one of the popular research areas in computer vision because it can be applied to solve many problems especially in security surveillance, behavior analysis, healthcare and so on. Some of the well-known Convolutional Neural Network (CNN) in action classification using 3D convolution are C3D, I3D and R(2 + 1)D. These 3D CNNs assume that the spatial and temporal dimensions of motion are uniform where the 3D filters are uniformly shaped. However, the path in motion can be in any directions and a uniform shape filter might not be able to capture nonuniform spatial motion and this limits the performance of the classification. To address the above problem, we incorporate a 3D deformable filter in a C3D network for action classification. The deformable convolution adds offsets to the regular grid sampling locations in the standard convolution resulting in non-uniform sampling location. We will also investigate the performance of the network when apply the 3D deformable convolution in different layers and the effect of different dilation size of the 3D deformable filter. UCF101 dataset is used in the experiments. From our experiments, it is found that applying the deformable convolution in lower layer yield better result compare to other layers. Our experiment shows that if we put the deformable convolution in Conv1a, the accuracy achieved is 48.50%.

We use deep learning technique for predicting temporary target position of each controlling agent in crowd simulation. Our deep learning model is based on recurrent neural network, which can predict a heatmap that represents the surrounding situation of an agent from a series of previous heatmaps of the agent. A heatmap contains positions and speeds of nearby agents and the temporary target position which represents the heading direction of the agent. Each agent is controlled based on the temporary target position that is chosen from an estimated heatmap using a force-based model.

In the last decades, depth estimation from multi-view is treated as an ill-posed problem. This problem becomes severe with limited data such as sparse-view cases. However, with the presence of Convolutional Neural Network (CNN), recent learning-based depth estimation methods prove effectiveness on occluded and texture-less area where prior works still suffer on handling such issues. They utilize features from CNN layer for constructing cost volume and regress input volume with regression network. To overcome those concerns, we introduce a unique approach by combining hand-crafted and learning-based strategies. Specifically, we utilize the Normalized Cross-Correlation (NCC) cost volume, which is more robust to noise than simple L1 and L2 costs, to improve the photo-consistency between local patches. The entire construction pipeline is implemented by pyOpenCL to speed up the processing time. Finally, we employ the network that estimates depth by regressing handcrafted cost-based plane sweeping volume.

Digital watermarking is the main method by which prevent illegal copying of proprietary images. In addition, to reduce the amount of damage caused by illegal distribution of proprietary images, a fast search technique for illegally distributed images on the Internet is necessary. We have proposed a two-stage search method for digital watermarked images. In the present paper, we propose a new high-efficiency pre-search algorithm for proprietary images that are suspected of being copied and distributed on the Internet. The proposed algorithm is capable of quickly identifying illegal copies from large image groups using mainly the correlation coefficient of compressed representations between the original image and a target image by convolutional autoencoder. The proposed search method has tolerance for attacks such as scaling, image compression, color reduction, and intensity conversion. The performance of the proposed search algorithm is evaluated through computer simulations.

A two-layer image coding method compatible with JPEG XS is proposed. JPEG XS is a new international standard for still image coding that has the characteristics of very low latency and very low complexity. However, the image quality is saturated at a certain quality level in general, although JPEG XS compression may be able to achieve visual lossless coding. The proposed method has a two-layer structure similar to JPEG XT, which consists of JPEG XS coding as the base layer and the extension layer. In an experiment, the proposed coding is demonstrated to outperform JPEG XT in terms of the coding efficiency, while maintaining compatibility with JPEG XS.

In video coding, the authors have proposed several rate control techniques based on a maximization of the spatiotemporal estimated MOS (EMOS). This paper extends the techniques to 2K240P videos, and demonstrates effectiveness of these techniques in 2K240P videos. The experimental results indicate that a frame rate of 240 fps is not always necessary, and that the overall spatio-temporal video quality can be improved by adaptively controlling the frame rate, compared with a fixed-frame-rate approach.

The CST (Chroma Separate Tree) coding tool in VVC (Versatile Video Coding) allows different partition structures in the luma and chroma channels, and it gives significant coding efficiency for the chroma channel. Under CST, the DM mode can be less efficient when the current chroma block and its co-located luma block have different block shapes and sizes, thus less correlated. To solve this potential problem, we investigate a slightly modified DM mode, so called MPDM (Most Probable Direct Mode), to increase coding efficiency of the chroma intra coding. The proposed method is shown to achieve BDBR gain of -0.03% in the Y channel, -0.25% and -0.24% in the Cb and Cr channels, respectively.

Recently a filtering method based on side windows has been reported. Although this method has strong ability to preserve edge structure, its ability to smooth textures is weaker than using a full window. In this paper, we propose a scale adaptive side window based bilateral filter (SASWBF). An expanding ratio parameter is designed to control the side window varying to the full window. To adapt the size of spatial kernel at each pixel, the scale of filtering window is estimated using a structure measure. A bilateral filter using Fourier basis is adopted to accelerate the filter processing. The computational complexity of the proposed smoothing filter does not depend on the window size and thus is in almost constant time. Our experimental results demonstrates that the proposed filter performs well.

Illumination estimation is a fundamental prerequisite for many computer vision applications. In this paper, we combine some previous methods for more effective estimation. SVR was used for the ensemble of previous methods. Instead of using the standard RGB image dataset, we have used hyperspectral images as the dataset, because we can freely set varieties of illuminant colors with them and can get accurate ground truth values. To render the hyperspectral image, we prepare spectral distribution with 21 different color temperatures and generate illuminant spectrums using Planck blackbody radiation equation with color temperature ranging from 2,000 [K] to 12,000 [K] at 500 [K] intervals. The number of hyperspectral images used for the training is 16. Each hyperspectral image contains 33 reflection data per each pixel. Illumination estimation methods combined in this paper are 6 methods in total, five traditional illuminant estimation methods, and one deep learning-based approach. We have compared the conventional illumination estimation methods with the proposed methods, and have concluded that the proposed method can achieve higher prediction accuracy.

Because of the proliferation of surveillance camera and the wide range of its utilization, person re-identification (re-ID) technology has been drawing attention. Although video-based re-ID to extract spatial and temporal features has become the mainstream these days, the number of input frames for most of video-based re-ID architectures is usually fixed. Thus, frame looping or frame duplication is applied if the number of input frames is less than the specified value. However, by applying CNN to the entire looped frames, the temporal features cannot be well trained. Therefore, in this paper, we propose a method to apply CNN not to the entire looped frames but several times to the neighboring frames by shifting frames (shifting-subclip), then the average of each CNN output is processed for further temporal feature extraction. The evaluation using the MARS dataset shows that mAP (Mean Average Precision score) and CMC (Cumulative Match Curve) with the proposal is higher than that without it. It suggests that it is effective to apply the proposed shifting-subclip in the MARS dataset.

In this work, we propose a heuristic to solve two-dimensional cutting and packing problems using a raster representation of shapes and container. We use discrete mathematical morphology operators and representations to prevent overlaps and to reduce the gaps between shapes. During the shapes placement, our method executes a continuous snapping of polygons to bring recently positioned shapes closer to others and to the container boundaries, thereby making our method a hybrid method that takes advantage of different techniques in the continuous and discrete domains. The results show that our approach is competitive to other works in the literature, since it can find optimal solutions for 7 instances among the 15 instances included in our dataset, the same number as reported by other works in the literature.

Texture synthesis has been researched for several decades, and a good texture can improve the quality of 3D objects. However, it is very difficult to obtain good textures through realistic scenes or the process of time-varying weathering. This paper introduces a method using a label map to control texture synthesis. The labels, which are extracted from source images, represent the different degrees of weathering in an image, meaning this method can synthesize a new label map that controls the region of weathering, while preserving the boundary feature. Furthermore, we can synthesize a more or less weathered texture from the source texture by image analogy.

It is one of the important issues to represent wettability that decides a liquid shape on a solid surface. There are some previous methods related to wettability using FEM (Finite Element Method), FVM (Finite Volume Method), and particle method. In these methods, particle method has a merit of robustness for topological change such as a droplet that comes out of a tube and reaches on a surface after being broken by gravity. In particle method, a liquid shape is decided by specifying the contact angle between the liquid and the solid, and by modifying the angle using CSF (Continuum Surface Tension) model or potential model. However, these methods have to specify the contact angle in advance, and the angle is not decided with the physical properties of the liquid and the solid. Therefore, this paper proposes a method to represent wettability not by specifying the contact angle, but by calculating the interfacial tension with the surface free energy between the liquid and the solid. As the result of the simulations with some liquid and solid, we have confirmed that wettability can be represented without specifying the contact angles.

We propose a novel method for the illumination robust template matching in unconstrained environments, named illumination robust template matching (IRTM). IRTM is based on the patch diversity of the Nearest-Neighbor. A core component is to calculate the similarity between patch pairs. Without illumination change, the Euclidean distance of appearance and geometric features jointly measure the similarity well. However, the illumination change will significantly affect the confidence of measure. In this paper, based on the relative pixel values that do not change with illumination, we define an illumination-robust distance function. With experiments, IRTM is shown to be a well-performing similarity measure against illumination changes, non-rigid geometric deformations, as well as background clutter, and occlusions.

High-resolution ocean remote sensing images are of vital importance in the research field of ocean remote sensing. However, the available ocean remote sensing images are composed of averaged data, whose resolution is lower than the instant remote sensing images. In this paper, we propose a very deep super-resolution learning model for remote-sensing image super-resolution. In our research, we target satellite-derived sea surface temperature (SST) images, a typical kind of ocean remote sensing image, as a specific case study of super-resolution on remote sensing images. In this paper, we propose a novel model architecture based on the very deep super-resolution (VDSR) model, to further enhance its performance. Furthermore, we evaluate the peak signal-to-noise ratio (PSNR) and perceptual loss of the model trained on the natural images and SST frames. We designed and applied our model to the China Ocean SST database, the Ocean SST database, and the Ocean-Front databases, all containing remote sensing images captured by advanced very high resolution radiometers (AVHRR). Experimental results show that our model performs better than the state-of-the-art models on SST frames.

This paper presents a new interior coordination image retrieval method using object-detection-based and color features. Interior coordination requires consideration of objects’ positional information and the overall atmosphere of the room simultaneously. However, similar image retrieval methods considering the coordination characteristics have not been proposed. In the proposed method, we extract different types of features from interior coordination images and realize the similar interior coordination image retrieval based on our newly derived features.

Classification of defect patterns that occur on semiconductor wafers is very important in the manufacturing process. Although CNN (Convolutional Neural Networks) is used in much of the research, these are not robust for rotation. We, therefore, propose a CNN model that is robust to rotation by performing data augmentation with rotation. The proposed method achieves higher classification accuracy than the conventional study of CNN using data augmentation.

This article proposes a training support system of holding chopsticks both for beginners and people with the wrong holding. Although eating with chopsticks is a part of East and Southeast Asian cultures, many people does not hold the chopsticks correctly. Since the posture of the holding is complex form in three dimensional space, two dimensional media such as textbooks and videos is insufficient for learning and training. Therefore, this article proposes a system, which acquires a trainee’s holding posture from a camera and a contactless motion sensor, and evaluates and represents the correctness of the posture in comparison of the correct one. Since the augmented reality (AR) technology is introduced into the representation, the system makes the trainee understand the difference easily.

In recent years, virtual reality (VR) and augmented reality (AR) have been developed and applied to various simulations for business and commercial use. In these simulations, computer graphics (CG) becomes very important to express virtual objects, and there are many studies on the expression of cloth. Some optical properties of an object are necessary to represent cloth with CG. These optical properties depend on the material of the thread, the number of threads, and the thickness. Therefore it is difficult to represent clothes corresponding to these changes. This study proposes a method to formulate the reflection and transmittance that depend on the component of the cloths. To formulate the reflection, we use the Kubelka-Munk theory and the component of the cloth that can be easily obtained using a smartphone, etc.

In this study, we propose a projection mapping method using contours viewed from multiple viewpoints to achieve highprecision tracking of objects that can be grasped and moved freely by hand. Recently, dynamic projection mapping, which changes the appearance of a moving object, has been actively studied. In particular, tracking a grasped object is difficult because the object is shielded. We have proposed a method for achieving high-precision object tracking using only ordinary infrared cameras. Nonetheless, there were still challenges that made it difficult to keep track. In this study, we propose a more robust and accurate object tracking by adding contours obtained from different viewpoints by increasing the number of cameras, which is more robust to shielding and accurate.

Normal map is an important and efficient way to represent complex 3D models. A designer may benefit from the auto-generation of high quality and accurate normal maps from freehand sketches in 3d content creation. This paper proposes a deep generative model for generating normal maps from users’ sketch with geometric sampling. Our generative model is based on conditional generative adversarial network with the curvature-sensitive points sampling of conditional masks. This sampling process can help eliminate the ambiguity of generation results as network input. It is verified that the proposed framework can generate more accurate normal maps.

In this paper, we consider the problem of affine subspace clustering, which requires to estimate the corresponding subspaces and assign the corresponding labels to data points on or near a union of low-dimensional affine subspaces. To address this problem, we propose a framework based on Nearest Subspace Neighbor (NSN). NSN is originally designed to estimate the geometric structure of clusters that can not be adequately performed by conventional approaches based on a general distance metric such as K-means, and has been applied in solving the linear subspace clustering problem. However, in real-world scenarios, the vast majority of data exist in the affine subspace rather than linear subspaces. To make better use of NSN, we construct an affinity matrix by incrementally picking the points considering affine subspaces in a greedy fashion. Statistical experiments demonstrate that our method outperforms both the original NSN and an affine subspace clustering method.

Removing shadows in a single image has been a challenging problem because shadows can appear in various forms due to complex physical situations, influenced by many factors such as light sources and the material’s transparency. In order to remove shadows precisely, most previous works utilized shadow mask information, which indicates the shadow region in a given image using binary representation. However, shadow mask utilization inevitably induces multiple problems, including shadow removal performance dependency and additional shadow detection process requirements. To solve these problems, the proposed algorithm is based on an image-to-image translation algorithm, which does not require additional shadow mask information. In this deep neural network , the convergence of fast learning is induced by utilizing various normalization layers. However, in a case that is very sensitive to various spatial features of an input image, such as shadow removal, the normalization process causes a problem of losing a large amount of information existing in the input image data. So, we utilize spatially adaptive denormalization(SPADE) to prevent loss of spatial features of input image data. Therefore, not only does it fundamentally solve the problem that various feature information constituting the input image is lost in the normalization process, but also enables precise shadow region removal by combining the feature map of multi-resolutions with the feature map of the decoder. In evaluation, the proposed algorithm shows that it exceeds the existing approach by about 20~30% in both PSNR and RMSE based on the ISTD large data set.

With the advent of the ultra-high-definition television (UHDTV) era, display resolutions and brightness of televisions have been augmented, and 4K and high dynamic range (HDR) videos are being publicly distributed. However, research on the psychological effects induced by these high-quality videos is still in its initial stages, and the video production methods that utilize these effects are still under investigation. Past studies have shown that expanding the dynamic range and improving the image resolution can enhance the impression of a video. In this study, we verified the effect of enhancing these impressions under the condition that the dynamic range and image resolution coexist in order to address the psychological effects of viewing UHDTV images. Four types of test images (2K/SDR, 2K/HDR, 4K/SDR, and 4K/HDR) were created for each of the six types of original images. Subsequently, subjective evaluation experiments were performed with a professional 4K monitor to measure the extent to which viewers sensed various impressions from the abovementioned images. Statistical analyses were conducted for the collected data, and it was found that high resolution and HDR were individually or collectively effective for enhancing a certain impression depending on the content. Furthermore, it was confirmed that the dynamic range and resolution had a synergistic effect. These findings can contribute toward more impressive video production and elucidation of the mechanisms influencing visual psychology.

The correct posture and procedure are essential for effective and injury-free strength training. Although it is necessary to know the differences between the postures and procedures of the trainer and the trainee, it is difficult for the trainee to know them during training without the presence of the trainer. Therefore in this article, we propose a mixed reality (MR) system which uses motion capture to recognize a trainee's posture on a HMD. Through the system, the trainee can train while looking his/her own postures on a HMD. Also the PC with the motion sensor can be placed some distance away since the HMD is connected with the PC wirelessly.

There are a variety of sizes and thicknesses of holds used in climbing, and the ease of holding them varies greatly. However, the difficulty of holding has not been considered in previous studies on route exploration. In this study, we improved the search algorithm A* used in previous studies and incorporated the difficulty of the hold into the fitness. We also used the improved A* as the evaluation function to estimate the difficulty of holds using a genetic algorithm (GA). There was also no discussion on how many divisions of the hold should be divided by difficulty, so we assumed four divisions: 2, 4, 8, and 16 divisions. After adjusting the parameters during interviews with expert climbers, we compared the algorithm of the four divisions with that of previous studies using a questionnaire online. The results showed that the route of the algorithm, which considers the difficulty of the hold, was rated higher by expert climbers and that the 8-division algorithm was the best among the proposed methods.

Fisheries are one of the few disciplines in biology where data collection continues to rely on the removal and destruction of the objects being studied. This practice has come under increasing scrutiny in recent years as research projects have been terminated due to denied permits. In some instances, research budgets have had to absorb the cost of purchasing quota for the fish captured, with difficulty in publishing results due to animal welfare concerns. In this paper, we proposed a non-extractive sampling system to localize fishes from underwater images obtained at aquaculture farms, which suffered from several issues; being, 1) low luminance issues, which could be significantly in detecting fishes, 2) severe water‟s turbidity due to fish mass caged in an area, and 3) protection / enclosure designed for camera to ensure fishes shy away from it. These images acquired in highly turbid waters are difficult to be recovered; due to 1) the fish feeding process (fish feed) which add noises to the already turbid water, and 2) the existing healthy biodiversity at the aquaculture farms. In this work, we investigate the performance of Faster R-CNN to localize fishes in the highly turbid dataset, under different network architectures at the base network. Different base network such as MobileNet, MobileNetV2, DenseNet, and ResNet, are employed. Experimental results show that MobileNetV2, with learning rate 0.01, with 500 iteration and 15 epoch, and with 87.52 % classification accuracy, is the most feasible to be deployed in the resource constrained environment, with about 6.7M parameters requiring 27.2 MB storage. These findings will be useful when the equipment is embedded with the Faster R-CNN when placed underwater for monitoring purposes.

This paper aims to detect compositional features from room images and present appropriate compositions for attractive photographic assistance. In this paper, we define the boundaries between walls and floor, the boundaries between walls and ceiling as compositional feature lines and vanishing points as compositional feature points. The proposed method was able to properly detect the composition of 27 images out of 36 room images. Especially for the two vanishing point images, the detection was successful in more than 80% of the images. Vanishing points were detected in all the room images with one and two vanishing points. We proposed a method that can easily take attractive indoor images by presenting an appropriate composition by detecting the compositional features. With this composition and features, a photography support system would be promoted to take effective room images easily. As a feature works, this method will be optimized to real time and interactive support for shooting seen.

To a achieve highly immersive auditory experience, physically accurate sound field reproduction is required. Many local sound field synthesis (SFS) methods have been proposed to accurately reproduce the sound field around a listener. However, the narrow listening area in local SFS prevents the listener from moving freely. In this study, we developed a dynamic 2.5-dimensional local SFS system, which moves the listening area to follow the movements of the listener using a body-tracking camera. For the evaluation the proposed system, the azimuth localization accuracy was investigated by measuring interaural level differences in an anechoic chamber. The results show that when the distance between the listener and the virtual point source is 1.0 m, the proposed system has a small azimuthal localization error compared with conventional SFS.

This paper proposes a shot detection method using the poses of a player in a badminton video sequence. In the proposed method, the hit timing is detected by focusing on the arm movements of the player and analysing the swing movement using skeletal information. The simple shot information is estimated by connecting player positions in the hit timing frame. By performing an experiment to verify hit timing detection, we confirmed that the detection is highly accurate and shot information detection is achieved.

Audiovisual feedback is one of the most essential elements in video games. Although most feedback does not affect game mechanics, it is believed to have an impact on a player’s experience. One of the concepts in audiovisual feedback is “juiciness,” or providing excessive positive feedback. A few empirical studies have analyzed how juiciness affects the performance and behavior of players. Yet there are few studies on how the kind of feedback and its timing affect player behavior and performance in video games. This paper focuses on the audiovisual feedbacks given to players when they fail in a game. If players experience many failures during gameplay, their behavior may change depending on the audiovisual feedback they receive at the time of failure. We hypothesized that if the feedback to failures was “feelgood,” a player would be motivated to replay the game and try to improve performance. In the first phase of the experiment, five different feedback patterns were prepared and measured by quantifying impressions using the semantic differential method. An A/B test was then conducted using a simple web action game. Players were presented with different audiovisual feedback patterns for each of five groups when they failed in the game. The analysis in multiple comparisons revealed no significant differences in spontaneous replay behavior. The test did, however, result in differences in players’ average scores, suggesting that audiovisual feedback on failure could affect player performance.

In this paper, we propose an estimation method of a rally scene from a table tennis video for broadcasting. The video is constructed of entry, rally, replay, and break scenes, the scenes are detected by comparison of adjacent frames. The rally scene shows a ping-pong table, and the table is extracted in the detected scene. In addition, an appearance of the rally scene is higher than the others, and the top number group of the scenes is defined as the rally scene. As a result of an evaluation experiment, the estimation accuracy of the proposed method is 96.8 % from 711 rally scenes of ten videos.

Human observers are able to allocate visual attention to multiple locations in a dynamic environment simultaneously. However, this ability depends on an individuals’ capacity to monitor the tracked objects' number, size, speed, and distance between objects. Moving objects in the real-world require that we independently allocate attentional resources to them. Recent studies have demonstrated that observers can focus attention on one location and ignore other locations; tracking performance improves when moving objects are rendered on a range of depth planes, and observers cannot track a moving object that abruptly changes depth in a 3D environment. We investigated whether an object’s abrupt or gradual changes improved attentive tracking under stereoscopic viewing in two experiments. These experiments indicated a similar pattern regardless of the depth-changing characteristics of the objects. Results demonstrated that abrupt- or gradual-changes of moving objects under stereoscopic viewing do not increase a tracking task's difficulty.

In this study, we propose a projection mapping using a rotating volumetric display. In this proposal, a light source synchronized with the object of projection including its position and shape is dynamically generated by a simple structure of a rotating volume display. The light is projected onto a moving object using a recursive through optical systems. In this paper, we verify the applicability to projection mapping by implementing a prototype.

There is an urgent need for three-dimensional analysis of cardiomyocytes using a computer to clarify the mechanism of heart disease. However, because microscopic images include cells other than cardiomyocytes, it is necessary to classify the cells before analysis. Cardiomyocytes are characterized by a relatively low volume fraction of cell nuclei in the cytoplasm compared with other cells. In this study, these features were utilized to extract cell nuclei and cytoplasm from fluorescence microscopy images of neonatal mouse hearts and to classify cardiomyocytes and other cells based on volume ratio. The accuracy of the classification was approximately 90% when the correct answer data were created using the images of fluorescent cardiomyocytes, and the experimental results were compared. This method is considered, based on the experimental results, to be an effective approach for cardiomyocyte classification.

This paper proposes a method for estimating 3D information, such as shape, orientation, size, and position of objects in a monocular image, and reproduce scenes in 3D point clouds using Convolutional Neural Network (CNN). This study proposes a network that combines depth estimation, object detection, and point cloud estimation to estimate 3D information of objects. The proposed network requires networks for object detection and segmentation, and a point cloud estimation for object shape estimation. The point cloud estimation network is robust to the reproduction of the object's surface and can deal with unknown objects through a semantic understanding of the object’s shape. In addition to these networks, we combine a depth estimation network for estimating the depth of the entire scene and the distance between the camera and object. In this paper, we consider the point cloud estimation network. We estimate the point clouds for real objects in the images of the dataset and evaluate the output point clouds.

We propose“ KUI Based Modeling ”which is a new method of real-world oriented 3D modeling based on KUI. Our proposed method allows objects to be moved by the KUI for painting-like 3D modeling with the trajectory of the objects. In this paper, at the first we describe and implement a system for shadow touch detection and a movement mechanism for object. Next, we experiment the operation of the system based on the proposed method, and show how the model is actually created and printed on a 3D printer. Finally, we discuss the issues that emerged from the behavioral experiments.

This paper attempts to improve denoising efficiency of BM3D technique for videos, i.e., VBM3D. VBM3D constructs 3D cubes from target video frames by a block matching algorithm that minimizes the residual matching error. However, such a cube formation results in sacrificing the pixel correlation in the temporal direction. This paper thus modifies this step to preserve the sub-pixel alignment, which makes the Fourier coefficients of each cube located on a vicinity of a certain plane in the 3-D Fourier domain. Then, SURE-shrinkage technique is separately applied to the inside and outside of the vicinity of the plane to denoise each cube. The experimental results given in this paper demonstrate the validity of our approach.

In the scenario of a VR hunting game, behaviour and autonomy of in-game animals are crucial in minimizing users’ uncomfortableness and maximizing their experience while gaming. In this study, forward and backward reaching inverse kinematics (IK) is extended and used as the IK solver to handle the locomotion of the autonomous animals. The IK model presented is seamlessly combined with the steering behaviour algorithms so that the animals can perform all sorts of autonomous actions efficiently, supported with realistic and smooth animations. The algorithms presented in this research can also be used for motion behaviour and control of other quadrupedal and multipedal objects.

Among characteristic club activities in Japanese fisheries high schools, demonstration of dismantling tunas and other fishes dismantling is attractive to consumers as well as to students. In spite of the extracurricular education, the school has difficulties to provide enough chances the students to practice of the dismantling due to their cost. To improve the situation, this article proposes a virtual reality support system of self-study for tuna dismantling. The system provides visual sense through Head-Mounted-Display (HMD) with interactive manipulation of some kinds of knives by both hands. Through the experience of the virtual world, students can study correct procedure with safety and accuracy of the dismantling. From the user test for fisheries high school students, the system shows its effectiveness on the learning outcome of the dismantling.

This paper describes a smart method to model bobsleigh track surface. On real bobsleigh tracks, the cross-sectional shape is generally rectangular in the strait section, and in contrast, circular and banked in the curve section. In the first step, three-dimensional shapes of track surface are generated automatically based on these two kinds of cross-sectional shapes along the track centerline. In the second step, the most suitable parameters of track surface modeling, which consist of triangular patches is decided by evaluating the smoothness of sliding in dynamic simulation with Unity.

This study offers a comparative analysis of user experience and levels of immersion in regards to screen based solutions for virtual museums and virtual reality based (VR) solutions with Head mounted displays (HMD). The aim of this study is to better inform design and research and development of virtual museums using real-time 3DCG for achieving an experience of presence and immersion for viewer that comparable to viewing experiences in an actual museum. We have developed a virtual museum that can be experienced both on a large screen display and also via VR on an HMD. Users can experience the viewing space where paintings, picture scrolls, and 3D-objects are placed in the virtual museum, and the virtual space by freely walking through the virtual environment (VE). The development environment used for the study is the Unity’s game engine, created in-game objects, and introduced output content for outputting as VR in Oculus. Then, Compare the results of the comparison experiment of the virtual museum conducted in Sweden and the comparison experiment of the virtual museum conducted in Japan in the previous research, and evaluate what is close to actual viewing.

We propose a new verification approach for subjective evaluation of actions and reactions in virtual reality. We have been working on subject behavior analysis in VR. A head tracking HMD allows users to move around and make actions up to a room size, and their experiences can be measured by their head position, motion estimation, and gaze tracking when they are available. But in most cases, subjective evaluation for total VR experience are conducted after their experiences are over. We have succeeded in integrating a new EEG device that can measure EEG and mind indexes in real time with gaze-trackable and head-trackable HMD. By referring the gaze tracking results and EEG analysis and/or mind indexes when a situation is presented in VR, we can check the subject is surely watching the objects with their attention in the situation and their attention level in his/her mind. It may support the reliability of the score of subjective evaluation. This is very important when we need to conduct a number of subjective evaluations with a fewer number of subjects. We have developed our preliminary system for two applications – sport action analysis and traffic safety evaluation – and confirmed that it is a promising approach.

In recent years, with the development of virtual reality (VR) technology, VR content has increased and is becoming more popular in homes. On the other hand, VR sickness becomes a serious problem. We have developed a sensory interface that mimics the "swimming motion" and obtained good results in reducing VR sickness. In this paper, we analyze the relationship between the motion sickness susceptibility to motion sickness and the sickness caused by "swimming motion" by using the MSSQ, which measures general motion sickness susceptibility, and verify the relationship between the motion sickness susceptibility and VR sickness.

In this article, a hybrid type of virtual keyboard with multimodal feedback using a physical tabletop (or any hard plane around the user) is proposed. A user, wearing a VR Head-Mounted-Display (HMD), sees a keyboard in the given virtual world. Tracking his/her finger’s motion on the tabletop, the proposed system determines the key pressed by him/her and gives multimodal feedback to him/her. A pair of changing colors and playing sounds are given to the user as visual and audio feedback, in conjunction with physical tactile feedback from the tabletop. The system is expected to improve performance of keyboard input in the virtual environment.

Regular evacuation drills from fire have the advantage that many people can participate at the same time. However, during training, you just walk down the corridors and passageways as usual, and it lacks a realistic feel, especially with flames and smoke. Since the number of facilities with realistic flames and smoke is limited, it is difficult to conduct drills with such realities in schools and workplaces. In this paper, we propose a system that uses mixed reality (MR) technology to simulate fire and smoke during an evacuation drill at a school or workplace. The proposed system uses a transparent head-mounted-display (HMD) to move through the real world, so that the system provides trainees with a highly realistic image of flames and smoke superimposed on the evacuation route. In addition, the appropriateness of the evacuation action is assessed by recording the route and time of the action, and the results are presented to the trainees at the end of the exercise to improve the effectiveness of the exercise.

When we try on eyeglasses at a store, the eyeglasses does not have optical lenses. However, a facial impression should differ between when wearing eyeglasses with optical lenses and when wearing eyeglasses without one. The different is caused by a distortion of the facial contour and the eyes from optical lens power. Our proposed system can help users choose optimal eyeglasses frame and lens by rendering user’s 3D face model and optical lens eyeglasses with refraction simulation in real-time.

Virtual Reality technology has been becoming popular. As fascinating virtual worlds have become more accessible, we may feel compelled to photograph the virtual world. However, real world photography and virtual world photography are not the same experiences. For example, traditional virtual world photography methods do not allow for failures to occur, which is a factor in enjoying the photographic experiences. In this paper, we propose a virtual space photography system that can simulate shooting with a real camera as an interface and allow for shooting failures. We specifically generate images that consider the motion blur of the exposure by the shutter. We demonstrate the possibilities of generating failed photos that function as a kind of memory by simulating the effect of taking a picture with a real world camera.

A gas burner is often used in science experiments in Japanese lower secondary school education, and its correct operating procedures are included in the study content. However, students are given only inadequate learning opportunities for the operating procedures with the actual gas burner, due to time and instrumentation constraints. Therefore, this article proposes an operation learning system of a gas burner using virtual reality technology with high realistic sense to supplement the learning with the real one. The proposed system provides interactivity through a dial controller and a pair of hand controllers as an input device, and a stereoscopic HMD as an output device. The dial controller imitates an adjusting screw, so that a learner can easily get a realistic sense of turning screws for air and gas. Also judging whether the user’s operation is correct or not, the system displays necessary information on the virtual scene of the gas burner for learning to promote the learner’s understanding.

In virtual and augmented reality, there has been a lot of research on interaction with virtual objects. In particular, eyetracking techniques are used to achieve hands-free interaction. In this paper, we focus on quick eye movements and intuitive pointing to move a virtual object with the gaze only. This study is basic research about the method for interacting with virtual objects in an augmented reality environment by the gaze. In this paper, we explore interaction by a typical controller and the gaze and investigate each characteristic of the interaction. The results show that the gaze may be useful as same as the controller in pointing to the distant position. To achieve more accurate hands-free interaction using the gaze, it is necessary to understand how to perceive the distance between the virtual object and the real object. Also, we need to examine the gaze interaction that uses other than the closing motion of the eye.

In this paper, manhole detection problem in pedestrian path from an image for visually impaired people or blind people is discussed. In many developing countries, it occurs many times in sidewalks that the manhole exists on the tactile paving surface and sometimes it is under construction or its cover keeps open. It means something unsafe for visually impaired or blind people or kids that they may fell in such manholes. So in this work, all manholes are divided into two types, which is respectively labeled as “normal” and “unsafe”, and we propose a method for both detection and classification with convolutional neural network (CNN). Some experiments with actually existing manhole images in Mongolia showed that our method classifies two types of manhole effectively.

Recently, object recognition using CNN is widespread. Still, medical images do not have a sufficient number of images because they require the doctor’s findings in the training dataset. On such a small-scale dataset, there is a problem that CNN cannot realize enough high recognition accuracy. As a solution to this problem, there is a method called transfer learning that reuses the weights learned on a large dataset. In addition, there is research on a method of pruning parameters unimportant for the target task during transfer learning. In this study, after transfer learning is performed, the convolution filter is evaluated using pruning, and the low evaluation filter is replaced with the high evaluation filter. In order to confirm the usefulness of the proposed method in recognition accuracy, we compare it with the three methods, i.e., transfer learning only, pruning, and initializing the filter. As a result, we were able to obtain a high recognition accuracy compared to other methods. We confirmed that CNN might be affected by replacing the filter in object recognition of small-scale datasets.

With the increasing accuracy of Deep Neural Network (DNN) models, a high-performance DNN model becomes more and more expensive to train. Moreover, like common multimedia resources, DNN models will be distributed or shared over the Internet. For these reasons, a high-performance model can be considered as an Intellectual Property (IP). However, because of the nature of deep learning, conventional watermarks for DNN models are usually vulnerable to attack. To protect the authentication of these DNN models, we proposed a watermarking technique that performs satisfactorily in terms of security, robustness, and embedding capacity without impairing the accuracy of the host DNN model. However, it has not been compared with other techniques in terms of robustness against overwriting attack. To this end, we extend our previous work5 by conducting more comparison experiments to evaluate the performance of our method.

Detection of main persons in a snapshot is proposed in this paper. The proposed method uses feature maps from Mask RCNN [1] and depth information from the depth estimation model [2] to detect main persons in the photo. Our research aims to construct a deep neural network model to estimate the main person degree map from these two inputs. Based on the estimated importance map, we performed post-processing and confirmed that only important persons could be extracted.

Nowadays, a precise video quality assessment (VQA) model is essential to maintain the quality of service (QoS). However, most existing VQA metrics are designed for specific purposes and ignore the spatiotemporal features of nature video. This paper proposes a novel general-purpose no-reference (NR) VQA metric adopting Long Short-Term Memory (LSTM) modules with the masking layer and pre-padding strategy, namely VQA-LSTM, to solve the above issues. First, we divide the distorted video into frames and extract some significant but also universal spatial and temporal features that could effectively reflect the quality of frames. Second, the data preprocessing stage and pre-padding strategy are used to process data to ease the training for our VQA-LSTM. Finally, a three-layer LSTM model incorporated with masking layer is designed to learn the sequence of spatial features as spatiotemporal features and learn the sequence of temporal features as the gradient of temporal features to evaluate the quality of videos. Two widely used VQA database, MCL-V and LIVE, are tested to prove the robustness of our VQA-LSTM, and the experimental results show that our VQA-LSTM has a better correlation with human perception than some state-of-the-art approaches.

Affine motion compensation (AMC) techniques are attracting much attention as one of promising tools in video coding. As an introduction of AMC increases the number of MC modes, an efficient mode selection technique is necessary to maximize the potential of AMC, which actually requires quite high computation cost. This paper thus tries to estimate the optimal mode in an AMC scenario using a neural network (NN). The experimental result indicates that our NN gives an efficient model selection technique in term the rate-distortion-based criteria.

This paper proposes a Neural Architecture Search (NAS) method for multimodal sequential data using a gradient-based neural architecture search method named Differentiable Neural Architecture Search (DARTS). Because Deep Neural Networks (DNNs) for multimodal data require task-specific network architecture, there is a high need for NAS for them to reduce the labor of architecture design. Experimental results using an emotion recognition dataset containing sequential data showed that the proposed method succeeded in automatically designing a network architecture with competitive performance to manually designed networks.

Digital Hologram is a very high-value-added image content, whose intellectual property should be protected for its distribution. Therefore its watermarking technology has become much important. This paper is to propose a training scheme of a deep neural network to perform a digital watermarking for a digital hologram. We construct various training datasets according to the distribution of holograms and train them on the proposed network. The network consists of preprocessing networks for both host data and watermark, watermark embedding network, and watermark extraction network, all of which consist of simple convolutional neural network (CNN) modules. The dataset is constructed with the digital holograms from JPEG Pleno and they are classified by distribution of the hologram pixel values. The network is trained with each of the classified holograms. With each of the trained weight sets, the invisibility and watermark extraction rate are calculated and compared with the results from testing the test dataset that has not been used for training. From the result, we explain the effects of the data distribution of the training dataset on the invisibility and robustness of the watermarking to present the best training scheme for a digital watermarking of a digital hologram.

To solve the problem of the huge amount of data in computer-generated hologram (CGH), we propose a system that applies fractal compression to object light data on the distributor side and decompresses the data on the receiver side to obtain hologram data. By applying fractal compression to the object light data on the distributor side, we succeeded in obtaining high quality reconstructed images with a high compression rate.

Many researchers are eager to study the stone tool production process based on ancient stone tools that were excavated from ruins. The assembly and disassembly of joining materials are repeated many times for investigation that is a better way to understand the stone tool production process. In our previous work, we have implemented an interactive assembly guiding program using the contours of stone tool images taken by an RGB camera. However, this method was not suitable for thick stone tools. In this work, we examine stone tool identification methods by using an RGB-D camera and 3D point cloud processing.

A stone chamber is a stone burial facility built inside a mounded tomb (Kofun). A corridor-style stone chamber is a stone chamber built on the side of a Kofun. A corridor-style stone chamber is composed of a main burial chamber for storing a body and a tunnel-shaped passageway to the chamber of a mounded tomb. Many corridorstyle stone chambers have basically the same structure. However, many of stone chambers are different in shapes such as the number of beams in the main burial chamber, and the shapes have been changed after stone on the ceiling is collapsed and sediment is accumulated inside. Archaeologists have been attempting to store and record 3D-measured digital data of stone chambers to protect and manage them¹. In [1], a depth sensor in a smartphone is used for 3D measurement, and multiple stone chambers are measured. In addition, the shape similarity between the measured point clouds is visually compared by any 3D CG Viewing software. Meanwhile, the visual comparison is based on subjective and qualitative observation approaches; thus a method for objectively evaluating the similarity is required. A matching method using the scaling iterative closest point (sICP) algorithm is one of the methods for evaluating the similarity between the point clouds. It enables distance minimization of closest point pairs between the point clouds after adjusting the point clouds in a same scale. Then the similarity between the point clouds can be evaluated as this obtained minimum distance. When a corridor-style stone chamber is measured, however, not only the stone chamber in the ground but also the ground surface part will be obtained together as a point cloud. The difference of the ground surface shapes makes shape matching difficult. Owing to the above-mentioned issue, this paper proposes an examination of the stone chamber extraction. It is the base of similarity evaluation between 3D-measured corridor-style stone chambers.

Recently, many researches on autostereoscopic displays have been carried out. In this work, we propose a new display configuration using the Pepper's ghost effect and a dynamic image adjustment method for a large scale autostereoscopic display. We also propose a method for extending the number of viewpoints for simultaneous viewing by multiple people. In the experiment, after constructing the device, we acquired the light transport matrix using two cameras and dynamically adjusted the video images. We observed parallax in the adjusted images, and we confirmed that the images were stereoscopic. At present, only one person can view the stereoscopic images at the same time because only two parallax images are presented. However, when acquiring the light transport matrix, we can create the video from multiple viewpoints by increasing the number of cameras. In the future, we will experiment using three cameras and confirm that the three-dimensional image presentation is possible by three parallaxes.

In this paper, we propose a preprocessing for the ICP algorithm to avoid convergence failures. Using a depth camera with the IMU, the movement of it can be measured. We use it for the pre-adjustment of two point-clouds as a preprocessing of the ICP. Our method consists of the following steps: (1) Measurement of the camera orientation from IMU, (2) Adjustment of the measured point-clouds using the camera orientation, (3) Parallel projection of the points onto one plane and 2D alignment, and (4) Depth alignment and overlapping of the projected models as the initial positions for the ICP. We also show several experimental results to indicate the effectiveness of our proposed methods.

A Piano is a popular musical instrument. Development of playing piano requires frequent and suitable practice. However, the learners often have difficulty on space and time for practice due to less portable size and weight of the piano. In this article, a portable and virtual-reality (VR) based self-training system of the piano is proposed, which is installed on a standalone see-through Head-Mounted-Display (HMD). A learner wearing the HMD can see both of superimposed virtual piano keyboard and virtual his/her own fingers. According to his/her keying by the fingers, the system represents the situation acoustically. Evaluation of training outcome (number of incorrect keying) and subjective assessment shows advantages of the proposed system in comparison of a piano playing application on a tablet device.

Some objects that can be assembled are disassembled and then reassembled for purposes such as maintenance and research of each part. When reassembling such objects, we reassemble it using images and illustrations that have been assembled, records of assembly methods. But if the assembly information is not sufficient, it may takes time to assemble. One solution is to record image the disassembly procedure. However, in this case, it is very difficult to associate the image with the parts, and if there are many similar parts, some parts may be mistaken or the exact direction of the parts may not be known. In this paper, we propose a method to display reassembling procedure by recording disassembling procedure of products. Specifically, the state when the parts are separated from the product is recorded together with the augmented reality (AR) marker, and the separated parts are associated with the AR marker. The AR marker are made to record the reading order so that the AR marker can be read and sort in the disassembled order when reassembly. By sorting the AR markers, the order of the parts used during reassembly becomes clear. By reproducing the disassembled information in the reverse order, the finished product reassembly can be finished without trial and error.

Rapid progress has been made in both augmented and virtual reality technologies. However, it is still challenging to seamlessly connect the virtual world with the real world, such as locating virtual three-dimensional models in the real environment and directly interacting with them. In this study, we propose a wearable augmented reality system with a proposed head-mounted device to enable the projection of anamorphic images. The proposed system can track the user's head movements, and then project the designated scene in real space in real time. To achieve this goal, our system consists of three steps: room scaling, blur correction for the projected contents, and calibration using dynamic mesh generation. We evaluated the proposed system by the interaction with virtual contents and characters. The evaluation results showed that the interaction with the virtual character was highly evaluated. This system can be used for a wide range of applications in daily life and entertainment, such as relieving loneliness and serving as a guide in museums.

Many services on user's location information are developed by using GPS (Global Positioning System). However, the GPS’s positioning accuracy drops significantly indoors. We have proposed and developed an indoor positioning method without GPS. We have already proposed a method to estimate the position and direction indoor by using short-range wireless communication and electronic compass based on directivity antennas without indoor map data. In this paper, in order to evaluate our method, we measure directional characteristics of the directivity antennas of the transmitter and receiver.

The so-called “Kanbun,” the Japanese reading of the classical Chinese text, has been studied as a type of the classical Japanese in secondary education in Japan. In Kanbun the word order is changed based on symbols called “return mark,” which is converted into a Japanese reading. This change is often so complicated that it may cause phobias in learning for junior and senior high school students in Japan. Therefore in this paper, a visualization system of step-by-step change of word order is proposed, which recognizes the return marks on the classical Chinese text and visually superimposes the converted text beside the original one, in the use of the augmented reality (AR) technology. The proposed system is applicable to any Kanbun sentences so that it is expected to contribute to improved student understanding.

Virtual studios, that synthesize video-camera images with CG objects of studio sets and props, are commonly used in professional program production. Since the actors cannot directly see the CG objects in the studio, they must estimate the positions of the CG objects by watching synthesized video displayed on a monitor placed within their field of view. However, it is difficult for actors to look at a monitor while acting because their acting becomes unnatural if gaze goes toward the monitor. Vibration feedback can be used as a way of indicating the positions of CG objects without the actor having to look at the synthesized video and without affecting the shooting or sound pickup. This method attaches a small vibration device to the actor’s body and communicates the distance to the CG objects by varying the intensity and pattern of vibration. A system to verify this method was developed. Subjects evaluated whether the surface of a CG object can be recognized and whether the distance from the surface of the CG object can be estimated on a four-level scale for four cases in an experiment that simulates interactions with a studio set and props. The experimenter observed whether the subject's hand penetrated the CG object in the synthesized video. As a result, it was found that vibration feedback is very useful for interactions with studio sets except for close interactions like handling props.

A process of 2-dimensional (2D) character animation is to segment an illustration into parts such as a face outline or hair. This is a simple and time-consuming process, which involves cutting out the parts to be animated from an illustration. In this study, we focus on automation of the segmentation of the parts of a 2D character animation. Moreover, to improve the accuracy of the process, we combine U-net and pix2pix. Based on the accuracy evaluation, we confirmed that the accuracy of the F-measure of the proposed method is 81.1%, which outperformed U-net and pix2pix by 15.5% and 0.9%, respectively.

In Computer-Generated Hologram (CGH), random phases are often added to diffuse the object light. However, due to this effect, speckle noise, which is a deterioration in image quality, occurs. In this study, we generated a hologram that suppresses the influence of speckle by calculation so that the light wave converges to the viewpoint. The viewing zone is limited by this method, it can be applied to Head-Mounted Display (HMD) type CGH or Eye tracking type electronic holography. The effectiveness of the proposed method was confirmed by simulation and actual optical reproduction.

In order to develop a technical process to display holographic reconstructed images in colors, it seems to be very important to extend the holographic displayable region of colors, and so the study of displaying process of the reconstructed images in colors with the use of blue-violet laser light (wavelength: 405 nm)seems to be crucial. We have studied how to improve the characteristics of the displayable region of the reconstructed images using blue-violet laser light when time-sharing multiplex reconstructing technique is employed. In this report, we are going to show some improved displaying characteristics of the object with the curved shape, and to propose some useful basic suggestions for a colorful display of reconstructed holographic images when blue-violet laser light is applied.

When inspecting defects such as scratches due to image processing, if we can obtain an image before defect occurrence, the defect can be detected by simply comparing the image pair before and after defect occurrence. However, this idea is generally unrealistic because it is impossible to obtain an image before defect occurrence from an image after the defect occurred unless we go back in time. Therefore, we propose a method of training a generation-based model to detect scratches based on subtraction between an input image and output image during testing. We obtained a large amount of ideal image pairs before and after defect occurrence (i.e., image pairs in which only defect regions are different and the others are almost completely the same) using an image-capturing device. An image with a background texture of high reconstruction performance was generated with our method by using this dataset. Although we used image pairs of only an aluminum plate for training, the area under the curve of the receiver operating characteristic (ROC-AUC) measure was 0.9973 for a copper plate and 0.9904 for a stainless steel plate. This shows that our method is also effective for background textures different from those used for training. We also compared our generation-based method with a method of training a classificationbased model (ResNet18). We confirmed that the ROC-AUC measure with the proposed method was 0.9824 for an acrylic plate and 0.6156 for a carbon fiber sheet, which is better than those with the classification-based method (0.9650 and 0.5000, respectively). The proposed method has much higher reconstruction performance than previous training of defect-free images only, and is effective for scratch detection of background texture similar to those used for training.

In this paper, we improve the extraction method by GrabCut in order to perform highly accurate stone contour extraction from stone wall images. In previous studies, there was a problem that over-segmentation and under-segmentation occurred because the area considered to be the background characteristics could not be specified properly. As a countermeasure to this problem, in order to make the method of specifying the restrict range for the background characteristics more suitable for stone materials, we aimed to improved the extraction accuracy by using the convex hull of the area obtained by ordinal GrabCut. Good results were obtained by limiting the characteristics due to the contraction of the convex hull, but some stones had insufficient segmentation, the cause was investigated, and future measures were described.

It has become popular for non-professional people to enjoy creating a digital image with a 3DCG software or a high functional paint tool. It is assumed that making an animation is next step, however, it is difficult to make an animation with images of illustration. The water flow, in particular, is a fluid and doesn’t have a fixed contour, therefore animating the water flow is one of the challenging subjects. In this paper, a method for animating the water flow portion included in an illustration by synthesizing motion information extracted from the water flow video material with that portion is proposed. The proposed method had been implemented on a PC and its usefulness was confirmed through applying to some water flow illustration images.

A facial caricature is an art form that captures the facial features of a person and exaggerates those features. Caricatures are commonly used to convey humor and sarcasm, as gifts, souvenirs, and social networking avatars. There are attempts to generate facial caricature, but these attempts have just focused on transforming the style into artistic style and do not exaggerate facial features. We focus on the exaggeration of facial features and propose a facial feature exaggeration system for generating portraits. Feature detection by comparing facial landmarks detected by the facial landmark detector with the mean face, and feature exaggeration by transforming the facial landmarks further away from the mean face. We also experiment with a new exaggeration method that combines normalization to exaggerate facial features. The results show that features can be exaggerated without collapsing the shape.

The performance of conventional deep learning models for liquid pouch inspection was analyzed. Pictures of liquid pouches were captured using soft X-ray transmission to generate our training, evaluation, and test datasets that included defective and defect-free products. Nine conventional deep learning models were trained using the training dataset, and their accuracy was computed using the evaluation and test datasets; ResNet50 and MobileNet demonstrated better performance. Because the difference between our defective and defect-free images was small, the learning effect was difficult to propagate to the deep layers. Therefore, the skip connections were indispensable to realize a deep layer for our application. While changing the discrimination threshold, it was possible to detect all defective products by allowing 5.59 % defect-free products to be mistakenly made defective in our experiments.

Mobile robots used in public spaces require safe path planning, such as avoiding obstacles. In this paper, we propose a method of generating a drivable path in a graph from a drivable region excluding moving obstacles, small obstacles, regions such as grooves and slopes that hinder traveling by LiDAR. Then, the effectiveness of the proposed method is shown by experiments.

Light field (LF) image is captured by plenoptic cameras which suffers from trade-off between spatial and angular resolutions. Numerous methods have been proposed to enhance the spatial resolution of images captured by LF cameras. Among the state-of-the-art methods, there is an approach to super resolve LF images using the graph-based regularization. However, it has a problem of taking too much time for execution. In this paper, we propose a method to simplify the process in computing graph. The experimental results show that our proposed method can reduce up to 18% of time complexity compared to the original approach while maintaining the image quality of LF images.

The Geospatial Information Authority of Japan (GSI) publishes digital elevation model (DEM) data by dividing Earth’s surface into equally spaced square meshes, and an elevation value is assigned to the center of a mesh element (cell). Currently, the available open data are 5-m mesh data obtained from laser measurements (DEM5A) and photogrammetry (DEM5B) and 10-m mesh data estimated from contour lines (DEM10B). The 5-m mesh data have significant issues over water areas, such as the sea and inland water surfaces. Specifically, the water portion of the DEM5A data, which cover much of Japan, is assigned a marker value of −9999. To solve this problem, this paper proposes a method for assigning an appropriate elevation value to every cell of the water portion in 5-m mesh data with different algorithms according to its classification: sea, lake and marshes, or river. To confirm its usefulness, the proposed method was applied to actual DEM data.

Predicting the occurrence of a natural disaster with high accuracy and providing an evacuation warning at the appropriate level are important for reducing any casualties. In the case of a landslide, for example, the behavior of rainwater after falling on the ground needs to be estimated. This estimation requires information on not only the local ground slope but also topographic features such as ridges and valleys. To obtain the latter, a method was previously proposed for using digital elevation model (DEM) data to obtain the overground and underground openness of an area. However, this method is not suitable for high-precision DEM data because it does not allow for dense sampling. To address this problem, this paper proposes a method for calculating the openness that uses a computer graphics technique to reflect the features of high-precision DEM data. The proposed method was applied to both simple-shape artificial terrain data and actual DEM data, and its usefulness was confirmed.

As a video encoding standard, High Efficiency Video Coding (HEVC) achieves excellent performance while causing a dramatic increase in coding complexity. Especially, the coding tree unit (CTU) depth decision process is the most complicated section, which takes heavy computation complexity in the entire HEVC intra coding process. Therefore, a deep learning-based method is applied to directly predict the CTU depth level for each frame in this study. In addition, a large-scale dataset that contains the coding unit image files and the corresponding depths was generated by HM16.15 to train and test the deep learning model. Besides, a Convolutional Neural Network called LeNet is fine-tuned by modifying the original architecture, and then the model with a more complicated structure is evaluated and compared on an acquired dataset. The experiments show that the fine-tuned deep learning model has the ability to identify accurately the depth level of CTU, the recognition accuracy reaches over 98.6%.

K-means is a widely used clustering technique that seeks to minimize the squared distance among points in the same cluster. K-means is an appealing clustering method in terms of computational speed. Also, K-means is the simplest clustering; thus, researchers select K-means as a first choice. Therefore, accelerating K-means is essential. In this study, we transform data structure from the array of structures to the structure of arrays for accelerating K-means by SIMD vectorization. Experimental results show that our implementation is faster than OpenCV’s implementation.

Aspect based sentiment analysis (ABSA) is a methodology used in various fields for analyzing the reviews of customers to project an overall opinion on any different product or service. With the evolution of the internet, people are provided with a time saving cheap procedure like this, to express viewpoints to a mass audience. With the intension of gaining market value, numerous fields are bestowed the chance in collecting information from the internet, free of charge. The application of Machine Learning (ML) methods for the procedure of evaluation regarding aspects connected to TV series and movies, has not been initiated. This would be a novel progression for the film industry. Research under this study focuses on implementing ABSA on TV series or movies based on sub aspects of genre as well as sub aspects of cast and crew. As the initial step, web scraping can be used for data collection from social media. Next, data is processed to generate adequate results for anyone who needs a broader understanding on significance of a TV series or a movie related to the aspects mentioned earlier. To collect information belonging to each aspect is further analyzed. The accuracy of the results of the suggested system has been gained high as 79% and above. In conclusion, the accuracy of the solution is greatly successful compared to the previous tasks with high market value.

A method for minor dry-eye detection based on deep learning was developed. Images of eyes were captured using a smartphone digital camera, and tear volume tests were performed using phenol red thread, to generate datasets of eye images and tear volumes used in our experiments. Our simple neural network models and ResNet 50 were applied to 2-class classification and evaluated. According to the results, it was difficult for humans to distinguish minor dry-eyes from our datasets, whereas the accuracy of ResNet 50 exceeded 0.80 and 0.75 on our evaluation and test datasets, respectively. Because there are few differences between the eye images in the dataset, the gradient in the deep layer was lost, and learning did not proceed to the deeper layers. However, when skip connection was applied, dry-eye detection via ResNet 50 became possible and resulted in the maximum accuracy among all tested methods. After the training, it was possible to detect minor dry-eyes using only a smartphone.

With the massive improvement in deep learning, application areas are also expanding significantly particularly for multimedia information forensic, antiforensic, and counter anti-forensic in the field of forensic analysis. Generative adversarial network (GAN) is one of the most popular deep learning models which is widely being used for anti-forensic JPEG compressed images to generate ground truth like images for making fool the JPEG compression detector. In this paper, we analyze the generator and discriminator in the GAN model to generate more realistic images and detect between generated and original images. We investigate the proposed method using two different GAN models in which the generators and discriminators are trained separately in each model. Then we use the generated images produced by the generator in one model to detect whether it is generated or original images with discriminator in another model and vice-versa. The reconstructed images produced by both generators are more realistic in visual perception and have the better quality that can deceive the JPEG compression detector. The discriminators are capable of differentiating between the generated and real images only if the generated images are reconstructed by their own generators. If we try to classify the generated images obtained by the generator in one model using discriminator in another model, the discrimination results reduce at an alarming rate.

This paper presents cross-domain recommendation based on multilayer graph analysis using subgraph representation. The proposed method constructs two graphs in source and target domains utilizing user-item embedding and trains link relationships between the users’ embedding features on each above graph via graph convolutional networks considering subgraph representation. Thus, the proposed method can obtain features with high representation ability, and this is the main contribution of this paper. Then the proposed method can estimate the user’s embedding features in the target domain from those in the source domain and recommend items to users by using the estimated features. Experiments on real-world e-commerce datasets verify the effectiveness of the proposed method.

We propose an autonomous flight control method for secure navigation of Unmanned Aerial Vehicles (UAVs). We focus on the relation between flight command given to UAV and the response. According to the flight commands sampled in world coordinate system, we acquire the corresponding 3D moving vectors, which represent moving distances and orientations in 3D space, by employing motion capture system. As the input/output relation implemented in UAV is a black box, our approach acquires the sequential input/output relation between flight command and 3D moving vector by using recurrent neural network. Given sequential flight commands, the model predicts the 3D moving vectors that correspond to the command sequence. We demonstrate that our proposed method yields the desirable command sequence for autonomous control.

We are researching navigation for the visually impaired. We propose a new interface that utilizes sound and vibration to support turn-by-turn navigation for visually impaired people. In our proposed interface, the target path is divided into straight segments and points of change direction. The navigation instruction given by the sound and vibration is carefully designed to give minimum yet sufficient clues on the visually impaired walking. We have implemented a preliminary system based on our proposal and conducted a subject experiment for visually impaired people. The results imply that our proposed approach is useful for visually impaired people.

Drivers’ distraction is one of the leading causes for road accidents. To reduce driver distraction, the real-time gesture recognition system for ADAS is aimed to simplify and enhance the interaction between human and computer by implementing the vision-based technique that allows driver to interact with the vehicle infotainment system functions using natural mid-air hand gestures. Therefore, in this paper, we proposed to track and recognize human static hand gestures. The system process is separated into five steps which include the image acquisition, the background subtraction, the hand segmentation, the features extraction, and the gesture recognition. Firstly, the image frame will be captured and resized. Thereafter, the region of interest is determined to minimize the required processing to increase performance. Next, the foreground model is being extracted and being converted to HSV color space. Then, the skin filter is applied to extract skin region. In the next step, the image is being transformed into binary image by thresholding and smoothened by applying the morphological transformation. Contour detection and approximation were being implemented. Finally, the hand features will be extracted to build the gesture recognition model such as hand center, palm radius, fingertips, defect points, hull area, hand area and angle of finger. Experimental results show that the system is able to achieve 86.25% recognition accuracy in room environment and 80% recognition accuracy in car environment. In comparison, the average classification accuracy is 92.5% and 90%, for room and car environment, respectively.

In Japan, many passengers use railway lines for commuting to work or school. In the morning and evening hours, chronic delays often occur, and the time to reach the destination increases. This paper proposes a method to store and analyze open data related to railway operations for several months. This paper also visualizes the tendency of chronic delays to occur.

In this paper, a new night dehazing strategy is proposed by modifying the transmission and atmospheric light conditions. To suppress the local light sources, a Gaussian based atmospheric light model is developed along with erosion filters to improve the traditional atmospheric light factors. A similar erosion operator is also utilized for transmission to refine, and the model is combined with a multi-scale clipping limit of ICLAHE to obtain an optimal nighttime dehazing effect. The experimental results confirm that the proposed strategy performs better in comparison to state-of-the-art techniques and is computationally less intensive.

The purpose of this study is to determine whether microsaccades and pupil diameter can be used to estimate human covert attention when subjects are observing natural scenes. In this experiment, natural images were presented, and eye movements and pupil responses were measured. The analysis focused on microsaccades and gaze time. As a result, the histogram at the angle of microsaccades showed that microsaccades may be attracted to the object of interest. In addition, the rank of the objects of interest was not proportional to the gaze time. This indicates that the factor determining the rank of the objects of interest did not depend on the gaze time.

On detecting people in a video of a wide area, there are three problems. First, the apparent sizes of people are likely to be very different due to their positions in the scene. Second, the density of people is unevenly distributed in the scene. Detecting the entire image implies a high video processing cost. Third, there are regions where people do not permanently appear. These regions should not be processed. To solve these problems, we aim to detect people regardless of people's positions and sizes, focusing on the regions where people could appear. To achieve this aim, we propose a video dividing method based on people's positions and sizes in the video for high-resolution video detection and a method to eliminate the divided regions where people will not appear. We adopt a pyramid representation for video division and integration. We skip processing the regions where no people are detected for a certain amount of time. Furthermore, we use semantic segmentation to eliminate regions where people do not appear permanently. We choose 4K high-resolution videos looking down wide areas from a fixed camera for the experiment. We test the proposed method on PANDA dataset and the videos taken at Tsukuba.