Rewiring Filterbank

Rewiring FilterbankThis work describes a series of new results outlining equivalences between certain “rewirings” of filterbank system block diagrams, and the corresponding actions of convolution, modulation, and downsampling operators. This gives rise to a general framework of reverse-order and convolution subband structures in filterbank transforms, which we show to be well suited to the analysis of filterbank coefficients arising from subsampled or multiplexed signals. These results thus provide a means to understand time-localized aliasing and modulation properties of such signals and their subband representations—notions that are notably absent from the global viewpoint afforded by Fourier analysis—as well as signal recovery from sampled sequences based on their filterbank characterizations.

Display Color Filter Array

Display Color Filter ArrayIn digital image display devices, data are typically presented via a spatial subsampling procedure implemented as a color filter array, a physical construction whereby each light emitting element controls the intensity level of only a single color. In this work, we examine the problem of color filter array design with respect to spatial resolution and human vision; in doing so we quantify the fundamental limitations of existing designs by explicitly considering the spectral wavelength representation induced by the choice of array pattern, and propose a framework for designing and analyzing alternative patterns that minimize aliasing. An empirical evaluation on color images confirms our theoretical results, and indicates the potential of these patterns to significantly increase spatial resolution while at the same time improving color image fidelity.

Optimal Color Filter Array

Optimal Color Filter ArrayIn digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array—a physical construction whereby only a single color value is measured at each pixel location. We consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem formally as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, and—after proving sub-optimality of a wide class of existing array patterns—provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple color image test sets support our theoretical results, and indicate the potential of these patterns to increase spatial resolution for fixed sensor size, and to contribute to improved reconstruction fidelity as well as significantly reduced hardware complexity.

Optimal Exposure Bracketing

Optimal Exposure BracketingA common technique used to acquire high dynamic range image data is that of exposure bracketing—short exposure times are required to capture bright regions of the image without saturation, whereas long exposure times are needed to capture darker image regions effectively. This work investigates how to take into account the statistics of the photon arrival process to derive optimal exposure control for maximizing signal recoverability in high dynamic range imaging.

Image Quality Assessment

Image Quality Assessment We developed a foundation for assessing visual quality with “corrupted reference” (CR-QA)—a new quality assessment (QA) paradigm for reasoning about human vision and image restoration problems jointly. In most image restoration problems, the notion of an ideal reference that exist only in theory and not in practice. CR-QA assesses the visual quality of a restored image signal relative to this ideal reference image (not provided) with the help of observed image (i.e. corrupted reference). This is in contrast to today’s QAs, which either require direct access to the ideal reference image (full-reference QA) or disregards the ideal reference entirely (no-reference QA).

Noise Parameter Estimation

Noise Parameter EstimationNoise is present in all images captured by image sensors. Due to photon emission and photoelectric effects that are the foundations of the ways in which quantum mechanics enable image sensors, in fact, random noise is a “necessary evil” of image sensors that will continue to require our attention. The goal of this work is to provide a comprehensive characterization of random noise in ways that enhance post-image-capture signal processing steps. We derive the Poisson approximation to model the measurement noise that is the result of photon arrival and photon recapture. A novel methodology to learn the parameters that describe the noise is developed. We conclude by presenting preliminary evidence that accurate noise modeling would improve image denoising, especially in the low photon count/high noise regimes.

3D Display Crosstalk

3D Display CrosstalkIn this work, we analyze the reproduction of light fields on multiview 3D displays. A three-way interaction between the input light field signal (which is often aliased), the joint spatioangular sampling grids of multiview 3D displays, and the interview light leakage in modern multiview 3D displays is characterized in the joint spatioangular frequency domain. Reconstruction of light fields by all physical 3D displays is prone to light leakage, which reduces sharpness of the images shown in the 3D displays. Stereoscopic image recovery is recast as a problem of joint spatioangular signal demodulation, where the combination of the 3D display point spread function and human visual system provides the narrow-band low-pass filter which removes spectral replicas in the reconstructed light field on the multiview display. The proposed light field reconstruction method performs light field antialiasing as well as angular sharpening to compensate for the nonideal response of the 3D display.

Single Shot HDR

Single Shot HDRA combination of photographic filter placed over the lens and the color filter array on image sensor induces differences in red, green, and blue channel sensitivities. Spectrally selective single-shot HDR (S4HDR) imaging treats this as an exposure bracketing. Optimally exposed regions of low dynamic range red/green/blue color components are merged in a principled manner to yield a single HDR color image. Though not expected to yield results superior to the traditional time multiplexing counterparts, the single-shot HDR solution we propose is a robust alternative that can be realized with conventional camera hardware.

Denoising + Demosaicking

Denoising + DemosaickingWe propose new approaches to demosaicing of spatially sampled image data observed through a color filter array, in which the correlation of color components are exploited in order to reconstruct a subsampled image. Two frameworks for applying existing image denoising algorithms to color filter array data are developed—one for wavelet domain processing, and the other for pixel domain processing. Recent advances in spatio-spectral sampling and panchromatic pixels have contributed to increased spatial resolution and enhanced noise performance. As such, it is necessary to consider the universality of demosaicking design principles—instead of CFA-specific optimization for signal recovery.

Object Motion from Blur

Object Motion from BlurThere are three types of blur in images: defocus, camera shake, and object motion. While defocus and camera shake have received considerable attention, object motion is by far the most difficult feature to detect due to its nonstationarity. Object motion blur, however, is potentially useful for scene analysis because it provides temporal cues from a single image. We developed a technique to infer object motion by detecting spatially varying motion blur from a single image. We are able to remove the blur without introducing ringing (an artifact that all conventional methods suffer from) or iterative steps (hence faster than most deconvolution methods).

Multispectral Sensor Design

Multispectral Sensor DesignColor filter arrays (CFA) have enjoyed immense popularity in single sensor imaging, including consumer, professional and scientific imaging. Owing in part to the technical breakthroughs that led to exceedingly high sensor resolution and high quality CFA interpolation algorithms, color image sensors now deliver high quality images. In this work, we investigate the plausibility of extending CFA sampling to enable multispectral imaging capabilities. In contrast to the prior work in this area that have been based on heuristics, we take a principled modeling approach based on a three dimensional Fourier transform (2D space, 1D spectrum) that reveal surprising degree of “spatial-spectral” structure. We propose a new spectral filter array (SFA) design aimed at maximizing the “recoverability” of the multispectral image signals. We provide concrete trade-offs between spatial and spectral resolution stemming from SFA sampling strategies.

Color Constancy

Color ConstancyThe color of a scene recorded by a trichromatic sensor varies with the spectral distribution of the illuminant. For recognition and many other applications, we seek a color representation that is unaffected by illumination changes. Achieving such color constancy is an ill-posed problem because both the spectral distribution of the illuminant and the scene reflectance are unknown. Most methods have approached this problem by leveraging the statistics of individual pixel measurements, independent from their spatial contexts. In this work, we show that the strong spatial correlations that exist between measurements at neighboring image points encode useful information about the illuminant and should not be ignored. We develop a method to encode these correlations in a statistical model and exploit them for color constancy. The method is computationally efficient, allows for the incorporation of prior information about natural illuminants, and performs well when evaluated on a large database of natural images.

Universal Demosaicking

Universal DemosaickingRecent advances in spatio-spectral sampling and panchromatic pixels have contributed to increased spatial resolution and enhanced noise performance. As such, it is necessary to consider the universality of demosaicking design principles—instead of CFA-specific optimization for signal recovery. In this article, we introduce a new universal demosaicking method that draws from the lessons learned in Bayer demosaicking designs, but can be applied to arbitrary array patterns. We recast the data-dependence of Bayer demosaicking as a parsimonious reconstruction of the underlying image signal that is inherently sparse in some representation. Using properties of filterbanks, we generalize this principle to yield a nonlinear recovery method that is consistent with the state-of-the-art Bayer demosaicking methods.

Binning Artifact Removal

Binning Artifact RemovalPixel binning refers to the concept of combining the electrical charges of neighboring pixels together to form a superpixel. The main benefit of this technique is that the combined charges would overcome the read noise at the sacrifice of spatial resolution. Binning in color image sensors results in superpixel Bayer pattern data, and subsequent demosaicking yields the final, lower resolution, less noisy image. It is common knowledge among the practitioners and camera manufacturers, however, that binning introduces severe artifacts. Our in-depth analysis proves that these artifacts differ from the ones stemming from loss of resolution or demosaicking, and therefore it cannot be eliminated simply by increasing the sensor resolution. We propose a post-capture binning data processing solution that succeeds in suppressing noise and preserving image details. We verify experimentally that the proposed method outperforms the existing alternatives by a substantial margin.


Intelligent Signal Systems Laboratory, Dr. Keigo Hirakawa, Director

Kettering Laboratories 241 C 
300 College Park 
Dayton, Ohio 45469 - 0232