Pseudo Video Sequence Coding of Integral Imaging based 3D Images

There is a significant
amount of inherent redundancy in an integral image. Applying a 2D-image coding
scheme to an integral image fail to extract and reduce this redundancy
efficiently. We have proposed a pre-processing and compression scheme that aims
to enhance the compression efficiency of integral images. The scheme first
transforms the still integral image into a pseudo video sequence consisting of
sub-images, which is then compressed using an H.264 video encoder. Thereby
leveraging on the efficient temporal and spatial coding tools of the
H.264-standard for coding integral images.

The improvement in compression efficiency of using this scheme is evaluated and
presented. Different parameterizations are also investigated, which affects the
coding efficiency at low range bitrates. An average PSNR increase of 5.7 dB or
more, compared to JPEG 2000, is observed on a set of reference images. Compared
to other coding schemes the introduced artifacts are distributed more
homogenously within the depicted 3D-volume.

**Proposed coding
scheme**

To lay the groundwork we first define an integral imaging based 3D-image (II-frame) with a resolution of M x N pixels as

** II**(

where *m = 0, 1, ..., M-1 *and *n = 0, 1, ...
N-1* are the
horizontal and vertical position of a II-frame pixel, which constitute a
RGB-triplet. In our simulations we assume a lens array with rectangular
lenslets, but the proposed scheme is applicable to hexagonal and lenticular
lenslets as well provided that the number of views are sufficiently high. This
gives K*·*L
elementary images (EI) with a resolution of U x V pixels that are defined as

* EI_{k,l}
*
(

where
*k**
= 0, 1, ..., K-1 *and *l = 0, 1, ... L-1*
is the position of each EI and
*u= 0, 1, ..., U-1 *and *v
= 0, 1, ... V-1*
is the pixel positions within every EI.

As a first step in our proposed coding scheme we extract the set of sub-images (SI) according to

* SI_{u,v}*(

where
consequently each of the U*·V *SI*s* have a resolution of K x L
pixels. Figure 1 shows examples of the above defined images II, EI and SI.

(a) II-frame | (b) EI | (c) SI | (d) 2D-image |

Figure 1. Synthesized II-frame divided into its component parts. (d) shows the scene captured by a 2D-camera merely for reference.

The SIs are then considered as pictures in a pseudo video sequence (PVS) according to

** PV S**(

where *t*
is the pseudo time index, or picture number, of the PVS. Selecting different
functions *u(t)* and *v(t)*, or selection orders, results in PVSs
with different coding properties. The PVS is then coded using a H.264/MPEG-4
AVC encoder. The three steps are illustrated in Figure 2.

Figure 2. Block diagram of the proposed coding scheme

**Results**

The proposed coding scheme outperforms JPEG2000 and other proposed PVS-schemes for resolution prioritizing II-techniques where the number of lenslets is large. For depth prioritizing II, with a lower number of high resolution EI, the benefits are less. A proper choice of selection order for the proposed scheme becomes more important at low bit rates. Compared to the references the proposed coding scheme distributes the coding artifacts homogenously over the 3D-scene's depth.

Proposed
scheme vs. 2D-image coding and other PVS-based coding schemes ("Car")

Different
selection orders' effect on coding efficiency ("Twins")

Lens array properties effect on coding efficiency
("Cuboid")

Induced coding artifacts ("Apples", front view @ 0.15 bpp)

Original (24
bpp)
Proposed SI-based PVS coding scheme

EI-based PVC coding
scheme
JPEG2000

**Publications**

"Multiview image coding scheme transformations: artifact characteristics and effects on perceived 3D quality"

Roger Olsson and Mårten Sjöström

*Proceedings of Stereoscopic Displays and Applications, January 2010*
Web page accompanying paper

"Empirical Rate-Distortion Analysis of JPEG2000 3D
and H.264/AVC Coded Integral Imaging Based 3D-Images"

Roger Olsson

*Proceedings of 3DTV Conference, IEEE/EURASIP/MPEG-IF,
May 2008*
Web page accompanying paper

"A novel quality metric for evaluating
depth distribution of artifacts in coded still 3D images", *
*Roger Olsson and
Mårten Sjöström

"A depth dependent
quality metric for evaluation of coded integral imaging based 3D-images",
*
*
Roger Olsson and Mårten
Sjöström

"Evaluation
of Combined Pre-Processing and H.264-Compression Schemes for 3D Integral Images",
*
*Roger Olsson, Mårten
Sjöström
and Youzhi Xu,

Proceedings of Electronic Imaging - VCIP,

"A
Combined Pre-processing and H.264-compression Scheme for 3D Integral Images"

Roger Olsson, Mårten Sjöström and Youzhi Xu,

*Proceedings of ICIP, *IEEE, October 2006