Object based Hybrid Video Compression

Rhoda Gbadeyan and Chris Joslin

Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada

Keywords: Video Compression, Object-based Coding, Background Subtraction, Object Detection.

Abstract: Standard video compression techniques have provided pixel-based solutions that have achieved high

compression performance. However, with new application areas such as streaming, ultra-high definition

TV(UHDTV) etc., expectations of end user applications are at an all-time high. Never the less, the issue of

stringent memory and bandwidth optimization remains. Therefore, there is a need to further optimize the

performance of standard video codecs to provide more flexibility to content providers on how to encode video.

In this paper, we propose replacing pixels with objects as the unit of compression while still harnessing the

advantages of standard video codecs thereby reducing the bits required to represent a video scene while still

achieving suitable visual quality in compressed videos. Test results indicate that the proposed algorithm

provides a viable hybrid video coding solution for applications where pixel level precision is not required.

1 INTRODUCTION

With more sophisticated end user devices constantly

being introduced onto the market at rapid pace, there

is an increasing demand for higher quality video e.g.

UHDTV, high quality streaming services etc. The

challenge for meeting these requirements necessitate

creating video content at higher spatial resolutions

and frame rates resulting in the need for a higher

bitrate to transmit, distribute and/or store this content.

Overtime, research has focused on the development

of both standard and non-standard video compression

techniques to handle this content. Of these two areas

of research, standard video compression solutions

have been by far the more successful solution.

For the purposes of video processing, to the

human observer, it is safe to assume that there are

only minimal identifiable changes as we navigate

within a video scene from one frame to the next.

However, when observed at the pixel level by a

standard video compression algorithm, there can be

significant differences when one frame is compared

to the next sequence of frames. These pixel variations

could be due to object motion, lighting variations,

shadowing or occlusion etc. and tracking all of these

changes at the pixel level results in a lot of data, which

even after compression, take up a lot of valuable

memory space as well as transmission bandwidth.

By handling scene variations at the pixel or block

level and storing this information, even in its encoded

form, the limited memory available is quickly

exhausted making compression mandatory. Video

compression standards have demonstrated

increasingly high performance starting from the

H.261 and MPEG-1 standards to the HEVC standard.

However, their existing uniform and lengthy schema

is fast becoming unable to meet all the changing

needs and dynamic nature of the video coding

community (Mattavelli, Jorn, & Mickaël, 2019) and

there is a need to find other ways to improve

compression by adding new functionalities or

including content adaptation methods which may

boost their coding efficiency. In this paper, we

propose a hybrid video compression solution that

replaces pixels with objects as the unit of

compression while still harnessing the advantages of

standard video codecs.

This paper is presented as follows: in section 2, a

literature review of existing work in this area of

research is provided. Section 3 outlines the details of

the proposed algorithm. Results are presented in

section 4 and the paper concludes in section 5.

2 LITERATURE REVIEW

Video coding solutions can be categorized into 3

groups on the basis of their approach: standard video

coding, non-standard video coding and a hybrid of

both standard and non-standard video coding. Both

Gbadeyan, R. and Joslin, C.

Object based Hybrid Video Compression.

DOI: 10.5220/0010207607850792

In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP, pages

785-792

ISBN: 978-989-758-488-6

785

standard and non-standard video compression

solutions have been actively investigated for years

resulting in the implementation of several techniques.

Standard video compression solutions have been the

more successful of the two solutions by far.

While standard codecs have enjoyed large scale

success, there have also been a few drawbacks, some

of which are that:

• While block-based solutions offer great

flexibility, they also suffer the draw back of

blocking artifacts in some implementations

• Existing block-based coding standards, such as

H.264 and H.265, do not consider the arbitrary

shape of moving objects and, as a result, their

prediction efficiency is compromised

While standard codecs remain the gold performance

criterion for video compression technologies, there is

still room for further improvement as well as a need

for alternative solutions as these codecs are not

always suitable for every application. There is also a

need to explore non-pixel/block-based alternatives

either as a standalone solution or in combination with

existing standard codecs as they may offer a more

efficient way to utilize the stringent memory

available. Several video compression solutions have

been proposed which either adapt existing standard

solutions or propose new ways to solve the video

compression problem, these are reviewed next.

Mosaics are created by stitching together images

from a given scene taken at different instances in

time, thereby creating a panoramic view of that scene.

Mosaics often contain spatially overlapping

information about the scene. Authors in (Hsu &

Anandan, 1996) & (Irani & Anandan, 1998) proposed

a hybrid mosaic-based video compression algorithm

that combines mosaics with standard video

compression. This solution replaces pixels with

mosaics as the basic unit for compression. By using

mosaics, the authors were able to address untapped

redundancies still present in video scenes.

Pattern based coding (PBC) for video was

standardized by the introduction of predefined pattern

code books in the H.263 and H.264 standards

(Sullivan & Wiegand, 2005). The problem with

predefined codebooks is that using them does not

achieve optimal coding efficiency (Manoranjan,

Murshed, & Dooley, 2003). Authors in (Manoranjan,

Murshed, & Dooley, 2003) & (Murshed &

Manoranjan, 2004) introduced the use of dynamically

extracted patterns from video content thereby

achieving superior coding efficiencies. Their hybrid

video coding solution combined the use of

customized pattern code books with existing standard

video codecs.

Authors in (Galpin, Balter, Morin & Pateux,

2004) & (Balter, Gioia & Morin, 2006) proposed a

non-standard model-based approach to video

compression. Without utilizing any standard codecs,

the authors developed a scalable compression method

for scenes with unknown content. They harnessed the

redundancy in the 3D models by only intra coding the

key frames while all others were compressed in a

predicted mode. Their approach avoided having to

transmit texture coordinates by computing the texture

information at the decoder side.

Over the last few years, deep learning-based

video coding has been an actively developing area of

research (Dong, Jianping, Li & Wu, 2020). This

emerging area of research has yielded techniques that

can be considered to be hybrid deep learning-based

video coding methods as well as non-standard deep

learning-based coding methods. Non-standard deep

learning can be divided into either pixel probability

(PP) modeling-based techniques or auto-encoder (A-

E) based techniques. PP modeling uses deep learning

to solve prediction problems and is representative of

predictive coding. Some representative work includes

(Wu, Singhai, Krahenbuhl, 2018) & (Chen, He, Jin &

Wu, 2019). Conversely, A-E based techniques work

by training a deep learning network to encode by

converting high dimensional signals to low

dimensional ones and decode by recovering the high

dimensional signals on the basis of the low

dimensional signals generated by the encoder. A-Es

are representative of transform coding. Some

representative work includes (Chen, Liu, Shen, Yue,

Cao, & Ma, 2017) & (Lu, Ouyang, Xu, Zhang, Cai,

& Gao, 2019).

Hybrid deep learning-based video coding

methods work by utilizing trained deep networks as

tools within standard coding schemes or in

conjunction with standard coding tools such as

HEVC. As demonstrated in (Dong, Jianping, Li &

Wu, 2020), deep trained networks can replace almost

all the modules in standard video codecs. Authors in

(Li, Li, Xu, Xiong, & Gao, 2018) & (Hu, Yang, Li, &

Liu, 2019) have proposed replacing the traditional

intra-coding module with a deep trained network

version while authors in (Lin, Liu, Li, & Wu, 2018)

& (Zhao, Wang, Zhang, Wang, Ma, & Gao, 2019)

utilized unique inter-coding modules.

Today, while deep learning-based video coding

continues to show promising results in this field, it is

important to note that it is an area of research that is

still in its infancy with a lot of room to develop into a

mature area of research (Dong, Jianping, Li & Wu,

2020). Next, we review the proposed technique.

VISAPP 2021 - 16th International Conference on Computer Vision Theory and Applications

786

3 OVERVIEW OF PROPOSED

OBJECT SEGMENTATION AND

DETECTION METHOD

When there are changes in a video scene, such

changes are often associated with specific objects,

affecting either the whole object or only a subsection

of the pixels that compose the object. Therefore,

rather than pixels, objects form the foundation for

each phase of this proposed algorithm.

We propose a background subtraction (BGS)

object-based hybrid video compression scheme. The

algorithm is made up of three key phases:

1. BGS Phase

2. Object Detection and Tracking Phase

3. Compression and Scene Reconstruction Phase

In each phase, an object-based algorithm was

implemented. To achieve a high level of consistency,

regardless of the phase, a consistent color space was

used. The Hue Saturation Value (HSV) color space

was chosen as it is closely aligned with the human

visual system. An overview of the proposed hybrid

object-based algorithm is provided in Figure 1. Next,

each phase of the algorithm is reviewed.

3.1 BGS Module

An adaption of the Statistical and Knowledge Based

Object Detection (SAKBOT) algorithm which was

proposed by authors in (Cucchiara, Grana, Piccardi &

Prati, 2003) as well as the improved SAKBOT

approach proposed by authors in (Calderara, Melli,

Prati & Cucchiara, 2006) was implemented. The

SAKBOT BGS algorithm was chosen as the primary

BGS method because it combines the advantages of

statistical methods with those of adaptive methods to

generate a resilient BGS algorithm. While this BGS

approach met the basic requirements for this

implementation, there were a few issues that had to

be remediated to improve the performance of the

overall object-based compression algorithm. These

include its inability to (1) provide a measure of

variance, (2) take advantage of the spatial information

inherent in a video scene and (3) apply post

processing to the achieved results. When combined,

these factors resulted in a higher incident of false

positives (Prati, Mikic, Grana & Trivedi, 2001).

Our implementation of the SAKBOT algorithm

is referred to as the adapted SAKBOT algorithm. It

added an additional data point to the computation

which was the spatial location of the moving visual

objects (MVO), their shadows and any ‘ghosts’ (i.e.

the set of connected points identified as being in

motion by the BGS process but which do not belong

to any real moving objects) in the video scene. This

change contributed to the reduction of the instances

of false positives. This adapted SAKBOT has 3 key

components:

1. Background Initialization aka Bootstrapping

2. Foreground Detection

3. Selective Background Maintenance

The background was initialized by building the

background model 𝐵



from the video sample. The

initialization buffer size was set to 120 frames. This

was optimized by initially using a buffer size of ~ 50

frames and then subsequently updating the model as

required throughout the computation. The advantage

of this modification to the SAKBOT algorithm was a

reduction in the memory required upfront to create

the BGS model. The generated background model

was designated as the background subtractor object

and served as the input to the next phase.

The foreground was computed on the frame

level as the difference between the current frame

𝐼



and the background model 𝐵



thereby generating

the foreground mask 𝑀



(𝑖,𝑗) containing the grey-

level information of the foreground. The selective

model updates were achieved by applying a temporal

median to a circular buffer that stored pixel values

over time. This approach benefits from the ability to

capture statistical dependencies between color

channels. The input video was converted from its

native RGB color space to HSV to support the

implementation of the shadow detection module.

Next, a binarized motion mask was computed

and MVO objects at time t were extracted from this

final binarized motion mask. This generated the first

list of candidate objects and shadows to be tracked.

This list was subsequently vetted in the object

detection module enabling the generation of the final

list of objects which was later fed as input to the

object tracking module.

3.2 Object Detection Module

As no one feature can provide invariance to all scene

changes, a multi-feature-based approach was used to

achieve reliable object identifications. The local

features selected in this phase were object color and

object motion. These features are considered to be

mutually independent as one feature cannot be

predicted on the basis of another feature (Khan &

Shah, 2001). The HSV color space was used because

it is able to explicitly separate luminosity and

chromaticity. Within the HSV color space, the hue of

a pixel was assigned a larger weight than its saturation

and brightness and was used for clustering pixels into

Object based Hybrid Video Compression

787

regions for the purpose of segmentation and object

detection. This involved extracting this feature as

well as generating an HSV Histogram of Oriented

Gradient (HOG) based on the hue channel which was

then used to detect and segment objects in the

foreground.

The object detection problem was considered to

be a form of the classification problem in which

classifications were considered to be good if they

were homogeneous with respect to a specific feature

within the region of interest (ROI) and dissimilar

outside the focus area. Therefore, since the hue of a

particular color is considered to be at its largest value

in the center of an object where it is least sensitive to

colors from adjacent regions, the center hill of the

histogram was identified as the center of the cluster

of the color which was then determined to be the

center of the object to be segmented. As the HSV

color space has the added advantage of being able to

identify color shade and intensity value variations in

the area of the object’s edge, thereby sharpening the

object boundaries, it was possible to accurately

identify the object of interest from its identified center

to its boundary thereby increasing the accuracy of the

object segmentation.

To apply object motion as a feature, thereby

validating the information provided by the color

feature, there was a need to compute the motion

parameters. By assuming that the color segment of the

detected object was a superset of the motion segment,

this computation focused only on generating the

motion parameters for pixels identified as belonging

to a candidate object/ROI via the dense Farneback

optical flow-based approach thereby simplifying the

otherwise cumbersome computation process.

Assuming that the motion field within each

candidate object’s ROI was smooth and that the

optical flow constraint assumption that the intensity

of a pixel in an image is constant along the trajectory

of the pixel’s motion holds true, then, motion

estimation was computed by generating a parametric

motion model for each candidate object in the scene.

The motion segmentation approach used key

representations highlighted in (Chang, Tekalp &

Sezan, 1997) because by defining the segmentation

based on a parametric motion model, physically

meaningful ROIs could be achieved.

As such, the motion field was jointly represented

as both the sum of a parametric field ‘p’ as well as a

residual field ‘r’. From the current frame 𝑔



to the

search frame 𝑔



, a 2-D motion vector 𝑑

(

𝑚,𝑛

)

for

pixel location

(

𝑚,𝑛

)

was defined as

𝑑

(

𝑚,𝑛

)



𝑢(𝑚,𝑛),𝑣(𝑚,𝑛)



(1)

Where u and v are the vectors for each pixel location

associated with candidate objects in each frame.

Then, k, the set of independently moving objects was

set equal to the list of candidate objects generated in

the previous phase. Provided that a segmentation

label 𝑥

(

𝑚,𝑛

)

is used to assign motion vector 𝒅(𝑚,𝑛)

to a pixel belonging to one of the objects in the set k,

the motion of each object was approximated by a 6 to

8 parameter affine parametric mapping of 𝜙, the

result of which produced the parametric motion

vector component 𝒅



as well as the residual motion

vector 𝒅



at pixel (𝑚,𝑛) as expressed below

𝒅

(

𝑚,𝑛

)

= 𝒅



(

𝑚,𝑛

)

+ 𝒅



(𝑚,𝑛) (2)

Where 𝒅



(

𝑚,𝑛

)

is dependent on the segmentation

label 𝑥

(

𝑚,𝑛

)

, thereby computing both the parametric

motion vectors as well as the residual motion vectors.

The Bayesian rule was applied to the a-posteriori

Probability Density Function (PDF) of u, v, and x,

given 𝑔



and 𝑔



to obtain the Maximum A-

Posteriori (MAP) estimates. Therefore,

• Given the best estimates of the motion and

segmentation fields, the mapping parameters 𝜙

where obtained by computing a least squares

procedure

• Given the best estimate of the parametric field,

constrained by the assumption that the motion

field is smooth within each segment, the motion

field was updated through an estimation of the

minimum-norm residual field and

• Given the best estimate of the motion field via

Gibbsian priors, the segmentation field was

updated to yield the minimum-norm residual

field

A conditional pdf was then used to quantify how well

the estimates fit the given candidate objects. After

computation, the following was obtained: a

parametric model per object, residual motion vectors

for each 4 by 4 block of pixels belonging to the object,

a dense motion field, a motion segmentation field and

a set of mapping parameters. The resulting parametric

motion model as well as the motion segmentation

field were overlaid on to the foreground, if there was

a match between the identified motion areas as well

as the candidate object, this object was identified as a

real detected object else it was discarded as a false

positive. Every detected object was then segmented

based on the outlined edges identified. The

segmented object was added to the object list and a

candidate blob was added to the blob list. A blob was

generated on the basis of its connectivity and stored

in the blob list. Once this task was completed, the

color space was converted back from the HSV color

VISAPP 2021 - 16th International Conference on Computer Vision Theory and Applications

788

space to the RGB color space for the next phase of

processing.

The tracking module received as input the list of

detected objects 𝐴𝑙𝑖𝑠𝑡 in addition to the list of

candidate blobs 𝐵𝑙𝑖𝑠𝑡 which were considered to be

associated with the tracking objects. A list, 𝐶𝑙𝑖𝑠𝑡 was

created to track the number of consecutive frames that

each object in the detected object list had not

appeared in the scene. If an object was identified as a

tracking object at some point in the scene but had

disappeared from the scene and over the next 50

consecutive frames, the object had not reappeared in

the scene, then that object was assumed to have exited

the scene and such an object was then removed from

𝐴𝑙𝑖𝑠𝑡 and only stored in 𝐶𝑙𝑖𝑠𝑡. However, if the i-th

object in 𝐴𝑙𝑖𝑠𝑡 is recognized as a tracking object, a

new blob was added to 𝐵𝑙𝑖𝑠𝑡 and the loop was

repeated. We assigned a candidate blob in 𝐵𝑙𝑖𝑠𝑡 to

each object in 𝐴𝑙𝑖𝑠𝑡 by computing a distance matrix

from 𝐴𝑙𝑖𝑠𝑡 to 𝐵𝑙𝑖𝑠𝑡 on the basis of 3 measures -

position distance, blob distance and color variance

measure. The position distance, i.e. the distance from

the center of the current tracking object to the center

of the previous tracked object was computed. On the

basis of this known value, the distance between the

position of the current tracking object and its next

position was then estimated using the Unscented

Kalman Filter (UKF). A similar distance metric was

computed between the blobs as well to effectively

match blobs to each object’s position in the scene.

The color variance measure was computed as the

square of the distance between the histogram of Hue

of the HSV color space of the previous tracking object

and that of the current tracking object. Finally, after

computing the position, blob and color measures, the

overall distance 𝐷



was calculated as

𝐷

,

=𝐶𝑜𝑙𝑜𝑟 𝑀𝑒𝑎𝑠𝑢𝑟𝑒 ∗

𝑠𝑞𝑟𝑡

(

𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒

)

∗𝐵𝑙𝑜𝑏 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 (3)

The value of 𝐷



for each object and blob position x,

y was stored in the D array D = 

𝐷



 (4)

where, 𝐷



is the distance from 𝐴



to 𝐵



In cases where a tracked object 𝐴



was never

recognized as a blob 𝐵



, the distances were

individually set to infinity and 𝐷



was assigned to

infinity. To solve this problem of assigning a tracked

object to a blob, the Hungarian algorithm (Kuhn,

1955) was used. Based on the distance matrix D, this

algorithm was used to achieve the optimal assignment

thereby associating blobs in 𝐵𝑙𝑖𝑠𝑡 with tracking

objects in 𝐴𝑙𝑖𝑠𝑡. The achieved results were updated

based on 𝐶𝑙𝑖𝑠𝑡 which contained skipped objects.

There were candidate blobs in 𝐵𝑙𝑖𝑠𝑡 which had not

yet been matched to any tracking object as the blobs

were not recognized. The blobs in this category were

then matched to the objects in 𝐶𝑙𝑖𝑠𝑡 and based on

their distance and statistics information, the object list

was updated with the newly matched objects.

The tracking algorithm developed in this

research is able to handle single object tracking as

well as multi-object tracking. In the single object

tracking case, it was identified that due to possible

computational errors in earlier phases of the

algorithm, a single object may have incorrectly been

identified as two tracking objects. To address this, a

distance threshold value ‘d’ was defined and set equal

to 0.5 such that if the distance between any two

trackers was found to be less than d, the objects

previously identified to be two distinct tracking

objects were considered to be just one object, and the

objects were merged into a single tracking object. On

the other hand, the multi-tracking algorithm was

triggered if the computed distance between any two

objects was found to be greater than d.

To enable reconstruction, the spatial location of

each object relative to other objects in a frame was

computed for each instance of that object across all

frames. To generate stable tracks of objects across the

scene, each newly identified object was compared to

the list of existing validated objects, if a new object

was matched to an existing object, it was tagged with

the same tracking ID as the existing object, else it was

tagged with a new unique tracking ID. Finally, all

stable object tracks were written into object files.

3.3 Compression and Object Layer

Reconstruction Module

The main goal of this module of the proposed

algorithm was to iteratively recombine the objects in

a frame, sequentially compress each recombined

frame, feed the recombined frame as input into the

prediction loop of a video codec and ultimately,

reconstruct the original input video. As the proposed

algorithm is an object-based hybrid compression

algorithm, an H.265 codec was utilized for the

compression phase. This reconstruction module

received as input a color image of the background,

scene objects and their stable tracks as well as the

tracking text file containing the statistics of each

frame in addition to the objects in the frame.

Individual object bitstream files were created for each

object’s stable track across the scene on the basis of

this data and objects were properly placed in the

resulting bitstream relative to their location details.

Object based Hybrid Video Compression

789

Once the object files were created, the next step was

to encode and reconstruct the video sample by

recombining the object files after which the video

sample was decoded. The algorithm worked as

follows:

Step 1 – Initialize the video sample

reconstruction phase by reading the

colored background image.

Step 2 – Pass the background as input to

the standard codec, encode this file as

a key frame and store the encoded file

in the buffer. The number of bitstream

files in the buffer is set = 1,

set Current bitstream = Recombined

Bitstream; set bitstream# = 1

If bitstream# < Total # of

bitstream -1

Step 3 – Pass the previously obtained

Recombined Bitstream as input to the

inter-frame prediction loop.

Step 4 - Read the next object bitstream

from the folder, pass the object

bitstream as input to the standard codec

to be encoded and pass its associated

metadata to the buffer.

Step 5 – If the number of bitstreams in

the buffer is > 1, then

Using the associated metadata

information, combine the New Object

bitstream with the current bitstream to

form the new Recombined Bitstream

Set Current bitstream = Recombined

Bitstream; bitstream# ++,

repeat steps 3-4,

else

Step 6 – Write overall recombined

bitstream to file. This bitstream was

then passed on to the decoder and the

video sample was decoded

end

4 RESULTS

This algorithm uses objects as the minimum unit of

processing, as such, pixel level changes from one

frame to another were not fully captured. Therefore,

the Structural Similarity Index Measure (SSIM)

proposed by in (Wang, Bovik, Sheikh & Simoncelli,

2004) was used. This metric takes into account the

similarity of the edges between the original frame and

the reconstructed frame. According to the authors in

(Hore & Ziou, 2010), there is a correlation between

SSIM and the quality perception of the human visual

system (HVS) and is computed by modeling any

image distortions resulting from the comparison of

the reference image against the test image as a

combination of three factors – Contrast distortion,

loss of correlation and luminance distortions.

This algorithm has been designed to integrate with

any standard video compression algorithm.

Therefore, its performance when integrated with

HEVC was assessed against the standard HEVC

implementation using a few sample videos. The

results obtained are promising while identifying areas

for improvement. The test results shown in Figure 2

are from the pedestrian traffic video sample which is

an outdoor video scene of medium complexity,

showing pedestrians in motion, with partial and full

occlusion at various points in the scene and the people

in the scene moving in unusual patterns. The scene

has moderate lighting variations resulting in shadows

being associated with the objects in motion. As shown

in Table 1, this algorithm achieved comparable SSIM

results compared to HEVC while achieving bit

reduction over H.265. Testing results have

demonstrated that when the complexity of the scene

is high, i.e. in scenes where 80% or more of the pixels

are changing over the course of the scene, this

proposed algorithm did not achieve the level of bit

reduction compared to HEVC as it did with less

complex scenes. A contributing factor is that the BGS

algorithm which forms the foundation of this

technique is not as effective at handling the

complexity of this type of scenes.

5 CONCLUSIONS

The main contribution of this work of research is a

non-pixel-based approach to video coding that

harnesses the strength of existing pixel based standard

codecs while eliminating the requirement to process

a video sample on the basis of pixels, rather offering

an opportunity to process such samples on the basis

of scene objects. This algorithm is able to efficiently

encode all the background areas of each video frame

i.e. areas observed for the first time; areas revisited by

the capture camera’s Field of View (FOV) following

a long period of time when that portion of the scene

was unavailable as well as areas of the scene that have

been uncovered by a moving object. Due to the fact

that this algorithm does not track scene changes at the

pixel level, it is suitable for use in applications where

performance is not measured at the pixel level.

VISAPP 2021 - 16th International Conference on Computer Vision Theory and Applications

790

Future work is still required to optimize the

performance of this algorithm when processing

complex scenes. This algorithm offers an alternative

solution suitable for use in bit constrained

environments.

REFERENCES

Mattavelli, Marco, Jorn W. Janneck, and Mickaël Raulet,

2019. "MPEG reconfigurable video coding."

In Handbook of signal processing systems, pp. 213-

249. Springer, Cham, 2019.

Hsu, S., & Anandan, P., 1996. Hierarchical representations

for mosaic-based video compression. In Proc. Picture

Coding Symp (Vol. 395).

Irani, M., & Anandan, P., 1998. Video indexing based on

mosaic representations. Proceedings of the IEEE,

86(5), 905-921.

ITU-T Rec. H.263 (01 /2005) Video Coding For Low Bit

Rate Communications

Sullivan, Gary J., and Thomas Wiegand, 2005. "Video

compression-from concepts to the H. 264/AVC

standard." Proceedings of the IEEE 93, no. 1 (2005):

18-31.

Paul, Manoranjan, Manzur Murshed, and Laurence Dooley,

2003. "An arbitrary shaped pattern selection algorithm

for very low bit-rate video coding focusing on moving

regions." Proc. of 4th IEEE Pacific-Rim Int. Con. on

Multimedia (PCM-03), Singapore (2003).

Murshed, M., Manoranjan, P., 2004. Pattern Identification

VLC for Pattern-based Video Coding using Co-

occurrence Matrix. In: International Conference on

Computer Science, Software Engineering, Information

Technology, e-Business & Applications, Egypt (2004)

Galpin, Franck, Raphaele Balter, Luce Morin, and Stéphane

Pateux, 2004. "Efficient and scalable video

compression by automatic 3d model building using

computer vision." In Picture Coding Symposium,

PCS’2004, San Francisco, USA. 2004.

Balter, Raphale, Patrick Gioia, and Luce Morin, 2006.

"Scalable and efficient video coding using 3-d

modeling." IEEE Transactions on Multimedia 8, no. 6

(2006): 1147-1155.

Liu, Dong, Yue Li, Jianping Lin, Houqiang Li, and Feng

Wu, 2020. "Deep learning-based video coding: A

review and a case study." ACM Computing Surveys

(CSUR) 53, no. 1 (2020): 1-35.

Claude Elwood Shannon, 1948. A mathematical theory of

communication. Bell System Technical Journal 27, 3

(1948), 379–423.

Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu., 2019.

Learning for video compression. IEEE Transactions on

Circuits and Systems for Video Technology.

DOI:10.1109/TCSVT.2019.2892608

Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl.,

2018. Video compression through image interpolation.

In ECCV. 416–431.

Tong Chen, Haojie Liu, Qiu Shen, Tao Yue, Xun Cao, and

Zhan Ma., 2017. DeepCoder: A deep neural network-

based video compression. In VCIP. IEEE, 1–4.

Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei

Cai, and Zhiyong Gao., 2019. DVC: An end-to-end

deep video compression framework. In CVPR. 11006–

11015.

Jiahao Li, Bin Li, Jizheng Xu, Ruiqin Xiong, and Wen

Gao., 2018. Fully connected network-based intra

prediction for image coding. IEEE Transactions on

Image Processing

27, 7 (2018), 3236–3247.

Yueyu Hu, Wenhan Yang, Mading Li, and Jiaying Liu.,

2019. Progressive spatial recurrent neural network for

intra prediction. IEEE Transactions on Multimedia 21,

12 (2019), 3024 – 3037.

DOI:10.1109/TMM.2019.2920603

Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu., 2018.

Generative adversarial network-based frame

extrapolation for video coding. In VCIP. 1–4.

Lei Zhao, Shiqi Wang, Xinfeng Zhang, Shanshe Wang,

Siwei Ma, and Wen Gao., 2019. Enhanced motion-

compensated video coding with deep virtual reference

frame generation. IEEE Transactions on Image

Processing 28, 10 (2019), 4832–4844.

Cucchiara, R., Grana, C., Piccardi, M., & Prati, A., 2003.

Detecting moving objects, ghosts, and shadows in video

streams. IEEE transactions on pattern analysis and

machine intelligence, 25(10), 1337-1342.

Calderara, S., Melli, R., Prati, A., & Cucchiara, R., 2006.

Reliable background suppression for complex scenes.

In Proceedings of the 4th ACM international workshop

on Video surveillance and sensor networks (pp. 211-

214). ACM.

Prati, A., Mikic, I., Grana, C., & Trivedi, M. M., 2001.

Shadow detection algorithms for traffic flow analysis: a

comparative study. In ITSC 2001. 2001 IEEE

Intelligent Transportation Systems. Proceedings (Cat.

No. 01TH8585) (pp. 340-345). IEEE.

Khan, S., & Shah, M., 2001. Object based segmentation of

video using color, motion and spatial information. In

Proceedings of the 2001 IEEE Computer Society

Conference on Computer Vision and Pattern

Recognition. CVPR 2001 (Vol. 2, pp. II-II). IEEE.

Chang, M. M., Tekalp, A. M., & Sezan, M. I., 1997.

Simultaneous motion estimation and segmentation.

IEEE transactions on image processing, 6(9), 1326-

1333.

Kuhn, H. W., 1955. The Hungarian method for the

assignment problem. Naval research logistics

quarterly, 2(1‐2), 83-97.

Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P.,

2004. Image quality assessment: from error visibility to

structural similarity. IEEE transactions on image

processing, 13(4), 600-612.

Hore, A., & Ziou, D., 2010. Image quality metrics: PSNR

vs. SSIM. In 2010 20th International Conference on

Pattern Recognition (pp. 2366-2369). IEEE.

Object based Hybrid Video Compression

791

Table 1: Performance Metrics for Proposed Object Based Video Compression Technique (Highlighting algorithm’s bit

reduction compared to H.265).

Input Video Sample

Background

Subtraction

Color

Conversion

from RGB to

HSV

Background

Image

Foreground

Bitstream

Stable Track

Bitstream

Generation

……….

Bitstream 1

DCT Based Standard

Encoder &

Reconstruction Loop

Bitstream 2

Bitstream n

DCT Based Standard

Decoder

Output Video

Object

Detection

stage 1 (Color

Feature)

Object

Detection

stage 2

(Motion

Feature)

Object

Tracking

Algorithm

Color

Conversion

HSV to RGB

Figure 1: An Overview of the Proposed Hybrid Object-based Algorithm.

Input Video Frame Foreground Mask Tracking Result

Corresponding

Blob Result

Final Output

Figure 2: Algorithm Results Across all Stages.

VISAPP 2021 - 16th International Conference on Computer Vision Theory and Applications

792