Video Data Hiding for Managing Privacy Information in Surveillance Systems
 2.3k Downloads
 3 Citations
Abstract
From copyright protection to error concealment, video data hiding has found usage in a great number of applications. In this work, we introduce the detailed framework of using data hiding for privacy information preservation in a video surveillance environment. To protect the privacy of individuals in a surveillance video, the images of selected individuals need to be erased, blurred, or rerendered. Such video modifications, however, destroy the authenticity of the surveillance video. We propose a new ratedistortionbased compressiondomain video data hiding algorithm for the purpose of storing that privacy information. Using this algorithm, we can safeguard the original video as we can reverse the modification process if proper authorization can be established. The proposed data hiding algorithm embeds the privacy information in optimal locations that minimize the perceptual distortion and bandwidth expansion due to the embedding of privacy data in the compressed domain. Both reversible and irreversible embedding techniques are considered within the proposed framework and extensive experiments are performed to demonstrate the effectiveness of the techniques.
Keywords
Discrete Cosine Transform Privacy Information Data Hiding Discrete Cosine Transform Coefficient Error Concealment1. Introduction
Video Surveillance has become a part of our daily lives. Closedcircuit cameras are mounted in countless shopping malls for deterring crimes, at toll booths for assessing tolls, and at traffic intersections for catching speeding drivers. Since the 9–11 terrorist attack, there have been much research efforts directed at applying advanced pattern recognition algorithms to video surveillance. While the objective is to turn the labor intensive surveillance monitoring process into a powerful automated system for counterterrorism, there is a growing concern that the new technologies can severely undermine individual's rights of privacy. The combination of ubiquitous cameras, wireless connectivity, and powerful recognition algorithms makes it easier than ever to monitor every aspect of our daily activities.
M. W. Hail has conducted a recent survey assessing citizens across demographic groups to see if they were comfortable with the expansion of government video surveillance if it protected privacy rights. (The survey was a cooperative effort through the University of Kentucky annual Kentucky Survey and the research was sponsored by a grant from the US Department of Homeland Security through the National Institute for Hometown Security.) The survey research was conducted utilizing a modified listassisted WaksbergMitofsky randomdigit dialing procedure for sampling and the population surveyed was noninstitutionalized Kentuckians eighteen years of age and older. The margin of error is Open image in new window 3.3% at the Open image in new window confidence interval. The respondents were asked, "Do you have a video security system that is used routinely?'' The results reflected that Open image in new window of employed Kentuckians have an operative video surveillance system at their workplace. We then asked of those employed, "Would you be interested in a video surveillance system at work if you knew it could protect an individual's privacy?'' The solid majority of Open image in new window expressed that they were interested in privacy protecting video surveillance. Urban residents, those in higher income levels, and those with advanced education attainment all were more disposed to privacy protecting video technology. Additionally, focus groups of law enforcement, first responders, hospitals, and public infrastructure managers have all reflected strong interest in privacy protecting video technology.
To mitigate public's concern on privacy violation, it is thus imperative to make privacy protection a priority in developing new surveillance technologies. There have been many recent work in enhancing privacy protection in surveillance systems [1, 2, 3, 4, 5, 6, 7, 8]. Many of them share the common theme of identifying sensitive information and applying image processing schemes for obfuscating that sensitive information. However, the security flaw overlooked in most of these current systems is that they fail to consider the security impact of modifying the surveillance videos. There are a number of security measures that must be incorporated before such modifications can be deployed. Firstly, mechanisms must be in place to authenticate modified videos so that no one can falsify a different modified video by adding and deleting images of objects or individuals. We call this measure privacy data authentication. The second measure is that the original video must be preserved and can only be retrieved under proper authorization. This is of paramount importance to any privacy protection schemes as all schemes are selective in the sense that the sensitive content are intended to a certain group for a certain purpose. No content should be permanently erased. For example, in a corporation, the security camera officer may have access to video contents of all visitors but not the employees; the chief privacy officer will have access to video contents of visitors and all employees except for the executive team but the law enforcement, with a proper order from the court, will have access to the true original footage. It has been postulated that such a static privacy policy would not be sufficient in more sophisticated environments or other sharing applications like teleconference where each participant might need to control the accessibility capability of each consumer of the content as in [9]. We call this measure privacy data preservation.
As explained earlier, except for the simplest organization, merely keeping the original video in encrypted form will not be sufficient in addressing these needs. On the other hand, it is advantageous to reuse the infrastructure of existing standard based video surveillance systems as much as possible. In this work, we propose using video data hiding for preserving the privacy information in the modified video itself in a seamless fashion. Using data hiding, the video bit stream will be accessible for both regular and authorized decoders but only the later can retrieve the hidden privacy information. The use of data hiding for privacy data preservation makes it completely independent from the obfuscation step unlike in some other work [10, 11]. Also, the presence of a single bit stream makes the process of video authentication much simpler to handle. Digitally signing the data hidden bit stream will authenticate the original video as well as all levels of privacy protected data.
 (1)Propose
a PrivacyProtected Video Surveillance System which can authenticate and preserve the privacy information.
 (2)Propose
a data hiding framework for managing privacy information which can support any kind of video modification.
 (3)Propose
a compression domain data hiding algorithm which offers high level of hiding capacity by embedding privacy information in selected transform coefficients optimized in terms of distortion and bitrate.
The rest of the paper is organized as follows. First in Section 2, we briefly review the stateoftheart in privacy protection and management systems and video data hiding. In Section 3, we describe the higher level design of our privacy protection system and its components. Section 4 introduces the data hiding framework for managing privacy information and various embedding techniques and perceptual distortion and rate models. Keeping the special constraints of data hiding for this application in consideration, we propose the optimization framework to find the embedding locations in Section 5. Experimental results are presented in Section 6 followed by conclusions in Section 7.
2. Related Work
In this section, we review existing work on visual privacy protection technologies followed by video data hiding techniques. There is a recent surge of interest in selective protection of visual objects in video surveillance. The PrivacyCam surveillance system developed at IBM protects privacy by revealing only the relevant information such as object tracks or suspicious activities [8]. Such a system is limited by the types of events it can detect and may have problems balancing privacy protection with the particular needs of a security officer. Alternatively, one can modify the video to obfuscate the appearance of individuals for privacy protection. In [1], the authors propose a privacy protecting video surveillance system which utilizes RFID sensors to identify incoming individuals, ascertains their privacy preference specified in an XMLbased privacy policy database, and finally uses a simple video masking technique to selectively conceal authorized individuals and display unauthorized intruders in the video. While [1] may be the first to describe a privacy protected video surveillance system, there are a large body of work that utilize such kinds of video modification for privacy protection. They range from the use of black boxes or large pixels in [2, 3] to complete object removal as in [1]. New techniques have also been proposed recently to replace a particular face with a generic face [6, 12] or a body with a stick figure [7] or complete object removal followed by inpainting of background and other foreground objects [13, 14].
All the aforementioned work target only at the modification of the video but not at the feasibility of recovering original video securely. To securely preserve the original video, selective scrambling of sensitive information using a private key have been recently proposed in [10, 11, 15]. These schemes differ in terms of the types of information scrambled which leads to different complexity and compression performances—spatial pixels are scrambled in [10], DCT signs and Wavelet coefficients are used in [11, 15], respectively. With the appropriate private key, the scrambling can be undone to retrieve the original video. These techniques have the advantages of simplicity with modified regions clearly marked. However, there are a number of drawbacks. First, similar to pixelation and blocking, scrambling is unable to fully protect the privacy of individuals, revealing their routes, motion, shape, and even intensity levels [6]. Second, as obfuscation is usually the first step in a complex process chain of a smart surveillance system, it introduces artifacts that can affect the performance of subsequent image processing. Lastly, the coupling of scrambling and data preservation prevents other obfuscation schemes like object replacement or removal to be used.
Using data hiding for privacy data preservation is more flexible as it completely isolates preservation from modification. Since our introduction of using data hiding for privacy data preservation in [16], there have been other work like [9, 17, 18, 19, 20] that employ a similar approach. Data hiding has been used in various applications such as copyright protection, authentication, fingerprinting, and error concealment. Each application imposes a different set of constraints in terms of capacity, perceptibility, and robustness [21]. Privacy data preservation certainly demands a large embedding capacity as we are hiding an entire video bitstream in the modified video. Perceptual quality of the embedded video is also of great importance as it effects the usability of the video for further processing. Robustness refers to the survivability of the hidden data under various processing operations. While it is a key requirement for applications like copyright protection and authentication, it is of less concern to a wellmanaged video surveillance system targeted to serve a single organization. Thus, we are focusing mainly on highcapacity fragile data hiding schemes. Another dimension is the reversibility of the hiding process which dictates if the embedded video can be fully restored after the hidden data is removed. While irreversible data hiding usually produces higher hiding capacity, reversible data hiding may be important for maintaining the authenticity of the original video. We shall consider both in this work.
Most irreversible data embedding and extracting approaches can be classified into two classes—spread spectrum and quantization index modulation (QIM). Spread spectrum techniques treats the data hiding problem as the transmission of the hidden information over a communication channel corrupted by the covered data [22]. QIM techniques use different quantization codebooks to represent the covered data with the selection of codebooks based on the hidden information [23]. QIMbased techniques usually have higher capacities than spreadspectrum schemes. The capacity of any QIM scheme is determined by the design of the quantization schemes. In [24], the authors propose to hide large volume of information into the nonzero DCT terms after quantization. This method cannot provide sufficient embedding capacity for our application because surveillance videos have high temporal correlation with a very large fraction of DCT coefficients being zero in the intercoded frames. In [25], the authors propose to implement the embedding in both zero and nonzero DCT coefficients but only in macro blocks with low inter frame velocity. This framework deals only with minimizing perceptual distortion without considering the increase in bit rate. Our initial scheme in [16] embeds the watermark bits at the highfrequency DCT coefficients during the compression process. Similar to [25], this method works well in terms of maintaining the output video quality but at an expense of much higher output bit rate.
Reversible data embedding can be broadly classified into three categories. The first class of methods like [26, 27] basically use lossless compression to create space for data hiding. The key idea is to embed the recovery information along with the hidden data to enable the reversibility at the decoder. This method is not suitable for our application because of its low capacity and that the information to be embedded is already a compressed bit stream. The second class of methods like [28, 29] work on residual expansion between pairs of coefficients in various transform domains. These methods assume high correlation between coefficients, hence most of the pairs would not overflow even after expanding the difference. The drawback of these schemes is the higher perceptual distortion caused due to significant changes in coefficient values. The third category of algorithms like [30] work on the concept of histogram bin shifting. This is suitable for our application because the histogram of DCT residue is Laplacian so that we can hide information at smallmagnitude coefficients without imposing significant perceptual distortion.
In Section 5, we describe a new approach of optimally placing hidden information in the DCT domain that simultaneously minimizes both the perceptual distortion and output bitrate. Our algorithm considers both rate and distortion and produces an optimal distribution of hidden bits among various DCT blocks. Our main contribution in the data hiding algorithm is an optimization framework to combine both the distortion and rate together as a single cost function and to use it in identifying the optimal locations to hide data. This allows a significant amount of information to be embedded into compressed bitstreams without disproportional increase in either output bit rate or perceptual distortion. This algorithm works for both irreversible and reversible embedding approaches.
3. Privacy Protected Video Surveillance
In this paper, we limit our discussion to the data hiding unit used for integrating the privacy information with the modified video. The privacy information contains the image objects of the individuals carrying the RFID tags, each padded with a black background to make a rectangular frame and compressed using a H.263 version 2 video encoder [32]. The embedding process is performed at frame level so that the decoder can reconstruct the privacy information as soon as the compressed bitstream of the same frame has arrived. Before the embedding, the compressed bitstream for each object is encrypted using the Advanced Encryption Standard (AES) with a 128bit key and appended with a small fixedsize header. Details of the encryption process, key management and the header format can be found in [31]. It is this encrypted data stream that is embedded into the modified video. The data hiding scheme is combined with the video encoder and produces a H.263compliant bitstream of the protected video to be stored in the database. The privacy protected video can be accessed without any restriction with a standard decoder as all the privacy information are encrypted and hidden in the bitstream. With a special decoder, the hidden data can be retrieved and the authorized user can decrypt the privacy information corresponding to his access level.
4. Hiding Privacy Information
4.1. Perceptual Distortion
To identify the embedding locations that cause the minimal disturbance to visual quality, we need a distortion metric to input into our optimization framework. Mean square distortion does not work for our goal of finding the optimal DCT coefficients to embed data bits—as DCT is an orthogonal transform, the mean square distortion for the same number of embedded bits will always be the same regardless of which DCT coefficients are used. Instead, we adopt the DCT perceptual model proposed by Watson [33], which has been shown to better correlate with the human visual system than standard mean square distortion. While there are other more sophisticated videobased perceptual models such as the one in [34], we adopt the Watson model for its simplicity to be included in our optimization algorithm.
4.2. Irreversible Embedding Process
This embedding, however, is not invertible. Since the quantization is altered to a coarser level as part of data embedding, it causes irrecoverable loss of data. For a single bit embedding, the maximum quantization noise doubles compared to that of without embedding. Beside the irreversible changes to the coefficient, the modified reference frame in the motion loop propagates the effect of data hiding into future frames, making the changes permanent. This implies that the reconstructed video will be slightly different from the originally compressed version. Such an irreversible embedding method is not suitable for certain applications that demand the original video to be unaltered by the data hiding process.
4.3. Reversible Embedding Process
Using the previous embedding technique, the decoder has no way to remove the distortion introduced by the embedding process. In this subsection, we explain a reversible embedding algorithm whose effect can be reversed on the decoder side after data extraction. A key requirement for our application is that the output bitstream with hidden data must be decodable with good quality by a standardcompliant decoder unaware of the embedding. This implies that we need to avoid any error caused by drifting and as such, the decoded frame with the hidden data must be used in the feedback path in the motion loop. As the motion compensation does not respect the DCT block boundary, the effect of hiding one bit in a DCT coefficient may spread to different spatial areas after many frames. It is an open question on how to make this temporal spreading reversible. In our current implementation, we focus on making the DCT embedding process reversible and prevent temporal spreading by restricting our attention to either intracoded frames or intracodedenhanced frames in a twolayer scalable codec.
 (1)If
Open image in new window is zero, extract Open image in new window bits from the privacy data buffer and set Open image in new window where Open image in new window is the decimal value of these Open image in new window privacy data bits.
 (2)If
Open image in new window is negative, no embedding is done and Open image in new window .
 (3)If
Open image in new window is positive, no embedding is done and Open image in new window .
The embedding is done only at zero coefficients while all the other coefficients visited in the scan order are displaced in either positive or negative direction. Compared with the irreversible embedding, the capacity here is smaller as data can only be embedded to zero coefficients. Also reversible embedding induces higher distortion as even some nonzero coefficients must be altered by Open image in new window without actually embedding at that position.
 (1)If
Open image in new window , Open image in new window hidden bits can be obtained as the binary equivalent of the decimal number Open image in new window and Open image in new window .
 (2)If
Open image in new window , no bit is hidden in this coefficient and Open image in new window .
 (3)If
Open image in new window , no bit is hidden in this coefficient and Open image in new window .
4.4. Rate Model
Data hiding effects the compression performance—simply choosing the distortionoptimal locations based on the perceptual model may increase the output bitrate manyfold. As surveillance video is typically quite static, many DCT blocks do not have any nonzero coefficients. Hiding bits into these zero blocks, while perceptual optimal, may significantly increase the bitrate. This is caused by the fragmentation of the long runlength patterns which are assumed to be frequent by the entropy coder. One possible approach to mitigate this problem is to limit the number of blocks to be modified [16]. However, the fewer blocks used for embedding, the more spatially concentrated the embedding becomes which will make the distortion more visible. As such, we need to measure the increase in rate by different embedding strategies so as to produce the optimal tradeoff with the distortion. The rate increase for a particular embedding is calculated using the actual entropy coder used for compression. As both the encoder and the decoder need to compute the rate function so as to derive the optimal data hiding positions, the actual privacy data cannot be used as it is not available at the decoder. Instead, we approximate the embedding by assuming the "worstcase'' embedding, that is, we choose the hidden bit value that causes the higher increase in bitrate.
5. RateDistortionOptimized Data Hiding
where Open image in new window is the variable that denotes the number of coefficients to be modified, Open image in new window is the target number of bits to be embedded, Open image in new window is the cost function as described in (9), and Open image in new window is any selection of Open image in new window DCT coefficients for embedding the data. We assume that a constant number of bits are embedded at each DCT coefficient and focus the optimization on choosing the coefficients for embedding (with the exception of the last DCT coefficient for embedding which may contain less than the target number). While it is entirely feasible to explore the dimension of embedding different numbers of bits to different coefficients, our preliminary experiments indicate that the gain is too small to justify the significant expansion of the search space for the optimization.
The second problem is an efficient way to search for Open image in new window that provides an optimal allocation of embedded bits to each block. The following two subsections describe our approach in tackling these problems.
5.1. Cost Function Computation for DCT Blocks
There are two components to the cost function introduced in (9): distortion and rate increase due to data hiding. Our distortion function as described in (4) is additive with each coefficient having an independent contribution. The rate increase due to the modification of a coefficient is far more complex. It depends on neighboring coefficients as consecutive coefficients along the zigzag scan are encoded together as a single runlength pattern. In the H.263 standard, a runlength pattern is defined as a run of zero coefficients followed by a nonzero coefficient. The length of the run and the nonzero coefficient determine the length of the codeword, and the longer the runlength, the shorter the codeword in the Huffman table becomes. Embedding a bit in any zero coefficients will break the runlength pattern into two and the bitrate increase will depend on the original and the resulting runlength patterns.
At first glance, the interdependency created by the runlength coding seems to evade any structural exploitation of the optimization problem. Exhaustive search of Open image in new window patterns, where Open image in new window is the number of candidate coefficients and Open image in new window is the number of embedded bits, seems inevitable. For a Open image in new window DCT block, such an exhaustive search will need to encode more than Open image in new window patterns in order to determine all the optimal positions for embedding Open image in new window bits. This is clearly impossible in practice. Fortunately, the "worstcase'' embedding assumption in our rate model as described in Section 4.4 provides a DynamicProgramming (DP) based solution to the optimization problem. In the actual embedding procedure as described in (6) and (7), embedding a specific bit may turn a nonzero DCT coefficient into zero and actually reduces the bitrate by making a runlength pattern longer. The "worstcase'' embedding, which is employed without the knowledge of the hidden bit, assumes the worst case and never makes a nonzero coefficient zero. This simple observation enables us to develop a recursive solution to the optimization problem based on the position of the last embedded bit.
(since the approach of computing the cost function is the same for each block, we drop the block index Open image in new window in representing the block cost function Open image in new window ).
This is precisely the Bellman principle that leads to a dynamic programming formulation to solve for Open image in new window [36]. Now we can state the full algorithm to compute Open image in new window for Open image in new window as follows.
(1)There are 64 stages with each stage representing the embedding of one bit. At stage Open image in new window where Open image in new window , there are Open image in new window states representing all possible DCT coefficients in the zigzag order that can store the Open image in new window th embedded bit. The minimum cost function Open image in new window will be computed at stage Open image in new window and state Open image in new window . The trellis depicting this construction is shown in Figure 3.
(2)The calculation starts from stage one. At stage Open image in new window , we compute the cost function at state Open image in new window by first worstcase embedding a bit at the Open image in new window th coefficient and then identifying the minimum combined cost among all the states up to Open image in new window in stage Open image in new window plus the extra cost incurred by the embedding at the Open image in new window th coefficient.
Number of DCT patterns examined by different algorithms in computing C*(M).
Approach  Number of DCT patterns examined 

Exhaustive search  Open image in new window 10^{19} 
Dynamic programming  43,744 
Greedy  2,080 
Fixed pattern  64 
5.2. Bit Allocation by Lagrangian Approximation
Sweeping through Open image in new window from Open image in new window to Open image in new window will examine the convex hull of all the block cost functions Open image in new window . While there exist efficient tree pruning techniques to search for the optimal value Open image in new window , the large number of DCT blocks in a frame can still render such techniques computationally intensive. As we will demonstrate in Section 6, the block cost functions in most cases can be well approximated by a second order curve. This allows us to devise a simple search strategy to quickly identify the appropriate value of Open image in new window .
Since the actual problem is a discrete one, we can only use Open image in new window from (21) as an initial slope and search for the exact slope in its neighborhood to match our target embedding requirement. At this optimal slope on each curve, we can identify the number of embedding locations Open image in new window for each DCT block. These Open image in new window embedding locations within each block are chosen from the same optimal order which are already calculated during the cost cuve generation process.
6. Experiments

Minnesota [37]. Two persons walk towards and cross each other while the camera is slowly panning (39 frames).

Board. One person walk across the scene, briefly occluded by a partition board (101 frames).

Twopersons. Two persons walk towards and cross each other (89 frames).

Threepersons. Two persons walk towards the right and one to the left, occluding each other briefly (73 frames).

Conference. Five persons sit around a conference table with two leaving one after the other (356 frames).

Hall. A standard sequence used in video compression (299 frames).
The data hiding algorithm is implemented based on the TMN Coder Version 3.0 of the ITUT H.263 version 2 by University of British Columbia. All sequences are compressed using a constant quantization parameter with the first frame intracoded and the remaining intercoded. Despite the differences in the original framerates among the sequences, the compression frame rate has been set to 30 Hz. The encoding performance is measured based on running the program on a Windows XP Professional machine with Intel Xeon Processor at 2 GHz with 4 GB memory.
6.1. Selection of DCT Coefficients for Embedding
In the first experiment, we consider the performances among different schemes in selecting DCT coefficients to embed hidden data. The three tested schemes are the DPbased optimal scheme, the greedy scheme and the fixed reversed zigzag patterns as described in Section 5.
Comparing the performances among DPoptimal, Greedy, and Zigzag on four different inpainted sequences.
Inpainted sequences  Minnesota  Board  Twopersons  Threepersons  

Bitrate (kbps)  DP optimal  927.3  96.9  344.8  472.3 
Greedy  933.5  96.1  345.8  473.1  
Zigzag  937.2  97.0  340.0  463.2  
PSNRY (dB)  DP optimal  31.83  37.73  35.37  34.30 
Greedy  31.84  37.80  35.36  34.31  
Zigzag  31.62  37.73  35.33  34.28  
Distortion  DP optimal  30.69  18.50  22.88  30.61 
Greedy  30.84  18.83  22.95  30.46  
Zigzag  42.96  22.43  29.67  38.55  
Cost ( Open image in new window )  DP optimal  731.7  252.5  438.4  591.6 
Greedy  736.7  256.3  439.7  590.2  
Zigzag  889.9  301.7  520.8  686.3  
Speed (sec/frame)  DP optimal  904.3  890.1  892.0  892.9 
Greedy  35.5  35.2  34.8  35.1  
Zigzag  1.6  1.5  1.5  1.5 
We should point out that the computational speeds provided in Table 2 are based on a nonoptimized implementation of the algorithms and also include the entire compression process, which amounts to roughly 0.6 second. Significant speedup can be achieved, for example, by updating only those blocks that are different from the previous frames as indicated by the macroblock modes. In fact, as the motion in typical surveillance videos is scarce, it is conceivable to update the cost function only occasionally rather than at every frame without losing much optimality. Furthermore, the complexity is mainly due to the computation of the cost functions for different DCT blocks which are certainly amenable to parallel implementation. While it is not the focus of this paper on the realtime implementation of the data hiding process, we believe that significant improvement in computational speed is indeed possible.
6.2. Bit Allocation for DCT Blocks
Comparing the performances between Lagrangian and Equal distribution of hidden data among DCT blocks.
Inpainted sequences  Minnesota  Board  Twopersons  Threepersons  

Bitrate (kbps)  Lagrangian  945.81  97.84  361.64  511.06 
Equal  1696.04  127.71  1018.67  1208.75  
PSNRY (dB)  Lagrangian  31.84  37.69  35.41  34.16 
Equal  31.34  37.68  34.28  33.45  
Distortion  Lagrangian  27.72  15.48  20.15  26.52 
Equal  18.86  14.80  16.65  16.26 
6.3. Different Privacy Protection Schemes
Comparing the performances among different video obfuscation schemes.
Inpainted sequences  Minnesota  Board  Twopersons  Threepersons  Conference  Hall  

PSNRY (dB)  Silhouette  34.87  38.47  38.13  37.54  35.72  35.15 
Scrambled  31.58  35.74  34.65  33.77  34.61  33.49  
Inpainted  31.84  37.80  35.36  34.31  34.96  33.43  
PSNRY drop %  Silhouette  9.7  1.8  8.4  9.7  2.4  3.8 
Scrambled  8.3  1.2  5.0  5.5  2.0  3.2  
Inpainted  9.7  1.3  9.7  11.2  2.5  3.6  
Distortion  Silhouette  28.14  18.29  22.71  28.84  27.11  28.97 
Scrambled  25.47  14.72  17.76  17.96  19.30  20.35  
Inpainted  30.85  18.83  22.95  30.46  27.29  28.53  
Bitrate (kbps)  Silhouette  1087.0  92.4  387.5  587.2  145.2  315.0 
Scrambled  1301.2  822.9  798.4  1124.3  359.8  1113.0  
Inpainted  933.5  96.1  345.8  473.1  127.7  285.2  
Bitrate increase %  Silhouette  61.9  29.2  91.6  78.9  31.1  45.5 
Scrambled  27.1  1.4  
Inpainted  87.0  26.2  115.8  111.2  44.6  59.6  
Marktowork bitrate  Silhouette  0.66  0.44  1.46  1.07  0.53  0.60 
Scrambled  0.35  0.02  0.17  0.18  0.12  0.08  
Inpainted  1.16  0.40  3.00  3.13  0.77  0.83 
6.4. Different Operating Parameters
Rate and Distortion for irreversible embedding HallMonitor at varying QP and δ values.
QP  Open image in new window (kbps)  Open image in new window (kbps)  Open image in new window (kbps)  Rate increase %  PSNRY (dB)  PSNRY drop %  Distortion  

5  359.72  161.38  0  754.5  44.79  36.77  3.29  15.76 
0.5  698.04  33.96  36.76  3.31  21.06  
1  690.54  32.52  36.73  3.39  58.61  
10  97.36  81.26  0  314.32  75.97  33.45  3.57  22.59 
0.5  285.15  59.64  33.43  3.63  28.53  
1  267.50  49.76  33.42  3.66  98.76  
15  59.12  54.77  0  202.98  78.22  31.37  3.77  28.64 
0.5  186.38  63.65  31.42  3.62  34.95  
1  170.50  49.71  31.30  3.99  138.77  
20  44.9  42.82  0  152.87  74.27  29.93  3.64  35.39 
0.5  141.9  61.76  29.97  3.51  41.92  
1  129.1  47.17  29.82  3.99  178.27 
Rate and distortion for irreversible embedding for Conference at varying QP and δ values.
QP  Open image in new window (kbps)  Open image in new window (kbps)  Open image in new window (kbps)  Rate increase %  PSNRY (dB)  PSNRY drop %  Distortion  

5  122.42  76.17  0  309.27  55.73  38.29  3.28  18.02 
0.5  279.37  40.68  38.34  3.16  28.67  
1  275.02  38.49  38.29  3.28  60.38  
10  49.81  38.44  0  135.50  53.54  34.94  2.57  28.16 
0.5  123.63  40.09  34.95  2.54  39.58  
1  120.93  37.03  34.84  2.84  94.18  
15  36.41  28.66  0  94.12  44.64  33.19  2.35  39.99 
0.5  87.97  35.19  33.22  2.27  47.93  
1  85.87  31.97  33.05  2.77  134.61  
20  30.33  23.96  0  74.23  36.73  31.89  2.39  51.33 
0.5  70.66  30.15  31.86  2.48  58.95  
1  68.41  26.01  31.85  2.51  161.62 
6.5. Reversible Embedding
Rate and distortion for reversible embedding using either intracoded frames (I) or enhanced intraframes (EI) at varying QP and δ values.
QP  Open image in new window (kbps)  Rate increase  Distortion  

I  EI  I  EI  I  EI  
Hall monitor  
5  0  6357.18  2952.49  34.54  11.86  448.80  137.89 
0.5  6225.73  2895.05  31.76  9.69  449.70  149.19  
1  6171.53  2877.52  30.61  9.02  791.91  186.96  
10  0  4486.32  1460.35  42.65  6.86  177.87  157.18 
0.5  4236.80  1441.40  34.72  5.47  205.28  170.34  
1  4227.27  1439.44  11.94  5.33  216.24  182.05  
15  0  3734.78  1025.34  48.02  5.54  235.81  140.01 
0.5  3548.35  1023.92  40.63  5.40  249.32  167.56  
1  3520.27  1016.06  39.52  4.59  263.15  244.73  
20  0  3228.89  822.79  47.82  7.25  290.19  140.45 
0.5  3154.32  812.12  44.41  5.86  291.26  165.05  
1  3104.45  810.54  42.12  5.65  306.59  309.36  
Conference  
5  0  4637.92  2190.74  30.90  3.28  104.69  80.79 
0.5  4624.90  2172.82  30.53  2.44  120.08  82.53  
1  4602.82  2164.65  29.91  2.05  131.70  103.17  
10  0  3292.24  1143.23  41.21  2.19  154.80  118.28 
0.5  3280.63  1131.16  40.71  1.11  167.42  120.66  
1  3277.95  1128.78  40.60  0.90  248.62  143.75  
15  0  2849.20  868.24  47.19  2.02  187.94  106.50 
0.5  2845.70  859.10  47.11  0.95  193.69  146.05  
1  2820.19  857.51  45.79  0.76  287.39  195.25  
20  0  2630.75  739.25  51.21  3.25  207.57  125.42 
0.5  2555.75  735.21  46.90  2.68  210.97  157.84  
1  2514.27  729.63  44.52  1.90  362.06  251.81 
7. Conclusions
In this paper, we have presented a privacyprotecting video surveillance system which offers multiple levels of secure privacy information preservation. Novel irreversible and reversible data hiding methods have been proposed to hide large amount of privacy information into the host video. An optimization framework has been proposed to identify DCT coefficients for hiding information that simultaneously minimize the perceptual distortion and the rate increase caused due to embedded information. Extensive experimental results have been presented to demonstrate the efficient implementation of our algorithms and their effectiveness in preserving privacy data.
Notes
Acknowledgments
The authors would like to acknowledge the support from Department of Justice for this work and also thank the anonymous reviewers for their constructive comments.
References
 1.Wickramasuriya J, Datt M, Mehrotra S, Venkatasubramanian N: Privacy protecting data collection in media spaces. Proceedings of the 12th ACM International Conference on Multimedia, October 2004, New York, NY, USA 4855.CrossRefGoogle Scholar
 2.Berger AM: Privacy mode for acquisition cameras and camcorders. US patent 6067399, Sony Corporation, May 2000Google Scholar
 3.Wada J, Kaiyama K, Ikoma K, Kogane H: Monitor camera system and method of displaying picture from monitor camera thereof. European patent, EP 1081955 A2, Matsushita Electric Industrial, April 2001Google Scholar
 4.Schiff J, Meingast M, Mulligan D, Sastry S, Goldberg K: Respectful cameras: detecting visual markers in realtime to address privacy concerns. Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS '07), April 2007, Beijing, China 971978.Google Scholar
 5.Chen D, Chang Y, Yan R, Yang J: Tools for protecting the privacy of specific individuals in video. EURASIP Journal on Advances in Signal Processing 2007, 2007:9.Google Scholar
 6.Newton EM, Sweeney L, Main B: Preserving privacy by deidentifying face images. IEEE Transactions on Knowledge and Data Engineering 2005, 17(2):232243. 10.1109/TKDE.2005.32CrossRefGoogle Scholar
 7.Wactlar H, Stevens S, Ng T: Enabling personal privacy protection preferences in collaborative video observation. NSF Award Abstract 0534625, http://www.nsf.gov/awardsearch/showAward.do
 8.Senior A, Pankanti S, Hampapur A, Brown L, Tian YL, Ekin A: Blinkering surveillance: enabling video privacy through computer vision. Security and Privacy 2005, 3: 5057. 10.1109/MSP.2005.65CrossRefGoogle Scholar
 9.Cheung SC, Paruchuri JK, Nguyen T: Managing privacy data in pervasive camera networks. Proceedings of the 15th IEEE International Conference on Image Processing (ICIP '08), October 2008, San Diego, Calif, USAGoogle Scholar
 10.Boult TE: Pico: privacy through invertible cryptographic obscuration. In Proceedings of the Computer Vision for Interactive and Intelligent Environments, November 2005, The Dr. Bradley D. Carter Workshop Series Edited by: Bradley DC. 2738.CrossRefGoogle Scholar
 11.Dufaux F, Ebrahimi T: Scrambling for video surveillance with privacy. Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW '06), June 2006, New York, NY, USA 160.Google Scholar
 12.Yu X, Babaguchi N: Privacy preserving: hiding a face in a face. Proceedings of the 8th Asian Conference on Computer Vision (ACCV '07), November 2007, Tokyo, Japan, Lecture Notes in Computer Science 4844: 651661.Google Scholar
 13.Cheung SC, Zhao J, Venkatesh MV: Efficient objectbased video inpainting. Proceedings of the IEEE International Conference on Image Processing (ICIP '06), October 2006, Atlanta, Ga, USA 705708.Google Scholar
 14.Venkatesh MV, Cheung SC, Zhao J: Efficient object based video inpainting. Pattern Recognition Letters 2009, 30(2):168179. 10.1016/j.patrec.2008.03.011CrossRefGoogle Scholar
 15.Martin K, Plataniotis KN: Privacy protected surveillance using secure visual object coding. IEEE Transactions on Circuits and Systems for Video Technology 2008, 18: 11521162. 10.1109/TCSVT.2008.927110CrossRefGoogle Scholar
 16.Zhang W, Cheung SC, Chen M: Hiding privacy information in video surveillance system. Proceedings of the 12th IEEE International Conference on Image Processing (ICIP '05), September 2005, Genova, ItalyGoogle Scholar
 17.Yu X, Babaguchi N: Hiding a Face in a Face, Lecture Notes in Computer Science. Volume 4844. Springer, Heidelberg, Berlin; 2007.Google Scholar
 18.Li G, Ito Y, Yu X, Nitta N, Babaguchi N: A discrete wavelet transform based recoverable image processing for privacy protection. Proceedings of the International Conference on Image Processing (ICIP '08), 2008 13721375.Google Scholar
 19.Paruchuri JK, Cheung SC: Joint optimization of data hiding and video compression. Proceedings of the IEEE International Symposium on Circuists and Systems (ISCAS '08), May 2008, Washington, DC, USAGoogle Scholar
 20.Meuel P, Chaumont M, Puech W: Data hiding in h.264 video for lossless reconstruction of region of interest. In Proceedings of the 15th European Signal Processing Conference (EUSIPCO '07), September 2007, Poznan, Poland. HAL—CCSD; 120124.Google Scholar
 21.Cox I, Miller M, Bloom J: Digital Watermarking. Morgan Kaufmann, San Fransisco, Calif, USA; 2002.Google Scholar
 22.Cox I, Kilian J, Leighton T, Shamoon T: Secure spread spectrum watermarking for multimedia. IEEE Transactions on Image Processing 1997, 6(12):16731687. 10.1109/83.650120CrossRefGoogle Scholar
 23.Chen B, Wornell GW: Quantization index modulation: a class of provably good methods for digital watermarking and information embedding. Proceedings of the IEEE International Symposium on Information Theory (ISIT '00), June 2000, Sorrento, ItalyGoogle Scholar
 24.Solanki K, Jacobsen N, Chandrasekaran S, Madhow U, Manjunath B: Highvolume data hiding in images: introducing perceptual criteria into quantization based embedding. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '02), May 2002, Orlando, Fla, USA 4: 34853488.Google Scholar
 25.Sur A, Mukherjee J: Adaptive data hiding in compressed video domain. Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP '06), 2006 738748.CrossRefGoogle Scholar
 26.Celik MU, Sharma G, Tekalp AM, Saber E: Lossless generalizedlsb data embedding. IEEE Transactions on Image Processing 2005, 14(2):253266. 10.1109/TIP.2004.840686CrossRefGoogle Scholar
 27.Goljan M, Fridrich J, Du R: Distortionfree data embedding for images. Proceedings of the 4th International Workshop on Information Hiding, 2001, Pittsburgh, Pa, USA 2741.CrossRefGoogle Scholar
 28.Alattar AM: Reversible watermark using the difference expansion of a generalized integer transform. IEEE Transactions on Image Processing 2004, 13(8):11471156. 10.1109/TIP.2004.828418MathSciNetCrossRefGoogle Scholar
 29.Thodi D, Rodriguez J: Reversible watermarking by predictionerror expansion. Proceedings of the 6th IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI '04), September 2004, Porto, Portugal 6: 2125.Google Scholar
 30.Chang CC, Tai WL, Lin MH: A reversible data hiding scheme with modified side match vector quantization. Proceedings of the 19th International Conference on Advanced Information Networking and Applications (AINA '05), March 2005, Taipei, Taiwan 1: 947952.Google Scholar
 31.Cheung SC, Venkatesh MV, Paruchuri JK, Zhao J, Nguyen T: Protecting and managing privacy information in video surveillance systems. In Protecting Privacy in Video Surveillance. Springer, New York, NY, USA; 2009.Google Scholar
 32.Video Coding for Low Bitrate Communication Version 2, ITUT Recommendation H.263 Version 2, 1998Google Scholar
 33.Watson A: Dct quantization matrices optimized for individual images. Human Vision, Visual Processing, and Digital Display IV, October 1993, Proceedings of SPIE 1913: 202216.CrossRefGoogle Scholar
 34.Seshadrinathan K, Bovik AC: A structural similarity metric for video based on motion models. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '07), May 2007, Honolulu, Hawaii, USA 1: 869872.Google Scholar
 35.Shoham Y, Gersho A: Efficient bit allocation for an arbitrary set of quantizers. IEEE Transactions on Acoustics, Speech, and Signal Processing 1988, 36(9):14451453. 10.1109/29.90373CrossRefMATHGoogle Scholar
 36.Bellman R: On the theory of dynamic programming. Proceedings of the National Academy of Sciences 1952, 38(8):716719. 10.1073/pnas.38.8.716MathSciNetCrossRefMATHGoogle Scholar
 37.Patwardhan KA, Sapiro G, Bertalmio M: Video inpainting under constrained camera motion. IEEE Transactions on Image Processing 2007, 16(2):545553. 10.1109/TIP.2006.888343MathSciNetCrossRefGoogle Scholar
Copyright information
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.