Joint Optimization for Compressive Video Sensing and Reconstruction Under Hardware Constraints
Compressive video sensing is the process of encoding multiple sub-frames into a single frame with controlled sensor exposures and reconstructing the sub-frames from the single compressed frame. It is known that spatially and temporally random exposures provide the most balanced compression in terms of signal recovery. However, sensors that achieve a fully random exposure on each pixel cannot be easily realized in practice because the circuit of the sensor becomes complicated and incompatible with the sensitivity and resolution. Therefore, it is necessary to design an exposure pattern by considering the constraints enforced by hardware. In this paper, we propose a method of jointly optimizing the exposure patterns of compressive sensing and the reconstruction framework under hardware constraints. By conducting a simulation and actual experiments, we demonstrated that the proposed framework can reconstruct multiple sub-frame images with higher quality.
KeywordsCompressive sensing Video reconstruction Deep neural network
This work was supported by JSPS KAKENHI (Grant Number 18K19818).
- 2.Sonoda, T., Nagahara, H., Endo, K., Sugiyama, Y., Taniguchi, R.: High-speed imaging using CMOS image sensor with quasi pixel-wise exposure. In: International Conference on Computational Photography (ICCP), pp. 1–11 (2016)Google Scholar
- 3.Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., Nayar, S.K.: Video from a single coded exposure photograph using a learned over-complete dictionary. In: International Conference on Computer Vision (ICCV), pp. 287–294 (2011)Google Scholar
- 6.Iliadis, M., Spinoulas, L., Katsaggelos, A.K.: DeepBinaryMask: learning a binary mask for video compressive sensing. arXiv preprint arXiv:1607.03343 (2016)
- 7.Sarhangnejad, N., Lee, H., Katic, N., O’Toole, M., Kutulakos, K., Genov, R.: CMOS image sensor architecture for primal-dual coding. In: International Image Sensor Workshop (2017)Google Scholar
- 12.Majidzadeh, V., Jacques, L., Schmid, A., Vandergheynst, P., Leblebici, Y.: A (256–256) pixel 76.7 mW CMOS imager/compressor based on real-time in-pixel compressive sensing. In: International Symposium on Circuits and Systems (ISCAS) (2010)Google Scholar
- 14.Spinoulas, L., He, K., Cossairt, O., Katsaggelos, A.: Video compressive sensing with on-chip programmable subsampling. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (2015)Google Scholar
- 16.Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: The Twenty-Seventh Asilomar Conference on Signals, Systems and Computers (1993)Google Scholar
- 18.Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or \(-1\). arXiv preprint arXiv:1602.02830 (2016)
- 19.Hamamatsu Photonics K.K. Imaging device. Japan patent JP2015-216594A (2015)Google Scholar