JP7620308B2 - Imaging device and measurement coding device - Google Patents

Description

The present invention relates to an imaging device and a measurement encoding device.

The Nyquist-Shannon sampling theorem states that the sampling rate (SR) must be at least twice the frequency of the signal, but conventional imaging technology faces implementation limitations. Recently developed CMOS image sensors (CIS) have resolutions of 4K, 8K, and 16K, and are expensive and consume high power. Furthermore, they require huge memory space to store large amounts of image data.

To reduce the cost of storing and transmitting image data, multimedia compression tools such as JPEG/JPEG2000, AVC (Advanced Video Coding), HEVC (High-Efficiency Video Coding), and VVC (Versatile Video Coding) are widely used. Software and hardware implementations can reach up to 10 times the complexity with each generation, which is the result of CIS design bottlenecks.

All data compression algorithms reduce spatial and temporal redundancy. The original signal is effectively compressed into a sparse vector of coefficients using a common transformation basis such as the Discrete Cosine Transform (DCT), a variant of the Fourier Transform, or the Discrete Wavelet Transform (DWT), which is lossy.

When performing lossy compression to meet low bitrates, the question arises as to why it is necessary to collect high-dimensional and redundant measurements in the first place. Now, instead of collecting and compressing high-dimensional measurements, we can take the compressed measurements and derive the sparsest high-dimensional signal that matches the measurements. This is called "compressive sensing" or "compressive sampling", and will be abbreviated as "CS" below.

Such compressed sensing is a new perspective that has the potential to revolutionize data collection and processing, enabling several enhanced approaches in signal processing, data compression, solving inverse problems, designing radiating systems, radar, imaging through walls, and synthetic aperture radar (SAR) antennas.

The advancement of block-based computational imaging (BCI) using compressed sensing has attracted attention in recent years. This paradigm will enable next-generation image sensors to achieve superior performance in terms of shorter readout latency and reduced power consumption.

Nevertheless, there is an increasing need for new encoding algorithms for BCI measurements with minimal processing. In general, compressed sensing is built on two main fundamental conditions consisting of sparsity and incoherence.

An important condition for applying CS in a particular application is that the signal property, when represented in a particular orthogonal transform basis, must be sparse. This property means that only a small number of coefficients contain most of the signal information and can be expressed as follows:

ここで、

Where:

はベクトル化された信号である。

is the vectorized signal.

は、２次元信号である基底

is a two-dimensional signal basis

におけるＸの射影を含むスパースベクトル、すなわち、画像又はビデオがラスタースキャンによって列ベクトルに変形される。

A sparse vector containing the projection of X in x, y, i.e., an image or video is transformed into a column vector by raster scanning.

The theory of compressed sensing claims that incoherence is key, related to the quality of the measurement and the way the signal is sampled. Therefore, the measurement matrix used to sample X

はコヒーレンスが低くなければならない。

must have low coherence.

Various distributions such as Bernoulli, Gaussian, and binary

が得られる。圧縮された測定値

The compressed measurement value is obtained.

を劣決定系の射影を用いて得ることができる。これは数学的に

can be obtained by using the underdetermined projection. This is mathematically

と記述できる。ここで、ｍの次元はｎ未満でなければならない。

where the dimension of m must be less than n.

To ensure good signal recovery, a property of the matrix called the Restricted Isometry Property (RIP) is important. The following formula can be used to determine the lower bound dimension of m for a non-uniform variance measurement matrix:

ここで、

Where:

である。したがって、不均一な分散測定行列に対するｍの下限の次元は、凸最適化を介して非適切（ill-posed）な線形逆問題を解くことによって、次のように記述できる。

Therefore, the dimensionality of the lower bound of m for non-uniform variance measurement matrices can be written as follows by solving an ill-posed linear inverse problem via convex optimization:

信号を復元するために、圧縮センシングは、信号Ｘが疎変換

To recover the signal, compressed sensing assumes that the signal X is transformed into a sparse

により圧縮可能であり、

It can be compressed by

は、

teeth,

と非常にインコヒーレントである必要がある。したがって、係数ドメイン（測定ドメイン）の不完全な測定値の次元ｍから以下のように高精度に復元できる。

Therefore, it can be accurately reconstructed from the dimension m of the imperfect measurements in the coefficient domain (measurement domain) as follows:

ここで、

Where:

は逆変換である。

is the inverse transformation.

Therefore, using a convex optimization called l ₁ minimization, we can recover the measurements in a classical way:

これは次のように単純化できる。

This can be simplified as follows:

As greedy-based restoration algorithms, OMP (Orthogonal Matching Pursuit), its extensions Stepwise OMP, A*Omp, CoSaMP, and TwIST2 have been proposed for compressed sensing.

Compared to _l1 minimization, greedy-based restoration algorithms minimize a series of subspace problems and are generally faster. However, they require higher prior knowledge and good measurements, and it is not practical to obtain such information in some applications where SR needs to be reduced at the sensing device.

Scalar quantization (SQ) offered a simple measurement coding approach (see non-patent literature 1, 2). It has been established that SQ coding is very inefficient in terms of rate-distortion (RD) performance by information theory. Compared to traditional coding, SQ coding produces low bit-rate utilization but results in high reconstruction errors. Therefore, complex reconstruction algorithms are required to predict the corrupted encoded measurements, such as Quantized Iterative Hard Thresholding (QIHT), Quantized Compressed Sampling Matching Pursuit (QCoSaMP), and Adaptive Outlier Pursuit For Quantized Iterative Hard Thresholding (AOP-QIHT). The quality degradation is finally visible in the reconstructed image/video.

Differential Pulse-Code Modulation (DPCM) is also a simple approach that can be applied to reduce the bit rate on the transmission channel (see non-patent document 3). In DPCM coding, one previous coded block is used to predict the current block. However, the previous coded block may contain irrelevant information that is not related to the current block, resulting in an unnecessary increase in the bit rate. Nevertheless, DPCM coding provides a practical coding method.

The coding method in Non-Patent Document 4 performs DCT coding on BCI measurements. This approach improves coding efficiency over SQ and DPCM.

In the study of non-patent literature 5, we proposed an alternative approach from the state-of-the-art coding schemes investigated in the quantizer dead zone. In non-patent literature 5, we stated that the quality degradation mainly comes from the quantizer, and the distortion is related to both quantization and SR in the measurement coding scheme. Therefore, in this approach, we designed system parameters including the selection of the measurement matrix, the trade-off between quantization and SR, and a restoration algorithm using deep learning. Furthermore, an effective method can jointly control the quantization step and SR to achieve near-optimal quality at any bit rate.

The directional coding approach was first introduced in Non-Patent Document 6. Spatially Directional Predictive Coding (SDPC) is an efficient measurement coding approach that achieves SQ and DPCM coding with high recovery probability and low bit rate. In this work, a random Gaussian is used as the measurement matrix to sample natural images. However, the Gaussian measurement matrix is applied, which results in unstable utilization of the bit rate. Importantly, this kind of measurement matrix could not be used for BCI, which only supports binary.

In Non-Patent Document 7, a partial modification of the measurement matrix to generate intra prediction candidates is proposed by combining the concept of intra prediction borrowed from H.264/AVC with measurement coding (Modified SRM). It is a four-way intra prediction approach using a partially modified random binary measurement matrix, and the work mainly focuses on transmitting the coded measurements to the endpoint. A specific sampling pattern is embedded in the measurement matrix to obtain boundary information while sensing. In Non-Patent Document 7, the obtained information is used to

を乗算し、予測テンプレートを生成する。ＳＱ符号化、ＤＰＣＭ符号化、ＳＤＰＣ符号化と比較して軽量なアプローチである。しかし、ＤＰＣＭ符号化やＳＤＰＣ符号化よりもビットレートを低くすることができなかった。この提案は、最先端の研究に比べて予測候補生成の複雑さを低減できるが、測定行列の修正により視覚的品質の大幅な劣化を引き起こし、ＲＩＰ条件及びその逆変換を満たすことができなかった。

to generate a prediction template. This is a lightweight approach compared to SQ coding, DPCM coding, and SDPC coding. However, it is not possible to reduce the bit rate below DPCM coding and SDPC coding. Although this proposal can reduce the complexity of prediction candidate generation compared to state-of-the-art research, it causes a significant degradation of visual quality due to the modification of the measurement matrix, and it cannot meet the RIP condition and its inverse transformation.

A. Zymnis, S. Boyd and E. Candes, "Compressed Sensing With Quantized Measurements," in IEEE Signal Processing Letters, vol. 17, no. 2, pp. 149-152, Feb. 2010, doi: 10.1109/LSP. 2009.2035667 W. Dai and O. Milenkovic. Information Theoretical and Algorithmic Approaches to Quantized Compressive Sensing. IEEE Transactions on Communications, 59(7):1857-1866, 2011 S. Mun and J. E. Fowler, "DPCM for quantized block-based compressed sensing of images," 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), Bucharest, 2012, pp. 1424-1428 Yifu Zhang, Shunliang Mei, Quqing Chen and Zhibo Chen, "A novel image/video coding method based on Compressed Sensing theory," 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV, 2008, pp. 1361- 1364, doi: 10.1109/ICASSP.2008.4517871 C. Zhao, H. Bai and Y. Zhao, "Optimized Measurements Coding for Compressive Sensing Reconstruction Network," 2018 14th IEEE International Conference on Signal Processing (ICSP), Beijing, China, 2018, pp. 596-599, doi: 10.1109 /ICSP.2018.8652427 J. Zhang, D. Zhao and F. Jiang, "Spatially directional predictive coding for block-based compressive sensing of natural images," 2013 IEEE International Conference on Image Processing, Melbourne, VIC, 2013, pp. 1021-1025, doi: 10.1109/ICIP.2013.6738211 J. Zhou, D. Zhou, L. Guo, Y. Takeshi and S. Goto, "Measurement-domain intra prediction framework for compressively sensed images," 2017 IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, MD, 2017 , pp. 1-4, doi: 10.1109/ISCAS.2017.8050262

The present invention aims to provide an imaging device and a measurement encoding device that can efficiently compress images while suppressing degradation of image quality when using compressed sensing.

The imaging device according to the first aspect includes an image sensor unit that outputs an image in units of an image block, a compressed sensing unit that performs a compressed sensing process by applying a measurement matrix with m rows and n columns (m<n) to pixel values of n pixels that constitute the image block to output m measured values, and an analog/digital conversion unit that outputs measurement data by performing an analog/digital conversion process on the m measured values. The compressed sensing unit performs the compressed sensing process using a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequency ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is arranged in sequential order as the measurement matrix. Each row of the CoWH matrix has a different projection pattern when projected onto the image block. The CoWH matrix is configured such that the number of consecutive elements with the same value in the projection pattern decreases from the upper row to the lower row in the CoWH matrix.

The measurement coding device according to the second aspect includes the imaging device according to the first aspect, an intra prediction unit that outputs predicted measurement data obtained by predicting the measurement data of the image block using one of a plurality of intra prediction modes including a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the image block, and a quantization unit that performs a quantization process on residual data representing the difference between the measurement data of the image block and the predicted measurement data output by the intra prediction unit.

The present invention provides an imaging device and a measurement encoding device that can efficiently compress images while suppressing degradation of image quality when using compressed sensing.

FIG. 1 is a diagram illustrating a configuration of a measurement transmission system according to an embodiment. FIG. 1 is a diagram illustrating a configuration of a measurement encoding device according to an embodiment. FIG. 4 is a diagram illustrating an operation of a compressed sensing unit according to the embodiment. FIG. 2 is a diagram illustrating vertical prediction modes and horizontal prediction modes according to an embodiment. FIG. 1 is a diagram illustrating an example of the configuration of an imaging apparatus according to an embodiment. FIG. 13 is a diagram illustrating a projection pattern of a measurement matrix according to an embodiment. FIG. 13 is a diagram illustrating a CoWH matrix generation algorithm according to an embodiment. FIG. 11 is a diagram showing a comparison of restoration results using _l1 minimization and the inverse fast Walsh-Hadamard transform (IFWHT) when a NoH matrix, a SoWH matrix, and a CoWH matrix are used, respectively. FIG. 2 is a diagram illustrating an example of the configuration of a portion of an encoding device according to an embodiment. FIG. 2 is a diagram illustrating a configuration of a measurement decoding device according to an embodiment. FIG. 11 is a diagram for explaining the operation and effect of the embodiment.

The measurement transmission system according to the embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

(1) Configuration of the measurement transmission system First, the configuration of the measurement transmission system according to this embodiment will be described. Fig. 1 is a diagram showing the configuration of a measurement transmission system 1 according to this embodiment. As shown in Fig. 1, the measurement transmission system 1 has a measurement encoding device 100 and a measurement decoding device 200.

The measurement coding device 100 generates image data by image sensing and compresses the image data by compressed sensing. The measurement coding device 100 then further compresses the measurement data compressed by compressed sensing by compression coding, and transmits the compressed coded data (bit stream) via a transmission path. For example, the transmission path is a wireless transmission path. The transmission path may also include a wired transmission path.

The measurement decoding device 200 receives the compressed encoded data from the measurement encoding device 100 and decodes the measurement data from the received encoded data. The measurement decoding device 200 then uses a restoration algorithm to restore the original image data from the measurement data and outputs the image data to, for example, a display.

The measurement transmission system 1 may constitute a remote monitoring system. For example, the measurement encoding device 100 may be a battery-powered sensing device, and the measurement decoding device 200 may be provided in a server. The server may collect image data from the measurement encoding device 100 and perform remote monitoring of the monitoring target. Such sensing devices are required to have low power consumption in order to extend their operating time.

Note that although only one measurement encoding device 100 is shown in FIG. 1, the measurement decoding device 200 may receive encoded data from multiple measurement encoding devices 100.

In addition, the transmission of encoded data from the measurement encoding device 100 to the measurement decoding device 200 is not limited to being performed via a transmission path, and the encoded data may be transmitted indirectly from the measurement encoding device 100 to the measurement decoding device 200 via a storage medium.

(2) Measurement Encoding Apparatus Next, the measurement encoding apparatus 100 according to this embodiment will be described.

(2.1) Overall Configuration of Measurement Encoding Apparatus FIG. 2 is a diagram showing the configuration of a measurement encoding apparatus 100 according to this embodiment.

As shown in FIG. 2, the measurement coding device 100 has an imaging device 110, a coding device 120, and a transmission unit 130. The imaging device 110, the coding device 120, and the transmission unit 130 may be configured separately or integrally. The measurement coding device 100 may have a battery (not shown).

The imaging device 110 has an image sensor unit 111, a compressed sensing unit 112, and an analog/digital (A/D) conversion unit 113.

The image sensor unit 111 outputs images in units of image blocks. The image sensor unit 111 includes a CMOS image sensor (CIS). However, the image sensor unit 111 may also include a CCD image sensor.

The compressed sensing unit 112 outputs m measurement values by applying a measurement matrix with m rows and n columns (m<n) to the pixel values of n pixels that make up the image block output by the image sensor unit 111. This type of processing corresponds to a conversion process from the pixel domain to the measurement domain.

Figure 3 is a diagram showing the operation of the compressed sensing unit 112 according to this embodiment. As shown in Figure 3, one image is divided into multiple image blocks, and the image blocks are acquired in raster scan order. Each image block consists of pixel values for n pixels (n pixels). The compressed sensing unit 112 acquires pixel values from each image block in raster scan order.

3, pixel value sequence _X1 corresponding to image block _B1 , pixel value sequence _X2 corresponding to image block _B2 , pixel value sequence _X3 corresponding to image block _B3 , and pixel value sequence _Xi corresponding to image block _Bi are illustrated. The pixel value sequence is made up of n pixel values ( _x1 to _xn ).

The compressed sensing unit 112 multiplies each pixel value sequence X by a measurement matrix with m rows and n columns, and outputs the result. As a result, each pixel value sequence X is converted into a measurement value sequence Y consisting of m measurement values ( _y1 to _ym ). Here, since m is smaller than n, the amount of data is compressed.

Returning to FIG. 2, the A/D conversion unit 113 performs A/D conversion processing on the m measurement values Y output by the compressed sensing unit 112, thereby outputting measurement data Y, which is a digital signal. Because the number of measurement values is reduced (compressed) by compressed sensing, the power consumption of the A/D conversion unit 113 can be kept low.

The measurement data Y of each image block output by the A/D conversion unit 113 is further compressed by the encoding device 120 through compression encoding using intra prediction. The encoding device 120 has an intra prediction unit 121, a residual data generation unit 122, a quantization unit 123, a Huffman encoding unit 124, a Huffman decoding unit 125, an inverse quantization unit 126, and a synthesis unit 127.

The intra prediction unit 121 outputs predicted measurement data _Yp obtained by predicting measurement data Y of the current image block using any one of a plurality of intra prediction modes. The plurality of intra prediction modes include a vertical prediction mode using an upper adjacent block adjacent to the upper side of the current image block, and a horizontal prediction mode using a left adjacent block adjacent to the left side of the current image block, as shown in Fig. 4. The plurality of prediction modes further includes an average prediction mode that calculates an average of the predicted measurement data of the vertical prediction mode and the predicted measurement data of the horizontal prediction mode as predicted measurement data.

The intra prediction unit 121 has a reference data generation unit 1211, a prediction data generation unit 1212, and a selection unit 1213.

The reference data generation unit 1211 generates reference data for the upper adjacent block and reference data for the left adjacent block to obtain predicted measurement data.

The predicted data generating unit 1212 uses the reference data generated by the reference data generating unit 1211 to generate predicted measurement data _Yv in the vertical prediction mode, predicted measurement data _Yh in the horizontal prediction mode, and predicted measurement data _Yavg in the average prediction mode.

The selection unit 1213 calculates the similarity between the measurement data Y of the current image block and each of the predicted measurement data _Yv of the vertical prediction mode, the predicted measurement data _Yh of the horizontal prediction mode, and the predicted measurement data _Yavg of the average prediction mode, and selects the predicted measurement data with the highest similarity, and outputs the selected predicted measurement data _Yp to the residual data generation unit 122. As an index value representing the similarity, for example, the sum of absolute differences (SAD) can be used.

However, if the similarity of any of the predicted measurement data with the measurement data Y of the current image block is low, the selection unit 1213 may output the measurement data Y of the current image block as is to the quantization unit 123. For example, if the SAD of each of the predicted measurement data _Yv of the vertical prediction mode, the predicted measurement data _Yh of the horizontal prediction mode, and the predicted measurement data _Yavg of the average prediction mode is greater than a threshold, the selection unit 1213 outputs the measurement data Y of the current image block as is to the quantization unit 123.

The residual data generation unit 122 generates residual data _Yr representing the difference between the measurement data Y of the current image block and the predicted measurement data _Yp output by the intra prediction unit 121 , and outputs the generated residual data _Yr to the quantization unit 123 .

The quantization unit 123 performs a quantization process on the residual data Y _r (or the measurement data Y of the current image block), and outputs the quantized data Y _q to the Huffman coding unit 124 .

The Huffman coding unit 124 generates coded data by performing a Huffman coding process on the quantized data _Yq , and outputs the generated coded data to the transmission unit 130 and the Huffman decoding unit 125. Note that the coding is not limited to Huffman coding, and entropy coding other than Huffman coding may be used.

The transmitting unit 130 transmits the coded data output by the Huffman coding unit 124 to the measurement decoding device 200 via a transmission path. The transmitting unit 130 may include a wireless transmitter that transmits a wireless signal including the coded data. The transmitting unit 130 also transmits the coded data including mode information indicating the prediction mode selected by the selecting unit 1213.

The Huffman decoding unit 125 performs Huffman decoding processing on the coded data output by the Huffman coding unit 124 to restore the quantized data _Yq , and outputs the restored quantized data _Yq to the inverse quantization unit 126 .

The inverse quantization unit 126 performs an inverse quantization process on the restored quantized data _Yq to restore the residual data _Yr , and outputs the restored residual data _Ydeq to the synthesis unit 127.

The synthesis unit 127 reconstructs the measurement data Y by synthesizing the reconstructed residual data Y _deq and the predicted measurement data Y _p , and outputs the reconstructed measurement data to the reference data generation unit 1211 of the intra prediction unit 121 .

In the exemplary configuration of the encoding device 120 shown in Fig. 2, the encoded data output by the Huffman encoding unit 124 is input to the Huffman decoding unit 125. However, the Huffman decoding unit 125 may be unnecessary. In this case, instead of inputting the encoded data output by the Huffman encoding unit 124 to the Huffman decoding unit 125, the quantized data _Yq output by the quantization unit 123 is input to the inverse quantization unit 126.

(2.2) Imaging Device Next, the imaging device 110 according to this embodiment will be described.

Figure 5 is a diagram showing an example of the configuration of an imaging device 110 according to this embodiment. The imaging device 110 according to this embodiment is a CMOS image sensor that uses compressed sensing, i.e., a CS-CIS.

As shown in FIG. 5, the imaging device 110 has a pixel array 111a, a row selection unit 111b, a column selection unit 111c, a matrix operation trigger unit 112a, a matrix operation unit 112b, and an A/D conversion unit 113.

The pixel array 111a, row selection unit 111b, and column selection unit 111c constitute the image sensor unit 111 shown in FIG. 2. The pixel array 111a has a plurality of pixels arranged in an array. Each pixel receives light and outputs a pixel value, which is an electrical signal. The pixel array 111a is divided into blocks each consisting of n pixels. The row selection unit 111b and the column selection unit 111c select pixels from which pixel values are to be read, and read the pixel values from the selected pixels.

The matrix operation trigger unit 112a and the matrix operation unit 112b constitute the compressed sensing unit 112 shown in FIG. 2. The matrix operation trigger unit 112a triggers a matrix operation on a pixel value sequence X consisting of n pixels. The matrix operation unit 112b performs a matrix operation on the pixel value sequence X using a measurement matrix, and outputs a measurement value sequence Y consisting of m measurement values to the A/D conversion unit 113. The A/D conversion unit 113 converts the measurement value sequence Y into a digital signal, and outputs measurement data Y.

Figure 6 shows the projection pattern of the measurement matrix according to this embodiment. Here, the size of the image block is 8 pixels x 8 pixels, i.e., the value of n is 64. Also, the value of m is 8 (there are eight patterns in the figure, from low frequency to high frequency), i.e., eight patterns are shown for use when compressing to 12.5% (m/n=8/64=12.5%), converting from n=64 to eight measurement values (m=8). In Figure 6, each open area represents an element value of "1" in the measurement matrix, and each filled area represents an element value of "0" in the measurement matrix.

As shown in FIG. 6, the compressed sensing unit 112 performs compressed sensing processing using a CoWH (Continuously ordered Walsh-Hadamard) matrix as a measurement matrix based on a SoWH (Sequency ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is arranged in sequential order. Note that sequence refers to the frequency at which values change in one row of the matrix.

Each row of the CoWH matrix has a different projection pattern when projected onto an image block. The CoWH matrix is constructed so that the number of consecutive elements with the same value in the projection pattern decreases from the upper rows to the lower rows in the CoWH matrix.

For example, the top row (row #1) of the CoWH matrix has 64 consecutive pixels of "1" elements in the projection pattern. The measurement calculated using row #1 is called _y1 .

In the second row (row #2) and the third row (row #3) from the top of the CoWH matrix, the element "1" is consecutive for 32 pixels and the element "0" is consecutive for 32 pixels in the projection pattern. The measurement calculated using row #2 is called _y2 , and the measurement calculated using row #3 is called _y3 .

In the fourth row from the top of the CoWH matrix (row #4), the element "1" occurs consecutively for 16 pixels, and the element "0" occurs consecutively for 32 pixels in the projection pattern.

In the fifth row from the top (row #5) of the CoWH matrix, the element "1" occurs consecutively for 16 pixels in the projection pattern, and the element "0" occurs consecutively for 16 pixels.

In the sixth row from the top of the CoWH matrix (row #6), the element "1" occurs consecutively for 16 pixels, and the element "0" occurs consecutively for 32 pixels in the projection pattern.

In the seventh row from the top of the CoWH matrix (row #7), the element "1" occurs consecutively for 16 pixels, and the element "0" occurs consecutively for 16 pixels in the projection pattern.

The eighth row from the top (row #8) of the CoWH matrix has an area in the projection pattern where the element "1" is continuous for 8 pixels and an area where the element "1" is continuous for 16 pixels. It also has an area where the element "0" is continuous for 8 pixels and an area where the element "0" is continuous for 16 pixels.

The CoWH matrix has rows constituting a first projection pattern that obtains all pixel values of an image block, rows constituting a second projection pattern that obtains pixel values of the upper half of the image block, and rows constituting a third projection pattern that obtains pixel values of the left half of the image block. This enables smooth intra-prediction of the vertical prediction mode and the horizontal prediction mode in the measurement domain. Specifically, the topmost row (row #1) in the CoWH matrix constitutes the first projection pattern, and the second and third rows from the top (rows #2 and #3) in the CoWH matrix constitute the second and third projection patterns.

In addition, by adopting CS theory to CIS, the measurement matrix

は、スクランブルマスクの確率が高いバイナリ｛０，１｝又はワーストケースでも相補｛－１，１｝でなければならないという実装上の制約がある。

There is an implementation constraint that the scrambling mask must be highly probable binary {0,1} or in the worst case complementary {−1,1}.

The continuous order Walsh-Hadamard (CoWH) of this embodiment is an optimization of the binary sequential order Walsh-Hadamard (SoWH) that can be derived by the natural order Hadamard (NoH). Note that the columns of the NoH matrix and the SoWH matrix are orthogonal. These matrices are n-th order matrices.

を与えることによって得ることができる。ＮｏＨ行列は次のように表現できる。

The NoH matrix can be expressed as follows:

ここで、Ｉ_ｎ×ｎは単位行列である。

Here, I _n×n is the identity matrix.

は、

の転置であり、その次数はＸの次元に等しくなければならない。これは、図６（ａ）に示すような投影パターンを生じる。これは繰り返し適用することができ、次のように、ｎ次の

teeth,

, the order of which must be equal to the dimension of X. This results in a projection pattern as shown in FIG. 6(a). This can be applied repeatedly to produce a projection pattern of order n as follows:

に対してSylvester's constructionを適用することにより次の行列のシーケンスにつながり、これはウォルシュ・アダマール行列と呼ばれ、次のように表現できる。

Applying Sylvester's construction to leads to the following sequence of matrices, known as Walsh-Hadamard matrices, which can be expressed as:

但し、

however,

であり、

and

はクロネッカー積である。その結果、図６（ｂ）に示すような投影パターンを生じる。

is the Kronecker product. This results in the projection pattern shown in FIG.

The sequential rows of the Walsh-Hadamard matrix are derived from the NoH matrix by applying bit reversal and gray-code permutation. This results in

で表現されるＳｏＷＨ測定行列が得られる。

The SoWH measurement matrix expressed as:

In this embodiment, we reorganize the projection pattern of the SoWH measurement matrix by grouping low-frequency coefficients at the top of the measurement matrix. In the CoWH arrangement, the projection pattern is organized to increase the number of connected elements, as shown in Figure 6(c). This allows for improved measurement quality while sensing. Related components correspond to groups of contiguous pixels with corresponding values, in terms of richer prior knowledge that influences the restoration algorithm and inverse transformation.

Also, the good continuity pattern of the CoWH matrix shown in Figure 6(c) allows extending the lower limit of SR compared to NoH and SoWH matrices, which are usually limited by 50% for good visual quality.

の次数とし、探索基準位置（search criteria position）ｄはｎ×ｎに等しい。次に、各ｄについて、ｒ_ｉ，ｊの除算後の剰余を求める。次数が奇数か偶数かを判定するために、

を計算する。各次数についてｒ_ｉ，ｊを求めた後、

を再配置するために、Ａを空の行列

とすると、

である。 The CoWH matrix is generated by the algorithm shown in FIG.

Let the degree be r i,j, and the search criteria position d is equal to n×n. Then, for each d, find the remainder after division of r _i,j . To determine whether the degree is odd or even,

After finding r _i,j for each degree, calculate

To rearrange A, let A be an empty matrix.

Then,

It is.

A (the CoWH matrix) satisfies the RIP with probability at least .

where β is a positive constant.

For hardware implementation, this expanded matrix A is binarized from its complementary matrix to binary. The sampling mask (projection pattern) of the CoWH matrix is arranged gradually from low-frequency coefficients to high-frequency coefficients compared to the NoH and SoWH matrices.

FIG. 8 is a diagram showing a comparison of the restoration results using l ₁ minimization and inverse fast Walsh-Hadamard transform (IFWHT) when using NoH matrix, SoWH matrix, and CoWH matrix, respectively. As shown in FIG. 8, the CoWH matrix outperforms NoH matrix and SoWH matrix in terms of visual quality at the same SR. Note that FIG. 8(a) shows the constructed measurement matrix, FIG. 8(b) shows the restored image, and FIG. 8(c) shows the result of 400% zoom. Note that n=65536 (256 pixels×256 pixels), m=8192 (compressed to 12.5%), and B (i.e., the size of the block used in BCI)=8.

(2.3) Encoding Device Next, a description will be given of the encoding device 120 according to this embodiment. Fig. 9 is a diagram showing an example of the configuration of a portion of the encoding device 120 according to this embodiment.

As shown in FIG. 9, the reference data generation unit 1211, the prediction data generation unit 1212, and the selection unit 1213 constitute the intra prediction unit 121 shown in FIG. 2. An input buffer 141 is provided on the input side of the intra prediction unit 121. An output buffer 142 is provided on the output side of the quantization unit 123.

The reference data generation unit 1211 includes a package splitter 1211a that separates the measurement values _y1 , _y2 , and _y3 from the restored measurement data Y output by the synthesis unit 127, a subtractor 1211b that subtracts the measurement value _y2 from the measurement value _y1 , a subtractor 1211c that subtracts the measurement value _y3 from the measurement value _y1 , a register 1211d that stores the output of the subtractor 1211b, and a register 1211e that stores the output of the subtractor 1211c.

Here, the measurement value _y1 is the measurement value obtained by row #1 constituting a projection pattern that obtains all pixel values of an image block in the CoWH matrix, and corresponds to the sum of all pixel values of the image block.

The measurement _y2 is the measurement obtained by row #2 constituting the projection pattern that obtains pixel values in the left half of the image block in the CoWH matrix, and corresponds to the sum of the pixel values in the left half of the image block.

Measurement value _y3 is the measurement obtained by row #3 constituting a projection pattern that captures pixel values in the upper half of the image block in the CoWH matrix, and corresponds to the sum of the pixel values in the upper half of the image block.

Therefore, the output of subtractor 1211b, which subtracts measurement value _y2 from measurement value _y1 , corresponds to the sum of pixel values in the right half of the image block, and is held in register 1211d. This sum of pixel values in the right half of the image block is used as reference data in the horizontal prediction mode shown in Figure 4. Register 1211d only needs to hold reference data corresponding to the left adjacent block of the current image block, and its size is set to 16 bits here.

On the other hand, the output of the subtractor 1211c, which subtracts the measured value _y3 from the measured value _y1 , corresponds to the sum of the pixel values of the lower half of the image block, which is stored in the register 1211e. This sum of the pixel values of the lower half of the image block is used as reference data in the vertical prediction mode shown in Figure 4. The register 1211e stores reference data for each image block that belongs to the row one row above the row to which the current image block belongs. If the width of the image is W and the size of the image block is B, the register 1211e stores reference data for W/B blocks.

The prediction data generator 1212 calculates the pixel value (average value) per pixel from the sum of the pixel values of the lower half of the image block output by the register 1211e, and then multiplies this average value by the transpose of the CoWH matrix.

を乗算することにより、垂直予測モードの予測測定データＹ_ｖを出力する。このようにして、イントラ予測部１２１は、垂直予測モードについて、上側隣接ブロックについて得られた測定値ｙ_１から、当該上側隣接ブロックについて得られた測定値ｙ_３を減算することにより、画素ドメインにおける当該上側隣接ブロックの下半分の合計の画素値を得る処理と、下半分の合計の画素値を有効画素数で除算することにより、当該上側隣接ブロックの下半分の画素値の平均を得る処理と、この平均からＣｏＷＨ行列に基づいて垂直予測モードの予測測定データＹ_ｖを得る処理とを実行する。

In this way, for the vertical prediction mode, the intra prediction unit 121 executes a process of subtracting _the measurement value _y3 obtained for the upper adjacent block from the measurement value _y1 obtained for the upper adjacent block to obtain a total pixel value of the lower half of the upper adjacent block in the pixel domain, a process of dividing the total pixel value of the lower half by the number of valid pixels to obtain an average of the pixel values of the lower half of the upper adjacent block, and a process of obtaining prediction measurement data _Yv for the vertical prediction mode from this average based on the CoWH matrix.

The prediction data generator 1212 also calculates the pixel value (average value) per pixel from the sum of the pixel values in the right half of the image block output by the register 1211d, and applies the transpose of the CoWH matrix to this average value.

を乗算することにより、水平予測モードの予測測定データＹ_ｈを出力する。このようにして、イントラ予測部１２１は、水平予測モードについて、左側隣接ブロックについて得られた測定値ｙ_１から、左側隣接ブロックについて得られた測定値ｙ_２を減算することにより、画素ドメインにおける左側隣接ブロックの右半分の合計の画素値を得る処理と、右半分の合計の画素値を有効画素数で除算することにより、左側隣接ブロックの右半分の画素値の平均を得る処理と、この平均からＣｏＷＨ行列に基づいて水平予測モードの予測測定データＹ_ｈを得る処理とを実行する。

In this way, for the horizontal prediction mode, the intra prediction unit 121 executes a process of obtaining a total pixel value of the right half of the left adjacent block in the pixel domain by subtracting the measurement value _y2 obtained for the left adjacent block from the measurement value _y1 obtained for the left adjacent block, a process of obtaining an average of the pixel values of the _right half of the left adjacent block by dividing the total pixel value of the right half by the number of valid pixels, and a process of obtaining predicted measurement data _Yh for the horizontal prediction mode from this average based on the CoWH matrix.

Furthermore, the predicted data generating unit 1212 calculates the average of the predicted measurement data _Yv and the predicted measurement data _Yh as predicted measurement data _Yavg in the average prediction mode.

This type of intra prediction method can be expressed as follows:

Vertical prediction mode:

水平予測モード:

Horizontal prediction mode:

平均予測モード:

Average prediction mode:

In order to select the final intra-prediction mode prediction measurement data Y _p , the selection unit 1213 uses the SAD as follows to find the minimum difference between the prediction candidate and Y:

を検索し、次のように予測測定データＹ_ｐを選択する。

and select the predicted measurement data Y _p as follows:

However, if no predicted candidate is selected, the selection unit 1213 assigns zero to the predicted measurement data Y _p .

Then, the residual data generating unit 122 calculates the residual data _Yr as follows.

The quantization unit 123 quantizes the residual data _Yr by scalar quantization as follows, and outputs quantized data _Yq .

where _Bp is a bit depth parameter, for example 4 bits.

The inverse quantization unit 126 performs the following calculations:

Of these processes, quantization is lossy, while intra prediction and entropy coding are lossless.

(3) Measurement Decoding Apparatus Next, a description will be given of the measurement decoding apparatus 200 according to this embodiment. Fig. 10 is a diagram showing the configuration of the measurement decoding apparatus 200 according to this embodiment.

As shown in FIG. 10, the measurement decoding device 200 has a receiving unit 210, a decoding device 220, and a restoration unit 230.

The receiving unit 210 receives encoded data from the measurement encoding device 100 via a transmission path, and outputs the received encoded data to the decoding device 220. The encoded data includes mode information indicating the prediction mode selected by the measurement encoding device 100. The receiving unit 210 also outputs the mode information to the decoding device 220.

The decoding device 220 has an intra prediction unit 221, a Huffman decoding unit 222, an inverse quantization unit 223, and a synthesis unit 224. The intra prediction unit 221 has a reference data generation unit 2211 and a prediction data generation unit 2212.

The reference data generating unit 2211 is configured similarly to the reference data generating unit 1211 of the measurement coding device 100, and the predicted data generating unit 2212 is configured similarly to the reference data generating unit 1211 of the measurement coding device 100. However, the predicted data generating unit 2212 generates predicted measurement data _Yp only for the prediction mode indicated by the mode information transmitted from the measurement coding device 100, and outputs the generated predicted measurement data _Yp to the synthesis unit 224.

The Huffman decoding unit 222 performs Huffman decoding processing on the coded data output by the receiving unit 210 to restore the quantized data _Yq , and outputs the restored quantized data _Yq to the inverse quantization unit 223 .

The inverse quantization unit 223 performs an inverse quantization process on the restored quantized data _Yq to restore the residual data _Yr , and outputs the restored residual data _Ydeq to the synthesis unit 224.

The combining section 224 combines the reconstructed residual data Y _deq and the predicted measurement data Y _p to reconstruct the measurement data Y, and outputs the reconstructed measurement data to the reconstruction section 230 .

The restoration unit 230 restores pixel values X (image block) of the original n pixels from the measurement data Y using a restoration algorithm and outputs the restored pixel values. The CS theory asserts that m-dimensional measurement data is sufficient to restore the measurement of the original signal through an efficient restoration algorithm. For example, the restoration unit 230 restores pixel values X (image block) of the original n pixels using the l ₁ algorithm and IFWHT.

(4) Actions and Effects According to the present embodiment, a lightweight coding framework for a new high-resolution BCI can be realized, including an optimized binary structural measurement matrix (SRM) based on the Walsh-Hadamard transform (WHT). In addition, spatial redundancy can be reduced by performing intra prediction in the measurement domain using the features of the CoWH matrix. Therefore, images can be efficiently compressed for storage and transmission.

In the following, we present simulation results for efficiently storing and transmitting the encoded measurements in memory space. The quantitative metrics are peak-to-noise-ratio (PSNR), structural similarity index measure (SSIM), and bits-per-pixel (bpp).

We use a 4K dataset in the color space Beauty, ReadySetGo, Bosphorus, and HoneyBee defined in "A. Mercat, M. Viitanen, and J. Vanne, "UVG dataset: 50/120fps 4K sequences for video codec analysis and development," in Proc. ACM Multimedia Syst. Conf., Istanbul, Turkey, June 2020". Here, n = 24883200 (3840 x 2160 x 3), SR ∈ {4/4, 1/8, 1/16, 1/32, 1/64}, and B = 16.

Table 1 below shows a comparison of minimum process BCI measurements for PSNR (dB), SSIM, and bpp without coding. The comparison of minimum process BCI measurements in Table 1 shows that the CoWH matrix provides an average degradation of PSNR of 5-8% and SSIM of 0.2-0.5 for each reduction in SR.

Furthermore, when SR is equal to 1/16, the CoWH matrix provides a better lower bound with fewer undersampling artifacts compared to 1/32 and 1/64. In the following, we fix SR to 1/16 to evaluate the coding performance and compare it with state-of-the-art work.

Furthermore, we prove the hypothesis that by increasing the relationship between each projected pattern, higher visual quality can be obtained, although this results in a loss of edge detail.

Furthermore, intra prediction-based measurement coding (IPMC) by the measurement coding device 100 according to the present embodiment is compared with state-of-the-art work on spatially coding minimal process BCI measurements. For these comparisons, state-of-the-art codes were selected. However, to make the comparison of coding performance fair, the quantization method and parameters were changed to be the same. SQ was used and used as a baseline.

を用いて測定値をサンプリングした。

The measurements were sampled using

As shown in the simulation results in Table 2, IPMC achieved an overall PSNR of 19.96% dB, SSIM of 20.25%, and SQ of 10.34% bpp.

Compared to DPCM, it achieved PSNR of 14.50%, SSIM of 9.19%, and bpp of 8.77%, outperforming DPCM in all respects.

Compared to similar coding such as SDPC, the PSNR was 14.37%, SSIM was 9.19%, and bpp was 7.19%.

Moreover, compared to Modified SRM, which has more advanced sampling and coding, the present embodiment outperforms it by 11.73% in PSNR, 13.09% in SSIM, and 6.74% in bpp. Therefore, Modified SRM is not very efficient because it modifies binary SRM to obtain information from neighboring blocks and finally generates intra prediction candidates as H.264.

Thus, the IPMC of this embodiment exhibits an overall superior coding performance compared to state-of-the-art research, with PSNR of 15.06%, SSIM of 18.74%, and bpp of 3.7%.

Figure 11 shows an example of 400% zoomed detail using the HoneyBee dataset. In Figure 11, (a) is SQ (baseline), (b) is DPCM, (c) is SDCP, (d) is Modified SRM, and (e) is the present embodiment. As shown in Figure 11, the cropped results using the HoneyBee dataset show that when SR is 1/16, the present embodiment provides richer visual quality compared to state-of-the-art research.

Thus, according to the overall performance in Table 2, this embodiment provides all-around trade-off performance relative to state-of-the-art research.

Furthermore, we implemented the IPMC algorithm in an FPGA, where the resolution is 3820x2140, the total logic utilization is 3,421, and the maximum operating frequency is 102.65MHz. This architecture consumed 28.32mW of power. This resulted in the possibility of real-time encoding 100 times faster than the software implementation, as summarized in Table 3.

(5) Other embodiments A program may be provided that causes a computer to execute each process according to the above-described embodiments. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

The circuits that execute each process performed by the measurement coding device 100 (coding device 120) may be integrated, and the measurement coding device 100 (coding device 120) may be configured as a semiconductor integrated circuit (chip set, SoC).

The above describes the embodiment in detail with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the invention.

1: Measurement transmission system 100: Measurement encoding device 110: Imaging device 111: Image sensor unit 111a: Pixel array 111b: Row selection unit 111c: Column selection unit 112: Compressive sensing unit 112a: Matrix operation trigger unit 112b: Matrix operation unit 113: A/D conversion unit 120: Encoding device 121: Intra prediction unit 122: Residual data generation unit 123: Quantization unit 124: Huffman encoding unit 125: Huffman decoding unit 126: Inverse quantization unit 127: Synthesis unit 130: Transmission unit 141: Input buffer 142: Output buffer 200: Measurement decoding device 210: Reception unit 220: Decoding device 221: Intra prediction unit 222: Huffman decoding unit 223: Inverse quantization unit 224: Synthesis unit 230 : Restoration unit 1211 : Reference data generation unit 1211a : Package splitter 1211b : Subtractor 1211c : Subtractor 1211d : Register 1211e : Register 1212 : Prediction data generation unit 1213 : Selection unit 2211 : Reference data generation unit 2212 : Prediction data generation unit

Claims (8)

an image sensor unit that outputs images in units of image blocks;
a compressed sensing unit that performs a compressed sensing process by applying a measurement matrix with m rows and n columns (m<n) to pixel values of n pixels that constitute the image block to output m measurement values;
an analog/digital conversion unit that outputs measurement data by performing analog/digital conversion processing on the m measurement values,
The compressed sensing unit performs the compressed sensing process by using, as the measurement matrix, a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequency ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is arranged in sequential order;
Each row of the CoWH matrix has a different projection pattern when projected onto the image block,
An imaging apparatus, comprising: a CoWH matrix configured such that the number of consecutive elements with the same value in the projection pattern decreases from upper rows to lower rows in the CoWH matrix. The CoWH matrix is
a row constituting a first projection pattern for acquiring all pixel values of the image block;
rows constituting a second projection pattern for acquiring pixel values of an upper half of the image block;
and a third projection pattern of rows that capture pixel values in a left half of the image block. the topmost row in the CoWH matrix constitutes the first projection pattern;
The imaging apparatus of claim 2 , wherein the second and third rows from the top of the CoWH matrix constitute the second and third projection patterns. An imaging device according to claim 2 or 3;
an intra prediction unit that outputs predicted measurement data obtained by predicting the measurement data of the image block using one of a plurality of intra prediction modes including a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the image block;
a quantization unit that performs a quantization process on residual data representing a difference between the measurement data of the image block and the predicted measurement data output by the intra prediction unit. The intra prediction unit, with respect to the vertical prediction mode,
obtaining a total pixel value for a lower half of the upper adjacent block in the pixel domain by subtracting measurements obtained by applying the second projection pattern to the upper adjacent block from measurements obtained by applying the first projection pattern to the upper adjacent block;
obtaining an average of the pixel values of the lower half of the upper adjacent block by dividing the total pixel value of the lower half by the number of valid pixels;
and obtaining predicted measurement data for the vertical prediction mode from the average based on the CoWH matrix. The intra prediction unit, with respect to the horizontal prediction mode,
obtaining a sum of pixel values for a right half of the left adjacent block in the pixel domain by subtracting measurements obtained by applying the third projection pattern to the left adjacent block from measurements obtained by applying the first projection pattern to the left adjacent block;
obtaining an average of the pixel values of the right half of the left adjacent block by dividing the sum of the pixel values of the right half by the number of valid pixels;
6. The measurement coding device according to claim 4, further comprising a process of obtaining predicted measurement data for the horizontal prediction mode from the average based on the CoWH matrix. The plurality of intra prediction modes further include an average prediction mode that calculates an average of prediction measurement data of the vertical prediction mode and prediction measurement data of the horizontal prediction mode as prediction measurement data;
The measurement encoding device described in any one of claims 4 to 6, characterized in that the intra prediction unit calculates a similarity between the measurement data of the image block and each of the prediction measurement data of the vertical prediction mode, the prediction measurement data of the horizontal prediction mode, and the prediction measurement data of the average prediction mode, and selects the prediction measurement data with the highest similarity. an encoding unit that encodes the residual data quantized by the quantization unit;
A transmitter that transmits the residual data encoded by the encoding unit to a decoding side,
The measurement encoding device of claim 7 , wherein the transmission unit further transmits mode information indicating a prediction mode of the predicted measurement data selected by the intra prediction unit.

Images