JP4243034B2 - Encoder - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to an encoding device and a decoding device, and in particular, an encoding device for a stereoscopic television using an IP (integral photography) system that can be viewed from an arbitrary viewpoint.In placeRelated.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, an IP method using a lens group arranged on a plane or a spherical surface is used as one of imaging methods for a stereoscopic television that can be viewed from an arbitrary viewpoint.
[0003]
The output signal from a stereoscopic TV camera adopting this IP system is a collection of stereoscopic imaging signals obtained from a plurality of small imaging lenses. Therefore, the amount of information is much larger than that of a normal flat TV camera. It is.
FIG. 8 is an explanatory diagram showing the imaging principle of a stereoscopic TV camera adopting a conventional IP system. In the conventional stereoscopic television camera shown in FIG. 8, a photographic film 84 is arranged behind a lens group 82 composed of a plurality of lenses (convex lens 82i is one of them) arranged on the same plane. The subject 81 located at is photographed.
[0004]
With this configuration, a plurality of images (the element pixel 83i is one of them) of the subject 81 imaged by each of the convex lenses 82i are formed on the photographic film 84.
An image obtained by developing a plurality of images formed on the photographic film 84 is placed on the lens group 82 at the same position as when the image was taken, and when the image is viewed from the front of the lens group 82 in this state, The reproduced image can be visually recognized.
[0005]
9 and 10 are block diagrams showing the structure of a conventional practical IP 3D television camera.
The structure of the stereoscopic television camera shown in FIGS. 9 and 10 increases the resolution of the television camera having the imaging principle shown in FIG. 8 so that a moving image of a stereoscopic image can be taken.
First, as a configuration in which the imaging area can be increased and the number of pixels can be extremely increased, a plurality of imaging elements (the imaging element 9e is one of them) are installed on one wafer 9w as an alternative to the photographic film 84. In addition, an element lens 9g is disposed on the front surface of each of a plurality of image sensors including the image sensor 9e (the element lens 9g is one of them and corresponds to the image sensor 9e). Each element lens 9g is conveniently provided with optical shielding means 9b so that adjacent ones do not interfere with each other.
[0006]
In the television camera shown in FIG. 9, in order to simplify the drawing, there are four image sensors represented by the image sensor 9e and four element lenses represented by the element lens 9g in the vertical direction and five elements in the horizontal direction, respectively. Although a structure in which a total of 20 optical images indicated by reference numerals D11 to D45 are formed is shown on these image pickup devices (array of 5 columns × 4 rows). In the stereoscopic television camera, the number of image pickup elements represented by the image pickup element 9e and element lenses represented by the element lenses 9g each reach 1,000 to 2,000.
[0007]
The number of the image pickup element and the element lens is related to the resolution of the moving image of the three-dimensional image to be picked up. If the number of the image pickup element and the element lens is increased, the resolution of the moving image of the three-dimensional image to be picked up is also increased.
A conventional stereoscopic television camera has the above-described structure, but an actual image is output from the stereoscopic television camera via a transmission system and further displayed on a display device such as a television receiver. Or recorded on a recording device such as a video.
[0008]
Here, the image signals (element image signals) obtained by photographing each of the lens groups are sequentially arranged in the order in which the lens groups are arranged to form one continuous signal. In this case, these existing general transmission devices, display devices, and recording devices (all of which are compatible with signals handled by existing general transmission devices, display devices, and recording devices) It is possible to apply a signal compression / reproduction device (specifically, an encoding device and a decoding device) for a flat television system).
[0009]
[Problems to be solved by the invention]
However, in the conventional general transmission apparatus, display apparatus, and signal compression / reproduction apparatus (encoding apparatus and decoding apparatus) for the conventional flat TV system, the television camera adopting the IP system is used. There is a limit to the compression ratio that can be obtained because it does not use the unique properties.
Therefore, there has been a problem that it cannot be used as a signal compression apparatus for a stereoscopic television system that needs to compress an enormous amount of information much more than a flat-screen television system.
[0010]
  The present invention has been made in view of the above problems in the conventional encoding device and decoding device, and efficiently compresses (encodes) an output image signal from a television camera adopting the IP system.)Encoding equipment that canPlaceThe purpose is to provide.
[0011]
[Means for Solving the Problems]
  In the present invention, in order to solve the above-described problem, in the encoding device according to claim 1, an integral photography signal output from a television camera that captures an element image connected to each of a plurality of integrated imaging lenses is received. In an encoding apparatus that inputs and encodes, a code using the correlation between the image signals using the image signal of the current display frame and the image signal of the display frame temporally prior to the current display frame Using inter-frame encoding means for encoding, intra-frame encoding means for encoding using an image signal within a range closed to the current display frame, and a plurality of image signals corresponding to a plurality of element images Inter-element image encoding means for performing encoding using correlation between the image signals,According to the content of the input integral photography signal, any one of the inter-frame coding means, the intra-frame coding means and the inter-element image coding means is appropriately selected, and the selected means And image characteristic analysis means for controlling the encoding of the inputted integral photography signal to be executed. In the encoding device, the image characteristic analysis means verifies whether the characteristic of the input integral photography signal is a still image, a moving image, or a scene change command, According to the verification result, any one of the inter-frame coding unit, the intra-frame coding unit, and the inter-element image coding unit is appropriately selected, and the input is performed by the selected unit. If the input integral photography signal is a signal with a scene change command, the scene is changed and the current image is encoded. If there is no occlusion in the image, the elemental inter-image encoding means has the occlusion and corresponds to the current image. When there is no other element image, the intra-frame encoding means causes the occlusion, and when there is another element image corresponding to the current image, the inter-element image encoding means Control the input integral photography signal to be encodedThe configuration.
[0012]
  With this configuration, the encoding device uses the correlation between the image signals using the image signal of the current display frame and the image signal of the display frame temporally prior to the current display frame. Inter-frame encoding means for performing the encoding, intra-frame encoding means for encoding using an image signal within a range closed to the current display frame, and a plurality of image signals corresponding to a plurality of element images Inter-element image encoding means for performing encoding using correlation between the image signals usingCan be appropriately selected and executed according to the characteristics of the input integral photography signal by the image characteristic analysis means, and thus optimum encoding corresponding to the characteristics of the IP signal can be performed.
[0013]
  Also,In the encoding apparatus, the image characteristic analysis unit verifies whether the characteristic as the content of the input integral photography signal is a still image, a moving image, or a scene change command, and responds to it. In view of the characteristics of the IP signal, it is possible to perform optimum encoding using the characteristics obtained by appropriately entering the contents of the image for each IP signal actually input in units of frames. it can.
[0014]
  Furthermore, in the encoding apparatus, if there is no occlusion even if there is a scene change, it is possible to perform encoding using correlation between element images, and when there is a scene change and there is also occlusion, After searching for an element image corresponding to the current image, encoding can be performed using the correlation between the current image and the searched element image.
[0021]
BEST MODE FOR CARRYING OUT THE INVENTION
  Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(Reference exampleFIG. 1 shows the present invention.Reference exampleIt is a block diagram which shows the structure of this encoding apparatus.
[0022]
MarkThe encoding device includes an inter-frame encoding unit 11 (inter-frame encoding means) that performs inter-frame encoding on the input integral photography (hereinafter referred to as IP) signal P1, and a normal in the frame. An intra-frame encoding unit 12 (intra-frame encoding unit) that performs encoding of the above, an inter-element image encoding unit 13 (inter-element image encoding unit) that performs encoding between element images, and the most efficient encoding And an encoding selection unit 14 (encoding selection means) that selects a high encoding output and outputs an encoding output C1. FIG. 2 is an explanatory diagram showing a configuration from the shooting to display of the stereoscopic television system using the IP system (the formation of the IP signal).
[0023]
In the stereoscopic television system shown in FIG. 2, an image of the subject 61 through the convex lens 62 is formed in the vicinity of the compound eye lens 63 (using an optical fiber). The image is captured by the television camera 64 and converted into an image signal output, and then displayed on the liquid crystal panel 65. Since the compound eye lens 66 is also provided in the vicinity of the liquid crystal panel 65, the operator can use the image via the compound eye lens 66 to display a three-dimensional image (a three-dimensional image that can be seen around the screen without wearing special glasses). ).
[0024]
Reference numeral 67 in FIG. 2 shows an enlarged view of an image obtained on the compound eye lens 63 (66). As can be seen from the enlarged image 67, each of the element lenses included in the compound eye lens 63 (66) has one of the images of the subject 61 taken from various angles as an element image.
[0025]
  Hereinafter, referring to FIG. 1 and FIG., MarksThe function of the encoding device will be described. The inter-frame encoding unit 11 receives the IP signal P1 and receives an image signal that is one frame before the current display frame (current frame) (or an image signal that is one field before the interlaced display). Is used for encoding using correlation between frames.
[0026]
The intraframe encoding unit 12 performs normal encoding of an image signal limited to a display frame (current frame).
The inter-element image encoding unit 13 performs encoding using image signals corresponding to a plurality of element images. Here, the principle of inter-element image encoding will be described with reference to FIG. As is clear from the structure of the compound eye lens 63 (66) shown in FIG. 2, each of the element lenses included in the compound eye lens 63 (66) has one of images obtained by photographing the subject 61 from various angles. Therefore, the correlation between the image signals between these multiple element images is high, and in the inter-element image coding, the high correlation between the multiple element images is used to improve the efficiency from the multiple element images. Encoding can be performed automatically.
[0027]
The encoding selection unit 14 compares the encoding outputs from the inter-frame encoding unit 11, the intra-frame encoding unit 12, and the inter-element image encoding unit 13, and outputs the encoding output with the highest encoding efficiency (that is, The encoded output with the smallest code amount) is selected and output as the encoded output.
[0028]
  Figure 3, MarksIt is a flowchart which shows operation | movement of an encoding apparatus. Hereinafter, referring to FIG. 1 and FIG.MarkThe operation of the encoding device will be described. However, in the following, the encoding process will be described in the order of steps S11, S12, and S13. However, since each of the processes of steps S11, S12, and S13 is executed by starting an independent processing unit, step S11 is executed. , S12, and S13 are executed at the same time (however, the above-mentioned activation simply means that the flow of control is transferred to the corresponding processing unit, and the same meaning is used hereinafter) To do).
[0029]
The inter-frame encoding unit 11 shown in FIG. 1 is activated, and encoding from a frame (previous frame) located immediately before the current frame to be displayed (current frame) is executed (ie, inter-frame encoding). (Step S11).
The intra-frame encoding unit 12 is activated, and normal encoding (that is, intra-frame encoding) within a range closed to the current frame to be displayed (current frame) is executed (step S12).
[0030]
The inter-element image encoding unit 13 is activated, and encoding using the level of correlation between a plurality of element images (that is, inter-element image encoding) is executed (step S13).
Thereafter, the encoding selection unit 14 is activated to select an encoded output with the highest encoding efficiency (that is, an encoded output with the smallest code amount) and output as an encoded output (encoding selection process). Is executed (step S21).
According to the present embodiment, it is possible to perform optimum encoding utilizing the characteristics of each IP signal that is actually input in units of frames in consideration of the characteristics of the IP signal.
[0031]
Figure4 in the present inventionMarkIt is a block diagram which shows the structure of an encoding apparatus. The encoding apparatus according to the present embodiment is shown in FIG.YouAn image characteristic analysis unit 21 (image characteristic analysis) that selects an optimal encoding unit in a part positioned in front of the interframe encoding unit 11, the intraframe encoding unit 12, and the inter-element image encoding unit 13 of the encoding device. Means) and the coding selection unit 14 is deleted.
[0032]
  The functions of the interframe coding unit 11, the intraframe coding unit 12, and the inter-element image coding unit 13 are the same as those of the coding apparatus according to the first embodiment of the present invention. In addition, the configuration from the shooting to display of the 3D television system using the IP system (the formation of the IP signal) is as shown in FIG. Hereinafter, in the present invention,MarkThe function of the image characteristic analysis unit 21 of the encoding device will be described.
[0033]
The image characteristic analysis unit 21 performs encoding according to the characteristics of the input IP signal P2 (that is, the inter-frame encoding unit 11, the intra-frame encoding unit 12, and the inter-element image encoding unit 13). And the inputted IP signal P2 is delivered to the selected encoding unit.
[0034]
  FIG. 5 shows the present invention.MarkIt is a flowchart which shows operation | movement of an encoding apparatus. Hereinafter, the operation of the coding apparatus according to the present embodiment shown in FIG. 5 will be described with reference to FIGS. When the image characteristic analyzing unit 21 recognizes that the input IP signal P2 is a still image, the image characteristic analyzing unit 21 prepares for encoding for a still pixel (step A1). In this case, since it is obvious that the image of the current frame is the same as the image of the previous frame, the inter-frame encoding unit 11 is activated and encoding from the previous frame is performed (step A11).
[0035]
When the image characteristic analyzing unit 21 recognizes that the input IP signal P2 is a moving image, the image characteristic analyzing unit 21 prepares for encoding for moving pixels (step B1). In this case, between the element images after encoding in the current frame (step B11), encoding in the previous frame after performing motion correction (step B12), and searching for a corresponding pixel (step B13) Are encoded (step B21). Hereinafter, step B11 to step B13 will be described in more detail.
[0036]
In step B11, normal encoding is performed in a range in which the intra-frame encoding unit 12 is activated and closed in the current frame. In step B12, the inter-frame encoding unit 11 is activated to predict the movement of the subject from the image of the previous frame, and encoding based on the prediction result is performed. In step B <b> 13, the image characteristic analysis unit 21 searches for an element image in which the same image as the image to be displayed is captured by the corresponding pixel search process, and activates the inter-element image encoding unit 13. Thereafter, the inter-element image encoding unit 13 performs encoding between the image to be displayed and the found element image.
[0037]
In the final step (step B31) of encoding for moving pixels after step B1, the encoding selection unit (not shown) having the same function as the encoding selection unit 14 shown in FIG. An encoded output with high encoding efficiency, that is, an encoded output with the smallest code amount is selected and output.
[0038]
When detecting the scene change command included in the input IP signal P2, the image characteristic analysis unit 21 changes the scene and prepares for encoding corresponding to the scene change (step C1). As is well known, scene change is a broadcasting technique frequently used in television broadcasting. However, when this is done, there is no correlation between the current frame that has been changed and the previous frame, so in this case the frame changes. Inter-frame coding by the inter-coding unit 11 has no effect. Therefore, in the conventional encoding apparatus, intra-frame encoding is performed in this case. However, the encoding apparatus according to the present embodiment has a possibility that encoding between element images can be applied even in this case. Have.
[0039]
In order to confirm the possibility, the image characteristic analysis unit 21 first verifies whether occlusion exists in the image of the current frame that has been changed. Here, occlusion is a phenomenon in which an image of a subject is hidden by the action of an image in front of the subject or an edge attached to the subject itself. If there is occlusion, the phenomenon may be avoided by changing the direction of viewing the subject, that is, by changing the direction of viewing around.
[0040]
Therefore, when this occlusion is detected, the image characteristic analysis unit 21 first searches for an element image in the vicinity of the subject image, and searches for an element image (corresponding image) in which the same image as the subject image is captured. Search (step C21). Next, when the corresponding image exists by the search, the image characteristic analysis unit 21 activates the inter-element image encoding unit 13. Thereafter, the inter-element image encoding unit 13 performs encoding between the element images (step C32). When this corresponding image does not exist, the image characteristic analysis unit 21 activates the intra-frame coding unit 12. Thereafter, the normal encoding in the range closed in the current frame is performed in the intra-frame encoding unit 12 (step C31).
[0041]
On the other hand, when there is no occlusion, since encoding between element images can be applied unconditionally, the image characteristic analysis unit 21 activates the inter-element image encoding unit 13 unconditionally. Thereafter, the inter-element image encoding unit 13 performs encoding between the element images (step C32).
[0042]
According to the present embodiment, considering the characteristics of the IP signal, each of the IP signals that are actually input in units of frames is analyzed by going into the contents of the image, and utilizing the resulting characteristics It can be performed.
[0043]
  FIG., RecoveryIt is a block diagram which shows the structure of an encoding apparatus.. RecoveryThe encoding device includes an encoding analysis unit 31 (encoding analysis unit) that selects a decoding unit based on an input IP encoded signal, and an inter-frame decoding unit that decodes the encoded output output by the inter-frame encoding method. A decoding unit 32 (interframe decoding unit), an intraframe decoding unit 33 (intraframe decoding unit) that decodes an encoded output output by the intraframe encoding method, and an inter-element image encoding method. An inter-element image decoding unit 34 (inter-element image decoding means) that decodes the encoded output output by.
[0044]
  Hereinafter, referring to FIG., RecoveryThe function of the encoding device will be described. The encoding analysis unit 31 verifies the encoded classification signal included in the input IP encoded signal C3. This encoding classification signal is a signal for identifying which encoding method the input IP encoded signal is encoded by, and specifically, three types of encoding methods, That is, the encoding method employed in the interframe coding unit 11 shown in FIGS. 1 and 4, the coding method employed in the intraframe coding unit 12, and the inter-element image coding unit 13 are employed. This is a signal for identifying the encoding method.
[0045]
Based on the verification result, the encoding analysis unit 31 selects and activates any one of the inter-frame decoding unit 32, the intra-frame decoding unit 33, and the inter-element image decoding unit 34, and the IP encoded signal Deliver C3.
When the inter-frame decoding unit 32 is selected, the inter-frame decoding unit 32 performs the IP code by the inverse operation method of the encoding method employed by the inter-frame encoding unit 11 shown in FIGS. The converted signal C3 is converted into a decoded output P3.
[0046]
When the intra-frame decoding unit 33 is selected, the intra-frame decoding unit 33 uses the inverse operation method of the encoding method adopted by the intra-frame encoding unit 12 shown in FIGS. The converted signal C3 is converted into a decoded output P3.
When the inter-element image decoding unit 34 is selected, the inter-element image decoding unit 34 performs the inverse operation method of the encoding method employed by the inter-element image encoding unit 13 shown in FIGS. The IP encoded signal C3 is converted into a decoded output P3.
[0047]
  FIG., RecoveryIt is a flowchart which shows operation | movement of an encoding apparatus. Hereinafter, referring to FIG.RecoveryThe operation of the encoding device will be described. The input IP encoded signal C3 is analyzed by the encoding analysis unit 31, and the encoded classification signal included therein is verified. The encoding analysis unit 31 selects and activates any one of the inter-frame decoding unit 32, the intra-frame decoding unit 33, and the inter-element image decoding unit 34 based on the verification result, and selects the selected decoding The IP encoded signal C3 is delivered to the unit (step D1). Thereby, inter-frame decoding by the inter-frame decoding unit 32 (step D11), intra-frame decoding by the intra-frame decoding unit 33 (step D12), and inter-element image decoding by the inter-image decoding unit 34 (step D12). D13) is executed.
[0048]
  in this way,An encoded output (IP encoded signal) encoded by an optimal encoding method considering the characteristics of the IP signal can be decoded into the original IP signal with high fidelity.
[0049]
A program for causing the processing apparatus to perform the above-described processing, such as a program for executing the processing shown in the flowcharts of FIGS. 3, 5, and 7, is a computer such as a semiconductor memory, a CD-ROM, or a magnetic tape. May be stored and distributed in a readable recording medium. A computer including at least a microcomputer, a personal computer, and a general-purpose computer may read the program from the recording medium and execute the program.
[0050]
【The invention's effect】
  As described above, in the encoding device according to the present invention,The following excellent effects are exhibited.
[0051]
BookIn the encoding device according to the invention, encoding using the correlation between the image signals using the image signal of the current display frame and the image signal of the display frame temporally prior to the current display frame, The characteristics of the input IP signal include encoding using an image signal within a range closed by a display frame and encoding using correlation between image signals using a plurality of image signals corresponding to a plurality of element images. Accordingly, it is possible to selectively execute appropriately, and thus it is possible to perform optimum encoding appropriately corresponding to the characteristics of the IP signal.
[0052]
Further, it is verified whether the input IP signal is a still image, a moving image, or a scene change command, and encoding of the input IP signal is executed according to the verification result. Considering the characteristics of the signal, it is possible to perform optimum encoding utilizing the characteristics obtained by appropriately entering the contents of the image for each IP signal actually input in units of frames.
[0053]
If the input IP signal is a still image signal, the input IP signal is encoded by inter-frame encoding, so that the still image having a high correlation with the previous frame is used. Certainly, it is possible to apply encoding using the correlation with the previous frame.
[0054]
If the input IP signal is a moving image signal, intra-frame coding, inter-frame coding after motion correction, and searching for other element images corresponding to the current image are performed. In addition, the inter-element image encoding is performed in parallel, and the encoded output having the maximum encoding efficiency is output from the plurality of encoded outputs. As a result, it is possible to obtain an encoded output by an optimal encoding method.
[0055]
Furthermore, if the input IP signal is a signal with a scene change command, and if there is no occlusion in the current image, inter-element image encoding, occlusion exists, and other corresponding to the current image When there is no element image, intra-frame coding and occlusion exist, and when there is another element image corresponding to the current image, element image coding can be applied. If there is no occlusion, encoding using correlation between element images, and if there is scene change and occlusion, search for the element image corresponding to the current image. Coding using the correlation between the current image and the searched element image can be performed.
[Brief description of the drawings]
[Figure 1]Reference exampleIt is a block diagram which shows the structure of the encoding apparatus which concerns on.
FIG. 2 is an explanatory diagram showing a configuration from the shooting to display of a stereoscopic television system using the IP system according to the present invention (the formation of IP signals).
[Fig. 3]For reference exampleIt is a flowchart which shows operation | movement of the encoding apparatus which concerns.
[Figure 4] Main departureClearlyIt is a block diagram which shows the structure of the encoding apparatus which concerns.
[Figure 5] DepartureClearlyIt is a flowchart which shows operation | movement of the encoding apparatus which concerns.
FIG.] RecoveryIt is a block diagram which shows the structure of an encoding apparatus.
FIG.] RecoveryIt is a flowchart which shows operation | movement of an encoding apparatus.
FIG. 8 is an explanatory diagram showing an imaging principle of a conventional IP system stereoscopic television camera.
FIG. 9 is a block diagram showing the structure of a conventional practical stereoscopic TV camera.
FIG. 10 is another configuration diagram showing the structure of a conventional practical IP 3D television camera.
[Explanation of symbols]
11: Interframe encoding unit,
12: Intra-frame encoding unit,
13 ... Inter-element image encoding unit,
14: Encoding selection unit,
21 …… Image characteristic analysis unit,
31 …… Encoding analysis unit,
32 …… Interframe decoding unit,
33 …… Intra-frame decoding unit,
34 .. Decoding unit between element images,
C1, C2 ... Coding output
C3 …… IP coded signal
P1, P2 ... IP signal
P3 …… Decoded output

Claims (2)

In an encoding apparatus that inputs and encodes an integral photography signal output from a television camera that captures an element image connected to each of a plurality of integrated imaging lenses,
Inter-frame encoding means for performing encoding using the correlation between the image signals using the image signal of the current display frame and the image signal of the display frame temporally prior to the current display frame;
Intra-frame encoding means for encoding using an image signal within a range closed to the current display frame;
Inter-element image encoding means for performing encoding using correlation between the image signals using a plurality of image signals corresponding to a plurality of element images;
According to the content of the inputted integral photography signal, any one of the inter-frame coding means, the intra-frame coding means and the inter-element image coding means is appropriately selected and the selected have a, an image characteristic analysis means for controlling so that encoding of the input integral photography signal is performed by means,
The image characteristic analysis means includes
As the content of the input integral photography signal, it is verified whether the characteristic is a still image, a moving image, or with a scene change command, and according to the verification result, the interframe coding means, Any one of intra-frame coding means and inter-element image coding means is appropriately selected, and coding of the input integral photogalagh signal is realized by the selected means. Control to
When the input integral photography signal is a signal with a scene change command, the scene is changed, and when there is no occlusion in the current image, the inter-element image encoding means,
When there is occlusion and there is no other element image corresponding to the current image, the intra-frame coding means,
When there is occlusion and there is another element image corresponding to the current image, the inter-element image encoding means,
An encoding apparatus that controls to execute encoding of the input integral photography signal.   Of the encoded outputs respectively output from the inter-frame encoding unit, the intra-frame encoding unit, and the inter-element image encoding unit, an encoding selection unit that outputs an encoded output with the maximum encoding efficiency The encoding device according to claim 1, further comprising:

Images