JP4854580B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more specifically to an image processing apparatus that improves the image quality of a moving image when a still image is captured during moving image shooting.

  In recent years, digital cameras capable of shooting both still images and moving images have been commercialized. The user can view the recorded image using these devices or a personal computer. Some digital cameras that can shoot both still images and moving images can shoot still images during moving image shooting (hereinafter referred to as simultaneous still image shooting). With this simultaneous still image shooting function, the user can shoot a still image in a favorite scene even during moving image shooting.

  On the other hand, an encoding method such as MPEG2 (Moving Picture Experts Group Phase 2) has been established as a technique for highly efficient encoding of moving images shot with a digital video camera or the like.

  One of the techniques for improving the encoding efficiency of the MPEG2 encoding method is a variable bit rate (VBR) encoding method. In variable bit rate (VBR) encoding, the encoding rate is changed in accordance with the complexity of the image and the intensity of motion in each frame of the moving image.

However, not only changing the encoding rate according to the complexity of the image and the intensity of movement, but also by increasing the encoding rate for the period specified by the user, Encoding with high image quality is also known (for example, Patent Document 1).
JP 2006-121574 A

  The technique described in Patent Document 1 requires a user's intentional operation. That is, the user has to consciously input a specific instruction to the camera during a period when it is desired to shoot with high image quality, and the operation becomes complicated.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus that increases the coding rate of a moving image without a user's conscious instruction.

An image processing apparatus according to the present invention is an image processing apparatus, and generates first moving image data from the image signal using an imaging unit that converts an optical image of a subject into an image signal and a first encoding rate. First generating means, second generating means for generating still image data from the image signal, and the second generating means in a state where the first moving image data is recorded on a recording medium. When it is detected that a predetermined instruction for generating image data is input to the image processing apparatus, the second instruction is generated using a second encoding rate higher than the first encoding rate. and a to that control means so that the second moving image data is recorded on the recording medium, the second moving image data, the time from the detection of said predetermined instruction until a predetermined time has elapsed It corresponds to.

  According to the present invention, when a still image is shot during moving image shooting, the encoding rate of the moving image during the period in which the still image was shot is increased. Thereby, it is possible to record a moving image with high image quality during a period in which simultaneous still image shooting is performed without troublesome work.

  Also, in the re-encoding of moving images, it is possible to record moving images with high image quality during the period in which simultaneous still image shooting was performed without troublesome work.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a schematic configuration block diagram of an imaging apparatus 10 as an image processing apparatus that is Embodiment 1 of the present invention. In the first embodiment, the moving image encoding rate is increased in accordance with a still image shooting operation during moving image shooting.

  The camera unit 12 includes a photographic lens 14, an image sensor 16, an A / D converter 18, and a camera signal processing device 20. The taking lens 14 has an optical zoom function and a focusing function. The image sensor 16 is composed of an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), for example. The image sensor 16 converts an optical image of a subject by the photographing lens 14 into an electric image signal. The A / D converter 18 converts the analog image signal from the image sensor 16 into a digital image signal. The camera signal processing device 20 performs processing such as γ correction and white balance on the digital image signal from the A / D converter 18 and outputs a video signal in a predetermined format.

  The moving image encoding device 22 compresses and encodes the video signal from the camera unit 12 as a moving image at the encoding rate controlled by the encoding rate control device 24. The moving image recording device 26 records the moving image data encoded by the moving image encoding device 22 on a recording medium. The moving image recording device 26 adds the recording time to moving image data as incidental information of the moving image and records it.

  On the other hand, the still image encoding device 28 compresses and encodes one frame of the video signal output from the camera unit 12 as a still image. The still image recording device 30 records the still image data encoded by the still image encoding device 28 on a recording medium. The still image recording device 30 adds the recording time to the still image data as incidental information of the still image and records it. The recording medium on which the still image recording device 30 records still image data may be the same as or different from the recording medium on which the moving image recording device 26 records moving image data.

  The moving image encoding device 22, the coding rate control device 24, and the moving image recording device 26 constitute a moving image recording system, and the still image encoding device 28 and the still image recording device 30 constitute a still image recording system.

  The system control device 32 starts or stops moving image recording by the moving image recording system in accordance with the operation of the moving image recording / stop switch 34. The system control device 32 also causes the still image recording system to execute still image recording in accordance with the operation of the release switch 36. The moving image recording / stop switch 34 functions as moving image recording instruction means in the claims. The release switch 36 functions as a still image recording instruction unit in the claims.

  As a characteristic operation of the present embodiment, for a still image recording operation by the release switch 36 during moving image recording, the system control device 32 controls the coding rate of moving image compression via the coding rate control device 24. , Temporarily increase the quality of the video being recorded.

  Link information for associating each other may be additionally recorded as supplementary information on one or both of the moving image and the still image that are simultaneously shot. Information about the shooting time of a still image shot at the same time may be added to the moving image.

  The release switch 36 used for still image shooting is a two-stage switch. As is well known, the exposure and focus of the camera unit 12 are adjusted in the half-pressed state (still image shooting preparation operation or first pressing step). Control is performed, and still image shooting is executed when the button is fully pressed (second pressing stage). In response to the release switch 36 being fully pressed, the still image encoding device 28 takes in the output video signal of the camera unit 12 and compresses it, and the still image recording device 30 records the compressed still image data on the recording medium. To do. In the case of shooting only a still image, the camera unit 12 may output an image signal having a larger number of pixels than when shooting a moving image.

  FIG. 2 shows a schematic block diagram of the moving picture encoding device 22. A configuration in which an MPEG (Moving Picture Experts Group) encoding method is adopted as an encoding method will be described. However, the encoding method is not limited to this. H.264 encoding may be used.

  The input buffer 52 of the moving image encoding device 22 temporarily stores the video signal from the camera unit 12 for subsequent processing. The capacity of the input buffer 52 affects the delay time of the video encoding device 22. The subtractor 54 outputs the image data from the input buffer 52 as it is in the case of intra-frame coding, and subtracts the predicted image data from the image data from the input buffer 52 in the case of inter-frame coding. , Output image difference data. The predicted image data will be described later.

  A discrete cosine transform (DCT) device 56 performs a discrete cosine transform on the data from the subtractor 54 and outputs the transform coefficient. The quantization device 58 quantizes the transform coefficient data from the DCT device 56 based on the quantization scale. By changing the quantization scale, the amount of generated code, and hence the encoding rate, changes greatly.

  The variable length coding device 72 performs variable length coding on the transform coefficient quantized by the quantization device 58 and the motion vector from the motion detection device 70. The output buffer 74 temporarily stores the output code of the variable length encoding device 72 and outputs it to the subsequent stage (here, the moving image recording device 26).

  The code amount control device 76 monitors the storage data amount or occupation ratio of the output buffer 74. The code amount control device 76 basically controls the quantization scale of the quantization device 58 so that the amount of data stored in the monitored output buffer 74 becomes substantially constant. In the present embodiment, the code amount control device 76 controls the quantization scale of the quantization device 58 by the variable bit rate (VBR) method in accordance with the control signal from the coding rate control device 24.

  The transform coefficient quantized by the quantizing device 58 is locally decoded and used to generate predicted image data. The inverse quantization device 60 inversely quantizes the transform coefficient quantized by the quantization device 58. The inverse DCT device 62 performs inverse discrete cosine transform on the transform coefficient inversely quantized by the inverse quantization device 60. The adder 64 outputs the output data of the inverse DCT device 62 as it is in the case of intra-frame coding, and predictive image data as the output data (difference data) of the inverse DCT device 62 in the case of inter-frame coding. Is added. Thereby, the image data compressed by the DCT device 56 and the quantization device 58 is expanded and restored. The reconstructed image data restored locally in this way is stored in the frame memory 66. Among the reconstructed image data, image data that may be referred to in the subsequent interframe coding is stored in the frame memory 66 for a period of time.

  The motion detection device 70 detects the motion of the image by referring to the current image from the input buffer 52 and the image stored in the frame memory 66, and the motion compensation device 68 and the variable length encoding of the detected motion vector. Supply to device 72. The motion compensation device 68 compensates for the motion of the image from the frame memory 66 in accordance with the motion vector from the motion detection device 70. The motion compensated image data is supplied to the subtractor 54 and the adder 64 as predicted image data.

  The operation of the coding rate control device 24 for still image shooting during moving image recording will be described. FIG. 3 is a schematic diagram illustrating an example of a change in the coding rate when still image shooting is performed during moving image shooting. However, FIG. 3 shows a case where a still image is shot 20 seconds after the start of shooting during 40 seconds of moving image shooting.

  First, when shooting is started (time: 0 second), the encoding rate control device 24 controls the encoding rate of the moving image encoding device 22 to a predetermined encoding rate, for example, 5 Mbps. When the release switch 36 is half-pressed by the user during video recording (time: 10 seconds), the encoding rate control device 24 gradually increases the video encoding device 22 to a certain maximum encoding rate, for example, 10 Mbps. Increase the encoding rate. When the maximum encoding rate (10 Mbps) is reached (time: 15 seconds), the maximum encoding rate (10 Mbps) is maintained.

  If the user fully presses the release switch 36 within a predetermined time after the release switch 36 is pressed halfway (time: 20 seconds), the encoding rate control device 24 sets the maximum encoding rate (10 Mbps) to a predetermined time, for example, Maintain for 5 seconds (time: 25 seconds). Thereafter, the coding rate control device 24 gradually lowers the coding rate to the original coding rate, in this example, 5 Mbps. When the encoding rate is lowered to the original predetermined encoding rate (5 Mbps) (time: 30 seconds), the encoding rate control device 24 maintains the predetermined encoding rate (5 Mbps).

  By such an operation of the coding rate control device 24, when a still image is shot during moving image shooting, the encoding rate is set higher than a predetermined encoding rate during the period before and after the still image is shot. To do. As a result, it is possible to encode and record a moving image of a predetermined period in which simultaneous still image shooting is performed with high image quality.

  The reason why the coding rate is gradually changed is to prevent the viewer from recognizing a change in image quality due to a sudden change in the coding rate. In the example of FIG. 3, the coding rate is changed over 5 seconds both when the coding rate is raised and when the coding rate is lowered. It is obvious that the predetermined encoding rate or the maximum encoding rate can be changed according to the user's selection of the image quality mode or the like.

  FIG. 4 is an operation flowchart of the coding rate control device 24. In response to the operation of the moving image recording / stop switch 34, moving image shooting is started (S11). When the release switch 36 is not half-pressed by the user (S12), the encoding rate control device 24 controls the encoding rate of the moving image encoding device 22 to a predetermined encoding rate, for example, 5 Mbps (S13). ).

  When the release switch 36 is half-pressed by the user (S12), the encoding rate control device 24 gradually increases the encoding rate of the moving image encoding device 22 toward the maximum encoding rate, for example, 10 Mbps. (S14). When the set maximum encoding rate (10 Mbps) is reached, it is maintained as it is. However, when the half-pressed state of the release switch 36 elapses for a predetermined time or when the half-press is released (S15), the encoding rate control device 24 sets the encoding rate to a predetermined encoding rate (5 Mbps). ) Is gradually lowered toward S) (S16). On the other hand, when the release switch 36 is fully pressed within a predetermined time from the start of half-pressing in S15 (S17), the encoding rate control device 24 keeps the encoding rate of the moving image encoding device 22 at the maximum encoding rate. Maintain for a predetermined time (5 seconds). In the case where it is fully pressed immediately after half pressing, the maximum encoding rate is maintained for a predetermined time (5 seconds) after reaching the maximum encoding rate. Thereafter, the coding rate control device 24 gradually lowers the coding rate of the moving image coding device 22 to a predetermined coding rate (5 Mbps) (S18). If the release switch 36 is not fully depressed in S17, the process returns to S12.

  The above processing is repeated until the moving image shooting is completed (S19).

  In this way, in the present embodiment, a moving image in the vicinity at the time of still image shooting can be recorded with high image quality only by an operation accompanying the still image shooting.

  Next, a second embodiment of the present invention will be described. FIG. 5 is a schematic configuration block diagram of Embodiment 2 of the present invention. As in the first embodiment, the imaging apparatus 210 according to the second embodiment can simultaneously capture still images during moving image shooting. However, this is not a configuration for improving the image quality of a moving image being simultaneously shot in accordance with still image shooting, but a configuration for improving the image quality of a moving image simultaneously shot by re-encoding moving image data.

  The camera unit 212 of the imaging device 210 has the same configuration as the camera unit 12. The moving image recording / reproducing device 214 compresses and encodes an image output from the camera unit 212 as a moving image and records it. The still image recording device 216 compresses and encodes an image at the timing instructed by the user among the images output from the camera unit 212 as a still image and records it.

  In one or both of the moving image recording / reproducing device 214 and the still image recording device 216, link information regarding simultaneous still image shooting is recorded as supplementary information so that it can be referred to later. The supplementary information is, for example, information on the file name and shooting time of a simultaneously shot still image for a moving image, and information on the file name and frame number of the simultaneously shot moving image for a still image.

  The moving picture decoding apparatus 218 decodes the compressed and encoded moving picture data recorded in the moving picture recording / reproducing apparatus 214 and supplies the decoded video signal to the moving picture encoding apparatus 220. The simultaneous still image shooting information creation device 222 refers to the accompanying information of the moving image and still image recorded in the moving image recording / playback device 214 and the still image recording device 216, and detects the moving image and the still image captured simultaneously. The simultaneous still image shooting information creation device 222 outputs simultaneous still image shooting information indicating the detection result to the re-encoding rate control device 224.

  The re-encoding rate control device 224 determines and controls the encoding rate when the moving image encoding device 220 performs re-encoding according to the simultaneous still image shooting information from the simultaneous still image shooting information creation device 222. . The video encoding device 220 re-encodes the decoded video signal from the video decoding device 218 at the encoding rate controlled by the re-encoding rate control device 224. The re-encoded moving image data is recorded on a recording medium by the moving image recording / reproducing device 214. At that time, the previous encoded moving image data may be deleted or stored as it is.

  FIG. 6 shows a schematic block diagram of the moving picture decoding apparatus 218. The configuration of the MPEG decoding method for decoding moving image data encoded by the MPEG encoding method will be described. However, the decoding method is not limited to this. When decoding a moving image encoded by the H.264 encoding method, H.264 decoding method.

  The video decoding device 218 includes an input buffer 282, a variable length decoding device 284, an inverse quantization device 286, an inverse DCT device 288, an adder 290, a frame memory 292, and a motion compensation device 294.

  The input buffer 282 temporarily stores the encoded moving image data reproduced from the moving image recording / reproducing device 214. The variable length decoding device 284 performs variable length decoding on the encoded moving image data from the input buffer 282 and outputs the quantized transform coefficient to the inverse quantization device 286. The variable length decoding device 284 extracts the motion vector information multiplexed in the encoded moving image data, and outputs it to the motion compensation device 294.

  The inverse quantizer 286 performs inverse quantization on the quantized transform coefficient, and the inverse DCT device 288 performs inverse discrete cosine transform on the output of the inverse quantizer 286. The adder 290 outputs the output data of the inverse DCT device 288 as it is to the moving image data encoded in the frame. The adder 290 also adds the motion-compensated predicted image data from the motion compensation device 294 to the output data of the inverse DCT device 288 for the inter-frame encoded moving image data. Through these processes, the image data of each screen of the moving image is restored.

  The frame memory 292 temporarily stores the image data inter-frame prediction predicted image restored in this way. The motion compensation device 294 refers to the motion vector information from the variable length decoding device 284, compensates for the motion of the image data temporarily stored in the frame memory 292, and supplies it to the adder 290 as predicted image data.

  The operation of the simultaneous still image shooting information creation device 222 will be described. The simultaneous still image shooting information creation device 222 refers to the moving image auxiliary information of the moving image recording / reproducing device 214 and the still image auxiliary information of the still image recording device 216, and It is determined whether or not the movie was recorded during shooting of the moving image recorded in 214. Then, the simultaneous still image shooting information creation device 222 provides the re-encoding rate control device 224 with the shooting time information of the still image shot during moving image shooting as simultaneous still image shooting information when there is a still image shot simultaneously. Output. The time information of the still image may be, for example, an elapsed time from the start of the simultaneous shooting moving image to the shooting of the still image.

  The operation of the re-encoding rate control device 224 will be described with reference to FIG. FIG. 7 shows an example of changes in the encoding rate when re-encoding moving image data in the second embodiment. In the example illustrated in FIG. 7, in a 40-second moving image, a still image is simultaneously captured 20 seconds after the beginning of the moving image. The simultaneous still image shooting information creation device 222 supplies the re-encoding rate control device 224 with simultaneous still image shooting information indicating that a still image has been simultaneously shot after 20 seconds from the start of moving image recording. In FIG. 7, the broken line indicates the encoding rate before transcoding (first encoding rate), and the solid line indicates the encoding rate after transcoding (second encoding rate). In this embodiment, a moving image that has been encoded at an encoding rate of 10 Mbps is re-encoded (transcoded) at an encoding rate of 5 Mbps, which is lower than 10 Mbps, except for the still image simultaneous shooting portion.

  When transcoding of moving image data is started (time: 0 second), the re-encoding rate control device 224 sets the encoding rate of the moving image encoding device 220 to a predetermined encoding rate (5 Mbps in this example). Control. That is, a moving image whose encoding rate was 10 Mbps is re-encoded at 5 Mbps.

  According to the simultaneous still image shooting information from the simultaneous still image shooting information creating device 222, still images are simultaneously shot 20 seconds after the start of shooting. Therefore, the re-encoding rate control device 224 sets the set maximum encoding rate from the time (for example, time: 10 seconds) that is a predetermined time before the time (time: 20 seconds) when the still image was captured. The encoding rate of the moving image encoding device 220 is gradually increased toward the original encoding rate (10 Mbps here). When the encoding rate reaches the maximum encoding rate (10 Mbps) (time: 15 seconds), the maximum encoding rate is maintained. Thereafter, the encoding rate is gradually lowered toward a predetermined encoding rate (5 Mbps) from 5 seconds after the time when the still image was captured (time: 25 seconds). When the encoding rate is lowered to a predetermined encoding rate (5 Mbps) (time: 30 seconds), the predetermined encoding rate (5 Mbps) is maintained. The reason why the coding rate is gradually changed is to prevent the viewer from recognizing a change in image quality due to a sudden change in the coding rate. In the example of FIG. 7, the coding rate is changed over 5 seconds each time the coding rate is increased or decreased.

  In this way, if a still image is being shot at the same time that the moving image to be re-encoded is shot, the encoding rate of the moving image can be set to other values in the period before and after the still image was shot. Set higher than the encoding rate of the period. As a result, it is possible to re-encode a moving image of a predetermined period based on the time when simultaneous still image shooting is performed with high image quality.

  FIG. 8 shows an operation flowchart of the re-encoding rate control device 224. Video transcoding is started (S51). Based on the simultaneous still image shooting information from the simultaneous still image shooting information creation device 222, it is determined whether there is a still image shot simultaneously (S52). When there is no simultaneously shot still image (S52), the re-encoding rate control device 224 controls the encoding rate of the moving image encoding device 220 to a predetermined encoding rate (5 Mbps) (S53).

  If there are still images that have been shot simultaneously (S52), the re-encoding rate control device 224 performs moving image encoding until a predetermined time (10 seconds before) the shooting time of the still images that have been shot simultaneously (S54). The encoding rate of the apparatus 220 is maintained at a predetermined encoding rate (5 Mbps) (S53). When it becomes within the predetermined time (within 10 seconds) until the shooting time of the still image taken at the same time (S54), the re-encoding rate control device 224 sets the encoding rate of the moving image encoding device 220 to the maximum encoding rate (10 Mbps). The height is gradually increased (S55).

  The re-encoding rate control device 224 sets the encoding rate of the moving image encoding device 220 to the maximum encoding rate (10 Mbps) until a predetermined time (5 seconds later) from the shooting time of the still image captured simultaneously (S56). To maintain.

  When a predetermined time (5 seconds) elapses from the shooting time of the still image taken at the same time (S56), the re-encoding rate control device 224 sets the encoding rate of the moving image encoding device 220 to a predetermined encoding rate (5 Mbps). Gradually lower (S57).

  The above processing is repeated until the end of transcoding (S58).

  In this embodiment, when a still image is shot during moving image shooting, the moving image encoding bit rate is set to be relatively higher than that of other scenes in the period before and after the time when the still image was shot. Therefore, it is possible to record a moving image in the vicinity of still image shooting with high image quality. Thereby, the moving image of the favorite scene which the user performed still image photography can be recorded with high image quality without complicated processing.

It is a schematic block diagram of Example 1 of the present invention. 3 is a block diagram of a schematic configuration of a moving image encoding device 22. FIG. It is a schematic diagram of the relationship between simultaneous still image photography and the encoding rate of a moving image. 5 is an operation flowchart of the coding rate control device 24. It is a schematic block diagram of Example 2 of this invention. 3 is a block diagram of a schematic configuration of a video decoding device 218. FIG. It is a schematic diagram of the relationship between the simultaneous still image photography in Example 2, and the encoding rate of a moving image. 6 is an operation flowchart of the re-encoding rate control device 224.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image pickup device 12 Camera unit 14 Shooting lens 16 Image pick-up element 18 A / D converter 20 Camera signal processing device 22 Movie encoding device 24 Coding rate control device 26 Movie recording device 28 Still image encoding device 30 Still image recording device 32 System controller 34 Movie recording / stop switch 36 Release switch 52 Input buffer 54 Subtractor 56 DCT (discrete cosine transform)
58 Quantizer 60 Inverse Quantizer 62 IDCT (Inverse Discrete Cosine Transform)
64 Adder 66 Frame memory 68 Motion compensation device 70 Motion detection device 72 Variable length coding device 74 Output buffer 76 Code amount control device 210 Imaging device 212 Camera unit 214 Movie recording / playback device 216 Still image recording device 218 Movie decoding device 220 Video encoding device 222 Simultaneous still image shooting information creation device 224 Recoding rate control device 282 Input buffer 284 Variable length decoding device 286 Inverse quantization device 288 Inverse DCT (discrete cosine transform) device 290 Adder 292 Frame memory 294 Motion Compensation device

Claims (9)

An image processing apparatus,
Imaging means for converting an optical image of a subject into an image signal;
First generation means for generating first moving image data from the image signal using a first encoding rate ;
Second generation means for generating still image data from the image signal;
When it is detected that a predetermined instruction for causing the second generation means to generate the still image data is input to the image processing apparatus while the first moving image data is recorded on a recording medium in, and a first coding rate higher second second moving image data to that control means so that recorded on the recording medium produced by using the coding rate than,
The image processing apparatus according to claim 2, wherein the second moving image data corresponds to a time from when the predetermined instruction is detected until a predetermined time elapses . The control means has third moving image data generated using a third encoding rate that is equal to or higher than the first encoding rate and lower than the second encoding rate on the recording medium. To be recorded,
The image processing apparatus according to claim 1, wherein the third moving image data corresponds to a time after the predetermined time has elapsed. The image processing apparatus according to claim 1, wherein the second moving image data is not generated using a fourth encoding rate lower than the first encoding rate. The said control means controls the encoding rate used in order to make a said 1st production | generation means produce | generate moving image data from the said image signal in steps. The image processing apparatus described. The control means uses the first moving image data corresponding to the time from when the predetermined instruction is detected until the predetermined time elapses, and the second encoding rate, and the recording medium. 5. The image processing apparatus according to claim 1, wherein the second moving image data to be recorded is generated. The control means uses the information indicating the still image data recorded in response to the predetermined instruction and the first moving image data recorded on the recording medium after the predetermined instruction is detected. The image processing apparatus according to claim 5, wherein the first moving image data corresponding to a time until a predetermined time elapses is detected. The image processing according to any one of claims 1 to 6, wherein the first moving image data and the second moving image data are encoded in accordance with MPEG (Moving Picture Experts Group). apparatus. The first moving image data and the second moving image data are H.264 data. The image processing apparatus according to claim 1, wherein the image processing apparatus is encoded according to H.264. When it is detected that the predetermined instruction is input to the image processing apparatus in a state where the first moving image data is recorded on the recording medium, the control unit generates the still image data. The image processing according to any one of claims 1 to 8, wherein the second generation unit is controlled as described above, and the second moving image data is recorded on the recording medium. apparatus.

Images