JP7148236B2 - Encoding device and its control method and program - Google Patents

Description

The present invention relates to video encoding technology .

A moving image has a very large amount of information. Therefore, in a device for capturing moving images (such as a video camera or a portable communication device with a built-in camera), captured moving images are compression-encoded and then recorded in a storage medium.

In recent years, devices capable of reproducing various sizes (resolutions) such as 1920×1080 pixels and beyond such as 8192×4320 pixels have appeared. With such diversification of reproduction environments, methods for realizing encoding of moving images of a plurality of image sizes have been proposed, as in Patent Document 1, for example. Japanese Patent Application Laid-Open No. 2002-200000 discloses a technique of encoding two moving images, one of which is a small-sized image in which a region of interest is cut out, and the other of which is not cut out. In Patent Literature 1, moving image data of two different image sizes are encoded while suppressing the amount of code by replacing pixel values corresponding to a region of interest of a moving image with a single color.

JP 2010-212811 A

However, since the moving image data encoded in Patent Document 1 is obtained by reducing the original image, the image quality is degraded. Furthermore, when reproducing a moving image of a region of interest, it is necessary to decode two pieces of encoded data and synthesize them into one reproduced image.

SUMMARY OF THE INVENTION The present invention has been made in view of such problems, and it is an object of the present invention to provide a technique for generating encoded data capable of decoding a partial area including a main subject while encoding a high-resolution image. .

In order to solve this problem, for example, the encoding device of the present invention has the following configuration. i.e.
An encoding device for encoding a moving image,
Determination means for determining the position of a sub-image area of a preset size containing the main subject from the frame of interest to be encoded;
dividing means for dividing the frame of interest into a plurality of regions based on boundaries of the sub-image regions determined by the determining means;
encoding means for encoding an image of each region obtained by dividing by the dividing means;
The encoded data of the frame of interest is obtained from the encoded data obtained by the encoding means, the position of the area of the sub-image in the frame of interest, and additional information indicating the storage address of the encoded data of the sub-image. and generating means for generating
The encoding means are
For the area of the sub-image in the frame of interest, encoding is performed with reference to the area of the sub-image or the sub-image area in the previously encoded frame,
Encoding an area other than the area of the sub-image in the frame of interest with reference to the frame of interest or a previously encoded frame,
The determining means determines the position of the main subject and the position of the sub-image region using update timing when the frame of interest to be coded is an I picture or a picture referring to the I picture. death,
When the frame of interest to be encoded is not at the update timing, the encoding means performs encoding assuming that the position of the area of the sub-image is the same as that of the previous frame.

According to the present invention, it is possible to generate encoded data capable of decoding a partial area including a main subject while encoding a high-resolution image.

Also, according to the present invention, it is possible to decode and reproduce only a partial area including the main subject from high-resolution encoded data.

The figure which shows an example of the system in embodiment. FIG. 2 is a block configuration diagram of an encoding device and a decoding device according to the embodiment; FIG. 2 is a block configuration diagram of an encoding processing unit according to the embodiment; FIG. 4 is a diagram showing an example of tile division in encoding processing according to the embodiment; 4 is a flowchart showing processing of the encoding device in the embodiment; 4 is a flowchart showing tile configuration determination processing and encoding processing of the encoding device according to the embodiment. 4 is a flowchart showing processing of the decoding device in the embodiment; FIG. 10 is a diagram for explaining a problem when tile positions change between frames; FIG. 11 is a diagram showing an example of trimming the screen of a decoded image according to the second embodiment; 9 is a flowchart showing processing of the encoding device in the second embodiment; 9 is a flowchart showing processing of the decoding device in the second embodiment; 10 is a flowchart showing an example of processing for determining a trimming position of a decoded image according to the second embodiment;

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

<System configuration diagram>
As shown in FIG. 1, the system of this embodiment has an encoding device 100, a decoding device 200, and a display device 300. FIG.

The encoding device 100 performs imaging in response to a shooting instruction by the user through the shutter switch of the operation unit 104, and obtains image data. The encoding device 100 can compress the amount of image data by encoding image data obtained by imaging. The encoding device 100 records encoded image data obtained by encoding in a removable non-volatile recording medium, or transfers the data to a device such as the decoding device 200 using wireless communication. The decoding device 200 receives encoded image data using a removable non-volatile recording medium or wireless communication. The decoding device 200 reproduces by outputting the image data obtained by decoding the encoded image data to the display device 300 by operating the operation unit 204 or a remote controller (not shown). The display device 300 receives the image data (image signal) from the decoding device 200 and displays the image on the display section.

The encoding device 100 is an imaging device such as a digital still camera, a smart phone, a camera-equipped mobile phone, or a digital video camera. The decoding device 200 may be a reproducing device such as a player that reproduces image data or audio data, or may be an imaging device such as a digital still camera, a smart phone, a mobile phone with a camera, or a digital video camera. The display device 300 is a display device such as liquid crystal or organic EL. Note that the encoding device 100 may perform decoding processing similar to that of the decoding device 200 . The decoding device 200 may also include a display device 300 .

<Overall block diagram>
FIG. 2 is a diagram showing block configurations of the encoding device 100 and the decoding device 200. As shown in FIG.

The encoding device 100 has a CPU 101 , a memory 102 , a nonvolatile memory 103 , an operation section 104 , an imaging lens 111 , an imaging section 112 , an image processing section 113 and an encoding processing section 114 . The encoding device 100 also has a display control unit 115 , a display unit 116 , a communication control unit 117 , a communication unit 118 , a recording medium control unit 119 , a recording medium 120 and an internal bus 130 .

The CPU 101 controls the operation of each unit of the encoding device 100 by executing computer programs stored in the nonvolatile memory 103 . The CPU 101 shoots a person or the like as the main subject in advance in response to a signal such as a shutter button or a menu button input from the operation unit 104 operated by the user, and records it in the nonvolatile memory 103 as main subject information. In addition, the CPU 101 determines the size of the small image size screen area including the specific main subject based on a signal such as a menu button input from the operation unit 104 by the user. The CPU 101 stores the size information of the screen area specified by the user in the memory 102 .

The memory 102 is a rewritable volatile memory, and temporarily stores a computer program for controlling the operation of each unit of the encoding apparatus 100, information such as parameters related to the operation of each unit, information received by the communication control unit 117, and the like. do. The memory 102 has a sufficient storage capacity for temporarily recording images and information processed by the imaging unit 112, the image processing unit 113, the encoding processing unit 114, and the like.

The nonvolatile memory 103 is an electrically erasable/recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 103 stores information such as a computer program for controlling the operation of each unit of the encoding apparatus 100 and parameters related to the operation of each unit. The computer program referred to here is a program for executing various flowcharts described later in this embodiment.

An operation unit 104 provides a user interface for operating the encoding device 100 . An operation unit 104 has a power button, a menu button, a shutter button, and the like of the encoding apparatus 100, and each button is configured by a switch, a touch panel, and the like. The CPU 101 controls the encoding device 100 according to user instructions input via the operation unit 104 . Note that the operation unit 104 may control the encoding apparatus 100 according to a remote control signal received from a remote controller (not shown) or a request notified from a portable terminal (not shown) via the communication control unit 117 .

The imaging lens 111 includes a lens group including a zoom lens and a focus lens, a lens control section, an aperture, and the like. The imaging lens 111 has a lens control unit (not shown), and controls focus adjustment and an aperture value (F value) according to a control signal transmitted from the CPU 101 .

The imaging unit 112 includes an imaging device that converts an optical image of a subject into an electrical signal. The imaging device is an area image sensor composed of, for example, a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor) device. The imaging unit 112 converts an analog image signal obtained by imaging into a digital image signal (image data), and outputs the digital image signal (image data) to the image processing unit 113 or the memory 102 .

The image processing unit 113 performs resizing processing such as predetermined pixel interpolation and reduction, color conversion processing, and the like on the image data from the imaging unit 112 or the image data read from the memory 102 . Further, the image processing unit 113 performs predetermined arithmetic processing using captured image data, and the CPU 101 performs exposure control and distance measurement control based on the obtained arithmetic results. As a result, AE (auto exposure) processing, AWB (auto white balance) processing, and AF (auto focus) processing are performed. Further, the image processing unit 113 compares the feature amount of the face or person recorded in advance in the nonvolatile memory 103 with the feature amount calculated from the image data received from the imaging unit 112 . The image processing unit 113 identifies the coordinate position and range of the main subject in the image data from the compared feature amount and pixel coordinates.

The main subject detection function will now be described. The CPU 101 sends image data for face detection to the image processing unit 113 . Under the control of the CPU 101, the image processing unit 113 applies a horizontal bandpass filter to the image data. Under the control of the CPU 101, the image processing unit 113 applies a vertical bandpass filter to the processed image data. Edge components are detected from the image data by these horizontal and vertical bandpass filters. After that, the CPU 101 performs pattern matching on the detected edge components and extracts a candidate group. Then, the CPU 101 narrows down candidates that satisfy a preset condition from the group of extracted candidates. Then, the CPU 101 detects a person by associating the group of narrowed-down candidates with the person corresponding thereto, and by passing through a preset non-person condition filter. The CPU 101 outputs the person information according to the person detection result, and ends the process. At this time, the feature amount such as a person is stored in the memory 102 .

As described above, it is possible to detect the main subject (person) by performing image analysis on image data displayed in live view display or reproduced display, and extracting feature amounts of the image data.

The encoding processing unit 114 compresses the amount of image data by performing intra-frame prediction and inter-frame prediction coding on the input image data. The encoding processing unit 114 is, for example, H.264. 265 (ITU H.265 or ISO/IEC23008-2). A detailed description of the encoding processing unit 114 will be given later with reference to FIG.

The display control section 115 controls the display section 116 . The display control unit 115 performs resizing processing, color conversion processing, etc. so that the image can be displayed on the display unit 116 , and outputs an image signal to the display unit 116 . The display unit 116 is composed of a liquid crystal display, organic EL, or the like, and displays image signals sent from the display control unit 115 . The display unit 116 is a touch panel and also serves as an operation unit for receiving operations from the user.

Communication control section 117 is controlled by CPU 101 to generate a modulated signal conforming to a predetermined wireless communication standard such as IEEE 802.11 and outputs the modulated signal to communication section 118 . Further, the communication control unit 117 receives and decodes a modulated signal conforming to the wireless communication standard from the communication unit 118 , converts the analog signal into a digital signal, and notifies the CPU 101 of the digital signal. Further, the communication control unit 108 has a register for setting communication, and is controlled by the CPU 101 to adjust transmission/reception sensitivity during communication and transmit/receive with a predetermined modulation method. The communication unit 118 includes an antenna for outputting the modulated signal sent from the communication control unit 117 to the outside or receiving the modulated signal from the outside, an analog circuit, and the like.

A recording medium control unit 119 is a control unit for controlling the recording medium 120 , and receives a request from the CPU 101 to output a control signal for controlling the recording medium 120 . The recording medium 120 is composed of a detachable or built-in nonvolatile memory, magnetic disk, or the like for recording captured and encoded image data. Note that when the CPU 101 records on the recording medium 120, it saves as file data in a format adapted to the file system of the recording medium. The file data here refers to containers such as MP4 files (ISO/IEC 14496-14:2003) and MXF (Material Exchange Format) files.

An internal bus 130 is an internal bus for accessing the CPU 101 and the memory 102 from each processing unit. The encoding device 100 according to the embodiment has been described above.

As shown in FIG. 2, the decoding device 200 includes a CPU 201, a memory 202, a nonvolatile memory 203, an operation unit 204, a communication control unit 211, a communication unit 212, a decoding processing unit 213, a recording medium control unit 214, a recording medium 215, It has a display control unit 216 and an internal bus 220 .

The CPU 201 controls the operation of each unit of the decoding device 200 by executing computer programs stored in the nonvolatile memory 203 .

The memory 202 is a rewritable volatile memory, and temporarily records a computer program that controls the operation of each unit of the decoding device 200, information such as parameters related to the operation of each unit, information received by the communication unit 212, and the like. The nonvolatile memory 203 controls the operation of each unit of the decoding device 200 by executing the computer programs stored in the nonvolatile memory 203 .

An operation unit 204 provides a user interface for operating the decoding device 200 . The operation unit 204 has a power button, a play button, a stop button, and the like of the decoding device 200, and each button is configured by a switch, a touch panel, and the like. The CPU 201 controls the decoding device 200 according to user's instructions input via the operation unit 204 . Note that the operation unit 204 may control the decoding device 200 according to a remote control signal received from a remote controller (not shown) or a request notified from a portable terminal (not shown) via the communication control unit 211 .

The communication control unit 211 is controlled by the CPU 201 to generate a modulated signal conforming to a predetermined wireless communication standard such as IEEE802.11, and outputs the modulated signal to the communication unit 212 . Further, the communication control unit 211 receives and decodes a modulated signal conforming to the wireless communication standard from the communication unit 212 , converts the analog signal into a digital signal, and notifies the CPU 201 of the digital signal. Further, the communication control unit 212 has a register for setting communication, and is controlled by the CPU 201 to adjust the transmission/reception sensitivity during communication and perform transmission/reception according to a predetermined modulation method. The communication unit 212 includes an antenna for outputting the modulated signal sent from the communication control unit 211 to the outside or receiving the modulated signal from the outside, an analog circuit, and the like.

The decoding processing unit 213 performs decoding processing of the encoded image data (encoded stream) that has undergone intra-frame prediction and inter-frame prediction decoding. For example, H.I. H.265-encoded image data is decoded, and the decoded image data is temporarily stored in the memory 202 .

A recording medium control unit 214 is a control unit for controlling the recording medium 215 , and receives a request from the CPU 201 to output a control signal for controlling the recording medium 215 . A recording medium 215 is composed of a detachable or built-in nonvolatile memory, magnetic disk, or the like for recording encoded image data.

The display control unit 216 is a control unit for outputting image signals to the display device 300 . The display control unit 216 performs resizing processing, color conversion processing, and the like so that the image can be displayed on the display device 300 , and outputs an image signal to the display device 300 . The output to the display device 300 is performed via a cable such as HDMI (High Definition Multimedia Interface) (registered trademark).

The internal bus 220 is an internal bus for accessing the CPU 201 and the memory 202 by each processing unit and control unit.

<Details of Encoding Processing Unit 114>
Next, the configuration and processing flow of the encoding processing unit 114 in the encoding device 100 according to the embodiment will be described with reference to FIG. The encoding processing unit 114 uses H.264. It is assumed that encoding is performed according to the H.265 system.

The screen dividing unit 301 divides the screen into several screen sizes smaller than the screen size of the input image to be encoded. Specifically, based on the size of the display screen set by the user and the position information of the main subject, several slices or slices smaller than the screen size of the input image to be encoded are included so that the main subject is included. and determine the division size of the tiles. Note that the position of the main subject is acquired from the CPU 101 or the image processing unit 113 . Division into slices or tiles corresponds to division of the screen. Note that the number of slice divisions is determined in the horizontal direction by num_tile_columns_minus1, and in the vertical direction by num_tile_rows_minus1. In this embodiment, the tiles_fixed_structure_flag is set to 0 when the tile division method is unique for all bitstreams.

The screen size shall be defined by the number of pixels in the horizontal and vertical directions. However, since the screen size can be specified by the number of pixels on one side and the aspect ratio, this is not the case.

The screen dividing unit 301 assigns a number (hereinafter referred to as a tile number) to each area (hereinafter simply referred to as a tile) obtained by dividing the screen to be encoded into tiles at the determined coordinate position. Then, the tile number indicated by the tile number is associated with the position of the tile on the screen to be encoded. The size of each tile is determined by various information indicating the number of CTBs (Coding Tree Blocks).

Then, the screen dividing unit 301 outputs the predictive encoding condition for each tile to the predictive encoding method determining unit 302 during the encoding process. This predictive coding condition includes information defining a range for obtaining a predicted pixel block of a target pixel block (encoding target block) when encoding the target pixel block in a certain tile (details will be described later). ).

The predictive encoding method determination unit 302 determines the predictive method for each pixel block in the encoding target screen based on the screen division information and the predictive encoding conditions input from the screen division unit 301 . The predictive encoding method determination unit 302 determines a screen including simple intra-screen prediction or motion detection from the input image signal and the encoded pixel values read from the memory 102 that stores the decoded image of the encoded area. An evaluation value indicating coding efficiency is calculated by performing inter-prediction processing. Then, a prediction method that optimizes the coding efficiency is determined.

When the pixel block to be encoded is an I slice, the intra-screen prediction pixel block size and prediction mode are determined. In the case of P-slice or B-slice, the one with higher coding efficiency is selected from intra-frame prediction and inter-frame prediction. A frame in which all slices are I-slices is called an I-frame, a frame in which all slices are P-slices is called a P-frame, and a frame in which all slices are B-slices is called a B-frame.

Here, in the case of inter prediction, parameters for intra prediction coding such as intra prediction pixel block size and intra prediction mode are determined. In the case of inter-prediction, parameters for inter-predictive coding such as reference image frames, pixel block division patterns, motion vectors, etc. are determined.

The predictive encoding method determination unit 302 outputs the determined predictive encoding parameters to the predictive encoding processing unit 303 .

When encoding a pixel block of interest in a tile of interest in a frame to be encoded, the predictive encoding processing unit 303 stores data in the memory 102 according to the predictive encoding parameter determined by the predictive encoding method determination unit 302 . A prediction pixel block is generated from the encoded image read from the . Then, the predictive encoding processing unit 303 outputs the predictive residual block, which is the difference between the pixel block of interest and the predictive pixel block, to the orthogonal transform/quantization unit 305 . The prediction pixel block is also output to the local decoding unit 306 .

The orthogonal transform/quantization unit 305 performs orthogonal transform processing on the given prediction residual block. Further, the orthogonal transform/quantization unit 305 quantizes the coefficient obtained by orthogonal transform using a quantization step corresponding to a quantization parameter set by the code amount control unit 307, which will be described later. The quantized coefficients (quantized data) are supplied to the entropy coding section 308 and the local decoding section 306 .

The local decoding unit 306 generates prediction residual data by performing inverse quantization processing and inverse orthogonal transform processing on the input quantized data. Then, the local decoding unit 306 performs decoding processing by adding the prediction image input from the prediction encoding processing unit 303 to the prediction residual data, and stores the image data indicated by the obtained pixel block in the memory 102. do. The decoded image data stored in the memory 102 is used for subsequent intra-frame prediction processing. Furthermore, the decoded image data subjected to deblocking filter processing is held in the memory 102 . The decoded data after the deblocking filtering process held in the memory 102 is also used for the subsequent inter-screen prediction process.

The entropy coding unit 308 performs entropy coding processing by CABAC (context adaptive binary arithmetic coding) on the input quantized data in units of slices. The entropy coding unit 308 includes a binarization unit that converts input quantized data (multilevel data) into binary data, and a binary code that stores the binarized data generated by the binarization unit. has a modified data memory. Further, the entropy coding unit 308 calculates the occurrence probability of the binarized data according to the context, and performs arithmetic coding according to the held context calculation unit and the occurrence probability supplied by the context calculation unit. It has a chemical part. The encoded data is supplied to the multiplexing processing unit 309 in tile units, and the generated code amount is output to the code amount control unit 307 .

The code amount control unit 307 is a processing unit that controls the code amount of encoded data so as not to overflow or underflow the encoded picture buffer. The code amount control unit 307 generates a quantization parameter for the subsequently input frame based on the generated code amount after entropy coding supplied from the entropy coding unit 308 , and sends it to the orthogonal transform/quantization unit 305 . supply.

At the start of image encoding, the multiplexing processing unit 309 stores, in the file header, information such as the size of the frame to be encoded, the division position of each tile, the set screen size, the tile number, and Stores information indicating the tile number where the main subject exists.

Note that if it is common for both the encoding and decoding apparatuses that the tile numbers are assigned in raster scan order in units of tiles, the tile numbers may be unnecessary.

<Example of tile division by screen division unit 301>
FIG. 4 shows an example of a tile configuration according to this embodiment.

FIG. 4(a) is a diagram showing an example of a tile configuration at a certain time for input image data. FIG. 4A is composed of nine tiles, tiles A1 to A8 and tile B. FIG. As shown in FIG. 4A, the screen dividing unit 301 determines the position of tile B including the person who is the main subject, and determines other tile configurations according to the position of tile B. As shown in FIG. Specifically, when the position of tile B is determined, tile A is determined by extending the line segments representing the four sides of tile B as boundary lines for dividing the main screen. A tile including a main subject, such as tile B, is hereinafter referred to as a "main tile". An input image is defined as a "main image", and a screen smaller than the main screen represented by a "main tile" in the main image is defined as a "sub-image".

FIG. 4(b) is a diagram showing an example of a tile configuration in a frame subsequent in time series to FIG. 4(a) in the embodiment. As shown in FIG. 4B, the screen dividing unit 301 moves the position of the tile B (main tile) due to the movement of the person who is the main subject. The screen dividing unit 301 also changes the size and position of tile A1, tile A2, tile A3, tile A4, and tile A5 according to the position of tile B, and tiles A6 to 8 do not exist in the illustrated case.

Note that the size of the tile B is the same size in FIGS. 4(a) and 4(b). When the size of the main screen is, for example, 8192×4320 pixels, tile B (main tile) is 1/4 of that size, such as 4096×2160 pixels. It may be set to a determined commonly used screen size.

Also, when the main subject moves further to the left from the state of FIG. 4B and tile B moves to the lower left corner position of the main screen, the total number of divided tiles becomes the minimum "4". That is, the number of tiles into which the main screen is divided varies in the range of 4 to 9 depending on the position of the main tile. Note that the encoding is performed in tile units. An increase in the number of tiles is a factor in lowering coding efficiency. Therefore, when the main screen is divided into tiles under the above conditions, if the size (area) of the tiles is equal to or less than a preset threshold value and the rectangular shape can be maintained even when the tiles are connected to each other, You may merge those tiles into one. For example, in the state shown in FIG. 4A, the sizes of the tiles A6, A7, and A8 are equal to or less than the threshold value, and the rectangular shape can be maintained even if the tiles A6, A7, and A8 are connected to each other. You may make it decide.

<Overall Processing Flow of Encoding Apparatus 100>
An example of overall processing in the encoding device 100 is shown in the flowchart of FIG. Note that the control program of the flowchart in this embodiment is stored in the nonvolatile memory 103 . Then, while the encoding device 100 is powered on, the CPU 101 expands the program stored in the nonvolatile memory 103 into the memory 102 and executes it. The control program of the encoding device 100 in this embodiment may repeatedly execute the process periodically.

In S<b>501 , the CPU 101 determines the screen size of the sub-image that is smaller than the screen size of the image captured by the imaging unit 112 . By operating the operation unit 104, the CPU 101 uses the screen size set by the user as a setting value, and selects a secondary image having a smaller screen size than the main screen including the main subject, from the setting information stored in the nonvolatile memory 103. Determines the screen size of the image. The CPU 101 advances the processing in this flowchart from S501 to S502.

In S502, the CPU 101 designates a main subject. The CPU 101 determines a main subject from a plurality of pieces of main subject information that are selected by the user through an operation from the operation unit 104 and stored in advance in the nonvolatile memory 103 . Further, when main subject information does not exist in the nonvolatile memory 103, the CPU 101 superimposes a display prompting a touch operation on the live view image on the display unit 116, and displays the subject (object) touched by the user as the main subject. and Alternatively, a subject photographed such that the main subject falls within the frame for registering the subject superimposed on the display unit 116 is registered in the nonvolatile memory 103 as the main subject. The CPU 101 advances the processing in this flowchart from S502 to S503.

In S503, the CPU 101 determines whether or not the moving image recording start button of the operation unit 104 has been pressed. When the CPU 101 determines that the moving image recording start button has been pressed (YES in S503), the processing in this flowchart proceeds from S503 to S504. Further, when the CPU 101 determines that the moving image recording start button has not been pressed (NO in S503), the processing in this flowchart continues in S503.

In S<b>504 , the CPU 101 performs settings before encoding processing by the encoding processing unit 114 . The CPU 101 writes main subject information to the memory 102 for reference by the screen division unit 301 of the encoding processing unit 114 . The CPU 101 also sets whether to record only the coded image of the main screen in a file, or to record the coded images of two screens, the main screen and the sub-screen, in separate files. Furthermore, the CPU 101 performs various register settings and the like for performing encoding processing, makes the encoding processing unit 114 operable, and advances the processing in this flowchart from S504 to S505.

In S505, the CPU 101 notifies the encoding processing unit 114 of the address of the image data stored in the memory 102 by the image processing unit 113, and instructs execution of encoding processing. Note that detailed encoding processing in the encoding processing unit 114 will be described later using FIG. Then, the CPU 101 advances the processing in this flowchart from S505 to S506.

In S506, the CPU 101 causes the recording medium control unit 119 to control the recording medium 120 to sequentially record the encoded image data and the syntax information (additional information) generated by the encoding processing unit 114 in S505. When only the main screen is recorded in the file, the encoded image data of the main screen and the syntax information of the main screen are recorded. When the main screen and sub-screen are recorded in separate files, the encoded image data for the main screen and the syntax information for the main screen, and the encoded image data for the sub-screen and syntax information for the sub-screen are stored in separate files. Record so that The CPU 101 advances the processing in this flowchart from S506 to S507.

In S507, the CPU 101 determines whether or not shooting has ended. The CPU 101 receives a signal from the operation unit 104 according to the user's operation, and determines that the shooting has ended. When the CPU 101 determines that the shooting has ended (YES in S508), the processing in this flowchart proceeds from S507 to S508. When the CPU 101 determines that the shooting has not ended (NO in S507), the processing in this flowchart returns from S507 to S505.

In S<b>508 , the CPU 101 performs encoding processing termination processing on the encoding processing unit 114 . Specifically, an end value is set in a register (not shown) in the encoding processing unit 114 . The CPU 101 advances the processing in this flowchart from S508 to S509.

In S509, the CPU 101 controls the recording medium control unit 119 to complete generation of the file recorded on the recording medium 120, and makes it readable as one file. The CPU 101 ends the processing in this flowchart.

Although the file is recorded on the recording medium 120 in the flowchart shown in FIG.

<Tile Configuration Determination Processing and Encoding Processing of Encoding Apparatus 100>
FIG. 6 shows an example of tile configuration determination and encoding processing in the encoding device 100 . This processing is a detailed description of the processing of the encoding processing unit 114, which is executed in response to a request from the CPU 101 in S505 described above with reference to FIG.

In S<b>601 , the encoding processing unit 114 determines whether or not the frame of interest is the frame at the update timing of the tile configuration. This is because the tile division positions are determined not for each frame but at certain frame intervals. In the embodiment, when the predictive encoding method determining unit 302 determines to encode the frame of interest as an I picture, it is assumed that the tile configuration is updated. Note that if the accuracy is to be improved, not only the I picture but also the P and B pictures that refer to the I picture may be used as the update timing.

When the encoding processing unit 114 determines that it is time to update the tile configuration of the frame of interest (YES in S601), the processing in this flowchart proceeds from S601 to S602. If the encoding processing unit 114 determines that it is not the tile configuration update timing for the frame of interest (NO in S601), the processing advances to S611 to perform encoding based on the same tile configuration as the previous frame.

In S602, the encoding processing unit 114 acquires the main subject range information NEW (information indicating the size and position of the main subject in the frame of interest) written at a predetermined address in the memory 102 by the image processing unit 113 or the CPU 101. The encoding processing unit 114 advances the processing in this flowchart from S602 to S603. Note that the encoding processing unit 114 may detect the range of the main subject using the screen division unit 301 . Also, the range of the main subject is assumed to be the circumscribed rectangle of the main subject.

In S<b>603 , the encoding processing unit 114 determines whether the range of the main subject is larger than the image size of the sub-image by the screen division unit 301 . In an embodiment, it is determined that the extent of the main subject is larger than the image size of the sub-image if either the number of pixels in the horizontal or vertical direction defining the extent of the main subject is larger than that of the sub-image. If the range of the main subject is larger than the image size of the sub-image (YES in S603), the encoding processing unit 114 advances the processing in this flowchart from S603 to S608. If the range of the main subject is equal to or smaller than the image size of the sub-image (NO in S603), the encoding processing unit 114 advances the processing in this flowchart from S603 to S604.

In S604, the encoding processing unit 114 determines whether the screen dividing unit 301 has moved the position of the main subject. The encoding processing unit 114 determines whether or not the position of the main subject range information NEW stored in the memory 102 and the position of the main subject range information OLD in the picture (previous I-picture) at the previous tile configuration update timing have been moved. to judge whether When the encoding processing unit 114 determines that the main subject has moved (YES in S604), the processing in this flowchart proceeds from S604 to S605. When the encoding processing unit 114 determines that the main subject has not moved (NO in S604), the encoding processing unit 114 determines not to change the tile configuration, and advances the processing in this flowchart from S604 to S611. Note that the encoding processing unit 114 may determine whether or not the main subject has moved based on the motion vector amount of the image block in which the main subject exists, which is obtained by the predictive encoding processing unit 303 .

In S<b>605 , the encoding processing unit 114 uses the screen dividing unit 301 to calculate the movement amount from the main subject range information NEW and the main subject range information OLD. The encoding processing unit 114 compares the amount of movement with a predetermined value prerecorded in the nonvolatile memory 103, and if the amount of movement is equal to or less than the predetermined value, advances the processing in this flowchart from S605 to S607. If the movement amount is larger than the predetermined value recorded in the nonvolatile memory 103, the encoding processing unit 114 advances the processing in this flowchart from S605 to S606.

In S606, the encoding processing unit 114 converts the position of the main tile by the screen dividing unit 301 into a new position obtained by moving the sub-image by a predetermined value in the direction from the main subject range information OLD to the main subject range information NEW. Position the main tile. Then, the encoding processing unit 114 determines the positions of the tiles other than the main tile according to the positions of the main tiles, as shown in FIG. The encoding processing unit 114 advances the processing in this flowchart from S606 to S611.

In S607, the encoding processing unit 114 uses the screen division unit 301 to determine the main tile according to the position of the main subject range information NEW. For example, a typical process is to align the center position of the main subject with the center position of the subject tile. Also, the encoding processing unit 114 determines the positions of tiles other than the main tile according to the positions of the main tiles, as shown in FIG. The encoding processing unit 114 advances the processing in this flowchart from S606 to S611.

In S<b>608 , the encoding processing unit 114 determines whether or not the screen division unit 301 has generated a moving image of the sub-image as a separate file. The encoding processing unit 114 confirms the register set by the CPU 101 . If the encoding processing unit 114 determines in S608 not to generate the moving image of the sub-image as a separate file, the processing in this flowchart proceeds from S608 to S610. When the encoding processing unit 114 determines not to generate the moving image of the sub-image as a separate file, the processing in this flowchart proceeds from S608 to S609.

In S<b>609 , the encoding processing unit 114 causes the screen dividing unit 301 not to generate the main tile. That is, the encoding processing unit 114 determines the entire main image as one tile without dividing the main image into a plurality of tiles. The encoding processing unit 114 advances the processing in this flowchart from S606 to S611.

In S610, the encoding processing unit 114 requests the image processing unit 113 to perform reduction processing. The image processing unit 113 reduces the main image to the image size of the sub-image according to the request and saves it in the memory 102 . Note that the main image is left in the memory 102 as it is. The encoding processing unit 114 advances the processing in this flowchart from S610 to S611.

In S611, the encoding processing unit 114 stores the main subject range information NEW in the memory 102 as the main subject range information OLD. The encoding processing unit 114 advances the processing in this flowchart from S611 to S612.

In S612, the encoding processing unit 114 sets a preset small value for the main tile as a value in the quantization table set by the code amount control unit 307, and sets a smaller value for the tiles other than the main tile. Set a large value. The encoding processing unit 114 advances the processing in this flowchart from S612 to S613.

In S613, the encoding processing unit 114 performs encoding processing on the main image. The encoding process is as described above with reference to FIG. 3, but encoding of the main tile is performed without referring to areas outside the main tile regardless of the type of frame of interest. This is so that the main tile can be decoded independently. As with the determination process in S608, when the moving image of the sub-image is generated in a separate file, the encoding process for the sub-image is separately performed. If the sub-image exists as separate data due to the reduction processing in S610, the encoding processing unit 114 performs encoding processing on the reduced image as the sub-image.

As will be described later, in order for the decoding device 200 to be able to decode the main tile alone, it is necessary to know the storage address position of the encoded data of the main tile. Therefore, the encoding processing unit 114 cumulatively adds the code amount of the encoded data of each tile when encoding is performed in raster scan order from the tile at the upper left corner of the main image. Then, the result of cumulative addition when the decoding device 200 starts the encoded data of the main tile (more precisely, the result of cumulative addition +1) is included in the syntax information of the frame of interest as the start address of the encoded data of the main tile. make it

After completing the encoding process for the frame of interest described above, the encoding processing unit 114 advances the process in this flowchart from S613 to S614.

In S<b>614 , the encoding processing unit 114 adds information indicating the tile configuration to the syntax information of the encoded main image data, and records the information in the memory 102 . Also, the presence of the sub-image and the range information of the main tile are added to the syntax information of the main image and stored in the memory 102 . When the moving image is generated in a separate file, information indicating the tile configuration is added to the syntax information of the encoded sub-image data and recorded in the memory 102 . In addition to the syntax, the existence of the sub-image and the range information of the main tile may be included in the container information when generating the file. The encoding processing unit 114 ends the processing in this flowchart in S614.

Note that the processing described above with reference to FIG. 6 may be performed by the CPU 101 . Also, although the process is described as generating both the main image and the sub-image, only either the main image or the sub-image may be generated. Also, although the coding apparatus 100 described with reference to FIGS. 5 and 6 is configured to perform processing using tiles, it is possible to configure slices instead of tiles as long as it is a horizontally long sub-image that can be configured in the slice direction. do. Furthermore, slices may be included in tiles and main tiles.

<Overall Processing Flow of Decoding Device 200>
An example of overall processing when decoding and reproducing a sub-image (main tile) in the decoding device 200 is shown. This is easily understood by considering the case where the display resolution of the display device 300 is the same as that of the main tiles. Note that the control program of the flowchart in the present embodiment is executed by the CPU 201 after expanding the program stored in the nonvolatile memory 203 into the memory 202 while the decoder 200 is powered on. The control program of the decryption device 200 in this embodiment may periodically and repeatedly execute the process. It is assumed that this control program is started when the user presses the reproduction start button of the operation unit 204 . Further, the present control program controls the recording medium 215 by the recording medium control unit 214 to read the file information, store it in the memory 202, and then execute the process. Alternatively, the control program may control the communication unit 212 by the communication control unit 211 to receive the file information, store it in the memory 202, and then execute the process.

In S<b>701 , the CPU 201 acquires tile information in the frame to be reproduced from the container information or syntax information of the file and records it in the memory 202 . The CPU 201 advances the processing in this flowchart from S701 to S702.

In S702, the CPU 201 determines whether or not there is a sub-image in the frame to be reproduced from the tile information acquired in S701. If the sub-image exists (YES in S702), the CPU 201 advances the processing in this flowchart from S702 to S703. If the sub-image does not exist (NO in S702), the CPU 201 advances the processing in this flowchart from S702 to S704.

In S703, the CPU 201 controls the decoding processing unit 213 using the storage address of the encoded data of the sub-image included in the syntax information to perform decoding processing only for the main tiles forming the sub-image. Data decoded by the decoding processing unit 213 is stored in the memory 202 . The CPU 201 advances the processing in this flowchart from S703 to S706.

In S704, the CPU 201 controls the decoding processing unit 213 to perform decoding processing of the encoded image data of the main image. The main image obtained by decoding by the decoding processing unit 213 is stored in the memory 202 . The CPU 201 advances the processing in this flowchart from S704 to S705. In S<b>705 , the CPU 201 reduces the image data saved in the memory 202 to the image size of the sub-image and saves it in the memory 202 . In an embodiment, the main image is 8192×4320 pixels and the sub-image is 4096×2160 pixels. Therefore, the CPU 201 performs processing to obtain one pixel of the reduced image from, for example, 2×2 pixels of the main image. For example, the CPU 201 may set one pixel at a preset position within 2×2 pixels in the main image or the average value of the 2×2 pixels as one pixel of the reduced image. The CPU 201 advances the processing in this flowchart from S704 to S705.

In S<b>706 , the CPU 201 controls the display control unit 216 to output the image data obtained by the decoding process stored in the memory 202 to the display device 300 . Note that the CPU 201 converts the image into an image format that can be displayed by the display device 300 and outputs the image. The CPU 201 advances the processing in this flowchart from S706 to S707.

In step S<b>707 , the CPU 201 determines whether the user has pressed the playback stop button of the operation unit 204 . When the user presses the playback stop button of the operation unit 204 (YES in S707), the CPU 201 ends the processing in this flowchart in S707. If the user has not pressed the playback stop button of the operation unit 204 (NO in S707), the CPU 201 returns the processing in this flowchart from S707 to S702.

Although the decoding apparatus 200 described with reference to FIG. 7 is configured to perform processing using tiles, it is assumed that horizontally long sub-images that can be configured in the slice direction may be configured using slices instead of tiles. Additionally, slices may be included within tiles and main tiles.

As described above, according to the encoding apparatus 100 of the embodiment, it is possible to generate encoded image data capable of decoding not only the main image but also only the sub-image area including the main subject in the main image. Also, the decoding device 200 having a display device with the same resolution as the sub-image can decode and reproduce only the sub-image.

In the above embodiment, the frame to be encoded is 8192×4320 pixels, and the size of the sub-image (main tile) is 4096×2160 pixels. The resolution may be set according to the resolution of the display device to be used. As a result, it becomes possible to perform partial decoding of a moving image specialized for the display device owned by the user.

[Second embodiment]
As a second embodiment, the screen size of the sub screen is the same as the screen size of the main tile, the screen size of the main tile is fixed, and the tile position is variable. H.265 encoding and decoding of the main tile to trim and display the screen. When the position of the main tile changes, the trimming position is gradually changed from within the decoded screen. Note that the configuration of the device is assumed to be the same as that of the above-described embodiment (first embodiment), and the description thereof is omitted.

H. In H.265, tile boundary positions and tile sizes are in units of CTU (Coding Tree Unit). Possible settings for CTU are 16×16, 32×32, and 64×64 pixels. Therefore, when only the main tile is decoded and displayed, if the tile position changes between frames, the display position changes by at least 16 pixels between frames, which may result in an unnatural moving image.

With reference to FIG. 8, an example in which tile positions change between frames and an unnatural moving image is produced will be described. FIG. 8 shows images for four consecutive frames N to N+3, reference numerals 801 to 804 in FIG. 8 indicate main images, and reference numerals 811 to 814 indicate subimages (main tiles). In the N and N+1 frames, the horizontal size of the tile to the left of the sub-image was L1, whereas in the N+2 and N+3 frames, the horizontal size of the tile to the left was L2 (L2< L1). In other words, the position of the sub-image is different between the N+1 frame and the N+2 frame. If only the sub-image is decoded and reproduced in this state, the main subject and background are displayed at positions that are suddenly shifted between the sub-image of the N+1th frame and the sub-image of the N+2th frame, resulting in an unnatural moving image. put away.

Therefore, in this embodiment, sub-images (main tiles) 911 to 914 obtained by decoding continuous main images 901 to 904 are trimmed and displayed as shown in FIG. At this time, when the tile position changes, by gradually changing the trimming position, the unnaturalness when the tile position changes is reduced. 9 will be described later together with the flowchart of FIG. Here, let the tile position of the (N+1)th frame in FIG. 9 be the tile position α, and the tile position of the (N+2)th frame be the tile position β.

The overall processing flow of the encoding apparatus 100 is the same as that of the first embodiment, and description thereof is omitted. However, in the second embodiment, the screen size of the sub-image (main tile) is fixed, and only the main screen is recorded in the file.

<Tile Configuration Determination and Encoding Processing Flow of Encoding Apparatus 100>
FIG. 10 is a flow chart showing tile configuration determination and encoding processing in the encoding device 100 according to the second embodiment. This processing is a detailed description of the processing of the encoding processing unit 114, which is executed in response to a request from the CPU 101 in S505 described above with reference to FIG.

In S1001, the encoding processing unit 114 determines whether it is time to update the tile. As in the first embodiment, the tile update timing is set when the predictive encoding method determination unit 302 determines that the frame of interest is to be coded as an I picture. Of course, in addition to this, the coding timing of the P and B pictures that refer to the I picture may be used as the update timing. When it is determined that the tile configuration should be updated for the frame of interest, the encoding processing unit 114 advances the processing in this flowchart from S1001 to S1002. When the encoding processing unit 114 determines that it is not the update timing, the processing in this flowchart proceeds from S1001 to S1006.

In S<b>1002 , the encoding processing unit 114 acquires main subject range information NEW written at a predetermined address in the memory 102 by the image processing unit 113 or the CPU 101 . The encoding processing unit 114 advances the processing in this flowchart from S1002 to S1003. Note that the encoding processing unit 114 may detect the range of the main subject using the screen division unit 301 .

In S1003, the encoding processing unit 114 determines whether the screen dividing unit 301 has moved the position of the main subject. The encoding processing unit 114 calculates the difference between the position indicated by the main subject range information NEW stored in the memory 102 and the position indicated by the main subject range information OLD at the previous update timing (previous I-picture encoding timing). Based on the above, it is determined whether or not it has moved. When the encoding processing unit 114 determines that the main subject has moved, the processing in this flowchart proceeds from S1003 to S1004. If the encoding processing unit 114 determines that the main subject has not moved, the encoding processing unit 114 determines not to change the tile configuration, and advances the processing in this flowchart from S1003 to S1006. Note that the encoding processing unit 114 may determine whether or not the main subject has moved based on the motion vector amount of the image block in which the main subject exists, which is obtained by the predictive encoding processing unit 303 .

In S<b>1004 , the encoding processing unit 114 uses the screen division unit 301 to calculate the amount of movement from the main subject range information NEW and the main subject range information OLD. Then, if the movement amount is equal to or less than the predetermined value recorded in the nonvolatile memory 103, the encoding processing unit 114 advances the processing in this flowchart from S1004 to S1006. If the movement amount is greater than the predetermined value recorded in the nonvolatile memory 103, the encoding processing unit 114 advances the processing in this flowchart from S1004 to S1005.

In S1005, the encoding processing unit 114 determines the position where the sub-image is moved by a predetermined value in the direction from the main subject range information OLD to the main subject range information NEW as the updated main tile position. Then, the encoding processing unit 114 determines the positions of the tiles other than the main tile according to the position of the main tile, as described with reference to FIG. The encoding processing unit 114 advances the processing in this flowchart from S1005 to S1006.

In S1006, the encoding processing unit 114 stores the main subject range information NEW in the memory 102 as the main subject range information OLD. The encoding processing unit 114 advances the processing in this flowchart from S1006 to S1007.

In S1007, the encoding processing unit 114 sets a preset small value for the main tile as a value in the quantization table set by the code amount control unit 307, and sets a small value for the tiles other than the main tile. set a value greater than that. The encoding processing unit 114 advances the processing in this flowchart from S1007 to S1008.

In S1008, the encoding processing unit 114 performs encoding processing on the main image. The encoding process is as described above with reference to FIG. The encoding processing unit 114 stores the encoded image data in the memory 102 . The encoding processing unit 114 advances the processing in this flowchart from S1008 to S1009.

In S<b>1009 , the encoding processing unit 114 adds information indicating the tile configuration to the syntax information of the encoded main image data, and records the information in the memory 102 . Also, the presence of the sub-image and the range information of the main tile are added to the syntax information of the main image and stored in the memory 102 . In addition to the syntax, the range information of the sub-image (main tile) may be included in the container information when generating the file. The encoding processing unit 114 ends the processing in this flowchart in S1009.

Note that the processing described above with reference to FIG. 10 may be performed by the CPU 101 . It is also assumed that slices may be included in tiles and main tiles.

<Overall Processing Flow of Decoding Device 200>
FIG. 11 shows processing when decoding and reproducing a sub-image in the decoding device 200 according to the second embodiment. Note that the control program of the flowchart in the present embodiment is executed by the CPU 201 after expanding the program stored in the nonvolatile memory 203 into the memory 202 while the decoder 200 is powered on. The control program of the decryption device 200 in this embodiment may periodically and repeatedly execute the process. It is assumed that this control program is started when the user presses the reproduction start button of the operation unit 204 . Further, the present control program controls the recording medium 215 by the recording medium control unit 214 to read the file information, store it in the memory 202, and then execute the process. Alternatively, the control program may control the communication unit 212 by the communication control unit 211 to receive the file information, store it in the memory 202, and then execute the process.

In S<b>1101 , the CPU 201 acquires tile information in the frame to be reproduced from the container information or syntax information of the file and records it in the memory 202 . The CPU 201 advances the processing in this flowchart from S1101 to S1102.

In S1102, the CPU 201 determines whether or not there is a sub-image in the frame to be reproduced from the tile information acquired in S1101. When the CPU 201 determines that the sub-image exists, the process in this flowchart proceeds from S1102 to S1105. When the CPU 201 determines that the sub-image does not exist, the process in this flowchart proceeds from S1102 to S1103.

In S<b>1103 , the CPU 201 controls the decoding processing unit 213 to perform decoding processing of image data obtained by encoding the main image. Data decoded by the decoding processing unit 213 is stored in the memory 202 . The CPU 201 advances the processing in this flowchart from S1103 to S1104.

In S<b>1104 , the CPU 201 reduces the image data saved in the memory 202 to the image size of the sub-image and saves it in the memory 202 . The CPU 201 advances the processing in this flowchart from S1104 to S1108.

In S1105, the CPU 201 controls the decoding processing unit 213 to decode only the main tiles forming the sub-image. Data decoded by the decoding processing unit 213 is stored in the memory 202 . The CPU 201 advances the processing in this flowchart from S1105 to S1106.

In S1106, the CPU 201 determines the trimming position of the decoded image. The image trimming position determination process will be described later with reference to FIG. The CPU 201 advances the processing in this flowchart from S1106 to S1107.

In S<b>1107 , the CPU 201 trims the image using the image trimming position information determined in S<b>1106 and the image processing unit 113 , and stores the trimmed image in the memory 202 . The CPU 201 advances the processing in this flowchart from S1107 to S1108.

In S<b>1108 , the CPU 201 controls the display control unit 216 to output the image data saved in the memory 202 to the display device 300 . Note that the CPU 201 converts the image into an image format that can be displayed by the display device 300 and outputs the image. The CPU 201 advances the processing in this flowchart from S1108 to S1109.

In S<b>1109 , the CPU 201 determines whether or not the user has pressed the playback stop button of the operation unit 204 . When the CPU 201 determines that the user has pressed the playback stop button of the operation unit 204, the processing in this flowchart ends in S1109. When the CPU 201 determines that the user has not pressed the playback stop button of the operation unit 204, the process in this flowchart proceeds from S1109 to S1110.

In S1110, the CPU 201 determines whether reproduction of all frames has ended. When the CPU 201 determines that reproduction of all frames has ended, the processing in this flowchart ends in S1110. If the CPU 201 determines that the processing of all frames has not been completed, the processing in this flowchart returns from S1110 to S1102.

Although the decoding apparatus 200 described with reference to FIG. 11 is configured to perform processing using tiles, it is also possible to configure sub-images that are oblong sub-images that can be configured in the slice direction using slices instead of tiles. Furthermore, slices may be included in tiles and main tiles.

<Processing to determine trimming position>
The flow of the trimming position determination process according to the second embodiment will be described with reference to the flowchart of FIG.

In S1201, the CPU 201 determines whether the frame of interest to be decoded is the first frame. When the CPU 201 determines that the frame of interest is the first frame, the processing in this flowchart proceeds from S1201 to S1202. If the CPU 201 determines that the frame of interest to be decoded is not the first frame, the CPU 201 advances the process from S1201 to S1204 in the flowchart.

In S1202, the CPU 201 sets the trimming position change flag stored in the memory 202 to 0, and advances the processing in this flowchart from S1202 to S1203.

In S1203, the CPU 201 determines the center portion of the sub-image as the border position for trimming, like the sub-images 911 and 912 in FIG. 9, and ends the processing in this flowchart in S1203.

In S<b>1204 , the CPU 201 determines whether the trimming position change flag is 1 or not. When the CPU 201 determines that the trimming position change flag is 1, the processing in this flowchart proceeds from S1204 to S1208. If the CPU 201 determines that the trimming position change flag is not 1, it advances the processing in this flowchart from S1204 to S1205.

In S1205, the CPU 201 determines whether the position of the main tile of the current frame has changed from the position of the main tile of the previous frame based on the syntax information regarding the tile. When the CPU 201 determines that the position of the main tile has changed, the process in this flowchart proceeds from S1205 to S1206. If the CPU 201 determines that the position of the main tile has not changed, it advances the processing in this flowchart from S1204 to S1203.

In S1206, the CPU 201 sets the trimming position change flag to 1, and advances the processing in this flowchart from S1206 to S1207.

In S1207, the CPU 201 sets the trimming counter to 0, and advances the processing in this flowchart from S1207 to S1208.

In S1208, the CPU 201 determines the trimming position according to the value of the trimming counter, and advances the processing in this flowchart from S1208 to S1209. The trimming position varies from the center of the tile position α to the center of the tile position β when considering the position based on the main screen, like the sub-images 913 and 914 in FIG. 9, according to the value of the trimming counter. Let it change gradually. The trimming position according to the value of the trimming counter may be variable according to the frame rate or moving speed of the main subject.

In S1209, the CPU 201 increments the trimming counter and advances the processing in this flowchart from S1209 to S1210.

In S1210, the CPU 201 determines whether the value of the trimming counter is greater than or equal to the threshold. When the CPU 201 determines that the value of the trimming counter is equal to or greater than the threshold, the processing in this flowchart proceeds from S1210 to S1211. Further, when the CPU 201 determines that the value of the trimming counter is less than the threshold value, the processing in this flowchart ends in S1210.

In S1211, the CPU 201 sets the trimming position change flag to 0, and ends the processing in this flowchart in S1211.

As described above, according to the second embodiment, the sub-image within the main image can be reproduced as a more natural moving image than in the first embodiment.

(Other embodiments)
The encoding device 100 according to the present invention is not limited to the encoding device 100 described in this embodiment. Also, the decoding device 200 according to the present invention is not limited to the decoding device 200 described in this embodiment. For example, the encoding device 100 and decoding device 200 according to the present invention can also be realized by a system composed of a plurality of devices.

Also, various processes and functions described in this embodiment can be realized by a computer program. In this case, the computer program according to the present invention can be executed by a computer (including a CPU, etc.), and implements various functions described in this embodiment.

It goes without saying that the computer program according to the present invention may implement the various processes and functions described in the present embodiment using an OS (Operating System) running on the computer.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

DESCRIPTION OF SYMBOLS 100... Encoding apparatus, 101... CPU, 102... Memory, 103... Nonvolatile memory, 104... Operation unit, 111... Lens, 112... Imaging unit, 113... Image processing unit, 114... Encoding processing unit, 115... Display Control unit 116 Display unit 117 Communication control unit 118 Communication unit 119 Recording medium control unit 120 Recording medium 130 Internal bus 200 Decoding device 201 CPU 202 Memory 203 Non-volatile memory 204 operation unit 210 communication control unit 211 communication unit 212 decryption processing unit 213 recording medium control unit 214 recording medium 215 display control unit 220 internal bus, 300... Display device 301... Screen dividing unit 302... Predictive encoding method determining unit 303... Predictive encoding processing unit 305... Orthogonal transform quantizing unit 306... Local decoding unit 307... Code amount control unit 308 ... entropy coding unit, 309 ... multiplexing processing unit

Claims (11)

An encoding device for encoding a moving image,
Determination means for determining the position of a sub-image area of a preset size containing the main subject from the frame of interest to be encoded;
dividing means for dividing the frame of interest into a plurality of regions based on boundaries of the sub-image regions determined by the determining means;
encoding means for encoding an image of each region obtained by dividing by the dividing means;
The encoded data of the frame of interest is obtained from the encoded data obtained by the encoding means, the position of the area of the sub-image in the frame of interest, and additional information indicating the storage address of the encoded data of the sub-image. and generating means for generating
The encoding means are
For the area of the sub-image in the frame of interest, encoding is performed with reference to the area of the sub-image or the sub-image area in the previously encoded frame,
Encoding an area other than the area of the sub-image in the frame of interest with reference to the frame of interest or a previously encoded frame,
The determining means determines the position of the main subject and the position of the sub-image region using update timing when the frame of interest to be coded is an I picture or a picture referring to the I picture. death,
When the frame of interest to be encoded is not at the update timing, the encoding means performs encoding assuming that the position of the area of the sub-image is the same as that of the previous frame. Device. The determining means is
(1) if the main subject in the frame of interest has not moved from the position of the main subject at the previous update timing, without changing the position of the sub-image area;
(2) If the main subject in the frame of interest has moved from the position of the main subject at the previous update timing and the amount of movement is equal to or less than a preset threshold value, the position of the main subject in the frame of interest is determined. determining a location of a region of said sub-image;
(3) When the main subject in the frame of interest has moved from the position of the main subject at the previous update timing, and the amount of movement exceeds the threshold, 2. The encoding device according to claim 1, wherein a position obtained by moving the sub-image by a predetermined value in a direction toward the position of the object is set as the position of the area of the sub-image after updating. setting means for setting characteristics of the main subject;
detection means for detecting, within a frame to be encoded, a region of the main subject having the characteristics of the main subject set by the setting means;
3. The encoding apparatus according to claim 1, wherein the determining means determines the area of the sub-image when the size of the area of the main subject is equal to or smaller than the size of the sub-image. The determination means determines the frame of interest as one region without dividing the frame of interest into the plurality of regions when the size of the region of the main subject is larger than the size of the sub-image. 4. Encoding device according to claim 3. 4. The method according to claim 3, wherein the setting means includes means for determining an object selected by the user in the moving image displayed on the display means as the main subject and extracting features of the determined main subject. Encoding apparatus as described. 6. The encoding apparatus according to any one of claims 1 to 5, further comprising second setting means for setting the size of the sub-image area. The encoding means performs encoding using a first quantization parameter when encoding the area of the sub-image, and uses the first quantization parameter when encoding an area other than the area of the sub-image. The encoding device according to any one of claims 1 to 6, wherein encoding is performed using a second quantization parameter larger than the parameter. further comprising designating means for designating generation of a separate file of encoded data of the region of the sub-image;
The generation means, when the designation means designates to generate separate files, generates a file representing the encoded data obtained from the frame to be encoded and the encoded data obtained from the area of the sub-image. 8. An encoding device as claimed in any one of claims 1 to 7, characterized in that it generates a file representing the . 9. The encoding apparatus according to any one of claims 1 to 8, further comprising imaging means for capturing an image to be encoded. A control method for an encoding device that encodes a moving image,
a determination step of determining the position of a sub-image region of a preset size including the main subject from the frame of interest to be encoded;
a dividing step of dividing the frame of interest into a plurality of regions based on boundaries of regions of the sub-image determined in the determining step;
an encoding step of encoding an image of each region obtained by dividing in the dividing step;
The encoded data of the frame of interest is obtained from the encoded data obtained in the encoding step, the position of the area of the sub-image in the frame of interest, and additional information indicating the storage address of the encoded data of the sub-image. and a generating step of generating,
The encoding step includes:
For the area of the sub-image in the frame of interest, encoding is performed with reference to the area of the sub-image or the sub-image area in the previously encoded frame,
Encoding an area other than the area of the sub-image in the frame of interest with reference to the frame of interest or a previously encoded frame,
The determining step determines the position of the main subject and the position of the sub-image region, using update timing when the frame of interest to be encoded is an I picture or a picture that refers to an I picture. death,
In the encoding step, when the frame of interest to be encoded is not at the update timing, encoding is performed assuming that the position of the region of the sub-image is the same as that of the previous frame. How to control the device. A program that is read and executed by a computer to cause the computer to perform each step of the method according to claim 10.

Images