JP7388737B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Held on February 8, 2021 University of Tsukuba, College of Information and Media Creation, Graduation Research Final Presentation (WEB Conference)

The present invention relates to image processing technology.

Interframe predictive coding is one of the important techniques for compressing moving images. In interframe predictive coding, the image of the target frame is predicted based on an encoded frame (hereinafter also referred to as the "reference frame") at a different time from the frame to be encoded (hereinafter also referred to as the "target frame"). , encodes the difference between the actual image and the predicted image of the target frame. By encoding the difference between the target frame and the encoded frame, the data amount of the frame can be reduced.

In interframe predictive coding, the following processes (1) to (3) are performed for each block obtained by dividing a frame into multiple parts.

(1) Search for blocks (hereinafter referred to as "reference blocks") similar to blocks in the target frame (hereinafter referred to as "target blocks") from within the reference frame using image matching technology

(2) Find the difference image between the target block and the reference block obtained by search

(3) Find a motion vector indicating the movement from the reference block obtained through the search to the target block.

When decoding a frame of a moving image compressed in this manner, the image of the target block can be restored from the reference block, difference image, and motion vector.

As described above, in (1) above, image matching technology is used when searching for a reference block similar to the target block. Various methods such as feature-based matching and region-based matching are known as image matching techniques. Movement of an object within a video image includes movement, expansion/contraction (change in scale), and rotation.

Patent Document 1 discloses a technique for calculating motion in moving image compression. The coordinates of the three points in the area of the previous frame are determined using the three points in the known area of the current frame on the image plane and the three translation vectors searched for these three points. Furthermore, the horizontal and vertical components of each vector are calculated from the position coordinates of the six points in the area of the current frame and the previous frame, and from the x and y components of these vectors, the rotation angle in the horizontal and vertical directions, Six affine transformation parameters including expansion/contraction magnification and translation amount are calculated.

Further, Non-Patent Document 1 discloses a technique related to scale change and rotation of an image. A histogram regarding the gradient of pixels included in an image block is created, and the peak that is the maximum value of the histogram is assigned as the orientation of the key point. Then, the image is rotated in the orientation direction of the key point.

In the technique disclosed in Patent Document 1, the process of searching for a translation vector is complicated. In the technique disclosed in Non-Patent Document 1, the process of creating a histogram using the gradient strength and gradient direction of an image is complicated. Therefore, the amount of calculation required to determine the relative enlargement or reduction ratio between the target block and the reference block increases. Enlargement or reduction may be referred to as "enlargement/contraction" hereinafter. Further, the ratio of enlargement or reduction may be hereinafter referred to as "enlargement/contraction ratio."

On the other hand, Patent Document 2 defines a plurality of sub-regions that correspond to each other as a target region (corresponding to a target block) and a reference region (corresponding to a reference block), and a relative positional relationship between the sub-regions, that is, a sub-region is defined. A method has been disclosed in which the scaling ratio of a target area with respect to a reference area is determined by primarily forming a vector indicating the movement of the area. This reduces the amount of calculation for scaling factors.

In the method of Patent Document 2, when defining a plurality of sub-regions that correspond to each other in a target area and a reference area, a search range is determined within the reference area for each sub-target area set within the target area, and the search range is determined within the reference area, and Search for a sub-reference region that has a high degree of similarity.

Japanese Patent Application Publication No. 9-98424 Japanese Patent Application Publication No. 2020-030502

Gradient-based feature extraction - SIFT and HOG - Hironobu Fujiyoshi Information Processing Society of Japan Research Report CVIM 160, pp. 211-224, September, 2007.

In the method of Patent Document 2, in the process of searching for a sub-reference area within a reference area, by enlarging the search range, a sub-reference area with a high degree of similarity to the sub-target area can be found, and the accuracy of the scaling ratio can be improved. Go up. However, if the search range is increased, the processing for calculating the similarity will increase.

One objective of the present disclosure is to provide a technique for calculating the enlargement or reduction ratio between regions with a smaller amount of calculation.

An image processing device according to one aspect of the present disclosure is configured to set a plurality of second sub-regions corresponding to each of a plurality of first sub-regions in a first region, which is one of a target region or a reference region, to the target region or the reference region. a search unit that searches from a search range in a second area that is the other reference area; a relative positional relationship between the first sub-area in the first area and the second sub-area in the second area; a calculation unit that calculates a ratio of expansion or reduction of the first area to the second area based on the ratio; and a calculation unit that updates the search range based on the ratio, and searches the second sub-area by the search unit. and an updating unit that causes the calculation unit to calculate the ratio again.

According to one aspect of the present disclosure, the ratio of enlargement or reduction of the target region to the reference region can be calculated with a smaller amount of calculation.

FIG. 2 is a block diagram showing the functional configuration of an image processing device. FIG. 2 is a block diagram showing the hardware configuration of an image processing device. 3 is a flowchart of overall processing by the image processing device. FIG. 3 is a diagram showing an example of a target image. FIG. 3 is a diagram showing an example of a reference image. It is a flowchart of the 1st expansion/contraction calculation process. FIG. 7 is a diagram showing how a sub-target area is created within the target area. FIG. 3 is a diagram showing an initial search range set as a reference area. FIG. 6 is a diagram showing how a sub-reference area is determined within a reference area. FIG. 7 is a diagram illustrating an example of a sub-target region normalized using a sub-reference region as a reference. FIG. 7 is a diagram showing eigenvectors of principal components of a plurality of vectors obtained in step S203. It is a flowchart of the 2nd expansion/contraction calculation process. FIG. 6 is a diagram showing how a sub-reference area is created within the reference area. FIG. 3 is a diagram showing an initial search range set in a target area. FIG. 7 is a diagram showing how a sub-target area is determined within the target area. FIG. 7 is a diagram showing eigenvectors of principal components of a plurality of vectors obtained in step S303. It is a flowchart of enlargement/reduction determination processing.

Embodiments of the present invention will be described below with reference to the drawings.

In the present embodiment, an image encoding device that performs differential encoding of an image will be exemplified as an example of a device that calculates the ratio of enlargement or reduction of an image. Differential encoding encodes the difference between the still image frame image to be encoded (target image) and the image predicted from the still image frame image (reference image) used for encoding (predicted image). This is an encoding method. For example, the target image is an image of a still frame that is a target of encoding in a moving image, and the reference image is an image of a previous frame that has already been encoded in the moving image.

If a predicted image as close as possible to the target image is generated from the reference image, the difference between the target image and the predicted image becomes smaller, and as a result, the amount of data after encoding can be reduced. For example, if a predicted image as close as possible to the target image is obtained, the difference will be almost eliminated and the amount of data after encoding will be small. On the other hand, if prediction is impossible, for example, the difference between the target image and the reference image is encoded as is, and the amount of data after encoding becomes large.

FIG. 1 is a block diagram showing the functional configuration of an image processing device. FIG. 2 is a block diagram showing the hardware configuration of the image processing device.

Referring to FIG. 1, the image processing device 1 includes a control section 10 and a storage section 20. The storage unit 20 stores frame information 21 and differential code information 22. The frame information 21 is image data of each frame making up a moving image. The differential code information 22 is image data of each frame encoded by differential encoding.

The control unit 10 encodes the image of the frame using the frame information 21 stored in the storage unit 20, and records the encoded image data in the storage unit 20 as differential code information 22.

Referring to FIG. 2, the hardware of the image processing device 1 is a computer device 400 as an example. The computer device 400 includes a control circuit 401, a storage device 402, a reading device 403, a recording medium 404, a communication interface 405, an input/output interface 406, an input device 407, and a display device 408. . Control circuit 401, storage device 402, reading device 403, communication interface 405, input/output interface 406, and display device 408 are connected to each other by bus 410. Recording medium 404 is connected to reading device 403 . Input device 407 is connected to input/output interface 406 . Further, the communication interface 405 is connected to a network 409.

A control circuit 401 controls the entire computer device 400. The control circuit 401 is, for example, a processor. The control circuit 401 operates as the control unit 10 shown in FIG. 1 by executing an image processing program (not shown) recorded in the storage device 402.

The storage device 402 stores an image processing program that causes the control circuit 401 to function as the control unit 10. Furthermore, the storage device 402 stores various data for the control circuit 401 to operate. Furthermore, the storage device 402 functions as the storage unit 20 in FIG.

The reading device 403 reads data from a removable recording medium 404 and writes data to the recording medium 404 .

The recording medium 404 is a non-temporary storage medium such as an SD (Secure Digital) memory card, an FD (Floppy Disk), a CD (Compact Disc), a DVD (Digital Versatile Disk), a BD (Blu-ray Disk (registered trademark)), and a flash memory. It is a recording medium. The recording medium 404 stores, for example, data given to the computer device 400, data generated by the computer device 400, and data acquired by the computer device 400. Further, the recording medium 404 may store information in the storage unit 20 shown in FIG. 1.

Communication interface 405 communicably connects computer device 400 and other devices (not shown) via network 409.

The input/output interface 406 is an interface that connects to various input/output devices such as the input device 407. Input/output interface 406 outputs a signal input from input device 407 to control circuit 401 via bus 410. Further, the input/output interface 406 outputs the signal output from the control circuit 401 to the input/output device via the bus 410. Input device 407 is an example of an input/output device.

The input device 407 is an input device such as a keyboard or a mouse. The input device 407 accepts an information input operation by a user, and outputs a signal corresponding to the input information to the input/output interface 406. For example, when a user inputs an instruction to start differential encoding of a video into the input device 407, the input device 407 outputs a signal indicating the instruction to the input/output interface 406.

The display device 408 displays various information on the screen under control from the control circuit 401.

Network 409 communicably connects computer device 400 and other devices.

The details of the control unit 10 in FIG. 1 will be described below.

Referring to FIG. 1, the control unit 10 includes a reference area determination unit 11, a scaling calculation unit 12, a predicted image generation unit 16, and a difference encoding unit 17.

The reference area determining unit 11 acquires image data of a target image to be encoded from the frame information 21 of the storage unit 20, and selects an object in the reference image to be used for encoding a block area (target area) in the target image. A region similar to the region is determined as a reference region. Being similar means that the value of the index indicating the degree of similarity is large. Hereinafter, an index indicating the degree of similarity will also be referred to as similarity. Although the method for evaluating the similarity between images is not particularly limited, it can be evaluated using, for example, the sum of squared errors of pixel values. For example, a region whose degree of similarity to the target region exceeds a predetermined threshold may be used as the reference region. Alternatively, a region having a higher degree of similarity to the target region than other regions in the reference image may be used as the reference region. When determining a reference region, the reference region determination unit 11 sequentially slides reference region candidates within the reference image and calculates the degree of similarity between the candidate region and the target region in a round-robin manner. Search for areas with high similarity. In creating a predicted image, which will be described later, the image of the target area is predicted by enlarging or reducing the reference area, so the reference area serves as a reference for enlarging or reducing the target area.

The scaling calculation unit 12 scales the target area with respect to the reference area based on the relative positional relationship between the multiple sub-target areas within the target area and the multiple sub-reference areas within the reference area corresponding to each sub-target area. Determine the percentage of expansion or contraction. The enlargement/reduction calculation unit 12 includes a search unit 13, a calculation unit 14, and an update unit 15, which work together to determine the ratio of enlargement or reduction of the target area to the reference area.

The search unit 13 searches for a plurality of sub-reference regions corresponding to each of the plurality of sub-target regions within the target region from a search range within the reference region. The calculation unit 14 calculates the ratio of expansion or reduction of the target area to the reference area based on the relative positional relationship between the sub-target area in the target area and the sub-reference area in the reference area. The update unit 15 updates the search range in the reference area based on the ratio calculated by the calculation unit 14, and causes the search unit 13 to search for the sub-reference area and the calculation unit 14 to perform the calculation of the ratio again. A series of processes including the search for the sub-reference area by the search unit 13, the calculation of the ratio by the calculation unit 14, and the update of the search range by the update unit 15 shown here are repeated multiple times. In this way, the enlargement or reduction ratio is determined once, the sub-region is searched within the search range determined based on that ratio, and the enlargement or reduction ratio is calculated again using the obtained sub-region. A suitable sub-region can be found without enlarging the search range, and the ratio of expansion or contraction between regions can be calculated with a smaller amount of calculation.

Note that here we have shown an example in which a sub-reference area is searched within a reference area based on a sub-target area within a target area, and the ratio of enlargement or reduction of the target area to the reference area is calculated. Even if a sub-target area is searched within the target area based on the sub-reference area, it is possible to calculate the enlargement or reduction ratio of the target area with respect to the reference area. In this embodiment, the sub-reference area is searched within the target area based on the ratio calculated by searching the sub-reference area within the reference area based on the sub-target area within the target area, and the sub-reference area is searched within the target area based on the sub-reference area within the reference area. Based on the calculated ratio, the ratio of enlargement or reduction of the target area to the reference area is determined. However, it is also possible to perform only one of them and use the resulting ratio. For example, if the target image and the reference image are not temporally separated, or if it is known that the ratio is close to 1, only one of them may be performed to simplify the calculation. Details of the processing in this embodiment will be described later.

The predicted image generation unit 16 generates a predicted image in which the target image is predicted from the reference image using an image in which the reference area is enlarged or reduced at a ratio determined by the enlargement/reduction calculation unit 12.

The difference encoding unit 17 encodes the difference between the predicted image generated by the predicted image generation unit 16 and the target image. Code data obtained by encoding the difference is recorded in the storage unit 20 as difference code information 22.

According to this embodiment, a ratio is calculated with a smaller amount of calculation, and a predicted image of the target area is generated using an image in which the reference area is enlarged or reduced by that ratio, so that good differential encoding can be performed with less amount of calculation. It can be performed.

Note that the reference region determination unit 11 uses a plurality of regions in the reference image that can overlap with each other as candidate regions, and calculates the degree of similarity between a rotated candidate region obtained by rotating the candidate region by a predetermined rotation angle and the target region. , a reference region may be determined from among a plurality of candidate regions based on the degree of similarity. Since the reference area is searched using the similarity between the rotated area in the reference image and the target area, it is possible to determine a good reference area, and as a result, it is possible to perform good differential encoding. can. At this time, for example, multiple rotation angles are determined in advance, a rotation candidate area is created by rotating each candidate area by each rotation angle, and the similarity between the multiple rotation candidate areas and the target area is calculated by round-trip calculation. The original candidate region of the rotation candidate region with the highest degree of similarity may be used as the reference region. Further, it can be said that the rotation angle of the rotation candidate area at this time is the relative rotation angle between the target area and the reference area.

FIG. 3 is a flowchart of the overall processing by the image processing device.

Referring to FIG. 3, first, in step S101, the reference area determination unit 11 determines an area in the reference image that is similar to the target area in the target image as a reference area.

FIG. 4 is a diagram showing an example of a target image. FIG. 5 is a diagram showing an example of a reference image. Here, it is assumed that a region having a high degree of similarity to the target region 110 in the target image 100 shown in FIG. 4 is searched from within the reference image 200 shown in FIG. 5. In this example, the reference area determining unit 11 sequentially slides the candidate area 211 that is a reference area candidate in the reference image 200, rotates the candidate area 211 by a predetermined rotation angle to make it a rotated candidate area, and sets the target area 110 as a rotated candidate area. The candidate region 211, which is the source of the rotation candidate region with the highest similarity, is set as the reference region 210.

Next, the scaling calculation unit 12 executes scaling calculation processing in steps S102 to S106, which will be described below.

First, in step S102, the scaling calculation unit 12 executes a first scaling calculation process. The first enlargement/contraction calculation process is a calculation process related to enlargement or reduction when the reference area is viewed from the target area.

FIG. 6 is a flowchart of the first scaling calculation process.

Referring to FIG. 6, first, in step S201, the scaling calculation unit 12 creates a plurality of sub-target regions within the target region. FIG. 7 is a diagram showing how a sub-target area is created within the target area. As shown in FIG. 7, the enlargement/reduction calculation unit 12 creates a plurality of sub-target regions (first sub-target regions) 121 at equal intervals on a circumference centered on the center of the target region 110. Here, eight sub-target regions 121 numbered from 1 to 8 are created.

Next, in step S202, the scaling calculation unit 12 sets a search range in the reference area 210, and the search unit 13 searches for areas similar to each of the plurality of sub-target areas 121 (sub-reference areas) within the search range. ) to explore.

FIG. 8 is a diagram showing an initial search range set as a reference area. Referring to FIG. 8, a search range 220 is set in the reference area 210. The search range 220 is individually set for each of the sub-target regions 121. The initial search range 220 is calculated from the position in the reference area 210 that is the same as the position where each sub-target area 121 in the target area 110 was cut out, and from the relative position between the target area 110 and the reference area 210 with the center of the reference area 210 as the axis. It is set in a predetermined shape and size near the position rotated by the rotation angle.

The scaling calculation unit 12 searches for regions similar to each of the sub-target regions 121 shown in FIG. (first sub-reference area). As a specific example, the scaling calculation unit 12 slides a candidate area within the search range 220 and evaluates the degree of similarity between the candidate area and the sub-target area 121 each time. The region with the highest candidate value may be defined as the sub-reference region.

FIG. 9 is a diagram showing how the sub-reference area is determined within the reference area. Referring to FIG. 9, sub-reference areas 221 corresponding to each sub-target area 121 are determined within the reference area 210.

Once the sub-reference areas 221 corresponding to all the sub-target areas 121 have been determined, next in step S203, the scaling calculation unit 12 causes the calculation unit 14 to set the center of the target area 110 as the origin and set the multiple sub-target areas 121. Find a plurality of vectors (first vectors) by normalizing the coordinate vectors indicating the positions of the plurality of sub-reference areas 221 with the center of the reference area 210 as the origin so that the magnitude of the coordinate vector indicating the position of is 1. . The set of vectors obtained here indicates the distribution of the ratio of enlargement or reduction of the target region to the reference region in each direction. FIG. 10 is a diagram illustrating an example of a normalized sub-target region. FIG. 10 shows a sub-target region 122 whose distance from the origin O has been expanded or reduced relative to the corresponding sub-reference region 221 as a result of normalization.

Next, in step S204, the scaling calculation unit 12 uses the calculation unit 14 to perform principal component analysis on the plurality of vectors obtained in step S203 to obtain a principal component (principal component A). The eigenvector of the principal component found here indicates the direction of expansion or contraction, and the eigenvalue indicates the ratio of expansion or contraction. The eigenvalue of the first principal component is α', and the eigenvalue of the second principal component is β'. Let the eigenvector of the first principal component be Vα', and the eigenvector of the second principal component be Vβ'. FIG. 11 is a diagram showing the eigenvectors of the principal components of the plurality of vectors obtained in step S203. Referring to FIG. 11, the eigenvector Vα' of the first principal component and the eigenvector Vβ' of the second principal component are shown.

Returning to FIG. 3, further, in step S103, the scaling calculation unit 12 executes a second scaling calculation process. The second enlargement/contraction calculation process is a calculation process related to enlargement or reduction when the target area is viewed from the reference area.

FIG. 12 is a flowchart of the second scaling calculation process.

Referring to FIG. 12, first, in step S301, the scaling calculation unit 12 creates a plurality of sub-reference areas within the reference area. FIG. 13 is a diagram showing how a sub-reference area is created within the reference area. As shown in FIG. 13, the scaling calculation unit 12 creates a plurality of sub-reference areas (second sub-reference areas) 222 at equal intervals on a circumference centered on the center of the reference area 210. Here, eight sub-reference areas 222 numbered from 1 to 8 are created.

Next, in step S302, the scaling calculation unit 12 sets a search range in the target area 110, and the search unit 13 sets an area similar to each of the plurality of sub-reference areas 222 (sub-target area).

FIG. 14 is a diagram showing the initial search range set in the target area. Referring to FIG. 14, a search range 120 is set in the target area 110. The search range 120 is individually set for each sub-reference area 221. The initial search range 120 is calculated from the position in the target area 110 that is the same as the position where each sub-reference area 221 in the reference area 210 was cut out, and from the relative position between the reference area 210 and the target area 110 with the center of the target area 110 as an axis. It is set in a predetermined shape and size near the position rotated by the rotation angle.

The scaling calculation unit 12 searches for regions similar to each of the sub-reference regions 222 shown in FIG. (second sub-target area). As a specific example, the scaling calculation unit 12 slides a candidate area within the search range 120 and evaluates the degree of similarity between the candidate area and the sub-reference area 222 each time. The candidate region with the highest value may be determined as the sub-target region.

FIG. 15 is a diagram showing how a sub-target area is determined within the target area. Referring to FIG. 15, sub-target regions 122 corresponding to each sub-reference region 221 are determined within the target region 110.

Once the sub-target regions 122 corresponding to all the sub-reference regions 222 have been determined, next, in step S303, the scaling calculation unit 12 causes the calculation unit 14 to set the center of the reference region 210 as the origin and select the sub-reference regions 222 from the plural sub-reference regions 222. A plurality of vectors (second vectors), which are coordinate vectors indicating the positions of the plurality of sub-target regions 122 with the center of the target region 110 as the origin, are normalized so that the magnitude of the coordinate vectors indicating the positions of are 1. demand. The set of vectors obtained here indicates the distribution of the ratio of enlargement or reduction of the target region to the reference region in each direction.

Next, in step S304, the scaling calculation unit 12 causes the calculation unit 14 to perform principal component analysis on the plurality of vectors obtained in step S303 to obtain a principal component (principal component B). The eigenvector of the principal component found here indicates the direction of expansion or contraction, and the eigenvalue indicates the ratio of expansion or contraction. The eigenvalue of the first principal component is α, and the eigenvalue of the second principal component is β. Let the eigenvector of the first principal component be Vα, and the eigenvector of the second principal component be Vβ. FIG. 16 is a diagram showing the eigenvectors of the principal components of the plurality of vectors obtained in step S303. Referring to FIG. 16, the eigenvector of the first principal component is shown as Vα, and the eigenvector of the second principal component is shown as Vβ.

Note that the order in which the first scaling calculation process and the second scaling calculation process described above are executed is not particularly limited. The second scaling calculation process may be executed before the first scaling calculation process.

Returning to FIG. 3, further, in step S104, the scaling calculation unit 12 executes scaling determination processing. The scaling determination process is a process that determines the direction and ratio of expansion or contraction of the target area with respect to the reference area based on the processing results of the first scaling calculation process and the processing results of the second scaling calculation process.

FIG. 17 is a flowchart of the enlargement/reduction determination process.

Referring to FIG. 17, in step S401, the enlargement/reduction calculation unit 12 determines whether αα′≦1 and ββ′<1. If the determination in step S401 is YES, the scaling calculation unit 12, in step S402, reduces the size in the direction indicated by the eigenvector Vα at the ratio indicated by the eigenvalue α, and in the direction indicated by the eigenvector Vβ at the ratio indicated by the eigenvalue β. It is determined that

If the determination in step S401 is NO, the scaling calculation unit 12 determines in step S403 whether αα′>1 and ββ′≧1. If the determination in step S403 is YES, in step S404, the expansion/contraction calculation unit 12 expands in the direction indicated by the eigenvector Vα' at the ratio indicated by the eigenvalue α', and in the direction indicated by the eigenvector Vβ' as indicated by the eigenvalue β'. It is judged that the ratio is expanding.

If the determination in step S403 is NO, the scaling calculation unit 12 determines in step S405 that αα′>1 and ββ′<1, and the angle formed by the eigenvector Vα and the eigenvector Vα′ is smaller than a predetermined angle (for example, 45°). Determine whether or not. If the determination in step S405 is YES, the enlargement/reduction calculation unit 12 recalculates enlargement or reduction in step S406. In the recalculation, the expansion/contraction calculation unit 12 performs principal component analysis on the union of the first vector near the eigenvector Vα' and the second vector near the eigenvector Vβ, and expands or expands the eigenvalue of the obtained principal component. Let the ratio of reduction be the ratio of reduction, and let the eigenvector be the direction of expansion or reduction. Note that a vector near the eigenvector Vα' means a vector whose direction is close to the eigenvector Vα', that is, a vector whose direction is within a predetermined range of the direction of the eigenvector Vα'. Furthermore, in this example, the reason why the angle between the eigenvector Vα and the eigenvector Vα′ is required to be smaller than a predetermined angle (for example, 45°) is because the eigenvector Vα and the eigenvector Vα′ should originally be in close directions. , this excludes cases where the angles formed by these are large.

If the determination in step S405 is NO, the scaling calculation unit 12 determines in step S407 whether αα′=1 and ββ′=1. If the determination in step S407 is YES, the enlargement/reduction calculation unit 12 determines in step S408 that the target area is neither enlarged nor reduced with respect to the reference area. If the determination in step S407 is NO, the enlargement/reduction calculation unit 12 determines in step S409 that it is unclear whether the target area is enlarged or reduced relative to the reference area.

Returning to FIG. 3, further in step S105, the enlargement/reduction calculation unit 12 determines whether a predetermined termination condition is satisfied. The termination condition is not particularly limited, but here, as an example, it is assumed that the process is terminated after the ratio determination in step S104 has been executed a predetermined number of times.

If the termination condition is not met, in step S106, the scaling calculating unit 12 updates the search range using the updating unit 15, and returns to step S102. Here, the search ranges used for the first scaling calculation process and the second scaling calculation process are updated.

If the ratio calculated in the scaling determination process is such that the target area 110 is expanded relative to the reference area 210, the scaling calculation unit 12 sets the search range used in the first scaling calculation process to the reference area 210. If the target area 110 is updated in a direction closer to the center, and if the ratio is such that the target area 110 is reduced with respect to the reference area 210, the search range used for the first scaling calculation process is moved in the direction away from the center of the reference area 210. There is no particular limitation on how much the search range is moved. For example, the amount of movement may be a predetermined fixed value or may be a value according to a ratio.

Further, if the ratio calculated in the scaling determination process is such that the target area 110 is expanded with respect to the reference area 210, the scaling calculation unit 12 sets the search range used in the second scaling calculation process to the target area. 110, and if the target area 110 is at a reduced ratio with respect to the reference area 210, the search range used for the second scaling calculation process is moved in the direction away from the center of the target area 110. . There is no particular limitation on how much the search range is moved. For example, the amount of movement may be a predetermined fixed value or may be a value according to a ratio.

After the search range is updated, the first scaling calculation process, the second scaling calculation process, and the scaling determination process are executed again.

If the end condition is satisfied in step S105, the predicted image generation unit 16 further generates a predicted image by predicting the target image 100 from the reference image 200 in step S107. At this time, the predicted image generation unit 16 uses an image obtained by enlarging or reducing the reference area 210 at a ratio finally determined by the enlargement/reduction calculation unit 12 for predicting an area corresponding to the target area 110 of the predicted image. For this purpose, the predicted image generation unit 16 generates a motion vector indicating the amount of movement from the reference area to the target area, and a difference image between the image predicted from the image of the reference area 210 and the image of the target area 110. The generated data is included in the predicted image data. Note that an image of the reference area at the same size may be used for the area corresponding to the target area for which the expansion/reduction calculation unit 12 has determined that it is unclear whether it is being expanded or reduced.

Subsequently, in step S108, the differential encoding unit 17 encodes the target image 100 using the difference image between the predicted image generated by the predicted image generation unit 16 and the target image 100, and generates a code obtained by encoding the difference. The data is recorded in the storage unit 20 as differential code information 22.

The embodiment described above includes the following matters. However, the matters included in this embodiment are not limited to those shown below.

(Item 1)
Searching for a plurality of second sub-regions corresponding to each of the plurality of first sub-regions in a first region, which is one of the target region or the reference region, within a second region, which is the other of the target region or the reference region. A search section that searches from a range,
Based on the relative positional relationship between the first sub-region in the first region and the second sub-region in the second region, calculate an expansion or contraction ratio of the first region with respect to the second region. a calculation unit that performs
an updating unit that updates the search range based on the ratio and causes the search unit to search the second sub-region and the calculation unit to calculate the ratio again;
An image processing device having:
According to this, the ratio of expansion or reduction is determined once, a sub-area is searched within the search range determined based on that ratio, and the ratio of expansion or reduction is calculated again using the obtained sub-area. , a suitable sub-region can be found without enlarging the search range, and the ratio of expansion or contraction between regions can be calculated with a smaller amount of calculation.

(Item 2)
The search unit creates the first sub-region on a predetermined circumference in the first region, and creates an initial search range in the vicinity of a position corresponding to the first sub-region in the second region,
The updating unit updates the search range in a direction closer to the center of the circumference if the ratio is increasing, and updates the search range in a direction away from the center if the ratio is decreasing.
The image processing device according to item 1.

(Item 3)
The search unit sets a plurality of regions in the search range that can overlap with each other as candidate sub-regions, calculates a similarity between the candidate sub-region and the first sub-region, and selects the candidate sub-region based on the similarity. determining one of them as the second sub-region,
The image processing device according to item 1.
According to this, in a configuration in which a second sub-region is searched using the similarity between a candidate sub-region in a search area and a first sub-region, the ratio of expansion or reduction is once determined, and the ratio is updated based on that ratio. Since the second sub-region is searched within the search range, a sub-region with high similarity can be found without enlarging the search range.

(Item 4)
The calculation unit is configured to perform principal component analysis on a plurality of vectors based on the relative positional relationship between the first sub-region in the first region and the second sub-region in the second region. calculating the ratio based on the eigenvalues of the principal components;
The image processing device according to item 1.

(Item 5)
The calculation unit performs principal component analysis on the plurality of vectors to calculate the eigenvalues and eigenvectors of the principal components, and the first region is configured to move the first region relative to the second region in a direction determined based on the eigenvector. Calculate that it is expanding or contracting at a rate determined based on the eigenvalues,
The image processing device according to item 4.
According to this, not only the ratio of enlargement or reduction but also the direction of enlargement or reduction can be determined, so it is possible to determine in which direction and how much the image is enlarged or reduced.

(Item 6)
The target area is an area included in a target image to be encoded, and the reference area is an area included in a reference image used for encoding the target image,
The image processing device includes:
an area determining unit that determines, for the target area, the reference area that serves as a reference for enlarging or contracting the target area;
a predicted image generation unit that generates a predicted image that predicts the target image using an image in which the reference area is enlarged or reduced based on the ratio;
further comprising a difference encoding unit that encodes a difference between the predicted image and the target image;
The image processing device according to item 1.
According to this, in an image processing device that performs predictive differential encoding using enlarged and reduced images, it is possible to calculate the enlargement or reduction ratio with a smaller amount of calculation and realize good differential encoding.

The embodiments of the present invention described above are illustrative examples of the present invention, and are not intended to limit the scope of the present invention only to those embodiments. Those skilled in the art can implement the invention in various other ways without departing from the scope of the invention. Although the devices of each of the above embodiments have been exemplified as being realized by executing a software program on a processor, the present invention is not limited to this. Some or all of the functions of the control unit 10 may be realized by hardware. Further, part or all of the functions of the control unit 10 may be realized by a dedicated processor.

DESCRIPTION OF SYMBOLS 1... Image processing device, 10... Control part, 11... Reference area determination part, 12... Enlargement/reduction calculation part, 13... Search part, 14... Calculation part, 15... Update part, 16... Predicted image generation part, 17... Difference code converting unit, 20... storage unit, 21... frame information, 22... differential code information, 100... target image, 110... target area, 120... search range, 121... sub target area, 122... sub target area, 200... reference image , 210...Reference area, 211...Candidate area, 220...Search range, 221...Sub reference area, 222...Sub reference area, 400...Computer device, 401...Control circuit, 402...Storage device, 403...Reading device, 404... Recording medium, 405... Communication interface, 406... Input/output interface, 407... Input device, 408... Display device, 409... Network, 410... Bus

Claims (6)

A block area in a target image, which is an image of a frame to be encoded in a moving image, is defined as a target area, and an area similar to the target area in a reference image, which is an image of a frame that has already been encoded in the moving image, is defined as a target area. a reference area, one of the target area or the reference area is a first area and the other is a second area, and within the first area are arranged at equal intervals on the circumference of a circle centered on the center of the first area. A plurality of discretely arranged first sub-regions are created with the radius of the circle and the shape and size of the first sub-region that enable updating of the search range, which will be described later , and are the same as the first sub-region. A plurality of second sub-regions similar in shape and size to each of the plurality of first sub-regions are searched within the second region, the initial search range being the second sub-region in the first region. The second sub-region is rotated from a position in the second region that is the same as the first sub-region to a position rotated about the center of the second region by a relative rotation angle between the first region and the second region. a search unit that searches from each of a plurality of discrete search ranges set in a shape and size that can create a plurality of candidate sub-regions that are candidates for the sub-region;
Based on the size of a coordinate vector whose origin is the center of the first area and indicates the position of the first sub-area, and the size of the coordinate vector whose origin is the center of the second area and indicates the position of the second sub-area. a calculation unit that calculates a ratio of expansion or contraction of the first area to the second area;
If the ratio is increasing, the search range is updated by a predetermined fixed value in a direction closer to the center of the second area, and if the ratio is reduced, the search range is updated by a predetermined fixed value in a direction away from the center. , and the search for the second sub-area by the search unit and the calculation of the ratio by the calculation unit require less calculation than the case where the search range is fixed and the second sub-area is searched. an updating unit that repeats until a predetermined end condition is met that limits the number of times such that the ratio is reduced , and outputs the finally determined ratio;
An image processing device having: The search unit calculates the similarity of pixel values between the images of the candidate sub-region and the first sub-region while moving the candidate sub-region within the search range, and selects a candidate sub-region with the highest similarity. determining a region as the second sub-region;
The image processing device according to claim 1. The calculation unit is configured to calculate the second sub-region with the center of the second region as the origin so that the magnitude of a coordinate vector indicating the position of the first sub-region with the center of the first region as the origin is 1. The first vector, which is a coordinate vector that indicates the position, has a magnitude changed, and the coordinate vector that indicates the position of the second sub-area, with the origin at the center of the second area, has a magnitude of 1. The ratio is determined based on the eigenvalue of the principal component obtained by principal component analysis of one or both of the second vectors, which have the origin at the center of the region and vary the length of the coordinate vector indicating the position of the first sub-region. calculate,
The image processing device according to claim 1. The calculation unit calculates the eigenvalue and eigenvector of the principal component, and expands or reduces the first region with respect to the second region in a direction indicated by the eigenvector at a ratio represented by the eigenvalue. calculating the ratio of expansion or contraction of the first area to the second area based on the fact that
The image processing device according to claim 3. The search unit sets a block area in a target image that is an image of a frame to be encoded in a moving image as a target area, and sets a block area in a reference image that is an image of a frame that has already been encoded in the moving image as a target area. A similar area is a reference area, one of the target area or the reference area is a first area and the other is a second area, and within the first area, on the circumference of a circle centered on the center of the first area. A plurality of first sub-areas arranged discretely at equal intervals are created with the radius of the circle and the shape and size of the first sub-area that enable updating of the search range, which will be described later. A plurality of second sub-regions having the same shape and size as the sub-regions and similar to each of the plurality of first sub-regions are searched within the second region, the initial search range being the first sub-region. from a position in the second region that is the same as the position of the first sub-region in one region to a position rotated by a relative rotation angle between the first region and the second region around the center of the second region. , searching from each of a plurality of discrete search ranges set in a shape and size that can create a plurality of candidate sub-regions that are candidates for the second sub-region;
The calculation unit calculates the size of a coordinate vector whose origin is the center of the first area and indicates the position of the first sub-area, and the size of a coordinate vector whose origin is the center of the second area and indicates the position of the second sub-area. and calculating the ratio of expansion or contraction of the first area to the second area based on
The updating unit updates the search range by a predetermined fixed value in a direction closer to the center of the second area if the ratio is increasing, and updates the search range in a direction away from the center if the ratio is decreasing. When searching a second sub-area with a fixed search range, the searching unit searches the second sub-area and the calculating unit calculates the ratio by updating a predetermined fixed value. Repeat until a predetermined termination condition is met that limits the number of calculations so that the amount of calculation is less than that , and output the finally determined ratio;
Image processing method. The search unit sets a block area in a target image that is an image of a frame to be encoded in a moving image as a target area, and sets a block area in a reference image that is an image of a frame that has already been encoded in the moving image as a target area. A similar area is a reference area, one of the target area or the reference area is a first area and the other is a second area, and within the first area, on the circumference of a circle centered on the center of the first area. A plurality of first sub-areas arranged discretely at equal intervals are created with the radius of the circle and the shape and size of the first sub-area that enable updating of the search range, which will be described later. A plurality of second sub-regions having the same shape and size as the sub-regions and similar to each of the plurality of first sub-regions are searched within the second region, the initial search range being the first sub-region. from a position in the second region that is the same as the position of the first sub-region in one region to a position rotated by a relative rotation angle between the first region and the second region around the center of the second region. , searching from each of a plurality of discrete search ranges set in a shape and size that can create a plurality of candidate sub-regions that are candidates for the second sub-region;
The calculation unit calculates the size of a coordinate vector whose origin is the center of the first area and indicates the position of the first sub-area, and the size of a coordinate vector whose origin is the center of the second area and indicates the position of the second sub-area. and calculating the ratio of expansion or contraction of the first area to the second area based on
The updating unit updates the search range by a predetermined fixed value in a direction closer to the center of the second area if the ratio is increasing, and updates the search range in a direction away from the center if the ratio is decreasing. When searching a second sub-area with a fixed search range, the searching unit searches the second sub-area and the calculating unit calculates the ratio by updating a predetermined fixed value. Repeat until a predetermined termination condition is met that limits the number of calculations so that the amount of calculation is less than that , and output the finally determined ratio;
An image processing program that allows a computer to perform processing.

Images