JP5207941B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for causing a computer to execute the image processing method for synthesizing a real image in a real space and a CG image.

  Conventionally, a real world image (image) and a 3D modeled CG image (image) are combined and displayed, as if a CG object (virtual object) exists in the real world. There is a mixed reality (MR) presentation device shown in FIG. As such an apparatus, for example, one shown in Patent Document 1 below can be cited. Specifically, Patent Document 1 discloses a real video shooting unit for shooting a real video (image) and a CG video generation for creating a CG video (CG image) as seen from the position and orientation of the real video shooting unit. An apparatus comprising a means and a composition means for synthesizing a real image and a CG image is shown.

  In such a mixed reality presentation device, the virtual object image is superimposed and drawn on the real image, so that there is a problem that the depth relationship between the real object and the virtual object is not correctly displayed. In particular, a situation where an experienced person of mixed reality (MR) presents his / her hand against a virtual object can frequently occur, but a virtual object that should be behind the hand is drawn in front of the hand, Experienced people often felt a strong sense of incongruity.

  In order to solve this problem, it is necessary to acquire depth information of the real space viewed from the shooting point of the real image. In that case, a special measurement device is required. In addition, depending on the accuracy of the depth information, there is a problem that the contour of the real object and the boundary of the CG are not clearly expressed. In order to solve this problem, Patent Document 1 attempts to solve the problem by having a configuration in which CG is not drawn in an image region having substantially the same color as the skin color of the hands and arms of the experiencer. According to this configuration, although it is a simple configuration that does not require acquisition of depth information of the real space, it is possible to solve a problem of a situation that frequently occurs. In addition, there is an advantage that the contours of the hand and the arm are expressed beautifully by judging from the information of the image instead of the judgment based on the depth information. Further, Patent Document 1 also shows a configuration in which it is determined whether or not a region where CG is to be drawn is based on an area occupied by a region having substantially the same color as the skin color.

JP 2006-343953 A

  However, in the conventional technology, for example, when another person is in the back of the CG object, the face area of the person is extracted as an area having the same color as the skin color of the hand or arm. there were. Even when there is a means for judging by the area of the skin color area, there are many situations where the area of the face area is photographed in a wide area that is not much different from the area of the hand and arm areas, so that it is as intended. In some cases, it could not be processed.

  The present invention has been made in view of such problems, and realizes the generation of a more natural composite image without a sense of incongruity when a composite image is generated by combining a real image and a CG image. An object is to provide an image processing apparatus, an image processing method, and a program.

The image processing apparatus of the present invention includes a color area extraction unit that extracts a color area included in a specific color range from a real image, a CG image generation unit that generates a CG image based on virtual scene data, A shape area extraction unit that extracts a region having a specific shape as a specific shape region from a real image, and a specific shape region extracted by the shape region extraction unit among the regions extracted by the color region extraction unit And a synthesizing unit that synthesizes the real image and the CG image by excluding a region other than .
According to another aspect of the image processing apparatus of the present invention, color area extraction means for extracting a color area included in a specific color range from a real image, and CG image generation for generating a CG image based on virtual scene data are provided. Means corresponding to the CG drawing area out of the real image, and CG drawing area extracting means for extracting the area where the CG object of the virtual scene data is drawn as a CG drawing area from the CG image A real image cutout unit that cuts out a region to be processed, a shape region extraction unit that extracts a region having a specific shape from the real image cut out by the real image cutout unit, and the color region extraction unit. Based on the extracted color area and the specific shape area extracted by the shape area extraction means, the real image and the CG image are combined. And a forming means.

According to the image processing method of the present invention, the color region extracting unit extracts a color region included in a specific color range from the real image, and the CG image generating unit is based on the virtual scene data. A CG image generation step for generating a CG image, a shape region extraction unit for extracting a region having a specific shape from the real image as a specific shape region, and a synthesis unit for the color region of the extracted in the extraction step region, and a synthesizing step to exclude regions other than the particular shape region extracted by said shape region extraction step, synthesizing processing and the CG image and the real image.
According to another aspect of the image processing method of the present invention, the color region extracting unit extracts a color region included in a specific color range from the real image, and the CG image generating unit includes a virtual scene. A CG image generation step for generating a CG image based on the data, and a CG drawing area extracting means for extracting, from the CG image, an area in which the CG object of the virtual scene data is drawn as a CG drawing area The drawing area extraction step and the real image cutout means cut out the area corresponding to the CG drawing area from the real image, and the shape area extraction means was cut out by the real image cutout step. A shape region extraction step for extracting a region having a specific shape as a specific shape region from a real image; Has a color region extracted by the color region extraction step, the shaped region extracted based on the particular shape region extracted in step, and a synthesis step of synthesizing processing and the CG image and the real image.

The program of the present invention includes a color region extraction step for extracting a color region included in a specific color range from a real image, a CG image generation step for generating a CG image based on virtual scene data, and the real image A region having a specific shape as a specific shape region , and a region other than the specific shape region extracted in the shape region extraction step among the regions extracted in the color region extraction step . This is for causing a computer to execute a synthesizing step of synthesizing the real image and the CG image by excluding a region .
According to another aspect of the program of the present invention, a color area extraction step for extracting a color area included in a specific color range from a real image, and a CG image generation step for generating a CG image based on virtual scene data, A CG drawing area extracting step for extracting, as a CG drawing area, an area in which the CG object of the virtual scene data is drawn from the CG image, and an area corresponding to the CG drawing area from the real image A real image cutout step, a shape region extraction step for extracting a region having a specific shape as a specific shape region from the real image cut out by the real image cutout step, and the color region extraction step. The real image based on the color area and the specific shape area extracted in the shape area extraction step. It is intended for executing a combining step of combining processing and the CG image into the computer.

  According to the present invention, when a synthesized image is generated by synthesizing a real image and a CG image, it is possible to realize generation of a more natural synthesized image without a sense of incompatibility compared to the conventional method.

  The best mode (embodiment) for carrying out the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic diagram illustrating an example of a schematic configuration of an image processing apparatus 100 according to an embodiment of the present invention.
As shown in FIG. 1, the image processing apparatus 100 includes a CPU 110, a display 120, a video camera 130, a first memory 140, a second memory 150, an information input device 160, and a bus 170. Configured.

  The CPU 110 controls the operation of the image processing apparatus 100 as a whole, and controls each component (120 to 160) of the image processing apparatus 100 as necessary.

  The display 120 constitutes a display device that displays various images and various information based on the control of the CPU 110.

  The video camera 130 constitutes an imaging device that captures an image of a subject or the like based on the control of the CPU 110.

  A program 1400 is stored in the first memory 140. For example, when the CPU 110 executes the program 1400, a color area extraction unit 1401, a shape area extraction unit 1402, a synthesis unit 1403, a CG drawing area extraction unit 1404, a real image cutout unit 1405, a shape peripheral area extraction unit 1406, A shape peripheral area determination unit 1407 and a shape peripheral area reduction unit 1408 are configured.

  Various data and various images are stored in the second memory 150. Specifically, FIG. 1 shows virtual scene data 1501, first color range data 1502, second color range data 1511, specific shape area pixel number data 1512, specific shape peripheral area pixel number data as various data. 1513 and determination parameter data 1514 are stored. As various images, a real image 1503, a specific color region image 1504, a CG image 1505, a CG drawing region image 1506, a superimposed real image 1507, a specific shape region image 1508, a specific shape peripheral region image 1509, a composite image 1510, A mask image 1515 is stored.

  In FIG. 1, the first memory 140 and the second memory 150 are only distinguished for convenience of explanation, and may be configured by one memory, for example.

  The information input device 160 inputs information to the image processing device 100.

  The bus 170 is for connecting the CPU 110, the display 120, the video camera 130, the first memory 140, the second memory 150, and the information input device 160 so that they can communicate with each other.

  Here, in this embodiment, 3D data representing a virtual scene to be drawn is stored in advance as virtual scene data 1501 in the second memory 150 via the information input device 160 or the like. is there. Data for expressing a specific color range is stored in advance as first color range data 1502 in the second memory 150 via the information input device 160 or the like. . As the first color range data 1502, for example, the upper and lower limits of three values representing colors in appropriate color space representation (for example, YUV, RGB, etc.) are set as color ranges. That's fine. As a result, the color region extraction unit 1401 determines whether the three values representing the pixel value to be presented are included in the upper and lower limit color ranges respectively provided, thereby determining the color Region extraction can be performed. In addition, a large number of sample colors representing a specific color range (first color range) are stored in advance as first color range data 1502, and the color area extraction unit 1401 approximates the color to the sample color. The color area may be extracted by comparing the degrees.

  Furthermore, in this embodiment, data for expressing a second color range different from the first color range of the first color range data 1502 is stored in advance via the information input device 160 or the like. The second color range data 1511 is stored in the memory 150. Further, data to be used for determination by the shape peripheral area determination unit 1407 is stored in advance as determination parameter data 1514 in the second memory 150 via the information input device 160 or the like.

  Hereinafter, the processing procedure of the image processing method by the image processing apparatus 100 of the present embodiment will be described.

  FIG. 2 is a flowchart showing an example of the processing procedure of the image processing method by the image processing apparatus 100 according to the embodiment of the present invention.

  First, in step S <b> 101, for example, the CPU 110 captures and acquires a real video (real image) using the video camera 130, and stores this as a real image 1503 in the second memory 150.

  FIG. 3 is a schematic diagram showing an example of the real image 1503 shown in FIG. The real image 1503 is, for example, a full color image.

  Subsequently, in step S102, the CPU 110 executes the program 1400, and the color area extraction unit 1401 selects colors included in the specific color range (first color range) from the real image 1503 acquired in step S101. Extracted image areas (color areas) are extracted. Specifically, the color area extraction unit 1401 determines whether or not all pixels in the real image 1503 are in a specific color range (first color range) indicated in the first color range data 1502. And a color region of a color included in the specific color range is extracted. At this time, as is conventionally done, pixels that are in a specific color range are labeled, and the area of the labeled area is smaller than a predetermined value or relative to other areas. For example, a process of not extracting a small area as a color area may be added. Thereafter, for example, the CPU 110 (color area extracting unit 1401) stores the color area extracted in step S102 in the second memory 150 as the specific color area image 1504.

FIG. 4 is a schematic diagram showing an example of the specific color area image 1504 shown in FIG.
The specific color area image 1504 is an image having 0 or 1 as a pixel value. Here, the pixel values of the color regions (image regions 402, 403, and 404 in FIG. 4) extracted as being a specific color range (first color range) from the real image 1503 shown in FIG. The pixel value of the other area (image area 401 in FIG. 4) is set to 0. Here, the image regions 402 and 403 indicate the image region of the observer's hand, and the image region 405 indicates the image region of the face of the person standing in front.

  Subsequently, in step S <b> 103, for example, the CPU 110 renders the virtual scene data 1501 stored in the second memory 150 to generate a CG image, and this is generated as a CG image 1505 in the second memory 150. Remember. Here, for example, the CPU 110 that performs the process of generating the CG image (the process of S103) constitutes a CG image generating unit. Here, as is done in the conventional mixed reality system, the position / posture of the actual video camera 130 may be matched with the position / posture of the virtual camera for CG rendering. For example, a position / posture sensor that acquires the position / posture of the video camera 130 or a method of estimating the position / posture of the video camera 130 based on the characteristics of the real image 1503 can be used.

  FIG. 5 is a schematic diagram showing an example of the CG image 1505 shown in FIG. FIG. 5 shows a spherical object image as the CG image 1505.

  Subsequently, in step S104, the CPU 110 executes the program 1400, and an area (CG drawing area) in which the CG object in the virtual scene data 1501 is rendered from the CG image 1505 by the CG drawing area extraction unit 1404. Extract. Specifically, the CG drawing area extraction unit 1404 determines a portion (area) where the CG object is drawn and a portion (area) where the CG object is drawn, and extracts a CG drawing area where the CG object is drawn. Do. At this time, whether or not the CG object is a pixel (area) on which the CG object is drawn is determined by, for example, checking the value of the CG Z buffer, whether or not there is a change from the pixel value cleared before CG drawing, etc. It may be determined by the following. Thereafter, for example, the CPU 110 (CG drawing area extraction unit 1404) stores the CG drawing area extracted in step S104 in the second memory 150 as a CG drawing area image 1506.

FIG. 6 is a schematic diagram showing an example of the CG drawing area image 1506 shown in FIG.
This CG drawing area image 1506 is an image having 0 or 1 as a pixel value. Here, the pixel value of an area (image area 601 in FIG. 6) extracted from the CG image 1505 shown in FIG. The pixel value of the sixth image area 602) is set to zero.

  Subsequently, in step S105, the CPU 110 executes the program 1400, and an image area (an area corresponding to a CG drawing area) on which a CG object is to be superimposed from the real image 1503 during synthesis by the real image cutout unit 1405. ). Thereafter, for example, the CPU 110 (real image cutout unit 1405) stores the image region cut out in step S105 in the second memory 150 as a superimposed real image 1507. This superimposed real image 1507 can be generated by multiplying each pixel value of the real image 1503 by each pixel value of the CG drawing region image 1506. In other words, the superimposed real image 1507 is generated as the value of the actual image 1503 in the region where the pixel value is 1 in the CG drawing region image 1506 (601 in FIG. 6), and the pixel value is generated in the CG drawing region image 1506. The pixel value of the area that is 0 (image area 602 in FIG. 6) is generated as 0.

FIG. 7 is a schematic diagram showing an example of the superimposed real image 1507 shown in FIG.
In FIG. 7, an image area 701 is an area having a pixel value as it is in the real image 1503 shown in FIG. 3, and an image area 702 is an area where the pixel value is 0 in the CG drawing area image 1506 shown in FIG. This is an area having a pixel value of 0 corresponding to the area 602). When multiplying each pixel value of the real image 1503 with each pixel value of the CG drawing area image 1506, the area having the pixel value of 1 in the CG drawing area image 1506 is expanded by an appropriate width, and then multiplied. You may make it match. In this case, for example, it may be expanded outward by a predetermined number of pixels. By doing so, it becomes a measure against a case where the face area of the person photographed as the real image 1503 is partially overlapped at the end of the CG drawing area.

  Subsequently, in step S106, the CPU 110 executes the program 1400, and the shape region extraction unit 1402 extracts a region (specific shape region) recognized as having a specific shape from the superimposed real image 1507. To do. Here, the specific shape is, for example, the shape of a human face. At this time, a technique for extracting an area recognized as a human face from a certain image is known. Here, what kind of region is extracted depends on a face recognition algorithm. For example, if feature extraction is performed in the skin color area on the image and there is a part that can be recognized as the eyes and mouth, a rectangular area including the positions of both eyes and mouth can be extracted as the face area. Thereafter, for example, the CPU 110 (shape area extraction unit 1402) stores the specific shape area extracted in step S106 in the second memory 150 as the specific shape area image 1508.

FIG. 8 is a schematic diagram showing an example of the specific shape region image 1508 shown in FIG.
FIG. 8 shows a recognition process of a specific shape (here, a face) on the superimposed real image 1507 shown in FIG. 7, and the position of the recognized shape is represented by a rectangle. Here, the pixel value of an area (image area 801 in FIG. 8) extracted as having a specific shape from the superimposed real image 1507 shown in FIG. 7 is 1, and the other area (image in FIG. 8) is selected. The pixel value in the area 802) is set to 0. In the example shown in FIG. 8, the extracted region is rectangular for ease of explanation, but it may be indefinite. For example, it may be possible to extract a continuous skin-color region including a portion recognized as an eye / mouth in an irregular shape.

  Subsequently, in step S107, the CPU 110 executes the program 1400, is continuous with the specific shape area extracted in step S106 by the shape peripheral area extraction unit 1406, and is based on the color range of the second color range data 1511. Extract the area around the specific shape. Details of the processing in step S107 will be described later with reference to FIG. Thereafter, for example, the CPU 110 (shape peripheral region extraction unit 1406) stores the specific shape peripheral region extracted in step S107 as the specific shape peripheral region image 1509 in the second memory 150.

FIG. 9 is a schematic diagram showing an example of the specific shape peripheral area image 1509 shown in FIG.
In FIG. 9, the pixel value of the region (image regions 901, 902, and 903 in FIG. 9) extracted as the peripheral region of the specific shape is 1, and the pixel value of the other region (image region 904 in FIG. 9) is 0. It is. An image area 901 is an area in the forehead of the person's face in the real image 1503 shown in FIG. 3, an image area 902 is an area in the ear of the person in the real image 1503 shown in FIG. 3, and an image area 903 is the reality shown in FIG. This is an area on the chin / neck of the person in the image 1503. These areas are areas that are continuous with an area recognized as a face (image area 801 in FIG. 8) and have a color in the second color range (here, skin color). Note that the region where the hand is photographed is a skin color, but is not extracted here because it is not continuous with the region recognized as a face.

  Subsequently, in step S108, the CPU 110 executes the program 1400 to operate the shape peripheral area determination unit 1407. Then, the shape peripheral area determination unit 1407 compares the number of pixels having a pixel value of 1 in the specific shape area image 1508 with the number of pixels having a pixel value of 1 in the specific shape peripheral area image 1509. Then, it is determined whether or not the ratio is a certain value or more. That is, the process of step S108 determines (determines) whether or not the area ratio of the specific shape region to the specific shape peripheral region is a certain value or more. Details of the processing in step S108 will be described later with reference to FIG.

  As a result of the determination in step S108, the ratio between the number of pixels having a pixel value of 1 in the specific shape region image 1508 and the number of pixels having a pixel value of 1 in the specific shape peripheral region image 1509 is greater than or equal to a certain value. If there is (S108 / YES), the process proceeds to step S109.

  In step S109, the CPU 110 executes the program 1400, and the shape peripheral area reduction unit 1408 performs a reduction process for reducing the specific shape peripheral area extracted in step S107. Details of the processing in step S109 will be described later with reference to FIG.

  When the process of step S109 is completed, or when the ratio of the number of pixels in a predetermined pixel value between the specific shape area image 1508 and the specific shape peripheral area image 1509 is not a predetermined value or more in step S108 (S108 / NO). The process proceeds to step S110.

  In step S110, the CPU 110 executes the program 1400, and the synthesizing unit 1403 synthesizes the real image 1503 and the CG image 1505 to generate a synthesized image. Details of the processing in step S110 will be described later with reference to FIG. Thereafter, for example, the CPU 110 (compositing means 1403) stores the composite image generated in step S110 as the composite image 1510 in the second memory 150.

  Subsequently, in step S111, the CPU 110 outputs and displays the composite image 1510 generated in step S110 on the display 120.

  Through the series of processes shown in FIG. 2 described above, a composite image 1510 for one frame is output and displayed on the display 120. If necessary, the composite image 1510 can be displayed as a moving image on the display 120 by repeatedly performing a series of processes shown in FIG.

Next, detailed processing in step S107 of FIG. 2 performed by the shape peripheral area extraction unit 1406 will be described.
FIG. 10 is a flowchart illustrating an example of a detailed processing procedure of step S107 illustrated in FIG.

  First, in step S201 of FIG. 10, the shape peripheral area extraction unit 1406 uses the second color range data 1511 to select a color area having a pixel value included in the second color range from the real image 1503. Extract.

FIG. 11 is a schematic diagram showing an example of an image of the color region extracted in step S201 of FIG.
In the image of the color area shown in FIG. 11, the pixel value of the area determined to be the second color range (image areas 1101, 1102, and 1103 in FIG. 11) is 1, and the other area (image area in FIG. 11). 1104) is set to 0.

  Subsequently, in step S202, the shape peripheral area extraction unit 1406 ORs (that is, adds) the pixel value of the specific shape area image 1508 to the color area image (FIG. 11) extracted in step S201.

  FIG. 12 is a schematic diagram illustrating an example of an image obtained by the process of step S202 of FIG. The image shown in FIG. 12 is obtained by adding the color region image shown in FIG. 11 and the specific shape region image 1508 shown in FIG. That is, the image area 1201 in FIG. 12 is an area obtained by adding the image area 1101 in FIG. 11 and the image area 801 in FIG.

  Subsequently, in step S203, the shape peripheral region extraction unit 1406 labels the image (FIG. 12) obtained by the processing in step S202 so that continuous regions can be collected and distinguished from each other. For example, the label number assigned here may be included in the pixel value. Here, for example, the pixel value of the image region 1201 in FIG. 12 is set to 1, the pixel value of the image region 1202 is set to 2, and the pixel value of the image region 1203 is set to 3.

  Subsequently, in step S204, the shape peripheral region extraction unit 1406 performs pixel processing in the region having the same label as the region corresponding to the region having the pixel value 1 in the specific shape region image 1508 (image region 801 in FIG. 8). The value is 1. Then, the shape peripheral region extraction unit 1406 changes the pixel value of the region with another label to 0.

  FIG. 13 is a schematic diagram illustrating an example of an image obtained by the process of step S204 of FIG. In FIG. 13, an image area 1301 is an area having a pixel value of 1, and an image area 1302 is an area having a pixel value of 0.

  Subsequently, in step S205, the shape peripheral area extraction unit 1406 converts the image obtained in the process of step S204 into an area having a pixel value of 1 in the specific shape area image 1508 (image area 801 in FIG. 8). The corresponding pixel value is set to zero. As a result, a specific shape peripheral area image 1509 shown in FIG. 9 is obtained.

  Through the processes in steps S201 to S205 described above, the specific shape peripheral area image 1509 is generated.

  Here, in the present embodiment, as described above, the color range (second color range) used when extracting the color region in step S201 is set to the specific color range (first color range) of the first color range data 1502. Color range). The reason is that the peripheral area of the specific shape is extracted more stably. Specifically, when the hand / arm region is to be extracted by the first color range data 1502, if the color range is widened, pixels that are not the hand / arm are extracted, so that the color range is not much. Cannot be set widely. On the other hand, the region desired to be extracted by the shape peripheral region extraction unit 1406 is trying to extract a color region that is continuous with a region that is not a hand / arm, so there is no problem even if the color range is a little wide. For example, when a skin color region continuous to a region recognized as a face is extracted, if a color range slightly wider than the first color range data 1502 is set, there is an advantage that the skin color region continuous to the face region is easily connected. is there. As described above, it is advantageous to use the second color range data 1511 different from the first color range data 1502 in the process of step S201. However, for the sake of simplicity, the color range used in step S201 may use a specific color range (first color range) in the first color range data 1502 instead of the second color range. .

Next, detailed processing in step S108 of FIG. 2 performed by the shape peripheral area determination unit 1407 will be described.
FIG. 14 is a flowchart illustrating an example of a detailed processing procedure of step S108 illustrated in FIG.

  First, in step S <b> 301, the shape peripheral region area determination unit 1407 calculates the number of pixels having a pixel value of 1 (specific shape region pixel number) for the specific shape region image 1508. For example, the CPU 110 (shape peripheral region area determination unit 1407) stores the data obtained as a result of the calculation in step S301 in the second memory 150 as the specific shape region pixel number data 1512.

  Subsequently, in step S302, the shape peripheral region area determination unit 1407 calculates the number of pixels having a pixel value of 1 (number of specific shape peripheral region pixels) for the specific shape peripheral region image 1509. Then, for example, the CPU 110 (shape peripheral region area determination unit 1407) stores the data obtained as a result of the calculation in step S302 in the second memory 150 as the specific shape peripheral region pixel number data 1513.

  Subsequently, in step S303, the shape peripheral region area determination unit 1407 calculates the ratio of the number of specific shape region pixels to the number of specific shape peripheral region pixels calculated in steps S301 and S302. Compare with value. Then, the shape peripheral area determination unit 1407 determines whether or not the ratio is equal to or greater than a certain value of the determination parameter data 1514.

Here, the determination parameter data 1514 will be described.
The determination parameter data 1514 is used to determine a case where a specific shape, for example, a region recognized as a human face is connected to other regions, for example, regions that are hands and arms. That is, whether or not it is connected to another skin color area by determining whether the surrounding skin color area connected to the face is relatively larger than the area of the area recognized as a face having a specific shape Is to judge.

  Therefore, for example, when the area ratio of the specific shape region recognized as the face to the specific shape peripheral region such as the skin color region continuous thereto is equal to or larger than the predetermined value of the determination parameter data 1514 (S303 / YES), step S304. Proceed to In step S304, the shape peripheral region area determining unit 1407 determines that the specific shape peripheral region of another skin color object (for example, a hand / arm) has overlapped the specific shape region such as the face. In this case, in the subsequent step, step S109, the shape peripheral area reduction unit 1408 performs processing so that the specific shape peripheral area falls within an appropriate range (for example, a range that does not overlap the specific shape area).

  On the other hand, if a negative determination is made as a result of the determination in step S303 (S303 / NO), in step S305, the shape peripheral area determination unit 1407 does not overlap the specific shape peripheral area with the specific shape area such as the face. Is determined.

  In the present embodiment, the determination method here uses a simple area ratio. However, if it can be determined that the specific shape peripheral region is too large compared to the specific shape region, a separately defined method is used. It may be due to.

  The process of step S108 of FIG. 2 is completed through the processes of steps S301 to S305.

Next, detailed processing in step S109 of FIG. 2 performed by the shape peripheral area reduction unit 1408 will be described.
FIG. 15 is a flowchart illustrating an example of a detailed processing procedure of step S109 illustrated in FIG.

  First, in step S <b> 401, the shape peripheral area reduction unit 1408 sets the center of gravity P of the specific shape area image 1508. Here, the gravity center position is set as a temporary variable as the gravity center P.

Subsequently, in step S402, the shape peripheral area reduction unit 1408 obtains the distance and coordinates from the position corresponding to the center of gravity P for all the pixels having a pixel value of 1 in the specific shape peripheral area image 1509, and arranges them. Store in A. Here, the array A is an array composed of three contents, for example, the distance from the center of gravity P, the x coordinate of the pixel, and the y coordinate of the pixel. In addition, the size of the array is equal to the value of the specific shape peripheral area pixel number data 1513.

  Subsequently, in step S403, the shape peripheral area reduction unit 1408 sorts the array A in ascending order of distance.

  Subsequently, in step S404, the shape peripheral area reduction unit 1408 obtains a value N calculated by N = (specific shape area pixel number data 1512) × (determination parameter data 1514). This value N is obtained by obtaining the number of pixels in the specific shape surrounding area such that the ratio between the number of pixels in the specific shape area and the number of pixels in the peripheral area of the specific shape is less than (or less than) the determination parameter data 1514 is there.

  Subsequently, in step S405, the shape peripheral area reduction unit 1408 determines whether the value N is less than the specific shape peripheral area pixel number.

  As a result of the determination in step S405, if the value N is not less than the number of pixels in the peripheral area of the specific shape, that is, it is more than that (S405 / NO), the process proceeds to step S406.

  In step S406, the shape peripheral area reduction unit 1408 sets 0 to the pixel corresponding to the Nth element An of the array A sorted in step S403. Thereafter, in step S407, the shape peripheral area reduction unit 1408 adds 1 to the current value N, changes the current value N, and then returns to step S405.

  That is, in steps S405 to S407, pixels from the Nth element An of the array A sorted in step S403 to the last element (the total number of elements of the array A is equal to the value of the specific shape peripheral area pixel number data 1513). The pixel value corresponding to is set to 0. Here, since each element of the array A stores the x-coordinate and y-coordinate in the specific shape peripheral area image 1509 of the pixel, the value of the corresponding pixel in the specific shape peripheral area image 1509 is set. Can do.

  On the other hand, if the result of determination in step S405 is that the value N is less than the specific shape peripheral area pixel number (S405 / YES), the processing of the flowchart shown in FIG.

  Through the processes of steps S401 to S407, the ratio between the number of pixels having a pixel value of 1 in the specific shape area image 1508 and the number of pixels having a pixel value of 1 in the specific shape peripheral area image 1509 is The determination parameter data 1514 falls below a certain value.

  In the present embodiment, in the specific shape peripheral area image 1509, an object close to the centroid P of the specific shape area image 1508 is left as a pixel value 1 and an object far from the centroid P is deleted (the pixel value becomes 0). However, other methods may be used. In this case, for example, in the processing of the present embodiment, the region having the pixel value of 1 in the specific shape peripheral region image 1509 is circular as a result, but the target recognized as the specific shape is a human face. In view of the above, it may be possible to process the remaining area to be elliptical.

Next, detailed processing in step S110 of FIG. 2 performed by the synthesizing unit 1403 will be described.
FIG. 16 is a flowchart illustrating an example of a detailed processing procedure of step S110 illustrated in FIG.

  First, in step S501, the synthesizing unit 1403 copies the specific color area image 1504 (FIG. 4) to the mask image 1515 of the second memory 150.

  Subsequently, in step S502, the synthesizing unit 1403 performs a process of inverting all the pixel values of the specific shape region image 1508 (FIG. 8).

  Subsequently, in step S503, the synthesizing unit 1403 ANDs all the pixels of the mask image 1515 with the pixels of the specific shape region image 1508 at the corresponding position (inverted in step S502), and obtains the result. Reflected in the mask image 1515.

  FIG. 17 is a schematic diagram showing an example of the mask image 1515 obtained by the process of step S503 of FIG.

  Subsequently, in step S504, the synthesizing unit 1403 performs a process of inverting all the pixel values of the specific shape peripheral area image 1509 (FIG. 9).

  Subsequently, in step S505, the synthesizing unit 1403 ANDs all the pixels of the mask image 1515 with the pixels of the specific shape peripheral area image 1509 at the corresponding position (inverted in step S504), and the result. Is reflected in the mask image 1515.

  FIG. 18 is a schematic diagram showing an example of a mask image 1515 obtained by the process of step S505 of FIG.

  Subsequently, in step S506, the synthesizing unit 1403 performs a process of inverting all the pixel values of the mask image 1515.

  Subsequently, in step S507, the synthesizing unit 1403 multiplies corresponding pixel values in the CG image 1505 and the mask image 1515 inverted in step S506, and reflects the result in the CG image 1505.

  Subsequently, in step S <b> 508, the synthesizing unit 1403 copies the real image 1503 to the synthesized image 1510 inside the second memory 150.

  Subsequently, in step S509, the synthesizing unit 1403 performs processing for overwriting and superimposing the CG image 1505 processed in step S507 on the synthesized image 1510 to synthesize the real image 1503 and the CG image 1505. At this time, if the CG image 1505 includes a semi-transparent region, processing such as blending is appropriately performed. Then, the process of the flowchart shown in FIG.

  Through the processes in steps S501 to S509, the composite image 1510 is combined in step S110 in FIG.

FIG. 19 is a schematic diagram showing an example of a composite image 1510 obtained by the composite processing in step S110 (or FIG. 16) in FIG.
In the synthesized image 1510 shown in FIG. 19, the CG image 1505 is masked by extracting the hand image area that overlaps the CG image 1505 shown in FIG. 5 as a skin color area. The hand of the real image 1503 shown in FIG. In addition, since the face area that overlaps the CG image 1505 and the skin color area that is continuous with the face area are skin colors, they are extracted as having a specific shape and a predetermined color area that is continuous with the skin color. They are superimposed as they are without being masked.

  Instead of multiplying the inverted mask image 1515 and the CG image 1505 as in step S507, the mask image 1515 is stored in the stencil buffer without being inverted, and then the virtual scene is rendered to obtain the CG image 1505. A method is also conceivable. In the present embodiment, the CG image 1505 has already been generated in order to limit the region for shape recognition. However, if the processing for limiting the shape recognition region and improving the efficiency is not performed, the virtual scene may not be rendered until this processing is reached. In such a case, it is also effective to perform CG drawing using a stencil buffer during the synthesis process.

  According to the present embodiment, when a synthesized image is generated by synthesizing a real image and a CG image, it is possible to realize generation of a more natural synthesized image without a sense of incompatibility compared to the conventional method.

(Other embodiments of the present invention)
Each unit (1401 to 1408) included in the above-described image processing apparatus 100 according to the embodiment of the present invention and each step of FIGS. 2, 10, and 14 to 16 showing the image processing method are programmed by the CPU 110. This can be realized by executing 1400. The program 1400 and a computer-readable recording medium (for example, the first memory 140) on which the program 1400 is recorded are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  In the present invention, a software program that realizes the functions of the above-described embodiments (in the embodiment, programs corresponding to the flowcharts shown in FIGS. 2, 10, and 14 to 16) is directly or directly stored in a system or apparatus. Includes remote supplies. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  Note that each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 is a schematic diagram illustrating an example of a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows an example of the process sequence of the image processing method by the image processing apparatus which concerns on embodiment of this invention. It is a schematic diagram which shows an example of the real image shown in FIG. It is a schematic diagram which shows an example of the specific color area image shown in FIG. It is a schematic diagram which shows an example of the CG image shown in FIG. It is a schematic diagram which shows an example of the CG drawing area | region image shown in FIG. It is a schematic diagram which shows an example of the to-be-superimposed real image shown in FIG. It is a schematic diagram which shows an example of the specific shape area | region image shown in FIG. It is a schematic diagram which shows an example of the specific shape periphery area | region image shown in FIG. It is a flowchart which shows an example of the detailed process sequence of step S107 shown in FIG. It is a schematic diagram which shows an example of the image of the extracted color area | region of step S201 of FIG. It is a schematic diagram which shows an example of the image obtained by the process of step S202 of FIG. It is a schematic diagram which shows an example of the image obtained by the process of step S204 of FIG. It is a flowchart which shows an example of the detailed process sequence of step S108 shown in FIG. It is a flowchart which shows an example of the detailed process sequence of step S109 shown in FIG. It is a flowchart which shows an example of the detailed process sequence of step S110 shown in FIG. It is a schematic diagram which shows an example of the mask image obtained by the process of step S503 of FIG. It is a schematic diagram which shows an example of the mask image obtained by the process of step S505 of FIG. It is a schematic diagram which shows an example of the synthesized image obtained by the synthesis process of step S110 (or FIG. 16) of FIG.

Explanation of symbols

100 Image processing apparatus 110 CPU
120 Display 130 Video camera 140 First memory 1401 Color area extraction means 1402 Shape area extraction means 1403 Composition means 1404 CG drawing area extraction means 1405 Real image extraction means 1406 Shape peripheral area extraction means 1407 Shape peripheral area area determination means 1408 Shape periphery Area reduction means 150 Second memory 1501 Virtual scene data 1502 First color range data 1503 Real image 1504 Specific color area image 1505 CG image 1506 CG drawing area image 1507 Overlaid real image 1508 Specific shape area image 1509 Specific shape peripheral area Image 1510 Composite image 1511 Second color range data 1512 Specific shape region pixel number data 1513 Specific shape peripheral region pixel number data 1514 Determination parameter data 1515 Mask image 160 Information input Power device 170 bus

Claims (16)

Color area extraction means for extracting a color area included in a specific color range from a real image;
CG image generation means for generating a CG image based on virtual scene data;
A shape region extracting means for extracting a region having a specific shape from the real image as a specific shape region;
Combining means for synthesizing the real image and the CG image by excluding areas other than the specific shape area extracted by the shape area extracting means from among the areas extracted by the color area extracting means. An image processing apparatus. CG drawing area extracting means for extracting, as a CG drawing area, an area in which the CG object of the virtual scene data is drawn from the CG image;
A real image cutout unit for cutting out an area corresponding to the CG drawing area from the real image;
The image processing apparatus according to claim 1, wherein the shape area extraction unit extracts the specific shape area from a real image cut out by the real image cutout unit. A shape peripheral area extracting unit that extracts a color area that is continuous with the specific shape area and that is included in the specific color range as a specific shape peripheral area;
In addition to the color region extracted by the color region extraction unit and the specific shape region extracted by the shape region extraction unit, the synthesizing unit further adds a specific shape peripheral region extracted by the shape peripheral region extraction unit. The image processing apparatus according to claim 1, wherein the composition processing is performed based on the image processing device. A shape peripheral region area determining means for determining whether a ratio of an area of the specific shape region to the specific shape peripheral region is a certain value or more;
When the ratio of the areas is a certain value or more, further comprising a shape peripheral area reduction means for reducing a region far from the center of gravity of the specific shape area among the specific shape peripheral area,
The said synthesis | combination means performs the said synthetic | combination process based on the specific shape peripheral area reduced by the said shape peripheral area reduction means, when performing the said synthetic | combination process based on the said specific shape peripheral area. An image processing apparatus according to 1.   The image processing apparatus according to claim 4, wherein the shape peripheral area determination unit uses a ratio between the number of pixels in the specific shape area and the number of pixels in the specific shape peripheral area as the ratio of the areas.   The shape surrounding area extracting unit, when extracting the specific shape surrounding area, replaces the color area included in the specific color range with a color included in a second color range different from the specific color range. The image processing apparatus according to claim 3, wherein a region is used. The shape area extraction means extracts a predetermined area of the human body as the specific shape area,
The synthesizing unit is configured to synthesize the real image and the CG image by excluding a region other than the predetermined region of the human body from the regions extracted by the color region extracting unit. The image processing apparatus according to any one of claims 1 to 6. A color region extracting step in which the color region extracting means extracts a color region included in a specific color range from the real image;
A CG image generation step in which the CG image generation means generates a CG image based on the virtual scene data;
A shape region extracting step, wherein the shape region extracting means extracts a region having a specific shape from the real image as the specific shape region;
A synthesis unit that synthesizes the real image and the CG image by excluding a region other than the specific shape region extracted in the shape region extraction step from the regions extracted in the color region extraction step. An image processing method comprising: steps. A color region extraction step for extracting a color region included in a specific color range from a real image;
A CG image generation step for generating a CG image based on the virtual scene data;
A shape region extraction step for extracting a region having a specific shape as a specific shape region from the real image;
Among the color area extraction step is extracted in the region, the shaped region extracted by excluding an area other than the specific shaped region extracted in step, the real image and the CG image and the computer and synthesis steps of synthesis process A program to make it run. Color area extraction means for extracting a color area included in a specific color range from a real image;
CG image generation means for generating a CG image based on virtual scene data;
CG drawing area extracting means for extracting, as a CG drawing area, an area in which the CG object of the virtual scene data is drawn from the CG image;
A real image cutout means for cutting out an area corresponding to the CG drawing area from the real image;
A shape area extracting means for extracting an area having a specific shape as a specific shape area from the real image cut out by the real image cutout means;
A combining unit configured to combine the real image and the CG image based on the color region extracted by the color region extracting unit and the specific shape region extracted by the shape region extracting unit;
An image processing apparatus comprising: A shape peripheral area extracting unit that extracts a color area that is continuous with the specific shape area and that is included in the specific color range as a specific shape peripheral area;
In addition to the color region extracted by the color region extraction unit and the specific shape region extracted by the shape region extraction unit, the synthesizing unit further adds a specific shape peripheral region extracted by the shape peripheral region extraction unit. The image processing apparatus according to claim 10, wherein the composition processing is performed based on the image processing. A shape peripheral region area determining means for determining whether a ratio of an area of the specific shape region to the specific shape peripheral region is a certain value or more;
When the ratio of the areas is equal to or greater than a certain value, a shape peripheral area reduction unit that performs reduction processing of an area that is far from the center of gravity of the specific shape area among the peripheral areas of the specific shape;
Further comprising
12. The synthesizing unit, when performing the synthesizing process based on the specific shape peripheral area, performs the synthesizing process based on the specific shape peripheral area reduced by the shape peripheral area reducing unit. An image processing apparatus according to 1. 13. The image processing apparatus according to claim 12, wherein the shape peripheral area determination unit uses a ratio between the number of pixels in the specific shape area and the number of pixels in the specific shape peripheral area as the ratio of the areas. The shape surrounding area extracting unit, when extracting the specific shape surrounding area, replaces the color area included in the specific color range with a color included in a second color range different from the specific color range. The image processing apparatus according to claim 11, wherein a region is used. A color region extracting step in which the color region extracting means extracts a color region included in a specific color range from the real image;
A CG image generation step in which the CG image generation means generates a CG image based on the virtual scene data;
A CG drawing area extracting means for extracting, from the CG image, an area in which the CG object of the virtual scene data is drawn as a CG drawing area;
A real image cutout unit that cuts out an area corresponding to the CG drawing area from the real image;
A shape region extracting unit that extracts a region having a specific shape as a specific shape region from the real image cut out by the real image cutting step;
A synthesis step for synthesizing the real image and the CG image based on the color region extracted in the color region extraction step and the specific shape region extracted in the shape region extraction step;
An image processing method comprising: A color region extraction step for extracting a color region included in a specific color range from a real image;
A CG image generation step for generating a CG image based on the virtual scene data;
A CG drawing area extracting step of extracting, from the CG image, an area where a CG object of the virtual scene data is drawn as a CG drawing area;
A real image cutout step of cutting out an area corresponding to the CG drawing area from the real image;
A shape region extraction step for extracting a region having a specific shape as a specific shape region from the real image cut out by the real image cutting step;
A synthesis step for synthesizing the real image and the CG image based on the color region extracted in the color region extraction step and the specific shape region extracted in the shape region extraction step;
A program that causes a computer to execute.

Images