JP4438557B2 - Imaging apparatus, information processing apparatus, information processing method, and computer program - Google Patents

Description

  The present invention relates to an information processing apparatus, information processing method, and computer program for processing digital data, and more particularly to an imaging apparatus, information processing apparatus, information processing method, and computer for processing captured image data.・ Regarding the program.

  More specifically, the present invention relates to an imaging apparatus, an information processing apparatus, an information processing method, and a computer program that extract an image of a specific object included in a captured image, and in particular, using reflected light from the object. The present invention relates to an imaging apparatus, an information processing apparatus, an information processing method, and a computer program that extract an image of a specific object included in a captured image.

  Along with the recent rapid progress in the information technology (IT) field, various forms of computer systems have been developed and manufactured, and are widely used in universities, other research institutions, corporate offices, and even general households. . On a computer, in addition to text-format document files, various media such as audio, images, and natural languages are digitized and handled mathematically, so that information editing / processing, storage, management, transmission, sharing, etc. are more advanced. It is possible to perform a wide variety of processing.

  For example, image processing techniques for performing digital image processing such as transformation, color change, image quality improvement, re-encoding, etc. on an image using a computer are widely spread. This image processing technology includes special effects that remove objects such as utility poles that impair the beauty of landscape photographs, and morphing that smoothly changes from human faces to animal faces. In addition, image processing techniques such as processing of photographic images sent from artificial satellites and processing of diagnostic images read by CT scanners are also applied to various specialized fields such as science and medicine.

  Further, according to the image processing technique, it is possible to easily input a gesture or a motion without wearing a special device (see, for example, Patent Document 1). In this case, an object image is extracted from input by reflected light or radiant light, the shape of the reflected light image is interpreted, and the shape is converted into an instruction (user interaction). Further, in order to cut out the character from the background image, the specific image is decomposed, and the specific portion is extracted based on the obtained decomposed image. Further, the extracted image and background are stored, the arrangement position of the reflected light image is designated, and when the arrangement position of the stored reflected light image is designated, the reflected light is reflected at the designated arrangement position with respect to the stored background. Arrange the images. As a result, pointing in the three-dimensional space and change of the starting point can be easily performed. In addition, the user's gestures and movements can be used as they are, and direct operations such as adding natural movements to animated characters can be performed. Furthermore, it is possible to cut out only a specific character or to input character depth information.

  FIG. 24 shows a configuration example of an information input device provided with means for extracting only a specific object. The object extraction unit 100 includes a lens 102, a reflected light extraction unit 101, a timing control unit 108, and a light emitting unit 107. Further, the reflected light extraction unit 101 includes a first light receiving unit 103, a second light receiving unit 104, a difference calculation unit 105, an amplifier 106, and an A / D conversion unit 109. Below, the extraction method of the object 110 using this object extraction part 100 is demonstrated.

  The light emitting unit 107 blinks under the control of the timing control unit 108, and the blinking light 111 illuminates the object 110. The blinking light 111 may be visible light or infrared light.

  The first light receiving unit 103 acquires the image of the object 110 and its background when the light emitting unit 107 is turned off. Next, the second light receiving unit 104 acquires the image of the object 110 and its background when the light emitting unit 107 is lit.

  When there is no movement in the image, the image acquired by the second light receiving unit 104 is obtained by adding light emitted from the light emitting unit 107 to the image acquired by the first light receiving unit 103. This addition amount increases as the object is closer to the light emitting unit 107.

  Next, the image data acquired by the light receiving units 103 and 104 is transferred to the difference calculation unit 105. Here, the difference in the amount of received light is calculated from each image data. The difference value becomes larger as the object strongly receives the light from the light emitting unit 107 (that is, the object is strongly reflected). That is, assuming that the reflectance of the object is equal, the difference value is proportional to the distance from the light emitting unit 107 to the object. Thereafter, the difference value is amplified by the amplifier 106 and further converted into digital data by the A / D conversion unit 109 before proceeding to the next step.

  Since the first light receiving unit 103 and the second light receiving unit 104 do not acquire images at the same time, they may be the same light receiving device. Further, as the light receiving units 103 and 104, solid-state imaging devices in which detection units (pixels) generally formed of photodiodes are two-dimensionally arranged are used. Therefore, the above-described difference value can be obtained for each pixel. That is, the difference value of a pixel that images an object that is close to the light emitting unit 107 is large, and the difference value of a pixel that images a far object is small. As described above, since each pixel has a difference value proportional to the distance, an appropriate threshold value is set, and an object is easily extracted by displaying only a value greater than (or less) than that value. be able to. According to the apparatus configuration shown in FIG. 24, the object 110 can be cut out from the background and displayed, and interaction using this data is also possible.

  Here, the present inventors consider that the object extraction method as described above has the following problems.

  Normally, the object extraction unit 100 is built in the camera device. In this case, it is assumed that the user moves and operates the camera device itself to capture an image of the object 110, for example. However, the amount of light changes according to the movement of the camera itself, which may lead to erroneous recognition. If the background image from which the flashing light 111 does not reach is fixed, the brightness difference is not detected in the pixels that capture this portion. On the other hand, if the entire background moves due to the movement of the camera itself, some brightness change occurs in the pixels that image them. This change in brightness is misrecognized as a distance, and correct distance data cannot be acquired. This makes object extraction difficult.

  Furthermore, if there is flashing light in the background, there is also a temporal change in brightness here, which is erroneously recognized as a distance. For example, when a CRT display is placed in the background, there is a possibility that the shape of the object is erroneously recognized because of a display screen that switches at a predetermined frame rate.

  As described above, in the conventional image processing system, when there is a change in brightness in the background, there is a problem that all of these are misrecognized as distance information, so that object extraction is difficult. .

Japanese Patent No. 3321053

  An object of the present invention is to provide an excellent imaging apparatus, information processing apparatus, information processing method, and computer program capable of suitably extracting an image of a specific object included in a captured image.

  A further object of the present invention is to provide an excellent imaging apparatus, information processing apparatus, information processing method, and computer that can suitably extract an image of a specific object included in a captured image using reflected light from the object.・ To provide a program.

  A further object of the present invention is to more accurately extract an image of a specific object contained in a photographed image using reflected light from the object even when the brightness changes in the background. Another object of the present invention is to provide an excellent imaging apparatus, information processing apparatus, information processing method, and computer program.

  The present invention has been made in consideration of the above-described problems, and a first aspect thereof includes an irradiation unit that emits arbitrarily-coded modulated light, and reflected light from a specific object with respect to the modulated light. The imaging apparatus includes an imaging unit that captures an image and an arithmetic unit that processes the captured image based on decoding of reflected light with respect to the modulated light.

  If an image processing technique using a computer is used, an image of an object included in the input photographed image can be extracted. In this case, generally, an object image is extracted from an input by reflected light or radiant light, and the shape of the reflected light image is recognized. However, in this case, there is a possibility that it may be erroneously recognized due to the influence of the change in the amount of light due to the movement of the camera itself, or there is a problem that the shape of the object is erroneously recognized due to the influence of the flashing light in the background.

  On the other hand, according to the present invention, the arithmetic unit performs the decoding of the modulated light, and determines that the portion of the captured image that has been correctly decoded is the portion irradiated with the modulated light, Only the image portion of the specific object can be extracted more accurately from the captured image.

  That is, according to the present invention, since the object is recognized based on whether or not the modulated light with an arbitrary coding has been correctly decoded, the method of knowing the distance by the light difference as in the prior art Unlike this, it is possible to provide a robust system that is not erroneously recognized even when the brightness changes in the background.

  Here, the imaging unit uses a CCD (Charge Coupled Device) or MOS (Metal Oxide Semiconductor) type imaging device configured by two-dimensionally arranging micro photoelectric conversion devices. It is done. Then, the arithmetic unit tries to decode the modulated light for each image sensor, and based on whether or not the original code pattern applied to the modulated light from the output signal of the image sensor can be correctly decoded, It can be separated into a portion irradiated with the modulated light and a portion not irradiated.

  You may further provide the memory | storage part which memorize | stores the captured image for several frames in the said imaging part. The arithmetic unit can decode the code pattern by comparing the output signals of the corresponding pixels between the preceding and succeeding frames. In addition, by suitably utilizing a semiconductor mounting technology, the imaging unit, the storage unit, and the arithmetic unit can be configured by a single semiconductor circuit chip.

  The irradiating unit irradiates modulated light including a wavelength component of invisible light. The irradiating unit may irradiate two or more types of modulated light having at least one of a code, a coding method, a light intensity, and a light emission angle.

  The irradiating unit irradiates the modulated light by coding the irradiation light from the light source with a predetermined code pattern according to a predetermined light emission frequency, for example. In this case, the imaging unit captures an image at a frame rate equal to or higher than the emission frequency so that the reflected light with respect to the coded modulated light can be decoded. The arithmetic unit decodes a code included in the reflected light with respect to the modulated light from the specific object, based on the output signal from the imaging unit. For example, when the modulated light is coded in the irradiation unit and the modulated light is decoded in the arithmetic unit, the imaging unit is switched to a higher frame rate.

  The imaging apparatus according to the present invention includes a normal image capturing mode in which the image capturing unit performs normal image capturing, and a decoding / decoding in which the image capturing unit performs image capturing for decoding the modulated light in the arithmetic unit. A mode may be provided. For example, the normal image capturing mode and the decoding mode may be alternately operated in a time division manner. In this case, in order to obtain an image of the object that is not affected by the modulated light component, acquisition of a normal image in which only the ambient light component is irradiated on the object and image acquisition for object extraction are performed in a time-sharing manner.

  In the normal image capturing mode, the irradiating unit does not irradiate modulated light, and the imaging unit captures an image composed of an object with respect to ambient light (or natural light) or reflected light from a landscape. In the decode mode, the irradiating unit irradiates modulated light, so that the imaging unit captures an image composed of ambient light and reflected light with respect to the modulated light.

  For example, the arithmetic unit can specify the specific object portion in the captured image based on the result of decoding the modulated light from the captured image under the decoding mode. Then, a corresponding portion of the captured image under the normal image capturing mode can be extracted as the image of the specific object.

  Here, for the convenience of photographing the object in time division as described above, a spatial shift may occur in the position of the object in the image frame. Therefore, the calculation unit is configured to identify the specific image from the captured image under the normal image capturing mode while considering the influence of the movement of the object due to the temporal difference in the captured image between the decode mode and the normal image capturing mode. An image of the object may be extracted.

  In this case, the calculation unit obtains a motion vector representing a spatial difference of the specific object due to a temporal difference in the captured image between the decode mode and the normal image capturing mode. Then, based on the motion vector, address conversion is performed from the specific object portion in the captured image under the decoding mode to the specific object portion in the captured image under the normal image capturing mode. Thus, the image portion of the specific object can be more accurately extracted from the normal captured image.

  Further, the irradiating unit emits two or more types of modulated light having different codes, and the arithmetic unit performs decoding of the modulated light for each code and extracts a portion irradiated with each modulated light. Also good. In other words, illuminate an object using multiple types of modulated light, attempt to decode the captured image for each pixel, and if it is decoded correctly, detect which code pattern can be further decoded can do. And the said calculating part can apply an individual image process for every part to which the modulated light from which a code differs was irradiated. Alternatively, it is possible to provide a system that is more convenient to use, for example, by displaying and outputting only a portion irradiated with modulated light of a specific code.

  At this time, the irradiation unit may irradiate two or more types of modulated light having different light intensities for each code. If the light intensity of each modulated light is changed relatively, the reachable distance is also different. Therefore, the modulated light having a higher light intensity reaches farther, and the reflected light from a distant object can be correctly decoded. Therefore, the arithmetic unit decodes the modulated light for each code, extracts a portion irradiated with each modulated light as an object, and the distance to the irradiated object based on the light intensity of the modulated light. Can be identified. In other words, distance information can be obtained by using a plurality of modulated lights, and an object at an arbitrary distance can be extracted therefrom. A system that is more convenient to use can be provided by, for example, outputting a display corresponding to the distance of the object.

  Further, the irradiating unit may irradiate two or more types of modulated light having different emission angles for each code. Each modulated light is emitted in different directions, and these directions may be left, right, up, down, concentric, or any direction. In this case, the calculation unit can decode the modulated light for each emission angle and perform object display processing according to the angle of view.

  The second aspect of the present invention is described in a computer-readable format so that processing of image information obtained by imaging a specific object irradiated with arbitrarily coded modulated light is executed on a computer system. A decoding step for decoding modulated light on a captured image, a portion of the captured image irradiated with the modulated light based on whether the decoded image is correctly decoded, and the others And an object extraction step of extracting the portion irradiated with the modulated light as the specific object.

  The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize predetermined processing on a computer system. In other words, by installing the computer program according to the second aspect of the present invention in the computer system, a cooperative action is exhibited on the computer system, and the imaging apparatus according to the first aspect of the present invention. The same effect can be obtained.

  ADVANTAGE OF THE INVENTION According to this invention, the outstanding imaging device, information processing apparatus, information processing method, and computer program which can extract the image of the specific object contained in a picked-up image suitably can be provided.

  In addition, according to the present invention, it is possible to suitably extract an image of a specific object included in a captured image using reflected light from the object, an excellent imaging apparatus, information processing apparatus, information processing method, and A computer program can be provided.

  Further, according to the present invention, it is possible to more accurately extract an image of a specific object included in a captured image using reflected light from the object even when the brightness changes in the background. An excellent imaging device, information processing device, information processing method, and computer program can be provided.

  According to the present invention, since a specific object is extracted using modulated light, it is possible to extract more robustly against changes in ambient light, etc. than when extraction is simply performed based on changes in brightness. Can do. Thereby, even when the camera itself moves or when blinking light existing in the living environment is photographed, it is possible to extract a good object without being influenced by these. Further, by preparing a plurality of code patterns, it is possible to know the distance and angle to the subject.

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

First embodiment:
FIG. 1 schematically shows a functional configuration of the object extraction device according to the first embodiment of the present invention. The illustrated object extraction apparatus 1 includes a modulated light decoding unit 2, a timing control unit 7, a light emitting unit 8, and a lens 9. Further, the modulated light decoding unit 2 includes an imaging unit 3, a buffer memory 5, and a calculation unit 6.

  The object extraction device 1 according to the present embodiment emits modulated light 10 from the light emitting unit 8 and demodulates (decodes) the reflected light 12 from the object 11 by the modulated light decoding unit 2. Here, there is a feature in that the distance of the object is known based on whether or not it is normally decoded. Unlike the conventional method of knowing the distance based on the difference in light, the use of modulated light provides a robust system that does not misrecognize even when the brightness changes in the background. can do.

  In FIG. 2, the structure of the light emission part 8 is shown. As illustrated, the light emitting unit 8 includes a light emission control unit 13 and a light source 14. The modulated light 10 is light obtained by modulating (coding) a certain code into a brightness signal. In the illustrated example, an 8-bit binary value “0101010010” is modulated, and “0” is expressed as unlit (dark) and “1” is lit (bright) (the same applies hereinafter). The light emission control unit 13 is responsible for controlling such a blinking pattern (modulated light). The light source 14 may be anything as long as the brightness can be changed, but it is appropriate to use an LED or a laser in terms of response speed and brightness. The wavelength of light may be in the visible light region or in the infrared light region. In this embodiment, the brightness of the modulated light is binary, but of course, it may be changed in an analog manner.

  FIG. 3 shows the procedure of the decoding process in the modulated light decoding unit 2 in the form of a flowchart.

  First, an electrical signal corresponding to light is generated in the imaging unit 3 (step S1).

  FIG. 4 schematically illustrates the configuration of the imaging surface of the imaging unit 3. In order to generate an electric signal based on light, photoelectric conversion devices are two-dimensionally arranged. As the photoelectric conversion device, a CCD (Charge Coupled Device) or a MOS (Metal Oxide Semiconductor) type imaging device is used.

  The pixel 15 is a minimum unit that can be photoelectrically converted. In general, it is desirable that the frame rate of the imaging unit 3 is equal to or higher than the frequency of the modulated light 10. In the present embodiment, for the sake of simplification of description, the case where the modulated light 10 and the imaging unit 3 operate in synchronization at the same frequency will be described. That is, when the modulated light is operating at 100 Hz, it is assumed that the imaging unit 10 operates at 100 fps (the number of frames per second).

  Next, the electrical signal output from the imaging unit 3 is sent to the buffer memory 5, where it is stored for several frames (step S2). The electrical signal at this time may be an analog value or a digital value (generally, an output digital value from the imaging apparatus is about 10 to 14 bits), but a digital value is more preferable because of easy handling of data. In this embodiment, since 8-bit binary data is used, signals for 8 frames are stored.

  Next, the data stored in the buffer memory 5 is transferred to the arithmetic unit 6. Here, a decoding operation is performed based on a digital (or analog) signal for 8 frames (step S3).

  FIG. 5 shows an example of the calculation method of the electric signal data here. In the example shown in the figure, a certain threshold value is provided, and determination processing is performed for each pixel so that “0” is set below this threshold and “1” is set above. It is convenient to binarize the output using the comparator 16. In the same figure, the reason why the signal amount is changed even when the value is the same “0” or “1” is that the signal goes up and down due to factors other than the modulated light (environment light). is there. From this figure, it can be understood that the object extraction processing according to the present embodiment is robust against changes in ambient light. The control unit 7 in FIG. 1 controls the operations of the modulated light decoding unit 2 and the light emission control unit 13 as described above.

  As described above, in the present invention, the modulated reflected light 12 can be decoded at the level of the pixel 15 in the minimum unit. Then, based on whether or not the decoded code is correct (that is, whether or not it matches the code given to the modulated light 10 output from the light emitting unit 8), whether or not the reflected light is incident on the pixel. It becomes possible to know accurately. For example, when only the object 11 is in the range where the modulated light 10 reaches, only the pixel 15 that has received the reflected light 12 from the object 11 can perform correct decoding. Therefore, it is possible to extract the shape of the object 11 from the captured image by paying attention only to the correctly decoded pixels. FIG. 6 shows a state in which the shape of the object 11 is extracted by focusing on only correctly decoded pixels.

  The first embodiment of the present invention has been described in detail above. Each part in the object extraction device 1 is shown in a disassembled manner for convenience of explanation, but even if several modules or all of the functional modules are integrated in one chip or module, the same applies. It goes without saying that the effect of can be obtained. For example, JP-A-2002-292379, JP-A-2002-326857, and JP-A-2003-33962, which have already been assigned to the present applicant, describe an imaging apparatus having an arithmetic function for arithmetic processing of pixel data. Is disclosed. Since the arithmetic function in this type of imaging apparatus can be used to store electrical signal data for several frames and to perform decoding processing (comparison with the threshold value of electrical signal data) based on the electrical signal data, the above-mentioned All the functional modules can be configured with one chip. Further, the arithmetic processing or the like may be performed by hardware or software.

Second embodiment:
FIG. 7 shows a functional configuration of an object extraction apparatus according to the second embodiment of the present invention. The illustrated object extraction apparatus 1 includes a modulated light decoding unit 2, a timing control unit 7, a light emitting unit 8, and a lens 9. Further, the modulated light decoding unit 2 includes an imaging unit 3, an interframe comparison unit 4, a buffer memory 5, and a calculation unit 17. The difference between the present embodiment and the first embodiment described above is that there is an inter-frame comparison unit 4 and the handling of data in the calculation unit 17.

  FIG. 8 shows a decoding process procedure in the modulated light decoding unit 2 in the form of a flowchart.

  First, an electrical signal corresponding to light is generated in the imaging unit 3 (step S11). The electrical signal may be an analog value or a digital value (generally, an output digital value from the imaging device is about 10 to 14 bits).

  Thereafter, this signal is transferred to the inter-frame comparison unit 4. Here, the magnitude of the signal is compared between the previous and next frames (step S12). FIG. 9 shows an example of the comparison calculation method here. FIG. 10 shows a processing procedure for realizing this calculation method in the form of a flowchart. In this method, the magnitude and sign of the difference 18 between frames as shown in FIG. 9 are calculated. At this time, even if the signals are the same, for example, “0”, signal differences occur. This difference (error) is defined as a threshold value 19 and it is defined that a data change occurs when a difference 18 larger than this is seen. As shown in FIG. 10, the magnitude of the pixel signal is compared in units of pixels between the previous and next frames. If the signal in the later frame is larger than that in the previous frame, a change of “0” → “1” occurs in the pixel, and vice versa, a change of “1” → “0” occurs. You can see that it occurred. However, as described above, when the comparison amount is equal to or less than the threshold value 19, it is defined as unchanged. In the present embodiment, the output data from the inter-frame comparison unit 4 is defined by 2 bits, “01” when the former changes, “10” when the latter changes, and “00” if it does not change. The input signal is usually an analog value or a digital value of 10 bits or more, but is compressed to 2 bits by the operation of the comparison unit 4. In the invariant data “00”, it cannot be determined whether “0” → “0” or “1” → “1”. However, since the modulated light 10 is used, “01” or “ 10 ”is present (ALL“ 0 ”and ALL“ 1 ”are difficult to determine as modulated light, so these are not used). From feedback from this data, whether“ 0 ”→“ 0 ” It is determined whether “1” → “1”.

  As shown in FIG. 8, the interframe comparison unit 4 result is stored in the buffer memory 5 (step S13). The data after the calculation is stored for several frames. However, the amount of data may be smaller than that of the buffer memory in the first embodiment as much as the data is compressed. As in the first embodiment, since the code assumes 8-bit binary data, the buffer memory 5 stores data for 8 frames.

  Next, the data stored in the buffer memory 5 is transferred to the arithmetic unit 17. The arithmetic unit 17 decodes the data compressed by the inter-frame comparison unit 4 (step S14). FIG. 11 shows the flow of frame data. The data output from the imaging unit 3 is compressed by the inter-frame comparison unit 4 and then decoded, whereby 8-bit data can be obtained from the modulated light 10.

  As described above, in the present invention, the modulated reflected light 12 can be decoded at the pixel 15 level in the minimum unit. Whether or not the reflected light is incident on the pixel is accurately determined based on whether or not the decoded code is correct (that is, whether or not it matches the code given to the modulated light 10 output from the light emitting unit 8). It becomes possible to know. For example, when only the object 11 is in the range where the modulated light 10 reaches, only the pixel 15 on which the reflected light 12 from the object 11 is incident can be decoded correctly. Therefore, it is possible to extract the shape of the object 11 from the captured image by paying attention only to the correctly decoded pixels. FIG. 13 shows how the shape of the object 11 can be extracted by paying attention only to correctly decoded pixels.

  Hereinabove, the second embodiment of the present invention has been described in detail. Each part in the object extracting apparatus 1 is illustrated in a disassembled manner for convenience of explanation, but the same effect can be obtained even if some or all of these parts are integrated in one chip or module. Needless to say, the same as above. Also, the arithmetic processing or the like may be performed by hardware or software.

Third embodiment:
Here, an embodiment will be described in which only the object clipped by the information for object clipping using the above-described means and the information of the normal image is displayed on the display device.

  FIG. 12 shows a processing procedure for displaying and outputting the clipped object according to the present embodiment in the form of a flowchart.

  The output data from the imaging unit 3 (see FIGS. 1 and 7) is transferred to the above-described object extraction device 1 (see FIGS. 1 and 7) (step S21), and a normal video signal. Is transferred to the image processing apparatus for generating (step S22). After that, decoding success / failure data and normal image data are output from each device for each pixel or at a level equivalent thereto. At this time, it is desirable that the mutual data correspond one-to-one, but this is not a limitation.

  Next, it is determined whether or not the decoding is correctly performed for each pixel (step S23), and arbitrary processing 1 and processing 2 are individually performed based on the information (steps S24 and S25). Here, what corresponds to process 1 is transferred to the display device, and what corresponds to process 2 is not transferred to the display device. Thus, only normal image data corresponding to correctly decoded pixels can be displayed on the display device.

  FIG. 13 shows a screen when the present embodiment is applied and a specific object included in the captured image is displayed.

  FIG. 13A shows a subject. Here, it is assumed that there is a human hand at a position where the modulated light can reach sufficiently, and a tree exists at a position where the modulated light does not reach thereafter. Further, FIG. 13B shows a normal image when the image is taken with a normal camera. FIG. 13C shows an example in which the present embodiment is applied and only a specific object (human hand) is cut out and displayed. If only video data corresponding to correctly decoded pixels is displayed, such extraction is possible.

  Here, areas of pixels that are not correctly decoded are displayed in gray. In addition, by blurring an area that has not been correctly decoded by image processing and displaying it, for example, nearby persons can be clearly displayed and the background can be blurred. Alternatively, completely different image data may be prepared and displayed. By using the decoding information processed by the object extraction device 1 as described above, only a specific object can be displayed on the display device.

Fourth embodiment:
By the way, in the third embodiment described above, the extracted object can be easily displayed, while the output data from the imaging unit includes modulated light in addition to ambient light. For this reason, when displaying normal image data, there may be a problem that a specific object irradiated with modulated light becomes brighter than the surroundings or flicker is observed by the modulated light. In the present embodiment, the extracted object is for solving these problems caused by exposure to the modulated light irradiated for object extraction.

  FIG. 14 shows an image acquisition operation sequence according to the present embodiment. As shown in the figure, in the present embodiment, acquisition of a normal image in which only an ambient light component is irradiated on the object, and an image in which the object is irradiated with light on which the modulated light component is also added to the environmental light component are obtained. Acquisition is performed by time division.

  Normal image processing is performed on the normal image obtained during the normal image acquisition operation. For example, it is converted into a video signal for screen output to a display device such as a CRT display. Also, during the image acquisition and decoding processing operation for extracting an object irradiated with light having a modulated light component in addition to the ambient light component, the captured image is processed by the object extraction device described above.

  By such an operation sequence, a normal image that is not affected by the modulated light can be displayed.

  FIG. 15 shows a processing procedure from data output to image display by the imaging unit 3 (see FIGS. 1 and 7) in the form of a flowchart.

  Normal image data and object extraction data are alternately output in a time-sharing manner (steps S31 and S32). Depending on whether normal image processing was performed before in time and decoding was correctly performed in the subsequent decoding processing (step S33), which pixel image data is processing 1 (step S34) or processing 2 (step It is determined whether it is sent to S35) (the reverse relationship may be reversed).

  Here, the normal image data does not coincide with the object extraction data in terms of time. Therefore, when the object to be extracted is moving, there is a spatial shift in the position of the object in the image frame between the acquisition of the normal image for image display and the acquisition of the image for object extraction. May occur. However, this can be reduced by increasing the frame rate of the imaging unit 3 and reducing the time difference between the normal image capturing for object display and the image capturing for object extraction.

  In the present embodiment, the frame rate is changed between the normal image operation and the operation including the decoding process. For example, in the example shown in FIG. 14, the former: the latter = 1: 8. Thereby, each operation time becomes equal. As described above, in the present embodiment, when the normal image data and the object extraction data are alternately output in time division, the spatial shift of the object may occur, but the modulated light is mounted. An image of a particular object that is not illuminated (ie, only ambient light) can be displayed on the display device.

Fifth embodiment:
In the fourth embodiment, in order to obtain an image of an object that is not affected by the modulated light component, a normal image in which only the ambient light component is irradiated on the object and an image acquisition for object extraction are sometimes obtained. This is done by dividing. In the latter image acquisition for object extraction, an image in which an object is irradiated with light having a modulated light component in addition to the ambient light component is acquired. Thus, for the purpose of photographing the object in time division, a spatial shift may occur in the position of the object in the image frame (described above).

  In the present embodiment, the problem of spatial displacement of objects caused by the fact that the normal image data and the object extraction data do not coincide with each other in time is applied by applying a motion prediction method. Is. However, since the object motion compensation process is added to the system configuration according to the fourth embodiment, the load on the computer increases accordingly.

  FIG. 16 shows a state where shooting of a normal image and shooting for object extraction (decoding) are alternately performed in time division on a moving object. In the illustrated example, the subject 20 is a moving circular object, and the case where the object extraction according to the present invention is applied to the subject 20 will be described below.

  Assume that the subject 20 (during image acquisition and decoding processing) is currently at the position shown in FIG. The subject 20 is moving along the trajectory indicated by the arrow 26. The subject 20 is moved to the subject 21 in the previous frame (at the time of normal image acquisition), and further to the subject 22 in the previous frame (at the time of image acquisition and decoding processing). It is assumed that the frame is present at the subject 23 in the previous frame (at the time of normal image acquisition).

  It is assumed that a pixel corresponding to the subject 20 is correctly decoded by the decoding process and an ordinary image corresponding to this pixel is to be displayed. However, since the image acquisition and the normal image acquisition for decoding processing are alternately performed in a time division manner, between the current position of the subject 20 and the position of the subject 21 in the normal image acquired most recently. There is a spatial shift. As a result, only the display portion 28 that is the overlap (AND) portion of the subject 21 is displayed (see FIG. 16B), and the other portions are not displayed.

  Therefore, in the present embodiment, in order to avoid such a situation, a motion vector of the subject is extracted, and an attempt is made to achieve a desired display using the result of motion prediction.

  In FIG. 16, pixels 24 to 27 are pixels corresponding to the same position with respect to the subjects 21 to 23. Here, a pixel correspondence relationship between image frames in which the normal image corresponding to the pixel 24 of the subject 20 at the time of decoding is the pixel 25 of the subject 21 is derived, and it corresponds to the subject 20 in the most recently captured normal image. It can be seen that a good object extraction without missing can be performed by displaying the pixel portion to be displayed.

  In order to realize such object extraction, it is necessary to obtain the vector of the arrow 26, that is, the motion vector of the object. The motion vector is extracted by using the subject 20 and the subject 22 which are decoded data. For example, if the subject is a simple circle as shown in FIG. 16, first, the center of gravity of each subject may be obtained, and then a vector connecting these centers of gravity may be derived.

  Alternatively, as another method for extracting motion vectors, a general method used in a standard image compression technique such as MPEG (Moving Picture Experts Group) 4 can be applied. For example, the subject 23 is treated as one polygon, and a vector is obtained by performing polygon matching processing with the next (actually next) frame (normal image frame in which the subject 21 is photographed). Ask. Based on this motion vector, pixel address conversion of pixel 24⇒pixel 25 may be performed.

  FIG. 17 shows a processing procedure for extracting the motion vector of the object and performing address conversion based on the motion vector in the form of a flowchart.

  First, the center of gravity of the subject 22 is obtained in the latest decoded photographed image (step S41). Next, the center of gravity of the subject 20 in the current decoded photographed image is obtained (step S42). Then, a motion vector is extracted based on each obtained center of gravity (step S43).

  Next, assuming that the subject 21 exists at the center of the motion vector in the most recent normal image (step S44), the subject 20 is formed based on the obtained motion vector, but its address is converted (step S45). . By using this address conversion, pixel data corresponding to the subject 20 in the decoding image can be acquired.

  As described above, in the present embodiment, the specific object can be satisfactorily extracted by estimating the video pixel 25 corresponding to the decoded pixel 24 from the motion vector.

Sixth embodiment:
So far, each embodiment has been described in which an object is irradiated with modulated light composed of one type of modulation code pattern and the object is extracted based on the binary value of whether or not the captured image is correctly decoded for each pixel. I have done it. In contrast, in the sixth embodiment of the present invention, an object is irradiated using a plurality of types of modulated light having different modulation codes. If the captured image is decoded for each pixel and correctly decoded, it is further detected which code pattern is decoded, and which reflected light is reflected by which modulated light. Is identified. In this case, since individual image processing can be applied to each object extracted by each modulated light, a user-friendly system can be provided. Hereinafter, as an application of object extraction using modulated light, an embodiment using modulated light composed of two or more types of modulation code patterns will be described.

  FIG. 18 shows a configuration of the light emitting unit according to the present embodiment. In the first to fifth embodiments, as shown in FIG. 2, one type of modulation code pattern is emitted as the modulated light 10. On the other hand, in this embodiment, modulated light 30 and 31 composed of two or more types of modulation code patterns are emitted. FIG. 18 shows an example in which the light source 33 alternately emits two modulated lights of “0101010010” and “010101001”.

  Here, as the light source 33, one light source may output a plurality of modulation code patterns, or a light source may be provided for each modulation code pattern. The light emission controller 32 controls the light source 33.

  Further, the two modulated lights 30 and 31 are set so that the light intensity is relatively changed, the former is weak and the latter is strong. In this case, the distance that each modulated light 30 and 31 can reach also varies depending on the light intensity, and the latter reaches far.

  FIG. 19 shows an image acquisition operation sequence according to the present embodiment. In the illustrated operation sequence, image acquisition for object extraction is performed in a time-sharing manner as in the case of the fourth embodiment (see FIG. 14). In the latter image acquisition for object extraction, an image in which an object is irradiated with light having a modulated light component in addition to the ambient light component is acquired.

The difference from the operation sequence shown in FIG. 14 is that the decoding process is performed twice, that is, the decoding process is performed for each modulation code pattern. This is because each of the modulated light 30 and 31 are assumed to interfere. Of course, if modulation methods (coding methods) that do not interfere with each other are used, it is also possible to perform decoding simultaneously.

  In the example shown in FIG. 19, in the normal image acquisition operation and the image acquisition and decoding processing operation, the latter operation period is drawn longer. However, considering the spatial deviation as described above, the latter operation period is drawn. It is desirable to handle the operation as fast as possible.

  When the light emission operation and operation sequence as described above are used, the object irradiated with each modulated light is extracted at the stage where the number of codes with different light intensities is decoded, and the distance to the object Can be identified. FIG. 18 shows a state in which two or more types of modulated light with different light intensities are irradiated for each code. As shown in the figure, the object 34 at a closer distance can decode the reflected light from the modulated light 30 having a low light intensity, and the reflected light from the strong modulated light 31 can be decoded farther than this. It is the object 35 in the distance. Further, it is the object 36 that is further away that cannot be correctly decoded. In the illustrated example, there are two types of modulated light, but naturally more than that may be used.

  As described above, by using a plurality of modulated lights, distance information can be obtained, and an object at an arbitrary distance can be extracted and displayed.

  FIG. 20 shows a processing procedure from data output to image display in the form of a flowchart in the present embodiment.

  Normal image data and object extraction image acquisition are alternately output in a time-sharing manner (steps S51 and S52). Normal image processing is performed before in time, and in the subsequent decoding processing, it is determined whether or not each pixel of the acquired image for object extraction has been correctly decoded by each coding method (step S53).

  Here, processing 2 is applied to pixels that have not been correctly decoded by any coding method (step S54). An example of the process 2 is not to display and output the pixel signal.

  Further, for a pixel that can be correctly decoded by any coding scheme (step S53), a processing method is determined according to which coding scheme is correctly decoded (step S55).

  For example, processing 1 (for example, display output) is applied to a pixel obtained by decoding the code pattern “010100010” of the modulated light 30 set to a low light intensity as a pixel signal of an object at a closer location. (Step S56).

  Further, processing 4 (for example, display output) is applied to a pixel obtained by decoding the code pattern “010101001” of the modulated light 31 set to a high light intensity as a pixel signal with an object at a farther place. (Step S57).

  Further, processing 3 (for example, display output) is applied to the pixels for which any of the code patterns “0101010010” and “010101001” can be decoded (step S58).

Seventh embodiment:
In the above-described sixth embodiment, a plurality of modulated lights having different light intensities are used. On the other hand, in this embodiment, a plurality of modulated lights having different emission angles are used.

  FIG. 21 shows a configuration of the light emitting unit and modulated light according to the present embodiment. The modulated lights 37 and 38 are emitted in different directions. The direction may be left, right, up, down, concentric, or any direction. The light source 39 may be configured such that one light source emits modulated light having different light emission angles in a time-sharing manner, or a light source may be arranged for each light emission angle.

  The reason why decoding processing is performed in a time division manner for each modulated light having a different modulation code pattern is because it is assumed that the respective modulated lights 37 and 38 interfere with each other, and a modulation method (coding method) that does not interfere with each other. ) Can be simultaneously decoded.

  Further, the light emission control unit 40 controls the operation of the light source 39. The operation sequence is the same as that in FIG. 19, and image acquisition for object extraction can be performed by time division. In the latter image acquisition for object extraction, an image in which an object is irradiated with light having a modulated light component in addition to the ambient light component is acquired. The decoding process is performed for each emission angle.

  According to this embodiment, it is possible to specify the direction in which object extraction is desired. FIG. 22 shows the relationship between the light emitting area 50 (modulated light 37) and the light emitting area 51 (modulated light 38) and the imaging field angle 52 of the image sensor when different modulated lights are emitted in the left-right direction. . By setting as shown in the figure, for example, as shown in FIG. 23, both the right hand 53 irradiated to the modulated light 37 and the left hand 54 irradiated to the modulated light 38 can be recognized by different codes. Become. This makes it possible to input information separately with both hands. In applications such as games, the application range of how to play can be expanded.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  The electrical signal data obtained from the image sensor is stored for several frames, and a part or all of the functional modules that perform the decoding process by comparing the electrical signal data in the previous and subsequent frames are integrated on one chip or on-chip module. By this, the present invention can be preferably realized.

  For example, JP-A-2002-292379, JP-A-2002-326857, and JP-A-2003-33962, which have already been assigned to the present applicant, describe an imaging apparatus having an arithmetic function for arithmetic processing of pixel data. Is disclosed. Some or all of the functional modules related to the present invention can be mounted on this type of imaging apparatus.

  Also, the processing for decoding the electrical signal data for each pixel by comparison with the threshold value, or the arithmetic processing for performing the decoding processing by comparing the electrical signal data in the preceding and following frames, is performed by software rather than by hardware. May be.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

FIG. 1 is a diagram schematically illustrating a functional configuration of an object extraction device according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating a configuration of the light emitting unit 8. FIG. 3 is a flowchart showing a procedure of decoding processing in the modulated light decoding unit 2. FIG. 4 is a diagram schematically illustrating the configuration of the imaging surface of the imaging unit 3. FIG. 5 is a diagram for explaining a method of calculating an electrical signal output from the imaging unit 3. FIG. 6 is a diagram showing a state in which the shape of the object 11 is extracted by focusing on only correctly decoded pixels. FIG. 7 is a diagram showing a functional configuration of an object extraction device according to the second embodiment of the present invention. FIG. 8 is a flowchart showing the procedure of the decoding process in the modulated light decoding unit 2. FIG. 9 is a diagram for explaining a calculation method for comparing the electrical signal data output from the imaging unit 3 in the previous and subsequent frames. FIG. 10 is a flowchart illustrating a processing procedure for comparing the electrical signal data output from the imaging unit 3 in the previous and subsequent frames. FIG. 11 is a diagram showing the flow of frame data. FIG. 12 is a diagram showing a state in which the shape of the object 11 can be extracted by paying attention only to correctly decoded pixels. FIG. 13 is a diagram showing an example of a screen when the present invention is applied and a specific object included in the photographed image is displayed. FIG. 14 is a diagram showing an operation sequence of image acquisition according to the fourth embodiment of the present invention. FIG. 15 is a flowchart showing a processing procedure from data output to image display in the fourth embodiment of the present invention. FIG. 16 is a diagram showing a state where shooting of a normal image and shooting for object extraction (decoding) are alternately performed in time division on a moving object. FIG. 17 is a flowchart showing a processing procedure for extracting an object motion vector and performing address conversion based on the motion vector. FIG. 18 is a diagram showing a configuration of a light emitting unit according to the sixth embodiment of the present invention. FIG. 19 is a diagram showing an operation sequence of image acquisition according to the sixth embodiment of the present invention. FIG. 20 is a flowchart showing a processing procedure from data output to image display in the sixth embodiment of the present invention. FIG. 21 is a diagram showing a configuration of a light emitting unit and modulated light according to the seventh embodiment of the present invention. FIG. 22 is a diagram illustrating a relationship between a light emitting area and an imaging field angle when different modulated light is issued in the left-right direction. FIG. 23 is a diagram illustrating an example of an object recognition result when the relationship between the light emitting area and the imaging angle of view is set as illustrated in FIG. FIG. 24 is a diagram illustrating a configuration example (prior art) of an information input device including means for extracting only a specific object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Object extraction apparatus 2 ... Modulated light decoding part 3 ... Imaging part 4 ... Inter-frame comparison part 5 ... Buffer memory 6 ... Calculation part 7 ... Timing control part 8 ... Light emission part 9 ... Lens 17 ... Calculation part

Claims (16)

An irradiating unit that emits two or more types of modulated light coded based on different data values ;
An image pickup unit configured by two-dimensionally arranged image pickup devices, picking up an image including reflected light from a specific object with respect to the two or more types of modulated light, and outputting a video signal for each image pickup device ;
Attempts to decode the two or more types of modulated light with respect to the video signal for each image sensor, and based on which of the different data values is correctly decoded, the two or more types of modulated light in the captured image. Is a calculation unit that separates the irradiated part and the other part ,
A display for displaying an image;
Equipped with,
Extracting a video signal of a portion irradiated with each of the two or more types of modulated light, cutting out only the video of the specific object from the captured image, and displaying the video on the display unit;
That imaging device. The irradiation unit codes the modulated light according to a predetermined emission frequency,
The imaging unit captures an image at a frame rate equal to or higher than the emission frequency,
The arithmetic unit decodes a code included in reflected light with respect to modulated light from the specific object based on an output signal from the imaging unit.
The imaging apparatus according to claim 1. When the modulated light is coded in the irradiating unit and the modulated light is decoded from the captured image in the arithmetic unit, the imaging unit switches a frame rate.
The imaging apparatus according to claim 2 . A normal image capturing mode in which the image capturing unit performs normal image capturing; and a decoding mode in which the image capturing unit performs image capturing for decoding the modulated light in the arithmetic unit.
The imaging apparatus according to claim 1. In the normal image capturing mode, the irradiating unit does not irradiate modulated light, the imaging unit captures an image composed of reflected light with respect to ambient light,
In the decoding mode, the irradiating unit emits modulated light, and the imaging unit captures an image composed of ambient light and reflected light with respect to the modulated light.
The imaging apparatus according to claim 4. A storage unit for storing captured images for several frames in the imaging unit ;
The imaging apparatus according to claim 1. The irradiation unit irradiates modulated light including a wavelength component of invisible light.
The imaging apparatus according to claim 1 . The irradiation unit irradiates two or more kinds of modulated light with different light intensities for each cord,
The arithmetic unit decodes modulated light for each code, extracts a portion irradiated with each modulated light as an object, and identifies a distance to each object based on the light intensity for each modulated light.
The imaging apparatus according to claim 1. The irradiation unit irradiates two or more types of modulated light with different emission angles for each cord,
The arithmetic unit performs decoding of modulated light for each emission angle, and performs object display processing according to the angle of view.
The imaging apparatus according to claim 1. An information processing apparatus that processes a captured image obtained by capturing an object irradiated with two or more types of modulated light coded based on different data values,
Decoding means for decoding the two or more types of modulated light from the video signal for each pixel of the captured image;
Based on which of the different data values is correctly decoded, the captured image is separated into a portion irradiated with the two or more types of modulated light and a portion other than that, and the two or more types of modulation are separated. Object extracting means for extracting each part irradiated with light as an object,
An information processing apparatus comprising:   The captured image is captured at a frame rate equal to or higher than the emission frequency for coding the modulated light,
The decoding means decodes a code included in reflected light with respect to modulated light from an object based on a captured image.
The information processing apparatus according to claim 10. Input both a normal captured image consisting of reflected light with respect to ambient light and an object extraction captured image consisting of reflected light with respect to ambient light and coded modulated light.
The information processing apparatus according to claim 10. Input image information obtained by imaging a specific object irradiated with two or more types of modulated light with different light intensities for each code,
The decoding means decodes the modulated light for each code on the captured image,
The object extraction means extracts the object irradiated to each modulated light based on the result of decoding for each code, and identifies the distance to each object based on the light intensity for each modulated light.
The information processing apparatus according to claim 10. Enter image information obtained by imaging a specific object irradiated with two or more types of modulated light with different emission angles for each code,
The decoding means decodes modulated light for each emission angle with respect to the captured image,
The object extraction means extracts, for each angle of view, an object irradiated with each modulated light based on a result decoded for each code.
The information processing apparatus according to claim 10 . An information processing method for processing a captured image obtained by imaging an object irradiated with two or more kinds of modulated light coded based on different data values,
A decoding step of decoding the two or more types of modulated light from the video signal for each pixel of the captured image;
Based on which of the different data values is correctly decoded, the captured image is separated into a portion irradiated with the two or more types of modulated light and a portion other than that, and the two or more types of modulation are separated. An object extraction step of extracting each portion irradiated with light as an object,
An information processing method characterized by comprising: A computer program written in a computer-readable format so as to execute processing of a captured image obtained by imaging an object irradiated with two or more kinds of modulated light coded based on different data values on a computer. The computer
Decoding means for decoding the two or more types of modulated light from the video signal for each pixel of the captured image;
Based on which of the different data values is correctly decoded, the captured image is separated into a portion irradiated with the two or more types of modulated light and a portion other than that, and the two or more types of modulation are separated. An object extracting means for extracting each part irradiated with light as an object,
Computer program to function as

Images