JP7677699B2 - Processing Equipment - Google Patents

Description

The present invention relates to a processing device, a processing method, a program, and a processing system.

Captured image data may be displayed for purposes such as expanding vision.

For example, Patent Document 1 describes a visual expansion device that expands human vision. According to Patent Document 1, the visual expansion device has an imaging means, an image processing means, and a display means. For example, the imaging means captures an image of a range of a visual field that is substantially the same as the range of the user's visual field. Furthermore, the image processing means performs a predetermined process on the image captured by the imaging means. Then, the display means displays the processed image visibly within the user's visual field.

JP 2003-287708 A

The technology described in Patent Document 1 captures an image of a range of visual field that is approximately the same as the range of the user's visual field, so if an obstruction is present within the visual field, for example, it is not possible to recognize a person or object hidden behind the obstruction. This creates a problem in that there is a risk of an area becoming invisible when an obstruction is present.

The object of the present invention is to provide a processing device, processing method, and program that solve the problem of the risk of invisible areas being created when an obstruction or the like is present.

In order to achieve this object, a processing device according to one embodiment of the present disclosure includes:
an estimation unit that estimates an occluded region, which is a region in which an object that cannot be confirmed in the image data is possibly hidden, based on the image data;
a detection unit that detects an object that exists in the blocked area in the image data by using second image data acquired by an imaging device that is located at a position different from the imaging device that acquired the image data, when the estimation unit estimates the existence of the blocked area; and
The configuration has the following:

In addition, a processing method according to another embodiment of the present disclosure includes:
An information processing device,
Based on the image data, an occlusion region is estimated, which is a region in which an object that cannot be confirmed in the image data may be hidden;
When the existence of the occluded area is estimated, an object existing within the occluded area in the image data is detected using second image data acquired using an imaging device located at a different position from the imaging device that acquired the image data.

In addition, a program according to another aspect of the present disclosure includes:
In the information processing device,
Based on the image data, an occlusion region is estimated, which is a region in which an object that cannot be confirmed in the image data may be hidden;
This is a program for realizing the following: when the existence of the occluded area is estimated, an object existing within the occluded area in the image data is detected using second image data acquired using an imaging device located at a different position from the imaging device that acquired the image data.

In addition, a processing system according to another embodiment of the present disclosure includes:
a processing device having: an estimation unit that estimates an obstructed area based on image data, the obstructed area being an area in which an object that cannot be confirmed in the image data may be hidden; and a detection unit that, when the estimation unit estimates the existence of the obstructed area, detects an object that exists in the obstructed area in the image data using second image data acquired by an imaging device that is located at a position different from the imaging device that acquired the image data;
a display device that displays the detection result by the detection unit together with the image data;
The configuration has the following:

The above-mentioned configurations make it possible to provide a processing device, processing method, and program that can solve the problem of the risk of invisible areas being created when an obstruction or the like is present.

FIG. 1 is a diagram for explaining a processing system according to a first embodiment of the present disclosure. FIG. 1 illustrates an example of the overall configuration of a processing system. FIG. 2 is a block diagram showing a configuration example of a processing device. 11A and 11B are diagrams for explaining a processing example of a masked region estimation unit; 11A and 11B are diagrams for explaining an example of processing by an extension area confirmation unit; 11A and 11B are diagrams for explaining an example of processing by an invisible object detection unit. 11 is a diagram for explaining an example of processing by an integration unit; FIG. 13A and 13B are diagrams for explaining another example of processing by the integration unit; 10 is a flowchart showing an example of the operation of the processing device. FIG. 11 is a diagram illustrating an example of a hardware configuration of a processing device according to a second embodiment of the present disclosure. FIG. 2 is a block diagram showing a configuration example of a processing device.

[First embodiment]
A first embodiment of the present disclosure will be described with reference to Figs. 1 to 9. Fig. 1 is a diagram for explaining a processing system 100. Fig. 2 is a diagram showing an example of the configuration of the entire processing system 100. Fig. 3 is a block diagram showing an example of the configuration of a processing device 400. Fig. 4 is a diagram for explaining an example of processing by the occluded area estimation unit 433. Fig. 5 is a diagram for explaining an example of processing by the extended area confirmation unit 434. Fig. 6 is a diagram for explaining an example of processing by the invisible object detection unit 435. Fig. 7 is a diagram for explaining an example of processing by the integration unit 436. Fig. 8 is a diagram for explaining another example of processing by the integration unit 436. Fig. 9 is a flowchart showing an example of the operation of the processing device 400.

In the first embodiment of the present disclosure, a processing system 100 capable of displaying an object hidden in a masked area by detecting an object in a masked area present in image data acquired by a certain imaging device using second image data, which is image data acquired by another imaging device, will be described. As shown in FIG. 1, for example, when a masked area exists in image data acquired by an imaging device owned by person A, the processing system 100 checks whether there is another imaging device that captures the masked area. For example, in the case of FIG. 1, a device D owned by person D, who exists in a different position from person A, or installed in a different position, captures the masked area. Therefore, the processing system 100 detects an object such as a person based on image data acquired by device D, which is an imaging device, and estimates the position of the detected object. In addition, the processing system 100 checks whether an object such as a person exists in the masked area present in the image data acquired by person A using an imaging device based on the estimated position. Then, if it is determined that an object such as a person exists in the masked area present in the image data acquired using the imaging device owned by person A, the processing system 100 integrates the detection result into the image data acquired using the imaging device owned by person A, and outputs the integrated result to the device owned by person A. For example, in the case of FIG. 1, person C exists in the masked area in the image data acquired using the imaging device owned by person A. Therefore, the processing system 100 integrates information indicating that person C exists in the masked area into the image data acquired using the imaging device owned by person A, and outputs the integrated result to the device owned by person A.

Note that an occluded area is an area where an object that cannot be seen in image data may be hidden due to the presence of an obstructing object such as a wall, vehicle, or large luggage. As shown in Figure 1, even if an invisible object is hidden within an occluded area and cannot be seen in image data acquired from a certain direction, it may be possible to see it in image data acquired from another position or direction.

Figure 2 shows an example of the overall configuration of the processing system 100. Referring to Figure 2, the processing system 100 can include, for example, a moving device 200, an installation device 300, and a processing device 400. For example, the processing system 100 can include multiple moving devices 200 and installation devices 300. The processing system 100 does not have to include either the moving device 200 or the installation device 300.

As shown in FIG. 2, the mobile device 200 and the processing device 400 are connected so as to be able to communicate with each other via wire or wirelessly. The installation device 300 and the processing device 400 are connected so as to be able to communicate with each other via wire or wirelessly. The mobile device 200 and the processing device 400 may communicate with each other via a relay device (not shown). Similarly, the installation device 300 and the processing device 400 may communicate with each other via a relay device (not shown).

The mobile device 200 is a device (display device) capable of acquiring image data and displaying the image data. The mobile device 200 is possessed by a moving object such as a user, and moves in accordance with the movement of the moving object. For example, the mobile device 200 is a mobile information terminal such as a smartphone or tablet having an imaging device such as a camera and a screen display device such as a display with a touch panel. The mobile device 200 may be composed of multiple devices such as an imaging device such as a camera or a 360-degree camera and a screen display device such as AR/MR (Augmented Reality/Mixed Reality) glasses.

The mobile device 200 is owned, for example, by a user of the processing system 100, and acquires image data at a predetermined viewing angle, such as 120 degrees or 360 degrees, from the user's position. The mobile device 200 also transmits the acquired image data to the processing device 400. The mobile device 200 may associate time information indicating the time the image data was acquired with the image data, and transmit the associated data to the processing device 400.

The mobile device 200 can also transmit location data indicating the user's location (location of the mobile device 200) to the processing device 400. For example, the mobile device 200 has a GPS (Global Positioning System) function and transmits location data acquired using the GPS function to the processing device 400 at predetermined intervals. The mobile device 200 may associate time information indicating the time when the location data was acquired with the location data and transmit the associated data to the processing device 400. The processing device 400 may be configured to be able to grasp the user's location (location of the mobile device 200) by performing positioning using a beacon or the like.

The mobile device 200 also has a screen display device and displays image data on the screen display device. For example, the mobile device 200 can display image data acquired using its own imaging device or image data received from the processing device 400 on the screen.

The mobile device 200 may be a device that only has the function of acquiring image data. In other words, some of the mobile devices 200 included in the processing system 100 may not have a screen display device. In addition, the mobile device 200 may be a moving object other than a person.

The installation device 300 is a device that acquires image data. For example, the installation device 300 is an imaging device such as a surveillance camera installed at a predetermined position. For example, the installation device 300 acquires image data at a predetermined viewing angle in a predetermined direction. The installation device 300 then transmits the acquired image data to the processing device 400. The installation device 300 may associate time information indicating the time the image data was acquired with the image data, and transmit the associated data to the processing device 400.

The imaging direction of the installation device 300 may be variable. When the imaging direction of the installation device 300 is variable, the imaging direction of the installation device 300 can be grasped by the processing device 400, for example, by transmitting information indicating the imaging direction of the installation device 300 to the processing device 400. The installation device 300 may also have a function of displaying image data.

The processing device 400 is an information processing device that estimates an obstructed area present in image data acquired from the mobile device 200 or the like, and detects an object present in the estimated obstructed area based on image data acquired by another imaging device. For example, the processing device 400 is an information processing device such as a server device. The processing device 400 may be configured as a single information processing device, or may be realized, for example, on a cloud.

Figure 3 shows an example of the configuration of the processing device 400. Referring to Figure 3, the processing device 400 has, as its main components, for example, a communication I/F unit 410, a memory unit 420, and a calculation processing unit 430.

The communication I/F unit 410 is composed of a data communication circuit. The communication I/F unit 410 performs data communication with an external device connected via a communication line.

The storage unit 420 is a storage device such as a hard disk or memory. The storage unit 420 stores processing information and programs 427 necessary for various processes in the calculation processing unit 430. The programs 427 are read into the calculation processing unit 430 and executed to realize various processing units. The programs 427 are read in advance from an external device or recording medium via a data input/output function such as the communication I/F unit 410, and are stored in the storage unit 420. Examples of main information stored in the storage unit 420 include viewing angle information 421, installation position/orientation information 422, image information 423, position information 424, behavior vector information 425, and integrated image information 426. Note that the storage unit 420 may include information other than the exemplified information, such as map information.

The viewing angle information 421 indicates the viewing angle of the imaging device and the installation device 300 possessed by the moving device 200. For example, in the viewing angle information 421, identification information for identifying the moving device 200, the installation device 300, etc. is associated with information indicating the viewing angle. The viewing angle information 421 is acquired in advance from the moving device 200, the installation device 300, etc., and stored in the memory unit 420.

The installation position/orientation information 422 indicates the installation position of the installation device 300 and the direction in which the installation device 300 is installed. For example, the installation position/orientation information 422 associates identification information for identifying the installation device 300 with information indicating the installation position and installation direction of the installation device 300. The installation position/orientation information 422 is acquired in advance from the installation device 300 and the like and stored in the memory unit 420.

Image information 423 indicates image data acquired by the imaging unit of mobile device 200, installation device 300, etc. For example, image information 423 stores time-series image data for each piece of identification information for identifying mobile device 200, installation device 300, etc. Image information 423 is generated and updated by acquiring image data from mobile device 200, installation device 300, etc.

The location information 424 indicates the location of the mobile device 200 (the user's location). For example, the location information 424 stores time-series location data for each piece of identification information for identifying the mobile device 200. The location information 424 is generated and updated by acquiring location data from the mobile device 200 or the like. The location information 424 may also be acquired and generated and updated by performing location measurement using a beacon or the like.

The behavior vector information 425 indicates the direction in which the user moves, the speed at which the user moves, and the like. For example, the behavior vector information 425 indicates the time-series direction of movement and the speed at which the user moves for each piece of identification information for identifying the mobile device 200. The behavior vector information 425 is obtained, generated, and updated, for example, by the behavior vector calculation unit 432 described below calculating the speed and direction of movement based on the position information 424.

The integrated image information 426 indicates image data that integrates information indicating objects present within the obstructed area. For example, the integrated image information 426 stores integrated image data in a time series for each piece of identification information for identifying the moving device 200, the installation device 300, etc. The integrated image information 426 is acquired, generated, and updated by the integration unit 436, which will be described later, performing an integration process.

The calculation processing unit 430 has a calculation device such as a CPU and its peripheral circuits. The calculation processing unit 430 reads and executes the program 427 from the storage unit 420, thereby implementing various processing units by having the above hardware and the program 427 work together. The main processing units implemented by the calculation processing unit 430 include, for example, a receiving unit 431, a movement vector calculation unit 432, an occluded area estimation unit 433, an extended area confirmation unit 434, an invisible object detection unit 435, an integration unit 436, and an output unit 437.

The receiving unit 431 receives various data from the mobile device 200 and the installation device 300. The receiving unit 431 then stores the received data in the storage unit 420.

For example, the receiving unit 431 receives image data from the mobile device 200 or the installation device 300. The receiving unit 431 then stores the received image data in the storage unit 420 as image information 423. The receiving unit 431 may store the image data in the storage unit 420 in association with time information. The time information may be the time associated with the image data, or may be the time when the receiving unit 431 receives the image data.

The receiving unit 431 also receives position data from the mobile device 200. The receiving unit 431 then stores the received position data in the storage unit 420 as position information 424. The receiving unit 431 may associate the position data with time information and store them in the storage unit 420. The time information may be a time associated with the position data, or may be the time when the receiving unit 431 received the position data. The receiving unit 431 may be configured to receive from a relay device the position data acquired by the relay device or the like performing position positioning based on the transmission from the mobile device 200. In this case as well, the receiving unit 431 stores the received position data in the storage unit 420 as position information 424.

The movement vector calculation unit 432 calculates information indicating the movement direction and movement speed of the user based on the position information 424. The movement vector calculation unit 432 then stores the information indicating the calculated movement direction and movement speed in the storage unit 420 as movement vector information 425.

As described above, the position information 424 stores time-series position data for each user. The movement vector calculation unit 432 acquires multiple pieces of position data from the time-series position data included in the position information 424, and calculates the movement direction and movement speed of the user using the acquired position data. Note that the movement vector calculation unit 432 may calculate the movement direction and movement speed using a general method. The number of pieces of position data used by the movement vector calculation unit 432 when calculating the movement direction and movement speed may also be arbitrary.

The movement vector calculation unit 432 may calculate the movement direction and movement speed at any timing. For example, the movement vector calculation unit 432 can calculate information indicating the movement direction and movement speed of the user periodically at a predetermined interval.

The occluded area estimation unit 433 estimates an occluded area present in the image data based on the image data included in the image information 423. If the occluded area estimation unit 433 estimates that an occluded area exists in the image data, confirmation is performed by the extended area confirmation unit 434. On the other hand, if it is not estimated that an occluded area exists in the image data, for example, the output unit 437 described below can instruct the mobile device 200 to display image data in which no occluded area exists. For example, the occluded area estimation unit 433 estimates an occluded area for image data received from the mobile device 200. The occluded area estimation unit 433 may also be configured to estimate an occluded area for image data received from the installed device 300 when a predetermined condition is satisfied.

For example, the occluded area estimation unit 433 estimates the depth based on the image data. For example, the occluded area estimation unit 433 can estimate the depth using known techniques, such as a neural network for depth estimation. The occluded area estimation unit 433 then estimates the occluded area based on the estimation result. For example, the occluded area estimation unit 433 estimates an area where the estimated depth satisfies a predetermined condition, such as an area where the change in depth is less than a predetermined value (which may be any value), as an occluded area. For example, in the case shown in Figure 4, the occluded area estimation unit 433 estimates that a specified range on the right side in the image data is an occluded area because the change in depth is less than a predetermined value.

The occluded area estimation unit 433 may estimate the occluded area using a method other than the above examples. For example, the occluded area estimation unit 433 may detect an object having a size equal to or larger than a predetermined standard by using a model learned in advance or by performing pattern matching, and estimate the location where the detected object exists as the occluded area. The occluded area estimation unit 433 may also estimate the occluded area based on map information acquired in advance. For example, the occluded area estimation unit 433 can grasp the position of the user based on the position information 424, and can estimate that the user is facing the movement direction based on the action vector information 425. It can also estimate that image data corresponding to the direction in which the user is facing has been acquired. Therefore, the occluded area estimation unit 433 can estimate the area of the map that will be reflected in the image data based on the map information, the position and direction of the user, and the viewing angle indicated by the viewing angle information 421, thereby identifying the area in the image data where a blocking object such as a wall exists and estimating the occluded area. The occluded area estimation unit 433 may estimate the occluded area using a method other than the above examples, such as estimating the occluded area using depth information acquired from the moving device 200, the installation device 300, etc. together with the image data.

The shielded area estimation unit 433 may estimate the shielded area at any timing. For example, the shielded area estimation unit 433 can check whether or not a shielded area exists in the stored image data each time image data is stored as the image information 423. The shielded area estimation unit 433 may estimate the shielded area periodically at a predetermined interval. The shielded area estimation unit 433 may be configured to estimate the shielded area when it is estimated that the user is present in a location that satisfies a predetermined condition, such as near a shield such as a wall, based on the map information and the position information 424. The shielded area estimation unit 433 may determine whether or not to estimate the shielded area using the estimation result of the shielded area for another user, such as estimating the shielded area when it is estimated that the user is present in the vicinity of a shielded area estimated based on image data held by the other user using the position information 424.

When the occluded area estimation unit 433 estimates that an occluded area exists in the image data, the extended area confirmation unit 434 (identification unit) confirms whether or not the occluded area can be complemented based on second image data, which is image data acquired by an imaging device other than the imaging device that acquired the image data in which the occluded area exists. When the extended area confirmation unit 434 determines that complementation is possible, detection is performed by the invisible object detection unit 435. On the other hand, when complementation is determined to be impossible, for example, the output unit 437 described below can instruct the corresponding mobile device 200 to display the image data.

For example, as shown in FIG. 5, the extension area confirmation unit 434 checks whether complementation is possible by checking whether there are other mobile devices 200 or installation devices 300 capturing images within the estimated obstruction area.

Specifically, for example, the extended area confirmation unit 434 refers to the viewing angle information 421, the installation position/orientation information 422, the position information 424, the behavior vector information 425, etc., as well as the results of estimation by the occluded area estimation unit 433. As a result, the extended area confirmation unit 434 identifies the position of the imaging area, which is the area captured by the mobile device 200 or the like that has acquired the image data that has estimated the existence of the occluded area, and the position that will become the occluded area. In other words, the extended area confirmation unit 434 identifies the position, facing direction, viewing angle, etc. of the mobile device 200 or the like that has acquired the image data, thereby identifying the position of the occluded area, etc.

The extended area confirmation unit 434 also confirms the positions of the imaging areas of other installation devices 300 included in the processing system 100 by referring to the viewing angle information 421 and the installation position/orientation information 422. That is, the extended area confirmation unit 434 confirms the positions of the imaging areas of each installation device 300 by identifying the installation positions, facing directions, and viewing angles of the installation devices 300 present in the processing system 100. Then, the extended area confirmation unit 434 identifies the installation device 300 capturing an image within the shielded area based on the results of the above confirmation. For example, when the identified position of the shielded area and the position of the imaging area of the installation device 300 at least partially overlap, the extended area confirmation unit 434 determines that the installation device 300 is capturing an image of the shielded area, and identifies the installation device 300 as the installation device 300 capturing an image within the shielded area. When the installation device 300 capturing an image of the shielded area can be identified, the extended area confirmation unit 434 determines that the shielded area can be complemented based on second image data acquired by an imaging device other than the imaging device that captured the image data in which the shielded area exists. For example, in the case of FIG. 5, the imaging area of the installation device 300α overlaps with the shielded area present in the image data acquired using the mobile device 200 owned by person A. Therefore, the extended area confirmation unit 434 identifies the installation device 300α as the installation device 300 capturing an image of the shielded area, and determines that the installation device 300α can complement the shielded area based on the second image data.

For example, the extended area confirmation unit 434 confirms the imaging area of the mobile device 200 included in the processing system 100 by referring to the viewing angle information 421, the position information 424, and the user's orientation determined based on the behavior vector information 425. That is, the extended area confirmation unit 434 confirms the position of the imaging area of each mobile device 200 by identifying the position, the facing direction, and the viewing angle of the mobile device 200. Then, based on the result of the above confirmation, the extended area confirmation unit 434 identifies the mobile device 200 that is imaging within the shielded area. The method of identification may be the same as in the case of the installed device 300. When the mobile device 200 that is imaging within the shielded area can be identified, the extended area confirmation unit 434 determines that it is possible to complement the inside of the shielded area based on the second image data acquired by the imaging device possessed by the identified mobile device 200.

For example, as described above, the extended area confirmation unit 434 confirms whether the inside of the shielded area can be complemented by confirming whether there is a mobile device 200 or an installation device 300 capturing an image of the inside of the shielded area based on information indicating the position and orientation of the mobile device 200 or the installation device 300. Then, for example, if the extended area confirmation unit 434 can identify even one mobile device 200 or installation device 300 capturing an image of the inside of the shielded area, the process proceeds to processing by the invisible object detection unit 435.

When the extended area confirmation unit 434 determines that the area can be complemented, the invisible object detection unit 435 detects an object present in the area corresponding to the shielded area based on the second image data acquired from the mobile device 200 or the installation device 300 capturing an image in the shielded area. That is, the invisible object detection unit 435 detects an invisible object, which is an object hidden behind the shielded area in the image data in which the shielded area exists, using the second image data acquired by another imaging device. For example, the invisible object detection unit 435 detects an object using the latest second image data among the time-series second image data included in the image information 423. When the invisible object detection unit 435 detects an object present in the area corresponding to the shielded area, the integration unit 436 proceeds to integration processing. On the other hand, when an object is not detected, for example, the output unit 437 described later can instruct the corresponding mobile device 200 to display the image data.

For example, the invisible object detection unit 435 detects an object such as a person present in the second image data by using a model previously learned for the second image data or by performing pattern matching. The invisible object detection unit 435 also estimates the actual position of the detected object using information indicating the positions of the moving device 200 and the installation device 300. For example, the invisible object detection unit 435 estimates the position of the detected object using depth information estimated using a known technique such as a neural network for depth estimation. Then, the invisible object detection unit 435 checks whether the detected object exists in an area corresponding to a shielded area based on the estimated object position. If it is estimated that the object exists in an area corresponding to a shielded area, the invisible object detection unit 435 detects that the object exists in an area corresponding to the shielded area. In other words, the invisible object detection unit 435 detects an invisible object. For example, in the case shown in FIG. 6, it is estimated that the object, a person C, exists in an area corresponding to a shielded area from the estimated object position. Therefore, the invisible object detection unit 435 detects the presence of an invisible object, person C, in the occluded area. Note that the invisible object detection unit 435 may estimate the position of the detected object using a method other than the above examples, such as estimating the three-dimensional position of the object in world coordinates from the camera parameters estimated by self-camera calibration or the like and the position of the object in the image, or using depth information acquired from the moving device 200 or the installation device 300.

When the invisible object detection unit 435 detects an object present in an area corresponding to a blocked area, the invisible object detection unit 435 may calculate information indicating the moving speed and moving direction of the detected object by using multiple pieces of time-series second image data.

When the invisible object detection unit 435 detects that an object exists in an area corresponding to a masked area, the integration unit 436 integrates the detection result into the image data in which the masked area exists. The integration unit 436 then stores the integrated image data, which is the image data resulting from the integration, in the storage unit 420 as integrated image information 426.

FIG. 7 shows an example of processing by the integration unit 436. For example, as shown in FIG. 7, the integration unit 436 integrates an abstraction display indicating that an object has been detected in the occluded area into the image data. For example, the abstraction display is an oval display or the like in which information indicating the type of object detected, such as a person, is displayed. The abstraction display may be an abstract expression corresponding to the detected object other than the above examples. Also, referring to FIG. 8, the integration unit 436 may simply add a speech bubble or the like indicating the detected object. The integration unit 436 may transparently integrate image data of the detected object into image data in which the occluded area is present.

The integration unit 436 may perform integration other than the above examples. For example, when the extension area confirmation unit 434 determines that complementation is not possible, the integration unit 436 may integrate information indicating that it is unknown whether an object exists in the masked area into the image data. On the other hand, when the invisible object detection unit 435 does not detect an object in the masked area, the integration unit 436 may integrate information indicating that no object exists in the masked area into the image data. Furthermore, when the invisible object detection unit 435 calculates information indicating the moving speed or moving direction of an object, the integration unit 436 may also integrate information indicating the calculated moving speed or moving direction into the image data. In this way, the integration unit 436 may integrate information according to the processing results of each of the above-mentioned processing units.

The output unit 437 outputs according to the results of processing by each processing unit. For example, when integration processing is performed by the integration unit 436, the output unit 437 transmits the integrated image data to the mobile device 200 or the like that is the source of the image data in which the occluded area is present. On the other hand, when the occluded area estimation unit 433 does not estimate an occluded area, when the extended area confirmation unit 434 determines that it is not possible to complement the occluded area, when the invisible object detection unit 435 does not detect an object in the occluded area, etc., the output unit 437 instructs the corresponding mobile device 200 or the like to display the image data.

In addition, if the integration unit 436 has integrated information according to the processing results of each processing unit, the output unit 437 may output integrated image data according to the results of each integration process.

The above is an example of the configuration of the processing device 400. Next, an example of the operation of the processing device 400 will be described with reference to FIG. 9.

Figure 9 shows an example of the operation of the processing device 400. Referring to Figure 9, the occlusion area estimation unit 433 estimates an occlusion area present in the image data based on the image data included in the image information 423 (step S101). The occlusion area estimation unit 433 may estimate the occlusion area using any method.

When it is estimated that a blocked area exists in the image data (step S101, Yes), the extended area confirmation unit 434 confirms whether or not the inside of the blocked area can be complemented based on the second image data, which is image data acquired by an imaging device other than the imaging device that acquired the image data in which the blocked area exists (step S102). For example, the extended area confirmation unit 434 confirms whether or not complementation is possible by checking whether or not there are other mobile devices 200 or installed devices 300 capturing images of the inside of the blocked area.

If it is estimated that completion is possible (step S102, Yes), the invisible object detection unit 435 detects an object present in the area corresponding to the occluded area based on the second image data. For example, the invisible object detection unit 435 detects an object present in the second image data and estimates the position of the detected object (step S103), thereby detecting an object present in the area corresponding to the occluded area.

If an object is detected in an area corresponding to the occluded area, that is, if an invisible object is present (step S104, Yes), the integration unit 436 integrates the detection result into the image data in which the occluded area is present (step S105). In response to this, the output unit 437 transmits the integrated image data, which is the integration result, to the corresponding mobile device 200, etc.

On the other hand, if it is not estimated that an occluded area exists in the image data (step S101, No), if it is not estimated that complementation is possible (step S102, No), or if an invisible object does not exist (step S104, No), the output unit 437 transmits the image data to the mobile device 200 or the like to display it.

In this way, the processing device 400 has a shading area estimation unit 433, an extension area confirmation unit 434, and an invisible object detection unit 435. With this configuration, when the existence of a shading area is estimated by the shading area estimation unit 433, the invisible object detection unit 435 can detect an invisible object that is an object hidden behind the shading area in the image data in which the existence of the shading area is estimated, using the second image data, which is image data acquired from the moving device 200 or the installation device 300 identified by the extension area confirmation unit 434. As a result, it is possible to integrate information indicating the existence of an invisible object into image data in which a shading area exists. This makes it possible to make an invisible area, for example, pseudo-visible even when an obstruction or the like is present.

The processing device 400 may be configured to perform object detection in the image data at the timing when the image data is acquired from the moving device 200 or the installation device 300. In this case, the invisible object detection unit 435 may detect the invisible object using the results of detection performed in advance.

[Second embodiment]
Next, a second embodiment of the present disclosure will be described with reference to Fig. 10 and Fig. 11. In the second embodiment of the present disclosure, an overview of the configuration of a processing device 500 that is an information processing device will be described.

Fig. 10 shows an example of a hardware configuration of the processing device 500. Referring to Fig. 10, the processing device 500 has, as an example, the following hardware configuration.
・CPU (Central Processing Unit) 501 (arithmetic unit)
ROM (Read Only Memory) 502 (storage device)
RAM (Random Access Memory) 503 (storage device)
Program group 504 loaded into RAM 503
A storage device 505 for storing the programs 504
A drive device 506 that reads and writes data from and to a recording medium 510 outside the information processing device.
A communication interface 507 that connects to a communication network 511 outside the information processing device
Input/output interface 508 for inputting and outputting data
A bus 509 that connects each component

In addition, the processing device 500 can realize the functions of the estimation unit 521 and the detection unit 522 shown in FIG. 11 by the CPU 501 acquiring the program group 504 and executing it. The program group 504 is stored in advance in the storage device 505 or the ROM 502, for example, and is loaded into the RAM 503 or the like by the CPU 501 as necessary for execution. The program group 504 may be supplied to the CPU 501 via the communication network 511, or may be stored in advance in the recording medium 510, and the drive device 506 may read out the programs and supply them to the CPU 501.

Note that FIG. 10 shows an example of the hardware configuration of the processing device 500. The hardware configuration of the processing device 500 is not limited to the above-described case. For example, the processing device 500 may be configured with only a part of the above-described configuration, such as not having the drive device 506.

Based on the image data, the estimation unit 521 estimates an occluded area, which is an area where an object that cannot be confirmed in the image data may be hidden.

When the estimation unit 521 estimates the presence of a blocked area, the detection unit 522 detects an object that exists in the blocked area in the image data using second image data acquired using an imaging device that is located at a different position from the imaging device that acquired the image data.

In this way, the processing device 500 has an estimation unit 521 and a detection unit 522. With this configuration, when the estimation unit 521 estimates the existence of a blocked area, the detection unit 522 can detect an object that exists in the blocked area in the image data by using second image data acquired by an imaging device that is located at a different position from the imaging device that acquired the image data. As a result, even when an obstruction or the like is present, it becomes possible to make an area that cannot be seen, for example, pseudo-visible.

Incidentally, an information processing device such as the processing device 500 described above can be realized by incorporating a predetermined program into the information processing device. Specifically, a program that is another form of the present invention is a program for implementing a process in an information processing device that estimates, based on image data, a masked area, which is an area in which an object that cannot be confirmed in the image data may be hidden, and, if the existence of a masked area is estimated, detects an object that exists in the masked area in the image data using second image data acquired using an imaging device that is located at a different position from the imaging device that acquired the image data.

The processing method executed by the information processing device described above is such that the information processing device estimates, based on image data, an occluded area, which is an area in which an object that cannot be confirmed in the image data may be hidden, and when it estimates the existence of an occluded area, detects an object that exists within the occluded area in the image data using second image data acquired using an imaging device that is located at a different position from the imaging device that acquired the image data.

Inventions such as a program (or recording medium) having the above-mentioned configuration, or a processing method, or a processing system including a processing device and a display device having the above-mentioned configuration, have the same effects and advantages as those described above, and can therefore achieve the above-mentioned object of the present invention.

<Additional Notes>
A part or all of the above-described embodiments can be described as follows: The processing apparatus and the like of the present invention will be outlined below. However, the present invention is not limited to the following configuration.

(Appendix 1)
an estimation unit that estimates an occluded region, which is a region in which an object that cannot be confirmed in the image data is possibly hidden, based on the image data;
a detection unit that detects an object that exists in the blocked area in the image data by using second image data acquired by an imaging device that is located at a position different from the imaging device that acquired the image data, when the estimation unit estimates the existence of the blocked area;
A processing device having
(Appendix 2)
The processing device according to claim 1, wherein the detection unit detects an object present in the obstructed region in the image data by detecting an object using the second image data and estimating a position of the detected object.
(Appendix 3)
an identification unit that identifies an imaging device that acquires the second image data based on information indicating a position of the imaging device and information indicating an imaging direction;
The processing device according to claim 1 or 2, wherein the detection unit detects an object present in the blocked region in the image data by using the second image data acquired by using the imaging device identified by the identification unit.
(Appendix 4)
The processing device according to claim 3, wherein the identification unit is configured to identify, as the imaging device acquiring the second image data, an imaging device having an imaging area that overlaps with a position of the blocked area identified according to a position of the imaging device that acquired the image data.
(Appendix 5)
The processing device according to any one of claims 1 to 4, further comprising an integration unit that integrates a result of the detection by the detection unit into the image data.
(Appendix 6)
6. The processing device of claim 5, wherein the integrating unit is configured to integrate an abstracted indication of a detection of an object within the obstructed region into the image data.
(Appendix 7)
The processing device according to claim 5 or 6, further comprising an output unit configured to output a result of the integration by the integration unit to an imaging device that acquired the image data.
(Appendix 8)
An information processing device,
Based on the image data, an occlusion region is estimated, which is a region in which an object that cannot be confirmed in the image data may be hidden;
A processing method for detecting an object existing in the occluded area in image data when the existence of the occluded area is estimated, using second image data acquired by an imaging device located at a different position from the imaging device that acquired the image data.
(Appendix 9)
In the information processing device,
Based on the image data, an occlusion region is estimated, which is a region in which an object that cannot be confirmed in the image data may be hidden;
A program for implementing a process of detecting an object existing in the occluded area in image data when the existence of the occluded area is estimated, using second image data acquired by an imaging device located at a different position from the imaging device that acquired the image data.
(Appendix 10)
a processing device having: an estimation unit that estimates an obstructed area based on image data, the obstructed area being an area in which an object that cannot be confirmed in the image data may be hidden; and a detection unit that, when the estimation unit estimates the existence of the obstructed area, detects an object that exists in the obstructed area in the image data using second image data acquired by an imaging device that is located at a position different from the imaging device that acquired the image data;
a display device that displays the detection result by the detection unit together with the image data; and
A processing system comprising:

The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium. For example, the recording medium may be a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The present invention has been described above with reference to the above-mentioned embodiments, but the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

100 Processing system 200 Moving device 300 Installation device 400 Processing device 410 Communication I/F unit 420 Memory unit 421 Viewing angle information 422 Installation position/orientation information 423 Image information 424 Position information 425 Movement vector information 426 Integrated image information 427 Program 430 Calculation processing unit 431 Receiving unit 432 Movement vector calculation unit 433 Occluded area estimation unit 434 Extended area confirmation unit 435 Invisible object detection unit 436 Integration unit 437 Output unit 500 Processing device 501 CPU
502 ROM
503 RAM
504 Program group 505 Storage device 506 Drive device 507 Communication interface 508 Input/output interface 509 Bus 510 Recording medium 511 Communication network 521 Estimation unit 522 Detection unit

Claims (9)

an estimation unit that estimates an occluded area, which is an area in which an object that cannot be confirmed in the image data may be hidden, based on image data acquired by the moving first imaging device ;
a calculation unit that calculates information indicating a moving direction based on time-series position data acquired from the first imaging device;
an identification unit that identifies a position of the blocked area estimated by the estimation unit by using information indicating the moving direction calculated by the calculation unit and position data of the first imaging device, and identifies a second imaging device that is capturing an image of the inside of the blocked area according to the identified position of the blocked area;
a detection unit that detects an object present in the blocked area in second image data acquired by the second imaging device identified by the identification unit; and
A processing device having The processing device according to claim 1 , wherein the detection unit detects an object present in the obstructed region in the image data by detecting the object using the second image data and estimating a position of the detected object. The processing device according to claim 1 , wherein the identification unit is configured to identify an imaging device having an imaging area that overlaps with the identified position of the blocking area as a second imaging device that acquires the second image data. The processing device according to claim 1 , further comprising an integration unit that integrates a result of the detection by the detection unit into the image data. The processing device of claim 4 , wherein the integrating unit is further configured to integrate an abstraction indicating that an object has been detected within the occluded region into the image data. The processing device according to claim 4 , further comprising an output unit that outputs a result of the integration by the integration unit to a first imaging device that acquired the image data. An information processing device,
Based on the image data acquired by the moving first imaging device , an occluded area is estimated, which is an area in which an object that cannot be confirmed in the image data may be hidden;
Calculating information indicating a moving direction based on time-series position data acquired from the first imaging device;
specifying a position of the shielded area estimated by using information indicating the calculated moving direction and position data of the first imaging device, and specifying a second imaging device capturing an image of the inside of the shielded area according to the specified position of the shielded area;
A processing method comprising: detecting an object present in the obstructed region in second image data obtained by the identified second imaging device using the second image data. In the information processing device,
Based on the image data acquired by the moving first imaging device , an occluded area is estimated, which is an area in which an object that cannot be confirmed in the image data may be hidden;
Calculating information indicating a moving direction based on time-series position data acquired from the first imaging device;
specifying a position of the shielded area estimated by using information indicating the calculated moving direction and position data of the first imaging device, and specifying a second imaging device capturing an image of the inside of the shielded area according to the specified position of the shielded area;
A program for implementing a process of detecting an object present in the blocked area in second image data acquired by the identified second imaging device using the image data. a processing device having: an estimation unit that estimates a blocked area, which is an area where an object that cannot be confirmed in image data may be hidden, based on image data acquired by a moving first imaging device; a calculation unit that calculates information indicating a moving direction based on time-series position data acquired from the first imaging device; an identification unit that identifies a position of the blocked area estimated by the estimation unit by using the information indicating the moving direction calculated by the calculation unit and position data of the first imaging device, and identifies a second imaging device that is capturing an image within the blocked area according to the identified position of the blocked area; and a detection unit that detects an object present in the blocked area in the image data, using second image data acquired using the second imaging device identified by the identification unit;
a display device that displays the detection result by the detection unit together with the image data;
A processing system comprising:

Images