JP7709932B2 - Object identification system and object identification method - Google Patents

Description

The present invention relates to an object identification system and an object identification method.

When performing maintenance and inspection work on equipment such as servers, workers may point and call out the object. When pointing and calling, the worker extends their arm forward, points their fingertips at the object, and calls out the command.

Patent Document 1 discloses an information processing device that uses an imaging unit worn by the worker to capture an image of an object and the fingertip of a worker who is pointing at the object while standing away from the object, and identifies the intended pointing position of the person on the captured image.

JP 2016-224890 A

During maintenance and inspection work, when a worker wears a camera on their body, it is possible to determine whether the worker is pointing and calling out, and what object they are pointing at, based on the image captured by the camera. However, parallax occurs between the worker's line of sight and the direction of the camera worn by the worker. As a result, even if the fingertip and the object overlap in the visual field (field of view) of the worker who points and calls out, there will be a discrepancy in the image captured by the camera worn by the worker. As such, it is not easy to identify the object the worker is pointing at from the image captured by the camera.

Therefore, the present invention has been made in consideration of the above problems, and its purpose is to provide a technology that appropriately identifies an object based on an image captured by an imaging unit worn by a worker.

In order to achieve the above object, the present invention provides an object identification system that identifies an object pointed at by a worker based on an image captured by an imaging unit worn by the worker, and includes a calculation unit that calculates a positional relationship between the eyes of the worker and the imaging unit based on a reference image captured by the imaging unit while the worker is facing a reference object located at a predetermined distance away, a creation unit that creates an environmental map of the space captured by the imaging unit based on the spatial image captured by the imaging unit, a setting unit that sets a line of sight plane that includes the eyes and fingertips of the worker and the object on the environmental map based on the positional relationship, an estimation unit that estimates a line of sight on a specified image captured by the imaging unit of the fingertip and the object while the worker is pointing at the object, a correction unit that corrects the specified image based on the line of sight, and an identification unit that identifies the object pointed at by the worker based on the corrected image corrected by the correction unit.
In addition, in the present invention, "wearing" includes a state where the device is attached via a helmet or the like worn by a worker.

According to the present invention, it is possible to appropriately identify an object based on an image captured by an imaging unit worn by a worker.

1 is a schematic diagram of a target object identification system at a work site according to a first embodiment; 5A and 5B are diagrams comparing the field of view of a worker and a camera image according to the first embodiment. FIG. 4 is a diagram for explaining the position of a fingertip of a worker who points and calls out according to the first embodiment. FIG. 1 is a functional block diagram of an object identification system according to a first embodiment. 5A to 5C are diagrams for explaining an object identification process according to the first embodiment. FIG. 4 is a schematic diagram illustrating an approach distance according to the first embodiment. FIG. 2 is a diagram showing a screen of a display device according to the first embodiment. 5A to 5C are schematic diagrams illustrating a calibration operation according to the first embodiment. 5A to 5C are diagrams for explaining a parallax calculated in a calibration operation according to the first embodiment. 11 is a flowchart showing an object identification process according to the first embodiment. FIG. 11 is a diagram of an object identification system at a work site according to a second embodiment. 10 is a flowchart showing an object identification process according to a second embodiment. FIG. 11 is a diagram of a target object identification system at a work site according to a third embodiment.

Specific examples of an object identification system according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the examples, but is defined by the claims.

Figure 1 is a schematic diagram of a target identification system for a work site according to the first embodiment.

The object identification system 10 is a system that identifies an object T pointed at by a worker W. The object T is, for example, a device or facility such as a server. The worker W points and calls out the object T while performing maintenance and inspection of the object T. When pointing and calling out, the worker W extends his arm forward, points his fingertips F towards the object T, and yells out a command.

The worker W wears a helmet H on his head. A monocular camera 11 is attached to the upper front of the helmet H as an example of an "imaging unit" that constitutes the object identification system 10. The camera 11 may be attached to other parts of the worker W, such as the chest, in addition to the head. Furthermore, the object identification system 10 other than the camera 11 may be provided integrally with the camera 11, or may be provided in the pocket of the worker W or elsewhere in a state capable of communicating with the camera 11 wirelessly or via a wire.

Here, the camera 11 is attached to a helmet H worn by a worker W, and is positioned above the line of sight L1 of the worker W. The direction L2 of the camera 11 is fixed diagonally downward. For this reason, a parallax θ occurs between the line of sight L1 of the worker W and the direction L2 of the camera.

Figure 2 is a diagram comparing the field of view of the worker and the camera image in Example 1.

In the viewing angle (field of vision) V of worker W pointing at and calling out object T, the fingertip F and object T overlap at the center of viewing angle V. Meanwhile, in captured image i1, which is an example of a "predetermined image" captured by camera 11 of worker W pointing at and calling out object T, the fingertip F is displayed at the center of captured image i1 and object T is displayed at the top of captured image i1. The difference (deviation) in the positions of the fingertip F and object T displayed in captured image i1 is proportional to the magnitude of the parallax θ between the line of sight of worker W and camera 11, and the distance from camera 11 to object T. In this embodiment, camera 11 is above the line of sight of worker W, so object T is displayed at the top of captured image i1.

Figure 3 is a diagram explaining the position of the worker's fingertips when pointing and calling out in Example 1, where (a) shows the state in which the worker's fingertips are located on the line of sight, and (b) shows the state in which the worker's fingertips are located to the left of the line of sight.

In general, when worker W points and calls out, the fingertip F of worker W is located near the vertical center of worker W's visual field angle V. In other words, assuming that worker W or the action of worker W (angle of bending arm) is different for each pointing and calling, the fingertip F of worker W who points and calls out the object T is located somewhere near the vertical center of worker W's visual field angle V.

Figure 4 is a functional block diagram of the object identification system according to the first embodiment, and Figure 5 is a diagram illustrating the object identification process according to the first embodiment.

The object identification system 10 executes an object identification process to identify the object T pointed at by the worker W. The object identification system 10 includes a monocular camera 11, a memory 12 as an example of a "storage unit", a processor 13, a warning unit 14, and a display unit 15.

The camera 11 is attached to a helmet H worn by a worker W, and captures an image of an object T. The camera 11 transmits an image i1 of the object T to the processor 13.

The memory 12 includes, for example, a semiconductor storage medium such as a RAM (Random Access Memory), a ROM (Read Only Memory), or an SSD (Solid State Drive). The memory 12 may store personal parallax for each worker W, vertical parallax of the worker W for each of a plurality of heights relative to the worker W, and a predetermined range from the object T for each of a plurality of objects T. The personal parallax is the individual difference in the position of the fingertip F of each worker W when pointing and naming. The vertical parallax is the tendency of the position of the fingertip F of the worker W when the worker W points and names objects T of different heights. The predetermined range is the range within which the worker W is allowed to approach the object T.

The processor 13 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. The processor 13 includes a calculation unit 21, a creation unit 22, a setting unit 23, an estimation unit 24, and an identification unit 25.

The calculation unit 21 calculates the positional relationship between the eyes of the worker W and the camera 11 through a calibration operation described below that is performed in advance before the worker W starts the maintenance and inspection work. The calculation unit 21 may calculate the distance between the fingertip F and the target object T based on an environmental map described below of the space captured by the camera 11. Furthermore, the calculation unit 21 may calculate the distance between the eyes of the worker W and the fingertip F based on the positional relationship between the eyes of the worker W and the camera 11.

The creation unit 22 creates an environmental map of the space captured by the camera 11 using SLAM (Simultaneous Localization and Mapping) or the like based on the spatial image captured by the camera 11.

The setting unit 23 sets a line of sight plane P (FIG. 5(b)) including the eyes and fingertips F of the worker W and the target object T on the environmental map based on the positional relationship calculated by the calculation unit 21. The setting unit 23 may set the line of sight plane P based on the positional relationship and personal parallax. Furthermore, the setting unit 23 may set the line of sight plane P based on the positional relationship and vertical parallax. Furthermore, the setting unit 23 may set the line of sight plane P based on the positional relationship, personal parallax, and vertical parallax.

The estimation unit 24 estimates the line of sight L1 of the worker W on the line of sight plane P based on the coordinate difference between the central coordinate image on the captured image i1 and the coordinates of the fingertip F (see FIG. 5(c)).

The identification unit 25 identifies the object T pointed to by the worker W based on the line of sight L1 of the worker W. At this time, the eyes and fingertips F of the worker W and the target object T are lined up in this order on the line of sight L1 of the worker W.

Figure 6 is a schematic diagram illustrating the approach distance in Example 1.

The warning unit 14 generates an alert A1 when the distance between the fingertip F of the worker W and the object T approaches a predetermined range. Furthermore, the warning unit 14 generates an alert A1 when the distance between the worker's eyes and the fingertip F is less than a first distance (e.g., 15 cm) or a second distance greater than the first distance (e.g., 1.5 times or more longer than the average arm length of an adult). Furthermore, the warning unit 14 generates an alert A1 when the position of the fingertip F on the environmental map is a third distance or more away from the previous position of the fingertip F.

Figure 7 shows the screen of the display device in Example 1.

The display unit 15 may be a display device such as a display. The display unit 15 displays the object T on the captured image i1 and a processed image i2 in which the line of sight L1 of the worker W is emphasized. The display unit 15 may also emphasize and display either the object T or the line of sight L1 of the worker W in the processed image i2.

Next, we will explain the calibration process for calculating the parallax θ between the line of sight L1 of the worker W and the direction L2 of the camera 11.

Figure 8 is a schematic diagram illustrating the calibration process according to the first embodiment, and Figure 9 is a diagram illustrating the parallax calculated in the calibration process according to the first embodiment.

As shown in FIG. 8, the camera 11 is attached to the body (e.g., the forehead) of the worker W. A marker M (e.g., an AR marker) is provided at a position a predetermined distance away from the worker W as an example of a "reference object." The marker M is provided vertically at a predetermined height. This allows the worker W to easily perform calibration. The worker W stands upright facing the marker M from a position a predetermined distance away from the marker M, and aligns his/her line of sight L1 with the marker M.

The camera 11 captures an image of the marker M. The calculation unit 21 recognizes the marker M in the captured image i1 captured by the camera 11, and calculates the two-dimensional coordinates pM of the marker M in the coordinates of the captured image i1, the rotation vector R indicating the positional relationship between the marker M and the camera 11, and the translation vector t = (tx, ty, tz). At this time, when the marker M on the line of sight plane P (line of sight L1) is away from the camera 11 by t (m), it is captured at the coordinates pM in the coordinate system of the camera 11 (captured image i1).

The calculation unit 21 calculates the parallax θ between the line of sight L1 and the direction L2 of the camera 11 based on the rotation vector R. The calculation unit 21 is able to convert from the image coordinate system to the line of sight plane P based on the coordinate pM and the parallax θ, and can convert the fingertip coordinates in the captured image i1 into three-dimensional coordinates. Note that since the eyes of the worker W are located vertically below the camera 11, the eye coordinates are (0, 0, tz).

Figure 10 is a flowchart showing the object identification process according to Example 1.

First, the calculation unit 21 calculates the positional relationship between the eyes of the worker W and the camera 11 through a calibration operation before the worker W starts the maintenance and inspection work (S1).

Next, the creation unit 22 creates an environmental map of the space captured by the camera 11 using SLAM or the like based on the spatial image captured by the camera 11 (S2). Next, the setting unit 23 sets a line of sight plane P including the eyes and fingertips F of the worker W and the target object T on the environmental map based on the positional relationship calculated by the calculation unit 21 (S3).

Next, the estimation unit 24 estimates the line of sight L1 of the worker W on the line of sight plane P based on the coordinate difference between the central coordinate image on the captured image i1 and the coordinates of the fingertip F (S4). Next, the identification unit 25 identifies the object T pointed to by the worker W based on the line of sight L1 of the worker W (S5).

According to this configuration, the object identification system 10, which identifies the object T pointed at by the worker W based on an image captured by the camera 11 worn by the worker W, includes a calculation unit 21, a creation unit 22, a setting unit 23, an estimation unit 24, and an identification unit 25. The calculation unit 21 calculates the positional relationship between the eyes of the worker W and the camera 11 based on the marker M captured by the camera 11 while the worker W faces the marker M set at a predetermined distance. The creation unit 22 creates an environmental map of the space captured by the camera 11 based on the spatial image captured by the camera 11. The setting unit 23 sets a line of sight plane P including the eyes and fingertips F of the worker W and the object T on the environmental map based on the positional relationship. The estimation unit 24 estimates the line of sight L1 of the worker W on the line of sight plane P based on the coordinate difference between the central coordinate and the coordinate of the fingertip on the captured image i1 captured by the camera 11 of the fingertip F and the object T while the worker W is pointing at the object T. The identification unit 25 identifies the object T pointed at by the worker W based on the line of sight L1. This makes it possible to appropriately identify the object T pointed at by the worker W based on the image captured by the camera 11 worn by the worker W.

The system further includes a memory 12 that records the personal parallax for each of multiple workers W, and the setting unit 23 sets the line of sight plane P based on the positional relationship between the eyes of the worker W and the camera 11 and the personal parallax. This allows the line of sight plane P to be set appropriately even if there are individual differences among the workers W.

Furthermore, the system is provided with a memory 11 that records the vertical parallax of the worker W for each of a number of heights relative to the worker W, and the setting unit 23 sets the line of sight plane based on the positional relationship between the eyes of the worker W and the camera 11 and the vertical parallax. This makes it possible to appropriately set the line of sight plane even in a work site where the worker W looks up or down at the target object T.

Furthermore, the device includes a memory 12 that records a predetermined range from the target object T for each of a plurality of targets T, and a warning unit 14 that issues an alert A1. The calculation unit 21 calculates the distance between the fingertip F and the target object T based on the environmental map, and the warning unit 14 issues an alert A1 when the distance between the fingertip F and the target object T approaches the predetermined range. This makes it possible to prevent the worker W from approaching the target object T that is energized more than necessary.

The system further includes a warning unit 14 that issues an alert A1. The calculation unit 21 calculates the distance between the eyes of the worker W and the fingertip F based on the positional relationship between the eyes of the worker W and the camera 11. The warning unit 14 issues an alert A1 when the distance between the eyes of the worker W and the fingertip F is less than the first distance or equal to or greater than a second distance greater than the first distance, or when the position of the fingertip F on the environmental map is a third distance or greater from the previous position of the fingertip F. This allows the worker W to confirm that the position and orientation of the camera 11 have changed due to the camera 11 colliding with another object or the fastener of the camera 11 becoming loose.

An object identification system 30 according to Example 2 of the present invention will be described. Note that the object identification system 30 according to the second embodiment differs from the object identification system 10 according to the first example only in the configuration of the processor 33, and the other configurations are the same as those of the object identification system 10 according to Example 1. Therefore, the differences from Example 1 will be mainly described.

Figure 11 is a diagram of an object identification system at a work site according to Example 2. Figure 12 is a flowchart showing an object identification process according to Example 2.

The processor 33 of the object identification system 30 includes a calculation unit 21, a creation unit 22, an estimation unit 44, and an identification unit 45.

In Example 1, it was assumed that when worker W points and calls out, the fingertip F of worker W is located near the vertical center of the field of view V of worker W. In Example 2, by assuming that the fingertip F is located near the vertical and horizontal center of the field of view V of worker W, it becomes possible to estimate the line of sight L1 without setting the line of sight plane P. Therefore, the estimation unit 44 estimates the line of sight L1 from the above assumption and the positional relationship between the eyes of worker W and the camera 11. The identification unit 45 identifies the object T that worker W is gazing at near the vertical and horizontal center of the field of view based on the line of sight L1. Furthermore, when the above assumption is made, information on the fingertip F is not required to estimate the line of sight L1, so pointing and calling out is not necessary in this example.

With this configuration, the object T that the worker W is gazing at can be easily and appropriately identified based on the image captured by the camera 11 worn by the worker W.

An object identification system 50 according to Example 3 of the present invention will be described. Note that the object identification system 50 according to the third embodiment differs from the object identification system 10 according to Example 1 only in the position where it is attached to the worker W, and the rest of the configuration is the same as the object identification system 10 according to Example 1. Therefore, the differences from Example 1 will be mainly described.

Figure 13 is a diagram of a target identification system at a work site according to Example 3.

The camera 11 of the object identification system 50 is attached to the right side of the lumet H that the worker W wears on his/her head. In other words, the camera 11 of the object identification system 50 is positioned to the right of the eye of the worker W.

With this configuration, when the fingertip F of a worker W who points at and calls out an object T is located either to the left or right of the center of the worker W's viewing angle V, the object T pointed to by the worker W can be appropriately identified based on the image captured by the camera 11 worn by the worker W.

The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Also, a configuration of another embodiment can be added to the configuration of one embodiment. Also, other configurations may be added to, deleted from, or substituted for part of the configuration of another embodiment.

10... object identification system, 11... camera, 12... memory, 14... warning unit, 15... display unit, 21... calculation unit, 22... creation unit, 23... setting unit, 24... estimation unit, 25... identification unit, 30... object identification system, 44... estimation unit, 45... identification unit, 50... object identification system, Al... alert, F... fingertip, i1... captured image, i2... processed image, L1... line of sight, P... line of sight plane, T... object, W... worker, θ... parallax

Claims (10)

1. An object identification system that identifies an object pointed to by an operator based on an image captured by a monocular imaging unit worn by the operator,
a calculation unit that calculates a positional relationship between the eyes of the worker and the imaging unit based on a reference image captured by the imaging unit in a state where the worker faces a reference object provided at a predetermined distance from the worker;
a creation unit that creates an environmental map of the space captured by the imaging unit based on the spatial image captured by the imaging unit;
a setting unit that sets a line of sight plane including the eyes and fingertips of the worker and the object on the environmental map based on the positional relationship;
an estimation unit that estimates the line of sight of the worker on the line of sight plane based on a coordinate difference between a center coordinate of the fingertip and the object in a predetermined image captured by the imaging unit while the worker is pointing at the object, and a coordinate of the fingertip;
and an identification unit that identifies the object pointed to by the worker based on the line of sight. 1. An object identification system for identifying an object that an operator is gazing at based on an image captured by a monocular imaging unit worn by the operator, comprising:
a calculation unit that calculates a positional relationship between the eyes of the worker and the imaging unit based on a reference image captured by the imaging unit in a state where the worker faces a reference object provided at a predetermined distance from the worker;
a creation unit that creates an environmental map of the space captured by the imaging unit based on the spatial image captured by the imaging unit;
an estimation unit that estimates a line of sight of the worker on the environmental map based on the positional relationship;
An object identification system comprising: an identification unit that identifies an object that the worker is gazing at based on the line of sight. The reference object is a marker provided vertically at a predetermined height.
The object identification system according to claim 1 or 2. A storage unit is provided for recording personal disparities for each of the plurality of workers,
The setting unit sets the line of sight plane based on the positional relationship and the personal parallax.
The object identification system of claim 1 . a storage unit configured to record vertical parallax of the worker for each of a plurality of heights relative to the worker,
The setting unit sets the line of sight plane based on the positional relationship and the vertical parallax.
The object identification system of claim 1 . A storage unit that records a predetermined range from each of the plurality of objects;
an alert section for generating an alert;
Equipped with
The calculation unit calculates a distance between the fingertip and the target object based on the environmental map,
The warning unit generates an alert when the distance between the fingertip and the target object approaches the predetermined range.
The object identification system of claim 1 . Equipped with a warning unit that generates an alert,
the calculation unit calculates a distance between the eye of the worker and the fingertip based on the positional relationship between the eye of the worker and the imaging unit;
The warning unit issues an alert when a distance between the operator's eyes and the fingertip is less than a first distance or equal to or greater than a second distance greater than the first distance, or when a position of the fingertip on the environmental map is separated by a third distance or more from a previous position of the fingertip.
The object identification system of claim 1 . A display unit is provided for displaying a processed image in which at least one of the object and the line of sight of the worker is emphasized on the predetermined image.
The object identification system of claim 1 . 1. A method for identifying an object by an object identification system, the method comprising:
calculating a positional relationship between the eyes of the worker and the imaging unit based on a coordinate difference between a center coordinate on a reference image captured by the imaging unit while the worker faces a reference object set at a predetermined distance from the reference image and a coordinate of the object;
creating an environmental map of the space captured by the imaging unit based on the spatial image captured by the imaging unit;
Based on the positional relationship, a line of sight plane is set that includes the eyes and fingertips of the worker and the object on the environmental map;
estimating a line of sight of the worker on the line of sight plane based on a coordinate difference between a center coordinate of the fingertip and the target object in a predetermined image captured by the imaging unit while the worker is pointing at the target object, and a coordinate of the fingertip;
An object identification method for identifying an object pointed to by the worker based on the line of sight. 1. A method for identifying an object by an object identification system, the method comprising: identifying an object that an operator is gazing at based on an image captured by a monocular imaging unit worn by the operator, the method comprising:
calculating a positional relationship between the eyes of the worker and the imaging unit based on a coordinate difference between a center coordinate on a reference image captured by the imaging unit while the worker faces a reference object set at a predetermined distance from the reference image and a coordinate of the object;
creating an environmental map of the space captured by the imaging unit based on the spatial image captured by the imaging unit;
estimating a line of sight of the worker on the environmental map based on the positional relationship;
An object identification method for identifying an object that the worker is gazing at based on the line of sight.