JP7637249B2 - Object Search System - Google Patents

Description

The present invention relates to an object search system, an object search device, and an object search method used when a user searches for an object, such as a fallen object, on the ground.

There are known systems that use radar or similar to detect obstacles and fallen objects (hereafter referred to as "falling objects") in vast areas (work sites, grounds, facilities, parking lots, roads, railroad tracks, runways, etc.). In such systems, the position (coordinates, etc.) of a fallen object can be identified with a certain degree of accuracy based on the detection results of the radar or similar.

Conventionally, when a fallen object is detected by radar, a worker from the management company would then search for the fallen object while looking at a map based on the coordinate information (location information) of the fallen object. In this case, for example, the map can be displayed on a tablet device held by the worker, and the location of the fallen object can be highlighted, making it easier for the worker to perform this task.

In this case, as described in Patent Document 1, for example, a scope (head-mounted display) worn by the worker is used instead of a tablet terminal, and the position information of the falling object recognized by the radar is recognized on the scope side, and a display showing the positional relationship between the worker and the falling object is displayed on the scope in real time, making this work particularly easy to perform.

International Publication WO2019/065045

However, there are various environments in which workers must search for fallen objects. For example, the road conditions may be poor and make normal walking difficult, or the worker may be on a road with vehicle traffic. In such cases, the worker must search for fallen objects while visually checking the safety of their feet and surroundings. In such situations, it is not safe for the worker to keep their eyes fixed on the scope screen, and even when a head-mounted display is used, it is not easy to search for fallen objects and ensure safety at the same time. For this reason, there was a demand for technology that would allow for easy and safe searching for fallen objects.

The present invention has been developed in consideration of the above situation and aims to solve the above problems.

The present invention provides an object search system including a detection device that detects position information of an object, and a display device worn by a user, the detection device including an object position information detection unit that detects the object and acquires position information of the object, and a transmission unit that transmits the position information of the object to the display device, the display device including a reception unit that receives the position information of the object from the detection device, a position and orientation recognition unit that recognizes the position and orientation of the display device, a sound output unit that outputs a plurality of types of sounds, and a control unit that recognizes a positional relationship between the display device and the object from the position and orientation of the display device and the position information of the object, and selects one of the plurality of types of sounds in accordance with the positional relationship and causes the sound output unit to output the selected sound, the display device displays a substantial image of an environment including the object to the user, and the control unit receives the substantial image and transmits the substantial image to the display device. The distance between the display device and the object is recognized from the position information of the object and the position of the display device recognized by the position/orientation recognition unit , and a target recognition range, which is an area within the field of view of the solid image on the display device , is set as an area of a certain range around the horizontal center of the field of view at a single point in time, and at a time point different from the single point in time, the horizontal width is set narrower than at the single point in time if the distance is greater than at the single point in time, and wider than at the single point in time if the distance is smaller than at the single point in time, and the target recognition range is displayed superimposed on the solid image , and the sound is switched depending on the result of determining whether the object is present within the target recognition range, and if the object is present within the target recognition range, the sound pattern is changed depending on the distance.

The present invention makes it possible to search for objects easily and safely.

1 is a diagram showing a configuration of an object search system according to an embodiment. 1 is a block diagram showing a configuration of a scope (display device: object searching device) in an object searching system according to an embodiment. 4 is an example of an image recognized by a scope in the object search system according to the embodiment. 11 is a typical example (part 1) of a positional relationship between a recognized fallen object and a scope in the object search system according to the embodiment. 13 is a typical example (part 2) of a positional relationship between a recognized fallen object and a scope in the object search system according to the embodiment. 13 is a typical example (part 3) of a positional relationship between a recognized fallen object and a scope in the object search system according to the embodiment. 10 is an example of a change in the positional relationship between a fallen object and a scope when an operator searches for the fallen object using the object search system according to the embodiment. 1 is an example of a flowchart illustrating an object searching method according to an embodiment.

Hereinafter, an object search system according to an embodiment of the present invention will be described. This object search system is used, for example, to search for fallen objects on the ground, similar to the technology described in Patent Document 1.

Figure 1 is a diagram showing the schematic configuration of an object search system 1 according to this embodiment. This object search system 1 has a detection device 10 that uses radar to detect a falling object (object) 100, recognizes the positional relationship between itself and the falling object 100, and captures an image of the falling object 100, and a scope (display device: object search device) 20 that is worn or carried by an operator (user) 200 and wirelessly connected to the detection device 10. Figure 1 also shows a block diagram of the detection device 10.

The detection device 10 is composed of a detection head 10A installed in an area where a falling object 100 can be searched, and a detection device main body 10B connected to the detection head 10A by wire or wirelessly. The detection head 10A is provided with a radar 11 and a camera 12. As is well known, the radar 11 transmits a predetermined electromagnetic wave (for example, millimeter wave band radio waves of 60 GHz or more) in a certain direction and receives the reflected wave for each direction, thereby recognizing the distance to the reflector for each direction. The presence of the falling object 100, which is a reflector, can be recognized based on the relationship between the direction and the distance at this time. The control of the radar 11 and its signal processing (calculation processing for recognizing the distance based on the time difference between the transmission timing and the detection timing of the reflected wave, etc.) are performed by the radar control unit 13 of the detection device main body 10B.

At this time, the radar control unit 13 recognizes the position of the detection head 10A (or radar 11) in advance, and can recognize the position information of the recognized falling object 100 from this position and the detection results by the radar 11. Furthermore, if the falling object 100 is sufficiently larger than the detection resolution of the radar 11, the radar control unit 13 can also recognize the shape and size (shape information) of the falling object 100. In other words, the radar 11 and the radar control unit 13 function as an object position information detection unit that recognizes the position information, etc. of the falling object 100.

In addition, the camera 12 can photograph the falling object 100 by having its field of view controlled by the camera control unit 14 based on the detection results of the radar 11.

In addition to the radar control unit 13 and the camera control unit 14, the detection device main body 10B is provided with a detection device side display unit 15, which is a display that displays various information and images, a detection device side wireless communication unit (transmitter) 16 that wirelessly transmits acquired data to the scope 20, and a detection device control unit 17 that controls the entire detection device 10. The detection device control unit 17 can display, in real time, on the detection device side display unit 15, radar images in which the radar detection results (detection times) for each direction by the radar 11 are mapped, and images including the falling object 100 obtained by the camera 12. In addition, the detection results of the radar 11, position information and shape information of the recognized falling object 100, and image data captured by the camera 12 are transmitted to the scope 20 side by wireless communication by the detection device side wireless communication unit (transmitter) 16.

Figure 2 is a block diagram showing the configuration of the scope (display device: object search device) 20 in Figure 1. The object search method according to the embodiment of the present invention is executed by this scope 20. The scope 20 is provided with a scope-side wireless communication unit (receiving unit) 21 that receives data transmitted from the detection device 10. In addition, a position and attitude recognition unit 22 that recognizes the position and attitude (direction) of the scope 20 in three-dimensional space is also provided. The position and attitude recognition unit 22 is provided with a position recognition unit 221 that recognizes the position information of the scope 20 by receiving a GNSS signal. In addition, in order to recognize the direction (direction) of the scope 20, a gyro sensor 222 that recognizes the rotation angle of the scope 20 is provided. Here, the position (position information) of the scope 20 is substantially equal to the position (position information) of the worker 200 wearing the scope 20, and the attitude of the scope 20 is set as the direction of the line of sight of the worker 200 wearing the scope 20. By using the gyro sensor 222, it is possible to detect changes in this attitude, especially over a short period of time.

The scope 20 is also provided with a storage unit 23, which is a non-volatile memory that stores map information within the area in which the fallen object 100 is searched, a scope side display unit 24, which is a head-mounted display, and a scope control unit (control unit) 25 that controls the entire scope 0. As described in Patent Document 1, the scope side display unit 24 allows the worker 200 to directly view a real image (actual image) through the scope side display unit 24, and can display AR (Augmented Reality) information superimposed on the image. Therefore, the scope control unit 25 can display map information recognized as being visible from the current situation of the scope 20 on the scope side display unit 24 as AR information based on the recognized position information and attitude of the scope 20. As shown in FIG. 1, the scope 20 has a glasses-like shape, and the scope side display unit 24 actually occupies most of the area of the scope 20.

At this time, the scope control unit 25 can recognize in advance the field of view of the actual image in the scope 20 (scope side display unit 24) based on the detection results of the position/posture recognition unit 22. On the other hand, the scope control unit 25 can determine whether or not the falling object 100 is currently within the field of view of the scope 20 (scope side display unit 24) based on the position information of the falling object 100 obtained from the detection device 10 via the scope side wireless communication unit 21, and if so, can use a method such as highlighting the range corresponding to the falling object 100 as AR information on the scope side display unit 24. This allows the worker 200 to easily search for the falling object 100.

However, what the worker 200 perceives through the scope side display unit 24 is the actual image that the worker actually sees, and the two-dimensional image displayed as AR information as described above, and the worker 200 is unable to directly perceive the actual distance between himself/herself and the falling object 100 inside. In this regard, in the technology described in Patent Document 1, a display (specific display) according to the positional relationship (distance) between the worker 200 (scope 20) and the falling object 100 is performed on the scope side display unit 24, and the worker 200 can perceive the distance (positional relationship) between himself/herself and the falling object 100 simply by looking at the scope side display unit 24. Therefore, the worker 200 can gradually approach the falling object 100 while looking at the scope side display unit 24.

However, the environments (situations related to the road surface R in FIG. 1) in which fallen object 100 is searched for vary. For example, the road surface R may not be a well-maintained paved road but may be rough, or the environment may be dangerous, with many vehicles running on the road surface R. In such cases, it is not desirable for the worker 200 to constantly stare at the screen of the scope side display unit 24.

For this reason, the scope 20 is provided with a sound output unit 26 that outputs sound (sound signal) rather than visually displaying an image or the like to indicate the positional relationship between the scope 20 and the falling object 100, particularly the distance between them. The sound may include voice. In order for the worker 200 to easily determine the situation of the change in the positional relationship between the worker 200 and the falling object 100 when the relationship changes from moment to moment, it is preferable to use, as such a sound signal, multiple types of sound output simplified according to rules and switched between, rather than using, for example, a voice output that emits text information. It is preferable that the switching of the sound signal is performed based on the relationship between the recognized falling object 100 and the field of view of the worker 200 (scope 20).

For this reason, the object search method executed in this scope 20 comprises: (1) an object position information acquisition process for obtaining position information of the falling object 100 from the detection device 10; (2) a position/attitude recognition process for recognizing the position and attitude of the scope 20 by the position/attitude recognition unit 22; (3) a positional relationship recognition process for recognizing the positional relationship between the scope 20 and the falling object 100 from the position and attitude of the scope 20 and the position information of the falling object 100; and (4) a sound output process for selecting and outputting one of multiple types of sound signals preset in accordance with this positional relationship.

As described above, the scope control unit 25 can recognize the position and size of the falling object 100 from the data received from the detection device 10 (object position information acquisition process). Meanwhile, the scope control unit 25 can recognize the position and orientation of the scope 20 itself by the position and orientation recognition unit 22 (position and orientation recognition process), and can therefore recognize the position and size of the falling object 100 in the actual image (the field of view of the worker 200) that the worker 200 sees directly on the scope side display unit 24. Alternatively, it can recognize whether the falling object 100 is within the field of view of the actual image. Here, the position in the actual image displayed on the scope side display unit 24 corresponds to the direction (azimuth) as seen from the scope 20, and the size of the object in this actual image corresponds to the angle of view of this object as seen from the scope 20.

Figure 3 is an example of a solid image (two-dimensional image) seen from the worker 200 side on the scope 20 (scope side display unit 24) in this case. Here, a falling object 100 is present within this field of view. Figures 4a to 4c show three typical situations of a falling object 100 as seen from the scope 20. The size of the falling object 100 in the image on the scope side display unit 24 corresponds to its angle of view; if the distance between the falling object and the object is large, this angle of view is small, and if the distance is small, this angle of view is large. Figure 4a shows the state where the distance is the greatest (smallest angle of view), Figure 4c shows the state where the distance is the smallest (largest angle of view), and Figure 4b shows an intermediate state between these two.

As described above, the scope control unit 25 obtains the position information and shape (size) information of the falling object 100 from the detection device 10, but there is a limit to the accuracy, and it is actually difficult to accurately recognize the position and size of the falling object 100 in the image of FIG. 3 from only this information, and to accurately show the falling object 100 in the image of FIG. 3, for example. For this reason, it is preferable to simply recognize the distance range to the falling object 100 and the range in the field of view of the falling object 100 (positional relationship recognition process) and communicate this to the worker 200, as shown in FIGS. 4a to 4c. In this case, the positional relationship between the worker 200 and the falling object 100 can be more appropriately expressed by setting a narrow field of view when the distance is large and a wide field of view when the distance is small, in accordance with the size of the falling object 100 recognized in the image as described above.

The scope control unit 25 can recognize the positional relationship between the scope 20 and the falling object 100 in a simplified manner by dividing the range of the distance R in Fig. 4a to Fig. 4c. In Fig. 4a to Fig. 4c, Fig. 4a shows a situation when the distance D between the scope 20 and the falling object 100 is large (20 m or more), Fig. 4b shows a situation when the distance D is smaller (less than 20 m, 5 m or more), and Fig. 4c shows a situation when the distance D is the smallest (less than 5 m) as viewed from above, and the oscillation patterns of the sound signal emitted by the sound output unit 26 when each of these situations is recognized are shown on the lower side.

In the upper part of Figures 4a to 4c, a range of a predetermined angle (target recognition range X) around the central axis Y is set as the field of view range within which the presence or absence of a falling object 100 is recognized as described above. This angle is, for example, 5° (target recognition range X1) in Figure 4a where distance D is the greatest, 90° (target recognition range X3) in Figure 4c where distance D is the smallest, and 45° (target recognition range X2) in Figure 4b where distance D is intermediate between the above.

When the situation in the upper part of Fig. 4a is recognized (when the distance D between the scope 20 and the falling object 100 is 20 m or more and the object 100 is recognized to be within the target recognition range X1), the scope control unit 25 causes the sound output unit 26 to output a periodic sound as shown in the lower part of Fig. 4a. This sound output is actually recognized by the worker 200 as a periodic sound such as "beep beep beep". Similarly, when the situation in the upper part of Fig. 4b is recognized (when the distance D between the scope 20 and the falling object 100 is less than 20 m but 5 m or more and the object 100 is recognized to be within the target recognition range X2), the scope control unit 25 causes the sound output unit 26 to output a periodic sound such as "beep beep beep beep" as shown in the lower part of Fig. 4b. When the situation in the upper part of Figure 4c is recognized (when the distance D between the scope 20 and the falling object 100 is recognized to be less than 5 m and the object 100 is recognized to be within the target recognition range X3), a continuous "beep" sound as shown in the lower part of Figure 4c is output (this is the sound output process).

The worker 200 can easily identify these sound signals. On the other hand, when it is recognized that the falling object 100 does not exist within each of the target recognition ranges X in Figures 4a to 4c, these sound signals are not emitted and a silent state is established. For this reason, for example, when the falling object 100 does not exist within the target recognition range X in Figures 4a to 4c, and the worker 200 (scope 20) changes direction or moves, causing the falling object 100 to enter the target recognition range X set according to the distance D, the worker 200 can easily recognize this fact by the sound signal. After that, the worker 200 can move straight from this state to get closer to the falling object 100.

Figure 5 shows an example of such a work situation in correspondence with Figures 4a to 4c. In Figure 5, the state (1) is when the falling object 100 does not exist within the target recognition range X (X1) in Figure 4a. In this case, since no sound is emitted, the worker 200 can recognize this and change his/her direction. As a result, as shown in (2), when the falling object 100 enters the target recognition range X1, the worker 200 can recognize this by the sound. In this state, the distance D to the falling object 100 is not small, but the falling object 100 exists almost in front of the worker in this state, so the worker can move forward as it is and sequentially reach the states (3) and (4), and can recognize this by the sound. When the falling object X is no longer present within the target recognition range X corresponding to the distance D between them when moving forward, the sound is no longer emitted, so the worker 200 can realize the states (3) and (4) by changing his/her direction, as in the case of (1).

By performing such operations, the worker 200 can approach the falling object 100. At this time, the worker 200 can recognize the positional relationship (distance, direction of travel) between himself and the falling object 100 by sound alone. Note that the image in FIG. 3 also shows the target recognition range X. In FIGS. 4a to 4c and 5, the target recognition range X is shown as a range of horizontal azimuth angles, but the horizontal range of the elliptical target recognition range X in FIG. 3 corresponds to the target recognition range X in FIGS. 4a to 4c and 5. The vertical width of the elliptical shape that is the target recognition range X in the image in FIG. 3 can be set appropriately depending on the size of the recognized falling object 100, etc. Also, the display method of the target recognition range X can be set appropriately.

Figure 6 is a flowchart showing this object search method. First, as described above, the scope 20 performs an object position information acquisition process to obtain position information of the falling object 100 from the detection device 10 (S1). Next, the scope 20 performs a position/attitude recognition process to recognize its own position and attitude by the position/attitude recognition unit 22 (S2), and thereby performs a positional relationship recognition process to recognize the positional relationship between the scope 20 and the falling object 100 from its own position and attitude and the positional information of the falling object 100 (S3). Next, the scope 20 performs a sound output process to select and output one of a plurality of types of sound signals set in advance according to this positional relationship (S5). The worker 200 recognizes the sound output and performs an action such as moving forward or changing direction as described above (S5). The above actions are repeated until the worker 200 finds the falling object 100 (the worker 200 gets close enough to the falling object 100) (S6: Yes).

In the above example, the pattern of the sound signal was changed depending on the relative positions of the falling object 100 and the worker 200 (scope 20), but instead of or in addition to changing the pattern, the volume may also be changed.

Furthermore, the scope control unit 25 can display, as AR information on the scope display unit 24, information obtained from the detection device 10 that is useful when the worker 200 searches for the falling object 100. As in the case of Patent Document 1, such information includes the shape of the falling object 100 captured by the camera 12 (or an image including the falling object 100).

In the above example, the worker 200 can recognize the positional relationship between himself and the falling object 100 by the sound signal, but the positional relationship shown in Figures 4a to 4c may be displayed by a display method on the scope (display device) side, in addition to the sound signal. One such example is a method of changing the blinking pattern when blinking the display depending on the positional relationship.

Even in such a case, as mentioned above, the worker can recognize the positional relationship between the fallen object and himself (the scope) only by the sound signal, so the worker does not need to constantly watch the screen and can simply check the contents as needed. This allows the worker to easily perform this work while ensuring the safety of the surroundings.

In the above example, the position and orientation recognition unit 22 is provided only on the scope (display device) 20 side, but it may also be provided on the detection device (detection head). In this case, even if the detection device moves, its own position and orientation are accurately recognized, and even if the detection device moves, the display device can obtain accurate position information of the fallen object.

In the above example, the radar 11 was used to obtain position information of the falling object 100 in the detection device 10. However, the position information of the falling object can also be recognized by other methods, such as optical methods and acoustic methods, and the same operation as described above is possible.

Furthermore, even if the display device itself can recognize the position information of the falling object by, for example, a signal emitted by the falling object itself, the display device can perform a similar operation. In such a case, the above-mentioned detection device is not necessary, and the above-mentioned display device alone can be used to perform a similar operation. In other words, the above-mentioned display device alone can be used as an object search device. Furthermore, while the display device allows the user to visually recognize the actual image as described above, the search can be made easier even if the user cannot visually recognize the actual image on the display device (object search device), as long as sound output is performed as described above.

In addition, in the above object search method, the positional relationship between the falling object 100 and the scope 20 is recognized on the scope (display device) 20 side. However, this positional relationship may be recognized on a device other than the display device, and the display device may only output a sound corresponding to this. In other words, in the above object search method, the object position information acquisition process, position/attitude recognition process, and positional relationship recognition process may be performed inside or outside the display device as appropriate.

The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components, and that such modifications are also within the scope of the present invention.

1 Object search system 10 Detection device 10A Detection head 10B Detection device main body 11 Radar (object position information detection unit)
12 Camera 13 Radar control unit (object position information detection unit)
14 Camera control unit 15 Detection device side display unit 16 Detection device side wireless communication unit (transmitter)
17 Detection device control unit 20 Scope (display device: object search device)
21 Scope side wireless communication unit (receiver)
22 Position and orientation recognition unit 23 Memory unit 24 Scope side display unit 25 Scope control unit (control unit)
26 Sound output unit 100 Falling object (object)
200 Worker (user)
221 Position recognition unit 222 Gyro sensor R Road surface X, X1 to X3 Target recognition range Y Central axis

Claims (1)

An object search system including a detection device for detecting position information of an object and a display device worn by a user,
The detection device includes:
an object position information detection unit that detects the object and acquires position information of the object;
a transmitting unit that transmits position information of the object to the display device;
Equipped with
The display device includes:
a receiving unit that receives position information of the object from the detection device;
a position and orientation recognition unit that recognizes a position and orientation of the display device;
a sound output unit that outputs a plurality of types of sound;
a control unit that recognizes a positional relationship between the display device and the object based on the position and attitude of the display device and position information of the object, and selects one of the plurality of types of sounds in accordance with the positional relationship and outputs the selected sound to the sound output unit;
Equipped with
The display device displays to the user a real-world image of an environment including the object;
The control unit is
a distance between the display device and the object is recognized from the received position information of the object and the position of the display device recognized by the position/orientation recognition unit , and a target recognition range, which is an area within the field of view of the solid image on the display device , is set as an area of a certain range around the center of the field of view in the horizontal direction at a single point in time, and at a time point different from the single point in time, the horizontal width is set narrower than at the single point in time if the distance is greater than at the single point in time, and wider than at the single point in time if the distance is smaller than at the single point in time, and the target recognition range is displayed superimposed on the solid image , and the sound is switched depending on the result of determining whether the object is present within the target recognition range,
An object searching system characterized in that, when the object is present within the target recognition range, the sound pattern is changed depending on the distance.

Images