JP7739062B2 - Periphery monitoring system, periphery monitoring method, and periphery monitoring device - Google Patents

Description

This disclosure relates to a perimeter monitoring system, a perimeter monitoring method, and a perimeter monitoring device.

For example, technology is known for detecting moving objects within a detection area during warehouse work performed using industrial vehicles such as forklifts (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2021-033701

Work machines travel in places such as warehouses or factories where there are obstacles, including people, in the surrounding area. If all obstacles, including people, around the work machine were detected and an alarm were output, the alarm would be issued frequently. Therefore, it is desirable to appropriately determine the possibility of a collision between the work machine and nearby obstacles, including people.

Aspects of the present disclosure aim to provide a perimeter monitoring system, perimeter monitoring method, and perimeter monitoring device that appropriately monitor the possibility of a collision between a work machine and obstacles, including people in the vicinity.

According to an aspect of the present disclosure, there is provided a perimeter monitoring system comprising: a photographing unit that photographs a work machine and its surroundings; a recognition unit that recognizes the work machine and obstacles in its surroundings from the images photographed by the photographing unit; a calculation unit that calculates the movement direction of the work machine and the movement direction of the obstacle from the recognition results of the recognition unit; and a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit.

According to an aspect of the present disclosure, there is provided a periphery monitoring method that includes photographing a work machine and its surroundings, recognizing the work machine and obstacles in its surroundings from the photographed image, calculating the movement direction of the work machine and the movement direction of the obstacle from the recognition results, and determining the possibility of a collision between the work machine and the obstacle based on the calculated movement direction of the work machine and the movement direction of the obstacle.

According to an aspect of the present disclosure, there is provided a periphery monitoring device comprising: a recognition unit that recognizes the work machine and obstacles in its vicinity from video footage of the work machine and its vicinity; a calculation unit that calculates the direction of movement of the work machine and the direction of movement of the obstacle from the recognition results of the recognition unit; and a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the direction of movement of the work machine and the direction of movement of the obstacle calculated by the calculation unit.

According to aspects of the present disclosure, a perimeter monitoring system, perimeter monitoring method, and perimeter monitoring device are provided that can appropriately monitor the possibility of a collision between a work machine and obstacles, including people in the vicinity.

1 is a schematic diagram illustrating an example of a location monitored by a perimeter monitoring system according to an embodiment of the present invention. 1 is a functional block diagram illustrating an example of a perimeter monitoring system according to an embodiment of the present invention. FIG. 10 is a schematic diagram illustrating a recognition result of a forklift. 3 is a schematic diagram illustrating an example of a monitoring image output by the perimeter monitoring system according to the present embodiment. FIG. FIG. 2 is a schematic diagram illustrating an example of the positional relationship between a forklift and a person. 6 is a schematic diagram illustrating the possibility of a collision between the forklift and the person in the positional relationship shown in FIG. 5 . FIG. FIG. 10 is a schematic diagram illustrating another example of the positional relationship between the forklift and the person. FIG. 8 is a schematic diagram illustrating the possibility of a collision between the forklift and the person in the positional relationship shown in FIG. 7 . 4 is a flowchart illustrating an example of processing of the perimeter monitoring system according to the present embodiment. FIG. 1 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention.

Embodiments of the present disclosure will be described below with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be combined as appropriate. In addition, some components may not be used.

In this embodiment, a forklift 100 (see Figure 4) driven by power from a battery is used as an example of a work machine, but the work machine is not limited to this. For example, the work machine may be a wheel loader driven by power from a battery. Furthermore, the work machine is not limited to one driven by electricity, and may be a forklift driven by hydraulics, etc.

<Perimeter monitoring system>
The perimeter monitoring system 1 monitors the perimeter of the forklift 100 at a work site of the forklift 100. The perimeter monitoring system 1 appropriately monitors the possibility of a collision between the work machine and obstacles, including nearby people. In the following explanation, a person will be described as an example of an obstacle. The perimeter monitoring system 1 monitors the perimeter of the forklift 100 in places where a person 110 (see FIG. 4 ) may be working or moving around the forklift 100, such as an inspection area or a material handling area in a warehouse. The perimeter monitoring system 1 also monitors the perimeter of the forklift 100 in places with poor visibility on the travel path along which the forklift 100 and the person 110 move, such as corners of a travel path in a factory.

Figure 1 is a schematic diagram illustrating an example of a location monitored by a perimeter monitoring system 1 according to this embodiment. Figure 1 shows a parking area S for a forklift 100 in a warehouse as an example of a location to be monitored. The warehouse has, for example, a height of h [m] from the floor to the ceiling and a depth of d [m]. Within the warehouse, the forklift 100 is parked at the parking area S and lifts and lowers cargo. A person 110 also works and moves around within the warehouse. In the example shown in Figure 1, the perimeter monitoring system 1 monitors the area around the forklift 100 in the vicinity of the parking area S.

Figure 2 is a functional block diagram showing an example of a perimeter monitoring system 1 according to this embodiment. In this embodiment, the perimeter monitoring system 1 determines the possibility of a collision between the forklift 100 and a person 110 based on the direction of movement of the forklift 100 and the direction of movement of a person 110 around the forklift 100.

The perimeter monitoring system 1 comprises a camera 2 as a capture unit, a monitor 3, a vibration unit 4 as an alarm unit, a video memory unit 5, a learning memory unit 6, and a perimeter monitoring device 10. The camera 2, monitor 3, vibration unit 4, video memory unit 5, learning memory unit 6, and perimeter monitoring device 10 are connected to each other via wired or wireless connections so that data can be communicated.

Camera 2 is capable of capturing images of the forklift 100 and person 110. Camera 2 is placed in a location, such as a warehouse, around the forklift 100 where person 110 may be working or moving. In the example shown in Figure 1, camera 2 is placed near the ceiling of the warehouse and captures images of parking lot S and the area around parking lot S. Camera 2 is placed on the path along which the forklift 100 and people move, such as the corner of a path within a factory. Camera 2 transmits the video data of the captured image to the perimeter monitoring device 10. Camera 2 is capable of capturing images over a range of, for example, several meters.

In this embodiment, camera 2 is a monocular camera. Camera 2 has an optical system and an image sensor. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The video captured by camera 2 includes multiple images per unit time depending on the frame rate.

The monitor 3 is located, for example, in a warehouse or a remote monitoring room away from the travel path. The monitor 3 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). The monitor 3 displays the video transmitted from the perimeter monitoring device 10. In this embodiment, as shown in FIG. 4, the monitor 3 displays a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 superimposed on the video 200 captured by the camera 2. The movement area 102 and the movement area 112 will be described later. The monitor 3 may also be located inside the forklift 100.

The vibration unit 4 is a vibration generating device. The vibration unit 4 generates vibrations to alert the operator. The vibration unit 4 generates vibrations in response to a control signal from the periphery monitoring device 10. In this embodiment, the vibration unit 4 generates vibrations when the periphery monitoring device 10 determines that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, the vibration unit 4 is worn on the body of the operator of the forklift 100. The vibration unit 4 may be installed in another location, such as on the seat of the driver's seat, as long as the vibrations can be recognized by the operator in the driver's seat.

After generating vibrations, the vibration unit 4 may stop the vibrations in response to a control signal from the periphery monitoring device 10. After generating vibrations, the vibration unit 4 may stop the vibrations based on an operation by the operator to stop the vibrations. After generating vibrations, the vibration unit 4 may stop the vibrations after a predetermined time has elapsed.

In addition to or instead of the vibration unit 4, another alarm unit (not shown) may be provided. The alarm unit is installed near the driver's seat of the forklift 100. The alarm unit may also be installed in another location as long as it can issue an alarm to the operator in the driver's seat or to the area around the forklift 100. The alarm unit uses, for example, a speaker or lamp to emit a sound or light to alert the operator to the possibility of a collision. The alarm unit uses, for example, a display panel located in the driver's cab to display an image to alert the operator to the possibility of a collision.

The video memory unit 5 is a storage device that stores video data, including data from before and after the time when it is determined that a collision is possible. The video memory unit 5 stores video under the control of the video storage control unit 23. The video data to be stored will be described later. The video memory unit 5 may be, for example, at least one of non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM, a magnetic disk, a flexible disk, or a magneto-optical disk.

The learning memory unit 6 is a storage device that stores learning image data, which is image data used for learning. The learning memory unit 6 stores learning image data under the control of the image storage control unit 24. Learning image data will be described later. The learning memory unit 6 uses, for example, at least one of non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Random Only Memory), flash memory, EPROM (Erasable Programmable Random Access Memory), EEPROM (Electrically Erasable Programmable Random Access Memory), a magnetic disk, a flexible disk, and a magneto-optical disk.

<Perimeter monitoring system control system>
The perimeter monitoring device 10 monitors the perimeter of the forklift 100 and determines the possibility of contact between the forklift 100 and a person 110. If the perimeter monitoring device 10 determines that there is a possibility of contact between the forklift 100 and a person 110, it outputs an alarm. A camera 2, a monitor 3, a vibration unit 4, a video storage unit 5, and a learning storage unit 6 are connected to the perimeter monitoring device 10. The perimeter monitoring device 10 includes a numerical calculation unit (processor) such as a CPU (Central Processing Unit). The perimeter monitoring device 10 includes an image acquisition unit 11, a recognition unit 12, a tracking unit 13, a calculation unit 14, an image generation unit 15, a determination unit 16, an output control unit 21, an alarm output unit 22, a video storage control unit 23, an image storage control unit 24, and a storage unit 29.

The video acquisition unit 11 acquires video data of the video captured by the camera 2. The video data acquired by the video acquisition unit 11 includes the forklift 100 and person 110 within the monitored area. The video acquisition unit 11 converts the acquired video data into image data for each frame and outputs it to the recognition unit 12. The video acquisition unit 11 stores the video data in the memory unit 29.

The recognition unit 12 recognizes the forklift 100 and the person 110 from the video data of the video acquired by the video acquisition unit 11. More specifically, the recognition unit 12 performs recognition processing of the forklift 100 and the person 110 on the image data of each frame extracted from the video data. The recognition unit 12 performs recognition processing using artificial intelligence (AI). The recognition unit 12 uses a learning model to recognize the forklift 100 and the person 110 using image data as input data, and outputs the recognition results as output data.

The learning model is generated by learning the feature quantities of the forklift 100 and the person 110 from multiple learning image data. By performing machine learning using learning images including the forklift 100 and the person 110 as training data, a learning model is generated in which the feature quantities of the forklift 100 and the person 110 are input and the recognition results of the forklift 100 and the person 110 are output.

The training image data includes a forklift 100 and a person 110. The training image data includes, for example, forklifts 100 and people 110 displayed in various positions in the image. The training image data includes, for example, forklifts 100 and people 110 in various orientations and sizes. The training image data includes, for example, image data captured in various lighting conditions, weather conditions, seasons, and time periods. The training image data includes, for example, image data in which part of the forklift 100 and part of the person 110 are missing.

In this embodiment, the learning model may be trained using an average image obtained by averaging multiple images extracted from video captured by camera 2. Images extracted from the captured video may contain noise due to the forklift 100 and person 110 being moving objects. By using the average image data as training image data, noise is reduced. The training image data includes, for example, average image data of an average image obtained by averaging multiple consecutive frame images to reduce noise.

The recognition unit 12 associates the detection frame and center point of the detection frame of the recognized forklift 100 and person 110 with the image on which the recognition process was performed, and stores them in the memory unit 29. In this embodiment, the recognition unit 12 associates the detection frame 101 and center point C1 of the detection frame 101 of the recognized forklift 100 (see FIG. 5), and the detection frame 111 and center point C2 of the detection frame 111 of the person 110 (see FIG. 5), with the image, and stores them in the memory unit 29. The detection frames of the recognized forklift 100 and person 110 are, for example, rectangular in shape, surrounding the outlines of the forklift 100 and person 110.

Figure 3 is a schematic diagram illustrating the recognition results of the forklift 100. Figure 3 is a schematic diagram of an image of the forklift 100 viewed from directly above. Note that the image actually captured by the camera 2 is an image captured from diagonally above, as shown in Figure 4. In the example of Figure 3, the detection frame 101 of the forklift 100 is rectangular and surrounds the outer shape of the forklift 100. The center point of the detection frame 101 is designated C1.

The tracking unit 13 tracks the forklift 100 and person 110 recognized by the recognition unit 12. In this embodiment, it tracks the center point C1 of the forklift 100 and the center point C2 of the person 110 recognized by the recognition unit 12. The tracking unit 13 stores the tracking results in the memory unit 29.

The calculation unit 14 calculates the movement direction of the forklift 100 and the movement direction of the person 110 from the recognition result of the recognition unit 12. More specifically, the calculation unit 14 calculates the movement direction and speed of the forklift 100 and person 110 recognized by the recognition unit 12. The calculation unit 14 associates the calculated movement direction and speed of the forklift 100 and person 110 with the image that underwent recognition processing and stores them in the memory unit 29.

In this embodiment, the movement direction and speed of the forklift 100 are the movement direction and speed of the center point C1 of the detection frame 101 of the forklift 100. In this embodiment, the movement direction and speed of the person 110 are the movement direction and speed of the center point C2 of the detection frame 111 of the person 110. The movement direction and speed of the center point can be calculated using a known method using a Kalman filter.

The image generation unit 15 generates an image showing the detection frames of the forklift 100 and person 110 recognized by the recognition unit 12, as well as the direction and speed of movement. The image generation unit 15 associates the generated image with the image and stores it in the memory unit 29.

In this embodiment, the movement areas indicating the movement direction and speed of the forklift 100 and person 110 are shown in the shape of a sector. The movement area of the forklift 100 is the area through which the forklift 100 is likely to move within a few seconds. The movement area of the person 110 is the area through which the person 110 is likely to move within a few seconds. By defining the movement area as a sector, errors in the movement direction are taken into account. The center line of the sector is a line extending from the center point along the movement direction. The radius is, for example, approximately the distance traveled within a few seconds. The radius of the sector is determined according to the movement speed. The radius becomes longer as the movement speed increases. The central angle changes depending on the area of the obstacle being monitored. For the forklift 100, for example, it is approximately 60°. In this embodiment, the central angle for the person 110 is approximately 10°, as the area of the person 110 is smaller than that of the forklift 100.

In this way, in this embodiment, the image generation unit 15 generates a rectangular image indicating the detection frame of the forklift 100 and person 110, and a sector-shaped image indicating the movement area.

In Figure 3, the movement direction and speed of the forklift 100 are indicated by a sector-shaped movement area 102 centered on a central point C1. The length of the sector-shaped movement area of the forklift 100 in the front-to-rear direction is R1 [m]. The length R1 [m] is, for example, several meters.

Figure 4 is a schematic diagram illustrating an example of a monitoring image 200 output by the perimeter monitoring system 1 according to this embodiment. In the example shown in Figure 4, an image showing the detection frame 101 of the forklift 100, an image showing a sector-shaped movement area 102 indicating the movement direction and speed of the forklift 100, an image showing the detection frame 111 of the person 110, and an image showing the sector-shaped movement area 112 indicating the movement direction and speed of the person 110 are superimposed on the image 200 captured by the camera 2.

If the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the image generation unit 15 may generate an image to be superimposed on the video 200, indicating the possibility of a collision using text, graphics, etc.

The determination unit 16 determines the possibility of a collision between the forklift 100 and the person 110 based on the movement direction of the forklift 100 and the movement direction of the person 110 calculated by the calculation unit 14. As a result, the determination unit 16 determines that there is no possibility of a collision, for example, even if the person 110 is present near the forklift 100, if the movement directions of the forklift 100 and the person 110 are opposite each other.

In this embodiment, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when the direction of travel of the forklift 100 and the person 110 intersect. For example, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when a line extending in the direction of travel of the forklift 100 intersects with a line extending in the direction of travel of the person 110. Furthermore, the determination unit 16 may take travel speed into consideration when making a determination. For example, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when a line segment extending in the direction of travel of the forklift 100 and having a length corresponding to the speed of the forklift 100 intersects with a line segment extending in the direction of travel of the person 110 and having a length corresponding to the speed of the person 110. As a result, the determination unit 16 determines that there is no possibility of a collision when the direction of travel of the forklift 100 and the direction of travel of the person 110 do not intersect.

In this embodiment, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. By using the movement area 102 and the movement area 112, the determination unit 16 can determine the possibility of a collision while taking into account the error in the calculated movement direction.

Figure 5 is a schematic diagram illustrating an example of the positional relationship between a forklift 100 and a person 110. Figure 6 is a schematic diagram illustrating the possibility of a collision in the positional relationship between the forklift 100 and the person 110 in Figure 5. Figure 6 is a diagram that extracts the movement area 102 of the forklift 100 and the movement area 112 of the person 110 from Figure 5. In the example shown in Figures 5 and 6, the movement area 102 of the forklift 100 and the movement area 112 of the person 110 do not overlap. In this case, the determination unit 16 does not determine that there is a possibility of a collision between the forklift 100 and the person 110.

Figure 7 is a schematic diagram illustrating another example of the positional relationship between the forklift 100 and the person 110. Figure 8 is a schematic diagram illustrating the possibility of a collision in the positional relationship between the forklift 100 and the person 110 in Figure 7. Figure 8 is a diagram that extracts the movement area 102 of the forklift 100 and the movement area 112 of the person 110 from Figure 7. In the example shown in Figures 7 and 8, the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. In this case, the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110.

The output control unit 21 controls the display of a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110, superimposed on the video 200 captured by the camera 2. In this embodiment, the marker indicating the movement area 102 and the marker indicating the movement area 112 are fan-shaped images. In this embodiment, the output control unit 21 controls the display of an image generated by the image generation unit 15, superimposed on the video 200 captured by the camera 2. More specifically, the output control unit 21 controls the display of a rectangular image indicating the detection frames of the forklift 100 and the person 110 and a fan-shaped image indicating the movement area, superimposed on the video 200 captured by the camera 2, so as to output the image.

In this embodiment, the output control unit 21 controls the output to the monitor 3. The output control unit 21 may also control the distribution of video to the monitor 3 via a network (not shown).

The alarm output unit 22 controls the vibration unit 4 to vibrate when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, the alarm output unit 22 controls the vibration unit 4 to vibrate when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110.

After outputting the alarm, the alarm output unit 22 may be controlled to stop the alarm if the possibility of a collision has disappeared. In this embodiment, after outputting the alarm, the alarm output unit 22 may be controlled to stop the vibration of the vibration unit 4 if the possibility of a collision has disappeared.

When the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the video storage control unit 23 controls the video storage unit 5 to store the video 200 captured by the camera 2 with the image generated by the image generation unit 15 superimposed on it. More specifically, the video storage control unit 23 controls the video 200 captured by the camera 2 to store the video in the video storage unit 5 with a rectangular image indicating the detection frame of the forklift 100 and the person 110 and a sector-shaped image indicating the movement area superimposed on it. The video storage control unit 23 may also control the video storage unit 5 to store the sector-shaped image indicating the movement area of the forklift 100 and the person 110 without superimposing it.

When the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the video storage control unit 23 may control the video storage unit 5 to store the video captured by the camera 2, including the time before and after the determination. The video storage control unit 23 may also control the video storage unit 5 to store the video 200 captured by the camera 2, including the time before and after the determination, with a rectangular image indicating the detection frame of the forklift 100 and the person 110 and a sector-shaped image indicating the movement area superimposed on the video 200.

The image storage control unit 24 controls the learning storage unit 6 to store images that have been recognized by the recognition unit 12, in order to improve the accuracy of the learning model. The image storage control unit 24 controls the learning storage unit 6 to store, for example, image data captured under specified conditions. "Under specified conditions" refers to conditions similar to those under which image data with low recognition accuracy of the forklift 100 and person 110 was captured. "Under specified conditions" refers to conditions such as brightness, weather, season, and time of day that satisfy specified conditions.

The image storage control unit 24 may also process the captured image data and control the storage of multiple processed image data in the learning storage unit 6.

For example, if the perimeter monitoring system 1 determines that there is a possibility of a collision and the operator's vibration unit 4 vibrates, and the operator determines that there is no possibility of a collision, the recognition accuracy is predicted to be low. For example, if the perimeter monitoring system 1 determines that there is a possibility of a collision and the monitor monitoring the video displayed on the monitor 3 determines that there is no possibility of a collision, the recognition accuracy is predicted to be low.

The memory unit 29 includes at least one of RAM, ROM, flash memory, and a hard disk drive. The memory unit 29 stores data used in the processing of the perimeter monitoring device 10.

<Perimeter monitoring system processing>
It is assumed that the perimeter monitoring system 1 is activated during the operating hours of the forklift 100. While the perimeter monitoring system 1 is activated, the camera 2 captures images of the surroundings, and the perimeter monitoring device 10 determines the possibility of a collision between the forklift 100 and a person 110.

Figure 9 is a flowchart showing an example of processing by the perimeter monitoring system 1 according to this embodiment. The perimeter monitoring device 10 acquires video (step S101). More specifically, the perimeter monitoring device 10 acquires video data of video captured by the camera 2 using the video acquisition unit 11. The perimeter monitoring device 10 converts the acquired video data into image data for each frame using the video acquisition unit 11 and outputs the image data to the recognition unit 12. The perimeter monitoring device 10 proceeds to step S102.

The periphery monitoring device 10 recognizes the forklift 100 and the person 110 (step S102). More specifically, the periphery monitoring device 10 recognizes the forklift 100 and the person 110 using the recognition unit 12 from the image data acquired from the video acquisition unit 11. The periphery monitoring device 10 proceeds to step S103.

The periphery monitoring device 10 tracks the forklift 100 and the person 110 (step S103). More specifically, the periphery monitoring device 10 uses the tracking unit 13 to track the forklift 100 and the person 110 recognized by the recognition unit 12. The periphery monitoring device 10 proceeds to step S104.

The perimeter monitoring device 10 calculates the direction and speed of movement (step S104). More specifically, the perimeter monitoring device 10 calculates the direction and speed of movement of the forklift 100 and the direction and speed of movement of the person 110 using the calculation unit 14. The perimeter monitoring device 10 proceeds to step S105.

The periphery monitoring device 10 generates an image (step S105). More specifically, the periphery monitoring device 10 uses the image generation unit 15 to generate an image showing the detection frames of the forklift 100 and person 110 recognized by the recognition unit 12, as well as the direction and speed of movement. The periphery monitoring device 10 proceeds to step S106.

The periphery monitoring device 10 controls the output of the video (step S106). More specifically, the periphery monitoring device 10 controls the output control unit 21 to superimpose a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 on the video 200 captured by the camera 2. The periphery monitoring device 10 proceeds to step S107.

The perimeter monitoring device 10 determines the possibility of a collision (step S107). More specifically, the perimeter monitoring device 10 determines, via the determination unit 16, the possibility of a collision between the forklift 100 and the person 110 based on the movement direction of the forklift 100 and the movement direction of the person 110 calculated by the calculation unit 14. The perimeter monitoring device 10 may determine, via the determination unit 16, that there is a possibility of a collision between the forklift 100 and the person 110 if the movement direction of the forklift 100 and the movement direction of the person 110 intersect. The perimeter monitoring device 10 may also determine, via the determination unit 16, that there is a possibility of a collision between the forklift 100 and the person 110 if the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. If the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110 (Yes in step S107), the process proceeds to step S108. If the determination unit 16 does not determine that there is a risk of a collision between the forklift 100 and the person 110 (Yes in step S107), the process proceeds to step S110.

If it is determined that there is a possibility of a collision (Yes in step S107), the periphery monitoring device 10 controls the output of an alarm (step S108). More specifically, the periphery monitoring device 10 controls the alarm output unit 22 to vibrate the vibration unit 4. The periphery monitoring device 10 proceeds to step S109.

The operator recognizes that the vibrations of the vibration unit 4 may cause the forklift 100 to collide with the person 110. The operator then takes action to avoid a collision with the person 110. For example, the operator applies the brakes, releases the accelerator, or operates the steering wheel.

The perimeter monitoring device 10 controls to store the video (step S109). More specifically, when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the perimeter monitoring device 10 controls the video storage control unit 23 to superimpose a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 on the video 200 captured by the camera 2, including the time before and after the determination, and store the video in the video storage unit 5. Note that the perimeter monitoring device 10 may also control the video storage control unit 23 to store the marker indicating the movement area 102 and the marker indicating the movement area 112 of the person 110 in the video storage unit 5 without superimposing them.

If it is determined that there is no possibility of a collision (No in step S107), the periphery monitoring device 10 controls the alarm to stop (step S110). More specifically, the periphery monitoring device 10 controls the alarm output unit 22 to stop the vibration of the vibration unit 4.

<Computer System>
FIG. 10 is a block diagram showing an example of a computer system 1000 according to this embodiment. The above-described periphery monitoring device 10 includes the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU, a main memory 1002 including a nonvolatile memory such as a ROM and a volatile memory such as a RAM, a storage 1003, and an interface 1004 including an input/output circuit. The functions of the above-described periphery monitoring device 10 are stored as a program in the storage 1003. The processor 1001 reads the program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing in accordance with the program. Note that the program may be distributed to the computer system 1000 via a network.

<Effects>
As described above, in this embodiment, the possibility of a collision between the forklift 100 and the person 110 is determined based on the moving direction of the forklift 100 and the moving direction of the person 110. This embodiment can determine that there is no possibility of a collision, for example, even if the person 110 is present near the forklift 100, if the moving directions of the forklift 100 and the person 110 are opposite to each other. This embodiment can appropriately determine the possibility of a collision between the forklift 100 and the nearby person 110. In this way, this embodiment can appropriately monitor the possibility of a collision between the forklift 100 and the nearby person 110.

In this embodiment, if the direction of movement of the forklift 100 and the direction of movement of the person 110 intersect, it is determined that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, if the direction of movement of the forklift 100 and the direction of movement of the person 110 do not intersect, it can be determined that there is no possibility of a collision. This embodiment can more appropriately determine the possibility of a collision between the forklift 100 and a nearby person 110.

In this embodiment, if the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap, it is determined that there is a possibility of a collision between the forklift 100 and the person 110. This embodiment can more appropriately determine the possibility of a collision by taking into account errors in the calculated movement direction.

In this embodiment, an alarm is output when it is determined that there is a possibility of a collision between the forklift 100 and the person 110. This embodiment can notify the operator of the forklift 100, etc., of the possibility of a collision by issuing an alarm.

In this embodiment, when it is determined that there is a possibility of a collision between the forklift 100 and the person 110, a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 are superimposed on the video captured by the camera 2 and stored. This embodiment can store the video when it is determined that there is a possibility of a collision. This makes it possible to later confirm the validity of the determination of the possibility of a collision.

In this embodiment, when it is determined that there is a possibility of a collision between the forklift 100 and the person 110, the video captured by the camera 2 is stored, including the period before and after the determination. This embodiment can store video before and after the determination that there is a possibility of a collision. This makes it possible to later confirm the validity of the determination of the possibility of a collision.

This embodiment uses a learning model to recognize the forklift 100 and people 110 around the forklift 100 using images extracted from video as input data. This embodiment can improve the recognition accuracy of the forklift 100 and people 110.

In this embodiment, the learning model is trained using an average image obtained by averaging multiple images extracted from video. According to this embodiment, learning can be performed using an average image with appropriately reduced noise. This embodiment can improve the recognition accuracy of the forklift 100 and the person 110.

<Modification>
In the above, the camera 2 is placed near the ceiling of the warehouse or on the travel path, but this is not a limitation. The camera 2 may be placed on the forklift 100. In this case, the direction of movement of the person, etc., is calculated from the image captured by the camera 2 placed on the forklift 100. The direction of movement of the forklift 100 can be determined, for example, from the steering angle. The speed of the forklift 100 can be determined, for example, from a speedometer provided on the forklift 100. The movement area of the forklift 100 can be determined from the direction of movement and vehicle speed determined as described above.

Step S105 in the flowchart shown in Figure 9 is not essential and may be omitted. The movement direction and speed may be calculated without generating a detection frame and an image indicating the movement direction and speed.

Steps S106, S108, S109, and S110 in the flowchart shown in Figure 9 are not required steps and may be omitted.

In the above, a person is an example of an obstacle. Other examples of obstacles include stationary objects such as other work machines and cargo in a factory. In the example of other work machines, the possibility of a collision may be determined when the direction of movement of the forklift 100 intersects with the direction of movement of the other work machine. Also, the possibility of a collision may be determined when the movement area of the forklift 100 overlaps with the movement area of the other work machine. In the example of cargo, the recognition unit 12 may recognize the cargo, and the possibility of a collision may be determined when the center point of the cargo intersects with the movement direction of the forklift 100. Also, the possibility of a collision may be determined when the center point of the cargo overlaps with the movement area of the forklift 100.

In the above description, the work machine is described as a forklift 100, but this is not limited to this. The work machine may also be a wheel loader or hydraulic excavator that is powered by battery power or an engine, etc.

1...Periphery monitoring system, 2...Camera (photographing unit), 3...Monitor, 4...Vibration unit (alarm unit), 5...Video memory unit, 6...Learning memory unit, 10...Periphery monitoring device, 11...Video acquisition unit, 12...Recognition unit, 13...Tracking unit, 14...Calculation unit, 15...Image generation unit, 16...Determination unit, 21...Output control unit, 22...Alarm output unit, 23...Video memory control unit, 24...Image memory control unit, 29...Memory unit, 100...Forklift (work machine), 101...Detection frame, 102...Movement area, 110...Person, 111...Detection frame, 112...Movement area.

Claims (11)

an imaging unit that images the work machine and the surrounding area of the work machine;
a recognition unit that recognizes the work machine and obstacles around the work machine from the image captured by the imaging unit;
a calculation unit that calculates the movement direction and speed of the work machine and the movement direction and speed of the obstacle from the recognition result of the recognition unit;
a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit;
Equipped with
the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and the speed of the work machine overlaps with a movement area indicating the movement direction and the speed of the obstacle;
the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction,
The central angle of the sector changes depending on the area of the obstacle to be monitored,
The radius of the sector increases as the moving speed increases.
Perimeter surveillance system. the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when the movement direction of the work machine and the movement direction of the obstacle intersect.
The perimeter monitoring system according to claim 1 . an alarm unit that outputs an alarm when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle;
The perimeter monitoring system according to claim 1 or 2 , comprising: The alarm unit is a vibration unit that generates vibration to alert an operator,
the vibration unit vibrates when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle.
The perimeter monitoring system according to claim 3 . an image storage control unit that, when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle, controls the image captured by the image capturing unit to store a marker indicating a movement area in the movement direction of the work machine and a marker indicating a movement area in the movement direction of the obstacle, superimposed on the image;
The perimeter monitoring system according to claim 1 , comprising: an image storage control unit that, when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle, stores the image captured by the image capture unit, including the period before and after the time point at which the determination is made;
The perimeter monitoring system according to claim 1 , comprising: the recognition unit uses a learning model to recognize the work machine and obstacles around the work machine using an image cut out from the video as input data.
The perimeter monitoring system according to any one of claims 1 to 6 . The learning model is trained using an average image obtained by averaging a plurality of images extracted from the video.
The perimeter monitoring system according to claim 7 . The perimeter monitoring system according to claim 1 , wherein the obstacle is a person. Taking an image of the work machine and the surrounding area of the work machine;
Recognizing the work machine and obstacles around the work machine from the captured video;
calculating a moving direction and speed of the work machine and a moving direction and speed of the obstacle from the recognition result;
determining, based on the calculated movement direction of the work machine and the movement direction of the obstacle, that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and speed of the work machine overlaps with a movement area indicating the movement direction and speed of the obstacle;
Including,
the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction,
The central angle of the sector changes depending on the area of the obstacle to be monitored,
The radius of the sector increases as the moving speed increases.
A perimeter monitoring method performed by a perimeter monitoring system. a recognition unit that recognizes obstacles around the work machine and the surrounding area of the work machine from images of the work machine and the surrounding area of the work machine;
a calculation unit that calculates the movement direction and speed of the work machine and the movement direction and speed of the obstacle from the recognition result of the recognition unit;
a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit;
Equipped with
the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and the speed of the work machine overlaps with a movement area indicating the movement direction and the speed of the obstacle;
the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction,
The central angle of the sector changes depending on the area of the obstacle to be monitored,
The radius of the sector increases as the moving speed increases.
Perimeter monitoring device.