JP7692716B2 - Workplace monitoring system - Google Patents

Description

The present invention relates to a monitoring system for work sites where civil engineering works are carried out using hydraulic excavators and the like.

Patent document 1 describes an image recognition device equipped with an object discrimination unit that uses a neural network method to determine whether a candidate object detected from an input image is a pedestrian or not.

Specifically, the object discrimination unit performs a judgment (part judgment) to compare the contour and size of the candidate object in the input image currently being input from the imaging device with the size and shape of any of the patterns of predetermined parts that make up a pedestrian (e.g., head, torso, limbs, etc.) stored in the learning model storage unit, and obtains a judgment output value that indicates the degree of match. If the judgment output value exceeds a predetermined discrimination threshold, the object discrimination unit judges the candidate object to be a pedestrian, since the candidate object is any of the predetermined parts that make up a pedestrian (e.g., head, torso, limbs, etc.).

JP 2008-21034 A

In order to improve the accuracy of object recognition using neural network techniques, etc., it is effective to collect learning data corresponding to various situations and update the recognition model (learning model) required for object recognition processing. However, when creating learning data based on image data captured by an imaging device, if the amount of image data becomes enormous, creating the learning data takes time and effort, which can lead to problems such as delays in updating the recognition model required to improve the accuracy of object recognition.

The present invention aims to provide a work site monitoring system that can easily create learning data and improve the accuracy of object recognition by appropriately updating the recognition model.

A monitoring system for a work site according to one aspect of the present invention includes a monitoring device having a memory unit and executing a recognition process to recognize the presence or absence of an object within a monitoring range using image data captured by an imaging device that captures images of the surroundings of a work machine and a recognition model stored in the memory unit , and a learning device that generates update data for updating the recognition model . When the reliability of a recognition process result of an object recognized by the recognition process is equal to or higher than a predetermined threshold , the monitoring device stores and holds in the memory unit the image data used in the recognition process. The learning device generates the update data by learning using learning data including image data selected by operating an input device from the image data stored and held in the memory unit and supervised data created based on the image data .

The present invention provides a work site monitoring system that can easily create learning data and appropriately update the recognition model to improve the accuracy of object recognition.

FIG. 1 is a diagram showing the configuration of a work site monitoring system. FIG. 2 is a functional block diagram showing main functions of the work site monitoring system. FIG. 13 is a diagram showing an example of a display image displayed on a display screen of a display device, showing a display image of a worker working while standing near a photographing device. FIG. 13 is a diagram showing an example of a display image displayed on a display screen of a display device, showing a display image in which a worker is working in a crouched position. 5 is a flowchart showing the contents of a monitoring process executed by a controller of the monitoring device. 5 is a flowchart showing the contents of an image data transmission process executed by a controller of the monitoring device. 6 is a flowchart showing the contents of a recognition model update process executed by a controller of the monitoring device. 13 is a diagram showing an example of a display image displayed on a display screen of a display device in the monitoring device according to the first modification. FIG. 13A and 13B are diagrams showing another example of a display image displayed on the display screen of the display device in the monitoring device according to the first modification.

A monitoring system for a work site according to an embodiment of the present invention will be described with reference to the drawings. At the work site, a work machine is in operation. The work machine is a machine used for various types of work such as civil engineering work, construction work, and demolition work. In this embodiment, an example will be described in which the work machine is a crawler-type hydraulic excavator 1.

Figure 1 is a diagram showing the configuration of a work site monitoring system 5. As shown in Figure 1, the work site monitoring system 5 has a hydraulic excavator 1 that performs work at the work site, and a management server 53 installed in a management center 52. The management center 52 is installed, for example, in a facility such as the head office, branch office, or factory of the manufacturer of the hydraulic excavator 1, a rental company for the hydraulic excavator 1, a data center that specializes in operating servers, or a facility of the owner of the hydraulic excavator 1. The management server 53 is an external device that remotely manages (understands and monitors) the status of the hydraulic excavator 1.

The monitoring system 5 is configured to enable two-way communication between the hydraulic excavator 1 performing work at the work site and a management server 53 installed at a location remote from the work site or at the work site via a wide area network communication line 50. That is, the hydraulic excavator 1 and the management server 53 can transmit and receive information (data) via the communication line 50. The communication line 50 is a mobile phone communication network (mobile communication network) deployed by a mobile phone operator or the like, the Internet, or the like. For example, as shown in the figure, when the hydraulic excavator 1 and the wireless base station 51 are connected via a mobile phone communication network (mobile communication network), when the wireless base station 51 receives predetermined information from the hydraulic excavator 1, it transmits the received information to the management server 53 via the Internet.

The management server 53 receives vehicle body data such as operation data, warning data, and image data from the hydraulic excavator 1 and stores the data in a memory unit 54 such as a hard disk drive. The management server 53 displays the information (data) stored in the memory unit 54 on a display device 55 such as a liquid crystal display device. The administrator can operate the management server 53 using an input device 56 such as a keyboard or mouse to display specific information about the hydraulic excavator 1 on the display device 55, thereby understanding the status of the hydraulic excavator 1. For example, the management server 53 stores image data transmitted from the hydraulic excavator 1 in the memory unit 54, and displays an image of the surroundings of the hydraulic excavator 1 on the display screen of the display device 55 based on the stored image data.

The hydraulic excavator 1 comprises a machine body (vehicle body) 4 and a working device 10 attached to the machine body 4. The machine body 4 comprises a running body 2 and a rotating body 3 rotatably mounted on the running body 2, with the working device 10 attached to the front of the rotating body 3. The running body 2 comprises a left-side running hydraulic motor that drives the left crawler and a right-side running hydraulic motor that drives the right crawler. The running body 2 runs by driving a pair of left and right crawlers with the running hydraulic motors. The rotating body 3 rotates by driving a rotating hydraulic motor provided on the machine body 4.

The rotating body 3 has a rotating frame 8, a cab 7 provided on the front left side of the rotating frame 8, a counterweight 9 provided at the rear of the rotating frame 8, and an engine room 6 provided on the rotating frame 8 behind the cab 7. The engine room 6 houses an engine as a prime mover, and hydraulic equipment such as a hydraulic pump driven by the engine. A working device 10 is rotatably connected to the center of the front of the rotating frame 8.

The working device 10 is a multi-joint working device having multiple rotatably connected front members and multiple hydraulic cylinders (actuators) that drive the front members. In this embodiment, the three front members, a boom 11, an arm 12, and a bucket 13, are connected in series. The base end of the boom 11 is rotatably connected to the front of the rotating frame 8. The base end of the arm 12 is rotatably connected to the tip of the boom 11. The bucket 13 is rotatably connected to the tip of the arm 12.

The boom 11 is driven by a hydraulic cylinder (hereinafter also referred to as boom cylinder 11a) and rotates relative to the revolving frame 8. The arm 12 is driven by a hydraulic cylinder (hereinafter also referred to as arm cylinder 12a) and rotates relative to the boom 11. The bucket 13 is driven by a hydraulic cylinder (hereinafter also referred to as bucket cylinder 13a) and rotates relative to the arm 12. By driving each hydraulic cylinder of the working device 10, work such as excavating the ground and leveling the ground is performed.

The monitoring system 5 comprises a monitoring device 40 mounted on the hydraulic excavator 1, and a learning device 60 installed outside the hydraulic excavator 1, for example, at the work site where the hydraulic excavator 1 is operating. The monitoring system 5 is a system for monitoring the monitoring range (i.e., the range photographed by the photographing device 30) based on image data photographed by the photographing device 30. The monitoring system 5 is capable of monitoring the surroundings of the machine body 4 by attaching multiple photographing devices 30 to the rotating body 3.

As shown in FIG. 1, in this embodiment, a camera 30 that captures images of the rear of the rotating body 3 is attached to the top of the counterweight 9. The camera (rear camera) 30 continuously captures images of the area behind the rotating body 3 in a diagonal downward direction with a viewing angle of approximately 180° to the left and right. In addition to the rear camera, the hydraulic excavator 1 may also be equipped with a camera (left camera) that captures images of the left side of the rotating body 3, a camera (right camera) that captures images of the right side of the rotating body 3, and a camera (front camera) that captures images of the front side of the rotating body 3. The camera 30 is, for example, a wide-angle video camera equipped with a durable, weather-resistant imaging element such as a CCD or CMOS, and a wide-angle lens.

In the cab 7, there are provided an operating device for operating each part of the hydraulic excavator 1, and a display device 190 for displaying predetermined information. When the display device 190 determines that an object is present around the hydraulic excavator 1 (when an object is detected) through processing by the controller 100 described below, the display device 190 displays the information on a display screen to notify the operator of the hydraulic excavator 1 that an object has been detected. The display device 190 is a touch panel monitor having a touch sensor on a liquid crystal display. In other words, the display device 190 also functions as an input device 180 (see FIG. 2) that can input predetermined information to the controller 100 by performing a predetermined operation.

The first communication device 160 is a wireless communication device capable of wireless communication with a wireless base station 51 connected to a communication line 50, which is a wide area network, and has a communication interface including a communication antenna with a specific frequency band as its sensitive band. The first communication device 160 may also directly or indirectly send and receive information with the learning device 60 using a communication method such as Wi-Fi (registered trademark), ZigBee (registered trademark), or Bluetooth (registered trademark).

The controller 100 and the management server 53 are configured as a computer having a CPU (Central Processing Unit) as an operating circuit, a ROM (Read Only Memory) as a storage unit, a RAM (Random Access Memory) as a storage unit, an input/output interface (I/O interface), and other peripheral circuits. Note that the controller 100 and the management server 53 may each be configured as a single computer, or may each be configured as multiple computers.

The ROM of the controller 100 and the management server 53 is a non-volatile memory such as an EEPROM, and stores programs capable of executing various calculations and data such as thresholds. In other words, the ROM of the controller 100 and the management server 53 is a storage medium (storage device) capable of reading programs that realize the functions of this embodiment. The RAM is a volatile memory, and is a work memory that directly inputs and outputs data to and from the CPU. The RAM temporarily stores necessary data while the CPU is executing the program. The controller 100 and the management server 53 are equipped with non-volatile memory such as a flash memory or a hard disk drive as a storage unit.

The CPU is a processing device that loads a control program stored in ROM into RAM and executes calculations, and performs predetermined calculations on signals received from the input/output interface, ROM, and RAM in accordance with the control program. Signals from various devices are input to the input/output interface. The input/output interface converts the input signals so that they can be calculated by the CPU. The input/output interface generates output signals according to the results of calculations by the CPU, and outputs these signals to various devices.

Figure 2 is a functional block diagram showing the main functions of the work site monitoring system 5. As shown in Figure 2, the monitoring device 40 includes a camera device 30 attached to the rotating body 3 and taking pictures of the surroundings of the machine body 4, a display device 190 that displays images taken by the camera device 30 on a display screen, a first communication device 160 for wireless communication with a learning device 60 that is an external device installed outside the hydraulic excavator 1, a controller 100 that controls the display device 190 and the first communication device 160, and an input device 180 that inputs predetermined information to the controller 100.

The controller 100 functions as a display control device that displays image data captured by the image capture device 30 on the display device 190, a communication control device that transmits the image data to the outside via the first communication device 160, and a recognition processing device that recognizes objects in the image data captured by the image capture device 30. These are explained in detail below.

By executing a program stored in the ROM, the controller 100 functions as an object recognition unit 101, a display image generation unit 102, a memory retention control unit 103, and a communication control unit 104. The non-volatile memory of the controller 100 also functions as a model storage unit 131 and an image storage unit 132. The model storage unit 131 and the image storage unit 132 may be configured as a single non-volatile memory, or each may be configured as an individual non-volatile memory.

The model storage unit 131 stores a recognition model for recognizing objects. The recognition model is a YOLO (You Only Look Once) model, which is a type of convolutional neural network. The YOLO model is a recognition model created as a result of deep learning based on machine learning using a neural network method, and is stored in advance in the model storage unit 131.

The object recognition unit 101 acquires image data captured by the imaging device 30, and recognizes (estimates) the presence of an object within the monitoring range from the acquired image data through recognition processing using a single recognition model (YOLO model). The object recognition unit 101 determines whether or not an object exists within the monitoring range based on the reliability of the recognition processing result of the object recognized by the recognition processing. The object recognition unit 101 determines whether or not the reliability of the recognition processing result of the recognized object is equal to or higher than a first reliability as a result of the recognition processing on the image data. If the reliability of the recognition processing result of the recognized object is equal to or higher than the first reliability, the object recognition unit 101 determines that an object exists within the monitoring range. If the reliability of the recognition processing result of the recognized object is less than the first reliability, the object recognition unit 101 does not determine that an object exists within the monitoring range.

The reliability is a judgment output value obtained by the recognition process of the object recognition unit 101, and corresponds to the degree of match (degree of agreement) with the parameters of the object preset in the recognition model, and is 100% in the case of perfect agreement. The first reliability is a threshold value that is preset to determine the presence or absence of an object. The first reliability is set to a value of, for example, about 90%.

In this embodiment, the recognition model has a group of parameters related to a human head, and the object recognition unit 101 calculates the likelihood that it is estimated to be a human head as the reliability. In other words, the object recognition unit 101 executes a recognition process to recognize a human head from the acquired image data using the recognition model. The object recognition unit 101 generates information on the object whose existence is recognized by the recognition process, such as a "class representing the type of object (classification information of the recognition target)," "position coordinate information of the object (coordinates in the horizontal and vertical directions in the image)," and "size information of the object (lengths in the horizontal and vertical directions of a rectangular frame surrounding the object)."

Based on information about an object 192 that has been determined by the object recognition unit 101 to be present within the monitoring range (information about the object's position coordinates and size), the display image generation unit 102 synthesizes a frame image 193 surrounding the object 192 with the image captured by the imaging device 30 and displays the frame image 193 on the display screen of the display device 190 (see Figures 3 and 4).

Figures 3 and 4 show examples of display images displayed on the display screen of the display device 190. Figure 3 shows a display image captured of a worker working in a standing position near the image capture device 30 (i.e., near the rear of the hydraulic excavator 1), and Figure 4 shows a display image captured of a worker working in a crouched position at a certain distance from the image capture device 30.

3 and 4, the display image generating unit 102 displays the captured image 191 captured by the imaging device 30 on the display screen of the display device 190. When the object recognizing unit 101 determines that an object 192 exists within the monitoring range, the display image generating unit 102 generates a rectangular frame image 193 surrounding the object (human head) 192, synthesizes the frame image 193 with the captured image 191, and displays the result on the display screen of the display device 190. In this way, by displaying the rectangular frame image 193 representing the position and size of the object (human head) 192 superimposed on the captured image 191, the operator can easily grasp the position of the worker (person) present around the hydraulic excavator 1.

It is preferable that the display image generating unit 102 gradually changes the color of the frame image 193 according to the distance from the hydraulic excavator 1 to the worker. For example, the display image generating unit 102 displays the frame image 193 in red when the distance from the hydraulic excavator 1 to the worker is less than a first threshold, displays the frame image 193 in yellow when the distance from the hydraulic excavator 1 to the worker is equal to or greater than the first threshold and less than a second threshold, and displays the frame image 193 in blue when the distance from the hydraulic excavator 1 to the worker is equal to or greater than the second threshold. Note that the second threshold is a value greater than the first threshold. In this way, by gradually changing the frame image 193 according to the distance from the hydraulic excavator 1 to the worker, the operator can more easily grasp the position of the worker.

An example of a method for calculating the distance from the hydraulic excavator 1 to the worker is as follows. The storage unit of the controller 100 stores conversion parameters for converting the position coordinates of the image 191 captured by the image capture device 30 into position coordinates in the excavator reference coordinate system (e.g., a three-dimensional coordinate system with an origin at any point on the rotation center axis of the revolving body 3). The controller 100 converts (coordinate conversion) the coordinates of the center position of the lower side of the frame image 193 into position coordinates in the excavator reference coordinate system using the conversion parameters, and calculates the distance from the hydraulic excavator 1 (e.g., the distance from the origin of the excavator reference coordinate system). Since the frame image 193 represents the position and size of the person's head, the coordinates after the coordinate conversion may be corrected using correction parameters for estimating the position of the person's feet from the position of the person's head, and the distance from the hydraulic excavator 1 to the worker (person) may be calculated based on the corrected coordinates.

The memory retention control unit 103 determines whether the reliability of the recognition processing result of the object recognized by the object recognition unit 101 is equal to or higher than the second reliability. If the reliability of the recognition processing result of the object recognized by the object recognition unit 101 is equal to or higher than the second reliability, the memory retention control unit 103 causes the image data to be stored and retained in the image storage unit 132. If the reliability of the recognition processing result of the object recognized by the object recognition unit 101 is less than the second reliability, the memory retention control unit 103 does not cause the image data to be stored and retained in the image storage unit 132.

The controller 100 acquires video data consisting of 30 image data frames per second captured by the image capture device 30. Many of the multiple image data acquired in one second are similar image data. For this reason, in this embodiment, the memory retention control unit 103 thins out the image data acquired from the image capture device 30 and stores and retains the thinned image data in the image storage unit 132. The memory retention control unit 103 stores and retains image data from the acquired image data in the image storage unit 132 at predetermined time intervals t0 (e.g., 0.2 seconds). Note that the image data includes not only information about the captured image, but also information such as the identification number of the hydraulic excavator 1 and the date and time of capture.

The second reliability is a threshold for selecting image data to be used in re-training the recognition model (a threshold for determining whether image data requires re-training), and is set in advance to an arbitrary value lower than the first reliability. The second reliability is set to a value of, for example, about 50-70%. The second reliability can be determined by whether a person can determine whether an object is present in the selected image data and whether it is possible to efficiently create learning data. In other words, the second reliability can be said to be a threshold for determining whether a specified object is likely to exist. Note that if learning data cannot be efficiently created based on the selected image data, the value of the second reliability can be increased.

The communication control unit 104 determines whether or not communication with the learning device 60 is possible through the first communication device 160. When communication is possible, the communication control unit 104 transmits the image data stored in the image storage unit 132 to the learning device 60 through the first communication device 160. The learning device 60 receives image data from the monitoring device 40 through the second communication device 150. When the learning device 60 has successfully received the image data from the monitoring device 40, the learning device 60 transmits reception completion information indicating this to the monitoring device 40 of the hydraulic excavator 1 through the second communication device 150. When the monitoring device 40 receives the reception completion information, the monitoring device 40 registers the image data for which it has been determined that the transmission has been successfully completed as transmitted image data. The transmitted image data is periodically erased or made in a state in which it can be overwritten. The transmitted image data is not transmitted to the learning device 60.

The monitoring process executed by the controller 100 of the monitoring device 40 will be described with reference to FIG. 5. The process of the flowchart shown in FIG. 5 is started, for example, when the ignition switch of the hydraulic excavator 1 is turned on (i.e., the key is turned on), and after initial settings are made, it is repeatedly executed at a predetermined control period (e.g., 1/30 seconds).

In step S101, the controller 100 acquires data of an image captured by the image capture device 30 from the image capture device 30, and proceeds to step S104.

In step S104, the controller 100 performs recognition processing using a recognition model from the single image data acquired in step S101, calculates the reliability of the recognition processing result (in this embodiment, the estimated accuracy of a human head), and proceeds to step S107. In the recognition processing, the controller 100 also calculates position coordinate information and size information of an object estimated to be present within the monitoring range (a recognized object).

In step S107, the controller 100 determines whether the time t since the memory retention process S113 (also referred to as the previous memory retention process) in the calculation cycle that is prior to this calculation cycle and is closest to this calculation cycle has elapsed a predetermined time t0 (for example, 0.2 seconds). If it is determined in step S107 that the time t since the previous memory retention process S113 has been executed has elapsed a predetermined time t0, the process proceeds to step S110. If it is determined in step S107 that the time t since the previous memory retention process S113 has been executed has not elapsed a predetermined time t0, the process proceeds to step S131. If step S107 is executed for the first time, the process proceeds to step S110. The time t is measured by the timer function of the controller 100, and is reset (t=0) each time the memory retention process in step S113 described later is executed.

In step S110, the controller 100 determines whether the reliability calculated in step S104 is equal to or greater than the second reliability. If it is determined in step S110 that the reliability is equal to or greater than the second reliability, the process proceeds to step S113, and if it is determined that the reliability is less than the second reliability, the process proceeds to step S137.

In step S113, the controller 100 stores the image data acquired in step S101 in the image storage unit 132, and proceeds to step S131. In step S131, the controller 100 determines whether the reliability calculated in step S104 is equal to or greater than the first reliability. If it is determined in step S131 that the reliability is equal to or greater than the first reliability, the object is considered to exist and the process proceeds to step S134, and if it is determined that the reliability is less than the first reliability, the object is considered to not exist and the process proceeds to step S137.

In step S134, the controller 100 generates a frame image 193 based on the position coordinate information and size information of the object determined to be present within the monitoring range in step S104, and proceeds to step S137.

In step S137, the controller 100 generates a display image (captured image 191) based on the image data acquired in step S101, and displays it on the display screen of the display device 190. If the frame image generation process S134 is being executed in this calculation cycle, a composite image in which the display image (captured image 191) generated based on the image data acquired in step S101 is composited with the frame image 193 is displayed on the display screen of the display device 190.

When the display process of step S137 is completed, the process shown in the flowchart of FIG. 5 ends and proceeds to step S101 of the next calculation cycle.

The image data transmission process executed by the controller 100 of the monitoring device 40 will be described with reference to FIG. 6. The process of the flowchart shown in FIG. 6 is started, for example, when the ignition switch of the hydraulic excavator 1 is turned on (i.e., the key is turned on), and after initial settings are made, it is repeatedly executed at a predetermined control period.

In step S116, the controller 100 determines whether or not communication with the learning device 60 is possible via the first communication device 160. If it is determined in step S116 that communication is possible, the process proceeds to step S119, and if it is determined that communication is not possible, the process proceeds to step S128.

In step S128, the controller 100 registers image data stored in the image storage unit 132 other than the registered image data (image data registered as sent image data and image data registered as unsent image data) as unsent image data, and ends the process shown in the flowchart of FIG. 6.

In step S119, the controller 100 transmits the image data registered as unsent image data among the image data stored in the image storage unit 132 to the management server 53 via the first communication device 160, and proceeds to step S122.

In step S122, the controller 100 determines whether the transmission of the unsent image data has been completed successfully. In step S122, if the controller 100 acquires from the management server 53 reception completion information regarding the unsent image data transmitted in step S119, the controller 100 determines that the transmission of the unsent image data has been completed successfully, and proceeds to step S125. In step S122, if the controller 100 does not acquire from the management server 53 reception completion information regarding the unsent image data transmitted in step S119 within a predetermined time, the controller 100 determines that the transmission of the unsent image data has not been completed successfully, and proceeds to step S128.

In step S125, the controller 100 registers the image data stored in the image storage unit 132 that has been completely transmitted to the management server 53 as transmitted image data, and ends the process shown in the flowchart of FIG. 6.

As shown in FIG. 1, the learning device 60 includes a management server 53, a second communication device 150, a display device 55, and an input device 56. The second communication device 150 has the same configuration as the first communication device 160. As shown in FIG. 2, the learning device 60 functions as a control unit 57 and a memory unit 54. The memory unit 54 corresponds to a non-volatile memory such as a flash memory or a hard disk drive mounted on the management server 53. A recognition model is stored in the memory unit 54. The control unit 57 controls the display device 55 and the second communication device 150.

The control unit 57 acquires the image data stored in the image storage unit 132 of the monitoring device 40 of the hydraulic excavator 1 through reception processing by the second communication device 150, and stores the image data in the storage unit 54. The control unit 57 displays the acquired image data on the display screen of the display device 55 based on information input from the input device 56.

The control unit 57 performs learning based on the learning data related to the acquired image data to generate a new version of the recognition model and executes a process of generating update data for the recognition model (update data generation process). The update data for the recognition model is an update program for updating the recognition model stored in the model storage unit 131 of the hydraulic excavator 1 to a new version of the recognition model. Note that the update data for the recognition model is not limited to the update program, and may be the entire new version of the recognition model.

The learning data is composed of image data in which the administrator has visually confirmed that a specific object (e.g., a person) is present among the acquired image data, position coordinate information of the object in the image data (horizontal and vertical coordinates in the image), size information of the object (horizontal and vertical lengths of a rectangular frame surrounding the object), and a class (classification information) indicating the type of object. The learning data can be created by the administrator operating the input device 56 while looking at the image displayed on the display device 55, selecting image data in which an object is present, and performing a process (learning data creation process) in which the position coordinate information, size information, and class of the object in the image data are stored in the memory unit 54 in association with the selected image data.

As described above, the image data received by the learning device 60 has been selected by the monitoring device 40 of the hydraulic excavator 1 as having a high probability of indicating the presence of a person. This allows the administrator to efficiently create learning data based on a small amount of image data.

The control unit 57 of the learning device 60 performs machine learning using a well-known neural network method such as YOLOv3 based on the learning data to generate a new version of the recognition model and generate data for updating the recognition model.

The communication control unit 104 of the monitoring device 40 communicates with the learning device 60 to acquire information about the recognition model stored in the memory unit 54 of the management server 53. Based on the acquired information about the recognition model, the communication control unit 104 determines whether the learning device 60 has a newer version of the recognition model than the version of the recognition model stored in the model memory unit 131. In other words, the communication control unit 104 of the monitoring device 40 determines whether the recognition model stored in the memory unit 54 of the management server 53 includes a new parameter group that does not exist in the recognition model stored in the model memory unit 131.

If the learning device 60 has a newer version of the recognition model than the version of the recognition model stored in the model storage unit 131, the communication control unit 104 transmits a request command to the learning device 60 via the first communication unit 160 to obtain update data for the recognition model. When the control unit 57 of the learning device 60 receives the request command, it transmits the update data for the recognition model to the monitoring device 40 via the second communication unit 150.

The communication control unit 104 acquires data for updating the recognition model from the learning device 60 through reception processing by the first communication device 160. The communication control unit 104 performs an update process (upgrade) for updating the recognition model stored in the model storage unit 131 based on the acquired data for updating the recognition model. In other words, the controller 100 updates the parameter group constituting the recognition model stored in the model storage unit 131.

The contents of the recognition model update process executed by the controller 100 of the monitoring device 40 will be described with reference to FIG. 7. The process of the flowchart shown in FIG. 7 is started, for example, when the ignition switch of the hydraulic excavator 1 is turned on (i.e., the key is turned on), and is executed after initial settings are made. Note that the process of the flowchart shown in FIG. 7 is not limited to being executed when the ignition switch is turned on and the monitoring device 40 of the hydraulic excavator 1 is started, but may also be executed when the operator operates the input device 180 and a command to execute the update process is input from the input device 180 to the controller 100.

In step S180, the controller 100 determines whether or not communication with the learning device 60 is possible via the first communication device 160. If it is determined in step S180 that communication is possible, the process proceeds to step S185, and if it is determined that communication is not possible, the process shown in the flowchart of FIG. 7 ends.

In step S185, the controller 100 acquires version information of the recognition model from the learning device 60, and determines whether the version of the recognition model of the learning device 60 is newer than the version of the recognition model stored in the model storage unit 131 of the monitoring device 40. If it is determined in step S185 that the version of the recognition model of the learning device 60 is newer than the version of the recognition model stored in the model storage unit 131 of the monitoring device 40, the process proceeds to step S190. If it is determined in step S185 that the version of the recognition model of the learning device 60 is not newer than the version of the recognition model stored in the model storage unit 131 of the monitoring device 40, the process shown in the flowchart of FIG. 7 is terminated.

In step S190, the controller 100 acquires data for updating the recognition model from the learning device 60 via the first communication device 160, and proceeds to step S195.

In step S195, the controller 100 executes an update process to update the currently used recognition model (current model) with the recognition model update data acquired in step S190, and ends the process shown in the flowchart of FIG. 7.

As described above, the monitoring method according to this embodiment includes an acquisition process (S101) for acquiring image data captured by the imaging device 30, and a recognition process (S104) for recognizing an object present within the monitoring range (shooting range) from the acquired image data using a recognition model stored in the model storage unit 131. This monitoring method includes a presence determination process (S131) for determining that an object exists within the monitoring range (shooting range) when the reliability of the recognition process result of the object recognized in the image data is equal to or higher than a first reliability, a storage and retention process (Y in S110 to S113) for storing and retaining the image data in the image storage unit 132 when the reliability of the recognition process result of the object recognized in the image data is equal to or higher than a second reliability lower than the first reliability, a learning data creation process for creating learning data based on the image data stored and retained in the image storage unit 132, an update data generation process for generating update data for the recognition model based on the learning data, and a model update process (S195) for updating the recognition model stored in the model storage unit 131 based on the update data. This monitoring method makes it easy to create learning data, and by appropriately updating the recognition model, it is possible to improve the accuracy of object recognition.

The above-described embodiment provides the following advantages:

(1) The monitoring system 5 for the work site includes a monitoring device 40 having an imaging device 30 that captures images of the surroundings of the machine body (vehicle body) 4, a controller (recognition processing device) 100 that acquires image data captured by the imaging device 30 and recognizes the presence of an object within a monitoring range based on the acquired image data, and a learning device 60 that performs learning based on the image data acquired by the controller (recognition processing device) 100. The monitoring device 40 includes a model storage unit 131 that stores a recognition model for recognizing an object (e.g., a human head), and an image storage unit 132 that stores and holds predetermined image data. When the reliability of the recognition processing result of the object recognized by the controller 100 is equal to or higher than the first reliability, the monitoring device 40 determines that an object exists within the monitoring range. When the reliability of the recognition processing result of the object recognized by the controller 100 is equal to or higher than the second reliability that is lower than the first reliability, the monitoring device 40 stores and holds the image data that was the subject of the recognition processing in the image storage unit 132. The learning device 60 generates update data for the recognition model stored in the model storage unit 131 by performing learning based on the learning data related to image data stored and held in the image storage unit 132. The monitoring device 40 updates the recognition model stored in the model storage unit 131 based on the update data generated by the learning device 60.

In this configuration, image data required for creating training data is extracted (selected) and stored, so that training data can be created with a small amount of image data. This allows training data to be created efficiently, and the recognition model can be updated in a short period of time. In other words, according to this embodiment, training data can be easily created, and the recognition model can be appropriately updated, thereby improving the accuracy of object recognition (detection).

(2) The monitoring device 40 is mounted on the hydraulic excavator (work machine) 1. The monitoring device 40 includes a first communication device 160 that transmits image data stored in the image memory unit 132 to the learning device 60 and receives data for updating the recognition model from the learning device 60. The learning device 60 is provided outside the hydraulic excavator 1. The learning device 60 includes a second communication device 150 that receives image data from the monitoring device 40 and transmits data for updating the recognition model to the monitoring device 40. The monitoring device 40 updates the recognition model stored in the model memory unit 131 based on the data for updating the recognition model received by the first communication device 160.

In this configuration, the first communication device 160 and the second communication device 150 can quickly provide image data used to create learning data from the monitoring device 40 to the learning device 60, and can quickly provide data for updating the recognition model from the learning device 60 to the monitoring device 40. Therefore, according to this embodiment, the recognition model can be updated at a more appropriate time than when data is exchanged between the monitoring device 40 and the learning device 60 using a storage medium or the like, thereby improving work efficiency.

(3) The monitoring device 40 recognizes the presence of an object within the monitoring range from the acquired image data through recognition processing using a single recognition model. This improves the accuracy of the recognition processing compared to a case where multiple recognition models are used to perform multiple stages of recognition processing.

(4) The monitoring device 40 executes a recognition process that uses a recognition model to recognize parts of a person's body, including the head, from the acquired image data. This allows the position of a person within the monitoring range to be properly grasped.

The following modified examples are also within the scope of the present invention, and it is possible to combine the configurations shown in the modified examples with the configurations described in the above embodiments, or to combine the configurations described in the different modified examples below.

<Modification 1>
In the above embodiment, an example has been described in which the controller 100 executes a process for recognizing the head of a person (worker) and displays a frame image 193 representing the position and size of the head of a person (detected person's head) determined to be present within the monitoring range on the display screen of the display device 190, but the present invention is not limited to this. In an image captured by the imaging device 30, the center of the bottom edge of the image is closest to the imaging device 30, and distances are added radially from the center of the bottom edge of the image. For this reason, when estimating the distance to a person (worker) who is a three-dimensional object, it is preferable that the controller 100 calculates the position coordinates in the excavator reference coordinate system based on the position coordinates of the feet of the detected person (worker) and calculates the distance from the hydraulic excavator 1 to the worker.

The monitoring device 40 according to this first modification executes a recognition process using a recognition model from the acquired image data to recognize the position and size of three classes (types) of a person's head, upper body including the head, and whole body including the upper body as parts of the person's body including the head. In other words, in this modification, the recognition model stored in the model storage unit 131 includes parameter sets for the three classes of head, upper body, and whole body.

As shown in FIG. 8, the controller 100 displays rectangular frame images 193A, 193B, 193C representing the position and size of the person's head, upper body, and entire body on the display screen of the display device 190. When the person's head, upper body, and entire body are detected, it is also possible to display only the largest frame image, that is, frame image 193C representing the position and size of the entire body, without displaying all of the frame images 193A, 193B, 193C representing their positions and sizes. The controller 100 calculates the distance from the hydraulic excavator 1 to the person based on the coordinates of the center position of the bottom end of frame image 193C.

When a person (worker) is crouching down to work, the controller 100 causes the display screen of the display device 190 to display rectangular frame images 193A, 193B representing the position and size of the person's head and upper body, as shown in FIG. 9. When the person's head and upper body are detected, it is also possible to display only the largest frame image, that is, frame image 193B representing the position and size of the upper body, without displaying all of the frame images 193A, 193B representing their positions and sizes. In this case, the controller 100 calculates the distance from the hydraulic excavator 1 to the person based on the coordinates of the center position of the bottom end of frame image 193B.

The configuration of this modified example 1 allows the position of the worker (person) to be grasped more accurately, thereby improving the work efficiency of the hydraulic excavator 1.

<Modification 2>
In the above embodiment, an example has been described in which the monitoring device 40 and the learning device 60 transmit and receive data via the communication devices 150 and 160, but the present invention is not limited to this. Data may be exchanged between the monitoring device 40 and the learning device 60 via a portable storage medium such as a memory card or USB memory, or an information terminal such as a tablet PC that has a built-in storage medium.

In this case, for example, the monitoring device 40 accumulates image data in the image storage unit 132 and moves the image data accumulated in the storage medium. The administrator uses the storage medium to store image data in the storage unit 54 of the learning device 60 and create learning data. The learning device 60 re-learns the recognition model based on the learning data, generates data for updating the recognition model, and stores the data in the storage medium. The operator uses the storage medium to input the data for updating the recognition model to the monitoring device 40. The monitoring device 40 updates the recognition model stored in the model storage unit 131 based on the input data for updating the recognition model. Therefore, the monitoring system 5 according to this modification 2 can omit the communication devices 150 and 160. Furthermore, according to this modification 2, there is no need to communicate via the communication devices 150 and 160, so there is no communication cost for re-learning the recognition model and updating the recognition model.

<Modification 3>
In the above embodiment, an example has been described in which image data stored in the image storage unit 132 is transmitted to the learning device 60 by the process shown in the flowchart of FIG. 6, but the present invention is not limited thereto. For example, untransmitted image data stored in the image storage unit 132 may be transmitted to the learning device 60 based on an operation on the input device 180. When an operator performs an operation to transmit data using the input device 180, a data transmission command is input from the input device 180 to the controller 100. As shown in FIG. 2, when the data transmission command is input, the communication control unit 104 controls the first communication device 160 to transmit the untransmitted image data stored in the image storage unit 132 to the management server 53.

The controller 100 may determine whether the operating time of the hydraulic excavator 1 has exceeded a predetermined threshold, and if the operating time has exceeded the threshold, may automatically transmit unsent image data stored in the image storage unit 132 to the management server 53. The controller 100 may determine whether the amount of accumulated unsent image data has exceeded a predetermined threshold, and if the amount of unsent image data has exceeded the threshold, may automatically transmit unsent image data stored in the image storage unit 132 to the management server 53.

<Modification 4>
In the above embodiment, an example has been described in which a rectangular frame image 193 representing the position and size of an object 192 is displayed on the display screen of the display device 190, but the present invention is not limited to this. The frame image 193 is not limited to a rectangular shape, and may be a polygonal shape such as a circle, an ellipse, or a pentagon, or may be any shape made by connecting curves and straight lines, or may be a shape that follows the shape of the object. The frame image 193 may also be displayed in a transparent color with an arbitrary alpha value set. The frame image 193 may not always be displayed when an object is detected, but may be displayed (blinked) at an arbitrary cycle.

<Modification 5>
In the above embodiment, an example has been described in which, when it is determined that an object exists in an image captured by the image capture device 30, the frame image 193 is composited with the captured image 191 and displayed, but the method for notifying the operator that it has been determined that an object exists is not limited to this. For example, when it is determined that an object exists in an image captured by the image capture device 30, the controller 100 may display an icon indicating the existence of an object (e.g., a worker) outside the captured image 191.

<Modification 6>
In the above embodiment, an example in which the learning device 60 is provided in the management center 52 outside the hydraulic excavator 1 has been described, but the present invention is not limited to this. The monitoring device 40 mounted on the hydraulic excavator 1 may have the functions of the learning device 60 described in the above embodiment. In this case, data exchange by the communication devices 150 and 160 described in the above embodiment and data exchange by the storage medium described in Modification 2 become unnecessary. Note that when the monitoring device 40 and the learning device 60 are mounted on the hydraulic excavator 1, the recognition model is re-learned in the controller 100, so a computer with high processing power (high performance) is required. On the other hand, by providing the learning device 60 outside the hydraulic excavator 1 as described in the above embodiment, the cost of the controller 100 mounted on the hydraulic excavator 1 can be reduced.

<Modification 7>
In the above embodiment, an example has been described in which the image data stored in the monitoring device 40 mounted on the hydraulic excavator 1 is transmitted to the learning device 60 in the management center 52 by the first communication device 160, but the present invention is not limited to this. As shown in FIG. 1, the monitoring device 40 may transmit image data to a data server 58 installed in a location separate from the hydraulic excavator 1 and the management center 52 by the first communication device 160. The data server 58 stores the received image data in the storage device 59. In this case, learning data is created using the image data stored in the storage device 59 of the data server 58, and the learning data is stored in the storage device 59 of the data server 58. The learning device 60 receives the learning data from the data server 58 by the second communication device 150. The learning device 60 generates update data for the recognition model based on the learning data. According to this seventh modification, the capacity of the storage unit 54 of the learning device 60 can be made smaller than that of the above embodiment.

<Modification 8>
In the above embodiment, an example has been described in which data of an image captured by the image capture device 30 is stored and held in the image storage unit 132 when a predetermined condition is met, but the present invention is not limited to this. As shown in Fig. 1, image data captured by an image capture device 33 attached to the tip of a support 34 installed at a work site where the hydraulic excavator 1 is operating may be stored and held in the image storage unit 132 so as to have the same height and mounting angle as the image capture device 30 attached to the hydraulic excavator 1.

In this case, the monitoring system has a photographing device 33 and a monitoring simulator 240 connected to the photographing device 33. The monitoring simulator 240 has the same configuration and functions as the monitoring device 40, so detailed description thereof will be omitted. In this modified example, similar to the above embodiment, the monitoring simulator 240 acquires image data photographed by the photographing device 33, and executes recognition processing to recognize the presence of an object within the monitoring range (photographing range) from the acquired image data. When the reliability of the recognition processing result of the object recognized in the image data is equal to or higher than the second reliability, the monitoring simulator 240 extracts the image data that was the subject of the recognition processing and stores and holds it in the image storage unit of the monitoring simulator 240.

In other words, in this modification 8, the imaging range of the imaging device 33 is simulated as the imaging range of the imaging device 30 of the hydraulic excavator 1, and imaging is performed by the imaging device 33, and image data that can be used to efficiently create learning data can be selected from the captured image data. Therefore, according to this modification 8, it is possible to obtain the image data necessary to create learning data without using the hydraulic excavator 1.

<Modification 9>
In the above embodiment, an example has been described in which the monitoring device 40 executes a process for recognizing a person present within the monitoring range, but the present invention is not limited to this. The monitoring device 40 may execute a process for recognizing obstacles such as a work vehicle, a sign, a rock, etc. as objects present within the monitoring range.

<Modification 10>
In the above embodiment, the working machine is a crawler type hydraulic excavator 1, but the present invention is not limited to this. The present invention can be applied to various working machines such as wheeled hydraulic excavators, wheel loaders, dump trucks, and cranes.

<Modification 11>
In the above embodiment, the YOLO model is used as an example of the recognition model, but the recognition model is not limited to this. Various convolutional neural network (CNN) models for object recognition, such as a single shot multibox detector (SSD), may be used.

Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

1... hydraulic excavator (working machine), 4... machine body (vehicle body), 5... monitoring system, 10... working device, 30... imaging device, 33... imaging device, 40... monitoring device, 52... management center, 53... management server, 54... storage unit, 55... display device, 56... input device, 60... learning device, 100... controller (recognition processing device), 101... object recognition unit, 102... display image generation unit, 103... memory retention control unit, 104... communication control unit, 131... model storage unit, 132... image storage unit, 150... second communication device, 160... first communication device, 180... input device, 190... display device, 240... monitoring simulator (monitoring device)

Claims (4)

A monitoring system for a work site, comprising: a monitoring device having a memory unit , which executes a recognition process for recognizing the presence or absence of an object within a monitoring range using image data captured by an imaging device that captures images of the surroundings of a work machine and a recognition model stored in the memory unit; and a learning device that generates update data for updating the recognition model,
The monitoring device includes:
When the reliability of the recognition processing result of the object recognized by the recognition processing is equal to or greater than a predetermined threshold , the image data used in the recognition processing is stored and held in the storage unit;
The learning device includes:
The update data is generated by performing learning using learning data including image data selected by operating an input device from the image data stored and held in the storage unit and answer data created based on the image data .
A work site monitoring system comprising: 2. The work site monitoring system according to claim 1,
The image data stored and held in the storage unit by the monitoring device includes image data in which it is determined by the recognition process that no object is present.
A work site monitoring system comprising: 2. The work site monitoring system according to claim 1,
The monitoring device is mounted on a work machine ,
the learning device is provided outside the work machine ,
The monitoring device and the learning device transmit and receive data to each other via a communication device.
A work site monitoring system comprising: 4. The work site monitoring system according to claim 3,
The monitoring device transmits the image data stored in the storage unit to the learning device, and periodically deletes image data that has been completely transmitted to the learning device among the image data stored in the storage unit.
A work site monitoring system comprising:

Images