JP7758520B2 - OBJECT RECOGNITION DEVICE, ENVIRONMENTAL DATA GENERATION DEVICE, MODEL GENERATION DEVICE, OBJECT RECOGNITION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an object recognition device, an environmental data generation device, a model generation device, an object recognition method, and a program.

In recent years, efforts have been made to develop autonomous construction robots at construction sites, with the aim of reducing the number of workers required. To achieve autonomous construction by robots, it is necessary for them to correctly recognize the environment at the construction site. However, at construction sites, the environmental conditions change as work progresses, such as when excavated soil is piled up, making it difficult to prepare a static map in advance and have the robot recognize the environment. In particular, when using autonomous bulldozers, it is necessary to recognize the relative distance to objects in order to avoid contact with other construction machinery, drainage pipes, etc. Additionally, it is necessary to recognize the shape and relative distance of the embankment in order to determine where to level the embankment and to control the leveling process.

In the past, to recognize the environment at a construction site, images captured by a camera, for example, were used to recognize the environment (see, for example, Patent Document 1). However, while recognition using camera images can recognize objects and embankments, it is difficult to accurately measure the distance to the target object or the size of that object. Furthermore, the recognition accuracy when using camera images is heavily dependent on external factors such as the weather.

JP 2019-044480 A

To address these issues, one method for accurately measuring the size and relative distance of an object is to develop artificial intelligence (AI) and recognize objects using point cloud data measured by 3D-LiDAR (3D-Light Detection and Ranging, hereafter referred to as LiDAR). Using point cloud data for object recognition makes it possible to calculate the size, shape, and relative distance of a recognized object from the set of recognized points. This makes it possible to use the size, shape, and relative distance of an object to determine control values for autonomous operation of construction machinery.

To build AI that can recognize point cloud data, it is necessary to prepare a large amount of point cloud data as training data, with each point annotated with construction equipment, obstacles, etc. However, there is no large amount of point cloud data annotated with construction equipment, obstacles, etc., and annotating large amounts of point cloud data, for example, requires a great deal of effort. This makes it difficult to build AI and improve the accuracy of object recognition.

The present invention was developed based on the above-mentioned problem recognition, and aims to provide an object recognition device, environmental data generation device, model generation device, object recognition method, and program that can improve the accuracy of object recognition at construction sites.

The object recognition device, the environment data generation device, the model generation device, the object recognition method, and the program according to the present invention employ the following configuration.
(1): An object recognition device according to one aspect of the present invention is an object recognition device including: a model acquisition unit that acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work; and a recognition unit that recognizes objects present in the construction environment using the acquired model.

(2): In the aspect (1) above, the point cloud data generation device includes a simulator that simulates the construction environment, and the simulator generates the point cloud data based on the object.

(3): In the above aspect (2), the data set includes, as input data for the model, construction environment data relating to the object tagged with tag data in the simulation of the simulator, and includes, as output data for the model, annotation data assigned to each point of the point cloud data based on the construction environment data.

(4): Furthermore, an environmental data generation device according to one aspect of the present invention is an environmental data generation device that includes an object recognition device according to any one of (1) to (3) above, and an environmental data generation unit that generates environmental data representing the state of the construction environment based on the object recognized by the object recognition device.

(5): Furthermore, a model generation device according to one aspect of the present invention includes a learning unit that generates a model by learning using, as a dataset, construction environment data of a construction environment for civil engineering work and point cloud data generated by a point cloud data generation device based on the construction environment data, wherein the dataset includes, as input data for the model, construction environment data relating to objects tagged with tag data in a simulation of a simulator that virtualizes the construction environment, and includes, as output data for the model, annotation data assigned to each point of the point cloud data based on the construction environment data.

(6): Another aspect of the present invention is an object recognition method in which a computer of an object recognition device acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment for civil engineering work, and recognizes objects present in the construction environment using the acquired model.

(7): Another aspect of the present invention provides a program that causes a computer of an object recognition device to acquire a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment for civil engineering work, and recognizes objects present in the construction environment using the acquired model.

According to the above-mentioned aspects (1) to (7), the accuracy of object recognition at construction sites can be improved.

1 is a configuration diagram of an environmental data generation system 1 according to an embodiment. FIG. 10 is a diagram showing an image of how the environmental data generating device 100 recognizes an embankment. 10 is a flowchart showing an example of processing in the model generating device 200. 10 is a flowchart illustrating an example of processing in the environmental data generating device 100. FIG. 2 is a diagram illustrating an example of a hardware configuration of an environmental data generating apparatus.

Embodiments of the object recognition device, environmental data generation device, model generation device, object recognition method, and program of the present invention will be described below with reference to the drawings.

[Overall configuration]
1 is a configuration diagram of an environmental data generation system 1 according to an embodiment. The environmental data generation system 1 includes, for example, an environmental data generation device 100 and a model generation device 200. The environmental data generation device 100 is mounted on, for example, a construction machine 10. The construction machine 10 and the model generation device 200 are capable of communicating with each other via a communication network NW.

The construction machine 10 on which the environmental data generating device 100 is mounted is, for example, a bulldozer, hydraulic excavator, or vibratory roller. In the following explanation, we will assume that the construction machine 10 on which the environmental data generating device 100 is mounted is a bulldozer that levels piled up soil (embankment) at a construction site.

The construction machine 10 includes, for example, a driving operator 11, a driving control device 12, a traveling driving force output device 13, a construction driving force output device 14, a LiDAR 15, a camera 16, a communication device 17, a positioning device 18, and an environmental data generation device 100. The configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other components may be added.

For example, the positioning device 18 may be omitted from the construction machine 10 shown in Figure 1. Alternatively, sensors such as a speed sensor that detects the speed of the construction machine 10, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, and a direction sensor that detects the orientation of the construction machine 10 may be added.

The driving operators 11 are operators that are used by workers at the construction site to manually control the operation of the civil engineering and construction machine 10. The driving operators 11 include multiple operators for operating the civil engineering and construction machine 10, such as operators that are operated when traveling (moving) the civil engineering and construction machine 10 and operators that are operated when performing construction work using the construction equipment equipped on the civil engineering and construction machine 10.

Operators include, for example, control levers, joysticks, hand grips, and operation buttons. The driving operator 11 is fitted with a sensor that detects the amount of operation of the operator or whether or not it is being operated, and the detection results are output to the driving control device 12 or some or all of the traveling driving force output device 13 and construction driving force output device 14.

When the construction machine 10 is manually operated by a worker at the construction site, the operation control device 12 controls the traveling drive force output device 13 and the construction drive force output device 14 in accordance with the detection results output by the sensor attached to the driving operator 11. On the other hand, when the construction machine 10 is an autonomous construction machine and performs construction operations autonomously, the operation control device 12 controls the traveling drive force output device 13 and the construction drive force output device 14 in accordance with the environmental data and control values output by the environmental data generation device 100.

The driving force output device 13 outputs driving force (torque) to, for example, crawlers or wheels, for the civil engineering/construction machine 10 to travel within a construction site. The driving force output device 13 includes, for example, an internal combustion engine and a control device. The control device controls the internal combustion engine in accordance with information input from the operation control device 12 or information input from the driving operator 11.

When the civil engineering and construction machine 10 performs construction work at a construction site, the construction driving force output device 14 outputs the driving force (torque) of the internal combustion engine as driving force to generate hydraulic pressure in the cylinders of hydraulic devices connected to work equipment such as blades, arms, and baguettes. The construction driving force output device 14 outputs driving force during construction in accordance with information input from the operation control device 12 or information input from the operation operator 11.

At the construction site where construction is being carried out by the civil engineering and construction machine 10, the LiDAR 15 irradiates light into the space of the construction environment, which is a predetermined range in front of the civil engineering and construction machine 10, and receives the reflected light (scattered light) that is reflected by objects. The LiDAR 15 measures the distance to objects in the construction environment based on the time between irradiating the light and receiving the reflected light. The LiDAR 15 irradiates light to multiple measurement points within the construction environment, i.e., the measurement range for measuring the distance to objects, and measures the distance to the objects for each measurement point. In this way, the LiDAR 15 measures the state of the construction environment.

The LiDAR 15 measures the distance to the object at each measurement point, for example, periodically and repeatedly. The light emitted by the LiDAR 15 is, for example, pulsed laser light. The LiDAR 15 outputs point cloud data of the construction environment representing the distance to the object measured at each measurement point to the environmental data generating device 100. The LiDAR 15 can be attached to any location on the civil engineering and construction machine 10. For example, the LiDAR 15 can be attached to the roof of the operator's cabin where the workers are seated.

Camera 16 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 16 captures images of the construction environment (or an area wider than the construction environment) periodically, for example. Camera 16 measures the state of the construction environment by capturing images of the construction environment.

The camera 16 may be a mono camera or a stereo camera. The camera 16 outputs captured image data to the environmental data generating device 100. The camera 16 is mounted, for example, in the same location as the LiDAR 15 or at the front of the ceiling inside the cockpit. It may also be mounted at any other location on the construction machine 10.

The communication device 17 communicates, for example, via a cellular network or Wi-Fi network, with a management device used by an administrator to determine a construction plan, a server device that manages the construction plan, and the like. The communication device 17 receives, for example, information transmitted by the model generation device 200. The communication device 17 outputs the received information to the environmental data generation device 100. The communication device 17 may be provided in the environmental data generation device 100.

The positioning device 18 measures the position of the construction machine 10 based on signals received from GPS (Global Positioning System) satellites, for example. The positioning device 18 outputs information indicating the measured position of the construction machine 10 to the environmental data generating device 100. The environmental data generating device 100 references the position information of the construction machine 10 output by the positioning device 18, generates control values for autonomously operating the construction machine 10, and outputs them to the construction machine 10. The positioning device 18 may measure the position of the construction machine 10 in response to a request from the control values generated by the environmental data generating device 100.

The environmental data generating device 100 mounted on the construction machine 10 recognizes surrounding objects, such as embankments. Figure 2 is a diagram illustrating how the environmental data generating device 100 recognizes embankments. The environmental data generating device 100 acquires point cloud data output by the LiDAR 15 and image data output by the camera 16. The environmental data generating device 100 inputs the acquired point cloud data into a trained model and recognizes objects such as embankment M based on the output results, as well as the distance to and height of the embankment M. The trained model is one example of a model. The environmental data generating device 100 may input image data along with the point cloud data into the trained model and recognize objects based on the output results.

The environmental data generating device 100 includes, for example, an object recognition unit 110, an environmental data generation unit 120, and a control value generation unit 130. The object recognition unit 110, the environmental data generation unit 120, and the control value generation unit 130 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the environmental data generating device 100, or may be stored on a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the environmental data generating device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

The object recognition unit 110 includes, for example, an acquisition unit 111 and a recognition unit 112. The acquisition unit 111 includes, for example, a model acquisition unit 111A, a point cloud data acquisition unit 111B, and an image data acquisition unit 111C. The model acquisition unit 111A acquires the trained model transmitted by the model generation device 200 and received by the communication device 17. The object recognition unit 110 is an example of an object recognition device. In the embodiment, the object recognition device is included in the environment data generation device 100 as the object recognition unit 110, but the object recognition device may also be provided independently from the environment data generation device 100.

The point cloud data acquisition unit 111B acquires the point cloud data output by the LiDAR 15. The point cloud data acquisition unit 111B outputs the acquired point cloud data to the recognition unit 112. The image data acquisition unit 111C acquires image data output by the camera 16. The image data acquisition unit 111C outputs the acquired image data to the recognition unit 112. The point cloud data acquisition unit 111B and the image data acquisition unit 111C sequentially acquire the point cloud data periodically output by the LiDAR 15 and the image data periodically output by the camera 16, and output them to the recognition unit 112. The point cloud data acquisition unit 111B and the image data acquisition unit 111C may store the acquired point cloud data and image data in memory (not shown). In this case, the recognition unit 112 performs the following functions based on the point cloud data and image data stored in memory.

The recognition unit 112 inputs the point cloud data acquired by the point cloud data acquisition unit 111B as input data into the trained model acquired by the model acquisition unit 111A. The recognition unit 112 performs object recognition based on the output results of the trained model to which the point cloud data has been input. Through object recognition, the recognition unit 112 recognizes objects that exist in the construction environment, such as construction targets for embankments and other construction work, other construction machinery, and materials.

The recognition unit 112 performs object recognition based on the image data acquired by the image data acquisition unit 111C. Object recognition based on image data is performed, for example, by inputting the image data into a second trained model. The recognition unit 112 may also perform object recognition based on image data by other means, such as image analysis using a template. The recognition unit 112 performs object recognition based on image data based on the output result of the second trained model. The recognition unit 112 corrects the result of object recognition based on the output result of the trained model using the result of object recognition based on image data. The recognition unit 112 outputs object information, in which at least information about the objects recognized in the corrected recognition result is linked to point cloud data, to the environmental data generation unit 120. The object information includes, for example, information about the relative distance to the object measured by LiDAR 15, as well as information about the attributes, size, height, shape, etc. of the object recognized by the recognition unit 112. The attributes of an object are information that indicates, for example, whether the object is an embankment or a change in soil other than an embankment, i.e., a change in the topography of the construction site.

The object information may include information representing the relative distance to other objects such as construction machinery and materials recognized by the recognition unit 112, as well as information representing the attributes of the object, such as the type of other construction machinery, e.g., a dump truck, or the type of material, e.g., piping. Each time point cloud data is output from the point cloud data acquisition unit 111B, the recognition unit 112 executes the object recognition function described above, sequentially updates the object information, and outputs it to the environmental data generation unit 120. The recognition unit 112 may recognize objects in the construction environment using the point cloud data acquired by the point cloud data acquisition unit 111B and the image data acquired by the image data acquisition unit 111C as input data.

The environmental data generation unit 120 generates environmental data representing the state of the construction environment based on the object information output by the recognition unit 112 included in the object recognition unit 110. The environmental data is, for example, a map of the construction site that includes the distance between the civil engineering construction machine 10 and the object being constructed, i.e., information on the location within the construction site where the object is located, and information on the locations of other construction machinery, materials, and other objects within the construction site that are not the target of construction. The environmental data generation unit 120 outputs the generated environmental data to the control value generation unit 130. Each time the object information is updated by the recognition unit 112, the environmental data generation unit 120 sequentially updates the environmental data and outputs it to the control value generation unit 130.

Based on the environmental data output by the environmental data generation unit 120, the control value generation unit 130 generates control values for controlling the operation of the civil engineering and construction machine 10 when the civil engineering and construction machine 10 autonomously performs construction operations. The control values are control information that represent, for example, instructions for causing the civil engineering and construction machine 10 to travel (move) within a construction site or instructions for causing the civil engineering and construction machine 10 to perform construction work at a target location. The control value generation unit 130 outputs the generated control values to the operation control device 12. The operation control device 12 controls the traveling driving force output device 13 and the construction driving force output device 14 in accordance with the control values input from the control value generation unit 130.

The model generation device 200 includes, for example, a point cloud data generation unit 210 and a learning unit 220. The point cloud data generation unit 210 includes, for example, an input interface 211 and a simulation unit 212. The point cloud data generation unit 210 generates a dataset including point cloud data used for learning in the learning unit 220. The point cloud data generation unit 210 is an example of a point cloud data generation device.

The input interface 211 outputs data provided based on input operations by workers, etc. to the simulation unit. Examples of the input interface 211 include a mouse, keyboard, touch panel, drag ball, switch, button, joystick, camera, infrared sensor, microphone, etc.

The simulation unit 212 executes a simulation based on the reproduction data of the construction site provided based on the input operation of the input interface 211. By executing the simulation, the simulation unit 212 generates construction environment data that reproduces a virtual construction site based on the reproduction data of the construction site.

Examples of reconstruction data of a construction site include 3D data created using BIM (Building Information Modeling)/CIM (Construction Information Modeling) and 3D point cloud map data measured using a drone or similar. Instead of a virtual construction site, the construction environment data may also be a construction site based on construction environment data generated by the environmental data generating device 100.

In the construction site simulation, there are multiple objects, such as embankments and civil engineering machinery. Each object in the simulation is tagged with attribute information. The construction environment data includes the attribute information tagged to the objects.

The simulation unit 212 performs a simulation of measuring the recreated construction site using 3D LiDAR based on the generated construction environment data, and generates point cloud data as a result of the simulation. The simulation unit 212 adds annotations to the generated point cloud data based on attribute information tagged for each of the multiple objects. The attribute information is an example of tag data.

The simulation unit 212 recreates a large number of construction sites, assuming a variety of situations. The simulation unit 212 recreates a large number of construction sites, and for each of these construction sites, performs a simulation similar to the above-mentioned simulation using construction environment data, and generates annotated point cloud data.

The simulation unit 212 generates a large amount of annotated point cloud data as a result of executing the simulation. The point cloud data generation unit 210 outputs to the learning unit 220 a dataset including a large amount of construction environment data in which objects generated by the simulation unit 212 are tagged with attribute information and a large amount of annotated point cloud data. The simulation unit 212 is an example of a simulator. The simulator (simulation unit) may be provided independently of the point cloud data generation unit 210 and the model generation device 200.

The learning unit 220 performs machine learning by learning the large amount of construction environment data and point cloud data generated by the simulation unit 212 as a data set. The learning unit 220 performs machine learning, for example, using construction environment data tagged with attribute information as input data and point cloud data with annotations as output data. The learning unit 220 generates a trained model as a result of the machine learning. The learning unit 220 transmits the generated trained model to the environmental data generation device 100.

A trained model is a model used for object recognition in the environmental data generating device 100. A trained model is a model for object recognition by AI (Artificial Intelligence), generated through learning such as deep learning or pattern matching. A trained model may be generated through learning using a support vector machine (SVM), decision tree, k-nn (k-nearest neighbor) classifier, or the like. Learning may also be rule-based.

Next, we will explain the processing in the environmental data generation system 1. The environmental data generation system 1 generates environmental data when carrying out construction at a construction site, but before generating the environmental data, the model generation device 200 generates a trained model for generating the environmental data. Therefore, we will explain the generation of the trained model, and then explain the generation of the environmental data.

Figure 3 is a flowchart showing an example of processing in the model generation device 200. The processing shown in Figure 3 is initiated when starting to generate a trained model. To generate a trained model, the model generation device 200 first generates construction environment data that reproduces a construction site in the simulation unit 212 of the point cloud data generation unit 210 based on information input to the input interface 211 (step S101). The construction site includes objects. The objects are represented by a set of points, and each point of the object is tagged with attribute information.

The simulation unit 212 then performs a simulation in which the LiDAR 15 illuminates the recreated construction site, and generates point cloud data based on the simulation. Each point in the generated point cloud data corresponds to one of the points in the construction environment data that constitutes the object.

The simulation unit 212 determines the attribute information tagged to the points of the object corresponding to each point in the point cloud data, and assigns annotations to each point in the point cloud data according to the determined attribute information (step S103). For example, if a point included in the construction environment data is tagged with attribute information of "embankment," the simulation unit 212 assigns the annotation "embankment" to the corresponding point in the point cloud data. The simulation unit 212 similarly assigns annotations to all points included in the point cloud data, thereby generating point cloud data (step S105).

The simulation unit 212 then generates a large amount of data set that includes attribute information in the construction environment data and annotation data in the point cloud data (step S107). The simulation unit 212 then outputs the generated large amount of data set to the learning unit 220 (step S109).

The learning unit 220 performs machine learning using the large amount of data set output by the simulation unit 212 to generate a trained model (step S111). The learning unit 220 transmits the generated trained model to the environmental data generating device 100 (step S113). In this way, the model generating device 200 ends the processing shown in FIG. 3.

Next, the processing in the environmental data generating device 100 will be described. Figure 4 is a flowchart showing an example of the processing in the environmental data generating device 100. The processing in this flowchart begins when the construction machine 10 begins autonomous operation. In the following explanation, it is assumed that the construction machine 10 has already arrived at the position (location) of the object where construction work will be performed.

First, when the autonomous operation of the civil engineering construction machine 10 begins, the environmental data generating device 100 acquires the determined construction plan, for example, via the communication device 17 (step S201). Next, the model acquisition unit 111A acquires the trained model transmitted by the model generating device 200, for example, via the communication device 17 (step S203). The model acquisition unit 111A outputs the acquired trained model to the recognition unit 112 (step S205).

Next, the environmental data generating device 100 causes the LiDAR 15 and camera 16 to measure the state of the construction environment (step S207). The LiDAR 15 and camera 16 output point cloud data and image data corresponding to the measured state of the construction environment. Next, the point cloud data acquisition unit 111B acquires the point cloud data output by the LiDAR 15 (step S209). The point cloud data acquisition unit 111B outputs the acquired point cloud data to the recognition unit 112.

The image data acquisition unit 111C also acquires image data output by the camera 16 (step S211). The image data acquisition unit 111C outputs the acquired image data to the recognition unit 112 (step S213). The processing of steps S207 and S209 and the processing of steps S211 and S213 may be performed simultaneously or may be interchanged.

Next, the recognition unit 112 inputs the point cloud data output by the point cloud data acquisition unit 111B as input data to the trained model output by the model acquisition unit 111A. The recognition unit 112 recognizes the object to be constructed (e.g., an embankment) based on the output data output by the trained model (step S215). The recognition unit 112 outputs object information, in which information about the recognized object is linked to the point cloud data, to the environmental data generation unit 120 (step S217).

The environmental data generation unit 120 generates environmental data representing the state of the construction environment where construction will be performed, based on the object information output by the recognition unit 112 (step S219). Next, the environmental data generation unit 120 outputs the generated environmental data to the control value generation unit 130 (step S221).

The control value generation unit 130 compares the construction plan with the environmental data, that is, compares the state of the construction environment represented by the construction plan with the current state of the construction environment represented by the environmental data (step S223). The control value generation unit 130 then generates control values based on the results of comparing the construction plan with the environmental data (step S225). The control value generation unit 130 outputs the generated control values to the operation control device 12, which controls the traveling driving force output device 13 and the construction driving force output device 14, thereby causing the civil engineering and construction machine 10 to operate autonomously.

The environmental data generation system 1 described above generates environmental data based on the results of inputting point cloud data indicating the state of the construction environment measured by LiDAR 15 and camera 16 into a trained model. This allows for more accurate recognition of the construction environment at the construction site.

Furthermore, the trained model for generating environmental data is generated by the model generation device 200. The model generation device 200 generates large amounts of point cloud data using the point cloud data generation unit 210 equipped with the simulation unit 212, making it easy to generate large amounts of point cloud data used to generate trained models. This makes it easy to build AI, thereby improving the accuracy of object recognition at construction sites.

Furthermore, as construction work and other activities are carried out at construction sites, the construction environment at the construction site changes from moment to moment. Under such circumstances, it is necessary to generate environmental data in accordance with changes in the construction environment, but generating environmental data every time the construction environment changes is time-consuming. In this regard, the environmental data generation system 1 of the embodiment generates a trained model for generating environmental data by performing a simulation using the model generation device 200, making it easy to provide a highly accurate trained model. Therefore, at construction sites where the rate of change is high, environmental data can be easily generated in accordance with changes in the construction environment.

In the above embodiment, it is assumed that the civil engineering and construction machine 10 is a bulldozer that levels piled-up earth (embankment) at a construction site, but the same applies if the civil engineering and construction machine 10 is a construction machine other than a bulldozer. For example, even if the civil engineering and construction machine 10 is a vibrating roller that autonomously performs construction work by compacting earth and sand with a roller, the environmental data generating device 100 can generate environmental data (e.g., a map of the construction site) tailored to the target construction work.

In the above embodiment, the target is a civil engineering work environment in which construction machinery such as bulldozers is used, but the civil engineering work environment may be other environments. For example, the civil engineering work environment may be a tunnel construction site to reduce accidents within the tunnel. In this case, the simulation unit 212 can, for example, run a simulation in which construction machinery and people are placed within the tunnel to generate point cloud data, which can then be learned by the learning unit 220. In this case, it becomes possible to recognize objects within the tunnel, and the system can function as a contact prevention system to prevent contact between construction machinery and people.

[Hardware configuration]
FIG. 5 is a diagram illustrating an example of the hardware configuration of an object recognition device 800 according to an embodiment. As illustrated, the object recognition device 800 includes a communication controller 800-1, a CPU 800-2, a random access memory (RAM) 800-3 used as a working memory, a read-only memory (ROM) 800-4 for storing a boot program and the like, a storage device 800-5 such as a flash memory or a hard disk drive (HDD), and a drive device 800-6, all interconnected via an internal bus or a dedicated communication line. The communication controller 800-1 communicates with components other than the object recognition device 800. The storage device 800-5 stores a program 800-5a executed by the CPU 800-2. This program is loaded into the RAM 800-3 by a direct memory access (DMA) controller (not shown) or the like and executed by the CPU 800-2. This realizes part or all of the acquisition unit 111 (model acquisition unit 111A, point cloud data acquisition unit 111B, image data acquisition unit 111C) and recognition unit 112 in the object recognition unit 110.

The above-described embodiment can be expressed as follows.
a storage device storing a program;
a hardware processor;
The hardware processor executes the program stored in the storage device,
a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work;
Recognizing objects present in the construction environment using the acquired model.
The object recognition device is configured as follows.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1...Environmental Data Generation System, 10...Construction Machine, 11...Driving Operator, 12...Driving Control Device, 13...Travel Driving Force Output Device, 14...Construction Driving Force Output Device, 15...LiDAR, 16...Camera, 17...Communication Device, 18...Positioning Device, 100...Environmental Data Generation Device, 110...Object Recognition Unit, 111...Acquisition Unit, 111A...Model Acquisition Unit, 111B...Point Cloud Data Acquisition Unit, 111C...Image Data Acquisition Unit, 112...Recognition Unit, 120...Environmental Data Generation Unit, 130...Control Value Generation Unit, 200...Model Generation Device, 210...Point Cloud Data Generation Unit, 211...Input Interface, 212...Simulation Unit, 220...Learning Unit

Claims (7)

a model acquisition unit that acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work;
a recognition unit that recognizes an object present in the construction environment using the acquired model ,
the point cloud data generation device includes a simulator that simulates the construction environment,
The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of irradiating light on the construction environment using the simulator.
Object recognition device. The simulation includes a simulation using a distance measured based on a time from the irradiation of the irradiated light to the reception of the reflected light,
The object recognition device according to claim 1 . The data set includes, as input data of the model, construction environment data relating to the object tagged with tag data in the simulation of the simulator, and includes, as output data of the model, data of annotations assigned to each point of the point cloud data based on the construction environment data.
The object recognition device according to claim 1 or 2. An object recognition device according to any one of claims 1 to 3;
an environment data generation unit that generates environment data representing a state of the construction environment based on the object recognized by the object recognition device,
Environmental data generator. a learning unit that generates a model by learning using data sets of construction environment data of a construction environment of civil engineering work and point cloud data generated by a point cloud data generating device based on the construction environment data;
The data set includes, as input data of the model, construction environment data relating to objects tagged with tag data in a simulation in which light is irradiated onto the construction environment of a simulator that virtualizes the construction environment, and includes, as output data of the model, data of annotations assigned to each point of the point cloud data based on the construction environment data.
Model generation device. The computer of the object recognition device
a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work;
Using the acquired model, recognize objects present in the construction environment;
the point cloud data generation device includes a simulator that simulates the construction environment,
The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of irradiating light on the construction environment using the simulator.
Object recognition method. The object recognition device's computer
acquiring a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work;
Using the acquired model, objects present in the construction environment are recognized ;
the point cloud data generation device includes a simulator that simulates the construction environment,
The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of light irradiation using the simulator.
program.