JP7612460B2 - Construction machinery motion control system - Google Patents

Description

The present invention relates to a construction machine operation control system that is applied to construction machines such as hydraulic excavators.

As a technology for automatically operating multiple construction machines performing different tasks at a construction site, for example, Patent Document 1 discloses a construction method for multiple construction machines performing different tasks that have an automatic driving function and that is managed by a construction management unit, the construction method comprising a construction information transmission process in which the construction management unit transmits construction position information for each construction machine to the construction machine, a construction information reception process in which the construction machine receives the construction position information transmitted in the construction information transmission process for each construction machine, and an automatic driving operation process in which the construction machine performs work in automatic driving using the construction position information received in the construction information reception process for each construction machine.

JP 2016-132912 A

In the above conventional technology, the construction management unit transmits construction position information to each of the multiple construction machines at the construction site. The multiple construction machines at the construction site perform work by automatic driving based on the construction position information they receive. This allows the multiple construction machines to operate by automatic driving under the management of the construction management unit.

In addition, when an operator operating the construction system visually inspects the area in which the construction machinery is working and recognizes that an unusual situation has occurred, the operator issues a command to the construction machinery to stop working according to the situation, i.e., a work stop command. The work stop command is sent from the construction management unit via the transmitting/receiving device to each construction machinery, causing the construction machinery to stop working. This makes it possible to avoid a decrease in construction efficiency due to the occurrence of unusual situations during work by construction machinery.

However, in the above conventional technology, the correspondence between the target of the command in the system and the actual construction machine to which the command is applied is not clearly stated. In particular, when there are multiple construction machines of the same type working in parallel, it takes time to identify the target to which the command is to be issued, which may reduce construction efficiency. In addition, since the target of the command is determined based on the operator's visual inspection, there is a possibility that the target may be misjudged and a command may be issued to a different construction machine, resulting in unintended results. Furthermore, in order to confirm whether a command has been issued to the correct target, it is necessary to wait until the target construction machine actually starts operating according to the command, which may reduce construction efficiency.

The present invention has been made in consideration of the above, and aims to provide a construction machine operation control system that can more reliably specify the construction machine to be operated, and can more easily and quickly confirm whether a command has been issued to the correct operation target.

The present application includes a number of means for solving the above-mentioned problems, and an example thereof is a construction management terminal having a terminal control device that creates control information for operating a construction machine, a terminal display device that displays the control information created by the terminal control device, a terminal operation device that detects the coordinates of the specified control information as input coordinate values and outputs the detected input coordinate values when an operator specifies at least a part of the control information displayed on the terminal display device, and a terminal transmission/reception device that transmits and receives the control information, a machine transmission/reception device that transmits and receives the control information, a machine position detection device that acquires the position of the construction machine within the construction site space and outputs it as machine position information, a machine control device that outputs a control signal according to the input control information, and a construction machine having a control target that is a control mechanism of the construction machine that operates in response to the control signal input from the machine control device, the construction management terminal being configured to detect the orientation of the construction management terminal within the construction site space, and outputting the terminal attitude information; a terminal position detection device that acquires the position of the construction management terminal in the construction site space and outputs it as terminal position information; a terminal camera that is provided integrally with the construction management terminal and captures the surroundings of the construction management terminal and outputs it as a surrounding image; and a terminal state calculation device that calculates and outputs terminal state information indicating the state of the construction management terminal based on the terminal attitude information detected by the terminal attitude detection device and the terminal position information detected by the terminal position detection device, and identifies the construction machine present in the construction site space corresponding to the input coordinate value specified by the terminal operation device as an operation target based on the input coordinate value specified by the terminal display device and the terminal state information detected by the terminal state calculation device when the image detected by the terminal camera is displayed on the terminal display device, and generates stop control information that is control information for stopping the construction machine identified as the operation target.

The present invention makes it possible to more reliably specify the construction machine to be operated, and to more easily and quickly confirm whether a command has been issued to the correct target.

1 is a functional block diagram illustrating a schematic configuration of a construction machine operation control system according to a first embodiment. 4 is a flowchart showing the processing of the operation control system for a construction machine according to the first embodiment. FIG. 1 is a diagram illustrating an example of a construction machine operating at a construction site. 1A and 1B are diagrams illustrating the basic principle of camera attitude detection. FIG. 4 is a diagram showing how a construction machine is designated by the operation device according to the first embodiment. FIG. 13 is a diagram showing a construction machine displayed on a display device of a construction management terminal. FIG. 13 is a diagram for explaining the principle of determining whether or not a construction machine can be stopped, and illustrates a case where the zero moment point is inside an indication polygon. FIG. 11 is a diagram showing an example of information displayed on a display device of the construction management terminal. FIG. 13 is a diagram for explaining the principle of determining whether or not a construction machine can be stopped, and illustrates a case where the zero moment point is outside the instruction polygon. FIG. 11 is a diagram showing an example of information displayed on a display device of the construction management terminal. FIG. 11 is a diagram showing how a construction machine is designated by an operation device according to a second embodiment. 10 is a flowchart showing the processing of a construction machine operation control system according to a second embodiment. 13A and 13B are diagrams illustrating how a construction machine is designated by an operation device according to a third embodiment. 13 is a flowchart showing the processing of a construction machine operation control system according to a third embodiment. 13A and 13B are diagrams showing how a construction machine is designated by an operation device according to a fourth embodiment. 13 is a flowchart showing the processing of a construction machine operation control system according to a fourth embodiment. FIG. 13 is a diagram showing an example of a display of an estimated vertical error on a display device according to a fifth embodiment.

The following describes an embodiment of the present invention with reference to the drawings. Note that in this embodiment, a hydraulic excavator is used as an example of a construction machine, but the present invention is not limited to this and can be applied to other construction machines operating at construction sites.

First Embodiment
A first embodiment of the present invention will be described with reference to FIGS.

Figure 1 is a functional block diagram that shows the schematic configuration of a construction machine operation control system according to this embodiment.

In FIG. 1, the construction machinery operation control system allows a user to operate a specified construction machinery by specifying the construction machinery on the camera image, and has a functional unit provided in the construction machinery 300 (e.g., a hydraulic excavator) operating at the construction site and a functional unit provided in the construction management terminal 100 operated by the user who operates the construction machinery 300.

The construction management terminal 100 is a device (e.g., a mobile terminal such as a tablet or smartphone) that is carried and operated by a user, such as a site manager who manages multiple construction machines (in Figure 1, construction machine 300 is shown as a representative) operating at a construction site, and is roughly composed of a control device 101, a display device 102, an operation device 103, a transmission/reception device 104, an attitude detection device 109, a position detection device 110, a camera 111, and a status calculation device 112.

The control device 101 and the state calculation device 112 are configured, for example, by an arithmetic device having an input interface provided in the construction management terminal 100, a central processing unit (CPU) which is a processor, a read-only memory (ROM) or random access memory (RAM) which is a storage device, an output interface, etc. In other words, each function of the control device 101 and the state calculation device 112 is configured by performing a predetermined arithmetic process according to a control program stored in the ROM on signals input via the input interface and information stored in the ROM and RAM, etc., and outputting the result via the output interface.

The control device 101 (terminal control device) performs calculations based on operational input by the user, the status of the construction management terminal, camera images, information input from the construction machinery, etc., and creates control information for operating the construction machinery 300, for example, and outputs the result.

The display device 102 (terminal display device) displays the information created and output by the control device 101 on a screen to show it to the user (operator). The display device 102 is, for example, an LCD monitor.

The operation device 103 (terminal operation device) detects user input and inputs the detected information to the control device 101. The operation device 103 is an input device such as a touch panel, and is placed on the surface of the display device 102 to form a GUI (Graphical User Interface) together with the display device 102.

The attitude detection device 109 (terminal attitude detection device) detects the direction in which the construction management terminal is physically facing in a coordinate system set in the construction site space, and outputs the detection result as terminal attitude information. As the attitude detection device 109, for example, sensors such as a magnetic sensor that detects geomagnetism, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) can be used alone or in combination.

The position detection device 110 (terminal position detection device) detects the position of the construction management terminal within the construction site space and outputs the detection result as terminal position information. The position detection device 110 is, for example, a GNSS (Global Navigation Satellite System).

Camera 111 (terminal camera) is attached to the construction management terminal, captures images within the construction site space, and outputs the captured images to control device 101. Camera 111 is, for example, a photographing device using a CMOS image sensor.

The state calculation device 112 calculates the position and attitude of the construction management terminal within the construction site space based on the terminal attitude information input from the attitude detection device 109 and the terminal position information input from the position detection device 110, and outputs the calculation result to the control device 101 as terminal state information.

The transmission/reception device 104 transmits information output by the control device 101 (such as control information described later) to the transmission/reception device 105 of the construction machine 300 via the network 200, and also receives information output from the transmission/reception device 105 of the construction machine 300 (such as unstoppable information described later) and inputs it to the control device 101. The transmission/reception device 104 is, for example, a communication device that performs wireless communication.

The construction machine 300 is, for example, a work machine such as a hydraulic excavator operating at a construction site, and is generally composed of a transmitting/receiving device 105, a position detection device 106, a control device 107, a control target 108, an operating status detection device 113, and a stop possibility determination device 114.

The control device 107, the movable state detection device 113, and the stoppability determination device 114 are configured by, for example, an arithmetic device having an input interface provided on the construction machine 300, a central processing unit (CPU) which is a processor, a read-only memory (ROM) or random access memory (RAM) which is a storage device, an output interface, etc. In other words, each function of the control device 107, the movable state detection device 113, and the stoppability determination device 114 is configured by performing a predetermined arithmetic process according to a control program stored in the ROM on signals input via the input interface and information stored in the ROM and RAM, etc., and outputting the results via the output interface.

The transmitter/receiver 105 (machine transmitter/receiver) receives information (such as control information described below) output from the transmitter/receiver 104 of the construction management terminal 100 via the network 200 and inputs it to the control device 107, and also transmits information generated by the construction machine 300 (such as unstoppable information described below) to the transmitter/receiver 104 of the construction management terminal 100. The transmitter/receiver 105 is, for example, a communication device that performs wireless communication.

The position detection device 106 (machine position detection device) detects the position of the construction machine within the construction site space and outputs the resulting position information. The position detection device 106 is, for example, a GNSS (Global Navigation Satellite System).

The control device 107 (machine control device) generates control amounts according to the input control information in order to operate the construction machine 300, and outputs the control amounts as signals to each device (control target) of the vehicle body that can be operated.

The controlled object 108 is a control mechanism of the construction machine that operates based on the control amount input from the control device 107. In the hydraulic excavator exemplified as the construction machine 300 in this embodiment, the controlled object is, for example, the engine and various hydraulic cylinders.

The operating status detection device 113 (machine operating status detection device) detects the status of each part of the construction machine 300 and the posture of the construction machine 300, and outputs as the detection result machine operating status information consisting of machine posture information, which is information indicating the posture of the construction machine 300, and the control amount currently being applied.

The stop possibility determination device 114 determines whether the construction machine 300 is in a state where stopping its operation would result in unstable behavior, such as tipping or slipping, based on the machine operation status information output by the operation status detection device 113, and outputs stop impossible information to the transmission/reception device 105 if the determination result indicates that stopping the operation would result in unstable behavior, and outputs stop control information to the control device 107 if stopping the operation would not result in unstable behavior.

In the construction machine operation control system configured as described above, the operator specifies the desired construction machine 300 on the construction management terminal 100, and it is determined whether the operating state of the specified construction machine 300 can be stopped, and the operation is stopped according to the determination result.

Figure 2 is a flowchart showing the processing of the construction machine operation control system.

In FIG. 2, when the construction management terminal 100 starts operation, it first detects input from the operation device 103 (step S201).

Figure 3 is a diagram illustrating construction machinery operating at a construction site.

Figure 3 illustrates an example in which a total of three shovels, a first shovel 301, a second shovel 302, and a third shovel 303, are present within the construction site space.

In this state, the construction management terminal is equipped with a liquid crystal display as the display device 102, and the image of the construction site space input from the camera 111 is displayed on the liquid crystal display. In addition, a touch panel is provided as the operation device 103, overlaid on the surface of the display device 102. By operating the operation device 103, the user can specify a point on the image of the construction site space captured by the camera 111. As a result of the specification, the operation device 103 detects this input.

Next, the orientation of the camera 111 is calculated (step S202).

Figure 4 is a diagram explaining the basic principles of camera orientation detection.

As shown in FIG. 4, the position (xc, yc, zc) of the camera 111 on the construction management terminal in the coordinate system that indicates the construction site space is called the camera position 401. The camera position 401 is obtained using the position detection device 110. For example, the position detection device 110 may calculate the position within the construction site space from the latitude and longitude on the earth obtained using a GPS (Global Positioning System). As another example, a total station or the like may be used to obtain the position within the construction site space.

The direction vector (xe, ye, ze) indicating the direction of the camera 111 is called the camera direction vector 402. The camera direction vector 402 is obtained using the orientation detection device 109. The orientation detection device 109 may calculate the direction from sensors such as a geomagnetic sensor or an angle sensor.

Next, the position specified by the user (the operator of the construction management terminal 100) within the construction site space is calculated from the position on the display device 102 specified by the user by operating the operation device 103 (step S203).

Figure 5 shows how a construction machine is specified using the operating device.

In FIG. 5, the camera 111 is attached to the back of the construction management terminal. The candidate construction machine is assumed to be the construction machine photographed at the position of a point on the display device 102 that is specified by the user by operating the operation device 103. This point is called the specified point 501. The coordinates of the specified point 501 on the display device 102 are Ps. The candidate construction machine indicated by the specified point 501 exists on a half line extending from the focal position of the camera 111 in the direction corresponding to the specified point 501. This half line is called the specified line 502. The construction machine that has an intersection with the specified line 502 is assumed to be the construction machine to be operated (the construction machine to be operated 503). In the subsequent processing, the construction machine to be operated 503 will be acquired, and the specified point 501 used for this purpose is processed as the position specified by the user.

Next, it is determined whether or not verification processing has been performed on all construction machines connected to the network 200 (step S204), and if the determination result is YES, the process proceeds to step S212.

If the result of the determination in step S204 is NO, i.e., if it is determined that there is a construction machine for which the verification process has not been performed, the position of one construction machine to be subject to the verification process is obtained (step S205).

The position information of one of the construction machines connected to the network 200 detected by the position detection device 106 is transmitted by the transmission/reception device 105 of the construction machine, and is received by the transmission/reception device 104 of the construction management terminal 100 via the network 200 and input to the control device 101. The control device 101 converts the received position of the construction machine from the three-dimensional coordinate system of the construction site space to the coordinate system of the camera 111 using the camera position 401, the camera direction vector 402, and the geometric characteristics of the lens and image sensor held by the camera 111. The coordinate system of the camera 111 is equivalent to the coordinate system indicated by the planar image of the construction site space displayed on the display device 102.

Figure 6 shows how construction machinery is displayed on the display device of the construction management terminal.

As shown in FIG. 6, the position of the construction machine detected by the position detection device 106 is expressed as the position 601 of the construction machine on the image in the coordinate system of the camera 111.

Next, it is determined whether the position 601 of the construction machine on the image of the display device 102 overlaps with the specified point 501 (step S206).

The construction machine occupies a certain amount of space in the construction site space. This is called the occupied space V. In the example of FIG. 6, the occupied space V is represented by a rectangular parallelepiped, but in reality, it represents the space occupied when the shape of the current construction machine 503 to be operated, obtained from the information output by the operating state detection device 113 of the construction machine 503 to be operated, is placed at the position obtained from the information output by the position detection device 106 of the construction machine 503 to be operated. When this occupied space V is projected onto the coordinate system of the camera 111, it occupies a certain area on the two-dimensional image displayed on the display device 102, as shown in FIG. 6. This is called the occupied area 602 of the construction machine on the image. It is represented by a symbol A. The relationship between A and the occupied space V can be expressed as A = P (V). Here, P is the projection from the occupied space V, which is a three-dimensional space, to A, which is a two-dimensional area. If the specified point 501 is included within the occupation area 602 of the construction machine on the image, i.e., if Ps ⊂ A, it is determined that the two overlap.

If the determination result in step S206 is NO, i.e., if it is determined that the position of the construction machine on the image does not overlap with the specified point, the process returns to step S204.

If the determination result in step S206 is YES, information to be used for operational control of the construction machine 503 to be operated that has been determined to overlap with the specified point 501, such as the current position of the construction machine 503 to be operated obtained from the position detection device 106, is obtained (step S207).

Next, the operating status of the construction machine 503 to be operated is acquired (step S208). The operating status detection device 113 acquires information such as the positions of the boom, arm, and bucket of the shovel, the rotation angle of the upper rotating body, and the inclination of the lower traveling body.

Next, it is determined whether or not it is possible to stably stop the construction machine 503 being operated at the current processing time (step S209).

Figure 7 is a diagram explaining the principle of determining whether or not a construction machine can be stopped, and shows an example of a case where the zero moment point is inside the indication polygon.

In FIG. 7, the construction machine 503 to be operated is shown descending a slope. The ground contact position 701 of the construction machine 503 to be operated is the position where a pair of parallel tracks of the construction machine 503 to be operated, obtained from the operating state detection device 113, contacts the ground. In this state, the support polygon 702 of the construction machine 503 to be operated indicates a rectangular area including the space between the parallel tracks. Next, the center of gravity 703 of the construction machine 503 to be operated is obtained from the state of the boom, arm, bucket, upper rotating body, lower traveling body, etc. of the construction machine 503 to be operated, which is input from the operating state detection device 113. The center of gravity 703 is used to obtain the ZMP (Zero Moment Point) 704, which is the center of pressure of the floor reaction force. If the ZMP 704 is within the area of the support polygon 702, it can be said that the construction machine 503 to be operated is in a stable state. In the example of FIG. 7, the ZMP 704 exists within the area of the support polygon 702. When the construction machine 503 to be operated is in this state, it is determined that the construction machine 503 to be operated can be stopped stably.

If the determination result in step S209 is YES, i.e., if it is determined that the construction machine can be stopped, the control device 107 sends a stop signal (a control amount equivalent to stopping) to the controlled object 108, and stops the construction machine 503 to be operated (step S210).

Figure 8 shows an example of information displayed on the display device of the construction management terminal.

As shown in FIG. 8, when stop control information is sent from the construction management terminal 100 to the construction machine 503 to be operated, a display content (message) 901 indicating that the transmission of the specified operation signal (i.e., the stop control signal) has been completed is displayed near the position 601 of the construction machine on the image on the display device 102. This allows the operator of the construction management terminal to more easily and quickly confirm whether a command has been issued to the correct operation target.

In addition, if the determination result in step S209 is NO, that is, if it is determined that the construction machine cannot be stopped, the reason why operation of the construction machine 503 to be operated is not possible is sent to the control device 101 via the transmission/reception device 105 of the construction machine, the network 200, and the transmission/reception device 104 of the construction management terminal 100 (step S211).

Figure 9 is a diagram explaining the principle of determining whether or not a construction machine can be stopped, and illustrates a case where the zero moment point is outside the instruction polygon.

In FIG. 9, the construction machine 503 to be operated is shown descending a slope with a greater angle than in the example shown in FIG. 7. Based on the center of gravity 801 in the different state, the ZMP 802 in the different state is determined. The ZMP 802 in the different state exists outside the area of the support polygon 702. This result is sent to the control device 101. The sent message 803 contains the name identifying the construction machine that sent the message 803, operation feasibility information indicating whether a stop operation can be performed, and, if a stop operation cannot be performed, the reason for this. When the control device 101 receives this message 803, it displays the contents on the display device 102.

Figure 10 shows an example of information displayed on the display device of the construction management terminal.

In FIG. 10, the display content (message) 902 created based on the telegram 803 is displayed near the position 601 of the construction machine on the image on the display device 102. This allows the user to be informed of the reason why the construction machine 503 being operated is not stopped, and makes it possible to encourage the user to take other measures.

When the processing of step S210 or step S211 is completed, it is then determined whether or not to terminate the operation of the construction machine operation control system (step S212).

For example, it is possible to determine that the operation is to be terminated by the user turning off the power of the construction management terminal. If the determination result in step S212 is YES, that is, if it is determined that the operation is to be terminated, the system is terminated. If the determination result in step S212 is NO, the process returns to step S201.

In this embodiment, an example is shown in which the ZMP 704 is used to determine whether the construction machine 503 to be operated can be stopped, but other methods can also be used for the determination. This process stops the construction machine 503 to be operated specified by the user, or if stopping the construction machine 503 would cause the operation of the construction machine 503 to become unstable, the construction machine is not stopped and the reason is displayed to the user, making it possible to operate the construction machine stably.

In the present embodiment configured as described above, a construction management terminal 100 having a terminal control device (control device 101) that creates control information for operating a construction machine, a terminal display device (display device 102) that displays the control information created by the terminal control device, a terminal operation device (operation device 103) that detects the coordinates of the specified control information as input coordinate values and outputs the detected input coordinate values when the operator specifies at least a part of the control information displayed on the terminal display device, and a terminal transmission/reception device (transmission/reception device 104) that transmits and receives the control information, a machine transmission/reception device (transmission/reception device 105) that transmits and receives the control information, a machine position detection device (position detection device 106) that acquires the position of the construction machine in the construction site space and outputs it as machine position information, a machine control device (control device 107) that outputs a control signal according to the input control information, and a construction machine having a control target that is a control mechanism of the construction machine that operates in response to the control signal input from the machine control device, in the operation control system for the construction machine, the construction management terminal detects the orientation of the construction management terminal in the construction site space and outputs it as terminal attitude information. The construction management terminal further includes a terminal position detection device (attitude detection device 109), a terminal position detection device (position detection device 110) that acquires the position of the construction management terminal in the construction site space and outputs it as terminal position information, a terminal camera (camera 111) that is provided integrally with the construction management terminal and captures the situation around the construction management terminal and outputs it as a surrounding image, and a terminal state calculation device (state calculation device 112) that calculates and outputs terminal state information indicating the state of the construction management terminal based on the terminal attitude information detected by the terminal attitude detection device and the terminal position information detected by the terminal position detection device. In a state in which the surrounding image output by the terminal camera is displayed on the terminal display device, the construction machine present in the construction site space corresponding to the input coordinate value specified by the terminal operation device is identified as the operation target based on the input coordinate value specified on the terminal display device and the terminal state information calculated by the terminal state calculation device, and stop control information, which is control information for stopping the construction machine identified as the operation target, is generated. This makes it possible to more reliably specify the construction machine to be operated, and to more easily and quickly confirm whether a command to the correct operation target has been issued.

Second Embodiment
A second embodiment of the present invention will be described with reference to FIGS.

This embodiment shows a case where, when there are multiple construction machines to be operated at a location specified by a user (the operator of the construction management terminal), one machine is selected from the multiple machines and processed.

Figure 11 is a diagram showing how a construction machine is specified by the operating device. Also, Figure 12 is a flowchart showing the processing of the construction machine operation control system. In Figures 11 and 12, the same reference numerals are used for the same components as in the first embodiment, and the description will be omitted.

In FIG. 11, the point on the display device 102 that the user specifies by operating the operation device 103 is called the specified point 1001 in the second example. The coordinates of the specified point 1001 in the second example on the display device 102 are P's. The candidate construction machine indicated by the position where the specified point 1001 in the second example exists is on a half line extending from the focal position of the camera 111 in the direction corresponding to the specified point 1001 in the second example. This half line is called the specified line 1002 in the second example. In this example, there are three construction machines that have an intersection with the specified line 1002 in the second example. These have a spatial anteroposterior relationship on the specified line 1002 in the second example. This is because a physical object such as a construction machine cannot exist simultaneously in exactly the same position in space. In the image captured by the camera 111, the construction machine that exists in the position closest to the camera 111 in the construction site space is shown. In other words, in this case, the user specifies the construction machine that is closest to the construction management terminal. This construction machine to be operated is called the construction machine to be operated 1003 in the second example.

In FIG. 12, when the construction management terminal 100 starts operation, it first detects input from the operation device 103 (step S201).

Next, the orientation of the camera 111 is calculated (step S202).

Next, the position specified by the user (the operator of the construction management terminal 100) within the construction site space is calculated from the position on the display device 102 specified by the user by operating the operation device 103 (step S203).

Next, it is determined whether or not verification processing has been performed on all construction machines connected to the network 200 (step S204), and if the determination result is YES, the process proceeds to step S212.

If the result of the determination in step S204 is NO, i.e., if it is determined that there is a construction machine for which the verification process has not been performed, the position of one construction machine to be subject to the verification process is obtained (step S205).

Next, it is determined whether the position 601 of the construction machine on the image of the display device 102 overlaps with the specified point 501 (step S206). If the specified point 1001 in the second example is included within the occupation area 602 of the construction machine on the image, i.e., if P's ⊂ A, it is determined that the two overlap.

If the result of the determination in step S206 is YES, the position in the construction site space where the construction machine is located, where P's ⊂ A, is stored in the control device 101 (step S1101), and the process returns to step S204.

Also, if the determination result in step S206 is NO, the process returns to step S204.

If the determination result in step S204 is YES, the construction machine closest to the position of the camera 111 is extracted from the information on the construction machines stored in step S1101 as the construction machine 1003 to be operated in the second example (step S207).

Next, the operating status of the construction machine to be operated is obtained (step S208).

Next, it is determined whether or not it is possible to stably stop the construction machine being operated at the current processing time (step S209).

If the determination result in step S209 is YES, i.e., if it is determined that the construction machine can be stopped, the control device 107 sends a stop signal (a control amount equivalent to stopping) to the controlled object 108, and stops the construction machine being operated (step S210).

In addition, if the determination result in step S209 is NO, i.e., if it is determined that the construction machine cannot be stopped, the reason why operation of the construction machine to be operated is not possible is sent to the control device 101 via the transmission/reception device 105 of the construction machine, the network 200, and the transmission/reception device 104 of the construction management terminal 100 (step S211).

When the processing of step S210 or step S211 is completed, it is then determined whether or not to terminate the operation of the construction machine operation control system (step S212).

If the determination in step S212 is YES, i.e., if it is determined that the process is to end, the system is terminated. If the determination in step S212 is NO, the process returns to step S201.

The rest of the configuration is the same as in the first embodiment.

The present embodiment, configured as described above, can achieve the same effects as the first embodiment.

In addition, when multiple construction machines are present on the specified line 1002 of the second example obtained by specifying the specified point 1001 of the second example, it becomes possible to obtain the construction machine displayed at the position of the specified point 1001 of the second example on the construction management terminal as the construction machine 1003 to be operated of the second example, making it possible to operate the construction machine specified from the user's viewpoint without error.

Third Embodiment
A third embodiment of the present invention will be described with reference to FIGS.

This embodiment shows how to extract and process multiple construction machines that are to be operated when they are in an area specified by a user (operator of the construction management terminal).

Figure 13 is a diagram showing how a construction machine is specified by the operating device. Also, Figure 14 is a flowchart showing the processing of the construction machine operation control system. In Figures 13 and 14, the same reference numerals are used for components similar to those in the first and second embodiments, and descriptions thereof will be omitted.

In FIG. 13, the area on the display device 102 that is specified by operating the operation device 103 is called the specified area 1201 of the third example. The specified area 1201 of the third example is symbolically represented as Rs. The specified area 1201 of the third example can be expressed, for example, as a rectangular area, and can be expressed as a rectangle with two points specified on the display device 102 as the vertices of the upper left corner and the lower right corner. The candidate construction machine indicated by the position where the specified area 1201 of the third example exists is inside a pyramid that is extended from the focal position of the camera 111 in the direction corresponding to the specified area 1201 of the third example. This pyramid is called the specified pyramid 1202 of the third example. In this example, there are two construction machines that intersect with the specified pyramid 1202 of the third example. These construction machines to be operated are called the first operation target construction machine 1203 of the third example and the second operation target construction machine 1204 of the third example, respectively.

The first operation target construction machine 1203 of the third example and the second operation target construction machine 1204 of the third example occupy a certain area on the two-dimensional image displayed on the display device 102, similar to the explanation of FIG. 6 of the first embodiment. The occupied area of the first operation target construction machine 1203 of the third example is symbolically represented as A1, and the occupied area of the second operation target construction machine 1204 of the third example is symbolically represented as A2. Here, neither A1∩Rs nor A2∩Rs is an empty set.

In FIG. 14, when the construction management terminal 100 starts operation, it first detects input from the operation device 103 (step S201).

Next, the orientation of the camera 111 is calculated (step S202).

Next, the specified area 1201 of the third example is calculated (step S1301). As described above, two points are specified on the display device 102, and the area Rs is represented by a rectangle with these points as the upper left and lower right corners.

Next, it is determined whether or not verification processing has been performed on all construction machines connected to the network 200 (step S204), and if the determination result is YES, the process proceeds to step S212.

If the result of the determination in step S204 is NO, i.e., if it is determined that there is a construction machine for which the verification process has not been performed, the position of one construction machine to be subject to the verification process is obtained (step S205).

Next, it is determined whether the designated area 1201 in the third example overlaps with the occupied area (step S1302). For the construction machine acquired in step S205, the occupied area on the two-dimensional image displayed on the display device 102 is set as A, and A∩Rs is calculated. If A∩Rs is not an empty set, it is determined that the designated area 1201 in the third example overlaps with the occupied area.

If the determination result in step S1302 is YES, the position in the construction site space where the construction machine is located, where P's ⊂ A, is stored in the control device 101 (step S1101), and the process returns to step S204.

Also, if the determination result in step S1302 is NO, the process returns to step S204.

If the determination result in step S204 is YES, the construction machine information stored in step S1101 is extracted one by one (step S207).

Next, the operating status of the construction machine to be operated is obtained (step S208).

Next, it is determined whether or not it is possible to stably stop the construction machine being operated at the current processing time (step S209).

If the determination result in step S209 is YES, i.e., if it is determined that the construction machine can be stopped, the control device 107 sends a stop signal (a control amount equivalent to stopping) to the controlled object 108, and stops the construction machine being operated (step S210).

In addition, if the determination result in step S209 is NO, i.e., if it is determined that the construction machine cannot be stopped, the reason why operation of the construction machine to be operated is not possible is sent to the control device 101 via the transmission/reception device 105 of the construction machine, the network 200, and the transmission/reception device 104 of the construction management terminal 100 (step S211).

When the processing of step S210 or step S211 is completed, it is next determined whether all information has been extracted in step S207 from the information on the construction machines stored in step S1101, i.e., the first operation target construction machine 1203 in the third example in FIG. 13 and the second operation target construction machine 1204 in the third example, i.e., whether all targets have been verified (step S1303).

If the determination result in step S1303 is NO, the process returns to step S207. If the determination result in step S1303 is YES, a determination is made as to whether or not to terminate the operation of the construction machine operation control system (step S212).

If the determination in step S212 is YES, i.e., if it is determined that the process is to end, the system is terminated. If the determination in step S212 is NO, the process returns to step S201.

The rest of the configuration is the same as in the first and second embodiments.

The present embodiment, configured as described above, can achieve the same effects as the first embodiment.

In addition, when multiple construction machines are present on the specified pyramid 1202 of the third example obtained by specifying the specified area 1201 of the third example, it becomes possible to operate all of the corresponding construction machines, making it possible to quickly avoid events that may occur due to the related operations of multiple construction machines.

<Fourth embodiment>
A fourth embodiment of the present invention will be described with reference to FIGS.

This embodiment shows a case where there are multiple construction machines to be operated in an area specified by a user (operator of the construction management terminal), and the user selects the one to be operated from among these.

Figure 15 is a diagram showing how a construction machine is specified by the operating device. Also, Figure 16 is a flowchart showing the processing of the construction machine operation control system. In Figures 15 and 16, the same reference numerals are used for the same components as in the first to third embodiments, and the description will be omitted.

In FIG. 15, the area on the display device 102 designated by operating the operation device 103 is called the designated area 1401 of the fourth example. The designated area 1401 of the fourth example is symbolically represented as R's. The designated area 1401 of the fourth example can be expressed, for example, as a rectangular area, and can be represented as a rectangle with two points designated on the display device 102 as the vertices of the upper left and lower right corners, respectively. The construction machine candidate indicated by the position where the designated area 1401 of the fourth example exists is inside a pyramid that extends from the focal position of the camera 111 in the direction corresponding to the designated area 1401 of the fourth example. This pyramid is called the designated pyramid 1402 of the fourth example. In this example, there are three construction machines that intersect with the designated pyramid 1402 of the fourth example. These construction machines to be operated are respectively referred to as the first construction machine to be operated 1403 in the fourth example, the second construction machine to be operated 1404 in the fourth example, and the third construction machine to be operated 1405 in the fourth example.

The first operation target construction machine 1203 in the third example and the second operation target construction machine 1204 in the third example occupy a certain area on the two-dimensional image displayed on the display device 102, similar to the explanation of FIG. 6 in the first embodiment. The occupied area of the first operation target construction machine 1403 in the fourth example is represented by a symbol A'1, its name is M1, and its distance from the camera 111 is 5 m. Similarly, the occupied area of the second operation target construction machine 1404 in the fourth example is represented by a symbol A'2, its name is M2, and its distance from the camera 111 is 12 m. Also, the occupied area of the third operation target construction machine 1405 in the fourth example is represented by a symbol A'3, its name is M3, and its distance from the camera 111 is 6 m. Here, A'1∩R's, A'2∩R's, and A'3∩R's are not empty sets. The information on each of the above-mentioned construction machines is displayed as a construction machine list 1406 on the display device 102. In this example, the name of each construction machine and the distance from the camera 111 are shown, and a check mark indicates whether each construction machine has been selected. There is also an execute button for executing an operation.

In FIG. 16, when the construction management terminal 100 starts operation, it first detects input from the operation device 103 (step S201).

Next, the orientation of the camera 111 is calculated (step S202).

Next, the specified area 1201 of the third example is calculated (step S1301). As described above, two points are specified on the display device 102, and the area Rs is represented by a rectangle with these points as the upper left and lower right corners.

Next, it is determined whether or not verification processing has been performed on all construction machines connected to the network 200 (step S204), and if the determination result is YES, the process proceeds to step S212.

If the result of the determination in step S204 is NO, i.e., if it is determined that there is a construction machine for which the verification process has not been performed, the position of one construction machine to be subject to the verification process is obtained (step S205).

Next, it is determined whether the designated area 1201 in the third example overlaps with the occupied area (step S1302). For the construction machine acquired in step S205, the occupied area on the two-dimensional image displayed on the display device 102 is set as A, and A∩Rs is calculated. If A∩Rs is not an empty set, it is determined that the designated area 1201 in the third example overlaps with the occupied area.

If the determination result in step S1302 is YES, the position in the construction site space where the construction machine is located, where P's ⊂ A, is stored in the control device 101 (step S1101), and the process returns to step S204.

Also, if the determination result in step S1302 is NO, the process returns to step S204.

Also, if the determination result in step S204 is YES, the information on the construction machines stored in the control device 101 in step S1101 is displayed in list format on the display device 102 (step S501). This results in the display contents being like the construction machine list 1406. The user operates the operation device 103 to switch the check marks in the construction machine list 1406 on or off. Here, the construction machines with the check marks on are the ones to be operated. In this example, the check marks are on in the rows for the first operation target construction machine 1403 in the fourth example and the third operation target construction machine 1405 in the fourth example, and these construction machines are the ones to be operated. Next, the execute button is pressed to proceed to the next step of the process.

Next, the construction machine information stored in step S1101 is extracted one by one (step S207).

Next, the operating status of the construction machine to be operated is obtained (step S208).

Next, it is determined whether or not it is possible to stably stop the construction machine being operated at the current processing time (step S209).

If the determination result in step S209 is YES, i.e., if it is determined that the construction machine can be stopped, the control device 107 sends a stop signal (a control amount equivalent to stopping) to the controlled object 108, and stops the construction machine being operated (step S210).

In addition, if the determination result in step S209 is NO, i.e., if it is determined that the construction machine cannot be stopped, the reason why operation of the construction machine to be operated is not possible is sent to the control device 101 via the transmission/reception device 105 of the construction machine, the network 200, and the transmission/reception device 104 of the construction management terminal 100 (step S211).

When the processing of step S210 or step S211 is completed, it is next determined whether all information has been extracted in step S207 from the information on the construction machines stored in step S1101, i.e., the first operation target construction machine 1203 in the third example in FIG. 13 and the second operation target construction machine 1204 in the third example, i.e., whether all targets have been verified (step S1303).

If the determination result in step S1303 is NO, the process returns to step S207. If the determination result in step S1303 is YES, a determination is made as to whether or not to terminate the operation of the construction machine operation control system (step S212).

If the determination in step S212 is YES, i.e., if it is determined that the process is to end, the system is terminated. If the determination in step S212 is NO, the process returns to step S201.

The rest of the configuration is the same as in the first to third embodiments.

The present embodiment, configured as described above, can achieve the same effects as the first embodiment.

In addition, it becomes possible to specify only the construction machine selected by the user from among the construction machines that were candidates before executing step S1501, and to make it the target of operation. When operating multiple construction machines, construction machines that are assumed to be out of range, for example construction machines that are significantly far away from the camera 111, can be excluded from the target of operation, making it possible to operate the construction machines efficiently.

Fifth embodiment
A fifth embodiment of the present invention will be described with reference to FIG.

This embodiment shows a case where a construction machine is operated using equipment other than the camera 111 mounted on the construction management terminal.

Figure 17 is a functional block diagram that shows the schematic configuration of a construction machine operation control system according to this embodiment. In the figure, the same components as those in the first embodiment are given the same reference numerals, and the description will be omitted.

This embodiment shows the case where imaging equipment is added to the configuration of the first embodiment (see Figure 1).

In FIG. 17, the imaging equipment 400 is roughly composed of a transmitting/receiving device 1601 (equipment transmitting/receiving device), a camera 1602 (equipment camera), an attitude detection device 1603 (equipment attitude detection device), and a position detection device 1604 (equipment position detection device).

The transmission/reception device 1601 of the imaging equipment transmits the information acquired by the imaging equipment to the construction management terminal via the network 200.

Camera 1602 captures images of the construction site space and outputs the results to the transmitting/receiving device 1601.

The orientation detection device 1603 detects the direction in which the imaging equipment is physically facing and outputs directional information.

The position detection device 1604 detects the location of the imaging equipment within the construction site space and outputs the resulting position information.

The control device 101 of the construction management terminal 100 acquires the position and posture of the imaging equipment and the images captured by the camera 1602 via the transmitting/receiving device 104, the network 200, and the transmitting/receiving device 1601.

The processing performed by the construction management terminal 100 using the information received uses, for example, the image captured by the camera 111 and the information output by the attitude detection device 109 and the position detection device 110 as explained in the first embodiment using FIG. 2, but these are replaced with the image captured by the camera 1602 and the information output by the attitude detection device 1603 and the position detection device 1604 for processing.

The rest of the configuration is the same as in the first embodiment.

The present embodiment, configured as described above, can achieve the same effects as the first embodiment.

In addition, it will be possible to operate operating construction machinery in places where there is no construction management terminal, for example, in remote locations where imaging equipment is present but out of the user's line of sight, eliminating the need for users to carry around a construction management terminal and allowing operations to be performed efficiently.

<Additional Notes>
The present invention is not limited to the above-described embodiment, and includes various modifications and combinations that do not deviate from the gist of the present invention. Furthermore, the present invention is not limited to those having all of the configurations described in the above-described embodiment, and includes those in which some of the configurations are omitted.

For example, in the fifth embodiment, an example is shown in which an operator operates a specified construction machine at a construction site by operating the construction management terminal 100, but this is not limited to the above. A terminal having some of the functions of the construction management terminal 100 (control device 101, display device 102, operation device 103, transmission/reception device 104, status calculation device 112) may be installed in a management office at the construction site, and the construction machine may be operated from the management office using information from the imaging equipment 400.

Furthermore, each of the above configurations, functions, etc. may be realized in part or in whole by, for example, designing them as an integrated circuit. Furthermore, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function.

100...construction management terminal, 101...control device, 102...display device, 103...operation device, 104...transmitting/receiving device, 105...transmitting/receiving device, 106...position detection device, 107...control device, 108...control target, 109...posture detection device, 110...position detection device, 111...camera, 112...state calculation device, 113...operation state detection device, 114...stop possibility determination device, 200...network, 300...construction machine, 1601...transmitting/receiving device, 1602...camera, 1603...posture detection device, 1604...position detection device

Claims (7)

A construction machine operation control system including a construction management terminal that creates control information for operating a construction machine, and a machine control device that controls the operation of the construction machine based on the control information,
A terminal attitude detection device provided in the construction management terminal and detecting an orientation of the construction management terminal;
A terminal position detection device provided in the construction management terminal and detecting the position of the construction management terminal;
A machine position detection device is provided on the construction machine and detects position information of the construction machine within a construction site space ,
The construction management terminal has a terminal camera that photographs the construction site, a terminal display device that displays the construction site photographed by the terminal camera as a surrounding image, a terminal operation device that allows an operator to specify a predetermined position within the surrounding image displayed on the terminal display device, and a terminal control device that creates the control information,
the terminal display device displays the control information created by the terminal control device;
the terminal operation device, when an operator designates at least a part of the control information displayed on the terminal display device, detects coordinates of the designated control information as input coordinate values, and outputs the detected input coordinate values;
The transmission and reception of the control information is performed by a terminal transmission/reception device of the construction management terminal and a machine transmission/reception device of the construction machine,
The machine control device is provided in the construction machine, and outputs a control signal corresponding to the control information input via the machine transmitting/receiving device;
The control signal input from the machine control device operates a control target, which is a control mechanism of the construction machine,
The construction management terminal includes:
calculates a position of a designated area at the construction site corresponding to the predetermined position specified by the terminal operation device based on the predetermined position in the surrounding image corresponding to the input coordinate value specified by an operator using the terminal operation device while the surrounding image output by the terminal camera is displayed on the terminal display device , and based on terminal state information indicating a state of the construction management terminal calculated based on the orientation of the construction management terminal detected by the terminal attitude detection device and the position of the construction management terminal detected by the terminal position detection device;
determining whether the construction machine is present within the designated area based on the position of the designated area and position information of the construction machine detected by the machine position detection device;
A construction machinery operation control system characterized in that, when it is determined that the construction machinery is present within the designated area, stop control information for stopping the construction machinery is created as the control information, and the stop control information is output to the machinery control device. 2. The construction machine operation control system according to claim 1,
a machine operation state detection device that detects the posture of the construction machine and machine operation information consisting of the control information for operating the construction machine;
Further equipped with
the machine control device, the machine position detection device, and the machine operating state detection device are attached to the construction machine,
The terminal control device includes:
Calculating an outer shape of the construction machine from the machine operation information detected by the machine operation state detection device;
determining whether the outer shape is within the designated area based on the position of the designated area, the position information of the construction machine , and the outer shape;
A construction machine operation control system characterized in that, when the outer shape is present within the designated area, it is determined that the construction machine is present within the designated area. 2. The construction machine operation control system according to claim 1,
a machine operation state detection device that detects the posture of the construction machine and machine operation information consisting of the control information for operating the construction machine ;
a stop possibility determination device that determines whether the construction machine can be stopped stably based on the machine operation information detected by the machine operation state detection device;
Further equipped with
the machine control device, the machine position detection device, the machine operation state detection device, and the stop possibility determination device are attached to the construction machine,
The stop possibility determination device includes:
When the machine control device receives the stop control information, it determines whether the construction machine can be stopped stably,
When it is determined that the construction machine can be stopped stably, the machine control device is caused to stop the construction machine;
A construction machine operation control system characterized in that, when it is determined that the construction machine is stable and cannot be stopped, unstoppable information indicating that the construction machine is in an unstoppable state is transmitted to the construction management terminal. 2. The construction machine operation control system according to claim 1,
The terminal control device includes:
A construction machinery operation control system characterized in that when it is determined that there are multiple construction machines within the designated area, stop control information is output to stop the construction machine among the multiple construction machines that is located closest to the construction management terminal based on the position of the construction management terminal detected by the terminal position detection device and the position information of the multiple construction machines detected by the machine position detection device. 2. The construction machine operation control system according to claim 1,
The terminal control device includes:
A construction machinery operation control system characterized by outputting stop control information for stopping all of the construction machines determined to be present within the designated area when it is determined that multiple construction machines are present within the designated area. 2. The construction machine operation control system according to claim 1,
The terminal control device includes:
When it is determined that a plurality of the construction machines are present within the designated area, information of all the construction machines determined to be present within the designated area is output to the terminal display device, and the information of the construction machines is displayed in a list;
A construction machine operation control system characterized by outputting stop control information for stopping a construction machine selected by the operator from the list display by using the terminal operation device among the plurality of construction machines. A construction machine operation control system including a construction management terminal that creates control information for operating a construction machine, and a machine control device that controls the operation of the construction machine based on the control information,
A camera to photograph the construction site,
a camera attitude detection device provided in the camera and detecting an orientation of the camera;
a camera position detection device provided in the camera and detecting a position of the camera;
A machine position detection device is provided on the construction machine and detects position information of the construction machine within a construction site space ,
The construction management terminal has a terminal display device that displays the construction site photographed by the camera as a surrounding image, a terminal operation device for an operator to specify a predetermined position within the surrounding image displayed on the terminal display device, and a terminal control device that creates the control information,
the terminal display device displays the control information created by the terminal control device;
the terminal operation device, when an operator designates at least a part of the control information displayed on the terminal display device, detects coordinates of the designated control information as input coordinate values, and outputs the detected input coordinate values;
The transmission and reception of the control information is performed by a terminal transmission/reception device of the construction management terminal and a machine transmission/reception device of the construction machine,
The machine control device is provided in the construction machine, and outputs a control signal corresponding to the control information input via the machine transmitting/receiving device;
The control signal input from the machine control device operates a control target, which is a control mechanism of the construction machine,
The terminal control device includes:
Calculate the position of a designated area at the construction site corresponding to the predetermined position specified by the terminal operation device based on the predetermined position in the surrounding image corresponding to the input coordinate value specified by an operator using the terminal operation device while the surrounding image output by the camera is displayed on the terminal display device , and based on terminal state information indicating the state of the construction management terminal calculated based on the orientation of the camera detected by the camera attitude detection device and the position of the camera detected by the camera position detection device ;
determining whether the construction machine is present within the designated area based on the position of the designated area and position information of the construction machine detected by the machine position detection device;
A construction machinery operation control system characterized in that, when it is determined that the construction machinery is present within the designated area, stop control information for stopping the construction machinery is created as the control information, and the stop control information is output to the machinery control device.

Images