JP7762099B2 - Work machine video transmission system - Google Patents

Description

The present invention relates to a video transmission system for a work machine.

In recent years, video transmission systems have come into use, in which multiple cameras are mounted on construction machinery such as hydraulic excavators, and the images of the surrounding area captured by the cameras are sent via a network to a remote terminal and displayed on a monitor. This system can be used for remotely controlling construction machinery and monitoring the surrounding area.

In this type of video transmission system, if the amount of camera video data sent from the work machine (camera) to the remote terminal (monitor) increases and reaches the data communication limit on the communication path, the image on the remote monitor may become distorted (including insufficient resolution) or may be missing. For example, when a hydraulic excavator rotates, the subject being captured by the camera mounted on the rotating body moves significantly, which increases the amount of video data transmitted and can cause distorted or missing images on the monitor.

To solve this type of problem, Patent Document 1 discloses a video transmission system in which multiple cameras that capture surveillance footage of the construction machine's surroundings, an operating lever that remotely controls the construction machine, and a display (monitor) that displays the surveillance footage are communicably connected via a network, and the video transmission system includes an encoder and decoder arranged on the network, a connection/disconnection member (switch) that switches between connecting and disconnecting communication between the encoder and decoder, and a control device that, when an operation signal from the operating lever corresponding to the target speed of the construction machine's actuator is equal to or exceeds an allowable judgment threshold, operates the connection/disconnection member to connect only the encoder and decoder connected to the display (monitor) that displays surveillance footage of the surroundings in the direction the construction machine is driven by the operation signal.

Japanese Patent Application Laid-Open No. 2017-092908

In other words, in Patent Document 1, of the multiple camera images mounted on a single construction machine, which camera images are transmitted to a decoder via a communication line and which are not are selected in accordance with the operator's operation. For example, when a right turn operation is input to the control lever, only the camera image showing the right side of the rotating unit is transmitted from the construction machine to the communication line, and other camera images (for example, camera images showing the left side of the rotating unit) are not transmitted from the construction machine to the communication line. This maintains the quality of the image showing the right side of the rotating unit, which is necessary when turning right.

However, simply limiting the camera footage transmitted within a single construction machine may not be enough to avoid straining communication line capacity. In other words, even if the amount of data transmitted for the camera footage of that construction machine can be reduced, if the amount of data transmitted for the camera footage of other construction machines using the same network (for example, construction machines operating at the same construction site) is large, there is still a risk that the camera footage of that construction machine will be distorted or missing due to the upper limit of the network's communication capacity.

The present invention was made in consideration of the above circumstances, and its purpose is to provide a work machine video transmission system that is less likely to interfere with the remote control of each work machine or the monitoring of its surroundings, even when multiple work machines transmit camera video data to remote locations.

The present application includes a plurality of means for solving the above-mentioned problems, and one example thereof is a video transmission system for work machines comprising a plurality of on-board controllers that are mounted on a plurality of work machines each having a plurality of cameras and that transmit video data from the plurality of cameras to a network, an image management server connected to the network so as to be able to communicate with the plurality of on-board controllers, and a computer connected to the network so as to be able to communicate with the plurality of on-board controllers and that receives the video data from the plurality of cameras transmitted from the plurality of on-board controllers and displays it on a monitor, wherein the image management server determines the priorities of the plurality of on-board controllers based on the work states of the plurality of work machines and a correspondence relationship between the work states of the work machines and priorities that is defined by setting a priority for each work state of the work machine in advance, and the computer calculates an upper limit of the amount of transmission data for each on-board controller based on the priority of the plurality of on-board controllers and transmits the calculated upper limit of the amount of transmission data to each on-board controller; the plurality of on-board controllers each adjusts the image quality of the video data from the plurality of cameras so that the image quality is equal to or less than the upper limit of the amount of transmission data transmitted from the image management server and transmits the image quality adjusted video data to the computer; the plurality of work machines are a plurality of hydraulic excavators, and the working states of the work machines include "loading" which indicates that the hydraulic excavator is loading the bed of a dump truck and "digging" which indicates that the hydraulic excavator is excavating, and the priority defined by the correspondence between the working state and priority of the work machine is set to "loading" highest, and the image management server calculates the upper limit of the amount of transmission data so that the upper limit of the amount of transmission data is larger for on-board controllers with relatively higher priorities .

According to the present invention, the priority of the amount of video data to be transmitted is changed depending on the work status of the work machine, and the quantity and quality of video data corresponding to the work being performed by the work machine is transmitted to the management computer, thereby reducing the possibility of interference with the remote control of the work machine or surrounding monitoring.

1 is a diagram showing the overall configuration of a video transmission system according to a first embodiment of the present invention; FIG. 2 is a schematic diagram of an image management server 200. FIG. 1 is a schematic diagram of a management computer 100. FIG. 2 is a schematic diagram of an in-vehicle controller 300. FIG. 10 is a diagram showing an example of part of data stored in a storage device 312 of an in-vehicle controller 300A in a table format. FIG. 2 is a diagram showing part of the data stored in the storage device 212 of the image management server 200 in table format. FIG. 10 is a diagram showing an example of part of data stored in a storage device 312 of an in-vehicle controller 300B in a table format. FIG. 10 is a diagram showing an example of part of data stored in a storage device 312 of an in-vehicle controller 300C in a table format. 10 is a flowchart of a process executed by the image management server 200. FIG. 10 is a diagram showing an example of an equation used to calculate an upper limit of transmission data. 4 is a flowchart of a process executed by each in-vehicle controller 300 according to the first embodiment. FIG. 10 is a diagram showing the overall configuration of a video transmission system according to a second embodiment of the present invention. FIG. 11 is a diagram showing, in table format, part of the data stored in the storage device 212 of the image management server 200 according to the second embodiment. FIG. 10 is an explanatory diagram for setting the working status of a hydraulic excavator to "digging" or "loading" using position data of the hydraulic excavator and dump truck. 10 is a flowchart of a process executed by the image management server 200 according to the second embodiment. 16 is a detailed flowchart of the process performed in S1012 of FIG. 15; 10 is a flowchart of a process executed by each on-board controller 300 according to the second embodiment.

The following describes an embodiment of the present invention with reference to the drawings.

Note that in the following description, when there are multiple identical components, the reference numeral may be suffixed with an uppercase or lowercase letter, but the multiple components may be collectively referred to without the uppercase or lowercase letter. For example, when there are three identical computers 112A, 112B, and 112C, they may be collectively referred to as computer 112.

First Embodiment
Fig. 1 is a diagram showing the overall configuration of a video transmission system according to a first embodiment of the present invention. The video transmission system of Fig. 1 includes a plurality of on-board controllers 300A, 300B, 300C, and 300D mounted, for example, one on each of a plurality of work machines (hydraulic excavators in the example of Fig. 1) 1A, 1B, 1C, and 1D operating at the same work site, an image management server 200, and at least one management computer (referred to as a "management PC" in Fig. 1) 100A, 100B, 100C, and 100D.

The image management server 200 calculates an upper limit for the amount of transmission data (transmission data upper limit 406 (see Figure 6)) for each on-board controller 300 so that the total amount of video data from multiple cameras 310 (see Figure 4) transmitted from multiple on-board controllers 300 to the management computer 100 is within a predetermined value, and transmits the calculated upper limit for the amount of transmission data (transmission data upper limit 406) to each on-board controller 300.

Each vehicle controller 300 adjusts the image quality of the video data from multiple cameras 310 (see Figure 4) so that the amount of data sent from the image management server 200 is within the upper limit (upper limit 406 for data sent (see Figures 5, 7, and 8)), and sends the image data after the image quality adjustment to the corresponding management computer 100.

The management computer 100 displays the image data (camera images) after image quality adjustment sent from the on-board controller 300 on the monitor 120 (see Figure 3), providing the operator in front of the monitor 120 with a video of the situation around the work machine 1.

Each onboard controller 300 is connected to a wide area network (WAN) via the nearest base station 50A, 50B, for example, using a mobile phone communication standard line. Base stations 50A, 50B are the first base stations through which data transmitted from multiple onboard controllers 300 (including video data from cameras 310) passes along the communication path (WAN) from the onboard controller 300 to the management computer 100. Because radio wave conditions can vary depending on the location of the work machine 1, each work machine 1 (onboard controller 300) may connect to a different base station 50. In the example of Figure 1, three work machines 1A, 1B, and 1C (onboard controllers 300A, 300B, and 300C) are connected to base station 50A, and work machine 1D (onboard controller 300D) is connected to base station 50B.

The WAN is further connected to an image management server 200 and management computers 100A, 100B, 100C, and 100D, and each of the terminals 100, 200, and 300 connected to the WAN can communicate data with each other. The WAN may include the Internet 10, which is a public line network, as shown in Figure 1, or it may consist of only dedicated lines.

Furthermore, each on-board controller 300 and the image management server 200 may be connected to the same local area network (LAN), for example, via Wi-Fi, as shown in FIG. 1. Using a LAN connection makes it easy to identify work machines 1 that are located near the image management server 200 (for example, at the same work site).

Note that, since each terminal 100, 200, 300 is connected to a WAN, the use of a LAN is not required, and it is also possible to use only the WAN. When only the WAN is used, a work machine 1 that is present at the same work site as the image management server 200 can be identified, for example, by comparing position data from a GNSS receiver mounted on the work machine 1 with map data that includes position data for the work site where the image management server 200 is installed.

(Configuration of image management server 200)
2 is a schematic diagram of the image management server 200. The image management server 200 is a server that includes a processor such as a CPU 211, a storage device 212 such as a ROM, RAM, flash memory, or hard disk drive that can store programs and various data that can be executed by the processor, and a communication device 215 that is connected to a WAN and a LAN. The image management server 200 calculates an upper limit value for the amount of data that can be transmitted to each on-board controller 300 and transmits the calculated upper limit value for the amount of data that can be transmitted to the target on-board controller 300. The communication device 215 can be connected to both a LAN and a WAN, and different communication devices may be used for the LAN and the WAN.

(Configuration of management computer 100)
3 is a schematic diagram of each management computer 100. Each management computer 100 is a computer (e.g., a personal computer) that includes a processor such as a CPU 111, a storage device 112 such as ROM, RAM, flash memory, or a hard disk drive in which programs to be executed by the processor and various data can be stored, and a communication device 115 connected to a WAN. Connected to the management computer 100 are a monitor 120 that displays camera image data (described below) transmitted from the corresponding on-board controller 300, and an input device 130 (e.g., a mouse, keyboard, touch panel, etc.) for providing data, instructions, etc. to the management computer 100.

When the management computer 100 is used to remotely control each work machine 1, an operation device (e.g., an operation lever) 150 may be connected to output an operation signal to the work machine 1 (on-board controller 300) to be remotely controlled in response to an input operation. In this embodiment, the operation device 150 is connected to the management computer 100 because the management computer 100 is used for remote control. In this embodiment, there is a one-to-one correspondence between one management computer 100 and one on-board controller 300, and those with the same alphabet (uppercase letter) at the end of their reference numerals correspond. For example, when the operation device 150 of the management computer 100A is operated, the work machine 1A equipped with the corresponding on-board controller 300A operates in accordance with the operation of the operation device 150.

(Configuration of the in-vehicle controller 300)
4 is a schematic diagram of each on-board controller 300. Each on-board controller 300 includes a processor such as a CPU 311, a storage device 312 such as a ROM, RAM, flash memory, or hard disk drive in which programs that can be executed by the processor and various data can be stored, and a communication device 315 connected to a WAN and a LAN.

The on-board controller 300 is also connected via a controller area network (CAN) to, for example, multiple cameras 310a, 310b, 310c mounted on the work machine 1, an engine 305 that drives a hydraulic pump that serves as a hydraulic supply source for hydraulic actuators mounted on the work machine 1, a rotation speed sensor 306 for detecting the rotation speed (engine rotation speed) of the engine 305, multiple solenoid valves (solenoid proportional valves) 302 that generate pilot pressure for control valves that control the hydraulic oil supplied to and discharged from the hydraulic actuators, and multiple pressure sensors 303 that detect the pilot pressures generated by the multiple solenoid valves 302.

Multiple cameras 310a, 310b, 310c are mounted on the work machine 1 and capture images of the surroundings of the work machine 1. The image data captured by each camera 310 is first stored in a storage device 312 in the on-board controller, for example, via the CAN, and then transmitted to a predetermined management computer 100 via the WAN. In this embodiment, the cameras 310 are mounted on the rotating body of each hydraulic excavator (work machine) 1, and each captures an image in a different direction (for example, the front, right side, or left side of the rotating body). The management computer 100 displays the received camera image data on the monitor 120. There is no particular limit to the number of cameras 310 mounted on a single work machine 1, and four or more cameras may be mounted.

When the management computer 100 is used to remotely control the work machine 1, there is a one-to-one correspondence between the on-board controller 300 and the management computer 100, and an operator seated in front of the monitor 120 operates the operation device 150 connected to the management computer 100 while watching the image (camera image) on the monitor 120 to operate the work machine 1. When the operation device 150 is operated, an operation signal corresponding to the operation is transmitted from the management computer 100 to the corresponding on-board controller 300 via the WAN. The on-board controller 300 receives the operation signal and operates the solenoid valve 302 corresponding to the operation signal based on the operation signal. This drives the control valve and operates the hydraulic actuator, allowing the operator to remotely control the work machine 1.

(Data of the in-vehicle controller 300A)
5 is a diagram showing an example of a table of some of the data stored in the storage device 312 of the on-board controller 300A. As shown in this diagram, the storage device 312 of each on-board controller 300 stores work status data 401 indicating the work status of the work machine on which it is mounted (hereinafter sometimes referred to as "own machine"), first camera image quality data 402 indicating the image quality of the video data from the first camera 310a of the own machine to be transmitted to the WAN, second camera image quality data 403 indicating the image quality of the video data from the second camera 310b of the own machine to be transmitted to the WAN, third camera image quality data 404 indicating the image quality of the video data from the third camera 310c of the own machine to be transmitted to the WAN, connection destination data 405 indicating the base station to which the own machine is connected while using the WAN, and transmission data upper limit data 406 indicating the upper limit of the amount of data transmitted to the WAN by the own machine.

Examples of work status data 401 include "loading," which indicates that the vehicle is currently loading soil and sand onto the bed of a dump truck; "digging," which indicates that the vehicle is currently digging soil and sand using the front work implement; "leveling," which indicates that the vehicle is currently leveling the construction surface (finishing work); "cleaning," which indicates that the vehicle is currently cleaning its body; and "moving to garage," which indicates that the vehicle has completed its scheduled work and is currently being moved to the garage.

In this embodiment, the work status data 401 is set by the operator of the management computer 100 (in this embodiment, the operator who remotely controls the work machine 1 using the operation device 150). If the operator has not set the work status after starting up the on-board controller 300, the work status is maintained at the default value of "not specified" (see Figure 6).

Each camera's image quality data 402, 403, 404 is preset with multiple image quality grades, such as high, medium, and low quality. The data volume (transfer speed) for each image quality is roughly 100 Mbps for high quality, 50 Mbps for medium, and 10 Mbps for low quality. Note that in this article, the image quality of each piece of video data is explained as indicating the image quality at the time of shooting, but it is also possible to shoot at a uniform high image quality during shooting, and then compress the image to a lower quality as needed when transmitting over the WAN.

Connection destination data 405 may, for example, include the first base station 50A and second base station 50B shown in Figure 1, and may also store the identifier of the currently connected base station. Generally, there are multiple base stations near a single work site. Since radio wave conditions vary depending on the location of the work machine at the work site, each work machine may connect to a different base station in order to find a base station with better radio wave conditions.

The transmission data upper limit data 406 can be used to set the upper limit of the transfer speed when the device itself transmits video data from multiple cameras 310 to the WAN. For example, the transmission data upper limit of the device itself in Figure 5 is "500 Mbps," indicating that the device itself can use the WAN at a maximum of 500 Mbps.

(Image management server 200 data)
6 is a diagram showing in table form some of the data stored in the storage device 212 of the image management server 200. The storage device 212 of the image management server 200 stores a vehicle information table 501 that compiles information on multiple work machines 1 (for example, work machines at the same work site) for which the image management server 200 manages transmission data upper limits, and a priority table 502 that predefines the correspondence between the work status of the work machine 1 and the priority of transmission data.

The priority table 502 is a table into which a preset priority is input for each work state of the work machine 1. In other words, the priority table 502 defines the correspondence between the work state of the work machine 1 and priority. "Priority" indicates the priority of the amount of data that each onboard controller 300 can transmit to the WAN. In this embodiment, the higher the priority of the work state, the higher the value in the table (priority score 550) is set, and as a result, the higher the priority score of the onboard controller 300 (work machine 1), the relatively larger the upper limit of the amount of data that can be transmitted. Note that the correspondence between work state and priority may be defined by something other than a table.

In the priority table 502 of FIG. 6, "Loading" has a priority score of 5, "Excavating" and "Leveling" have a priority score of 3, "Cleaning" and "Moving to Depot" have a priority score of 2, and if no work state is specified ("No Designation"), the priority score is set to 1. In other words, in this embodiment, "Loading" (loading work), which primarily operates the front work implement and upper rotating body in cooperation with the dump truck, has the highest priority score; "Excavating" and "Leveling," which primarily operate the front work implement, have the second highest priority score; "Cleaning" and "Moving to Depot," which do not actively operate the front work implement, have the third highest priority score; and "No Designation" has the lowest priority score.

The vehicle body information table 501 stores work status data 401, connection destination data 405, priority score data 550, and transmission data upper limit data 406 for each work machine (vehicle body 1A-vehicle body 1D) managed by the image management server 200. The image management server 200 receives the work status data 401 and connection destination data 405 from each on-board controller 300 and stores them in the vehicle body information table 501.

The priority score data 550 is input with a priority score determined by the image management server 200 based on the work status data 401 acquired from each on-board controller 300 and the priority table 502. For example, since the work status of vehicle body 1A is "loading" and the priority score for "loading" in the priority table 502 is "5," the image management server 200 inputs "5" as the priority score data 550 for vehicle body 1A.

When loading work is performed by remote control, the operator needs to accurately view the position of the bucket tip of the hydraulic excavator 1 and the position of the dump truck via the monitor 120, necessitating high-resolution video data from the camera 310. In the case of Figure 6, the upper limit for transmission data for vehicle body 1A during "loading" is "500 Mbps," which is set higher than the upper limit for transmission data for vehicle body 1B (300 Mbps) and vehicle body 1C (200 Mbps), allowing the work machine to be operated with higher resolution.

Details of the calculation of the transmission data upper limit data 406 by the image management server 200 will be explained later using Figure 9.

(Data of the in-vehicle controllers 300B and 300C)
Figure 7 is a diagram showing an example of a table showing some of the data stored in the storage device 312 of the on-board controller 300B. The table items are the same as those of the on-board controller 300A in Figure 5, but in the example shown in this figure, the work status data 401 of the on-board controller 300B (work machine 1B) is set to "digging" and the transmission data upper limit data 406 is set to "300 Mbps".

Figure 8 is an example diagram showing in table format some of the data stored in the storage device 312 of the on-board controller 300C. The table items are the same as those for the on-board controller 300A in Figure 5, but in this example, the work status data 401 for the on-board controller 300C (work machine 1C) is set to "cleaning," and the transmission data upper limit data 406 is set to "200 Mbps." Furthermore, "camera off" has been entered as the image quality data for the third camera 310c, which indicates that the third camera 310c has stopped capturing video data around the work machine.

(Processing of the image management server 200)
Next, the process by which the image management server 200 sets the transmission data upper limit data 406 of each on-board controller 300A-300C to the value shown in Figures 5, 7, and 8 when the working status data 401 and connection destination data 405 of the on-board controllers 300A-C are as shown in Figures 5, 7, and 8 will be explained using a flowchart (Figure 9) of the processing executed by the image management server 200.

Figure 9 is a flowchart of the processing executed by the image management server 200. The image management server 200 starts the processing of the flowchart in Figure 9 at a predetermined interval. When processing starts, the image management server 200 collects information about the vehicle bodies (work machines) 1 that it manages. Specifically, in S1001, the image management server 200 first acquires the identification information of the work machines 1A-1D connected to the same LAN, creates columns in the vehicle body information table 501 for the number of work machines 1A-1D, and assigns work machines (vehicle bodies) 1A-1D to each column. Note that the on-board controllers 300A-300D may also be assigned to each column.

In S1002, the image management server 200 requests each managed work machine 1A-1D (each on-board controller 300A-300D) to send work status data 401, and then stores the work status data 401 of each vehicle (each work machine) 1A-1D sent from each on-board controller 300A-300D in the vehicle information table 501.

In S1003, the image management server 200 requests each managed work machine 1A-1D (each on-board controller 300A-300D) to send connection destination data 405, and then stores the connection destination data 405 for each vehicle (each work machine) 1A-1D sent from each on-board controller 300A-300D in the vehicle information table 501.

In S1101, the image management server 200 extracts the priority score corresponding to the work status data 401 acquired in S1002 from the priority table 502 and stores the extracted priority score in the priority score data 550 of the corresponding vehicle body (work machine) 1A-1D in the vehicle body information table 501. In other words, the priority score becomes the priority of the corresponding on-board controller 300. For example, since the work status of the on-board controller 300A is "loading" as shown in FIG. 5, a priority score of "5" is entered as the priority score data 550 of the vehicle body 1A in the vehicle body information table 501 (see FIG. 6). Similarly, priority score data 550 is extracted for the other vehicle bodies 1B-1D based on the work status data 401 and the priority table 502, and the extracted priority score data 550 is entered into the vehicle body information table 501.

In S1102, the image management server 200 references the connection destination data 405 for each vehicle body 1A-1D in the vehicle body information table 501 to determine whether there are multiple vehicle bodies 1 connected to the same base station 50. If there are multiple work machines 1 connected to the same base station 50, the image management server 200 calculates the transmission data upper limit data 406 for each of the multiple work machines 1 based on the work status data 401 and priority score data 405 of the multiple work machines 1.

The reason for this calculation is that there is an upper limit to the amount of data that can be exchanged by one base station 50. In the case of Figure 6, the data volume of base station 50A is shared among three vehicles (work machines) 1A, 1B, and 1C, so adjustments to the transmission data upper limit are only made to the three vehicles (work machines) 1A, 1B, and 1C that are connected to the same "first base station 50A."

On the other hand, if there are not multiple work machines 1 connected to the same base station 50, there is no need to impose a limit on the amount of data that can be transmitted, so the transmission data upper limit data 406 for the work machines 1 connected to that base station 50 is not calculated. In the case of Figure 6, only the vehicle body 1D is connected to the "second base station 50B," and there is no need to adjust the amount of data that can be transmitted.

When calculating the transmission data upper limit data 406, the image management server 200 uses formula (1) shown in Figure 10. The denominator of formula (1) is the sum of the priority scores of the vehicles to be managed, which in the above example is the sum (10) of the priority scores of the three vehicles 1A-1C. The numerator of formula (1) is the product of the priority score of the work machine (host machine) that is the target of the "transmission data upper limit" calculation and a predetermined transmission data volume (1000 Mbps), which in the above example is the value obtained by multiplying any one of the priority scores 550 of vehicles 1A-1C by 1000 Mbps. It is preferable to set the "predetermined transmission data volume" to a value equal to or less than the upper limit of the transmission data volume that the target base station 50 can handle, and in the above example, it is set to 1000 Mbps, which is the upper limit of the transmission data volume for base station 50A. In this case, the transmission data upper limit 406 for vehicle body (work machine) 1A is 500 Mbps, the transmission data upper limit 406 for vehicle body (work machine) 1B is 300 Mbps, and the transmission data upper limit 406 for vehicle body (work machine) 1C is 200 Mbps. As a result, the higher the priority score of the on-board controllers 300A-300C (vehicle bodies 1A-1C), the higher the calculated upper limit of the transmission data volume, and the transmission data upper limit 406 can be allocated to each on-board controller 300A-300C within the range of the upper limit of the transmission data volume for the first base station 50A.

Note that formula (1) in Figure 10 is merely one example of a formula for calculating the transmission data upper limit 406, and any formula can be used as long as it allows the transmission data upper limit 406 to be larger for in-vehicle controllers 300 with higher priority scores, within the upper limit of the amount of data that a certain base station 50 can process.

Once the calculation of the transmission data upper limit 406 is complete, in S1103 the image management server 200 transmits the transmission data upper limit data 406 to each vehicle body 1A-1C (on-board controller 300A-300C) whose transmission data upper limit was calculated in S1102.

(Processing of the in-vehicle controller 300)
Next, we will explain the flow of processing executed by each on-board controller 300. Fig. 11 shows a flowchart of processing executed by each on-board controller 300 according to this embodiment. When the power supply of the work machine 1 is turned on from off to on, the on-board controller 300 starts the processing of the flowchart in Fig. 11 .

First, in S2001, the in-vehicle controller 300 initializes the tables in the storage device 312 shown in Figures 5, 7, and 8. As a result, each data 401-406 in the tables in the storage device 312 is set to its initial value. The initial value of the work status data 401 is "not specified." The initial values of the image quality data 402, 403, and 404 of the first to third cameras are all "medium image quality," the initial value of the connection destination data 405 is "none," and the initial value of the transmission data limit 406 is "100 Mbps." Note that the initial values of the image quality data 402, 403, and 404 of the first to third cameras and the transmission data limit 406 may be set to other values.

Next, in S2002, the on-board controller 300 performs a setting process for work status data 401 indicating the work status of its own machine. In this embodiment, the setting process is performed by the on-board controller 300 accepting input of work status data 401 from the operator of the corresponding management computer 100 via the operation device 150. While viewing the video data from the camera 310 displayed on the monitor 120, the operator of the management computer 100 operates the operation device 150 to input the work status of the work machine 1 to be remotely controlled into the management computer 100. Based on the work status input by the operator, the management computer 100 generates work status data 401 (e.g., "loading") for the corresponding on-board controller 300 and transmits the generated work status data 401 to the on-board controller 300 via the WAN.

In S2101, the in-vehicle controller 300 checks whether work status data 401 has been input (transmitted) from the corresponding in-vehicle controller 300. If work status data 401 has been input, the in-vehicle controller 300 stores the work status data 401 in a table (S2102) and proceeds to S2203. On the other hand, if work status data 401 has not been input, the process proceeds directly to S2203.

In S2203, the in-vehicle controller 300 determines whether there is a request from the image management server 200 to send the work status data 401. If there is a request, the in-vehicle controller 300 sends the work status data 401 to the image management server 200 (S2204) and proceeds to S2205. On the other hand, if there is no request, the in-vehicle controller 300 proceeds directly to S2205.

In S2205, the in-vehicle controller 300 determines whether there is a request from the image management server 200 to send the connection destination data 405. If there is a request, the in-vehicle controller 300 sends the connection destination data 405 to the image management server 200 (S2206) and proceeds to S2207. On the other hand, if there is no request, the in-vehicle controller 300 proceeds directly to S2207.

In S2207, the in-vehicle controller 300 determines whether or not the transmission data upper limit data 406 has been transmitted from the image management server 200. If the transmission data upper limit data 406 has been transmitted, the in-vehicle controller 300 stores the transmission data upper limit data 406 in a table in the storage device 312 (S2208) and proceeds to S2004. On the other hand, if the transmission data upper limit data 406 has not been transmitted, the process proceeds to S2204.

In S2004, the on-board controller 300 sets the image quality of the video data from the multiple cameras 310a, 310b, and 310c mounted on the vehicle within the transmission data upper limit set in the table in the storage device 312. Image quality is set using a known method. In addition to lowering the image quality of the video data from each camera 310 to match the transmission data upper limit, the amount of data transmitted can also be reduced by setting cameras 310 to suspend filming depending on the work status. The image quality set for each camera 310 here is stored in camera image quality data 402, 403, and 404.

In S2005, the in-vehicle controller 300 captures video data using each camera 310 with image quality according to the settings of the camera image quality data 402, 403, and 404.

In S2006, the on-board controller 300 transmits the video data captured by each camera 310 to the corresponding management computer 100 via the WAN. This causes the video data from the camera 310 to be displayed on the monitor 120 connected to the management computer 100, with image quality corresponding to the work status of the machine itself. The operator can then operate the control device 150 while viewing the video data to remotely control the work machine 1.

After transmitting the video data in S2006, the in-vehicle controller 300 returns to S2101 and repeats the subsequent processing steps according to the flowchart in Figure 11.

(Actions and Effects)
In the video transmission system configured as described above, the image management server 200 determines the priority of the transmission data volume of each on-board controller 300 based on the work status of each work machine 1 and the correspondence (priority table 502) between the work status and priority (priority score 550), and calculates an upper limit value (transmission data upper limit 406) for the transmission data volume of each on-board controller 300 based on the priority (priority score 550) of the transmission data volume of each on-board controller 300 so that the total data volume of video data from the cameras 310 transmitted from the multiple on-board controllers 300 to the network (WAN) is equal to or less than a predetermined value (e.g., 1000 Mbps). The calculated transmission data upper limit 406 is transmitted to the corresponding on-board controller 300, and each on-board controller 300 adjusts the image quality of the video data from the multiple cameras 310 so that the data volume is equal to or less than the data volume specified by the transmission data upper limit 406, and transmits the image quality-adjusted video data via the network to the management computer 100.

In this way, by determining the transmission data upper limit 406 for each on-board controller 300 based on the priority (priority score 550) preset for each work state of the work machine 1, video data with image quality appropriate for the work state of each work machine 1 can be transmitted to the management computer 100. For example, during a "loading operation" in which a hydraulic excavator loads earth and sand into the bed of a dump truck, it is preferable that the operator of the management computer 100 be able to accurately grasp the position of the bucket toe, the position of the dump truck bed, and the load amount on the monitor 120. Therefore, by setting the priority (priority score 550) of the "loading" work state higher than other work states, as in this embodiment, the image quality of the video data from the camera 310 viewed by the operator through the monitor 120 during loading operation is prioritized to be high, thereby enabling remotely controlled loading operations to be carried out smoothly.

In this embodiment, the operator of the management computer 100 sets the work status (work status data 401) of each work machine 1 each time the work changes. However, if the work status data 401 is left at its initial value (unspecified), the priority score 550 is set to the lowest value (1) (see S2001 in Figure 11). Therefore, if the operator performs a high-priority task (e.g., loading) while forgetting to set the work status, the image quality of the image on the monitor 120 is likely to be poor. In other words, according to this embodiment, the image quality of the monitor 120 can make the operator aware that they have forgotten to set the work status, thereby prompting the operator to set the correct work status.

In this embodiment, the image management server 200 is configured to calculate the transmission data limit 406 when there are multiple vehicle controllers 300 connected to a certain base station 50. However, it is also possible to configure the image management server 200 to calculate the transmission data limit 406 when there are multiple vehicle controllers 300 using the same communication route on the WAN.

Second Embodiment
In the first embodiment, the operator of the management computer 100 manually set the working status of each hydraulic excavator 1 (see S2002 in FIG. 11), but the main feature of this embodiment is that the working status of each hydraulic excavator 1 is automatically set based on the position data of the work machine 1 and the map information table 3202 (see FIG. 13).

Figure 12 is an overall configuration diagram of a video transmission system according to a second embodiment of the present invention. This system includes multiple dump trucks 2A, 2B as work machines for transporting soil and sand excavated by a hydraulic excavator 1, and each dump truck 2A, 2B is equipped with an on-board controller 400A, 400B connected to a LAN. Each of the hydraulic excavators 1A, 1B, 1C, 1D and dump trucks 2A, 2B is equipped with an antenna and receiver (collectively referred to as a "positioning device 20") for detecting its own position using a positioning satellite system. The on-board controllers 300, 400 of each work machine 1, 2 store position data calculated by its own positioning device 20. This position data can be transmitted from the on-board controllers 300, 400 of each work machine 1, 2 to the image management server 200 via the LAN.

(Image management server 200 data)
13 is a diagram showing, in table format, some of the data stored in the storage device 212 of the image management server 200 according to the second embodiment of the present invention. As shown in this figure, in addition to the tables 501 and 502 shown in FIG. 6, the storage device 212 of the image management server 200 stores a position information table 3201 and a map information table 3202. The position information table 3201 stores position data for each work machine (hydraulic excavators (vehicle bodies) 1A, 1B, 1C, and 1D and dump trucks (trucks) 2A and 2B) connected to the same LAN as the image management server 200. Each piece of position data is calculated by the positioning device 20, and the position is defined by latitude and longitude. For example, the position of vehicle body 1A is expressed as latitude Gx and longitude Gy.

The map information table 3202 stores position data for multiple areas obtained by dividing a work site into multiple areas. In this embodiment, each area is defined by a rectangle, and the position of each area is defined by the upper left vertex (first vertex) and the lower right vertex (second vertex) of the rectangle, as shown in Figure 14. A working status of the hydraulic excavator 1 is assigned to each area in advance. The working status data 401 for each hydraulic excavator 1 (each on-board controller 300) in the vehicle body information table 501 (Figure 6) is set to the working status assigned to the area in which the excavator is located.

However, it is difficult to distinguish between the two work states, "Excavating" and "Loading," using only the map information table 3202 and the hydraulic excavator 1 position data. Therefore, in this embodiment, the work state is first provisionally set to "Excavating/Loading," and then the distance between the hydraulic excavator 1 and the dump truck 2 is taken into consideration to finally determine whether it is "Excavating" or "Loading" (details will be described later using Figure 16). Other work states (for example, leveling, cleaning, moving to garage, etc.) follow the settings in the map information table 3202.

(Processing of the image management server 200)
15 is a flowchart of the processing executed by the image management server 200 of this embodiment. Here, steps S1011 and S1012, which are different from the flowchart in Fig. 9, will be explained, and other steps that are the same as those in Fig. 9 will be assigned the same reference numerals and will not be explained.

In S1011, the image management server 200 requests each on-board controller 300, 400 (work machine 1, 2) connected to the same LAN to send the position data (their own position data) obtained by the positioning device 20, and then stores the position data of each work machine 1, 2 sent from each on-board controller 300, 400 in the position information table 3201.

In S1012, the image management server 200 calculates the work status data 401 of the managed hydraulic excavator 1 based on the position data of each work machine 1, 2 acquired in S1011 and the data in the map information table 3202, and stores the calculated work status data 401 in the vehicle information table 501 (Figure 6). Specifically, the processing shown in Figure 16 is performed.

Figure 16 is a flowchart showing the details of the processing performed by the image management server 200 in this embodiment at S1012.

In S3501, the image management server 200 selects one hydraulic excavator 1 from the multiple hydraulic excavators 1 under management, extracts the position data of the selected hydraulic excavator 1 (selected excavator) 1 from the position information table 3201, identifies the area in which the selected excavator 1 is located from the map information table, and determines whether the work status assigned to that area is "digging and loading in progress." If the work status assigned to the area is "digging and loading in progress," the process proceeds to S3502; if the work status is any other type (including cases where no work status is assigned), the process proceeds to S3505.

In S3502, the image management server 200 obtains the position data of the selected shovel 1 and the position data of the dump truck 2 closest to the selected shovel 1 from the position information table, and determines whether the distance between them is within a predetermined threshold (e.g., 10 m). In the example of Figure 14, the distance between the selected shovel 1 and dump truck 2 can be calculated from the "latitude Gx, longitude Gy" of shovel 1A and the "latitude Ux, longitude Uy" of dump truck 2A. If the distance between them is within the threshold (10 m), the selected shovel is considered to be loading the closest dump truck 2, and its work status data 401 is set to "loading" (S3503). On the other hand, if the distance exceeds the threshold (10 m), the selected shovel is considered to be excavating, and its work status data 401 is set to "excavating" (S3504).

Note that the threshold value used in S3502 is not limited to the example of 10 m, and can be changed as appropriate depending on the size of the hydraulic excavator or dump truck, the shape of the work site, etc.

In S3505, the image management server 200 determines whether a work status has been assigned to the area in which the selected shovel 1 is located. If a work status has been assigned to the area, that work status is set as the work status data 401 (S3506). If a work status has not been assigned, predetermined work status data (e.g., "unspecified") is set as the work status data 401 (S3607).

In S3508, the image management server 200 determines whether it has checked the areas to which all managed hydraulic excavators 1 belong, and if checking has been completed for all hydraulic excavators 1, it proceeds to S1003; if not, it returns to S3501.

(Processing of the in-vehicle controller 300)
Next, we will explain the flow of processing executed by each on-board controller 300. Figure 17 shows a flowchart of processing executed by each on-board controller 300 according to this embodiment. Here, we will explain S2209 and S2210, which are processes different from those in the flowchart of Figure 11, and other processes that are the same as those in Figure 11 will be assigned the same reference numerals and will not be explained.

In S2209, the in-vehicle controller 300 determines whether there is a request from the image management server 200 to send location data. If there is a request, the in-vehicle controller 300 sends the location data to the image management server 200 (S2210) and proceeds to S2205. On the other hand, if there is no request, the in-vehicle controller 300 proceeds directly to S2205.

Although no explanation using a flowchart will be provided, the on-board controller 400 of the dump truck 2 is configured to transmit position data in response to a request from the image management server 200, just like the on-board controller 300 of the hydraulic excavator 1.

(Actions and Effects)
In this embodiment, the working status of each hydraulic excavator 1 is automatically set based on the position data of the work machine 1 and the map information table 3202 (see FIG. 13 ), which eliminates the need for the operator of the management computer 100 to set the working status, improving work efficiency. In addition, the fact that the working status will no longer be maintained at its initial value due to the operator forgetting to set the working status can also contribute to improving work efficiency.

In addition, in this embodiment, the position data of not only the hydraulic excavator 1 but also the dump truck 2 is referenced, making it possible to accurately classify two work states, "loading" and "excavating," which are difficult to distinguish based on the hydraulic excavator 1 position data alone.

However, the working status of the hydraulic excavator 1 may be set using only the position data of the hydraulic excavator 1 and the map information table, without referencing the position data of the dump truck 2. In this case, "loading" and "digging" are classified using only the position data and the map information table.

(others)
In each of the above embodiments, the case where video data from the camera 310 mounted on the hydraulic excavator 1 is transmitted has been described, but the present invention can also be applied to the case where video data from a camera mounted on a work machine other than the hydraulic excavator 1, such as a wheel loader, a crane, or a dump truck, is transmitted.

In the above embodiments, the case where the work machine 1 is remotely controlled by the management computer 100 has been described, but the present invention can also be applied to cases where the operating status of each work machine 1 is managed based on video data transmitted from each on-board controller 300.

In the second embodiment, the image management server 200 determines the working status of the hydraulic excavator 1 based on the position data of each work machine 1, 2 and the map information table 3202. However, the map information table 3202 may be stored in the on-board controller 300, so that the working status of the hydraulic excavator 1 is determined on the on-board controller 300 side. In this case, for example, the algorithm of each on-board controller 300 may be configured to execute the working status determination process based on the position data instead of the process of S2002 in the flowchart of FIG. 11.

The present invention is not limited to the above-described embodiments, and includes various modifications within the scope of the spirit of the invention. For example, the present invention is not limited to those having all of the configurations described in the above-described embodiments, and also includes those in which some of the configurations are omitted. Furthermore, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.

Furthermore, the components, functions, and execution processes of the above-mentioned management computer 100, image management server 200, and on-board controllers 300, 400 (hereinafter referred to as control devices 100, 200, 300, 400) may be partially or entirely implemented in hardware (e.g., by designing the logic that executes each function as an integrated circuit). Furthermore, the components of the above-mentioned control devices 100, 200, 300, 400 may be implemented as programs (software) that are read and executed by an arithmetic processing device (e.g., a CPU) to implement the functions of the components of the control devices 100, 200, 300, 400. Information related to the programs can be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (magnetic disks, optical disks, etc.).

Furthermore, in the above explanation of each embodiment, the control lines and information lines shown are those that are considered necessary for explaining the embodiment, but this does not necessarily mean that all control lines and information lines related to the product are shown. In reality, it is safe to assume that almost all components are interconnected.

1 (1A, 1B, 1C, 1D)... Hydraulic excavator (work machine), 2 (2A, 2B) Dump truck (work machine), 20... Positioning device, 50 (50A, 50B)... Base station, 100 (100A, 100B, 100C, 100D)... Management computer, 112... Storage device, 115... Communication device, 120... Monitor, 130... Input device, 150... Operation device, 200... Image management server, 212... Storage device, 215... Communication device, 300 (300A, 300B, 300C, 300D)... On-board controller, 302... Solenoid valve, 303... Pressure sensor, 305... Engine 306... RPM sensor, 310 (310a, 310b, 310c)... Camera, 312... Storage device, 315... Communication device, 400 (400A, 400B)... On-board controller, 401... Work status data, 402... First camera image quality data, 403... Second camera image quality data, 404... Third camera image quality data, 405... Connection destination data, 405... Priority score data, 406... Transmission data limit data, 501... Vehicle information table, 502... Priority table, 550... Priority score data, 3201... Location information table, 3202... Map information table

Claims (6)

a plurality of on-board controllers that are mounted on a plurality of work machines each having a plurality of cameras and that transmit video data from the plurality of cameras to a network;
an image management server connected to the network so as to be able to communicate with the plurality of vehicle-mounted controllers;
a computer connected to the network so as to be able to communicate with the plurality of on-board controllers, receiving video data from the plurality of cameras transmitted from the plurality of on-board controllers and displaying the video data on a monitor,
The image management server
determining the priorities of the plurality of on-board controllers based on the work states of the plurality of work machines and a correspondence relationship between the work states of the work machines and priorities that is defined by setting a priority in advance for each work state of the work machines;
calculating an upper limit value for the amount of data transmitted from each of the in-vehicle controllers based on the priority of the in-vehicle controllers so that the total amount of video data from the plurality of cameras transmitted from the plurality of in-vehicle controllers to the computer is equal to or less than a predetermined value;
The calculated upper limit of the amount of data to be transmitted is transmitted to each vehicle controller.
each of the plurality of in-vehicle controllers adjusts the image quality of the video data from the plurality of cameras so that the image quality is within the upper limit value of the amount of data transmitted from the image management server, and transmits the image data after the image quality adjustment to the computer ;
the plurality of work machines are a plurality of hydraulic excavators,
the work status of the work machine includes "loading" indicating that the hydraulic excavator is performing loading work onto the bed of a dump truck, and "excavating" indicating that the hydraulic excavator is performing excavation work,
The priority defined by the correspondence relationship between the work state of the work machine and the priority is set highest for "loading,"
The image management server calculates the upper limit value of the transmission data amount so that the upper limit value of the transmission data amount is larger for an in-vehicle controller with a relatively higher priority.
A video transmission system for a work machine. 2. The video transmission system for a work machine according to claim 1,
The plurality of on-board controllers store position data of their own vehicle calculated by a positioning device mounted on their own vehicle,
The image management server
setting the work status of the plurality of work machines based on the position data of the own machine transmitted from the plurality of on-board controllers and a map information table that defines the work statuses performed by the work machines in a plurality of areas set in the work site where the plurality of work machines are operating;
a video transmission system for a work machine, characterized in that the priorities of the plurality of on-board controllers are determined based on the set work states of the plurality of work machines and the correspondence between the work states of the work machines and priorities. 3. The video transmission system for a work machine according to claim 2 ,
The image management server
when the working state of the hydraulic excavator, which has been set based on the position data and the map information table, is "digging and loading" and when the distance between the hydraulic excavator and the dump truck closest to the hydraulic excavator is equal to or less than a predetermined threshold, the working state of the hydraulic excavator is set to "loading,"
and when the working status of the hydraulic excavator, which is set based on the position data and the map information table, is "digging and loading," and when the distance between the hydraulic excavator and the nearest dump truck exceeds the threshold value, the working status of the hydraulic excavator is set to "digging." 2. The video transmission system for a work machine according to claim 1,
a communication path from the plurality of on-board controllers to the computer through which the video data from the plurality of cameras transmitted from the plurality of on-board controllers first passes is the same base station. 2. The video transmission system for a work machine according to claim 1,
an operation device is connected to the computer and outputs an operation signal via the network to a work machine on which the image data from the plurality of cameras is displayed on the monitor;
The operation device is operated by an operator who watches the video data from the plurality of cameras displayed on the monitor. 2. The video transmission system for a work machine according to claim 1,
The working state of the work machine is set by an operator,
the work state of the work machine includes “not specified,” which indicates that the work state has not been set by the operator after the on-board controller is started,
The priority defined in the correspondence relationship between the work state of the work machine and the priority is set to "not specified" as the lowest.
A video transmission system for a work machine.