JP7660922B2 - Information Processing System - Google Patents

Description

The present invention relates to an information processing system.

Patent Document 1 discloses a technology in which a construction site photo album creator takes photos of the construction work using a digital camera, transmits the taken construction photo data from a work site terminal to a server at a management and control center, and classifies and organizes the construction photos and inputs information to explain the construction photos on the server side and/or on an input request terminal requested by the server, all while viewing the contents of an explanatory blackboard displayed enlarged on the display unit.

JP 2011-65321 A

At construction sites and other work sites, digitization is desirable to improve work efficiency, etc.

In view of the above circumstances, the present invention aims to provide an information processing system that can promote the digitization of work sites.

According to one aspect of the present invention, an information processing system is provided. The information processing system includes a processor, and in a site acquisition step, the processor acquires measurement results obtained by a measuring device measuring a site where work related to a structure is being performed, the measuring device having a sensor that measures electromagnetic waves arriving from an object. In an analysis step, a predetermined analysis process is executed based on the acquired measurement results. In a storage step, the results of the executed analysis process are stored.

This type of approach can facilitate the creation of data related to the work site.

1 is a diagram showing the overall configuration of a site management support system 1. FIG. FIG. 2 is a diagram illustrating a hardware configuration of the support server device 10. FIG. 2 is a diagram illustrating a hardware configuration of a field terminal 20. FIG. 2 is a diagram illustrating a hardware configuration of the imaging device 30. 2 is a diagram showing a hardware configuration of the supervisor terminal 40. FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit of the support server device 10. FIG. FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit of each device. FIG. 11 is an activity diagram showing an example of captured image processing. FIG. 13 is a diagram showing an example of a displayed area image. FIG. 11 is a diagram illustrating an example of a specific image database. FIG. 13 is a diagram showing an example of a displayed candidate image. FIG. 11 is a diagram showing an example of displayed candidate information. FIG. 13 is a diagram showing an example of displayed specific image data. FIG. 13 is a diagram showing an example of a displayed area image. FIG. 13 is a diagram illustrating an example of a flow diagram of an automatic extraction process. FIG. 13 is a diagram showing an example of a displayed check image.

The following describes embodiments of the present invention with reference to the drawings. The various features shown in the following embodiments can be combined with each other.

The program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

In this embodiment, a "unit" may also include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, this embodiment handles various types of information, which may be represented, for example, by physical values of signal values representing voltage and current, high and low signal values as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations may be performed on a circuit in the broad sense.

In the broad sense, a circuit is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, and memory. In other words, it includes application specific integrated circuits (ASICs), programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

Figure 1 shows the overall configuration of the site management support system 1. In Figure 1, an overview of each device equipped in the site management support system 1 and the users who use those devices is shown. Each overview will be explained from time to time with reference to other figures.

The site management support system 1 is an information processing system that executes processes to support the management of work at a site such as architecture, construction, or construction. The site management support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power source 5, a breaker 6, a support server device 10, a site terminal 20, an imaging device 30, a supervisor terminal 40, an operator terminal 50, and a business server device 60.

The communication line 2 includes the Internet, etc., and mediates data exchange between devices connected to the line. The support server device 10 and the business server device 60 are connected to the communication line 2 by wire, and the field terminal 20 and the supervisor terminal 40 are connected wirelessly. In this embodiment, the field terminal 20 communicates with the communication line 2 by mobile communication. The field terminal 20 also communicates wirelessly with the imaging device 30 using two communication methods. In this embodiment, the two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication.

The on-site terminal 20 and the imaging device 30 are, for example, terminals used by on-site worker W1, installed at the work site, and perform processing related to on-site photography. The supervisor terminal 40 is, for example, a terminal used by the work supervisor W2 in charge of the work site, and performs processing for managing the progress of work, etc. The operator terminal 50 is, for example, a terminal used by operator W3 of the on-site management support system 1, and accepts various operations related to images captured by the imaging device 30.

The on-site terminal 20 is connected to the external power source 5 via the external battery 4 by a cable. The imaging device 30 is detachably connected to the external power source 5 via the external battery 4 by a cable and a connector 7. In other words, the external power source 5 also supplies power to the detachably connected imaging device 30. The external battery 4 has a so-called pass-through function that allows it to supply power while charging, and when the breaker 6 is on, it supplies power to the on-site terminal 20 and imaging device 30 while being charged with power supplied from the external power source 5.

The imaging device 30 is a digital camera equipped with an image sensor, and captures an image represented by the light measured by the image sensor. In this embodiment, the imaging device 30 is a 360-degree camera (also called an omnidirectional camera or spherical camera) that can capture images in all directions: up, down, left, right, front, back, and so on. The imaging device 30 is attached to a selfie stick 3, which can be inserted into a stand 8 installed at the work site and fixed there.

When the site worker W1 removes the connector 7 and walks around the work site holding the selfie stick 3 removed from the stand 8, captured image data is generated that shows images captured by a 360-degree camera with the work site as the capture area. In this embodiment, the images captured by the imaging device 30 in the site management support system 1 are moving images, but they may also be still images captured continuously as long as images of various parts of the work site can be obtained. The imaging device 30 transmits the generated captured image data to the site terminal 20.

The on-site terminal 20 is a terminal that serves as the main user interface for the on-site worker W1, and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30, for example, using one of the above two communication methods (BLE communication in this embodiment). In addition, the on-site terminal 20 transfers the captured image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in this embodiment) to the support server device 10 using yet another wireless communication (mobile communication in this embodiment).

The support server device 10 performs image processing using the image of the work site shown in the captured image data sent from the on-site terminal 20, and generates area image data that shows, for example, an image of the work site as seen from the position where the image was captured by the imaging device 30 as an image of the captured area. The supervisor terminal 40 refers to the generated area image data and displays images of various parts of the work site. The work supervisor W2 understands the situation at the work site from the displayed image of the work site, and gives instructions on the work to the on-site worker W1 as necessary.

Depending on the work site, the breaker 6 may be turned off after work is completed for reasons such as power saving. In that case, power is no longer supplied from the external power source 5 after the breaker 6 is turned off. In this way, the external power source 5 can switch between supplying power and not supplying power. The on-site terminal 20 and the imaging device 30 also have built-in batteries, so they do not stop immediately, but some processes, such as sending image data, take time. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on-site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

The operator terminal 50 accepts an operation by the operator W3 to extract a portion showing a specific location from an image of the work site captured by the imaging device 30 as a specific image. In this embodiment, the specific location is a location that needs to be checked to confirm the progress of the work process. The support server device 10 extracts the specific image based on the operation of the operator W3 on the operator terminal 50, and outputs specific image data showing the extracted specific image. In this embodiment, the specific image data is a checklist that lists the specific locations. The supervisor terminal 40 displays the output specific image data (checklist), and accepts, for example, an operation to check the work progress by the work supervisor W2.

Figure 2 is a diagram showing the hardware configuration of the support server device 10. The support server device 10 includes a control unit 11, a memory unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects each unit included in the support server device 10.

(Control unit 11)
The control unit 11 is, for example, a central processing unit (CPU) not shown. The control unit 11 is a computer that realizes various functions related to the site management support system 1 by reading out a predetermined program stored in the storage unit 12. That is, information processing by the software stored in the storage unit 12 can be specifically realized by the control unit 11, which is an example of hardware, and executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have multiple control units 11 for each function. Also, a combination of these may be used.

(Memory unit 12)
The memory unit 12 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the site management support system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program calculations. The memory unit 12 stores various programs, variables, etc. related to the site management support system 1 executed by the control unit 11.

(Communication unit 13)
The communication unit 13 is configured to be capable of transmitting various electrical signals from the supporting server device 10 to external components. The communication unit 13 is also configured to be capable of receiving various electrical signals from the external components to the supporting server device 10. More preferably, the communication unit 13 has a network communication function, which allows various information to be communicated between the supporting server device 10 and external devices via the communication line 2.

Like the support server device 10, the business server device 60 has hardware such as a control unit 61 (only the control unit 61 is given a different reference number from that of the support server device 10).

Figure 3 is a diagram showing the hardware configuration of the on-site terminal 20. The on-site terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects the various units included in the on-site terminal 20. The control unit 21 and the memory unit 22 are similar hardware to the control unit 11 and the memory unit 12 shown in Figure 2, although their performance differs.

(Communication unit 23)
The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication in this embodiment as the first wireless communication. The second communication unit 232 performs wireless communication by BLE in this embodiment as the second wireless communication that has a slower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication in this embodiment as the third wireless communication that has a wider communication area than the first wireless communication and the second wireless communication.

(Input unit 24)
The input unit 24 includes keys, buttons, a touch screen, a mouse, etc., and receives input from the user.
(Output unit 25)
The output unit 25 has a display (including a touch screen) and a speaker, and displays visual information generated in a manner that is visible to the user, such as a screen, image, icon, text, etc., on the display surface, and outputs sound including voice.

(Internal power supply unit 26)
The internal power supply unit 26 is a battery built into the device itself, i.e., a repeatedly rechargeable battery, and supplies stored power to each part of the device. The internal power supply unit 26 is an example of a portable battery that can be carried around together with the device itself. The internal power supply unit 26 is charged by power supplied from the external power supply 5. The internal power supply unit 26 has a pass-through function like the external battery 4, and when the breaker 6 is on, it supplies power to each part while being charged by power supplied from the external power supply 5.

Figure 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects the various units included in the imaging device 30. The various units from the control unit 31 to the internal power supply unit 36 are similar hardware to the various units from the control unit 21 to the internal power supply unit 26 shown in Figure 3, although their performance differs.

However, the communication unit 33 only includes a first communication unit 331 and a second communication unit 332. The first communication unit 331, like the first communication unit 231 of the communication unit 23, performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 332, like the second communication unit 232 of the communication unit 23, performs wireless communication by BLE in this embodiment as the second wireless communication which has a slower communication speed and lower power consumption than the first wireless communication. The output unit 35 also has a light in addition to a display etc., and emits light to ensure the amount of light required for shooting. The input unit 34 has a switch for turning on the light.

(Imaging unit 37)
The imaging unit 37 is a sensor that has an optical system including a lens, an image sensor, etc., and measures light incident from the lens to generate captured image data. In this embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and multiple image sensors to generate captured image data captured in all directions, including up and down, left and right, and front and back.

Figure 5 is a diagram showing the hardware configuration of the supervisor terminal 40. The supervisor terminal 40 includes a control unit 41, a memory unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects the various units of the supervisor terminal 40. The units from the control unit 41 to the output unit 45 are similar hardware to the units from the control unit 31 to the output unit 35 shown in Figure 4, although there are differences in performance. The operator terminal 50, like the supervisor terminal 40, includes hardware such as a control unit 51 (only the control unit 51 is given a different reference number from that of the supervisor terminal 40).

2. Functional Configuration In this section, the functional configuration of the present embodiment will be described. As described above, information processing by software stored in the storage unit of each device is specifically realized by a control unit, which is an example of hardware, so that each functional unit included in the control unit can be executed.

Figure 6 is a diagram showing an example of the functional configuration of the control unit of the support server device 10. The control unit 11 of the support server device 10 includes a site acquisition unit 101, a task acquisition unit 102, a site analysis unit 103, a result accumulation unit 104, a dashboard unit 105, and a tool collaboration unit 106.

The site acquisition unit 101 is an example of a site acquisition unit that acquires measurement results obtained by a measuring device measuring a site where work related to a structure is being carried out. A structure is an object constructed using a plurality of materials and supplies, and includes architectural structures, civil engineering structures, ships, and the like. The measuring device is a device having a sensor that measures electromagnetic waves arriving from an object, and in this embodiment, is an imaging device 30 equipped with an image sensor. The imaging device 30 outputs an image measured by the image sensor (hereinafter referred to as a "site image") as a measurement result. The site acquisition unit 101 acquires the output site image as a measurement result.

The work acquisition unit 102 is an example of a work acquisition unit that acquires work information related to the work described above. The work information includes on-site drawing information, a schedule, material information, component information, cost information, and environmental load information. This work information is stored in the business server device 60 as information related to the business. The on-site acquisition unit 101 and the work acquisition unit 102 execute processing in the data acquisition phase in which data related to on-site work is acquired.

The site analysis unit 103 is an example of an analysis unit that executes a predetermined analysis process based on the measurement results (site images in this embodiment) acquired by the site acquisition unit 101 and the work information acquired by the work acquisition unit 102. The predetermined analysis processes include a progress assessment process, a construction capability assessment process, a critical path assessment process, and the like. Details of each process will be explained later.

The result accumulation unit 104 is an example of an accumulation unit that accumulates the results of the analysis process executed by the site analysis unit 103. The result accumulation unit 104 accumulates the results of the analysis process, for example, in association with the site where the site image used in the analysis process was taken and the date and time of taking the site image. The result accumulation unit 104 accumulates the respective analysis results for each type of analysis process (data A, data B, ..., data X in the example of FIG. 6). The site analysis unit 103 and the result accumulation unit 104 execute the process in the database phase in which the acquired data is organized and accumulated.

The dashboard unit 105 is an example of a dashboard function that collectively displays the results of the analysis processing accumulated by the result accumulation unit 104, making it easy to access each result. The tool linkage unit 106 is an example of a linkage function that executes processing to link various tools used in business with the accumulated results of the analysis processing. The dashboard unit 105 and the tool linkage unit 106 execute processing in the data utilization phase that utilizes the accumulated data.

Next, the site images acquired by the site acquisition unit 101 and the work information acquired by the work acquisition unit 102 will be described in detail. As described above, in the site management support system 1, the work supervisor W2 grasps the situation at the site and gives instructions to the site worker W1 on the work, so area image data is repeatedly generated and related information is generated. An example of a series of information processes for generating such information about the work site and an example of a functional configuration for executing the information processing will be described with reference to FIG. 7 onwards.

Figure 7 is a diagram showing an example of the functional configuration of the control unit of each device. The control unit 11 of the support server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, a user reception unit 117, and an association processing unit 118. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a shooting control unit 313, and a transmission control unit 314. The control unit 41 of the supervisor terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

The DB control unit 111 (DB: Database) of the support server device 10 controls the storage of data in the database and the reading of data from the database. The database here may be not only a database stored in the support server device 10, but also a database stored in an external device. The database stores, for example, captured image data showing an image captured by the imaging device 30 and the above-mentioned area image data.

The server display unit 112 executes processing for displaying a system screen related to the site management support system 1 on each terminal. For example, the server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, which is display data, and displays a web page showing the system screen on the supervisor terminal 40, etc. The server display unit 112 may also perform processing such as generating and transmitting display data for an application for using the site management support system 1.

The image acquisition unit 113 acquires an image of the shooting area captured by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. For example, the image processing unit 114 performs processing related to a technology called Visual SLAM (Simultaneous Localization and Mapping), which simultaneously estimates the self-position and creates an environmental map based on the image of the shooting area.

The data generation unit 115 generates data based on the results of image processing by the image processing unit 114. The data generation unit 115 generates the above-mentioned area image data based on, for example, the results of Visual SLAM. The data output unit 116 outputs the data generated by the data generation unit 115. The data output unit 116 outputs, for example, the generated area image data to the supervisor terminal 40. The user reception unit 117 receives a specific operation by the user via a terminal used by the user (for example, the operator terminal 50). The association processing unit 118 executes processing for associating images with information. The association of specific operations with images and information will be explained in detail later.

The display control unit 511 of the operator terminal 50 controls the display processing on the display means of the own device. The operation reception unit 512 receives operations from a user (e.g., operator W3). The operations received by the operation reception unit 512 include the specific operations described above. The display control unit 411 of the supervisor terminal 40 controls the display processing on the display means of the own device. The operation reception unit 412 receives operations from a user (e.g., work supervisor W2).

The display control unit 211 of the on-site terminal 20 controls the display processing on the display means of the device itself. The operation reception unit 212 receives operations from a user (e.g., field worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission processing of captured image data by the device itself and the imaging device 30.

The display control unit 311 of the imaging device 30 controls the display processing on the display means of the device itself. The operation reception unit 312 receives operations from a user (e.g., field worker W1). The shooting control unit 313 controls the shooting processing by the imaging unit 37. The transmission control unit 314 controls the transmission processing of the captured image data by the device itself.

In this section, information processing will be described that is executed by a computer according to a program in the present embodiment. The site management support system 1 executes image processing for performing various processes based on images captured in a shooting area of a work site, for example.

Figure 8 is an activity diagram showing an example of captured image processing. The activity shown in Figure 8 is started when the support server device 10 receives captured image data of a work site captured by the imaging device 30. First, in A11, the support server device 10 acquires multiple images captured by the imaging device 30 using the image acquisition unit 113. Below, these multiple images are also simply referred to as "captured images".

In this embodiment, the captured images are each frame shown in the video data, and are omnidirectional images captured by a 360-degree camera. The captured images are also images captured at each position on the path of movement by the imaging device 30 carried by the site worker W1 while moving around the work site. The work site is an example of an imaging area captured by the imaging device 30. The image acquisition unit 113 is an example of an acquisition unit that acquires the captured images.

<Generation of area image data>
Next, the supporting server device 10 performs processing for generating the above-mentioned area image data. First, in A12, the supporting server device 10 causes the image processing unit 114 to estimate the movement path of the imaging device 30 based on the captured image acquired by the image acquisition unit 113. The image processing unit 114 is an example of an estimation unit. The image processing unit 114 estimates the movement path using VSLAM technology. There are, for example, the following two methods for VSLAM technology.

The first method is an indirect position estimation method (indirect method), which estimates the positions of feature points and the camera position by matching feature points between multiple images. When using the indirect method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the matching of feature points of objects shown in each of the multiple captured image data obtained.

The second method is a direct position estimation method (direct method), which estimates the depth of each pixel and the position of the imaging device 30 by minimizing the error in the luminance values of corresponding pixels between multiple images. When using the direct method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the error in the values of corresponding pixels indicated by each of the multiple acquired captured image data.

Next, in A13, the support server device 10 generates the above-mentioned area image data, i.e., data showing the work site as seen from the position where the image was taken by the imaging device 30, as an imaging area, using the data generation unit 115 based on the captured image acquired in A11 and the movement path estimated in A12. The area image data is output to a terminal such as the supervisor terminal 40. The terminal to which the area image data is output displays the area image shown by the area image data.

Fig. 9 is a diagram showing an example of a displayed area image. In Fig. 9, a display image A11 and a floor plan C11 are shown. The display image A11 is an image showing a part of the area image A1, which is one of a plurality of images (photographed images) taken of the construction site C1. The floor plan C11 is a plan view showing the structure of the construction site C1. The floor plan C11 is shown superimposed on the display image A11. The display image A11 and the floor plan C11 show position images P1, P2, P3, P4, etc., which show the photographing positions along the estimated travel path.

The area image A1 shown in FIG. 9 is an omnidirectional image captured by the imaging device 30 from the shooting position indicated by the position image P9. The display image A11 is a rectangular image showing a certain angle of view within the area image A1. Note that the area image A1 is distorted because the omnidirectional image is represented as a rectangle, but in the display image A11, the distortion has been corrected so that the object is captured in the same way as in a normal photograph.

When an operation is performed to move the display image A11 up and down, left and right, or diagonally, the area image A1 moves in the direction of the operation, and the portion displayed as the display image A11 changes. In addition, when an operation is performed to select the position image shown in the area image A1 and drawing B1, the area image photographed from the shooting position indicated by the selected position image is displayed. In this way, by displaying various parts of the area image at each shooting position indicated by the area image data as a display image, the overall state of the construction site C1 can be confirmed.

<Extracting specific images>
Next, the support server device 10 performs a process for extracting a specific image from the captured image. As described above, the specific image is a portion of the area image that shows a specific location. In this embodiment, the specific image is extracted through a manual process in which the user specifies a specific range of one image included in the captured image, and an automatic process in which candidate images that indicate candidates for the specific range are presented to the user.

First, in A21, the support server device 10 uses the image processing unit 114 to extract candidate images of a specific range from one area image included in the captured image. The image processing unit 114 extracts candidate images based on information about check points that have been determined by the business operator performing the work on-site to improve the quality of the work. In this embodiment, the support server device 10 stores a specific image database that includes information about check points.

Figure 10 is a diagram showing an example of a specific image database. The specific image database DB1 shown in Figure 10 stores check location information, specific image history, the current specific image, the history of specified information, the current specified information, and specific image data. The check location information stores the name and shooting information. The name stores a character string indicating the name of the check location, such as "entrance", "corridor", "stairs", "kitchen wall", and "kitchen floor". The shooting information also stores information indicating the shooting position where the check location can be photographed.

The shooting information is, for example, angle information indicating the angle of view of the check location as viewed from the height at which the imaging device 30 normally shoots. If vertically upward is 0 degrees and vertically downward is 180 degrees, the range information indicates the range of the angle of view, for example, 0 degrees to 30 degrees for a ceiling, 45 degrees to 135 degrees for a wall, and 150 degrees to 180 degrees for a floor. The shooting information also includes a range of distances suitable for shooting the check location. The distance suitable for shooting is, for example, 0 m (directly above or below) for a ceiling or floor, and 1 m to 5 m for a wall, which is the distance at which the check location is captured at an appropriate size. In addition, for a location that faces a specific direction, such as a wall, the direction in which the location faces is also included in the shooting information.

The image processing unit 114 first identifies a shooting position from the estimated travel path where the check point can be photographed. For example, the image processing unit 114 identifies a position on the travel path that is a distance from the check point on the drawing that is suitable for shooting, and in the case of a wall, is in the direction in which the check point faces, as the shooting position. The image processing unit 114 then extracts, from among the area images photographed from the identified position, an image with the angle of view indicated by the shooting information as a candidate image.

In the case of locations with distinctive appearances, such as doors and stairs, pattern images showing those locations may be used as the photographic information. In this case, the image processing unit 114 uses well-known image recognition technology to extract images of locations similar to the pattern image as candidate images. The server display unit 112 transmits the extracted candidate images, for example, to the operator terminal 50. The operator terminal 50 displays the transmitted candidate images by the display control unit 511 at A22.

Fig. 11 is a diagram showing an example of a displayed candidate image. In Fig. 11, the display control unit 511 displays a candidate image selection screen D1. On the candidate image selection screen D1, a character string saying "Please select a specific image from the candidate images" is displayed, along with candidate images E11, E12, and E13 (referred to as "candidate image E10" when there is no need to distinguish between them), selection buttons B11, B12, and B13 (referred to as "selection button B10" when there is no need to distinguish between them), and adjustment buttons B21, B22, and B23 (referred to as "adjustment button B20" when there is no need to distinguish between them).

In this way, the server display unit 112 and the image processing unit 114 function as an example of a first presentation unit that presents candidate images of a specific range to the user as candidate images (candidate images E1, E2, and E3 in the example of FIG. 11). According to this aspect, it is possible to make it easier to select an image of a specific location compared to a case where candidate images are not presented.

The selection button B10 is a button for accepting an operation to select a candidate image displayed in association with that selection button as a specific image. The adjustment button B20 is a button for accepting an operation to adjust the range displayed as a candidate image. These operations are accepted, for example, by the operation acceptance unit 512 of the operator terminal 50, and operation data indicating the operation content is transmitted to the assistance server device 10. The user acceptance unit 117 of the assistance server device 10 accepts the operation indicated by the transmitted operation data as an operation performed by the user.

The user accepting unit 117 functions as an example of a specification accepting unit that accepts a first operation by the user to specify a specific range of one of the images included in the multiple images acquired by the image acquiring unit 113. The user accepting unit 117 accepts, for example, an operation to select a candidate image E10 presented by the server display unit 112 (an operation to press one of the selection buttons B10) as the first operation. The first operation here is an operation to specify the range indicated by the candidate image E10 as a specific range.

Specifically, for example, when an operation of pressing the adjustment button B20 is performed and then an operation of dragging the candidate image E10 is performed, the user acceptance unit 117 accepts the user operation, and the server display unit 112 moves the area image displayed as the candidate image E10 in the dragged direction, causing a previously undisplayed portion of the area image to be displayed. In this way, the user acceptance unit 117 accepts the operation of spatially fine-tuning the image showing a specific location as the first operation.

In addition to the above operations, the user reception unit 117 may also receive, as the first operation, an operation for finely adjusting an image showing a specific location over time. For example, when a user drags the candidate image E1 while pressing a specific button (such as the Ctrl key), the user reception unit 117 receives the user operation as the first operation, and the server display unit 112 displays the same portion of an area image captured earlier or later than the area image displayed as the candidate image E1.

The user reception unit 117 may also accept, as the first operation, an operation to specify a specific range on a screen displaying the area image data shown in FIG. 9. In this case, the data output unit 116 first outputs the area image data generated by the data generation unit 115 to, for example, the operator terminal 50. The display control unit 511 of the operator terminal 50 displays the output area image data. The operation reception unit 512 accepts an operation to specify a specific range from the display image displayed as shown in FIG. 9, and the user reception unit 117 accepts the operation as the first operation.

When operator W3 determines that an appropriate range has been specified as a specific image by the above-mentioned first operation, he or she performs an operation of pressing selection button B10 as a first operation. In A24, the support server device 10 receives a series of first operations by the user reception unit 117, and the image processing unit 114 extracts, as a specific image, a candidate image selected by the received first operation or an area image of the specified specific range. The DB control unit 111 stores the extracted specific image in the specific image database DB1 as the "current specific image" shown in FIG. 10.

In this way, the image processing unit 114 functions as an extraction unit that extracts, as a specific image, a portion showing a specific location in the shooting area from one or more images included in the multiple images acquired by the image acquisition unit 113. In this embodiment, the image processing unit 114 extracts, as the specific image, an image of the range specified by the first operation as described above. According to this aspect, it is possible to reflect the user's judgment in the extraction of the image of the specific location.

When a new captured image is acquired and new area image data is generated, the DB control unit 111 moves the extracted specific image from "current specific image" to "specific image history." In this way, the specific image extracted by the image processing unit 114 is stored as a history image in the specific image history. The image processing unit 114 may extract candidate images based on the history images thus stored. In that case, the image processing unit 114, for example, identifies the shooting position and angle of view of each of the stored history images.

The image processing unit 114 then extracts images with angles of view within the specified range from among the newly acquired captured images taken at shooting positions within the specified range as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this manner, the candidate images may be images extracted based on the history of the extracted specific images. According to this aspect, images that are close to the actual specific image are presented as candidate images, so the usefulness of the candidate images can be increased compared to a case in which the history of the specific images is not taken into consideration.

<Matching of specified information>
Next, the support server device 10 performs a process for associating the extracted specific image with the predetermined information. The predetermined information is information, such as the name of a specific location, that is defined in the site management support system 1 as information for associating with the specific image. In this embodiment, the specific information is associated with the specific image through a manual operation in which the user specifies the predetermined information in association with the specific image, and an automatic process prior to this in which candidate information indicating candidates for the predetermined information is presented to the user.

First, in A31, the support server device 10 uses the association processing unit 118 to determine candidates for the specified information to be associated with the specific image as candidate information. For example, the association processing unit 118 determines the name of the check point information stored in the specific image database DB1 as the candidate information. The server display unit 112 transmits display data for displaying the determined candidate information to, for example, the operator terminal 50. In A32, the operator terminal 50 uses the display control unit 511 to display the candidate information indicated by the transmitted display data.

Figure 12 is a diagram showing an example of displayed candidate information. In Figure 12, the display control unit 511 is displaying a candidate information selection screen D2. The candidate information selection screen D2 displays a character string saying "Please select specific information from the candidate information," a specific image F14, a selection button B31, and a candidate information input field J11. Below the candidate information input field J11, a candidate information list J12 is displayed as a pull-down list, and candidate information selected from the candidate information list J12 is displayed in the candidate information input field J11.

In this way, the server display unit 112 and the association processing unit 118 function as a second presentation unit that presents candidate information indicating candidates for the specified information to the user. According to this aspect, it is possible to reduce the effort required to associate a specific image with the specified information compared to a case in which candidate information is not presented. The operation of selecting candidate information and pressing the selection button B31 is accepted as an operation performed by the user by the user acceptance unit 117 of the support server device 10, similar to the first operation described above.

The user reception unit 117 functions as an example of a correspondence reception unit that receives a second operation by the user to correspond the specific image extracted by the image processing unit 114 with specified information (in the example of FIG. 12, the specific image F14 and the name of the checked area "kitchen wall"). The user reception unit 117 receives, as the second operation, an operation to select candidate information presented by the server display unit 112 (in the example of FIG. 12, the candidate information included in the candidate information list J12). This second operation is an operation to correspond the candidate information with the specific image as specified information.

When operator W3 determines that appropriate candidate information to be associated with the specific image has been selected by the above-mentioned second operation, he/she performs the operation of pressing selection button B31 as the second operation. At A33, the operator terminal 50 accepts these second operations as selection operations of candidate information by the operation acceptance unit 512. At A34, the support server device 10 accepts the series of second operations by the user acceptance unit 117, and stores the candidate information associated with the specific image by the accepted second operations in the specific image database DB1 as the "current specified information" shown in FIG. 10 in association with the specific image by the DB control unit 111.

Then, the data generation unit 115 reads out the "current specific image" and the "current specified information" from the specific image database DB1, and generates a list in which they are associated with each other as specific image data. Now, for example, assume that the specific image data is requested from the supervisor terminal 40. In this case, the support server device 10 outputs the generated specific image data to the supervisor terminal 40 by the data output unit 116 at A34. The supervisor terminal 40 displays the output specific image data by the display control unit 411 at A35.

Figure 13 is a diagram showing an example of displayed specific image data. In the example of Figure 13, the display control unit 411 displays a checklist H1. Checklist H1 displays the names of the check locations "Entrance", "Hallway", "Stairs", "Kitchen Wall", and "Kitchen Floor", specific images F11, F12, F13, F14, and F15 of each check location (referred to as "specific image F10" when not distinguishing between them), and the check results of each check location. The check results include a symbol indicating whether the work is progressing well or not, and a comment.

Specific images F10 include those extracted from images taken on different dates, "mm/d1," "mm/d2," "mm/d3," and "mm/d4." For example, specific images F11-1, F11-2, F11-3, and F11-4 are all images extracted with the "entrance" as a specific location, and are extracted from images taken on "mm/d1," "mm/d2," "mm/d3," and "mm/d4."

In this way, the data output unit 116 outputs a list of multiple specific images (checklist H1 in the example of FIG. 13) as specific image data. These multiple specific images (specific images F11-1, F11-2, F11-3, and F11-4 in the example of FIG. 13, etc.) have a specific location in common but were photographed at different times. According to this aspect, it is easier to grasp the transition in the state of the specific location compared to when only specific images photographed at a single time are output.

Note that, since the on-site worker W1 takes the images using the imaging device 30 while walking, the route he or she moves during the image capture is not the same each time. Therefore, the specific images do not necessarily show the exact same area as the specific location, but it is sufficient that at least a portion of the specific location is shown. This is because by looking at that portion of the image, the situation of the specific location at the work site can be understood.

As described above, the data output unit 116 functions as an example of a specific output unit that outputs specific image data including a specific image extracted by the image processing unit 114. According to this aspect, a list showing specific locations can be easily created. In this embodiment, the data output unit 116 outputs, as specific image data, a list that lists the specific image extracted by the image processing unit 114 and the specific information associated with the specific image by the second operation described above.

When a specific enlargement operation (e.g., a single click) is performed on a specific image F10 included in the checklist H1, the display control unit 411 enlarges and displays the specific image F10. This makes it easier to see the situation at the site. Furthermore, in this embodiment, the specific image F10 included in the checklist H1 is a link to area image data generated based on the captured image from which the specific image F10 was extracted. Furthermore, the specific image F10 is a link to area image data in which a specific location shown in the specific image F10 is displayed as a display image.

At A41, the supervisor terminal 40 accepts a predetermined operation (e.g., double-clicking) on the specific image F10 as a link operation via the operation acceptance unit 412. The operation acceptance unit 412 transmits operation data indicating the specific image F10 for which the link operation has been performed to the assistance server device 10. At A42, the assistance server device 10 outputs the linked area image data to the supervisor terminal 40 via the data output unit 116. At A43, the supervisor terminal 40 displays the output area image data via the display control unit 411.

For example, when a link operation is performed to specific image F11-4 shown in FIG. 13, area image data generated based on the image captured at mm/d4 is output with the entrance as the display image and displayed on the supervisor terminal 40. The supervisor terminal 40 accepts, at A44, an operation to move the location shown as the display image of the area image data as a display operation. For example, since the work supervisor W2 is unable to fully determine the work situation from the specific image F11-4 alone, he determines the work situation by manipulating the area image data in this way to view the surrounding image.

As described above, the data output unit 116 outputs data including a link from an image of a specific location (each specific image F10 in the example of FIG. 13) to an image showing that location among the images shown by the area image data (for example, in the case of specific image F11-4, an image showing the entrance taken at mm/d4) as specific image data. According to this aspect, it is easier to confirm the image around the specific location compared to when the specific image data does not include the above link.

When the work supervisor W2 judges the work status by displaying the area image data, he/she returns to the checklist H1 to check the work status. In this embodiment, the area image data includes a link for switching the display to the specific image data (the checklist H1 in the example of FIG. 13).
Fig. 14 is a diagram showing an example of a displayed area image. In the example of Fig. 14, the display control unit 411 displays a display image A41 that represents a part of the area image A4.

Display image A11 shows link images L14-4 and L15-4 (referred to as "link image L10" when there is no need to distinguish between them). Link image L14-4 is a link to specific image F14-4 shown in FIG. 13. Link image L15-4 is a link to specific image F15-4 shown in FIG. 13. When the operation reception unit 412 of the supervisor terminal 40 receives a link operation to link image L10 at A51, it transmits operation data indicating the operated link image L10 to the assistance server device 10.

At A52, the support server device 10 outputs the specific image data that is the link destination indicated by the transmitted operation data to the supervisor terminal 40 by the data output unit 116. At A53, the supervisor terminal 40 displays the output specific image data by the display control unit 411. When an operation is performed on the link image L14-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F14-4 shown in FIG. 13. When an operation is performed on the link image L15-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F15-4.

As described above, the data output unit 116 outputs data including a link from an image taken at a position corresponding to a specific location to an image of that location in the specific image data as area image data. According to this aspect, even when the area image indicated by the area image data is displayed, it is easier to output the image of the specific location compared to a case where the area image data does not include a link.

Next, at A54, the supervisor terminal 40 accepts the check operation through the operation acceptance unit 412 and transmits the operation data to the assistance server device 10. At A55, the assistance server device 10 uses the DB control unit 111 to store the specific image data reflecting the check results indicated by the transmitted operation data in the specific image database DB1. The captured image processing up to A34 shown in FIG. 8 is executed each time photography is performed at the work site. As a result, the area image data and specific image data are saved and accumulated each time photography is performed.

As described above, the data output unit 116 uses multiple images (photographed images) acquired by the image acquisition unit 113 to output area image data showing the photographed area as seen from each position on the estimated travel path. The data output unit 116 also uses the photographed images to output specific image data, as described above. In this manner, the site management support system 1 utilizes the photographed images for both area image data as shown in FIG. 9 and specific image data as shown in FIG. 13. When photographing images of the photographed area to generate area image data and specific image data, respectively, the effort of photographing is required twice. In this embodiment, the photographed images are shared as described above, and therefore the effort of the work associated with photographing the site can be reduced compared to when the photographed images are not shared.

Furthermore, if a site worker W1 takes still images with a camera with a normal angle of view while moving around the shooting area, taking pictures of specific locations one by one does not necessarily result in an image that makes it easy for the work supervisor W2 to judge the situation, and taking pictures again just because an image is undesirable is very time-consuming. Furthermore, even if the picture is taken successfully, the surrounding situation outside the range of the camera's angle of view cannot be seen. In this embodiment, as described above, a video is taken with a 360-degree camera that captures everything in all directions, and specific locations are extracted, making it possible to select a desired image. Furthermore, because omnidirectional images are highly comprehensive, it is rare that a desired image cannot be selected, and even if a specific location that needs to be extracted arises after shooting, that location can be extracted.

Although the area image data also shows a specific location, it is time-consuming to search for the specific location in the area image shown by the area image data when one wants to see only the required location. In this embodiment, the specific location is extracted and the specific image data is output, making it possible to display an image of the specific location without such time-consuming steps. On the other hand, by outputting not only the specific image data but also the area image data, even if it is difficult to tell from the specific image data which part of the shooting area the specific image shows, the position of the location shown in the specific image can be ascertained from the area image data.

As described above, when matching a specific image with predetermined information, by presenting candidate information, the effort required to match appropriate candidate information can be reduced. In addition, in this embodiment, the check points shown in FIG. 10 are always extracted as the specific image. As a result, when the specific image data is used as a checklist, the specific points are always checked at every work site, ensuring the quality of the work.

<Accumulation of analysis results>
The site acquisition unit 101 shown in Fig. 6 acquires area images and the like at each shooting position indicated by area image data generated by the information processing described in Fig. 7 and subsequent figures as a measurement result of the site by the imaging device 30 (site images in this embodiment). The site acquisition unit 101 acquires images captured by the imaging device 30 capable of capturing images in all directions as a measurement result of the site. According to this aspect, the range shown in the site image is wider than when capturing images by an imaging device that cannot capture images in all directions, so more site information can be digitized.

The work acquisition unit 102 acquires, as work information, drawing data and the like used in the information processing described in FIG. 7 and subsequent figures. In addition to the above, the work acquisition unit 102 acquires, as work information, a process schedule, material information, component information, cost information, environmental load information, and the like stored in the business server device 60. The work information acquired in this way includes schedule information (e.g., a process schedule or work plan, etc.) that indicates the schedule for the work.

The site analysis unit 103 executes a process of analyzing the progress of work as a predetermined analysis process based on the site situation shown in the site image acquired by the site acquisition unit 101 and the schedule shown in the work information acquired by the work acquisition unit 102. For example, the site analysis unit 103 analyzes the site image to determine whether each component part of a structure such as a wall, floor, ceiling, and pillars is completed. The site analysis unit 103 then generates, as information indicating the progress of the site work, the proportion of components that have been completed among the components that are shown to be completed according to the schedule in the work information. According to this aspect, the progress of work can be managed.

The work information also includes information about the worker who will perform the work (e.g., a personnel schedule, etc.). The worker here may be an individual worker or an organization (such as a construction company) that undertakes the work. The site analysis unit 103 executes an analysis process to analyze the construction capabilities of the workers indicated by the work information based on the results of the progress analysis process and the acquired work information. The site analysis unit 103 calculates the amount of work that can be completed in a unit period as the analysis result, for example, based on the amount of work indicated in each worker's schedule and the progress rate of that worker. According to this aspect, the construction capabilities of the workers can be grasped.

The result storage unit 104 stores the results of the analysis process executed by the site analysis unit 103 as described above. As described with reference to FIG. 7 etc., site images are photographed and acquired daily, so new analysis processes are performed daily and the analysis results are stored. According to this embodiment, it is possible to promote the digitization of on-site work compared to a case where a system such as the site management support system 1 is not introduced.

<Example of analysis results>
The site analysis unit 103 may execute various analysis processes other than those described above. For example, the site analysis unit 103 may not only analyze the progress of work on the site drawing indicated by the work information, but also analyze construction errors. For example, the site analysis unit 103 compares the site drawing with a site image (360-degree image) and extracts the difference between the drawing and the actual construction. The site analysis unit 103 utilizes the extracted difference to output work progress data and construction error data indicating the progress of work on the drawing.

The site analysis unit 103 may also output work progress data for a process as an analysis result. For example, the site analysis unit 103 may output work progress data indicating the progress of work for a planned process by combining the work progress data obtained by comparing the site drawings and the site images (360-degree images) with the schedule.

The on-site analysis unit 103 may output the construction ability data of the contractor as the analysis result. For example, the on-site analysis unit 103 analyzes the construction accuracy data of the contractor by combining data related to the contractor or contractor with construction error data. The on-site analysis unit 103 also outputs the construction speed data of the contractor as the analysis result by combining data related to the contractor or contractor, the corresponding work content, and work progress data for the process.

The site analysis unit 103 may also output data on the causes of construction errors as the analysis results. In this case, for example, a user creates corrective instruction data on the system using site images (360-degree images) acquired by the site acquisition unit 101, and the result storage unit 104 stores the created corrective instruction data. The site analysis unit 103 then combines the corrective instruction data with a schedule to output data on the causes of construction errors in a specific construction project as the analysis results. The accumulated data on the causes of construction errors is presented, for example, by the dashboard unit 105 to a construction manager carrying out similar construction projects, together with the accumulated construction error data.

The on-site analysis unit 103 may also output waiting period data as the analysis result. For example, the on-site analysis unit 103 outputs data on waiting periods during which work is not progressing as the analysis result based on work progress data for a process.

The site analysis unit 103 may also output total construction capacity data for a specific area as the analysis result. For example, the site analysis unit 103 outputs total construction capacity data obtained by aggregating construction capacity data for contractors in a specific area as the analysis result. Based on the aggregated total construction capacity data, the balance between demand and supply for each area can be optimized.

The on-site analysis unit 103 may also output optimal personnel allocation as the analysis result. For example, the on-site analysis unit 103 uses accumulated construction ability data of the contractors to calculate the personnel allocation that can minimize the construction period, and outputs the calculated personnel allocation as the analysis result.

The on-site analysis unit 103 may also output the optimal process as the analysis result. For example, the on-site analysis unit 103 calculates a process that can minimize the construction period appropriately based on the accumulated progress status data for the process and taking into account other projects handled by the construction manager, and outputs the calculated optimal process as the analysis result.

The on-site analysis unit 103 may also output an optimal material delivery schedule as the analysis result. For example, the on-site analysis unit 103 calculates an optimized material delivery schedule based on accumulated construction progress data for each process as follows. First, the on-site analysis unit 103 calculates a material delivery schedule that minimizes waiting time by just-in-time material delivery. The on-site analysis unit 103 also calculates a material delivery schedule that always maximizes work space by just-in-time material delivery, improves work efficiency, and minimizes the time spent by on-site workers tidying up materials.

The on-site analysis unit 103 also calculates a material delivery schedule that minimizes the environmental load. Specifically, the on-site analysis unit 103 calculates a material delivery schedule that optimizes the amount of materials and minimizes the amount of waste. The on-site analysis unit 103 also calculates a material delivery schedule that minimizes the number of deliveries and minimizes the amount of emissions. The on-site analysis unit 103 also calculates a material delivery schedule that minimizes the delivery distance and minimizes the amount of emissions.

<Other embodiments>
The shooting area is not limited to the above. For example, it may be inside an existing building, inside a vehicle, or a predetermined area outdoors. Furthermore, the imaging device 30 is not limited to a 360-degree camera, and a wide-angle camera or a camera that shoots at a normal angle of view may be used. Furthermore, the captured images are not limited to moving images, and may be still images that are captured continuously.

In the embodiment, the imaging device 30 is an example of a measuring device, but this is not limiting. The measuring device may be equipped with, for example, a distance image sensor that measures the distance to an object, and output point cloud data indicating the measurement results as measurement data. The measuring device may also be equipped with an infrared sensor that measures the temperature of the object, or a millimeter wave sensor that measures the distance to the object or the speed of the object.

In addition, in the embodiment, the measurement results (in the embodiment, site images) generated by the functions shown in FIG. 7 are analyzed by the site analysis unit 103 and stored by the result storage unit 104, but for example, a predetermined analysis process (such as a process of estimating a travel path) may be performed by the image processing unit 114 and stored by the DB control unit 111. In this case, the image processing unit 114 is an example of a site analysis unit, and the DB control unit 111 is an example of a storage unit. Also, some of the functions shown in FIG. 7, including the image processing unit 114 and the DB control unit 111, will execute processing in the database creation phase shown in FIG. 6.

In the embodiment, the work acquisition unit 102 acquires the information stored in the business server device 60 as the work information as is, but for example, information that has been processed to make it easier to perform a specified analysis process may be acquired as the work information. For example, the work acquisition unit 102 performs a process of classifying the information (drawing information, schedule, material information, etc.) stored in the business server device 60 by site or area as a processing process, and acquires the classified information as the work information.

The area image data is not limited to that shown in FIG. 9. For example, it may be data showing only an area image without a drawing or a position image superimposed. In that case, it is sufficient if the viewpoint can be moved along the movement route by an operation different from the operation of selecting the position image. The specific image data is not limited to that shown in FIG. 13. For example, it may be data in which a specific location is changed each time shooting is performed, or it may be data showing only a specific image extracted from the most recently shot image.

The specified information is not limited to the name of the check location, but may be, for example, the name of the work site (floor number or room number, etc.), the work process, the name of the photographer or work supervisor, the date and time of shooting, the weather at the time of shooting, or the type of imaging device, etc. In this way, the specified information may be information indicating what kind of location the specific location is or what kind of image the specific image is. However, information unrelated to the specific image may also be used as the specified information.

<Candidate image>
The image processing unit 114 may extract candidate images in a manner different from that of the embodiment. For example, when a history of a specific range designated by the operator W3 is accumulated, the image processing unit 114 extracts candidate images based on the history. For example, when the same part of the shooting area is designated as a specific range a predetermined number of times or more, the image processing unit 114 extracts an image including the part as a candidate image.

In addition, when multiple rooms with the same layout are photographed as the photographing area, such as in an apartment building, the image processing unit 114 extracts images of the range specified in the other rooms with the same layout as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this case, the candidate images are images extracted based on the history of the specific range specified by the user. According to this aspect, images similar to the specific images of the range previously specified by the user are presented as candidate images, so that the usefulness of the candidate images can be increased compared to a case where the history of the specific range is not taken into consideration.

<Candidate information>
The association processing unit 118 may determine candidate information in a manner different from that of the embodiment. For example, the association processing unit 118 recognizes a location shown in a specific image extracted by the image processing unit 114, and determines information on the recognized location as candidate information. For example, the association processing unit 118 recognizes doors, stairs, lights, windows, washrooms, etc. by image recognition using a pattern image, and determines their names as candidate information.

In this case, the candidate information is information that is determined based on the location shown in the specific image extracted by the image processing unit 114. According to this aspect, for example, even if the specific location is changed due to the situation at the work site, information that matches that location is presented as candidate information, so the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

The association processing unit 118 may also determine, for example, specific information that has been associated with a specific image in the past as candidate information. The association processing unit 118 may also determine, as candidate information, specific information that has been used in the past not only at the same work site, but also at work sites with the same layout. In either case, the candidate information is information that is determined based on the past specific information. According to this aspect, for example, even specific information that has been added depending on the situation at the work site is presented as candidate information, so that the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

<Image Filter>
In the site management support system 1, both the estimation of the travel path and the extraction of the specific image are performed based on a plurality of images (photographed images) captured by the imaging device 30. At that time, the captured images may be narrowed down to those with better images. The quality of the images may be determined, for example, using a known technique (such as fast Fourier transform, machine learning, or open source) for determining whether a photograph is out of focus.

For example, the image processing unit 114 estimates a travel path based on an image narrowed down based on a first rule from among the multiple images acquired by the image acquisition unit 113. The image processing unit 114 also extracts a specific image from the images narrowed down based on a second rule from among the multiple images acquired by the image acquisition unit 113. The second rule may be a rule different from the first rule.

The first rule, for example, is a rule that narrows down the captured images to those that have a good overall appearance. On the other hand, the second rule is a rule that narrows down the captured images to those that have a good appearance in a specific part. Even a single image may contain a mixture of good and bad parts. As a result, it may happen that the image is good overall but a specific part is bad, or that a specific part is good but the image is bad overall. Therefore, by using the first and second rules as described above, it is possible to increase the accuracy of the estimation of the travel path while extracting a specific image that has a good appearance, thereby achieving both the accuracy of the estimation of the travel path and the quality of the extracted image.

Note that the first and second rules are not limited to the above rules. For example, the first rule is a rule that narrows down images to those with feature points equal to or greater than a threshold, and the second rule is a rule that narrows down images regardless of the number of feature points. The more feature points an image has, the more accurate the self-location estimation using VSLAM becomes. On the other hand, when a specific location is a wall or floor, for example, there will be very few feature points. Therefore, by making the first and second rules different as described above, it is possible to achieve both the estimation accuracy of the travel path and the quality of the extracted image.

The first rule and the second rule may be the same rule. Even in this case, it is possible to improve both the estimation accuracy of the travel path and the quality of the extracted image compared to the case where no narrowing down is performed. The above-mentioned narrowing down of images may also be used when extracting candidate images. In this case, the image processing unit 114 narrows down the captured images based on the second rule (which may be the first rule), and extracts candidate images from the narrowed down captured images, as in the embodiment. In this case, it is also possible to improve the quality of the extracted image.

<Automatic extraction>
In the embodiment, the image processing unit 114 extracts the specific image through a manual operation of the user specifying a specific range, but the specific image may be extracted automatically without manual operation. In that case, the image processing unit 114 may extract the specific image, for example, by using the same method for extracting the candidate image as it is. For example, the image processing unit 114 extracts a new specific image based on the shooting information of the check point shown in FIG. 10, as in the embodiment.

The image processing unit 114 also extracts a new specific image that includes the same location as a specific location, for example, based on the history of extracted specific images. The image processing unit 114 also extracts a new specific image that includes a range that was previously specified as a specific location, based on the history of specific ranges specified by the user.

Figure 15 is a diagram showing an example of a flow diagram of the automatic extraction process. First, in S11, the support server device 10 acquires a captured image by the image acquisition unit 113. Next, in S12, the support server device 10 estimates the movement path of the imaging device 30 by the image processing unit 114. Next, in S13, the support server device 10 generates area image data by the data generation unit 115. Next, in S14, the support server device 10 extracts a specific image from the captured image by the image processing unit 114. Next, in S15, the support server device 10 determines the specified information by the association processing unit 118.

In S14 and S15, unlike the example in FIG. 8, a specific image is extracted and the specified information is determined without any operation by the user. Next, in S16, the support server device 10 generates specific image data by the data generation unit 115. Then, in S17, the support server device 10 outputs the area image data and the specific image data in response to a request from the supervisor terminal 40 or the operator terminal 50.

According to this aspect, the effort required to generate specific image data can be further reduced compared to when candidate images are presented. When extracting specific images, the image processing unit 114 may use the second rule described above to narrow down the captured images, thereby extracting images that look good as specific images. Furthermore, the image processing unit 114 may extract specific images using AI (Artificial Intelligence) technology that performs machine learning using past specific images as training data.

In addition, drawing information such as a floor plan of a work site may contain location information indicating the location of a specific location. In this case, when a specific location is specified from the drawing information, the image processing unit 114 may identify a captured image taken from a shooting position where the location is likely to be included in the angle of view, based on the location information of the specified location and the estimated movement path. Note that the server display unit 112 may present the drawing information and movement path to the user, and the user may identify a similar shooting location, thereby identifying a captured image taken from the identified shooting position.

Then, the image processing unit 114 recognizes the portion of the identified photographed image that shows the specific location, and extracts the recognized portion as the specific image. Note that the server display unit 112 may present the identified photographed image to the user, allowing the user to specify the portion that shows the specific location, and the image processing unit 114 may extract the specified portion as the specific image. According to these aspects, the specific image can be easily extracted if it is a location shown in the drawing information.

<Automatic mapping>
In the embodiment, the association processing unit 118 associates the specific information with the specific image through a manual operation of designating the specific information in association with the specific image, but the specific information may be associated automatically without a manual operation. In that case, the association processing unit 118 determines, for example, the specific information associated with the history of specific images that have captured the same specific location as the extracted specific image, as the specific information to be associated with the extracted specific image.

Furthermore, when the association processing unit 118 determines the predetermined information to be associated with a specific image as described above, it determines predetermined information that exists near the predetermined information as the predetermined information to be associated with an image of another location extracted from the same shooting information as the specific image. For example, when the predetermined information "kitchen wall" is associated with a specific image, the association processing unit 118 determines "kitchen floor" or "kitchen ceiling" that exists near the "kitchen wall" as the predetermined information to be associated with an image of another location (e.g., kitchen floor or ceiling).

The association processing unit 118 may also identify the location shown in the specific image using image recognition technology and determine the name, etc. of the identified location as the specified information. Specifically, the association processing unit 118 identifies the location shown in the specific image using a pattern image associated with the specified information (if the specified information is "entrance", pattern images of the entrance photographed from various angles) and determines the specified information associated with the pattern image of the identified location as the specified information to be associated with the specific image.

According to this aspect, the effort required to generate specific image data can be further reduced compared to when candidate information is presented. The matching processing unit 118 may match specific information using AI technology that performs machine learning using specific information that has been matched to specific images in the past as training data.

In addition, drawing information such as a floor plan of a work site may contain the name of each location as the specified information. In such a case, the correspondence processing unit 118 may estimate the specified information corresponding to the specific location indicated by the specific image from the drawing information. For example, the correspondence processing unit 118 identifies the position of the specific location on the drawing from the shooting position and angle of view information at which the specific image was taken, and determines the name of the place including that position as the specified information.

The server display unit 112 may present the drawing information to the user, and the association processing unit 118 may determine the name of a location specified by the user (operator W3 or work supervisor W2, etc.) from the presented drawing information as the specified information. According to these aspects, if the specified information is shown in the drawing information, association with a specific image can be easily performed.

<Extraction of wide-area images>
When extracting a specific image, the image processing unit 114 may extract an image having a range that is a predetermined ratio (hereinafter referred to as "image ratio") wider than the range in which the specific location is shown as the specific image. In this case, the data generating unit 115 generates data showing the specific location and the surroundings of the location when a predetermined operation (such as a flick) is performed as specific image data. This makes it easy to display the surroundings of a specific location when you want to see the surroundings of the specific location.

The image ratio may be a value according to the size of the photographed area, the usage of the outputted area image data, or the usage of the outputted specific image data. For example, the image processing unit 114 increases the image ratio to extract the specific image as the photographed area becomes larger. The larger the photographed area, the larger the space around the specific location, and therefore the larger the surrounding image to be confirmed.

Furthermore, the image processing unit 114 extracts a specific image with a larger image ratio the higher the usage of the area image data or specific image data. The usage here is higher the longer the usage time or the more frequently the area image data or specific image data is used. The higher the usage, the more likely it is that there will be opportunities to look around a specific location, and the more likely it will be that a situation will arise where you want to look at a wider range. As described above, by varying the image ratio, it is possible to check the situation in a range as wide as necessary, compared to when the image ratio is constant.

<Recommendation of specific images>
A specific image may be recommended. For example, when a specific range is specified by the user, the image processing unit 114 not only extracts an image of the specified range, but also extracts an image that is likely to be used as a specific image from an area spatially adjacent to the range as a recommended image. For example, when a kitchen wall is specified, the image processing unit 114 extracts an image of the kitchen floor and ceiling as a recommended image. Also, when a specific range is specified from an image taken at a certain shooting position, the image processing unit 114 extracts images of the same range from images taken at positions before and after the position as recommended images.

The server display unit 112 presents the recommended image thus extracted to the user together with the specific image. The user performs an operation to instruct the user to designate the recommended image as the specific image in addition to or instead of the specific image. The data generation unit 115 generates specific image data that includes the instructed recommended image as the specific image. According to this aspect, the usefulness of the specific image data can be increased compared to a case in which the recommended image is not presented.

<Check with area image data>
When a check area is extracted as a specific image, the data generation unit 115 may generate area image data that is displayed together with a check image for inputting the check result when the specific area is displayed.

FIG. 16 is a diagram showing an example of a displayed check image. In the example of FIG. 16, the display control unit 411 displays check images M14-4 and M15-4 on the display image A41. Check image M14-4 is an image for inputting the check results of specific image F14-4, and check image M15-4 is an image for inputting the check results of specific image F15-4. When input is made to the check images, the DB control unit 111 also reflects the input content in the check results of the check list H1 shown in FIG. 13. This aspect can reduce the effort required to check specific locations compared to a case where check results can only be input into specific image data.

<Other>
When a new range is designated as a specific range, the image processing unit 114 may extract an image of the same range from a previously captured image as a previous specific image. The data generating unit 115 generates specific image data including the extracted current specific image and the previous specific image. This makes it possible to compare the newly designated range with the previous image.

The data generating unit 115 may also generate specific image data that represents, in a comparable manner, specific images showing common locations extracted from images taken in multiple shooting areas with a common structure (for example, rooms with the same layout in the same apartment building). By arranging comparison targets in this way, it is possible to make it easier to judge the status of the work.

When displaying a checklist such as that shown in FIG. 13, the server display unit 112 may also display the check results for similar specific images from the past. For example, the server display unit 112 displays an image of an entranceway extracted from other captured images and its check results alongside the specific image F11-4 of the entranceway and its check results. This makes it easier to input the check results.

For example, the image processing unit 114 may determine the check result for the specific image being checked based on the check results for similar specific images in the past. For example, the image processing unit 114 determines the check result to be "○" if the work progress is similar to a specific image marked "○", and determines the check result to be "×" if the work progress is similar to a specific image marked "×".

The server display unit 112 may then display the judgment results made by the image processing unit 114 in a checklist, or may display an alert if the check results entered by the user differ from the judgment results made by the image processing unit 114, thereby assisting the user in checking. Furthermore, the DB control unit 111 may reflect the results made by the image processing unit 114 as check results in the checklist H1, even if there is no checking by the user. According to these aspects, it is possible to reduce the effort required to check specific locations, compared to a case in which the check results are not judged by the image processing unit 114.

The imaging device 30 does not have to be installed at the work site. For example, a user such as a field worker W1 takes the field terminal 20 and imaging device 30 to the work site to take pictures, has the imaging device 30 send the captured image data to the field terminal 20, and connects the field terminal 20 to a Wi-Fi router in the office or at home to transfer the captured image data to the support server device 10 (the image data may be transferred via mobile communication). In this case, the field management support system 1 only needs to include a communication line 2, a selfie stick 3, a support server device 10, a field terminal 20, an imaging device 30, a supervisor terminal 40, and an operator terminal 50.

The imaging device 30 may be, for example, a device with a wide-angle camera attached to a helmet. In this case, the imaging device can capture images in other directions, although the vertically downward direction where people are present is a blind spot. The captured image data may be transferred from the imaging device 30 to the on-site terminal 20 via a recording medium such as a MicroSD card, rather than wireless communication, or via a communication cable such as a Type C. The imaging device 30 may transmit the captured image data directly to the support server device 10, or may transfer the captured image data to the support server device 10 via a device other than the on-site terminal 20.

The user who performs the manual tasks described in the embodiment (the tasks of specifying a specific range and specifying specific information in association with a specific image) is not limited to the operator W3. For example, the tasks may be performed by the work supervisor W2 or the on-site worker W1. They may also be performed by other workers belonging to the work business operator or by an outsourced company commissioned by the work business operator.

<Configuration variations>
The configuration shown in FIG. 1 is an example, and other configurations may be used as long as there is no inconvenience in implementation. For example, one device may be distributed to two or more devices, or may be replaced by a cloud computing system. In addition, the functions of one device may be distributed to two or more devices, or the functions of two or more devices may be concentrated and realized by one device. In addition, the operation performed by one function may be distributed to two or more functions, or two or more functions may be integrated into one function. In short, as long as each function required for the entire site management support system 1 is realized, the devices that realize those functions may have any configuration.

Although the above-mentioned embodiment is an information processing device such as the support server device 10, or an information processing system such as the site management support system 1 including the support server device 10, it may be an information processing method. The information processing method includes each step executed by the site management support system 1. The above-mentioned embodiment may be a program. The program causes a computer to execute each step executed by the site management support system 1.

<Other variations>
The output destination of information or data (hereinafter referred to as "information, etc.") may be another device, a display, a memory unit (including a built-in memory unit and an external memory unit), etc. Acquisition of information, etc. includes acquiring information, etc. generated by the device itself, in addition to acquiring information, etc. transmitted from another device. The table in which parameters are associated is not limited to the table shown in the figure, and the number of parameters may be reduced or increased. Furthermore, information, etc. corresponding to parameters may be obtained by a formula, a conditional formula, etc. without using a table.

<Additional Notes>
Furthermore, it may be provided in the following aspects:

(1) An information processing system comprising a processor, in which, in a site acquisition step, the processor acquires measurement results obtained by a measuring device measuring a site where work related to a structure is being performed, the measuring device having a sensor that measures electromagnetic waves arriving from an object, in an analysis step, a predetermined analysis process is executed based on the acquired measurement results, and in a storage step, the information processing system stores the results of the executed analysis process.

This aspect can promote the digitization of on-site work.

(2) In the information processing system described in (1) above, the processor acquires work information related to the work in the work acquisition step, and performs the analysis process based on the acquired measurement results and the acquired work information in the analysis step.

In this manner, a wider variety of items can be digitized.

(3) In the information processing system described in (1) or (2) above, the processor acquires, in the on-site acquisition step, images captured by an imaging device capable of capturing images in all directions as the measurement results.

This approach allows more on-site information to be digitized.

(4) In the information processing system described in (2) above, the work information includes schedule information indicating the schedule of the work, and in the analysis step, the processor executes, as the analysis process, a process of analyzing the progress of the work based on the site conditions indicated by the acquired measurement results and the schedule indicated by the acquired work information.

This allows you to manage the progress of work.

(5) In the information processing system described in (4) above, the work information includes information about the worker performing the work, and in the analysis step, the processor executes, as the analysis process, a process of analyzing the construction capability of the worker indicated by the work information based on the results of the progress analysis process and the acquired work information.

According to this embodiment, the construction capabilities of the worker can be grasped.
Of course, this is not the case.
Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. New embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiments and their modifications are within the scope and gist of the invention, and are also within the scope of the invention and its equivalents as set forth in the claims.

1: On-site management support system 2: Communication line 3: Rod 4: External battery 5: External power supply 6: Breaker 7: Connector 8: Stand 10: Support server device 11: Control unit 20: On-site terminal 21: Control unit 30: Imaging device 31: Control unit 40: Supervisor terminal 41: Control unit 50: Operator terminal 51: Control unit 60: Business server device 61: Control unit 101: On-site acquisition unit 102: Work acquisition unit 103: On-site analysis unit 104: Result accumulation unit 105: Dashboard unit 106: Tool collaboration unit 111: DB control unit 112: Server display unit 113: Image acquisition unit 114: Image processing unit 115: Data generation unit 116: Data output unit 117: User reception unit 118: Correlation processing unit 211: Display control unit 212: Operation reception unit 213: Operation control unit 214 : Transmission control unit 231 : First communication unit 232 : Second communication unit 233 : Third communication unit 311 : Display control unit 312 : Operation reception unit 313 : Photography control unit 314 : Transmission control unit 331 : First communication unit 332 : Second communication unit 411 : Display control unit 412 : Operation reception unit 511 : Display control unit 512 : Operation reception unit

Claims (5)

An information processing system,
a processor, the processor comprising:
In the site acquisition step, an image of the site where the work on the structure is performed is acquired by an imaging device;
In the estimation step, a moving path of the imaging device is estimated based on the acquired photographed images;
In the generation step, on the basis of the acquired photographed image and the estimated moving path, site image data is generated which shows the photographed image photographed by the imaging device from a position on the moving path as an image of the work site as seen from the position;
In the analysis step, a predetermined analysis process is performed based on the generated on-site image data .
In the storage step, the generated on-site image data and a result of the analysis process executed based on the on-site image data are stored.
Information processing system. 2. The information processing system according to claim 1,
The processor,
In the work acquisition step, work information relating to the work is acquired, and the work information includes drawing information of the site;
In the analysis step, a process of analyzing an error in the work is executed as the analysis process based on a difference between a situation of the work site shown in the acquired photographed image and a drawing of the work site shown in the acquired work information.
Information processing system. 3. The information processing system according to claim 2,
The work information includes information about a contractor performing the work,
The processor,
In the analysis step, a process of analyzing the construction accuracy of the constructor indicated by the work information is executed as the analysis process based on a result of the analysis process of the work error and the acquired work information.
Information processing system. 2. The information processing system according to claim 1,
The processor,
In the task acquisition step, task information relating to the task is acquired, and the task information includes schedule information indicating a schedule for the task and information regarding a worker who will perform the task;
In the analyzing step,
A process of analyzing the progress of the work based on the site situation indicated by the acquired photographic images and the schedule indicated by the acquired work information;
A process of analyzing the construction capability of a worker indicated by the work information based on a result of the process of analyzing the progress and the acquired work information;
A process of analyzing a total construction capacity obtained by aggregating the construction capacity of the construction workers in a specific area;
as the analysis process.
Information processing system. 2. The information processing system according to claim 1,
The processor,
In the task acquisition step, task information relating to the task is acquired, and the task information includes schedule information indicating a schedule for the task;
In the analyzing step,
A process of analyzing the progress of the work based on the site situation indicated by the acquired photographic images and the schedule indicated by the acquired work information;
A process of analyzing a material transportation schedule that minimizes waiting time or material preparation time at the site based on a result of the process of analyzing the progress;
as the analysis process.
Information processing system.