JP7705152B2 - Information Processing System - Google Patents

Description

The present invention relates to an information processing system.

In Patent Document 1, a mobile terminal stores a construction photo checklist including the photographed drawings of each construction project to be photographed, the construction name of each construction project to be photographed, and the construction design values. Shooting data is created including the photographed drawings of each construction project selected from the stored photographed drawings of each construction project and each construction project in the construction photo checklist, the construction name, the construction design values, and the actual measurement values of the construction project that are input. Then, photo data of the construction project is obtained according to the camera shooting operation while the shooting data is displayed. The photo number and actual measurement values of the photographed construction project are written into the construction photo checklist. Finally, a technology is disclosed in which the photo data of each construction project and the updated construction photo checklist are sent to an external construction photo editing device.

JP 2006-146682 A

At construction sites, etc., work is carried out that involves photographing specific locations on the site, for example to record them. The images captured can be used in a variety of ways, such as to show the on-site situation, but a considerable amount of effort is required to obtain the images necessary for that purpose.

In consideration of the above circumstances, the present invention aims to reduce the amount of work involved in photographing the site.

According to one aspect of the present invention, an information processing system is provided. The information processing system includes a processor capable of executing a program to perform the following steps. In the acquisition step, a plurality of images are acquired, which are captured by the imaging device at each position on the movement path while the imaging device moves through the shooting area. In the estimation step, the movement path is estimated based on the plurality of acquired images. In the area output step, area image data showing the shooting area as seen from each position on the estimated movement path is output using the plurality of acquired images. In the extraction step, a portion showing a specific location in the shooting area is extracted as a specific image from one or more images included in the plurality of acquired images. In the specific output step, specific image data including the extracted specific image is output.

This approach can reduce the amount of work required to photograph the site.

1 is a diagram showing the overall configuration of a site management support system 1. FIG. FIG. 2 is a diagram illustrating a hardware configuration of the server device 10. FIG. 2 is a diagram illustrating a hardware configuration of a field terminal 20. FIG. 2 is a diagram illustrating a hardware configuration of the imaging device 30. 2 is a diagram showing a hardware configuration of the supervisor terminal 40. FIG. FIG. 2 is a diagram illustrating the functional configuration of a control unit of each device. FIG. 11 is an activity diagram showing an example of captured image processing. FIG. 13 is a diagram showing an example of a displayed area image. FIG. 11 is a diagram illustrating an example of a specific image database. FIG. 13 is a diagram showing an example of a displayed candidate image. FIG. 11 is a diagram showing an example of displayed candidate information. FIG. 13 is a diagram showing an example of displayed specific image data. FIG. 13 is a diagram showing an example of a displayed area image. FIG. 13 is a diagram illustrating an example of a flow diagram of an automatic extraction process. FIG. 13 is a diagram showing an example of a displayed check image.

The following describes embodiments of the present invention with reference to the drawings. The various features shown in the following embodiments can be combined with each other.

The program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

In this embodiment, a "unit" may also include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, this embodiment handles various types of information, which may be represented, for example, by physical values of signal values representing voltage and current, high and low signal values as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations may be performed on a circuit in the broad sense.

In the broad sense, a circuit is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, and memory. In other words, it includes application specific integrated circuits (ASICs), programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

1. Hardware Configuration In this section, the hardware configuration of the construction support system according to this embodiment will be described.

Figure 1 shows the overall configuration of the site management support system 1. In Figure 1, an overview of each device equipped in the site management support system 1 and the users who use those devices is shown. Each overview will be explained from time to time with reference to other figures.

The site management support system 1 is an information processing system that executes processes to support the management of work at a site such as architecture, construction, or construction. The site management support system 1 includes a communication line 2, a selfie stick 3, an external battery 4, an external power source 5, a breaker 6, a server device 10, a site terminal 20, an imaging device 30, a supervisor terminal 40, and an operator terminal 50.

The communication line 2 includes the Internet, etc., and mediates data exchange between devices connected to the line itself. The server device 10 is connected to the communication line 2 by wire, and the field terminal 20 and supervisor terminal 40 are connected wirelessly. In this embodiment, the field terminal 20 communicates with the communication line 2 by mobile communication. The field terminal 20 also communicates wirelessly with the imaging device 30 using two communication methods. In this embodiment, the two communication methods are Wi-Fi communication and BLE (Bluetooth (registered trademark) Low Energy) communication.

The on-site terminal 20 and the imaging device 30 are, for example, terminals used by on-site worker W1, installed at the work site, and perform processing related to on-site photography. The supervisor terminal 40 is, for example, a terminal used by the work supervisor W2 in charge of the work site, and performs processing for managing the progress of work, etc. The operator terminal 50 is, for example, a terminal used by operator W3 of the on-site management support system 1, and accepts various operations related to images captured by the imaging device 30.

The on-site terminal 20 is connected to the external power source 5 via the external battery 4 by a cable. The imaging device 30 is detachably connected to the external power source 5 via the external battery 4 by a cable and a connector 7. In other words, the external power source 5 also supplies power to the detachably connected imaging device 30. The external battery 4 has a so-called pass-through function that allows it to supply power while charging, and when the breaker 6 is on, it supplies power to the on-site terminal 20 and imaging device 30 while being charged with power supplied from the external power source 5.

The imaging device 30 is a digital camera equipped with an image sensor, and captures an image represented by the light measured by the image sensor. In this embodiment, the imaging device 30 is a 360-degree camera (also called an omnidirectional camera or spherical camera) that can capture images in all directions: up, down, left, right, front, back, and so on. The imaging device 30 is attached to a selfie stick 3, which can be inserted into a stand 8 installed at the work site and fixed there.

When the site worker W1 removes the connector 7 and walks around the work site holding the selfie stick 3 removed from the stand 8, captured image data is generated that shows images captured by a 360-degree camera with the work site as the capture area. In this embodiment, the images captured by the imaging device 30 in the site management support system 1 are moving images, but they may also be still images captured continuously as long as images of various parts of the work site can be obtained. The imaging device 30 transmits the generated captured image data to the site terminal 20.

The on-site terminal 20 is a terminal that serves as the main user interface for the on-site worker W1, and is, for example, a smartphone. The on-site terminal 20 controls the operation of the imaging device 30, for example, using one of the above two communication methods (BLE communication in this embodiment). In addition, the on-site terminal 20 transfers the captured image data transmitted from the imaging device 30 by one of the above two communication methods (Wi-Fi communication in this embodiment) to the server device 10 using yet another wireless communication (mobile communication in this embodiment).

The server device 10 performs image processing using the image of the work site shown in the captured image data sent from the on-site terminal 20, and generates area image data that shows, for example, an image of the work site as seen from the position where the image was captured by the imaging device 30 as an image of the capture area. The supervisor terminal 40 references the generated area image data and displays images of various parts of the work site. The work supervisor W2 understands the situation at the work site from the displayed image of the work site, and gives instructions on the work to the on-site worker W1 as necessary.

Depending on the work site, the breaker 6 may be turned off after work is completed for reasons such as power saving. In that case, power is no longer supplied from the external power source 5 after the breaker 6 is turned off. In this way, the external power source 5 can switch between supplying power and not supplying power. The on-site terminal 20 and the imaging device 30 also have built-in batteries, so they do not stop immediately, but some processes, such as sending image data, take time. Therefore, in this embodiment, an external battery 4 is provided to increase the operating time of the on-site terminal 20 and the imaging device 30 after the breaker 6 is turned off.

The operator terminal 50 accepts an operation by the operator W3 to extract a portion showing a specific location from an image of the work site captured by the imaging device 30 as a specific image. In this embodiment, the specific location is a location that needs to be checked to confirm the progress of the work process. The server device 10 extracts the specific image based on the operation of the operator W3 on the operator terminal 50, and outputs specific image data showing the extracted specific image. In this embodiment, the specific image data is a checklist that lists the specific locations. The supervisor terminal 40 displays the output specific image data (checklist), and accepts, for example, an operation to check the work progress by the work supervisor W2.

Figure 2 is a diagram showing the hardware configuration of the server device 10. The server device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a bus 14. The bus 14 electrically connects each unit included in the server device 10.

(Control unit 11)
The control unit 11 is, for example, a central processing unit (CPU) not shown. The control unit 11 is a computer that realizes various functions related to the site management support system 1 by reading out a predetermined program stored in the storage unit 12. That is, information processing by the software stored in the storage unit 12 can be specifically realized by the control unit 11, which is an example of hardware, and executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being single, and may be implemented to have multiple control units 11 for each function. Also, a combination of these may be used.

(Storage unit 12)
The memory unit 12 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the site management support system 1 executed by the control unit 11, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program calculations. The memory unit 12 stores various programs, variables, etc. related to the site management support system 1 executed by the control unit 11.

(Communication unit 13)
The communication unit 13 is configured to be capable of transmitting various electrical signals from the server device 10 to external components. The communication unit 13 is also configured to be capable of receiving various electrical signals from the external components to the server device 10. More preferably, the communication unit 13 has a network communication function, which allows various information to be communicated between the server device 10 and external devices via the communication line 2.

Figure 3 is a diagram showing the hardware configuration of the on-site terminal 20. The on-site terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an input unit 24, an output unit 25, an internal power supply unit 26, and a bus 27. The bus 27 electrically connects the various units included in the on-site terminal 20. The control unit 21 and the memory unit 22 are similar hardware to the control unit 11 and the memory unit 12 shown in Figure 2, although their performance differs.

(Communication unit 23)
The communication unit 23 includes a first communication unit 231, a second communication unit 232, and a third communication unit 233, and is an example of a wireless communication unit that performs three types of wireless communication. The first communication unit 231 performs wireless communication by Wi-Fi communication in this embodiment as the first wireless communication. The second communication unit 232 performs wireless communication by BLE in this embodiment as the second wireless communication that has a slower communication speed and lower power consumption than the first wireless communication. The third communication unit 233 performs wireless communication by mobile communication in this embodiment as the third wireless communication that has a wider communication area than the first wireless communication and the second wireless communication.

(Input unit 24)
The input unit 24 includes keys, buttons, a touch screen, a mouse, etc., and receives input from the user.
(Output unit 25)
The output unit 25 has a display (including a touch screen) and a speaker, and displays visual information generated in a manner that is visible to the user, such as a screen, image, icon, text, etc., on the display surface, and outputs sound including voice.

(Internal power supply section 26)
The internal power supply unit 26 is a battery built into the device itself, i.e., a repeatedly rechargeable battery, and supplies stored power to each part of the device. The internal power supply unit 26 is an example of a portable battery that can be carried around together with the device itself. The internal power supply unit 26 is charged by power supplied from the external power supply 5. The internal power supply unit 26 has a pass-through function like the external battery 4, and when the breaker 6 is on, it supplies power to each part while being charged by power supplied from the external power supply 5.

Figure 4 is a diagram showing the hardware configuration of the imaging device 30. The imaging device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, an output unit 35, an internal power supply unit 36, an imaging unit 37, and a bus 38. The bus 38 electrically connects the various units included in the imaging device 30. The various units from the control unit 31 to the internal power supply unit 36 are similar hardware to the various units from the control unit 21 to the internal power supply unit 26 shown in Figure 3, although their performance differs.

However, the communication unit 33 only includes a first communication unit 331 and a second communication unit 332. The first communication unit 331, like the first communication unit 231 of the communication unit 23, performs wireless communication by Wi-Fi communication as the first wireless communication in this embodiment. The second communication unit 332, like the second communication unit 232 of the communication unit 23, performs wireless communication by BLE in this embodiment as the second wireless communication which has a slower communication speed and lower power consumption than the first wireless communication. The output unit 35 also has a light in addition to a display etc., and emits light to ensure the amount of light required for shooting. The input unit 34 has a switch for turning on the light.

(Imaging unit 37)
The imaging unit 37 is a sensor that has an optical system including a lens, an image sensor, etc., and measures light incident from the lens to generate captured image data. In this embodiment, as described above, the imaging unit 37 uses an ultra-wide-angle lens and multiple image sensors to generate captured image data captured in all directions, including up and down, left and right, and front and back.

Figure 5 is a diagram showing the hardware configuration of the supervisor terminal 40. The supervisor terminal 40 includes a control unit 41, a memory unit 42, a communication unit 43, an input unit 44, an output unit 45, and a bus 46. The bus 46 electrically connects the various units of the supervisor terminal 40. The units from the control unit 41 to the output unit 45 are similar hardware to the units from the control unit 31 to the output unit 35 shown in Figure 4, although there are differences in performance. The operator terminal 50, like the supervisor terminal 40, includes hardware such as a control unit 51 (only the control unit 51 is given a different reference number from that of the supervisor terminal 40).

2. Functional Configuration In this section, the functional configuration of the present embodiment will be described. As described above, information processing by software stored in the storage unit of each device is specifically realized by a control unit, which is an example of hardware, so that each functional unit included in the control unit can be executed.

Figure 6 is a diagram showing the functional configuration of the control unit of each device. The control unit 11 of the server device 10 includes a DB control unit 111, a server display unit 112, an image acquisition unit 113, an image processing unit 114, a data generation unit 115, a data output unit 116, a user reception unit 117, and an association processing unit 118. The control unit 21 of the on-site terminal 20 includes a display control unit 211, an operation reception unit 212, an operation control unit 213, and a transmission control unit 214. The control unit 31 of the imaging device 30 includes a display control unit 311, an operation reception unit 312, a shooting control unit 313, and a transmission control unit 314. The control unit 41 of the supervisor terminal 40 includes a display control unit 411 and an operation reception unit 412. The control unit 51 of the operator terminal 50 includes a display control unit 511 and an operation reception unit 512.

The DB control unit 111 (DB: Database) of the server device 10 controls the storage of data in the database and the reading of data from the database. The database referred to here may be not only a database stored in the server device 10, but also a database stored in an external device. The database stores, for example, captured image data showing an image captured by the imaging device 30 and the above-mentioned area image data.

The server display unit 112 executes processing for displaying a system screen related to the site management support system 1 on each terminal. For example, the server display unit 112 performs processing such as generating and transmitting an HTML (Hyper Text Markup Language) file, which is display data, and displays a web page showing the system screen on the supervisor terminal 40, etc. The server display unit 112 may also perform processing such as generating and transmitting display data for an application for using the site management support system 1.

The image acquisition unit 113 acquires an image of the shooting area captured by the imaging device 30. The image processing unit 114 executes image processing on the image acquired by the image acquisition unit 113. For example, the image processing unit 114 performs processing related to a technology called Visual SLAM (Simultaneous Localization and Mapping), which simultaneously estimates the self-position and creates an environmental map based on the image of the shooting area.

The data generation unit 115 generates data based on the results of image processing by the image processing unit 114. The data generation unit 115 generates the above-mentioned area image data based on, for example, the results of Visual SLAM. The data output unit 116 outputs the data generated by the data generation unit 115. The data output unit 116 outputs, for example, the generated area image data to the supervisor terminal 40. The user reception unit 117 receives a specific operation by the user via a terminal used by the user (for example, the operator terminal 50). The association processing unit 118 executes processing for associating images with information. The association of specific operations with images and information will be explained in detail later.

The display control unit 511 of the operator terminal 50 controls the display processing on the display means of the own device. The operation reception unit 512 receives operations from a user (e.g., operator W3). The operations received by the operation reception unit 512 include the specific operations described above. The display control unit 411 of the supervisor terminal 40 controls the display processing on the display means of the own device. The operation reception unit 412 receives operations from a user (e.g., work supervisor W2).

The display control unit 211 of the on-site terminal 20 controls the display processing on the display means of the device itself. The operation reception unit 212 receives operations from a user (e.g., field worker W1). The operation control unit 213 controls the operation of the imaging device 30. The transmission control unit 214 controls the transmission processing of captured image data by the device itself and the imaging device 30.

The display control unit 311 of the imaging device 30 controls the display processing on the display means of the device itself. The operation reception unit 312 receives operations from a user (e.g., field worker W1). The shooting control unit 313 controls the shooting processing by the imaging unit 37. The transmission control unit 314 controls the transmission processing of the captured image data by the device itself.

In this section, information processing will be described that is executed by a computer according to a program in the present embodiment. The site management support system 1 executes image processing for performing various processes based on images captured in a shooting area of a work site, for example.

Figure 7 is an activity diagram showing an example of captured image processing. The activity shown in Figure 7 is started when the server device 10 receives captured image data of a work site captured by the imaging device 30. First, in A11, the server device 10 acquires multiple images captured by the imaging device 30 using the image acquisition unit 113. Below, these multiple images are also simply referred to as "captured images".

In this embodiment, the captured images are each frame shown in the video data, and are omnidirectional images captured by a 360-degree camera. The captured images are also images captured at each position on the path of movement by the imaging device 30 carried by the site worker W1 while moving around the work site. The work site is an example of an imaging area captured by the imaging device 30. The image acquisition unit 113 is an example of an acquisition unit that acquires the captured images.

<Generation of area image data>
Next, the server device 10 performs processing for generating the above-mentioned area image data. First, in A12, the server device 10 causes the image processing unit 114 to estimate the movement path of the imaging device 30 based on the captured image acquired by the image acquisition unit 113. The image processing unit 114 is an example of an estimation unit. The image processing unit 114 estimates the movement path using VSLAM technology. There are, for example, the following two methods for VSLAM technology.

The first method is an indirect position estimation method (indirect method), which estimates the positions of feature points and the camera position by matching feature points between multiple images. When using the indirect method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the matching of feature points of objects shown in each of the multiple captured image data obtained.

The second method is a direct position estimation method (direct method), which estimates the depth of each pixel and the position of the imaging device 30 by minimizing the error in the luminance values of corresponding pixels between multiple images. When using the direct method, the image processing unit 114 estimates the movement path of the imaging device 30 based on the error in the values of corresponding pixels indicated by each of the multiple acquired captured image data.

Next, in A13, the server device 10 generates the above-mentioned area image data, i.e., data showing the work site as seen from the position where the imaging device 30 took the image, as an imaging area, using the data generation unit 115 based on the captured image acquired in A11 and the movement path estimated in A12. The area image data is output to a terminal such as the supervisor terminal 40. The terminal to which the area image data has been output displays the area image shown by the area image data.

Figure 8 is a diagram showing an example of a displayed area image. In Figure 8, a display image A11 and a floor plan C11 are shown. The display image A11 is an image showing a part of the area image A1, which is one of multiple images (photographed images) taken of the construction site C1. The floor plan C11 is a plan view showing the structure of the construction site C1. The floor plan C11 is shown superimposed on the display image A11. The display image A11 and the floor plan C11 show position images P1, P2, P3, P4, etc., which indicate the photographing positions along the estimated travel path.

The area image A1 shown in FIG. 8 is an omnidirectional image captured by the imaging device 30 from the shooting position indicated by the position image P9. The display image A11 is a rectangular image showing a certain angle of view within the area image A1. Note that the area image A1 is distorted because the omnidirectional image is represented as a rectangle, but in the display image A11, the distortion has been corrected so that the object is captured in the same way as in a normal photograph.

When an operation is performed to move the display image A11 up and down, left and right, or diagonally, the area image A1 moves in the direction of the operation, and the portion displayed as the display image A11 changes. In addition, when an operation is performed to select the position image shown in the area image A1 and drawing B1, the area image photographed from the shooting position indicated by the selected position image is displayed. In this way, by displaying various parts of the area image at each shooting position indicated by the area image data as a display image, the overall state of the construction site C1 can be confirmed.

<Extracting specific images>
Next, the server device 10 performs a process for extracting a specific image from the captured image. As described above, the specific image is a portion of the area image that shows a specific location. In this embodiment, the specific image is extracted through a manual process in which the user specifies a specific range of one image included in the captured image, and an automatic process in which candidate images that indicate candidates for the specific range are presented to the user.

First, in A21, the server device 10 uses the image processing unit 114 to extract candidate images of a specific range from one area image included in the captured image. The image processing unit 114 extracts candidate images based on information about check points that have been determined by the business operator performing the work on-site to improve the quality of the work. In this embodiment, the server device 10 stores a specific image database that includes information about check points.

Figure 9 is a diagram showing an example of a specific image database. The specific image database DB1 shown in Figure 9 stores check location information, specific image history, the current specific image, the history of specified information, the current specified information, and specific image data. The check location information stores the name and shooting information. The name stores a character string indicating the name of the check location, such as "entrance," "corridor," "stairs," "kitchen wall," and "kitchen floor." In addition, the shooting information stores information indicating the shooting position where the check location can be photographed.

The shooting information is, for example, angle information indicating the angle of view of the check location as viewed from the height at which the imaging device 30 normally shoots. If vertically upward is 0 degrees and vertically downward is 180 degrees, the range information indicates the range of the angle of view, for example, 0 degrees to 30 degrees for a ceiling, 45 degrees to 135 degrees for a wall, and 150 degrees to 180 degrees for a floor. The shooting information also includes a range of distances suitable for shooting the check location. The distance suitable for shooting is, for example, 0 m (directly above or below) for a ceiling or floor, and 1 m to 5 m for a wall, which is the distance at which the check location is captured at an appropriate size. In addition, for a location that faces a specific direction, such as a wall, the direction in which the location faces is also included in the shooting information.

The image processing unit 114 first identifies a shooting position from the estimated travel path where the check point can be photographed. For example, the image processing unit 114 identifies a position on the travel path that is a distance from the check point on the drawing that is suitable for shooting, and in the case of a wall, is in the direction in which the check point faces, as the shooting position. The image processing unit 114 then extracts, from among the area images photographed from the identified position, an image with the angle of view indicated by the shooting information as a candidate image.

In the case of locations with distinctive appearances, such as doors and stairs, pattern images showing those locations may be used as the photographic information. In this case, the image processing unit 114 uses well-known image recognition technology to extract images of locations similar to the pattern image as candidate images. The server display unit 112 transmits the extracted candidate images, for example, to the operator terminal 50. The operator terminal 50 displays the transmitted candidate images by the display control unit 511 at A22.

Fig. 10 is a diagram showing an example of a displayed candidate image. In Fig. 10, the display control unit 511 displays a candidate image selection screen D1. On the candidate image selection screen D1, a character string saying "Please select a specific image from the candidate images" is displayed, along with candidate images E11, E12, and E13 (referred to as "candidate image E10" when there is no need to distinguish between them), selection buttons B11, B12, and B13 (referred to as "selection button B10" when there is no need to distinguish between them), and adjustment buttons B21, B22, and B23 (referred to as "adjustment button B20" when there is no need to distinguish between them).

In this way, the server display unit 112 and the image processing unit 114 function as an example of a first presentation unit that presents candidate images of a specific range to the user as candidate images (candidate images E1, E2, and E3 in the example of FIG. 10). According to this aspect, it is possible to make it easier to select an image of a specific location compared to a case where candidate images are not presented.

The selection button B10 is a button for accepting an operation to select a candidate image displayed in association with that selection button as a specific image. The adjustment button B20 is a button for accepting an operation to adjust the range displayed as a candidate image. These operations are accepted, for example, by the operation acceptance unit 512 of the operator terminal 50, and operation data indicating the operation content is transmitted to the server device 10. The user acceptance unit 117 of the server device 10 accepts the operation indicated by the transmitted operation data as an operation performed by the user.

The user accepting unit 117 functions as an example of a specification accepting unit that accepts a first operation by the user to specify a specific range of one of the images included in the multiple images acquired by the image acquiring unit 113. The user accepting unit 117 accepts, for example, an operation to select a candidate image E10 presented by the server display unit 112 (an operation to press one of the selection buttons B10) as the first operation. The first operation here is an operation to specify the range indicated by the candidate image E10 as a specific range.

Specifically, for example, when an operation of pressing the adjustment button B20 is performed and then an operation of dragging the candidate image E10 is performed, the user acceptance unit 117 accepts the user operation, and the server display unit 112 moves the area image displayed as the candidate image E10 in the dragged direction, causing a previously undisplayed portion of the area image to be displayed. In this way, the user acceptance unit 117 accepts the operation of spatially fine-tuning the image showing a specific location as the first operation.

In addition to the above operations, the user reception unit 117 may also receive, as the first operation, an operation for finely adjusting an image showing a specific location over time. For example, when a user drags the candidate image E1 while pressing a specific button (such as the Ctrl key), the user reception unit 117 receives the user operation as the first operation, and the server display unit 112 displays the same portion of an area image captured earlier or later than the area image displayed as the candidate image E1.

The user reception unit 117 may also accept, as the first operation, an operation to specify a specific range on a screen displaying the area image data shown in FIG. 8. In this case, the data output unit 116 first outputs the area image data generated by the data generation unit 115 to, for example, the operator terminal 50. The display control unit 511 of the operator terminal 50 displays the output area image data. The operation reception unit 512 accepts an operation to specify a specific range from the display image displayed as shown in FIG. 8, and the user reception unit 117 accepts the operation as the first operation.

When operator W3 determines that an appropriate range has been specified as a specific image by the above-mentioned first operation, he or she performs an operation of pressing selection button B10 as a first operation. At A24, the server device 10 receives a series of first operations by the user reception unit 117, and the image processing unit 114 extracts, as a specific image, a candidate image selected by the received first operation or an area image of the specified specific range. The DB control unit 111 stores the extracted specific image in the specific image database DB1 as the "current specific image" shown in FIG. 9.

In this way, the image processing unit 114 functions as an extraction unit that extracts, as a specific image, a portion showing a specific location in the shooting area from one or more images included in the multiple images acquired by the image acquisition unit 113. In this embodiment, the image processing unit 114 extracts, as the specific image, an image of the range specified by the first operation as described above. According to this aspect, it is possible to reflect the user's judgment in the extraction of the image of the specific location.

When a new captured image is acquired and new area image data is generated, the DB control unit 111 moves the extracted specific image from "current specific image" to "specific image history." In this way, the specific image extracted by the image processing unit 114 is stored as a history image in the specific image history. The image processing unit 114 may extract candidate images based on the history images thus stored. In that case, the image processing unit 114, for example, identifies the shooting position and angle of view of each of the stored history images.

The image processing unit 114 then extracts images with angles of view within the specified range from among the newly acquired captured images taken at shooting positions within the specified range as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this manner, the candidate images may be images extracted based on the history of the extracted specific images. According to this aspect, images that are close to the actual specific image are presented as candidate images, so the usefulness of the candidate images can be increased compared to a case in which the history of the specific images is not taken into consideration.

<Matching of specified information>
Next, the server device 10 performs a process for associating the extracted specific image with the predetermined information. The predetermined information is information, such as the name of a specific location, that is defined as information for associating with the specific image in the site management support system 1. In this embodiment, the specific information is associated with the specific image through a manual operation in which the user specifies the predetermined information in association with the specific image, and an automatic process prior to this in which candidate information indicating candidates for the predetermined information is presented to the user.

First, in A31, the server device 10 uses the association processing unit 118 to determine candidates for the specified information to be associated with the specific image as candidate information. For example, the association processing unit 118 determines the name of the check point information stored in the specific image database DB1 as the candidate information. The server display unit 112 transmits display data for displaying the determined candidate information to, for example, the operator terminal 50. In A32, the operator terminal 50 uses the display control unit 511 to display the candidate information indicated by the transmitted display data.

Figure 11 is a diagram showing an example of displayed candidate information. In Figure 11, the display control unit 511 is displaying a candidate information selection screen D2. The candidate information selection screen D2 displays a character string saying "Please select specific information from the candidate information," a specific image F14, a selection button B31, and a candidate information input field J11. Below the candidate information input field J11, a candidate information list J12 is displayed as a pull-down list, and candidate information selected from the candidate information list J12 is displayed in the candidate information input field J11.

In this way, the server display unit 112 and the association processing unit 118 function as a second presentation unit that presents candidate information indicating candidates for the specified information to the user. According to this aspect, it is possible to reduce the effort required to associate a specific image with the specified information compared to a case in which candidate information is not presented. The operation of selecting candidate information and pressing the selection button B31 is accepted by the user acceptance unit 117 of the server device 10 as an operation performed by the user, similar to the first operation described above.

The user reception unit 117 functions as an example of a correspondence reception unit that receives a second operation by the user to correspond the specific image extracted by the image processing unit 114 with specified information (in the example of FIG. 11, the specific image F14 and the name of the checked area "kitchen wall"). The user reception unit 117 receives, as the second operation, an operation to select candidate information presented by the server display unit 112 (in the example of FIG. 11, the candidate information included in the candidate information list J12). This second operation is an operation to correspond the candidate information with the specific image as specified information.

When operator W3 determines that appropriate candidate information to be associated with the specific image has been selected by the above-mentioned second operation, he/she performs the operation of pressing selection button B31 as the second operation. At A33, the operator terminal 50 accepts these second operations as selection operations of candidate information by the operation acceptance unit 512. At A34, the server device 10 accepts the series of second operations by the user acceptance unit 117, and stores the candidate information associated with the specific image by the accepted second operations by the DB control unit 111 in association with the specific image as the "current specified information" shown in FIG. 9 in the specific image database DB1.

Then, the data generation unit 115 reads out the "current specific image" and "current specified information" from the specific image database DB1, and generates a list in which they are associated with each other as specific image data. Now, for example, assume that the specific image data is requested from the supervisor terminal 40. In that case, the server device 10 outputs the generated specific image data to the supervisor terminal 40 by the data output unit 116 at A34. The supervisor terminal 40 displays the output specific image data by the display control unit 411 at A35.

Figure 12 is a diagram showing an example of displayed specific image data. In the example of Figure 12, the display control unit 411 displays a checklist H1. Checklist H1 displays the names of the check locations "Entrance", "Hallway", "Stairs", "Kitchen Wall", and "Kitchen Floor", specific images F11, F12, F13, F14, and F15 of each check location (referred to as "specific image F10" when not distinguishing between them), and the check results of each check location. The check results include a symbol indicating whether the work is progressing well or not, and a comment.

Specific images F10 include those extracted from images taken on different dates, "mm/d1," "mm/d2," "mm/d3," and "mm/d4." For example, specific images F11-1, F11-2, F11-3, and F11-4 are all images extracted with the "entrance" as a specific location, and are extracted from images taken on "mm/d1," "mm/d2," "mm/d3," and "mm/d4."

In this way, the data output unit 116 outputs a list of multiple specific images (checklist H1 in the example of FIG. 12) as specific image data. These multiple specific images (specific images F11-1, F11-2, F11-3, and F11-4 in the example of FIG. 12, etc.) have a specific location in common but were photographed at different times. According to this aspect, it is easier to grasp the transition in the state of the specific location compared to when only specific images photographed at a single time are output.

Note that, since the on-site worker W1 takes the images using the imaging device 30 while walking, the route he or she moves during the image capture is not the same each time. Therefore, the specific images do not necessarily show the exact same area as the specific location, but it is sufficient that at least a portion of the specific location is shown. This is because by looking at that portion of the image, the situation of the specific location at the work site can be understood.

As described above, the data output unit 116 functions as an example of a specific output unit that outputs specific image data including a specific image extracted by the image processing unit 114. According to this aspect, a list showing specific locations can be easily created. In this embodiment, the data output unit 116 outputs, as specific image data, a list that lists the specific image extracted by the image processing unit 114 and the specific information associated with the specific image by the second operation described above.

When a specific enlargement operation (e.g., a single click) is performed on a specific image F10 included in the checklist H1, the display control unit 411 enlarges and displays the specific image F10. This makes it easier to see the situation at the site. Furthermore, in this embodiment, the specific image F10 included in the checklist H1 is a link to area image data generated based on the captured image from which the specific image F10 was extracted. Furthermore, the specific image F10 is a link to area image data in which a specific location shown in the specific image F10 is displayed as a display image.

At A41, the supervisor terminal 40 accepts a predetermined operation (e.g., double-clicking) on the specific image F10 as a link operation via the operation acceptance unit 412. The operation acceptance unit 412 transmits operation data indicating the specific image F10 for which the link operation has been performed to the server device 10. At A42, the server device 10 outputs area image data of the linked destination to the supervisor terminal 40 via the data output unit 116. At A43, the supervisor terminal 40 displays the output area image data via the display control unit 411.

For example, when a link operation is performed to specific image F11-4 shown in FIG. 12, area image data generated based on the image captured at mm/d4 is output with the entrance as the display image and displayed on the supervisor terminal 40. The supervisor terminal 40 accepts, at A44, an operation to move the location shown as the display image of the area image data as a display operation. For example, since the work supervisor W2 is unable to fully determine the work situation from the specific image F11-4 alone, he determines the work situation by manipulating the area image data in this way to view the surrounding image.

As described above, the data output unit 116 outputs data including a link from an image of a specific location (each specific image F10 in the example of FIG. 12) to an image showing that location among the images shown by the area image data (for example, in the case of specific image F11-4, an image showing the entrance taken at mm/d4) as specific image data. According to this aspect, it is easier to confirm the image around the specific location compared to when the specific image data does not include the above link.

When the work supervisor W2 judges the work status by displaying the area image data, he/she returns to the checklist H1 to check the work status. In this embodiment, the area image data includes a link for switching the display to the specific image data (the checklist H1 in the example of FIG. 12).
Fig. 13 is a diagram showing an example of a displayed area image. In the example of Fig. 13, the display control unit 411 displays a display image A41 that represents a part of the area image A4.

Display image A11 shows link images L14-4 and L15-4 (referred to as "link image L10" when there is no need to distinguish between them). Link image L14-4 is a link to specific image F14-4 shown in FIG. 12. Link image L15-4 is a link to specific image F15-4 shown in FIG. 12. When the supervisor terminal 40 accepts a link operation to link image L10 at A51 via the operation acceptance unit 412, it transmits operation data indicating the operated link image L10 to the server device 10.

At A52, the server device 10 outputs the specific image data that is the link destination indicated by the transmitted operation data to the supervisor terminal 40 by the data output unit 116. At A53, the supervisor terminal 40 displays the output specific image data by the display control unit 411. When an operation is performed on the link image L14-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F14-4 shown in FIG. 12. When an operation is performed on the link image L15-4, the display control unit 411 displays, for example, the checklist H1 and an enlarged image of the specific image F15-4.

As described above, the data output unit 116 outputs data including a link from an image taken at a position corresponding to a specific location to an image of that location in the specific image data as area image data. According to this aspect, even when the area image indicated by the area image data is displayed, it is easier to output the image of the specific location compared to a case where the area image data does not include a link.

Next, at A54, the supervisor terminal 40 accepts the check operation through the operation acceptance unit 412 and transmits the operation data to the server device 10. At A55, the server device 10 uses the DB control unit 111 to store the specific image data reflecting the check results indicated by the transmitted operation data in the specific image database DB1. The captured image processing up to A34 shown in FIG. 7 is executed each time photography is performed at the work site. As a result, the area image data and specific image data are saved and accumulated each time photography is performed.

As described above, the data output unit 116 uses multiple images (photographed images) acquired by the image acquisition unit 113 to output area image data showing the photographed area as seen from each position on the estimated travel path. The data output unit 116 also uses the photographed images to output specific image data, as described above. In this manner, the site management support system 1 utilizes the photographed images for both area image data as shown in FIG. 8 and specific image data as shown in FIG. 12. When photographing images of the photographed area to generate area image data and specific image data, respectively, the effort of photographing is required twice. In this embodiment, the photographed images are shared as described above, and therefore the effort of the work associated with photographing the site can be reduced compared to when the photographed images are not shared.

Furthermore, if a site worker W1 takes still images with a camera with a normal angle of view while moving around the shooting area, taking pictures of specific locations one by one does not necessarily result in an image that makes it easy for the work supervisor W2 to judge the situation, and taking pictures again just because an image is undesirable is very time-consuming. Furthermore, even if the picture is taken successfully, the surrounding situation outside the range of the camera's angle of view cannot be seen. In this embodiment, as described above, a video is taken with a 360-degree camera that captures everything in all directions, and specific locations are extracted, making it possible to select a desired image. Furthermore, because omnidirectional images are highly comprehensive, it is rare that a desired image cannot be selected, and even if a specific location that needs to be extracted arises after shooting, that location can be extracted.

Although the area image data also shows a specific location, it is time-consuming to search for the specific location in the area image shown by the area image data when one wants to see only the required location. In this embodiment, the specific location is extracted and the specific image data is output, making it possible to display an image of the specific location without such time-consuming steps. On the other hand, by outputting not only the specific image data but also the area image data, even if it is difficult to tell from the specific image data which part of the shooting area the specific image shows, the position of the location shown in the specific image can be ascertained from the area image data.

As described above, when matching a specific image with predetermined information, candidate information is presented, thereby reducing the effort required to match appropriate candidate information. In this embodiment, the check points shown in FIG. 9 are always extracted as the specific image. As a result, when the specific image data is used as a checklist, the specific points are always checked at every work site, ensuring the quality of the work.

<Other embodiments>
The shooting area is not limited to the above. For example, it may be inside an existing building, inside a vehicle, or a predetermined area outdoors. The imaging device is not limited to a 360-degree camera, and a wide-angle camera or a camera that shoots at a normal angle of view may be used. The captured images are not limited to moving images, and may be still images that are continuously captured.

The area image data is not limited to that shown in FIG. 8. For example, it may be data showing only the area image, without any drawings or positional images superimposed. In that case, it is sufficient if the viewpoint can be moved along the movement route by an operation different from the operation of selecting the positional image. The specific image data is not limited to that shown in FIG. 12. For example, it may be data in which a specific location is changed each time an image is taken, or it may be data showing only a specific image extracted from the most recently taken image.

The specified information is not limited to the name of the check location, but may be, for example, the name of the work site (floor number or room number, etc.), the work process, the name of the photographer or work supervisor, the date and time of shooting, the weather at the time of shooting, or the type of imaging device, etc. In this way, the specified information may be information indicating what kind of location the specific location is or what kind of image the specific image is. However, information unrelated to the specific image may also be used as the specified information.

<Candidate image>
The image processing unit 114 may extract candidate images in a manner different from that of the embodiment. For example, when a history of a specific range designated by the operator W3 is accumulated, the image processing unit 114 extracts candidate images based on the history. For example, when the same part of the shooting area is designated as a specific range a predetermined number of times or more, the image processing unit 114 extracts an image including the part as a candidate image.

In addition, when multiple rooms with the same layout are photographed as the photographing area, such as in an apartment building, the image processing unit 114 extracts images of the range specified in the other rooms with the same layout as candidate images. The server display unit 112 presents the candidate images thus extracted to the user. In this case, the candidate images are images extracted based on the history of the specific range specified by the user. According to this aspect, images similar to the specific images of the range previously specified by the user are presented as candidate images, so that the usefulness of the candidate images can be increased compared to a case where the history of the specific range is not taken into consideration.

<Candidate information>
The association processing unit 118 may determine candidate information in a manner different from that of the embodiment. For example, the association processing unit 118 recognizes a location shown in a specific image extracted by the image processing unit 114, and determines information on the recognized location as candidate information. For example, the association processing unit 118 recognizes doors, stairs, lights, windows, washrooms, etc. by image recognition using a pattern image, and determines their names as candidate information.

In this case, the candidate information is information that is determined based on the location shown in the specific image extracted by the image processing unit 114. According to this aspect, for example, even if the specific location is changed due to the situation at the work site, information that matches that location is presented as candidate information, so the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

The association processing unit 118 may also determine, for example, specific information that has been associated with a specific image in the past as candidate information. The association processing unit 118 may also determine, as candidate information, specific information that has been used in the past not only at the same work site, but also at work sites with the same layout. In either case, the candidate information is information that is determined based on the past specific information. According to this aspect, for example, even specific information that has been added depending on the situation at the work site is presented as candidate information, so that the usefulness of the candidate information can be increased compared to when the candidate information is fixed.

<Image Filter>
In the site management support system 1, both the estimation of the travel path and the extraction of the specific image are performed based on a plurality of images (photographed images) captured by the imaging device 30. At that time, the captured images may be narrowed down to those with better images. The quality of the images may be determined, for example, using a known technique (such as fast Fourier transform, machine learning, or open source) for determining whether a photograph is out of focus.

For example, the image processing unit 114 estimates a travel path based on an image narrowed down based on a first rule from among the multiple images acquired by the image acquisition unit 113. The image processing unit 114 also extracts a specific image from the images narrowed down based on a second rule from among the multiple images acquired by the image acquisition unit 113. The second rule may be a rule different from the first rule.

The first rule, for example, is a rule that narrows down the captured images to those that have a good overall appearance. On the other hand, the second rule is a rule that narrows down the captured images to those that have a good appearance in a specific part. Even a single image may contain a mixture of good and bad parts. As a result, it may happen that the image is good overall but a specific part is bad, or that a specific part is good but the image is bad overall. Therefore, by using the first and second rules as described above, it is possible to increase the accuracy of the estimation of the travel path while extracting a specific image that has a good appearance, thereby achieving both the accuracy of the estimation of the travel path and the quality of the extracted image.

Note that the first and second rules are not limited to the above rules. For example, the first rule is a rule that narrows down images to those with feature points equal to or greater than a threshold, and the second rule is a rule that narrows down images regardless of the number of feature points. The more feature points an image has, the more accurate the self-location estimation using VSLAM becomes. On the other hand, when a specific location is a wall or floor, for example, there will be very few feature points. Therefore, by making the first and second rules different as described above, it is possible to achieve both the estimation accuracy of the travel path and the quality of the extracted image.

The first rule and the second rule may be the same rule. Even in this case, it is possible to improve both the estimation accuracy of the travel path and the quality of the extracted image compared to the case where no narrowing down is performed. The above-mentioned narrowing down of images may also be used when extracting candidate images. In this case, the image processing unit 114 narrows down the captured images based on the second rule (which may be the first rule), and extracts candidate images from the narrowed down captured images, as in the embodiment. In this case, it is also possible to improve the quality of the extracted image.

<Automatic extraction>
In the embodiment, the image processing unit 114 extracts the specific image through a manual operation of the user specifying a specific range, but the specific image may be extracted automatically without manual operation. In that case, the image processing unit 114 may extract the specific image, for example, by using the same method for extracting the candidate image as it is. For example, the image processing unit 114 extracts a new specific image based on the shooting information of the check point shown in FIG. 9, as in the embodiment.

The image processing unit 114 also extracts a new specific image that includes the same location as a specific location, for example, based on the history of extracted specific images. The image processing unit 114 also extracts a new specific image that includes a range that was previously specified as a specific location, based on the history of specific ranges specified by the user.

Figure 14 is a diagram showing an example of a flow diagram of the automatic extraction process. First, in S11, the server device 10 acquires a captured image by the image acquisition unit 113. Next, in S12, the server device 10 estimates the movement path of the imaging device 30 by the image processing unit 114. Next, in S13, the server device 10 generates area image data by the data generation unit 115. Next, in S14, the server device 10 extracts a specific image from the captured image by the image processing unit 114. Next, in S15, the server device 10 determines the specified information by the association processing unit 118.

In S14 and S15, unlike the example in FIG. 7, a specific image is extracted and the specified information is determined without any operation by the user. Next, in S16, the server device 10 generates specific image data by the data generation unit 115. Then, in S17, the server device 10 outputs the area image data and the specific image data in response to a request from the supervisor terminal 40 or the operator terminal 50.

According to this aspect, the effort required to generate specific image data can be further reduced compared to when candidate images are presented. When extracting specific images, the image processing unit 114 may use the second rule described above to narrow down the captured images, thereby extracting images that look good as specific images. Furthermore, the image processing unit 114 may extract specific images using AI (Artificial Intelligence) technology that performs machine learning using past specific images as training data.

In addition, drawing information such as a floor plan of a work site may contain location information indicating the location of a specific location. In this case, when a specific location is specified from the drawing information, the image processing unit 114 may identify a captured image taken from a shooting position where the location is likely to be included in the angle of view, based on the location information of the specified location and the estimated movement path. Note that the server display unit 112 may present the drawing information and movement path to the user, and the user may identify a similar shooting location, thereby identifying a captured image taken from the identified shooting position.

Then, the image processing unit 114 recognizes the portion of the identified photographed image that shows the specific location, and extracts the recognized portion as the specific image. Note that the server display unit 112 may present the identified photographed image to the user, allowing the user to specify the portion that shows the specific location, and the image processing unit 114 may extract the specified portion as the specific image. According to these aspects, the specific image can be easily extracted if it is a location shown in the drawing information.

<Automatic mapping>
In the embodiment, the association processing unit 118 associates the specific information with the specific image through a manual operation of designating the specific information in association with the specific image, but the specific information may be associated automatically without a manual operation. In that case, the association processing unit 118 determines, for example, the specific information associated with the history of specific images that have captured the same specific location as the extracted specific image, as the specific information to be associated with the extracted specific image.

Furthermore, when the association processing unit 118 determines the predetermined information to be associated with a specific image as described above, it determines predetermined information that exists near the predetermined information as the predetermined information to be associated with an image of another location extracted from the same shooting information as the specific image. For example, when the predetermined information "kitchen wall" is associated with a specific image, the association processing unit 118 determines "kitchen floor" or "kitchen ceiling" that exists near the "kitchen wall" as the predetermined information to be associated with an image of another location (e.g., kitchen floor or ceiling).

The association processing unit 118 may also identify the location shown in the specific image using image recognition technology and determine the name, etc. of the identified location as the specified information. Specifically, the association processing unit 118 identifies the location shown in the specific image using a pattern image associated with the specified information (if the specified information is "entrance", pattern images of the entrance photographed from various angles) and determines the specified information associated with the pattern image of the identified location as the specified information to be associated with the specific image.

According to this aspect, the effort required to generate specific image data can be further reduced compared to when candidate information is presented. The matching processing unit 118 may match specific information using AI technology that performs machine learning using specific information that has been matched to specific images in the past as training data.

In addition, drawing information such as a floor plan of a work site may contain the name of each location as the specified information. In such a case, the correspondence processing unit 118 may estimate the specified information corresponding to the specific location indicated by the specific image from the drawing information. For example, the correspondence processing unit 118 identifies the position of the specific location on the drawing from the shooting position and angle of view information at which the specific image was taken, and determines the name of the place including that position as the specified information.

The server display unit 112 may present the drawing information to the user, and the association processing unit 118 may determine the name of a location specified by the user (operator W3 or work supervisor W2, etc.) from the presented drawing information as the specified information. According to these aspects, if the specified information is shown in the drawing information, association with a specific image can be easily performed.

<Extraction of wide-area images>
When extracting a specific image, the image processing unit 114 may extract an image having a range that is a predetermined ratio (hereinafter referred to as "image ratio") wider than the range in which the specific location is shown as the specific image. In this case, the data generating unit 115 generates data showing the specific location and the surroundings of the location when a predetermined operation (such as a flick) is performed as specific image data. This makes it easy to display the surroundings of a specific location when you want to see the surroundings of the specific location.

The image ratio may be a value according to the size of the photographed area, the usage of the outputted area image data, or the usage of the outputted specific image data. For example, the image processing unit 114 increases the image ratio to extract the specific image as the photographed area becomes larger. The larger the photographed area, the larger the space around the specific location, and therefore the larger the surrounding image to be confirmed.

Furthermore, the image processing unit 114 extracts a specific image with a larger image ratio the higher the usage of the area image data or specific image data. The usage here is higher the longer the usage time or the more frequently the area image data or specific image data is used. The higher the usage, the more likely it is that there will be opportunities to look around a specific location, and the more likely it will be that a situation will arise where you want to look at a wider range. As described above, by varying the image ratio, it is possible to check the situation in a range as wide as necessary, compared to when the image ratio is constant.

<Recommendation of specific images>
A specific image may be recommended. For example, when a specific range is specified by the user, the image processing unit 114 not only extracts an image of the specified range, but also extracts an image that is likely to be used as a specific image from an area spatially adjacent to the range as a recommended image. For example, when a kitchen wall is specified, the image processing unit 114 extracts an image of the kitchen floor and ceiling as a recommended image. Also, when a specific range is specified from an image taken at a certain shooting position, the image processing unit 114 extracts images of the same range from images taken at positions before and after the position as recommended images.

The server display unit 112 presents the recommended image thus extracted to the user together with the specific image. The user performs an operation to instruct the user to designate the recommended image as the specific image in addition to or instead of the specific image. The data generation unit 115 generates specific image data that includes the instructed recommended image as the specific image. According to this aspect, the usefulness of the specific image data can be increased compared to a case in which the recommended image is not presented.

<Check with area image data>
When a check area is extracted as a specific image, the data generation unit 115 may generate area image data that is displayed together with a check image for inputting the check result when the specific area is displayed.

Figure 15 is a diagram showing an example of a displayed check image. In the example of Figure 15, the display control unit 411 displays check images M14-4 and M15-4 on the display image A41. Check image M14-4 is an image for inputting the check results of specific image F14-4, and check image M15-4 is an image for inputting the check results of specific image F15-4. When input is made to the check images, the DB control unit 111 also reflects the input content in the check results of the check list H1 shown in Figure 12. This aspect makes it possible to reduce the effort required to check specific locations compared to a case in which check results can only be input into specific image data.

<Other>
When a new range is designated as a specific range, the image processing unit 114 may extract an image of the same range from a previously captured image as a previous specific image. The data generating unit 115 generates specific image data including the extracted current specific image and the previous specific image. This makes it possible to compare the newly designated range with the previous image.

The data generating unit 115 may also generate specific image data that represents, in a comparable manner, specific images showing common locations extracted from images taken in multiple shooting areas with a common structure (for example, rooms with the same layout in the same apartment building). By arranging comparison targets in this way, it is possible to make it easier to judge the status of the work.

When displaying a checklist such as that shown in FIG. 12, the server display unit 112 may also display the check results for similar specific images from the past. For example, the server display unit 112 displays an image of an entranceway extracted from other captured images and its check results alongside the specific image F11-4 of the entranceway and its check results. This makes it easier to input the check results.

For example, the image processing unit 114 may determine the check result for the specific image being checked based on the check results for similar specific images in the past. For example, the image processing unit 114 determines the check result to be "○" if the work progress is similar to a specific image marked "○", and determines the check result to be "×" if the work progress is similar to a specific image marked "×".

The server display unit 112 may then display the judgment results made by the image processing unit 114 in a checklist, or may display an alert if the check results entered by the user differ from the judgment results made by the image processing unit 114, thereby assisting the user in checking. Furthermore, the DB control unit 111 may reflect the results made by the image processing unit 114 as check results in the checklist H1, even if there is no checking by the user. According to these aspects, it is possible to reduce the effort required to check specific locations, compared to a case in which the check results are not judged by the image processing unit 114.

The imaging device 30 does not have to be installed at the work site. For example, a user such as a field worker W1 takes the field terminal 20 and imaging device 30 to the work site to take pictures, has the imaging device 30 send the captured image data to the field terminal 20, and connects the field terminal 20 to a Wi-Fi router in the office or at home to transfer the captured image data to the server device 10 (the image data may be transferred via mobile communication). In this case, the field management support system 1 only needs to include the communication line 2, the selfie stick 3, the server device 10, the field terminal 20, the imaging device 30, the supervisor terminal 40, and the operator terminal 50.

The imaging device 30 may be, for example, a device with a wide-angle camera attached to a helmet. In this case, the imaging device may have a blind spot vertically below where people are, but can capture images in other directions. The captured image data may be transferred from the imaging device 30 to the on-site terminal 20 via a recording medium such as a MicroSD card, rather than wireless communication, or via a communication cable such as a Type C. The imaging device 30 may transmit the captured image data directly to the server device 10, or may transfer the captured image data to the server device 10 via a device other than the on-site terminal 20.

The user who performs the manual tasks described in the embodiment (the tasks of specifying a specific range and specifying specific information in association with a specific image) is not limited to the operator W3. For example, the tasks may be performed by the work supervisor W2 or the on-site worker W1. They may also be performed by other workers belonging to the work business operator or by an outsourced company commissioned by the work business operator.

<Configuration variations>
In the site management support system 1, if the work site is one in which the breaker 6 is not turned off, the site terminal 20 and the imaging device 30 may be connected to the external power source 5 without going through the external battery 4. In the embodiment, the site terminal 20 transmits measurement data to the server device 10 by mobile communication even when the breaker 6 is turned off, but if the breaker 6 is not turned off, a Wi-Fi router may be installed at the work site to transmit measurement data to the server device 10 by Wi-Fi communication.

For example, the server device 10 may be distributed across two or more devices, or may be replaced by a cloud computing system. The on-site terminal 20 and the imaging device 30 may be integrated. The functional configuration shown in FIG. 6 is also an example, and is not limited to this. For example, the functions of the server device 10, the on-site terminal 20, and the imaging device 30 may each be distributed across two or more devices.

In addition, the operations performed by one function may be distributed among two or more functions, or two or more functions may be integrated into one function. In short, as long as the functions shown in FIG. 6 are realized in the entire site management support system 1, the devices that realize those functions may have any configuration.

Although the above-described embodiment is an information processing device such as the server device 10, or an information processing system such as the site management support system 1 including the server device 10, it may also be an information processing method. The information processing method includes steps of each process executed by the information processing system. Also, the above-described embodiment may be a program. The program causes a computer to execute each process executed by a similar information processing system.

<Additional Notes>
Furthermore, it may be provided in the following aspects:

(1) An information processing system, comprising a processor capable of executing a program to perform each of the following steps: in the acquisition step, a plurality of images captured by an imaging device at each position of a moving path while the imaging device moves through a shooting area are acquired; in the estimation step, the moving path is estimated based on the acquired plurality of images; in the area output step, area image data showing the shooting area as seen from each position of the estimated moving path is output using the acquired plurality of images; in the extraction step, a portion showing a specific location in the shooting area is extracted as a specific image from one or more images included in the acquired plurality of images; and in the specific output step, specific image data including the extracted specific image is output.

This approach can reduce the amount of work required to photograph the site.

(2) In the information processing system described in (1) above, in the specific output step, a list of multiple specific images is output as the specific image data, and the multiple specific images have the specific location in common but were taken at different times.

This aspect makes it easy to understand the state transitions of specific locations.

(3) In the information processing system described in (1) or (2) above, the specification receiving step receives a first operation by a user that specifies a specific range of one of the acquired images, and the extraction step extracts the image of the range specified by the first operation as the specific image.

In this manner, the user's judgment can be reflected in the extraction of images of specific locations.

(4) In the information processing system described in (3) above, in the first presentation step, candidate images of the specific range are presented to the user as candidate images, and in the designation reception step, an operation of selecting the presented candidate image is received as the first operation, and the first operation is an operation of designating the range indicated by the candidate image as the specific range.

This aspect makes it easier to select an image of a specific location.

(5) In the information processing system described in (4) above, the candidate images are determined based on the history of the extracted specific image or the history of the specified specific range.

This aspect can increase the usefulness of the candidate images.

(6) In the information processing system described in any one of (1) to (5) above, in the response receiving step, a second operation by the user that associates the extracted specific image with predetermined information is received, and in the specific output step, a list that lists the extracted specific image and the predetermined information that has been associated with the specific image by the second operation is output as the specific image data.

In this manner, human-determined information can be associated with a specific image.

(7) In the information processing system described in (6) above, in the second presentation step, candidate information indicating candidates for the specified information is presented to the user, and in the response reception step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation of associating the candidate information as the specified information with the specific image.

This aspect can reduce the effort required to match specific images with predetermined information.

(8) In the information processing system described in (7) above, the candidate information is information determined based on the location shown in the extracted specific image or the past specified information.

This aspect can increase the usefulness of the candidate information.

(9) In the information processing system described in any one of (1) to (8) above, in the estimation step, the estimation is performed based on an image narrowed down from the plurality of acquired images based on a first rule, and in the extraction step, the specific image is extracted from an image narrowed down from the plurality of acquired images based on a second rule, and the second rule is different from the first rule.

This aspect makes it possible to achieve both high estimation accuracy of the travel path and high quality of the extracted image.

(10) In the information processing system described in any one of (1) to (9) above, in the specific output step, data including a link from the image of the specific location to an image showing the location among the images shown by the area image data is output as the specific image data.

This allows you to easily check the image around a particular location.

(11) In the information processing system described in any one of (1) to (10) above, in the area output step, data including a link from an image taken at a position corresponding to the specific location to an image of the specific location in the specific image data is output as the area image data.

This aspect makes it easy to output an image of the relevant area.

(12) In the information processing system described in any one of (1) to (11) above, in the extraction step, an image having a range that is a predetermined percentage wider than the range in which the specific location is captured is extracted as the specific image, and the percentage is a value corresponding to the size of the shooting area, the degree of utilization of the outputted area image data, or the degree of utilization of the outputted specific image data.

According to this embodiment, it is possible to check the situation in a range as required.
Of course, this is not the case.
Furthermore, the above-described embodiments and modifications may be combined in any desired manner.

Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. New embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiments and their modifications are within the scope and gist of the invention, and are also within the scope of the invention and its equivalents as set forth in the claims.

1: On-site management support system 2: Communication line 10: Server device 20: On-site terminal 30: Imaging device 40: Supervisor terminal 50: Operator terminal 111: DB control unit 112: Server display unit 113: Image acquisition unit 114: Image processing unit 115: Data generation unit 116: Data output unit 117: User reception unit 118: Correlation processing unit 411: Display control unit 412: Operation reception unit 511: Display control unit 512: Operation reception unit

Claims (10)

An information processing system,
A processor capable of executing a program to perform the following steps:
In the acquisition step, the imaging device acquires a plurality of images captured at each position along a moving path while moving through the imaging area;
In the estimation step, the travel path is estimated based on the plurality of acquired images;
In the area output step, area image data indicating the photographing area as viewed from each position of the estimated moving path is output using the acquired multiple images;
In the extraction step, a portion showing a specific location in the photographed area is extracted as a specific image from one or more images included in the plurality of acquired images;
In the specific output step, specific image data including the extracted specific image is output;
In the first narrowing down step, the acquired plurality of images are narrowed down to a first image group based on a first rule;
In a second narrowing down step, the acquired images are narrowed down to a second image group based on a second rule;
In the estimating step, the estimation is performed based on the first image group that has been narrowed down;
In the extraction step, the specific image is extracted from the narrowed-down second image group, and the second rule is different from the first rule,
The first rule is a rule for narrowing down the acquired multiple images to images that have a good overall appearance,
The second rule is a rule for narrowing down the acquired plurality of images to images in which the specific portion is clearly visible.
Information processing system. An information processing system,
A processor capable of executing a program to perform the following steps:
In the acquisition step, the imaging device acquires a plurality of images captured at each position along a moving path while moving through the imaging area;
In the estimation step, the travel path is estimated based on the plurality of acquired images;
In the area output step, area image data indicating the photographing area as viewed from each position of the estimated moving path is output using the acquired multiple images;
In the extraction step, a portion showing a specific location in the photographed area is extracted as a specific image from one or more images included in the plurality of acquired images;
In the specific output step, specific image data including the extracted specific image is output;
In the first narrowing down step, the acquired plurality of images are narrowed down to a first image group based on a first rule;
In a second narrowing down step, the acquired images are narrowed down to a second image group based on a second rule;
In the estimating step, the estimation is performed based on the first image group that has been narrowed down;
In the extraction step, the specific image is extracted from the narrowed-down second image group, and the second rule is different from the first rule,
the first rule is a rule for narrowing down the acquired plurality of images to images having a feature point equal to or greater than a threshold;
The second rule is a rule for narrowing down the acquired images regardless of the number of feature points.
Information processing system. 3. The information processing system according to claim 1,
In the designation receiving step, a first operation of a user is received to designate a specific range of one of the acquired images, and
In the extraction step, an image within a range designated by the first operation is extracted as the specific image,
In the first presentation step, candidates of images in the specific range are presented to a user as candidate images;
In the designation receiving step, an operation of selecting the presented candidate image is received as the first operation, and the first operation is an operation of designating a range indicated by the candidate image as the specific range,
An information processing system, wherein the candidate images are determined based on a history of the extracted specific images or a history of the specified specific range. 3. The information processing system according to claim 1,
In the correspondence receiving step, a second operation of the user for corresponding the extracted specific image with predetermined information is received;
In the specific output step, a list in which the extracted specific image and the predetermined information associated with the specific image by the second operation are arranged is output as the specific image data,
In the second presentation step, candidate information indicating candidates for the predetermined information is presented to the user;
In the response receiving step, an operation of selecting the presented candidate information is received as the second operation, and the second operation is an operation of associating the candidate information with the specific image as the specified information, in an information processing system. 5. The information processing system according to claim 4 ,
An information processing system, wherein the candidate information is information determined based on a location shown in the extracted specific image or the specified information from the past. 3. The information processing system according to claim 1,
In the specific output step, data including a link from an image of the specific location to an image showing the specific location among images represented by the regional image data is output as the specific image data. 3. The information processing system according to claim 1,
In the area output step, data including a link from an image captured at a position corresponding to the specific location to an image of the specific location in the specific image data is output as the area image data. 3. The information processing system according to claim 1,
In the extraction step, an image having a range that is a predetermined percentage wider than the range in which the specific location is captured is extracted as the specific image, and the percentage is a value corresponding to the size of the shooting area, the utilization rate of the outputted area image data, or the utilization rate of the outputted specific image data, in an information processing system. A program,
A method for causing a computer to execute each step of the information processing system according to claim 1 or 2 ,
program. 1. An information processing method, comprising:
3. An information processing system that executes each step according to claim 1 or 2 .
Information processing methods.