JP7425406B2 - Image inspection device, learned model generation device, image inspection system, image inspection program, learned model generation program, and learned model - Google Patents

Description

The present invention relates to an image inspection device, a learned model generation device, an image inspection system, an image inspection program, a learned model generation program, and a learned model. The present invention relates to an image inspection device that inspects whether or not it is included in image data.

Patent Document 1 describes a method for specifying geographic location information of an inspection object from image data of a road, tunnel, etc., even if the exact shooting point is unknown. In this method, the image data for which the geolocation is to be determined (target image) is compared with a large number of image data (matching images) for which the geolocation information is known, and if the similarity is high, the matching image is Geographic location information is used as location information of the target image. In order to quickly and accurately specify the position of the object to be inspected in the target image, images for comparison are narrowed down using attribute information of the object to be inspected as a narrowing condition.

Patent Document 2 describes a method for more accurately identifying the position and orientation of an observation target by comparing an image included in reference data that associates the position, orientation, and photographed image in the observation target space with an image photographed by the observation target. It is stated that As a result of the comparison, the position and orientation associated with the images determined to match or be similar are specified as the position and orientation of the observation target.

Japanese Patent Application Publication No. 2017-102672 Japanese Patent Application Publication No. 2018-185182

In both of the methods described in Patent Documents 1 and 2, the degree of similarity is determined by performing feature amount matching between an image to be identified and an image for comparison. Feature amount matching is a process of extracting feature amounts from an image to be identified, extracting feature amounts from an image for comparison, and determining similarity based on the degree of match between these extracted feature amounts.

In methods based on feature matching, even if the features of the same object exist in the image to be identified and the image for comparison, if there is a mismatch in size or resolution, there is a possibility that the images will be determined to be different features. There is. In order to avoid this, it is conceivable to perform matching using a plurality of matching images having different sizes and resolutions.

However, if a plurality of images with different sizes and resolutions for one identification target are prepared as images for matching, matching will take that much time. In this way, when determining similarity by performing feature value matching, it can be said that there is a trade-off relationship between determination accuracy and matching time.

Furthermore, in feature matching, the accuracy of matching is affected by what features are extracted as targets. When extracting features from images that do not change, such as logos or marks, it can be determined with good accuracy, but when extracting features from objects that move or change over time, It is difficult to use.

As described above, it has not been easy to quickly identify the photographing location based on a photographed image of a landscape that changes over time. In the case of a photographed image of a landscape that changes over time, if it is outdoors, it is possible to have information on the photographed location identified by GPS as the attribute information of the photographed image, so that the photographed location etc. It is also possible to quickly identify the However, positioning using GPS is difficult indoors (inside a building).

The present invention has been made in view of the above-mentioned conventional problems, and an object of the present invention is to locate the photographed location based on the photographed image, despite the simple configuration, even in a building where it is difficult to specify the location using GPS etc. The purpose is to quickly identify the

An image inspection apparatus according to a typical embodiment of the present invention includes an inspection data reception unit that receives image data of an inspection target photographed in a building consisting of a plurality of areas, and a a first trained model for causing a computer to function so as to output an area where image data to be inspected was taken; and a plurality of second trained models corresponding to each of the plurality of areas, a second trained model for causing the computer to function so as to input the image data of the inspection object and output the location and direction in the area where the image data of the inspection object was taken; , based on the first trained model stored in the storage unit, output the area where the image data of the inspection target received by the inspection data reception unit was captured, and create a target area corresponding to the output area. The present invention is characterized by comprising a photographing state specifying section that outputs the location and direction at which the image data of the inspection object received by the inspection data receiving section was photographed, based on the second learned model.

According to the image inspection apparatus according to the present invention, even in a building where it is difficult to specify the location using GPS or the like, it is possible to quickly identify the photographing location based on the photographed image, despite the simple configuration.

1 is a diagram showing the configuration of an image inspection system according to Embodiment 1. FIG. 1 is a diagram showing a functional block configuration of a trained model generation device 2 according to Embodiment 1. FIG. 1 is a diagram showing a hardware configuration of a trained model generation device 2 according to Embodiment 1. FIG. FIG. 3 is a diagram for explaining zenith correction. FIG. 3 is a diagram illustrating cutting out image data of multiple magnifications and multiple angles of view from 360-degree imaging data. It is a figure explaining classification. 7 is a diagram showing an example of an algorithm for the first trained model 204 or the second trained models 205_1 to 205_n. FIG. 1 is a diagram showing a functional block configuration of an image inspection apparatus 3 according to Embodiment 1. FIG. 1 is a diagram showing a hardware configuration of an image inspection apparatus 3 according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating the functions of a first trained model 305 (204). FIG. 7 is a diagram illustrating the functions of second trained models 306_1 to 306_n (205_1 to 205_n). 2 is a flow diagram showing the flow of processing by the trained model generation device 2 of the image inspection system 1. FIG. FIG. 3 is a diagram illustrating the flow of image data generation processing in more detail. 3 is a flow diagram showing the flow of first processing by the image inspection device 3 of the image inspection system 1. FIG. 3 is a flowchart showing the flow of second processing by the image inspection device 3 of the image inspection system 1. FIG.

1. Overview of Embodiments First, an overview of typical embodiments of the invention disclosed in this application will be described. In the following description, as an example, reference numerals on the drawings corresponding to constituent elements of the invention are written in parentheses.

[1] An image inspection apparatus (3) according to a typical embodiment includes an inspection data reception unit (33) that receives image data of an inspection target photographed in a building consisting of a plurality of areas, and a first trained model (305) for causing a computer to function so as to receive image data as input and output an area where the image data of the inspection target was photographed; and a plurality of trained models corresponding to each of the plurality of areas. 2, the trained model (306) is a trained model (306) for causing a computer to function so as to input the image data of the inspection object and output the location and direction in the area where the image data of the inspection object was taken. 2 trained model (306), and the test data reception unit (33) based on the first trained model (305) stored in the storage unit (38). ) outputs the area in which the image data of the inspection target received by The present invention is characterized by comprising a photographing state specifying unit (34) that outputs the location and direction in which the image data of the inspection object was photographed.

[2] In [1] above, the first trained model (305) is generated by labeling image data taken within the building with information indicating the area where the image data was taken. The second learned model (306) includes learned parameters adjusted by machine learning of a plurality of learning image data, and the second learned model (306) is configured to combine the image data taken within the building with the learned parameters. The learning parameters may include learned parameters adjusted by machine learning of a plurality of learning image data generated by labeling information indicating the location and direction.
[3] In the above [1] or [2], further comprising a detection object determination section (35) that determines whether or not a predetermined detection object is included in the image data of the inspection object, and the storage section (38) ) further stores detection target determination data that is a combination of area, location, and direction in which image data including a predetermined detection target can be photographed, and the detection target determination unit (35) is configured to determine the photographing state. When the combination of area, location, and orientation output in section (34) is included in the detection target determination data, a predetermined detection is applied to the image data of the inspection target related to the output combination of area, location, and orientation. It may be determined that the target object is included.

[4] In [3] above, the apparatus further includes an image data deletion section (36) for deleting the image data of the inspection object determined by the detection object determination section (35) to include a predetermined detection object. Good too.

[5] The trained model generation device (2) according to the representative embodiment generates data at multiple magnifications and angles of view from 360-degree imaging data taken by a 360-degree camera in a building consisting of multiple areas. A learning image data generation unit (22A) that generates a plurality of learning image data in which a plurality of cut out image data is labeled with the area, location, and direction in which the image was taken; a first trained model generation unit (26A) for causing the computer to function so as to output the areas that have been trained, and a plurality of second trained model generation units (26B) corresponding to each of the plurality of areas. and a second trained model generation unit (26B) for causing a computer to function to input the image data and output the location and direction in the area where the image data was photographed. It is characterized by

[6] The image inspection system (1) according to the representative embodiment extracts data at multiple magnifications and angles of view from 360-degree imaging data taken by a 360-degree camera in a building consisting of multiple areas. a learning image data generation unit (22A) that generates a plurality of learning image data in which a plurality of image data are labeled with photographed areas, locations, and directions; A first trained model generation unit (26A) that generates a first trained model (204, 305) for making the computer function to output the areas; A second trained model (205, 306), which is a second learning model for causing a computer to function so as to input the image data and output the location and direction in the area where the image data was taken. A trained model generation unit (22B) that generates trained models (205, 306), an inspection data reception unit (33) that receives image data of an inspection target, and the first trained model generation unit (26A). Based on the first learned model (305), the area where the image data of the inspection target received by the inspection data reception unit (33) was captured is output, and the area corresponding to the output area is output. The image data of the inspection object received by the inspection data reception unit (33) is photographed based on the second trained model (306) generated by the second trained model generation unit (26B). The present invention is characterized by comprising a photographing state specifying section (34) that outputs the location and direction.

[7] In [6] above, a detection object determination unit (35) that determines whether or not the image data of the inspection target includes a predetermined detection object; The detection target determination unit (35) further includes the storage unit (38) that further stores detection target determination data that is a combination of areas, places, and orientations that can be photographed, and the detection target determination unit (35) includes the photographing state specifying unit (34). If the combination of area, location, and orientation outputted in the detection target determination data is included in the detection target determination data, the image data of the inspection target related to the output combination of area, location, and orientation includes the predetermined detection target object. It may be determined that the

[8] In [7] above, the apparatus further comprises an image data deletion section (36) for deleting the image data of the inspection object determined by the detection object determination section (35) to include a predetermined detection object. Good too.

[9] The image inspection program according to the representative embodiment includes a computer, an inspection data reception unit (33) that receives image data of an inspection target photographed in a building consisting of a plurality of areas, and a first trained model (305) for making a computer function so as to input the image data of the object and output the area where the image data of the inspection target was taken; and a plurality of trained models corresponding to each of the plurality of areas. a second learned model (306) for causing a computer to function so as to input the image data of the inspection object and output the location and direction in the area where the image data of the inspection object was taken; a storage unit (38) that stores a second trained model (306); and a storage unit (38) that stores a second trained model (306); 33) outputs the area where the image data of the inspection target received by The first trained model (305) functions as a photographing state identification unit (34) that outputs the location and direction in which the received image data of the inspection target was photographed, and the first trained model (305) is configured to output image data photographed within the building. The second trained model (306) is generated by machine learning of a plurality of learning image data generated by labeling information indicating the area where the image data was taken. It is characterized by being generated by machine learning of a plurality of training image data generated by labeling image data taken inside a building with information indicating the location and direction in which the image data was taken. do.

[10] In [9] above, the apparatus further includes a detection object determination unit (35) that determines whether or not the image data of the inspection target includes a predetermined detection target object, and the storage unit (38) The detection object determination section (35) further stores detection object determination data that is a combination of areas, locations, and directions in which image data including the detection object can be photographed, and the detection object determination section (35) If the combination of area, location, and orientation outputted in the detection target determination data is included in the detection target determination data, the image data of the inspection target related to the output combination of area, location, and orientation includes the predetermined detection target object. It may be determined that the

[11] In [10] above, the apparatus further comprises an image data deletion section (36) for deleting image data of the inspection object determined by the detection object determination section (35) to include a predetermined detection object. Good too.

[12] The trained model generation program according to the representative embodiment causes a computer to generate images at multiple magnifications and angles of view from 360-degree imaging data taken by a 360-degree camera in a building consisting of multiple areas. A learning image data generation unit (22A) generates a plurality of learning image data by labeling the plurality of image data cut out with the photographed area, location, and direction; A first trained model generation unit (26A) for causing a computer to function to output photographed areas, and a plurality of second trained model generation units (26B) corresponding to each of the plurality of areas. and functioning as a second trained model generation unit (26B) for causing the computer to function so as to input the image data and output the location and direction in the area where the image data was photographed. It is characterized by

[13] The trained model according to the representative embodiment is a plurality of images generated by labeling image data taken inside a building consisting of a plurality of areas with information indicating the area in which the image data was taken. includes a first learned parameter adjusted by machine learning learning image data of , performs a calculation based on the first learned parameter on the input image data, and inputs the image data. A first trained model (204, 305) that causes the computer to function so as to output the area where the image data was taken, and a first trained model (204, 305) that corresponds to the area output by the first trained model (204, 305). and a second method for causing the computer to function so as to input the image data output from the area in the first learned model (204, 305) and output the location and direction in which the image data was taken. A plurality of trained models (205, 306) that are generated by labeling image data taken within the output area with information indicating the location and direction in which the image data was taken. It includes a second learned parameter adjusted by machine learning of image data, performs a calculation based on the second learned parameter on the input image data, and outputs the location and direction. , and a second trained model (205, 306) that causes the computer to function.

2. Specific Examples of Embodiments Hereinafter, specific examples of embodiments of the present invention will be described with reference to the drawings. In addition, in the following description, the same reference numerals are given to the same component in each embodiment, and repeated description is omitted. Furthermore, it should be noted that the drawings are schematic, and the dimensional relationship of each element, the ratio of each element, etc. may differ from reality. Drawings may also include portions that differ in dimensional relationships and ratios.

≪Embodiment 1≫
FIG. 1 is a diagram showing the configuration of an image inspection system according to the first embodiment.

The image inspection system 1 shown in the figure uses machine learning to learn image data taken inside a building consisting of multiple areas to generate a learned model, and then uses the learned model to generate an input image. This is a system for specifying the area, place, and direction in which data was photographed, and checking whether a predetermined detection target is included based on the specified area, place, and direction.

In this embodiment, a case where the object is installed within a predetermined area will be described as an example. In this specification, an installed object refers to a structure that is fixedly placed in a building and whose position does not usually change. Examples include wall clocks, ceiling lights, emergency lights, security cameras, doors, windows, and stairs, and the concept also includes the structure of the building itself.

As shown in FIG. 1, the image inspection system 1 is configured to be connectable to a network 4, and connects to a plurality of external information processing devices such as client terminal devices 5a, 5b, 5c, 5d, etc. via the network 4. Communication is possible.

The network 4 is, for example, a wide area network (WAN) typified by the Internet. The image inspection system 1 is configured to be connectable to a network 4 via wired or wireless communication.

The client terminal devices 5a, 5b, 5c, 5d, . . . are client-side information processing devices for requesting predetermined processing (services) from the image inspection system 1 as a server. The client terminal devices 5a, 5b, 5c, 5d, . It has become.

Examples of the client terminal devices 5a, 5b, 5c, 5d, . . . include various information processing devices (computers) such as a personal computer, a tablet terminal, or a smartphone.

Further, an imaging device 51 such as a 360-degree camera is connected to the client terminal devices 5a, 5b, 5c, 5d, etc., and transmits the captured 360-degree imaging data to the image inspection system 1 via the network 4. be able to. Of course, the client terminal devices 5a, 5b, 5c, 5d... themselves may have a 360-degree camera imaging function.

As shown in FIG. 1, the image inspection system 1 identifies the imaging state (imaging area, location, direction) of the input image data based on the learned model, and uses the image inspection system 1 to determine the image data based on the identified imaging state. The image inspection device 3 includes an image inspection device 3 capable of inspecting whether or not the object includes a predetermined detection target (for example, a wall clock), and a learned model generation device 2 that generates the learned model.

The trained model generation device 2 and the image inspection device 3 are connected, for example, via a predetermined network (for example, LAN: Local Area Network), and are capable of transmitting and receiving data to and from each other.

The trained model generation device 2 and the image inspection device 3 will be described in detail below.

First, the trained model generation device 2 will be explained.

FIG. 2 is a diagram showing a functional block configuration of the trained model generation device 2 according to the first embodiment.

FIG. 3 is a diagram showing the hardware configuration of the trained model generation device 2 according to the first embodiment.

The trained model generation device 2 is, for example, an information processing device (computer) such as a PC or a server, and generates learning image data according to an installed trained model generation program, and also generates learning image data. This is a device that learns data using machine learning and generates a trained model. In this way, the trained model generation device 2 may be implemented as a trained model generation program that runs on a computer.

As shown in FIG. 2, the trained model generation device 2 includes a communication unit 21, a learning image data generation unit 22A, a storage unit 27, and a learned model generation unit 22B as functional blocks for generating a trained model. have. Each of these functional blocks is created by the hardware resources that make up the information processing device as the trained model generation device 2 working together with the software (trained model generation program) installed in the information processing device. , realized.

As shown in FIG. 3, the trained model generation device 2 includes a calculation device 101, a storage device 102, an input device 103, an I/F (interface) device 104, an output device 105, and a bus 106 as hardware resources. ing.

The program 1021 stored in the storage device 102 includes a trained model generation program that causes the computer to function as each functional unit of the trained model generation device 2 according to the present embodiment, and includes, for example, a trained model generation program that is installed in the storage device 102 in advance. ing.

The data 1022 also includes 360-degree imaging data 201_1 to 201_p (p is an integer of 2 or more), folders 202_1 to 202_n of multiple areas storing image data (n is an integer of 2 or more), and image data, which will be described later. Folders 203_1 to 203_m (m is an integer of 2 or more) containing multiple stored locations and orientations, a first trained model 204 that is a first trained model, and a second trained model that is a second trained model It includes at least one of models 205_1 to 205_n (n is an integer of 2 or more).

Note that the program 1021 and the data 1022 may be distributed via a network, or may be distributed by being written on a non-transitory computer readable medium such as a CD-ROM. It's okay.

Each functional block of the trained model generation device 2 will be described below.

The communication unit 21 is a functional unit for communicating with an external information processing device. Specifically, the communication unit 21 transmits and receives data to and from the image inspection apparatus 3 in the image inspection system 1 via the above-described predetermined network (for example, LAN).

The communication unit 21 is connected to the network 4 and can send and receive data to and from external information processing devices such as a plurality of client terminal devices 5a, 5b, 5c, 5d, etc. via the network 4. It may be configured as follows.

The learning image data generation unit 22A is a functional unit that generates various data for generating a learned model. Specifically, the learning image data generation unit 22A generates a plurality of learning image data (teacher data) generated by labeling image data taken inside a building with information indicating the area where the image data was taken. ), and a plurality of learning image data (teacher data) are generated by labeling information indicating the location and direction where the image data was taken. Image data taken inside a building as a learning target by the learned model generation device 2 of this embodiment can be provided with the area and location where the image was taken at the time of the first image. The learning image data generation unit 22A can perform labeling using the area and location given to the image data. Furthermore, when cutting out a plurality of image data at a plurality of angles of view from the photographed image data, the learning image data generation unit 22A performs labeling using the orientation estimated from the angle of view cut out from the photographed image data. be able to. The photographed image data is photographed with a specific direction as a reference, and the direction can be estimated based on the angle of view from the reference position in the photographed image data.

Note that in this embodiment, the area, location, and direction will be exemplified as information indicating the state when the image data was captured, but the information is not limited thereto.

As shown in FIG. 2, the learning image data generation section 22A includes a zenith correction section 23, an image data generation section 24, a first classification section 25A, and a second classification section 25B as functional blocks. .

The zenith correction unit 23 is a functional unit that performs zenith correction of the 360-degree imaging data 201_1 to 201_p (p is an integer of 2 or more).

The image data generation unit 24 is a functional unit that cuts out a plurality of image data at a plurality of magnifications and a plurality of angles of view from the 360-degree imaging data 201_1 to 201_p. Here, the angle of view refers to the size occupied by image data in each of the horizontal and vertical directions out of 360 degrees.

The first classification unit 25A is a functional unit that classifies a plurality of cut out image data by area. Classifying the image data by area includes labeling the image data with information indicating the area. Specifically, the image data is classified into folders 202_1 to 202_n (n is an integer of 2 or more) in a plurality of areas and saved. The image data will be labeled with information indicating the area corresponding to the saved folder.

The second classification unit 25B is a functional unit that classifies the plurality of cut out images by location and orientation. Classifying the image data by location and orientation includes labeling the image data with information indicating the location and orientation. Specifically, the image data is classified into folders 203_1 to 203_m (m is an integer of 2 or more) of a plurality of locations and orientations and saved. The image data will be labeled with information indicating the location and orientation corresponding to the saved folder. The folders 203_1 to 203_m for a plurality of locations and orientations may be stored in any of the folders 202_1 to 202_n of the area including the location, as shown in FIG.

FIG. 4 is a diagram for explaining zenith correction. FIG. 5 is a diagram illustrating cutting out image data of a plurality of magnifications and a plurality of angles of view from 360-degree imaging data. FIG. 6 is a diagram explaining classification. Here, generation of image data in the learning image data generation section 22A will be described with reference to FIGS. 2 to 6.

The 360-degree imaging data can be obtained, for example, by taking pictures while moving inside the target building using an imaging device equipped with two lenses on the front and back sides capable of taking 180-degree pictures. Information about the area and location (position within the area) where the image was taken is associated with the 360-degree imaging data as metadata.

The learning image data generation unit 22A performs zenith correction in the zenith correction unit 23 when the 360-degree imaging data is obtained and stored in the storage unit 27. Zenith correction is, as shown in FIG. 4, a correction process that corrects the vertical and horizontal directions by rotating the image, when the vertical and horizontal directions of the image are misaligned. With this function, when photographing 360-degree imaging data, it is possible to perform photographing without adjusting the vertical and horizontal directions.

As shown in FIG. 5, the image data generation unit 24 cuts out a plurality of image data from one piece of 360-degree imaging data that has been subjected to zenith correction as necessary. In this cutting, it is preferable to cut out as much image data as possible so that a plurality of celestial spheres having different radii centered on the imaging position are completely filled with the image data after cutting. As shown in FIG. 6, the cut out image data is immediately passed to the first classification section 25A and classified by area, and the second classification section 25B classifies the plurality of cut out images by location and direction. be done. The folders used for classification may be nested, as shown in FIG. 2, or may be organized into folders for each combination of area, location, and orientation, as shown in FIG.

In this embodiment, a first trained model 204 that is a first trained model and second trained models 205_1 to 205_n (n is an integer of 2 or more) that are second trained models are stored. It is stored in section 27.

The learned model generation unit 22B performs machine learning on the plurality of learning image data generated by the learning image data generation unit 22A, and uses the image data as input to determine the area, location, and direction in which the image data was captured. Generate a first trained model 204 and second trained models 205_1 to 205_n as a first trained model and a second trained model for making the information processing device (computer) function to output. It is a functional part.

The first trained model 204 and the second trained models 205_1 to 205_n are machine learning programs based on a predetermined technique (algorithm).

Here, as the predetermined technique (algorithm), for example, a neural network such as CNN (Convolutional Neural Network), a decision tree, a random forest, a logistic regression, a support vector machine (SVM), a Naive Bayes classifier, a k-nearest neighbor method, It includes at least one of the k-means method, AdaBoost, and Markov chain.

That is, the first trained model 204 performs calculations based on predetermined learned parameters on the input image data, and outputs the area where the image data was photographed. ) is a program to make it work. Similarly, the second trained models 205_1 to 205_n perform calculations based on predetermined learned parameters on the input image data, and output the location and direction where the image data was taken. It is a program for making an information processing device (computer) function.

Here, the learned parameters mean that the image data is created by using the learning image data stored in the predetermined folders 202_1 to 202_n and 203_1 to 203_n as input to the learning program (a program based on the above-mentioned predetermined algorithm). These are parameters that are mechanically adjusted to output the photographed area. For example, in the case of a neural network, the learned parameters are variables such as weighting coefficients. The output area can include the probability of being in that area.

FIG. 7 is a diagram showing an example of an algorithm for the first trained model 204 or the second trained models 205_1 to 205_n. FIG. 7 shows an example in the case of a neural network. The first trained model 204 and the second trained models 205_1 to 205_n each have an input layer consisting of a plurality of nodes N1 to N5... and an output layer consisting of a plurality of nodes M1 to M5... This algorithm has a structure in which a plurality of intermediate layers exist between them, and each node is connected to a node in an adjacent layer using an arbitrary weighting parameter. For example, if you raster scan image data labeled with the captured area or location and orientation, and input each raster scanned block (pixel) to the input layer, the value of the weighting parameter will be adjusted based on the output corresponding to the input. do. That is, the trained model generation unit 22B uses the image data used for learning and the correct data (identification information: labeled area or location and direction), and calculates the error between the input result and the correct data using the error back propagation method. Thus, the learned parameters are sequentially updated to generate the first learned model 204 or the second learned models 205_1 to 205_n, which are learned models.

The first trained model 204 or the second trained models 205_1 to 205_n may include an inference program in addition to the learned parameters described above. The inference program is a program that incorporates the learned parameters and outputs a certain result for input image data.

The first trained model 204 or the second trained models 205_1 to 205_n generated by the trained model generation unit 22B are stored in the storage unit 27, and the first It is registered as the trained model 204 or the second trained models 205_1 to 205_n. For example, the image inspection device 3 communicates with the learned model generation device 2, so that the image inspection device 3 receives the first learned model 204 or the second learned models 205_1 to 205_n from the learned model generation device 2. It is acquired and stored in the storage unit 38 as the first trained model 305 or the second trained models 306_1 to 306_n.

Thereby, the image inspection apparatus 3 uses the first trained model 305 or the second trained models 306_1 to 306_n) stored in the storage unit 38 to determine the area, location, and location where the input image data was photographed. It becomes possible to identify the location.

Note that the learning image data generation unit 22A and the trained model generation unit 22B in the trained model generation device 2 may be realized by program processing by one processor, but they may be realized by program processing by separate processors. It's okay. That is, the arithmetic unit 101, which is one of the hardware resources of the trained model generation device 2, includes at least one processor that executes data processing for generating image data for learning, and a processor for generating the trained model. At least one processor that performs data processing may be provided separately.

FIG. 8 is a diagram showing a functional block configuration of the image inspection apparatus 3 according to the first embodiment.

FIG. 9 is a diagram showing the hardware configuration of the image inspection apparatus 3 according to the first embodiment.

The image inspection device 3 is an information processing device (computer) such as a PC (personal computer) or a server, similar to the learned model generation device 2, and generates a learned model according to an installed image inspection program. This is a device that detects image data that includes a predetermined object to be detected by specifying the area, location, and direction in which the input image data was photographed. In this way, the image inspection device 3 may be implemented as an image inspection program running on a computer.

As shown in FIG. 8, the image inspection device 3 serves as a functional block for performing various data processing on input image data using a trained model to detect image data that includes a predetermined detection target object. As functional blocks, an image inspection section 32 including a communication section 31, an inspection data reception section 33, an imaging state identification section 34, a detection target determination section 35, an image data deletion section 36, and an inspection result data transmission processing section 37; It has a storage section 38. Each of these functional blocks is realized by the hardware resources constituting the information processing device as the image inspection device 3 working together with the software (image inspection program) installed in the information processing device. .

As shown in FIG. 9, the image inspection device 3, like the trained model generation device 2, includes a calculation device 101, a storage device 102, an input device 103, an I/F (Interface) device 104, and an output device as hardware resources. A device 105 and a bus 106 are provided.

The arithmetic unit 101 includes a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The storage device 102 has a storage area that stores a program 1023 for causing the arithmetic device 101 to execute various data processes, and data 1024 such as parameters and calculation results used in data processing by the arithmetic device 101. For example, it is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD, a flash memory, and the like.

Here, the program 1023 includes an image inspection program for causing the computer (processor) to function as each functional unit related to the image inspection apparatus 3, and is installed in the storage device 102 in advance, for example.

In addition, the data 1024 includes inspection target image data 301_1 to 301_q (q is an integer of 2 or more), identification results 302_1 to 302_r (r is an integer of 2 or more), deletion target determination data 303, and inspection result data 304_1 to 301_q, which will be described later. 304_s (s is an integer of 2 or more), a first trained model 305, and second trained models 306_1 to 306_n (n is an integer of 2 or more).

Note that the program 1023 and the data 1024 may be distributed via a network, or may be distributed by being written on a non-transitory computer readable medium such as a CD-ROM. It's okay.

The input device 103 is a functional unit that detects input of information from the outside, and includes, for example, a keyboard, a mouse, a pointing device, a button, a touch panel, or the like. The I/F device 104 is a functional unit that transmits and receives information to and from the outside, and is composed of a communication control circuit, an input/output port, an antenna, etc. for wired or wireless communication.

The output device 105 is a functional unit that outputs information etc. obtained by data processing by the arithmetic device 101. Examples of the output device 105 include external storage devices such as SSD and HDD, and display devices such as LCD (Liquid Crystal Display) and organic EL (Electro Luminescence). The bus 106 is a functional unit that interconnects the arithmetic device 101, the storage device 102, the input device 103, the I/F device 104, and the output device 105, and enables data to be exchanged between these devices.

In the image inspection device 3, the arithmetic device 101 executes calculations according to the program 10231 stored in the storage device 102, and controls the storage device 102, the input device 103, the I/F device 104, the output device 105, and the bus 106. By doing so, each functional block of the image inspection apparatus 3 shown in FIG. An image inspection section 32 including a transmission processing section 37, and a storage section 38) are realized.

Each functional block of the image inspection apparatus 3 will be explained below.

The communication unit 31 is a functional unit for communicating with an external information processing device.

Specifically, the communication unit 31 is connected to the network 4 and transmits and receives data to and from external information processing devices such as a plurality of client terminal devices 5a, 5b, 5c, 5d, etc. via the network 4. I do.

The communication unit 31 also transmits and receives data to and from the learned model generation device 2 in the image inspection system 1 via the above-described predetermined network (for example, LAN).

The inspection data receiving unit 33 is a functional unit that receives image data to be inspected. Here, accepting the image data to be inspected means inputting the image data to be inspected to the image inspection apparatus 3 from the outside, and also includes acquiring the image data to be inspected. Image data to be inspected can be input from a plurality of client terminal devices 5a, 5b, 5c, 5d, . . . via the network 4, or can be input directly to the image inspection device 3. The inspection data receiving unit 33 stores the received image data to be inspected in the storage unit 38 as image data to be inspected 301_1 to 301_q.

The imaging state specifying unit 34 inputs the inspection target image data 301_1 to 301_q stored in the storage unit 38, and generates a first trained model 305 and a second trained model 306_1 generated by the trained model generation device 2. 306_n (algorithm) is used to identify the area, location, and direction in which the inspection target image data 301_1 to 301_q were photographed. The imaging state specifying unit 34 can associate the inspection target image data with the identified area, location, and direction and store it in the storage unit 38 as identification results 302_1 to 302_r.

Specifically, the photographing state identifying unit 34 identifies the area where the inspection target image data 301_1 to 301_q have been photographed by the first trained model 305. Further, the location and direction in which the image data to be inspected 301_1 to 301_q were photographed are identified using the second trained model 306 corresponding to the area identified among the second trained models 306_1 to 306_n.

Here, the first trained model 305 and the second trained models 306_1 to 306_n stored in the storage unit 38 are the first trained model 305 and the second trained models 306_1 to 306_n stored in the storage unit 27, which are generated by the trained model generation device 2 and stored in the storage unit 27. This is the same as the trained model 204 or the second trained models 205_1 to 205_n. For example, the imaging state specifying unit 34 of the image inspection device 3 communicates with the learned model generating device 2, so that the first learned model 204 or the second learned model is stored from the storage unit 27 of the learned model generating device 2. The trained models 205_1 to 205_n are acquired and stored in the storage unit 38 as the first trained model 305 or the second trained models 306_1 to 306_n.

FIG. 10 is a diagram explaining the functions of the first trained model 305 (204), and FIG. 11 is a diagram explaining the functions of the second trained models 306_1 to 306_n (205_1 to 205_n).

In FIGS. 10 and 11, left and right arrows indicate machine learning in the trained model generation device 2, and upper and lower arrows indicate identification in the image inspection device 3. As shown by the left and right arrows in FIGS. 10 and 11, in machine learning in the trained model generation device 2, labeled image data is input and parameters are adjusted. On the other hand, as shown by the up and down arrows in FIGS. 10 and 11, in the identification in the image inspection device 3, the image data to be inspected is input, and (in the case of the first trained model) its imaging area or (the second (in the case of a trained model), its imaging location and orientation are output.

The detection target determination unit 35 determines whether or not the combination of area, location, and orientation related to the image data of the identification results 302_1 to 302_r matches the combination of area, location, and orientation included in the deletion target determination data 303. , a functional unit that detects image data that includes a predetermined detection target object. The combination of area, location, and orientation included in the deletion target determination data 303 is stored in advance as a combination of area, location, and orientation where image data including a predetermined detection target can be photographed.

The image data deletion unit 36 is a functional unit that deletes image data detected as image data including a predetermined detection target object. The image data deletion unit 36 stores the image data that has not been deleted in the storage unit 38 as test result data 304_1 to 304_s.

The test result data transmission processing unit 37 can transmit the test result data 304_1 to 304_s to the client terminal devices 5a, 5b, 5c, 5d, . . . as necessary.

Next, the flow of processing by the image inspection system 1 will be explained.

FIG. 12 is a flow diagram showing the flow of processing by the trained model generation device 2 of the image inspection system 1. FIG. 13 is a diagram illustrating the flow of image data generation processing in more detail.

First, as shown in FIG. 12, in the image inspection system 1, the trained model generation device 2 acquires 360-degree imaging data (step S101). The 360-degree imaging data may be acquired via the network 4, or may be acquired by being directly input to the trained model generation device 2 itself.

When the 360-degree imaging data is acquired, the learning image data generation unit 22A generates image data (step S103). As shown in FIG. 13, image data generation includes zenith correction processing in the zenith correction section 23 (step S201), generation of images with a plurality of different zoom magnifications in the image data generation section 24 (step S203), and a plurality of This includes cutting out image data at the viewing angle (step S205).

The cut out image data is classified by area by the first classification unit 25A (step S105), and classified by location and direction by the second classification unit 25B (step S107).

Next, the classified image data is subjected to first learning (machine learning) in the first trained model 204 by the first trained model generation unit 26A (step S109), and then to the second trained model generation unit 26A. 26B performs second learning (machine learning) on the second trained models 205_1 to 205_n (step S111).

Through the above processing, the trained model generation device 2 of the image inspection system 1 generates the first trained model 204 and the second trained models 205_1 to 205_n.

FIG. 14 is a flow diagram showing the flow of the first process by the image inspection apparatus 3 of the image inspection system 1. The first processing flow is a process from receiving image data to be inspected to identifying it.

As shown in FIG. 14, in the image inspection apparatus 3, the inspection data reception unit 33 receives inspection target image data (step S301). Receiving the image data to be inspected can include acquiring a photographed image.

The imaging state specifying unit 34 inputs the inspection target image data to the first trained model 305 (step S303), and outputs the area (step S305), so the second learning is performed according to the output area. Image data to be inspected is further input to the completed model (step S307). As a result, the location and orientation are output from the second learned model (step S309). The area, location, and orientation of the output image data are stored in the storage unit 38 in association with the image data as identification results 302_1 to 302_r.

FIG. 15 is a flowchart showing the flow of the second process by the image inspection apparatus 3 of the image inspection system 1. The second processing flow is a process from identifying image data to be inspected to deleting image data including a predetermined detection object.

As shown in FIG. 14, in the image inspection apparatus 3, the detection target determination unit 35 acquires the identification results 302_1 to 302_r (step S401). It is compared with the combination of area, location, and direction included in the deletion target determination data 303 (step S403).

As a result of the comparison in step S403, if it is determined that the combination of area, location, and orientation included in the deletion target determination data 303 is determined (step S405: Yes), the image data deletion unit 36 deletes the image that is determined to be a match. The data is deleted from the identification results 302_1 to 302_r (step S407). The image data deletion unit 36 can store the image data that has not been deleted in the storage unit 38 as inspection result data 304_1 to 304_s.
The image inspection device 3 can output the inspection result data 304_1 to 304_s stored in the storage unit 38 to the client terminal devices 5a to 5d, etc., as necessary.

According to the embodiments described above, even in a building where it is difficult to specify the location using GPS or the like, it is possible to quickly identify the shooting location based on the shot image, despite the simple configuration. For example, in a building at a nuclear power plant, equipment that should not be photographed can be a predetermined detection target. In this case, image data taken inside a building at a nuclear power plant is input to the image inspection device 3 of the image inspection system 1 as image data to be detected, and the area, location, and direction (location, location, etc.) of the image data to be detected are etc.), and from the combination of identified areas, locations, and orientations (shooting locations, etc.), it is determined whether equipment that should not be photographed is included in the image data to be detected, and equipment that should not be photographed is determined. It is possible to delete image data that is determined to include.

(Modified example of embodiment)
In the above embodiments, image data is deleted based on image data that includes a predetermined detection target object, but the present invention is not limited to this, and predetermined processing can be performed.

Further, the configurations of each of the image inspection system 1, trained model generation device 2, and image inspection device 3 are merely examples, and the specific configurations are not limited thereto.

Furthermore, the above-described flowchart shows an example for explaining the operation, and is not limited thereto. That is, the steps shown in each figure of the flowchart are specific examples, and the flowchart is not limited to this flow. For example, the order of some processes may be changed, other processes may be inserted between each process, or some processes may be performed in parallel.

In this embodiment, the first trained model and the second trained model are described as separate trained models, but depending on the output of the first trained model, a plurality of trained models By providing a configuration for re-inputting an appropriate one of the two trained models, a trained model including a first trained model and a second trained model may be configured. .

DESCRIPTION OF SYMBOLS 1... Image inspection system, 2... Learned model generation device, 3... Image inspection device, 4... Network, 5a... Client terminal device, 5b... Client terminal device, 5c... Client terminal device, 5d... Client terminal device, 51... Imaging device, 21...Communication unit, 22A...Learning image data generation unit, 22B...Learned model generation unit, 23...Zenith correction unit, 24...Image data generation unit, 25A...First classification unit, 25B...Second classification unit, 26A...first trained model generation unit, 26B...second trained model generation unit, 27...storage unit, 201...360 degree imaging data, 202...area folder, 203...location, orientation folder, 204...First trained model, 205...Second trained model, 101...Arithmetic unit, 102...Storage device, 1021...Program, 1022...Data, 103...Input device, 104...I/F device, 105... Output device, 106...Bus, 1023...Program, 1024...Data, 31...Communication unit, 32...Image inspection unit, 33...Inspection data reception unit, 34...Photography state identification unit, 35...Detection target determination unit, 36...Image Data deletion unit, 37... Inspection result data transmission processing unit, 38... Storage unit, 301... Inspection target image data, 302... Identification result, 303... Deletion target determination data, 304... Inspection result data, 305... First learning trained model, 306...second trained model

Claims (7)

an inspection data reception unit that accepts image data of inspection targets taken inside a building consisting of multiple areas;
a first trained model for causing a computer to function so as to input the image data of the inspection target and output an area where the image data of the inspection target was taken; and a plurality of trained models corresponding to each of the plurality of areas. A second trained model for causing a computer to function so as to input the image data of the inspection object and output the location and direction in the area where the image data of the inspection object was taken. a storage unit that stores the trained model;
Based on the first trained model stored in the storage unit, the area where the image data of the inspection object received by the inspection data reception unit was captured is output, and the first trained model corresponding to the output area is output. a photographing state specifying unit that outputs the location and direction in which the image data of the inspection object received by the inspection data receiving unit was photographed based on the learned model of No. 2;
a detection target determination unit that determines whether a predetermined detection target object is included in the image data of the inspection target;
Equipped with
The storage unit further stores detection object determination data that is a combination of an area, a location, and a direction in which image data including a predetermined detection object can be photographed;
The detection target determination unit performs an inspection on the combination of area, location, and orientation output by the imaging state identification unit when the combination of area, location, and orientation is included in the detection target determination data. determining that the target image data includes a predetermined detection target;
Image inspection equipment. The image inspection apparatus according to claim 1,
The first trained model is created by machine learning a plurality of training image data generated by labeling image data taken inside the building with information indicating the area where the image data was taken. Contains tuned learned parameters,
The second trained model performs machine learning on a plurality of training image data generated by labeling image data taken within the building with information indicating the location and direction in which the image data was taken. Contains learned parameters adjusted by
Image inspection equipment. The image inspection apparatus according to claim 1 or 2 ,
further comprising an image data deletion unit that deletes image data of the inspection target determined by the detection target determination unit to include a predetermined detection target object;
Image inspection equipment. Multiple image data are extracted from 360-degree imaging data taken by a 360-degree camera in a building consisting of multiple areas at multiple magnifications and multiple angles of view, and are labeled with the area, location, and direction in which the images were taken. a learning image data generation unit that generates learning image data;
a first trained model generation unit that generates a first trained model for causing a computer to function to input the image data and output an area where the image data is taken; and each of the plurality of areas. a plurality of second trained models corresponding to the image data, the second learning for causing the computer to function so as to input the image data and output the location and direction in the area where the image data was photographed; a trained model generation unit that generates a trained model;
an inspection data reception unit that accepts image data of the inspection target;
Based on the first trained model generated by the first trained model generation unit, output an area where the image data of the inspection target received by the inspection data reception unit was captured; the location and direction in which the image data of the inspection object received by the inspection data reception unit was captured based on the second trained model generated by the second trained model generation unit corresponding to the area a shooting state identification section that outputs the
a detection target determination unit that determines whether a predetermined detection target object is included in the image data of the inspection target;
a storage unit that stores detection object determination data that is a combination of areas, locations, and directions in which image data including a predetermined detection object can be photographed;
Equipped with
The detection target determination unit performs an inspection on the combination of area, location, and orientation output by the imaging state identification unit when the combination of area, location, and orientation is included in the detection target determination data. determining that the target image data includes a predetermined detection target;
Image inspection system. The image inspection system according to claim 4 ,
further comprising an image data deletion unit that deletes image data of the inspection target determined by the detection target determination unit to include a predetermined detection target object;
Image inspection system. computer,
an inspection data reception unit that accepts image data of inspection targets taken inside a building consisting of multiple areas;
a first trained model for causing a computer to function so as to input the image data of the inspection target and output an area where the image data of the inspection target was taken; and a plurality of trained models corresponding to each of the plurality of areas. A second trained model for causing a computer to function so as to input the image data of the inspection object and output the location and direction in the area where the image data of the inspection object was taken. a storage unit that stores the trained model;
Based on the first trained model stored in the storage unit, the area where the image data of the inspection object received by the inspection data reception unit was captured is output, and the first trained model corresponding to the output area is output. a photographing state specifying unit that outputs the location and direction in which the image data of the inspection object received by the inspection data receiving unit was photographed based on the learned model of No. 2 ;
functioning as a detection target determination unit that determines whether a predetermined detection target is included in the image data of the inspection target ;
The first trained model is created by machine learning a plurality of training image data generated by labeling image data taken inside the building with information indicating the area where the image data was taken. has been generated,
The second trained model performs machine learning on a plurality of training image data generated by labeling image data taken within the building with information indicating the location and direction in which the image data was taken. It is generated by
The storage unit further stores detection object determination data that is a combination of an area, a location, and a direction in which image data including a predetermined detection object can be photographed;
The detection target determination unit performs an inspection on the combination of area, location, and orientation output by the imaging state identification unit when the combination of area, location, and orientation is included in the detection target determination data. determining that the target image data includes a predetermined detection target;
Image inspection program. The image inspection program according to claim 6 ,
further comprising an image data deletion unit that deletes image data of the inspection target determined by the detection target determination unit to include a predetermined detection target object;
Image inspection program.

Images