JP7744037B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to an image processing device, an image processing method, and a program.

Patent Document 1 discloses a technology that compares a reference image of an object with a newly captured image of the object, and determines whether there are any abnormalities in the object captured in the newly captured image.

Patent Document 1 discloses that a comparison reference image and an image on the day of inspection are used to extract multiple feature points from each image, and the feature points of one image are matched to those of the other image, and the other image is projected and changed so that the deviation of the feature points between the images is minimized for the matched feature points (Patent Document 1, paragraphs 0047 to 0051, etc.).

Japanese Patent Application Laid-Open No. 2019-3574

As mentioned above, in technology that compares a reference image with images taken at different times to determine whether an object is abnormal, there is a demand for technology that can accurately determine whether an object is abnormal even when the object is moving at high speed.

The purpose of this disclosure is to provide an image processing device, an image processing method, and a program that solve the above problems.

According to a first aspect of this disclosure, the image processing device includes a correspondence relationship identification means for identifying correspondence relationships between identical feature points in a base image captured of a moving object moving in a predetermined movement direction and an evaluation image captured of the moving object at a timing different from the timing of capturing the base image; a section identification means for identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the base image at a predetermined interval in the movement direction based on changes in the difference in position in the movement direction of the identical feature points captured in the base image and the evaluation image; and a new evaluation image generation means for generating a new evaluation image in which images of the cut-out section are arranged sequentially in the movement direction.

According to a second aspect of this disclosure, an image processing method identifies correspondences between identical feature points in a base image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from that of the capture of the base image; identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the base image at a predetermined interval in the direction of movement based on changes in the difference in position in the direction of movement of the identical feature points captured in the base image and the evaluation image; and generates a new evaluation image in which images of the cut-out section are arranged sequentially in the direction of movement.

According to a third aspect of this disclosure, the program causes a computer of an image processing device to function as: a correspondence relationship identification means for identifying correspondence relationships between identical feature points in a base image captured of a moving object moving in a predetermined movement direction and an evaluation image captured of the moving object at a timing different from the timing of capturing the base image; a section identification means for identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the base image at a predetermined interval in the movement direction based on changes in the difference in position in the movement direction of the identical feature points captured in the base image and the evaluation image; and a new evaluation image generation means for generating a new evaluation image in which images of the cut-out section are arranged sequentially in the movement direction.

FIG. 1 is a first diagram illustrating a configuration of an image processing system according to an embodiment of the present disclosure. FIG. 1 is a hardware configuration diagram of an image processing device according to an embodiment of the present disclosure. FIG. 1 is a functional block diagram of an image processing device according to an embodiment of the present disclosure. 1A and 1B are diagrams illustrating examples of a base image and an evaluation image according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating a processing flow of an image processing device according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating a relationship between feature points in a base image and an evaluation image according to an embodiment of the present disclosure. FIG. 10 is a first diagram illustrating an example of a correspondence relationship specifying process according to an embodiment of the present disclosure. FIG. 10 is a second diagram illustrating an example of a correspondence relationship specifying process according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an overview of a correspondence removal process according to an embodiment of the present disclosure. FIG. 1 is a first diagram illustrating an overview of a section identification process according to an embodiment of the present disclosure. FIG. 10 is a second diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. FIG. 10 is a third diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating another configuration of an image processing device according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating another processing flow of the image processing device according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating another configuration of an image processing device according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating another processing flow of the image processing device according to an embodiment of the present disclosure.

FIG. 1 is a first diagram showing the configuration of an image processing system according to an embodiment of the present disclosure.
The image processing system 100 is configured by a communication connection between an image processing device 1 and a line scan camera 2. The line scan camera 2 is installed so as to capture an image of a moving object 5 moving in a predetermined direction. In FIG. 1 , the line scan camera 2 is installed in a fixed position and captures an image of the moving object 5 moving to the right. The moving object 5 may be, for example, a train. The line scan camera 2 repeatedly captures images of the moving object 5 at predetermined short time intervals. With each capture, the image captured by the line scan camera 2 becomes a long, narrow image. The line scan camera 2 sequentially transmits the images generated by the capture to the image processing device 1. By using the line scan camera 2, it is possible to capture the entire moving object 5, which is long in the direction of movement.

The image processing device 1 combines images acquired from the line scan camera 2 to generate an image that captures a predetermined range of the moving object 5. The image of the predetermined range may be the entire moving object 5, or if the moving object 5 is a train with multiple cars, it may be one car or a predetermined number of cars of the train. The image processing device 1 may compare each image (in this disclosure, a base image and an evaluation image) generated each time the same moving object passes in front of the line scan camera 2 and output the comparison result. The comparison result may be the presence or absence of an abnormality in the moving object 5, etc.

Here, the image processing device 1 generates one of the images each time the same moving object 5 passes in front of the line scan camera 2 as a base image, and the other as an evaluation image. The image processing device 1 may determine whether there is a difference (such as an abnormality) between the evaluation image and the base image. The base image may be an image taken when the moving object 5 is normal, and the evaluation image may be an image taken when the moving object 5 is inspected after a period of time has passed.

In this disclosure, the image processing device 1 performs the following processing to generate a base image and an evaluation image. Specifically, the image processing device 1 identifies correspondences between identical feature points in the base image and the evaluation image. Based on changes in the difference in position between the identical feature points that have correspondences between the base image and the evaluation image, the image processing device 1 identifies cut-out sections in the evaluation image that correspond to sections obtained by dividing the base image at predetermined intervals in the movement direction. Each cut-out section in the evaluation image is reduced or enlarged so that each cut-out section in the evaluation image matches the length of an image in each section of the base image. A new evaluation image is generated in which the images of each cut-out section after being reduced or enlarged are arranged sequentially in the movement direction.

FIG. 2 is a hardware configuration diagram of an image processing device according to an embodiment of the present disclosure.
As shown in FIG. 2, the image processing device 1 is a computer equipped with various hardware components, such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, other storage devices 104, and a communication module 105.

FIG. 3 is a functional block diagram of an image processing device according to an embodiment of the present disclosure.
The CPU 101 of the image processing device 1 executes an image processing program, which causes the image processing device 1 to perform the functions of an acquisition unit 11, an image generation unit 12, a correspondence relationship identification unit 13, a correspondence relationship removal unit 14, a section identification unit 15, an image conversion unit 16, a new evaluation image generation unit 17, and an abnormality detection unit 18.

The acquisition unit 11 acquires image data from the line scan camera 2 .
The image generating unit 12 generates a base image and an evaluation image.
The correspondence specifying unit 13 specifies the correspondence between the same feature points in the base image and the evaluation image.
The correspondence removal unit 14 identifies identical feature points that have a correspondence relationship between the basic image and the evaluation image and that result in an outlier due to a change in the difference in position of the identical feature points, and removes the correspondence relationship for the identical feature points from the correspondence identification result of the correspondence identification unit 13.
The section identification unit 15 identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the direction of movement of the moving body, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image.
The image conversion unit 16 reduces or enlarges each of the cut-out sections in the evaluation image so that the length of each cut-out section in the evaluation image matches the length of one section of each of the basic images.
The new evaluation image generating unit 17 generates a new evaluation image by sequentially arranging the images of each section after reducing or enlarging the cut-out section in the direction of movement of the moving object.
The abnormality detection unit 18 compares the base image with the new evaluation image to detect abnormalities in the moving object.

The image processing device 1 may be configured such that each of the above functions is performed by a single computer, or such that multiple computers work together to perform each of the above functions. When multiple computers work together to configure the image processing device 1, any one of the computers will perform any of the above functions, allowing the entire system to perform each of the above functions. The function of the anomaly detection unit 18 does not have to be included in the image processing device 1. The present disclosure may be capable of generating a new evaluation image to be compared with the basic image based on at least the evaluation image generated by the image generation unit 12.

FIG. 4 is a diagram illustrating an example of a base image and an evaluation image according to one embodiment of the present disclosure.
The base image and the evaluation image are images generated at different times. The base image is an image generated by combining the image data captured by the line scan camera 2. Similarly, the evaluation image is an image generated by combining the image data captured by the line scan camera 2.

Line scan camera 2 is fixedly installed in a position where it can capture the moving object 5, and captures the moving object 5 by releasing the shutter at high speed as the moving object 5 passes by line scan camera 2. As a result, the multiple pieces of image data output by line scan camera 2 are image data that include a partial section of the moving direction of the moving object 5, and by sequentially combining these pieces of image data so that they match the moving direction of the moving object 5 captured in the image, it is possible to generate a base image and an evaluation image that include the entire moving direction of the moving object 5. The moving speed and change in moving speed of the moving object 5 when capturing the image data used to generate the base image may differ from the moving speed and change in moving speed of the moving object 5 when capturing the image data used to generate the evaluation image. Furthermore, the shutter speeds used when capturing the image data used to generate the base image and the evaluation image may differ. As a result, the lengths of the base image and the evaluation image, which capture the entire moving direction of the moving object 5, differ as shown in Figure 4.

FIG. 5 is a diagram showing a processing flow of the image processing device according to an embodiment of the present disclosure.
Next, the processing flow of the image processing device 1 will be explained step by step.
The acquisition unit 11 outputs multiple pieces of image data acquired from the line scan camera 2 to the image generation unit 12 at the timing of generating a basic image. The image generation unit 12 synthesizes each piece of shooting data to generate a basic image (step S101). Similarly, the acquisition unit 12 outputs multiple pieces of image data acquired from the line scan camera 2 to the image generation unit 12 at the timing of generating an evaluation image. The image generation unit 12 synthesizes each piece of shooting data to generate an evaluation image (step S102). In this way, the basic image and evaluation image as shown in FIG. 4 are generated. Note that since the basic image and evaluation image are generated at different times, they may be stored in a storage unit or the like.

The correspondence relationship identification unit 13 detects the timing for generating a new evaluation image (step S103). For example, the correspondence relationship identification unit 13 may detect the timing for generating a new evaluation image based on a user operation. When the correspondence relationship identification unit 13 detects the timing for generating a new evaluation image, it acquires the basic image and evaluation image generated by the image generation unit 12 from the storage unit or the image generation unit 12.

The correspondence identification unit 13 detects feature points of the moving object 5 that appear in the base image (step S104). Feature points indicate points where edges, such as the corners of a specific part of the moving object 5 or the corners of a screw, stand out. The correspondence identification unit 13 extracts the boundaries between the parts that make up the moving object 5 that appear in the base image and other adjacent parts or objects using edge processing, and detects feature points where the edges stand out. The feature point may be part of a specific part that makes up the moving object 5. In this case, the correspondence identification unit 13 may identify the specific part using pattern matching and identify a part of that part as the feature point. The correspondence identification unit 13 identifies multiple feature points throughout the entire base image. The more feature points, the better. For example, the correspondence identification unit 13 identifies multiple feature points within a section that is set in advance in the base image. The correspondence identification unit 13 similarly identifies feature points in the evaluation image (step S105).

Note that, when identifying feature points, the correspondence identification unit 13 may identify feature points that have no similar objects within a predetermined range nearby. This can reduce erroneous determinations in the subsequent matching process for identical feature points between the base image and the evaluation image. In other words, this eliminates the difficulty of identifying the correspondence between feature points in the base image and feature points in the evaluation image when there are many similar objects nearby. This process is an example of a process in which the correspondence identification unit 13 determines whether there are any similar objects to a specified object within a predetermined range including the specified object, and, if no similar objects are found, detects the specified object as a candidate for identical feature points.

The correspondence identification unit 13 compares the feature points identified in the base image with the feature points identified in the evaluation image and matches identical feature points (step S106). Specifically, if the degree of match between the edge patterns indicated by the feature points is equal to or greater than a predetermined threshold, the feature points are considered to be identical, and the correspondence between the feature points in the base image and the feature points in the evaluation image is identified.

FIG. 6 is a diagram showing the relationship between feature points in a base image and an evaluation image according to an embodiment of the present disclosure.
In both images 6A and 6B shown in FIG. 6 , the top row shows the base image 51, and the bottom row shows the evaluation image 52. When lines are drawn to connect the feature points in the base image 51 with the corresponding feature points in the evaluation image 52, the lines appear as shown in FIG. 6 . Image 6B shows a case in which more feature points are identified than in image 6A. For convenience of explanation, FIG. 6 uses an example in which the lengths of the base image 51 and the evaluation image 52 are the same. However, in this disclosure, the lengths of the base image 51 and the evaluation image 52 are different, as shown in FIG. 4 . However, in other processes, the lengths of the base image 51 and the evaluation image 52 may be matched first. When the lengths of the base image and the evaluation image are equal or nearly equal, as shown in FIG. 6 , and the moving speed of the moving object 5 in the base image and the moving speed of the moving object 5 in the evaluation image are the same, the line connecting the feature point in the base image and the corresponding feature point in the evaluation image will be perpendicular to the upper limit. On the other hand, if the moving speed or change in moving speed of the moving object 5 in the base image and the evaluation image are different, the corresponding identical feature points will be shifted in the direction of movement of the moving object 5, as shown in Figure 6, and the line connecting these feature points will be diagonal.

The correspondence relationship identification unit 13 identifies feature points in both the base image and the evaluation image (the most recent new evaluation image generated in the past) each time a new evaluation image is generated, and matches identical feature points. If feature points in the base image are identified in advance and an attempt is made to find identical feature points in the evaluation image that correspond to those feature points, it will be impossible to find the corresponding feature points in the evaluation image if the identical feature points are not clearly visible in the evaluation image due to shadows or other factors. Therefore, by identifying feature points in both the base image and the evaluation image each time a new evaluation image is generated, it becomes possible to identify identical feature points from the base image and the evaluation image in a manner that is robust to changes in shooting conditions.

In some cases, the correspondence between the feature points in the base image and the evaluation image cannot be adequately determined in the correspondence determination process described above. Therefore, the image processing device 1 adds the following process to the correspondence determination process.
FIG. 7 is a first diagram illustrating an example of the correspondence relationship specifying process.
7 shows four process summaries of the correspondence relationship identification process, namely, a first process summary 91, a second process summary 92, a third process summary 93, and a fourth process summary 94. The first process summary 91 shows how three feature points 51a, 51b, and 51c are identified in the base image 51, and three feature points 52a, 52b, and 52c are identified in the evaluation image 52. Now, in the processing by the correspondence relationship identification unit 13, it should be determined that the feature points 51a and 52a are the same feature point, the feature points 51b and 52b are the same feature point, and the feature points 51c and 52c are the same feature point. However, it is assumed that an identical feature point in the evaluation image corresponding to the feature point 51b is not detected, and the relationship between the feature points 51a and 52a and the relationship between the feature points 51a and 52b are both determined to be the same feature point.

In other words, the first processing overview 91 indicates that feature point 51a of the base image has been identified as having an identical feature point relationship with each of two feature points 52a and 52c of the evaluation image. In this case, feature point 51a and feature point 52a are identical feature point relationships, but feature point 51a and feature point 52b are not a correct identical feature point correspondence relationship, and an error has occurred in identifying them. If similar feature points exist in positions near the moving object 5, errors may repeatedly occur in identifying such identical feature point correspondence relationships.

In this case, as shown in the second processing outline 92, the correspondence identification unit 13 identifies the range in which errors occur consecutively when identifying correspondence between identical feature points in both the base image 51 and the evaluation image 52. The range in which errors occur consecutively identified in the base image 51 is designated 51x, and the range in which errors occur consecutively identified in the evaluation image 52 is designated 52x.

As shown in the third processing outline 93, the correspondence identification unit 13 identifies the range 51x where the identified errors occur consecutively in the base image 51 as the center point 51p, and the range 52x where the identified errors occur consecutively in the evaluation image 52 as the center point 52p. When the correspondence identification unit 13 sets a range where errors occur consecutively as shown in the fourth processing outline 94, it provisionally identifies the center points 51p and 52p of each range within that range as points that correspond to the same feature point, and generates correspondence data. This correspondence data is then used to perform the outlier removal process described above by the correspondence removal unit 14.

The processing performed by the correspondence identification unit 13 is an example of processing in which ranges in each of the base image and the evaluation image where errors occur in identifying the correspondence between identical feature points in the base image and the evaluation image are identified, and predetermined positions in each of the ranges identified in the base image and the evaluation image are identified as correspondences between identical feature points.

The above processing avoids, as much as possible, generating correspondence data that contains errors in the correspondence between the same feature points in the base image and the evaluation image, thereby preventing the occurrence of feature points that are not identified as outliers in the outlier removal process in the correspondence removal unit 14, which is described below.

The range where errors occur repeatedly may be identified by the image processing device 1 outputting a display screen showing the relationship between identical feature points on the base image and the evaluation image by superimposing lines, and receiving a user's specification of the range where errors occur repeatedly on that display. Alternatively, the correspondence relationship identification unit 13 of the image processing device 1 may automatically identify the range where errors in the correspondence relationship between identical feature points occur repeatedly in each of the base image and the evaluation image.

FIG. 8 is a second diagram showing an example of the correspondence relationship specifying process.
Next, an example will be described in which the image processing device 1 automatically identifies a range in which errors in the correspondence between the same feature points occur repeatedly in each of the base image and the evaluation image. In the correspondence identification process, the image processing device 1 may perform the following process.

Figure 8 shows four processing overviews: first automatic processing overview 111, second automatic processing overview 112, third automatic processing overview 113, and fourth automatic processing overview 114. First automatic processing overview 111 shows how three feature points 51a, 51b, and 51c are identified in base image 51, and three feature points 52a, 52b, and 52c are identified in evaluation image 52. Now, in the processing of correspondence identification unit 13, it should be determined that feature points 51a and 52a are the same feature point, that feature points 51b and 52b are the same feature point, and that feature points 51c and 52c are the same feature point. However, it is assumed that the same feature points in the evaluation image corresponding to feature points 51b and 51c are not detected, and the relationship between feature points 51a and 52a, the relationship between feature points 51a and 52b, and the relationship between feature points 51a and 52c are determined to be the same feature point, respectively.

In other words, in the example of the first automatic processing overview 111, feature point 51a of the base image is identified as having the same feature point relationship as each of the three feature points 52a, 52b, and 52c of the evaluation image. In this case, feature point 51a and feature point 52a are the same feature point relationship, but feature point 51a and feature point 52b, and feature point 51a and feature point 52c are not the same feature point corresponding relationships, and an error has occurred in the identification. If similar feature points are present in positions near the moving object 5, errors may repeatedly occur in the identification of such correspondences of the same feature points.

In this case, the correspondence identification unit 13 detects that multiple feature points in the evaluation image have been determined to be the same feature point as one feature point in the base image (first automatic processing overview 111). In this case, the correspondence identification unit 13 determines whether the distance between each of the multiple feature points in the evaluation image that have been determined to be the same feature point as one feature point in the original image is less than a threshold value and therefore close to each other (second automatic processing overview 112). If the distance between each of the multiple feature points in the evaluation image is less than a threshold value and therefore close to each other, the correspondence identification unit 13 identifies a range that includes those feature points 52a, 52b, and 52c (third automatic processing overview 113).

The correspondence identification unit 13 identifies the center point of the range identified in the evaluation image as feature point 52p in the evaluation image that corresponds to feature point 51a in the base image. It then identifies feature point 51a in the base image and feature point 52p identified in the evaluation image as the same feature point, and generates correspondence data that includes the relationship between them.

The correspondence identification unit 13 may also display information about the range in the evaluation image that includes feature points 52a, 52b, and 52c, allowing the user to modify that range. In this case, the correspondence identification unit 13 acquires information about the range modified by the user (fourth automatic processing overview 114). The correspondence identification unit 13 identifies the center point of that range as feature point 52p in the evaluation image that corresponds to feature point 51a in the base image.

By performing the above process, the image processing device 1 can automatically identify the range in which errors in the correspondence between the same feature points occur repeatedly.

FIG. 9 is a diagram showing an overview of the correspondence removal process according to an embodiment of the present disclosure.
The correspondence identification unit 13 outputs correspondence data holding multiple correspondences between identical feature points identified in the base image and the evaluation image to the correspondence removal unit 14. The correspondence data is data including multiple correspondences between the coordinates of identical feature points identified in the base image and the evaluation image in each image. The correspondence removal unit 14 reads the coordinates of the identical feature points in the base image and the evaluation image included in the correspondence data and calculates the difference (xDiff) in coordinates in the movement direction of the moving object 5 in the image (x-axis direction in this disclosure). The correspondence removal unit 14 calculates the difference (xDiff) in the x-coordinate in the movement direction for each correspondence between identical feature points. The correspondence identification unit 13 identifies outliers with a steep change in the difference (xDiff) between the x-coordinate value of the feature point identified in the base image and the x-coordinate of the identical feature point in the corresponding evaluation image (step S107).

The moving object 5 being photographed here is moving in the real world, but due to the constraints of inertial force, it is assumed that the speed and acceleration of the moving object 5 can be approximated as being almost constant over an extremely short period of time. In other words, it is assumed that the speed of the moving object 5 changes linearly over a certain short period of time. The correspondence removal unit 14 uses this to remove outliers using linear approximation (step S108).

In graph 7A of Figure 9, the correspondence removal unit 14 displays a two-dimensional graph showing the relationship between the x-coordinate value of a feature point identified in the base image and the difference (y-axis) between the x-coordinate of the same feature point in the corresponding evaluation image. For example, in graph 7A of Figure 9, p1 and p2 are outliers. The correspondence identification unit 13 identifies such outliers using a known technique such as the RANSAC (RANdom SAmple Consensus) algorithm. This generates correspondence data (7B) from which the outliers have been removed. This process is an example of the correspondence removal unit 14 identifying identical feature points that correspond to an outlier due to a change in the difference in position between the identical feature points in the base image and the evaluation image, and removing the correspondence between those identical feature points.

The results of the correspondence identification unit 13's identification of correspondence between identical feature points may include information that has been mistakenly identified as identical feature points. If such erroneous information is left, it will cause problems in the calculations of the subsequent processing by the section identification unit 15 and image conversion unit 16, and therefore processing by the correspondence removal unit 14 is required.

FIG. 10 is a first diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure.
FIG. 11 is a second diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure.
FIG. 12 is a third diagram illustrating an overview of the section identification process according to an embodiment of the present disclosure.
The correspondence removal unit 14 outputs the correspondence data from which the outliers have been removed to the section identification unit 15. Here, in the basic image, one section is predetermined to have a width of 20 px (pixels) in the direction of movement of the moving object 5. The width (20 px) of one section set in the basic image is determined so that image processing for one section does not take much time, depending on the shutter speed of the line scan camera 2 and the processing capacity of the image processing device 1. The section identification unit 15 identifies the correspondence of one section of the evaluation image corresponding to one section of the basic image (step S109). Now, as shown in FIG. 10 , the difference in the x-coordinate direction between each of the sections in the evaluation image corresponding to each of the three sections of 20 px each in the basic image is unknown.

Therefore, the section identification unit 15 identifies the correspondence between each section of the evaluation image corresponding to each of the multiple sections set in the basic image using the correspondence data acquired from the correspondence removal unit 14. The correspondence data 71 shown in FIG. 11 indicates the difference between the x-coordinate of a feature point identified in the basic image and the x-coordinate of a feature point identified in the evaluation image corresponding to that feature point. For example, the correspondence data 71 records that the x-coordinate value of feature point α identified in the basic image in the basic image is 10.2 (px), and the difference in x-coordinate between feature point α and the corresponding feature point in the evaluation image is "-1.2." The correspondence data 71 also records that the x-coordinate value of feature point β identified in the basic image in the basic image is 15.6 (px), and the difference in x-coordinate between feature point β and the corresponding feature point in the evaluation image is "-0.5." Furthermore, in the correspondence data 71, the x-coordinate value of the feature point γ identified in the basic image is recorded as 19.5 (px), and the difference in x-coordinate between that feature point γ and the corresponding feature point in the evaluation image is recorded as " 2.2 ." Furthermore, in the correspondence data 71, the x-coordinate value of the feature point δ identified in the basic image is recorded as 20.2 (px), and the difference in x-coordinate between that feature point δ and the corresponding feature point in the evaluation image is recorded as "2.9." Note that the x-coordinate value is considered to correspond to the number of px (pixels). If the origin of the x-coordinate in the basic image and the evaluation image is the left end, when the difference between the feature point in the basic image and the feature point in the basic image that corresponds to that feature point is negative, the x-coordinate position of the feature point in the evaluation image is closer to the origin than the x-coordinate position of the feature point in the corresponding basic image.

Now, the x-coordinate in the evaluation image corresponding to the end position (x-coordinate 20.0 (px)) of the first section of the basic image determined by a width of 20 px is calculated using the correspondence data 71. In this case, since the feature point at the x-coordinate indicating x-coordinate = 20.0 (px) of the basic image is not identified in the basic image, attention is focused on feature point γ (x-coordinate = 19.5) and feature point δ (x-coordinate = 20.2), which are neighboring points before and after x-coordinate 20.0 (px). In this case, the section identification unit 15 can calculate the virtual feature point in the evaluation image corresponding to the virtual feature point equivalent to x-coordinate 20.0 in the basic image using the following linear interpolation formula:

<Linear interpolation formula>
Difference in x coordinate = 2.2 + ( ( 2.9 - 2.2 ) ÷ ( 20.2 - 19.5 ) ) x (20.0 - 19.5)
= 2.7

Similarly, the section identification unit 15 uses linear interpolation to calculate the x-coordinate in the evaluation image corresponding to the end of the second section of the basic image (x-coordinate 40.0 (px)), the x-coordinate in the evaluation image corresponding to the end of the third section (x-coordinate 60.0 (px)), and so on, sequentially calculating the difference in position between the x-coordinate of the basic image at the end of each section and the x-coordinate of the corresponding evaluation image.

The section identification unit 15 generates cut-out section identification data 81 that indicates, for each section, the difference between the x-coordinate of the end of one section set in the base image and the x-coordinate in the evaluation image that corresponds to that x-coordinate (step S110). The section identification unit 15 outputs the cut-out section identification data 81 to the image conversion unit 16.

The image conversion unit 16 identifies the start coordinates to the end coordinates of one section in the evaluation image based on the cut-out section identification data 81 (step S111). Then, as shown in FIG. 12 ( 10A ), the image conversion unit 16 generates cut-out section data by reducing or enlarging the image width from the start coordinates to the end coordinates of one section in the evaluation image to the width (20 px) of one section set in the basic image (step S112). As a result, the image width from the start coordinates to the end coordinates of one section in the evaluation image is reduced or enlarged in the direction of movement of the moving object depicted in the image. This causes the length of one section in the basic image to match the length of one section in the evaluation image.

As shown in Fig. 6 , the length of the original image of the evaluation image acquired from the line scan camera 2 may be converted in advance so that the lengths of the basic image and the evaluation image match. In this case, as shown in Fig. 12 ( 10B ), the image conversion unit 16 first identifies the width of the corresponding original image based on the length from the start coordinate to the end coordinate of one section in the evaluation image based on the cut-out section identification data 81 and the conversion rate when the original image of the evaluation image is adjusted to the length of the basic image. The image conversion unit 16 generates cut-out section data by converting the width of the original image of the evaluation image identified in Fig. 12 ( 10B ) to the width (20 px) of one section set in the basic image (Fig. 12 ( 10C )).

The image conversion unit 16 outputs each of the image data (cut-out section data) for one section generated from the evaluation image to the new evaluation image generation unit 17. The new evaluation image generation unit 17 generates a new evaluation image by arranging and combining the image data (cut-out section data) for each section generated from the evaluation image in order (step S113). The length of the moving object 5 in the new evaluation image in the direction of movement matches the length of the moving object 5 in the direction of movement in the base image.

The above-described processing allows for the generation of a new evaluation image that matches the length of the moving object 5 in the direction of movement in the base image, even when the subject captured by the line scan camera 2 is a moving object 5.

Furthermore, with the above-described processing, even if the subject, a moving object 5, moves at high speed and the speed of the moving object 5 changes as it passes by the line scan camera 2, a new evaluation image can be generated in which the same feature points appear at positions in the direction of movement that roughly correspond to the feature points of the basic image. This makes it easier to compare and evaluate two images generated based on images taken at different times by the line scan camera 2, since the same feature points will be in roughly the same positions.

Furthermore, with the above-described processing, even if the moving object 5 is long in the direction of movement, such as a train, and the evaluation image becomes longer in the direction of movement depending on the shutter speed, it is possible to generate a new evaluation image that matches the length of the basic image. Therefore, when performing evaluations such as discovering abnormalities in the entire new evaluation image, the number of image processes for each section divided into a specified size of the new evaluation image can be reduced, thereby reducing the effort required for such image processing.

Furthermore, the above-described processing identifies many relationships between identical feature points in the base image and the evaluation image. As a result, even if the change in speed of the moving object 5 when photographed to generate the base image does not match the change in speed of the moving object 5 when photographed to generate the evaluation image, it is possible to generate a new evaluation image in which the same feature points appear at approximately the same positions in the direction of movement.

FIG. 13 is a diagram showing another configuration of an image processing device according to an embodiment of the present disclosure.
FIG. 14 is a diagram showing another processing flow of the image processing device according to an embodiment of the present disclosure.
The image processing device 1 shown in Figure 13 performs the functions of a correspondence relationship identification means, a section identification means, an image conversion means, and a new evaluation image generation means.The correspondence relationship identification means identifies the correspondence relationship between the same feature points in a basic image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image (step S1301).
The section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image (step S1302).
The image conversion means converts each of the cut-out sections in the evaluation image so that each cut-out section in the evaluation image matches the length of one section of each of the basic images (step S1303).
The new evaluation image generating means generates a new evaluation image by arranging the images of each section after the conversion of the cut-out sections in order in the direction of movement of the moving object shown in the images (step S1304).

FIG. 15 is a diagram showing another configuration of an image processing device according to an embodiment of the present disclosure.
FIG. 16 is a diagram showing another processing flow of the image processing device according to an embodiment of the present disclosure.
The image processing device 1 shown in FIG. 15 performs the functions of a correspondence relationship specifying means, a section specifying means, and a new evaluation image generating means.
The correspondence relationship identification means identifies the correspondence relationship between the same feature points in a basic image captured of a moving object moving in a predetermined direction of movement and an evaluation image captured of the moving object at a timing different from the timing of capturing the basic image (step S1501).
The section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on the change in the difference in position of the same feature point that has a corresponding relationship between the basic image and the evaluation image (step S1502).
The new evaluation image generating means generates a new evaluation image by sequentially arranging the images of the cutout section in the movement direction (step S1503).

The image processing device 1 of this disclosure has been described above, but this disclosure is not limited to the above-described embodiment. Various modifications that would be understandable to a person skilled in the art can be made to the configuration and details of this disclosure within the scope of this disclosure.

Note that some or all of the above embodiments can also be described as, but are not limited to, the following notes.

(Appendix 1)
a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the timing of photographing the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having the length of one section in each of the basic images;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction;
An image processing device comprising:

(Appendix 2)
a correspondence removal means for identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence between the identical feature points;
The image processing device described in Appendix 1, wherein the section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position in the movement direction of the same feature point that appears in the basic image and the evaluation image after the removal.

(Appendix 3)
The image processing device described in Appendix 2, wherein the correspondence identification means identifies, in each of the basic image and the evaluation image, a range where errors occur in identifying the correspondence between the same feature points in the basic image and the evaluation image, and identifies predetermined positions in each of the identified ranges in the basic image and the evaluation image as the correspondence between the same feature points.

(Appendix 4)
an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera;
2. The image processing device according to claim 1, comprising:

(Appendix 5)
The image processing device according to claim 1, wherein the correspondence specifying means specifies a plurality of correspondences between the same feature points in the base image and the evaluation image in a movement direction.

(Appendix 6)
The image processing device according to claim 5, wherein the correspondence specifying means detects predetermined objects in each of the base image and the evaluation image as candidates for the identical feature point.

(Appendix 7)
The image processing device according to claim 6, wherein the correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the identical feature point.

(Appendix 8)
an abnormality detection means for detecting an abnormality in the moving object by comparing the basic image with the new evaluation image;
2. The image processing device according to claim 1, comprising:

(Appendix 9)
Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image;
identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having a length of one section in each of the basic images;
and generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction.

(Appendix 10)
identifying an identical feature point whose change in the difference in position of the identical feature point in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence relationship for the identical feature point;
and identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position in the movement direction of the same feature point that appears in the basic image and the evaluation image after the removal.

(Appendix 11)
The image processing method according to claim 9 or 10, further comprising: identifying, in each of the basic image and the evaluation image, a range in which errors occur in identifying the correspondence between the same feature points in the basic image and the evaluation image; and identifying predetermined positions in each of the identified ranges in the basic image and the evaluation image as the correspondence between the same feature points.

(Appendix 12)
12. The image processing method according to any one of Supplementary Note 9 to Supplementary Note 11, further comprising: generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera.

(Appendix 13)
The image processing method according to any one of Supplementary Note 9 to Supplementary Note 12, wherein a plurality of correspondences between the same feature points in the base image and the evaluation image are identified in the movement direction.

(Appendix 14)
The image processing method according to any one of Supplementary Note 9 to Supplementary Note 13, further comprising detecting predetermined objects in the base image and the evaluation image as candidates for the identical feature point.

(Appendix 15)
The image processing method according to any one of Supplementary Note 9 to Supplementary Note 14, wherein it is determined whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, the predetermined object is detected as a candidate for the identical feature point.

(Appendix 16)
16. The image processing method according to any one of claims 9 to 15, further comprising: comparing the base image with the new evaluation image to detect an abnormality in the moving object.

(Appendix 17)
The computer of the image processing device,
a correspondence relationship specifying means for specifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in one predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the movement direction between the basic image and the evaluation image;
an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having the length of one section in each of the basic images;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging images of each of the sections after the reduction or enlargement of the cut-out sections in the movement direction;
A program that functions as a

(Appendix 18)
A computer of the image processing device,
identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and causing the device to function as correspondence relationship removal means for removing the correspondence relationship between the identical feature points;
The program described in Appendix 17, wherein the section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position in the movement direction of the same feature point that appears in the basic image and the evaluation image after the removal.

(Appendix 19)
The program according to claim 17 or 18, wherein the correspondence identification means identifies, in each of the basic image and the evaluation image, a range in which errors occur in identifying the correspondence between the same feature points in the basic image and the evaluation image, and identifies predetermined positions in each of the identified ranges in the basic image and the evaluation image as the correspondence between the same feature points.

(Appendix 20)
A computer of the image processing device,
an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera;
20. The program according to any one of appendices 17 to 19,

(Appendix 21)
21. The program according to claim 17, wherein the correspondence specifying means specifies a plurality of correspondences between the same feature points in the base image and the evaluation image in a movement direction.

(Appendix 22)
The program according to any one of Supplementary Note 17 to Supplementary Note 21, wherein the correspondence specifying means detects predetermined objects in the base image and the evaluation image as candidates for the identical feature point.

(Appendix 23)
The program according to any one of Supplementary Note 17 to Supplementary Note 22, wherein the correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the identical feature point.

(Appendix 24)
A computer of the image processing device,
24. The program according to claim 17, which functions as an abnormality detection means for detecting an abnormality in the moving body by comparing the basic image with the new evaluation image.

(Appendix 25)
a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the timing of photographing the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of the cutout section in the movement direction;
An image processing device comprising:

(Appendix 26)
an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that each of the cut-out sections in the evaluation image matches an image having a length of one section in each of the basic images;
The image processing device according to claim 25, wherein the new evaluation image generating means generates a new evaluation image by sequentially arranging images of each of the cut-out sections after reduction or enlargement in the movement direction.

(Appendix 27)
Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined one moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image;
identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
generating a new evaluation image by sequentially arranging the images of the cut-out section in the movement direction.

(Appendix 28)
The computer of the image processing device,
a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from that of photographing the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the movement direction between the basic image and the evaluation image;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of the cutout section in the movement direction;
A program that functions as a

1... Image processing device 2... Line scan camera 5... Moving object 11... Acquisition unit 12... Image generation unit 13... Correspondence relationship identification unit 14... Correspondence relationship removal unit 15... Section identification unit 16... Image conversion unit 17... New evaluation image generation unit 18... Abnormality detection unit

Claims (9)

a correspondence specifying means for specifying a correspondence between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the timing of photographing the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that the image length of each of the cut-out sections in the evaluation image matches the image length of the corresponding section in the basic image;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction;
An image processing device comprising: a correspondence removal means for identifying identical feature points whose change in the difference in position of the identical feature points in the movement direction between the basic image and the evaluation image is an outlier, and removing the correspondence between the identical feature points;
2. The image processing device according to claim 1, wherein the section identification means identifies a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in a difference in position in the movement direction of the same feature point appearing in the basic image and the evaluation image after the removal. 3. The image processing device according to claim 2, wherein the correspondence specifying means specifies, in each of the basic image and the evaluation image, a range in which errors occur in specifying the correspondence between the same feature points in the basic image and the evaluation image, and specifies a predetermined position in the range specified in the basic image and a predetermined position in the range specified in the evaluation image as the correspondence between the same feature points. an image generating means for generating the basic image and the evaluation image by combining images of the moving object captured by a line scan camera;
The image processing device according to claim 1 , comprising: The image processing device according to claim 1 , wherein the correspondence specifying means specifies a plurality of correspondences between the same feature points in the basic image and the evaluation image in the movement direction. The image processing device according to claim 5 , wherein the correspondence specifying means detects predetermined objects in each of the base image and the evaluation image as candidates for the same feature point. The image processing device according to claim 6, wherein the correspondence specifying means determines whether or not there is an object similar to the predetermined object within a predetermined range including the predetermined object, and if there is no similar object, detects the predetermined object as a candidate for the identical feature point. Identifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in a predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image;
identifying a cut-out section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction based on a change in the difference in position of the same feature point in the basic image and the evaluation image in the movement direction;
reducing or enlarging each of the cutout sections in the evaluation image so that the image length of each of the cutout sections in the evaluation image matches the image length of the corresponding section in the basic image;
and generating a new evaluation image by sequentially arranging the images of each of the cut-out sections after reduction or enlargement in the movement direction. The computer of the image processing device,
a correspondence relationship specifying means for specifying a correspondence relationship between the same feature points in a basic image obtained by photographing a moving object moving in one predetermined moving direction and an evaluation image obtained by photographing the moving object at a timing different from the photographing timing of the basic image;
a section specifying means for specifying a section in the evaluation image corresponding to one section obtained by dividing the basic image at a predetermined interval in the movement direction, based on a change in the difference in position of the same feature point in the movement direction between the basic image and the evaluation image;
an image conversion means for reducing or enlarging each of the cut-out sections in the evaluation image so that the image length of each of the cut-out sections in the evaluation image matches the image length of the corresponding section in the basic image;
a new evaluation image generating means for generating a new evaluation image by sequentially arranging images of each of the sections after the reduction or enlargement of the cut-out sections in the movement direction;
A program that functions as a