CN110349135A - Object detection method and device - Google Patents

Abstract

The invention discloses a target detection method and device. The method comprises: performing feature extraction on a target image containing a target to be detected to obtain feature vector data reflecting the features of the target to be detected; setting the number of paths and the number of network levels; An aggregation transformation network is constructed in the framework, wherein, the number of paths in the aggregation transformation network is more than two, and data transmission is performed between corresponding paths between different levels; the feature vector data is input into the aggregation transformation network, and the feature vector data It is divided into different levels, and the data of each level is divided into feature groups corresponding to each path; in the aggregation transformation network, the feature groups of each level on each path are processed in parallel to obtain multiple processing results; Based on the processing result, the feature image of the target image is obtained.

Description

Target detection method and device

technical field

The invention relates to the technical field of image target detection and segmentation, in particular to a target detection method and a target detection device.

Background technique

In the field of manufacturing and processing, etc., it is usually necessary to perform defect detection on the produced products. At present, when performing defect detection on products, artificial intelligence technology is usually used to replace the traditional artificial appearance detection of product defects. The basic idea is: by collecting/collecting a certain amount of relevant original image data, and then feeding the instance segmentation framework Mask R-CNN, based on the convolutional feature extraction network backbone in the instance segmentation framework, obtains the feature image of the product to be tested, and then processes the feature image to determine whether the product to be tested is qualified.

However, the existing convolutional feature extraction network backbone uses a deep residual network structure to process the original image containing the product to be detected to obtain the feature image of the target object. Since the feature information of the original image is complicated and large in number, therefore, When using the existing deep residual network structure to process the original image, the processing speed is slow, and the processing efficiency is low.

Contents of the invention

The purpose of the present invention is to provide a new technical solution for target detection.

According to a first aspect of the present invention, a method for target detection is provided, comprising:

Feature extraction is performed on the target image containing the target to be detected to obtain feature vector data reflecting the characteristics of the target to be detected;

Set the number of paths and the number of network levels, and construct an aggregation transformation network in the instance segmentation framework, wherein the number of paths in the aggregation transformation network is more than two, and data transmission is performed between corresponding paths between different levels;

Input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into feature groups corresponding to each path;

In the aggregation transformation network, the feature groups of each level on each path are processed in parallel to obtain multiple processing results;

Combine multiple processing results to obtain the feature image of the target image.

Optionally, set the number of paths and the number of network layers, and build an aggregation transformation network in the instance segmentation framework, including:

According to the number of paths, set more than two paths;

According to the number of network layers, each path includes an input layer, one or more transition layers and an output layer.

Optionally, in the aggregation transformation network, the feature groups of each level on each path are processed in parallel to obtain multiple processing results, including:

Perform dimensionality reduction processing on the input feature vector data through the input layer on each path to obtain the feature group after dimensionality reduction;

Input the dimension-reduced feature group into the transition layer on the corresponding path, perform feature extraction on the dimension-reduced feature group through the transition layer, and obtain the intermediate feature group;

The intermediate feature group is input to the output layer on the corresponding path, and the intermediate feature group is processed through the output layer to increase the dimension, and the feature group after the dimension increase is obtained as the processing result of the corresponding path.

Optionally, the instance segmentation framework includes a regional attention network RPN, and after obtaining the feature image of the target image, the method further includes:

Input the feature image into the regional attention network, through the regional attention network, slide the feature image according to the preset fixed window to identify the region with defects in the feature image, and obtain the identified defective region as the initial region of interest, and A feature image is generated identifying each initial region of interest.

Optionally, sliding on the feature image according to a preset fixed window to identify defective regions in the feature image, including:

Provide at least one N×1 filter and one 1×N filter for a fixed window with a window size of N×N; where N is the size of the fixed window;

The feature image is subjected to dimensionality reduction processing through the N×1 filter and the 1×N filter in turn;

The region with defects is marked in the feature image after dimensionality reduction.

Optionally, the instance segmentation framework also includes a fully connected layer FC for positioning, and the method also includes:

Obtain the matrix of image data processed in the fully connected layer;

Construct a processing matrix according to the matrix of image data;

Extract the weight matrix and offset matrix corresponding to the processing matrix, decompose the matrix of the image data, and use the decomposed matrix to perform operations in the fully connected layer.

According to the second aspect of the present invention, there is also provided a target detection device, characterized in that it includes:

The extraction module is used to perform feature extraction on the target image containing the target to be detected, and obtain feature vector data reflecting the characteristics of the target to be detected;

The network building module is used to set the number of paths and the number of network levels, and construct an aggregation transformation network in the instance segmentation framework, wherein the number of paths in the aggregation transformation network is more than two, and the corresponding paths between different levels carry out data transmission;

The input module is used to input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into each feature group corresponding to each path;

The parallel processing module is used to process the feature groups of each level on each path in parallel in the aggregation transformation network to obtain multiple processing results;

The fusion module is used to fuse multiple processing results to obtain the feature image of the target image.

Optionally, the instance segmentation framework includes a regional attention network RPN, and the device also includes a region extraction module, which is used for:

Input the feature image into the regional attention network, through the regional attention network, slide the feature image according to the preset fixed window to identify the region with defects in the feature image, and obtain the identified defective region as the initial region of interest, and A feature image is generated identifying each initial region of interest.

Optionally, a region extraction module, specifically for:

Provide at least one N×1 filter and one 1×N filter for a fixed window with a window size of N×N; where N is the size of the fixed window;

performing dimensionality reduction processing on the feature image through filters and filters in sequence; and,

The region with defects is marked in the feature image after dimensionality reduction.

Optionally, the instance segmentation framework includes a fully connected layer FC for positioning, and the device also includes a matrix processing module, which is used for:

Obtain the matrix of image data processed in the fully connected layer;

Construct a processing matrix according to the matrix of image data;

Extract the weight matrix and offset matrix corresponding to the processing matrix, decompose the matrix of the image data, and use the decomposed matrix to perform operations in the fully connected layer.

A beneficial effect of the present invention is that, according to the method and device of the embodiment of the present invention, it improves the convolutional feature extraction network in the traditional instance segmentation framework, and the convolutional feature extraction network backbone in the traditional instance segmentation framework Constructing an aggregation transformation network in , the aggregation transformation network has a multi-layer and multi-group structure, and then can process the input feature vector data reflecting the characteristics of the target to be detected in parallel, that is, it can use the idea of layered parallel processing to optimize the existing The convolutional feature extraction network further improves the processing efficiency of the convolutional feature extraction network without increasing the complexity.

Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following will briefly introduce the drawings used in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the invention and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained based on these drawings without creative effort.

1 is a schematic diagram of a hardware structure of a target detection system according to an embodiment of the present invention;

2 is a schematic flow diagram of a target detection method according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of an instance segmentation framework according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a traditional residual network structure according to an embodiment of the present invention;

Fig. 5 is a schematic diagram of a polymerization operation according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a converged transformation network according to an embodiment of the present invention;

Fig. 7 is a schematic diagram of a convolution structure according to an embodiment of the present invention;

Fig. 8 is a schematic flow chart of a target detection method according to an example of the present invention;

FIG. 9 is a block diagram of an object detection device according to an embodiment of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and in no way taken as limiting the invention, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, techniques, methods and devices should be considered part of the description.

In all examples shown and discussed herein, any specific values should be construed as exemplary only, and not as limitations. Therefore, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters denote like items in the following figures, therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.

<Example 1>

FIG. 1 is a block diagram of a hardware configuration of an object detection system 100 according to an embodiment of the present invention.

As shown in FIG. 1 , the target detection system 100 includes an image acquisition device 1000 and a target detection device 2000 .

The image capture device 1000 is used to capture target images, and provide the captured target images to the target detection device 2000 .

The image acquisition device 1000 may be any imaging device capable of taking pictures of the target to be detected, such as a camera.

The object detection apparatus 2000 may be any electronic equipment, such as a PC, a notebook computer, a server, and the like.

In this embodiment, referring to FIG. 1 , the target detection device 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, a speaker 2700, a microphone 2800, and the like.

The processor 2100 may be a mobile processor. The memory 2200 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile memory such as a hard disk, and the like. The interface device 2300 includes, for example, a USB interface, an earphone jack, and the like. The communication device 2400 is capable of wired or wireless communication, for example, and the communication device 2400 may include a short-distance communication device, such as based on Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, Any device that performs short-range wireless communication using a short-range wireless communication protocol such as LiFi, and the communication device 2400 may also include a remote communication device, such as any device that performs WLAN, GPRS, or 2G/3G/4G/5G long-distance communication. The display device 2500 is, for example, a liquid crystal display screen, a touch screen display, etc., and the display device 2500 is used to display the target image collected by the image collection device. The input device 2600 may include, for example, a touch screen, a keyboard, and the like. A user can input/output voice information through the speaker 2700 and the microphone 2800 .

In this embodiment, the memory 2200 of the object detection apparatus 2000 is used to store instructions, and the instructions are used to control the processor 2100 to operate to at least execute the object detection method according to any embodiment of the present invention. A skilled person can design instructions according to the solutions disclosed in the present invention. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.

Although multiple devices of the target detection device 2000 are shown in FIG. 1 , the present invention may only involve some of them. For example, the target detection device 2000 only involves the memory 2200 , the processor 2100 and the display device 2500 .

In this embodiment, the image acquisition device is used to collect target images and provide them to the target detection device 2000, and the target detection device 2000 implements the method according to any embodiment of the present invention based on the target images.

It should be understood that although FIG. 1 only shows one image acquisition device 1000 and one target detection device 2000, it does not mean to limit the respective quantities, and the target detection system 100 may include multiple image capture devices 1000 and/or target detection devices 1000 and 2000. Device 2000.

<Example 2>

The embodiment of the present invention also provides a target detection method. According to FIG. 2, the target detection method in this embodiment specifically includes the following steps S2100-S2500:

Step S2100, performing feature extraction on the target image including the target to be detected, to obtain feature vector data reflecting the features of the target to be detected.

The target image is an image obtained by taking pictures of the target to be detected, and may be a product that needs to be determined to be qualified or unqualified as the target to be detected. For example, it is necessary to determine whether a certain component is qualified. Here, the component may be used as the target to be detected, that is, the image of the component may be collected as the target image.

In an example, the target image may be a digital image, and the digital image may be composed of countless pixels. Here, the digital image can be displayed based on different color modes, such as but not limited to RGB (Red, Green, Blue) color mode, CMYK (Cyan, Magenta, Yellow, Black) color mode, HSB (HueSaturate Bright) color mode and bitmap mode, etc.

In another example, the target image can also be a simulated image.

In this embodiment, feature extraction may be performed on the target image containing the target to be detected to obtain feature vector data reflecting the features of the target to be detected, wherein the feature vector data may be 256-dimensional feature vector data, and of course, It can be feature vector data of other dimensions, which is not limited here.

After executing step S2100 to perform feature extraction on the target image containing the target to be detected, and obtain feature vector data reflecting the features of the target to be detected, proceed to step S2200.

In step S2200, the number of paths and the number of network levels are set, and an aggregation transformation network is constructed in the instance segmentation framework.

The number of paths in the aggregation transformation network is more than two, and the corresponding paths between different levels perform data transmission. The number of paths and the number of network levels can be set according to specific application scenarios and simulation experiments.

At present, most of the convolutional feature extraction networks in the instance segmentation framework use a deep residual network structure to extract semantic features from the feature vector data obtained in step S2100, so as to abstract the input target image into Feature Maps, which are useful in The region is activated, irrelevant or ambiguous feature information is weakened.

However, as shown in Figure 3 and Figure 4, most of the existing deep residual network structures adopt the "Resnet50+FPN" structure, where 256-d in represents the feature vector data of the input deep residual network structure, and 256-d out Represents the eigenvector data output after being processed by the deep residual network structure. The first layer uses a 256, 1×1 structure to reduce the dimension of the input eigenvector data, that is, the 256 dimension is reduced to 64 dimensions, and the second layer passes 64 , 3×3 structure for local feature extraction, and the third layer uses 64, 1×1 structure to expand the output dimension, that is, expand the 64 dimension to 256 dimension, and all the “×” operations that appear above come from volume The inner product operation of the product. Combining with Figure 3, it can be seen that the existing deep residual network structure adopts a single-input and single-output network structure, that is, 256-d in are all processed through the same path, resulting in slow processing speed, while this implementation In the example, it is the convolutional feature extraction network in the instance segmentation framework to construct the aggregation transformation network, so that the input feature vector data is processed in parallel through multiple paths to improve the processing speed.

The aggregation transformation network provided in this embodiment changes the inner product operation shown in Figure 4, and transforms the inner product operation into a combination of splitting-transforming-aggregating in the form of aggregation transformation, as shown in Formula 1:

Among them, the value of x is every natural number from 1 to D, D represents the number of input dimensions, x=[x ₁ , x ₂ ,...,x _D ] represents the input vector, w _i represents the i-th The filter weights of input vectors, where the "Splitting" operation in the transformation divides the input vector x into low-dimensional embedding vector features, that is, x _i in the single-dimensional space; the "Transforming" operation in the transformation transforms the low-dimensional space into Low-dimensional representation, that is, w _i x _i ; the "Aggregating" operation in the transformation performs transformation aggregation by adding all the embedded vector features, and this formula 1 can be represented by the topology diagram shown in Figure 5.

According to the above analysis, the element-by-element transformation w _i x _i can be replaced by the function T(x), as shown in formula 2:

Among them, T(x) can be any function, and x is projected into an embedding space (generally a low-dimensional space) and transformed, where C is the size of the transformation set to be aggregated, similar to D in formula 1, C Dimensionality determines the number of complex transformations. For the design of the transformation function, the adopted method requires that all T(i) topological structures are the same.

Refer to the idea of deep residual network design, such as formula 3, and replace the F inner product expression in formula 3 with the F aggregation expression in formula 2 to obtain the overall transformation of the aggregation transformation of formula 4, which is the aggregation transformation The steps to replace the inner product operation "×" provide theoretical support.

y=F(x, {w _i })+x (Formula 3)

Exemplarily, the number of paths is M, M is a natural number greater than 1, and the number of network levels is 3. Here, according to the number of paths M and the number of network levels 3, it can be constructed in the instance segmentation framework as shown in Figure 6. The aggregation transformation network shown.

After performing step S2200 to set the number of paths and the number of network levels, and constructing an aggregation transformation network in the instance segmentation framework, enter step S2300.

Step S2300, input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into feature groups corresponding to each path.

As shown in Figure 6, taking the feature vector data input into the aggregation transformation network as 256-dimensional feature vector data, and the number of paths is M, and the number of network levels is 3 as an example, here, the 256-dimensional feature vector data can be Divided into M feature groups, each feature group includes eigenvector data.

After performing step S2300 to input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into feature groups corresponding to each path, enter step S2400.

In step S2400, in the aggregation transformation network, the feature groups of each level on each path are processed in parallel to obtain multiple processing results.

Exemplarily, for example, in the aggregation transformation network shown in FIG. 6 , the feature groups of each level on the M paths may be processed in parallel, so as to obtain processing results corresponding to the M paths.

After step S2400 is executed in the aggregation transformation network, feature groups of each level on each path are processed in parallel, and multiple processing results are obtained, and then step S2500 is entered.

Step S2500, merging multiple processing results to obtain a feature image of the target image.

In this embodiment, for example, multiple processing results may be superimposed to obtain the feature image of the target image. As shown in Figure 6, it can be superimposed on the output layer output on each path Dimension feature vector data to obtain the feature image of the target image.

According to the method of the embodiment of the present invention, it improves the convolutional feature extraction network in the traditional instance segmentation framework, constructs an aggregation transformation network in the convolutional feature extraction network backbone in the traditional instance segmentation framework, and the aggregation transformation network It has a multi-layer and multi-group structure, and can then process the input feature vector data reflecting the characteristics of the target to be detected in parallel, that is, it can use the idea of layered parallel processing to optimize the existing convolution feature extraction network, so that the convolution The processing efficiency of the feature extraction network is further improved without increasing the complexity.

<Example Three>

The embodiment of the present invention also provides a target detection method. In this embodiment, the number of paths and the number of network levels are set in the above step S2200, and the construction of an aggregation transformation network in the instance segmentation framework may further include the following steps S2210-S2220:

Step S2210, according to the number of paths, set more than two paths.

As shown in FIG. 6 , the number of paths is M, that is, corresponding M paths can be set according to the number M of paths.

Step S2220, according to the number of network layers, set each path to include an input layer, one or more transition layers and an output layer.

As shown in Figure 6, the number of layers in the network is 3. Here, several paths can be set, for example, M paths include an input layer, a transition layer and an output layer. Of course, each path can also be set to include, for example are two or other transition layers, which are not limited here.

In this embodiment, more than two paths are set based on the number of paths, and, according to the number of network levels, each path is set to include an input layer, one or more transition layers and an output layer, which is conducive to the construction efficiency of the aggregation transformation network, And it is further beneficial to process the corresponding number of feature vectors in parallel through multiple paths, so as to improve the processing speed.

<Example 4>

The embodiment of the present invention also provides a target detection method. In this embodiment, in the above step S2400, in the aggregation transformation network, the feature groups of each level on each path are processed in parallel, and multiple processing results can be further obtained. Including the following steps S2410-S2430:

Step S2410, performing dimensionality reduction processing on the input feature vector data through the input layer on each path to obtain feature groups after dimensionality reduction.

As shown in Figure 6, there are a total of M paths, and the total input feature vector data is 256 dimensions. Here, the input layer input on each path The eigenvector data of the dimension is passed through the 1x1 filter set in the input layer on each path to the input Dimensionality reduction processing is performed on the eigenvector data of the dimension, and the obtained Dimensional eigenvector data.

In step S2420, input the dimension-reduced feature group into the transition layer on the corresponding path, perform feature extraction on the dimension-reduced feature group through the transition layer, and obtain an intermediate feature group.

As shown in Figure 6, the output of the input layer on each path The eigenvector data input of the dimension corresponds to the transition layer on the path, and the input 3x3 filter is set in the transition layer. Dimension feature vector data for feature extraction, obtained Dimensional eigenvector data.

In step S2430, the intermediate feature group is input to the output layer on the corresponding path, and the intermediate feature group is processed through the output layer to increase the dimension, and the dimension-enhanced feature group is obtained as the processing result of the corresponding path.

As shown in Figure 6, the output of the transition layer on each path The eigenvector data of one dimension is input to the output layer on the corresponding path, and the input 4-dimensional eigenvector data is processed through the 1x1 filter set in the output layer to obtain Dimensional eigenvector data.

According to this embodiment, it may process the corresponding number of feature vectors in parallel through multiple paths, further improving the processing speed.

<Embodiment 5>

The embodiment of the present invention also provides a target detection method, as shown in Figure 3, the instance segmentation framework also includes a region attention network (Region Proposal Network, RPN), after obtaining the feature image of the target image, the target detection method of the present invention Methods also include:

Input the feature image into the regional attention network, through the regional attention network, slide the feature image according to the preset fixed window to identify the region with defects in the feature image, and obtain the identified defective region as the initial region of interest, and A feature image is generated identifying each initial region of interest.

In this embodiment, the regional attention network will slide a fixed window on the feature image generated in step S2500 with a receptive field of a specific size. The size of the fixed window can be 3, and the potential existence of The feature region of the target is identified, and a certain number of initial regions of interest are generated. These initial regions of interest are regions that the regional attention network believes may have defects in the feature image.

In this embodiment, the step of sliding on the feature image according to the preset fixed window to identify the defect area in the feature image may further include the following steps S2610-S2630:

Step S2610, providing at least one N×1 filter and one 1×N filter for the fixed window whose size is N×N.

N is the size of the extraction window, which can be set according to specific application scenarios and specific application requirements, for example, N can be 3.

As shown in Figure 7, it provides an N×1 filter and a 1×N filter for a fixed window of N×N. For example, if N is 3, it can be 3 in the regional attention network RPN in Figure 3 The ×3 filter is replaced with a 3×1 filter and a 1×3 filter.

Step S2620, performing dimensionality reduction processing on the feature image through the N×1 filter and the 1×N filter in sequence.

As shown in Figure 7, the feature image is input into the regional attention network, and the dimensionality reduction process is performed on the feature image through the N×1 filter and the 1×N filter in turn.

Alternatively, after passing through the N×1 filter and the 1×N filter, two 1×1 filters are respectively connected as shown in FIG. 3 to perform dimensionality reduction processing on the feature image.

In step S2630, mark regions with defects in the feature image after dimensionality reduction processing.

In this step S2630, the region with defects may be marked in the feature image after dimensionality reduction processing as the initial region of interest.

According to the above steps S2610-S2630, it replaces the existing N×N filter with a two-part structure of an N×1 filter and a 1×N filter, thereby reducing the amount of model parameters.

<Example 6>

The embodiment of the present invention also provides a target detection method. The instance segmentation framework also includes a fully connected layer FC for positioning. The target detection method of the present invention also includes the following steps S2710-S2730:

Step S2710, acquiring a matrix of image data processed in the fully connected layer.

As shown in Figure 3, the fully connected layer used for positioning can be the second "FC layer" (box reg) in the coordinates "coordinates" sub-branch in the "three branches" branch structure shown in Figure 3, where , which may be the image data output by the first "FC layer" in the "three branches" branch structure as the image data input by the second "FC layer" in the "coordinates" sub-branch.

Step S2720, constructing a processing matrix according to the image data matrix.

Step S2730, extracting the weight matrix and offset matrix corresponding to the processing matrix, decomposing the matrix of the image data, and using the decomposed matrix to perform operations in the fully connected layer.

In this step S2730, the weight matrix and offset matrix corresponding to the processing matrix may be extracted by means of singular value decomposition. It can be understood that this matrix decomposition processing method may only be applied to some branches of the FC layer, such as only applied to the second "FC layer" (box reg).

According to the above steps S2710-S2730, it can decompose the complex matrix, significantly reduce the computational complexity on the basis of ensuring the accuracy, and thus also improve the efficiency of coordinate positioning in a disguised form.

<Embodiment 7>

FIG. 8 shows an example of a target detection method. Referring to FIG. 3 and FIG. 8, in this example, the target detection method may include the following steps:

Step S8100, input (input) a target image containing the target to be detected, and perform feature extraction on the target image to obtain feature vector data reflecting the features of the target to be detected.

Step S8200, setting the number of paths and the number of network layers, and constructing an aggregation transformation network in the backbone of the convolutional feature extraction network.

Through this step S8200, the "Resnet50+FPN" structure in the backbone of the convolutional feature extraction network shown in FIG. 3 can be optimized by using an aggregation transformation network.

Step S8300, input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into feature groups corresponding to each path.

Step S8400, in the aggregation transformation network, process feature groups of each level on each path in parallel to obtain multiple processing results, and fuse the multiple processing results to obtain a feature image of the target image.

As shown in FIG. 3 , through the above steps S8200 - S8400 , the feature image FeatureMaps of the target image can be obtained.

Step S8500, input the feature image into the regional attention network RPN, through the regional attention network, slide on the feature image according to the preset fixed window to identify the defect area in the feature image, and obtain the identified defect area as the initial regions of interest and generate feature images identified with each initial region of interest.

As shown in Figure 3 and Figure 8, it can be after replacing the 3×3conv in the regional attention network RPN in Figure 3 with a 3×1conv and a 1×3conv, and respectively using a 1×1conv to the feature image Dimensionality reduction is performed to extract candidate regions that may have defects as initial regions of interest.

Step S8600, input the feature image marked with each initial region of interest into the region of interest alignment network ROI Align, through the region of interest alignment network ROI Align, pool each initial region of interest according to the preset pooling window, And generating a feature image with a corresponding region of interest identified.

As shown in Figure 3, each initial region of interest can be aligned through the region of interest alignment network ROI Align to output a feature map with a fixed size of 7x7.

Step S8700, input the feature image marked with the corresponding region of interest into the fully connected layer FC for positioning and classification, obtain the category information and coordinate information of the defect, and obtain the mask information of the defect through the Fully Convolution Nets.

As shown in Figure 3, through the first "FC layer" in the "three branches" branch structure and the second "FC layer" (bbox reg) in the coordinates "coordinates" sub-branch, the location information to which the defect belongs is obtained. Through the first "FC layer" in the "three branches" branch structure and the second "FC layer" (softmax) in the coordinate "Category" sub-branch, the category information of the defect is obtained, and, through the fully convolutional network Fully ConvolutionNets(Mask), to obtain mask information of defects.

According to this example, it can not only detect and locate the specific position of the defect in the target to be detected, but also segment the contour information of the defect, so that defects of different types and instances can be divided one by one. Moreover, since this example optimizes the convolutional feature extraction network in the instance segmentation framework, and replaces the filter in the regional attention network, the detection accuracy can be further improved, and the training time and Model inference time.

<Embodiment Eight>

FIG. 8 is a functional block diagram of an object detection device 8000 according to an embodiment of the present invention.

As shown in FIG. 8 , the object detection apparatus 8000 of this embodiment may include an extraction module 8100 , a network construction module 8200 , an input module 8300 , a parallel processing module 8400 and a fusion module 8500 .

The extraction module 8100 is used to perform feature extraction on the target image containing the target to be detected, and obtain feature vector data reflecting the features of the target to be detected.

The network construction module 8200 is used to set the number of paths and the number of network levels, and construct an aggregation transformation network in the instance segmentation framework, wherein, the number of paths in the aggregation transformation network is more than two, and the corresponding paths between different levels for data transfer.

The input module 8300 is used to input the feature vector data into the aggregation transformation network, divide the feature vector data into different levels, and divide the data of each level into feature groups corresponding to each path.

The parallel processing module 8400 is configured to process feature groups of various levels on each path in parallel in the aggregation transformation network to obtain multiple processing results.

The fusion module 8500 is used to fuse multiple processing results to obtain the feature image of the target image.

In one embodiment, the network construction module 8200 is specifically used for:

According to the number of paths, set more than two paths;

According to the number of network layers, each path includes an input layer, one or more transition layers and an output layer.

In one embodiment, the parallel processing module 8400 is specifically used for:

The dimensionality reduction processing is performed on the input feature vector data through the input layer on each path to obtain the feature group after dimensionality reduction.

Input the dimension-reduced feature group into the transition layer on the corresponding path, and perform feature extraction on the dimension-reduced feature group through the transition layer to obtain the intermediate feature group.

The intermediate feature group is input to the output layer on the corresponding path, and the intermediate feature group is processed through the output layer to increase the dimension, and the feature group after the dimension increase is obtained as the processing result of the corresponding path.

In one embodiment, the instance segmentation framework includes a regional attention network RPN, and the device 8000 also includes a region extraction module (not shown in the figure), and the region extraction module is used for:

Input the feature image into the regional attention network, through the regional attention network, slide the feature image according to the preset fixed window to identify the region with defects in the feature image, and obtain the identified defective region as the initial region of interest, and A feature image is generated identifying each initial region of interest.

In one embodiment, the region extraction module (not shown in the figure) is specifically used for:

Provide at least one N×1 filter and one 1×N filter for a fixed window with a window size of N×N; where N is the size of the fixed window;

performing dimensionality reduction processing on the feature image through an N×1 filter and a 1×N filter in sequence; and,

The region with defects is marked in the feature image after dimensionality reduction.

In one embodiment, the instance segmentation framework includes a fully connected layer FC for positioning, and the device also includes a matrix processing module (not shown in the figure), and the matrix processing module is used for:

Obtain the matrix of image data processed in the fully connected layer;

constructing a processing matrix from the matrix of image data; and,

Extract the weight matrix and offset matrix corresponding to the processing matrix, decompose the matrix of the image data, and use the decomposed matrix to perform operations in the fully connected layer.

For the specific implementation manner of each module in the device embodiment of the present invention, reference may be made to relevant content in the method embodiment of the present invention, and details are not repeated here.

The present invention can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present invention.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as “C” or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the invention are implemented by executing computer readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, a program segment, or a portion of an instruction that contains one or more executable instruction. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by means of hardware, implementation by means of software, and implementation by a combination of software and hardware are all equivalent.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein. The scope of the invention is defined by the appended claims.

Claims (10)

1. a kind of object detection method, wherein include:

Feature extraction is carried out to the target image comprising target to be detected, obtains the feature vector number for reflecting target signature to be detected According to；

Number of paths and network level quantity are set, cluster translation network is constructed in example segmentation framework, wherein cluster translation Number of paths in network is two or more, is carried out data transmission between corresponding path between different levels；

Described eigenvector data are inputted into cluster translation network, characteristic vector data are divided into different levels, by each The data of level correspond to each path and are divided into each feature group；

In cluster translation network, the feature group of each level on each path of parallel processing obtains multiple processing results；

Multiple processing results are merged, the characteristic image of the target image is obtained.

2. according to the method described in claim 1, wherein, the setting number of paths and network level quantity are divided in example Cluster translation network is constructed in frame, comprising:

According to number of paths, more than two paths are set；

According to network level quantity, being arranged includes an input layer on each path, one or more transition zones and an output Layer.

3. it is described in cluster translation network according to the method described in claim 2, wherein, on each path of parallel processing The feature group of each level, obtains multiple processing results, comprising:

Dimension-reduction treatment is carried out by characteristic vector data of the input layer on each path to input, the feature after obtaining dimensionality reduction Group；

By the transition zone in the feature group input respective path after the dimensionality reduction, by the transition zone to the spy after the dimensionality reduction Sign group carries out feature extraction, obtains intermediate features group；

The intermediate features group is inputted into the output layer in respective path, the intermediate features group is carried out by the output layer Dimension processing is risen, the feature group after rising dimension, the processing result as respective path are obtained.

4. according to the method described in claim 1, including region attention network RPN, institute in the example segmentation framework wherein Method is stated after the characteristic image for obtaining target image, further includes:

The characteristic image is inputted into the region attention network, by the region attention network, according to default fixation Window carries out the region that sliding identifies existing defects in the characteristic image, the presence after being identified on the characteristic image The region of defect generates the characteristic image for being identified with each initial area-of-interest as initial area-of-interest.

5. described to be slided on the characteristic image according to default fixed window according to the method described in claim 4, wherein The region of existing defects in the dynamic identification characteristic image, comprising:

The fixation window that size for window is N × N at least provides the filter of N × 1 and a 1 × N filter；Wherein, N For the size of the fixed window；

The filter of the N × 1 and the 1 × N filter are passed sequentially through, dimension-reduction treatment is carried out to the characteristic image；

The region of existing defects is marked in the characteristic image after the dimension-reduction treatment.

6. according to the method described in claim 1, further including full articulamentum used for positioning in the example segmentation framework wherein FC, the method also includes:

Obtain the matrix of image data handled in the full articulamentum；

According to the matrix of described image data, processing array is constructed；

The corresponding weight matrix of the processing array and bias matrix are extracted, the matrix of described image data is decomposed, is made Operation is carried out in the full articulamentum with the matrix after decomposition.

7. a kind of object detecting device characterized by comprising

Extraction module obtains reflecting that target to be detected is special for carrying out feature extraction to the target image comprising target to be detected The characteristic vector data of sign；

Network struction module constructs cluster translation in example segmentation framework for number of paths and network level quantity to be arranged Network, wherein the number of paths in cluster translation network is two or more, is carried out between corresponding path between different levels Data transmission；

Characteristic vector data is divided into difference for described eigenvector data to be inputted cluster translation network by input module The data of each level are corresponded to each path and are divided into each feature group by level；

Parallel processing module, in cluster translation network, the feature group of each level on each path of parallel processing to be obtained Obtain multiple processing results；

Fusion Module obtains the characteristic image of the target image for merging multiple processing results.

8. device according to claim 7, which is characterized in that include attention network in region in the example segmentation framework RPN, described device further include region extraction module, and the region extraction module is used for:

The characteristic image is inputted into the region attention network, by the region attention network, according to default fixation Window carries out the region that sliding identifies existing defects in the characteristic image, the presence after being identified on the characteristic image The region of defect generates the characteristic image for being identified with each initial area-of-interest as initial area-of-interest.

9. device according to claim 8, which is characterized in that the region extraction module is specifically used for:

The fixation window that size for window is N × N at least provides the filter of N × 1 and a 1 × N filter；Wherein, N For the size of the fixed window；

The filter of the N × 1 and the 1 × N filter are passed sequentially through, dimension-reduction treatment is carried out to the characteristic image；And

The region of existing defects is marked in the characteristic image after the dimension-reduction treatment.

10. device according to claim 7, which is characterized in that include used for positioning complete in the example segmentation framework Articulamentum FC, described device further include matrix disposal module, and the matrix disposal module is used for:

Obtain the matrix of image data handled in the full articulamentum；

According to the matrix of described image data, processing array is constructed；And

The corresponding weight matrix of the processing array and bias matrix are extracted, the matrix of described image data is decomposed, is made Operation is carried out in the full articulamentum with the matrix after decomposition.