JP7647597B2 - How to operate an electric furnace - Google Patents

Description

The present invention relates to a moving object identification device, a moving object identification method, and an electric furnace operation method.

In the steel industry, the use of scrap iron is expected to increase in order to reduce _CO2 emissions. Scrap iron is steel-based products such as discarded automobiles, rails, and building materials, and iron generated during the steelmaking process, and its shape is various, such as plate, bar, and block. Most of the scrap iron is melted in an electric furnace using arc discharge, and various composition adjustments are performed and processed into the desired shape before being reused. In an electric furnace, since the shapes of the scrap iron are various, it is expected that the operating state will change depending on the shape of the scrap iron. For example, if the scrap iron is thin and plate-shaped, it takes a short time to reach the melting temperature and melts early, so it is necessary to suppress the heat input. In contrast, if the scrap iron is block-shaped, it takes a long time for the heat to be transmitted to the center, so it is necessary to heat it up for a long time. Scrap iron is sometimes separated by shape, but scrap iron of various shapes is often mixed together. Therefore, in order to perform more efficient operation, a technique is required to recognize the shape of scrap iron charged into the electric furnace and control the operating state of the electric furnace according to the shape of the scrap iron. Here, as a method for identifying the shape of scrap iron, a method for identifying the shape of scrap iron from an image of the scrap iron charged into the electric furnace is effective. Specifically, as a method for identifying the shape of scrap iron from an image of the scrap iron, it is considered to use image segmentation as described in Patent Document 1. In image segmentation, a teacher image in which an object is shown and a label image in which a label according to the shape of the object is attached to the pixel in the teacher image in which the object is shown are prepared, and the shape of the object is model-learned using the teacher image and the label image. Then, the shape of the object is identified by using the learned model.

JP 2021-61065 A

EMMANUEL J. Candes, XIAODONG LI, YI MA, JOHN WRIGHT, Journal of the ACM, Vol.58, No3, Article 11

However, it is difficult to use image segmentation to identify the shape of scrap iron in images taken of scrap iron in factories with electric furnaces. More specifically, in factories with electric furnaces, large amounts of scrap iron are stored in the scrap yard. The area near the charging cart that transports scrap iron to be loaded into the electric furnace, which is a desirable location for taking images of scrap iron, is often close to the scrap yard, so the scrap iron in the scrap yard is reflected in the background image. Since the scrap iron reflected in the background image and the scrap iron in the charging cart are the same scrap iron, they cannot be distinguished by image segmentation. Therefore, even if image segmentation is used, the shape of the scrap iron cannot be properly identified, and there is a possibility that the shape of the scrap iron will be falsely detected.

The present invention has been made to solve the above problems, and its object is to provide a moving object identification device and a moving object identification method that can accurately identify the shape of a moving object. Another object of the present invention is to provide an electric furnace operating method that can operate the electric furnace efficiently.

The moving object identification device according to the present invention is characterized by comprising a preprocessing unit that converts video data of a moving object into a two-dimensional data matrix, a calculation unit that performs robust principal component analysis on the two-dimensional data matrix to obtain a sparse matrix, a binary image processing unit that generates a binary image by binarizing element values of the sparse matrix with a predetermined threshold, a first image correction unit that generates a modified binary image by performing a morphological transformation on the binary image to expand a region of a predetermined pixel value and then contract the region of the predetermined pixel value, a second image correction unit that generates a mask image by performing a morphological transformation on the modified binary image to contract a region of a predetermined pixel value and then expand the region of the predetermined pixel value, and a mask processing unit that extracts an image corresponding to the mask image from the video data of the moving object as an image of the moving object.

The moving object identification method according to the present invention includes the steps of converting video data of a moving object into a two-dimensional data matrix, performing robust principal component analysis on the two-dimensional data matrix to obtain a sparse matrix, binarizing the element values of the sparse matrix with a predetermined threshold to generate a binary image, performing a morphological transformation on the binary image to expand a region of a predetermined pixel value and then contract the region of the predetermined pixel value to generate a modified binary image, performing a morphological transformation on the modified binary image to contract a region of a predetermined pixel value and then contract the region of the predetermined pixel value to generate a mask image, and extracting an image corresponding to the mask image from the video data of the moving object as an image of the moving object.

The electric furnace operating method according to the present invention is characterized by including a step of identifying the shape of scrap iron using the moving object identification method according to the present invention and controlling the electric furnace according to the identified shape.

The moving object identification device and moving object identification method of the present invention can accurately identify the shape of a moving object. Furthermore, the electric furnace operation method of the present invention can operate the electric furnace efficiently.

FIG. 1 is a schematic diagram showing the configuration of a scrap iron recycling plant to which a shape identification system according to an embodiment of the present invention is applied. FIG. 2 is a block diagram showing the configuration of a shape identification system according to an embodiment of the present invention. FIG. 3 is a flowchart showing the flow of the shape identification process according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a histogram of sparse matrix values. FIG. 5 is a diagram for explaining the shape identification process in the embodiment.

Below, a shape identification system according to one embodiment of the present invention will be described with reference to the drawings. Note that, although this embodiment is for identifying the shape of scrap iron, the object to be identified by shape is not limited to scrap iron, and can be any moving object. In addition, the shape identification system can be installed anywhere as long as it can capture video of the moving object.

[Configuration of recycling plant]
First, with reference to FIG. 1, a configuration of a scrap iron recycling plant to which a shape identification system according to an embodiment of the present invention is applied will be described.

Figure 1 is a schematic diagram showing a scrap iron recycling plant to which a shape identification system according to one embodiment of the present invention is applied. As shown in Figure 1, in a scrap iron recycling plant to which a shape identification system according to one embodiment of the present invention is applied, scrap iron S of various shapes is stored in a scrap yard 100, and the scrap iron S is transported into a loading cart 102 by a transporting device 101 such as a crane. The loading cart 102 then loads the scrap iron S inside into an electric furnace 103, which melts the scrap iron S loaded inside and reuses it.

[Configuration of shape recognition system]
Next, the configuration of a shape identification system according to an embodiment of the present invention will be described with reference to FIG.

Figure 2 is a block diagram showing the configuration of a shape identification system according to one embodiment of the present invention. The shape identification system according to one embodiment of the present invention is an information processing system that captures video of scrap iron S being transported into a loading cart 102 by a transport facility 101 and identifies the shape of the scrap iron S from the captured video. As shown in Figure 2, the shape identification system 1 according to one embodiment of the present invention includes a photography device 2, an input device 3, an output device 4, and an image processing device 5.

The photographing device 2 is composed of a video photographing device such as a video camera, and is installed in a position (see FIG. 1) where it can photograph video of the scrap iron S being transported by the transport equipment 101 into the loading cart 102. The photographing device 2 photographs video including images of the scrap iron S being transported by the transport equipment 101 into the loading cart 102, and outputs the photographed video data to the image processing device 5.

The input device 3 is composed of input devices such as a keyboard, mouse pointer, and touch pen, and inputs operation input information to the image processing device 5.

The output device 4 is composed of output devices such as an LCD display, a printer, and a speaker, and outputs various information according to output control signals from the image processing device 5.

The image processing device 5 is composed of an information processing device such as a personal computer or a workstation, and functions as a preprocessing unit 5a, a calculation unit 5b, a binary image processing unit 5c, a first image correction unit 5d, a second image correction unit 5e, and a mask processing unit 5f by executing a computer program with an arithmetic processing unit such as a CPU (Central Processing Unit) in the information processing device. The functions of each of these units will be described later.

Such a shape identification system 1 accurately identifies the shape of the scrap by executing the shape identification process described below. Below, the operation of the shape identification system 1 when executing the shape identification process will be described with reference to the flowchart shown in FIG. 3.

[Shape Identification Processing]
3 is a flowchart showing the flow of the shape recognition process according to an embodiment of the present invention. The shape recognition process shown in Fig. 3 starts when an execution command for the shape recognition process is input to the image processing device 5 via the input device 3, and the shape recognition process proceeds to step S1.

In the processing of step S1, the photographing device 2 photographs a video including an image of the scrap iron S being transported into the loading cart 102 by the transport equipment 101, and outputs the photographed video data to the image processing device 5. The video data includes image data of the scrap iron S, which is a moving object, and image data of the stationary background. This completes the processing of step S1, and the shape identification processing proceeds to the processing of step S2.

In the process of step S2, the preprocessing unit 5a converts the video data captured in the process of step S1 into grayscale video data and creates a two-dimensional data matrix from the grayscale video data. Specifically, the video data captured in the process of step S1 may be color video data or grayscale video data, but if it is color video data, the preprocessing unit 5a converts the video data into grayscale video data in which each pixel has one brightness value. In addition, since the video data captured in the process of step S1 may include images that do not contain scrap iron S, the preprocessing unit 5a extracts video data for a period specified by the input device 3 or the like as processing target period video data. Then, since the processing target period video data is a three-dimensional matrix of two-dimensional images arranged in a chronological order, the preprocessing unit 5a converts the two-dimensional image into a one-dimensional image and converts the processing target period video data, which is a three-dimensional matrix, into a two-dimensional data matrix in which one-dimensional images are arranged in a chronological order. There is no limit to the method for converting a two-dimensional image into a one-dimensional image, as long as the method is based on a conversion rule that has a restorable regularity, such as arranging pixel values in the row direction from the top of the two-dimensional image, and then arranging the pixel values below that in the row below. This completes the processing of step S2, and the shape identification process proceeds to processing of step S3.

In the process of step S3, the calculation unit 5b performs the robust principal component analysis described in Non-Patent Document 1 on the data matrix of the two-dimensional matrix obtained by the process of step S2 to obtain a sparse matrix. In detail, the principal component analysis is a method of obtaining the direction (principal component vector) and principal components in which the variance is maximum for a data matrix, and is a method of obtaining features from data. In the principal component analysis, for a data matrix containing outliers, the outliers increase the variance, so that the principal component vectors and principal components that are greatly influenced by the outliers are obtained. In contrast, the robust principal component analysis is a principal component analysis with enhanced resistance to outliers, and is a method of removing the outliers as a sparse matrix and calculating a low-rank matrix including the principal components.

In the shape identification process of this embodiment, a sparse matrix indicating outliers is used. That is, in a video in which scrap iron S appears in the background, the background appears in every image, so the background can be considered as a principal component. On the other hand, since the scrap iron S moves relative to the background, it may or may not be present in each pixel. Therefore, when viewed from the background, the scrap iron S can be considered as an outlier. In other words, the sparse matrix obtained by robust principal component analysis of the data matrix indicates the scrap iron S. The robust principal component analysis algorithm used in this embodiment is a method called Principal Component Pursuit described in Non-Patent Document 1, but any other method may be used as long as it can detect outliers in the principal components. In the sparse matrix obtained by robust principal component analysis of the data matrix, the elements indicating the background are 0, and the elements indicating the scrap iron S are non-zero. This completes the process of step S3, and the shape identification process proceeds to the process of step S4.

In the process of step S4, the binary image processing unit 5c creates a binary image by binarizing the sparse matrix indicating the scrap iron S obtained in the process of step S3 with a predetermined threshold value. Specifically, when each element value of the sparse matrix is plotted as a histogram, it has a mountain-shaped distribution with 0 at the apex as shown in FIG. 4. As described above, in the sparse matrix, 0 indicates the background, and values other than 0 indicate the scrap iron S. Therefore, in order to clearly distinguish between the background and the scrap iron S, the element value of the sparse matrix corresponding to the i pixel of the captured image is S _i , the threshold value is a, and the sparse matrix is binarized in accordance with the conditional formula shown in the following formula (1).

For example, the binary image processing unit 5c creates a binary matrix by setting the element value of the sparse matrix that satisfies the condition shown in formula (1) to 1 and the element value of the sparse matrix that does not satisfy the condition shown in formula (1) to 0. In this embodiment, the element value of the sparse matrix indicating the scrap iron S is set to 1 and the element value of the background is set to 0, but as long as the scrap iron S and the background can be clearly distinguished from each other, any numerical value can be used for the distinction. In addition, the threshold value a in formula (1) may be set within the range of 0.1σ to 3σ by calculating the standard deviation σ of the sparse matrix. The binary image processing unit 5c then converts the binary matrix into a binary image by performing the opposite operation to the conversion rule that converted the two-dimensional image into a one-dimensional image in the preprocessing unit 5a. This completes the process of step S4, and the shape identification process proceeds to the process of step S5.

In the process of step S5, the first image correction unit 5d performs morphological transformation on the binary image created by the process of step S4 to connect pixel values 1 indicating scrap iron S, thereby creating a corrected binary image. Specifically, in the binary image created by the process of step S4, pixel values 1 indicating scrap iron S may not be connected to each other and may be missing, or points that are not scrap iron S may have pixel values 1 as noise. For this reason, the first image correction unit 5d performs morphological transformation on the binary image. Here, morphological transformation is an image processing method that expands and contracts a certain pixel value area in an image to restore missing areas and remove noise. The first image correction unit 5d expands the pixel value 1 area based on the kernel matrix, and then contracts it to fill in the missing areas of the pixel value 1 area indicating scrap iron S, thereby processing the area including scrap iron S to have pixel value 1. The kernel matrix is a matrix that indicates the size at which dilation and contraction are performed; if the kernel matrix is large, distant regions with a pixel value of 1 are connected by dilation. The size of the kernel matrix should be large enough to compensate for missing parts and not connect noise regions together. Specifically, the size of the kernel matrix should be between a 3x3 square matrix and an 11x11 square matrix. Note that during this process, dilation and contraction operations are also performed on areas with a pixel value of 1 that are noise, but in the case of noise, the areas with a pixel value of 1 are not connected to each other because they are far apart. This completes the process of step S5, and the shape identification process proceeds to the process of step S6.

In the process of step S6, the second image correction unit 5e performs a morphological transformation on the corrected binary image created in the process of step S5, first shrinking the pixel value 1 area and then expanding it, thereby creating a mask image. When the pixel value 1 area is shrunk based on the size of the kernel matrix, the scattered pixel value 1 points are removed. When the area is then expanded, the non-noise areas with large pixel values 1 are restored. The size of the kernel matrix should be such that the non-noise pixel value 1 areas are not removed during shrinkage, and specifically, it should be set between the sizes of a 3x3 square matrix and an 11x11 square matrix, similar to the size of the kernel matrix used in the process of step S5. This completes the process of step S6, and the shape identification process proceeds to the process of step S7.

In the process of step S7, the mask processing unit 5f extracts an image of scrap iron S from the video data captured by the imaging device 2 using the mask image created in the process of step S6. Specifically, the mask processing unit 5f uses the mask image corresponding to the target frame of the captured image to perform a masking operation in which the pixels of the captured image corresponding to the positions of pixel value 1 in the mask image are filled with the pixel value 0 in the mask image, while leaving the pixel value unchanged, and fills the pixels of the captured image corresponding to the positions of numerical value 0 in the mask image with a pixel value of 0, thereby extracting the image of scrap iron S. Thereafter, if the extracted image is applied to a machine learning model that has previously learned the shape of scrap iron by semantic segmentation, instance segmentation, or the like, the shape of the scrap iron can be identified with high accuracy. The extracted image may also be output to the output device 4, and the shape of the scrap iron may be identified visually. This completes the process of step S7, and the series of shape identification processes ends.

An embodiment in which the method of the present invention is applied to a video of simulated scrap iron falling from an experiment is shown. The frame rate of the video was 30 fps, and the number of frames was 40. FIG. 5(a) shows the 20th frame, a grayscale image of simulated scrap iron falling in the center. This captured image was preprocessed by the preprocessing unit 5a, and then robust principal component analysis was performed by the calculation unit 5b to obtain a sparse matrix. FIG. 5(b) shows an image of the sparse matrix. The standard deviation σ was calculated for the sparse matrix in FIG. 5(b), and a binarized image was created in the 0.5σ region, as shown in FIG. 5(c), and the white part indicating the simulated scrap iron was obtained. In FIG. 5(c), since there is a missing part in the region thought to be the simulated scrap iron, a morphological transformation was performed in the first image correction unit 5d to expand and contract the white part using a 5×5 kernel, and a corrected binarized image was created. The corrected binary image is as shown in FIG. 5(d), and the missing parts were repaired and the white areas showing the simulated scrap iron were connected. Next, since the corrected binary image shown in FIG. 5(d) has scattered white parts that are noise, the second image correction unit 5e performs morphological transformation to shrink and expand the image, resulting in the image shown in FIG. 5(e). The noise in the corrected binary image of FIG. 5(d) was removed, and a mask image was created. Then, the mask processing unit 5f used the mask image to extract the simulated scrap iron part of the captured image, resulting in the image shown in FIG. 5(f), and an extracted image showing only the falling simulated scrap iron was created. Since this extracted image shows only the moving object and the simulated scrap iron with its characteristics, applying it to a machine learning model such as image segmentation that has been trained in advance can improve the accuracy of identifying the simulated scrap iron.

As is clear from the above description, in the shape identification process according to one embodiment of the present invention, the image processing device 5 converts the video data of the scrap iron S into a two-dimensional data matrix, performs robust principal component analysis on the two-dimensional data matrix to obtain a sparse matrix, binarizes the element values of the sparse matrix with a predetermined threshold to generate a binary image, performs a morphological transformation on the binary image to expand a region of a predetermined pixel value and then contract the region of the predetermined pixel value to generate a modified binary image, performs a morphological transformation on the modified binary image to contract a region of a predetermined pixel value and then expand the region of the predetermined pixel value to generate a mask image, and extracts an image corresponding to the mask image from the video data of the scrap iron S as an image of the scrap iron S. This allows the shape of the scrap iron S to be identified with high accuracy. In addition, the electric furnace can be operated efficiently by controlling the electric furnace according to the identified shape of the scrap iron S.

When a modified binary image is generated by performing a morphological transformation to expand a region of a predetermined pixel value on the binary image and then contract the region of the predetermined pixel value, and a mask image is generated by performing a morphological transformation to expand a region of a predetermined pixel value on the modified binary image and then contract the region of the predetermined pixel value, the binary image has a missing region, so the necessary part may be recognized as noise and removed when the modified binary image is generated. Therefore, the missing region cannot be complemented when the mask image is generated, and a mask image in which the moving object part is missing may be generated. In contrast, in this embodiment, a modified binary image is generated by performing a morphological transformation to expand a region of a predetermined pixel value on the binary image and then contract the region of the predetermined pixel value, and a mask image is generated by performing a morphological transformation to contract a region of a predetermined pixel value on the modified binary image and then expand the region of the predetermined pixel value. That is, in this embodiment, the missing region is complemented by the first morphological transformation, and noise is removed by the second morphological transformation. This makes it possible to remove noise without losing necessary image information.

The above describes an embodiment of the invention made by the inventors, but the present invention is not limited to the description and drawings that form part of the disclosure of the present invention according to this embodiment. In other words, other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

REFERENCE SIGNS LIST 1 Shape identification system 2 Photography device 3 Input device 4 Output device 5 Image processing device 5a Pre-processing unit 5b Calculation unit 5c Binarized image processing unit 5d First image correction unit 5e Second image correction unit 5f Mask processing unit 100 Scrap yard 101 Transport facility 102 Charging cart 103 Electric furnace S Scrap iron

Claims (1)

a step of converting video data of a moving object into a two-dimensional data matrix, a step of performing robust principal component analysis on the two-dimensional data matrix to obtain a sparse matrix, a step of binarizing element values of the sparse matrix with a predetermined threshold to generate a binary image, a step of performing a morphological transformation on the binary image to expand a region of a predetermined pixel value and then contract the region of the predetermined pixel value to generate a modified binary image, a step of performing a morphological transformation on the modified binary image to contract a region of a predetermined pixel value and then contract the region of the predetermined pixel value to generate a mask image, and a step of extracting an image corresponding to the mask image from the video data of the moving object as an image of the moving object.

Images