JP6768572B2 - Image processing system and image processing method - Google Patents

Description

Embodiments of the present invention relate to image processing systems and image processing methods.

A system that collects information used for quality determination of an object such as a product from captured moving images / still images is known. For example, in the communication from the imager that takes a picture to the outside, the photographed image is used, and the quality judgment is performed by the processor connected to the photographed image. In addition, there is a device that outputs image information to be recorded as a record to a recorder by an independent communication path from a device that captures an image. In addition, there is one that uses a technology for compressing data for the purpose of reducing the amount of communication and the amount of recording.

International Publication No. 01/038860 Pamphlet International Publication No. 02/023480 Pamphlet International Publication No. 2010/032562 Pamphlet

In the conventional technology, the moving image / still image is not discarded until the quality judgment is performed. Therefore, when collecting information without performing the quality judgment, the imager and the processor connected to the outside are continuously connected. The amount of communication for transmitting and receiving the acquired moving image / still image and the amount of information recorded in the processor become very large. As a result, the performance requirements for hardware become high, which affects the miniaturization of equipment and the cost of procurement.

In addition, in the verification of the relevance to the quality judgment result of the final product, since a huge amount of accumulated moving images / still images are processed, it takes a lot of time in the work of the worker or the processing by the software on the computer. This is a necessary result, and it takes a long time to improve the quality.

An object to be solved by the present invention is to provide an image processing system and an image processing method capable of reducing the amount of communication and recording of information.

The image processing system of the embodiment includes at least one imager having a function of photographing an object including a product, and a processor having a function of acquiring and storing image information generated by the imager. In the image processing system, the image processor processes the photographed image to generate product state information indicating the state of the product, which is information having a capacity smaller than that of the photographed image. A means for recording image information including the captured image on a predetermined recording medium, and a state index indicating the degree of influence on the quality evaluation of the product is received from the processor and the received state index. The processor receives and receives the state information of the product from the image pickup device, which comprises means for performing a process of selecting and deleting unintentional image information recorded on the predetermined recording medium based on the above. A means for generating the state index based on the state information and sending it to the camera, and a means for acquiring image information recorded on the predetermined recording medium from the camera and recording it on another recording medium. It has.

The figure which shows an example of the use scene to which the image processing system which concerns on embodiment is applied. The figure which shows an example of the functional structure of the image processing system which concerns on the same embodiment. The figure which shows the modification of the image processing system which concerns on the same embodiment. The figure for demonstrating the detection of presence / absence of a graphic element, inclination, and deviation. The figure for demonstrating the information of invasion in an area, deviation from a movement path, displacement, and passage position. A table showing examples of data structures of state information, state indicators, and image information. A flowchart showing the operation of the entire image processing system. The flowchart which shows the operation of the image processing unit 12. The flowchart which shows an example of the operation of the image state index generation processing unit 21. The flowchart which shows the operation of the image information selection processing unit 14. The flowchart which shows the operation of the image information transmission processing unit 15. The flowchart which shows the operation of the image acquisition processing unit 22. The flowchart which shows the operation of the image information generation processing unit 13. FIG. 5 is a flowchart showing a part of the operation of the temporary recording processing unit 16 (processing of input from the image processing unit 12). FIG. 5 is a flowchart showing a part of the operation of the temporary recording processing unit 16 (processing of input from the image information selection processing unit 14). FIG. 5 is a flowchart showing a part of the operation of the temporary recording processing unit 16 (processing of input from the image information transmission processing unit 15). FIG. 5 is a flowchart showing a part of the operation of the temporary recording processing unit 16 (processing for securing free space at an appropriate timing). The flowchart which shows a part of the operation (subflow of the process of securing free space) of the temporary recording processing unit 16. The flowchart which shows the operation of the whole image processing system by the structure of FIG. The flowchart which shows the operation of the image information reading processing unit 17. The figure which shows the use scene in 1st Example. The figure for demonstrating the inclination of a product. The table which shows the comparison of the communication amount / recording amount in the case of storing a 4K video and the case of applying the first embodiment. The flowchart which shows an example of the operation of generating the state index value. A flowchart showing another example of the operation of generating the state index value. The figure which shows a part of the screen for setting a threshold value in a processor. The figure which shows a part of the screen for setting a threshold value in a processor. The figure which shows the use scene in the 2nd Example. The figure which shows the use scene in 3rd Example. The flowchart which shows the operation of the relevance verification in the 3rd Example. The figure which shows the example of the display of the relevance verification in the 3rd Example.

Hereinafter, embodiments will be described with reference to the drawings.
(Outline of Embodiment)
In the present embodiment, in the moving image / still image camera, "product state information" (information indicating the state of the product) in which the amount of communication is much smaller than that of the moving image / still image obtained by shooting an object including the product. ) Is generated and transmitted to the processor to reduce the amount of communication, and in the processor, a "state index" is used for later detailed verification and relevance verification with the quality judgment result. By generating (information indicating the degree of influence on product quality evaluation) and based on this, limited acquisition and recording of video / still image information that may contain quality problems, Achieve reduction of communication volume / recording volume. As a result, not only one system but also a system composed of a large number of moving image / still image photographers can achieve a significant reduction in communication volume and recording volume, and realize a cost-effective system.

In addition, in the relevance verification with the quality judgment result, the state information of the product obtained sequentially and the image information including the part of the moving image / still image that may contain a quality problem can be targeted. It reduces the amount of recording and enables the time required for verification of relevance to the final product quality judgment result.

In addition, when shooting with a video / still image camera with a limited recording capacity, it is possible to acquire the shot video / still image due to the limitation of observation time due to environmental conditions, insufficient communication line capacity, etc. , Even if it is limited to a specific timing such as after the end of observation, we will realize a system that can preferentially acquire and record highly important moving image / still image information using state information.

(Image processing system usage scene)
FIG. 1 is a diagram showing an example of a usage scene to which the image processing system according to the embodiment is applied.

Here, as a usage scene, a case where a product or the like is continuously moved and processed on a production line composed of a plurality of work steps is illustrated. The specific embodiment will be described in detail later, and the outline thereof will be briefly described here.

The image processing system according to the present embodiment includes at least one moving image / still image camera 10 and a processor 20. Each moving image / still image capturing device 10 captures an object including a product and generates image information, and the processor 20 acquires the image information generated by each moving image / still image capturing device 10. It is to be saved.

In the usage scene shown in FIG. 1, information necessary for quality judgment in order to deal with rare and unpredictable quality problems of products manufactured by continuously moving on a production line. A plurality of video / still image photographers 10 (video / still image photographers (A) to (D)) are installed along the production line for the purpose of acquiring the image, and may be related to quality problems. A processor 20 for acquiring and recording moving images / still images used for verification of a certain product state is installed.

In this usage scene, the relationship between the image information obtained by the processor 20 and the information collected by other means and the final quality judgment result of the product is further verified and observed by offline processing. It is assumed that there are processes for quality improvement after clarifying the target / improvement target, but these processes are outside the scope of the image processing system according to the present embodiment.

(Configuration and operation of image processing system)
FIG. 2 is a diagram showing an example of the functional configuration of the image processing system according to the present embodiment.
Here, for the sake of simplicity, one moving image / still image camera 10 and a processor 20 are shown. Of course, even when there are a plurality of moving image / still image capturing devices 10, the relationship between the moving image / still image capturing device 10 and the processor 20 is the same. The elements in the dotted line in FIG. 2 are outside the scope of the image processing system according to the present embodiment.

The moving image / still image camera 10 is a product in which the capacity of information is smaller than that of the captured image by processing the moving image or still image (hereinafter referred to as captured image) obtained by shooting with the camera 11 by the image processing unit 12. The state information of the product is generated, the state information of this product is sent to the processor 20, and the image information including the captured image and the identifier of the state information is generated by the image information generation processing unit 13, and the generated image information. The temporary recording processing unit 16 has a function of recording the image on a predetermined recording medium (for example, a recording medium provided in the moving image / still image camera 10). The camera 11 may be externally configured without being included in the configuration of the moving image / still image camera 10.

Further, the moving image / still image capturing device 10 receives a state index indicating the degree of influence on the quality evaluation of the product from the processor 20, and the image information selection processing unit 14 receives the received state index to record the predetermined state. It is determined whether each image information recorded on the medium is unexpected or significant, and the temporary recording processing unit 16 is requested to delete the unexpected image information (image information deletion request), while the deletion is performed. For significant image information, it has a function of notifying the image information transmission processing unit 15 of the state index.

Further, the moving image / still image camera 10 selects and processes the corresponding image information from the temporary recording processing unit 16 by the image information transmission processing unit 15 using the state index notified from the image information selection processing unit 14. It has a function to send to the vessel 20.

Further, when the image information transmission processing unit 15 completes the transmission of the image information to the processing unit 20, the moving image / still image camera 10 requests the temporary recording processing unit 16 to delete the corresponding image information ( It has a function to delete image information).

On the other hand, the processor 20 has a function of receiving product status information from the moving image / still image capturing device 10, generating the status index based on the received status information, and sending the status index to each moving image / still image capturing device 10. I have.

Further, the processor 20 acquires the image information recorded in the predetermined recording medium from the moving image / still image capturing device 10 by communication with the moving image / still image capturing medium 10 or the like, and another recording medium (for example, It has a function of recording on a recording medium provided in the moving image / still image camera 10.

The image information (moving image / still image, state information, etc.) stored by the processor 20 is related in another processor 30 in order to verify the relevance with the quality determination result of the product from the inspection device or the like. It will be input to the sex verification processing unit 31 and used. From the result of this relevance verification, the observation target / improvement target will be clarified and quality improvement will be implemented.

In addition to including the main body of the captured image, each image information includes information indicating the shooting date and time, information indicating the shooting position, information indicating the shooting parameters used at the time of shooting, and information indicating the identifier of the state information. May be good.

Further, the moving image / still image capturing device 10 has a function of selecting significant image information recorded on the predetermined recording medium based on the received state index and sending it to the processor 20.

Further, the moving image / still image capturing device 10 outputs the image information recorded on the predetermined recording medium to a cable or the like connected to the processor 20 (or writes the image information to another recording medium) in response to a predetermined detection. ) Is provided.

Further, the processor 20 has a function of outputting image information acquired from the moving image / still image camera 10 and recorded on a recording medium to the outside as information used for product quality evaluation.
Further, the processor 20 has a function of recording the image information acquired from the moving image / still image camera 10 on the recording medium in an associated state together with the corresponding state information and the state index.

Further, the moving image / still image camera 10 has a function of preferentially deleting the image information recorded on the predetermined recording medium from the oldest one when the storage capacity of the predetermined recording medium exceeds a certain value. It has.

Further, when the storage capacity of the predetermined recording medium exceeds a certain value, the moving image / still image camera 10 captures the image information recorded on the predetermined recording medium with an older shooting date and more unexpectedly. It has a function to preferentially delete things.

Further, the moving image / still image capturing device 10 may use one or more of a binarized image obtained from the captured image, an edge detection result, and a difference image from the reference image as state information to be transmitted to the processor 20. It has the ability to apply combinations.

Further, the moving image / still image capturing device 10 has any one of a predetermined presence / absence of a graphic element, a tilt of the product from a predetermined position, and a deviation detection result as state information to be transmitted to the processor 20. Or it has a function to apply a plurality of combinations.

Further, the moving image / still image photographing device 10 uses the state information to be transmitted to the processing unit 20 as a predetermined movement of the product as a result of detecting the intrusion of an object into a predetermined single or a plurality of areas of the photographing range. It has a function to apply any one or a combination of information on the amount of deviation from the route or a predetermined position and the information on the passing position of the divided area.

If the communication line from the moving image / still image capturing device 10 to the processor 20 is not suitable for transmitting image information, the configuration shown in FIG. 3 is adopted instead of the configuration shown in FIG. 1, and the image transmission processing unit 15 is used. The recorded contents of the moving image / still image camera 10 are separately recorded by the image information reading processing unit 17 in a high-speed wireless environment, high-speed communication using a wired connection, or recorded without sending the image information to the processor 20. You may read it using media.

(Information generated by the image processing unit 12)
4 and 5 are diagrams for explaining a part of the image processing performed by the image processing unit 12. Hereinafter, the processing result obtained from this image processing is referred to as a "state information body".

The image processing unit 12 has a function of generating a post-processed image such as a binarized image, an edge detection result, and a difference image from a reference image by performing image processing on the input captured image.

In addition, the image processing unit 12 has a function of generating element analysis results as shown in FIGS. 4 (a) to 4 (d) by performing image processing on the input captured image. FIG. 4A shows the detection of the missing P of the drawing element. FIG. 4B shows the detection of the unnecessary element P of the graphic element. FIG. 4C shows the detection of the inclination P of the graphic element. FIG. 4D shows the detection of the deviation P of the graphic element.

In addition, the image processing unit 12 has a function of generating element analysis results as shown in FIGS. 5A to 5D by performing image processing on the input captured image. FIG. 5A shows the detection of the intruding portion P into the predetermined region of the product. FIG. 5B shows the detection of the deviation dY (the amount of deviation in the Y-axis direction) from the predetermined movement path. FIG. 5C shows the detection of deviation dX (amount of deviation in the X-axis direction) and deviation dY (amount of deviation in the Y-axis direction) from a predetermined position. FIG. 5D shows information on the divided passing positions (information such as the number of products passing through each passing position).

Further, the image processing unit 12 has a "state identifier" which is an identifier of the state information unique to the "state information main body" generated in this way, and a "captured image identifier" which is an identifier of the target captured image. ", And has the function of generating" state information "of products containing them. The data structure of "state information" is shown in the table of FIG. 6A.

(Information generated by the image state index generation processing unit 21)
The image state index generation processing unit 21 has a function of calculating a "state index value" by inputting a state information main body in the state information received via a communication line. The "state index value" is a numerical value of the degree of influence on the quality evaluation of the captured image. For example, a larger number means that it is significant, and a smaller number means that it is unintentional. The quantification method depends on the shape / material of the target product, the quality determination method, etc., and is outside the scope of this embodiment, and some examples will be shown in Examples described later.

The image state index generation processing unit 21 generates a "state index" including the "state index value" generated in this way and the "state identifier" and the "captured image identifier" included in the above-mentioned state information. .. The data structure of the "state index" is shown in the table of FIG. 6 (b).

(Information generated by the image information generation processing unit 13)
The image information generation processing unit 13 generates the state information of the product in the image processing unit 12 among the input captured images based on the captured image input from the camera 11 and the state identifier input from the image processing unit 12. In addition to the "captured image body" that summarizes the images used for this purpose, various information indicating the image processing contents of the image processing unit 12 such as the "shooting date and time" of the image, and input from the image processing unit 12. "Image information" including "state identifier" and "captured image identifier" is generated and output to the temporary recording processing unit 16. Further, in addition to the date and time information, position information obtained by another means and "shooting parameters" such as aperture / shutter speed may be included in the "image information", and the state information corresponding to the state identifier may be included in the "image". It may be included in "information". The data structure of "image information" is shown in the table of FIG. 6 (c).

(Operation of the entire system)
Next, the overall operation of the image processing system will be described with reference to FIG. 7.

The moving image / still image camera 10 generates image state information of the product, which is information having a smaller capacity than the captured image, by performing image processing on the captured image obtained by photographing with the camera 11 by the image processing unit 12. The state information of this product is sent to the processor 20 (S11).

Then, the processor 20 receives the state information from the moving image / still image capturing device 10, generates the state index based on the received state information, and sends it to each moving image / still image capturing device 10 (S12).

The moving image / still image camera 10 receives the state index from the processor 20, and is the corresponding image information recorded on the predetermined recording medium based on the state index by the image information selection processing unit 14 abrupt or significant? Is determined (S13).

On the other hand, the moving image / still image camera 10 generates image information including a photographed image obtained by photographing with the camera 11 and a state information identifier (state identifier) by the image information generation processing unit 13. (S14), the generated image information is recorded by the temporary recording processing unit 16 on a predetermined recording medium (for example, a recording medium provided in the moving image / still image camera 10) (S15).

When the determination result in step S16 is significant (YES in S16), the moving image / still image camera 10 processes the corresponding image information recorded on the predetermined recording medium by the image information transmission processing unit 15. Sending to 20 (S17). Then, the processor 20 receives the image information from the moving image / still image capturing device 10 by the image acquisition processing unit 22, and records the received image information on another recording medium (for example, the recording provided in the moving image / still image capturing device 10). Record on the medium) (S18).

On the other hand, when the determination result in step S16 indicates involuntary (NO in S16), the moving image / still image photographer 10 does not perform the processing of steps S17 and S18, and the temporary recording processing unit 16 puts it on the predetermined recording medium. The recorded corresponding image information is deleted (S19).

When the above-mentioned process is continued after this (YES in S20), the process is repeated. On the other hand, if it is not necessary to continue the processing (NO in S20), all the processing is terminated.

The processing flow of the image processing unit 12 is shown in FIG.

The image processing unit 12 cuts out a necessary image from the input captured image according to the subsequent processing (S21), and outputs the information of the captured image used. For example, the image processing unit 12 performs a process of generating one or more of the following three images on the captured image (S22, S23A to S23C), and after processing the moving image / still image to be input. Make the resulting capacity reduced. The processing content is not limited to the following as long as the capacity is reduced.

-A combination of one or more of the processed images such as the binarized image obtained from the captured image, the edge detection result, and the difference image from the reference image (S23A).
-As an element analysis result of the captured image, any one or a combination of a plurality of predetermined graphic element presence / absence, product inclination from a predetermined position, and deviation detection result (see FIG. 4) (S23B).
-Information on the detection result of intrusion of an object into a predetermined single or multiple areas of the shooting range, the movement path of the product or the amount of deviation from the predetermined position, and the passing position of the divided area (Fig.) 5) Any one or more combinations (S23C)
Further, the image processing unit 12 generates an identifier of state information (state identifier) and an identifier of a photographed image (photographed image identifier) (S24, S25), and generates state information including them (S26).

The image processing unit 12 sends the generated state information identifier (state identifier) and the captured image identifier (photographed image identifier) to the image information generation processing unit 13 (S27), and processes the generated state information via the communication line. It is sent to the vessel 20 (S28).

In this way, the moving image / still image camera 10 transmits the state information whose capacity is reduced as compared with the captured image to the processor 20 to increase the amount of communication output via the communication line. Reduce. As a result, the cost of the communication line can be reduced.

The processing flow of the image state index generation processing unit 21 is shown in FIG.

The image state index generation processing unit 21 calculates the state index value by inputting the state information main body in the state information received via the communication line (S31, S32).

Further, the image state index generation processing unit 21 generates a state index including the calculated state index value, the identifier (state identifier) included in the state information, and the identifier of the captured image (photographed image identifier). (S33), the image is sent to the moving image / still image camera 10 via the communication line (S34).

The processing flow of the image information selection processing unit 14 is shown in FIG.

The image information selection processing unit 14 inputs the state index from the processor 20 (S41), compares the state index value of the input state index with the preset determination standard (S42), and determines that it is significant. Regarding the thing (YES in S42), the state index is output to the image information transmission processing unit 15 (S43). For those determined to be unintentional (NO in S42), the temporary recording processing unit 16 is requested to delete the image information by using the captured image identifier in the state index (S44).

The determination criteria are set, for example, by operating a switch provided in the moving image / still image capturing device 10 or by remote control from the processor 20, and when there are a plurality of moving image / still image capturing devices 10. Adjustments are made according to individual installation environments such as lighting conditions and camera angles.

The moving image / still image capturing device 10 can reduce the recording amount required for the moving image / still image capturing device 10 by deleting the image information determined to be unintentional using the state index from the recording. As a result, it is possible to reduce the demand for hardware for the moving image / still image camera 10 and realize low cost and miniaturization.

FIG. 11 shows a processing flow of the image information transmission processing unit 15.

When the image information transmission processing unit 15 inputs the state index from the image information selection processing unit 14 (S51), the captured image identifier is extracted from the input state index (S52), and the captured image identifier is used for temporary recording processing. The corresponding image information is acquired from the unit 16 (S53) and sent to the processor 20 (S54).

Next, the image information transmission processing unit 15 confirms the presence or absence of a reception confirmation response from the processor 20 (S55). When the reception confirmation response from the processor 20 has not been received (NO in S56), the image information transmission processing unit 15 outputs the image information temporary storage request together with the status index to the temporary recording processing unit 16 (S57). Even if the transmission of image information fails, the data should be retrieved later as much as possible. On the other hand, when the reception confirmation response from the processor 20 is received and the transmission to the processor 20 is completed (YES in S56), the image information transmission processing unit 15 deletes the corresponding image information together with the state index. The information deletion request is output to the temporary recording processing unit 16 (S58).

In this way, the moving image / still image photographing device 10 sends a part selected from all the captured images to the processing unit 20. This makes it possible to reduce the amount of communication output via the communication line.

The processing flow of the image acquisition processing unit 22 is shown in FIG.

When the image acquisition processing unit 22 receives the image information via the communication line (S61), the image acquisition processing unit 22 outputs the received image information to the recording processing unit 23 (S62).
When the image information of the image acquisition processing unit 22 is input, the recording processing unit 23 records the input image information. The image information recorded here is used for verifying the relevance with the quality determination result obtained by another means.

The image information recorded by the recording processing unit 23 does not include the one determined to be unintentional, and the recording amount is reduced. The reduction of the recording amount of the information in the processor 20 and the subsequent relevance. The processing time of the relevance verification in the verification processing unit 31 is reduced. As a result, the demand for hardware for the processor 20 can be reduced, and low cost and miniaturization can be realized.

The processing flow of the image information generation processing unit 13 is shown in FIG.

The image information generation processing unit 13 inputs a moving image / still image from the image processing unit 12 (S71), cuts out the image (S72), and outputs a state information identifier (state identifier) and a captured image from the image processing unit 12. The identifier (photographed image identifier) of the above is input, image information including them is generated (S73), and output to the temporary recording processing unit 16 (S74).

The processing flow of the temporary recording processing unit 16 is shown in FIGS. 14A to 14E.

FIG. 14A shows a processing flow of input processing from the image processing unit 12.
As shown in FIG. 14A, when the temporary recording processing unit 16 receives the image information input from the image information generation processing unit 13 (S81), the temporary recording processing unit 16 confirms the free space for recording the image information (S82), and is free. After securing the capacity on the recording medium (S83), the image information is recorded (S84).

FIG. 14B shows a processing flow of input processing from the image information selection processing unit 14.
As shown in FIG. 14B, when the temporary recording processing unit 16 inputs a deletion request for image information determined to be unintentional from the image information selection processing unit 14 (S91), the temporary recording processing unit 16 uses the captured image identifier included in the corresponding state index. Then, the corresponding image information is deleted (S92).

FIG. 14C shows a processing flow of input processing from the image information transmission processing unit 15.
As shown in FIG. 14C, when the temporary recording processing unit 16 inputs a request from the image information transmission processing unit 15 (S101), the temporary recording processing unit 16 determines whether the request is an image information deletion request or a temporary holding request (S101). S102).

In response to the image information deletion request, the temporary recording processing unit 16 deletes the corresponding image information by using the captured image identifier in the state index included in the request (S103). On the other hand, in response to the image information temporary storage request, the storage of the corresponding image information is maintained (S104), and the input state index is held to prepare for the retrieval of the image information by means other than communication.

FIG. 14D shows a processing flow of processing for securing free space at an appropriate timing.
As shown in FIG. 14D, the temporary recording processing unit 16 secures a free space for recording the image information at an appropriate timing or when the image information is input from the image information generation processing unit 13. The process is performed (S111), and the same process is repeated as necessary.

FIG. 14E shows a subflow of the process of securing free space in FIG. 14D.
As shown in FIG. 14E, the temporary recording processing unit 16 confirms the free space for recording the image information (S121).

The temporary recording processing unit 16 determines the free space using a preset lower limit of the free space (S122). If the free space is greater than or equal to the lower limit, no processing is performed. However, if the free space is less than the lower limit or insufficient for recording new image information, check the recorded image information (S123) and temporarily. The deletion is repeated in the order of the oldest shooting date and time of the image information to be saved request until the free space exceeds the free space lower limit (S124).

The temporary recording processing unit 16 confirms the free space for recording the image information again (S125), and determines the free space using the preset lower limit value of the free space (S126). If the free space is greater than or equal to the lower limit, no processing is performed. However, if the free space is less than the lower limit or insufficient for recording new image information, the shooting date and time of the image information subject to the temporary storage request is set. The deletion is repeated in the order of oldest and smallest state index value included in the state index until the free space exceeds the free space lower limit value (S127).

As a result, the amount of information recorded in the moving image / still image camera 10 can be reduced, the demand for hardware can be reduced, cost reduction and miniaturization can be realized, and the processing unit 20 in the image information transmission processing unit 15 can be realized. Image information can be obtained as much as possible even when the reception confirmation response from the processor 20 is unconfirmed when communication with the processor 20 fails.

FIG. 15 shows the processing flow of the entire image processing system according to the configuration of FIG.

As described above, when the communication line from the moving image / still image capturing device 10 to the processor 20 is not suitable for transmitting image information, the configuration shown in FIG. 3 is adopted instead of the configuration shown in FIG. 1 and the image is transmitted. The image information reading processing unit 17 separately performs high-speed communication using a high-speed wireless environment and a wired connection for the recorded contents of the moving image / still image capturing device 10 without sending the image information to the processing unit 20 by the processing unit 15. Alternatively, it may be read out using a recording medium.

In this case, the processing flow of FIG. 7 described above changes like the processing flow of FIG.

The processing of S131 to S133 and S134 to S135 in FIG. 15 is basically the same as the processing of S11 to S13 and S14 to S15 of FIG. However, in S134 of FIG. 15, it is desirable that the moving image / still image photographer 10 confirms the free space for recording the image information when the image information generation processing unit 13 generates the image information.

Further, when S136 to 138 in FIG. 15 and S16 to S20 in FIG. 7 are compared, there is no processing corresponding to S17 and S18 in FIG. 7 in FIG.

In the processing flow of FIG. 15, when the repetitive processing is completed in S138 (YES in S138), the moving image / still image capturing device 10 is subsequently recorded by the image information reading processing unit 17 in the moving image / still image capturing device 10. Is read (S139). In this case, the recorded content of the moving image / still image camera 10 is read out separately using, for example, a high-speed wireless environment, high-speed communication using a wired connection, or a recording medium.

On the other hand, the processor 20 acquires the image information input from the connected cable or the image information from the recording medium by the image acquisition processing unit 22, and records it on a predetermined recording medium by the recording processing unit 23 (S140).

The processing flow of the image information reading processing unit 17 is shown in FIG.

The image information reading processing unit 17 reads the recorded image information from the temporary recording processing unit 16 by operating a switch provided in the moving image / still image camera 10, detecting a cable connection, or detecting the insertion of a recording medium. (S141), output to a cable or write to a recording medium (S142).

As a result, the amount of information recorded in the moving image / still image camera 10 can be reduced, the demand for hardware can be reduced, and cost reduction and miniaturization can be realized.

Hereinafter, each embodiment will be described.

(First Example)
FIG. 17 is a diagram showing an example of a usage scene to which the image processing system is applied in the first embodiment.

Here, as a usage scene, a specific example of a case where a product or the like is continuously moved and processed on a production line composed of a plurality of work steps is shown. The work places 1 to 3 and the inspection device 40 shown in FIG. 17 contribute to the manufacture of the product and are not included in the configuration of the image processing system.

In the system of this embodiment, as shown in FIG. 17, a plurality of the above-mentioned moving image / still image capturing devices 10 are installed between each work place, and only one processor 20 is installed.

The work places 1 to 3 are places where various processes are performed on the products flowing on the line by a machine or a worker, and the inspection device 40 determines the quality of the final product.

Here, it is necessary to verify the relationship between the state of the semi-processed product flowing on the production line including the work places 1 to 3 at a speed of, for example, 30 units / minute and the quality determination result in the inspection device 40. Think.

As shown in FIG. 2, each of the plurality of moving image / still image capturing devices 10 (moving / still image capturing devices (A) to (D)) shows images of products moving on the production line of the camera 11. The image information obtained by taking a picture is sent to the image processing unit 12 and the image information generation processing unit 13.

The image information generation processing unit 13 temporarily holds the input video, enables the image information to be specified by using the shooting date and time input from the image processing unit 12, and captures a still image cut out from the moving image. As the main body, image information is generated and output to the temporary recording processing unit 16.

The captured image body may be a partial moving image including a designated shooting date and time having a time width reflecting the amount of information recorded by the moving image / still image capturing unit 10 and the amount of communication with the processor.

The image processing unit 12 performs the above-mentioned processing of FIG. 8 using the input video. The shooting date and time of the video used for generating the state information is output to the image information generation processing unit 13, and the image information for later detailed verification of the state information can be generated. The details of the image processing unit 12 in this embodiment will be described later with reference to FIGS. 18 and 19.

As shown in FIG. 20, the processor 20 takes out the state information main body (S151) from the received state information by the image state index generation processing unit 21, and the state information main body (ABS) sets a threshold value specified in advance. It is determined whether or not it exceeds (S152). If the threshold value is not exceeded, the state index value is set to 0 (S153), while if it is exceeded, the state index value is set as the value of the state information main body (S154). Then, the processor 20 generates a state index including the state specified value, the state identifier and the captured image identifier (S155), and transmits the state index to the moving image / still image photographer 10. The threshold value may be set to a different value for each moving image / still image photographer 10.

The moving image / still image photographer 10 performs the above-described processing of FIG. 10 on the state index value in the received state index by the image information selection processing unit 14. Here, the significance / inaction determination of the state index value is performed by comparing the following equations, for example, using a numerical value as a determination criterion.

State index value> Judgment reference value: Significant state index value ≤ Judgment reference value: The result of the involuntary significant / random judgment is input to the image information transmission processing unit 15.

The determination reference value may be set to a different value individually for each moving image / still image photographer 10.

The moving image / still image camera 10 performs the processing of FIG. 11 described above by the image information transmission processing unit 15, and transmits the image information to the processing unit 20.

The processing unit 20 acquires image information from the moving image / still image capturing device 10 by the image acquisition processing unit 22 by the process of FIG. 12 described above, and records and saves the acquired image information on a recording medium by the recording processing unit 23. To do.

Next, the details of the image processing unit 12 and its effects in the examples are shown below.

The contents of the state information generated by the image processing unit 12 include the image after processing such as the binarization, the presence / absence of graphic elements, inclination, deviation detection, intrusion into the area, deviation from the movement path, displacement, and passing position. Any or combination of information can be mentioned, but the selection is set individually according to the shape of the target product and the processing content at each work place.

Here, it is assumed that the inclination of the graphic element shown in FIG. 4C described above is selected as the state information output by the moving image / still image photographer 10. The state of inclination in this case is shown in FIG.

The example of FIG. 18 shows that a product flowing on a production line has an inclination of an angle θ with respect to a predetermined inclination (direction of a dotted line in the vertical direction in FIG. 18) with counterclockwise as positive. There is.

Here, consider the case where the measurement resolution of the angle θ is 0.1 degrees. At this time, it is necessary when the vertical direction of the shooting range of the moving image / still image camera 10 coincides with the vertical direction of the manufacturing line (the direction indicated by L in FIG. 18) and the product has the vertical dimension of L / 3. The minimum number of pixels is obtained as follows.

L = (3 / tan 0.1 °) ~ 1718.9
Therefore, the vertical angle of view size of the moving image / still image camera 10 is required to be 1718.9 or more, and 4K (3840 * 2160) or more is required as a general image size. It should be noted that a larger image size is required for a smaller product or when a smaller measurement resolution is required.

Here, in the system of this embodiment, in the moving image / still image camera 10, among the state information shown in the table of FIG. 6A, the state information main body outputs the inclination of 0.1 degree resolution. Think. When the range of the state information body is [-180, 180], it may be a signed 13-bit integer. Therefore, here, as the state information main body, integer data with a 2-byte sign is used. Since the captured image identifier and the state identifier need only correspond to the product unit flowing on the production line, in the case of a speed of 30 units / minute, unsigned 16 bits for recording for one day, for one year. It can be handled with unsigned 24-bit data. Here, considering the case of recording for one day, the amount of data is 6 bytes as the state information sent from the moving image / still image capturing device 10 to the processor.

When there is nothing judged to be significant in the image information selection processing unit 14, the communication amount / recording amount when storing the 4K video is compared with the communication amount / recording amount when this embodiment is applied. It is shown in the table of FIG. From this table, it can be seen that when this embodiment is applied, the amount of communication and the amount of recording can be reduced to 1 / 32,000.

If the image information selection processing unit 14 determines that it is significant, the corresponding image information is transmitted from the moving image / still image capturing unit 10 to the processing unit 20. Since this determination sends and receives only significant information due to the setting of the threshold value for calculating the state index in the processor and the determination criteria in the moving image / still image camera 10, 4K video is transmitted when this embodiment is applied. It is self-evident that the amount of communication / recording is reduced compared to doing so.

It is also conceivable to use a dedicated cable for 4K video transmission, but when applied to this example, by reducing the amount of communication, a system is configured using a LAN communication network by combining various other sensors. This makes it possible to reduce the cost / time required for cable laying.

In addition, when re-verifying the status confirmation of products moving on the production line from the recorded and saved information, we are paying attention when applying this example compared to the case where a large amount of 4K video is targeted. It is possible to reduce the time required for re-verification and the demand for re-verification equipment by targeting the information in which the recorded amount is reduced by narrowing down according to the inclination of the product.

As described above, by collecting the product inclination and image information at the installation location of each moving image / still image capturing device 10 and verifying the relevance with the quality determination result in the inspection device 40, for example, by confirming the correlation with time. It can be carried out. For example, if there is a relationship between the positive tilt value of the product in the moving image / still image photographer 10 (A) and the poor quality determination result, the tilt is simply checked by checking the corresponding image information. It is possible to examine in more detail whether only the ones are related or whether there are events occurring in relation to the inclination. Based on the information obtained here, for example, by providing a mechanism for correcting the inclination of the product at the monitoring point of the moving image / still image camera 10 (A), the quality of the product can be improved.

Further, when this embodiment is applied, an example in which the communication amount / recording amount can be further reduced by reflecting the verification result of the relevance of the quality determination result in the inspection device 40 is shown below.

Consider a situation in which it is determined that there is no relationship between the case where the product inclination is negative and the case where the quality determination result is negative, based on the product inclination and the quality judgment result in the inspection device. In this case, there is no need to record and store information when the product tilt is negative, both for more detailed verification of the relevance and for continuous monitoring of the production line.

Correspondingly, in the process of FIG. 20 in which the image state index generation processing unit 21 of the processor uses the value of the state information main body to calculate the state index value, the state index is shown in the dotted frame of FIG. A process (S156) for determining the sign of the value is additionally provided, and the state index value is changed to 0 when the value is negative.

As described above, in the system to which this embodiment is applied, information having a negative inclination determined to be unnecessary can be communicated without modifying a plurality of moving image / still image photographing units 10 and only by making corrections on the processing unit side. It is possible to reduce the amount / recording amount.

Further, when this embodiment is applied, an example of a method of controlling image information to be recorded and saved while determining the recording amount is shown below.

Consider the case of collecting product inclination information at the initial stage of verification of the relationship between the quality judgment result in the inspection device and the product inclination. In the initial stage, collecting as much information as possible is superior to the accuracy of the results in the subsequent verification work.

Here, the determination criteria shown in FIG. 10 of the plurality of moving image / still image capturing devices 10 (moving / still image capturing devices (A) to (D)) shown in FIG. 17 are fixed values, for example, 0 as the initial state. Consider a given situation. In order to collect a lot of information, it is desirable that the threshold values used in FIGS. 9, 10, 20 and 21 are as small as possible.

In the system to which this embodiment is applied, for example, screens as shown in FIGS. 22A and 22B are provided to the operator, and the operator monitors and sets the recorded amount of the recorded information and the change in the tilt amount. The recording amount can be controlled by controlling the threshold value while predicting the expected increase in the recording amount when the threshold value is used.

The components of FIGS. 22A and 22B will be described.
On the screen of FIG. 22A, the result display of the moving image / still image capturing device 10 (moving image / still image capturing device (A) to (D)) and the quality determination result from the inspection device 40 are displayed with respect to the time direction vs. tilt value. There is a part to display a chart, and as soon as the processor 20 receives new results in sequence, the display is updated while scrolling in the horizontal direction.

Since the observation time for the same product differs between the moving image / still image camera 10 and the inspection device according to the speed of the manufacturing line, it is corrected so that the same product is displayed vertically on the display. It is assumed that In addition, the cursor K indicating the threshold value specified by the operator is displayed on the time direction vs. slope value chart, the threshold value is displayed on the chart at the same time, and the data Y of the slope value exceeding the threshold value is filled in and displayed. It is displayed so as to be easily distinguishable from the display of the slope value data M that does not exceed the threshold value. If there is a defect in the quality determination result from the inspection device 40, the data F indicating the defect is displayed.

On the screen of FIG. 22B, for example, one day's estimation when the total recording capacity available for information retention, the recording capacity of the information already recorded and saved, and the setting of the threshold value specified by the operator are used. The consumption amount and the estimated free space are displayed. The estimated consumption amount is calculated from the already recorded information by, for example, the following formula.

(Estimated daily consumption)
= (Number of data points exceeding the threshold per hour) x (Average size of image information) x 24
As described above, the operator confirms the estimated consumption amount while visually recognizing the distribution of the data points in which the image information is recorded in the time direction in conjunction with the specified threshold value. By determining the threshold value, it is possible to control the image information recorded and saved.

(Second Example)
In the second embodiment, an example in which the image processing system is applied to a manufacturing system including a plurality of parts factories and one assembly factory existing in geographically separated places as shown in FIG. 23 will be shown.

Each parts factory and assembly factory each has a production line similar to that shown in the first embodiment, and as in the first embodiment, moving image / still image capturing devices 10 are arranged in various places and processors. It is assumed that only one 20 is arranged. The moving image / still image camera 10 of each component factory is connected to the processor 20 via the communication line T1, and the moving image / still image camera 10 of the assembly factory is connected to the processor 20 via the communication line T2.

Here, not only the quality of the parts shipped from each parts factory to the assembly factory, but also the quality judgment result in the inspection device of the assembly factory and the state of the parts in the work step of each assembly factory are acquired by using image information and related. Consider verifying sex.

In this case, as in the case shown in the first embodiment, in the system to which this embodiment is applied, the amount of communication from each moving image / still image camera 10 to the processor 20 is reduced, and the information in the processor 20 is information. The amount of recording can be reduced.

In particular, in this embodiment, since a factory located in a geographically distant place is assumed, it is assumed that a communication line whose cost fluctuates according to the communication amount of the communication company is used for communication to the processor 20. Therefore, the cost advantage of the system to which this embodiment is applied becomes greater than that of the first embodiment.

Further, since the number of the target moving image / still image camera 10 also increases, the required reduction amount of the image recording amount and the reduction amount of the cost for the hardware are further larger than those of the first embodiment. Become.

(Third Example)
In the third embodiment, an example in which the image processing system is applied to a situation in which a large number of small products as target products flow on the production line is shown.

As an object for verifying the relevance to the quality judgment result of the inspection device, consider the case where the passing amount per minute of the product at the positions divided into a plurality of ranges on the production line as shown in FIG. 24 is selected. ..

The passing amount is measured by totaling the number of products that have passed the passing measurement line shown in FIG. 24 for each category.

In this case, the state information main body included in the state information is the number of products passed per minute for each category shown in FIG. 24.

The image processing unit 12 of the moving image / still image camera 10 measures the number of products passing through each category shown in FIG. 24 in 1-minute units, and uses the following contents as the state information main body.

(Number of passages in Category 1 and Category 1)
(Number of passages in Category 2 and Category 2)
(Number of passages in Category 3 and Category 3)
Generally, assuming that a large number of small products will flow, a threshold is set for the number of passages in each category, and in order to narrow down the image information to be recorded and saved, information is collected while changing the threshold for each category. And it is very difficult to find an association with the results of poor quality that rarely occur in inspection equipment.

On the other hand, in the system to which this embodiment is applied, a means is provided which enables verification of the relationship with the quality determination result while reducing the communication amount / recording amount. The procedure of processing by this means is shown in FIG.

First, the threshold value of the number of passages for each division in the image state index generation processing unit 21 of the processor 20 is set to a very large value, for example, a value slightly smaller than the total number of products flowing on the production line in one minute (S161). ).

As a result, in the information gathering carried out by photographing the first production line, the transmission of image information from each moving image / still image photographing device 10 is stopped in most cases, and the passing amount for each division which is the state information main body. Only get (S162).

In the system to which this embodiment is applied, image information is recorded even in an abnormal state in which the product flows intensively in a certain category, as compared with the case where the transmission of image information is simply stopped by setting the threshold value described above. You can save it and check the status later.

Subsequently, the relationship between the quality determination result of the inspection device and the number of passed products for each category in the recorded state information is verified (S163). For example, the display as shown in FIG. 26 determines whether or not the determination result in the inspection device is related to the number of items passing through each category. In FIG. 26, the symbols H1 and H2 indicate the determined points related to each of the quality determination results of the two inspection devices.

For example, when it becomes related to the increase in the passing amount of Category 1, the threshold value for Category 2 and Category 3 is set to the same value as above, and only the threshold value for Category 1 is recorded as shown in the first embodiment. Is taken into consideration (S164). As a result, in the second information collection (S165), the number of images passing through Category 1, which is suspected to be related to the quality judgment result of the inspection device, and the corresponding image information are collected, and more detailed verification is performed next. For example, it is possible to provide the information required for verification (S166) of the relationship between the quality determination result of the inspection device and the number of passed products of Category 1.

In the system to which this embodiment is applied as described above, it is possible to provide a means for easily narrowing down the image information to be recorded and to reduce the communication amount / recording amount.

(Fourth Example)
In the fourth embodiment, an example in which the image processing device is applied to an environment in which the communication line between the moving image / still image capturing device 10 and the processing unit 20 cannot be used for transferring image information will be shown.

For example, in an existing manufacturing line as shown in FIG. 17, when the existing communication line that can be used has a low speed and a small capacity, which hinders the transmission and reception of image information, or between a plurality of factories as shown in FIG. 23. It is assumed that it is difficult to add a new cost for sending and receiving image information to the communication line.

In this embodiment, the functional configurations in the moving image / still image capturing device 10 and the processor are as shown in FIG. 3, and the processing flow thereof is as shown in FIG.

Hereinafter, the difference from the first embodiment will be shown.

The operations of the image information acquisition processing unit 22, the image processing unit 12, and the image state index generation processing unit 21 are the same as those in the first embodiment.

The temporary recording processing unit 16 confirms the recording capacity of the moving image / still image camera 10 at an appropriate timing or when the image information is input from the image information generation processing unit 13, and the free space is a preset lower limit. If it is less than the value or insufficient for recording new image information, the image information whose shooting date and time is older and closer to unexpected is preferentially deleted based on the state index value included in the state index.

The image information reading processing unit 17 reads the recorded and saved image information from the temporary recording processing unit 16 by operating a switch provided in the moving image / still image camera 10, detecting a cable connection, or detecting the insertion of a recording medium. , Output to cable or write to recording media.

The image acquisition processing unit 22 acquires image information from the image information input from the connected cable or the recording medium, and records and saves the image information.

As a result, the system to which this embodiment is applied provides a means for inputting image information to the processor without depending on the communication line, enables verification of the relationship with the quality judgment result of the inspection device, and also enables the video / It is possible to preferentially obtain information of high importance according to the recording capacity of the still image photographer 10.

As described in detail above, according to the embodiment, it is possible to reduce the amount of information communication and the amount of recording.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1-3 ... Work place, 10 ... Movie / still image camera, 20 ... Processor, 30 ... Another processor, 40 ... Inspection device.

Claims (12)

In an image processing system composed of at least one imager having a function of photographing an object including a product and a processor having a function of acquiring and storing image information generated by the imager.
The camera
By performing image processing on the captured image, the state information of the product representing the state of the product, which is information having a smaller capacity than the captured image, is generated and sent to the processor, and the image information including the captured image is included. Is received from the processor and a state index indicating the degree of influence on the quality evaluation of the product, and is recorded on the predetermined recording medium based on the received state index. It has a means to select and delete unintentional image information.
The processor
A means for receiving product state information from the camera, generating the state index based on the received state information, and transmitting the state index to the camera, and an image recorded on the predetermined recording medium from the camera. It has a means to acquire information and record it on another recording medium.
Image processing system. The camera
A means for selecting significant image information recorded on the predetermined recording medium based on the received status index and sending it to the processor.
The image processing system according to claim 1. The imager includes means for outputting the image information recorded on the predetermined recording medium to a cable or writing the image information to another recording medium in response to a predetermined detection.
The image processing system according to claim 1. Each image information includes the main body of the captured image, information indicating the shooting date and time, information indicating the shooting position, information indicating the shooting parameters used at the time of shooting, and information indicating the identifier of the state information.
The image processing system according to any one of claims 1 to 3. The processor includes means for outputting image information acquired from the camera and recorded on a recording medium to the outside as information used for product quality evaluation.
The image processing system according to any one of claims 1 to 4. The processor includes means for recording image information acquired from the imager on a recording medium in an associated state together with corresponding state information and a state index.
The image processing system according to any one of claims 1 to 5. The imager is provided with a means for preferentially deleting the image information recorded on the predetermined recording medium from the oldest one when the storage capacity of the predetermined recording medium exceeds a certain value.
The image processing system according to any one of claims 1 to 6. When the storage capacity of the predetermined recording medium exceeds a certain value, the imager preferentially deletes the image information recorded on the predetermined recording medium from the one with the oldest shooting date and the one closer to the unexpected. Have the means to
The image processing system according to any one of claims 1 to 7. The camera
As the state information to be sent to the processor, it is provided with a means for applying any one or a plurality of combinations of a binarized image obtained from the captured image, an edge detection result, and a difference image from the reference image.
The image processing system according to any one of claims 1 to 8. The camera
As the state information to be sent to the processor, a means for applying any one or a plurality of combinations of the presence / absence of a predetermined graphic element, the inclination of the product from the predetermined position, and the detection result of the deviation is provided. Yes,
The image processing system according to any one of claims 1 to 8. The camera
The state information to be sent to the processor is classified into a predetermined intrusion detection result of an object in a single or a plurality of areas of the imaging range, a predetermined movement path of the product, or a deviation amount from a predetermined position. A means for applying any one or a combination of information on the passing position of the area is provided.
The image processing system according to any one of claims 1 to 8. It is applied to an image processing system composed of at least one imager having a function of photographing an object including a product and a processor having a function of acquiring and storing image information generated by the imager. Image processing method
By performing image processing on the captured image taken by the camera, the state information of the product representing the state of the product, which is information having a capacity smaller than that of the shot image, is generated and sent to the processor, and the shooting is performed. Performs processing to record image information including images on a predetermined recording medium,
The processor receives product status information from the camera, generates a status index indicating the degree of influence on the quality evaluation of the product based on the received status information, and sends it to the camera.
The camera receives the state index from the processor, and based on the received state index, performs a process of selecting and deleting unintentional image information recorded on the predetermined recording medium.
The processing unit acquires image information recorded on the predetermined recording medium from the photographing device and records it on another recording medium.
Image processing methods including that.