JP6987538B2 - Image processing equipment, work support systems, control methods and programs - Google Patents

Description

The present invention relates to an image processing device, a work support system, a control method and a program.

A remote work support system has been proposed in which a work supporter in a remote place provides remote support via a network to a worker who has little experience working in the field such as printer repair or construction. Patent Document 1 discloses the following remote support system. While the worker has a camera terminal with a shooting function and a display function, the worker sends a live view image of the work situation to the work support terminal owned by the work supporter, and the live view image displayed on the work support terminal is used for work support. Confirm by the person. Then, the work support terminal receives the instruction data instructed by the work supporter at a predetermined position in the live view image, and transmits the data in which the instruction data and the instruction position are associated with each other to the camera terminal. The camera terminal of the worker displays the received instruction data, so that the worker confirms the instruction content of the work supporter. Further, Patent Document 2 causes a work supporter's image sickness by limiting the switching of the image when displaying the image of the photographed subject when the image of the photographed subject is reciprocated by the camera attached by the operator. The device for mitigation is disclosed.

Japanese Unexamined Patent Publication No. 2011-34315 Japanese Unexamined Patent Publication No. 2011-109463 Japanese Unexamined Patent Publication No. 2007-06533 Japanese Unexamined Patent Publication No. 2010-74315 Japanese Unexamined Patent Publication No. 2007-166269

The remote support system disclosed in Patent Document 1 requires a display device that displays the content of instructions from the work supporter on the camera terminal on the worker side. In addition, the work support terminal needs a function to process the instruction contents of the work supporter, which increases the cost of constructing the remote work support system. Further, in the device disclosed in Patent Document 2, when the work supporter wants to observe the continuous operation state of the subject, if the worker's shooting is shaken, the switching of the image is restricted and the observation of the image becomes difficult. Become.

The present invention enables a work supporter to effectively support the work of the worker even if the worker does not have a display device, and improves the visibility of the image observed by the work supporter. The purpose is to provide an image processing device capable of performing.

The image processing apparatus according to the embodiment of the present invention has a recognition means for recognizing a subject used for designating a work object by analyzing a captured image, a tracking means for tracking the recognized subject, and the subject. The image is subjected to at least one of a setting means for setting a region of interest, which is a region to be subject to image correction processing, according to the work object, and at least one of the enlargement processing and the image blur correction processing of the captured image for the region of interest. A correction means performed as a correction process and a generation means for superimposing an image obtained by the image correction process on the captured image to generate a display image are provided.

According to the present invention, it is possible for a work supporter to effectively support the work of the worker even if the worker does not have a display device, and the visibility of the image observed by the work supporter is improved. Can be made to.

It is a figure which shows the system configuration of Example 1. FIG. It is a figure explaining the generation process of the display image by a camera terminal. It is a flowchart explaining the process of generating a display image by a camera terminal. It is a flowchart explaining the process of generating a display image by a camera terminal. It is a flowchart explaining the process of displaying an image by a work support terminal. It is a figure which shows the system configuration of Example 2. It is a figure explaining the process of transmitting a live view image to a work support terminal. It is a flowchart explaining the generation process of a display image. It is a flowchart explaining the generation process of a display image. It is a figure explaining the display image generated by a work support system. It is a figure explaining the display image generated by a work support system.

(Example 1)
FIG. 1 is a diagram showing a system configuration of the first embodiment.
The work support processing system shown in FIG. 1 includes a camera terminal 100 and a work support terminal 120. The camera terminal 100 is an example of an image processing device and is provided on the operator side. The camera terminal 100 is installed on, for example, a helmet or the like worn by a worker who works on a work object such as a printer. The work support terminal 120 is an information processing device operated by a work supporter who supports the work of the worker.

The camera terminal 100 includes an image pickup optical system 101, an image pickup element 102, an A / D conversion unit 103, a system control unit 104, a memory unit 105, a microphone / speaker 106, and an image processing unit 107. Further, the camera terminal 100 has a subject recognition unit 108, a subject tracking unit 109, a region of interest designation unit 117, a shaking amount detection unit 112, an image correction processing unit 113, a communication unit 114, a display image generation unit 115, and an internal bus 116. ..

The system control unit 104 controls the entire camera terminal 100. The system control unit 104 includes, for example, a programmable processor (hereinafter, CPU), a ROM, and a RAM. The CPU executes a program stored in the ROM and controls each part to realize the function of the camera terminal 100. The memory unit 105 is a volatile storage unit. The memory unit 105 functions as a buffer memory for temporarily storing image data, distance information, and the like based on a request from each block.

The image pickup optical system 101 has a lens, a shutter, and an aperture, and controls the focusing distance, the aperture amount of the aperture, and the opening / closing of the shutter based on the instruction of the system control unit 104. Based on the control of the system control unit 104, the image pickup element 102 photoelectrically converts the subject light passing through the image pickup optical system 101 to generate an analog signal of the subject image. Further, the image pickup device 102 includes an element having a distance measuring function. For example, as described in Patent Document 3, the luminous flux that has passed through different regions on the pupil of the lens of the image pickup optical system 101 is exposed to the image pickup element 102 in which the pixel having the distance measuring function is arranged. The system control unit 104 acquires the image shift amount of the exposed light image, and controls the distance image generation unit 110 to calculate the defocus amount.

The A / D conversion unit 103 generates a digital signal (hereinafter referred to as RAW data) by performing A / D conversion on the analog signal output from the image pickup device 102 based on the control of the system control unit 104. Then, it is stored in the memory unit 105. The microphone / speaker unit 106 is an acoustic input / output unit. The microphone / speaker unit 106 has a commonly used microphone device and speaker device, and has a function of collecting the contents spoken by the operator and surrounding sounds, and is collected by the microphone / speaker unit 123 of the work support terminal 120. It has a function to output the sounded sound.

Based on the control of the system control unit 104, the image processing unit 107 performs noise removal, gamma correction processing, white balance adjustment processing, and the like on the RAW data stored in the memory unit 105 to generate image data. Further, the image processing unit 107 performs coding processing and decoding processing as necessary, and stores the obtained image data in the memory 105. The image data is updated at a predetermined frame rate under the control of the system control unit 104. Further, the image data is transmitted to the work support terminal 120 via the network 130 and displayed on the display as a live view image.

Based on the control of the system control unit 104, the subject recognition unit 108 performs recognition processing of the main subject using the template image on the image data stored in the memory unit 105. The template image is image data of a subject previously photographed by a camera terminal 100 or the like. The template image is stored in a non-volatile memory (not shown). In this embodiment, the template image is an image of a subject (hereinafter referred to as a designated subject) used for designating an object (working object), and specifically, it is an image of a worker's finger.

The subject recognition unit 108 recognizes a designated subject by performing a matching process for analyzing the subject in the image data based on the template image. The subject recognition unit 108 calculates the degree of similarity by comparing the feature amount of the template image with the feature amount of the subject in the image data specified by the analysis during the matching process. The subject recognition unit 108 identifies a designated subject based on the degree of similarity. Then, the subject recognition unit 108 targets the specified designated subject as a tracking target, and supplies information indicating the similarity and position of the designated subject (hereinafter referred to as tracking target information) to the subject tracking unit 109.

The subject tracking unit 109 performs the following processing based on the control of the system control unit 104. The subject tracking unit 109 acquires tracking target information from the subject recognition unit 108 and performs tracking processing for the tracking target existing in the image data. The subject tracking unit 109 calculates the degree of correlation in the subject region specified by the tracking target information between the image frames by using, for example, the technique disclosed in Patent Document 4. The subject tracking unit 109 detects a subject region that has a tracking relationship with the subject region of the previous frame from the current frame, and calculates a motion vector thereof. Then, the subject tracking unit 109 calculates the coordinates of moving the subject area by the amount of the motion vector, and corrects the position information of the tracking target. The subject tracking unit 109 passes the position information of the tracking target to the subject distance determination unit 111.

The area of interest setting unit 117 sets an area of interest, which is an area to be image-corrected by the image correction processing unit 113, according to the distance between the designated subject and the work object. The region of interest setting unit 117 includes a distance image generation unit 110 and a subject distance determination unit 111.

The distance image generation unit 110 calculates the defocus amount for each subject area from the image shift amount obtained by the image pickup device 102 based on the control of the system control unit 104. The distance image generation unit 110 generates a distance image showing the distance in the depth direction for each area based on the calculated defocus amount, and stores it in the memory unit 105. The subject distance determination unit 111 performs the following processing based on the control of the system control unit 104. The subject distance determination unit 111 calculates the distance between the designated subject and the work object (hereinafter, distance information) based on the position information of the designated subject and the distance image stored in the memory unit 105. The subject distance determination unit 111 stores the calculated distance information in the memory unit 105. Further, the subject distance determination unit 111 stores the position information of the designated subject (hereinafter, the position information of interest) when the distance information is within the predetermined range in the memory 105.

The shaking amount detection unit 112 executes the following processing based on the control of the system control unit 104. The shaking amount detection unit 112 detects the shaking amount (camera shake) of the camera terminal 100 by using the angular velocity sensor, and stores the shaking amount information in the memory unit 105. In this embodiment, the amount of shaking is detected using a commonly used angular velocity sensor, but a method of comparing input images between two consecutive frames and detecting the motion vector on the image as the amount of shaking is applied. You may.

The image correction processing unit 113 performs image correction processing on the image data stored in the memory unit 105 based on the control of the system control unit 104. When the image correction processing unit 113 performs the crop enlargement processing, the image correction processing unit 113 crops (cuts out) a predetermined crop area centering on the attention position information stored in the memory unit 105 with respect to the image data, and has a predetermined size. Performs crop enlargement processing. The crop area is determined according to the preset enlargement ratio and the enlarged display angle of view size. Further, the image correction processing unit 113 may perform image shake correction processing (electronic vibration isolation processing) on the cropped image (crop image) based on the shaking amount information stored in the memory unit 105. For example, the image correction processing unit 113 realizes the electronic vibration isolation processing by performing the coordinate conversion processing using the technique described in Patent Document 5. The image correction processing unit 113 stores the corrected image generated by the image correction processing in the memory 105. The image correction processing unit 113 may execute at least one of the crop enlargement processing and the image blur correction processing.

The display image generation unit 115 superimposes the correction image generated by the image correction processing unit 113 on the image (captured image) stored in the memory unit 105 to generate a display image and stores it in the memory 105. The communication unit 114 transmits / receives data to / from the work support terminal 120 via the network 130 under the control of the system control unit 104. The network 130 connects a plurality of terminals so that they can communicate with each other. The network is, for example, WAN (Wide Area Network).

The work support terminal 120 has a system control unit 121, a memory unit 122, a microphone / speaker 123, a communication unit 124, a display unit 125, an operation unit 126, and an internal bus 127. The system control unit 121 controls the entire work support terminal 120. The system control unit 121 includes, for example, a programmable processor (hereinafter, CPU), a ROM, and a RAM. The CPU executes the program stored in the ROM and controls each part to realize the function of the work support terminal 120.

The memory unit 122 is a volatile storage unit. The memory unit 122 functions as a buffer memory for temporarily storing image data and the like based on a request from each block. The microphone / speaker unit 123 is an acoustic input / output unit. The microphone / speaker unit 123 has a commonly used microphone device and speaker device, and has a function of collecting the contents spoken by a work supporter and surrounding sounds, and the microphone / speaker unit 106 of the camera terminal 100. It has a function to output the sounded sound.

The communication unit 124 transmits / receives data to / from the camera terminal 100 via the network 130 under the control of the system control unit 121. The display unit 125 has a display and a display controller, and based on the control of the system control unit 121, reads out the contents of the display frame memory stored in the memory unit 122 and displays it on the display.

The operation unit 126 is an input device device such as a keyboard and a mouse that receives an operation instruction from a work supporter. The work support terminal 120 may be realized by downloading a dedicated application to a smartphone, tablet, personal computer, or the like.

FIG. 2 is a diagram illustrating a process of generating a display image by a camera terminal.
FIG. 2A shows a live view image 200 taken by the camera terminal 100, which is displayed on the display unit 125 of the work support terminal 120. The work supporter communicates with the worker through the microphone / speaker unit 123 while observing the live view image 200. FIG. 2B shows a live view image 210 in a state where the operator brings his finger as the designated subject 211 closer to the object 212 in order to specify a portion to be confirmed by the work supporter.

FIG. 2C shows a distance image 220 generated by the distance image generation unit 110. The subject distance determination unit 111 detects that the distance between the designated subject 221 and the object 222 is within a predetermined range, and obtains the position of interest 223. FIG. 2D shows a display image 230 generated by the display image generation unit 115. The display image 230 is generated by the image correction processing unit 113 superimposing the corrected image 231 electronically vibration-proof and cropped around the position of interest on the image data.

3 and 4 are flowcharts illustrating a process by which the camera terminal generates a display image.
In S301, the camera terminal 100 captures the operator's finger as a designated subject for designating an object and stores it as a template image. In S302, the system control unit 104 performs an imaging process. Specifically, the system control unit 104 controls the image pickup optical system 101, the image pickup element 102, the A / D conversion unit 103, and the image processing unit 107 to generate image data related to the image pickup image at a predetermined frame rate. It is stored in the memory unit 105.

In S314, the system control unit 104 determines whether or not the attention position information is calculated by the subject distance determination unit 111. If the position information of interest has been calculated, the process proceeds to S309 in FIG. If the focus position information has not been calculated, the process proceeds to S315. In S315, the system control unit 104 transfers the image data generated in S302 to the communication unit 114. The communication unit 114 transmits the image data to the communication unit 124 of the work support terminal 120 via the network 130.

In S303, the subject recognition unit 108 analyzes the image data generated by 302 to recognize the designated subject. Specifically, the subject recognition unit 108 performs matching processing on the image data using the template image acquired in S301. As a result, the designated subject is specified and the tracking target information is generated. Subsequently, in S304, the subject tracking unit 109 performs tracking processing of the finger of the operator who is the designated subject based on the tracking target information generated in S303. In S305, the subject tracking unit 109 determines whether or not the designated subject is in the image data based on the result of the tracking process. If the designated subject is in the image data, the process proceeds to S306 in FIG. If the designated subject is not in the image data, the process returns to S302.

Next, in S306, the distance image generation unit 110 calculates the defocus amount for each subject area based on the image shift amount of the captured image, and generates a distance image for each area. For example, the distance image 220 shown in FIG. 2C is generated. Subsequently, in S307, the subject distance determination unit 111 determines whether the distance between the designated subject and the object is within a predetermined range based on the tracking processing result in S304 and the distance image 220 generated in S306. If the distance between the designated subject and the object is not within the predetermined range, the process returns to S302 in FIG. When the distance between the designated subject and the object is within the predetermined range, the process proceeds to S308.

In S308, the system control unit 104 acquires the coordinate position of the designated subject in the image data when the distance between the designated subject and the object is within the predetermined range as the position of interest information. Subsequently, in S309, the shaking amount detection unit 112 detects the shaking amount of the camera terminal 100.

Next, in S310, the image correction processing unit 113 performs crop enlargement processing on the image data generated in S302. Specifically, the image correction processing unit 113 cuts out and enlarges a predetermined crop region centered on the position information of interest. In addition, the image correction processing unit 113 also executes image blur correction processing when performing crop enlargement. As a result, for example, the corrected image 231 shown in FIG. 2D is generated. In this embodiment, both the crop enlargement process and the image blur correction process are performed, but only one of them may be performed.

In S311 the display image generation unit 115 superimposes the corrected image generated in S310 on the image data generated in S302 to generate a display image. For example, the display image 230 shown in FIG. 2D is generated. Subsequently, in S312, the system control unit 104 transfers the display image generated in S311 to the communication unit 114. The displayed image is transmitted by the communication unit 114 to the communication unit 124 of the work support terminal 120 via the network 130.

Next, in S312, it is determined whether the system control unit 104 has received an instruction to end the process. When the system control unit 104 receives an instruction to end the process, the process ends. If the system control unit 104 has not received an instruction to end the process, the process returns to S302 in FIG.

FIG. 5 is a flowchart illustrating a process in which the work support terminal displays an image transmitted from the camera terminal.
In S401, the communication unit 124 receives the live view image (captured image) generated in S302 of FIG. 3 or the display image generated in S311 of FIG. 4 from the camera terminal 100. The live view image and the display image are transmitted at a predetermined frame rate. The system control unit 121 stores the received image in the memory unit 122. In S402, the display unit 125 displays the live view image or the display image stored in the memory unit 122 on the display at a predetermined frame rate. According to the work support system of the first embodiment, it is possible to easily generate a display image in which an object that the worker wants to pay attention to is emphasized. As a result, even if the camera terminal 100 does not have a display device, it is possible to accurately notify the operator of the subject of interest by communicating only with the work supporter by voice. Further, since the work support system of the first embodiment enlarges the image of the area of the subject of interest, the visibility of the work supporter who observes this image can be improved.

(Example 2)
FIG. 6 is a diagram showing a system configuration of the second embodiment.
Among the components of the work support system shown in FIG. 6, those having the same functions as the components of the work support system shown in FIG. 1 are indicated by the same reference numerals. In this embodiment, the work support terminal 520 functions as an image processing device that generates a display image.

The camera terminal 500 includes an image pickup optical system 101, an image pickup element 102, an A / D conversion unit 103, a system control unit 104, a memory unit 105, a microphone / speaker 106, an image processing unit 107, a shaking amount detection unit 112, and a communication unit 114. It has an internal bus 116.

The work support terminal 520 has a system control unit 121, a memory unit 122, a microphone / speaker 123, a communication unit 124, a display unit 125, an operation unit 126, an internal bus 127, and a subject recognition unit 108. Further, the work support terminal 520 has a subject tracking unit 109, an image correction processing unit 113, a display image generation unit 115, and a region of interest designation unit 117. The subject recognition unit 108, the subject tracking unit 109, and the image correction processing unit 113 have the same functions as the subject recognition unit 108, the subject tracking unit 109, and the image correction processing unit 113 included in the camera terminal 100 of FIG. Further, the display image generation unit 115 and the attention area designation unit 117 have the same functions as the display image generation unit 115 and the attention area designation unit 117 of the camera terminal 100 of FIG.

FIG. 7 is a flowchart illustrating a process in which the camera terminal transmits a live view image and various data to the work support terminal.
In S601, the camera terminal 500 takes a picture of the operator's finger in advance as a designated subject used for designating an object, and stores it as a template image. In S602, the system control unit 104 transfers the template image of the designated subject taken in S601 to the communication unit 114, and transmits the template image to the communication unit 124 of the work support terminal 520 via the network 130.

In S603, the system control unit 104 transmits the image data (live view image) generated by performing the image pickup process to the work support terminal 520. In S604, the system control unit 104 acquires the image shift amount. Then, the system control unit 104 transmits the image shift amount to the communication unit 124 of the work support terminal 520 via the network 130.

Next, in S605, the shaking amount detection unit 112 detects the shaking amount of the camera terminal 500. Subsequently, in S606, the system control unit 104 transfers the amount of shaking to the communication unit 114 and transmits it to the communication unit 124 of the work support terminal 520 via the network 130. In S607, it is determined whether the system control unit 104 has received an instruction to end the process. When the system control unit 104 receives an instruction to end the process, the process ends. If the system control unit 104 has not received an instruction to end the process, the process returns to S603.

8 and 9 are flowcharts illustrating a display image generation process by the work support terminal.
In S701, the communication unit 124 receives the template image of the designated subject taken in S601 of FIG. 7 from the camera terminal 500. The system control unit 121 stores the received template image in the memory unit 122. Subsequently, in S702, the communication unit 124 receives the image data generated in S603 of FIG. 7 from the camera terminal 500. Then, the system control unit 121 stores the received image data in the memory unit 122.

In S716, the system control unit 121 determines whether the focus position information has been calculated by the subject distance determination unit 111. When the attention position information is calculated, the process proceeds to S711 in FIG. If the focus position information has not been calculated, the process proceeds to S703. In S703, the display unit 125 displays the image data stored in the memory unit 122 in S702 on the display at a predetermined frame rate. In S704, the communication unit 124 receives the image shift amount transmitted from the camera terminal 500 at a predetermined frame rate. Then, the system control unit 121 stores the received image deviation amount in the memory unit 122.

In S705, the subject recognition unit 108 performs matching processing on the live view image received in S702 based on the template image acquired in S701, and generates tracking target information. In S706, the subject tracking unit 109 performs tracking processing of the finger of the operator who is the designated subject based on the tracking target information generated in S705. Subsequently, in S707, the subject tracking unit 109 determines whether or not the designated subject is in the image data based on the result of the tracking process. If the designated subject is in the image data, the process proceeds to S708 in FIG. If the designated subject is not in the image data, the process returns to S702.

In S708 of FIG. 9, the distance image generation unit 110 calculates the defocus amount for each subject area based on the image shift amount received in S704 of FIG. 8, and generates a distance image for each subject area. Subsequently, in S709, whether the subject distance determination unit 111 has within the predetermined range of the distance between the designated subject and the object based on the tracking processing result of the designated subject in S706 and the distance image 220 generated in S708. To judge. When the distance between the designated subject and the object is within the predetermined range, the process proceeds to S710. If the distance between the designated subject and the object is not within the predetermined range, the process returns to S702 in FIG.

In S710, the system control unit 121 acquires the coordinate position of the designated subject when the distance between the designated subject and the object is within the predetermined range as the position of interest information. Subsequently, in S711, the communication unit 124 receives the amount of shaking transmitted from the camera terminal 500. Then, the system control unit 121 stores the received shaking amount in the memory unit 122.

Next, in S712, the image correction processing unit 113 performs crop enlargement processing on the image data received in S702. Specifically, the image correction processing unit 113 cuts out and enlarges a predetermined crop region centered on the position information of interest. In addition, the image correction processing unit 113 also executes image blur correction processing when performing crop enlargement. As a result, a corrected image is generated. In this embodiment, both the crop enlargement process and the image blur correction process are performed, but only one of them may be performed.

Next, in S713, the display image generation unit 115 superimposes the corrected image generated in S712 on the image data received in S702 to generate a display image. Subsequently, in S714, the display unit 125 displays the display image generated in S713 on the display at a predetermined frame rate.

In S715, it is determined whether the system control unit 121 has received an instruction to end the process. When the system control unit 121 receives an instruction to end the process, the process ends. If the system control unit 121 has not received an instruction to end the process, the process returns to S702 in FIG. According to the work support system of the second embodiment, the display image generation process is performed by the work support terminal 520 instead of the camera terminal 500, so that the weight of the camera terminal can be reduced and the cost of the entire system can be reduced. ..

(Example 3)
The work support system of the third embodiment can change the enlargement ratio of the corrected image. The configuration and processing flow of the work support system of the third embodiment are the same as those of the work support system of the first embodiment. In the work support system of the third embodiment, the image correction processing unit 113 included in the camera terminal 100 changes the crop area to be cropped. Specifically, the image correction processing unit 113 changes the crop area according to the distance information between the designated subject and the object calculated by the subject distance determination unit 111 stored in the memory unit 122, and changes the crop area to a predetermined size. Expand to. For example, the image correction processing unit reduces the crop area and increases the enlargement ratio as the distance between the designated subject and the object becomes closer (smaller).

FIG. 10 is a diagram illustrating a display image generated by the work support system of the third embodiment.
FIG. 10A shows a display image 800 generated by the display image generation unit 115. The display image generation unit 115 superimposes the corrected image 801 cropped and enlarged around the position of interest on the image data by the image correction processing unit 113. As a result, the display image 800 is generated. Since the distance between the designated subject and the object is large, the crop area is large and the enlargement ratio of the corrected image 801 is small.

FIG. 10B shows a display image 810 generated when the distance between the designated subject and the object is smaller than that in the state shown in FIG. 10A. The crop area is smaller than in the state shown in FIG. 10 (A), and the enlargement ratio of the corrected image 811 is large. FIG. 10C is a display image 820 generated when the distance between the designated subject and the object is smaller than that in the state shown in FIG. 10B. The crop area is smaller than in the state shown in FIG. 10B, and the enlargement ratio of the corrected image 821 is further increased.

The work support system of the third embodiment makes the crop area variable when generating a corrected image according to the distance between the designated subject and the object. As a result, it is possible to generate a display image of a focus position having a small magnification in a wide range and a focus position having a large magnification in a narrow range. Therefore, since the flexibility of the display image transferred from the camera terminal is increased, the worker can communicate more smoothly with the work supporter. It should be noted that Example 3 can also be applied to the configuration of Example 2 (FIG. 6). That is, the image correction processing unit 113 included in the work support terminal 520 may change the enlargement ratio of the image according to the distance information between the designated subject and the object.

(Example 4)
The work support system of the fourth embodiment generates a high-quality corrected image. The configuration and processing flow of the work support system of the fourth embodiment are the same as those of the work support system of the first embodiment. In the work support system of the fourth embodiment, the system control unit 104 switches the drive for reading out the signal from the image pickup element 102. When the region of interest is set, the system control unit 104 controls the image sensor 102 to cut out and read a predetermined crop region around the region of interest (crop readout).

When generating a live view image, the system control unit 104 thins out the image pickup element 102 at regular intervals and reads it out so that the camera terminal 100 can process it within a predetermined frame rate. When reading the crop area, the system control unit 104 reads the image sensor 102 without thinning out. The number of pixels in the crop area is the same as the number of pixels read out by thinning out when generating the live view image.

FIG. 11 is a diagram illustrating a display image generated by the work support system of the third embodiment.
FIG. 11A shows image data generated by the image processing unit 107 before the position information of interest is determined. FIG. 11B is image data in a state where the position information of interest is fixed and the region to be cropped and read by the image sensor 102 is fixed. In the state shown in FIG. 11B, the focus position 911 is calculated by the subject distance determination unit 111. The region 912 indicates a region to be cropped and read by the image pickup device 102 centering on the position of interest. FIG. 11C shows image data 920 generated by the image processing unit 107 processing the image obtained by crop readout. FIG. 11D is a display image 930 generated by the display image generation unit 115. The display image 930 is an image in which the corrected image 931 that has been cropped and image-blurred corrected around the position of interest by the image correction processing unit 113 is superimposed on the image data 920. According to the work support system of the fourth embodiment, the image quality of the corrected image displaying the position of interest can be improved by switching the readout of the image pickup device to the crop readout.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Camera terminal 120 Work support terminal

Claims (13)

A recognition means that analyzes the captured image and recognizes the subject used to specify the work object,
The tracking means for tracking the recognized subject and
A setting means for setting an area of interest, which is an area to be subject to image correction processing, according to the subject and the work object, and
A correction means that performs at least one of the image enlargement processing and the image blur correction processing for the region of interest as the image correction processing.
An image obtained by the image correction process is superimposed on the captured image to generate a display image, and a generation means.
An image processing device characterized by being equipped with. The image processing apparatus according to claim 1, wherein the correction means changes the enlargement ratio according to the distance between the subject and the work object when enlarging the captured image. The image processing apparatus according to claim 2, wherein the correction means increases the enlargement ratio as the distance between the subject and the work object becomes smaller. The image processing apparatus according to any one of claims 1 to 3, wherein the correction means performs crop enlargement by cutting out and enlarging a crop region as the enlargement processing of the captured image. The image processing apparatus according to any one of claims 1 to 4 , wherein the information processing apparatus includes a transmission means for transmitting a display image generated by the generation means. The image processing apparatus according to any one of claims 1 to 5 , further comprising an image pickup device that photoelectrically converts subject light and outputs a signal related to the captured image. A recognition means that analyzes the captured image and recognizes the subject used to specify the work object,
The tracking means for tracking the recognized subject and
A setting means for setting an area of interest, which is an area to be subject to image correction processing, according to the subject and the work object, and
A correction means that performs at least one of the image enlargement processing and the image blur correction processing for the region of interest as the image correction processing.
A generation means for generating a display image based on the image obtained by the image correction process, and
An image sensor that photoelectrically converts subject light and outputs a signal related to the captured image,
Equipped with
When said region of interest is set, images processor you comprising a control means for switching the reading operation of the signals according to the captured image from the imaging device. A receiving means for receiving the captured image from the image pickup device, and
The image processing apparatus according to any one of claims 1 to 4 , further comprising a captured image and a display means for displaying the displayed image. A system having an image processing device and an information processing device that communicates with the image processing device.
The image processing device is
A transmission means for transmitting an captured image to the information processing device, and
A recognition means for recognizing a subject used for designating a work object by analyzing the captured image.
The tracking means for tracking the recognized subject and
A setting means for setting an area of interest, which is an area to be subject to image correction processing, according to the subject and the work object, and
A correction means that performs at least one of the enlargement processing and the image blur correction processing of the captured image as the image correction processing for the region of interest.
A generation means for generating a display image based on the image obtained by the image correction process is provided.
The transmission means transmits the display image to the information processing apparatus, and the transmission means transmits the display image to the information processing apparatus.
The information processing device is
A work support system including the captured image received from the image processing device and a display means for displaying the displayed image. A system having an image processing device and an information processing device that communicates with the image processing device.
The image processing device is
A transmission means for transmitting an captured image to the information processing device is provided.
The information processing device is
A recognition means for recognizing a subject used for designating a work object by analyzing an captured image received from the image processing device.
The tracking means for tracking the recognized subject and
A setting means for setting an area of interest, which is an area to be subject to image correction processing, according to the subject and the work object, and
A correction means that performs at least one of the image enlargement processing and the image blur correction processing for the region of interest as the image correction processing.
A generation means for generating a display image based on the image obtained by the image correction process, and
A work support system including the captured image and a display means for displaying the displayed image. The process of analyzing the captured image and recognizing the subject used to specify the work object,
The process of tracking the recognized subject and
A step of setting a region of interest, which is an region to be subject to image correction processing, according to the subject and the work object, and a step of setting the region of interest.
A step of performing at least one of the image enlargement processing and the image blur correction processing for the region of interest as the image correction processing, and
A control method for an image processing apparatus, comprising: superimposing an image obtained by the image correction process on the captured image to generate a display image. The process of recognizing the subject used to specify the work object by analyzing the captured image output by the image sensor by photoelectric conversion of the subject light.
The process of tracking the recognized subject and
A step of setting a region of interest, which is an region to be subject to image correction processing, according to the subject and the work object, and a step of setting the region of interest.
When the region of interest is set, the step of switching the reading drive of the signal related to the captured image from the image pickup element and the step of switching.
A step of performing at least one of the image enlargement processing and the image blur correction processing for the region of interest as the image correction processing, and
A step of generating a display image based on the image obtained by the image correction process, and
Have
A control method for an image processing device, characterized in that. A program characterized in that a computer functions as each means provided in the image processing apparatus according to any one of claims 1 to 8.

Images