JP6607243B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In recent years, image recognition technology that applies machine learning technology called deep learning has been developed to detect what the object (category) appears in the image and where the position (coordinates in the image) is. Many methods have been proposed.

  In order to construct an object detector using the deep learning technology, a large amount of images and teacher data (annotation data indicating what category of object is present in the image) are required.

  Also, the quality of the teacher data needs to be ensured. If the quality of the teacher data is low, learning is not performed correctly, and an object detector with low detection accuracy is constructed.

  Therefore, it is very important to label the object to be detected in one image without omission (specify the area where the object to be detected is reflected and what the object is). It has become.

  Further, when the size of the object to be detected is smaller than the image resolution, it is necessary to perform a labeling operation while enlarging and displaying each area of the image.

JP 2006-197055 A

  Patent Document 1 describes a function that enables electronic zooming while checking an entire image together with an enlarged image.

  Just like displaying the entire image and the enlarged image as in the function described in Patent Document 1, it is not possible to distinguish between an area that has been enlarged and an area that has never been enlarged. The possibility of omission cannot be determined.

  Therefore, an object of the present invention is to provide a mechanism for efficiently confirming whether there is any omission in labeling of a detection object.

  An information processing apparatus according to the present invention includes a receiving unit that receives a designation of a region of a detection target object included in an image and a designation of a label for the detection target object, a display magnification change receiving unit that receives a change in display magnification of the image, Display means for displaying the image at a display magnification changed by the display magnification change accepting means, and the display means is capable of distinguishing an enlarged display area and an undisplayed area in the image. It is characterized by displaying.

  According to the present invention, it is possible to efficiently check whether there is any omission in the labeling of the detection object.

The figure which shows an example of a structure of the information processing system in this invention The block diagram which shows an example of the hardware constitutions of the information processing apparatus (client terminal 101, server apparatus 102) of this invention Flowchart showing display rectangle creation / update processing The flowchart which shows the process which masks the bird's-eye view image The flowchart which shows the process which masks the bird's-eye view image The flowchart which shows the process which masks the bird's-eye view image A flowchart showing message display processing when an instruction to switch the target image is given The figure which shows an example of the data table in which various data were registered The figure which shows an example of the screen which performs labeling work The figure which shows an example of the screen which enlargedly displayed the image The figure which shows an example of the screen where the image by which the mask process was displayed is displayed The figure which shows an example of the screen where the message was displayed

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing an example of the configuration of an information processing system according to the present invention.

  As shown in FIG. 1, the information processing system according to the present invention has a configuration in which a client terminal 101 and a server device 102 are communicably connected.

  The client terminal 101 is a terminal used by a user who performs a labeling operation on a detected object, displays a bird's-eye view image, an enlarged image, and the like, and accepts a labeling instruction. The labeling operation is an operation for receiving designation of an area in which a detected object is shown in the displayed image and designation of what the detected object is. For example, the screen shown in FIG. 9 shows an example in which the face of a person shown in the image is labeled as male or female. An area corresponding to a human face is enclosed by a rectangle, and the face in the rectangle is labeled as male or female. At this time, since the face shown in a small size cannot be determined as it is as it is, it is enlarged and labeled (FIG. 10).

  The server apparatus 102 performs a process of determining a confirmation OK / confirmation NG / unconfirmed area, which will be described later, and generating a mask image for identifying each area.

  Details of processing executed by the client terminal 101 and the server apparatus 102 will be described later with reference to FIG.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus (client terminal 101, server apparatus 102) according to the present invention.

  As shown in FIG. 2, the information processing apparatus includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an input controller 205, a video controller 206, a system bus 204, and the like. A memory controller 207 and a communication I / F controller 208 are connected.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  The ROM 202 or the external memory 211 includes a BIOS (Basic Input / Output System) and an OS (Operating System) that are control programs executed by the CPU 201, and a computer-readable program and various necessary programs for realizing the information processing method. Holds data (including data table).

  The RAM 203 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for executing the processing from the ROM 202 or the external memory 211 to the RAM 203 and executing the loaded program.

  The input controller 205 controls input from an input device such as a keyboard 209 or a pointing device such as a mouse (not shown). When the input device is a touch panel, the user can perform various instructions by pressing (touching with a finger or the like) in accordance with an icon, a cursor, or a button displayed on the touch panel.

  The touch panel may be a touch panel capable of detecting a position touched with a plurality of fingers, such as a multi-touch screen.

  The video controller 206 controls display on an external output device such as the display 210. The display includes a display of a notebook computer integrated with the main body. The external output device is not limited to a display, and may be a projector, for example. An input device is also provided for the device that can accept the touch operation described above.

  Note that the video controller 206 can control a video memory (VRAM) for display control, and a part of the RAM 203 can be used as a video memory area, or a dedicated video memory can be provided separately. Is possible.

  The memory controller 207 controls access to the external memory 211. The external memory is connected via an adapter to an external storage device (hard disk), flexible disk (FD), or PCMCIA card slot that stores boot programs, various applications, font data, user files, editing files, and various data. A compact flash (registered trademark) memory or the like can be used.

  The communication I / F controller 209 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and communication using 3G lines of mobile phones are possible.

  Note that the CPU 201 enables display on the display 210 by executing outline font rasterization processing on a display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction with a mouse cursor (not shown) on the display 210.

  Next, display rectangle creation / update processing will be described with reference to the flowchart of FIG.

  In step S301, the CPU 201 of the client terminal 101 determines whether an instruction to change the enlargement magnification (display magnification) is received from the user on the screen shown in FIG. 9, and if the change instruction is received, the process proceeds to step S302. To do. The change of the enlargement factor can be accepted by operating the slider displayed in FIG. FIG. 10 is a diagram illustrating an example of a screen when enlarged display is performed.

  In step S302, the CPU 201 of the server apparatus 102 newly creates a display rectangle. The display rectangle is information for managing which region (coordinate) of which image (image path) is displayed at what magnification (enlargement magnification) as shown in the data table of “display rectangle” in FIG. It is. The display rectangle is newly created when the enlargement magnification is changed (the process described in steps S301 to S304). When the display area is changed at the same magnification, the coordinate information is updated (step S305). Processing described in S308).

  In step S <b> 303, the CPU 201 of the server apparatus 102 transmits the display rectangle created in step S <b> 302 to the client terminal 101.

  In step S304, the CPU 201 of the client terminal 101 receives the display rectangle transmitted in step S303.

  Next, display rectangle update processing will be described.

  In step S305, the CPU 201 of the client terminal 101 determines whether or not the display coordinates have been changed on the screen shown in FIG. 9, and if the change has been made, the process proceeds to step S306. The change of display coordinates can be accepted by an operation with a mouse or a keyboard.

  In step S306, the CPU 201 of the server apparatus 102 updates the coordinate information of the display rectangle.

  In step S307, the CPU 201 of the server apparatus 102 transmits the display rectangle updated in step S307 to the client terminal 101.

  In step S307, the CPU 201 of the client terminal 101 receives the display rectangle transmitted in step S307.

  Next, using the flowcharts of FIGS. 4 to 6, a process for displaying the areas determined to be confirmed and the areas determined to be unconfirmed in the overhead image in an identifiable manner will be described.

  Steps S401 and S404 to S406 in FIG. 4 are processes executed by the CPU 201 of the client terminal 101, and steps S402 (FIGS. 5 and 6) and S403 are processes executed by the CPU 201 of the server apparatus 102.

  In step S401, it is determined whether or not the bird's-eye view image display button shown in FIG. 9 is pressed. If the button is pressed, the process proceeds to step S402.

  In step S402, a mask image for displaying an overhead image is created. Details of the processing in this step will be described with reference to FIGS.

  In step S501 of FIG. 5, a display rectangle group corresponding to the target image is acquired. For example, when the target image is “1.jpg”, three display rectangles with IDs 1 to 3 whose image path is “/image/1.jpg” are acquired from the display rectangles shown in FIG. The

  In step S502, a detected object rectangle group corresponding to the target image is acquired. Among the detected object rectangles shown in FIG. 8, three of IDs 1 to 3 whose image path is “/image/1.jpg” are acquired.

  In step S503, it is determined whether the detected object rectangle has been acquired as a result of the process in step S502.

  If it can be obtained (step S503: YES), the process proceeds to step S504. If it cannot be obtained (step S503: NO), the process proceeds to step S518.

  In addition, the case where it cannot acquire is considered when the labeling operation | work is not performed in the target image, or the case where the detection object is not contained in the target image.

  Thereafter, the processes of steps S505 to S517 are repeatedly executed for each display rectangle (S504), and the processes of steps S506 to S509 are repeatedly executed for each detected object rectangle (S505).

  In step S506, an IoU (Intersection-over-Union) between the display rectangle and the detected object rectangle is calculated.

  In step S507, a preset IoU setting value is acquired. In this embodiment, it is assumed that “0.5” is set as shown in FIG.

  In step S508, the IoU value calculated in step S506 is compared with the IoU setting value acquired in step S507. If the IoU value calculated in step S506 is larger (that is, the display rectangle and the detected object rectangle If the overlap is greater than a preset value) (step S508: YES), the process proceeds to step S509. If the IoU set value is larger, the process proceeds to the next detected object rectangle.

  In step S509, the detected object rectangle to be processed is adopted as the representative size calculation rectangle.

  When the processes of S506 to S509 are executed for all the detected object rectangles acquired in step S502, the process proceeds to step S510.

  In step S510, it is determined whether there is a detected object rectangle adopted as the representative size calculation rectangle in S509. That is, it is determined whether there is a detected object rectangle that overlaps the display rectangle more than the IoU set value.

  If there is any (step S510: YES), the process proceeds to step S511.

  If not (NO in step S510), the process proceeds to step S517, and the display rectangle to be processed is managed as a check pending display rectangle.

  In step S511, a preset representative size calculation method is acquired. In this embodiment, the average is used as shown in FIG. In addition to the average, a method using minimum, maximum, median, etc. can be considered.

  In step S512, the representative size is calculated from the representative size calculation rectangle group. In the present embodiment, the average value of the size (area) of the detection object rectangle employed in S509 is calculated. Specifically, the apparent size on the screen is calculated by multiplying the average size of the representative size calculation rectangle (the true size on the image) by the magnification of the display rectangle.

  In step S513, a preset reference determination size is acquired. In the present embodiment, as shown in FIG.

  In step S514, the representative size (appearance size on the screen) calculated in step S512 is compared with the reference determination size acquired in step 513.

  If the representative size calculated in step S512 is larger (S514: YES), the process proceeds to step S515, and the display rectangle to be processed is adopted as the confirmation OK display area.

  If the reference determination size is larger (S514: NO), the process proceeds to step S516, and the display rectangle to be processed is adopted as the confirmation NG display area.

  Then, the process returns to step S504, and the process for the next display rectangle is executed. When the process for all display rectangles is completed, the process proceeds to step S601 in FIG.

  Next, processing when it is determined in S503 that there is no detected object rectangle will be described.

  In step S518, a preset reference magnification setting value is acquired. In this embodiment, it is doubled as shown in FIG.

  And the process of S520-S522 is repeatedly performed for every display rectangle.

  In step S520, the reference magnification setting value acquired in step S518 is compared with the enlargement magnification of the display rectangle to be processed.

  If the enlargement ratio of the display rectangle is larger (S520: YES), the process proceeds to step S521, and the display rectangle to be processed is adopted as the confirmation OK display area.

  If the reference magnification setting value is larger (S520: NO), the process proceeds to step S522, and the display rectangle to be processed is adopted as the confirmation NG display area.

  When the process for all display rectangles is completed, the process proceeds to step S601 in FIG.

  In step S601 in FIG. 6, the processing of S602 to S607 is executed for each display rectangle that has been set as the check pending display rectangle in step S517.

  In step S602, it is determined whether or not there is a display rectangle adopted as the confirmation OK display area in the process of step S515.

  If it exists (S602: YES), the process proceeds to step S603, and if it does not exist (S602: NO), the process proceeds to step S607.

  Then, the distance between the center coordinates of the confirmation hold display rectangle and the center coordinates of the confirmation OK display rectangle is calculated for all display rectangles adopted as the confirmation OK display area in step S515 (S603, S604).

  In step S605, the enlargement magnification of the confirmation OK display rectangle with the closest distance calculated in step S604 is compared with the enlargement magnification of the confirmation hold display rectangle.

  As a result of the comparison, when the enlargement magnification of the confirmation hold display rectangle is equal to or larger than the enlargement magnification of the nearest confirmation OK display rectangle (S605: YES), the process proceeds to step S606, and the confirmation hold If the enlargement magnification of the display rectangle is smaller than the enlargement magnification of the nearest confirmation OK display rectangle (S605: NO), the process proceeds to step S607.

  In step S606, the confirmation hold display rectangle to be processed is adopted as a confirmation OK display region, and in S607, the confirmation hold display rectangle to be processed is adopted as a confirmation NG display region. In other words, if it is viewed at a magnification equal to or greater than the confirmation OK display rectangle that is closest to the confirmation hold display rectangle, the confirmation OK display region is set.

  When the processing of S602 to S607 is completed for all confirmation hold display rectangles, the process proceeds to step S608.

  In step S608, a target image is acquired.

  In step S609, a mask image having the same resolution as the target image and a pixel value of (0, 0, 0) is created.

  In steps S610 and S611, the pixel values of the portions corresponding to all the confirmation NG display rectangles in the mask image are set to (255, 0, 0). That is, it sets so that the part corresponding to all the confirmation NG display rectangles can be identified in red.

  In steps S612 and S613, the pixel values of the portions corresponding to all the confirmation OK display rectangles in the mask image are set to (0, 0, 255). That is, settings are made so that all the portions corresponding to the confirmation OK display rectangles can be identified in blue.

  By performing the processing of S609 to S613, a region of the target image that is not enlarged is masked with no color, a confirmation NG display region is masked with red, and a confirmation OK display region is masked with blue. A mask image is created. Note that it is only necessary to be able to identify the area where the enlarged display is not performed, the confirmation NG display area, and the confirmation OK display area, and thus it is possible to freely set the coloring method. Instead of changing the color, the color may be identified by shading, identified by a pattern, or identified by a frame line display format.

  Then, the process proceeds to step S403 in FIG.

  In step S403, the mask image created in step S402 is transmitted to the client terminal 101.

  In step S404, the mask image transmitted in step S403 is received.

  In step S405, an image is created by superimposing the mask image received in step S404 and the target image on the face, and the superimposed image is displayed as an overhead image (step S406).

  As a result of the above processing, the area of the target image that has not been enlarged is displayed as it is, the confirmation NG display area is masked in red, and the confirmation OK display area is masked in blue. Can be displayed.

  As a result, it is possible to distinguish between the enlarged display area and the non-enlarged display area, and among the enlarged display areas, an area (or a moving image) enlarged at a predetermined magnification or more. Thus, it is possible to display an area that is magnified more than the calculated magnification) and other areas in an identifiable manner.

  In this embodiment, when there is a detection object rectangle, it is determined whether or not it is a confirmation OK area based on the size of the detection object rectangle. However, a predetermined reference magnification is used regardless of the presence or absence of the detection object rectangle. (Even if there is a detected object rectangle, it may be determined by the same processing as the processing of S518 to S522).

  Next, message display processing when a target image switching instruction is issued will be described using the flowchart of FIG.

  7 are processes executed by the CPU 201 of the client terminal 101, and processes of S702, S703, S707, and S708 are processes executed by the CPU 201 of the server apparatus 102.

  In step S701, it is determined whether or not an instruction to display the next image or an instruction to display the previous image has been given by the user on the screen shown in FIG. 9, and if the instruction has been given (S701: YES), the process is performed. The process proceeds to S702.

  In step S702, a mask image for displaying an overhead image is created. The processing in this step is the processing shown in the flowcharts of FIGS.

  In step S703, the mask image created in step S702 is transmitted to the client terminal 101.

  In step S704, the mask image transmitted in step S703 is received.

  In step S705, it is determined whether the pixel value (255, 0, 0) or the pixel value (0, 0, 0) exists. That is, it is determined whether there is an area that has not been enlarged or a confirmation NG display area.

  If it exists (S705: YES), the process proceeds to step S706, and a message indicating that there is an area for which enlarged display confirmation is insufficient or an undisplayed area is displayed. An example of a screen displaying a message is shown in FIG.

  If the OK button in FIG. 12 is pressed, the process proceeds to step S707.

  If it does not exist (S705: NO), the process proceeds to step S707.

  In step S707, an image to be edited (previous image or next image) is acquired and transmitted to the client terminal 101 (S708). The client terminal 101 receives the transmitted image (S709) and displays the image.

  As described above, it is possible to prevent omission of labeling of a detected object by notifying a message when an unconfirmed region or the like remains in the current image.

  FIG. 8 is a diagram illustrating an example of various data tables.

  The category label is a table in which the category of the detected object (information indicating what it is) is registered. The category name to be registered differs depending on what the object detector to detect is constructed. When constructing a detector for detecting the gender of a person included in an image as in this embodiment, categories such as male and female are registered as in the example of FIG.

  The image table is a table in which original images to be labeled are registered.

  The detected object rectangle is a table in which information indicating which category of object exists in which region of which image is registered. For example, the detected object rectangle identified by ID: 1 indicates that a male face exists in the area identified by (14, 20, 30, 50) of image: 1.jpg.

  The setting table is a table in which setting values necessary for the processing of the present invention are registered. These set values are also registered each time depending on what kind of object detector is constructed, what kind of image data the object is detected from, and the like.

  The display rectangle table is a table for managing information indicating which area of which image is displayed at which magnification.

  FIG. 11 is an example of a screen displayed in step S406. In the example of FIG. 11, identification is displayed by a pattern rather than by color.

  As shown in FIG. 11, the area (1101) where the vertical stripes are masked is a confirmation OK display area, and the area (1102) masked with light ink color is a confirmation NG display area. The other areas are areas that are not enlarged and displayed.

  As mentioned above, although the embodiment as the information processing system has been described, the present invention can take an embodiment as an apparatus, a method, a program, a recording medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The program according to the present invention is a program that allows a computer to execute the processing methods of the flowcharts shown in FIGS. 3 to 7. Is remembered. The program in the present invention may be a program for each processing method of each apparatus in FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

  As a recording medium for supplying the program, for example, flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 Client terminal 102 Server device

Claims (7)

An area designation accepting unit that accepts designation of a plurality of areas as an area to which attribute information in an image is given from a user ;
Display magnification setting accepting means for accepting setting of the display magnification of the image;
Display control means for displaying the image at a display magnification set by the display magnification setting accepting means;
With
The information processing apparatus, wherein the display control unit displays an area in the image that has been enlarged and an area that has not been enlarged in an identifiable manner. Wherein the display control unit, the region of what was the magnified and displayed, areas that may have been enlarged and areas that may have been enlarged and displayed at a predetermined condition is satisfied magnification at the magnification does not satisfy a predetermined condition 2. The information processing apparatus according to claim 1, wherein the information processing device is displayed in an identifiable manner.   The information processing apparatus according to claim 2, wherein whether or not the magnification satisfies the predetermined condition is determined based on an area to which the attribute information that has received the designation is added. In the case where there is an area that has not been enlarged and displayed at a magnification satisfying a predetermined condition in the image, the image processing apparatus further includes notification means for notifying that there is an area that has not been enlarged and displayed at a magnification that satisfies the predetermined condition. The information processing apparatus according to any one of claims 1 to 3. A switching instruction receiving means for receiving an image switching instruction;
When the notification means receives the image switching instruction, if there is an area that has not been magnified and displayed at a magnification that satisfies a predetermined condition in the image, the notification means is displayed at a magnification that satisfies the predetermined condition. 5. The information processing apparatus according to claim 4, wherein notification is made that there is a region that does not occur . An area designation accepting unit of the information processing apparatus receives an area designation accepting step for accepting designation of a plurality of areas as an area to which attribute information in an image is given from a user ;
A display magnification setting accepting step in which the display magnification setting accepting means of the information processing device accepts the setting of the display magnification of the image;
A display control step in which the display control means of the information processing apparatus displays the image at a display magnification set by the display magnification setting acceptance step;
With
The display control step displays an area in the image that has been enlarged and an area that has not been enlarged in an identifiable manner. Information processing device
An area designation accepting unit that accepts designation of a plurality of areas as an area to which attribute information in an image is given from a user ;
Display magnification setting accepting means for accepting setting of the display magnification of the image;
Function as display control means for displaying the image at a display magnification set by the display magnification setting accepting means;
A program for causing the display control means to function as a means for displaying an area in the image that has been enlarged and an area that has not been enlarged in an identifiable manner.

Images