JP7464509B2 - OBJECT DETECTION DEVICE, OBJECT DETECTION SYSTEM AND OBJECT DETECTION METHOD - Google Patents

Description

The present disclosure relates to an object detection device, an object detection system, and an object detection method.

In recent years, with the advancement of IT, numerous sensors have been installed in society and an extremely large amount of data has been accumulated. In this context, various methods for utilizing accumulated image data are being considered. In particular, as the amount of video content, such as photographs, videos, and images, increases, there is a growing need to freely identify objects in the video and accurately detect the object category and position.

One of the object detection methods is the so-called deep neural network (DNN). With the development of DNN, object detection can now be used in a variety of situations, such as X-ray image analysis.

However, to achieve highly accurate object detection using a DNN, a large amount of labeled learning data is required to train the DNN. Depending on the application of the DNN, it may be difficult to obtain such labeled learning data or it may require enormous cost and effort. As a result, while there is a large amount of learning data available for domains that are relatively easy to obtain, there is only a small amount of learning data available for domains that are difficult to obtain, and the accuracy of object detection in domains with little training data is limited.

To solve the above problems, for example, JP 2019-032821 A (Patent Document 1) discloses a method for training a DNN using easily available labeled learning data from a domain.

Patent document 1 describes a technology that "provides a method for reducing the need for costly, tedious, and error-prone manual labeling of training data. The method uses an image captured by a target camera as a style target image to train a style transfer network that performs style transfer to convert any photo-realistic input image into a transformed image. The transformed image has the content of the input image, maintains the photo-realistic quality of the input image, and is of a style that matches the style of the style target image. The trained style transfer network is used to convert training images of an original training dataset into transformed training images, and each transformed training image is labeled with the training label of a corresponding training image in the original training dataset to create an augmented training dataset, and the augmented training dataset is used to train a deep neural network (DNN) to perform a specific task."

JP 2019-032821 A

Patent document 1 describes a method of using a style transfer network to convert labeled images corresponding to a specific source domain into a target domain (the "target style" described in patent document 1), thereby obtaining labeled images corresponding to the target domain, and training a DNN using the image data obtained in this way.

However, Patent Document 1 is based on the premise that good approximation accuracy can be achieved between the pseudo target domain image (the "style target image" described in Patent Document 1) converted to the target domain and the actual target domain. In the case of images such as X-ray images where there is a large difference between the source domain and the target domain (so-called "domain gap" exists), even if a DNN is trained using the labels of the pseudo target domain image converted to the target domain, the object detection accuracy for the target domain image is limited.

Therefore, the present disclosure utilizes easily available labeled learning data from domains to train a DNN after shortening the domain gap between the source domain and the target domain. In this way, the present disclosure aims to provide a highly accurate object detection means even for images with a large difference between the source domain and the target domain, such as X-ray images.

In order to solve the above problem, one representative object detection device of the present disclosure is an object detection device for detecting objects in X-ray images, and includes an image input unit that accepts an input image set including a source domain image corresponding to a source domain and a target domain image corresponding to a target domain, a domain conversion unit that performs a domain conversion process on the input image set to generate a converted image set including a pseudo target domain image obtained by converting the source domain image to the target domain and a pseudo source domain image obtained by converting the target domain image to the source domain image, a pair generation unit that generates image pairs using the input image set and the converted image set, a feature extraction unit that extracts a feature map for each image included in the image pair, a detection prediction unit that generates a prediction result indicating the category and position of an object in each image included in the image pair based on the feature map, and an object detection unit that analyzes a specified X-ray image to generate a detection result indicating the category and position of an object in the X-ray image.

According to the present disclosure, by utilizing labeled learning data from easily available domains and shortening the domain gap between the source domain and the target domain before training a DNN, it is possible to provide a highly accurate object detection means even in the case of images with large differences between the source domain and the target domain, such as X-ray images.
Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention.

FIG. 1 illustrates a computer system for implementing an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of an object detection system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of an X-ray image according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a logical configuration of an object detection learning unit in the object detection device according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a domain conversion process performed by the domain conversion unit according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of an image pair generation process performed by the image pair generation unit according to an embodiment of the present disclosure. FIG. 7 illustrates an example of domain gap shortening according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example method for training an object detector according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of an object detection process according to an embodiment of the present disclosure.

Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

As described above, a large amount of labeled learning data is required to train a DNN, and depending on the application of the DNN, it may be difficult to obtain such labeled learning data or it may require huge costs and labor. Therefore, while a large amount of learning data is available for domains that are relatively easy to obtain, the amount of learning data for domains that are difficult to obtain is small, and the object detection accuracy in domains with little learning data is limited.
It would therefore be desirable to have a means to train a DNN that can generate highly accurate object detection results for any target domain (English: “target domain”) using only labeled training data from a domain that is readily available (hereafter referred to as the “source domain”), without collecting training data from the target domain that is difficult to obtain.

Here, "domain" refers to a set of parameters that define the display of an image. For example, various display settings such as color, sharpness, resolution, brightness, and contrast of an image are parameters that define the domain of an image. In other words, a domain refers to a collection of data.
When the display setting parameters described above differ significantly between a source domain image corresponding to the source domain and a target domain image corresponding to the target domain, a so-called "domain gap" exists between the source domain and the target domain.

When such a domain gap exists, for example, even if a DNN for object detection is trained with labeled data in the source domain, it cannot generate highly accurate object detection results for images in the target domain, which is an unlabeled domain.

In view of the above problems, in an embodiment of the present disclosure, in addition to an input image set including labeled source domain images and unlabeled target domain images, a DNN for object detection is trained using a transformed image set including pseudo target domain images in which source domain images are transformed into the target domain, and pseudo source domain images in which target domain images are transformed into the source domain. As a result, in an embodiment of the present disclosure, it is possible to shorten the domain gap between the source domain and the target domain, and to generate highly accurate object detection results for images in the target domain.

Next, referring to FIG. 1, a computer system 300 for implementing an embodiment of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 300 include one or more processors 302, memory 304, a terminal interface 312, a storage interface 314, an I/O (input/output) device interface 316, and a network interface 318. These components may be interconnected via a memory bus 306, an I/O bus 308, a bus interface unit 309, and an I/O bus interface unit 310.

Computer system 300 may include one or more general purpose programmable central processing units (CPUs) 302A and 302B, collectively referred to as processors 302. In some embodiments, computer system 300 may include multiple processors, and in other embodiments, computer system 300 may be a single CPU system. Each processor 302 executes instructions stored in memory 304 and may include an on-board cache.

In some embodiments, memory 304 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 304 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 304 may store an object detection application 350. In some embodiments, object detection application 350 may include instructions or descriptions that execute on processor 302 the functions described below.

In some embodiments, the object detection application 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the object detection application 350 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with the bus interface unit 309, the processor 302, or other hardware of the computer system 300.

Computer system 300 may include a bus interface unit 309 that provides communication between processor 302, memory 304, display system 324, and I/O bus interface unit 310. I/O bus interface unit 310 may be coupled to an I/O bus 308 for transferring data to and from various I/O units. I/O bus interface unit 310 may communicate via I/O bus 308 with multiple I/O interface units 312, 314, 316, and 318, also known as I/O processors (IOPs) or I/O adapters (IOAs).

The display system 324 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to the display device 326. The computer system 300 may also include one or more sensors or other devices configured to collect data and provide the data to the processor 302.

For example, computer system 300 may include biometric sensors to collect data such as heart rate data or stress level data, environmental sensors to collect data such as humidity data, temperature data, or pressure data, and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 324 may be connected to a display device 326, such as a standalone display screen, a television, a tablet, or a handheld device.

The I/O interface unit provides the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 312 can be fitted with user I/O devices 320, such as user output devices such as a video display, a speaker television, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing devices. A user may use a user interface to input input data or instructions to the user I/O devices 320 and the computer system 300 and receive output data from the computer system 300 by operating the user input devices. The user interface may be displayed on a display device, played through a speaker, or printed via a printer via the user I/O devices 320, for example.

The storage interface 314 allows attachment of one or more disk drives or direct access storage devices 322 (usually magnetic disk drive storage devices, but may be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, the storage device 322 may be implemented as any secondary storage device. The contents of the memory 304 may be stored in the storage device 322 and retrieved from the storage device 322 as needed. The I/O device interface 316 may provide an interface to other I/O devices such as printers, fax machines, etc. The network interface 318 may provide a communication path so that the computer system 300 and other devices can communicate with each other. This communication path may be, for example, a network 330.

In some embodiments, computer system 300 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 300 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, an object detection system according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 2 is a diagram showing an example of the configuration of an object detection system 200 according to an embodiment of the present disclosure. As shown in FIG. 2, the object detection system 200 according to the present disclosure mainly comprises an X-ray device 211, a communication network 202, and an object detection device 201. The X-ray device 211 and the object detection device 201 are connected via the communication network 202.

The communication network 202 may include, for example, a local area network (LAN), a wide area network (WAN), a satellite network, a cable network, a Wi-Fi network, or any combination thereof. In addition, the connection between the X-ray device 211 and the object detection device 201 may be wired or wireless.

The X-ray device 211 is a device that captures an X-ray image. The X-ray device 211 includes, for example, an X-ray generator that emits X-rays and an X-ray detector that detects and analyzes the reflected X-rays. The type of the X-ray device 211 according to the present disclosure is not particularly limited, and may be an X-ray device for luggage, a backscattered X-ray inspection device, or a medical X-ray device. The X-ray device 211 is configured to capture an X-ray image of a predetermined subject and transmit it to the object detection device 201 via the communication network 202.
Note that FIG. 2 shows an example of a configuration for processing X-ray images that includes an X-ray device 211, but the present disclosure is not limited thereto, and the X-ray device 211 may be, for example, any camera, sensor, or other device that provides an input image that is the subject of object detection.

The object detection device 201 is a computing device for detecting the category and location of an object in, for example, an X-ray image transmitted from the X-ray device 211. The object detection device 201 may be configured as, for example, a desktop computer, a server computer, a laptop computer, a tablet computer, a workstation, a mobile device, or other type of computing device.

As shown in FIG. 2, the object detection device 201 includes a processor 203 for executing instructions stored in a memory 207, an I/O interface 204 for controlling communication between internal and external devices of the object detection device 201, a network interface 205 for controlling communication via a communication network 202, a user I/O interface 206 for receiving input from a user, a memory 207 that stores functional units for executing each function of the object detection means according to an embodiment of the present disclosure, and a bus 212 for controlling bidirectional communication between these components.

As shown in FIG. 2, the memory 207 of the object detection device 201 includes an image input unit 208 that receives an input image set including a source domain image corresponding to the source domain and a target domain image corresponding to the target domain, an X-ray image transmitted from an X-ray device, etc., a domain conversion unit 210 that performs domain conversion processing on the input image set to generate a converted image set including a pseudo target domain image obtained by converting the source domain image into the target domain and a pseudo source domain image obtained by converting the target domain image into the source domain image, and a domain conversion unit 220 that converts the pseudo target domain image and the pseudo source domain image into a highly accurate 210, an object detection learning unit 225 that trains the object detection unit 220 to improve detection accuracy, the object detection unit 220 that, after being trained, detects the category and position of an object in the X-ray image transmitted from the X-ray device 211, a source domain image storage unit 230 for storing source domain images, a target domain image storage unit 235 for storing target domain images, a pseudo source domain image storage unit 240 for storing pseudo source domain images, and a pseudo target domain image storage unit 245 for storing pseudo target domain images.

The object detection unit 220 according to an embodiment of the present disclosure may be configured as a deep neural network. As described below, by training the deep neural network serving as the object detection unit 220 using an input image set including labeled source domain images and unlabeled target domain images, as well as a transformed image set including pseudo target domain images obtained by transforming source domain images into the target domain and pseudo source domain images obtained by transforming target domain images into the source domain, it is possible to shorten the domain gap between the source domain and the target domain and generate highly accurate object detection results for images in the target domain.

Each functional unit included in the object detection device 201 may be a software module constituting the object detection application 350 in the computer system 300 shown in FIG. 1, or may be an independent dedicated hardware device. Furthermore, the above functional units may be implemented in the same computing environment, or may be implemented in distributed computing environments.

According to the object detection system 200 described above, by utilizing labeled learning data from easily available domains, shortening the domain gap between the source domain and the target domain, and training the DNN, it is possible to detect the category and position of an object in an X-ray image received from an external device such as the X-ray device 211 via the communication network 202.

Next, referring to FIG. 3, we will explain the X-ray image according to an embodiment of the present disclosure.

FIG. 3 is a diagram illustrating an example of an X-ray image according to an embodiment of the present disclosure.
As described above, an object detection device according to an embodiment of the present disclosure (e.g., the object detection device 201 shown in FIG. 2 ) receives as input a source domain image, which is a labeled image corresponding to a source domain (e.g., a domain with a wealth of labeled training data), and a target domain image, which is an unlabeled image corresponding to a target domain (e.g., a domain with a small amount of labeled training data). These source domain image and target domain image become part of the training data used to train the object detection device, as described below.

3 illustrates an example of a source domain image 361 and a target domain image 362 according to an embodiment of the present disclosure. The source domain image 361 is a labeled image, such that metadata is attached to the source domain image 361 indicating the category and location of each object in the source domain image 361.
On the other hand, the target domain image 362 is an unlabeled image, so the category and position of each object in the target domain image 362 is unknown.

The source domain image 361 and the target domain image 362 are X-ray images with different domains because, for example, they were taken by different X-ray devices or taken by the same X-ray device with different shooting settings. Therefore, various display settings such as color and sharpness are different between the source domain image 361 and the target domain image 362, and a so-called "domain gap" exists between the source domain and the target domain.

When such a domain gap exists, for example, even if a DNN for object detection is trained with labeled data in the source domain, it cannot generate highly accurate object detection results for images in the target domain, which is an unlabeled domain.
Therefore, as described below, in the present disclosure, by shortening the distance between the source domain and the target domain and reducing the domain gap, it becomes possible to generate highly accurate object detection results even for images in an unlabeled domain.

Next, with reference to FIG. 4, the logical configuration of the object detection learning unit in the object detection device according to an embodiment of the present disclosure will be described.

FIG. 4 is a diagram illustrating a logical configuration of the object detection learning unit 225 in the object detection device 201 according to an embodiment of the present disclosure.
As described above, the object detection learning unit 225 in the object detection device 201 according to an embodiment of the present disclosure is a functional unit for training the object detection unit (e.g., the object detection unit 220 shown in FIG. 2) by adjusting parameters of the object detection unit so as to improve detection accuracy.

As shown in FIG. 4, the object detection learning unit 225 includes a pair generation unit 365, a feature extraction unit 368, an image disparity calculation unit 369, an adaptation loss calculation unit 370, a detection prediction unit 371, a detection loss calculation unit 372, and a parameter update unit 373.

First, the pair generation unit 365 inputs an input image set including a source domain image 361 and a target domain image 362, and a transformed image set including a pseudo source domain image 363 and a pseudo target domain image 364 obtained by performing a domain transformation process on the input image set. As described with reference to Fig. 3, the source domain image 361 is a labeled image corresponding to a predetermined source domain, and the target domain image 362 is an unlabeled image corresponding to a target domain that is a domain different from the source domain.
4, for convenience of explanation, the pair generation unit 365 inputs the source domain image 361, the target domain image 362, the pseudo source domain image 363, and the pseudo target domain image 364 one by one as an example, but the present disclosure is not limited to this. In reality, the pair generation unit 365 may input a batch including multiple images for each of the source domain image 361, the target domain image 362, the pseudo source domain image 363, and the pseudo target domain image 364.

The pseudo source domain image 363 is an image obtained by transforming the target domain image 362 into the source domain using the above-mentioned domain transformation unit (for example, the domain transformation unit 210 shown in FIG. 2).
Note that the pseudo source domain image 363 is an image converted from the target domain image 362, which is an unlabeled image, and is therefore an unlabeled image, just like the target domain image 362.

The pseudo target domain image 364 is an image obtained by transforming the source domain image 361 into the target domain using the domain transformation unit described above.
Note that the pseudo target domain image 364 is an image transformed from the source domain image 361, which is a labeled image, and is therefore a labeled image, similar to the source domain image 361. Also, in this disclosure, the images in the transformed image set obtained by performing a domain transformation process on the input image set are called "pseudo" because they do not completely match the actual source and target domains.

The pair generation unit 365 receives the source domain image 361, the target domain image 362, the pseudo source domain image 363, and the pseudo target domain image 364 and generates image pairs. More specifically, the pair generation unit 365 may generate pairs that combine each image included in the input image set with each image included in the transformation image set. The pair generation unit 365 may also generate pairs that combine each image included in the input image set with each image included in the transformation image set.
From the input image set and the transformed image set, images whose photographic contents (object category and arrangement) satisfy a predetermined similarity standard are designated as positive pairs 366, and from the input image set and the transformed image set, images whose photographic contents do not satisfy the predetermined similarity standard are designated as negative pairs 367.
The image pair will be described in detail with reference to FIG. 6, and therefore will not be described here.

The feature extraction unit 368 inputs the image pairs generated by the pair generation unit, and then extracts feature maps for each image included in these image pairs. Here, the means for extracting the feature maps of each image included in the image pairs may be an existing means such as a so-called convolutional neural network, and is not particularly limited in this disclosure. The feature maps of each image created by the feature extraction unit 368 are transferred to the image deviation calculation unit 369 and the detection prediction unit 371.

The image deviation calculation unit 369 calculates the deviation of the feature distribution of each image in each image pair by comparing the feature maps of each image in the image pair. The deviation here is a value indicating the distance of the feature distribution of the images, and the larger the deviation of the feature distribution of the image pair, the larger the domain gap described above.
The details of the calculation of the degree of deviation of the feature distribution of an image pair will be described later, and therefore will not be described here.

The adaptive loss calculation unit 370 calculates a first adaptive loss parameter that subtracts the disparity of a positive pair and a second adaptive loss parameter that improves the disparity of a negative pair among the image pairs generated by the pair generation unit 365.
The calculation of the adaptive loss parameters will be described in detail later, and therefore will not be described here.

The detection prediction unit 371 predicts the category and position of an object in each image from the feature map of each image pair generated by the feature extraction unit 368, and generates a prediction result indicating the predicted category and position of the object. As the detection prediction unit 371 here, for example, the deep neural network constituting the object detection unit 220 may be used in a pre-trained state.

The detection loss calculation unit 372 calculates a detection loss parameter indicating the detection loss by the detection prediction unit 371 by comparing the prediction result generated by the detection prediction unit 371 with ground truth indicating the actual category and actual location of the object.
The details of the calculation of the detection loss parameter will be described later, and therefore will not be described here.

The parameter update unit 373 trains the object detection unit by adjusting the parameters of the deep neural network constituting the object detection unit (e.g., the object detection unit 220 shown in FIG. 2) based on the first adaptation loss parameter and the second adaptation loss parameter calculated by the adaptation loss calculation unit 370 and the detection loss parameter calculated by the detection loss calculation unit 372.

According to the configuration of the object detection learning unit 225 described above, by training the object detection unit based on the adaptation loss parameter and the detection loss parameter, the object detection unit is trained to minimize the difference between the prediction result and the ground truth, and can generate highly accurate object detection results for images in the target domain. In this way, the object detection unit can be trained using only labeled learning data from easily available domains, eliminating the need to collect learning data from the target domain that is difficult to obtain, and reducing the cost of introducing an object detection means using a deep neural network.

Next, referring to FIG. 5, the domain conversion process performed by the domain conversion unit according to an embodiment of the present disclosure will be described.

FIG. 5 is a diagram illustrating an example of a domain conversion process by the domain conversion unit 210 according to an embodiment of the present disclosure. As described above, the domain conversion unit 210 according to an embodiment of the present disclosure generates a pseudo target domain image 511 obtained by converting a source domain image 505, which is a labeled image corresponding to a source domain (e.g., a domain with a wealth of labeled learning data), into a target domain, and a pseudo source domain image 509 obtained by converting a target domain image 507, which is an unlabeled image corresponding to a target domain (e.g., a domain with little labeled learning data), into a source domain.

5 shows an example of a source domain image 505, a target domain image 507, a pseudo source domain image 509, and a pseudo target domain image 511 according to an embodiment of the present disclosure. Also, the triangles and circles in each image shown in FIG. 5 represent two categories of objects (e.g., water bottles and watches) in the image.
Note that, for convenience of explanation, FIG. 5 shows an example of an image including objects of two categories, but the present disclosure is not limited to this, and the image may include any number of categories.

The source domain image 505 and the target domain image 507 are X-ray images with different domains, for example, because they were taken by different X-ray devices or taken by the same X-ray device with different shooting settings. For this reason, the source domain image 505 and the target domain image 507 differ in various display settings such as color and sharpness, and a domain gap exists between the source domain and the target domain, and the feature distributions of the respective images are significantly different.

When such a domain gap exists, for example, even if a DNN for object detection is trained with labeled data in the source domain, it cannot generate highly accurate object detection results for images in the target domain, which is an unlabeled domain.
Therefore, in the present disclosure, a domain conversion unit is used to perform domain conversion processing on each image, thereby shortening the domain gap and bringing the source domain and the target domain closer to each other.

In the domain conversion process according to the present disclosure, the domain conversion unit adjusts the display setting parameters such as color, brightness, sharpness, etc. of each image to match the domain of the other image without changing the positions of objects in the images. More specifically, the domain conversion unit 210 adjusts the color, brightness, sharpness, etc. of the source domain image 505 to match the target domain, and generates a pseudo target domain image 511. Similarly, the domain conversion unit 210 adjusts the display setting parameters such as color, brightness, sharpness, etc. of the target domain image 507 to match the source domain, and generates a pseudo source domain image 509.

This domain conversion process reduces the domain gap between images. In addition, by using images with a reduced domain gap to train a DNN for object detection, it becomes possible to generate highly accurate object detection results for images in the target domain, for example, without collecting a large amount of training data corresponding to the target domain.

Next, the image pair generation process performed by the image pair generation unit according to an embodiment of the present disclosure will be described with reference to FIG. 6.

FIG. 6 is a diagram showing an example of an image pair generation process by a pair generation unit according to an embodiment of the present disclosure. As described above, an image pair generation unit according to an embodiment of the present disclosure (e.g., image pair generation unit 365 shown in FIG. 4) may input an input image set including a source domain image and a target domain image, and a transformed image set including a pseudo source domain image and a pseudo target domain image, and generate pairs combining each image included in the input image set with each image included in the transformed image set.

As an example, suppose there are three source domain images, source domain image 605A, source domain image 605B, and source domain image 605C, and three target domain images, pseudo target domain image 610A, pseudo target domain image 610B, and pseudo target domain image 610C, which are obtained by converting these three source domain images into the target domain. In this case, the image pair generation unit generates nine image pairs: source domain image 605A and pseudo target domain image 610A, source domain image 605A and pseudo target domain image 610B, source domain image 605A and pseudo target domain image 610C, source domain image 605B and pseudo target domain image 610A, source domain image 605B and pseudo target domain image 610B, source domain image 605B and pseudo target domain image 610C, source domain image 605C and pseudo target domain image 610A, source domain image 605C and pseudo target domain image 610B, and source domain image 605C and pseudo target domain image 610C.

Furthermore, from among these nine pairs, the image pair generating unit selects images whose photographed contents (shape and arrangement of objects) satisfy a predetermined similarity standard as a positive pair 612, and selects images whose photographed contents do not satisfy the predetermined similarity standard as a negative pair 614. The similarity standard here may be set in advance by the user. Furthermore, the similarity of the images may be determined by an existing image similarity algorithm, and is not particularly limited here.
5, positive pairs 612 are shown with solid lines, and negative pairs 614 are shown with dotted lines. Thus, each source domain image and its corresponding pseudo target domain image are positive pairs 612, and the remaining image combinations are negative pairs 614.

More specifically, the source domain image 605A and the pseudo target domain image 610A, the source domain image 605B and the pseudo target domain image 610B, and the source domain image 605C and the pseudo target domain image 610C form positive pairs 612, while the source domain image 605A and the pseudo target domain image 610B, the source domain image 605A and the pseudo target domain image 610C, the source domain image 605B and the pseudo target domain image 610A, the source domain image 605B and the pseudo target domain image 610C, the source domain image 605C and the pseudo target domain image 610A, and the source domain image 605C and the pseudo target domain image 610B form negative pairs 614.

As described below, by subtracting the degree of deviation of the positive pair 612 and improving the degree of deviation of the negative pair 614 among the image pairs generated by the pair generation unit, it is possible to shorten the domain gap while improving the discrimination ability (i.e., detection accuracy) of the object category.

Next, referring to FIG. 7, we will explain domain gap shortening according to an embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of domain gap shortening according to an embodiment of the present disclosure. As described above, the parameters of the object detection unit are updated based on parameters generated by an adaptive loss calculation unit (e.g., adaptive loss calculation unit 370 shown in FIG. 3) and a detection loss calculation unit (e.g., detection loss calculation unit 372 shown in FIG. 3) according to an embodiment of the present disclosure, thereby making it possible to shorten the domain gap of images corresponding to different domains.

7 shows an example of domain gap shortening by the adaptive loss calculation unit and the detection loss calculation unit. As described above, the adaptive loss calculation unit calculates a first adaptive loss parameter that reduces the disparity of a positive pair and a second adaptive loss parameter that improves the disparity of a negative pair among the image pairs generated by the pair generation unit, and the object detection unit is trained based on these parameters.
7, when the deviation degree of the positive pair is subtracted, the feature distributions of the same category between the source domain image 505 and the pseudo target domain image 511 and the target domain image 507 and the pseudo source domain image 509 become closer to each other. Also, when the deviation degree of the negative pair is improved, the feature distributions of different categories between the source domain image 505 and the pseudo target domain image 511 and the target domain image 507 and the pseudo source domain image 509 become more distant from each other.

As described above, the detection loss calculation unit also calculates a detection loss parameter by comparing the prediction result generated by the detection prediction unit with ground truth indicating the actual category and actual location of the object, and the object detection unit is trained based on this parameter. This allows the object detection unit to recognize different categories with higher accuracy.

In this way, by updating the parameters of the object detection unit using the parameters from the adaptive loss calculation unit and the detection loss calculation unit and shortening the image domain gap, the object detection unit can generate highly accurate object detection results even for unlabeled target domain images.

Next, with reference to FIG. 8, a method for training an object detection unit according to an embodiment of the present disclosure will be described.

FIG. 8 is a diagram illustrating an example of an object detection unit training method 800 according to an embodiment of the present disclosure. The object detection unit training method 800 illustrated in FIG. 8 is a method executed by each functional unit of the object detection device 201 illustrated in FIG. 2, for example, to train the object detection unit.

First, in step S810, an image input unit (for example, the image input unit 208 shown in FIG. 2)
An input image set including a source domain image corresponding to the source domain and a target domain image corresponding to the target domain is accepted, where the source domain image and the target domain image may be, for example, images selected by a user for training, or may be images previously transmitted from an X-ray device connected to the object detection device and selected for training.
The details of the source domain image and the target domain image have been described with reference to FIG. 3, and therefore will not be described here.

Next, in step S820, a domain conversion unit (e.g., the domain conversion unit 210 shown in FIG. 2) performs a domain conversion process on the input image set accepted by the image input unit in step S810, and generates a converted image set including a pseudo target domain image obtained by converting the source domain image to the target domain, and a pseudo source domain image obtained by converting the target domain image to the source domain image.
The details of the domain conversion process have been described with reference to FIG. 5, and therefore will not be described here.

Next, in step S830, a pair generation unit (for example, the pair generation unit 365 shown in FIG. 4) inputs the source domain image, the target domain image, the pseudo source domain image, and the pseudo target domain image to generate image pairs. For example, as described above, the pair generation unit generates a positive pair of images from the input image set and the transformed image set whose captured content (object category and arrangement) meets a predetermined similarity standard, and generates a negative pair of images from the input image set and the transformed image set whose captured content does not meet the predetermined similarity standard.
The details of the process for generating image pairs have been described with reference to FIG. 6, and therefore will not be described here.

Next, in step S840, a feature extraction unit (e.g., feature extraction unit 368 shown in FIG. 4) extracts a feature map for each image included in the image pair generated in step S830. The feature extraction unit here may be a convolutional neural network trained to extract feature maps according to the categories and arrangements of objects in the images. The feature map here may be, for example, a one-dimensional vector, or a two- or three-dimensional matrix representation.

ここで、ｕ及びｖは、ソースドメインＳのｉ番目の画像の特徴マップfⁱ _s及び疑似ターゲットドメインＴ’のｉ番目の画像の特徴マップfⁱ _Ｔ’におけるベクトルであってもよい。このように、画像乖離度計算部、ペア生成部によって生成される各ポジティブペア及び各ネガティブペアについて、当該ペアの乖離度を計算する。
例えば、第１の画像に対応する第１の特徴マップ及び第２の画像に対応する第２の特徴マップとを含むポジティブペアがあり、第３の画像に対応する第３の特徴マップ及び第４の画像に対応する第４の特徴マップを含むネガティブペアがある場合、画像乖離度計算部は、第１の特徴マップと第２の特徴マップとの乖離度である第１の乖離度を計算し、第３の特徴マップと第４の特徴マップとの乖離度である第２の乖離度を計算してもよい。 Next, in step S850, an image disparity calculation unit (for example, the image disparity calculation unit 369 shown in FIG. 4) calculates the disparity of each image pair using the feature map generated in step S840. The disparity here is a value indicating the distance of the feature distribution of the images, and the larger the disparity of the feature distribution of the image pair, the larger the domain gap described above. As a method for calculating the disparity here, for example, a cosine similarity method may be used, which calculates the cosine of the angle between two vectors projected into a multidimensional space, as shown in Equation 1.

Here, u and v may be vectors in the feature map f ⁱ _s of the i-th image in the source domain S and the feature map f ⁱ _T ' of the i-th image in the pseudo target domain T'. In this way, the image disparity calculation unit calculates the disparity of each positive pair and each negative pair generated by the pair generation unit.
For example, if there is a positive pair including a first feature map corresponding to a first image and a second feature map corresponding to a second image, and a negative pair including a third feature map corresponding to a third image and a fourth feature map corresponding to a fourth image, the image discrepancy calculation unit may calculate a first discrepancy that is the discrepancy between the first feature map and the second feature map, and may calculate a second discrepancy that is the discrepancy between the third feature map and the fourth feature map.

Next, an adaptation loss calculation unit (e.g., the adaptation loss calculation unit 370 shown in FIG. 4) calculates a first adaptation loss parameter for subtracting the deviation (e.g., the first deviation) of the image included in the positive pair, and calculates a second adaptation loss parameter for improving the deviation (e.g., the second deviation) of the image included in the negative pair.

The deviation of the positive pair images is subtracted to train the feature extractor to extract domain invariant features to reduce the domain gap for images with similar shooting contents (object categories and arrangements), thereby approximating the feature distributions between the source domain and the pseudo target domain, and between the target domain and the pseudo source domain.
The purpose of improving the dissimilarity of images in the negative pair is to train the feature extractor to extract different features for images that are dissimilar in content (object categories and placement), thereby improving the discrimination (i.e., detection accuracy) of different objects.

数式２では、分子はポジティブペアの乖離度（例えば、第１の乖離度）であり、分母は、ネガティブペアの乖離度（例えば、第２の乖離度）である。従って、ポジティブペアの乖離度を減算し、ネガティブペアの乖離度を向上すると、適応損失パラメータがより小さい値となる。
なお、分母に示されるネガティブペアの乖離度は、複数のネガティブペアの乖離度の和であってもよい。１つのポジティブペアに対して、多数のネガティブペアの乖離度を合わせて数式２の分母とすることで、異なるオブジェクトの識別力（つまり、検出精度）を更に向上させることができる。 The adaptive loss parameter here is calculated, for example, from the following Equation 2.
The following Equation 2 is the adaptive loss parameter L _adp ^S,T′ of the source domain and the pseudo target domain.
, which may be used to calculate the adaptive loss parameters L _adp ^S',T in the target domain and pseudo source domain.

In Equation 2, the numerator is the positive pair disparity (e.g., the first disparity) and the denominator is the negative pair disparity (e.g., the second disparity). Thus, subtracting the positive pair disparity and improving the negative pair disparity results in a smaller adaptation loss parameter.
The deviation of the negative pair shown in the denominator may be the sum of the deviations of multiple negative pairs. By adding up the deviations of multiple negative pairs for one positive pair and using them as the denominator of Equation 2, the ability to distinguish between different objects (i.e., detection accuracy) can be further improved.

Next, in step S860, a detection prediction unit (e.g., detection prediction unit 371 shown in FIG. 4) predicts the category and location of an object in each image using the feature map of each image pair generated in step S840, and generates a prediction result indicating the predicted category and location of the object. The detection prediction unit here may be, for example, a Faster R-CNN including a Region Proposal Network (RPN) that generates region proposals for a given region of interest (ROI), a classifier (classification head) that identifies the category of each ROI, and a box regression head for adjusting the coordinates of the bounding box of each ROI.

Next, a detection loss calculation unit (e.g., the detection loss calculation unit 372 shown in FIG. 4) calculates a detection loss parameter by the detection prediction unit by comparing the prediction result generated by the detection prediction unit with the ground truth indicating the actual category and actual location of the object. The detection loss parameter here may include an RPN loss, a classification loss, and a regression loss obtained by calculating the distance between the prediction result generated by the detection prediction unit and the ground truth indicating the actual category and actual location of the object.
Note that the detection loss parameter here is calculated only on labeled data (i.e., source domain images and pseudo target domain images).

Next, in step S870, a parameter update unit (e.g., parameter update unit 373 shown in FIG. 4) trains the object detection unit by adjusting the parameters of the object detection unit based on the first adaptation loss parameter and the second adaptation loss parameter calculated by the adaptation loss calculation unit and the detection loss calculated by the detection loss calculation unit.

According to the object detection unit training method 800 described above, by adjusting the parameters of the object detection unit based on the adaptation loss parameter and the detection loss parameter, the object detection unit is trained to minimize the difference between the prediction result and the ground truth, and can generate highly accurate object detection results even for images in the target domain with little labeled training data, for example.

Next, the object detection process according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 9 is a diagram illustrating an example of an object detection process 900 according to an embodiment of the present disclosure. The object detection process 900 illustrated in FIG. 9 is executed by each functional unit of an object detection device trained, for example, by the object detection unit training method 800 illustrated in FIG. 8, and is a process for detecting the category and position of an object in a specified X-ray image.

First, in step S910, an image input unit (for example, the image input unit 208 shown in FIG. 2)
A target domain image is received, where the image input may for example input an image sent from an X-ray device configured to acquire an X-ray image in the target domain.

Next, in step S920, a feature extraction unit (e.g., feature extraction unit 368 shown in FIG. 4) extracts a feature map for the target domain image received by the image input unit in step S910. As described above, the feature extraction unit here may be a convolutional neural network trained to extract a feature map according to the category and arrangement of objects in the image. In addition, the feature map here may be, for example, a one-dimensional vector, or a two- or three-dimensional matrix representation.

Next, in step S930, an object detection unit (e.g., object detection unit 220 shown in FIG. 2) detects the category and position of each object in the target domain image based on the feature map extracted in step S920, and generates data indicating the category and position of each of these objects as detection results.

Next, in step S940, the object detection unit outputs the detection result information generated in step S930. Here, the object detection unit may transmit the detection result to a predetermined notification destination (such as the administrator of the X-ray device) via, for example, a communication network.

According to the object detection process 900 described above, the object detection device can generate highly accurate object detection results even for X-ray images of a target domain with little unlabeled training data.

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present disclosure.

200 Object detection system 201 Object detection device 202 Communication network 203 Processor 204 I/O interface 205 Network interface 206 User I/O interface 207 Memory 208 Image input unit 210 Domain conversion unit 211 X-ray device 215 Domain conversion learning unit 220 Object detection unit 225 Object detection learning unit 230 Source domain image storage unit 235 Target domain image storage unit 240 Pseudo source domain image storage unit 245 Pseudo target domain image storage unit 361 Source domain image 362 Target domain image 363 Pseudo source domain image 364 Pseudo target domain image 365 Pair generation unit 366 Positive pair 367 Negative pair 368 Feature extraction unit 369 Image deviation calculation unit 370 Adaptation loss calculation unit 371 Detection prediction unit 372 Detection loss calculation unit 373 Parameter update unit

Claims (8)

1. An object detection apparatus for detecting an object in an X-ray image, comprising:
an image input unit for accepting a set of input images including a source domain image corresponding to a source domain and a target domain image corresponding to a target domain;
A domain conversion unit that performs a domain conversion process on the input image set to generate a converted image set including a pseudo target domain image obtained by converting the source domain image into the target domain and a pseudo source domain image obtained by converting the target domain image into the source domain image;
a pair generation unit that generates image pairs using the input image set and the transformed image set;
a feature extractor for extracting a feature map for each image in the image pair;
a detection and prediction unit that generates a prediction result indicating a category and a position of an object in each image of the image pair based on the feature map;
an object detector configured to analyze a given X-ray image to generate detection results indicative of categories and locations of objects in the X-ray image;
1. An object detection device comprising: The pair generation unit
A first image and a second image, the contents of which satisfy a predetermined similarity criterion, are determined as a positive pair from among the input image set and the converted image set;
A third image and a fourth image, the photographed contents of which do not satisfy a predetermined similarity standard, are set as a negative pair from among the input image set and the converted image set.
The object detection device according to claim 1 . The feature extraction unit is
extracting a first feature map for a first image in the positive pair;
extracting a second feature map for a second image in the positive pair;
extracting a third feature map for a third image in the negative pair;
extracting a fourth feature map for a fourth image in the negative pair;
The object detection device includes:
calculating a first degree of discrepancy between the first feature map and the second feature map;
an image disparity calculation unit that calculates a second disparity between the third feature map and the fourth feature map;
The object detection apparatus according to claim 2 , further comprising: The object detection device includes:
Calculating a first adaptive loss parameter for subtracting the first discrepancy for the first image and the second image included in the positive pair;
an adaptation loss calculation unit for calculating a second adaptation loss parameter for improving the second discrepancy for a third image and a fourth image included in the negative pair;
The object detection apparatus according to claim 3 , further comprising: a detection loss calculation unit for calculating a detection loss parameter indicative of a detection loss by the object detection unit by comparing the prediction result with a ground truth indicative of an actual category and an actual location of the object;
a parameter update unit that uses the detection loss parameter, the first adaptation loss parameter, and the second adaptation loss parameter to update parameters of the object detection unit to train the object detection unit;
The object detection apparatus according to claim 4 , further comprising: The object detection unit is a deep neural network.
The object detection device according to claim 5 . 1. An object detection system for detecting an object in an X-ray image, comprising:
an object detection device for analyzing the X-ray image and detecting an object;
an X-ray device for capturing the X-ray image and transmitting the X-ray image to the object detection device is connected via a communication network;
The object detection device includes:
an image input unit for accepting a set of input images including a source domain image corresponding to a source domain and a target domain image corresponding to a target domain;
A domain conversion unit that performs a domain conversion process on the input image set to generate a converted image set including a pseudo target domain image obtained by converting the source domain image into the target domain and a pseudo source domain image obtained by converting the target domain image into the source domain image;
a pair generation unit that generates image pairs using the input image set and the transformed image set;
a feature extractor for extracting a feature map for each image in the image pair;
a detection and prediction unit that generates a prediction result indicating a category and a position of an object in each image of the image pair based on the feature map;
an object detection unit that analyzes the X-ray image received from the X-ray device to generate a detection result indicating a category and a position of an object in the X-ray image and transmits the detection result to a predetermined notification destination;
1. An object detection system comprising: 1. An object detection method for detecting an object in an X-ray image, comprising:
accepting a set of input images including a source domain image corresponding to a source domain and a target domain image corresponding to a target domain;
performing a domain transformation process on the input image set to generate a transformed image set including a pseudo target domain image obtained by transforming the source domain image into the target domain and a pseudo source domain image obtained by transforming the target domain image into the source domain image;
generating image pairs by selecting a first image and a second image from the input image set and the transformed image set, the first image and the second image having contents that satisfy a predetermined similarity criterion as a positive pair, and selecting a third image and a fourth image from the input image set and the transformed image set, the third image and the fourth image having contents that do not satisfy a predetermined similarity criterion as a negative pair;
extracting a first feature map for a first image in the positive pair, extracting a second feature map for a second image in the positive pair, extracting a third feature map for a third image in the negative pair, and extracting a fourth feature map for a fourth image in the negative pair;
calculating a first degree of discrepancy between the first feature map and the second feature map;
calculating a second degree of discrepancy between the third feature map and the fourth feature map;
Calculating a first adaptive loss parameter for subtracting the first discrepancy for a first image and a second image included in the positive pair, and calculating a second adaptive loss parameter for improving the second discrepancy for a third image and a fourth image included in the negative pair;
generating a prediction indicative of a category and a location of an object in each image of the image pair based on the first feature map, the second feature map, the third feature map, and the fourth feature map;
calculating a detection loss parameter indicative of a detection loss due to object detection by comparing the prediction result with a ground truth indicative of an actual category and an actual location of the object;
training a deep neural network for object detection by updating parameters of the deep neural network for object detection using the detection loss parameters, the first adaptive loss parameters, and the second adaptive loss parameters;
analyzing a given X-ray image using the trained deep neural network for object detection to generate detection results indicative of object categories and locations in the X-ray image;
1. A method for detecting an object, comprising:

Images