JP7534916B2 - Data Generation System - Google Patents

Description

One aspect of the present disclosure relates to a data generation system.

Technology for determining the condition of a person's oral cavity through machine learning is known. For example, Patent Document 1 describes a medical treatment device that includes a diagnosis unit that outputs oral cavity diagnostic information based on an image of the patient's oral cavity and diagnostic reference data, and an update unit that updates the diagnostic reference data through machine learning based on a disease image that represents an oral cavity disease. Patent Document 2 describes a machine learning device that learns a function for determining the probability of suffering from periodontal disease or the state of periodontal disease.

Annotation, a technology for generating training data used in machine learning, is also known. For example, Patent Document 3 describes a machine learning device that analyzes an image to determine the extension direction of a structure's image, determines the direction perpendicular to the extension direction, determines the contour portion in the vertical direction based on the contrast change and/or color change of the structure's image, performs image processing on the structure in the image to fill in the structure's contour portion in the vertical direction along the extension direction of the structure, and uses the filled image as annotation information.

International Publication No. 2018/193936 International Publication No. 2018/159712 JP 2020-35094 A

It is desirable to efficiently generate training data to be used in generating machine learning models.

A data generation system according to one aspect of the present disclosure includes at least one processor. The at least one processor acquires a reference record indicating a combination of a first intraoral image of a human oral cavity and label information related to the state of the oral cavity, inputs the first intraoral image to each of a plurality of trained models that estimate the state of the oral cavity from the intraoral image, each trained model being generated by machine learning using an intraoral image that has been subjected to corresponding image processing, calculates a plurality of output vectors, selects from the plurality of trained models a trained model that has obtained an output vector that has the highest similarity to the label information, performs image processing corresponding to the selected trained model on the first intraoral image to generate a second intraoral image, generates an additional record indicating a combination of the second intraoral image and the label information, and outputs the additional record as at least a part of training data for machine learning.

In this aspect, a trained model that has obtained an output vector that is most similar to the label information of the first oral cavity image is selected, and a second oral cavity image is generated from the first oral cavity image by image processing corresponding to the trained model. This second oral cavity image can be said to be qualified to have the same label information as the first oral cavity image. Therefore, by performing annotation using the second oral cavity image, training data that is used to generate a machine learning model that estimates the state of a person's oral cavity can be efficiently generated.

According to one aspect of the present disclosure, it is possible to efficiently generate training data used to generate a machine learning model.

FIG. 1 is a diagram illustrating an example of a functional configuration of a data generation system according to an embodiment. 4 is a flowchart illustrating an example of an operation of the data generation system according to the embodiment. FIG. 1 is a diagram showing a specific example of data augmentation by the data generation system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the data generation system according to the embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the drawings, identical or equivalent elements are given the same reference numerals, and duplicate descriptions will be omitted.

The data generation system 1 according to the embodiment is a computer system for generating at least a portion of the teacher data used in machine learning to determine the state of a person's oral cavity. A machine learning model for determining the state of a person's oral cavity is generated by machine learning using the teacher data, i.e., supervised learning.

Machine learning is a technique for autonomously finding laws or rules by repeatedly learning based on given information. A machine learning model is a computational model used in machine learning. A machine learning model can be constructed using algorithms and data structures. There are no limitations on the machine learning technique, and various techniques such as neural networks and deep learning may be used.

The "oral condition" determined by machine learning is not limited. For example, the "oral condition" may be the condition of the entire oral cavity, or the condition of partial areas (sites) that make up the oral cavity, such as the teeth, gums, tongue, palate, and lips. Examples of the "condition" of the oral cavity include the presence or absence of symptoms, classification of symptoms, and progression of symptoms, but information regarding the "condition" is not limited to these examples.

A machine learning model for determining the state of a person's oral cavity is generated autonomously by learning using training data. The training data is electronic data consisting of a large number of combinations of an input vector input to the machine learning model and a correct output vector (this correct answer is also called "label information") that should be obtained from the input vector. In this disclosure, the autonomously generated machine learning model is also called a "trained model". The process of generating a trained model (learning process) corresponds to the learning phase. It should be noted that a trained model is a computational model that is estimated to be optimal for performing some computation, and is not necessarily a "computational model that is optimal in reality". The trained model processes an input vector whose correct answer is unknown, and calculates an output vector that indicates the estimated result. This corresponds to the operation phase. A trained model is portable between computer systems. Therefore, a trained model generated in one computer system can be used in another computer system. Of course, one computer system may perform both the generation and use of a trained model.

In general, a large amount of training data must be prepared to generate a trained model that meets the desired estimation accuracy. Therefore, if annotation is performed manually, it takes a long time to generate training data. The data generation system 1 automatically performs at least a part of the annotation, thereby contributing to efficient generation of training data. Specifically, the data generation system 1 processes a reference record for automatically performing annotation, and automatically generates another record as an additional record. Both the reference record and the additional record are electronic data indicating a combination of an image of a human oral cavity and label information indicating the state of the oral cavity, and can be used as training data for machine learning. Therefore, it can be said that the data generation system 1 is a mechanism for realizing data expansion that automatically increases the number of records of training data. The label information indicates one or more labels. For example, each label of the label information may indicate the state of one or more partial regions that constitute the oral cavity. As a more specific example, the label information may indicate multiple states corresponding to multiple gingival regions (i.e., the state of each gingival region).

Figure 1 is a diagram showing an example of the functional configuration of data generation system 1. Data generation system 1 includes, as functional components, an acquisition unit 11, an estimation unit 12, a selection unit 13, an image generation unit 14, and a record generation unit 15. In one example, data generation system 1 can access a learning database 2, which is a device (storage unit) that stores teacher data. The learning database 2 may be part of data generation system 1, or may belong to a computer system different from data generation system 1. Data generation system 1 can access learning database 2 via a communication network such as the Internet, an intranet, or a WAN (Wide Area Network).

The acquisition unit 11 is a functional element that acquires a reference record. This reference record indicates a combination of a first oral cavity image that captures a person's oral cavity and label information that indicates the state of the oral cavity. The first oral cavity image may capture the entire oral cavity, or may capture a portion of the oral cavity. Examples of a portion of the oral cavity include one or more specific teeth, one or more specific gingival areas (gum areas), the entire or part of the tongue, the entire or part of the palate, etc. However, the range captured in the first oral cavity image is not limited to these.

The estimation unit 12 is a functional element that estimates the state of the oral cavity from the first oral cavity image using each of a plurality of given trained models. It should be noted that the trained models are not machine learning models generated by machine learning using the teacher data prepared by the data generation system 1, but are machine learning models used to generate the teacher data. Each trained model processes an input vector indicating the first oral cavity image to calculate an output vector indicating the state of the oral cavity. That is, each trained model estimates the state of the oral cavity from the first oral cavity image. The trained models may be models for solving regression problems or models for solving classification problems. Each trained model is generated by machine learning using oral cavity images that have been subjected to corresponding image processing as teacher data. Image processing refers to a process of editing an original image to generate a different image. Therefore, the oral cavity image used as the teacher data for the machine learning is not a simple copy of the original oral cavity image, but an image in which visible information such as color, contrast, and lines has been changed from the original oral cavity image. One trained model corresponds to one or more image processing. One image processing may correspond to multiple trained models.

The selection unit 13 is a functional element that selects from a plurality of given trained models a trained model that has obtained an output vector with the highest similarity to the label information of the reference record (in other words, the label information of the first oral cavity image). Similarity refers to the degree to which two comparison objects resemble each other. When the output vector matches the label information, the similarity indicates the highest value (for example, when the similarity is expressed as a numerical range from 0 to 1, the maximum value is 1). In short, the selection unit 13 selects the trained model that has obtained a result closest to the label information (i.e., the trained model with the highest accuracy).

The image generation unit 14 is a functional element that performs image processing corresponding to the selected trained model on the first oral cavity image to generate a second oral cavity image. Therefore, the second oral cavity image is not simply a copy of the first oral cavity image, but an image in which visible information such as color, contrast, and lines has been changed from the first oral cavity image. Like the first oral cavity image, the second oral cavity image is also an image that depicts all or part of a person's oral cavity. The selected trained model corresponds to n types of image processing (n is an integer equal to or greater than 1). The image generation unit 14 performs each of the n types of image processing on the first oral cavity image to generate n second oral cavity images. If n>1, the n second oral cavity images are different from each other.

The record generation unit 15 is a functional element that generates, for each of the at least one generated second intraoral image, an additional record that indicates a combination of the second intraoral image and the label information of the reference record. Therefore, the reference record and the additional record generated based on the reference record have the same label information. The record generation unit 15 outputs the generated at least one additional record.

The data generation method executed by the data generation system 1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing an example of the operation of the data generation system 1 as a process flow S1.

In step S11, the acquisition unit 11 acquires one reference record indicating a combination of a first oral cavity image showing a person's oral cavity and label information indicating the state of the oral cavity. The method of acquiring the reference record is not limited. For example, the acquisition unit 11 may access a given database to read the reference record, may acquire a reference record input by a user, or may receive a reference record sent from another computer. The first oral cavity image may be a still image (photograph). Alternatively, the first oral cavity image may be one frame image constituting a moving image (video) (i.e., an image extracted from a moving image).

In step S12, the estimation unit 12 estimates the state of the oral cavity from the first oral cavity image using each of the multiple trained models. Each trained model is generated in advance by machine learning and stored in a given storage unit. Each trained model may be stored in a memory or database in the data generation system 1, or in a memory or database outside the data generation system 1. For example, it is assumed that a trained model _M1 corresponding to image processing _K1 , a trained model _M2 corresponding to image processing _K2 and _K3 , a trained model _M3 corresponding to image processing _K2 and _K4 , and a trained model _M4 corresponding to image processing _K5 are prepared as multiple trained models. In this case, the estimation unit 12 inputs the first intraoral image into the trained model _M1 to calculate the output vector _V1 , inputs the first intraoral image into the trained model _M2 to calculate the output vector _V2 , inputs the first intraoral image into the trained model _M3 to calculate the output vector _V3 , and inputs the first intraoral image into the trained model _M4 to calculate the output vector _V4 .

In step S13, the selection unit 13 selects a trained model that has obtained an output vector with the highest similarity to the label information of the reference record (first oral cavity image). The method of calculating the similarity (i.e., accuracy) is not limited. For example, the selection unit 13 may calculate the rate at which one or more states (respective states of one or more partial regions) indicated by the output vector match one or more states indicated by the label information as the similarity (accuracy). If there are multiple trained models that have obtained output vectors with the highest similarity to the label information, the selection unit 13 may select the multiple trained models.

In step S14, the image generating unit 14 applies image processing corresponding to the selected trained model to the first intraoral image to generate a second intraoral image. For example, when the trained model _M1 is selected, the image generating unit 14 applies image processing _K1 to the first intraoral image to generate a second intraoral image. As another example, when the trained model _M3 is selected, the image generating unit 14 applies image processing _K2 to the first intraoral image to generate one second intraoral image, and applies image processing _K4 to the first intraoral image to generate another second intraoral image.

The specific method of image processing for generating the second intraoral image (i.e., image processing corresponding to the trained model used by the estimation unit 12) is not limited. For example, the image processing may include at least one of monochromatization, grayscaling, rotation/translation, resizing, and edge processing.

Monochromatization refers to a method of expressing an image by a specific component of a color space, and in this process, information on other components of the color space is deleted from the image. The specific method of monochromatization is not limited. As an example, the image generating unit 14 may perform monochromatization (monochromatization based on the RGB color space) on the first oral cavity image, which expresses an image by a specific component of the RGB color space (one of red, green, and blue), to generate a second oral cavity image. As another example, the image generating unit 14 may perform monochromatization (monochromatization based on the HSV color space), which expresses an image by a specific component of the HSV color space (one of hue, saturation, and value), to generate a second oral cavity image. The color space considered in monochromatization is not limited to the RGB color space and the HSV color space, and the image generating unit 14 may perform monochromatization based on other color spaces.

Grayscaling is a technique for representing an image using only brightness, without including color information. Typically, with grayscaling, each pixel value is represented by 8 bits, and can therefore take any integer value between 0 (black) and 255 (white).

Rotational movement refers to a technique for rotating a subject in an image by an arbitrary angle around an arbitrary reference point. The rotational movement may be a process for rotating the subject on a two-dimensional plane represented by the image. Alternatively, the rotational movement may be a process for making the subject appear to rotate three-dimensionally in the three-dimensional space represented by the image (so-called 3D rotation). The image generating unit 14 performs a rotational movement with a given reference point, a given angle, and a given rotational direction on the first oral cavity image (more specifically, the subject in the first oral cavity image) to generate the second oral cavity image.

Resizing refers to a technique for enlarging or reducing the size of a subject in an image without changing its shape. There are no limitations on the enlargement and reduction ratios, and they may be set according to any policy.

Edge processing refers to a process of identifying as edges the points in an image where pixel values change discontinuously (in other words, the points where pixel values change suddenly) and generating an image that emphasizes those edges. The specific method of edge processing is not limited. For example, the image generation unit 14 may perform edge detection using a differential operator or a Sobel operator.

In step S15, the record generating unit 15 generates an additional record based on each of the generated one or more second intraoral images. Specifically, the record generating unit 15 generates one or more additional records by associating the label information of the reference record with each of the second intraoral images. In other words, the record generating unit 15 performs annotation.

In step S16, the record generation unit 15 outputs the one or more additional records as at least a part of the teacher data for machine learning. In one example, the record generation unit 15 stores the additional records in the training database 2. If the reference record is not stored in the training database 2, the record generation unit 15 may store both the reference record and the one or more additional records in the training database 2. The method of outputting the additional records is not limited to storing them in a database. For example, the record generation unit 15 may display the additional records on a monitor or transmit them to another computer system. In either case, the additional records are used as at least a part of the teacher data for machine learning. In one example, both the reference record and the additional records are used as the teacher data.

The process of generating the second oral cavity image from the first oral cavity image by image processing (step S14) can be said to be a process of generating the second oral cavity image by focusing on a specific feature. The feature is, for example, a specific component of the color space, a grayscale value, a rotation angle, a magnification ratio, a reduction ratio, or an edge. A certain feature is effective in determining a specific condition related to the oral cavity (e.g., diseases such as periodontal disease, caries, and tongue cancer). As some examples of such effectiveness, blue (B) in the RGB color space may be effective in determining periodontal disease, saturation (S) in the HSV color space may be effective in determining caries, and edge processing may be effective in determining healthy gums. Since the image processing used to generate the second oral cavity image corresponds to a trained model that has obtained an output vector with the highest accuracy, the second oral cavity image is eligible to have the same label information as the first oral cavity image. Therefore, even if the second oral cavity image is given the same label information as the first oral cavity image to expand the training data, the quality of the training data can be maintained at or above the desired standard.

When the data generation system 1 processes multiple base records, the data generation system 1 repeatedly executes the processing flow S1 the number of times equal to the number of base records. When the data generation system 1 generates at least one additional record from each base record, it can expand the teacher data by more than twice as much as when the teacher data is constructed from only a group of base records.

The data expansion by the data generation system 1 will be described in more detail with reference to FIG. 3. FIG. 3 shows a specific example of the data expansion.

In this example, the acquisition unit 11 acquires a reference record 30 indicating a combination of a first intraoral image 31 showing the dentition and gingival region (gum region) and label information 32 which is a collection of labels 32a of the individual gingival regions (step S11). Each label 32a is shown diagrammatically in a dashed frame, but in reality each label 32a indicates the state of the gingival region, such as "healthy," "gingivitis," "periodontitis," etc.

The estimation unit 12 inputs the first oral cavity image 31 to each of the trained models to calculate a plurality of output vectors (step S12). In this example, the estimation unit 12 uses five trained models 41 to 45. The trained model 41 is a computational model generated by machine learning using an oral cavity image obtained by monochromatizing with blue in the RGB color space and a grayscaled oral cavity image. The trained model 42 is a computational model generated by machine learning using an oral cavity image obtained by monochromatizing with red in the RGB color space. The trained model 43 is a computational model generated by machine learning using a grayscaled oral cavity image. The trained model 44 is a computational model generated by machine learning using an oral cavity image obtained by edge processing. The trained model 45 is a computational model generated by machine learning using a rotated and moved oral cavity image. It is assumed that all of the trained models 41 to 45 are models for solving a classification problem regarding individual gingival regions. In this example, the estimation unit 12 inputs the first intraoral image 31 to a trained model 41 to calculate an output vector _V41 , inputs the first intraoral image 31 to a trained model 42 to calculate an output vector _V42 , and inputs the first intraoral image 31 to a trained model 43 to calculate an output vector _V43 . Furthermore, the estimation unit 12 inputs the first intraoral image 31 to a trained model 44 to calculate an output vector _V44 , and inputs the first intraoral image 31 to a trained model 45 to calculate an output vector _V45 .

The selection unit 13 selects the trained model that has obtained the output vector with the highest similarity to the label information, that is, the trained vector with the highest estimation accuracy (step S13). In this example, the selection unit 13 determines whether or not the value (vector element) indicated by the output vector matches the label 32a for each gingival region, and calculates the ratio of gingival regions where the two match as the similarity (accuracy). For example, assume that 20 gingival regions are identified from the first oral cavity image 31, and the number of gingival regions where the value indicated by the output vector matches the label 32a is 12. In this case, the selection unit 13 calculates the similarity as 0.6 (=12/20). In the example of FIG. 3, the similarities of the output vectors V ₄₁ , V ₄₂ , V ₄₃ , V ₄₄ , and V ₄₅ are 0.75, 0.66, 0.70, 0.55, and 0.66, respectively. Therefore, the selection unit 13 selects the trained model 41.

The image generating unit 14 applies image processing corresponding to the trained model 41 to the first oral cavity image 31 to generate a second oral cavity image (step S14). The trained model 41 corresponds to two types of image processing: monochromatization using blue in the RGB color space, and grayscaling. In one example, the image generating unit 14 applies monochromatization to the first oral cavity image 31 to generate a second oral cavity image 51, and applies grayscaling to the first oral cavity image 31 to generate a second oral cavity image 61.

The record generation unit 15 generates an additional record based on the generated second intraoral image (step S15). In this example, the record generation unit 15 generates an additional record 50 indicating a combination of the second intraoral image 51 and the label information 32 of the reference record 30, and generates an additional record 60 indicating a combination of the second intraoral image 61 and its label information 32. It should be noted that the label information indicated by the additional record is not the output vector calculated by the estimation unit 12, but the same label information 32 as the reference record 30. The record generation unit 15 outputs the additional records 50 and 60 (step S16).

The selection unit 13 may further select another trained model in addition to the trained model that has obtained an output vector with the highest similarity to the label information of the reference record (first oral cavity image). For example, the selection unit 13 may further select a trained model that has obtained an output vector with the second highest similarity. Alternatively, the selection unit 13 may select at least one trained model whose similarity is equal to or greater than a given threshold.

In this disclosure, the expression "at least one processor executes a first process, executes a second process, ... executes an nth process" or a corresponding expression indicates a concept including cases where the entity executing the n processes from the first process to the nth process (i.e., the processor) changes midway. In other words, this expression indicates a concept including both cases where all n processes are executed by the same processor and cases where the processor changes among the n processes according to an arbitrary policy.

The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and directly or indirectly connected (e.g., using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, regard, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or transmitter. In either case, as mentioned above, there are no particular limitations on the method of realization.

For example, the data generation system 1 in one embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure. FIG. 4 is a diagram showing an example of the hardware configuration of a computer 100 used in the data generation system 1. The computer 100 may physically include a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the term "apparatus" can be interpreted as a circuit, device, unit, etc. The hardware configuration of the data generation system 1 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function in the data generation system 1 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, registers, etc.

The processor 1001 also reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above-mentioned embodiments. For example, each functional element of the data generation system 1 may be stored in the memory 1002 and realized by a control program that runs on the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one structure (e.g., a touch panel).

In addition, each device such as the processor 1001 and memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

The computer 100 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

As described above, a data generation system according to one aspect of the present disclosure includes at least one processor. The at least one processor acquires a reference record indicating a combination of a first oral cavity image of a human oral cavity and label information related to the state of the oral cavity, inputs the first oral cavity image to each of a plurality of trained models that estimate the state of the oral cavity from the oral cavity image, each trained model being generated by machine learning using an oral cavity image that has been subjected to corresponding image processing, calculates a plurality of output vectors, selects from the plurality of trained models a trained model that has obtained an output vector that is most similar to the label information, performs image processing corresponding to the selected trained model on the first oral cavity image to generate a second oral cavity image, generates an additional record indicating a combination of the second oral cavity image and the label information, and outputs the additional record as at least a part of training data for machine learning.

In this aspect, a trained model that has obtained an output vector that is most similar to the label information of the first oral cavity image is selected, and a second oral cavity image is generated from the first oral cavity image by image processing corresponding to the trained model. This second oral cavity image can be said to be qualified to have the same label information as the first oral cavity image. Therefore, by performing annotation using the second oral cavity image, training data that is used to generate a machine learning model that estimates the state of a person's oral cavity can be efficiently generated.

In a data generation system according to another aspect, at least one of the multiple trained models may support monochromatization, which represents an image using one specific component of a color space. The one specific component of a color space may be effective in determining a specific condition related to the oral cavity. Thus, by monochromatization, it is possible to obtain an oral cavity image that is effective as training data.

In a data generation system according to another aspect, at least one of the multiple trained models may support grayscaling. Grayscale images may be useful for identifying specific conditions related to the oral cavity. Thus, grayscaling can provide oral cavity images that are useful as training data.

In a data generation system according to another aspect, at least one of the multiple trained models may support edge processing. Images that have been edge processed may be useful for identifying specific conditions related to the oral cavity. Thus, edge processing can provide oral cavity images that are useful as training data.

In a data generation system according to another aspect, at least one of the multiple trained models may be compatible with multiple types of image processing. In this case, multiple second intraoral images are generated from one first intraoral image in response to the selection of a trained model compatible with multiple types of image processing. Therefore, a larger number of additional records can be generated.

In a data generation system according to another aspect, the label information and the output vector may each indicate a plurality of states corresponding to a plurality of sub-regions in the oral cavity. At least one processor may calculate, as the similarity, the proportion at which the plurality of states indicated by the output vector match the plurality of states indicated by the label information. In this case, the similarity can be obtained more objectively by calculating such a proportion.

In a data generation system according to another aspect, the multiple partial regions may be multiple gingival regions. In this case, it becomes possible to objectively obtain similarity regarding the condition of the gingival region and efficiently generate training data for machine learning that estimates the condition of the gingival region based on the similarity.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended as an illustrative example and does not have any limiting meaning on the present disclosure.

The notification of information is not limited to the aspects/embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination of these. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

Each aspect/embodiment described in this disclosure may be applied to at least one of systems utilizing LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), or other suitable systems, and next generation systems enhanced based on these. Additionally, multiple systems may be combined (for example, a combination of at least one of LTE and LTE-A with 5G, etc.).

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

Specific operations that are described as being performed by a base station in this disclosure may also be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and other network nodes other than the base station (e.g., MME or S-GW, etc., but are not limited to these). Although the above example shows a case where there is one other network node other than the base station, it may also be a combination of multiple other network nodes (e.g., MME and S-GW).

Information, etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). It may also be input/output via multiple network nodes.

The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), then these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that the terms described in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). Also, the signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

As used in this disclosure, the terms "system" and "network" are used interchangeably.

In addition, the information, parameters, etc. described in this disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, radio resources may be indicated by an index.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

In this disclosure, terms such as "base station (BS)", "radio base station", "fixed station", "NodeB", "eNodeB (eNB)", "gNodeB (gNB)", "access point", "transmission point", "reception point", "transmission/reception point", "cell", "sector", "cell group", "carrier", and "component carrier" may be used interchangeably. A base station may also be referred to by terms such as macrocell, small cell, femtocell, and picocell.

A base station can accommodate one or more (e.g., three) cells. When a base station accommodates multiple cells, the entire coverage area of the base station can be divided into multiple smaller areas, and each smaller area can also provide communication services by a base station subsystem (e.g., a small indoor base station (RRH: Remote Radio Head). The term "cell" or "sector" refers to a part or the entire coverage area of at least one of the base station and base station subsystems that provide communication services in this coverage.

In this disclosure, terms such as "Mobile Station (MS)," "user terminal," "User Equipment (UE)," and "terminal" may be used interchangeably.

A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

At least one of the base station and the mobile station may be called a transmitting device, a receiving device, a communication device, etc. At least one of the base station and the mobile station may be a device mounted on a moving body, the moving body itself, etc. The moving body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned moving body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). At least one of the base station and the mobile station may include a device that does not necessarily move during communication operation. For example, at least one of the base station and the mobile station may be an IoT (Internet of Things) device such as a sensor.

In addition, the base station in the present disclosure may be read as a user terminal. For example, each aspect/embodiment of the present disclosure may be applied to a configuration in which communication between a base station and a user terminal is replaced with communication between multiple user terminals (which may be called, for example, D2D (Device-to-Device) or V2X (Vehicle-to-Everything)). In this case, the user terminal may be configured to have the functions of a base station. Furthermore, terms such as "uplink" and "downlink" may be read as terms corresponding to terminal-to-terminal communication (for example, "side"). For example, uplink channel, downlink channel, etc. may be read as side channel.

Similarly, the user terminal in this disclosure may be interpreted as a base station. In this case, the base station may be configured to have the functions that the user terminal has.

The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. "Determining" and "determining" may also include resolving, selecting, choosing, establishing, comparing, and the like. In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Also, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

The terms "connected" and "coupled", or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to elements using designations such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Additionally, the term "or," as used in this disclosure, is not intended to be an exclusive or.

In this disclosure, where articles have been added through translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are in the plural form.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

1...data generation system, 2...learning database, 11...acquisition unit, 12...estimation unit, 13...selection unit, 14...image generation unit, 15...record generation unit, 30...reference record, 31...first oral cavity image, 32...label information, 32a...label, 41-45...trained model, 50...additional record, 51...second oral cavity image, 60...additional record, 61...second oral cavity image.

Claims (7)

At least one processor,
the at least one processor:
obtaining a reference record indicating a combination of a first intraoral image of a person's intraoral cavity and label information related to a state of the intraoral cavity;
a plurality of trained models for estimating a state of the oral cavity from an oral cavity image, each of the trained models being generated by machine learning using a corresponding oral cavity image that has been subjected to image processing; inputting the first oral cavity image into each of the trained models to calculate a plurality of output vectors;
Selecting a trained model from the plurality of trained models that has obtained the output vector having the highest similarity to the label information;
The image processing corresponding to the selected trained model is performed on the first intraoral image to generate a second intraoral image;
generating an additional record indicating a combination of the second intraoral image and the label information;
outputting the additional record as at least a part of training data for machine learning;
Data generation system. At least one of the plurality of trained models corresponds to monochromatization that represents an image by a specific component of a color space.
The data generation system of claim 1 . At least one of the plurality of trained models corresponds to grayscaling;
3. The data generation system according to claim 1 or 2. At least one of the plurality of trained models corresponds to edge processing.
The data generation system according to any one of claims 1 to 3. At least one of the plurality of trained models corresponds to a plurality of types of image processing,
The data generation system according to any one of claims 1 to 4. the label information and the output vector each indicate a plurality of states corresponding to a plurality of partial regions in an oral cavity;
the at least one processor calculates, as the similarity, a rate at which the plurality of states indicated by the output vector coincide with the plurality of states indicated by the label information;
The data generation system according to any one of claims 1 to 5. The plurality of sub-regions are a plurality of gingival regions.
The data generation system of claim 6.

Images