JP7462534B2 - Image Recording System - Google Patents

Description

One aspect of the present disclosure relates to an image recording system.

Patent document 1 describes an intraoral camera that determines the imaging site of image data to be the oral cavity, and if a tooth area is detected in the image data, sets the autofocus area on that area.

JP 2011-110073 A

There is a need for a system that can easily obtain appropriate images for use in oral diagnosis.

An image recording system according to one aspect of the present disclosure includes at least one processor. The at least one processor receives an oral cavity image showing the oral cavity of a subject, identifies an oral cavity region, which is a region showing the oral cavity, from the oral cavity image, obtains a luminance value for each of a plurality of pixels in the oral cavity region, and stores the oral cavity image in a storage device if there are no highly reflective pixels, which are pixels whose luminance values are equal to or greater than a given threshold, and does not store the oral cavity image in a storage device if there are highly reflective pixels.

In this aspect, whether or not to store an oral cavity image in a storage device is determined depending on the magnitude of the luminance value of the pixels that make up the oral cavity region of the oral cavity image. Highly reflective pixels with luminance values higher than a given threshold value are likely to be the result of light reflection by saliva in the oral cavity. Therefore, an oral cavity image containing such highly reflective pixels may not capture the oral cavity well due to undesirable reflection. By recording an oral cavity image only when the luminance value of each pixel that makes up the oral cavity region is within an acceptable range, an appropriate image for use in diagnosing the oral cavity can be easily obtained.

According to one aspect of the present disclosure, it is possible to easily obtain appropriate images for use in diagnosing the oral cavity.

FIG. 1 is a diagram illustrating an example of a configuration of an image recording system according to an embodiment. 4 is a flowchart showing an example of an image recording method executed by the image recording system according to the embodiment. FIG. 13 is a diagram showing an example of a virtual line for determining the angle of the oral cavity. 10 is a flowchart showing another example of an image recording method executed by the image recording system according to the embodiment. 10 is a flowchart illustrating an example of a determination based on whether or not a highly reflective pixel exists. 13 is a flowchart illustrating an example of a process for outputting a guidance message. FIG. 13 is a diagram for explaining a camera direction. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the image recording system according to the embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the drawings, identical or equivalent elements are given the same reference numerals, and duplicate descriptions will be omitted.

The image recording system 1 according to the embodiment is a computer system that stores an image of a person's oral cavity (also referred to as an "oral cavity image" in this disclosure) in a specified storage device. "Storing an image in a storage device" refers to a process of permanently recording electronic data (i.e., image data) of an image at a certain point in time or a certain time span as an electronic file (i.e., an image file) on the storage device. The stored image may be a still image (photograph) or a moving image, and therefore the image data may be still image data (photograph data) or moving image data. In this disclosure, "storing an image in a storage device" is a concept that includes both a process of storing already generated image data (i.e., an image file) in a storage device and capturing an image. Capturing an image refers to a process of generating electronic data (i.e., image data) of an image at a certain point in time or a certain time span and permanently storing (recording) the image data as an image file on the storage device.

The image data may be used for any purpose. For example, the image data may be used for face-to-face or online diagnosis or treatment, or for care such as home nursing. The image data may be used by the image recording system 1, or by other computer systems or computers.

In this disclosure, a person whose oral cavity will be photographed or has already been photographed is referred to as a "subject." Also, a person who uses the image recording system 1 is referred to as a "user." The user and the subject may be the same person, in which case the user (subject) records an image of his or her own oral cavity. Alternatively, the user and the subject may be different people. For example, the user and the subject may have a relationship such as that of a medical professional and a patient, or a caregiver and a person receiving care. Alternatively, the user may be a family member or acquaintance of the subject.

In one example, the image recording system 1 accepts an oral cavity image showing the oral cavity of a subject. The oral cavity image may show a part of the subject's oral cavity (e.g., the jaw). The oral cavity image may be an already generated image file. Alternatively, the oral cavity image may be temporary image data obtained by automatically recording a capture area shown in a camera viewfinder without receiving user operation. In any case, the image recording system 1 determines whether the oral cavity image satisfies given recording conditions.

If the accepted oral cavity image satisfies the recording conditions, the image recording system 1 stores the oral cavity image. As described above, storing an oral cavity image is a concept that includes both saving an existing image file and photographing the oral cavity.

On the other hand, if the accepted oral cavity image does not satisfy the recording conditions, the image recording system 1 does not store the oral cavity image. In one example, the image recording system 1 then accepts a new oral cavity image, determines whether or not the oral cavity image satisfies the recording conditions, and decides whether or not to store the oral cavity image based on the determination result. In short, the image recording system 1 may repeatedly retry until it can store the oral cavity image of the subject.

The oral cavity image processed in the image recording system 1 may be acquired by any method. For example, the oral cavity image may be a still image or a frame image constituting a moving image. Thus, the oral cavity image may be extracted from the moving image.

The image recording system 1 is configured using at least one computer. The specific physical configuration of the image recording system 1 is not limited and may be designed according to any policy.

In one example, the image recording system 1 may be realized by a client-server system including a mobile terminal and a server connected to each other via a communication network. In the client-server system, whether each functional element is implemented on the mobile terminal or on the server may be designed according to any policy. The server side may be a distributed system made up of multiple computers. The configuration of the communication network is not limited and may be designed according to any policy. For example, each communication network may be configured to include at least one of a mobile communication network, the Internet, an intranet, and a WAN (Wide Area Network).

In another example, the image recording system 1 may be realized by a single computer, for example, a single mobile terminal. In this case, all of the above functional elements are implemented on that single computer.

Examples of mobile terminals include high-function mobile phones (smartphones), mobile phones, wearable devices, and laptops. However, specific types of mobile terminals are not limited to these.

Figure 1 shows an example of the configuration of an image recording system 1. The image recording system 1 includes a camera 2 having a viewfinder 3, and a display 4. The viewfinder 3 is a function that allows a user to visually confirm the capture area, which is the area reflected by the lens of the camera 2. The capture area of the camera 2 is, in short, the field of view of the camera 2. Typically, in a mobile terminal, the viewfinder 3 is displayed on the display 4.

When at least a part of the image recording system 1 is realized by a mobile terminal, an application program for recording the oral cavity is pre-installed in the mobile terminal. The mobile terminal starts the application program in response to a user operation, and the application program controls the camera 2, viewfinder 3, or display 4 as necessary.

The image recording system 1 includes an image processing unit 10 as a functional element. The image processing unit 10 is a functional element that executes control related to recording of oral cavity images, and cooperates with the camera 2, the viewfinder 3, or the display 4 as necessary. The image processing unit 10 includes a reception unit 11, a judgment unit 12, a recording unit 13, an image storage unit 14, and a guidance unit 15. The reception unit 11 is a functional element that receives oral cavity images. The judgment unit 12 is a functional element that judges whether the oral cavity image satisfies a given recording condition. The recording unit 13 is a functional element that stores the oral cavity image (more specifically, an image file) of the subject in the image storage unit 14. The image storage unit 14 is a functional element (storage device) that stores the oral cavity images. The guidance unit 15 is a functional element that outputs a guidance message to the user or the subject to obtain an appropriate oral cavity image.

An example of an image recording method executed by the image recording system 1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the example as process flow S1. In process flow S1, the image recording system 1 processes an existing oral cavity image file.

In step S11, the reception unit 11 receives one oral cavity image file. The method of receiving the oral cavity image file is not limited. For example, the reception unit 11 may access a given database to read the oral cavity image file, may obtain an oral cavity image file input by a user, or may receive an oral cavity image file sent from another computer. The first oral cavity image may be a still image (photograph). Alternatively, the first oral cavity image may be one frame image constituting a moving image (video) (i.e., an image extracted from a moving image).

In step S12, the determination unit 12 determines whether the oral cavity image satisfies the recording conditions. In one example, the determination unit 12 may determine at least one of five conditions: image brightness, reflection of light in the image by saliva, camera shake, capture of the oral cavity, and angle of the oral cavity. If the recording conditions are a combination of two or more conditions, the determination unit 12 determines whether the oral cavity image satisfies all of the two or more conditions.

When determining the brightness of an image, the determination unit 12 converts each pixel value of the oral cavity image into a grayscale value. Next, the determination unit 12 calculates the average of all the grayscale values and determines whether the average value is less than a given threshold value Ta. If the average value is less than the threshold value Ta, the determination unit 12 determines that the oral cavity image satisfies the condition regarding the brightness of the image (more specifically, the brightness of the oral cavity image is appropriate). On the other hand, if the average value is equal to or greater than the threshold value Ta, the determination unit 12 determines that the oral cavity image does not satisfy the condition regarding the brightness of the image (more specifically, the brightness of the oral cavity image is not appropriate).

When determining the saliva reflex, the determination unit 12 analyzes the oral cavity image and identifies the area in which the oral cavity is shown as the oral cavity area. In one example, the determination unit 12 estimates the oral cavity area using a trained model Ma for estimating the oral cavity area from the oral cavity image. The trained model Ma is an algorithm that processes an input vector indicating the oral cavity image and generates an output vector indicating the oral cavity area. The trained model Ma is generated by machine learning using training data composed of a large number of records in which data indicating the oral cavity image is associated with a label (correct answer) indicating the oral cavity area, and is implemented in advance in the image recording system 1.

The determination unit 12 acquires the luminance value of each of the pixels constituting the estimated oral cavity region, and determines whether or not there is a pixel whose luminance value is equal to or greater than a given threshold value Tb (also referred to as a "highly reflective pixel" in this disclosure). This highly reflective pixel indicates that light reflection by saliva has occurred. In one example, the determination unit 12 converts each pixel value of the oral cavity image or oral cavity region into a grayscale value, and acquires the grayscale value as the luminance value. In one example, the determination unit 12 identifies a set of multiple highly reflective pixels that exist consecutively as a reflective region. The existence of multiple highly reflective pixels consecutively refers to a situation in which, when any one of the multiple highly reflective pixels is selected as a reference pixel, another highly reflective pixel is located next to the reference pixel in at least one direction (e.g., horizontal, vertical, or diagonal) extending radially from the reference pixel. Therefore, in the reflective region, each highly reflective pixel is located next to one or more other highly reflective pixels.

In one example, if there is no highly reflective pixel, the determination unit 12 determines that the oral image satisfies the condition for reflection by saliva (more specifically, the reflection by saliva does not exist in the oral image). On the other hand, if there is a highly reflective pixel, the determination unit 12 determines that the oral image does not satisfy the condition for reflection by saliva (more specifically, the reflection by saliva exists in the oral image). In another example, if there is no reflection area (i.e., a set of multiple highly reflective pixels that exist in succession), the determination unit 12 determines that the oral image satisfies the condition for reflection by saliva (more specifically, the reflection by saliva does not exist in the oral image). On the other hand, if the reflection area exists, the determination unit 12 determines that the oral image does not satisfy the condition for reflection by saliva (more specifically, the reflection by saliva exists in the oral image). In this way, the determination unit 12 may perform a determination based on a certain highly reflective pixel, or may perform a determination based on a reflection area that spreads two-dimensionally.

When determining whether camera shake has occurred, the determination unit 12 acquires the acceleration of the camera 2 from the acceleration sensor of the mobile terminal. The determination unit 12 then determines whether the acceleration is less than a given threshold Tc. If the acceleration is less than the threshold Tc, the determination unit 12 determines that the oral cavity image satisfies the condition related to camera shake (more specifically, the oral cavity image is not blurred). On the other hand, if the acceleration is equal to or greater than the threshold Tc, the determination unit 12 determines that the oral cavity image does not satisfy the condition related to camera shake (more specifically, the oral cavity image is blurred).

When determining whether the oral cavity is captured, the determination unit 12 analyzes the oral cavity image to determine whether the entire open oral cavity (i.e., the entire area inside the lips in the oral cavity image) is captured. In one example, the determination unit 12 determines the number of teeth of the subject captured in the oral cavity image and determines whether the number of teeth is equal to or greater than a given threshold Td. The determination unit 12 may determine the number of teeth using a trained model. The determination unit 12 estimates the number of teeth using a trained model Mb for estimating the number of teeth from the oral cavity image. The trained model Mb is an algorithm that processes an input vector indicating the oral cavity image and generates an output vector indicating the number of teeth. The trained model Mb is generated by machine learning using teacher data composed of a large number of records in which data indicating the oral cavity image and labels (correct answers) indicating the number of teeth are associated with each other, and is pre-implemented in the image recording system 1. If the number of teeth is equal to or greater than the threshold Td, the determination unit 12 determines that the oral cavity image satisfies the condition for capturing the oral cavity (more specifically, the entire oral cavity is captured in the oral cavity image). On the other hand, if the number of teeth is less than the threshold value Td, the determination unit 12 determines that the oral cavity image does not satisfy the conditions for capturing the oral cavity (more specifically, the entire oral cavity is not shown in the oral cavity image).

When determining the angle of the oral cavity, the determination unit 12 analyzes the oral cavity image to identify the two-dimensional coordinates of each of the two central incisors. The two-dimensional coordinates are coordinates on the oral cavity image, which is a two-dimensional plane. In one example, the determination unit 12 estimates the two-dimensional coordinates of each central incisor using a trained model Mc for estimating the two-dimensional coordinates of each central incisor from the oral cavity image. The trained model Mc is an algorithm that processes an input vector indicating the oral cavity image and generates an output vector indicating the two-dimensional coordinates of each central incisor. The trained model Mc is generated by machine learning using teacher data composed of a large number of records in which data indicating the oral cavity image is associated with labels (correct answers) indicating the two-dimensional coordinates of each central incisor, and is pre-implemented in the image recording system 1. The determination unit 12 generates a virtual line connecting the two-dimensional coordinates of the two central incisors, and calculates the angle between the virtual line and the outer edge of the oral cavity image corresponding to the virtual line. The method of calculating the angle is not limited. For example, the determination unit 12 may generate either a virtual line passing through two left central incisors or a virtual line passing through two right central incisors, and calculate the angle between this virtual line and the vertical outer edge of the oral cavity image. In this case, the vertical outer edge corresponds to the "outer edge of the oral cavity image corresponding to the virtual line". Alternatively, the determination unit 12 may generate either a virtual line passing through two upper central incisors or a virtual line passing through two lower central incisors, and calculate the angle between this virtual line and the horizontal outer edge of the oral cavity image. In this case, the horizontal outer edge corresponds to the "outer edge of the oral cavity image corresponding to the virtual line". The determination unit 12 determines whether the angle is less than a given threshold value Te. If the calculated angle is less than the threshold value Te, the determination unit 12 determines that the oral cavity image satisfies the condition regarding the oral cavity angle (more specifically, the row of teeth is reflected along the horizontal direction of the oral cavity image). On the other hand, if the calculated angle is equal to or greater than the threshold Te, the determination unit 12 determines that the oral cavity image does not satisfy the condition regarding the oral cavity angle (more specifically, the dentition is shown at an angle).

Figure 3 shows examples of virtual lines for determining the angle of the oral cavity. In example (a), the determination unit 12 shows a virtual line 211 passing through two-dimensional coordinates 201 of the maxillary left central incisor and two-dimensional coordinates 202 of the mandibular left central incisor. The determination unit 12 calculates the angle between the virtual line 211 and the vertical outer edge of the oral cavity image, and compares this angle with a threshold value Te. In example (b), the determination unit 12 shows a virtual line 212 passing through two-dimensional coordinates 203 of the maxillary right central incisor and two-dimensional coordinates 204 of the maxillary left central incisor. The determination unit 12 calculates the angle between the virtual line 212 and the horizontal outer edge of the oral cavity image, and compares this angle with a threshold value Te.

As described above, in this embodiment, five specific recording conditions are shown. The condition to be met regarding image brightness is that "the brightness of the oral cavity image is appropriate." The condition to be met regarding saliva reflection is that "there is no saliva reflection in the oral cavity image." The condition to be met regarding camera shake is that "the oral cavity image is not blurred." The condition to be met regarding capture of the oral cavity is that "the entire oral cavity is shown in the oral cavity image." The condition to be met regarding the angle of the oral cavity is that "the row of teeth is shown along the horizontal direction of the oral cavity image."

Returning to FIG. 2, if the oral cavity image satisfies the recording conditions (YES in step S13), the process proceeds to step S14. As described above, if the recording conditions are a combination of two or more conditions, the process proceeds to step S14 if the oral cavity image satisfies all of the two or more conditions. In step S14, the recording unit 13 stores the oral cavity image file in the image storage unit 14.

On the other hand, if the oral cavity image does not satisfy the recording condition (NO in step S13), the process proceeds to step S15. In step S15, the recording unit 13 discards the oral cavity image file without storing it in the image storage unit 14. In step S15, the guidance unit 15 may output a guidance message to the user or the subject. This guidance message is information for prompting the user or the subject to input an appropriate oral cavity image file, and is set based on the unsatisfied recording condition. The guidance unit 15 may display the guidance message on the display 4 or output it from the speaker. The method of expressing the guidance message is not limited, and for example, the guidance message may be expressed by at least one of text, graphics, and sound. For example, in response to the condition regarding the capture of the oral cavity not being satisfied, the guidance unit 15 may output a guidance message saying, "The right side of the mouth is not visible. Please enter a photo that shows the entire mouth."

With reference to Figure 4, another example of an image recording method executed by the image recording system 1 will be described. Figure 4 is a flowchart showing that example as process flow S2. In process flow S2, the image recording system 1 processes the image displayed on the viewfinder 3.

In step S21, the application program launches camera 2. In one example, the application program launches camera 2 in response to a user launching a specific application program.

In step S22, the reception unit 11 receives the oral cavity image. The user aims the camera 2 at the subject so that the subject's face fits within the viewfinder 3. The reception unit 11 automatically records the subject's face shown in the viewfinder 3 without receiving any user operation, and acquires the oral cavity image.

In step S23, the determination unit 12 determines whether the oral cavity image satisfies the recording conditions. Step S23 is the same as step S12, except that the determination is made on image data obtained from the viewfinder 3, rather than on an existing oral cavity image file.

If the oral cavity image satisfies the recording conditions (YES in step S24), processing proceeds to step S25. If the recording conditions are a combination of two or more conditions, processing proceeds to step S25 if the oral cavity image satisfies all of the two or more conditions.

In step S25, the recording unit 13 takes an image of the oral cavity, generates an oral cavity image file, and stores the oral cavity image file in the image storage unit 14. In one example, the recording unit 13 may automatically release the electronic shutter in response to the oral cavity image satisfying the recording conditions without accepting a user operation, and generate photo data showing the oral cavity. Alternatively, the recording unit 13 may automatically start recording in response to the oral cavity image satisfying the recording conditions without accepting a user operation, and generate video data showing the oral cavity. In another example, the recording unit 13 may output a message to prompt the user to perform an operation for shooting in response to the oral cavity image satisfying the recording conditions, and release the electronic shutter according to the shooting operation of the user who responded to the message, and generate photo data showing the oral cavity. Alternatively, the recording unit 13 may output a message to prompt the user to perform an operation for shooting in response to the oral cavity image satisfying the recording conditions, and start recording according to the shooting operation of the user who responded to the message, and generate video data showing the oral cavity. In this way, the oral cavity image may be automatically or manually shot. In either case, the recording unit 13 stores the generated image data in the image storage unit 14 as an oral cavity image file.

On the other hand, if the oral cavity image does not satisfy the recording conditions (NO in step S24), the process proceeds to step S26. As described above, if the recording conditions are a combination of two or more conditions, the process proceeds to step S26 if the oral cavity image does not satisfy at least one of the conditions.

In step S26, the guidance unit 15 outputs a guidance message to the user or the subject. This guidance message is information for encouraging the user or the subject to properly capture the oral cavity within the viewfinder 3 (the capture area of the camera), and is set based on the unsatisfied recording condition. As in step S15, the output method and expression method of the guidance message are not limited. For example, in response to the fact that the condition for capturing the oral cavity is not satisfied, the guidance unit 15 may output a guidance message including the sentence "The right side of the mouth is not captured" and an arrow for directing the camera 2 in a desired direction. After step S26, the process returns to step S22. In step S22, the reception unit 11 receives a new oral cavity image, and in step S23, the judgment unit 12 judges whether the oral cavity image satisfies the recording condition. Then, depending on the judgment result, in step S25, the recording unit 13 takes an oral cavity image, or in step S26, the guidance unit 15 outputs a guidance message. If the new oral cavity image satisfies the recording condition, the recording unit 13 may output a message for stopping the facial movement of the subject (for example, "Please stop your face").

An example of a judgment using the recording conditions (steps S12 and S23) will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of a judgment based on whether or not a highly reflective pixel exists (i.e., an example of judging reflection due to saliva) as process flow S3.

In step S31, the determination unit 12 analyzes the oral cavity image to identify the oral cavity area. As described above, the determination unit 12 may estimate the oral cavity area using a trained model.

In step S32, the determination unit 12 acquires the luminance value of each of the multiple pixels in the oral cavity region. In one example, the determination unit 12 converts each pixel value of the oral cavity image or the oral cavity region into a grayscale value, and acquires the grayscale value as the luminance value.

In step S33, the determination unit 12 determines whether or not a highly reflective pixel is present in the oral cavity region. The determination unit 12 may determine whether or not a reflective region (a collection of multiple contiguous highly reflective pixels) is present.

If highly reflective pixels are present (or if reflective areas are present) (YES in step S33), the process proceeds to step S34. In step S34, the determination unit 12 determines that the oral cavity image does not satisfy the recording conditions.

On the other hand, if there are no highly reflective pixels (or no reflective areas) (NO in step S33), the process proceeds to step S35. In step S35, the determination unit 12 determines that the oral cavity image satisfies the recording conditions.

With reference to Figure 6, an example of outputting a guidance message (step S26) to change the orientation of the subject's face relative to the viewfinder 3 will be described. Figure 6 is a flowchart showing, as process flow S4, an example of processing to output the guidance message in response to the presence of a reflective area (a collection of multiple contiguous highly reflective pixels).

In step S41, the guide unit 15 estimates the reflection characteristics of light in the reflection area. The reflection characteristics refer to the type of light reflection. In one example, the reflection characteristics of a certain reflection area are either specular reflection or diffuse reflection. Specular reflection refers to reflection in which the angle of incidence and the angle of reflection are equal on the reflection surface. Diffuse reflection refers to the phenomenon in which incident light is reflected in various directions on the reflection surface. In general, specular reflection occurs on smooth surfaces, and diffuse reflection occurs on rough surfaces. In one example, the guide unit 15 estimates the reflection characteristics by analyzing an image showing the reflection area (also referred to as a "reflection area image" in this disclosure).

In one example, the guide unit 15 may estimate the part of the reflection area using a trained model Mx for estimating the part from the reflection area image, and determine the reflection characteristics corresponding to the part. The trained model Mx is an algorithm that processes an input vector indicating an image of a part related to the oral cavity, such as teeth, gingiva (gums), tongue, and lips, and generates an output vector indicating the part. The trained model Mx is generated by machine learning using teacher data composed of a large number of records in which data indicating an image of a part related to the oral cavity is associated with a label (correct answer) indicating the part, and is pre-implemented in the image recording system 1. In one example, in response to the estimation of teeth or gingiva (gums), the guide unit 15 determines that the reflection characteristics of the reflection area are specular reflection, and in response to the estimation of tongue or lips, determines that the reflection characteristics are diffuse reflection.

Alternatively, the guide unit 15 may estimate the reflection characteristics using a trained model My for estimating the reflection characteristics from the reflection area image. The trained model My is an algorithm that processes an input vector indicating an image of the reflection in the oral cavity to generate an output vector indicating the reflection characteristics (specular reflection or diffuse reflection). The trained model My is generated by machine learning using training data consisting of a large number of records in which data indicating the image of the reflection in the oral cavity is associated with a label (correct answer) indicating the reflection characteristics, and is pre-implemented in the image recording system 1.

Alternatively, the guide unit 15 may perform edge processing on the reflection area image to calculate feature amounts related to edges, and determine the reflection characteristics based on the feature amounts. Edge processing refers to processing for identifying, as edges, locations in an image where pixel values change discontinuously (in other words, locations where pixel values change abruptly). The guide unit 15 may perform edge detection using a differential operator, or may perform edge detection using a Sobel operator. The guide unit 15 estimates the reflection characteristics based on the feature amounts calculated by the edge processing. In one example, the guide unit 15 calculates a feature amount that is relatively low for edges, and calculates a feature amount that is relatively high for flat areas (in other words, areas that are not edges). In this case, the guide unit 15 estimates that the reflection characteristics are specular reflection if the feature amount is equal to or greater than a given threshold, and estimates that the reflection characteristics are diffuse reflection if the feature amount is less than the threshold.

In step S42, the guidance unit 15 determines a guidance direction based on the identified reflection characteristics. The guidance direction refers to the direction in which the subject's face shown in the viewfinder 3 should be turned.

When the reflection characteristic is estimated to be a specular reflection, the guide unit 15 estimates, for example, the direction from the reflection area to the camera as the camera direction, and sets the opposite direction to the camera direction as the guide direction. In one example, the guide unit 15 estimates the camera direction as follows. That is, the guide unit 15 estimates the two-dimensional coordinates of each central incisor using the above-mentioned trained model Mc for estimating the two-dimensional coordinates of each central incisor from the oral cavity image. Then, the guide unit 15 sets three-dimensional coordinates with the center of the oral cavity as the origin based on those two-dimensional coordinates. The three-dimensional coordinates are defined by setting the direction along the row of teeth (for example, the horizontal direction) as the X axis, the axis perpendicular to the X axis and dividing the left and right central incisors as the Y axis, and the axis perpendicular to both the X and Y axes as the Z axis. The guide unit 15 acquires the position obtained by projecting the center of the lens of the camera 2 in the three-dimensional coordinates onto the XY plane as the camera position. Then, the guide unit 15 specifies the direction from the coordinate origin on the XY plane toward the camera position. Since the coordinate origin is close to the position of the reflection area on the XY plane, the direction of the camera position as viewed from the coordinate origin is similar to the direction of the camera position as viewed from the reflection area. Therefore, the guidance unit 15 specifies the direction of the camera position as viewed from the coordinate origin as the direction of the camera position as viewed from the reflection area (camera direction). The guidance unit 15 sets the direction opposite to that camera direction (for example, a direction 180° or nearly 180° away from the camera direction) as the guidance direction.

Figure 7 is a diagram for explaining the camera direction. In the example of Figure 7, the camera 2 captures the subject 9 from the lower right side of the subject 9. In this example, the guidance unit 15 acquires the camera position 301 located in the third quadrant of the XY coordinate system, and specifies the direction 311 from the coordinate origin to the camera position 301 as the camera direction from the representative point 302 of the reflection area to the camera position 301. Here, the representative point 302 indicates the position of any one pixel in the reflection area. Since the coordinate origin is close to the representative point 302, it can be considered that the assumed camera direction 311 is approximate to the actual camera direction 312. Please note that the guidance unit 15 specifies the direction 311 as the camera direction, not the direction 312. The guidance unit 15 sets the direction opposite to the camera direction 311 (for example, a direction 180° or nearly 180° away from the camera direction 311) as the guidance direction.

If the reflection characteristics are estimated to be diffuse reflection, the guidance unit 15, for example, sets the guidance direction randomly.

Returning to FIG. 6, in step S43, the guidance unit 15 outputs a guidance message based on the determined guidance direction. For example, the guidance unit 15 outputs a guidance message such as "Please look a little further to the left."

As described above, the image recording system 1 may use a trained model. In this case, the trained model may be generated by the image recording system 1, or may be generated by a computer system different from the image recording system 1.

As described above, the image recording system 1 may use a trained model. In this case, the trained model may be generated by the image recording system 1, or may be generated by a computer system different from the image recording system 1.

Camera 2 (and viewfinder 3) may be prepared independently of the mobile terminal. For example, a single-lens reflex camera may be used as camera 2 (and viewfinder 3). In this case, the above embodiment may be realized using images obtained from the single-lens reflex camera and data obtained from various sensors of the mobile terminal.

In this disclosure, the expression "at least one processor executes a first process, executes a second process, ... executes an nth process" or a corresponding expression indicates a concept including cases where the entity executing the n processes from the first process to the nth process (i.e., the processor) changes midway. In other words, this expression indicates a concept including both cases where all n processes are executed by the same processor and cases where the processor changes among the n processes according to an arbitrary policy.

The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and directly or indirectly connected (e.g., using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, regard, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or transmitter. In either case, as mentioned above, there are no particular limitations on the method of realization.

For example, the image recording system 1 in one embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure. FIG. 8 is a diagram showing an example of the hardware configuration of a computer 100 used in the image recording system 1. The computer 100 may physically include a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the term "apparatus" can be interpreted as a circuit, device, unit, etc. The hardware configuration of the image recording system 1 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function in the image recording system 1 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) that includes an interface with peripheral devices, a control device, an arithmetic unit, registers, etc.

The processor 1001 also reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above-mentioned embodiments. For example, each functional element of the image recording system 1 may be realized by a control program stored in the memory 1002 and running on the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one structure (e.g., a touch panel).

In addition, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

The computer 100 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

As described above, an image recording system according to one aspect of the present disclosure includes at least one processor. The at least one processor receives an oral cavity image showing the oral cavity of a subject, identifies an oral cavity region, which is a region showing the oral cavity, from the oral cavity image, obtains a luminance value for each of a plurality of pixels in the oral cavity region, and stores the oral cavity image in a storage device if there are no highly reflective pixels, which are pixels whose luminance values are equal to or greater than a given threshold, and does not store the oral cavity image in a storage device if there are highly reflective pixels.

In this aspect, whether or not to store an oral cavity image in a storage device is determined depending on the magnitude of the luminance value of the pixels that make up the oral cavity region of the oral cavity image. Highly reflective pixels with luminance values higher than a given threshold value are likely to be the result of light reflection by saliva in the oral cavity. Therefore, an oral cavity image containing such highly reflective pixels may not capture the oral cavity well due to undesirable reflection. By recording an oral cavity image only when the luminance value of each pixel that makes up the oral cavity region is within an acceptable range, an appropriate image for use in diagnosing the oral cavity can be easily obtained.

In another aspect of the image recording system, at least one processor accepts an image of the oral cavity showing the oral cavity captured within the capture area of the camera, and if no highly reflective pixels are present, captures an image of the oral cavity to generate an oral cavity image file, stores the oral cavity image file in a storage device, and if highly reflective pixels are present, outputs a guidance message to change the orientation of the subject's face relative to the capture area. In this case, the oral cavity is photographed only if no highly reflective pixels are present, and if highly reflective pixels are present, guidance is output to enable the oral cavity to be photographed. Thus, an appropriate image to be used for diagnosing the oral cavity can be easily captured.

In an image recording system according to another aspect, when a reflective area is present that is a collection of multiple contiguous highly reflective pixels, at least one processor may estimate the light reflection characteristics in the reflective area, determine a guidance direction in which the subject's face reflected in the capture area should be turned based on the reflection characteristics, and output a guidance message based on the guidance direction. In this case, the characteristics of the reflection generated in the oral cavity are estimated, and the direction in which the subject's face should be turned is determined based on the reflection characteristics. This process allows appropriate guidance on the orientation of the subject's face in order to acquire an oral cavity image.

In an image recording system according to another aspect, the recording conditions may be such that at least one processor analyzes a reflection area image, which is an image showing a reflection area, to estimate whether the reflection characteristic is specular reflection or not, and if it estimates that the reflection characteristic is specular reflection, estimates the direction from the reflection area to the camera as the camera direction, and sets the direction opposite to the camera direction as the guidance direction. This series of processes makes it possible to appropriately guide the subject's face direction so that specular reflection does not appear.

In the image recording system according to another aspect, at least one processor may analyze a reflection area image, which is an image showing a reflection area, to estimate whether the reflection characteristic is diffuse reflection or not, and may randomly set the guide direction if it estimates that the reflection characteristic is diffuse reflection. In this case, the process for determining the direction of the subject's face so that diffuse reflection does not appear can be realized by a simple random decision. Therefore, the computational load on the image recording system for this process can be reduced.

In the image recording system according to another aspect, at least one processor may analyze the reflection area image, estimate the location of the reflection area, and estimate the reflection characteristics based on the location. Since the reflection characteristics may depend on the location in the oral cavity, the series of processes allows the reflection characteristics to be estimated with high accuracy.

In an image recording system according to another aspect, at least one processor may perform edge processing on the reflection area image to calculate edge feature quantities and estimate the reflection characteristics based on the feature quantities. Since the feature quantities reflect the smoothness of the reflection area, they are also related to the reflection characteristics. Therefore, the reflection characteristics can be estimated with high accuracy by taking the feature quantities into consideration.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended as an illustrative example and does not have any limiting meaning on the present disclosure.

The notification of information is not limited to the aspects/embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination of these. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

Each aspect/embodiment described in this disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other suitable systems, and next-generation systems extended based on these. In addition, multiple systems may be combined (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.).

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

Specific operations that are described as being performed by a base station in this disclosure may also be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and other network nodes other than the base station (e.g., MME or S-GW, etc., but are not limited to these). Although the above example shows a case where there is one other network node other than the base station, it may also be a combination of multiple other network nodes (e.g., MME and S-GW).

Information, etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). It may also be input/output via multiple network nodes.

The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean (true or false) value, or a comparison of numerical values (e.g., a comparison with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), then these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that the terms described in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). Also, the signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

As used in this disclosure, the terms "system" and "network" are used interchangeably.

In addition, the information, parameters, etc. described in this disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, radio resources may be indicated by an index.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

In this disclosure, terms such as "base station (BS)", "radio base station", "fixed station", "NodeB", "eNodeB (eNB)", "gNodeB (gNB)", "access point", "transmission point", "reception point", "transmission/reception point", "cell", "sector", "cell group", "carrier", and "component carrier" may be used interchangeably. A base station may also be referred to by terms such as macrocell, small cell, femtocell, and picocell.

A base station can accommodate one or more (e.g., three) cells. When a base station accommodates multiple cells, the entire coverage area of the base station can be divided into multiple smaller areas, and each smaller area can also provide communication services by a base station subsystem (e.g., a small indoor base station (RRH: Remote Radio Head). The term "cell" or "sector" refers to a part or the entire coverage area of at least one of the base station and base station subsystems that provide communication services in this coverage.

In this disclosure, terms such as "Mobile Station (MS)," "user terminal," "User Equipment (UE)," and "terminal" may be used interchangeably.

A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

At least one of the base station and the mobile station may be called a transmitting device, a receiving device, a communication device, etc. At least one of the base station and the mobile station may be a device mounted on a moving body, the moving body itself, etc. The moving body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned moving body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). At least one of the base station and the mobile station may include a device that does not necessarily move during communication operation. For example, at least one of the base station and the mobile station may be an IoT (Internet of Things) device such as a sensor.

In addition, the base station in the present disclosure may be read as a user terminal. For example, each aspect/embodiment of the present disclosure may be applied to a configuration in which communication between a base station and a user terminal is replaced with communication between multiple user terminals (which may be called, for example, D2D (Device-to-Device) or V2X (Vehicle-to-Everything)). In this case, the user terminal may be configured to have the functions of a base station. Furthermore, terms such as "uplink" and "downlink" may be read as terms corresponding to terminal-to-terminal communication (for example, "side"). For example, uplink channel, downlink channel, etc. may be read as side channel.

Similarly, the user terminal in this disclosure may be interpreted as a base station. In this case, the base station may be configured to have the functions that the user terminal has.

The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. "Determining" and "determining" may also include resolving, selecting, choosing, establishing, comparing, and the like. In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Also, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

The terms "connected" and "coupled", or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to elements using designations such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Additionally, the term "or," as used in this disclosure, is not intended to be an exclusive or.

In this disclosure, where articles have been added through translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are in the plural form.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

1...image recording system, 2...camera, 3...viewfinder, 4...display, 9...subject, 10...image processing unit, 11...reception unit, 12...determination unit, 13...recording unit, 14...image storage unit, 15...guidance unit, 301...camera position, 302...representative point of reflection area, 311...assumed camera direction, 312...actual camera direction.

Claims (7)

At least one processor;
the at least one processor:
accepting an oral cavity image of the subject's oral cavity;
Identifying an oral cavity area, which is an area showing the oral cavity, from the oral cavity image;
Obtaining a luminance value for each of a plurality of pixels in the oral cavity region;
If there is no highly reflective pixel, which is a pixel whose luminance value is equal to or greater than a given threshold value, the image of the oral cavity is stored in a storage device.
When the highly reflective pixel is present, the image of the intraoral cavity is not stored in the storage device.
Image recording system. the at least one processor:
receiving an image of the oral cavity showing the oral cavity within a capture area of a camera;
If the highly reflective pixel is not present, an image of the oral cavity is captured to generate an oral cavity image file, and the oral cavity image file is stored in the storage device;
outputting a guidance message for instructing the subject to change the orientation of his/her face relative to the capture area when the highly reflective pixel is present;
2. The image recording system according to claim 1. the at least one processor:
When a reflective area is present that is a collection of a plurality of the highly reflective pixels that are consecutively present, the reflective characteristics of light in the reflective area are estimated;
determining a guide direction, which is a direction in which the face of the subject reflected in the capture area should be turned, based on the reflection characteristics;
outputting the guidance message based on the guidance direction;
3. The image recording system according to claim 2. the at least one processor:
Analyzing a reflection area image, which is an image showing the reflection area, to estimate whether the reflection characteristic is a specular reflection;
If the reflection characteristic is estimated to be the specular reflection, a direction from the reflection area to the camera is estimated as a camera direction;
A direction opposite to the camera direction is set as the guide direction.
4. The image recording system according to claim 3. the at least one processor:
Analyzing a reflection area image, which is an image showing the reflection area, to estimate whether the reflection characteristic is diffuse reflection;
When it is estimated that the reflection characteristic is diffuse reflection, the guiding direction is set randomly.
5. The image recording system according to claim 3 or 4. the at least one processor analyzes the reflection area image to estimate a location of the reflection area, and estimates the reflection characteristic based on the location.
6. An image recording system according to claim 4 or 5. The at least one processor performs edge processing on the reflection area image to calculate feature amounts related to edges, and estimates the reflection characteristics based on the feature amounts.
6. An image recording system according to claim 4 or 5.

Images