JP7690684B2 - OBJECT IDENTIFICATION DEVICE, OBJECT IDENTIFICATION METHOD, PROGRAM, AND RECORDING MEDIUM - Google Patents

Description

The present disclosure relates to an object identification device, an object identification method and a program, and in particular to image recognition technology for identifying objects from images.

Patent document 1 describes drug identification software that places the drug to be identified in a drug imaging device, photographs the drug, and searches for the drug by referring to a database based on the data from the photographed drug image.

Patent Document 2 describes a tablet detection method that includes an imaging step of capturing a reflected light image and a transmitted light image of a packaging paper in which one or more tablets for one dose are packaged; an excision step of excising a tablet region in the reflected light image, which is a region corresponding to the tablet, based on the reflected light image and the transmitted light image; a first identification step of identifying the type of each tablet by comparing the dimensions and color of each tablet region excised by the excision step with model information regarding the shape and color of the tablet; and a second identification step of identifying the types of at least similar tablets that are different types but have similar features, based on a learning model generated by executing machine learning using learning data including the tablet images.

JP 2022-010060 A JP 2018-027242 A

When identifying the type of a tablet, the type of drug (drug type) can often be identified using the information on the characters and/or symbols engraved or printed on the tablet. On the other hand, for drugs such as capsule drugs and half tablets, it is difficult or impossible to identify the drug type by template matching between the master image and the captured image or machine learning-based methods because only a portion of the character and symbol information for identifying the drug type is captured in the captured image, or there are countless patterns of capsule interlocking and tablet division. In other words, there are drugs whose type can be identified from the captured image using image recognition technology, such as tablets with engraved or printed characters, and drugs whose type is difficult to identify from the captured image, such as capsule drugs and half tablets. In this specification, the former are called "drugs whose type can be identified" and the latter are called "drugs whose type is difficult to identify". In addition, letters and/or symbols attached to drugs by engraving or printing are referred to as "character symbols".

It is expected that the drug type of drugs that can be identified will be identified by image recognition based on the captured image. On the other hand, in order to identify the drug type of drugs that are difficult to identify, for example, the captured image may be enlarged and information on the character symbols attached to the drug may be presented to the user, and the user may visually check the character symbols that are partially visible and input the text or voice information, thereby searching a database such as an engraving text master, thereby making it possible to identify the drug type. In this way, with regard to drugs that are difficult to identify, a method that supports the drug type identification task by presenting the user with information useful for identifying the drug type is considered to be practically effective.

It is also possible to construct a system that distinguishes between identifiable drugs and drugs with difficult drug identification methods at the stage of photographing the drugs to be identified, photographs only identifiable drugs, and identifies the type of identifiable drug.

However, from a pharmaceutical and usability perspective, it would be preferable to be able to photograph multiple medications at once, in units of the same dosage time or same dispensing, and to be able to identify the drug type all at once.

However, when photographs are taken in units such as the same time of ingestion, a single photographed image will inevitably contain a mixture of drugs that are difficult to identify and drugs that can be identified, making it necessary to select for each drug whether it is a drug that is difficult to identify or a drug that can be identified. From the viewpoint of usability, it is preferable that this selection be performed automatically on the drug type identification device as far as possible. It is also preferable that this selection be performed within a series of drug type identification work flows on the drug type identification device.

These technical issues are not limited to applications in identifying medicines, but are recognized as common issues when identifying the type of various objects from their images. In this specification, an object whose type can be identified from its image is referred to as an "identifiable type object," and an object whose type is difficult to identify from its image but whose group to which it belongs can be identified is referred to as an "difficult-to-identify type object."

The present disclosure has been made in consideration of these circumstances, and aims to provide an object identification device, an object identification method, and a program that make it possible to identify the type of each object using an image of multiple objects, which may include a mixture of objects whose type can be identified and objects whose type is difficult to identify.

An object identification device according to one aspect of the present disclosure includes a detector that detects objects on an object-by-object basis from an image in which multiple objects are captured; a classifier that estimates the type of object from the image for type-identifiable objects among the objects detected by the detector, where the type of object can be identified from the image; and a classifier that estimates the group from the image for type-difficult objects where the type of object is difficult to identify from the image but the group to which the object belongs can be identified; and a processing unit that performs group-specific processing leading to the identification of the type of objects estimated as a group by the classifier.

According to this aspect, objects are detected from an image in which multiple objects are captured, and each object is estimated by a classifier. If an object detected from an image is an object whose type can be identified, the classifier estimates the type of the object. On the other hand, if an object detected from an image is an object whose type is difficult to identify, the classifier estimates the group to which the object belongs, and for objects whose groups have been estimated, the process proceeds to group-specific processing that contributes to identifying the type of object, leading to the identification of the type.

The image may be an image captured in a state in which identifiable objects and difficult-to-identify objects are mixed.

Objects whose types are difficult to identify may be classified into multiple groups, with group-specific processing defined for each group.

The detector may be configured to include a first learned model trained by machine learning using first training data that is labeled on an object-by-object basis without distinguishing between objects whose type can be identified and objects whose type is difficult to identify.

The classifier may be configured to include a second learned model trained by machine learning using second training data in which objects whose types can be identified are labeled by object type, and objects whose types are difficult to identify are labeled by group to which the objects belong.

Labels identifying groups may be defined in a hierarchical structure.

The input to the classifier may be at least one of an object image obtained by cutting out the area of an object detected by a detector from an image on an object-by-object basis, a character/symbol extraction image including at least one of characters and symbols extracted from the object image, an outline image of the object, and size information of the object.

The input to the classifier may further include magnification information indicating the enlargement or reduction ratio of the object image.

The group-specific processing may include processing to display a screen that accepts input of search conditions for searching for object types within the estimated group.

In another aspect of the present disclosure, the object is a medicine, the difficult-to-identify object includes at least one of a capsule medicine, a plain medicine, and a divided tablet, and the identifiable object includes a tablet having an imprint or marking.

The input to the classifier may be at least one of a drug image obtained by cutting out the area of the drug detected by a detector from an image on a drug-by-drug basis, a character/symbol extraction image including at least one of characters and symbols extracted from the drug image, an image of the drug's outline, and size information of the drug.

An object identification device according to another aspect of the present disclosure is an object identification device including one or more processors and one or more memories in which programs executed by the one or more processors are stored, and the one or more processors perform a detection process to detect objects on an object-by-object basis from an image in which a plurality of objects are photographed; a classification process to estimate the type of object from the image for type-identifiable objects whose type can be identified from the image among the objects detected by the detection process, and to estimate the group from the image for type-unidentifiable objects whose type is difficult to identify from the image but whose group to which the object belongs can be estimated; and a process to transition to group-specific processing leading to the identification of the type of objects estimated as a group by the classification process.

The one or more processors may be configured to perform the detection process using a detector including a first learned model trained by machine learning using first training data labeled on an object-by-object basis without distinguishing between objects whose type can be identified and objects whose type is difficult to identify.

The one or more processors may be configured to perform the classification process using a classifier including a second learned model trained by machine learning using second training data in which objects whose types can be identified are labeled by object type, and objects whose types are difficult to identify are labeled by group to which the objects belong.

The one or more processors may be configured to extract from the image an area of an object detected by the detection process, perform a process of generating an object image on an object-by-object basis, and perform an identification process based on the object image.

In another aspect of the present disclosure, the object is a medicine, the difficult-to-identify object includes at least one of a capsule medicine, a plain medicine, and a divided tablet, the identifiable object includes a tablet having an engraving or marking, and the group-specific processing may include processing for displaying a screen that accepts input of search criteria for searching for a type of medicine within the estimated group.

In another aspect of the present disclosure, a first database is provided in which character/symbol information, including at least one of characters and symbols indicated by an engraving or marking on a medicine, is linked to the type of medicine, and a second database in which a master image of the medicine is stored, and one or more processors are configured to search at least one of the first database and the second database based on received search conditions and output candidate medicines that meet the search conditions.

The one or more processors may be configured to perform processing to display a screen including a captured image display unit that displays an image, and a candidate display unit that displays information about candidate objects based on the estimated results of the classification process.

The one or more processors may be configured to perform processing to display information about the group to which the candidate object to be displayed in the candidate display unit belongs.

The one or more processors may be configured to receive an instruction specifying a group to which a candidate object to be displayed in the candidate display section belongs, and to control the display of the candidate display section in accordance with the received instruction.

In another aspect of the present disclosure, the system may be configured to include a camera and a display that displays an image captured by the camera and information about the object estimated from the image.

An object identification method according to another aspect of the present disclosure is an object identification method executed by one or more processors, which includes detecting objects on an object-by-object basis from an image in which a plurality of objects are captured, estimating the type of an object from the image for type-identifiable objects among the detected objects, for which the type of object can be estimated from the image, estimating a group from the image for type-difficult objects for which the type of object is difficult to identify from the image but the group to which the object belongs can be identified, and performing group-specific processing that leads to identification of the type of objects estimated as a group.

A program according to another aspect of the present disclosure provides a computer with the following functions: detect objects on an object-by-object basis from an image in which multiple objects are captured; estimate the type of object from the image for identifiable objects among the detected objects, where the type of object can be identified from the image; estimate the group from the image for difficult-to-identify objects, where the type of object is difficult to identify from the image but the group to which the object belongs can be identified; and perform group-specific processing that leads to the identification of the type of objects estimated as a group.

According to the present disclosure, it becomes possible to identify the type of individual objects in an image even when the image is captured in a state in which identifiable objects and difficult-to-identify objects are mixed.

FIG. 1 is a front perspective view of a smartphone. FIG. 2 is a rear perspective view of the smartphone. FIG. 3 is a block diagram showing the electrical configuration of the smartphone. FIG. 4 is a block diagram showing the internal configuration of the in-camera. FIG. 5 is a block diagram showing a functional configuration of a drug type identification device according to an embodiment. FIG. 6 is a flowchart showing an overview of a learning phase by machine learning for realizing a drug type identification device including a drug detector and a drug identifier. FIG. 7 is an explanatory diagram showing an example of an identification code given as training data for a drug-type-identifiable medicine. FIG. 8 is an explanatory diagram showing an example of an identification code that is assigned as training data to a drug whose drug type is difficult to identify. FIG. 9 is an explanatory diagram showing another example of an identification code given to a capsule medicine as training data. FIG. 10 is an explanatory diagram showing another example of an identification code that is assigned to a plain medicine as training data. FIG. 11 is a conceptual diagram showing an example of information to be input to the medicine identifier. FIG. 12 is a diagram showing an example of input information to the neural network. FIG. 13 is a diagram showing another example of input information to the neural network. FIG. 14 is a diagram showing an example of input information in the case of a capsule medicine having an identification code printed on the capsule. FIG. 15 is a diagram showing an example of input information for a capsule medicine without an identification code. FIG. 16 is a diagram showing an example of input information in the case of a half tablet. FIG. 17 is a block diagram showing the functional configuration of the machine learning system. FIG. 18 is a flowchart showing the operation of the drug type identification device according to the embodiment. FIG. 19 is a flowchart showing the operation of the drug type identification device according to the embodiment, and shows an example of the loop processing applied to steps S15 and S16 in FIG. 18. FIG. 20 is a diagram showing an example of a screen displayed on the touch panel display. FIG. 21 is a diagram showing an example of a screen display when editing a medicine region. FIG. 22 is an explanatory diagram showing an example of an operating method for editing an area. FIG. 23 is a diagram showing an example of a screen display in a state where the identification of the medicine has been confirmed. FIG. 24 is a diagram showing an example of a screen display of a recognition result confirmation GUI (Graphical User Interface). FIG. 25 is a diagram showing an example of a screen display of the candidate list display GUI. FIG. 26 is a diagram showing an example of a screen display of the engraving text GUI. FIG. 27 is a diagram showing an example of a screen display of a capsule search GUI. FIG. 28 is a diagram showing an example of a screen display of a plain medicine search GUI. FIG. 29 is a diagram showing an example of a screen display of a divided tablet search GUI. FIG. 30 shows examples of icons representing typical shapes of medicines. FIG. 31 is a diagram showing examples of icons used for color selection. FIG. 32 is a block diagram showing an example of a functional configuration for identifying the type of a drug whose drug type is difficult to identify in the drug type identification device according to the embodiment. FIG. 33 is a top view of the photography auxiliary device. FIG. 34 is a cross-sectional view taken along line 34-34 in FIG. FIG. 35 is a top view of the shooting assist device with the auxiliary light source removed. FIG. 36 is a top view of a shooting assistance device according to another embodiment. FIG. 37 is a cross-sectional view taken along line 37-37 in FIG. FIG. 38 is a top view of the shooting assist device with the auxiliary light source removed. FIG. 39 is a cross-sectional view showing an example of a lighting device. FIG. 40 is a top view of the medicine placing platform using a circular marker as the reference marker. FIG. 41 is a top view of a medicine placing table using a circular marker according to a modified example. FIG. 42 is a diagram showing a specific example of a reference marker having a rectangular outline. FIG. 43 is a top view of a medicine placing table using a circular marker according to another modified example. FIG. 44 shows another specific example of a rectangular reference marker. FIG. 45 is a diagram showing an example of a medicine placing table having a recessed structure. FIG. 46 is a diagram showing an example of a medicine placement platform having a recess structure for capsule medicine. FIG. 47 shows an example of a medicine placement platform having a recess structure for an oval tablet.

A preferred embodiment of the present invention is described in detail below with reference to the attached drawings.

<Outline of drug type identification device according to the embodiment>
The drug type identification device according to the embodiment is a device for identifying the type (drug type) of a drug from an image of the drug. In the present embodiment, it is assumed that a plurality of drugs are photographed in a state in which a drug type identifiable drug and a drug type difficult to identify are mixed, but the drug type identification device can function effectively even when both are not mixed or when only one drug is photographed.

The drug type identification device according to the embodiment detects each drug from an image of multiple drugs, automatically determines whether each detected drug is a drug with a difficult drug type or a drug with a drug type that can be identified, and automatically branches to an appropriate drug type identification flow for each drug, enabling efficient drug type identification. As an example, the drug type identification device is mounted on a mobile terminal device. The mobile terminal device includes at least one of a smartphone, a mobile phone, a PHS (Personal Handy-phone System), a PDA (Personal Digital Assistant), a tablet computer terminal, a notebook personal computer terminal, and a portable game machine. Below, a drug type identification device realized by the hardware and software of a smartphone is given as an example and will be described in detail with reference to the drawings.

[Appearance of the smartphone]
Fig. 1 is a front perspective view of a smartphone 10, which is a camera-equipped mobile terminal device that functions as a drug type identification device according to an embodiment. As shown in Fig. 1, the smartphone 10 has a flat housing 12. The smartphone 10 has a touch panel display 14, a speaker 16, a microphone 18, and an in-camera 20 on the front side of the housing 12.

The touch panel display 14 includes a display unit that displays images, etc., and a touch panel unit that is disposed in front of the display unit and accepts touch input. The display unit is, for example, a color LCD (Liquid Crystal Display) panel.

The touch panel unit is, for example, a capacitive touch panel provided in a planar manner on a light-transmitting substrate body, and having light-transmitting position detection electrodes and an insulating layer provided on the position detection electrodes. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to a user's touch operation. Touch operations include a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

The speaker 16 is an audio output unit that outputs audio during a call and when playing a video. The microphone 18 is an audio input unit to which audio is input during a call and when shooting a video. The in-camera 20 is an imaging device that captures videos and still images.

Figure 2 is a rear perspective view of the smartphone 10. As shown in Figure 2, the smartphone 10 has an outer camera 22 and a light 24 on the rear of the housing 12. The outer camera 22 is an imaging device that captures videos and still images. The light 24 is a light source that emits illumination light when capturing images with the outer camera 22, and is composed of, for example, an LED (Light Emitting Diode).

1 and 2, the smartphone 10 has switches 26 on the front and side of the housing 12. The switches 26 are input members that accept instructions from a user. The switches 26 are push-button switches that are turned on when pressed with a finger or the like and turned off when the finger is released by the restoring force of a spring or the like.

The configuration of the housing 12 is not limited to this, and a configuration having a folding structure or a sliding mechanism may also be adopted.

[Smartphone electrical configuration]
The smartphone 10 has, as its main function, a wireless communication function for performing mobile wireless communication with a base station device via a mobile communication network.

Figure 3 is a block diagram showing the electrical configuration of the smartphone 10. As shown in Figure 3, the smartphone 10 has the aforementioned touch panel display 14, speaker 16, microphone 18, in-camera 20, out-camera 22, light 24, and switch 26, as well as a CPU (Central Processing Unit) 28, a wireless communication unit 30, a call unit 32, a memory 34, an external input/output unit 40, a GPS receiver unit 42, and a power supply unit 44.

The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates according to the control program and control data stored in the memory 34, and controls each part of the smartphone 10. The CPU 28 has a mobile communication control function that controls each part of the communication system to perform voice communication and data communication through the wireless communication unit 30, and an application processing function.

The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function allows information such as still images, moving images, and text to be visually conveyed to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to the user's touch operation from the touch panel portion of the touch panel display 14. The CPU 28 also acquires an input signal from the switch 26.

The in-camera 20 and the out-camera 22 capture video and still images according to instructions from the CPU 28. FIG. 4 is a block diagram showing the internal configuration of the in-camera 20. The internal configuration of the out-camera 22 is the same as that of the in-camera 20. As shown in FIG. 4, the in-camera 20 has a photographing lens 50, an aperture 52, an image sensor 54, an AFE (Analog Front End) 56, an A/D (Analog to Digital) converter 58, and a lens driver 60.

The photographing lens 50 is composed of a zoom lens 50Z and a focus lens 50F. The lens driving unit 60 drives the zoom lens 50Z and the focus lens 50F forward and backward in response to commands from the CPU 28 to perform optical zoom adjustment and focus adjustment. The lens driving unit 60 also controls the aperture 52 in response to commands from the CPU 28 to adjust exposure. The lens driving unit 60 corresponds to an exposure correction unit that performs exposure correction of the camera based on the color gray, which will be described later. Information such as the positions of the zoom lens 50Z and the focus lens 50F and the opening degree of the aperture 52 is input to the CPU 28.

The image sensor 54 has a light receiving surface on which a large number of light receiving elements are arranged in a matrix. The subject light transmitted through the zoom lens 50Z, the focus lens 50F, and the aperture 52 is imaged on the light receiving surface of the image sensor 54. R (red), G (green), and B (blue) color filters are provided on the light receiving surface of the image sensor 54. Each light receiving element of the image sensor 54 converts the subject light imaged on the light receiving surface into an electrical signal based on the R, G, and B color signals. In this way, the image sensor 54 acquires a color image of the subject. The image sensor 54 can be a photoelectric conversion element such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge-Coupled Device).

The AFE 56 performs noise removal and amplification of the analog image signal output from the image sensor 54. The A/D converter 58 converts the analog image signal input from the AFE 56 into a digital image signal with a range of gradations. An electronic shutter is used as the shutter that controls the exposure time of the light incident on the image sensor 54. In the case of an electronic shutter, the exposure time (shutter speed) can be adjusted by controlling the charge accumulation period of the image sensor 54 by the CPU 28.

The in-camera 20 may convert the captured video and still image data into compressed image data such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group).

Returning to the explanation of FIG. 3, the CPU 28 stores the video and still images captured by the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input/output unit 40.

Furthermore, the CPU 28 displays the video and still images captured by the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the video and still images captured by the in-camera 20 and the out-camera 22 within application software.

In addition, when capturing an image with the outer camera 22, the CPU 28 may illuminate the subject with auxiliary light by turning on the light 24. The light 24 may be turned on and off by the user's touch operation on the touch panel display 14 or by operating the switch 26.

The wireless communication unit 30 performs wireless communication with a base station device accommodated in the mobile communication network in accordance with instructions from the CPU 28. Using this wireless communication, the smartphone 10 transmits and receives various file data such as voice data and image data, e-mail data, and receives Web (abbreviation for World Wide Web) data and streaming data.

The speaker 16 and microphone 18 are connected to the communication unit 32. The communication unit 32 decodes the voice data received by the wireless communication unit 30 and outputs it from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into voice data that can be processed by the CPU 28 and outputs it to the CPU 28.

The memory 34 stores instructions to be executed by the CPU 28. The memory 34 is composed of an internal storage unit 36 built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 and the external storage unit 38 are realized using known storage media.

The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

The external input/output unit 40 acts as an interface with external devices connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to other external devices by communication or the like via the external input/output unit 40. The external input/output unit 40 transmits data received from external devices to each component inside the smartphone 10, and also transmits data inside the smartphone 10 to the external devices.

The means of communication, etc., are, for example, Universal Serial Bus (USB), IEEE (Institute of Electrical and Electronics Engineers) 1394, the Internet, wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. The external device is, for example, a headset, an external charger, a data port, an audio device, a video device, a smartphone, a PDA, a personal computer, and an earphone.

The GPS receiver unit 42 detects the position of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, ..., STn.

The power supply unit 44 is a power supply source that supplies power to each part of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium ion secondary battery. The power supply unit 44 may also include an A/D conversion unit that generates a DC voltage from an external AC power source.

The smartphone 10 configured in this manner can be set to a shooting mode in response to user input of instructions using the touch panel display 14 or the like, and can capture video and still images using the in-camera 20 and the out-camera 22.

When the smartphone 10 is set to the shooting mode, it enters a shooting standby state, a video is captured by the in-camera 20 or the out-camera 22, and the captured video is displayed on the touch panel display 14 as a live view image.

The user can visually view the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject they want to capture, and set the shooting conditions.

When the smartphone 10 is in a standby state for shooting and is instructed to shoot by a user inputting instructions using the touch panel display 14 or the like, the smartphone 10 performs AF (Autofocus) and AE (Auto Exposure) control, and shoots and stores videos and still images.

[Functional configuration of drug type identification device]
5 is a block diagram showing a functional configuration of the drug type identification device 100 realized by the smartphone 10. The drug type identification device 100 includes a processor 102 and a storage device 104. The processor 102 includes a CPU 28. The processor 102 may include a GPU (Graphics Processing Unit). The storage device 104 is a computer-readable medium that is a non-transient tangible entity, and includes a memory 34. The processor 102 is connected to a touch panel display 14. The touch panel display 14 includes a display unit 14A that functions as a display device (display) and an input unit 14B that functions as an input device that accepts input by touch operation.

The drug type identification device 100 includes an image acquisition unit 112, a drug detector 114, an area correction unit 116, a drug area extraction unit 118, a drug identifier 120, a text search unit 122, a display control unit 124, and an input processing unit 126.

The image acquisition unit 112 acquires a captured image of the drug to be identified. The captured image is, for example, an image captured by the in-camera 20 or the out-camera 22. The captured image may be an image acquired from another device via the wireless communication unit 30, the external storage unit 38, or the external input/output unit 40. The captured image may include multiple drugs to be identified. The multiple drugs to be identified are not limited to drugs to be identified of the same drug type, and may be drugs to be identified of different drug types. In this embodiment, an example is described in which an image is processed in which multiple drugs are captured together at once (as one image) in a state in which different types of drugs are mixed.

The captured image may be an image of the drug to be identified and a marker. The marker may be, for example, an ArUco marker, a circular marker, or a square marker. It is preferable that the captured image includes multiple markers. The multiple markers are arranged, for example, at the four corners of a rectangular area in the drug placement area. The captured image may be an image of the drug to be identified and a reference gray color.

The captured image may be an image captured at a standard shooting distance and shooting viewpoint. The shooting distance can be expressed by the distance between the drug to be identified and the shooting lens 50 and the focal length of the shooting lens 50. The shooting viewpoint can be expressed by the angle between the marker printing surface and the optical axis of the shooting lens 50.

The image acquisition unit 112 includes an image correction unit (not shown). When a marker is included in the captured image, the image correction unit standardizes the shooting distance and shooting viewpoint of the captured image based on the marker to acquire a standardized image. The standardized image may be an image obtained by cutting out an area inside a rectangle whose vertices are the markers at the four corners after a standardization process is performed on the captured image. For example, the image correction unit specifies the coordinates of the four vertices of a rectangle whose coordinates are specified by the markers after standardization of the shooting distance and shooting viewpoint. The image correction unit obtains a perspective transformation matrix that transforms these four vertices into the positions of the specified coordinates. Such a perspective transformation matrix is uniquely determined if there are four points. For example, a transformation matrix can be obtained by the getPerspectiveTransform function of OpenCV (Open Source Computer Vision Library) if there is a correspondence between the four points.

The image correction unit uses the obtained perspective transformation matrix to perform perspective transformation on the entire original captured image, and obtains the transformed image. Such perspective transformation can be performed using the warpPerspective function of OpenCV. This transformed image may be a standardized image in which the shooting distance and shooting viewpoint are standardized.

In addition, when the captured image contains an area of a reference gray color, the image correction unit may perform color correction of the captured image based on the reference gray color.

The drug detector 114 includes a first trained model TM1, which is a learning model trained by machine learning. The first trained model TM1 is a model trained to perform a so-called object detection task. When an image (pre-standardized image or standardized image) is given as input, the first trained model TM1 outputs position information corresponding to the area of the detected object, the object class, and a score indicating the probability of the object's likelihood. The first trained model TM1 is an example of a "first trained model" in this disclosure.

The object class in drug detection includes at least "drug" and may further include "marker". Drug detector 114 detects drugs from the captured image acquired by image acquisition unit 112, and outputs information indicating the area of the detected drug. If a standardized image is acquired by the image correction unit, drug detector 114 detects the area of the drug from the standardized image. If the captured image contains multiple drugs, drug detector 114 detects the areas of each of the multiple drugs.

The output from the first trained model TM1 may be position information of a bounding box indicating the area of each drug detected within the captured image, or may be a segmentation mask image in which the area of each drug is filled in pixel by pixel. The training method for creating the first trained model TM1 and the contents of the detection process by the drug detector 114 will be described in detail below. The drug detector 114 is an example of a "detector" in this disclosure.

The results of the detection process by the drug detector 114 are displayed on the display unit 14A via the display control unit 124. The display control unit 124 generates a signal for display on the touch panel display 14 and controls the display. The display control unit 124 includes a magnification change unit 125 that changes the display magnification of the image to be displayed on the display unit 14A. When an instruction to change the display magnification is input, such as when a pinch-out or pinch-in operation is performed on the touch panel display 14, the magnification change unit 125 performs enlargement or reduction processing in accordance with the instruction. The input processing unit 126 accepts input from the input unit 14B or microphone 18, and sends the accepted input information to the corresponding processing unit.

The area correction unit 116 receives correction instructions from the user for the area of the drug detected by the drug detector 114, and performs processing to correct the drug area in accordance with the received instructions. That is, the area correction unit 116 makes corrections to the detection results of the drug detector 114 in accordance with the area correction instructions received from the input unit 14B. When an erroneous detection or missed detection occurs by the drug detector 114, the area correction unit 116 allows the user to input an instruction to correct the detection results from the input unit 14B and specify the correct area of the drug to be identified.

The drug region extraction unit 118 performs processing to extract a drug region for each drug from the captured image based on the detection results of the drug detector 114. When the drug region is corrected by the region correction unit 116, the drug region extraction unit 118 performs processing to extract the corrected drug region.

The drug identifier 120 includes a second trained model TM2, which is a learning model trained by machine learning. The second trained model TM2 is a model trained to perform a so-called object recognition task. The drug identifier 120 acquires an area image (hereinafter referred to as a drug image) of each drug cut out by the drug area cutout unit 118, estimates the type of the corresponding drug or the group to which the drug belongs, and performs labeling (i.e., multi-class classification). The class classification performed by the drug identifier 120 differs in fineness (granularity) of classification depending on whether the drug shown in the input image is a drug with a difficult drug type or a drug with a drug type that can be identified. The drug image is an example of an "object image" in this disclosure. The second trained model TM2 is an example of a "second trained model" in this disclosure.

When the input drug image is an image of a drug that is difficult to identify, the drug identifier 120 estimates the group to which the drug belongs and outputs the estimation result. Groups to which drugs that are difficult to identify may include, for example, "capsule drugs," "divided tablets," or "plain drugs." Groups may also be defined by further categorizing, or subgroups may be defined by a hierarchical classification. For example, groups may be defined as "single-color white capsule drugs," "two-color red and white capsule drugs," "half tablets," "quarter tablets," "plain white drugs," or "plain transparent drugs."

When the input drug image is an image of a drug with a drug type identifiable, the drug identifier 120 identifies the type (drug type) of the drug and outputs an estimation result of the drug type. The drug type identification information output by the drug identifier 120 may be, for example, a unique identification code defined for each drug type. The learning method for creating the second trained model TM2 and the contents of the identification process by the drug identifier 120 will be described in detail later. The drug identifier 120 searches a drug master database (not shown) from the identification code unique to the estimated drug type and acquires drug information on the corresponding drug or a similar drug. The drug identifier 120 is an example of an "identifier" in this disclosure.

The results of the identification process by the drug identifier 120 are displayed on the display unit 14A via the display control unit 124. After checking the identification result displayed on the display unit 14A, the user can confirm the identification result, correct the identification result, or perform a separate text search.

The text search unit 122 accepts search key input from the input unit 14B or microphone 18, accesses the engraving text master database 130 to search for relevant information, and outputs the search results. The engraving text master contains data linking text information of character symbols engraved or printed on various drugs with the type of drug. The text information of character symbols stored in the database 130 is an example of "character symbol information" in this disclosure.

The storage device 104 stores a database 130 of engraving text masters, a master image database 131 including master images of various drugs, and a drug master database (not shown). The master image database 131 may be included in the drug master database. Data such as the engraving text master, master image, and drug master may be stored on a network such as a cloud server (not shown).

The search results by the text search unit 122 are displayed on the display unit 14A via the display control unit 124. In addition, the master images read out from the master image database 131 are displayed on the display unit 14A via the display control unit 124.

After checking the search results and master images displayed on display unit 14A, the user can confirm the drug type identification results, further narrow the search, or perform a new search. Database 130 is an example of a "first database" in this disclosure. Master image database 131 is an example of a "second database" in this disclosure.

The storage device 104 includes an identification result storage unit 132. The identification result storage unit 132 is a storage area in which the identification results of drug types identified based on the drug identifier 120 and search results by the text search unit 122 are stored.

[Explanation of learning phase]
In the drug type identification device 100 according to this embodiment, each of the drug detector 114 and the drug identifier 120 is trained on a machine learning basis as follows, and these are combined to configure the drug type identification device 100. FIG. 6 is a flowchart showing an overview of the learning phase by machine learning for realizing the drug type identification device 100 including the drug detector 114 and the drug identifier 120.

The processing of each step shown in FIG. 6 may be performed, for example, by a computer executing a program. The machine learning method for obtaining the drug type identification device 100 includes a step of creating training data used to train the drug detector 114 (step S1: first training data creation step), a step of training the drug detector 114 by performing machine learning using the training data (step S2: first training step), a step of creating training data used to train the drug identifier 120 (step S3: second training data creation step), a step of training the drug identifier 120 by performing machine learning using the training data (step S4), and a step of configuring the drug type identification device 100 using the drug detector 114 trained in step S2 and the drug identifier 120 trained in step S4 (step S5).

The training data used in supervised learning includes input data and correct answer data (teacher data). Creating training data includes creating correct answer data (teacher data) that corresponds to the input data.

The process of creating the drug detector 114 (steps S1 and S2) and the process of creating the drug identifier 120 (steps S3 and S4) may be performed in parallel or sequentially. The execution timing of each step 1 to 5 is not particularly limited, and may be performed consecutively, or the process of each step may be performed at different times and using different computers. For example, a first computer may perform the process of creating the drug detector 114 (steps S1 and S2), and a second computer different from the first computer may perform the process of creating the drug identifier 120 (steps S3 and S4). In addition, the first computer may perform the process of creating training data (steps S1 and S3), and the second computer may perform the learning process (steps S2 and S4).

In step S1, training images are assigned training data (correct answer data) for the position information and class of each drug contained in the image. The position information of the drug given as training data may be information specifying the position of a bounding box surrounding the drug, or may be a mask filled with the drug's shape itself. Furthermore, the class label assigned as training data may be uniformly set to "drug" regardless of the type (drug type). In other words, the classification label is assigned to both drugs whose drug type can be identified and drugs whose drug type is difficult to identify, all of which are assigned the same classification label of "drug".

[Training of Drug Detector 114]
The drug detector 114 estimates object position information, class information, and the likelihood of the detected object (here, drug) for the purpose of identifying the position of each drug from the input photographed image, separating the drug region from the background, and cutting out the drug. The object position information may be, for example, "four vertex coordinates of a bounding box surrounding the drug without rotation,""center coordinates, height, and width of a bounding box surrounding the drug without rotation,""center coordinates, height, width, and rotation angle of a bounding box surrounding the drug with rotation," or "a mask filled with the shape of the drug itself," etc.

Many medications are ellipsoidal in shape, and when ellipsoidal medications are arranged vertically at an angle, a single bounding box without rotation may contain multiple medications. Therefore, the object position information is preferably the center coordinates, height, width, and rotation angle of a rotated bounding box surrounding the medication, or a mask filled with the medication's shape itself.

In step S2, machine learning is performed to train a learning model (hereinafter referred to as a first learning model) to be applied to the drug detector 114 using the data set of training data created in step S1. The first learning model is configured using, for example, a neural network. As a network model suitable for object detection, for example, a convolutional neural network (CNN) can be used. The drug detector 114 is trained to receive an input of an image and output object position information for each drug in the image.

In addition, when the drug detector 114 also detects markers, a dataset including training data in which the marker area is given as correct answer data (teacher data) is required to train the first learning model. The training data used to train the drug detector 114 is an example of the "first training data" in this disclosure.

[Learning of the drug identifier 120]
The drug identifier 120 is input with images of individual drugs, and is trained to estimate the type of drug or class information of the group to which the drug belongs, the likelihood of the identified class, and the like.

When creating training data for the drug identifier 120, for drugs whose drug type can be identified, a unique identification code defined for each drug is assigned as a correct label. On the other hand, for drugs whose drug type is difficult to identify, an identification code is assigned to the group to which the drug belongs. For drugs whose drug type is difficult to identify, for example, correct labels are assigned to groups in which the processing flow is to be branched in the drug type identification phase, such as "capsule drugs," "plain drugs," "half tablets," or "quarter tablets." Note that for "capsule drugs," correct labels may be assigned to smaller groups, such as "hard capsule drugs" and "soft capsule drugs."

In order to improve the discrimination performance for groups, it is preferable to use at least one, preferably a combination of a plurality of, as input to the drug identifier 120: an imprint extraction image, an image of the drug's outer shape, and drug size information. An imprint extraction image is an image in which only the imprinted or printed portion of a drug is extracted and the imprinted or printed portion is shown in white on a mainly black background. The training data used to train the drug identifier 120 is an example of the "second training data" in this disclosure.

Figure 7 is a diagram showing examples of identification codes assigned as training data to drug types that can be specified. For drug types that can be specified, a different identification code is assigned to each drug for each drug type. "P000001", "P000002", ... "P009999" in Figure 7 represent examples of identification codes that are uniquely defined to correspond to different drug types. The identification code shown in Figure 7 is an example of a "type unit" label in this disclosure.

Figure 8 shows an example of an identification code to be assigned as training data for drugs that are difficult to identify. For drugs that are difficult to identify, an identification code is assigned to the group to which the drug belongs. For example, an image of a capsule drug is assigned an identification code of "G000001" that represents the group of "capsule drugs" regardless of the drug type. An image of a plain drug is assigned an identification code of "G000002" that represents the group of "plain drugs" regardless of the drug type. An image of a half tablet (half tablet) is assigned an identification code of "G000003" that represents the group of "half tablets" regardless of the drug type. An image of a quarter tablet is assigned an identification code of "G000004" that represents the group of "quarter tablets" regardless of the drug type. If half tablets and quarter tablets are to be treated in the same way, the same identification code may be assigned to them. Alternatively, separate identification codes may be assigned to the half tablets and quarter tablets, and the program may perform the same processing on the identification codes "G000003" and "G000004."

In addition, the identification code assigned to a group may be defined as an identification code for a subgroup that further classifies the group according to a hierarchical structure.

9 is a diagram showing another example of an identification code to be assigned as training data for capsule medicines. For example, by grouping capsule medicines more finely according to capsule color or printed character color, it is expected that the accuracy of inference by the medicine identifier 120 will be improved. In addition, by performing such fine grouping, it may be used in discrimination processing, such as improving usability as a search attribute when searching for capsules. In FIG. 9, an example is shown in which a classification (subgroup) is defined for a group of "capsule medicines" in which the capsule color is a single color and a combination of two colors. In addition, the group of single-color capsules may be further classified according to the color of the capsule. For an image of a single-color white capsule medicine, for example, an identification code of "G000010" representing a group of single-color white capsule medicines is assigned. For an image of a single-color blue capsule medicine, an identification code of "G000011" representing a group of single-color blue capsule medicines is assigned. Although not shown in FIG. 9, individual identification codes representing each group may be assigned to images of single-color capsule medicines of other colors in the same manner.

In addition, groups of two-color capsules may be further classified according to the combination of capsule colors. For example, an image of a two-color red and white capsule medicine is assigned an identification code of "G000100" representing a group of red and white capsule medicines. An image of a two-color blue and white capsule medicine is assigned an identification code of "G000101" representing a group of blue and white capsule medicines. Although not shown in FIG. 9, images of capsule medicines with other two-color combinations may also be assigned individual identification codes representing their respective groups.

Figure 9 illustrates an example of capsule medication, but plain tablets can also be divided into smaller groups based on color and shape and assigned identification codes.

Figure 10 shows another example of an identification code that is assigned as training data to plain medicines. Figure 10 shows an example of further grouping plain medicines according to a combination of their shape and color. Figure 10 shows examples of the case where the shape of the plain medicine is circular and the case where the shape is ellipsoidal.

As shown in Figure 10, an image of a plain, circular, white medicine is assigned an identification code of, for example, "G001001" to represent a group of plain, circular, white medicines. An image of a plain, circular, yellow medicine is assigned an identification code of "G001002" to represent a group of plain, circular, yellow medicines. Although not shown in Figure 10, images of plain, circular medicines of other colors may also be assigned individual identification codes to represent their respective groups.

For an image of a plain drug whose shape is ellipsoidal and whose color is orange, for example, an identification code of "G002001" is assigned to indicate a group of plain drug whose shape is ellipsoidal and whose color is orange. For an image of a plain drug whose shape is ellipsoidal and whose color is transparent, an identification code of "G002002" is assigned to indicate a group of plain drug whose shape is ellipsoidal and whose color is transparent. Although not shown in FIG. 10, images of plain drug whose shape is ellipsoidal and whose color is other than ellipsoidal may be assigned individual identification codes to indicate their respective groups. For images of plain drug whose shape and color are not shown, individual identification codes to indicate their respective groups may be assigned.

[Information input to the drug identifier 120]
When taking pictures in various environments using the smartphone 10, color information may not necessarily be a reliable source of information due to the influence of the shooting environment. For this reason, when identifying a drug type-identifiable drug on a drug-by-drug basis, the stamp extraction image excluding color information is often important. It is preferable for the drug identifier 120 to place more importance on the stamp extraction image when making the identification. The stamp extraction image may be obtained by inputting the original image and performing inference using a trained machine learning model that inputs the original image and outputs the stamp extraction image.

On the other hand, even if color information is significantly affected by the shooting environment, it may be useful in some cases, such as for drugs with very unusual colors. In addition, drug shape information can also be a robust source of information that is less affected by the shooting environment. Images of the drug's outline can be useful shape information.

In addition, information on the major and minor axis sizes of drugs (size information) obtained by non-machine learning methods can also be a robust information source that is less susceptible to the effects of the imaging environment. For example, by extracting the major and minor axes of drugs using OpenCV or the like and using the numerical information, a robust information source that is less susceptible to the effects of the imaging environment can be obtained. In this way, size information measured for each drug from the captured image can be an important information source for identifying a huge number of drugs, even for drugs that are difficult to identify. In particular, for drugs that are difficult to identify, such as capsule drugs or plain tablets that do not have identification codes printed on them, size information can be a very important information source for group identification.

Considering these facts, it is preferable that the input to the drug identifier 120 is a combination of one or more, preferably a combination of a source image, which is a region image (drug image) cut out on a drug-by-drug basis from a captured image, an imprint extraction image extracted from the source image, an outline image, and size information. The term "imprint extraction image" includes not only imprints but also images extracted from imprints on tablets or capsules. The imprint extraction image may be referred to as a character symbol extraction image. The term "imprint" may be understood as a term including concepts such as "printing," "printed symbol," "identification symbol," or "character symbol" on tablet or capsule drugs, as necessary from the context.

The configuration may be such that all of the information, i.e., the original image including color information, the engraving extraction image, the outline image, and the size information, is combined and input to the drug identifier 120, or the configuration may be such that only some of this information is combined and input to the drug identifier 120.

Figure 11 is a conceptual diagram showing an example of information to be input to the drug identifier 120. The original image Org is a drug image per drug, and corresponds to an individual drug image cut out for each drug from the captured image. The imprint extraction image Egm is an image in which the imprint is extracted from the original image Org. The imprint extraction image Egm may be an image that has been subjected to an enhancement process to increase the visibility of the extracted imprint. The outline image Otw is an image showing the outline of the drug extracted from the original image Org. Note that the outline image Otw is not limited to one showing the exact outline of the drug, and may also show an approximate shape that conforms to the outline. The size information is, for example, numerical information obtained by measuring the major and minor axes of the drug based on the original image Org or the outline image Otw.

In order to build a system that is robust against the imaging environment, it is preferable to use a combination of these multiple pieces of information as input information to the drug identifier 120.

The learning model applied to the drug identifier 120 is constructed, for example, using a neural network. As a network model suitable for image recognition, for example, CNN can be used. When a combination of two or more images of the original image, the stamp extraction image, and the outline image is input as input to the neural network, there are two possible methods: combining these multiple images in the channel direction and inputting them, or combining them horizontally or vertically on the image (within the plane of the image) and inputting them as a composite image.

In neural networks, the input image size may be restricted to a fixed value, so there are two possible input image methods:

[Method 1] The first method is a method in which an image is enlarged or reduced to match the input image size accepted by the input layer of the neural network. Figure 12 shows an example of input information using the first method.

In the first method, each image is enlarged or reduced to the upper limit of the input image size that the neural network can accept, and then input. The information on the image enlargement or reduction ratio used in this image enlargement or reduction process (hereinafter referred to as "resizing process") is stored in case it becomes necessary in subsequent processes. In the first method, the magnification information applied to the resizing process is stored, and as input information for the neural network, in addition to a combination of one or more images of the original image Org1, the stamp extraction image Egm1, and the outline image Otw1 and size information, the magnification information of the resizing process is used as necessary. The original image Org1 shown here is an image of a drug unit cut out from a standardized image of a captured image, which has been resized to match the input image size of the neural network. In addition, the stamp extraction image Egm1 is obtained by performing a process to extract the stamp from the original image Org1. The outline image Otw1 is obtained by performing a process to extract the outline of the drug from the original image Org1.

According to the first method, for example, when the printed characters on a small tablet are small or fine, the image is enlarged and input to the neural network, which has the advantage of improving inference (recognition) accuracy. In addition, according to the first method, the input image size of the neural network can be designed to be small, which is expected to reduce execution time. In other words, since the input image size of the neural network can be designed to be appropriate for recognition performance without being restricted by the maximum size of drugs that actually exist in the world, the processing of unnecessary data is suppressed.

On the other hand, in the case of the first method, image resizing and other processes are performed at the time of input and output of the neural network, which makes the process somewhat complicated. Also, it is necessary to store information on the enlargement/reduction rate (magnification information) in case it is required in later processes (such as when presenting the drug identification results to the user with an image proportional to the size of the tablet).

[Method 2] The second method is to paste and input an image at its original size (without enlarging/reducing) at the center of the image of the input image size accepted by the input layer of the neural network. In this case, the input image size for the neural network is determined in advance, and the same original image cut out from the standardized image of the captured image is input to the input layer of the acceptable input image size. The standardized original image is an image whose correspondence with the actual size of the drug is understood.

Figure 13 shows an example of input information according to the second method. According to the second method, image resizing is not required. Therefore, in the case of the second method, input of magnification information indicating the enlargement/reduction ratio is also not required. In the second method, a combination of one or more images of the original image Org2, the stamp extraction image Egm2, and the outline image Otw2 and size information can be used as input information for the neural network. In addition, in the second method, the outline image Otw2 extracted from the original image Org2 itself contains information indicating the size of the drug, so when the outline image Otw2 is used for input, there is an advantage that numerical information on the major and minor diameters is not necessarily required.

On the other hand, in the case of the second method, the input image size accepted by the neural network is determined by the maximum drug size available in the world, so extra space may be created around the tablet, especially for small tablets, which may result in wasteful processing. In addition, since the input image size accepted by the neural network is larger than that of the first method, the execution time may be longer than that of the first method. Furthermore, in the case of the second method, small tablets and tablets with finely engraved characters are processed at their original size, so the identification accuracy of these tablets may be lower than that of the first method.

Comparing the first method with the second method, the first method is more preferable for identifying drug types using a smartphone 10.

[Example of input information for drugs that are difficult to identify]
14 to 16 show examples of input information on drugs that are difficult to identify. Here, an example is shown in which information on drugs that are difficult to identify is input to a neural network using the first method.

Figure 14 is an example of input information in the case of a capsule drug with an identification symbol printed on the capsule (with identification symbol). For a capsule drug with an identification symbol, as in Figure 12, a combination of one or more images from the original image Org3, the engraving extraction image Egm3, and the outline image Otw3 are input as input to the neural network. In addition to the images, numerical information on the size (long axis and short axis) of the capsule drug measured from the original image Org3 may be input. In addition to this information, magnification information for the resizing process may be input. The engraving extraction image Egm3 is an image of the identification symbol extracted from the original image Org3.

Figure 15 is an example of input information for a capsule drug without an identification mark. A capsule drug without an identification mark that does not appear in the original image Org4 may be a capsule drug that does not originally have an identification mark printed on the capsule, or may have an identification mark on the capsule but is not visible in the original image Org4 because the printing was hidden when the image was taken.

For capsule drugs without an identification symbol, a combination of one or more images from the original image Org4, the imprint extraction image Egm4, and the outline image Otw4 is input. However, for capsule drugs without an identification symbol, the imprint extraction image Egm4 is an image that does not contain any information about the identification symbol. The outline image Otw4 is an image that extracts the shape of the capsule drug shown in the original image Org4. In addition to these images, numerical information on the size (long diameter and short diameter) of the capsule drug measured from the original image Org4 can be input, as in FIG. 14. Furthermore, in addition to this information, magnification information for the resizing process can be input.

Figure 16 is an example of input information for a half tablet. Similarly, in the case of a half tablet, a combination of one or more images of the original image Org5, the stamp extraction image Egm5 extracted from the original image Org5, and the outline image Otw5 is input. In addition to the images, a combination of numerical information on the size (long diameter and short diameter) of the half tablet measured from the original image Org5 and magnification information for the resizing process may be input.

Figure 17 is a block diagram showing the functional configuration of the machine learning system 150. Here, an example is shown in which a combination of input information described in Figure 12 is input to the learning model 151.

The machine learning system 150 is an apparatus that generates a second trained model TM2 to be applied to the drug identifier 120, and is realized using a computer system including one or more computers. The machine learning system 150 includes a learning model 151, a loss calculation unit 152, and an optimizer 154. A neural network such as CNN is used for the learning model 151.

The learning model 151 receives as input information a combination of a drug image IMj, which is a region image of the drug DRj, an inscription extraction image IM1j, an outline image IM2j, size information SZj, which is numerical information indicating the size (dimension) of the drug, and magnification information MGj for the resizing process, and outputs an inference result PRj of the type of drug DRj or the group to which the drug DRj belongs. The subscript j represents the index number of the training data.

The machine learning system 150 shown in FIG. 17 includes an engraving extraction unit 140, an outer shape extraction unit 142, and a size measurement unit 144 in front of the learning model 151. The engraving extraction unit 140 extracts engraved or printed character symbols from the drug image IMj and generates an engraving extraction image IM1j, which is an image of the extracted character symbols. The engraving extraction unit 140 processes the drug region of the input image to eliminate the outer shape edge information of the drug DRj and extract the character symbols. The engraving extraction image IM1j is an image in which the character symbols are emphasized by expressing the brightness of the engraved or printed part relatively higher than the brightness of the parts other than the engraved or printed part.

The contour extraction unit 142 extracts the contour of the drug DRj from the drug image IMj and generates a contour image IM2j that is an image showing the contour of the drug DRj. The size measurement unit 144 measures the size of the drug DRj from the drug image IMj and/or the contour image IM2j and generates size information SZj that indicates the dimensions of each of the major and minor axes.

In the machine learning system 150, this information is generated from the drug image IMj and input to the learning model 151, but some or all of the mark extraction image IM1j, the external shape image IM2j, and the size information SZj may be created in advance at the stage of preparing the training data and included in the training data set. In that case, some or all of the mark extraction unit 140, the external shape extraction unit 142, and the size measurement unit 144 are not necessary in the machine learning system 150.

The loss calculation unit 152 calculates the loss value between the inference result output from the learning model 151 and the correct answer data (teacher data) GTj corresponding to the input information.

The optimizer 154 determines the amount of update of the parameters of the learning model 151 based on the calculation result of the loss value indicating the error between the output of the learning model 151 and the correct teacher signal so that the inference result PRj output by the learning model 151 approaches the correct data GTj, and performs the parameter update process of the learning model 151. The optimizer 154 updates the parameters based on an algorithm such as the gradient descent method. The parameters of the learning model 151 include the filter coefficients (weights of connections between nodes) of the filters used in processing each layer of the neural network and the biases of the nodes. The machine learning system 150 may acquire training data and update the parameters in units of mini-batches that combine multiple training data.

In this way, machine learning is performed using a large amount of training data, thereby optimizing the parameters of the learning model 151 and generating a learning model 151 having the desired inference performance. The learned (trained) learning model 151 for which an acceptable inference accuracy has been confirmed is used as the second trained model TM2 of the drug identifier 120.

In this case, in order to obtain an imprint extraction image, a contour image, and size information from the captured image, the drug type identification device 100 is configured to include processing units similar to the imprint extraction unit 140, contour extraction unit 142, and size measurement unit 144 between the drug area cut-out unit 118 and the drug identifier 120 described in Fig. 5. The drug type identification device 100 is an example of an "object identification device" in this disclosure.

[Utilization phase of the drug type identification device 100]
18 is a flowchart showing the operation of the drug type identification device 100 according to this embodiment. Here, as an example of a usage mode of the drug type identification device 100, a case where a drug brought by a certain patient is identified will be described.

In step S11, the user simultaneously photographs multiple medications to be identified. The user can photograph multiple medications in units that have pharmaceutical meaning, such as the same time of taking the medication. For example, the user removes a package of medications contained in the patient's own medication from a bag and photographs the multiple medications simultaneously (all at once) using the camera function of the smartphone 10. The processor 102 acquires the photographed image obtained by photographing. The multiple medications photographed at this time may include a mixture of identifiable and difficult to identify medications. An identifiable medication is an example of an "identifiable object" in this disclosure, and a difficult to identify medication is an example of an "object that is difficult to identify" in this disclosure.

Next, in step S12, the processor 102 detects individual drugs from the captured image using the drug detector 114. The drug detector 114 detects identifiable drugs and difficult-to-identify drugs on an individual drug basis from the captured image. The drug detector 114 estimates, for example, the "center coordinates, height, width, and rotation angle of a rotated bounding box surrounding the drug" or a "segmentation mask in which the drug area is filled with the shape of the drug itself" and outputs the estimation result. The estimation (detection) result by the drug detector 114 is displayed on the touch panel display 14 and is available for confirmation by the user. The processor 102 accepts input of an instruction to correct the detection result or an instruction to approve (confirm) the detection result from the input unit 14B of the touch panel display 14 or the like. If the detection result of the drug detector 114 includes overdetection or detection failure (missed detection), the user can input an instruction to correct the detection result from the input unit 14B of the touch panel display 14 and specify the correct area of each drug. The detection results can be corrected in units of detected regions or in units of detected medicines. A medicine unit is an example of an "object unit" in the present disclosure.

In step S13, the processor 102 determines whether the detection result by the drug detector 114 includes an overdetection or a detection failure. If the detection result by the drug detector 114 includes an overdetection or a detection failure and the determination result in step S13 is a Yes determination, the processor 102 proceeds to step S14 and performs a process to correct the drug area according to instructions received from the user. For example, if an overdetected area is included, a process to delete that area is performed. Also, if there is a detection failure where an area is not detected even though the drug is present, a process to add a new area for the drug is performed. After step S14, the processor 102 proceeds to step S15.

Also, if the judgment result in step S13 is a No judgment, that is, if there is no overdetection or detection failure in the detection result by the drug detector 114, the processor 102 proceeds to step S15. In step S15, the processor 102 performs identification using the drug identifier 120 for each drug in the image. The processor 102 cuts out the drug image in units of the detected drug, and inputs each drug image into the drug identifier 120 to identify the drug.

The processing content of step S15 will be described later using FIG. 19, but the outline is as follows. That is, for drugs with identifiable drug types, it is expected that the drug identifier 120 will infer on a drug type (drug type) basis. When it is confirmed by visual inspection that the drug type identification result obtained by the drug identifier 120 is correct, the inference result is confirmed. On the other hand, if the drug identifier 120 erroneously infers on a group basis of drugs with difficult drug types, the drug type is selected from drugs presented as other top inference candidates, or identified by voice search, text search, or the like.

For drugs with identifiable drug types, the drug identifier 120 is expected to infer the drug by group to which the drug belongs. If a drug is inferred by group as a drug with difficult drug type identification, and if the inference is confirmed to be correct by visual inspection of the captured image of the drug, the flow proceeds to the drug type identification flow defined for that group. For example, for a group of "capsule drugs," the character symbols attached to the capsules are visually inspected in the captured image, the text of the characters and/or symbols is entered as text or by voice, and the database of the engraved text master is searched to compare the drug to be identified with the master data, thereby moving on to the drug type identification flow, where the drug type is identified.

On the other hand, if the inference result of the drug identifier 120 for a drug whose drug type is difficult to identify is incorrect, the user corrects the inference result as appropriate and proceeds to the appropriate drug type identification flow (see Figure 19).

In step S16, the processor 102 determines whether the drug type has been determined for all drugs in the image. If the determination result in step S16 is No, the processor 102 returns to step S15, changes the drug to be identified, and continues processing. If the determination result in step S16 is Yes, the processor 102 ends the flowchart of FIG. 18.

Figure 19 is a flowchart showing an example of loop processing applied to steps S15 and S16 of Figure 18.

When the processing of step S15 is started, in step S21, the processor 102 uses the drug identifier 120 to identify the drug and determines whether the identification result is a drug that belongs to a drug type identifiable drug. The identification result by the drug identifier 120 is provided to the user through the touch panel display 14. The processor 102 accepts input of various instructions from the input unit 14B of the touch panel display 14, such as an instruction to confirm the drug type, an instruction to correct the identification result, or an instruction to move to other processing such as a text search. The user can check the presented identification result and input an instruction to confirm the drug type from the input unit of the touch panel display 14, or an instruction to correct the identification result or move to an engraving text search.

If the judgment result of step S21 is a Yes judgment, the processor 102 proceeds to step S22. In step S22, the processor 102 judges whether or not the result of the drug identifier 120 that the drug is a drug type identifiable drug is correct. If the judgment result of step S22 is a Yes judgment, the processor 102 proceeds to step S23 and judges whether or not the drug type identified (inferred) by the drug identifier 120 is correct. If the user can visually confirm that the drug type is correct, the user can input an instruction to confirm the drug type.

If the judgment result of step S23 is Yes, the processor 102 proceeds to step S29 and confirms the drug type.

If the judgment result of step S23 is No, the processor 102 proceeds to step S28. In step S28, the processor 102 performs a non-machine learning-based drug type identification flow using the engraving text, etc. After step S28, the processor 102 proceeds to step S29.

If the judgment result of step S21 is No, the processor 102 proceeds to step S24. In step S24, the processor 102 judges whether the result that the drug identified by the drug identifier 120 is a drug type difficult to identify is correct. If the judgment result of step S24 is No, the processor 102 proceeds to step S28.

If the judgment result of step S24 is a Yes judgment, the processor 102 proceeds to step S25 and judges whether the type of group to which the difficult-to-identify drug belongs as identified (inferred) by the drug identifier 120 is correct. If the judgment result of step S25 is a No judgment, or if the judgment result of step S22 is a No judgment, the processor 102 proceeds to step S26. In step S26, the processor 102 identifies the group of difficult-to-identify drugs to which the drug belongs. In step S26, the processor 102 accepts input of an instruction to specify the type of group to which the drug belongs from the input unit 14B of the touch panel display 14 or the like, and identifies the group in accordance with the accepted instruction.

After step S26, the processor 102 proceeds to step S27. Also, if the judgment result of step S25 is Yes, the processor 102 proceeds to step S27. In step S27, the processor 102 executes a drug type identification flow defined for the group of the drug whose drug type is difficult to identify. After step S27, the processor 102 proceeds to step S29 and determines the drug type.

Once the drug types have been determined for all drugs contained in the image, the loop process in FIG. 19 ends. The drug identification method described using the flowcharts in FIG. 18 and FIG. 19 is an example of an "object identification method" in this disclosure.

[Example of GUI (Graphical User Interface) of drug type identification device 100]
[Detection Result Display GUI]
FIG. 20 is a diagram showing an example of a screen displayed on the touch panel display 14. FIG. 20 shows an example of a detection result display GUI that displays the detection result of the drug area detected by the drug detector 114. The screen SC1 provided by the drug type identification device 100 is roughly divided into three areas: an entire image display section EDA, a candidate display section CDA, and a button display section BDA. The entire image display section EDA displays a standardized image generated from the acquired captured image. Note that the entire captured image including the marker may be displayed on the entire image display section EDA. The entire image display section EDA is an example of a "captured image display section" in the present disclosure.

The entire image display unit EDA can enlarge or reduce parts of the image by pinching out or in. Therefore, when the letters and symbols on the medicines are small and difficult to see, each medicine can be enlarged and displayed.

The overall image display unit EDA displays the detection results by the drug detector 114. After capturing an image, the area of each drug detected by the drug detector 114 is displayed as a rectangular frame (bounding box). Figure 20 shows an example of a captured image including three drugs DR1, DR2, and DR3. Frames BX1 and BX2 are displayed for the drugs DR1 and DR2 detected by the drug detector 114, respectively. In the example shown in Figure 20, detection has failed for drug DR3, and no frame is displayed for drug DR3. Also, in the example shown in Figure 20, there is an overdetection of the area Abs where no drug is present, and a frame BX3 based on the erroneous detection is displayed.

Each area surrounded by the frames BX1, BX2, and BX3 can be selected by the user, and for example, a selected area is displayed with a red frame, and an unselected area is displayed with a blue frame, with the frame line of a different color. Of course, a different color may be used. It is preferable that the frames BX1, BX2, and BX3 indicating the areas are displayed with a frame line of a different color depending on whether the area is selected or not.

Each of the boxes BX1, BX2, and BX3 has a check box CB. The check box CB displays a mark or is blank depending on the status, such as "drug type confirmed" or "unconfirmed."

At the bottom of the overall image display unit EDA, there are displayed an area edit switch SW1, an area add button BT2, and an area delete button BT3. The area edit switch SW1 is a switch that is operated when it is desired to correct the detection results of the drug area. When the area edit switch SW1 is slid to the right of the screen, the area of the drug can be edited on the screen of the overall image display unit EDA. When the area edit switch SW1 is turned on, the area add button BT2 and area delete button BT3 can be pressed, and the area correction GUI is displayed. When the area edit switch SW1 is slid to the left of the screen, the area cannot be edited. When editing is not possible, the area add button BT2 and area delete button BT3 are grayed out.

Note that the on/off setting of editability of the region is not limited to the region edit switch SW1 illustrated in FIG. 20, but may be implemented, for example, by a long press or double tap on the overall image display unit EDA. Pressing the region add button BT2 allows a region to be specified on the overall image display unit EDA and a drug region to be added. The region delete button BT3 is used when deleting an overdetected region, etc.

When region editing is off and each region is selected by tapping, the drug identifier 120 performs an identification process on the drug image cut out from that region, and drug information for the candidate drug is displayed in the candidate display area CDA below. In addition, the dimensions of the long and short diameters are displayed for each region. The on/off display of these dimensions may be configured to be switchable by a separate switch or the like. A candidate drug is an example of a "candidate object" in this disclosure.

The candidate display section CDA is an area that mainly displays drug information of candidate drugs. The candidate display section CDA displays drug information of candidate drugs based on the identification result estimated by the drug identifier 120. The drug information of candidate drugs includes, for example, a master image of the drug, a drug identification code, a drug surface, a pharmacological classification, and information on the group to which the drug belongs. FIG. 20 shows an example of drug information of candidate drugs for drug DR1. Note that the image can also be enlarged/reduced in the candidate display section CDA by pinching out or in.

The display field displaying group information is configured as a selection box SLB1. In addition, the candidate display area CDA displays an engraving text search button BT4.

On the button display unit BDA, various buttons are arranged, including, for example, a drug type confirmation button BT5, a drug type confirmation hold button BT6, a retake button BT7, and a done button BT8. These buttons can be changed to a pressable or unpressable state depending on the situation. The retake button BT7 is used to press and retake the image when there is a problem with the captured image, such as when the captured image is blurred.

[Area Editing GUI]
Fig. 21 is an example of a screen display when editing the drug region. As illustrated in Fig. 20, when overdetection and/or detection failure are confirmed in the detection result of the drug detector 114, the user slides the region edit switch SW1 to the right of the screen to make the region editable.

In this state, for example, if the region of drug DR1 is selected, the position, shape, and rotation angle of the selected region can be adjusted by dragging, tapping, and other operations. For example, as shown in Figure 22, the region can be enlarged or reduced by dragging an edge of frame BX, translated by dragging the region, or rotated by pinching the region with two fingers in the direction of rotation. Also, the region can be enlarged or reduced while maintaining a constant aspect ratio by dragging a corner of frame BX.

In the case of screen SC2 shown in Fig. 21, since an area Abs where no drug is present has been extracted (detected as a drug) due to overdetection, the user can select the area Abs shown in frame BX3 and press the area deletion button BT3 to delete the area Abs. Note that the "area deletion" function is not limited to the area deletion button BT3, and the GUI may be implemented so that deletion can be performed by pressing and holding the relevant area to display a pull-down menu (not shown) and selecting "area deletion" from the pull-down menu.

In the example shown in Fig. 21, the area for drug DR3 is not detected even though drug DR3 is present in the captured image, so the user can press the Add Area button BT2 and select an area to add it. To add an area, for example, tap any point and then drag it to the lower right to add a rectangular area, and adjust the width, height, rotation angle, etc. of the rectangle using the method described in Fig. 22.

Alternatively, the implementation can be such that a point on the screen of the overall image display unit EDA is pressed and held to display a default rectangle with an undefined area, and an area can be added, and this rectangle can then be translated to an appropriate position and corrected by adjusting the width, height, and rotation angle of the rectangle.

[GUI for displaying detection results after area determination]
Fig. 23 is an example of a screen display in a state where the identification of a drug has been confirmed. As shown in screen SC3 in Fig. 23, a check mark indicating the status of "drug type confirmed" is placed in check box CB for each of boxes BX1 and BX4 of drugs DR1 and DR3 whose drug type has been confirmed. It is possible to display a pull-down menu by pressing and holding the check box CB, and change the status from the pull-down menu. For example, it is possible to change the status from "drug type confirmed" to "unconfirmed state", etc.

In addition, for drugs that are difficult to identify, the drug type cannot necessarily be identified, so a status of "drug type determination pending" may be displayed. For example, an "X" mark indicating the status of "drug type determination pending" may be displayed in the check box CB of drug DR2, for which drug type determination is pending. Note that in the "unconfirmed state" where the drug type has neither been determined nor pending, nothing is displayed in the check box CB (see Figure 22). The default for the check box CB is blank, indicating an "unconfirmed state."

Once the identification status of all medications has been determined, the user presses the Complete button BT8 to complete the identification process for the captured image.

[Identification result confirmation GUI (for identifiable drugs)]
Fig. 24 is an example of a screen display of the identification result confirmation GUI. Fig. 24 shows an example of a GUI that displays drug information about drugs belonging to the group of "drug type identifiable drugs". Note that, although the illustration of the whole image display unit EDA is omitted in Fig. 24, the whole image display unit EDA exists above the candidate display unit CDA of the screen SC4 shown in Fig. 24 (see Fig. 23).

The candidate display section CDA displays drug information such as the front and back of the master image of the drug estimated by the drug identifier 120 for the drug cut-out image of the area selected in the overall image display section EDA, the drug code, drug name, and size (major axis and minor axis dimensions). In addition, the type of group to which the selected drug belongs is displayed at the bottom of the drug information display section where this drug information is displayed. The group type is displayed in a format in the selection box SLB1.

When the right arrow button BT9 on the right edge of the screen SC4 is pressed or swiped left, the candidate drugs with lower scores are displayed in order in the drug information display section of the candidate display section CDA. When the left arrow button BT10 on the left edge of the screen is pressed or swiped right, the candidate drugs with higher scores are displayed in order in the drug information display section of the candidate display section CDA.

Also, if the inferred group is incorrect, the pull-down menu PDM can be displayed by pressing the pull-down button in the selection box SLB1, and the correct group can be selected from the pull-down menu PDM to transition to a GUI specific to each group.

When the correct drug cannot be found among the candidate drugs output by the drug identifier 120, for example, when performing a search using a text-based method that does not rely on the drug type estimation by the drug identifier 120, pressing the engraving text search button BT4 will transition to the engraving text search GUI.

The Confirm Drug Type button BT5 is pressed when confirming the drug type for the drug displayed or selected in the candidate display area CDA. When the Confirm Drug Type button BT5 is pressed, the drug type is confirmed and a check mark is displayed in the check box CB for the corresponding drug in the overall image display area EDA. The Hold Drug Type Confirmation button BT6 is pressed when identification of the selected drug is to be completed without determining the drug type. When the Hold Drug Type Confirmation button BT6 is pressed, an "X" mark is displayed in the check box CB for the corresponding drug in the overall image display area EDA.

When the arrow button BT11 in the upper right corner of the candidate display section CDA is pressed, the screen transitions to the candidate list display GUI (see Figure 25).

[Candidate List Display GUI]
FIG. 25 is an example of a screen display of the candidate list display GUI. The candidate list display GUI is a GUI that displays a list of drugs with high inference scores by the drug identifier 120 when the drug related to the selection belongs to the group of "drugs that can be identified by drug type". As shown in FIG. 25, a plurality of candidate drugs are displayed in a list, so that comparison and consideration can be made. Note that although the illustration of the whole image display unit EDA and the button display unit BDA is omitted in FIG. 25, the whole image display unit EDA exists above the screen SC5 of the candidate display unit CDA shown in FIG. 25, and the button display unit BDA exists below the screen of the candidate display unit CDA (see FIG. 23). The same applies to each of FIGS. 26 to 29.

Figure 25 shows an example in which drug information for the top four drugs with the highest scores is displayed in a list. By operating the knob on the scroll bar SRB at the right end of screen SC5 downward or swiping the screen, lower-ranked candidates can also be displayed. The drug type can be confirmed by selecting any drug on this candidate image list screen SC5 and pressing the "Drug Type Confirmation Button BT5" in the button display section BDA.

The display order in the list may be in the order of the inference score by the drug identifier 120, or, for example, drugs similar to the imprinted text of the drug with the highest score by the drug identifier 120 may be searched from the imprinted text database and displayed in order of their similarity score.

In addition, the display format of the list display is not limited to a vertical line display as in Figure 25, but may be a multi-row, multi-column display using a wider area of the screen.

In the candidate image list, it is preferable to display the master image (front and back), text information indicating the drug name, and numerical size information as well as the engraving text information (master text information) registered in the engraving text database as drug information for each drug. In Figure 25, the term "master text information" is used for convenience of illustration, but the actual screen display will display the engraving text information registered for each drug.

When the arrow button BT11 in the upper right corner of the candidate display section CDA is pressed, the user returns to the identification result confirmation GUI (see Figure 24).

[Engraving Text Search GUI]
Fig. 26 is an example of the screen display of the engraving text GUI. The engraving text GUI is a GUI that displays a list of drugs with text similar to the text entered in the search box SBX1 when the drug related to the selection belongs to the group of "drug type identifiable drugs". When the engraving text search button BT4 is pressed on the screen of Fig. 24 or Fig. 25, the engraving text GUI screen SC6 including the search box SBX1 is displayed as shown in Fig. 26.

The user can search by engraved characters by entering text into search box SBX1 using keyboard input or voice input and pressing search button SBT1. Search results are displayed as a list, similar to Figure 25. The display format of the list of candidate drugs is also the same as Figure 25. A similarity score determination is performed each time a character is entered into search box SBX1, and the candidate list display may be updated immediately.

[Capsule Search GUI]
27 is an example of a screen display of a capsule search GUI. The capsule search GUI is a GUI that is displayed in the candidate display unit CDA when a “capsule drug” is identified by the drug identifier 120 or when “capsule” is selected in the pull-down menu PDM of the group selection box SLB1.

The capsule search screen SC7 includes a text search box SBX2, a size search box SZB, capsule color selection boxes SLB2, SLB3, a text color selection box SLB4, a capsule image display section SRD, a group selection box SLB1, and a capsule list button BT13.

By entering text into the text search box SBX2 using the keyboard or voice input and pressing the search button SBT2, you can search by the characters printed on the capsule. You can also search by the text of the drug name.

The size search box SZB includes an input box IB1 for inputting the numerical value of the long diameter, an input box IB2 for inputting the numerical value of the short diameter, and a search button BT12. By inputting numerical values into the input boxes IB1 and IB2 by keyboard input or voice input and pressing the search button BT12, it is possible to search by the size of the capsule medicine.

The search results are displayed on the capsule image display unit SRD. In the case of a text search, the front and back of the master image are displayed from the top in order of highest text similarity score. A similarity score determination is performed each time a character is entered into the text search box SBX2, and the candidate list display may be updated immediately. In a capsule search, the text search is limited to capsules. On the other hand, when searching by size, the front and back of the master image are displayed from the top in order of highest degree of match with the entered long axis and short axis values. In this case too, the search range is limited to capsules.

The order in which the master images of capsule drugs are displayed on the capsule image display unit SRD may refer to past identification history data and prioritize display of capsules with identification history. By operating the knob of the scroll bar SRB on the right edge of the screen SC7 downward or swiping the screen SC7, lower-level candidates can also be displayed.

The format of the candidate display is not limited to a vertical line display as shown in Figure 27, but may be a multi-row, multi-column display.

When the capsule list button BT13 at the bottom right of the screen SC7 is pressed, the capsule image display section SRD expands to the right, and more capsule drug candidates are displayed at once. In this case, the capsule list button BT13 is replaced with a "back button" (not shown), and pressing the back button returns to the original capsule candidate screen (Figure 27).

The type of drug can be confirmed by selecting any capsule from the capsule images displayed in the capsule image display unit SRD and pressing the "Drug Type Confirmation Button BT5" on the button display unit BDA.

The capsule color selection boxes SLB2 and SLB3 and the character color selection box SLB4 may be used to specify the capsule color and the color of the printed characters to narrow down the candidates. Alternatively, the capsule color may be automatically recognized from the captured image to narrow down the candidates in advance, and the selection boxes SLB2 and SLB3 may be displayed as default. In the list of candidate capsule images, it is preferable to display the master image (front and back), character information indicating the drug name, and numerical size information as well as the engraving text information (master character information) registered in the engraving text database as drug information for each drug.

[Blank Drug Search GUI]
28 is an example of a screen display of a plain medicine search GUI. The plain medicine search GUI is a GUI that is displayed in the candidate display unit CDA when a medicine is identified as a "plain medicine" by the medicine identifier 120 or when a "plain medicine" is selected in the pull-down menu PDM of the group selection box SLB1.

The plain drug search screen SC8 includes a color selection box SLB5, a plain drug image display section SRD2, a group selection box SLB1, and a plain drug list button BT14.

The plain medicine image display section SRD2 displays the front and back of the master image from top to bottom in order of the degree of match with the major axis and minor axis values of the medicine selected in the whole image display section EDA. In this case, the search range is limited to plain tablets.

The display order of the master images of plain drugs displayed on the plain drug image display unit SRD2 may refer to past discrimination history data and prioritize display of plain drugs with discrimination history. Lower-ranked candidates can also be displayed by operating the slider bar knob on the right edge of the screen downward or swiping the screen.

The format of the candidate display is not limited to a vertical line display as shown in Figure 27, but may be a multi-row, multi-column display.

When the plain drug list button BT14 at the bottom right of screen SC8 is pressed, the plain drug image display section SRD2 expands to the right, and more plain drug candidates are displayed at once. In this case, the plain drug list button BT14 is replaced with a "back button" (not shown), and when the back button is pressed, the screen returns to the original plain drug candidate screen (Figure 28).

The color selection box SLB5 may be used to specify the color of the plain medicine to narrow down the candidates. Also, the color of the plain medicine may be automatically recognized from the captured image to narrow down the candidates in advance, and the selection box SLB5 may be displayed as a default.

The user can confirm the type of drug by selecting one of the capsules from the capsule images displayed in the capsule image display unit SRD and pressing the drug type confirmation button BT5 on the button display unit BDA.

[Divided Tablet Search GUI]
29 is an example of a screen display of a divided tablet search GUI. The divided tablet search GUI is a GUI that is displayed in the candidate display section CDA when a "divided tablet" is identified by the medicine identifier 120 or when a "divided tablet" is selected from the pull-down menu of the group selection box SLB1.

The divided tablet search screen SC9 includes a divided tablet image display section LS1, a divided tablet call button BT15, a search box SBX3, a candidate drug image display section LS2, and a group selection box SLB1.

The split tablet image display unit LS1 displays a list of the split tablet image selected in the overall image display unit EDA and the split tablet image called up by pressing the split tablet call button BT15. For example, the split tablet image selected in the overall image display unit EDA is displayed at the top of the list display in the split tablet image display unit LS1. The split tablet image display unit LS1 may be configured to clearly distinguish between the display unit that displays the split tablet image called up by pressing the split tablet call button BT15 and the display unit that displays the split tablet image selected in the overall image display unit EDA.

When the user presses the divided tablet call button BT15, one or more divided tablet images can be selected from the list of previously registered divided tablet images and added to the list display in the divided tablet image display section LS1. The following implementation is possible as a method for registering divided tablet images. That is, when the group of drugs selected in the whole image display section EDA is in the "divided tablet" state, pressing and holding the selected area on the whole image display section EDA displays a pull-down menu, and the "divided tablet image registration" item can be selected from this pull-down menu. When "divided tablet image registration" is selected, the corresponding divided tablet image is stored in the shared section, and the image can be called up by pressing the divided tablet call button BT15 on this screen. The shared section is a part of the memory area of the storage device 104, and is a memory area in which data and the like that are shared in the processing in the drug type identification device 100 beyond the identification work unit are stored. The method for registering divided tablet images is not limited to this example, and other methods may be used.

You can search by engraved characters by entering text into the search box SBX3 using the keyboard or voice input and then pressing the search button SBT3 on the far right.

The candidate drug image display unit LS2 displays the front and back of the master image from top to bottom in order of highest text similarity score. A similarity score determination may be performed each time a character is entered into the search box SBX3, and the candidate list display may be updated immediately. The order in which the master images of drugs are displayed in the candidate drug image display unit LS2 may refer to past discrimination history data, and drugs corresponding to split tablets with discrimination history may be displayed preferentially. Lower-ranked candidates can also be displayed by operating the slider bar knob on the right edge of the screen downward or swiping the screen.

The format of the candidate display is not limited to a vertical line display as shown in Figure 27, but may be a multi-row, multi-column display.

The user can confirm the drug type by selecting one of the drugs displayed in the candidate drug image display section LS2 and pressing the “Drug Type Confirmation button BT5” in the button display section BDA.

In the list of candidate drug images in Figure 29, it is preferable to display, as drug information for each drug, the master image (front and back), text information indicating the drug name, and numerical size information, as well as the engraving text information (master text information) registered in the engraving text database.

Although not shown in FIG. 29, a color selection box may be used to specify the color of the medicine to narrow down the candidates. In this case, the color of the divided tablet may be automatically recognized from the captured image to narrow down the candidates in advance, and the color selection box may be displayed as a default.

[Improvements to improve usability in each search GUI]
In the search screen exemplified using Figures 20 to 29, multiple icons representing typical drug shapes may be further arranged, and when the user selects one or more icons, only drugs with the corresponding shape may be displayed on the candidate screen.

Figure 30 shows examples of icons representing typical drug shapes. Typical drug shapes may be circle, oval, ellipse, pentagon, and hexagon. As shown in Figure 30, graphic icons corresponding to each of these typical shapes and an "Other" button may be arranged on the search screen, allowing the user to specify a shape by selecting an icon.

Color selection is not limited to being selected from a pull-down menu; color options can also be provided using icons.

Figure 31 shows an example of icons used for color selection. In Figure 31, an example is shown in which, from the left, icons of white, yellow, orange, brown, red, blue, green, and transparent are arranged, along with an "Other" button. Of course, the order in which the colors are arranged and the types and number of colors displayed as icons can be designed as appropriate.

An icon corresponding to each color and an "Other" button may be placed on the search screen, and the color may be specified by selecting an icon. When the user selects one or more icons, only drugs of the corresponding color may be displayed on the candidate screen.

[Example of configuration for performing group-specific drug type identification processing]
32 is a block diagram showing an example of a functional configuration for identifying the type of a drug difficult to identify in the drug type identification device 100 according to the embodiment. The drug type identification device 100 includes a drug type identification processing control unit 160 that controls the processing contents based on the information estimated by the drug identifier 120, a drug type estimation result presentation processing unit 170, a capsule drug identification processing unit 172, a plain drug identification processing unit 174, and a dividable tablet identification processing unit 176.

If the drug to be identified is a drug with a drug type that can be identified, the drug identifier 120 outputs drug type estimation information that estimates the type of drug, and if the drug to be identified is a drug with a drug type that is difficult to identify, it outputs group estimation information that estimates the group to which the drug belongs. The drug type identification processing control unit 160 acquires the estimation information output from the drug identifier 120 and assigns subsequent processing according to the estimation information.

When the drug type identification processing control unit 160 acquires the drug type estimation information from the drug identifier 120, it executes the processing of the drug type estimation result presentation processing unit 170. The drug type estimation result presentation processing unit 170 performs processing to display drug information of the candidate drug in the candidate display unit CDA based on the drug estimation information estimated by the drug identifier 120. The processing of the drug type estimation result presentation processing unit 170 realizes the screen display described in Figures 24 and 25. The drug type estimation result presentation processing unit 170 works in conjunction with the text search unit 122, and can transition to engraving text search processing when the engraving text search button BT4 is pressed (see Figure 26).

The drug type identification processing control unit 160 includes a group discrimination unit 162. When the estimation information output from the drug identifier 120 is group estimation information, the group discrimination unit 162 discriminates the label of the estimated group. Here, an example is shown in which it is determined which group it belongs to: "capsule drug", "plain drug", or "divisible tablet".

When the group estimation information acquired from the drug identifier 120 indicates a group of "capsule drug", the drug type identification processing control unit 160 executes processing of the capsule drug identification processing unit 172. The capsule drug identification processing unit 172 performs processing to provide a capsule search GUI (see FIG. 27) for assisting in identifying the drug type of the capsule drug. The capsule drug identification processing unit 172 includes a capsule drug search unit 173. The capsule drug search unit 173 accepts input of search conditions, executes search processing based on the accepted search conditions, and outputs the search results.

When the group estimation information acquired from the drug identifier 120 indicates a group of "plain drugs", the drug type identification processing control unit 160 executes processing of the plain drug identification processing unit 174. The plain drug identification processing unit 174 performs processing to provide a plain drug search GUI (see FIG. 28) to assist in identifying the drug type of plain drugs. The plain drug identification processing unit 174 includes a plain drug search unit 175. The plain drug search unit 175 accepts input of search conditions, executes search processing based on the accepted search conditions, and outputs the search results.

If the group estimation information acquired from the drug identifier 120 indicates a group of "divided drugs", the drug type identification processing control unit 160 executes the processing of the divided tablet identification processing unit 176. The divided tablet identification processing unit 176 performs processing to provide a divided tablet search GUI (see FIG. 29) to assist in identifying the drug type of a divided tablet. The divided tablet identification processing unit 176 includes a divided tablet search unit 177 and a divided tablet image registration unit 178. The divided tablet search unit 177 accepts input of search conditions, executes search processing based on the accepted search conditions, and outputs the search results. The divided tablet image registration unit 178 is a processing unit that performs the registration processing and readout processing of the divided tablet image described in FIG. 29.

In addition, when the group labels are defined in a hierarchical structure, each of the capsule drug search unit 173, the plain drug search unit 175, and the divided tablet search unit 177 can narrow down the search conditions based on the estimated group labels.

The processing including providing a group-specific search GUI performed by each of the capsule drug identification processing unit 172, the plain drug identification processing unit 174, and the divided tablet identification processing unit 176 is an example of processing that leads to the identification of the type of drug in each group (group-specific processing).

<Shooting support equipment>
Fig. 33 is a top view of a photographing assistant device 70 for photographing an image to be input to the drug type identification device 100. Fig. 34 is a cross-sectional view taken along line 34-34 in Fig. 33. Fig. 34 also shows a smartphone 10 that photographs an image of a drug using the photographing assistant device 70.

33 and 34, the photography assistance device 70 has a housing 72, a medicine mounting stand 74, a main light source 75, and an auxiliary light source 78. In FIG. 33, the shape is depicted as being based on a square, but the housing 72, the medicine mounting stand 74, and the auxiliary light source 78 may be rectangular.

The housing 72 is composed of a horizontally supported square bottom panel 72A and four rectangular side panels 72B, 72C, 72D, and 72E fixed vertically to the ends of each side of the bottom panel 72A.

The drug placement stand 74 is fixed to the upper surface of the bottom plate 72A of the housing 72. The drug placement stand 74 is a member having a surface on which drugs are placed, and in this case is a thin, plastic or paper plate-like member that is square when viewed from above, and the placement surface on which the drug to be identified is placed has a reference gray color. When expressed in 256 gradation values from 0 (black) to 255 (white), the reference gray color is, for example, a gradation value in the range of 130 to 220, and more preferably a gradation value in the range of 150 to 190.

Generally, when a medicine is photographed with a smartphone 10 against a white or black background, the automatic exposure adjustment function can wash out the colors, making it difficult to obtain sufficient imprint information. With the medicine placement stand 74, the placement surface is gray, so the colors do not wash out and the details of the imprint can be captured. In addition, by obtaining the gray pixel values captured in the captured image and correcting them against the true gray gradation values, color correction or exposure correction of the captured image can be achieved.

Reference markers 74A, 74B, 74C, and 74D, each made of black and white, are attached or printed on the four corners of the mounting surface of the drug mounting table 74. Any material may be used for the reference markers 74A, 74B, 74C, and 74D, but simple circular markers with high detection robustness are used here.

It is preferable that the reference markers 74A, 74B, 74C, and 74D are 3 to 30 mm in size in the vertical and horizontal directions, and more preferably 5 to 15 mm in size.

In addition, it is preferable that the distance between reference marker 74A and reference marker 74B, and the distance between reference marker 74A and reference marker 74D are each 20 to 100 mm, and more preferably 20 to 60 mm.

Furthermore, as shown in Fig. 33, the four reference markers 74A, 74B, 74C, and 74D may be connected by straight lines to clearly indicate a rectangular drug placement range. Fig. 33 shows an example in which the four reference markers 74A, 74B, 74C, and 74D are arranged at the vertices of a square, but the arrangement of the reference markers 74A, 74B, 74C, and 74D is not limited to the example in Fig. 33. For example, the four reference markers 74A, 74B, 74C, and 74D may be arranged at the vertices of a rectangle.

The drug to be photographed is placed inside a rectangular area (drug placement range) with four reference markers 74A, 74B, 74C, and 74D as vertices. Figure 32 shows an example in which five drugs T1, T2, T3, T4, and T5 are photographed.

In addition, the drug placement stand 74 may also be served as the bottom plate 72A of the housing 72.

The main light source 75 and the auxiliary light source 78 constitute an illumination device used to capture an image of the drug to be identified. The main light source 75 is used to extract the markings on the drug to be identified. The auxiliary light source 78 is used to accurately capture the color and shape of the drug to be identified. The photography auxiliary device 70 does not need to be equipped with the auxiliary light source 78.

Figure 35 is a top view of the shooting assistance device 70 with the auxiliary light source 78 removed.

The main light source 75 is composed of multiple LEDs 76. Each LED 76 is a white light source with a light-emitting portion having a diameter of 10 mm or less. Here, six LEDs 76 are arranged horizontally at a constant height on each of the four rectangular side panels 72B, 72C, 72D, and 72E. This allows the main light source 75 to irradiate illumination light onto the drug to be identified from at least four directions. It is sufficient for the main light source 75 to be able to irradiate illumination light onto the drug to be identified from at least two directions.

The angle θ between the light emitted by the LED 76 and the top surface (horizontal plane) of the drug to be identified is preferably within the range of 10° to 20° in order to extract the marking. The main light source 75 may be composed of rod-shaped light sources with a width of 10 mm or less arranged horizontally on each of the four rectangular side panels 72B, 72C, 72D, and 72E.

The main light source 75 may be constantly on. This allows the photography assistance device 70 to irradiate the drug to be identified from all directions with illumination light. An image taken with all LEDs 76 turned on is called a fully illuminated image. A fully illuminated image makes it easy to extract the printing on the drug to be identified that has printing added thereto.

The main light source 75 may be an LED 76 that is switched on and off depending on the timing, or an LED 76 that is switched on and off by a switch (not shown). This allows the photography assistance device 70 to irradiate the drug to be identified with illumination light from multiple different directions using multiple main light sources 75.

For example, an image captured with only the six LEDs 76 on side panel 72B turned on is called a partial illumination image. Similarly, a partial illumination image captured with only the six LEDs 76 on side panel 72C turned on, a partial illumination image captured with only the six LEDs 76 on side panel 72D turned on, and a partial illumination image captured with only the six LEDs 76 on side panel 72E turned on can be captured to obtain four partial illumination images each illuminated with illumination light from a different direction. The partial illumination images illuminated with illumination light from a plurality of different directions facilitate the extraction of the markings of the identification target drug to which the markings are added.

The auxiliary light source 78 is a flat, planar white light source having a square outer shape and a square opening in the center. The auxiliary light source 78 may be an achromatic reflector that diffuses and reflects the light emitted by the main light source 75. The auxiliary light source 78 is disposed between the smartphone 10 and the drug placement table 74 so that the light is evenly emitted from the shooting direction (the optical axis direction of the camera) onto the drug to be identified. The illuminance of the light emitted by the auxiliary light source 78 onto the drug to be identified is relatively lower than the illuminance of the light emitted by the main light source 75 onto the drug to be identified.

Figure 36 is a top view of a photographing auxiliary device 80 according to another embodiment. Figure 37 is a cross-sectional view taken along line 37-37 in Figure 36. Figure 37 also shows a smartphone 10 that photographs an image of a drug using the photographing auxiliary device 80. Note that in Figures 36 and 37, parts common to Figures 33 and 34 are given the same reference numerals, and detailed descriptions thereof will be omitted. As shown in Figures 36 and 37, the photographing auxiliary device 80 has a housing 82, a main light source 84, and an auxiliary light source 86. The photographing auxiliary device 80 does not have to be equipped with the auxiliary light source 86.

The housing 82 has a cylindrical shape and is composed of a horizontally supported circular bottom plate 82A and a side plate 82B fixed vertically to the bottom plate 82A. The drug placement table 74 is fixed to the upper surface of the bottom plate 82A.

The main light source 84 and the auxiliary light source 86 constitute an illumination device used to capture images of the drug to be identified. Figure 38 is a top view of the imaging auxiliary device 80 with the auxiliary light source 86 removed.

The main light source 84 is configured by 24 LEDs 85 arranged in a ring shape at a constant height and at constant intervals in the horizontal direction on the side panel 82B. The main light source 84 may be always on, or the LEDs 85 that are turned on and off may be switched on and off.

The auxiliary light source 86 is a flat, planar white light source having a circular outer shape and a circular opening in the center. The auxiliary light source 86 may be an achromatic reflector that diffusely reflects the light emitted by the main light source 84. The illuminance of the light emitted by the auxiliary light source 86 on the drug to be identified is relatively lower than the illuminance of the light emitted by the main light source 84 on the drug to be identified.

The photographing assistant device 70 and the photographing assistant device 80 may be provided with a fixing mechanism (not shown) that fixes the smartphone 10 that photographs the drug to be identified at a standard shooting distance and shooting viewpoint position. The fixing mechanism may be configured to be able to change the distance between the drug to be identified and the camera according to the focal length of the photographing lens 50 of the smartphone 10.

[Lighting device]
Fig. 39 is a cross-sectional view showing the configuration of an illumination device 81 as a photographing auxiliary device according to another embodiment. The illumination device 81 shown in Fig. 39 has a configuration in which the bottom plate 82A and the medicine placement table 74 are removed from the configuration of the photographing auxiliary device 80 described in Figs. 37 and 38. The other configuration may be the same as that of the photographing auxiliary device 80. In addition, a diffusion plate cover (not shown) that covers the LED 85 on the side may be provided.

<Example of fiducial marker>
Fig. 40 is a top view of the medicine placement table 74 using a circular marker MC1. As shown in Fig. 40, the medicine placement table 74 has circular markers MC1 arranged at the four corners as reference markers 74A, 74B, 74C, and 74D. The left diagram F40A in Fig. 40 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a square, and the right diagram F40B in Fig. 40 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a rectangle. The reason for arranging four reference markers is that the coordinates of four points are required to determine a perspective transformation matrix for standardization.

Here, the four reference markers 74A, 74B, 74C, and 74D are the same size and color, but the sizes and colors may be different. If the sizes are different, it is preferable to arrange the adjacent reference markers 74A, 74B, 74C, and 74D so that their centers form the four vertices of a square or rectangle. By making the sizes or colors of the reference markers different, it becomes easier to identify the shooting direction.

In addition, at least four circular markers MC1 may be arranged, and five or more may be arranged. When five or more are arranged, it is preferable that the centers of the four circular markers MC1 form the four vertices of a square or a rectangle, and the centers of the additional circular markers MC1 are arranged on the sides of the square or the rectangle. By arranging five or more reference markers, it becomes easier to identify the shooting direction. In addition, by arranging five or more reference markers, even if detection of any of the reference markers fails, it is possible to increase the probability of simultaneously detecting the minimum four points required to obtain the perspective transformation matrix for standardization, and there is an advantage in that the effort of re-shooting is reduced.

The circular marker MC1 comprises an outer first perfect circle with a relatively large diameter, and an inner second perfect circle arranged concentrically with the first perfect circle and with a relatively smaller diameter than the first perfect circle. In other words, the first and second perfect circles are circles with different radii arranged at the same center. In addition, the inside of the second perfect circle of the circular marker MC1 is white, and the area inside the first perfect circle and outside the second perfect circle is filled in black.

It is preferable that the diameter of the first true circle is 3 to 20 millimeters. It is also preferable that the diameter of the second true circle is 0.5 to 5 millimeters. Furthermore, it is preferable that the ratio of the diameters of the first true circle and the second true circle (diameter of the first true circle/diameter of the second true circle) is 2 to 10.

There is a possibility that the coordinates of the center point of the true object in the pre-standardized image may differ from the coordinates of the center point estimated by machine learning. By providing the coordinates of the center point of the inner second perfect circle as training data, as in the case of the circular marker MC1, machine learning can accurately and easily estimate the center coordinates of the true object. In addition, the effect of the presence of the relatively large outer first perfect circle can significantly reduce the possibility of false detection due to dust or the like attached to the circular marker MC1.

Figure 41 is a top view of the medicine placement table 74 using a circular marker according to a modified example. The left diagram F41A of Figure 41 is a diagram showing an example in which the circular marker MC2 is used as the reference markers 74A, 74B, 74C, and 74D. The circular marker MC2 is arranged inside a perfect circle filled with black, with a cross-shaped figure consisting of two white straight lines that intersect at right angles, so that the intersection of the two straight lines coincides with the center of the perfect circle. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC2 are arranged so that their respective centers form the four vertices of a square, and the straight lines of the cross-shaped figure of the circular marker MC2 are arranged parallel to the sides of this square. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC2 may be arranged so that their respective centers form the four vertices of a rectangle. The thickness of the lines of the cross-shaped figure of the circular marker MC2 can be determined appropriately.

On the other hand, the right diagram F40B of FIG. 41 is a diagram showing an example in which a circular marker MC3 is used as the reference markers 74A, 74B, 74C, and 74D. The circular marker MC3 has two perfect circles with different radii arranged at the same center, the inside of the inner perfect circle is white, and the inside of the outer perfect circle and the outside of the inner perfect circle are filled in black. Furthermore, the circular marker MC3 has a cross-shaped figure consisting of two black straight lines that are perpendicular to each other arranged inside the inner perfect circle so that the intersection of the two straight lines coincides with the center of the perfect circle. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC3 are arranged so that their respective centers form the four vertices of a square, and the straight lines of the cross-shaped figure of the circular marker MC3 are arranged parallel to the sides of this square. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC3 may be arranged so that their respective centers form the four vertices of a rectangle. The thickness of the lines of the cross-shaped figure of the circular marker MC3 can be determined appropriately.

The circular markers MC2 and MC3 can improve the accuracy of estimating the center point coordinates. In addition, the circular markers MC2 and MC3 have a different appearance from the medication, making the markers easier to recognize.

Figure 42 is a diagram showing a specific example of a reference marker with a rectangular outline. The left diagram F42A in Figure 41 shows a rectangular marker MS1. The rectangular marker MS1 comprises an outer square SQ1 with a relatively long side length, and an inner square SQ2 that is arranged concentrically with the square SQ1 and has a relatively shorter side length than the square SQ1. In other words, the squares SQ1 and SQ2 are quadrangles with different side lengths that are arranged at the same center (center of gravity). Furthermore, in the rectangular marker MS1, the inside of the square SQ2 is white, and the area inside the square SQ1 and outside the square SQ2 is filled in black.

The length of one side of the square SQ1 is preferably 3 to 20 millimeters. The length of one side of the square SQ2 is preferably 0.5 to 5 millimeters. The ratio of the lengths of one side of the squares SQ1 and SQ2 (length of one side of the square SQ1/length of one side of the square SQ2) is preferably 2 to 10. In order to further improve the accuracy of estimating the central coordinates, a black rectangle (e.g., a square) (not shown) with a side length relatively smaller than that of the square SQ2 may be placed concentrically inside the square SQ2.

The right diagram F42B of Figure 42 shows a top view of the medicine mounting table 74 using a quadrilateral marker MS1. As shown in the right diagram F41B, the medicine mounting table 74 has quadrilateral markers MS1 arranged at its four corners as reference markers 74A, 74B, 74C, and 74D. Here, an example is shown in which the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D form a square, but the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D may also form a rectangle.

Figure 43 is a top view of the medicine placement table 74 using a circular marker according to another modified example. The left diagram F43A in Figure 42 is a diagram showing an example in which a circular marker MC4 is used as the reference markers 74A, 74B, 74C, and 74D. The circular marker MC4 includes an outer first perfect circle with a relatively large diameter, an inner second perfect circle arranged concentrically with the first perfect circle and with a relatively smaller diameter than the first perfect circle, and a black circle (third perfect circle) arranged concentrically with the second perfect circle inside the second perfect circle and with a relatively smaller diameter than the second perfect circle. The inside of the second perfect circle of the circular marker MC4 is white, and the area inside the first perfect circle and outside the second perfect circle is filled in black.

The diameter of the first true circle is preferably 3 to 20 millimeters. The diameter of the second true circle is preferably 5 to 18 millimeters. The diameter of the third true circle is preferably 0.5 to 5 millimeters. Furthermore, the ratio of the diameters of the first true circle to the second true circle (diameter of the first true circle/diameter of the second true circle) is preferably 1.1 to 3. Furthermore, the ratio of the diameters of the second true circle to the third true circle (diameter of the second true circle/diameter of the third true circle) is preferably 2 to 10.

The left diagram F43A in FIG. 43 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a square, and the right diagram F43B in FIG. 43 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a rectangle. Straight lines connecting the four circular markers MC4 in the vertical and horizontal directions are displayed. The circular markers MC4 can improve the estimation accuracy of the center point coordinates. In addition, by connecting the circular markers MC4 with straight lines, it becomes easier to recognize the markers and the drug placement range, and it is also possible to correct the distortion of the captured image using the straight lines in the captured image or specify the cut-out range of the standardized image. These straight lines are not necessary.

Figure 44 shows another specific example of a quadrilateral reference marker. The left diagram F44A in Figure 44 shows a quadrilateral marker MS2. The quadrilateral marker MS2 includes an outer square SQ1 with a relatively large side length, an inner square SQ2 arranged concentrically with the square SQ1 and with a relatively smaller side length than the square SQ1, and a black square SQ3 arranged concentrically inside the square SQ2 and with a relatively smaller side length than the square SQ2. The length of one side of the square SQ1 is preferably 3 to 20 millimeters. The length of one side of the square SQ2 is preferably 5 to 18 millimeters. The length of one side of the square SQ3 is preferably 0.5 to 5 millimeters. Furthermore, the ratio of the lengths of one side of the squares SQ1 and SQ2 (the length of one side of the square SQ1/the length of one side of the square SQ2) is preferably 1.1 to 3. It is preferable that the ratio of the length of one side of the square SQ2 to the length of one side of the square SQ3 (length of one side of the square SQ2/length of one side of the square SQ3) be 2-10.

The right diagram F44B of FIG. 44 shows a top view of the medicine placement table 74 using a quadrilateral marker MS2. As shown in the right diagram F44B, the medicine placement table 74 has quadrilateral markers MS2 arranged at the four corners as reference markers 74A, 74B, 74C, and 74D. Here, an example is shown in which the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D form a square, but the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D may form a rectangle. Also, as in FIG. 43, straight lines connecting the four quadrilateral markers MS2 in the vertical and horizontal directions are displayed.

The square marker MS2 can improve the accuracy of estimating the center point coordinates. In addition, by connecting the square markers MS2 with straight lines, it becomes easier to recognize the markers and the drug placement range. Note that these straight lines do not have to be included.

The medicine placement table 74 may contain a mixture of circular markers MC and square markers MS. By mixing circular markers MC and square markers MS, it is expected that the effect of making it easier to identify the shooting direction can be achieved.

Furthermore, it is preferable to employ circular markers rather than rectangular markers, because when detecting the reference markers using a mobile terminal device such as a smartphone, the following constraints (a) to (c) exist.
(a) Since there may be capacity limitations on apps on mobile terminal devices, it is desirable to perform marker detection and drug detection using the same trained model.
(b) In drug detection, it is desirable to use a bounding box with rotation so that parts of other drugs do not enter the cut-out image of the oval tablet. In this case, due to requirement (a), it is necessary to prepare reasonable training data for the rotation angle for marker detection as well.
(c) In the case of a rectangular marker, because of the arbitrariness of the rotation angle of the bounding box due to four-fold symmetry, it is difficult to create reasonable training data for the rotation angle of the rectangular marker in the pre-standardized image that is the input image when the marker is detected. In contrast, for a circular marker, it is possible to create reasonable training data with a rotation angle of 0 degrees.

In addition, by making circular markers concentric, the accuracy of estimating the center coordinates of the markers can be improved. In pre-standardized images, simple circular markers are distorted in shape, which makes it easy for errors to occur in estimating the center coordinates, but the inner circle of the concentric circles has a narrower range, making it easier for the trained model to identify the center coordinates even in distorted pre-standardized images. Furthermore, the outer circle of the concentric circles has the advantage that it has a large structure, making it easier for the trained model to find, and is robust against noise and debris. It is also possible to improve the accuracy of estimating the center coordinates of square markers by making them concentric.

<Other aspects of the medicine placement table>
The placement surface of the medicine placement table of the photographing assistant device may be provided with a recess structure for placing the medicine. The recess structure includes a recess, a groove, a depression, and a hole.

Figure 45 shows a medicine mounting table 410 having a recess structure, which is used in place of or in addition to the medicine mounting table 74 (see Figure 33). The medicine mounting table 410 is made of paper, synthetic resin, fiber, rubber, or glass. The left diagram F45A in Figure 45 is a top view of the medicine mounting table 410. As shown in the left diagram F45A, the mounting surface of the medicine mounting table 410 has a gray color, and reference markers 74A, 74B, 74C, and 74D are arranged at the four corners. In Figure 45, the shape of the marker is shown using a circular marker MC1, but it is not limited to the circular marker MC1, and may be a circular marker MC2, MC3, MC4, a square marker MS1, or MS2.

Furthermore, the placement surface of the medicine placement table 410 has a recess structure of 3 rows and 3 columns, totaling 9 recesses 410A, 410B, 410C, 410D, 410E, 410F, 410G, 410H, and 410I. The recesses 410A to 410I are each circular and of the same size when viewed from above.

The right diagram F45B of Figure 45 is a cross-sectional view of the medicine placement platform 410 taken along line 44-44. As shown in the right diagram F45B, the recesses 410A, 410B, and 410C have hemispherical bottom surfaces and are all the same depth. The same is true for the recesses 410D to 410I. The bottom surface does not have to be a perfect hemisphere, but may be a concave curved surface with a constant radius of curvature or a concave curved surface with a gradually changing radius of curvature.

Also, right figure F45B shows tablets T51, T52, and T53 placed in recesses 410A, 410B, and 410C, respectively. Tablets T51 and T52 are each a medicine that is circular when viewed from above and rectangular when viewed from the side. Also, tablet T53 is a medicine that is circular when viewed from above and elliptical when viewed from the side. When viewed from above, tablets T51 and T53 are the same size, and tablet T52 is relatively smaller than tablets T51 and T53. As shown in right figure F45B, tablets T51 to T53 are placed in place by fitting into recesses 410A to 410C. Tablets T51 to T53 may be circular when viewed from above, or may be linear on the left and right and arc-shaped on the top and bottom when viewed from the side.

In this way, the medicine placement table 410 has a hemispherical recess structure on the placement surface, which prevents the circular medicine from moving when viewed from above and allows it to remain stationary. In addition, the position of the medicine when photographing it can be determined to be the position of the recess structure, making it easy to detect the area of the medicine.

The medicine placement table may have a recess structure for rolling medicines such as capsule medicines. Figure 46 is a diagram showing a medicine placement table 412 provided with a recess structure for capsule medicines. Note that parts common to Figure 45 are given the same reference numerals, and detailed descriptions thereof are omitted.

The left diagram F46A in Figure 46 is a top view of the medicine mounting stand 412. As shown in the left diagram F46A, the mounting surface of the medicine mounting stand 412 has a recess structure with a total of six recesses 412A, 412B, 412C, 412D, 412E, and 412F arranged in three rows and two columns. The recesses 412A to 412F are each rectangular of the same size when viewed from above. The recesses do not need to be perfectly hemispherical, and may be concave curved surfaces with a constant radius of curvature or concave curved surfaces with a gradually changing radius of curvature.

The right diagram F46B of Figure 46 is a cross-sectional view of the medicine placement stage 412 taken along line 46-46. As shown in the right diagram F46B, the recesses 412A, 412B, and 412C have semi-cylindrical bottoms and are all the same depth. The same is true for the recesses 412D to 412F. Also, F63B shows capsule medicines CP1, CP2, and CP3 placed in recesses 412A, 412B, and 412C, respectively. The capsule medicines CP1 to CP3 are cylindrical with hemispherical ends (bottoms) and each has a different diameter. As shown in the right diagram F46B, the capsule medicines CP1 to CP3 are placed in place by fitting into the recesses 412A to 412C.

In this way, the medicine placement table 412 has a semi-cylindrical recess structure on the placement surface, which prevents cylindrical capsule medicine from moving or rolling and allows it to remain still. In addition, the position of the medicine when photographing can be determined to be the position of the recess structure, making it easy to detect the area of the medicine.

The medicine mounting table may also have a recess structure for oval tablets. Figure 47 shows medicine mounting table 414 having a recess structure for oval tablets. Note that parts common to Figure 46 are given the same reference numerals and detailed descriptions thereof are omitted.

The left diagram F47A in Fig. 47 is a top view of the medicine mounting stand 414. As shown in the left diagram F47A, the mounting surface of the medicine mounting stand 414 has a recess structure of three rows and two columns, totaling six recesses 414A, 414B, 414C, 414D, 414E, and 414F. Each of the recesses 414A to 414F is rectangular when viewed from above.

The recesses 414A and 414B are the same size. The recesses 414C and 414D are the same size and relatively smaller than the recesses 414A and 414B. The recesses 414E and 414F are the same size and relatively smaller than the recesses 414C and 414D.

Furthermore, the left-hand figure F47A shows tablets T61, T62, and T63 placed in recesses 414B, 414D, and 414F, respectively. As shown in the left-hand figure F47A, recesses 414B, 414D, and 414F are sized to correspond to tablets T61, T62, and T63, respectively.

The right-hand image F47B of Figure 47 is a cross-sectional view of the medicine placement table 414 along line 47-47. As shown in the right-hand image F47B, the recesses 414A and 414B have flat bottom surfaces. As shown in the right-hand image F47B, the tablet T61 fits into the recess 414B and is placed in place. The same is true for tablets T62 and T63.

In this way, the drug placement table 414 has a rectangular parallelepiped recess structure on the placement surface, which prevents oval tablets from moving and allows them to remain still. In addition, the position of the drug when photographing it can be determined to be the position of the recess structure, making it easy to detect the area of the drug.

The shape, number and arrangement of the recess structures are not limited to the aspects shown in Figures 45 to 47, and may be combined, enlarged or reduced as appropriate.

<Hardware configuration of each processing unit and control unit>
The hardware structure of the processing units that execute various processes, such as the image acquisition unit 112, the drug detector 114, the area correction unit 116, the drug area extraction unit 118, the drug identifier 120, the text search unit 122, the display control unit 124, the magnification change unit 125, and the input processing unit 126 described in FIG. 5, the mark extraction unit 140, the outline extraction unit 142, the size measurement unit 144, the loss calculation unit 152, the optimizer 154 described in FIG. 17, the drug type identification processing control unit 160, the group discrimination unit 162, the drug type estimation result presentation processing unit 170, the capsule drug identification processing unit 172, the capsule drug search unit 173, the plain drug identification processing unit 174, the plain drug search unit 175, the divided tablet identification processing unit 176, the divided tablet search unit 177, and the divided tablet image registration unit 178, are various processors as shown below.

Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and function as various processing units, GPUs (Graphics Processing Units), which are processors specialized for image processing, Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically to execute specific processes.

A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. In addition, multiple processing units may be composed of one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system including multiple processing units is used with one IC (Integrated Circuit) chip. In this way, various processing units are configured using one or more of the above various processors as a hardware structure.

More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

<Program for realizing the functions of the drug type identification device 100>
The processing function of the drug type identification device 100 is not limited to the smartphone 10, and can be realized by using various types of information processing devices, such as a tablet computer, a personal computer, or a workstation. A program for causing a computer to realize part or all of the processing functions of the drug type identification device 100 described in the above embodiment can be recorded on a computer-readable medium, which is a non-transitory information storage medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible information storage medium, and the program can be provided through this information storage medium. Instead of providing the program by storing it on such a tangible non-transitory information storage medium, it is also possible to provide a program signal as a download service using an electric communication line such as the Internet.

It is also possible to provide part of the processing functions of the drug type identification device 100 described in the above embodiment as an application server, and to provide a service that provides processing functions via telecommunications lines.

Effects of the embodiment
According to the drug type identification device 100 of the embodiment, even if an image is taken in a state where drugs with identifiable drug types and drugs with difficult drug types are mixed, the type of each drug in the image or the group to which the drug belongs can be automatically identified, and for drugs with difficult drug types, a GUI can be provided that transitions to processing specific to each group and assists in identifying the type. Therefore, it becomes possible to photograph multiple drugs collectively in units that are meaningful from a pharmaceutical perspective and identify each drug.

[Modification 1]
The medication identifier 120 may be configured to include an image recognizer using a template matching technique.

[Modification 2]
In the above embodiment, an example of identifying medicines has been described, but the technology of the present disclosure can also be applied to a case of inspecting medicines. In addition, in the above embodiment, an example of identifying an object to be identified is medicine, but the object to be identified is not limited to medicine. The technology of the present disclosure can be applied as a technology for identifying various objects from images, regardless of the type and use of the object.

<Terminology>
In this disclosure, an "object" refers to an object that can be classified according to a hierarchical structure. A specific depth in the hierarchical structure of the classification is called granularity, and an object belongs to one of the "types" at that granularity. "Identification" refers to determining which type an object belongs to at a certain granularity.

The hierarchical structure may be defined according to the purpose of identification.

The above "certain granularity" shall be determined according to the purpose of identification.

A group is a collection of objects that share a common characteristic. A group may or may not be related to a type. A group may refer to a higher level of the type in the above hierarchical structure, or it may be a newly defined collection (unrelated to the above hierarchical structure) for convenience. Groups may be determined and defined according to the purpose of identification.

[Specific examples]
Drugs can be classified into a hierarchical structure based on efficacy classification or a hierarchical structure based on appearance characteristics. Drug identification can be defined as the act of determining which YJ code the target drug is (this is an example of the definition of drug identification, and for example, the identification code may be defined using a type other than the YJ code). Each group of "capsule drug" and "plain tablet" described in the above embodiment is a collection of types newly defined for convenience. In addition, the "half tablet" group is a collection of drugs newly defined for convenience. Although it is a collection unrelated to the type of YJ code, the purpose of drug identification is to identify the YJ code of the half tablet.

"others"
The technical scope of the present invention is not limited to the scope of the above-mentioned embodiment and modified examples. The configurations and the like in the embodiment and modified examples can be changed without departing from the spirit of the present invention, and the embodiment and modified examples can be appropriately combined.

REFERENCE SIGNS LIST 10 Smartphone 12 Housing 14 Touch panel display 14A Display unit 14B Input unit 16 Speaker 18 Microphone 20 In-camera 22 Out-camera 24 Light 26 Switch 28 CPU
30 Wireless communication unit 32 Communication unit 34 Memory 36 Internal storage unit 38 External storage unit 40 External input/output unit 42 GPS receiver 44 Power supply unit 50 Shooting lens 50F Focus lens 50Z Zoom lens 54 Image sensor 58 A/D converter 60 Lens driver 70 Shooting auxiliary device 72 Housing 72A Bottom plate 72B Side plate 72C Side plate 72D Side plate 72E Side plate 74 Medicine placement table 74A, 74B, 74C, 74D Reference marker 75 Main light source 78 Auxiliary light source 80 Shooting auxiliary device 81 Illumination device 82 Housing 82A Bottom plate 82B Side plate 84 Main light source 86 Auxiliary light source 100 Drug type identification device 102 Processor 104 Storage device 112 Image acquisition unit 114 Drug detector 116 Area correction unit 118 Drug area cutout unit 120 Drug identifier 122 Text search unit 124 Display control unit 125 Magnification change unit 126 Input processing unit 130 Database 131 Master image database 132 Identification result storage unit 140 Engraving extraction unit 142 Outline extraction unit 144 Size measurement unit 150 Machine learning system 151 Learning model 152 Loss calculation unit 154 Optimizer 160 Drug type identification processing control unit 162 Group discrimination unit 170 Drug type estimation result presentation processing unit 172 Capsule drug identification processing unit 173 Capsule drug search unit 174 Plain drug identification processing unit 175 Plain drug search unit 176 Divided tablet identification processing unit 177 Divided tablet search unit 178 Divided tablet image registration unit 410 Drug placement table 410A, 410B, 410C, 410D Depression 410E, 410F, 410G, 410H, 410I Depression 412 Medicine placement stand 412A, 412B, 412C, 412D Indentation 412E, 412F Indentation 414 Medicine placement stand 414A, 414B, 414C, 414D Indentation 414E, 414F Indentation Abs Area BDA Button display area BT2 Add area button BT3 Delete area button BT4 Search engraved text button BT5 Confirm drug type button BT6 Hold confirmation of drug type button BT7 Retake photo button BT8 Done button BT9 Right arrow button BT10 Left arrow button BT11 Arrow button BT12 Search button BT13 Capsule list button BT14 Plain medicine list button BT15 Divided tablet call button BX, BX1, BX2, BX3, BX4 Frame CB Check box CDA Candidate display areas CP1, CP2, CP3 Capsule drugs DR1, DR2, DR3, DRj Drug EDA Whole image display area Egm, Egm1, Egm2, Egm3, Egm4, Egm5 Engraving extraction image F40A Left image F40B Right image F41A Left image F41B Right image F42A Left image F42B Right image F43A Left image F43B Right image F44A Left image F44B Right image F45A Left image F45B Right image F46A Left image F46B Right image F47A Left image F47B Right image GTj Correct data IB1 Input box IB2 Input box IM1j Engraving extraction image IM2j Outline image IMj Drug image LS1 Divided tablet image display area LS2 Candidate drug image display area MC, MC1, MC2, MC3, MC4 Circular marker MGj Magnification information MS, MS1, MS2 Square marker Org, Org1, Org2, Org3, Org4, Org5 Original image Otw, Otw1, Otw2, Otw3, Otw4, Otw5 Outline image PDM Pull-down menu PRj Inference result SBT1, SBT2, SBT3 Search button SBX1 Search box SBX2 Text search box SBX3 Search boxes SC1, SC2, SC3, SC4, SC5, SC6, SC7, SC8, SC9 Screen SLB1, SLB2, SLB3, SLB4, SLB5 Selection box SQ1, SQ2, SQ3 Square SRB Scroll bar SRD Capsule image display area SRD2 Plain drug image display section ST1 GPS satellite ST2 GPS satellite SW1 Area edit switch SZB Size search box SZj Size information T1, T2, T3, T4, T5 Drug T51, T52, T53 Tablet T61, T62, T63 Tablet TM1 First trained model TM2 Second trained models S1 to S5 Steps S11 to S16 of the learning method for constructing a drug type identification device Steps S21 to S29 of the drug type identification method Steps of the drug type identification method

Claims (24)

A detector that detects a plurality of objects on an object-by-object basis from an image in which the objects are captured;
a classifier that estimates a type of an identifiable object from the image among the objects detected by the detector, and estimates a group from the image for a type-unidentifiable object whose type is difficult to identify from the image but whose group to which the object belongs is identifiable;
a processing unit that performs group-specific processing on the objects estimated as the group by the classifier, the processing leading to identification of the type;
An object identification device comprising: the image is an image captured in a state in which the type-identifiable objects and the type-difficult-to-identify objects are mixed together,
The object identification device according to claim 1 . The object is classified into a plurality of groups,
A group-specific process is defined for each of the groups.
The object identification device according to claim 1 or 2. The detector includes a first learned model trained by machine learning using first training data labeled on an object-by-object basis without distinguishing between the identifiable object and the difficult-to-identify object.
The object identification device according to claim 1 or 2 . the classifier includes a second learned model trained by machine learning using second training data in which the identifiable objects are labeled by object type and the difficult-to-identify objects are labeled by group to which the objects belong;
The object identification device according to claim 1 or 2 . The labels for identifying the groups are defined in a hierarchical structure.
The object identification device according to claim 1 or 2 . As an input to the classifier, at least one of an object image obtained by cutting out a region of the object detected by the detector from the image on an object-by-object basis, a character/symbol extraction image including at least one of a character and a symbol extracted from the object image, an external image of the object, and size information of the object is used.
The object identification device according to claim 1 or 2 . Magnification information indicating a magnification or reduction ratio of the object image is further used as an input to the classifier.
The object identification device according to claim 7. the group-specific processing includes processing for displaying a screen for accepting input of a search condition for searching for the type of the object within the estimated group;
The object identification device according to claim 1 or 2 . the substance is a drug;
The difficult-to-identify object includes at least one of a capsule medicine, a plain medicine, and a divided tablet;
The type identifiable object includes a tablet having an imprint or print;
The object identification device according to claim 1 or 2 . As an input to the classifier, at least one of a drug image obtained by cutting out the region of the drug detected by the detector from the image on a drug-by-drug basis, a character/symbol extraction image including at least one of characters and symbols extracted from the drug image, an outline image of the drug, and size information of the drug is used.
The object identification device according to claim 10. An object identification device comprising one or more processors and one or more memories in which a program executed by the one or more processors is stored,
The one or more processors:
A detection process for detecting a plurality of objects on an object-by-object basis from an image in which the objects are captured;
a classification process for estimating the type of an identifiable object from the image among the objects detected by the detection process, and for a difficult-to-identify object whose type is difficult to identify from the image but whose group to which the object belongs can be estimated, estimating the group from the image;
A process of transitioning to a group-specific process leading to identification of the type of the object estimated as the group by the identification process;
An object identification device that performs the above. The one or more processors:
performing the detection process using a detector including a first trained model trained by machine learning using first training data in which the object is labeled on an object-by-object basis without distinguishing between the identifiable object and the difficult-to-identify object;
The object identification device according to claim 12. The one or more processors:
The classification process is performed using a classifier including a second trained model trained by machine learning using second training data in which the identifiable object type is labeled by the object type and the difficult-to-identify object type is labeled by the group to which the object belongs.
14. An object identification device according to claim 12 or 13. The one or more processors:
A process of extracting an area of the object detected by the detection process from the image and generating an object image for each object is performed;
The identification process is performed based on the object image.
14. An object identification device according to claim 12 or 13 . the substance is a drug;
The difficult-to-identify object includes at least one of a capsule medicine, a plain medicine, and a divided tablet;
The type identifiable object includes a tablet having an imprint or marking;
The group-specific processing includes a process of displaying a screen for accepting input of search conditions for searching for the type of the drug within the estimated group.
14. An object identification device according to claim 12 or 13 . A first database in which character and symbol information including at least one of characters and symbols indicated by a mark or print on the medicine is linked to the type of the medicine;
a second database in which master images of the medicine are stored;
Equipped with
The one or more processors:
searching at least one of the first database and the second database based on the received search conditions, and outputting the drug candidates that meet the search conditions;
The object identification device according to claim 16. The one or more processors:
a captured image display unit for displaying the image;
a candidate display unit that displays information on a candidate object based on an estimation result of the classification process; and
14. An object identification device according to claim 12 or 13 . The one or more processors:
performing a process of displaying information on the group to which the candidate object to be displayed on the candidate display unit belongs;
20. The object identification device according to claim 18. The one or more processors:
accepting an instruction to designate the group to which the candidate object to be displayed in the candidate display unit belongs;
Controlling the display of the candidate display unit in accordance with the received instruction.
20. The object identification device according to claim 19. A camera and
a display that displays the image captured by the camera and information about the object estimated from the image; and
The object identification device according to claim 1 or 12 , comprising: 1. A method for object identification executed by one or more processors, comprising:
the one or more processors:
Detecting a plurality of objects on an object-by-object basis from an image in which the objects are captured;
For a type-identifiable object among the detected objects, the type of the object can be estimated from the image, estimating the type of the object from the image, and for a type-difficult object whose type is difficult to identify from the image but whose group to which the object belongs can be identified, estimating the group from the image;
performing group-specific processing on the objects estimated as the group, leading to an identification of the type;
The object identification method includes: On the computer,
A function of detecting a plurality of objects on an object-by-object basis from an image in which the objects are captured;
a function of estimating the type of an identifiable object from the image among the detected objects, and estimating the group from the image for a difficult-to-identify object whose type is difficult to identify from the image but whose group to which the object belongs is identifiable;
A function of performing group-specific processing on the objects estimated as the group, which leads to identification of the type;
A program to achieve this. A non-transitory computer-readable recording medium on which the program according to claim 23 is recorded.

Images