JP7805749B2 - Information collection system, information collection method, program, information collection device, and electronic device - Google Patents

Description

The present invention relates to an information collection system, an information collection method, a program, an information collection device, and an electronic device.

Patients sometimes take photographs of the affected area to make it easier to explain to doctors during diagnosis. Patent Document 1 describes a technology in which a patient's face is photographed with a camera to obtain a facial image, and the content of the medical interview is changed depending on the patient's condition analyzed from the facial image.

Japanese Patent Application Laid-Open No. 2020-27364

In the method of Patent Document 1, the number of questions asked is reduced depending on the patient's condition in order to reduce the burden on the patient, which means there is a risk that insufficient information will be obtained.

Furthermore, even if a patient has saved images of the affected area, they may not be able to accurately remember information that cannot be determined from the images, such as the exact circumstances under which the injury or illness initially occurred, at the time of diagnosis. Furthermore, there may be other symptoms related to the injury or illness that the patient is unaware of. In such cases, there is a risk that useful information to assist the doctor in making a diagnosis will not be conveyed.

The present invention was made in consideration of the above-mentioned problems, and aims to make it possible to obtain information that supports diagnosis.

A first aspect of the present invention is an information collection system comprising: an image acquisition means for acquiring an image of a user's affected area; a determination means for determining which of a plurality of symptoms the image of the affected area shows; and a control means for presenting questions and controlling the system to accept answers to the questions, the control means presenting a first group of questions including questions regarding a first symptom determined based on the image of the affected area, and presenting a second group of questions including questions regarding a second symptom determined based on the image of the affected area and the answers to the first group of questions, wherein the control means, when prompting the user to take an image, presents imaging instruction information which is information instructing the user to take an image to assist in diagnosis, and the imaging instruction information is saved as metadata for an image taken based on the imaging instruction information .

A second aspect of the present invention is an information collection method comprising: an image acquisition step in which a computer acquires an image of a user's affected area; a determination step in which the computer determines which of a plurality of symptoms the image of the affected area shows; and a control step in which the computer presents questions and controls the computer to accept answers to the questions, wherein the computer presents a first group of questions including questions about a first symptom determined based on the image of the affected area, and the computer presents a second group of questions including questions about a second symptom determined based on the image of the affected area and the answers to the first group of questions.In the control step, the computer presents imaging instruction information, which is information instructing the user to take an image to assist in diagnosis, when prompting the user to take an image, and the imaging instruction information is saved as metadata of the image taken based on the imaging instruction information .

A third aspect of the present invention is a program for causing a computer to execute each step of the information gathering method described above.

The present invention makes it possible to obtain information that assists in diagnosis.

1 is a block diagram illustrating an example of the configuration of an information collection system according to a first embodiment. 10 is a flowchart of a diagnostic information collection process according to the first embodiment. FIG. 10 is an explanatory diagram of a first group of questions. FIG. 10 is an explanatory diagram of a second group of questions. FIG. 10 is an explanatory diagram of a second group of questions. These are examples of symptoms targeted by the information collection system. 10 is a display example of a screen related to a diagnostic information collection process in the first embodiment. 10 is a flowchart of a diagnostic information collection process according to the first embodiment. This is an example of a display when imaging the cause of the first symptom. FIG. 10 is an explanatory diagram of an example in which the next question to be presented is changed depending on the answer. 10 is a flowchart of a diagnostic information collection process according to the second embodiment. 10 is a display example of a screen relating to a diagnostic information collection process in the second embodiment. 11 is a flowchart of a diagnostic information collection process according to a third embodiment. 13 is a display example of a screen relating to a diagnostic information collection process in the third embodiment. 13 is a modified example of a screen relating to the diagnostic information collection process in the third embodiment. 10 is a flowchart of a diagnostic information collection process according to a fourth embodiment. 13 is a display example of a screen relating to a diagnostic information collection process in the fourth embodiment. This is an example of a display when a question is determined based on information about the user.

<Embodiment 1>
Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The technical scope of the present invention is determined by the claims and is not limited to the individual embodiments described below. Furthermore, not all combinations of features described in the embodiments are essential to the present invention. Furthermore, features described in separate embodiments can be combined as appropriate.

FIG. 1 is a block diagram showing an example configuration of an information collection system (image processing system) 1 according to a first embodiment. The information collection system 1 includes an operation terminal 100 and an analysis device 110. The operation terminal 100 accepts user operations, such as taking images and answering questions. The operation terminal 100 is, for example, a smartphone. The operation terminal 100 may be an electronic device such as a tablet terminal or a personal computer, or may be implemented by application software installed on a mobile terminal. The CPU 101 controls the entire operation terminal 100. The secondary memory unit 102 is, for example, a hard disk, and stores programs related to the operation of the CPU 101. The primary memory unit 103 is, for example, RAM, and is used as a work memory where programs read from the secondary memory unit 102 are deployed. The optical system 104 is composed of a lens, shutter, and aperture, and focuses light from a subject on the image sensor 105. The image sensor 105 converts the light focused by the optical system 104 into an image. The display unit 106 displays images and text. In the first embodiment, it is assumed that the display unit 106 is configured integrally with the operation terminal 100, but it may also be configured as a separate unit. The operation unit 107 accepts user operations on the operation terminal 100 and outputs operation information corresponding to the user operations. The operation unit 107 is a physical button, touch panel, or the like provided on the operation terminal 100. The communication unit 108 transmits images of the user's affected area (affected area images) to the analysis device 110. The communication unit 108 also receives information on a group of questions from the analysis device 110 and transmits the user's answers to the received group of questions to the analysis device 110. The storage medium 109 is a storage medium (storage unit) that may include a removable optical disk, a magnetic card, an optical card, an IC card, or the like.

The analysis device 110 is an information collection device (server) that determines which of multiple symptoms an image of an affected area indicates and determines the set of questions to present to the user. The CPU 111 controls the entire analysis device 110. The secondary memory unit 112 is, for example, a hard disk, and stores programs related to the operation of the CPU 111. The primary memory unit 113 is, for example, RAM, and is used as work memory where programs read from the secondary memory unit 112 are deployed. The communication unit 114 receives images of affected areas and answers to user questions from the operation terminal 100, and transmits information such as the set of questions to the operation terminal 100.

<Diagnostic Information Collection Process 200>
Next, the diagnostic information collection process 200 will be described with reference to Figures 2 to 5. Figure 2 is a flowchart of the diagnostic information collection process 200 performed by the operation terminal 100 and the analysis device 110. Figures 3(a) and 3(b) are explanatory diagrams of the first group of questions. Figures 4 and 5 are explanatory diagrams of the second group of questions.

In the diagnostic information collection process 200, the information collection system 1 acquires an image of the affected area, presents a first set of questions regarding a first symptom determined from the acquired image of the affected area, and controls the system to present a second set of questions based on the answers to the first set of questions. The diagnostic information collection process 200 in Figure 2 begins when the operation terminal 100 and the analysis device 110 are powered on.

In step S201, the CPU 101 of the operation terminal 100 photographs the affected area in response to a user operation and acquires an image of the affected area (image acquisition). Note that the CPU 101 may also acquire from the storage medium 109 an affected area image selected by the user from among previously captured affected area images.

In step S202, the CPU 101 transmits the image of the affected area acquired in step S201 to the analysis device 110.

In step S211, the CPU 111 of the analysis device 110 receives the image of the affected area transmitted from the operation terminal 100.

In step S212, the CPU 111 determines the symptom (first symptom) indicated by the affected area image obtained in step S211. The CPU 111 may determine the first symptom using rule-based processing. When determining the first symptom using rule-based processing, the CPU 111 may determine the first symptom based on feature quantities of the affected area image, such as the shape or color of the affected area contained in the image. The CPU 111 may also determine the first symptom using a trained model trained by machine learning, such as deep learning. The trained model is generated by having a training computer train a neural network using images captured for each symptom as training data. The CPU 111 may input the affected area image into the trained model and determine the first symptom based on the output (inference result) from the trained model.

If multiple first symptoms are acquired from the affected area image, the CPU 111 may determine the symptom for which the most likely result is obtained as the first symptom. For example, a score indicating the likelihood of the symptom may be acquired along with the acquired symptoms, and the first symptom may be determined based on the score. Note that the CPU 111 may terminate the diagnostic information collection process 200 if the first symptom is not acquired from the affected area image or if a likely result is not obtained.

In step S213, the CPU 111 determines a first set of questions based on the first symptom determined in step S212.

The first group of questions will now be described with reference to Figures 3(a) and 3(b). The table in Figure 3(a) shows the correspondence between first symptoms and questions related to the first symptoms. Symptoms 301 to 305 are examples of first symptoms. Note that the first symptoms are not limited to these and may be any symptoms that can be determined based on an image. Questions 306 to 311 are questions directly related to the first symptom. Directly related questions include, for example, questions about when the first symptom developed, the cause of the first symptom, and the severity of the first symptom. Questions 312 to 316 are questions about associated symptoms (second symptoms), which are symptoms caused in connection with the first symptom.

The CPU 111 determines the first group of questions to include questions related to the first symptom that correspond to rows marked with "O" in the column for the first symptom determined based on the image of the affected area. For example, if the CPU 111 determines that the first symptom is lymphoma, it determines the first group of questions to include questions 306, 310, 312-314, and 316, which are marked with "O" in the column for symptom 301 in Figure 3(a).

In this way, the first group of questions includes questions about the first symptom determined based on the image of the affected area. The first group of questions may also include questions about symptoms related to the first symptom. The questions about symptoms related to the first symptom may include questions about multiple possible related symptoms, or may prioritize questions about some of the multiple related symptoms. Some of the symptoms may be symptoms whose onset, whether early or not, influences the decision on subsequent treatment strategies. For example, if the first symptom is lymphoma, the first group of questions may be determined to include questions about symptoms that can help a doctor determine whether the disease is malignant or benign. This enables the information collection system 1 to provide doctors with information to assist in diagnosis.

The table in Figure 3(b) shows answer options corresponding to questions about the first symptom. For the first group of questions, items such as those shown in Figure 3(b) are determined as answer options. Note that the questions and answer options shown in Figure 3 are only an example, and questions and answer options with content and numbers appropriate to each of the other symptoms are prepared in advance.

Returning to the explanation of Figure 2, in step S214, the CPU 111 of the analysis device 110 transmits the first set of questions determined in step S213 to the operation terminal 100.

In step S203, the CPU 101 of the operation terminal 100 receives a first set of questions from the analysis device 110.

In step S204, the CPU 101 displays the first group of questions on the display unit 106. Note that the method of presenting the first group of questions to the user is not limited to display, and any method that can be recognized by the user, such as audio notification, may be used.

In step S205, the CPU 101 accepts answers from the user to the first group of questions. The CPU 101 obtains answers to the questions, for example, by accepting user operation from the operation unit 107 in response to answer options displayed on the display unit 106. If the first group of questions includes multiple questions, the CPU 101 repeatedly displays the questions and accepts answers to each question.

In step S206, the CPU 101 transmits the user's answers to the first group of questions to the analysis device 110. If the first group of questions includes multiple questions, the CPU 101 transmits the answers to the analysis device 110 once it has received answers from the user to all of the questions in the first group of questions. Note that the CPU 101 may transmit answers to the analysis device 110 each time it receives some of the answers to the first group of questions.

In step S215, the CPU 111 of the analysis device 110 receives answers to the first group of questions from the operation terminal 100.

In step S216, the CPU 111 determines a second set of questions based on the answers received to the first set of questions.

Here, we will explain the second symptom and the second group of questions. The second symptom is a symptom determined based on the image of the affected area and the answers to the first group of questions. For example, the second symptom is an associated symptom, which is a symptom caused in connection with the first symptom. In FIG. 3(a), of questions 306 to 316 regarding the first symptom, questions 312 to 316 are questions regarding symptoms associated with the first symptom, and are questions that confirm whether the user has the second symptom. The CPU 111 determines the symptoms the user is experiencing as the second symptom based on the answers to questions about the associated symptoms (second symptom). The CPU 111 determines the second group of questions so that it includes more detailed questions about the second symptom than the first group of questions.

The table in Figure 4 shows the correspondence between second symptoms and detailed questions about the second symptoms. Symptoms 401 to 405 are included in the first question group as questions 312 to 316, and are examples of symptoms for which the user has responded as to whether or not they have occurred. Questions 406 to 418 are detailed questions about the second symptom.

Based on the response received in step S215, CPU 111 determines the second symptom from symptoms 401 to 405. CPU 111 determines the second group of questions to include questions corresponding to rows marked with "O" in the column of the symptom determined to be the second symptom. For example, if CPU 111 obtains a response of "Yes" to question 312 in Figure 3(a) ("Do you feel something strange in your mouth?"), CPU 111 determines symptom 401 (discomfort in the mouth) to be the second symptom. CPU 111 determines the second group of questions to include questions 406 to 408 and question 410, which are marked with "O" in the column of symptom 401 in Figure 4.

In this way, the second group of questions includes questions that are more detailed about the second symptom determined based on the answers to the first group of questions than the first group of questions. If there are overlapping questions between the first group of questions and the second group of questions, the CPU 111 may determine the second group of questions so that the overlapping questions are not included. The second symptom may be one or more. The detailed questions about the second symptom may be, for example, questions that identify the cause of the second symptom, or questions about symptoms related to the second symptom. The second group of questions may also include more detailed questions about the first symptom.

The table in Figure 5 shows answer options corresponding to questions about the second symptom. For the second group of questions, items such as those shown in Figure 5 are set as answer options. Note that the questions and answer options shown in Figures 4 and 5 are only examples, and questions and answer options with content and numbers appropriate to each of the other symptoms are also prepared in advance.

Returning to the explanation of Figure 2, in step S217, the CPU 111 of the analysis device 110 transmits the second set of questions determined in step S216 to the operation terminal 100.

In step S207, the CPU 101 of the operation terminal 100 receives the second group of questions from the analysis device 110. In step S208, the CPU 101 displays the second group of questions on the display unit 106. In step S209, the CPU 101 accepts answers to the second group of questions from the user. If the second group of questions includes multiple questions, the CPU 101
The process of displaying the question and accepting the answer is repeated for each question.

In step S210, the CPU 101 displays the results acquired by the information collection system 1 on the display unit 106. The results acquired by the information collection system 1 include an image of the affected area, a first symptom, a first group of questions, answers to the first group of questions, a second symptom, a second group of questions, and answers to the second group of questions. When the CPU 101 receives an instruction from the user to end the diagnostic information collection process 200, it ends the diagnostic information collection process 200.

The results obtained by the information collection system 1 may be stored in the storage medium 109 of the operation terminal 100. This allows the user to, for example, display the stored data on the display unit 106 and explain it to the doctor. The results obtained by the information collection system 1 may also be sent to an external terminal device that the doctor can operate. This allows the doctor to check detailed information about the patient's symptoms at the time of or in advance of the examination. The user can also retake images of the affected area and revise answers to the first and second sets of questions from the information registered in the operation terminal.

Figure 6 shows examples of symptoms targeted by the information collection system 1. As shown in Figure 6, symptoms such as lymphoma, cuts, abrasions, burns, and hives have different characteristics of the affected area depending on the symptom. The analysis device 110 determines which symptom the photographed affected area is associated with based on the characteristics of the affected area, which differ depending on the symptom.

Figures 7(a) to (d) are examples of displays on the display unit 106 of the operation terminal 100 involved in the diagnostic information collection process 200. Display examples for each step of the diagnostic information collection process 200 shown in Figure 2 will be described with reference to Figures 7(a) to (d).

Figure 7 (a) is an example of a screen 700 (step S201) on which the user takes an image. Screen 700 displays a message prompting the user to take an image, area 701, button 702, and button 703. Area 701 is an area that displays an image of the subject. Area 701 displays a live view of the subject being photographed. In screen 700, area 701 displays a live view of a user with lymphoma in the neck. Area 701 also displays a photographed image of the affected area. Button 702 is a photographing button. When CPU 101 detects that button 702 has been selected by the user, it photographs an image of the affected area.

In embodiment 1, the image of the affected area is assumed to be a still image, but it may also be a video. The user can retake the image of the affected area by pressing (touching) button 702 again. Button 703 is a button for proceeding to the next process. For example, when CPU 101 detects that button 703 has been pressed, it transmits the captured image to analysis device 110. Note that CPU 101 may display a message asking the user whether or not to transmit the captured image to analysis device 110, and may control the transmission of the captured image to analysis device 110 when the user instructs transmission.

Figure 7 (b) is an example of a screen 710 (step S205) that displays the first group of questions (step S204) and accepts answers from the user. Screen 710 is an example of a display when the first symptom is determined to be lymphoma based on an image of the affected area. Screen 710 displays a question corresponding to question 310 in Figure 3 ("Do you have pain?") and answer options 711 to 714 for the question. Screen 710 shows a state in which option 712 ("It hurts a little") has been selected. Button 715 is a button for proceeding to the next process. For example, when the CPU 101 of the operation terminal 100 detects that button 715 has been pressed, it controls the display to display the next question from the first group of questions.

Figure 7 (c) is an example of a screen 720 (step S209) that displays the second group of questions (step S208) and accepts answers from the user. Screen 720 is an example of a display when the second symptom is determined to be "discomfort in the mouth." Screen 720 displays a question corresponding to question 707 in Figure 4 ("Do you have mouth ulcers?") and answer options 721 to 724 for the question. Screen 720 shows a state in which option 721 ("No") has been selected. Button 725 is a button for proceeding to the next process. For example, when CPU 101 detects that button 725 has been pressed, it controls the display to display the next question from the second group of questions.

Figure 7 (d) is an example of a screen 730 (step S210) that displays the results acquired by the information collection system 1. Screen 730 displays the user's answers to the first and second question groups (items 731, 733), the first symptom (item 732), an image of the affected area 734, button 735, and button 736. Item 733 includes the second symptom determined by the information collection system 1. Button 735 is a button for correcting the answers. The user can check the contents displayed on screen 730 and consider whether corrections are necessary. Button 736 is a button for confirming the user's answers and the image of the affected area. For example, when the CPU 101 detects that button 725 has been pressed, it stores the acquired results in the storage medium 109 of the operation terminal 100.

In this way, in the diagnostic information collection process 200, the information collection system 1 presents a first group of questions including questions about a first symptom determined based on the image of the affected area. The information collection system 1 presents a second group of questions including questions about a second symptom determined based on the image of the affected area and the answers to the first group of questions. This allows the information collection system 1 to obtain information to assist in diagnosis.

<Diagnostic Information Collection Process 800>
In the diagnostic information collection process 200, the CPU 111 of the analysis device 110 determines the first symptom based on the acquired image of the affected area. However, even if the symptom can be determined from the image, the first symptom may not be determined from the image of the affected area depending on how the image of the affected area was captured. In such cases, it is desirable for the information collection system 1 to retake an image that is useful for diagnosis. Therefore, a process including a case where the first symptom cannot be determined from the image of the affected area will be described with reference to FIG. 8.

Figure 8 is a flowchart of the diagnostic information collection process 800. In the diagnostic information collection process 800, if the first symptom is not determined from the affected area image, information instructing the operation terminal 100 to capture an image to assist in diagnosis (photography instruction information) is sent to the operation terminal 100. The user re-photographs the affected area in accordance with the photography instruction information and obtains a new image of the affected area. This allows the information collection system 1 to determine the first symptom and obtain an image to assist in diagnosis.

Regarding the processing of the operation terminal 100 in Figure 8, steps S801 to S802 and S805 to S811 are the same as the processing of steps S201 to S202 and S204 to S210 in Figure 2, respectively. Similarly, regarding the processing of the analysis device 110, steps S821 to S822 and S824 to S829 are the same as the processing of steps S211 to S212 and S213 to S217 in Figure 2, respectively. In explaining the diagnostic information collection process 800, we will omit explanations of the processing that is common to the diagnostic information collection process 200, and will instead explain the following processing that differs.

In step S823, the CPU 111 of the analysis device 110 determines whether the first symptom was determined in step S822. When determining the first symptom by rule-based processing using feature amounts of the affected area image, etc., the CPU 111 proceeds to step S826 if the CPU 111 determines "no symptoms," and proceeds to step S824 if the first symptom was determined. When determining the first symptom using deep learning, the CPU 111 proceeds to step S826 if the accuracy of the inference result is 70% or less, and proceeds to step S824 if the accuracy is higher than 70%, for example.

In step S826, the CPU 111 transmits information (imaging instruction information) to the operation terminal 100 instructing it to capture an image to assist in diagnosis. The imaging instruction information includes, for example, at least one of the area to be imaged, the imaging direction, the size of the subject to be imaged, and the setting parameters of the imaging device. For example, if the image of the affected area is out of focus, the CPU 111 may include in the imaging instruction information a message urging the user to adjust the focus. Furthermore, if the image of the affected area is dark, the CPU 111 may include in the imaging instruction information a message urging the user to capture an image that is brighter than the image received in step S821. The CPU 111 may determine and transmit the imaging instruction information based on the received image.

In step S803, the CPU 101 of the operation terminal 100 receives a first set of questions or shooting instruction information from the analysis device 110.

In step S804, the CPU 101 determines whether the information received in step S803 is the first group of questions. If the CPU 101 receives the first group of questions, it proceeds to step S805; if it receives shooting instruction information, it proceeds to step S822.

In step S812, the CPU 101 displays the imaging instruction information received in step S803 on the display unit 106. When re-imaging the affected area, the CPU 101 displays a message, for example, encouraging the user to adjust the focus during imaging or change the brightness of the image. By displaying such a message, the user of the operation terminal 100 can learn points to be careful of when re-imaging the affected area. After step S812, the CPU 101 returns to step S801, images the affected area, and obtains an image of the affected area.

In the diagnostic information collection process 800, control is exercised so that imaging instruction information is displayed if the first symptom is not determined based on the acquired affected area image, but control may also be exercised so that imaging instruction information is displayed when the affected area image is first captured. This allows the user of the operation terminal 100 to be aware of points to pay attention to from the beginning before capturing an image of the affected area.

In this way, in the diagnostic information collection process 800, the information collection system 1 displays imaging instruction information when the user photographs the affected area. By photographing the affected area in accordance with the imaging instruction information, the information collection system 1 can obtain images that support diagnosis.

In the diagnostic information collection process 200 and the diagnostic information collection process 800, the captured images were images of the affected area, but images of areas other than the affected area may also be useful for treatment. Therefore, the CPU 111 of the analysis device 110 may prompt the user to capture an image of an object that explains the cause of the first symptom. For example, if a cut occurs, capturing an image of the object that caused the cut as the subject, generating and recording the image, can be useful for the doctor's diagnosis.

A specific explanation will be given with reference to Figures 3(a) and 9. If the user answers question 307 in Figure 3(a) "What caused the cut?" by selecting an option other than option D "I don't know" in Figure 3(b), the CPU 111 of the analysis device 110 prompts the user to take a photo of the object that caused the cut.

9 shows an example of the display on the display unit 106 of the operation terminal 100 when photographing the cause of the first symptom. A screen 900 displays a photographing instruction 901, an area 902, and buttons 903 to 906.
The photographing instruction 901 includes a message prompting the user to photograph the object that caused the first symptom. In FIG. 9 , the photographing instruction 901 displays a message prompting the user to photograph the object that caused the cut. As shown in FIG. 9 , the photographing instruction 901 may also include a message of caution when photographing. Similar to the area 701 in FIG. 7A , the area 902 displays a live view of the subject to be photographed. The user selects one of buttons 903 to 906 depending on the image displayed in the area 902. In this way, the information collection system 1 also photographs images of areas other than the injured or ill area. This allows the doctor to more clearly identify the cause of the first symptom. If the user is unable to photograph the object that caused the injury because it is not nearby, for example, the user presses button 905 ("Not Nearby"). By pressing button 905, the user can skip photographing.

In the explanation of the diagnostic information collection process 200 and the diagnostic information collection process 800, the first group of questions is determined based on the symptoms determined from the affected area image. The operation terminal 100 displays questions based on the received first group of questions. Note that the operation terminal 100 may change the next question to be presented from the received first group of questions based on the user's answer. With reference to Figures 10(a) to 10(e), an example of changing the next question to be presented based on the user's answer will be described.

Figure 10(a) is an example of the display of screen 1000 (step S201) on which the user takes an image. Similar to screen 700 in Figure 7(a), screen 1000 displays an area 1001 that displays the subject, a capture button, and a button 1002 for proceeding to the next process. Screen 1000 shows a state in which an image of a hand being bitten by an animal is displayed in area 1001. When CPU 101 of operation terminal 100 detects that button 1002 has been pressed, it proceeds to the next process.

Figure 10(b) is an example of a screen 1010 (step S205) that displays the first set of questions (step S204) and accepts answers from the user. Screen 1010 is an example of a screen that is displayed when the first symptom is determined to be "animal bite wound." Screen 1010 displays the question ("What bit you?"), answer options 1011 to 1015 ("Dog," "Cat," "Snake," "I don't know"), and a button 1015 for proceeding to the next process. When option 1011 ("Dog") is selected and button 1015 is pressed, CPU 101 displays the screen shown in Figure 10(c) as the next screen. When option 1013 ("Snake") is selected and button 1015 is pressed, CPU 101 displays the screen shown in Figure 10(d) as the next screen.

Figure 10(c) is an example of screen 1020 that displays the question about "dogs" selected on screen 1010 ("Is it a stray dog?") and answer options.

Figure 10(d) is an example of screen 1021 that displays the question about "snakes" selected on screen 1010 ("Do you know what kind of snake it is?") and answer options.

After answering the questions shown in Figure 10(c) or Figure 10(d), CPU 101 displays screen 1030 shown in Figure 10(e). Screen 1030 is an example of a screen that displays a common question regardless of the type of animal that bitten the patient (the cause of the first symptom). Screen 1030 displays a question asking about the size of the wound and options for answering the question.

In this way, the CPU 101 changes the question presented on the next screen depending on the answer to the previous question. The question presented on the next screen may, for example, be a question that specifies more detailed content regarding the answer to the previous question. This allows the information collection system 1 to record detailed information about symptoms depending on the user's answers.

<Embodiment 2>
In the first embodiment, the information collection system 1 presents a second set of questions regarding the second symptom based on the image of the affected area or the answers to the first set of questions. In the second embodiment, the information collection system 1 prompts the patient to take an image of the area exhibiting the second symptom based on the image of the affected area or the answers to the first set of questions. This allows the information collection system 1 to acquire an image of the second symptom related to the first symptom and to acquire information to support the doctor's diagnosis.

<Diagnostic Information Collection Process 1100>
The configuration of the information collection system 1 of the second embodiment is the same as the configuration (FIG. 1) of the first embodiment. The diagnostic information collection process 1100 will be described below with reference to FIG.

Figure 11 is a flowchart of the diagnostic information collection process 1100. With regard to the processing of the operation terminal 100 in Figure 11, steps S1101 to S1106 are the same as the processing of steps S201 to S206 in Figure 2. Similarly, with regard to the processing of the analysis device 110, steps S1111 to S1115 are the same as the processing of steps S211 to S217 in Figure 2. In explaining the diagnostic information collection process 1100, we will omit explanations of the processing that is common to the diagnostic information collection process 200, and will explain the following processing that differs.

In step S1116, the CPU 111 of the analysis device 110 determines the second symptom based on the image of the affected area or the answers to the first group of questions, and determines information (imaging instruction information) that instructs the capture of an image to assist in diagnosis. The imaging instruction information includes the area showing the second symptom as an imaging item. There may be one imaging item or multiple imaging items. Furthermore, the imaging instruction information may include information regarding the size of the subject, the orientation of the subject, and the brightness of the image, in addition to the area showing the second symptom.

In step S1117, the CPU 111 transmits the image capture instruction information to the operation terminal 100. In step S1107, the CPU 101 of the operation terminal 100 receives the image capture instruction information from the analysis device 110.

In step S1108, the CPU 101 displays imaging instruction information on the display unit 106, prompting the user to capture an image of the area showing the second symptom. By prompting the user to capture an image of the area showing the second symptom, the information collection system 1 can acquire not only an image of the area showing the first symptom (affected area image; first image), but also an image of the area showing the second symptom (second image).

In step S1109, CPU 101 captures a second image based on the received image capture instruction information. Note that the second image captured does not necessarily have to be one. CPU 101 may control the capture of multiple second images depending on the first symptom or the answers to the first set of questions. When CPU 101 detects that the capture of the second image has ended, it proceeds to step S1110.

In step S1110, the CPU 101 displays the results acquired by the information collection system 1 on the display unit 106. The results acquired by the information collection system 1 include a first image, a first symptom, a first set of questions, answers to the first set of questions, a second image, and a second symptom. When the CPU 101 receives an instruction from the user to terminate the diagnostic information collection process 1100, it terminates the diagnostic information collection process 1100.

Figures 12(a) to 12(c) are examples of displays on the display unit 106 of the operation terminal 100 involved in the diagnostic information collection process 1100. Note that Figures 12(a) to 12(c) illustrate an example in which the first symptom is determined to be "lymphoma."

12A is a display example of screen 1200 (step S1105) that displays the first group of questions (step S1104) and accepts an answer from the user. Screen 1200 shows a state in which a question from the first group of questions, "Do you feel any discomfort in your mouth?", is displayed, and option 1202 ("Yes") is selected from answer options 1201 to 1203.

Figure 12(b) is an example of a screen 1210 (step S1109) that displays shooting instruction information (step S1108) and allows the user to capture a second image. Screen 1210 displays item 1211 and area 1212. Item 1211 displays a message urging the user to capture a second image ("Please take a picture so that your throat is clearly visible."). Area 1212, like area 701 in Figure 7(a), displays a live view of the subject being captured. On screen 1210, area 1212 displays the user's throat.

Figure 12(c) is an example display of the results obtained by information collection system 1 (step S1110). Screen 1220 displays the user's answers to the first set of questions (item 1221), area 1222, and obtained images 1223-1225. Image 1223 is the first image, and images 1224 and 1225 are second images. Images 1223-1225 are displayed as thumbnails, and when one of the images is selected, the selected image is enlarged and displayed in area 1222. The user can refer to screen 1220 to check the answers and the captured images.

In embodiment 2, the information collection system 1 prompts the patient to take an image of the area showing the second symptom based on the image of the affected area or the answers to the first set of questions. This allows the information collection system 1 to acquire images of the area showing the first symptom as well as images of the area showing the second symptom caused in association with the first symptom, thereby assisting the doctor in making a diagnosis.

<Embodiment 3>
An information collection system 1 according to the third embodiment performs processing that combines the processing according to the first and second embodiments. That is, the information collection system 1 determines a first symptom based on an image of the affected area and presents a first set of questions including questions related to the first symptom. The information collection system 1 determines a second symptom based on the image of the affected area and the answers to the first set of questions. The information collection system 1 presents a second set of questions including questions related to the second symptom and further prompts the patient to photograph an area exhibiting the second symptom. As a result, the information collection system 1 can obtain answers to the second set of questions and images of the area exhibiting the second symptom, and acquire information to support a doctor's diagnosis.

<Diagnostic Information Collection Process 1300>
The configuration of the information collection system 1 of the third embodiment is the same as the configuration (FIG. 1) of the first embodiment. The diagnostic information collection process 1300 will be described below with reference to FIG.

Figure 13 is a flowchart of the diagnostic information collection process 1300. With regard to the processing by the operation terminal 100 in Figure 13, steps S1301 to S1306 and steps S1308 to S1309 are the same as steps S201 to S206 and steps S208 to S209, respectively, in Figure 2. Similarly, with regard to the processing by the analysis device 110, steps S1321 to S1326 are the same as steps S211 to S216 in Figure 2. In explaining the diagnostic information collection process 1100, we will omit explanations of the processing that is common to the diagnostic information collection process 200, and will explain the following processing that differs.

In step S1327, the CPU 111 of the analysis device 110 determines information (photographing instruction information) that instructs that an image for assisting diagnosis be photographed. The photographing instruction information includes a region showing the second symptom as a photographing item. Step S1327 is the same as step S in FIG.
This is the same as the process in 1116.

In step S1328, the CPU 111 transmits the second set of questions determined in step S1326 and the shooting instruction information determined in step S1327 to the operation terminal 100.

In step S1307, the CPU 101 of the operation terminal 100 receives a second set of questions and shooting instruction information from the analysis device 110.

In step S1310, the CPU 101 displays imaging instruction information. In step S1311, the CPU 101 captures an image (second image) of the area exhibiting the second symptom. Steps S1310 to S1311 are the same as steps S1108 to S1109 in Figure 11.

In step S1110, the CPU 101 displays the results acquired by the information collection system 1 on the display unit 106. The results acquired by the information collection system 1 include an image of the affected area (first image), first symptoms, a first set of questions, answers to the first set of questions, a second image, second symptoms, a second set of questions, and answers to the second set of questions. When the CPU 101 receives an instruction from the user to terminate the diagnostic information collection process 1300, it terminates the diagnostic information collection process 1300.

Figures 14(a) to (c) are examples of displays on the display unit 106 of the operation terminal 100 involved in the diagnostic information collection process 1300. Note that Figures 14(a) to (c) illustrate an example in which the first symptom is determined to be "lymphoma."

Figure 14(a) is an example of screen 1400 (step S1309) that displays the second group of questions (step S1308) and accepts answers from the user. Screen 1400 displays the question "Do you have a sore throat?" from the second group of questions, and shows option 1402 ("A little sore") selected from answer options 1401 to 1404.

Figure 14(b) is an example of a display screen 1410 (step S1311) that displays imaging instruction information (step S1310) and allows the user to capture a second image. Screen 1410 displays a message prompting the user to capture a second image, item 1411, area 1412, and area 1413. Screen 1410 displays a message prompting the user to capture the second image, saying, "Please photograph the throat." Item 1411 displays the imaging point based on the imaging instruction information. Area 1412 displays a good example of the second image. Area 1413, similar to area 701 in Figure 7(a), displays a live view of the subject being captured. The user can capture the second image while referring to the imaging point and good example images. This allows information collection system 1 to obtain images that are useful for diagnosis.

Figure 14(c) is an example of the display of screen 1420 (step S1312) showing the results obtained by information collection system 1. Screen 1420 displays the user's answers (items 1421) to the first and second sets of questions, area 1422, and obtained images 1423-1424. Image 1423 is the first image, and image 1424 is the second image. Images 1423-1424 are displayed as thumbnails, and when one of the images is selected, the selected image is enlarged and displayed in area 1422. The user can refer to screen 1420 to check the answers and the first and second images that were taken.

Thus, in the diagnostic information collection process 1300, the information collection system 1 presents a second set of questions based on the image of the affected area and the answers to the first set of questions, and further prompts the patient to take an image of the area showing the second symptom. The information collection system 1 can obtain detailed information about the second symptom and an image of the area showing the second symptom from the answers to the second set of questions. The information collection system 1 can obtain detailed information and images that are useful for diagnosis.

Figures 15(a) and (b) are examples of displays on the display unit 106 of the operation terminal 100 involved in the diagnostic information collection process 1300. The basic screen layout of Figure 15(a), which is the capture screen for the second image, is the same as Figure 14(b), but the screen layout of Figure 15(b), which is the display screen for the results, is different from Figure 14(c). Below, we will explain Figures 15(a) and (b), focusing mainly on the differences from Figures 14(b) and (c).

Figure 15(a) is an example of a screen that displays shooting instruction information (step S1310) and allows the user to shoot a second image (step S1311), similar to Figure 14(b). Screen 1500 displays a message urging the user to shoot a second image, item 1501, area 1502, and area 1503. Screen 1500 displays a message urging the user to shoot a second image ("Please take a picture of your face."). Item 1501 displays the shooting point. Area 1502 displays good examples of facial images, and area 1503 displays the image being shot. As with Figure 14(b), the user can take a second image while referring to the shooting point and good examples of images.

Figure 15(b) is an example display of the results acquired by information collection system 1 (step S1312). Screen 1510 displays the user's answers to the first group of questions (item 1511), area 1512, and acquired images 1513-1515. Screen 1510 also displays the instructions given at the time of shooting (shooting point) in item 1516. Item 1516 displays the instructions given at the time of shooting for an image selected from the thumbnail images. On screen 1510, image 1515 is selected from the thumbnail images, and the instructions given at the time of shooting image 1515 are displayed in item 1516.

The instructions given at the time of imaging based on the imaging instruction information may be embedded as metadata in the captured image and saved. This allows the information collection system 1 to display the instructions given at the time of imaging along with the first or second image at the time of diagnosis. By displaying the instructions given at the time of imaging, the doctor can confirm the intention behind the image being captured.

<Fourth Embodiment>
In the first to third embodiments, the information collection system 1 first acquires an image and determines the first symptom based on the acquired image. In the fourth embodiment, unlike the first to third embodiments, the information collection system 1 first presents a set of preliminary questions and prompts the user to photograph the affected area based on the answers to the preliminary questions. The information collection system 1 determines the first symptom based on the answers to the questions and the image of the affected area. This allows the information collection system 1 to accurately determine the first symptom.

<Diagnostic Information Collection Process 1600>
The configuration of the information collection system 1 of the fourth embodiment is the same as the configuration (FIG. 1) of the first embodiment. The diagnostic information collection process 1600 will be described below with reference to FIG.

Fig. 16 is a flowchart of diagnostic information collection processing 1600. With regard to the processing of the operation terminal 100 in Fig. 16, step S1604 and steps S1606 to S1609 are the same as the processing of step S201 and steps S207 to S210 in Fig. 2, respectively. Similarly, with regard to the processing of the analysis device 110, step S1614 is the same as the processing of step S217 in Fig. 2. In the explanation of diagnostic information collection processing 1600, explanation of processing common to diagnostic information collection processing 200 will be omitted, and the following processing that differs will be explained.

In step S1601, the CPU 101 of the operation terminal 100 displays a first group of questions. The first group of questions displayed in step S1601 includes questions for confirming the symptoms the user is experiencing and questions for determining whether an image is required. Questions for confirming symptoms include questions for identifying the type of symptom, the location of the symptom, and the severity of the symptom. Questions for determining whether an image is required include, for example, questions for determining whether the symptom is visible on the surface.

In step S1602, CPU 101 accepts answers to the first group of questions from the user. In step S1603, CPU 101 displays the capture items determined based on the content accepted in step S1602, and prompts the user to capture an image of the affected area. The capture items may be items that do not specify a specific area, such as "affected area," or items that specify a specific area, such as "eyes."

Note that if the CPU 101 determines from the content received in step S1602 that images are not required, it may skip steps S1603 and S1604 and proceed to step S1605. It may determine that images of the affected area are not required if, for example, a response regarding a symptom that does not appear externally (e.g., abdominal pain) is received. In step S1605, the CPU 101 transmits the responses to the first group of questions and the images to the analysis device 110.

In step S1611, the CPU 111 of the analysis device 110 receives the answers to the first group of questions and the image from the operation terminal 100. In step S1612, the CPU 111 determines the first symptom based on the answers to the first group of questions, or based on the answers to the first group of questions and the image of the affected area. Thus, unlike embodiments 1 to 3 in which the first symptom is determined based on the image of the affected area, in embodiment 4 the CPU 111 determines the first symptom based on the answers to the first group of questions and the image of the affected area. This allows the CPU 111 to accurately determine the first symptom.

In step S1613, the CPU 111 determines a second group of questions. The second group of questions is determined based on the first symptom. For example, a case will be described where the first symptom is infectious conjunctivitis. Infectious conjunctivitis may be accompanied by a fever, such as pool fever. Therefore, if the CPU 111 determines that the first symptom is infectious conjunctivitis, it includes questions about fever in the second group of questions.

Figures 17(a) to (c) are examples of displays on the display unit 106 of the operation terminal 100 involved in the diagnostic information collection process 1600. Figure 17(a) is an example of a screen 1700 (step S1602) that displays a first group of questions (step S1601) and accepts answers from the user. Screen 1700 displays item 1701, which displays a question, and options 1702 to 1706. In Figure 17(a), item 1701 displays "What symptoms are you experiencing?" as a question from the first group of questions for confirming symptoms. Option 1702 is an option for identifying the type of symptom. Options 1703 to 1705 are options for narrowing down the location of the symptom. Option 1706 is an option for identifying the severity of the symptom. From the answers to options 1702 to 1706, the information collection system 1 can identify the symptoms the user is experiencing.

Figure 17(d) is an example of the options displayed in Figure 17(a). In Figure 17(d), item a corresponds to option 1702, item b corresponds to option 1703, item c corresponds to option 1704, item d corresponds to option 1705, and item e corresponds to option 1706.

Item a is an option that specifies the type of symptom. Item b is an option that specifies the body part where the symptom is present. Item c is an option that more accurately specifies the body part selected in item b. For example, if "face" is selected in option 1703 (item b), option 1704 (item c) displays "eyes," "nose," and "mouth." Item d is an option that more accurately specifies the body part selected in item c. For example, if "eyes" is selected in option 1704 (item c), option 1705 (item d) displays "left eye" and "right eye." Item e is an option that more accurately specifies the severity of the symptom selected in item a. For example, if "painful" is selected in option 1702 (item a), option 1706 (item e) displays "slight pain" and "very pain." Note that the number of items and options is merely an example, and items and options are prepared in advance according to various symptoms.

Figure 17(b) is an example of a screen 1710 (step S1604) that displays the imaging items (step S1603) and allows the user to capture an image. Screen 1710 displays items 1711 and area 1717. In Figure 17(b), item 1711 displays a message prompting the user to capture an image ("Please capture the affected area."). Area 1717 displays the image being captured.

Figure 17 (c) is an example of a screen 1720 (step S1608) that displays the second group of questions (step S1607) and accepts answers from the user. Screen 1720 displays item 1721, which displays the question, and answer options. On screen 1720, item 1721 displays "Do you have a fever?" as an example of the second group of questions.

In this way, in the diagnostic information collection process 1600, the information collection system 1 determines the first symptom based on the answers to the first group of questions and the images of the affected area. The information collection system 1 can accurately determine the first symptom.

Here, we will explain an example of changing the questions presented based on information about the user. Some symptoms are characterized by the gender or age of the patient who develops them. For example, approximately 98% of gout patients are male. When symptoms similar to those of gout are acquired, the information collection system 1 can accurately determine the symptoms by taking the user's gender into account. Furthermore, 80 to 90% of hand, foot, and mouth disease patients are children under the age of five. When symptoms similar to those of hand, foot, and mouth disease are acquired, the information collection system 1 can accurately determine the symptoms by taking the user's age into account. Therefore, the information collection system 1 acquires information about the user and changes the questions presented depending on the acquired information. This allows the information collection system 1 to accurately determine the symptoms.

Figures 18(a) to 18(e) are examples of the display on the display unit 106 of the operation terminal 100 when determining the first group of questions and the second group of questions based on information about the user. Figures 18(a) to 18(b) are examples of the screen displaying the first group of questions (step S1601) and accepting answers from the user (step S1602). Figures 18(c) to 18(d) are examples of the screen displaying the second group of questions (step S1606) and accepting answers from the user (step S1607).

In FIG. 18(a), screen 1800 displays item 1801, which displays a question about the user, and options 1802-1803. In FIG. 18(a), item 1801 displays "Please enter your gender and age" as an example of a first group of questions. Option 1802 is an option for specifying gender. Option 1803 is an option for specifying age. From the answers to options 1802-1803, information collection system 1 obtains information about the user's gender and age.

In FIG. 18(b), screen 1810 displays item 1811, which displays questions about symptoms, and options 1812 to 1817. From the answers to options 1812 to 1817, information collection system 1 obtains information regarding the type of symptom, the location of pain, and the degree of pain.

Figures 18(c) and 18(d) are examples of displaying questions about gout. If symptoms similar to gout are obtained from the responses to screens 1800 and 1810, the CPU 111 of the analysis device 110 includes questions about gout in the second group of questions. The CPU 101 of the operation terminal 100 presents questions about gout. For example, the CPU 101 may display a screen asking whether the user has joint pain, as shown in Figure 18(c). The CPU 101 may also display a screen asking whether the user has developed kidney stones, as shown in Figure 18(d). In this way, by presenting the first group of questions and the second group of questions based on information about the user, symptoms can be accurately determined.

The CPU 111 of the analysis device 110 may determine the questions to present taking into consideration not only the user's gender and age, but also other information. For example, the CPU 111 may record data previously answered by the user, such as medical history, and determine the questions to present based on the content of the past answers. For example, cerebral infarction is a disease with a high recurrence rate. Cerebral infarction is also one of the leading causes of death among Japanese people, so early detection is important. Symptoms of cerebral infarction include numbness in either half of the body or in the limbs, dizziness, vomiting, and slurred speech. If these symptoms are present, the information collection system 1 refers to the patient's past answer information and determines the questions to present. This allows the information collection system 1 to present the user with questions that will lead to early detection of cerebral infarction.

<Other embodiments>
Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or recording medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

100: Operation terminal 101: CPU 105: Image sensor 106: Display unit 107: Operation unit 108: Communication unit 110: Analysis device 111: CPU 114: Communication unit

Claims (12)

image acquisition means for acquiring an image of the affected area of the user;
a determining means for determining which of a plurality of symptoms the image of the affected area shows;
A control means for controlling to present a question and receive an answer to the question,
presenting a first group of questions including questions regarding a first symptom determined based on the image of the affected area;
a control means for presenting a second group of questions including questions regarding a second symptom determined based on the image of the affected area and the answers to the first group of questions;
and
the control means presents imaging instruction information, which is information instructing the user to capture an image to assist diagnosis, when prompting the user to capture an image;
the photographing instruction information is stored as metadata of an image photographed based on the photographing instruction information;
An information collection system comprising: the control means prompts the user to take a re-image of the affected area when the determination means has not determined that the first symptom exists based on the image of the affected area.
The information collection system according to claim 1 . the control means prompts the user to take a photograph of the subject to explain the cause of the first symptom.
3. The information collection system according to claim 1 . the control means prompts the imaging of the region exhibiting the second symptom.
The information collection system according to any one of claims 1 to 3 . The imaging instruction information includes at least one of an imaging region, an imaging direction, a size of the imaging target, and a setting parameter of the imaging device.
The information collection system according to any one of claims 1 to 4 . the control means determines the next question depending on the user's answer to the immediately preceding question.
The information collection system according to any one of claims 1 to 5 . the control means determines the first set of questions and the second set of questions based on information about the user or answers given by the user in the past;
The information collection system according to any one of claims 1 to 6 . The determination means determines which of a plurality of symptoms the image of the affected area represents by using a trained model trained using images acquired by photographing each symptom.
The information collection system according to any one of claims 1 to 7 . the control means presents a group of preliminary questions for determining which of a plurality of symptoms the subject is experiencing;
The information collection system according to claim 1 , wherein the first symptom is determined based on answers to the preliminary group of questions and an image of the affected area. The apparatus further comprises a recording means for recording in a recording unit, in association with the image of the affected area, the first symptom, the first group of questions, the answers to the first group of questions, the second symptom, the second group of questions, the answers to the second group of questions, and an image obtained by photographing the user in response to a prompt from the control means.
The information collection system according to any one of claims 1 to 9 . 1. A method of collecting information, comprising:
an image acquisition step in which a computer acquires an image of the affected area of the user;
a determining step in which the computer determines which of a plurality of symptoms the image of the affected area shows;
a control step of controlling the computer to present a question and receive an answer to the question,
the computer presents a first set of questions including questions regarding a first symptom determined based on the image of the affected area;
a control step in which the computer presents a second set of questions including questions regarding a second symptom determined based on the image of the affected area and the answers to the first set of questions;
and
In the control step, the computer presents imaging instruction information, which is information instructing the user to capture an image that assists diagnosis, when prompting the user to capture an image;
the photographing instruction information is stored as metadata of an image photographed based on the photographing instruction information;
An information gathering method comprising: A program for causing a computer to execute each step of the information gathering method according to claim 11 .