JP7654450B2 - Information processing device, information processing method, and program - Google Patents

Description

The embodiments disclosed in this specification and the drawings relate to an information processing device, an information processing method, and a program.

In the field of pathology and the like, there is a known technique for setting information such as annotations on tissue images in which multiple slices cut out from a specimen are captured, based on user operations or image processing results.

However, there were cases where the information displayed on the tissue image overlapped with the slice depicted on the tissue image, making it difficult for doctors and other medical professionals to see the area they wanted to observe. In addition, when the user had to manually adjust the display position so that the information displayed on the tissue image did not overlap with the slice, this could lead to a high workload.

Special table 2016-517559 publication

One of the problems that the embodiments disclosed in this specification and the drawings attempt to solve is to display information about slices depicted in a tissue image in a position that does not overlap with the area in which the slices are depicted. However, the problems that the embodiments disclosed in this specification and the drawings attempt to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems.

An information processing device according to an embodiment includes a recognition unit, a region of interest information acquisition unit, a generation unit, and a display control unit. The recognition unit recognizes an observation target region in a diagnostic image. The region of interest information acquisition unit acquires first region of interest information representing the range of one or more regions of interest in the observation target region in the diagnostic image. The generation unit generates second region information representing the range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information. The display control unit displays the second region information on the display unit at a position outside the observation target region .

FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system according to the first embodiment. FIG. 2 is a diagram for explaining a flow until a tissue image is captured according to the first embodiment. FIG. 3 is a diagram showing an example of a tissue image according to the first embodiment. FIG. 4 is a diagram illustrating an example of an abnormal portion according to the first embodiment. FIG. 5 is a diagram showing an example of a display annotation according to the first embodiment. FIG. 6 is a flowchart showing an example of the flow of a generation process of a display annotation according to the first embodiment. FIG. 7 is a diagram illustrating an example of an instruction annotation according to the second embodiment. FIG. 8 is a diagram showing an example of a display annotation according to the second embodiment. FIG. 9 is a diagram showing another example of instruction annotations and display annotations according to the second embodiment. FIG. 10 is a flowchart showing an example of the flow of a generation process of a display annotation according to the second embodiment. FIG. 11 is a diagram showing an example of instruction annotations and display annotations according to the first modification. FIG. 12 is a diagram showing an example of instruction annotations and display annotations according to the second modification. FIG. 13 is a diagram showing an example of instruction annotations and display annotations according to the third modification. FIG. 14 is a diagram showing an example of a screen display according to the fourth modification.

Below, embodiments of an information processing device, an information processing method, and a program will be described in detail with reference to the drawings.

(First embodiment)
Fig. 1 is a diagram showing an example of the overall configuration of an information processing system S according to the first embodiment. As shown in Fig. 1, the information processing system S includes an information processing device 100, a sample image storage device 201, and a tissue image storage device 202. The information processing device 100 is communicatively connected to the sample image storage device 201 and the tissue image storage device 202 via a network 300 such as an in-hospital LAN (Local Area Network).

The information processing system S may further include a hospital information system (HIS), a laboratory information system (LIS), a radiology information system (RIS), etc. Alternatively, the information processing system S may be part of a hospital information system. The information processing system S may further include a terminal device such as a PC (Personal Computer) or a tablet terminal.

The information processing system S is provided, for example, in a medical institution such as a hospital, a research institute such as a university, or an examination center. In addition, some or all of the devices constituting the information processing system S may be provided in a cloud environment.

The specimen image storage device 201 is a device that stores specimen images. A specimen image is an image of the entire tissue piece that is the specimen. The specimen image is a digital image captured by an imaging device such as a digital camera. A specimen image is also called a macro image.

In this embodiment, the specimen is a portion of tissue collected from a patient's body, an animal, etc. For example, the specimen is the mucosa and submucosa of the digestive tract removed by endoscopic submucosal dissection (ESD), but is not limited thereto, and may be a piece of tissue removed by laparotomy, etc.

The tissue image storage device 202 is a device that stores tissue images. A tissue image is an image of at least one slice cut from a tissue piece that is a specimen, captured at a higher magnification than the specimen image. For example, a tissue image is a Whole Slide Imaging (WSI) image in which the entire slice or a part of it placed on a glass slide is digitally imaged with high precision. The tissue image may be a micrograph captured by a Whole Slide Scanner (WSS) or a digital microscope, or an image captured by other methods. A tissue image is also called a micro image. A tissue image is an example of a diagnostic image in this embodiment.

The specimen image storage device 201 and the tissue image storage device 202 are, for example, a server device or a PC. The specimen image storage device 201 and the tissue image storage device 202 may be collectively referred to as an image storage device. Although the specimen image storage device 201 and the tissue image storage device 202 are shown as separate devices in FIG. 1, the specimen image storage device 201 and the tissue image storage device 202 may be configured as an integrated device. The specimen image storage device 201, the tissue image storage device 202, and the information processing device 100 may be configured as an integrated device.

Figure 2 is a diagram explaining the flow until a tissue image according to the first embodiment is captured. For example, a specimen 5 collected from a patient P by a diagnostician is transferred to a pathologist or a laboratory technician for pathological diagnosis. At this time, a pathology number is assigned for each request (order) for pathological diagnosis. Also, a different specimen number is assigned for each specimen to be the subject of pathological diagnosis. Note that, when there are multiple specimens to be diagnosed in one request for pathological diagnosis, multiple specimen numbers are associated with one pathology number. Hereinafter, in this embodiment, a pathologist or laboratory technician is referred to as a pathologist, etc.

The processing performed by the information processing device 100 in this embodiment does not necessarily have to be aimed at pathological diagnosis. For example, it may be aimed at research by a research institute, or at the creation of a report at an examination center that performs pathological examinations on behalf of a medical institution.

A specimen image ID is assigned as identification information for the specimen image 51 obtained by capturing an image of the specimen 5 using a digital camera or the like. The specimen image ID may be registered as additional information for the specimen image 51. Information related to the specimen image 51, such as the pathology number of the specimen 5 that is the subject of the image of the specimen image 51, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from whom the specimen 5 was obtained, and the image date and time of the specimen image 51, may be registered as additional information for the specimen image 51. The specimen image ID and other information related to the specimen image 51 may be displayed as text information on the specimen image 51.

The pathologist or the like cuts out the slices 60a to 60e from the specimen 5, and after performing processing such as staining, for example, slices the cross sections of the slices 60a to 60e thinly and places them on a glass slide 7. Hereinafter, when there is no need to distinguish between the individual slices 60a to 60e, they will simply be referred to as slices 60.

In this embodiment, the group of sections placed on the slide glass 7 is called a tissue specimen 6. The tissue specimen 6 includes at least one section 60.

In addition, in FIG. 2, each section 60 is placed directly on the glass slide 7, but if the section 60 is long or depending on the shape of the cutting position, one section 60 may be divided into multiple pieces and placed on the glass slide 7.

2, as an example, the slices 60a to 60e are imaged with their cross sections facing the imaging device, but the orientation of placement is not limited to this. For example, the slice 60 may be imaged in the same orientation as the specimen 5 when the specimen image 51 is imaged. Also, in FIG. 2, the slice 60 is placed on the glass slide 7 so that the horizontal direction of the tissue image 61 is the longitudinal direction of the tissue region 62, but the orientation is not limited to this. For example, the slice 60 may be placed on the glass slide 7 so that the vertical direction of the tissue image 61 is the longitudinal direction of the tissue region 62. Alternatively, the slice 60 may be placed at an angle on the glass slide 7. In this embodiment, the slice 60 is placed according to a method determined by the operating facility.

An image such as a WSI image captured of the tissue specimen 6 placed on the slide glass 7 is the tissue image 61. The tissue image 61 depicts the tissue specimen 6 including slices 60a to 60e.

A tissue image ID is assigned as identification information for the tissue image 61. The tissue image ID may be registered as additional information for the tissue image 61. Information related to the tissue image 61, such as the pathology number of the specimen 5 from which the slice that is the subject of imaging of the tissue image 61 was obtained, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from which the specimen 5 was obtained, and the imaging date and time of the tissue image 61, may be registered as additional information for the tissue image 61. The tissue image ID and other information related to the tissue image 61 may be depicted as text information on the tissue image 61.

The tissue image 61 includes a background region 70 depicting the glass slide 7, and a number of tissue regions 62a-62e depicting the tissue specimen 6. Hereinafter, when there is no need to distinguish between the individual tissue regions 62a-62e, they will simply be referred to as tissue regions 62.

Returning to FIG. 1, the information processing device 100 is, for example, a server device or a PC, and has a network interface 110, a memory circuit 120, an input interface 130, a display 140, and a processing circuit 150.

The NW interface 110 is connected to the processing circuit 150 and controls the transmission and communication of various data between the information processing device 100 and the sample image storage device 201 or the tissue image storage device 202. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc.

The memory circuitry 120 prestores various types of information to be used by the processing circuitry 150. The memory circuitry 120 also stores various programs.

The input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touch screen that integrates a display screen and a touchpad, a non-contact input circuit using an optical sensor, and a voice input circuit. The input interface 130 is connected to the processing circuit 150, and converts input operations received from an operator into electrical signals and outputs them to the processing circuit 150. Note that in this specification, the input interface is not limited to those that have physical operating parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs these electrical signals to the processing circuit 150 is also included as an example of the input interface 130.

The display 140 is a liquid crystal display, an organic electro-luminescence (OEL) display, or the like. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit.

The processing circuitry 150 is a processor that reads out programs from the memory circuitry 120 and executes them to realize functions corresponding to each program. The processing circuitry 150 of this embodiment includes an acquisition function 151, a recognition function 152, an image processing function 153, a generation function 154, a reception function 155, and a display control function 156. The acquisition function 151 is an example of an acquisition unit. The recognition function 152 is an example of a recognition unit. The image processing function 153 is an example of an image processing unit and an interest region information acquisition unit. The generation function 154 is an example of a generation unit. The reception function 155 is an example of a reception unit. The display control function 156 is an example of a display control unit.

Here, for example, each processing function of the processing circuit 150, the acquisition function 151, the recognition function 152, the image processing function 153, the generation function 154, the reception function 155, and the display control function 156, are stored in the storage circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes the function corresponding to each program by reading the program from the storage circuit 120 and executing it. In other words, the processing circuit 150 in the state in which each program is read has each function shown in the processing circuit 150 in FIG. 1. Note that, in FIG. 1, it has been described that the processing functions performed by the acquisition function 151, the recognition function 152, the image processing function 153, the generation function 154, the reception function 155, and the display control function 156 are realized by a single processor, but it is also possible to configure the processing circuit 150 by combining multiple independent processors, and realize the functions by each processor executing a program. In addition, in FIG. 1, a single memory circuit 120 is described as storing a program corresponding to each processing function, but multiple memory circuits may be distributed and the processing circuit 150 may read corresponding programs from individual memory circuits.

In the above description, an example was described in which the "processor" reads out and executes a program corresponding to each function from a storage circuit, but the embodiment is not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes the function by reading out and executing a program stored in a storage circuit. On the other hand, when the processor is an ASIC, instead of storing a program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in this embodiment is not limited to being configured as a single circuit for each processor, and may be configured as a single processor by combining multiple independent circuits to realize the function. Additionally, multiple components in FIG. 1 may be integrated into a single processor to achieve the functions.

The acquisition function 151 acquires the tissue image 61 from the tissue image storage device 202. The acquisition function 151 also acquires the sample image 51 from the sample image storage device 201. Note that the source of acquisition of the sample image 51 and the tissue image 61 is not limited to the sample image storage device 201 and the tissue image storage device 202. For example, the acquisition function 151 may acquire the tissue image 61 from an imaging device that captured the tissue image 61.

The specimen image 51 and tissue image 61 to be acquired are specified, for example, by a user operation accepted by the reception function 155. The user of the information processing device 100 is a pathologist, etc.

The user specifies the specimen image 51 or tissue image 61 to be acquired by, for example, inputting a pathology number, a patient ID, a specimen number, an order number, or an imaging date, or a combination of these. Furthermore, one pathology number corresponds to one order number, one pathology number is associated with one or more specimen numbers and specimen image IDs, and one specimen number is associated with one or more tissue image IDs. Therefore, when a user inputs one pathology number to process a request for a pathology diagnosis, one or more specimen images 51 and one or more tissue images 61 are acquired.

The user may also select the image to be acquired from a list of sample images 51 and tissue images 61 stored in the sample image storage device 201.

The specimen image 51 and tissue image 61 may be stored in advance in the memory circuitry 120 and read out by user operation.

The recognition function 152 uses image processing to recognize multiple tissue regions 62 in which the tissue specimen 6 is depicted in the tissue image 61.

Figure 3 is a diagram showing an example of a tissue image 61 according to the first embodiment. The left side of Figure 3 is a tissue image 61 in which slices 60a to 60c contained in a tissue specimen 6 are depicted. In the tissue image 61, the glass slide 7 reflected as the background and slices 60f to 60h are depicted. As shown on the right side of Figure 3, the recognition function 152 recognizes a background region 70 in which the glass slide 7 is depicted, and tissue regions 62f to 62h in which slices 60f to 60h are depicted.

The method for recognizing the tissue region 62 from the tissue image 61 is not particularly limited, but for example, the recognition function 152 may recognize the boundary between the background region 70 and the tissue region 62 by performing an image segmentation process on the tissue image 61. In addition, for example, the recognition function 152 may employ rule-based image processing such as Selective Search, which selects candidate regions by grouping similar regions at the pixel level.

Alternatively, image processing using machine learning such as deep learning may be employed as a method for recognizing the tissue region 62 from the tissue image 61. When deep learning is employed, the recognition function 152 inputs the tissue image 61 to a trained model stored in the memory circuitry 120, and obtains the recognition result of the tissue region 62 output from the trained model. The trained model used in the recognition process of the tissue region 62 is a model in which the correspondence between the multiple tissue images 61 and the tissue regions 62 in the multiple tissue images 61 has been learned by a method such as deep learning. The trained model may be incorporated into the recognition function 152.

The recognition function 152 also groups the recognized tissue regions 62 into one or more tissue regions 62 that correspond to one slice 60. For example, one slice 60 may be divided into multiple pieces and placed on the slide glass 7. In such a case, the multiple tissue regions 62 correspond to one slice 60. A group of one or more tissue regions 62 derived from one slice 60 is called a slice region. The method by which the recognition function 152 recognizes the slice region is not particularly limited, but for example, image processing using machine learning such as deep learning may be adopted.

In the example shown in FIG. 3, each of the tissue regions 62f-62h is an image of a single slice 60f-60h, so the tissue region 62 and the slice region are equal.

Returning to FIG. 1, the image processing function 153 acquires first region of interest information that represents a region of interest in the tissue image 61. In this embodiment, the term "acquire" includes obtaining information by image processing.

In this embodiment, the abnormal area in the tissue region 62 is an example of a region of interest. Also, in this embodiment, information indicating the position of the abnormal area on the tissue image 61 is an example of first region of interest information. The information indicating the position of the abnormal area is, for example, coordinate information of a pixel in the tissue image 61 where the abnormal area is depicted.

An abnormal area is, for example, an area where a lesion is depicted. Specifically, an area in tissue region 62 where a tumor such as an adenoma or adenocarcinoma is depicted is an abnormal area.

The image processing function 153 analyzes the degree of abnormality, the presence or absence of abnormality, the range of the abnormal area, etc. in the tissue region 62. The image processing method for analyzing the abnormality is not particularly limited, and any known method can be adopted. For example, the image processing function 153 recognizes an abnormal area from the tissue region 62 by image segmentation processing using a trained model generated by a technology such as deep learning. For example, when a tissue image 61 is input, the trained model outputs information indicating the position of the abnormal area.

When multiple tissue regions 62 are included in one slice region, the image processing function 153 may perform image processing to analyze abnormalities for each slice region. Information regarding the abnormal area analyzed by the image processing function 153 is associated with the slice region in which the abnormal area was detected.

Figure 4 is a diagram showing an example of an abnormal region according to the first embodiment. In the example shown in Figure 4, tissue region 62i included in tissue image 61 includes three abnormal regions 91a to 91c. In Figure 4, abnormal regions 91a to 91c are each surrounded by a contour line to indicate the range of abnormal regions 91a to 91c, but the contour line does not need to be displayed on an actual image viewer. Hereinafter, when there is no need to distinguish between the individual abnormal regions 91a to 91c, they will simply be referred to as abnormal regions 91. There is no particular limit to the number of abnormal regions 91 included in one tissue region 62i.

The generation function 154 generates second region information representing the region of interest based on the tissue region 62 and the first region of interest information. In this embodiment, the generation function 154 generates a display annotation representing the abnormal region based on the tissue region 62 and information indicating the position of the abnormal region on the tissue image 61.

In general, an annotation is an image or comment that notes or provides other information about the tissue region 62. In this embodiment, the display annotation is an image that indicates the location of an abnormality in the tissue region 62.

Figure 5 is a diagram showing an example of display annotations 81a to 81c according to the first embodiment. Figure 5 shows display annotations 81a to 81c corresponding to abnormal areas 91a to 91c described in Figure 4. Hereinafter, when there is no need to distinguish between the individual display annotations 81a to 81c, they will simply be referred to as display annotation 81.

The generation function 154 generates display annotations 81a-81c that represent the positions of the abnormal regions 91a-91c based on the positions of the abnormal regions 91a-91c and do not overlap the tissue region 62i. The display annotations 81a-81c are displayed on the background region 70 where the tissue region 62i is not depicted.

In this embodiment, the display annotations 81a to 81c are shaped to follow the outer contour of the tissue region 62i. The coordinates of the left and right ends of the display annotations 81a to 81c in the horizontal direction of the tissue image 61 match the coordinates of the left and right ends of the abnormal regions 91a to 91c.

In other words, the coordinates of the start and end of the display annotation 81 in the horizontal direction of the tissue image 61 match the coordinates of the start and end of the abnormal area 91. That is, in the horizontal direction of the tissue image 61, the range of the display annotation 81 matches the range of the abnormal area 91. The horizontal direction is an example of the first direction. If the tissue regions 62 are arranged vertically, the first direction may be the vertical direction. Note that the left end may be the start end and the right end may be the end end, or vice versa.

In the example shown in FIG. 5, the horizontal range of the display annotation 81a is from the left end of the tissue image 61 to the line L1, similar to the abnormal region 91a. The left ends of the display annotation 81a and the abnormal region 91a are both at the left end of the tissue image 61, and the right ends of the display annotation 81a and the abnormal region 91a are both at the position shown by the line L1.

The horizontal range of both the display annotation 81b and the abnormal region 91b is from line L2 to line L3. The horizontal range of both the display annotation 81c and the abnormal region 91c is from line L4 to the right end of the tissue image 61.

In addition, when the horizontal ranges of multiple display annotations 81 overlap as shown in FIG. 5, the generation function 154 moves one of the display annotations 81 away from the tissue area 62i to prevent the multiple display annotations 81 from overlapping and becoming invisible. Note that the display position of the display annotation 81 may be determined at the time of generation by the generation function 154, or may be adjusted by the display control function 156 when it is displayed.

The positions and shapes of the display annotations 81a to 81c are not limited to the example shown in FIG. 5, and may be any positions and shapes that do not overlap with the tissue region 62i.

In the example shown in FIG. 5, the display annotations 81a-81c extend horizontally because they extend in a direction along the mucosal surface of the tissue region 62i. If the tissue region 62i is placed so that the mucosal surface is vertical or diagonal to the image, the display annotations 81a-81c also extend vertically or diagonally. In this case, the first direction is the direction along the mucosal surface, and the coordinates of the end of the display annotation 81 in the first direction match the coordinates of the end of the abnormal area 91 in the first direction.

Returning to FIG. 1, the reception function 155 receives various operations by the user via the input interface 130. For example, when the tissue image 61 includes multiple tissue regions 62, the reception function 155 receives a user operation to select one of the tissue regions 62. The selection method is a click operation using a mouse, but is not limited to this.

The display control function 156 displays the display annotation 81 on the display 140 at a position that does not overlap with the tissue area 62. In this embodiment, as shown in FIG. 5, the display control function 156 displays the display annotation 81 on the tissue image 61.

Next, we will explain the flow of the display annotation generation process executed by the information processing device 100 configured as described above.

Figure 6 is a flowchart showing an example of the flow of the process for generating display annotations according to the first embodiment.

First, the acquisition function 151 acquires a sample image 51 from the sample image storage device 201 (S1).

Then, the recognition function 152 uses image processing to recognize multiple tissue regions 62 in which the tissue specimen 6 is depicted from the tissue image 61. Furthermore, when multiple tissue regions 62 correspond to one slice 60, the recognition function 152 uses image segmentation processing or the like to recognize a slice region that is a group of multiple tissue regions 62 that correspond to the same slice 60 (S2).

Next, the image processing function 153 recognizes an abnormal area 91 on the tissue image 61 through image processing (S3). The image processing function 153 identifies the position and range of the recognized abnormal area 91 on the tissue image 61.

Next, the generation function 154 generates a display annotation 81 based on the position and range of the abnormal area 91 on the tissue image 61 and the position on the tissue image 61 (S4). The generation function 154 stores the display annotation 81 in the memory circuitry 120 in association with the tissue region 62 or slice region that includes the abnormal area 91.

Then, the display control function 156 superimposes the display annotation 81 on the tissue image 61 and displays it on the display 140 (S5). The timing for displaying the display annotation 81 may be when the user selects the tissue image 61 as the display target, or when the user selects the display annotation 81 after the tissue image 61 is displayed. At this point, the processing of this flowchart ends.

In this manner, the information processing device 100 of this embodiment generates a display annotation 81 representing the abnormal area 91 based on information indicating the positions of the tissue area 62 and the abnormal area 91 in the tissue image 61. The information processing device 100 of this embodiment displays the display annotation 81 on the display 140 at a position that does not overlap the tissue area 62. Therefore, according to the information processing device 100 of this embodiment, information regarding the slice 60 depicted in the tissue image 61 is displayed at a position that does not overlap the tissue area 62 in which the slice 60 is depicted.

The information processing device 100 of this embodiment also acquires information indicating the position of the abnormal area 91 from the tissue image 61, generates a display annotation 81 that represents the position of the abnormal area 91 and does not overlap the tissue area 62 based on the information indicating the position of the abnormal area 91, and displays the display annotation 81 on the tissue image 61. Therefore, according to the information processing device 100 of this embodiment, the user can avoid the display annotation 81 interfering with the observation of the tissue area 62.

In addition, the display annotation 81 of this embodiment has a shape that follows the outside of the contour of the tissue region 62, and the range of the display annotation 81 in the first direction of the tissue image 61 coincides with the range of the abnormal area 91 in the first direction. Therefore, according to the information processing device 100 of this embodiment, the user can easily grasp the position and range of the abnormal area 91 in the tissue region 62.

Second Embodiment
In the above-described first embodiment, the display annotation is displayed based on the information indicating the position of the abnormal region 91 obtained by image processing. In this second embodiment, the display annotation is displayed based on the designation of a region of interest by the user.

The reception function 155 in this embodiment receives an input operation of an instruction annotation by a user. The reception function 155 is an example of an area of interest information acquisition unit and reception unit in this embodiment. In this embodiment, "acquire" also includes obtaining information from an input by a user. Also, the instruction annotation is an example of first area of interest information in this embodiment.

FIG. 7 is a diagram showing an example of an instruction annotation 91d according to the second embodiment. For example, the user draws the instruction annotation 91d by operating the mouse or the like on the tissue image 61 displayed on the display 140. The tissue image 61 is an example of a diagnostic image.

The instruction annotation 91d is an image that indicates the position and range of the region of interest in the tissue region 62.

The range of the instruction annotation 91d in the direction of the mucosal surface of the tissue region 62j coincides with the range of the region of interest in the direction of the mucosal surface of the tissue region 62j. In this embodiment, the region of interest is defined by the user. For example, a region that the user determines may be a lesion site becomes the region of interest. The tissue region 62j is an example of a region to be observed.

The generation function 154 of this embodiment generates a display annotation that does not overlap with the tissue region 62 based on the tissue region 62 and the instruction annotation 91d. The display annotation is an example of second region information.

Figure 8 is a diagram showing an example of a display annotation 81d according to the second embodiment. The position and range of the display annotation 81d correspond to the instruction annotation 91d. The range of the display annotation 81d in the horizontal direction of the tissue image 61 is equal to the range of the instruction annotation 91d in the horizontal direction.

The horizontal range of both the display annotation 81d and the instruction annotation 91d is from line L5 to line L6. The display annotation 81d is a linear image that is placed outside the tissue region 62j along the contour line of the tissue region 62j. The display annotation 81d does not overlap the tissue region 62j, and is displayed, for example, on the background region 70.

Figure 9 shows another example of instruction annotations 91e, 91f and display annotations 81e, 81f according to the second embodiment.

In addition, when the tissue image 61 includes multiple tissue regions 62, the tissue region 62 included in the slice region selected by the user becomes the target of the instruction annotations 91e, 91f. When the slice region includes only one tissue region 62, selecting the slice region is essentially the same as selecting the tissue region 62. In addition, when one slice region includes multiple tissue regions 62, the selection is made by the user on a slice region basis. The user does not need to be aware that the selection is being made on a slice region basis, and other tissue regions 62 that belong to the same slice region as the tissue region 62 selected by the user may be automatically selected.

In FIG. 9, it is assumed that tissue region 62k has been selected by the user. In this case, generation function 154 associates display annotations 81e and 81f corresponding to instruction annotations 91e and 91f with tissue region 62k. The selection of tissue region 62 by the user is, for example, but not limited to, clicking with a mouse.

Tissue regions 62k, 62l, and 62m are included on the tissue image 61, but display annotations 81e and 81f are not set in the tissue regions 62l and 62m that are not selected by the user because they are not the targets of instruction annotations 91e and 91f.

In addition, the display control function 156 of this embodiment causes the display 140 to display instruction annotations 91e, 91f and display annotations 81e, 81f corresponding to the tissue area 62 selected by the user.

The display control function 156 does not display instruction annotations 91e and 91f corresponding to tissue regions 62 that are not selected by the user on the display 140. For example, in the example shown in FIG. 9, if the user deselects tissue region 62k and no tissue region 62 is selected, or if the user selects other tissue regions 62l and 62m, the display control function 156 ends the display of instruction annotations 91e and 91f corresponding to tissue region 62. The display control function 156 displays display annotations 81e and 81f regardless of whether the user has selected them. Note that the display and non-display of display annotations 81e and 81f may be switchable by a user operation.

Also, the user may set multiple instruction annotations 91e, 91f in one tissue region 62, as shown in FIG. 9. When the ranges of the multiple instruction annotations 91e, 91f overlap, the generation function 154 changes the position or shape of the display annotations 81e, 81f corresponding to the overlapping multiple instruction annotations 91e, 91f so that they do not overlap. The generation function 154 may also change the color or line state of the multiple display annotations 81e, 81f. Also, when the user is in the middle of an operation to set multiple instruction annotations 91e, 91f for one of the tissue regions, the display control function 156 may hide other instruction annotations and display annotations during the operation.

Figure 10 is a flowchart showing an example of the flow of the process for generating a display annotation 81 according to the second embodiment.

The process from obtaining a tissue image in S1 to recognizing the tissue region 62 and slice region depicted in the tissue image 61 in S2 is the same as in the first embodiment.

The reception function 155 receives the selection of a slice region by the user (S21). The reception function 155 then acquires the instruction annotations 91d to 91f that the user has set for the tissue region 62 included in the selected slice region (S22).

The generation function 154 generates a display annotation 81 that does not overlap with the tissue area 62 based on the tissue area 62 and the instruction annotations 91d to 91f (S23).

Then, the display control function 156 superimposes the display annotation 81 on the tissue image 61 and displays it on the display 140 (S24). At this point, the processing of this flowchart ends.

In this way, the information processing device 100 of this embodiment can display the display annotation 81 that does not hide the tissue area 62, based on the instruction annotations 91d to 91f set by the user. Therefore, according to the information processing device 100 of this embodiment, when the user draws the instruction annotations 91d to 91f, the user does not need to take the time to avoid writing in the tissue area 62, and the work can be made more efficient.

In addition, in this embodiment, the instruction annotations 91d-91f are images that indicate the position and range of the region of interest in the tissue region 62, and the display annotations 81d-81f are images that are placed outside the tissue region along the contour of the tissue region 62. The positions and ranges of the display annotations 81d-81f correspond to the instruction annotations 91d-91f. Therefore, according to the information processing device 100 of this embodiment, the user can display the display annotations 81d-81f that indicate the position and range of the region of interest in a position that does not overlap with the tissue region 62, simply by performing an operation to specify the position and range of the region of interest.

The information processing device 100 of this embodiment also displays on the display 140 the instruction annotations 91d-91f and the display annotations 81d-81f that correspond to the tissue region 62 selected by the user. The information processing device 100 also does not display on the display 140 the instruction annotations 91d-91f that correspond to the tissue region 62 that is not selected by the user. Therefore, according to the information processing device 100 of this embodiment, it is possible to manage the display/non-display of the instruction annotations 91d-91f and the display annotations 81d-81f for each tissue region 62 or for each slice region based on the user's selection.

(Variation 1)
The display manner of the instruction annotations 91d to 91f is not limited to that of the second embodiment described above. For example, in the second embodiment, the instruction annotations 91d to 91f are displayed at the positions drawn by the user, but are not limited to this. FIG. 11 is a diagram showing an example of the instruction annotations 91g, 911g and the display annotation 81g according to the first modified example. As shown in FIG. 11, when the position where the user draws the instruction annotation 91g is on the tissue region 62n, the display control function 156 may move the instruction annotation 91g onto the background region 70 and display it as the instruction annotation 911g.

(Variation 2)
In addition, in the first and second embodiments described above, the display annotation 81 is linear, but is not limited to this. The display annotation 81 may be displayed not only along the contour of one side of the tissue image 61, but also on both sides of the tissue image 61. Fig. 12 is a diagram showing an example of display annotations 81h and 81i according to the second modification. As shown in Fig. 12, the display annotations 81h and 81i may be provided on both sides of the tissue image 61 along a range including a region of interest.

(Variation 3)
Moreover, Fig. 13 is a diagram showing an example of display annotations 81j and 81k according to Modification Example 3. As shown in Fig. 13, the display annotations 81j and 81k may be figures surrounding a range including a region of interest.

(Variation 4)
In addition, in each of the second embodiments described above, the screen on which the user operates the instruction annotations 91d to 91f and the screen on which the display annotations 81d to 81f are displayed are the same. In contrast, in this modified example, a window for operation and a window for display are provided separately.

FIG. 14 is a diagram showing an example of a screen display according to the fourth modified example. In the example shown in FIG. 14, the display control function 156 provides an operation window 141 and a display window 142 on the display 140. The same tissue image 61 is displayed in the operation window 141 and the display window 142, but the settings such as zooming can be changed for each individual window. For example, as shown in FIG. 14, the user may enlarge the tissue image 61 displayed in the operation window 141 more than the tissue image 61 displayed in the display window 142. In this case, the user can operate the instruction annotations 91x, 91y on the tissue image 61 enlarged and displayed in the display window 142, making it easy to make fine settings.

In FIG. 14, the same tissue image 61 is displayed in the operation window 141 and the display window 142, but the display control function 156 may display different images in the operation window 141 and the display window 142. For example, a three-dimensional image of the tissue image 61 may be displayed in the display window 142, and a two-dimensional tissue image 61 may be displayed in the operation window 141.

In the above-described embodiments, linear annotations have been exemplified, but the form of the annotation to be displayed is not limited to this. For example, text data or an image may be displayed on the display 140. The display control function 156 may display various meta information related to the selected slice region 8, for example, in the background region 70 near the slice region selected by the user. Also, a specific area in which meta information is displayed may be provided on the GUI displayed on the display 140.

(Variation 5)
In addition, in each of the above-described embodiments, the term "obtain" includes obtaining information by image processing and obtaining information from user input. In addition, receiving information from another information processing device is also included in "obtaining". For example, receiving first region of interest information representing a region of interest from a hospital information system or a clinical testing system by the acquisition function 151 via the NW interface 110 is also included in "obtaining". In this case, the acquisition function 151 is an example of a region of interest information acquisition unit.

(Variation 6)
In addition, in each of the above-described embodiments, the user determines the slice area on which the instruction annotation is to be set by performing an operation such as clicking within the slice area on the tissue image 61, but the association of the instruction annotation is not limited to this method. For example, the user may draw the instruction annotation on the desired slice area. In this case, the generation function 154 generates the display annotation in association with the slice area on which the instruction annotation is superimposed.

The generation function 154 may also determine the association between the instruction annotation and the slice area based on the proximity of the distance between the instruction annotation and the slice area, regardless of whether a selection operation has been performed. For example, the generation function 154 may generate a display annotation that corresponds to the instruction annotation by associating it with the slice area that is closest to the position where the user has drawn the instruction annotation.

(Variation 7)
In each of the above-mentioned embodiments, the extension direction of the instruction annotation and the display annotation was the horizontal direction of the tissue image 61, but depending on the direction of the mucosal surface of the tissue region 62, the extension direction of the instruction annotation and the display annotation may be the vertical direction or diagonal direction of the tissue image 61.

(Variation 8)
In the above-described embodiments, the information processing device 100 is a server device or a PC, but is not limited thereto. For example, the information processing device 100 may be a computer built into an imaging device that captures the tissue image 61. The imaging device is, for example, a WSS, a digital camera, or a digital microscope, but is not limited thereto.

(Modification 9)
Furthermore, in each of the above-described embodiments, the tissue image 61 is taken as an example of a diagnostic image, but the diagnostic image may be a medical image captured by a modality such as an X-ray diagnostic apparatus.

The various data discussed in this specification are typically digital data.

According to at least one of the embodiments described above, information about a slice depicted in a tissue image can be displayed in a position that does not overlap with the area in which the slice is depicted.

Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and spirit of the invention.

5 Specimen 6 Tissue specimen 7 Slide glass 51 Specimen image 60, 60a to 60e Section 61 Tissue image 62, 62a to 62p Tissue region 70 Background region 81a to 81k, 81x, 81y Display annotation 91d to 91g, 91x, 91y, 911g, 911x, 911y Instruction annotation 91a to 91c Abnormal region 100 Information processing device 110 NW interface 120 Memory circuit 140 Display 141 Operation window 142 Display window 150 Processing circuit 151 Acquisition function 152 Recognition function 153 Image processing function 154 Generation function 155 Reception function 156 Display control function S Information processing system

Claims (12)

a recognition unit for recognizing an observation target region in a diagnostic image;
a region-of-interest information acquisition unit that acquires one or more first region-of-interest information pieces that represent ranges of one or more regions of interest within the observation target region in the diagnostic image;
a generating unit that generates second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information;
a display control unit that displays the second region information at a position outside the observation target region on a display unit;
An information processing device comprising: When a plurality of pieces of first region-of-interest information are acquired and ranges of the regions of interest indicated by the plurality of pieces of first region-of-interest information overlap, the display control unit moves a position of the second region information corresponding to any one of the plurality of pieces of first region-of-interest information, thereby displaying the plurality of pieces of second region information corresponding to the plurality of pieces of first region-of-interest information on the display unit without overlapping each other.
The information processing device according to claim 1 . the region-of-interest information acquisition unit acquires one or more first region-of-interest information pieces that represent positions and ranges of one or more regions of interest within the observation target region in the diagnostic image;
The generating unit generates second region information representing a position and a range of the region of interest corresponding to the first region of interest information, based on the observation target region and the first region of interest information.
3. The information processing device according to claim 1 or 2. The diagnostic image is a tissue image depicting a tissue specimen including at least one slice;
the observation target region is a tissue region in which the tissue specimen is depicted in the tissue image,
the region of interest is an abnormal site in the tissue region;
the region-of-interest information acquisition unit acquires, by image processing, information indicating a position of the abnormal site from the tissue image as the first region-of-interest information;
the generating unit generates, as the second region information, a display annotation that indicates a position of the abnormal site and does not overlap the tissue region, based on the first region of interest information;
The display control unit causes the display annotation to be displayed on the diagnostic image.
The information processing device according to claim 1 . the display annotation has a shape that conforms to an outer contour of the tissue region;
a range of the display annotation in a first direction of the tissue image corresponds to a range of the abnormality region in the first direction;
The information processing device according to claim 4 . the first region of interest information is an instruction annotation indicating the region of interest,
the region of interest information acquisition unit receives an input operation of the instruction annotation by a user; and the generation unit generates, as second region information, a display annotation that does not overlap with the observation target region, based on the observation target region and the instruction annotation.
The information processing device according to claim 1 . The diagnostic image is a tissue image depicting a tissue specimen including at least one slice;
the observation target region is a tissue region in which the tissue specimen is depicted in the tissue image,
the instruction annotation is an image indicating a location and extent of the region of interest in the tissue region;
the display annotation is an image arranged outside the tissue region along a contour line of the tissue region,
The position and range of the display annotation correspond to the instruction annotation.
The information processing device according to claim 6 . the instructional annotation is an image of lines indicating the location and extent of the region of interest in the tissue region;
The information processing device according to claim 7. The display control unit is
displaying, on the display unit, the instruction annotation and the display annotation corresponding to a tissue region selected by a user;
a display form of the instruction annotation corresponding to a tissue region not selected by a user is made different from a display form of the instruction annotation corresponding to a tissue region selected by a user;
The information processing device according to claim 6 . The display control unit is
displaying, on the display unit, the instruction annotation and the display annotation corresponding to a tissue region selected by a user;
The instruction annotation corresponding to a tissue region not selected by a user is not displayed on the display unit.
The information processing device according to claim 8 . a recognition step of recognizing an observation target region in a diagnostic image;
an acquiring step of acquiring one or more first region of interest information representing ranges of one or more regions of interest within the observation target region in the diagnostic image;
a generating step of generating second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information;
a display step of displaying the second region information on a display unit at a position outside the observation target region ;
An information processing method comprising: a recognition step of recognizing an observation target region in a diagnostic image;
an acquiring step of acquiring one or more first region of interest information representing ranges of one or more regions of interest within the observation target region in the diagnostic image;
a generating step of generating second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information;
a display step of displaying the second region information on a display unit at a position outside the observation target region ;
A program that causes a computer to execute the following.

Images