JP7660421B2 - Information processing device, information processing method, and program - Google Patents

Description

The embodiments disclosed in this specification and the drawings relate to an information processing device, an information processing method, and a program.

In the field of pathology and the like, there is a known technique for setting information such as annotations on tissue images in which multiple slices cut out from a specimen are captured, based on user operations or image processing results.

Even within a single specimen, the condition of each section is different, so it is desirable to associate meta-information such as annotations with each section. However, when a tissue image contains multiple tissue regions, it can be difficult to identify the group of tissue regions that make up a single section.

JP 2019-095853 A

One of the problems that the embodiments disclosed in this specification and the drawings attempt to solve is to manage multiple tissue regions included in a tissue image on a slice-by-slice basis. However, the problems that the embodiments disclosed in this specification and the drawings attempt to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems.

The information processing device according to the embodiment includes an acquisition unit, a tissue region recognition unit, and a slice region recognition unit. The acquisition unit acquires a tissue image in which a tissue specimen including at least one slice is depicted. The tissue region recognition unit recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image. The slice region recognition unit recognizes one or more tissue regions corresponding to one slice among the plurality of tissue regions as one slice region. When a plurality of slices are included in a tissue specimen depicted in one tissue image, the slice region recognition unit recognizes that the plurality of tissue regions are included in the same slice region when the shortest distance between the contours of the plurality of tissue regions is equal to or less than a threshold value, or when the slice region recognition unit identifies a plurality of tissue regions that are relatively close to each other based on the average or standard deviation of the shortest distances between all tissue regions included in the tissue image.

FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system according to the first embodiment. FIG. 2 is a diagram for explaining a flow until a tissue image is captured according to the first embodiment. FIG. 3 is a diagram showing an example of a tissue image according to the first embodiment. FIG. 4 is a diagram showing an example of a state in which a plurality of tissue regions are recognized from the tissue image shown in FIG. FIG. 5 is a diagram showing an example of a recognition result of a slice region according to the first embodiment. FIG. 6 is a diagram showing an example of a tissue image based on the slice arrangement rule (1) according to the first embodiment. FIG. 7 is a diagram showing an example of a tissue image based on the slice arrangement rule (2) according to the first embodiment. FIG. 8 is a diagram showing an example of a tissue image based on the slice arrangement rule (3) according to the first embodiment. FIG. 9 is a diagram showing an example of a method for recognizing a segment area based on the segment arrangement rule (3) according to the first embodiment. FIG. 10 is a diagram showing an example of a segment arrangement that is not permitted by segment arrangement rule (3) in the first embodiment. FIG. 11 is a diagram illustrating an example of the image viewer according to the first embodiment. FIG. 12 is a diagram showing an example of a technique for selecting a slice region using a list box according to the first embodiment. FIG. 13 is a diagram showing an example in which the selected segment region has been changed from that shown in FIG. FIG. 14 is a diagram illustrating an example of annotation settings according to the first embodiment. FIG. 15 is a diagram showing an example in which the selected segment region has been changed from that shown in FIG. FIG. 16 is a diagram showing an example of a state in which the selection of a slice region is released in the first embodiment. FIG. 17 is a flowchart showing an example of the flow of the segment region recognition process according to the first embodiment. FIG. 18 is a flowchart showing an example of the flow of the segment region recognition process according to the first embodiment. FIG. 19 is a diagram showing an example of a recognition result of a slice region according to the second embodiment. FIG. 20 is a diagram showing an example of an analysis result of an image feature amount according to the second embodiment. FIG. 21 is a diagram illustrating an example of evaluation of periodicity of image feature amounts according to the second embodiment. FIG. 22 is a diagram illustrating an example of input and output of a trained model according to the third embodiment. FIG. 23 is a diagram illustrating an example of a bounding box according to the third embodiment. FIG. 24 is a diagram showing an example of a mask image according to the third embodiment. FIG. 25 is a diagram showing an example of a display of an image viewer according to the first modification.

Below, embodiments of an information processing device, an information processing method, and a program will be described in detail with reference to the drawings.

(First embodiment)
Fig. 1 is a diagram showing an example of the overall configuration of an information processing system S according to the first embodiment. As shown in Fig. 1, the information processing system S includes an information processing device 100, a sample image storage device 201, and a tissue image storage device 202. The information processing device 100 is communicatively connected to the sample image storage device 201 and the tissue image storage device 202 via a network 300 such as an in-hospital LAN (Local Area Network).

The information processing system S may further include a specimen management system, a hospital information system (HIS), a laboratory information system (LIS), and a radiology information system (RIS). Alternatively, the information processing system S may be part of a hospital information system. The information processing system S may further include a terminal device such as a PC (Personal Computer) or a tablet terminal.

The information processing system S is provided, for example, in a medical institution such as a hospital, a research institute such as a university, or an examination center. In addition, some or all of the devices constituting the information processing system S may be provided in a cloud environment.

The specimen image storage device 201 is a device that stores specimen images. A specimen image is an image of the entire tissue piece that is the specimen. The specimen image is a digital image captured by an imaging device such as a digital camera. A specimen image is also called a macro image.

In this embodiment, the specimen is a portion of tissue collected from a patient's body, an animal, etc. For example, the specimen is the mucosa and submucosa of the digestive tract removed by endoscopic submucosal dissection (ESD), but is not limited thereto, and may be a piece of tissue removed by laparotomy, etc.

The tissue image storage device 202 is a device that stores tissue images. A tissue image is an image of at least one slice cut from a tissue piece that is a specimen, captured at a higher magnification than a specimen image. For example, a tissue image is a WSI (Whole Slide Imaging) image in which the entire slice or part of the slice placed on a glass slide is digitally imaged with high precision. A tissue image may be a micrograph captured by a WSS (Whole Slide Scanner) or a digital microscope, or an image captured by other methods. A tissue image is also called a micro image.

The specimen image storage device 201 and the tissue image storage device 202 are, for example, a server device or a PC. The specimen image storage device 201 and the tissue image storage device 202 may be collectively referred to as an image storage device. Although the specimen image storage device 201 and the tissue image storage device 202 are shown as separate devices in FIG. 1, the specimen image storage device 201 and the tissue image storage device 202 may be configured as an integrated device. The specimen image storage device 201, the tissue image storage device 202, and the information processing device 100 may be configured as an integrated device.

Figure 2 is a diagram explaining the flow until a tissue image according to the first embodiment is captured. For example, a specimen 5 collected from a patient P by a diagnostician is transferred to a pathologist or a laboratory technician for pathological diagnosis. At this time, a pathology number is assigned for each request (order) for pathological diagnosis. Also, a different specimen number is assigned for each specimen to be the subject of pathological diagnosis. Note that, when there are multiple specimens to be diagnosed in one request for pathological diagnosis, multiple specimen numbers are associated with one pathology number. Hereinafter, in this embodiment, a pathologist or laboratory technician is referred to as a pathologist, etc.

The processing performed by the information processing device 100 in this embodiment does not necessarily have to be aimed at pathological diagnosis. For example, it may be aimed at research by a research institute, or at the creation of a report at an examination center that performs pathological examinations on behalf of a medical institution.

A specimen image ID is assigned as identification information for the specimen image 51 obtained by capturing an image of the specimen 5 using a digital camera or the like. The specimen image ID may be registered as additional information for the specimen image 51. Information related to the specimen image 51, such as the pathology number of the specimen 5 that is the subject of the image of the specimen image 51, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from whom the specimen 5 was obtained, and the image date and time of the specimen image 51, may be registered as additional information for the specimen image 51. The specimen image ID and other information related to the specimen image 51 may be displayed as text information on the specimen image 51.

The pathologist or the like cuts out the slices 60a to 60e from the specimen 5, and after performing processing such as staining, for example, slices the cross sections of the slices 60a to 60e thinly and places them on a glass slide 7. Hereinafter, when there is no need to distinguish between the individual slices 60a to 60e, they will simply be referred to as slices 60.

In this embodiment, the group of sections placed on the slide glass 7 is called a tissue specimen 6. The tissue specimen 6 includes at least one section 60.

In addition, in FIG. 2, each section 60 is placed directly on the glass slide 7, but if the section 60 is long or depending on the shape of the cutting position, one section 60 may be divided into multiple pieces and placed on the glass slide 7.

2, as an example, the slices 60a to 60e are imaged with their cross sections facing the imaging device, but the orientation of the slices is not limited to this. For example, the slice 60 may be imaged in the same orientation as the specimen 5 when the specimen image 51 is imaged. Also, in FIG. 2, the slice 60 is arranged on the glass slide 7 so that the horizontal direction of the tissue image 61 is the longitudinal direction of the tissue region 62, but the arrangement direction is not limited to this. For example, the slice 60 may be arranged on the glass slide 7 so that the vertical direction of the tissue image 61 is the longitudinal direction of the tissue region 62. Alternatively, the slice 60 may be arranged diagonally on the glass slide 7. In this embodiment, the arrangement of the slice 60 is determined by slice arrangement rules described below.

An image such as a WSI image captured of the tissue specimen 6 placed on the slide glass 7 is the tissue image 61. The tissue image 61 depicts the tissue specimen 6 including slices 60a to 60e.

A tissue image ID is assigned as identification information of the tissue image 61. The tissue image ID may be registered as additional information of the tissue image 61. Information about the tissue image 61, such as the pathology number of the specimen 5 from which the slice that is the subject of the imaging of the tissue image 61 was obtained, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from which the specimen 5 was obtained, and the imaging date and time of the tissue image 61, may be registered as additional information of the tissue image 61. The tissue image ID and other information about the tissue image 61 may be depicted as text information on the tissue image 61. The additional information of the tissue image 61 may include identification information of a facility such as a medical institution that captured the tissue image 61. Identification information of a facility such as a medical institution that captured the imaging process of the tissue image 61 may be acquired, for example, from a clinical examination system or a specimen management system.

The tissue image 61 includes a background region 70 depicting the glass slide 7, and a number of tissue regions 62a-62e depicting the tissue specimen 6. Hereinafter, when there is no need to distinguish between the individual tissue regions 62a-62e, they will simply be referred to as tissue regions 62.

Returning to FIG. 1, the information processing device 100 is, for example, a server device or a PC, and has a network interface 110, a memory circuit 120, an input interface 130, a display 140, and a processing circuit 150.

The NW interface 110 is connected to the processing circuit 150 and controls the transmission and communication of various data between the information processing device 100 and the sample image storage device 201 or the tissue image storage device 202. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc.

The memory circuitry 120 prestores various types of information to be used by the processing circuitry 150. The memory circuitry 120 also stores various programs.

The input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touch screen that integrates a display screen and a touchpad, a non-contact input circuit using an optical sensor, and a voice input circuit. The input interface 130 is connected to the processing circuit 150, and converts input operations received from an operator into electrical signals and outputs them to the processing circuit 150. Note that in this specification, the input interface is not limited to those that have physical operating parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs these electrical signals to the processing circuit 150 is also included as an example of the input interface 130.

The display 140 is a liquid crystal display, an organic electro-luminescence (OEL) display, or the like. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit.

The processing circuit 150 is a processor that reads out a program from the memory circuit 120 and executes it to realize a function corresponding to each program. The processing circuit 150 of this embodiment includes an acquisition function 151, a tissue area recognition function 152, a slice area recognition function 153, a display image generation function 154, a display control function 155, a reception function 156, an image processing function 157, a meta information setting function 158, and a correction function 159. The acquisition function 151 is an example of an acquisition unit. The tissue area recognition function 152 is an example of a tissue area recognition unit. The slice area recognition function 153 is an example of a slice area recognition unit. The display image generation function 154 is an example of a display image generation unit. The display control function 155 is an example of a display control unit. The reception function 156 is an example of a reception unit. The image processing function 157 is an example of an image processing unit. The meta information setting function 158 is an example of a meta information setting unit. The correction function 159 is an example of a correction section.

Here, for example, each of the processing functions of the processing circuit 150, namely, the acquisition function 151, the tissue area recognition function 152, the slice area recognition function 153, the display image generation function 154, the display control function 155, the reception function 156, the image processing function 157, the meta information setting function 158, and the correction function 159, are stored in the memory circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes the function corresponding to each program by reading and executing the program from the memory circuit 120. In other words, the processing circuit 150 in the state in which each program has been read has each function shown in the processing circuit 150 of FIG. 1. In FIG. 1, the processing functions performed by the acquisition function 151, tissue area recognition function 152, slice area recognition function 153, display image generation function 154, display control function 155, reception function 156, image processing function 157, meta information setting function 158, and correction function 159 are described as being realized by a single processor, but the processing circuit 150 may be configured to be realized by combining multiple independent processors and each processor executing a program. Also, in FIG. 1, the single storage circuit 120 is described as storing a program corresponding to each processing function, but multiple storage circuits may be distributed and the processing circuit 150 may be configured to read out the corresponding program from each storage circuit.

In the above description, an example was described in which the "processor" reads out and executes a program corresponding to each function from a storage circuit, but the embodiment is not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes the function by reading out and executing a program stored in a storage circuit. On the other hand, when the processor is an ASIC, instead of storing a program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in this embodiment is not limited to being configured as a single circuit for each processor, and may be configured as a single processor by combining multiple independent circuits to realize the function. Furthermore, multiple components in FIG. 1 may be integrated into a single processor to achieve the functions.

The acquisition function 151 acquires the tissue image 61 from the tissue image storage device 202. The acquisition function 151 also acquires the sample image 51 from the sample image storage device 201. Note that the source of acquisition of the sample image 51 and the tissue image 61 is not limited to the sample image storage device 201 and the tissue image storage device 202. For example, the acquisition function 151 may acquire the tissue image 61 from an imaging device that captured the tissue image 61.

The specimen image 51 and tissue image 61 to be acquired are specified, for example, by a user operation accepted by the reception function 156. The user of the information processing device 100 is a pathologist, etc.

The user specifies the specimen image 51 or tissue image 61 to be acquired by, for example, inputting a pathology number, a patient ID, a specimen number, an order number, or an imaging date, or a combination of these. Furthermore, one pathology number corresponds to one order number, one pathology number is associated with one or more specimen numbers and specimen image IDs, and one specimen number is associated with one or more tissue image IDs. Therefore, when a user inputs one pathology number to process a request for a pathology diagnosis, one or more specimen images 51 and one or more tissue images 61 are acquired.

The user may also select the object to be acquired from a list of sample images 51 and tissue images 61 stored in the sample image storage device 201.

The specimen image 51 and tissue image 61 may be stored in advance in the memory circuitry 120 and read out by user operation.

The tissue area recognition function 152 uses image processing to recognize multiple tissue areas 62 in which the tissue specimen 6 is depicted from the tissue image 61.

Figure 3 is a diagram showing an example of a tissue image 61 according to the first embodiment. Also, Figure 4 is a diagram showing an example of a state in which a plurality of tissue regions 62f-62g are recognized from the tissue image 61 shown in Figure 3. In the tissue image 61, the image region other than the plurality of tissue regions 62f-62g is a background region 70 in which the glass slide 7 is depicted.

The method for recognizing the tissue region 62 from the tissue image 61 is not particularly limited, but for example, the tissue region recognition function 152 may recognize the boundary between the background region 70 and the tissue region 62 by performing an image segmentation process on the tissue image 61. In addition, for example, the tissue region recognition function 152 may employ rule-based image processing such as Selective Search, which selects candidate regions by grouping similar regions at the pixel level.

Alternatively, image processing using machine learning such as deep learning may be employed as a method for recognizing the tissue region 62 from the tissue image 61. When deep learning is employed, the tissue region recognition function 152 inputs the tissue image 61 to a trained model stored in the memory circuitry 120, and obtains the recognition result of the tissue region 62 output from the trained model. The trained model used in the recognition process of the tissue region 62 is a model in which the correspondence between the multiple tissue images 61 and the tissue regions 62 in the multiple tissue images 61 has been trained by a method such as deep learning. The trained model may be incorporated into the tissue region recognition function 152.

Next, the slice area recognition function 153 recognizes one or more tissue areas 62 corresponding to one slice 60 from among the multiple tissue areas recognized by the tissue area recognition function 152 as one slice area. For example, one slice 60 may be divided into multiple pieces and placed on the slide glass 7. In such a case, multiple tissue areas 62 correspond to one slice 60. For this reason, the tissue area recognition function 152 groups the multiple tissue areas 62 on a slice 60 basis. A group of one or more tissue areas 62 derived from one slice 60 is called a slice area.

For example, when one slice 60 is divided into multiple tissue regions 62 and depicted, the slice region recognition function 153 recognizes the multiple tissue regions 62 corresponding to one slice 60 as one slice region.

A section region is an image region in which one section 60 is depicted in the tissue image 61. If one section 60 is divided into multiple pieces when placed on the slide glass 7, the single section 60 is depicted in the tissue image 61 as multiple tissue regions 62. In this case, the multiple tissue regions 62 become one section region. In other words, a section region is a group of tissue regions 62 derived from one section 60. Note that, since one section 60 may be depicted in the tissue image 61 as one tissue region 62 without being divided, a group of tissue regions 62 includes one or more tissue images 61.

Figure 5 is a diagram showing an example of the recognition result of slice regions 8a and 8b according to the first embodiment. In the example shown in Figure 5, of the tissue regions 62f to 62i included in the tissue image 61, tissue region 62f is slice region 8a, and tissue regions 62g to 62i are slice region 8b. The three tissue regions 62g to 62i are image regions in which fragments included in one slice 60 are captured. Furthermore, tissue region 62f is an image region in which a fragment included in a slice 60 different from the other tissue regions 62g to 62i is captured. Hereinafter, when there is no need to distinguish between the individual slice regions 8a and 8b, they will simply be referred to as slice region 8.

The slice area recognition function 153 recognizes the slice area 8 based on the specified slice placement rules.

The prescribed section placement rules are, for example, operational rules of a facility such as a medical institution or examination center that captures the tissue image 61. A technician or doctor who captures the tissue image 61 places the section 60 on the glass slide 7 in accordance with the operational rules.

In this embodiment, the following (1) to (3) are given as examples of the prescribed segment arrangement rules.
(1) Only one slice 60 is included in one tissue image 61. (2) When multiple slices 60 are included in the tissue specimen 6 depicted in one tissue image 61, the distance between multiple tissue regions 62 constituting the same slice is closer than the distance between multiple tissue regions 62 constituting other slices 60. (3) When multiple slices 60 are included in the tissue specimen 6 depicted in one tissue image 61, the multiple tissue regions 62 constituting one slice 60 are arranged along the shape of the slice 60 when the slice 60 is cut out from the specimen 5.

Which of the slice arrangement rules (1) to (3) is applied by the information processing device 100 may be determined in advance, or the user may be able to select one of the slice arrangement rules. For example, a system administrator or the like may set a default slice arrangement rule for the medical institution or the like where the information processing device 100 is used in the information processing device 100, and a general user may be able to change the slice arrangement rule by operating a GUI (Graphical User Interface) as necessary. Alternatively, the user may select the slice arrangement rule to be applied from a list of slice arrangement rules (1) to (3) each time the tissue image 61 is processed. The selection operation of the slice arrangement rule by the user is received by a reception function 156 described later. The slice area recognition function 153 may recognize the slice area 8 according to the slice arrangement rule selected by the user.

Alternatively, the information processing device 100 may be configured to support only one of the segment placement rules (1) to (3).

Alternatively, identification information of the facility where the tissue image 61 was captured may be registered in the supplementary information of the tissue image 61 or other metadata, and the slice area recognition function 153 may apply a slice arrangement rule corresponding to the facility. When this method is adopted, the memory circuitry 120 stores slice arrangement rule specification information in which identification information of the facility where the tissue image 61 is captured is associated with the slice arrangement rule adopted in the facility. Note that the facility where the tissue image 61 was captured may be the same as or different from the facility where the information processing device 100 is installed. The slice area recognition function 153 specifies the slice arrangement rule to be applied to the tissue image 61 based on the slice arrangement rule specification information.

Below, we explain the method for recognizing the segment area 8 according to each of the segment placement rules (1) to (3).

When slice placement rule (1) is applied, the slice area recognition function 153 recognizes all tissue areas 62 contained in the tissue image 61 as one slice area 8.

Figure 6 is a diagram showing an example of a tissue image 61 based on slice arrangement rule (1) according to the first embodiment. The tissue image 61 shown in Figure 6 includes three tissue regions 62g to 62i. According to slice arrangement rule (1), one tissue image 61 includes only one slice 60. Therefore, the slice area recognition function 153 recognizes all tissue regions 62 included in the tissue image 61 as one slice area 8. In the example shown in Figure 6, the slice area recognition function 153 recognizes the three tissue regions 62g to 62i as slice area 8b corresponding to one slice 60.

Figure 7 is a diagram showing an example of a tissue image 61 based on the slice arrangement rule (2) according to the first embodiment. According to the slice arrangement rule (2), when a tissue specimen 6 depicted in one tissue image 61 includes multiple slices 60, the distance between multiple tissue regions 62 constituting the same slice is closer than the distance between multiple tissue regions 62 constituting other slices 60. Therefore, the slice area recognition function 153 recognizes one or more tissue regions 62 corresponding to one slice 60 as one slice area 8 based on the distance between the multiple tissue regions 62.

For example, when the shortest distance between the contours of the tissue regions 62 is equal to or less than a threshold value, the slice region recognition function 153 recognizes that the tissue regions 62 are included in the same slice region 8. The threshold value is, for example, stored in advance in the memory circuitry 120. The shortest distance between the contours of the tissue regions 62 is the distance from the contour of one tissue region 62 to the contour of the other tissue region 62 at the point where the two tissue regions 62 are closest to each other.

In the example shown in FIG. 7, the shortest distance d1 between tissue region 62g and tissue region 62h, and the shortest distance d2 between tissue region 62h and tissue region 62i are less than or equal to a threshold value. In this case, the slice region recognition function 153 classifies tissue regions 62g to 62i into one group. Also, if the shortest distance d6 between tissue region 62g and tissue region 62i is greater than the threshold value, tissue region 62g and tissue region 62i are classified into the same group via tissue region 62h. The slice region recognition function 153 recognizes the tissue regions 62g to 62i included in one group as tissue regions 62g to 62i that constitute one slice region 8b.

The shortest distance d3 between tissue region 62f and tissue region 62g, the shortest distance d4 between tissue region 62f and tissue region 62h, and the shortest distance d5 between tissue region 62f and tissue region 62i are all greater than the threshold. In this case, the slice region recognition function 153 recognizes that tissue region 62f is a slice region 8 that is different from any of tissue regions 62g to 62i. Because tissue image 61 does not include any tissue region 62 other than tissue regions 62f to 62i, the slice region recognition function 153 recognizes that tissue region 62f is a single slice region 8a by itself.

The threshold value may not be a fixed value, but may vary according to the relative difference in the shortest distance between the tissue regions 62f-62i included in the tissue image 61. For example, the slice region recognition function 153 may group tissue regions 62 that are relatively close to each other based on the average or standard deviation of the shortest distance between the tissue regions 62f-62i.

Figure 8 is a diagram showing an example of a tissue image 61 based on the slice arrangement rule (3) according to the first embodiment. According to the slice arrangement rule (3), when a tissue specimen 6 depicted in one tissue image 61 includes multiple slices 60, the multiple tissue regions 62 constituting one slice 60 are arranged according to the shape of the slice 60 when the slice 60 is cut out from the specimen 5.

In the example shown in FIG. 8, tissue image 61 includes tissue regions 62f-62l. Of these, tissue regions 62g-62i are imaged as multiple tissue regions 62g-62i because one slice 60 is interrupted. Based on slice arrangement rule (3), slice 60 is arranged along the shape of slice 60 when it is cut out from specimen 5, so tissue regions 62g-i are arranged in a horizontal row along the original cut-out position of specimen 5.

When the slice arrangement rule (3) is applied, the slice area recognition function 153 estimates the direction in which the slices 60 are arranged and the range in which each slice 60 may be arranged based on the areas of the tissue areas 62 and the positions of the centers of gravity 65a-65g, and recognizes one or more tissue areas 62 included in the estimated range as one slice area 8. Hereinafter, when there is no particular need to distinguish between the individual centers of gravity 65a-65g, they will simply be referred to as the center of gravity 65.

The center of gravity 65 of the tissue region 62 is the center of the distribution of pixels contained in the tissue region 62. For example, the coordinates of the center of gravity 65 are the average value of the coordinates of the pixels contained in the tissue region 62. Note that the method of determining the center of gravity 65 can employ a known image processing method.

The slice area recognition function 153 determines the positions of the centers of gravity 65a-65g based on the pixels of each of the tissue areas 62g-62l. The positions of the centers of gravity 65a-65g are coordinates on the tissue image 61.

The slice area recognition function 153 also obtains the area of each of the tissue areas 62g to 62l, and estimates the arrangement direction of the slice 60 based on the position of the center of gravity 65 of the tissue area 62 having an area equal to or greater than the reference area calculated based on the areas of the tissue areas 62g to 62l. The tissue area 62 having an area equal to or greater than the reference area calculated based on the areas of the tissue areas 62g to 62l is also referred to as the representative tissue area. For example, the reference area may be determined by using the average value of the areas of the tissue areas. In the example shown in FIG. 8, as an example of using the average value of the areas of the tissue areas as the reference area, the center of gravity 65e of the tissue area 62g and the center of gravity 65g of the tissue area 62i are excluded because the areas of the tissue areas 62g to 62l are less than the average of the areas of the tissue areas 62g to 62l. The areas of the tissue areas 62h, 62j to 62l are equal to or greater than the average of the areas of the tissue areas 62g to 62l, and therefore are the representative tissue areas.

The slice area recognition function 153 estimates the angle of the direction in which the slices 60 are arranged based on the centers of gravity 65a to 65f of the tissue areas 62h, 62j to 62l. The direction in which the centers of gravity 65a to 65f are arranged may be determined, for example, by calculating the ratio of the variance of the X coordinate and the Y coordinate on the tissue image 61 in the coordinate group of the centers of gravity 65a to 65f, or by principal component analysis. The slice area recognition function 153 may perform principal component analysis on the coordinates of the centers of gravity 65a to 65f, and may set the first principal component direction as the direction in which the slices 60 are arranged. In the example shown in FIG. 8, the slice area recognition function 153 estimates the direction A as the direction in which the slices 60 are arranged. The slice area recognition function 153 also estimates the direction perpendicular to the direction A in which the slices 60 are arranged as the longitudinal direction of the slices 60. In addition, if the supplementary information of the tissue image 61 includes information regarding the arrangement of the slices 60, the slice area recognition function 153 may estimate the direction A in which the slices 60 are arranged based on that information.

The slice area recognition function 153 estimates the possible range in which each of the slices 60 may be located based on the centers of gravity 65a-65f of the tissue areas 62h, 62j-62l, which are representative tissue areas, and the angle of inclination of the direction A in which the slices 60 are aligned.

9 is a diagram showing an example of a recognition method of the slice region 8b based on the slice arrangement rule (3) according to the first embodiment. As shown in FIG. 9, when the direction A is the height direction of the tissue region 62h, the length of the tissue region 62h in the direction A at the center of gravity 65f is set as the height h1 of the tissue region 62h. The slice region recognition function 153 estimates the range in which the tissue image 61 originating from the same slice 60 as the tissue region 62h is included, based on the height h1. For example, the slice region recognition function 153 searches for a tissue region 62 at least partially included between two straight lines L1 and L2 that intersect perpendicularly with the direction A. The distance between the two straight lines L1 and L2 is equal to the height h1. In the example shown in FIG. 9, the tissue regions 62g and 62i are included between the two straight lines L1 and L2, so the slice region recognition function 153 recognizes that the tissue regions 62g to 62i are included in one slice region 8b.

In FIG. 9, the distance between the two straight lines L1 and L2 is equal to the height h1, but this is not limited to this. For example, the segment area recognition function 153 may set the distance between the two straight lines L1 and L2 to "n% of the height h1". The value of n may be determined in advance and stored in the memory circuitry 120.

Here, a supplementary explanation is given regarding the slice placement rule (3). FIG. 10 is a diagram showing an example of the placement of slices 60 that is not permitted by the slice placement rule (3) in the first embodiment. When the slice placement rule (3) is applied, the shape of each slice 60 is maintained while it is placed on the slide glass 7, so that, as shown in FIG. 10, one slice 60x is not cut into multiple pieces 60x1, 60x2 and placed in the forward direction. Therefore, when the slice placement rule (3) is applied, it is possible to estimate the placement direction of the slice 60 based on the direction in which the centers of gravity 65 are arranged.

The slice area recognition function 153 associates an identifier with each slice area 8 generated by grouping the tissue areas 62 contained in the slice area 8, and stores the identifier in the memory circuitry 120. The identifier is, for example, a numeric number. The identifier may also be assigned in association with the order of the cut-out positions at which the slices 60 were cut from the specimen 5. The identifier may also be set by the meta information setting function 158 described below.

Returning to FIG. 1, the display image generation function 154 generates a display image based on the slice area 8 recognized by the slice area recognition function 153. The display image is an image for highlighting the slice area 8, and is, for example, a mask image that covers a display slice area, which is an extension of one slice area 8, transparently and covers the other image areas with semi-transparent black. The display slice area is an image area that follows the contour of the slice area 8 and is slightly larger than the slice area 8. The reason that the display slice area is larger than the slice area 8 is to make it easier to see the boundary of the slice area 8 to be displayed when the tissue image 61 is covered by the mask image.

The display image generation function 154 generates a mask image for each individual slice region 8. Note that the display image is not limited to a mask image, and may be an image that depicts the outline of the display slice region with a dotted line, etc.

The display control function 155 causes the tissue image 61 to be displayed on the display 140.

Figure 11 is a diagram showing an example of the image viewer 14 according to the first embodiment. The image viewer 14 is a GUI that allows the user to perform operations such as viewing, editing, and processing of the tissue image 61 and the specimen image 51. In the example shown in Figure 11, the image viewer 14 displayed on the display 140 includes a micro image display area 141, a macro image display area 142, an operation tool display area 144, and an image list display area 143.

The micro image display area 141 displays a tissue image 61. The macro image display area 142 displays a specimen image 51. The operation tool display area 144 displays various tools that can be operated by the user. The image list display area 143 displays reduced versions of the tissue image 61 and specimen image 51 that can be displayed in the micro image display area 141 or the macro image display area 142.

In addition, in FIG. 11, for the purpose of explanation, dashed lines are shown indicating the slice regions 8n to 8r, but in reality, the dashed lines do not have to be displayed on the image viewer 14.

The display control function 155 also displays the tissue region 62 constituting one slice region 8, among the multiple tissue regions 62n-62s contained in the tissue image 61, in a manner different from the tissue regions 62 constituting the other slice regions 8. For example, when the user selects one of the multiple slice regions 8n-8r contained in the tissue image 61 displayed in the micro image display area 141, the display control function 155 highlights the tissue region 62 contained in the selected slice region 8.

When the user operates the micro image display area 141 of the display 140 by a reception function 156 described later, such as clicking a mouse, the display control function 155 determines that the slice region 8 corresponding to the clicked coordinates has been selected. In this case, the display control function 155 displays the tissue region 62 included in the slice region 8 selected by the user's operation in a manner different from the tissue regions 62 included in the other slice regions 8. For example, the display control function 155 displays on the display 140 a mask image generated by the display image generation function 154, in which only the display slice region including the selected slice region 8 is transparent and the other image regions are masked with semi-transparent black, superimposed on the tissue image 61. With this display, only the selected slice region 8 is highlighted. Note that the manner of highlighting is not limited to this.

The means by which the user selects the slice region 8 is not limited to clicking on the micro image display area 141. For example, FIG. 12 is a diagram showing an example of a method for selecting the slice region 8 using the list box 91 according to the first embodiment.

In the example shown in FIG. 12, the user selects the identifier of the desired slice region 8 from the drop-down menu of the list box 91. In FIG. 12, the display control function 155 displays a mask image 81p, in which the image region excluding the tissue region 62q and its surroundings is masked, superimposed on the tissue image 61 in order to highlight the tissue region 62q included in the slice region 8 corresponding to the selected identifier. This causes the tissue region 62q included in the selected slice region 8 to be highlighted.

The method of highlighting the tissue region 62q included in the selected slice region 8 is not limited to highlighting. For example, the display control function 155 may surround the tissue region 62q included in the selected slice region 8 with a frame such as a dashed or dotted line. The display control function 155 may also hide the tissue regions 62n-62p, 62r, and 62s included in the unselected slice region 8 and display only the tissue region 62q included in the selected slice region 8. The display control function 155 may also display the tissue regions 62n-62p, 62r, and 62s included in the unselected slice region 8 in a lighter color than the tissue region 62q included in the selected slice region 8.

The method of selecting the slice region 8 is not limited to the user clicking the mouse or specifying the slice region 8 using the list box 91. For example, the slice region 8 in which an abnormality such as a lesion has been detected by the image processing function 157 described below may be selected.

In the example shown in FIG. 12, the identifier of each section area 8 is associated with the number of the cut-out position where the section 60 was cut out from the specimen 5. Since the cut-out position number is displayed on the specimen image 51 displayed in the macro image display area 142, the user can select the identifier of the section area 8 by referring to the number displayed on the specimen image 51. The display control function 155 may also display the identifier of the section area 8 on the tissue image 61.

The display control function 155 may also change the display mode of the cut-out position on the sample image 51 that corresponds to the selected slice area 8. For example, the display control function 155 may highlight the cut-out position on the sample image 51 that corresponds to the selected slice area 8.

Figure 13 is a diagram showing an example of a case where the selected slice area 8 has changed from that shown in Figure 12. In Figure 13, the selected identifier has changed from "3" to "4." When the selection of slice area 8 has been changed, the display control function 155 removes the highlighting of the slice area 8 before the change and highlights the newly selected slice area 8.

In addition, the display control function 155 may display various metadata set by the meta information setting function 158 described below on the tissue image 61.

Returning to FIG. 1, the reception function 156 receives various operations by the user via the input interface 130. For example, when the tissue image 61 includes multiple slice regions 8, the reception function 156 receives a user operation to select one of the slice regions 8. As described in FIGS. 11 to 13, the selection method is a mouse click operation, a selection operation using the list box 91, etc.

The reception function 156 also receives input of annotations and other information by the user. Annotations are the addition of annotations or other information in the form of diagrams or text to the tissue image 61. Since the state of lesions and the like differs for each section 60 even within a single specimen 5, it is desirable to associate annotations and other information with each section region 8 unit rather than with each tissue image 61 unit. When the reception function 156 receives input of annotations and other information by the user, it sends the received content to the meta information setting function 158.

The reception function 156 also receives an operation by the user to modify the slice area 8. When the reception function 156 receives an operation by the user to modify the slice area 8, it sends the received content to the modification function 159.

The image processing function 157 performs an analysis by image processing for each slice region 8. Specifically, the image processing function 157 analyzes the degree of abnormality, the presence or absence of abnormality, the range of the abnormal region, etc., for each slice region 8. An abnormality is, for example, a lesion depicted in the tissue region 62 included in the slice region 8. The site where the lesion is depicted is, for example, a tumor region such as an adenoma or adenocarcinoma. The image processing method for analyzing the abnormality is not particularly limited, and any known method can be adopted. The image processing function 157 may also analyze the degree of abnormality, the presence or absence of abnormality, the abnormal region, etc., using a trained model such as deep learning.

The image processing function 157 may also analyze the area, major axis, and minor axis of each slice region 8. The image processing function 157 may also analyze the average, most frequent, and median pixel values of the pixels of each slice region 8.

The function of the image processing function 157 may be executed by another information processing device connected to the information processing device 100. In this case, the acquisition function 151 acquires the analysis results from the other information processing device.

When the tissue image 61 includes multiple slice regions 8, the meta information setting function 158 sets meta information for each slice region 8.

In this embodiment, the meta-information is information related to the section area 8, and includes, for example, an identifier capable of identifying the position where the section 60 corresponding to the section area 8 was cut out from the specimen 5. As shown in Figures 12 and 13, the identifier associates the section area 8 in the tissue image 61 with the cut-out position in the specimen image 51. The meta-information setting function 158 associates, as an identifier, a number corresponding to the number of each cut-out position with each section area 8, based on, for example, the order of the cut-out positions included in the specimen image 51 and the order of the section areas 8 in the tissue image 61.

The meta information may also include the analysis results of image processing performed by the image processing function 157 on the slice region 8 to which the meta information is to be added. The meta information may also include the doctor's diagnosis results on the slice region 8 to which the meta information is to be added. The doctor's diagnosis results include the presence or absence of abnormality, the degree of abnormality, the abnormal region, candidate diagnosis names, the basis for the diagnosis, and other comments. The doctor's diagnosis results on the slice region 8 may be acquired by the acquisition function 151 acquiring a report or the like registered in an external information processing device, or the reception function 156 accepting input of information related to the diagnosis by the user. Such information is also referred to as annotation. The user's operation of inputting such information may also be referred to as annotation.

FIG. 14 is a diagram showing an example of annotation setting according to the first embodiment. In the example shown in FIG. 14, the slice region 8 including the tissue region 62r is selected as a display target. Also, in FIG. 14, the reception function 156 is assumed to have received that the user has drawn region designation images 94a and 94b indicating the region of interest on the tissue image 61 using the drawing tool 92. In this case, the meta information setting function 158 sets the region of interest display images 93a and 93b at positions corresponding to the region designation images 94a and 94b in the tissue region 62r included in the selected slice region 8. Note that the meta information setting function 158 may set the region of interest display images 93a and 93b outside the contour of the tissue region 62r so that the region of interest display images 93a and 93b do not cover the tissue region 62r. The display control function 155 displays the region of interest display images 93a and 93b based on the setting by the meta information setting function 158. The region of interest display images 93a and 93b are included in the meta information.

Figure 15 is a diagram showing an example of a case where the selected slice region has changed from that shown in Figure 14. In Figure 15, slice region 8 including tissue region 62q is selected as the display target. In this case, region designation images 94a, 94b input in Figure 14 for setting annotations for slice region 8 including tissue region 62r are not displayed. Then, the reception function 156 receives a user operation to set a region designation image 94c for slice region 8 including currently selected tissue region 62q. Furthermore, the meta information setting function 158 sets a region of interest display image 93c at a position corresponding to the region designation image 94c in tissue region 62q included in the selected slice region 8.

The meta information setting function 158 may also associate information such as analysis results or diagnosis results associated with the slice region 8 as meta information with a corresponding cut-out position on the sample image 51. For example, the display control function 155 may display information such as analysis results or diagnosis results associated with the slice region 8 on the sample image 51 in association with a cut-out position corresponding to the slice region 8.

Figure 16 is a diagram showing an example of a state in which the selection of slice region 8 in the first embodiment has been cancelled. In the state shown in Figure 15, when the reception function 156 receives that the user has clicked on background region 70 with the mouse, the selection of slice region 8 is cancelled. In this case, the image processing function 157 cancels the display of the mask image. Note that the image processing function 157 continues to display region of interest display images 93a to 93c even when the selection has been cancelled.

Note that while linear annotations are shown as examples in Figures 14-16, the form of the displayed meta information is not limited to this. For example, text data or images may be displayed on the image viewer 14. The display control function 155 may display various types of meta information related to the selected slice region 8, for example, in a background region 70 near the slice region 8 selected by the user. Also, a specific area in which meta information is displayed may be provided on the image viewer 14.

Returning to FIG. 1, the correction function 159 corrects the slice area 8 recognized by the slice area recognition function 153 based on the user's operation accepted by the acceptance function 156. For example, if the user selects one or more tissue areas 62 on the image viewer 14, the correction function 159 may store the selected tissue areas 62 as one slice area 8 in the memory circuitry 120.

Next, we will explain the flow of the slice area recognition process executed by the information processing device 100 configured as described above.

Figure 17 is a flowchart showing an example of the flow of the segment area recognition process according to the first embodiment.

First, the acquisition function 151 acquires a sample image 51 from the sample image storage device 201 (S1).

Then, the tissue area recognition function 152 recognizes, through image processing, a plurality of tissue areas 62 in which the tissue specimen 6 is depicted from the tissue image 61 (S2).

Next, the slice area recognition function 153 groups the multiple tissue areas 62 according to a prescribed slice arrangement rule (S3). Note that the prescribed slice arrangement rule may be selected by the user when executing this flowchart, or may be determined in advance.

Then, the slice area recognition function 153 recognizes the tissue areas 62 in the same group as one slice area 8 (S4). The slice area recognition function 153 stores the association between the slice area 8 and the tissue area 62 in the memory circuitry 120.

The display image generating function 154 also generates a mask image 81 for highlighting each slice region 8 recognized by the slice region recognizing function 153 (S5). The display image generating function 154 stores the generated mask image 81 in the memory circuitry 120.

The meta information setting function 158 also associates a number corresponding to each cut-out position with each section area 8 as an identifier based on the order of the cut-out positions included in the sample image 51 and the order of the section areas 8 in the tissue image 61 (S6).

The image processing function 157 also performs image analysis for each slice region 8 to determine the degree of abnormality, the presence or absence of abnormality, the range of the abnormal region, the area, the major axis, the minor axis, the average pixel value, the most frequent pixel value, the median pixel value, and the like (S7). The meta-information setting function 158 associates the analysis results by the image processing function 157 with the slice region 8 as meta-information and stores them in the memory circuitry 120. At this point, the processing of this flowchart ends.

Next, we will explain the process flow for displaying a tissue image 61 in which a slice region 8 has been recognized.

Figure 18 is a flowchart showing an example of the flow of the segment area recognition process according to the first embodiment.

When the reception function 156 receives the user's selection of a tissue image to be displayed (S21 "Yes"), the display control function 155 causes the image viewer 14 to display the selected tissue image 61 (S22). Note that when the reception function 156 does not receive the selection of a tissue image to be displayed (S21 "No"), the process returns to S21.

If the reception function 156 receives a selection of a slice area 8 by the user (S23 "Yes"), the display control function 155 identifies the selected slice area 8 (S24) and highlights the selected slice area 8 (S25). Note that if the reception function 156 does not receive a selection of a slice area 8 (S23 "No"), the process returns to S23.

In addition, if meta information has already been set for the selected slice area 8, the display control function 155 displays the meta information in association with the selected slice area 8 (S26). Note that the user may be able to select whether to display or hide the meta information.

Furthermore, if the reception function 156 receives an operation to register or change meta information such as an annotation indicating an area of interest (S27 "Yes"), the meta information setting function 158 registers or changes the meta information in association with the selected slice area 8 (S28). The display control function 155 causes the image viewer 14 to display the newly registered or changed meta information. Note that if the reception function 156 does not receive an operation to register or change meta information (S27 "No"), the process returns to S27. At this point, the process of this flowchart ends.

In this manner, the information processing device 100 of this embodiment recognizes a plurality of tissue regions 62 in which the tissue specimen 6 is depicted from the tissue image 61, and recognizes one or more tissue regions 62 corresponding to one slice among the recognized plurality of tissue regions 62 as one slice region 8. Therefore, according to the information processing device 100 of this embodiment, the plurality of tissue regions 62 included in the tissue image 61 can be managed in units of slices 60.

For example, even for a single specimen 5, the condition of each section 60 is different, so it is desirable to associate meta-information such as annotations with each section 60 individually. However, as a typical comparative example, when a tissue image contains multiple tissue regions, it can be difficult to identify the group of tissue regions that make up a single section. In such an example, it can be difficult for the user to accurately specify the target region for which meta-information is to be set.

In contrast, according to the information processing device 100 of this embodiment, multiple tissue regions 62 included in the tissue image 61 are grouped by slice region 8 corresponding to one slice 60, making it possible to perform processing such as setting meta information for each individual slice 60.

The information processing device 100 of this embodiment also recognizes the slice area 8 based on the prescribed slice arrangement rules (1) to (3). Therefore, the information processing device 100 of this embodiment can recognize the slice area 8 with high accuracy in accordance with the rules of the facility where the tissue image 61 was captured.

Second Embodiment
In the above-described first embodiment, the slice area recognition function 153 recognizes the slice area 8 based on a prescribed slice arrangement rule. In this second embodiment, the slice area recognition function 153 recognizes the slice area 8 based on the periodicity of the image feature amount of the tissue area 62.

Figure 19 is a diagram showing an example of the recognition result of tissue regions 62t to 62y according to the second embodiment. In the example shown in Figure 19, the slice region recognition function 153 assigns a number to each of the tissue regions 62t to 62y included in the tissue image 61. The number is called a tissue region number.

Then, the slice area recognition function 153 analyzes the image features of the tissue areas 62t to 62y through image processing.

Figure 20 is a diagram showing an example of the analysis results of image features according to the second embodiment. In this embodiment, the image features are the area of tissue regions 62t to 62y, the average green color component, the green color standard deviation value, the position in the leftmost tissue image 61, and the position in the rightmost tissue image 61. Note that the indices representing the image features are not limited to these.

In the table of FIG. 20, the values listed in the "Area Area" column are the ratios of the individual tissue areas 62t-62y to the average area of the tissue areas 62t-62y.

The slice area recognition function 153 evaluates the similarity of the tissue areas 62t to 62y and the periodicity of the image features of the tissue areas 62t to 62y based on the image features of the tissue areas 62t to 62y shown in FIG. 20.

Figure 21 is a diagram showing an example of evaluation of the periodicity of image features according to the second embodiment. In Figure 21, the periodicity of the region area is evaluated. For example, the slice region recognition function 153 classifies the region area according to a predetermined threshold value. In the example shown in Figure 21, the threshold values are 25%, 50%, 150%, and 200%.

The slice area recognition function 153 determines whether there is periodicity in the evaluation of the area of the regions classified by the threshold value, and the boundary of the period. In FIG. 21, there is periodicity in the evaluation of the area of the regions for numbers 1 to 3 and numbers 4 to 6. For this reason, the slice area recognition function 153 recognizes the tissue areas 62t to 62v corresponding to numbers 1 to 3 as one group, and the tissue areas 62w to 62y corresponding to numbers 4 to 6 as another group. Then, as shown in FIG. 19, the slice area recognition function 153 recognizes the tissue areas 62t to 62v belonging to one group as slice area 8c, and the tissue areas 62w to 62y belonging to the other group as slice area 8d.

According to the method of this embodiment, even if the slice placement rules are not clearly defined or the slice placement rules at the time of imaging are unknown, it is possible to recognize the slice area 8.

In addition, an image feature detector that uses a trained model generated by a neural network may be used to evaluate the similarity and periodicity of the tissue regions 62t to 62y.

Third Embodiment
In the third embodiment, the slice region recognition function 153 recognizes the slice region 8 using a trained model.

FIG. 22 is a diagram showing an example of input/output of the trained model 90 according to the third embodiment. The slice area recognition function 153 inputs a tissue image 61 to the trained model 90, thereby grouping a plurality of tissue areas 62 in units of slice areas 8. Specifically, the slice area recognition function 153 inputs the tissue image 61 to the trained model 90, thereby obtaining a bounding box 82 and a mask image 81 for each slice area 8 output from the trained model 90. A known instance segmentation technique can be applied to generate the trained model 90.

The trained model 90 is, for example, a trained model generated by deep learning such as a neural network or other machine learning. As a deep learning method, a deep convolutional neural network (DCNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. can be applied, but are not limited to these. The trained model 90 is, for example, composed of a neural network and trained parameter data.

The trained model 90 is assumed to be stored in, for example, the memory circuitry 120. The slice area recognition function 153 reads out the trained model 90 from the memory circuitry 120 and inputs the tissue image 61. Alternatively, the trained model 90 may be incorporated in the slice area recognition function 153. Furthermore, the information processing device 100 may be provided with a learning function for generating the trained model 90, or may use a trained model 90 generated by another information processing device.

Figure 23 is a diagram showing an example of a bounding box 82 according to the third embodiment. The bounding boxes 82a and 82b are boxes that group one or more tissue regions 62 in units of slice regions 8. Within one bounding box 82a or 82b, slice regions 8f and 8g are detected as one instance. Furthermore, if part of a tissue region 62 belonging to another slice region 8 falls within one bounding box 82a or 82b, the trained model 90 is trained in advance so that the part of the tissue region 62 is not detected.

Figure 24 is a diagram showing an example of a mask image 81f according to the third embodiment. The trained model 90 outputs a mask image 81 for each slice region 8. The example shown in Figure 24 is a mask image 81f for highlighting the slice region 8f. In the mask image 81f, a display slice region 83f that is an extension of the slice region 8f is transparent, and the other image regions are covered with semi-transparent black. Note that in this embodiment, since the mask image 81 is generated by the trained model 90, the information processing device 100 does not need to be provided with a display image generation function 154.

According to the method of this embodiment, even if the slice placement rules are not clearly defined or the slice placement rules at the time of imaging are unknown, it is possible to recognize the slice area 8.

(Variation 1)
In the above-described first embodiment, an example has been described in which an annotation is set in the slice region 8, but the example in which meta information is displayed in association with the slice region 8 is not limited to this.

Figure 25 is a diagram showing an example of the display of the image viewer 14 according to the first modified example. In the example shown in Figure 25, the display control function 155 displays each lesion site 95 of the tissue region 62 contained in each slice region 8 in a manner different from other sites based on the lesion detection result for each slice region 8. The displayed lesion site 95 may be the analysis result by the image processing function 157 or the diagnosis result by a doctor.

(Variation 2)
In the above-described embodiments, the information processing device 100 is a server device or a PC, but is not limited thereto. For example, the information processing device 100 may be a computer built into an imaging device that captures the tissue image 61. The imaging device is, for example, a WSS, a digital camera, or a digital microscope, but is not limited thereto.

The various data discussed in this specification are typically digital data.

According to at least one of the embodiments described above, multiple tissue regions contained in a tissue image can be managed on a slice-by-slice basis.

Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and spirit of the invention.

Reference Signs List 5: specimen 6: tissue specimen 7: slide glass 8: 8a to 8g, 8n: slice area 14: image viewer 51: specimen image 60: 60a to 60e, 60x: slice 61: tissue image 62: 62a to 62y: tissue area 65: 65a to 65g: center of gravity 70: background area 81: 81f, 81p: mask image 82: 82a, 82b: bounding box 83f: slice area for display 90: trained model 91: list box 93a to 93c: region of interest display image 94a to 94c: area designation image 95: lesion site 100: information processing device 120: memory circuit 140: display 144: operation tool display area 150: processing circuit 151: acquisition function 152: tissue area recognition function 153: slice area recognition function 154 Display image generation function 155 Display control function 156 Reception function 157 Image processing function 158 Meta information setting function 159 Correction function S Information processing system

Claims (21)

an acquisition unit that acquires a tissue image depicting a tissue specimen including at least one slice;
a tissue region recognition unit that recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition unit that recognizes one or more tissue regions corresponding to one slice as one slice region among the plurality of tissue regions;
Equipped with
the section region recognition unit recognizes that, when the tissue specimen depicted in one tissue image includes multiple sections, the distance between multiple tissue regions constituting the same section is closer than the distance between multiple tissue regions constituting other sections, and based on the specified section arrangement rule, if the shortest distance between the contours of the multiple tissue regions is equal to or smaller than a threshold, or if multiple tissue regions that are relatively close to each other are identified based on the average or standard deviation of the shortest distances between all tissue regions included in the tissue image, the section region recognition unit recognizes that the multiple tissue regions are included in the same section region;
Information processing device. the tissue image includes a plurality of the tissue regions;
When one slice is divided into a plurality of tissue regions and depicted, the slice region recognition unit recognizes the plurality of tissue regions corresponding to the one slice as one slice region.
The information processing device according to claim 1 . an acquisition unit that acquires a tissue image depicting a tissue specimen including at least one slice;
a tissue region recognition unit that recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition unit that recognizes one or more tissue regions corresponding to one slice among the plurality of tissue regions as one slice region;
Equipped with
the segment area recognition unit recognizes the segment area based on a prescribed segment arrangement rule;
the specified slice arrangement rule is a rule that, when the tissue specimen depicted in one of the tissue images includes a plurality of slices, a plurality of tissue regions constituting one slice are arranged according to a shape of the slice when the slice is cut out from a specimen;
The slice region recognition unit estimates a direction in which the slices are arranged and a range in which each of the slices may be arranged based on areas and center positions of the tissue regions, and recognizes one or more tissue regions included in the estimated range as one of the slice regions.
Information processing device. an acquisition unit that acquires a tissue image depicting a tissue specimen including at least one slice;
a tissue region recognition unit that recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition unit that recognizes one or more tissue regions corresponding to one slice as one slice region among the plurality of tissue regions;
Equipped with
the slice region recognition unit groups the tissue regions based on periodicity of image feature amounts of the tissue regions, and recognizes one or more tissue regions included in one group as one slice region.
Information processing device. an acquisition unit that acquires a tissue image depicting a tissue specimen including at least one slice;
a tissue region recognition unit that recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition unit that recognizes one or more tissue regions corresponding to one slice as one slice region among the plurality of tissue regions;
Equipped with
The slice area recognition unit inputs the tissue image to a trained model to group the plurality of tissue areas in units of slice areas.
Information processing device. The slice area recognition unit inputs the tissue image to the trained model and obtains a recognition result of the tissue area output from the trained model.
The information processing device according to claim 5 . The trained model is a model in which a correspondence between a plurality of tissue images and the tissue regions in the plurality of tissue images is trained by a deep learning technique.
The information processing device according to claim 5 . The trained model is an image feature detector generated by a neural network that outputs an evaluation of the similarity of the tissue regions.
The information processing device according to claim 5 . The trained model is an image feature detector generated by a neural network that outputs an assessment of the periodicity of the tissue region.
The information processing device according to claim 5 . The slice area recognition unit inputs the tissue image to the trained model to obtain a bounding box output from the trained model and a mask image for each slice area,
The mask image is an image in which a display slice region, which is an extension of the slice region, of the tissue image is covered with transparency, and an image region other than the display slice region is covered with semi-transparent black.
The information processing device according to claim 5 . A display control unit that displays the tissue image on a display unit.
The information processing device according to claim 1 . the display control unit displays a tissue region constituting one slice region among the plurality of tissue regions included in the tissue image in a manner different from tissue regions constituting other slice regions;
The information processing device according to claim 11 . a reception unit that receives a user's operation to select one of the slice regions when the tissue image includes a plurality of slice regions;
the display control unit displays a tissue region included in the slice region selected by the user's operation in a manner different from tissue regions included in other slice regions.
The information processing device according to claim 12 . a meta-information setting unit for setting meta-information for each slice region when the tissue image includes a plurality of slice regions;
The information processing device according to claim 13 . the meta-information setting unit sets meta-information in a slice region selected by a user from among a plurality of slice regions included in the tissue image.
The information processing device according to claim 14 . The meta-information includes an analysis result of image processing on a slice region to which the meta-information is to be added.
16. The information processing device according to claim 14 or 15 . The meta-information includes a candidate diagnosis for the section region to which the meta-information is to be assigned.
The information processing device according to any one of claims 14 to 16 . The meta information includes information regarding a diagnosis entered by a user.
The information processing device according to any one of claims 14 to 17 . a correction unit that corrects the slice area recognized by the slice area recognition unit based on a user operation.
The information processing device according to claim 1 . An acquisition step in which an information processing device acquires a tissue image depicting a tissue specimen including at least one slice;
a tissue region recognition step in which the information processing device recognizes a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition step in which the information processing device recognizes one or more tissue regions corresponding to one slice among the plurality of tissue regions as one slice region;
Including,
In the section area recognition step, the information processing device recognizes that when the tissue specimen depicted in one tissue image includes multiple sections, the distance between multiple tissue areas constituting the same section is closer than the distance between multiple tissue areas constituting other sections, and if the shortest distance between the contours of the multiple tissue areas is equal to or smaller than a threshold, or if multiple tissue areas that are relatively close to each other are identified based on the average or standard deviation of the shortest distances between all tissue areas included in the tissue image, the information processing device recognizes that the multiple tissue areas are included in the same section area.
Information processing methods. acquiring a tissue image depicting the tissue specimen including at least one section;
a tissue region recognition step of recognizing a plurality of tissue regions in which the tissue specimen is depicted from the tissue image;
a slice region recognition step of recognizing one or more tissue regions corresponding to one slice among the plurality of tissue regions as one slice region;
Run the following on your computer :
In the section area recognition step, when the tissue specimen depicted in one tissue image includes multiple sections, the distance between multiple tissue areas constituting the same section is closer than the distance between multiple tissue areas constituting other sections based on a specified section arrangement rule. If the shortest distance between the contours of the multiple tissue areas is equal to or smaller than a threshold, or if multiple tissue areas that are relatively close to each other are identified based on the average or standard deviation of the shortest distances between all tissue areas included in the tissue image, the multiple tissue areas are recognized as being included in the same section area.
program .

Images