JP7121968B2 - Information display system, information display device, information display method and information display program - Google Patents

Description

The present invention relates to an information display system, an information display device, an information display method, and an information display program. The present invention relates to a display device, an information display method and an information display program.

Conventionally, as in Japanese Patent Laid-Open No. 2002-200010, convenience in operation and display has been achieved by providing a terminal device with a display unit that is equivalent to or larger than an actual display unit.

JP 2016-214496 A (paragraphs [0128] to [0129])

However, in many cases, medical devices are provided with mechanical operation switches, etc., and when the operation is performed using the operation switches, the technology disclosed in Patent Document 1 does not display operation information on the terminal device side. is limited.

In addition to electrical operations, there are cases where the flow progresses by means other than switch operations, such as patient introduction and positioning, and information on operations such as patient introduction is also required.

The present invention has been made in view of the circumstances described above, and provides an information display system, an information display device, an information display method, and an information display method capable of further improving the convenience of medical devices and reducing the risk of erroneous operation. An object of the present invention is to provide a display program.

In order to solve the above-described problems, an information display system includes a medical device having an operation unit, and an information display device that displays device-related information related to the medical device, wherein the operation unit includes the medical device. and a state information display area for displaying state information of the device, wherein the information display device includes a photographing unit, a display unit, and a control unit, and the control unit controls the photographing by the photographing unit. extracting the button of the operation unit and the state information display area appearing in the image obtained by recognizing the state information displayed in the state information display area ; is displayed on the display unit in association with the

INDUSTRIAL APPLICABILITY The present invention can further improve the convenience of medical devices and reduce the risk of erroneous operation.

1 is an external perspective view of an information display system according to an embodiment of the present invention; FIG. FIG. 3 is an external view of an operating portion of the medical device; 1 is a block diagram of an information display device; FIG. 4 is a data configuration diagram of a display information DB; FIG. It is a flow chart which shows operation of an information display system concerning an embodiment of the present invention. FIG. 10 is a diagram showing a display example of device-related information in a scanner reset process; FIG. 10 is a diagram showing a display example of device-related information in the imaging condition setting process; FIG. 10 is a diagram showing an example display of device-related information in a patient positioning process; FIG. 10 is a diagram showing a display example of device-related information in the imaging preparation process; FIG. 10 is a diagram showing a display example of apparatus-related information in an X-ray irradiation process; FIG. 10 is a diagram showing a display example of device-related information in the imaging end process;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each drawing is merely a schematic representation to the extent that the present invention can be fully understood. Accordingly, the present invention is not limited to the illustrated examples only. In addition, in each figure, the same code|symbol is attached|subjected about a common component and a similar component, and those overlapping description is abbreviate|omitted.

An information display system 1 according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 11. FIG. "Up and down", "left and right", and "back and forth" in the description of the information display system 1 follow the arrows in FIG. The direction is defined for convenience of explanation and does not limit the present invention.

The information display system 1 uses augmented reality (AR) technology to display information related to the medical device 2 (hereinafter referred to as “device related information”) to the operator. Display of device-related information to the operator is performed via the information display device 3 .

The device-related information may be any information related to the medical device 2, and includes, for example, descriptions of parts constituting the medical device 2, descriptions of operation methods, maintenance information, failure information, and the like. The site may be a partial area of the medical device 2, and may be an area distinguished by function or shape, for example. The maintenance information is information about maintenance (for example, maintenance method, etc.). The failure information is information about failure (for example, repair method, etc.).

<Overall composition>
The information display system 1 includes, for example, a medical device 2 used in a medical field and an information display device 3 that displays device-related information of the medical device 2, as shown in FIG.

<Medical device>
The medical device 2 is not particularly limited in type as long as it is a device used for medical treatment. The medical device 2 is, for example, a dental unit, an image diagnostic device, or the like. Here, the medical device 2 is assumed to be an X-ray CT (computer tomography) device. The medical device 2 here includes a body portion 21 , an elevating portion 22 , a scanner 23 and an operation portion 24 .

The medical device 2 may be connected to a management PC (Personal Computer) in which patient information is registered. In that case, the management PC stores the CT image captured by the medical device 2 in association with the patient's information.

The main body part 21 is a part that becomes the main body of the medical device 2 . The main body 21 includes a base 21a, a pillar 21b vertically provided at the rear of the base 21a, a patient chair 21c provided at the front of the base 21a, and a head fixing mechanism for fixing the patient's head. 21d. In addition, illustration of a patient is omitted in FIG.

The head fixing mechanism 21d has a structure in which sliding motion and rotational motion are combined, and is configured as a link mechanism having links 21da, 21db, and 21dc. A patient fixture 21dd for fixing the patient's head is provided at the tip of the link 21dc.

The elevating section 22 is a component that elevates the column section 21b. The elevating section 22 includes a scanner support section 22a that supports the scanner 23, and a positioning section 22b that positions the scanner 23 with the oral cavity of the patient.

A rotation mechanism for rotating the scanner 23 is accommodated inside the scanner support portion 22a, and rotates the scanner 23 with the vertical direction as a rotation axis.

The positioning portion 22b is a portion that irradiates an alignment beam (for example, a laser beam) forward. The beam emitted from the positioning unit 22b is parallel to the plane (horizontal plane) formed by the front/rear and left/right line segments, and is used for alignment of the scanner 23 in the vertical direction.

The scanner 23 is a component that takes an image of the patient's oral cavity using X-rays. The scanner 23 is capable of photographing while rotating, and by combining the image data photographed while rotating, it becomes three-dimensional volume data. By operating the operation unit 24, it is possible to set the imaging mode of the scanner 23 and imaging conditions (tube voltage, tube current, number of seconds for imaging, etc.).

The imaging modes of the medical device 2 are classified according to, for example, the type of data to be created and the imaging range. Here, there are four shooting modes: "oral mode", "dental mode", "standard panorama mode" and "pediatric panorama mode". The “oral mode” is an imaging mode for creating three-dimensional data of the entire oral cavity. "Dental mode" is an operation mode for creating local three-dimensional data in the oral cavity. The "standard panorama mode" is a photographing mode for creating two-dimensional data of the mouth. "Children's panorama mode" is an imaging mode that creates two-dimensional data of a narrower range than "standard panorama mode".

The operation unit 24 is a component operated by an operator, and is installed on the left side surface of the lifting unit 22 here. For example, as shown in FIG. 2, the operation unit 24 includes a reset button 24a, an imaging mode selection button 24b, up, down, left, and right direction buttons 24c, an X-ray irradiation button 24d, and a menu button 24e. . The operation unit 24 also includes, for example, an imaging condition display area 24f, a state information display area 24g, and an imaging mode display area 24h. Note that the configuration of the operation unit 24 shown in FIG. 2 is merely an example, and the layout and shape of buttons and the like may be arbitrary.

The reset button 24a is pressed, for example, when moving the scanner 23 to a specific position. The reset button 24a has a circular shape, and the letter "R" is displayed on the surface of the reset button 24a.

The shooting mode selection button 24b is pressed, for example, when the shooting mode is selected. The photographing mode selection button 24b has a circular shape, and the letter "M" is displayed on the surface of the photographing mode selection button 24b.

The up, down, left, and right direction buttons 24c are used, for example, to set imaging conditions (tube voltage, tube current, number of seconds for imaging, etc.) and to align the scanner 23. FIG. The direction button 24c has a substantially triangular shape (a shape in which one vertex of a triangle is rounded), and black triangles indicating respective directions are displayed on the direction buttons 24c.

The X-ray irradiation button 24d is pressed, for example, when X-rays are irradiated. The X-ray irradiation button 24d has a circular shape, and the character "B" is displayed on the X-ray irradiation button 24d.

The menu button 24e is used, for example, to determine the selected shooting mode or the set shooting conditions, or to make various settings other than when shooting. The menu button 24e has a rectangular shape.

The imaging condition display area 24f is a display, for example, a display composed of a plurality of segments (here, a 7-segment display is assumed). In the imaging condition display area 24f, the imaging conditions such as tube voltage (kV), tube current (mA), and imaging seconds (sec) are displayed.

The status information display area 24g is a display, for example, a display composed of a plurality of segments (here, a 7-segment display is assumed). State information of the medical device 2 is displayed in the state information display area 24g.

Here, the "state information" may be information indicating various states of the medical device 2. For example, information indicating operation steps during X-ray imaging (hereinafter referred to as "operation step information"), failure (hereinafter referred to as "failure identification information"), information for identifying the content of maintenance (hereinafter referred to as "maintenance identification information"), and the like.

The operation step information is represented by "letter 'S'+numeral" such as 'S1' and 'S2'. The failure identification information is represented by "letter 'E' + number" such as 'E1' and 'E2'. The maintenance identification information is represented by "letter 'M' + number" such as 'M1' and 'M2'.

The shooting mode display area 24h displays the selected shooting mode, and is composed of, for example, the name of the shooting mode and a lamp. When a shooting mode is selected, the corresponding shooting mode lamp lights up.

<Information display device>
The information display device 3 displays device-related information of the medical device 2 . The information display device 3 is not particularly limited in type as long as it can display device-related information using augmented reality technology. The information display device 3 is, for example, a head-mounted display (HMD) worn on the head, a mobile terminal operated by holding in hand, or the like. Here, the information display device 3 is assumed to be a head-mounted display of a method (video see-through method) in which device-related information is combined with an image at a signal level using video capture.

The information display device 3 includes, for example, an imaging unit 31, a display unit 32, a storage unit 34, and a control unit 35, as shown in FIG. Note that the control unit 35 is an example of a "control unit" in the scope of claims.

The imaging unit 31 is a camera including, for example, a CMOS (Complementary metal-oxide-semiconductor) image sensor or a CCD (Charged-coupled devices) image sensor. The photographing unit 31 is installed so as to photograph a video (including an image, the same shall apply hereinafter) in the line-of-sight direction of the operator while being worn by the operator.

The display unit 32 is, for example, a liquid crystal display or an organic EL display. The display unit 32 is arranged so as to block the field of view of the operator when worn by the operator. The display unit 32 displays an image captured by the image capturing unit 31 . In addition, device-related information corresponding to the state of the medical device 2 is displayed on the display unit 32 in association with the site of the medical device 2 . Details of the display of device-related information will be described later.

The storage unit 34 is, for example, a memory, and stores information necessary for displaying device-related information. The storage unit 34 here stores a part information DB (Data Base) 34a, a state information DB 34b, and an additional information DB 34c. Part or all of this information may be stored in a device other than the information display device 3, and the information display device 3 may acquire the information via the network as necessary. In that case, the information display device 3 includes, for example, a communication unit (for example, a wireless module) that realizes communication with other devices.

The part information DB 34a stores information used for recognition processing of parts (for example, various buttons and display areas of the operation unit 24) displayed in the captured image. The part information DB 34a, for example, associates the three-dimensional coordinates in the world coordinate system of the feature points of each part with the feature amounts. Here, the world coordinate system is a coordinate system whose origin is an arbitrary position in the physical space where the medical device 2 is installed. On the other hand, there is a camera coordinate system with respect to the world coordinate system, and the camera coordinate system is a coordinate system whose origin is the position of the photographing unit 31 . In the camera coordinate system, for example, the optical axis direction of the photographing unit 31 is the Z axis.

Creation of the part information DB 34a is performed as a pre-process for taking a CT image using the medical device 2. FIG. Preprocessing for creating the part information DB 34a is performed, for example, by the following processing. First, a group of images obtained by photographing a part from a plurality of directions is prepared, feature points of the part are detected from each image, and feature amounts of the feature points are calculated. The method for detecting feature points and the method for calculating feature amounts are not particularly limited. Next, the three-dimensional coordinates in the world coordinate system of the feature points detected by "Structure from Motion" or stereo vision are calculated, and the calculated three-dimensional coordinates are associated with the feature values to create the part information DB 34a. do.

The state information DB 34b stores information used for recognition processing of the state information displayed in the captured image (here, characters such as "S1" and "S2" displayed in the state information display area 24g). . The state information DB 34b, for example, associates three-dimensional coordinates in the world coordinate system of feature points of state information with feature amounts. The creation of the state information DB 34b is performed as a pre-process for taking a CT image using the medical device 2, like the part information DB 34a.

The additional information DB 34c stores information to be added to the captured video. For example, as shown in FIG. 4, the additional information DB 34c includes items of operation process information (state information), display target part information, and operation explanation information (apparatus-related information).

The operation process information is a kind of status information, and here, "scanner reset process (S1)", "imaging condition setting process (S2)", "patient positioning process (S3)", and "imaging preparation process (s4)". , “X-ray irradiation step (S5)” and “imaging end step (S6)” are exemplified.

The display target region information is information indicating a region to which device-related information is added. )”, “Positioning part”, “Patient fixture”, and “X-ray irradiation button” are exemplified. The display target part information is, for example, the coordinates of the representative point of the part in the world coordinate system.

The operation explanation information is a type of device-related information. It is done.” and so on. Here, information displayed on the operation unit 24, for example, is entered in the "○○" portion of the operation explanation information.

The control unit 35 is, for example, a CPU (Central Processing Unit), and implements a part recognition unit 35a, a camera position/orientation calculation unit 35b, a state recognition unit 35c, and a device-related information addition unit 35d by program execution processing. do. Programs for realizing these functions are stored in the storage unit 34, for example. Note that these programs may be stored in a device other than the information display device 3, and the information display device 3 may obtain the programs via a network as necessary.

The site recognition unit 35a recognizes the site appearing in the captured image. For example, the part recognition unit 35a detects feature points and calculates feature amounts from the captured image, and performs feature amount matching between the calculated feature amounts and the feature amounts stored in the part information DB 34a. The feature points detected by the part recognition unit 35a include the shape of the part and the characters and images displayed on the part. Various existing methods can be used for detection of feature points, calculation of feature amounts, and feature amount matching.

The camera position/orientation calculation unit 35b calculates the position and orientation of the photographing unit 31 in the world coordinate system. The camera position/orientation calculation unit 35b calculates, for example, the two-dimensional coordinates in the image coordinate system of the feature points detected from the captured image and the feature points of the parts stored in the part information DB 34a based on the result of the feature amount matching by the part recognition unit 35a. The position and orientation of the imaging unit 31 in the world coordinate system are calculated by associating them with the three-dimensional coordinates in the world coordinate system. Here, the image coordinate system is a coordinate system on the plane of the photographed image. The position and orientation of the imaging unit 31 in the world coordinate system are represented, for example, by a rotation matrix representing the difference in orientation between the world coordinate system and the camera coordinate system and a translation vector representing the difference in position. Various existing methods can be used to calculate the position and orientation of the imaging unit 31 in the world coordinate system.

The state recognition unit 35c recognizes state information appearing in the captured image. The state recognition unit 35c recognizes the state information by, for example, processing similar to the recognition of the part by the part recognition unit 35a. That is, the state recognition unit 35c detects feature points and calculates feature amounts from the captured image, and performs feature amount matching between the calculated feature amounts and the feature amounts stored in the state information DB 34b.

Also, the state recognition unit 35c may recognize state information from a captured image by a technique other than recognition using feature points. The state recognition unit 35c specifies, for example, the range of the state information display area 24g in the photographed image from the result of feature amount matching by the part recognition unit 35a, and the world coordinate system of the photographing unit 31 calculated by the camera position/orientation calculation unit 35b. , the image of the state information display area 24g is corrected to the state viewed from the front. Then, character recognition may be performed on the corrected image of the status information display area 24g. The character recognition method is not particularly limited, and for example, a method using AI (Artificial Intelligence) may be used.

The device-related information addition unit 35d adds device-related information to the captured image and displays it on the display unit 32. FIG. The device-related information addition unit 35d refers to the additional information DB 34c using, for example, the state information recognized by the state recognition unit 35c as a key, and adds predetermined operation explanation information to the position of the display target part information corresponding to the corresponding state information. Add For example, in the scanner resetting process, the device-related information adding unit 35d adds a message "Press the reset button to set the scanner to the initial position" at the position of the reset button 24a shown in the image.

When adding the device-related information to the captured video, the device-related information may be superimposed on the captured video in consideration of the position and orientation of the imaging unit 31 calculated by the camera position/orientation calculator 35b. . That is, the device-related information may be displayed as if it were actually drawn on the surface of the medical device 2 .

<Operation of information display system according to embodiment>
Next, the operation of the information display system 1 according to the embodiment of the present invention will be described in detail with reference to FIGS. 5 to 11 (see FIGS. 1 to 4 as appropriate).

<T1: Patient selection>
First, the operator of the medical device 2 uses a PC for management (not shown) to select a patient for whom a CT image is to be taken (T1). As a result, the CT image captured by the medical device 2 is stored in the management PC in association with the patient's information. Further, the medical device 2 displays “S1” in the state information display area 24g (see FIG. 2) of the operation unit 24 after the patient setting is performed.

<S1: Scanner Reset Process>
Next, the operator operates the operation unit 24 of the medical device 2 while wearing the information display device 3 . “S1” is displayed in the status information display area 24g (see FIG. 2) of the operation unit 24. FIG. The imaging unit 31 of the information display device 3 captures an image of the operation unit 24 in the line-of-sight direction of the operator, and the part recognition unit 35a of the control unit 35 is included in the operation unit 24 and the operation unit 24 of the captured image. Recognize various buttons and display areas. Also, the state recognition unit 35c recognizes the state information displayed in the state information display area 24g. Here, it is recognized that the state information displayed in the state information display area 24g is "S1" (that is, the operation process of the medical device 2 is the "scanner reset process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and associates the device-related information with the part appearing in the captured image. Here, the device-related information addition unit 35d adds, for example, the text "Press the reset button and set the scanner to the initial position." to the reset button 24a appearing in the captured image. As a result, the operator sees the image shown in FIG. 6, for example, when operating the operation unit 24 in "S1". Then, the operator presses the reset button 24a according to the wording indicated by reference numeral 41, and the medical device 2 moves the scanner 23 (see FIG. 1) to the initial position. Further, after moving the scanner 23 to the initial position, the medical device 2 displays "S2" in the state information display area 24g (see FIG. 2) of the operation unit 24. FIG.

<S2: Imaging condition setting process>
Next, the operator operates the operating section 24 of the medical device 2 again. "S2" is displayed in the state information display area 24g (see FIG. 2) of the operation unit 24. FIG. The photographing unit 31 of the information display device 3 photographs an image of the operation unit 24 in the direction of the line of sight of the operator, and the state recognition unit 35c of the control unit 35 detects the state information by the same processing as the scanner reset step (S1). It recognizes that the status information displayed in the display area 24g is "S2" (that is, the operation process of the medical device 2 is the "imaging condition setting process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and puts, for example, "Please select an imaging mode" on the imaging mode selection button 24b, which is the part to be displayed here. The shooting mode is set to "Aural mode". ” is added. Note that the method of recognizing the shooting condition display area 24f (see FIG. 2) of the operation unit 24 is not particularly limited. The photographing conditions (initial values at first) displayed in the display area 24f are recognized.

Further, the device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and sets the direction button 24c as the shooting condition setting button, for example, "Please set the shooting conditions. The shooting conditions are "60kV", "8.0mA" and "20.0sec". After setting the imaging conditions, introduce the patient. ” is added. Note that the method of recognizing the shooting mode display area 24h (see FIG. 2) of the operation unit 24 is not particularly limited. recognizing the lamp of the currently selected shooting mode (initially a predetermined shooting mode).

As a result, the operator sees the image shown in FIG. 7, for example, when operating the operation unit 24 in "S2". The operator presses the photographing mode selection button 24b according to the words indicated by reference numeral 42 to select the desired photographing mode. Also, the operator operates the direction button 24c according to the words indicated by reference numeral 43 to set the desired photographing conditions. Then, the operator presses the menu button 24e to complete the imaging condition setting process. The medical device 2 displays “S3” in the state information display area 24g (see FIG. 2) of the operation unit 24 after receiving the completion of setting the imaging mode and imaging conditions.

<T2: Patient introduction>
Next, after completing the setting of the imaging mode and imaging conditions, the operator introduces the patient into the medical device 2 and makes the patient sit on the chair 21c (see FIG. 1).

<S3: Patient positioning step>
Next, the operator operates the operating section 24 of the medical device 2 again. "S3" is displayed in the state information display area 24g (see FIG. 2). The photographing unit 31 of the information display device 3 photographs an image of the operation unit 24 in the direction of the line of sight of the operator, and the state recognition unit 35c of the control unit 35 detects the state information by the same processing as the scanner reset step (S1). It recognizes that the status information displayed in the display area 24g is "S3" (that is, the operation process of the medical device 2 is the "patient positioning process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and sets the direction button 24c, which is the up/down button, which is the display target site here, to, for example, "Add height to the imaging site. Please match." is added. In addition, the device-related information adding section 35d adds, for example, the text "Please confirm whether the positioning is correct" to the positioning section 22b. In addition, the device-related information addition unit 35d adds, for example, a message such as "When patient positioning is completed, fix the patient." to the patient fixture 21dd.

As a result, the operator sees, for example, the image shown in FIG. The description of the patient fixture 21dd is omitted). The operator operates the direction button 24c according to the words indicated by reference numeral 44 to bring the scanner 23 to a desired height. Further, the operator confirms the position of the positioning portion 22b according to the wording indicated by reference numeral 45. FIG. Also, the operator fixes the patient according to the words displayed on the patient fixing tool 21dd. The operator then presses the menu button 24e to complete the patient positioning process. After receiving the completion of patient positioning, the medical device 2 displays "S4" in the state information display area 24g (see FIG. 2) of the operation unit 24. FIG.

<S4: Imaging Preparation Process>
Next, the operator operates the operating section 24 of the medical device 2 again. "S4" is displayed in the state information display area 24g (see FIG. 2). The photographing unit 31 of the information display device 3 photographs an image of the operation unit 24 in the direction of the line of sight of the operator, and the state recognition unit 35c of the control unit 35 detects the state information by the same processing as the scanner reset step (S1). It recognizes that the status information displayed in the display area 24g is "S4" (that is, the operation process of the medical device 2 is the "imaging preparation process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4) and sets the reset button 24a, which is the display target portion here, to, for example, "Press the reset button to move the scanner to the imaging start position. Please set to .” is added. As a result, when the operator operates the operation unit 24 in "S4", the operator sees the image shown in FIG. 9, for example. Then, the operator presses the reset button 24a in accordance with the message indicated by reference numeral 47, and the medical device 2 moves the scanner 23 to the imaging start position. Further, the medical device 2 displays “S5” in the state information display area 24g (see FIG. 2) of the operation unit 24 after moving the scanner 23 to the imaging start position.

<S5: X-ray irradiation step>
Next, the operator operates the operating section 24 of the medical device 2 again. "S5" is displayed in the state information display area 24g (see FIG. 2). The photographing unit 31 of the information display device 3 photographs an image of the operation unit 24 in the direction of the line of sight of the operator, and the state recognition unit 35c of the control unit 35 detects the state information by the same processing as the scanner reset step (S1). It recognizes that the status information displayed in the display area 24g is "S5" (that is, the operation process of the medical device 2 is "X-ray irradiation process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and sets the X-ray irradiation button 24d, which is the display target site here, to, for example, "Press the X-ray irradiation button to take an image." Do not release the button during X-ray irradiation.” is added. As a result, the operator sees the image shown in FIG. 10, for example, when operating the operation unit 24 in "S5". Then, the operator presses the X-ray irradiation button 24d according to the words indicated by reference numeral 48, and the medical apparatus 2 performs X-ray irradiation and imaging. Further, the medical device 2 displays "S6" in the state information display area 24g (see FIG. 2) of the operation unit 24 after the imaging is completed.

<S6: Shooting end process>
"S6" is displayed in the state information display area 24g (see FIG. 2). The photographing unit 31 of the information display device 3 photographs an image of the operation unit 24 in the direction of the line of sight of the operator, and the state recognition unit 35c of the control unit 35 detects the state information by the same processing as the scanner reset step (S1). It recognizes that the status information displayed in the display area 24g is "S6" (that is, the operation process of the medical device 2 is the "imaging end process").

The device-related information addition unit 35d of the control unit 35 refers to the additional information DB 34c (see FIG. 4), and displays, for example, "Imaging has been completed. Please add the words, "Please release the device and let the patient out." As a result, the operator sees, for example, the image shown in FIG. 11 in "S6".

<T3: Patient Exit>
Next, the operator releases the patient fixture 21dd from the patient and allows the patient to leave the medical device 2 . This completes the intraoral CT imaging using the medical device 2 .

The information display system 1 according to the embodiment of the present invention configured as described above has the following effects.
That is, the information display system 1 uses augmented reality technology to display device-related information (description of functions, operation procedures, etc.) in association with the part (for example, the operation unit 24) of the medical device 2 shown in the captured image. do. Therefore, the operator can obtain additional information to be added in addition to the information obtained from the actual site. Therefore, the information display system 1 can further improve the convenience of the medical device and reduce the risk of erroneous operation.

In particular, when the display area of the medical device 2 is limited due to its small size, etc., the operator can determine the functions of the operation unit 24 from limited information such as the shapes and names of the buttons, and operate them. You have to remember the steps. Also, if you forget the operation procedure, you have to refer to the manual. However, the information display system 1 does not require that.

Furthermore, in the information display system 1, there is no need to pre-register device-related information when manufacturing the medical device 2, and additional information that needs to be displayed can be registered in the information display device 3 later. . Therefore, it is easy to update the device-related information.

The information display system 1 also recognizes the state information displayed on the medical device 2 and displays the device-related information corresponding to the state information in association with the site of the medical device 2 . Therefore, since different information can be displayed for each state of the medical device 2, the convenience of the medical device 2 can be further improved.

Furthermore, in the information display system 1, since the information display device 3 only needs to recognize the status information displayed on the medical device 2, data (status information) is transmitted between the information display device 3 and the medical device 2 by wireless communication. ) are not sent or received. Therefore, in the information display system 1 , there is no possibility that radio waves of wireless communication will adversely affect the medical device 2 . In other words, the information display system 1 is particularly suitable for sites where the medical device 2, which is susceptible to radio waves of wireless communication, is used.

Further, the information display system 1 uses the operation process information as the state information, and switches and displays the explanation of the operation unit 24 of the medical device 2 according to the operation process. Therefore, the minimum necessary information can be displayed at the optimum position for each operation step, so that the risk of erroneous operation by the operator can be further reduced.

In addition, the information display system 1 uses maintenance identification information or failure identification information as status information, and displays information corresponding to this information on the site of the medical device 2 . Therefore, the operator can fully recognize that maintenance is required or that a failure has occurred.

[Modification]
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can be implemented with appropriate modifications.

The information display system 1 of the present embodiment, as shown in FIG. associated with device-related information. That is, the control unit 35 has a part recognition unit 35a, a camera position/orientation calculation unit 35b, a state recognition unit 35c, and a device-related information addition unit 35d. However, the configuration of the present invention is not limited to this. For example, the information display system 1 includes a medical device 2, an information display device 3, and a control device (not shown). 2 may be associated with device-related information. That is, the configuration may be such that a control device different from the information display device 3 has a part or all of the functions of the control unit 35 of the present embodiment. In this case, the control device receives, for example, an image captured by the imaging unit 31 from the information display device 3 and transmits to the information display device 3 an image in which device-related information is associated with the parts constituting the medical device 2 . Note that the control device is an example of "control means" in the scope of claims.

Further, in the present embodiment, the medical device 2 is described assuming an X-ray CT (Computer Tomography) device. Therefore, the display is switched for each operation step, but the part of the medical device 2 and the device-related information may be in one-to-one correspondence. In that case, the medical device 2 does not display the status information, and the information display device 3 simply recognizes the part of the medical device 2 appearing in the captured image, and displays the device-related information in association with the part. In other words, the information display device 3 may not recognize the status information.

Further, in the present embodiment, status information is displayed on the operation unit 24 of the medical device 2, and the status of the medical device 2 is recognized by capturing an image of the display with the imaging unit 31 of the information display device 3. FIG. However, the method of recognizing status information is not limited to this, and it is also possible to use other methods instead of or in addition to the above method. For example, when the medical device 2 outputs (informs) state information by sound and the information display device 3 detects the sound, the state recognition unit 35c of the information display device 3 analyzes the sound to obtain a medical device. The state of the device 2 may be recognized. According to such a configuration, as long as the operator is in the vicinity of the medical device 2 (range of sound), the information display device 3 can recognize the status information regardless of the direction of the operator's line of sight. Therefore, the information display device 3 can recognize the state information without omission. The method of recognizing status information by sound is particularly effective in cases where the operator's line of sight moves to various parts (for example, a large medical device).

Further, in the present embodiment, the body parts appearing in the captured image are recognized by detecting feature points and calculating feature amounts from the image captured by the body part recognition unit 35a. That is, in this embodiment, natural feature-based (markerless) recognition was performed. However, marker-based (eg, AR markers such as barcodes and QR codes) recognition may also be performed.

In that case, for example, AR markers are attached to respective parts of the medical device 2 , and the positions (coordinates) of the AR markers in the world coordinate system are registered in advance in the information display device 3 . Then, the part recognition unit 35a of the information display device 3 recognizes the AR marker from the image captured by the image capturing unit 31, and the device-related information adding unit 35d adds the device-related information to the AR marker of the captured image and displays it. Displayed in section 32 .

Alternatively, for example, an AR marker is attached to a reference portion of the medical device 2, and the positions of other portions of the medical device 2 are registered in the information display device 3 in advance as coordinates based on the AR marker. Keep Then, the part recognition unit 35a of the information display device 3 recognizes the AR marker (reference point) from the image captured by the image capturing unit 31, and the device-related information addition unit 35d registers the AR marker of the captured image. Apparatus-related information is added to the position and displayed on the display unit 32 . At this time, the part recognition unit 35a calculates the size and distortion of the AR marker, and specifies the distance and direction from the imaging unit 31 to the AR marker.

Also, in the present embodiment, a video see-through type head-mounted display is assumed as the information display device 3, but the type of head-mounted display is not limited to this. For example, it may be an optical see-through system in which device-related information is superimposed on a transparent plate-like component (combiner). Further, it may be of a retinal projection type that scans a weak laser beam on the retina of the operator.

When the information display device 3 is one of these types, the device-related information adding unit 35d calculates, for example, the position of the part of the medical device 2 from the operator's line of sight from the image captured by the imaging unit 31, and displays the position. By displaying only the device-related information on the portion 32 (the transparent plate-like part or the retina), the operator feels as if the device-related information is associated with the part of the medical device 2 .

REFERENCE SIGNS LIST 1 information display system 2 medical device 21 main body 22 lifting section 22b positioning section (site)
23 scanner 24 operation unit 24a reset button (part)
24b Shooting mode selection button (site)
24c direction button (site)
24d X-ray irradiation button (site)
24e Menu button 24f Imaging condition display area 24g State information display area 24h Imaging mode display area 3 Information display device 31 Imaging unit 32 Display unit 34 Storage unit 34a Part information DB
34b State information DB
34c Additional information DB
35 control unit (control means)
35a part recognition unit 35b camera position/orientation calculation unit 35c state recognition unit 35d device-related information addition unit

Claims (8)

An information display system comprising: a medical device having an operation unit; and an information display device displaying device-related information related to the medical device,
The operation unit includes a state information display area for displaying state information of the medical device,
The information display device includes an imaging unit, a display unit, and a control unit,
The control unit extracts the buttons of the operation unit and the state information display area appearing in the image captured by the imaging unit , recognizes the state information displayed in the state information display area, and extracts the state information. displaying the device-related information corresponding to the button on the display unit in association with the button;
An information display system characterized by: The medical device comprises a notification unit that outputs status information of the medical device as sound,
Instead of or in addition to extracting the state information display area of the operation unit , the control unit detects sound from the notification unit and associates the device-related information corresponding to the state information with the button . displayed on the display unit by
The information display system according to claim 1, characterized by: The state information is operation process information indicating an operation process of the medical device,
the device-related information is a description of the operation unit;
wherein the control unit switches the description of the operation unit according to the operation step;
3. The information display system according to claim 1 or 2 , characterized in that: the status information is maintenance identification information or failure identification information of the medical device;
The device-related information is maintenance information or failure information of the medical device,
3. The information display system according to claim 1 or 2 , characterized in that: An information display system comprising: a medical device having an operation unit; an information display device that displays device-related information related to the medical device; and a control device,
The operation unit includes a state information display area for displaying state information of the medical device,
The information display device includes an imaging unit and a display unit,
The control device acquires an image captured by the imaging unit, extracts the buttons of the operation unit and the state information display area displayed in the image, and extracts the state information displayed in the state information display area. and causing the display unit to display the device-related information corresponding to the state information in association with the button .
An information display system characterized by: An information display device for displaying device-related information related to a medical device having an operation unit ,
The operation unit includes a state information display area for displaying state information of the medical device,
The information display device includes an imaging unit, a display unit, and a control unit,
The control unit extracts the buttons of the operation unit and the state information display area appearing in the image captured by the imaging unit , recognizes the state information displayed in the state information display area, and extracts the state information. displaying the device-related information corresponding to the button on the display unit in association with the button;
An information display device characterized by: An information display method for displaying device-related information related to a medical device having an operation unit on an information display device having an imaging unit and a display unit, comprising:
The operation unit includes a state information display area for displaying state information of the medical device,
extracting the button of the operation unit and the state information display area appearing in the image captured by the photographing unit , recognizing the state information displayed in the state information display area, and responding to the state information; displaying related information in association with the button on the display unit;
An information display method characterized by: An information display program for causing a control means to execute the information display method according to claim 7 .

Images