JP7599447B2 - Information control device, information control program, and information control method - Google Patents

Description

The present invention relates to an information control device that controls the information to be presented.

Extended reality (XR) technology is becoming more widespread. XR technology is a general term for virtual reality (VR), augmented reality (AR), mixed reality (MR), and other technologies. Research is being conducted into techniques to add useful information to XR, and various services are being considered.

When providing work support using AR technology, if images of all devices present in the actual work environment and AR content are displayed on the screen at the same time, the worker's cognitive load may increase and work efficiency may decrease. In light of this, a work support device has been proposed that presents information to support the worker's work in accordance with the work environment (Patent Document 1).

JP 2020-181456 A

However, the technology described in Patent Document 1 is intended to help workers easily reach the equipment they are working on in a work space such as a server room, but does not assist with the work itself.

The present invention was made in consideration of these circumstances. Its purpose is to provide an information control device that uses the information necessary for workers as a standard and deletes or reduces the visibility of unnecessary information.

The information control device according to one aspect of the present application is characterized by having an acquisition unit that acquires a plurality of pieces of display information constituting a portion of an XR space and information on an operation performed by a user, and an output unit that outputs the display information in a display mode different from normal when the degree of association between the display information and the operation satisfies a predetermined condition.

According to one aspect of the present invention, it is possible to delete information that is not necessary for the worker or to display it with reduced visibility.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of a work support system. 2 is a block diagram showing an example of a hardware configuration of an information control device. FIG. FIG. 2 is a block diagram showing an example of a hardware configuration of a display device. FIG. 11 is an explanatory diagram illustrating an example of a screen DB. FIG. 2 is an explanatory diagram illustrating an example of a part DB. FIG. 2 is an explanatory diagram illustrating an example of a classification master DB. FIG. 11 is an explanatory diagram illustrating an example of a classification DB. FIG. 11 is an explanatory diagram illustrating an example of a setting DB. FIG. 2 is an explanatory diagram showing an example of a work procedure DB. 13 is a flowchart showing an example of a procedure for a display process. FIG. 2 is an explanatory diagram showing an example of a display screen. FIG. 2 is an explanatory diagram showing an example of a display screen. FIG. 11 is an explanatory diagram showing an example of a display setting screen.

The following describes an embodiment with reference to the drawings. FIG. 1 is an explanatory diagram showing an example of the configuration of a work support system. The work support system 100 includes an information control device 1 and a display device 2. The information control device 1 and the display device 2 are communicatively connected via short-range wireless communication. The information control device 1 and the display device 2 do not necessarily need to be separate entities, and may be an integrated device in which the configuration and functions of the information control device 1 are incorporated into the display device 2.

The information control device 1 is configured with a tablet computer, a PC (Personal Computer), a smartphone, etc. The information control device 1 may also be configured with a server computer, a workstation, etc. The information control device 1 may also be configured with a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. Furthermore, the functions of the information control device 1 may be realized by a cloud service.

The display device 2 is a display device that is worn on the face, head, etc. The display device 2 is configured as a head-mounted display, VR goggles, AR goggles, MR goggles, SR goggles, smart glasses, etc. The display device 2 displays an image from the user's viewpoint in the XR space. The display device 2 may be configured as a smartphone fixed to a housing for goggles. The display device 2 may include an operation device used to operate the information control device 1.

Figure 2 is a block diagram showing an example of the hardware configuration of an information control device. The information control device 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 14, and a reading unit 15. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 14, and the reading unit 15 are connected by a bus B.

The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 11 reads and executes a control program 1P (program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc. related to the information control device 1, and realizing functional units such as an acquisition unit and an output unit.

The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data required for the control unit 11 to execute arithmetic processing.

The auxiliary storage unit 13 is a hard disk or SSD (Solid State Drive) or the like, and stores the control program 1P and various DBs (databases) necessary for the control unit 11 to execute processing. The auxiliary storage unit 13 stores a screen DB 131, a part DB 132, a classification master DB 133, a classification DB 134, a setting DB 135, and a work procedure DB 136. The auxiliary storage unit 13 may be an external storage device separate from the information control device 1. The various DBs and the like stored in the auxiliary storage unit 13 may be stored in a database server or cloud storage different from the information control device 1.

The communication unit 14 communicates simultaneously using multiple different communication standards. For example, the communication unit 14 communicates with the display device 2 using short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark). The communication unit 14 may communicate with the display device 2 via a USB (Universal Serial Bus) cable. On the other hand, the communication unit 14 communicates with the display device 2 via a wireless LAN router. The control unit 11 may also use the communication unit 14 to download the control program 1P from another computer via a network N or the like, and store it in the auxiliary storage unit 13.

The reading unit 15 reads portable storage medium 1a, including CD (Compact Disc)-ROM and DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 15 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

Figure 3 is a block diagram showing an example of the hardware configuration of a display device. The display device 2 includes a control unit 21, a main memory unit 22, an auxiliary memory unit 23, a communication unit 24, an input unit 25, an output unit 26, and an attitude detection unit 27. Each component is connected by a bus B.

The control unit 21 has one or more arithmetic processing devices such as a CPU, MPU, GPU, etc. The control unit 21 provides various functions by reading and executing a control program 2P (program, program product) stored in the auxiliary storage unit 23.

The main memory unit 22 is an SRAM, DRAM, flash memory, etc. The main memory unit 22 mainly temporarily stores data required for the control unit 21 to execute arithmetic processing.

The auxiliary storage unit 23 is a hard disk or SSD, etc., and stores various data necessary for the control unit 21 to execute processing. The various DBs, etc. stored in the auxiliary storage unit 23 may be stored in a database server or cloud storage.

The communication unit 24 communicates with the information control device 1 by short-range wireless communication, or wired serial communication (e.g., USB communication) or parallel communication. The communication unit 24 also communicates with an external computer via the network N. The control unit 21 may also use the communication unit 24 to download the control program 2P from another computer via the network N, etc., and store it in the auxiliary storage unit 23.

The input unit 25 is a keyboard or a mouse. A handheld controller may also be used as the input unit 25. The controller is equipped with an operation switch, a small joystick, etc. The output unit 26 includes a liquid crystal display panel, etc. The output unit 26 displays the XR images and the like output by the information control device 1. The input unit 25 and the output unit 26 may also be integrated to form a touch panel display. The XR images and the like output by the information control device 1 may also be displayed on a so-called normal PC monitor or liquid crystal display that is mainly used in connection with a PC.

The attitude detection unit 27 detects the attitude of the display device 2 and notifies the control unit 21 of the detection result. The attitude detection unit 27 is configured using an acceleration sensor, a gyro sensor, a magnetic sensor, or the like. The attitude detection unit 27 uses, for example, an acceleration sensor that detects acceleration in three axial directions to detect gravitational acceleration when the display device 2 is stationary. This allows the control unit 21 to calculate the degree to which the display device 2 is tilted with respect to the vertical direction from the detection result of the acceleration sensor (the direction of gravitational acceleration, etc.). Furthermore, when the display device 2 is moving, the acceleration sensor detects acceleration according to the movement, so that the attitude detection unit 27 can calculate the moving speed, moving direction, position, etc. of the display device 2. Furthermore, for example, when the attitude detection unit 27 has a gyro sensor, the tilt and position, etc. of the display device 2 can be calculated by calculating the time integral value (i.e., the rotation angle) of the angular velocity detected by the gyro sensor. However, the method of detecting the attitude of the display device 2 is not limited to this, and various other methods may be used. The tilt, position, direction, etc. of the display device 2 obtained from the detection result of the attitude detection unit 27 are called the attitude.

Next, the database used by the work support system 100 will be described. Figure 4 is an explanatory diagram showing an example of a screen DB. The screen DB 131 stores the design of the screen to be displayed on the display device 2, in particular the work screen. The screen DB 131 includes a screen ID column, a title column, a part ID column, and a position column. The screen ID column stores a screen ID that uniquely identifies a screen. The title column stores the title of the screen. The part ID column stores part IDs that identify the parts (display information) that make up the screen. Parts are also called controls, and are, for example, text boxes, pull-down menus, check boxes, etc. By specifying a screen as a part, it is also possible to create a nested structure of multiple screens. The position column stores the coordinates of the position where the part is placed on the screen.

Figure 5 is an explanatory diagram showing an example of a part DB. Part DB 132 stores information about parts. Part DB 132 includes a part ID column, a type column, and a property column. The part ID column stores a part ID that uniquely identifies a part. The type column stores the type of part. As described above, the types are text boxes, pull-down menus, check boxes, etc. The property column stores attributes related to parts. The attributes differ depending on the type of part. For example, for a text box, the attributes include the size, the content of the text to be displayed at the time of initial display, and a setting on whether or not the user is allowed to edit the text. For a pull-down menu, the attributes include the content to be displayed as a menu, the information specifying the menu to be displayed at the time of initial display, and the value to be returned when each menu is selected. For a check box, the attributes include the value of the label displayed together with the check box, and a setting on whether or not to check the initial display.

Figure 6 is an explanatory diagram showing an example of a category master DB. The category master DB 133 stores category definitions. A category is assigned to each element based on information held by the element to be displayed on the display device 2. The elements are, for example, screens and parts. The information control device 1 controls how each element is displayed on the display device 2 based on the category. The category master DB 133 includes a category ID column, a major category column, and a category column. The category ID column stores a category ID that uniquely identifies a category. The major category column stores the major categories of the categories. The category column stores the categories. A major category is a group of multiple categories. By defining the major categories, it becomes possible to control the display of multiple categories collectively.

Figure 7 is an explanatory diagram showing an example of a classification DB. Classification DB 134 stores the classification of parts. Classification DB 134 includes a part ID column and a classification ID column. The part ID column stores the part ID. The classification ID column stores the classification ID of the classification corresponding to the part.

Figure 8 is an explanatory diagram showing an example of a setting DB. The setting DB 135 stores display settings for each user. The setting DB 135 includes a user ID column, a noise removal column, an irrelevant item column, and a non-relevant item column. The user ID column stores a user ID that identifies a user. The noise removal column includes a background column, a person column, and a document column. The background column stores the display settings for the background. The person column stores the display settings for the person. In the case of VR display, the person includes an avatar. The document column stores the display settings for the document. The irrelevant item column includes a history column, a link column, and a popup column. The history column stores the display settings for the work history. The link column stores the display settings for the hyperlink. The popup column stores the display settings for the popup window. The non-relevant item column includes a reference column and a past non-reference column. The reference column stores the display settings for reference information. The past non-reference column stores the display settings for reference information that has not been referenced in the past. The display settings are 1 for display and 0 for non-display. Display settings may include settings other than display and non-display. For example, display settings may include displaying with low contrast, displaying in a reduced size, etc.

Figure 9 is an explanatory diagram showing an example of a work procedure DB. The work procedure DB 136 stores information about work procedures. A work procedure is a grouping of multiple tasks performed in a series. The work procedure DB 136 includes a work ID column, a number column, a content column, and a screen ID column. The work ID column stores a work ID that identifies a work procedure. The number column stores the sequence number of the task. The content column stores the task content. The screen ID column stores the screen ID of the screen to be displayed during the task.

Next, the information processing performed by the work support system 100 will be described. FIG. 10 is a flowchart showing an example of the procedure of the display processing. The control unit 11 of the information control device 1 acquires the display setting from the setting DB 135 (step S1). The control unit 11 acquires the information of the work to be performed by the user from the work procedure DB 136 (step S2). The control unit 11 searches the screen DB 131 using the screen ID included in the information of the work as a search key, and acquires the screen information (step S3). The control unit 11 selects a part included in the screen information (step S4). The control unit 11 determines the classification of the selected part (step S5). In this embodiment, the control unit 11 determines the classification of the part by the classification DB 134. The control unit 11 searches the classification DB 134 using the part ID of the selected part as a search key, and determines the classification of the part. The control unit 11 compares the acquired classification with the display setting and determines whether or not to change the display mode of the part (step S6). If the control unit 11 determines that the display mode of the part is not to be changed (NO in step S6), the process proceeds to step S8. When the control unit 11 determines that the display mode of the part is to be changed (YES in step S6), it changes the display mode of the part (step S7). The control unit 11 determines whether or not there is an unprocessed part (step S8). When the control unit 11 determines that there is an unprocessed part (YES in step S8), it returns the process to step S4 and processes the unprocessed part. When the control unit 11 determines that there is no unprocessed part (NO in step S8), it outputs the display screen to the display device 2 (step S9) and ends the process. Note that changing the display mode of the part means, for example, displaying the text lightly, displaying the text characters in a smaller size, or not displaying the text. On the other hand, the normal display mode is a mode in which all parts included in the screen are displayed. However, when a display control other than the display control in this embodiment is performed, there may be parts that are not displayed. The display screen output in step S9 is displayed on the display device 2, and the user recognizes the XR space by looking at it. Furthermore, when the display screen includes a part whose display mode has been changed, it can be said that the part whose display mode has been changed is output. Therefore, outputting a display screen that includes a part whose display mode has been changed is an example of "outputting display information in a display mode different from normal."

In the above description, if the classification of a part matches a pre-specified classification, it is determined that the part is unrelated to the work content and the display mode of the part is changed, but this is not limited to the above. As described below, it may be determined whether or not to change the display mode of the part based on the degree of association such as the degree of match or similarity. For example, if the content of the part is text, language analysis is performed. The control unit 11 divides the text into words by morphological analysis. It is determined whether each divided word is included in the description of the work content. If the proportion of matching words (degree of match) is equal to or greater than a predetermined threshold, the control unit 11 changes the display mode of the part, and if the degree of match is less than the threshold, the control unit 11 does not change the display mode.

The display mode may be changed differently depending on the type of XR space in which it is output. For example, when displaying in a VR space, if the degree of relevance is low as described above, it may be output in a manner that does not distract the user from their original work.

On the other hand, when displayed in an AR space, if the information to be displayed superimposed has a low degree of relevance to the task, it may be output in a manner that does not distract the user from the actual task. Furthermore, if the information superimposed in the real world has a low degree of relevance to the actual task, the information may be output in a manner that blocks out the information (e.g., by masking). For example, when working, everything except the computer screen is masked. When listening to a lecture, information other than the lecture presentation and the speaker is masked. When monitoring, information unrelated to the subject of monitoring is masked. Masking can be expected to allow users to concentrate more and become more immersed in their work, listening to a lecture, or monitoring.

The control unit 11 also determines whether the word is not included in the explanatory text, but a word with a similar meaning is included in the explanatory text. The control unit 11 determines the similarity between the text and the explanatory text indicating the work content from the degree of match and the ratio of similar words. If the similarity is equal to or greater than a predetermined threshold, the control unit 11 changes the display mode of the part, and if the similarity is less than the threshold, the control unit 11 does not change the display mode.

The similarity may be calculated using a natural language model such as BERT (Bidirectional Encoder Representations from Transformers). In addition, if the screen content contains text but the screen information does not contain text data, the control unit 11 acquires the screen content as an image. The control unit 11 uses a known character recognition technique to perform character recognition from the acquired image and acquire the displayed text.

Figures 11 and 12 are explanatory diagrams showing examples of display screens. Figures 11 and 12 are screens when working on batch 0002 in the flow diagram. Figure 11 is the normal display screen, displaying all parts (elements) to be displayed. Figure 12 displays a narrowed-down display of parts. In Figure 12, batches that are not the subject of work in the flow diagram do not display names and are displayed with dotted frames. Also, no detailed information is displayed.

Figure 13 is an explanatory diagram showing an example of a display setting screen. In the example shown in Figure 13, noise removal, irrelevant item removal, and non-work-related item removal are defined as major categories for which display control is possible. The noise removal subcategories are background, person, and document. The irrelevant item removal subcategories are history, link, and bop. The non-work-related item removal subcategories are reference and non-past reference. A slide switch is displayed for each subcategory. When the knob kn is moved to the right, removal is ON, and the parts corresponding to the subcategory are not displayed on the screen. When the knob kn is moved to the left, removal is OFF, and the parts corresponding to the subcategory are displayed on the screen. PT is a pointer. The display device 2 controls the position of the pointer PT by eye tracking. The position of the pointer PT may also be controlled by the posture of the display device 2. Input is possible by stopping the pointer PT at a specified position for a specified time or more. For example, when the pointer PT stops on the knob kn, ON changes to OFF and OFF changes to ON. By using eye tracking, the user can set the display even when they are working and need their hands free. The display setting screen shown in FIG. 13 may be displayed over the display setting screen shown in FIG. 11 or FIG. 12 (overlay display).

In the description of the display setting screen, only one of the two options, "delete ON" or "delete OFF," can be selected as the display control, but this is not limited to this. The threshold value that determines whether to display normally or change the display mode may be changed by the user using the knob kn. For example, in the process of determining that a document is noise, a numerical value indicating the degree of relevance between the contents of the document and the contents of the work is calculated, and it is determined whether the numerical value is equal to or greater than the threshold value or less than the threshold value. By allowing the user to change the threshold value, it is possible to display documents that are determined as noise in the initial setting in the normal display mode without determining them as noise. The same applies to other items that are determined using a threshold value. For example, in the setting that documents that have not been referenced in the past are not displayed, the user can set the range of the past to display documents that have not been referenced in the past month but have been referenced in the past three months.

In this embodiment, information that is not necessary for the work is not displayed, allowing the worker to easily identify the work target. This is expected to improve work efficiency.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.

REFERENCE SIGNS LIST 100 Work support system 1 Information control device 11 Control unit 12 Main memory unit 13 Auxiliary memory unit 131 Screen DB
132 Parts DB
133 Classification Master DB
134 Classification DB
135 Setting DB
136 Work procedure DB
14 Communication unit 15 Reading unit 1P Control program 1a Portable storage medium 1b Semiconductor memory 2 Display device 21 Control unit 22 Main storage unit 23 Auxiliary storage unit 24 Communication unit 25 Input unit 26 Output unit 27 Attitude detection unit 2P Control program B Bus N Network

Claims (6)

An acquisition unit that acquires a plurality of pieces of display information constituting a part of the XR space and information on an operation performed by a user;
an output unit that outputs the display information in a display mode different from a normal display mode when a degree of relevance between the display information and the task satisfies a predetermined condition ,
The information control device is characterized in that the degree of association is determined based on a similarity or a degree of agreement between a word included in the display information and a word associated with the task . The information control device according to claim 1, characterized in that when the relevance degree of the display information is determined to be noise, an irrelevant item, or unrelated to the task, the output unit outputs the display information in a display mode different from normal. 3. The information control device according to claim 1 , wherein the display mode different from the normal mode is a display with a low contrast. The information control device according to claim 1 or 2 , wherein the display mode different from the normal mode is no display output. Acquire a plurality of pieces of display information constituting the XR space and information on the work performed by the user;
When a degree of relevance between the display information and the task satisfies a predetermined condition, the display information is output to the XR space in a display mode different from a normal display mode ,
The degree of association is calculated based on a similarity or a match between a word included in the display information and a word associated with the task.
An information control program that causes a computer to execute a process. Acquire a plurality of pieces of display information constituting the XR space and information on the work performed by the user;
When a degree of relevance between the display information and the task satisfies a predetermined condition, the display information is output to the XR space in a display mode different from a normal display mode ,
The degree of association is calculated based on a similarity or a match between a word included in the display information and a word associated with the task.
An information control method, characterized in that the processing is executed by a computer.

Images