JP6531007B2 - Mark processing device, program - Google Patents

Description

  The present invention relates to a mark processing apparatus for processing a mark drawn on an image displayed on a terminal of a telecommunications system.

  At work sites where the knowledge of experts and experts is important, there is a scene where instructors such as experts and experts instruct (teacher) those who are not trained in the work or young workers There are many. For example, there is a scene that transmits know-how such as work procedures, judgment criteria, and how to deal with problems. At this time, if the instructor and the worker (the person who receives the instruction) are at the same place and can communicate the content of the work while communicating face to face, the instruction content can be efficiently transmitted.

  However, there are cases where the worker and the instructor are not at the same place at the same time, and in such a case, it is difficult for the worker to receive the instructions of the instructor and for the worker to receive the instructions of the instructor. Become. For example, when the work site is narrow and it is physically difficult for a group of people to work together, or the instructor is at a place far from the work site and immediately goes to the work site worker. There are cases where you can not do it. Even in these cases, if there is a manual that shows the judgment criteria and procedures for performing the work, the worker can proceed with the work according to the manual. However, in reality, situations may occur that are not described in the manual (a sudden problem or a situation that must be determined empirically depending on the situation), and inexperienced workers should handle it. Often not complete.

  On the other hand, conventionally, there has been known a solution method for performing work support from a remote place using a video phone (video phone) so that the operator can be instructed even when the instructor can not be present. There is. In this method, the worker shoots the work location and the work situation with the terminal, transmits the video obtained by this shooting to the instructor, and the instructor mainly performs voice communication based on the received video. It is intended to communicate instructions through.

  However, in this method, the instructor can not perform an explanation in voice (for example, an explanation in voice while pointing at an explanation target part) in combination with a visual instruction, and only in voice. There must be.

  Here, in the case of the explanation only by voice, if the point to be described is specified by vague expressions such as “here” and “that”, the intention of the instructor is not accurately transmitted to the worker on the site. Therefore, in the case of audio-only explanation, it is required to avoid the vague expression as much as possible, and to specify the explanation target part by universal expression such as "from the right end to what number from the top". .

  However, for example, in the case where the worker is constantly moving, even if it is the "third" place for the instructor, it will no longer be changed to the "fourth" or any other place for the worker. There is a possibility. Therefore, in the case of the explanation only by voice, it is often the case that the intention of the instructor is not accurately transmitted to the worker on the site even if the instructor specifies the explanation target part by the universal expression.

  Then, if the intention of the instructor is not accurately transmitted to the on-site worker, the recognition tends to be apt to occur between the two, resulting in a problem that the working efficiency is lowered. Also, even if it is possible to accurately convey to the worker the part to be described to the worker by a universal expression (for example, expression such as “from the right end to what number from the top”), normal work (with the worker Unlike the operation where the instructor is at the same place at the same time and does not use the above-mentioned solution, a detailed explanation (conversation) is required, which causes a problem of reduction in work efficiency.

  By the way, in recent years, research on augmented reality (AR) technology for drawing an image created by computer graphics (CG) superimposed on a captured image has become active. By using the AR technology, it is possible to draw a mark (pattern, code, characters, etc.) created by CG on an image captured in real time. In this way, it is possible to show a mark that is not actually present at the work site as if it is present at the work site in the image showing the work site. That is, if the AR technology is applied to a solution method for performing work support from a remote place using a video phone (video phone), the problems described above can be suppressed.

  For example, according to the technology disclosed in Non-Patent Document 1 shown below, the terminal of the instructor (operator) receives the video taken by the terminal of the worker, and the video is operated by the operation of the instructor. A mark such as a circle or an arrow is drawn at the upper target position. The mark drawn in this way is also displayed on the terminal of the worker (and, even if the shooting range is shifted due to the change in the direction or position of the terminal of the worker, The mark is displayed without following the target point (following the target point). As a result, even if the instructor talks using vague expressions such as "here" and "that", the target position becomes clear even for the operator by the mark, so that it is difficult for a conflict to occur between the two. Become. According to such a technique, merits such as efficiency of work, realization of highly specialized work not depending on skill proficiency, and work time reduction and cost reduction can be obtained.

Japanese Patent Application Publication No. 2013-518275

AR support function [Online], Nippon Telegraph and Telephone Corporation, [July 2, 2015 search], Internet, <URL: http://www.ntt-east.co.jp/release/detail/20131024_01.html 〉

  In general, in work support using AR technology, the instructor is the target of work by the worker, the target of conversation with the worker, and the target of explanation to the worker. The mark is drawn at the place where the mark is or related to these points.

  Here, it is assumed that the worker itself is always moving, and therefore, it is assumed that the terminal (photographing terminal) held by the worker is also moving, and the photographing range of the terminal is also moving. However, since the movement of the terminal corresponds to the movement of the worker, it is difficult for the instructor to predict the movement of the shooting range of the terminal, but under such circumstances, the target location on the image is A series of operations must be performed to identify and mark the target location.

  Therefore, it is very difficult for the instructor to set the mark accurately at the target location where the movement is difficult to predict on the display screen, so the image displayed on the display screen of the instructor's terminal is paused. It is desirable to carry out the above operation under the condition that the same frame is continuously displayed.

  On the other hand, the worker continues the work, and the terminal on the worker side does not pause the video, and the shooting and updating is continued, so the terminal on the worker side and the terminal on the instructor side Different scenes will be displayed (different frames will be displayed). This is because if the operator moves a lot while the instructor is performing a series of operations for marking the target location, the position of the marker newly set for the instructor is on the terminal side of the operator. It means that you get out of the display screen.

  In work support using telecommunications, immediate information transmission is important, and it is required to rapidly transmit highly important information related to the work content currently being worked.

  On the other hand, even if the instructor determines that it is important and draws a mark, as described above, the scene displayed on the terminal of the worker and the scene displayed on the terminal of the instructor Because the operator may not be able to recognize that a new mark has been drawn, the promptness of information transmission may be lost.

  The present invention aims at ensuring the speed of information transmission in a telecommunication system.

  In order to solve the above-mentioned subject, one mode of the present invention contains the 1st terminal and the 2nd terminal which can communicate mutually, and the picture picturized with the 1st terminal is both the 1st terminal and the 2nd terminal When the input operation for drawing a mark at the target position shown in the displayed video is performed in the second terminal, the above-described video is displayed in the first terminal and the second terminal. A mark processing apparatus included in a telecommunication system in which the mark is superimposed and displayed on the target position so as to move following the movement of the target position, and it is necessary to notify the user of the first terminal of the presence of the mark The apparatus is characterized by comprising a determination unit that causes the first terminal to execute notification processing of notifying the user of the presence of the mark when it is determined that the presence or absence of sex is present and the necessity is present.

  According to one aspect of the present invention, when an input operation of the mark is performed, the presence of the mark determined to be highly necessary to notify the user of the first terminal of the presence of the mark is determined. The user of the terminal can be notified promptly. Therefore, according to the present invention, it is possible to ensure the rapidity of information transmission in the telecommunication system.

It is a figure showing typically an example of the use scene of the telecommunication system concerning one embodiment of the present invention. FIG. 2 is a schematic diagram showing components of the telecommunication system of FIG. 1; It is the block diagram which showed the outline of a structure of the work terminal included in the telecommunication system of FIG. It is the block diagram which showed the outline of the information processing apparatus of the instruction | indication terminal contained in the telecommunication system of FIG. It is a block diagram showing an outline of composition of a terminal unit which can be used as a work terminal and can also be used as an instruction terminal. It is the block diagram which showed the structure of the mark tracking part of FIG. 4 or FIG. It is a flowchart which shows the flow of a process of a work terminal. It is a flowchart which shows the flow of the information processing apparatus of an instruction | indication terminal. It is the flowchart which showed the flow of a process of the determination part shown in FIG. It is the figure which showed typically an example of the image | video displayed on the display part of a work terminal. It is a figure which is an example of the image | video displayed on the display part of a working terminal, and has shown typically the example different from FIG. It is a figure which is an example of the image | video displayed on the display part of a working terminal, and has shown typically the example different from FIG.10 and FIG.11.

Embodiment 1
A telecommunication system (telecommunication system) according to an embodiment of the present invention will be described with reference to FIG. 1 to FIG.

  FIG. 1: is the figure which showed typically an example of the usage scene of the telecommunication system A which concerns on this embodiment. The left side of FIG. 1 is a work site 100, and the right side of FIG. 1 shows an instruction room 106, which are located apart from each other. The telecommunication system A is a system that realizes work support by communicating information regarding work between a worker and an instructor located apart from each other.

  FIG. 1 shows a scene in which a worker 101 at a work site 100 is performing a work while receiving a work instruction on a work target 102 from a work terminal 102 (first terminal) from an instruction person 107 in an instruction room 106. It shows. Explaining in more detail, it is an example in which the worker 101 who is repairing the work object 102 receives an instruction regarding the repair from the instructor 107 who supervises.

  The work terminal 103 is a tablet computer, and includes a display unit 103 b provided on the front side and a camera 103 a provided on the back side. The work terminal 103 can capture the work target 102 with the camera 103a, can display an image obtained by the capture on the display unit 103b, and send the image to the instruction terminal (second terminal) 108 at a remote location. It can be sent.

  The instruction terminal 108 installed in the instruction room 106 includes an information processing device (mark processing device) 108 a and a display device 108 b. The information processing device 108 a and the display device 108 b are connected, and display control of the display device 108 b is performed on the information processing device 108 a. The information processing apparatus 108 a is in the form of a desktop personal computer (Personal Computer) as shown in FIG. 1, but is not limited to this form, and is a tablet-like computer used by the worker 101. It may be

  The information processing apparatus 108a can receive the video sent from the work terminal 103 at a remote location, and can display the video on the display device 108b. The instructor gives a work instruction to the worker 101 using the display device 108 b and the instruction terminal 108 while watching the image 110 displayed on the display device 108 b.

  Specifically, when the instructor 107 issues an input instruction using a touch panel or a mouse, the instruction terminal 108 generates mark information indicating an instruction mark based on the input instruction. The instruction mark is a mark displayed superimposed on the designated position designated by the instructor 107 in the video displayed on the display device 108 b and the display unit 103 b, and the instructor 107 instructs the operator Used for In the present embodiment, the indication mark may be a pointer, a marker, a text, a pattern, or a combination of two or more of them, in addition to a simple mark.

  When generating the mark information, the instruction terminal 108 causes the display device 108 b to display an instruction mark based on the mark information, and transmits the mark information to the work terminal 103. The work terminal 103 also causes the display unit 103 b to display an instruction mark based on the mark information. That is, the work terminal 103 and the instruction terminal 108 share mark information, and superimpose the same instruction mark on a video based on the mark information.

  That is, in the instruction terminal 108, the input operation for superimposing the instruction mark on the work target video 102a (the video corresponding to the work target 102) shown in the video 110 of the display device 108b of FIG. The mark information indicating that the instruction mark is to be superimposed on the work target video 102 a is generated, the mark information is transmitted to the work terminal 103, and the mark information is shared with the work terminal 103.

  Then, based on the mark information, the instruction terminal 108 superimposes the instruction mark 111 on the work target image 102 a shown in the image 110 of the display device 108 b of FIG. 1, and the work terminal 103 applies the instruction mark information to the instruction mark information. Based on the instruction mark 105, the instruction mark 105 is superimposed on the work target image 102a shown in the image 104 of the display unit 103b of FIG.

  The operator 101 can grasp the instruction mark 105 and the portion (target position) designated by the instruction mark 105 by the display unit 103 b. As a result, the worker 101 can visually grasp the work instruction from the instruction room 106 located at the remote place. The work terminal 103 can also set mark information based on an input instruction from the worker 101, and display an instruction mark superimposed on the image 104 of the display unit 103b based on the mark information. At this time, as a matter of course, the work terminal 103 transmits the mark information to the instruction terminal 108, the work terminal 103 and the instruction terminal 108 share the mark information, and the instruction terminal 108 is sent from the work terminal 103. An instruction mark can be superimposed on the image 110 of the display device 108 b based on the mark information. As a result, both the instructor 107 and the worker 101 can recognize the instruction mark set by one of the instructor 107 and the worker 101.

  Further, as shown in FIG. 2, the telecommunication system A includes a work terminal 103, an instruction terminal 108, and a public communication network (for example, the Internet) NT, and the work terminal 103 and the instruction terminal 108 are public communication networks. It can communicate according to protocols, such as TCP / IP and UDP, via NT.

  As shown in FIG. 2, the telecommunication system A is provided with a management server 200 for collectively managing mark information, and the management server 200 is also connected to the public communication network NT. The work terminal 103 and the public communication network NT may be connected by wireless communication. In this case, wireless communication can be realized, for example, by the Wi-Fi (Wireless Fidelity) connection of the international standard (IEEE 802.11) defined by the Wi-Fi (registered trademark) Alliance (US industry group). It is. Furthermore, although the public communication network such as the Internet is exemplified as the communication network above, for example, it is also possible to use a LAN (Local Area Network) used in a company or the like, and a configuration in which they are mixed It may be Although FIG. 2 shows the configuration using the management server 200, there is no problem even if the work terminal 103 and the instruction terminal 108 directly communicate with each other, and in the following description, the work terminal 103 and the instruction terminal 108 And the direct communication form will be described. Further, with regard to general audio communication processing used in an ordinary video conference system and video communication processing other than the additional screen information, the description will be omitted within a range where there is no problem.

  First, each block which performs a process in the work terminal 103 is demonstrated using FIG. As illustrated in FIG. 3, the work terminal 103 includes a control unit 308, a communication unit 301, a display control unit 302, a video acquisition unit 303, and an encoding unit 304.

  The video acquisition unit 303 is connected to the camera 103a included in the work terminal 103, acquires a video from the camera 103a, and outputs the video to the encoding unit 304 based on the control of the control unit 308.

  The control unit 308 controls the entire process of the work terminal 103 and controls exchange of information between the units.

  The encoding unit 304 performs encoding processing to compress the video signal input from the video acquisition unit 303 so as to be smaller than the original signal amount. For the encoding process, for example, H.264 (international standard moving image compression standard) suitable for moving image encoding can be applied, or another method can be used. Also, the encoding unit 304 outputs a video code obtained by encoding a video signal to the communication unit 301.

  When receiving the video code from the encoding unit 304, the communication unit 301 shapes the data (video code) so that communication via the network can be performed, and the video code is sent to the outside of the work terminal 103 (for example, the instruction terminal 108 described later). Perform processing to send. The shaping of data for performing network communication is, for example, processing of adding information necessary for communication in accordance with a protocol such as TCP / IP or UDP.

  Communication unit 301 further receives mark information from the outside of work terminal 103 (for example, instruction terminal 108), and outputs the information to display control unit 302.

  When the display control unit 302 receives the mark information from the communication unit 301, the display control unit 302 refers to the mark information and superimposes the instruction mark on the video acquired by the video acquisition unit 303, and displays the video on which the instruction mark is superimposed. A process to display on 103 b is performed. Conventional techniques can be used to superimpose the indication mark.

  Then, each block which processes in the information processing apparatus 108a of the instruction | indication terminal 108 is demonstrated using FIG.

  The information processing apparatus 108 a of the instruction terminal 108 includes a control unit 408, a communication unit 401, a display control unit 402, a decoding unit 405, a video holding unit 406, a mark tracking unit 407, and an input unit 409.

  The control unit 408 controls the entire processing of the instruction terminal 108 and controls exchange of information between the units.

  When the information processing apparatus 108 a receives a video code from the outside of the instruction terminal 108 (for example, the work terminal 103 described later), the video code is input to the communication unit 401. The communication unit 401 performs a process of outputting the video code to the decoding unit 405.

  When receiving the video code from the communication unit 401, the decoding unit 405 performs processing (decoding processing) reverse to the coding processing in the encoding unit 304 described above. Thus, the video code is decoded into the original video signal. The decoding unit 405 performs a process of outputting the decoded video signal to the display control unit 402, the video holding unit 406, and the mark tracking unit 407.

  The display device 108 b connected to the information processing device 108 a is a touch panel, and the input unit 409 detects the position where the instructor 107 has touched the screen of the display device 108 b.

  The video holding unit 406 holds the video signal received from the decoding unit 405, and outputs a video signal one frame before the current frame to the mark tracking unit 407.

  The mark tracking unit 407 receives the video signal of the current frame from the decoding unit 405, the video signal of the frame one frame before the current from the video holding unit 406, and the detection result of the input unit 409, and generates mark information. Do the process. This process will be described later.

  The mark information generated by the mark tracking unit 407 is output to the display control unit 402 and the communication unit 401. When the communication unit 401 receives the mark information from the mark tracking unit 407, the communication unit 401 performs processing for transmitting the information to the outside (for example, the work terminal 103).

  When receiving the video signal from the decoding unit 405, the display control unit 402 causes the display device 108b to display a video according to the video signal. Further, when the display control unit 402 receives the mark information from the mark tracking unit 407, the display control unit 402 causes the display device 108b to display the video on which the instruction mark is superimposed.

  As described above, the configuration in FIG. 3 is each block for processing in the work terminal 103, and the configuration in FIG. 4 is each block for processing in the information processing apparatus 108a of the instruction terminal 108. The work terminal 103 and the instruction terminal 108 can also have a common configuration, and a terminal device 500 with this configuration is shown in FIG. That is, the terminal device 500 of FIG. 5 can be used as the work terminal 103 (see FIG. 3), and can also be used as the instruction terminal 108 (see FIG. 4).

  When the terminal device 500 in FIG. 5 is used as the work terminal 103, the communication unit 501 functions as the communication unit 301, the display control unit 502 functions as the display control unit 302, and the control unit 508 as the control unit 308. Function. When the terminal device 500 of FIG. 5 is used as the instruction terminal 108, the communication unit 501 functions as the communication unit 401, the display control unit 502 functions as the display control unit 402, and the control unit 508 as the control unit 408. Function. Further, among the blocks shown in FIG. 5, the blocks given the same reference numerals as the blocks in FIG. 3 or 5 perform the same processing as the blocks with the same reference numerals in FIG. 3 or 5. However, when the terminal device 500 of FIG. 5 is used as the work terminal 103, the blocks having the same reference numerals as the blocks shown in FIG. 3 among the blocks of FIG. 5 are paused, and the terminal device 500 of FIG. When it is used as the instruction terminal 108, among the blocks in FIG. 5, the blocks having the same reference numerals as the blocks shown in FIG. 3 are paused.

  Subsequently, the details of the mark tracking unit 407 in FIG. 4 or 5 will be described with reference to FIG. As shown in FIG. 6, the mark tracking unit 407 includes a video signal acquisition unit 601, a displacement vector calculation unit 602, a displacement vector addition unit (addition processing unit) 605, a mark management unit 606, and a position calculation unit ( An update processing unit) 607, a determination unit 608, and a mark information generation unit (position information generation unit) 610 are included. Further, as shown in the figure, the displacement vector calculation unit 602 includes an optical flow calculation unit 603 and an averaging processing unit 604.

  The video signal acquisition unit 601 acquires the video signal t of the current frame from the decoding unit 405 and the video signal t-1 of the frame one frame before the current frame from the video holding unit 406, and displaces these video signals. A process of passing to the vector calculation unit 602 is performed.

  Every time a combination of the video signal t and the video signal t-1 is input from the video signal acquisition unit 601 to the displacement vector calculation unit 602, the optical flow calculation unit 603 and the averaging processing unit 604 perform the processing described below.

  The optical flow calculation unit 603 executes an optical flow calculation process based on the video signal t and the video signal t-1. The optical flow is a two-dimensional vector composed of a component in the x direction and a component in the y direction, and is a vector indicating the movement amount and direction of the motion of the object shown in the image. The optical flow calculation unit 603 outputs all the optical flows calculated from the combination of the video signal t and the video signal t-1 to the averaging unit 604.

  The averaging processing unit 604 calculates the average of all the optical flows input from the optical flow calculating unit 603 as a displacement vector. That is, the displacement vector is a vector that indicates the movement amount and direction of movement of the entire video.

  The displacement vector obtained by the averaging processing unit 604 is sent to the position calculating unit 607. In addition, during a period in which the image to be displayed on the display device 108 b is temporarily stopped according to the command (command by the instructor) input from the input unit 409, the displacement vector obtained by the averaging processing unit 604 is Not only the position calculation unit 607 but also the displacement vector addition unit 605 is sent.

  When the instructor performs a touch operation for drawing a mark at a designated position of the video displayed on the display device 108b, the mark information generation unit 610 generates the mark information according to the touch operation. Set an instruction mark according to the mark information. The mark information indicates the position information of the designated position and the image data of the instruction mark attached to the designated position.

  When an instruction mark is set by the mark information generation unit 610, the mark management unit 606 registers the instruction mark by storing mark information of the instruction mark in a memory (not shown), and the mark management unit 606 registers the instruction mark. Manage mark information.

  The displacement vector addition unit 605 executes processing only during a period in which the display control unit 402 suspends the video to be displayed on the display device 108 b in accordance with the command (command by the instructor) input from the input unit 409. . Note that pause means a state in which the same frame continues to be displayed.

  Specifically, the displacement vector addition unit 605 continues to add (continue to add) the displacement vector calculated by the averaging processing unit 604 while the image displayed on the display device 108 b is temporarily stopped. . A vector obtained by this addition is referred to as an addition displacement vector.

  In addition, when the mark information generation unit 610 newly sets an instruction mark during a period in which the image displayed on the display device 108 b is temporarily stopped, the displacement vector addition unit 605 sets the addition displacement vector at the time of setting as mark information. The position information is corrected by adding to the indicated position information.

  That is, when the mark information generation unit 610 newly sets an indication mark while the pause of the image displayed on the display device 108b is canceled, the mark information corresponding to the indication mark is included in the mark information. The added displacement vector is stored in the memory by the mark management unit 606 without being added by the displacement vector addition unit 605 to the position information to be obtained. On the other hand, when the mark information generation unit 610 newly sets an indication mark while the video displayed on the display device 108 b is paused, the mark information corresponding to the indication mark is included in the mark information. After the addition displacement vector is added by the displacement vector addition unit 605 to the position information to be obtained, the mark management unit 606 stores the addition displacement vector in the memory.

  Note that the displacement vector addition unit 605 resets the addition displacement vector to zero when the temporary stop is released.

  The position calculation unit 607 refers to the displacement vector sent from the displacement vector calculation unit 602 every time the frame at the current time changes, and the mark information of the indication mark managed by the mark management unit 606. Update process of

  This update process is performed each time the displacement vector is calculated by the displacement vector calculation unit 602, and the position information indicated by the mark information of the registered instruction mark is updated according to the displacement vector. This is a process of moving the position of the instruction mark in accordance with the displacement vector.

  When the update process is performed, the mark information after the update process is sent from the position calculation unit 607 to the mark management unit 606, and the mark management unit 606 overwrites the mark information after the update process on the memory. As a result, the mark information of the instruction mark managed by the mark management unit 606 is always after the update processing. Also, the position calculation unit 607 outputs the mark information after the update processing to the determination unit 608.

  The determination unit 608 is a block that determines the presence or absence of necessity for notifying the worker of the presence of the registered instruction mark. The contents of the process will be described later.

  Subsequently, the flow of processing in the work terminal 103 shown in FIG. 3 will be described using FIG. 7. FIG. 7 is a flowchart showing the process flow of the work terminal 103.

  Step 701 (S701): The video acquisition unit 303 acquires a video from the camera 103a, and outputs the video signal of the obtained video to the encoding unit 304 and the display control unit 302. The encoding unit 304 performs an encoding process on the video signal input from the video acquisition unit 303. The encoding unit 304 outputs the video code generated by the encoding process to the communication unit 301. The control unit 308 then proceeds with the process to step 702.

  Step 702 (S702): When the communication unit 301 receives the video code from the encoding unit 304, the communication unit 301 reshapes the data so as to be able to communicate, and outputs (transmits) it to the outside of the work terminal 103. The control unit 308 then proceeds with the process to step 703.

  Step 703 (S703): When receiving the video signal from the video acquisition unit 303, the display control unit 302 causes the display unit 103b to display the video indicated by the video signal. The control unit 308 then proceeds with the process to step 704.

  Step 704 (S704): The communication unit 301 receives mark information from the outside, and outputs the mark information to the display control unit 302. The control unit 308 then proceeds with the process to step 705.

  Step 705 (S 705): Based on the mark information received from the communication unit 301, the display control unit 302 superimposes an instruction mark on a video and displays it. The control unit 308 then proceeds with the process to step 706.

  Step 706 (S706): The control unit 308 determines whether to end the process. The control unit 308 ends all processing when determining to end the processing. On the other hand, when the control unit 308 determines that the process is not to be ended (when the process is continued), the process returns to step 701, and step 701 and subsequent steps are repeatedly executed. For example, the control unit 308 ends all processing when an end command is input from the worker, and otherwise returns to S701.

  Subsequently, the flow of processing of the information processing device 108 a of the instruction terminal 108 shown in FIG. 4 will be described using FIG. 8. FIG. 8 is a flowchart showing the process flow of the instruction terminal 108.

  When the process is started, the control unit 408 provided in the information processing apparatus 108a simultaneously activates step 801 and step 808.

  Step 801 (S801): When the communication section 401 receives the video code sent from the work terminal 103, the communication section 401 outputs the video code to the decoding section 405. The control unit 408 then advances the process to step 802.

  Step 802 (S802): When receiving the video code from the communication unit 401, the decoding unit 405 performs the above-mentioned decoding process. The decoding unit 405 outputs the video signal obtained by the decoding process to the display control unit 402, the video holding unit 406, and the mark tracking unit 407. Thereafter, the control unit 408 advances the process to step 803.

  Step 803 (S803): When receiving the video signal from the decoding unit 405, the display control unit 402 causes the display device 108b to display the video of the video signal. The control unit 408 then advances the process to step 804.

  Step 804 (S804): When the mark tracking unit 407 receives the video signal t from the decoding unit 405 and the video signal t-1 from the video holding unit 406, the optical flow calculation unit 603 calculates the optical flow, and the average is calculated. The processing unit 604 calculates the displacement vector using the optical flow. At the same time, while the image of the display device 108 b is temporarily stopped according to a command from the instructor, the displacement vector addition unit 605 performs addition of displacement vectors to calculate an addition displacement vector. Subsequently, the process proceeds to step 805.

  Step 805 (S805): If there is an instruction mark registered in the mark management unit 606, the position calculation unit 607 of the mark tracking unit 407 updates the position information of the mark information of the registered instruction mark. Specifically, the position information is updated such that the position indicated by the position information is displaced in the direction and the direction corresponding to the displacement vector in S804. The mark management unit 606 overwrites the mark information stored in the memory with the updated content. The mark tracking unit 407 outputs mark information of the registered instruction mark to the display control unit 402 and the communication unit 401. The control unit 408 then advances the process to step 806.

  Step 806 (S806): When the display control unit 402 receives the mark information from the mark tracking unit 407, the display control unit 402 causes the display device 108b to display an instruction mark superimposed on the image based on the mark information. The control unit 408 then advances the process to step 807.

  Step 807 (S807): When the communication unit 401 receives the mark information from the mark tracking unit 407, the communication unit 401 outputs (sends) it to the outside of the instruction terminal 108. The control unit 408 then proceeds to step 810.

  Step 808 (S808): The mark information generation unit 610 generates mark information according to the position of the touch operation of the instructor on the display screen of the display device 108b, and outputs the mark information to the mark tracking unit 407. The control unit 408 then advances the process to step 809.

  Step 809 (S 809): When receiving the mark information from the mark information generation unit 610, the displacement vector addition unit 605 adds the added displacement vector calculated in step 804 to the position information indicated by the mark information to obtain position information. Make corrections for Specifically, the position information is corrected such that the position indicated by the position information is displaced in the direction and the direction corresponding to the addition displacement vector in S809. The mark management unit 606 registers the indication mark of the mark information by storing the mark information whose position information has been corrected in the memory. The control unit 408 then advances the process to step 810.

  Step 810 (S810): The control unit 408 determines whether to end the process. The control unit 408 ends all processing when determining to end the processing. On the other hand, when the control unit 408 determines that the process is not to be ended (in the case of continuing the process), the process returns to steps 801 and 808, and the flow of FIG. 8 is repeatedly executed. For example, when an end command is input from the instructor, the control unit 308 ends all processing, and in the other cases, the flow of FIG. 8 is repeatedly executed.

  Subsequently, steps performed by the mark tracking unit 407 will be described in detail. The steps performed by the mark tracking unit 407 include the steps of registering the indication mark and updating the mark information, and determining the presence or absence of necessity of notifying the worker of the work terminal 103 of the presence of the indication mark. It is.

  First, registration of an instruction mark and update of mark information will be described. The instruction mark may be set by the operator 101 touching the screen of the display unit 103 b of the work terminal 103 or may be set by the instructor 107 touching the screen of the display device 108 b. It is possible to handle any indication mark in the same manner, and the information processing device 108 a of the indication terminal 108 manages and updates them.

  It is necessary to update the position information indicated by the mark information at any time based on the movement of the video accompanying the movement of the actual worker 101. This is because, for example, in the case of the instruction mark set on the work target video 102 a, the instruction mark is displayed so as to follow the movement of the work target video 102 a. There are various update methods, but in the present embodiment, this is realized by a method using the optical flow of video.

  The optical flow is a vector representation of the motion of an object in a video, and an existing method can be applied. For example, OpenCV, which is one of a general-purpose image processing library group, has a function (cvCalcOpticalFlowLK) capable of calculating an optical flow, and the function can also be used. A plurality of optical flows are extracted as vectors from a video, and an average vector that is the average of all the optical flows is used as a representative vector (displacement vector) representing the motion of the entire video. By updating the position information indicated by the mark information of the registered indication mark using this displacement vector, the indication mark can be displayed so as to follow the movement of the scene being photographed.

  Here, in the present embodiment, the position calculation unit 607 updates the position information indicated by the mark information of the registered instruction mark, but can be realized by the following expression a.

  By the way, in the present embodiment, it is assumed that the video is paused when the instructor 107 performs the touch operation for setting the instruction mark. This is effective to increase the accuracy of the setting position of the instruction mark, and is a highly necessary function in the actual operation by the instructor 107.

  However, even if the video of the display device 108b that the instructor 107 is viewing is paused, the video of the work terminal 103 is not paused. Therefore, when the video of the display device 108b being viewed by the instructor 107 is paused, the scene of the video displayed on the display device 108b for the instructor and the scene of the video displayed on the work terminal 103 are displayed. As a result, the instruction mark set by the instruction person can not be appropriately reflected on the image.

  Therefore, in the present embodiment, when the video of the display device 108b of the instructor is temporarily stopped, the displacement vector addition unit 605 performs averaging processing of the displacement vector calculation unit 602 from the frame at the time of temporary suspension to the current frame. The displacement vector calculated at 604 is constantly added (accumulated), and the added displacement vector which is the addition result is held.

  Then, when the instruction mark information is newly set by the mark information generation unit 610 during the temporary stop, the displacement vector addition unit 605 adds an addition displacement vector to the position information indicated by the mark information of the instruction mark. The position information is corrected by adding.

  However, this correction is a process necessary only for an instruction mark newly set while being paused (the same frame is continuously displayed), and an instruction mark newly set in a state where it is not paused. Does not make this correction.

  In addition, the position information of the instruction mark newly set during the temporary stop can be accurately updated only by the update processing by the position calculation unit 607 after the correction by the displacement vector addition unit 605 is performed once. Becomes possible.

  Hereinafter, specific contents of the correction process performed by the displacement vector addition unit 605 will be described. When the video of the display device 108b used by the instructor 107 is paused by the input command of the instructor 107, the displacement vector addition unit 605 uses the following equation b to set the current time during the pause period. Every time the frame changes, the displacement vector output from the averaging unit 604 is added. The displacement vector addition unit 605 continues the above addition until the temporary rest is released.

  When the indication mark is newly set during the temporary stop, the displacement vector addition unit 605 corrects the position information indicated by the mark information of the indication mark, using the following equation c.

  Expression c means that position information in the current frame is calculated by adding the added displacement vector generated during the temporary stop to the position information corresponding to the frame displayed in the temporary stop. ing. When mark information including position information (corresponding to the current frame) calculated by equation c is stored in the memory by mark management unit 606, the following procedure is always performed on the current frame by using equation a. It becomes possible to update the location information to correspond.

  From the point described above, when an instruction mark is newly set, position information when the instruction mark is registered is as shown in [Equation 4].

  Subsequently, processing of the determination unit 608 illustrated in FIG. 6 will be described. The determination unit (unread / read determination unit) 608 determines the necessity of performing notification processing for notifying the user (worker) of the work terminal 103 of the presence of the registered instruction mark. It is a block.

  As described above, when the instructor 107 pauses the video of the display device 108b to set the instruction mark, the video of the work terminal 103 and the video of the display device 108b show different scenes. There is.

  Therefore, when the instructor 107 sets the instruction mark, the position of the instruction mark may be out of the display range of the work terminal 103. In this case, the worker 101 can not recognize the presence of the instruction mark. In addition, while the video of the display device 108 b is paused, the instructor 107 can not grasp the real-time video captured by the worker 101. Therefore, even if the operator 101 can not actually recognize the presence of the instruction mark, the instructor 107 may assume that the operator 101 recognizes the presence of the instruction mark.

  Therefore, in the present embodiment, when there is a high possibility that the worker 101 can not recognize the presence of the instruction mark, the determination unit 608 determines that “notification is necessary (necessity of performing the notification process)”.

  Specifically, when the instruction mark has never been displayed on the display unit 103b, the worker 101 can not recognize the presence of the instruction mark, so it is determined that “notification is necessary”. This determination is realized in S51 to S53 described later.

  Further, when the indication mark is displayed on the display unit 103b, the possibility that the worker 101 recognizes as the continuous display time is high is high, but the possibility that the worker 101 recognizes as the continuous display time is short Is low. Therefore, among the instruction marks that have been displayed on the display unit 103b, it is determined that "a notification is necessary" for an instruction mark whose continuous display time (continuous display time including the current frame) is less than or equal to the threshold. This determination is realized in S54 to S56 described later.

  Next, the flow of processing of the determination unit 608 will be described. FIG. 9 is a flow chart showing the flow of processing of the determination unit 608 for the indication mark when a certain indication mark is newly registered.

  When the instruction mark to be processed is registered, the determination unit 608 determines that "notification is required" while the instruction mark is not displayed on the display unit 103b (S51 to S53). This determination process will be described below.

  Assuming that the coordinates of the indication mark in the plane coordinates for specifying the position on the display screen of the display unit 103b are Qm (x, y), the number of vertical pixels of the display unit 103b is h, and the number of horizontal pixels is w, In the case of true, the instruction mark is positioned in the display screen of the display unit 103b, and in the case of false, the instruction mark is out of the display screen of the display unit 103b.

  Therefore, the determination unit 608 determines Qm (x, y) with reference to the position information calculated by the position calculation unit 607 for the current frame. Then, if it is false in the equation 5, that is, if the instruction mark is not located in the display screen of the display unit 103b, the determination unit 608 determines that “notification is necessary” (S52), and the next frame is again processed after S51. Are repeated (YES at S53).

  On the other hand, if it is true in the equation 5, that is, if the indication mark is located in the display screen of the display unit 103b, the indication mark is displayed on the display unit 103b. In this case, the determination unit 608 shifts the process to S54 and subsequent steps.

  In S54 to S56, when the continuous display time of the instruction mark to be processed does not exceed the threshold value, the determination unit 608 determines that “notification is necessary”. This determination process will be described below.

Let T _{m be} the continuous display time of the indication mark (the length of the period continuously displayed from the current frame), and let α _t be the threshold (note that m is the index of the indication mark). If the judgment unit 608 is false in [Equation 6], that is, if the continuous display time of the instruction mark does not exceed the threshold, the judgment unit 608 judges that “notification is necessary” (S55), and repeats S54 and subsequent steps again for the next frame YES at S56). In Equation 6, the value of α _t is not limited, but is empirically set to about 1 second. In addition, the determination unit 608 can, for example, set the period during which the determination of the equation 5 is continued to be true as T _m .

  The continuous display time is the length of a period in which the continuous display is continued from the current frame, as described above. Therefore, after the indication mark is once displayed and the process proceeds to S54, if the indication mark is separated from the display screen again before the continuous display time exceeds the threshold, the indication mark is continuous from the display screen again during the period. The display time is counted as zero and continues to be determined as NO at S54. Then, when the indication mark is displayed again, the continuous display time is counted with the displayed point as the starting point.

  If the determination unit 608 is true in [Equation 6], that is, if the continuous display time of the instruction mark exceeds the threshold, the determination unit 608 determines that “notification is unnecessary (no necessity to perform the above notification process)” in S57. Finish.

  When the instruction mark has never been displayed on the display unit 103 b by the process of FIG. 9 described above, and the display time is extremely short even if the instruction mark has been displayed, the determination unit 608 The indication mark can be determined as "requirement of notification (necessity of performing the notification process)".

  Even if the indication mark is displayed on the display unit 103b, if the moving speed (passing speed) of the indication mark displayed on the display unit 103b is too fast, it is difficult for the worker 101 to recognize the indication mark. It is difficult to read.

  Therefore, in S54, the process of Formula 7 shown below is performed instead of determination using Formula 6 (determination of whether or not the continuous display time of the instruction mark exceeds the threshold). It is also good.

In Expression 7, V _m is the moving speed of the indication mark, and t _m (x, y) is the position (for example, the center pixel) of the indication mark at the timing when the indication mark starts to be displayed on the display unit 103 b. It is a coordinate, and s _m (x, y) is a coordinate at the current frame after n frames from t _m , and a threshold is α _v . Then, in the case of “true” in [Equation 7], that is, when the moving speed of the instruction mark exceeds the threshold, the determination unit 608 determines “notification required” (S55), and repeats S54 and subsequent steps again for the next frame. (YES at S56). On the other hand, when the judgment unit 608 is false in [Equation 7], that is, when the moving speed of the instruction mark does not exceed the threshold, the judgment unit 608 judges that "notification is unnecessary" (S57), and ends the processing.

  Alternatively, in S54, both of the determination of [Equation 6] and the determination of [Equation 7] may be performed. In this embodiment, the judging unit 608 judges that “notification is necessary” when it is false in the equation 6 and true in the equation 7, and in other cases, it is determined as “the notification unnecessary”. It may be Alternatively, in the above-described embodiment, the determination unit 608 determines that “notification is unnecessary” if at least the number 6 is false or the number 7 is true, and in other cases, it is determined as “notification required” It may be

  Alternatively, in FIG. 9, S54 to S56 may be deleted, and in the case of YES at S51, the process may shift to S57 (that is, the determination of Equations 6 and 7 may not be performed).

  The determination unit 608 is configured to transmit each mark information to the communication unit 401 and the display control unit 402 when receiving each mark information of each registered instruction mark from the position calculation unit 607.

  At the same time, when an indication mark is newly registered, determination unit 608 starts the process of FIG. 9 for the indication mark (instruction mark to be processed), and it is determined that “notification is unnecessary” in S57. The process of FIG. Then, while the processing in FIG. 9 continues, if the determining unit 608 determines that “notification is required”, a flag indicating “notification required” is included in the mark information of the instruction mark to be processed. The mark information is transmitted to the communication unit 401 and the display control unit 402.

  On the other hand, if the determination unit 608 determines that "notification is unnecessary" in S57, the processing of FIG. 9 is terminated for the processing target indication mark, and a flag indicating "notification unnecessary" is the processing target indication mark. After being included in the mark information, the mark information is transmitted to the communication unit 401 and the display control unit 402. After that, the judgment unit 608 does not perform notification processing on the mark information even if the mark information is input from the position calculation unit 607 without performing the processing of FIG. And the mark information is transmitted to the communication unit 401 and the display control unit 402.

  When the communication unit 401 receives the mark information from the position calculation unit 607 of the mark tracking unit 407, the communication unit 401 transmits the mark information to the work terminal 103. In the work terminal 103, the communication unit 301 receives the mark information, and the communication unit 301 transmits the mark information to the display control unit 302. The display control unit 302 causes the display unit 103 b to display an instruction mark based on the mark information.

  The display control unit 302 superimposes the indication mark of the mark information including the flag of “notification unnecessary” on the image of the display unit 103 b as it is. For example, as shown in FIG. 1, the instruction mark 105 is displayed on the image of the display unit 103b.

  On the other hand, the display control unit 302 performs a notification process of notifying the operator of the presence of the instruction mark as described below for the instruction mark of the mark information including the flag of "notification required".

  First, the case where the instruction mark is out of the display screen of the display unit 103b will be described. In this case, the display control unit 302 identifies a coordinate value indicating a relative position with respect to the display screen of the display unit 103b based on the position information indicated by the mark information, and notifies the operator of the presence and the position of the instruction mark. The notification information is displayed on the display unit 103b.

  For example, as shown in FIG. 10, a message 550 indicating the position of the indication mark in a sentence may be displayed. Alternatively, information indicating a pattern or direction that can specify the position of the indication mark may be displayed. Further, as shown in FIG. 3 of Patent Document 2, an arrow may be displayed.

  As a result, even if an instruction mark is newly set at a position deviated from the display screen on the display unit 103b, the presence of the instruction mark can be recognized by the notification information. In addition, since the position of the instruction mark is also notified, the operator can quickly display the instruction mark by shifting the imaging range of the work terminal 103. For example, in the case shown in FIG. 10, an instruction mark can be displayed by shifting the imaging range to the right.

  Next, a case where the instruction mark is displayed on the display screen of the display unit 103b, but the mark information includes a flag of "notification required" will be described. In this case, the display control unit 302 specifies a coordinate value indicating a position on the display screen of the display unit 103b based on the position information indicated by the mark information, and displays an indication mark at the position of the coordinate value. Notification information for notifying the operator of the presence and position of the instruction mark is displayed on the display unit 103b.

  For example, as shown in FIG. 11, the indication mark 600 may be displayed on the display unit 103b, and a message 650 may be displayed to notify the operator of the presence and the position of the indication mark 600. Further, instead of the message, information indicating a pattern or direction for notifying the presence and position of the indication mark may be displayed. As a result, even if it is difficult for the operator to recognize the instruction mark 600 due to a short display time of the instruction mark 600, it is possible to quickly notify the user of the presence of the instruction mark 600.

  That is, according to the embodiment described above, the determining unit 608 determines whether there is a need to notify the user (worker) of the work terminal 103 of the presence of the instruction mark, and determines that there is the need The work terminal 103 is caused to execute a notification process of notifying the user of the presence of the instruction mark by outputting the mark information including the “necessity required” flag. Here, the notification process is a process of notifying the worker of the presence of the instruction mark by displaying the messages (notification information) 500 and 650 as shown in FIG. 10 and FIG.

  Thus, for example, since the indication mark is not yet displayed on the work terminal 103 or the display time on the work terminal 103 is short and the worker does not recognize the existence thereof, the presence of the indication mark is given to the worker With regard to an instruction mark that is determined to be highly necessary for notification, the presence of the instruction mark can be promptly notified to the user of the work terminal 103. Therefore, according to the present embodiment, it is possible to ensure the rapidity of information transmission in the telecommunication system.

  Further, in the embodiment described above, even if the instruction mark immediately after being set by the instruction person 107 is at a position deviated from the display screen on the display unit 103b of the work terminal 103, the operator 101 is performing it. It can be said that it is related to the work of That is, it can be said that the instruction mark set most recently is highly important and should be recognized with priority. In addition, the indication mark, whether or not currently displayed on the display screen, has already been recognized by the operator as a result of being displayed in the past, the importance is low, and the operator is still aware of it. Those that do not have high importance.

  In this respect, in the present embodiment, for the indication mark not recognized yet as well as for the indication mark not recognized by the operator due to the short display time or the like, the presence and the position of the indication mark are notified. Since notification information is displayed, quickness of information transmission can be secured.

  From the above, in the telecommunication system A using AR technology, even if the instruction mark is attached to a scene different from the scene of the video displayed on the work terminal 103, the operator 101 quickly recognizes the existence of the instruction mark. It will be possible to In addition, when the indication mark is out of the display screen of the display unit 103b, the indication mark may be near or far from the display screen, but the indication mark not displayed on the display screen is important. The instruction mark can be made to notice the operator 101 quickly.

  Further, in the present embodiment, the determination unit 608 determines whether "instruction required" or "not required" for the instruction mark, but the instruction mark is unread for the worker in the case of "notification required". This means a state of high possibility, and in the case of "notification not required" means a state in which the operator is highly likely to have read the instruction mark. Therefore, the determination unit 608 may determine "unread" instead of determining "notification required", and may determine "read" instead of determining "notification unnecessary".

  Further, in the present embodiment, the mark tracking unit 407 is included in the instruction terminal 108, and the instruction terminal 108 functions as a mark processing device, but the mark tracking unit 407 is included in the management server of FIG. However, it may be included in the work terminal 103. When the mark tracking unit 407 is included in the management server 200, the management server 200 functions as a mark processing device, and when the mark tracking unit 407 is included in the work terminal 103, the work terminal 103 functions as a mark processing device Do.

  In the present embodiment, the work terminal 103 notifies the worker by displaying the notification information for notifying the presence and the position of the instruction mark, but it is not the notification by display but the notification by voice. Of course it may be.

  The telecommunication system A of the present embodiment is configured to include one work terminal 103 and one instruction terminal 108, but it goes without saying that the work terminal (first terminal) 103 or the instruction terminal (second terminal) 108 The number is not limited. That is, the remote communication system A includes a plurality of work terminals (first terminals) 103, and the instruction is transmitted from the instruction terminal (second terminal) 108 to each of the work terminals (first terminals) 103. It may be done. Further, the remote communication system A includes a plurality of instruction terminals (second terminals) 108, and an instruction is transmitted from the plurality of instruction terminals (second terminals) 108 to the work terminal (first terminal) 103. It may be done. Alternatively, the telecommunication system A may include a plurality of instruction terminals (second terminals) 108 and a plurality of work terminals (first terminals) 103.

Second Embodiment
If there are a plurality of indication marks determined to be "notification required", notification information (for example, a message 550 in FIG. 10) for notifying the operator of the presence and position of the indication mark is displayed for all of them. Because there is too much information to notify the workers, efficient information transmission and work support may not be achieved.

  On the other hand, in the work support, the content instructed by the instructor 107 at the present time is the most important and should be prioritized. That is, the set time is more important for the indication mark closer to the present time and needs to be displayed with priority, and the indication mark having the set time away from the present time becomes less important.

  Therefore, in the present embodiment, when there are a plurality of instruction marks determined to be "notify necessary", the instruction mark determined to be "necessary to be notified" so that the instruction mark closer to the current point in time becomes higher in importance. Set the importance for each of. Then, the notification information of a predetermined number or more (for example, 2 or more) is not displayed, and the notification information is preferentially displayed from the instruction mark with high importance (that is, the instruction with high importance) Although the notification information is displayed on the mark, the notification information is not displayed on the less important instruction mark.

  In this case, the mark management unit 606 adds the registration time to the mark information when registering the instruction mark. Then, when there are two or more instruction marks determined as “notification required”, the determination unit 608 inquires the mark management unit 606 about the registration time of these instruction marks, and the importance becomes higher as the registration time is closer to the current time. The importance level is set for each instruction mark.

Assuming that the importance of the instruction mark at a certain time t is H _t , the high / low relation of the importance is set as shown in Equation 8. _{Incidentally, H t, H t-1} , H t-2, H t-3, ..., the order of _{H 0} is the order registration time is close to the current time.

  Further, as a method of setting the degree of importance, any other method can be used, and determination can also be made using the result of analyzing the instruction content. If the instruction mark is accompanied by a sentence (a sentence set by the instructor and indicating the instruction content), the determination unit 608 analyzes the syntax of the sentence to extract a word, and the extracted word is extracted at the work site 100. The matching may be performed with words related to the work being performed, and the importance may be higher as the number of matching is larger. This enables effective instruction mark selection. In addition, when the indicator 107 inputs in advance the category of the current operation and the category of the instruction mark, information indicating these categories is stored in the memory, and the determination unit 608 matches the category with the category of the operation. The degree of importance of the indication mark may be set higher than the degree of importance of the indication mark whose category does not match the category of work.

Third Embodiment
In the first embodiment, the point that the passing speed of the indication mark may be used as the index of the determination by the determination unit 608 is described (Equation 7). In this regard, in everyday life, there is a bias in the direction (direction) of a meaningful character string. Specifically, there are many horizontal writing and vertical writing, but diagonal writing is overwhelmingly less than these directions. Furthermore, the character string with movement has a large difference in readability depending on the direction of the character string (longitudinal, horizontal, diagonal) and the direction of movement (longitudinal, lateral, diagonal). Come. It is empirically known that when the direction of a string matches the direction of its movement, the string becomes easier to read.

  Therefore, in the present embodiment, when the direction of the character string and the direction of movement coincide with each other for the instruction mark accompanied by the character string, the threshold value to be compared with the passing speed is set lower than in the other cases. Control is performed so that the flag of "identification state" can be set more accurately. The method is described below.

  First, the relationship between the combination of the direction of the character string and the direction of movement and the readability is shown in Table 1. In Table 1, "o" indicates that it is easy to read, ".DELTA." Indicates that the readability is standard, and "x" indicates that it is difficult to read. The threshold control is performed according to Table 1.

  FIG. 12 is a view schematically showing an image displayed on the display unit 103 b of the work terminal 103 according to the present embodiment. As shown in FIG. 12, when the indication mark 903 with the horizontal writing character string 903a and the indication mark 902 with the vertical writing character string 902a are shown, the indication mark 903 and the indication mark 902 Different thresholds will be used.

  As shown in Table 1, since the readability is determined by the combination of the direction of the character string and the direction of movement, the threshold value is controlled according to the direction of the character string and the direction of movement.

  As for the direction of the character string, the mark information indicating the indication mark is electronic information, so the determination unit 608 can determine the direction by referring to the mark information.

Subsequently, although it is the direction of movement of the indication mark, a vector indicating the movement direction of the indication mark can be obtained by [Eq. 9]. For example, as shown in FIG. 12, it is assumed that the indication mark 903 displayed at the current frame is drawn at the position 903 'while pausing N frames earlier. Here, coordinates P _{t (x,} y) at the present time indication mark 903 and the coordinates of the indication mark 903 when drawn at a position 903 'during the pause before N frames from the present time P _{t- Assuming that N} (x, y), a vector 905 indicating the moving direction of the instruction mark 903 is D _{t of} Formula 9.

  Subsequently, the threshold will be described. In the present embodiment, the threshold value Th is adjusted (controlled) so that the threshold value Th becomes large when the movement direction of the indication mark matches the direction (longitudinal direction or lateral direction) of the character string attached to the indication mark. Yes (it becomes easy to be judged as "notification unnecessary (already read)" if it becomes large). Specifically, the inner product of the vector in the movement direction of the instruction mark and the vector related to the direction of the character string is obtained, and the product of the inner product and a constant is set as the threshold value Th. Specifically, the threshold Th can be calculated by equation 10.

In Equation 10, normalized (D _t ) is a vector obtained by normalizing D _t in Equation 9. That is, it is a vector of the movement direction of the instruction mark and is a normalized vector.

V _n is a unit vector of character strings (n = 0 or n = 1). If the character example is vertical, the unit vector of the character string is V ₀ = (0, 1), and if the character example is horizontal, the unit vector of the character string is V ₁ = (1, 0) is there. In FIG. 12, the vector denoted by 906 corresponds to V ₀ and the vector denoted by 907 corresponds to V ₁ (that is, the vectors denoted by 906 and 907 in FIG. 12 are for convenience of explanation of unit vectors. And does not indicate the content actually displayed on the display unit 103b).

  β is an arbitrary constant and is a parameter that controls the degree of determination. β may be set to a predetermined fixed value or may be automatically determined by image analysis. The smaller the value of β, the less likely it will be determined that “notification unnecessary (read)” when the indication mark appears on the screen, and the “notification unnecessary (read)” flag is displayed if the work terminal 103 is not moved slowly. I can not stand up.

Next, the flow of processing of the determination unit 608 will be described. First, the determination unit 608 obtains the threshold Th using Equations 9 and 10. Subsequently, the determination unit 608 substitutes the threshold value Th for α _v in Equation 7 and performs the determination in Equation 7.

  According to the above embodiment, the threshold Th is adjusted according to the movement direction of the indication mark and the direction of the character string, and when the direction of the character string matches the movement direction of the indication mark It becomes possible to control so that it becomes easy to be judged as "reading".

Fourth Embodiment
Furthermore, in the remote communication system A, it may be determined whether the indication mark to be processed is "notification required" or "notification unnecessary" according to the state of the worker itself.

  For example, the telecommunication system A of the present embodiment is premised on use at a work site, and it is assumed that a worker actually performs work while touching an object. That is, while holding the work terminal 103 with one hand, the worker may touch the object with the other hand. In that case, the operator's hand will be shown in the image taken by the camera 103 a of the work terminal 103. Here, if the operator's hand and the position of the instruction mark overlap in the video, it is considered that the operator recognizes the instruction mark.

  Therefore, the determination unit 608 inputs the video signal and performs skin color part extraction processing to extract the position of the operator's hand, and while the operator's hand and the position of the instruction mark have never coincided with each other, If it is determined that "notification is necessary" and the hand of the operator and the position of the instruction mark coincide with each other, it may be determined that "notification is unnecessary". The skin color part extraction process may be performed using OpenCV (Computer Vision) which is a general-purpose image processing library. For example, the determination unit 608 performs binarization processing on the image sent from the work terminal 103 using the cv :: Threshold function, and extracts the region of the skin color part of the hand. Since the skin color includes a reddish area, the hue value in the HSV color space falls within the range of 0-30. That is, the determination unit 608 converts the color space of the image into the HSV color space, and extracts the pixel whose numerical value of Hue is within the range by using the above-described binarization processing. Specifically, binarization processing is performed such that a hue value of 0 to 30 (skin color range) is a white pixel, and a hue value of 31 or more is a black pixel, and a pixel that corresponds to the hand is expressed in white. . Furthermore, the determination unit 608 approximates the hand region to a circle using the cv :: convexHull function, and sets the flag of “identification state” when the position of the instruction mark overlaps the approximate region (note that, the above In the processing, the skin color is a white pixel, and the other is a black pixel, but the skin color may be a black pixel and the other may be a white pixel).

  In addition, even if it is an instruction mark which is once determined as "notification unnecessary" by the determination unit 608, it is determined whether or not the worker is gazing at the image in which the instruction mark is indicated, and if not gazing The notification not required may be changed to "notification required". For example, an in-camera (a camera different from the camera 103a) is attached to the front surface side (the display unit 103b side) of the work terminal 103 so that the camera can capture an operator looking into the display unit 103b. It has become. Then, the photographed image of the in-camera of the work terminal 103 is transmitted to the instruction terminal 108, and the determination unit 608 performs image processing on the photographed image (an image showing the face image of the worker), Determines whether or not the user is gazing at the image for which the instruction mark is shown.

  The determination is performed as follows. The determination unit 608 estimates the line of sight using the OpenCV cascade classifier. In the OpenCV cascade classifier, the position and eye area of the worker's eye are estimated using a file previously learned about the eye shape. Then, the determination unit 608 performs binarization processing on the eye region to extract the black eye region, and further extracts the coordinates of the black eye using the cv :: Moments function, but the eye estimated by the cascade classifier If one of the left and right ends of the area overlaps with the coordinates of the eye, it is determined that the screen is not gazed (looking at the direction after the screen), otherwise it is gazed at the screen It is determined that Note that the present invention is not limited to the method of determining that "you are not gazing at the screen" when either of the left and right ends of the eye area overlaps with the coordinates of the eye; for example, the coordinates of the eye and the left and right ends of the eye area It may be determined that “you are not gazing at the screen” if the distance between the two and is not more than the threshold value, and it may be determined as “you are gazing at the screen” otherwise.

[Example of software implementation]
Each block shown in FIGS. 3 to 5 may be realized by software using a CPU, or may be realized by a logic circuit formed in an integrated circuit or the like. In the case of software, the information processing apparatus 108a includes a recording medium such as a ROM or a storage device in which the program that is the software is recorded so as to be readable by a computer (or CPU). The recording medium may be, for example, a “non-transitory tangible medium” such as a card, a disk, a semiconductor memory, a programmable logic circuit, or the like. Further, the program may be transmitted to the computer via any transmission medium (communication network, broadcast wave, etc.). Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Supplementary matter]
The following will supplement the problems of the background art. In the non-patent document 1 described above, it is possible to indicate the position graphically to the operator by the operator, who is the operator, installing a circle, an arrow, or the like on the instruction location on the image transmitted from the operator. . Thereby, even if an ambiguous directive word is used, the position becomes clear by the mark, and no wrinkles occur. In addition, there are merits such as improvement of work efficiency, realization of highly specialized work not depending on skill proficiency, and reduction of work period and cost.

  However, it is necessary for the worker to look at the superimposed instruction content and the work target through the screen, which causes a problem that the field of view is extremely limited. At that time, there are many cases where there is an instruction target out of the screen, that is, out of the field of view, and it is difficult to find the same mark in the same place again if the mark that is now visible disappears off the screen Is an issue.

  Patent Document 1 makes it possible to recognize even outside the screen by displaying guidance information (arrows) indicating the position of an object located outside the screen. Moreover, when there are a plurality of objects outside the field of view, in addition to squeezing and displaying the two nearest objects, the device is made to change the size of the display of the guidance information for each distance.

  In work support between remote locations using video as described above, the instructor generally presents information based on the work location where the worker is currently doing, or the location where conversation or explanation is being made. . However, since it is assumed that the worker is always moving, the instructor must perform a series of operations such as identifying the designated place based on the image which is not intended by his / her operation and installing information in the designated place. You must. At that time, it is very difficult to accurately set the mark on the detail of the object which is unintended movement, and the instructor needs to pause the image to carry out the operation. In the meantime, the worker continues the work and continues to shoot and update the video, so the video that the worker and the instructor are watching will be different scenes in time. This means that when the operator moves the information while setting up the information, the newly set information is out of the screen.

  In telecommunications, immediacy is important, and the transmission of highly important information related to current work is required. That is, even if the instructor installs the thought information as important, as described above, it may happen that the image of the scene the operator is watching differs from the scene of the instructor. In that case, the worker does not know when, where, and what information is installed.

  Patent Document 1 is merely a method of guiding an object that is not currently displayed on the display screen, regardless of whether or not it has already been displayed. Therefore, the problem is that it is difficult to determine whether the object has already been seen or an unknown object. Further, the distance between the object from the display screen and the importance to the user are irrelevant, and in the method of Patent Document 1, the guidance information of the object that is not important to the user is more prominent than the guidance information of the important object. It will be displayed.

[Summary]
Aspect 1 of the present invention includes at least a first terminal (working terminal 103) and a second terminal (instruction terminal 108) that can communicate with each other, and the first terminal and the second terminal can be used to capture an image captured by the first terminal. When an input operation for drawing a mark at the target position shown in the displayed video is performed in the second terminal, the video is displayed in the first terminal and the second terminal. A mark processing apparatus (information processing apparatus) included in the telecommunication system A in which the mark is superimposed and displayed on the target position so as to move following the movement of the target position, to the user of the first terminal The determination unit 608 is configured to cause the first terminal to execute a notification process of notifying the user of the presence of the mark by determining the presence or absence of the necessity of notifying the presence of the mark and determining that the necessity of the notification is present. What And butterflies.

  According to the first aspect of the present invention, when an input operation of the mark is performed, the presence of the mark determined to be highly necessary to notify the user of the first terminal of the presence of the mark is determined. The user of the terminal can be notified promptly. Therefore, according to aspect 1 of the present invention, it is possible to ensure the rapidity of information transmission in the telecommunication system.

  In the mark processing device according to aspect 2 of the present invention, the determining unit 608 determines that the necessity of notification is present when the mark has not been displayed on the first terminal.

  The first terminal and the second terminal are displaying different scenes when the video is being paused while the video being captured is displayed in real time on the first terminal There is. Therefore, when the input operation is performed when the video is paused at the second terminal, the mark may be set at a position deviated from the display screen of the first terminal, in this case, the first mark The user of the terminal is in a situation where the mark can not be viewed, and there is a high need to notify the user of the existence of the mark. In this respect, according to the second aspect of the present invention, in such a situation, the user of the first terminal can be notified of the presence of the mark quickly, and the promptness of information transmission can be secured.

  In the mark processing device of aspect 3 of the present invention, the determination unit 608 determines that there is a need for notification when the continuous display time in which the marks are continuously displayed on the first terminal is less than a first threshold. It is characterized by judging.

  Even if the mark is displayed on the first terminal, if the continuous display time is short, there is a high possibility that the user of the first terminal can not visually recognize the mark, and it is necessary to notify the user of the existence of the mark Sex is high. In this regard, according to aspect 3 of the present invention, in such a situation, the user of the first terminal can be promptly notified of the presence of the mark, and the promptness of information transmission can be secured.

  In the mark processing device of aspect 4 of the present invention, the determination unit 608 determines that the moving speed of the mark displayed on the display unit of the first terminal after the mark starts to be displayed on the first terminal is second If the threshold value is exceeded, it is determined that there is a need for notification.

  Even if the mark is displayed on the first terminal, if the moving speed of the mark displayed on the first terminal is too fast, there is a high possibility that the user of the first terminal can not visually recognize the mark, There is a high need to notify the user of the existence of the mark. In this regard, according to the fourth aspect of the present invention, in such a situation, the user of the first terminal can be promptly notified of the presence of the mark, and the promptness of information transmission can be secured.

  The mark processing device according to aspect 5 of the present invention calculates a displacement vector that calculates a displacement vector indicating the direction and amount of movement of the video with reference to the video signal of the video captured by the first terminal. Update processing unit that updates the position information indicating the position of the mark so that the position of the mark is displaced in the direction and the movement amount indicated by the displacement vector calculated by the unit 602 and the displacement vector calculation unit ( A position calculation unit 607), and the first terminal and the second terminal are configured to display the mark at the position of the position information updated by the update processing unit. Do.

  When the user holding the first terminal moves the first terminal or the user moves itself, the imaging range of the first terminal is shifted, so the images displayed on the first terminal and the second terminal move It will be. In this respect, according to the fifth aspect of the present invention, the position of the mark is displaced so that the position of the mark is displaced in the direction and the movement amount indicated by the displacement vector indicating the movement direction and the movement amount of the image. It is designed to update the indicated position information. Therefore, since the position of the mark is displaced following the movement of the image, the mark attached to the target position shown in the image can be superimposed and displayed without being separated from the target position.

  In the mark processing device of aspect 6 of the present invention, when the input operation is performed, a position information generation unit (mark information generation unit 610) that generates position information indicating the position of the mark according to the input operation; An addition processing unit (displacement unit that adds the displacement vector calculated by the displacement vector calculation unit during the pause period in which the same frame is continuously displayed on the second terminal by inputting the stop command to the second terminal A vector addition unit 605), and the addition processing unit generates the position information in the direction and the movement amount corresponding to the addition result by the addition processing unit when the input operation is performed in the temporary stop period The position information is corrected such that the position of the position information generated by the unit is displaced.

  While the video captured at the first terminal is displayed in real time, while the video is paused at the second terminal, the current frame is not displayed on the second terminal. Therefore, when the input operation is performed on the frame displayed at the second terminal, the target position (the position of the mark) specified in the input operation is adapted to the current frame. It needs to be corrected. In this respect, according to the sixth aspect of the present invention, the motion of the image between the current frame and the paused frame is caused by the addition processing unit adding the displacement vector during the pause period. Is quantified. Then, the position information is corrected such that the position of the position information generated by the position information generation unit is displaced in the direction and the movement amount corresponding to the addition result by the addition processing unit.

  As a result, while the video shot at the first terminal is displayed in real time, while the video is paused at the second terminal, the frame displayed at the second terminal is displayed. Even when the input operation is performed, the target position designated in the input operation can be corrected to conform to the current frame.

  In the mark processing device according to Aspect 7 of the present invention, when the mark includes a character string, the determination unit 608 determines the size of the second threshold according to the moving direction of the mark and the direction of the character string. It is characterized by changing.

  When the mark contains a character string, the readability of the character differs greatly depending on the combination of the direction of the character string and the movement direction of the character string. In this respect, according to the seventh aspect of the present invention, the magnitude of the second threshold is changed according to the moving direction of the mark and the direction of the character string, so that the user can easily recognize the character string. Accordingly, the magnitude of the second threshold is changed, and the determination by the determination unit can be made more appropriate.

  Further, the mark processing device of the aspects 1 to 7 of the present invention may be included in the second terminal.

  Further, the mark processing device according to the aspects 1 to 7 of the present invention may be realized by a computer, and in this case, by operating the computer as each unit included in the mark processing device according to the aspects 1 to 7 of the present invention. Also, a program for realizing the mark processing apparatus by a computer, and a computer readable recording medium recording the same fall within the scope of the present invention.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

103 work terminal (first terminal)
103 b Display unit 103 a Camera 108 Instruction terminal (second terminal)
Information processing apparatus (mark processing apparatus)
108 b Display device 407 mark tracking unit 602 displacement vector calculation unit 605 displacement vector addition unit (addition processing unit)
606 mark management unit 607 position calculation unit (update processing unit)
608 determination unit 610 mark information generation unit (position information generation unit)

Claims (7)

At least a first terminal and a second terminal capable of communicating with each other ,
Image captured by the first terminal is displayed in both the first terminal and the second terminal,
In the second terminal, an input operation to draw the mark at a target position in the image being displayed is carried out, the target in the image displayed by the first terminal and the second terminal A mark processing apparatus included in a telecommunication system in which the mark is superimposed and displayed on the target position so as to move following the movement of the position,
The first in accordance with the display mode of the mark that in terminal, e Bei determination unit to be executed on the first terminal a notification process of notifying the user of the first terminal of the presence of the mark,
The determination unit is configured to continuously display the mark on the first terminal.
If the continuous display time is less than the first threshold value, mark processing apparatus characterized by executing a notification process of notifying the presence of the mark to the user to the first terminal.   The determination unit causes the first terminal to execute a notification process of notifying the user of the presence of the mark when the mark has never been displayed on the first terminal. The mark processing device described in. The determination unit, when the moving speed of the mark which the mark is displayed at the end of the first end from the beginning to be displayed on the first terminal exceeds the second threshold value, before the existence of the mark's rating mark processing apparatus according to any one of claims 1 or et 2, characterized in that to execute the notification process to notify the User chromatography the said first terminal. Wherein the first terminal and the second terminal is either claim 1, characterized in that is adapted to display said mark at a position corresponding to the movement of the image being captured by the first terminal The mark processing apparatus according to any one of [ 3] . The pause command in the pause period the same frame to the second terminal by that input to the second terminal is continuously displayed, the input operation is performed, the second terminal, the current frame 5. The mark processing apparatus according to claim 4, wherein the mark is displayed at a position corresponding to. 4. The apparatus according to claim 3 , wherein, when the mark includes a character string, the determination unit changes the magnitude of the second threshold according to the moving direction of the mark and the direction of the character string. Mark processing apparatus as described. It is a program which operates the mark processing apparatus of any one of Claims 1-6, Comprising: The program which functions a computer as said each part.

Images