JP5489974B2 - Server, remote assist system and method - Google Patents

Description

  The present invention relates to a server, a remote assist system, and a method for providing a remote assist service.

  In recent years, superimposition processing of virtual objects using augmented reality technology, that is, AR (Augmented Reality) technology, has attracted attention. With this AR technology, it is possible to superimpose additional information (annotations) on the real environment using a computer and present them on a display. Typically, the annotation includes explanation and related information regarding a specific object in the real environment, and is presented near the object to be explained.

  On the other hand, nice assist mobile technology for remote operation support between mobile terminals is currently used. According to this nice assist mobile technology, the user can receive remote support using a portable terminal or the like.

JP 2007-279974 A JP 2000-101773 A

  An object of the present invention is to provide a server, a remote assist system, and a method for remotely assisting a device operation method using AR technology.

  In view of the above problems, one feature of the present invention is that an assist request receiving unit that receives an assist request for using a function of an assist target device from an assisted terminal, and a device information storage that stores manual information of the assist target device And in response to receiving the assist request, request a capture image indicating the current state of the assist target device from the assisted terminal, and specify a device image corresponding to the captured image received in the manual information. A device operation determining unit for determining a user operation for executing the function from the state of the assist target device indicated in the specified device image; and an operation instruction image for instructing the determined user operation or the target The assist terminal generates an information element for generating the operation instruction image, and transmits the information element to the assisted terminal. A server having an instruction image generation unit, wherein the device operation determination unit requests a post-operation capture image indicating a state of the assist target device after a user operation specified by the operation instruction image is executed. A server that compares the received post-operation captured image with a target image indicating a state in which the assist target device transitions after the user operation is correctly executed, and determines a next user operation according to the comparison result .

  Another feature of the present invention is that a remote terminal includes an assisted terminal that assists a user in using an assist target device, a server that is communicatively connected to the assisted terminal, and a data center that is communicatively connected to the server. In the assist system, the server obtains from the data center an assist request receiving unit that receives an assist request for using the function of the assist target device from a terminal to be assisted, and the assist target device. A device information storage unit for storing the acquired manual information; and in response to receiving the assist request, request a captured image indicating the current state of the assist target device from the assisted terminal and receive the received manual information. The device image corresponding to the captured image is identified, and the identified device image A device operation determining unit that determines a user operation for executing the function from the state of the assist target device, and an operation instruction image that instructs the determined user operation or the terminal to be assisted receives the operation instruction image. An operation instruction image generation unit that generates an information element for generation and transmits the information element to the assisted terminal, and the device operation determination unit is configured to execute a user operation instructed by the operation instruction image. A post-operation capture image indicating the state of the assist target device is requested, and the received post-operation capture image is compared with a target image indicating a state in which the assist target device shifts after the user operation is correctly executed. The present invention relates to a remote assist system that determines a next user operation according to a comparison result.

  Still another aspect of the present invention is a method in a server for providing a remote assist service, the step of receiving an assist request for using a function of an assist target device from a terminal to be assisted, and reception of the assist request. In response, the captured image indicating the current state of the assist target device is requested to the assisted terminal, the device image corresponding to the captured image received in the manual information stored in the server is identified, and the identification A step of determining a user operation for executing the function from the state of the assist target device indicated in the determined device image, and an operation instruction image for instructing the determined user operation or the assisted terminal receiving the operation instruction An information element for generating an image is generated and transmitted to the assisted terminal. The step of determining the user operation requests a post-operation capture image indicating a state of the assist target device after the user operation specified by the operation instruction image is executed, and has received the received The present invention relates to a method for comparing a captured image after operation with a target image indicating a state in which the assist target device shifts after the user operation is correctly executed, and determining a next user operation according to the comparison result.

  According to the present invention, it is possible to provide a server, a remote assist system, and a method for remotely assisting a device operation method using AR technology.

FIG. 1 shows a remote assist system according to an embodiment of the present invention. FIG. 2 shows a configuration of the assisted terminal according to an embodiment of the present invention. FIG. 3 shows a configuration of an AR server according to an embodiment of the present invention. FIG. 4 is a flowchart showing assist processing according to an embodiment of the present invention. FIG. 5 is a flowchart showing a current state specifying process according to an embodiment of the present invention. FIG. 6 is a flowchart showing an operation determination process according to an embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  A remote assist system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 shows a configuration of a remote assist system according to an embodiment of the present invention. As illustrated in FIG. 1, the remote assist system 100 includes an assist target device 150, an assisted terminal 200, an AR server 300, and a data center 400.

  The assist target device 150 is any device that can realize various functions in response to a user operation or set the device state of the assist target device 150 in response to a user operation. When the user tries to use the function of the assist target device 150 or sets the device state, the user may desire to receive assistance or assistance for the operation method and the setting method. The assist target device 150 is a device that is a target for which the user receives assistance with the operation method and the setting method.

  An example of the assist target device is a personal computer. Typically, in a personal computer, the operating state is changed in accordance with a user operation via an attached mouse or keyboard, and the operating state before and after the change can be visually recognized through the attached display. That is, in the personal computer, when the power is turned on, an initial screen showing a start state is displayed. For example, when a user wants to browse a web page of a site on the Internet, the user moves the pointer over the web browser icon displayed on the initial screen using the mouse and clicks the mouse on the icon. To do. After the web browser is started in response to the click, the user can display a desired web page by inputting the URL (Uniform Resource Locator) of the site in a predetermined area of the web browser using the keyboard. it can. In this case, the personal computer starts from the start state, shifts to a web browser activation state by a mouse movement and click operation by the user, and further displays a web page of a specific site by inputting a URL to the keyboard by the user. Migrate to However, the assist target device 150 according to the present embodiment is not limited to a personal computer. For example, an assist operation device, such as a built-in operation panel, an attached display, a blinking light, a button, or a switch, that changes according to a user operation. It is an arbitrary device in which the state of the target device 150 is visible from the outside.

  The assist target device 150 may receive a user operation instruction via an input device such as a button or a touch panel, or may receive a user operation instruction via an attached input device such as a remote controller, a keyboard, or a mouse. Also good. Further, the assist target device 150 may be capable of receiving an operation instruction from a user via a plurality of operation means.

  The assisted terminal 200 is a device that can visually recognize the visually recognizable state of the assist target device 150 and has an image capture function. Furthermore, the assisted terminal 200 has a communication function of transmitting captured image information to the AR server 300 and receiving various operation instruction images for assisting user operations of the assist target device 150 as will be described later from the AR server 300. . The assisted terminal 200 is a device that can be arranged and / or moved in the vicinity of the assist target device 150 in order to visually capture the state of the assist target device 150. Typically, the assisted terminal 200 is a mobile phone having a camera function, a mobile information terminal, a smartphone, or the like. In order to use the function of the assist target device 150, the user transmits an assist request for receiving an assist of an operation method for executing a desired function to the AR server 300 via the assisted terminal 200. The assist process is started.

  The AR server 300 is communicable with the assisted terminal 200, and upon receiving an assist request from the assisted terminal 200, an operation procedure for executing the function of the assist target device 150 desired by the user indicated in the assist request. And the image information indicating each user operation of the operation procedure is provided to the assisted terminal 200. For example, when the user next needs to click the right button of the mouse, the AR server 300 uses the AR (Augmented Reality) technology to click the right button of the displayed mouse in the image displaying the mouse. An image that superimposes an annotation such as a message, highlight, or animation that prompts the user to be displayed on the mouse is generated and transmitted to the assisted terminal 200. When such an operation instruction image is received via the assisted terminal 200, the user can visually recognize the operation method. The AR server 300 is communicatively connected to a data center 400 functioning as a database for storing manual information in which operation procedures of various devices and images indicating the device states of the devices are recorded, and the manual information of the assist target device 150 is obtained. Get and store.

  The data center 400 is typically a database provided by the manufacturer of the assist target device 150 and holds manual information of various devices manufactured by the manufacturer.

  The manual information includes operation procedure information prepared for each function of the assist target device 150 and image transition information. That is, the manual information includes one or more user operations for executing a certain function of a certain device, and an image indicating a device state before and after the execution of each user operation. For example, if the assist target device 150 is a video device and the function desired by the user is a TV reservation recording function of the video device, typically, the user first sets the TV reservation recording from the remote control in the start state after power-on. Press the function button. When this button is properly pressed, an electronic program guide is displayed on the attached television, prompting the user to select a desired television program. Next, the user designates a desired television program from the electronic program guide. When the television program is properly designated, the designated television program is highlighted in the electronic program guide, and the user is prompted to confirm whether the highlighted television program is correct. Thereafter, when the user designates a confirmation button, a reception completion screen indicating that the designated television program reservation has been set is displayed, and the television reserved recording function is completed. In this case, the operation procedure information of the TV reserved recording function of the video device indicates a transition of a series of user operations of “pressing the TV reserved recording function button → designating a desired TV program from the electronic program guide → designating a confirmation button”. The image transition information is a series of images indicating the state after execution of each corresponding operation of “display image of electronic program guide → highlight display image of designated television program → display image of reception completion screen”. It becomes information to show. Operation procedure information and image transition information for each function of the assist target device 150 are stored in the data center 400 as manual information.

  Next, the configuration of the assisted terminal 200 according to an embodiment of the present invention will be described with reference to FIG. As described above, when the user uses various functions of the assist target device 150, the user-assisted terminal 200 visually guides the user's operation procedure or setting procedure required to execute the function. In order to make the AR server 300 determine whether the user has performed an appropriate operation, the state of the assist target device 150 after the operation or setting by the user is captured, and the captured image is transmitted to the server 300. The assisted terminal 200 is typically any device having an image display function, an image capture function, and a communication function, such as a mobile phone, a portable information terminal, and a smartphone.

  The assisted terminal 200 typically includes one or more of various hardware resources such as an auxiliary storage device, a memory device, a CPU, a communication device, a display device, an input device, and a capture device. The auxiliary storage device is composed of a hard disk, a flash memory, and the like, and stores programs and data for realizing various processes described later. The memory device is composed of a RAM (Random Access Memory) or the like, and reads and stores the program from the auxiliary storage device when an instruction to start the program is given. The CPU functions as a processor for processing information, and realizes various functions described later according to a program stored in the memory device. The communication device includes various communication circuits for wired and / or wireless connection with other devices via a network. The communication device further includes a communication interface for connecting to other devices such as a server on the network. The display device includes a display and the like, and displays data received from the AR server 300. The input device typically includes an operation button, a keyboard, a mouse, and the like, and is used by a user to input various operation commands. The capture device converts an image acquired through a lens into image information by an imaging unit such as a CCD (Charge Coupled Device). The assisted terminal 200 is not limited to the hardware configuration described above, and may have any other appropriate hardware configuration having a function for bidirectional communication with the AR server 300.

  FIG. 2 shows a configuration of the assisted terminal 200 according to an embodiment of the present invention. As illustrated in FIG. 2, the assisted terminal 200 includes a user interface unit 210, an image capture unit 220, and a communication unit 230.

  The user interface unit 210 is realized by various input / output devices such as a display device and an input device. The user interface unit 210 receives various instructions such as an assist request from the user, and transmits them to the AR server 300 via the communication unit 230. In the assist request, the assist target device 150 that the user desires to receive assist and its function are designated. When an operation instruction image or the like is received from the AR server 300 in response to the assist request, the user interface unit 210 displays various image information such as the received operation instruction image to the user. Alternatively, the user interface unit 210 may receive various information elements for generating an operation instruction image from the AR server 300, generate an operation instruction image based on the received information element, and display it to the user.

  The image capture unit 220 is realized by an image capture device or the like, and captures the state of the assist target device 150 visible from the outside. The image capture unit 220 transmits the captured image captured to the AR server 300 via the communication unit 230. The image capture unit 220 captures various code images such as a barcode and a QR code, extracts information from the captured code image, and provides the extracted information to the communication unit 230. For example, in order to transmit the identification information of the assist target device 150 and the operation means such as a remote controller, a mouse, and a keyboard for operating the assist target device 150 to the AR server 300, the user uses the image capture unit 220 to assist the assist target device. The identification information of the assist target device 150 may be read by capturing the QR code attached to 150 or the operation unit, and provided to the AR server 300.

  The communication unit 230 is realized by a communication device, a communication circuit, or the like, and exchanges information with the AR server 300. The communication unit 230 transmits the assist request received from the user interface unit 210, the captured image received from the image capture unit 220, and the like to the AR server 300, and receives various information such as an operation instruction image from the AR server 300. In another embodiment, the communication unit 230 has a short-range wireless communication function for performing short-range wireless communication with a communication unit (not shown) of the assist target device 150, and uses the function to assist the target device. Information indicating the state of the assist target device 150 may be acquired from the communication unit 150 and transmitted to the AR server 300. In this case, the AR server 300 can use the received information indicating the state of the assist target device 150 as reference information, and determine the state of the assist target device 150 in combination with the received captured image. It is possible to determine the state of the assist target device 150 having a high value.

  Next, the configuration of the AR server 300 according to an embodiment of the present invention will be described with reference to FIG. As described above, when the AR server 300 receives an assist request from the assisted terminal 200, the AR server 300 receives one or more user operations necessary for executing the function of the assist target device 150 desired by the user indicated in the assist request. An operation instruction image indicating the user operation is provided. The operation procedure is determined based on manual information provided by the manufacturer of the assist target device 150, and one or more user operations from the current state of the assist target device 150 to completion of the requested function are determined. . The AR server 300 determines the current state of the assist target device 150 and provides an operation instruction image that visually represents the user operation to be operated by the user to the assisted terminal 200. This operation instruction image is composed of an image that allows the user to visually recognize the operation via the assisted terminal 200 using the AR technology. When the user operation notified from the AR server 300 is executed and a captured image indicating the state of the assist target device 150 after the execution is received from the assisted terminal 200, the AR server 300 receives the received captured image and the correct operation by the user. Is compared with a target image indicating a transition state when the image is executed, and it is determined that the user has performed a correct operation when the images match. Based on the determination result, the AR server 300 notifies the assisted terminal 200 of an instruction for the next user operation. The AR server 300 repeats the above-described processing until it receives capture information corresponding to the final target image related to the operation procedure for executing the requested function from the assisted terminal 200.

  The AR server 300 is typically composed of one or more of various hardware resources such as an auxiliary storage device, a memory device, a CPU, and a communication device. The auxiliary storage device is composed of a hard disk, a flash memory, and the like, and stores programs and data for realizing various processes described later. The memory device is composed of a RAM (Random Access Memory) or the like, and reads and stores the program from the auxiliary storage device when an instruction to start the program is given. The CPU functions as a processor for processing information, and realizes various functions described later according to a program stored in the memory device. As will be described later, the AR server 300 includes any appropriate image analysis program such as an image matching program for determining whether the captured image acquired from the assisted terminal 200 matches each target image. Further, the AR server 300 includes any appropriate image processing program for realizing an AR technique for superimposing an operation method annotation on the operation unit of the assist target device 150 as described above. The communication device includes various communication circuits for connecting to other devices via a network. The communication device typically includes a communication interface for connecting to various networks. The AR server 300 is not limited to the hardware configuration described above, and has any other appropriate hardware configuration having a function for bidirectional communication with the assisted terminal 200 via a network. Also good.

  FIG. 3 shows a configuration of the AR server 300 according to an embodiment of the present invention. As illustrated in FIG. 3, the AR server 300 includes an assist request receiving unit 310, a device information storage unit 320, a device operation determination unit 330, and an operation instruction image generation unit 340.

  The assist request receiving unit 310 receives an assist request from the user via the assisting terminal 200. The assist request designates an assist target device 150 that the user desires to receive assistance and its function. When receiving an assist request from the assisting terminal 200, the assist request receiving unit 310 extracts the assist target device 150 and the function that the user desires to assist from the received assist request, and transmits the assist target device 150 to the device operation determining unit 330. When the assist target device 150 can be operated via a plurality of operation means (for example, the operation panel of the device main body and the attached remote controller or mouse), the assist request receiving unit 310 specifies the operation means used by the user. The operation means identification information to be received may be received together.

  In this embodiment, the user designates the assist target device 150 and a desired function and transmits an assist request to the AR server 300, but the present invention is not limited to this. In another embodiment, the user designates only the assist target device 150 and transmits an assist request to the AR server 300, and the AR server 300 refers to the manual information stored in the device information storage unit 320 or the data center 400. Thus, the functions available to the user in the designated assist target device 150 are confirmed. After confirming the available functions, the AR server 300 presents menu information indicating these functions to the user via the assisted terminal 200, and allows the user to select a desired function from each function presented in the menu information. Thus, the function desired by the user may be specified. As described above, any appropriate method that enables the AR server 300 to specify the assist target device 150 and the function desired by the user can be used.

  The device information storage unit 320 stores manual information including operation procedure information and image transition information of various assist target devices provided by each manufacturer, and requested by other components of the AR server 300 in response to a request. Provide manual information for assistive devices. When the manual information of the requested assist target device is held, the device information storage unit 320 provides the manual information. On the other hand, when the manual information of the requested assist target device is not held, the device information storage unit 320 accesses the data center 400 of the requested assist target device manufacturer and acquires the manual information of the assist target device. .

  As described above, each user operation of the operation procedure is executed together with the operation procedure information indicating the operation procedure including one or more user operations for executing various functions available in the assist target device 150. The image transition information indicating the transition of the image indicating the state of the assist target device 150 is included. For example, in the case of a TV reservation recording function of a video device, typically, the operation procedure information is composed of “pressing a TV reservation recording function button → designating a desired TV program from an electronic program guide → designating a confirmation button”. The image transition information is composed of “display image of electronic program guide → highlight display image of designated television program → display image of reception completion screen”. Typically, the operation procedure information corresponding to each function indicates an operation procedure from the start state after power-on of the assist target device 150, and the image transition information corresponding to the operation procedure information is an image ( For example, in a video device, an image transition from a screen displayed after power-on is shown. In this case, each operation procedure corresponds to a tree-like path with the start state as the root and each device state as a node, and each user operation is performed from the device state node before the operation execution to the device state after the operation execution. It can be regarded as an edge connecting nodes. In this case, each image transition is a sequence of one or more images indicating each device state that transitions from the start state, and each image is transitioned by the assist target device 150 when the user performs a correct operation in the immediately preceding device state. This is the target image indicating the state.

  Upon receiving the assist target device 150 and the function that the user desires for assistance extracted by the assist request receiving unit 310, the device operation determining unit 330 accesses the device information storage unit 320 and receives the manual of the received assist target device 150. Get information. Further, the device operation determination unit 330 requests a capture image indicating the current state of the assist target device 150 from the requesting terminal 200 to confirm the current state of the assist target device 150. When the captured image is received from the terminal 200 to be assisted, the device operation determination unit 330 obtains which state of the assist target device 150 the received captured image corresponds to using any appropriate image matching program. Determine by referring to the manual information. The device operation determination unit 330 determines the next user operation necessary for executing the function requested by the user with reference to the acquired manual information from the determined state of the assist target device 150. Specifically, when the determined state of the assist target device 150 is on the operation procedure necessary for executing the function requested by the user, the device operation determination unit 330 determines the determined state on the operation procedure. The user operation for transitioning from one to the next state is determined. On the other hand, when the determined state of the assist target device 150 is not on the operation procedure for executing the function requested by the user, for example, the device operation determination unit 330 first returns from the determined state to the start state. Then, the operation procedure for executing the function is determined again from the start state. The device operation determination unit 330 transmits the determined user operation to the operation instruction image generation unit 340.

  Thereafter, the operation instruction image generated by the operation instruction image generation unit 340 is transmitted to the assisted terminal 200, and the user operates the access target device 150 according to the image, but the device operation determination unit 330 performs the assist after the operation is performed. The user is requested to capture an image indicating the state of the target device 150 and transmit the captured image to the AR server 300. When the captured image after execution of the requested operation is received, the device operation determination unit 330 compares the received captured image indicating the state after the execution of the operation with the target image of the image transition information, and the operation instructed by the user Determine whether or not was executed correctly. In other words, when the received captured image matches the target image, the device operation determination unit 330 determines that the operation instructed by the user has been correctly executed, and performs the next user operation on the operation procedure as the operation instruction image generation unit. 340 is notified. On the other hand, if the received captured image and the target image do not match, the device operation determination unit 330 determines that the operation instructed by the user has not been correctly executed, and performs a user operation for returning to the previous state as the operation instruction image. The data is transmitted to the generation unit 340. Specifically, the device operation determination unit 330 searches for a device image corresponding to the captured image from the manual information, and performs a user operation for returning from the state corresponding to the detected device image to the device state before execution of the operation. The operation instruction image generation unit 340 is instructed. For example, when the assist target device 150 is a video device, a user operation for pressing a “return” button or the like on the remote controller is instructed. The device operation determination unit 330 performs the above-described processing until the state of the assist target device 150 reaches the final state of the operation procedure for executing the requested function, that is, until the captured image matches the final target image. repeat.

  When receiving a user operation to be executed next by the user from the device operation determining unit 330, the operation instruction image generating unit 340 generates an operation instruction image indicating the operation method of the received user operation in the operation unit of the assist target device 150. . Using the AR technology, the operation instruction image generation unit 340 highlights an operation part to be operated by the user on the image indicating the operation unit, or displays a button to be pressed by the user on the image indicating the operation unit. An image that visually displays an operation to be performed by the user is generated by superimposing various annotations such as an arrow display. At this time, when the assist target device 150 has a plurality of operation means (mouse and keyboard), an operation instruction image corresponding to the operation means used by the user is generated. Further, when the AR server 300 receives the identification information of the operation means used by the user, the operation instruction image may be generated using the image of the operation means. For example, when the user uses a specific keyboard, an operation instruction image may be generated by acquiring an image of the keyboard and superimposing an annotation on the acquired image. The operation instruction image generation unit 340 transmits the generated operation instruction image to the assisted terminal 200. Note that the operation instruction image generation unit 340 may capture an image indicating the state of the assist target device 150 after the user operation and embed a message or the like for requesting the user to reply to the AR server 300 in the operation instruction image. Good. Further, the operation instruction image is not limited to a still image, and may be a moving image such as an animation showing a user's operation method in time series.

  In another embodiment, the operation instruction image generation unit 340 receives the user operation to be executed next by the user from the device operation determination unit 330 so that the assisted terminal 200 can generate the operation instruction image. An image (including a 3D object) indicating an operation method of the operation and information for specifying a part related to the operation of the assist target device 150 or the operation unit may be transmitted to the assisting terminal 200. When the information element for generating these operation instruction images is received, the assisted terminal 200 uses the information for specifying the portion of the assist target device 150 or the operation unit related to the operation, and uses the assist target device 150 or the operation unit. The operation method may be guided to the user by specifying a part related to the operation and superimposing and displaying an image showing the operation method of the user operation on the image showing the specified part.

  Next, with reference to FIG. 4, an assist process in the AR server 300 according to an embodiment of the present invention will be described. FIG. 4 is a flowchart showing assist processing in the AR server 300 according to an embodiment of the present invention.

  The process is started when the AR server 300 receives an assist request from the assisted terminal 200. As shown in FIG. 4, in step S401, the assist request receiving unit 310 identifies an assist target device 150 and a function that the user desires to receive assistance from the received assist request, and identifies the identified assist target device. 150 and the function are notified to the device operation determination unit 330.

  In step S <b> 403, the device operation determination unit 330 acquires manual information of the identified assist target device 150 from the device information storage unit 320. If the device information storage unit 320 does not have the manual information, the device information storage unit 320 acquires the manual information from the data center 400 of the manufacturer of the identified assist target device 150, and the device operation determination unit 330. To provide.

  In step S405, the device operation determination unit 330 requests the assisted terminal 200 for an image indicating the current state of the assist target device 150, and specifies the current state of the assist target device 150 based on the acquired captured image and manual information. . The current state specifying process in step S405 will be described in detail later with reference to FIG.

  In step S407, the device operation determination unit 330 determines the next user operation necessary for executing the requested function based on the current state of the identified assist target device 150 and the manual information. The operation determination process in step S407 will be described in detail later with reference to FIG. The device operation determination unit 330 notifies the operation instruction image generation unit 340 of the determined next user operation.

  In step S409, the operation instruction image generation unit 340 generates an operation instruction image corresponding to the received next user operation and transmits the operation instruction image to the assisted terminal 200. When receiving the operation instruction image from the AR server 300, the assisted terminal 200 displays the image to the user. The user executes the operation means of the assist target device 150 according to the operation instruction image displayed on the assisted terminal 200, and then uses the assisted terminal 200 to capture the state of the assist target device 150 after the execution of the operation. The captured image is transmitted to the AR server 300.

  In step S411, the device operation determination unit 330 determines whether the captured image received from the assisted terminal 200 matches the final target image of the image transition information related to the function requested by the user. If the received captured image matches the final target image (S411: YES), the device operation determination unit 330 determines that all operations for executing the requested function have been completed, and ends the assist process. . On the other hand, if the received captured image does not match the final target image (S411: NO), the flow moves to step S405, and the device operation determination unit 330 determines the current state of the assist target device 150 based on the received captured image. Is identified.

  Next, the current state specifying process according to an embodiment of the present invention will be described with reference to FIG. FIG. 5 is a flowchart showing a current state specifying process according to an embodiment of the present invention.

  The flow starts after execution of step S403 in FIG. As illustrated in FIG. 5, in step S <b> 501, the device operation determination unit 330 requests an image indicating the current state of the assist target device 150 from the assisted terminal 200.

  In step S503, upon receiving a captured image that captures the current state of the assist target device 150 from the assisted terminal 200, the device operation determination unit 330 searches for manual information and receives the captured image received from the device image included in the manual information. The device image that matches is identified. When the received captured image is a captured image after execution after the user operates according to the operation instruction image from the AR server 300, that is, the captured image acquired in the loop from step S405 to S411 after the second time. When the user performs a correct operation, the captured image matches the target image corresponding to the operation. On the other hand, when the user performs an incorrect operation, the captured image does not match the target image corresponding to the operation. , Will match other equipment images in the manual information.

  In step S505, the device operation determination unit 330 refers to the manual information and determines the state corresponding to the specified device image as the current state of the assist target device 150. For example, when the user performs a correct operation, the current state is a state corresponding to the target image that matches the captured image, and when the user performs an incorrect operation, the current state is the captured image detected in the manual information. The state corresponds to the matching device image, that is, any state that can be changed from the state before execution of the user operation.

  Next, an operation determination process according to an embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing an operation determination process according to an embodiment of the present invention.

  The flow starts after execution of step S405 in FIG. As illustrated in FIG. 6, in step S601, the device operation determination unit 330 refers to manual information and specifies operation procedure information and image transition information for a function requested by the user.

  In step S603, the device operation determination unit 330 determines whether the current state of the assist target device 150 specified in step S405 is on the operation procedure of the specified operation procedure information. Here, whether or not “the identified current state is in the operation procedure” will be described using a specific example in which the assist target device 150 is a video device and the TV reservation recording function is requested. For example, when the current state of the identified assist target device 150 is a TV electronic program guide display and waiting for the user to specify a TV program, the current state of the assist target device 150 is the TV reservation recording function. It can be determined that it is on the operating procedure. On the other hand, when the current state of the assist target device 150 displays a list of recorded TV programs and is waiting for designation from the user, the current state of the assist target device 150 is, for example, for a recorded program playback function. It can be determined that it is not on the operation procedure for the TV reservation recording function. That is, as described above, the operation procedure corresponds to a path from the start state to the final target state, and it is determined whether or not the current state is on this path.

  If the current state of the assist target device 150 is on the operation procedure (S603: YES), that is, if the user has performed a correct operation, the flow proceeds to step S605, and the next step on the operation procedure from this current state. The user operation for shifting to the state is determined. On the other hand, if the current state of the assist target device 150 is not in the operation procedure (S603: NO), that is, if the user performs an incorrect operation, the flow moves to step S607, and the current state is changed to the operation procedure. Determine the next user action to return. This determination can be performed by determining which function's operation procedure the current state is on, going backward in the determined operation procedure, and returning to the state immediately before the operation execution. If the user operation has not been executed yet, such as after receiving an assist request, there is a high possibility that the state before the operation is not in the operation procedure. In this case, the user may be instructed to return to the start state from the specified current state, and an operation procedure for executing the requested function again after returning to the start state may be instructed.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to the specific embodiment mentioned above, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

100 Remote Assist System 150 Assisted Device 200 Assisted Terminal 300 AR Server 400 Data Center

Claims (9)

An assist request receiving unit that receives an assist request for using the function of the assist target device from the assisted terminal;
A device information storage unit for storing manual information of the assist target device;
In response to receiving the assist request, request a captured image indicating the current state of the assist target device to the assisted terminal, identify a device image corresponding to the captured image received in the manual information, and A device operation determining unit that determines a user operation for executing the function from the state of the assist target device indicated in the device image that is displayed;
An operation instruction image for instructing the determined user operation or an information element for the assisted terminal to generate the operation instruction image, and an operation instruction image generating unit for transmitting to the assisted terminal;
A server having
The device operation determination unit requests a post-operation capture image indicating the state of the assist target device after the user operation specified by the operation instruction image is executed, and the received post-operation capture image and the user A server that compares a target image indicating a state in which the assist target device shifts after an operation is correctly executed, and determines a next user operation according to the comparison result.   The server according to claim 1, wherein the operation instruction image generation unit generates the operation instruction image by superimposing an annotation indicating the determined user operation on an image of an operation unit of the assist target device. The manual information is
Operation procedure information indicating an operation procedure including one or more user operations for executing each function from the start state of the assist target device;
Transition image information indicating an image sequence that is ordered from the device images indicating the start state of the assist target device and includes the device image before execution of each user operation of the operation procedure and the target image after execution;
The server according to claim 2, comprising:   The device operation determination unit determines whether the post-operation capture image matches the target image. If the post-operation capture image matches the target image, the device operation determination unit follows the target image in the image transition information. The server according to claim 3, wherein a user operation of the operation procedure information for transitioning to a next target image is determined as the next user operation.   The apparatus operation determination unit determines, as the next user operation, a user operation for returning to a state before the user operation is executed when the captured image after operation and the target image do not match. 4. The server according to 4.   The server according to claim 5, wherein the device operation determination unit continues to determine the next user operation until the post-operation capture image matches a final target image of the image transition information.   When the assist request receiving unit receives the assist target device, the assist request receiving unit refers to the manual information, transmits menu information indicating one or more functions available in the assist target device to the assisted terminal, and desires The server according to claim 1, which allows a user to select a function. An assisted terminal that assists the user in using the assist target device;
A server that is communicatively connected to the assisted terminal;
A data center communicatively connected to the server;
A remote assist system comprising:
The server
An assist request receiving unit that receives an assist request for using the function of the assist target device from an assisted terminal;
Obtaining manual information of the assist target device from the data center, a device information storage unit for storing the acquired manual information,
In response to receiving the assist request, request a captured image indicating the current state of the assist target device to the assisted terminal, identify a device image corresponding to the captured image received in the manual information, and A device operation determining unit that determines a user operation for executing the function from the state of the assist target device indicated in the device image that is displayed;
An operation instruction image for instructing the determined user operation or an information element for the assisted terminal to generate the operation instruction image, and an operation instruction image generating unit for transmitting to the assisted terminal;
Have
The device operation determination unit requests a post-operation capture image indicating the state of the assist target device after the user operation specified by the operation instruction image is executed, and the received post-operation capture image and the user A remote assist system that compares a target image indicating a state in which the assist target device shifts after an operation is correctly executed, and determines a next user operation according to the comparison result. A method in a server for providing a remote assist service,
Receiving an assist request for using the function of the assist target device from the assisted terminal;
In response to receiving the assist request, the captured image indicating the current state of the assist target device is requested to the assisted terminal, and the device image corresponding to the captured image received in the manual information stored in the server Determining a user operation for executing the function from the state of the assist target device indicated in the specified device image;
Generating an operation instruction image for instructing the determined user operation or an information element for the assisted terminal to generate the operation instruction image, and transmitting the information element to the assisted terminal;
Have
The step of determining the user operation requests the post-operation capture image indicating the state of the assist target device after the user operation specified by the operation instruction image is executed, and the received post-operation capture image, A method of comparing a target image indicating a state in which the assist target device shifts after the user operation is correctly executed, and determining a next user operation according to the comparison result.

Images