JP6969430B2 - User support devices, user support methods, programs, and information processing systems - Google Patents

Description

The present invention relates to a user support device, a user support method, a program, and an information processing system.

Currently, users must perform multiple different procedures using information processing devices such as automated teller machines (ATMs) installed at banks or convenience stores or ticket issuing machines installed at stations. Can be done. However, some of the procedures in these information processing devices require esoteric operations. Therefore, these ATMs or ticket issuing machines may support users by providing guidance by voice output or image display.

The following Patent Document 1 is disclosed with respect to a system for providing such guidance. In Patent Document 1, the facial expression and body movement of the user of the information processing terminal are detected from the image information, and the presentation of information by image or voice to the user is controlled based on the mental and physical condition determined based on the detection result. The technology to do is described.

Japanese Unexamined Patent Publication No. 2005-237561

In the technique disclosed in Patent Document 1 described above, the mental and physical state used for controlling the provision of information is limited to the mental and physical state of one user who is currently operating the information processing apparatus. However, for example, in an information processing device such as a ticket issuing machine or an ATM, one information processing device is sequentially operated by a plurality of users, and a user who is scheduled to operate after the currently operating user is not required to operate the information processing device. For example, it may form a queue. In this way, when one information processing device is sequentially operated by a plurality of users, not only the user who is currently operating the information processing device but also the queue for using the information processing device is placed. It is desirable to support users by reflecting the situation of people in line.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is not only the physical and mental state of each user operating the information processing apparatus, but also the use in a queue. It is an object of the present invention to provide a new and improved user support device, a user support method, a program, and an information processing system capable of supporting a user by reflecting the situation of the user.

In order to solve the above problems, according to a certain viewpoint of the present invention, the information processing device is a user support device for supporting the operation of an information processing device that is sequentially operated by a plurality of users. Acquisition to acquire the first mental and physical state information indicating the mental and physical state of the first user to be operated, and the queue information regarding the second user who operates the information processing apparatus after the first user. User support including a unit, and a support determination unit for selecting a support method for operating the information processing apparatus by the first user based on a determination based on the first mental and physical condition information and the queue information. Equipment is provided.

Further, the support determination unit may select the support method by using the user support determination table that associates the first mental / physical state information and the queue information with the support method.

A support control unit that controls the support of the first user may be further provided according to the selected support method.

The support control unit may generate support information regarding operation support of the information processing apparatus by the first user according to the selected support method.

The support information generated by the support control unit may include information indicating the selected support method.

The support information generated by the support control unit may include information indicating past operations by the first user.

The support information generated by the support control unit may include information indicating past exchanges with the first user.

When the selected support method is support by an agent, the support information generated by the support control unit is used to display the agent supporting the operation by the first user on the information processing apparatus. It may include agent control information.

The first mental and physical condition information may include information indicating the degree of confusion of the facial expression of the first user.

The queue information may include information on the number of the second users.

The queue information may include a second mental and physical condition information indicating the mental and physical condition of the second user.

The second mental and physical condition information may include information on the number of people who are highly dissatisfied among the second users.

Further, in order to solve the above problems, according to another viewpoint of the present invention, it is a user support method for supporting the operation of information processing devices sequentially operated by a plurality of users, and the above-mentioned information. The first mental and physical state information indicating the mental and physical state of the first user who operates the processing device, and the queue information regarding the second user who operates the information processing device after the first user. The user, including the acquisition, and the selection of the support method for the operation of the information processing apparatus by the first user based on the determination based on the first mental and physical state information and the queue information. Support methods are provided.

Further, in order to solve the above problems, according to another viewpoint of the present invention, the first method of operating the information processing device on a computer that supports the operation of the information processing device sequentially operated by a plurality of users. The function of acquiring the first mental and physical condition information indicating the mental and physical condition of the user, and the queue information regarding the second user who operates the information processing apparatus after the first user, and the first. A program for realizing a function of selecting a support method for operating the information processing apparatus by the first user based on a determination based on the mental and physical state information of 1 and the queue information is provided.

Further, in order to solve the above-mentioned problems, according to another viewpoint of the present invention, the information processing apparatus operated sequentially by a plurality of users, the information processing apparatus is operated after the first user who operates the information processing apparatus. To recognize the queue formed by the second user who operates the information processing device and to support the operation of the queue recognition device and the information processing device that generate the queue information about the second user. An information processing system including a user support device and a network that interconnects the information processing device, the queue recognition device, and the user support device, wherein the user support device is the first. Based on the acquisition unit for acquiring the first mental and physical state information indicating the physical and mental state of the user, the queue information regarding the second user, the first mental and physical state information, and the queue information. Based on the determination, an information processing system including a support determination unit for selecting a support method for operating the information processing apparatus by the first user is provided.

As described above, according to the present invention, not only the mental and physical condition of each user operating the information processing apparatus but also the situation of the users in the queue is reflected to support the user. Is possible.

It is explanatory drawing which shows an example of the schematic structure of the information processing system 1 by one Embodiment of this invention. It is a block diagram which shows an example of the structure of the information processing apparatus 100 which concerns on the same embodiment. It is a figure which shows an example of the screen which the output control unit 112 displays on the display unit 150. It is a figure which shows the example of the plurality of feature points obtained from the face image of the user U1. It is a figure for explaining the relationship between the neutral facial expression and the embarrassed facial expression. It is a block diagram which shows an example of the structure of the matrix recognition apparatus 200 which concerns on the same embodiment. It is a block diagram which shows an example of the structure of the monitoring apparatus 300 which concerns on the same embodiment. It is a block diagram which shows an example of the structure of the operator terminal 400 which concerns on the same embodiment. It is a block diagram which shows an example of the structure of the user support apparatus 500 which concerns on the same embodiment. It is a table which shows an example of the user support determination table stored in the storage unit 520. It is a flowchart which shows the process flow of the information processing system 1 by the same embodiment. It is a figure which shows an example of the screen displayed on the display unit 150 when a support method is selected. It is a figure which shows an example of the screen displayed on the display unit 150 when the support by an agent is performed. It is a figure which shows an example of the screen which connects to an operator. It is a figure which shows an example of the screen displayed on the display unit 150 when the support by an operator is performed. It is a block diagram which shows an example of the structure of the queue recognition apparatus 200 which concerns on modification 1. FIG. It is a table which shows an example of the user support determination table stored in the storage part 520 which concerns on this modification. It is a flowchart which shows the process flow of this modification. It is explanatory drawing which shows the hardware composition of the information processing apparatus 1000 which concerns on embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to particularly distinguish each of the plurality of components having substantially the same functional configuration, only the same reference numerals are given.

<< 1. Information processing system overview >>
First, with reference to FIG. 1, an outline of the information processing system 1 according to the embodiment of the present invention will be described. FIG. 1 is an explanatory diagram showing an example of a schematic configuration of the information processing system 1. The information processing system 1 includes an information processing device 100, a queue recognition device 200, a queue image pickup unit 250, a monitoring device 300, a communication terminal 350, an operator terminal 400, a user support device 500, and a network 600.

The information processing device 100 is an information processing device that is sequentially operated by a plurality of users. In this embodiment, an example in which the information processing device 100 is a ticket issuing machine installed at a station will be described. The information processing apparatus 100 determines the mental and physical state of the user U1 based on the captured image of the user U1 who operates the information processing apparatus 100 acquired by the imaging of the imaging unit 170. Further, the information processing device 100 transmits information indicating the mental and physical condition of the user U1 (hereinafter referred to as first mental and physical condition information) to the user support device 500 via the network 600. The detailed configuration of the information processing apparatus 100 will be described later with reference to FIG.

In the example shown in FIG. 1, users U2 to Un (second user) who are scheduled to operate the information processing device 100 after the user U1 (first user) who is operating the information processing device 100. ) Are lined up behind the user U1 to form a queue Q. However, the queue Q is not limited to the example formed immediately behind the user U1, and may be formed at a place away from the user U1. For example, the queue Q is different from the space where the information processing apparatus 100 exists. It may be formed in the space (another room, etc.).

The queue recognition device 200 recognizes the queue Q based on the image of the queue Q acquired by the imaging of the queue image pickup unit 250, and forms the queue Q. Users U2 to Un (second). Generates queue information about (users). The queue information may include, for example, the number of users U2 to Un included in the queue Q. Further, the queue recognition device 200 transmits the queue information to the user support device 500 via the network 600. The detailed configuration of the queue recognition device 200 will be described later with reference to FIG.

The queue image pickup unit 250 is connected to the queue recognition device 200, and provides the queue recognition device 200 with an image captured by capturing the queue Q formed by the users U2 to Un. The queue image pickup unit 250 is arranged at a position and direction in which an image captured by the queue Q can be acquired, and preferably can image the entire range in which the queue Q is expected to be formed. Be placed. For example, when there are many users U2 to Un forming the queue Q and it is assumed that it is difficult for one queue image pickup unit 250 to image the entire queue Q, a plurality of queue image captures are performed. A unit 250 may be arranged to provide a plurality of captured images to the queue recognition device 200.

The monitoring device 300 is installed, for example, at a station where the information processing device 100 is located. When the monitoring device 300 receives the support information indicating the support by the station staff from the user support device 500 via the network 600, the monitoring device 300 transmits an instruction to support the user U1 to the communication terminal 350 of the station staff E1. The detailed configuration of the monitoring device 300 will be described later with reference to FIG. 7.

As shown in FIG. 1, the communication terminal 350 is a terminal connected to the monitoring device 300 and possessed by the station staff E1. When the communication terminal 350 receives an instruction to support the user U1 of the information processing device 100 from the monitoring device 300, the communication terminal 350 displays a screen for guiding the station employee E1 to the information processing device 100 according to the instruction.

The operator terminal 400 is a terminal used by the operator O1, and for example, the operator terminal 400 may be installed in a remote place away from the station where the information processing apparatus 100 is present, for example. The operator terminal 400 has a so-called TV telephone function, and when it receives support information indicating support by an operator from the user support device 500 via the network 600, the operator terminal 400 is connected to the information processing device 100 via the network 600. Realize support using the TV phone function. The detailed configuration of the operator terminal 400 will be described later with reference to FIG.

The user support device 500 is a device for supporting the operation of the information processing device 100. The user support device 500 receives (acquires) the first mental and physical state information from the information processing device 100 and receives (acquires) the queue information from the queue recognition device 200 via the network 600. Further, the user support device 500 makes a determination based on the first mental and physical state information and the queue information, thereby supporting the operation of the information processing device 100 by the user U1 (hereinafter, also simply referred to as a support method). Call) is selected. Further, the user support device 500 controls the support of the user U1 according to the selected support method. The detailed configuration of the user support device 500 will be described later with reference to FIG.

The network 600 is a wired or wireless communication network that connects the information processing device 100, the queue recognition device 200, the monitoring device 300, the operator terminal 400, and the user support device 500. For example, the network 600 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), a WAN (Wide Area Network), and the like. Further, the network 600 may include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

The outline of the information processing system 1 according to the embodiment of the present invention has been described above. According to the information processing system 1 according to the present embodiment, the support method is provided based not only on the mental and physical condition of the user U1 operating the information processing apparatus 100 but also on the conditions of the users U2 and Un lined up in the queue Q. It is possible to select and assist the user. Hereinafter, the configurations of the information processing device 100, the queue recognition device 200, the monitoring device 300, the operator terminal 400, and the user support device 500 included in the above-mentioned information processing system 1 will be described in more detail.

<< 2. Configuration of each device >>
<2-1. Information processing device>
FIG. 2 is a block diagram showing an example of the configuration of the information processing apparatus 100 shown in FIG. The information processing device 100 includes a control unit 110, a storage unit 120, a speaker 130, a microphone 140, a display unit 150, an operation unit 160, an image pickup unit 170, and a communication unit 180.

The control unit 110 has a function of controlling each configuration of the information processing apparatus 100. Further, as shown in FIG. 2, the control unit 110 also functions as an output control unit 112, an operation processing unit 114, and an image processing unit 116.

The output control unit 112 controls the output by the information processing device 100. The output control unit 112 causes, for example, the speaker 130 to output a guidance voice or the like. Further, the output control unit 112 can function as a display control unit that causes the display unit 150 to display a screen related to the service provided by the information processing device 100 and a screen based on the support information received from the user support device 500 described later. ..

FIG. 3 is a diagram showing an example of a screen displayed on the display unit 150 by the output control unit 112. As described above, in the present embodiment, since the information processing device 100 is a ticket issuing machine installed at the station, the output control unit 112 issues a ticket as shown in FIG. 3 (information processing device 100). Display the screen for (an example) of the service provided by the display unit 150 on the display unit 150.

The screen shown in FIG. 3 includes an operation instruction area V12 including an explanation of the operation, and a content area V14 including an operation button of the service and the like. Further, the operation instruction area V12 includes the agent V121.

Note that the screen shown in FIG. 3 is an example, and the screen displayed on the display unit 150 by the output control unit 112 is not limited to the example shown in FIG. For example, the output control unit 112 changes the screen to be displayed on the display unit 150 by the output control unit 112 according to the operation information acquired by the operation processing unit 114 described later.

Further, as will be described later, the output control unit 112 causes the display unit 150 to display a screen corresponding to the support method selected by the user support device 500. For example, when the support by the agent is selected, the output control unit 112 supports the agent V121 shown in FIG. 3 based on the agent control information included in the support information received from the user support device 500 described later. Is displayed on the display unit 150. A specific example of the screen displayed according to the support method will be described later.

The operation processing unit 114 acquires the operation information input from the operation unit 160. The operation processing unit 114 stores the operation information input from the operation unit 160 in the storage unit 120. Further, the operation processing unit 114 calculates the display coordinate position on the display unit 150 corresponding to the position indicated by the operation by the user via the operation unit 160. Then, the operation processing unit 114 determines that the object corresponding to the calculated display position coordinates has been operated by the user.

The image processing unit 116 performs processing related to the captured image input from the imaging unit 170. The image processing unit 116 includes an image acquisition unit 1161, an image analysis unit 1162, and a mental and physical condition determination unit 1163.

The image acquisition unit 1161 acquires an image captured by the user U1 who operates the information processing apparatus 100 from the image pickup unit 170.

The image analysis unit 1162 extracts a region related to the user U1 (first user) who is currently operating the information processing apparatus 100 from the captured image acquired by the image acquisition unit 1161. For example, the image analysis unit 1162 according to the present embodiment may detect the face of the user U1 from the captured image and extract the face region of the user U1 based on the detection result.

The image analysis unit 1162 may extract candidate regions by, for example, edge detection or shape pattern detection, divide the extracted candidate regions into small regions, and collate the feature points of each region with a preset face region pattern. good. Further, the image analysis unit 1162 extracts the body candidate region when the density of each candidate region is a value corresponding to a predetermined threshold value, and uses the contrast of the density or saturation of the face and the body candidate region to obtain the face region. It may be extracted.

The mental / physical state determination unit 1163 detects the facial expression of the user U1 who is operating the information processing apparatus 100, and generates the first mental / physical state information indicating the mental / physical state of the user U1. As described above, the first mental and physical condition information generated by the mental and physical condition determination unit 1163 is transmitted to the user support device 500 and used for selecting the support method. For example, the mental and physical condition determination unit 1163 measures changes in eye and mouth movements in the face area detected by the image analysis unit 1162, and collates the measured changes with preset determination rules to obtain facial expressions. May be identified.

FIG. 4 is a diagram showing an example of a plurality of feature points obtained from the face image of the user U1. As shown in FIG. 4, by the image analysis unit 1162, the left eyebrows P1 to P3, the right eyebrows P4 to P6, the left eye P7 to P10, the right eye P11 to P14, and the mouth P15 to P18 are the feature points from the face image of the user U1. Is detected.

In the following, as a specific example, the characteristics of the embarrassed facial expression compared with the neutral facial expression will be described. However, other facial expressions have unique characteristics as well as embarrassed facial expressions. Therefore, each facial expression can be recognized based on its unique characteristics.

FIG. 5 is a diagram for explaining the relationship between a neutral facial expression and a confused facial expression. The image F1 is a left eyebrow image (P1 to P3) and a left eye image (P7 to P10) captured when the facial expression of the user U1 is a neutral facial expression. On the other hand, the image F2 is a left eyebrow image (P1 to P3) and a left eye image (P7 to P10) captured when the facial expression of the user U1 is a confused facial expression. Compared to the neutral facial expression shown in image F1, the distance between the eyebrows and the eyes is shorter in the confused facial expression shown in image F2 (in particular, the feature points P1 to P3 of the eyebrows and the feature points P7 to P9 above the eyes are. (It has changed), and the opening of the eyes has become narrower (in particular, the feature points P8 and P10 of the eyes have changed).

As shown in this example, each facial expression has its own characteristics. Therefore, once the position of each feature point is determined, it is possible to define the proximity to the facial expression for the position of each feature point. The mental and physical condition determination unit 1163 has feature points P1 to P3 for the left eyebrow, feature points P7 to P10 for the left eye, feature points P4 to P6 for the right eyebrow, feature points P11 to P14 for the right eye, and feature points P15 to P18 for the mouth, respectively. When the pre-change position and the post-change position of the above exceed a predetermined threshold value, it can be estimated that the degree of confusion of the user U1 has increased.

The first mental and physical condition information generated by the mental and physical condition determination unit 1163 may include information indicating the degree of confusion of the facial expression of the user U1 estimated as described above. However, the present invention is not limited to such an example, and the first mental and physical condition information may include information on various mental and physical states determined by another mental and physical condition determination unit 1163.

The explanation will be continued by returning to FIG. Various programs and data are stored in the storage unit 120 shown in FIG. 2. The storage unit 120 may be a semiconductor memory or a hard disk device, or may be composed of a RAM (Random Access Memory), a non-volatile memory, or the like. An OS (Operating System) and various applications may be stored in the hard disk device. The RAM may be a work memory for storing various data used by the control unit 110 to execute processing based on the program. The RAM may store various information such as a display screen displayed on the display unit 150, operation information input from the operation unit 160, and an image captured by the user U1 captured by the image pickup unit 170. The non-volatile memory is a memory (flash memory) in which the stored contents are not erased even when the power is turned off, and various setting information is stored.

The speaker 130 outputs sound according to the control of the output control unit 112 described above. For example, the speaker 130 outputs (reproduces) voice guidance stored in the information processing device 100, voice of the operator O1 received from the operator terminal 400, voice data included in the support information received from the user support device 500, and the like. ) May.

The microphone 140 acquires the voice of the user U1 who operates the information processing device 100. For example, when the user support device 500 selects support by an agent as the support method, the voice of the user U1 acquired by the microphone 140 is converted into text information by the control of the control unit 110, and the communication unit 180. Is transmitted to the user support device 500. Further, when the support by the agent is selected as the support method by the user support device 500, the voice of the user U1 acquired by the microphone 140 is transmitted from the communication unit 180 to the operator terminal 400 under the control of the control unit 110. NS.

The display unit 150 displays various screens under the control of the output control unit 112 described above.

The operation unit 160 may be, for example, a touch panel or a hard key such as a numeric keypad.

The image pickup unit 170 acquires an image captured by the user U1 who operates the information processing apparatus 100 by imaging. As described above, the captured image acquired by the imaging unit 170 can be used to determine the facial expression of the user U1. For example, the imaging unit 170 is a CCD (Charged).
It may be a Coupled Devices) camera.

The communication unit 180 is a communication interface that mediates communication between the information processing device 100 and other devices. The communication unit 180 supports any wireless or wired communication protocol and establishes a communication connection with, for example, via the network 600 described with reference to FIG. 1 or directly with other devices.

<2-2. Queue recognition device>
FIG. 6 is a block diagram showing an example of the configuration of the queue recognition device 200 shown in FIG. As shown in FIG. 6, the queue recognition device 200 includes a control unit 210, a storage unit 220, and a communication unit 230.

The control unit 210 has a function of controlling each configuration of the queue recognition device 200. In addition, as shown in FIG. 6, the control unit 210 also functions as an image acquisition unit 212 and an image analysis unit 214.

The image acquisition unit 212 acquires an image of the queue Q from the queue image pickup unit 250 shown in FIG.

The image analysis unit 214 recognizes the queue Q from the captured image acquired by the image acquisition unit 212, and generates queue information regarding the users U2 to Un (second user) forming the queue Q. .. In the present embodiment, the queue information generated by the image analysis unit 214 is, for example, information on the number of users U2 to Un (second users) arranged (included) in the queue Q of the information processing apparatus 100. May include. The queue information generated by the image analysis unit 214 is transmitted to the user support device 500 by the communication unit 230.

The image analysis unit 214 can detect a person from the video data by using a well-known image recognition method. For example, it is possible to retain the feature amount of the image, which corresponds to the detection range of a person, and detect a region similar to the feature amount from the video data as the detection range. For example, the image analysis unit 214 may detect a whole body of a person, or a part of a person such as a head or a face. The image analysis unit 214 identifies the number of users U2 to Un (second users) lined up (included) in the queue Q by using the result of the person detection as described above, and queue information. To generate.

Various programs and data are stored in the storage unit 220. The storage unit 220 may be a semiconductor memory or a hard disk device, or may be composed of a RAM, a non-volatile memory, or the like.

The communication unit 230 is a communication interface that mediates communication between the queue recognition device 200 and other devices. The communication unit 230 supports any wireless or wired communication protocol and establishes a communication connection with, for example, via the network 600 described with reference to FIG. 1 or directly with other devices.

<2-3. Monitoring device>
FIG. 7 is a block diagram showing an example of the configuration of the monitoring device 300 shown in FIG. As shown in FIG. 7, the monitoring device 300 includes a control unit 310 and a communication unit 320.

The control unit 310 has a function of controlling each configuration of the monitoring device 300. The control unit 310 also functions as a call control unit 312, as shown in FIG.

When the communication unit 320 receives the support information indicating the support by the station staff from the user support device 500, the call control unit 312 controls the communication unit 320 to the information processing device to the communication terminal 350 of the station staff E1. Have 100 users send instructions to support U1. For example, when there are a plurality of information processing devices including the information processing device 100 shown in FIG. 1 in the station, the call control unit 312 has information for identifying the information processing device 100 that requires support. (For example, the identification number) may be included in the instruction and transmitted to the communication terminal 350. Further, when the support information received from the user support device 500 includes text information indicating an exchange with the user U1, the call control unit 312 includes the text information in the instruction and causes the communication terminal 350 to transmit the text information. You may.

The communication unit 320 is a communication interface that mediates communication between the monitoring device 300 and other devices. The communication unit 320 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with, for example, via the network 600 described with reference to FIG. 1 or directly with other devices.

For example, the communication unit 320 may receive support information indicating support by a station employee from the user support device 500. Further, the communication unit 320 transmits an instruction to support the user U1 of the information processing device 100 to the communication terminal 350 according to the control of the call control unit 312.

<2-4. Operator terminal>
FIG. 8 is a block diagram showing an example of the configuration of the operator terminal 400 shown in FIG. As shown in FIG. 8, the operator terminal 400 includes a control unit 410, a storage unit 420, a videophone unit 430, and a communication unit 440.

The control unit 410 has a function of controlling each configuration of the operator terminal 400. Further, the control unit 410 has a TV telephone function, and information processing by the TV telephone function is performed, for example, when the communication unit 440 receives support information indicating support by the operator from the user support device 500 via the network 600. Support for the user U1 of the device 100 is realized. For example, the control unit 410 realizes a TV telephone function between the user U1 of the information processing apparatus 100 and the operator O1 by controlling the input / output with the TV telephone unit 430 described later and the communication by the communication unit 440. You may.

The storage unit 420 stores various programs and data. The storage unit 420 may be a semiconductor memory or a hard disk device, or may be composed of a RAM, a non-volatile memory, or the like.

As shown in FIG. 8, the videophone unit 430 includes a speaker 431, a microphone 433, a display unit 435, an operation unit 437, and an image pickup unit 439. The TV telephone unit 430 is controlled by the control unit 410 to provide the operator O1 shown in FIG. 1 with a TV telephone function with the user U1 of the information processing apparatus 100.

The speaker 431 outputs sound according to the control of the control unit 410. For example, the speaker 431 outputs the voice of the user U1 received by the communication unit 440.

The microphone 433 acquires the voice of the operator O1. The voice of the operator O1 acquired by the microphone 433 is transmitted from the communication unit 440 to the information processing device 100 under the control of the control unit 410.

The display unit 435 displays a screen including the face image of the user U1 under the control of the control unit 410. Further, the screen displayed on the display unit 435 may include the operation buttons displayed on the information processing apparatus 100 in the same layout as the information processing apparatus 100. With such a configuration, the operator can more easily support the user U1 of the information processing apparatus 100.

The operation unit 437 may be, for example, a touch panel, and accepts an input operation by the operator O1. The input operation received by the operation unit 437 may include an input operation by the operator O1 for the operation buttons included in the screen displayed on the display unit 435 described above.

The image pickup unit 439 acquires the captured image of the operator O1 by imaging. The captured image of the operator O1 acquired by the image pickup unit 439 is transmitted from the communication unit 440 to the information processing device 100 under the control of the control unit 410.

The communication unit 440 is a communication interface that mediates communication between the operator terminal 400 and other devices. The communication unit 440 supports any wireless or wired communication protocol and establishes a communication connection with, for example, via the network 600 described with reference to FIG. 1 or directly with other devices.

<2-5. User support device>
FIG. 9 is a block diagram showing an example of the configuration of the user support device 500 shown in FIG. As shown in FIG. 9, the user support device 500 includes a control unit 510, a storage unit 520, and a communication unit 530.

The control unit 510 has a function of controlling each configuration of the user support device 500. Further, as shown in FIG. 9, the control unit 510 also functions as a support determination unit 512 and a support control unit 514.

The support determination unit 512 makes a determination based on the first mental and physical state information received from the information processing device 100 and the queue information received from the queue recognition device 200, and supports the operation of the information processing device 100. Select.

For example, the support determination unit 512 may select a support method by a determination using the user support determination table stored in the storage unit 520. The user support determination table is a table for determining the support method, and is a table that associates the combination of the first mental and physical state information and the queue information with the support method.

FIG. 10 is a table showing an example of a user support determination table stored in the storage unit 520. By referring to the user support determination table shown in FIG. 10, the support determination unit 512 displays information indicating the degree of confusion of the user U1 included in the first mental and physical condition information, and the queue Q included in the queue information. The support method can be selected from the information on the number of users U2 to Un lined up in.

The support control unit 514 controls the support of the user U1 according to the support method selected by the support determination unit 512. For example, the support control unit 514 generates support information regarding the operation support of the information processing apparatus 100 according to the selected support method. The support information may include, for example, information indicating the selected support method. Further, the support information may include information indicating the past (up to the present) operation by the user U1, information indicating the past (up to the present) exchange with the user U1 and the like. With such a configuration, the support of the user U1 can be smoothly performed.

Further, the support control unit 514 controls the communication unit 530, and the support generated to at least one of the information processing device 100, the monitoring device 300, and the operator terminal 400 according to the selected support method. Have the information sent.

A specific example of the combination of the support information generated by the support control unit 514 and the destination device will be described later for each support method selected by the support determination unit 512.

The storage unit 520 stores various programs and data. The storage unit 520 may be a semiconductor memory or a hard disk device, or may be composed of a RAM, a non-volatile memory, or the like. The storage unit 520 stores, for example, the user support determination table shown in FIG. 10 and information for controlling the support of the user U1 by the support control unit 514.

The communication unit 530 is a communication interface that mediates communication between the user support device 500 and other devices. The communication unit 530 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with, for example, via the network 600 described with reference to FIG. 1 or directly with other devices. The communication unit 530 functions as an acquisition unit, receives (acquires) the first mental and physical state information from the information processing device 100, and receives (acquires) the queue information from the queue recognition device 200. Further, the communication unit 530 transmits the support information according to the support method selected by the support determination unit 512 to the device according to the support method under the control of the support control unit 514.

<< 3. Operation >>
The configuration examples of the information processing device 100, the queue recognition device 200, the monitoring device 300, the operator terminal 400, and the user support device 500 included in the information processing system 1 according to the present embodiment have been described above. Subsequently, an operation example of this embodiment will be described. In the following, first, the processing flow of the information processing system 1 will be described, and then specific examples of the processing related to the operation support will be described for each support method.

<3-1. Processing flow>
FIG. 11 is a flowchart showing a processing flow of the information processing system 1 according to the present embodiment. As shown in FIG. 11, first, the information processing apparatus 100 acquires a captured image of the user U1 who operates the information processing apparatus 100 (S102). Step S102 may be started with the detection of the user U1 as a trigger. For example, the information processing apparatus 100 detects a user by a human body sensor (not shown) or the like, and the operation screens stored in the storage unit 120 are displayed on the display unit 150 according to the operation order, and the image pickup unit 170 is displayed at the start of the operation. It may be activated and the acquisition of the captured image may be started. Alternatively, step S102 may be started in conjunction with the transition of the screen displayed on the display unit 150 instead of the detection of the user and triggered by the display of a predetermined screen.

Subsequently, the image processing unit 116 of the information processing apparatus 100 determines the mental and physical state of the user U1 from the captured image of the user U1 acquired in step S102 (S104). In step S104, the degree of confusion of the user U1 may be determined, for example, as a mental and physical condition.

Subsequently, the queue image pickup unit 250 images the queue Q formed by the users U2 to Un who are scheduled to use the information processing apparatus 100 after the user U1 and acquires the captured image of the queue Q. (S106).

Subsequently, the image analysis unit 214 of the queue recognition device 200 determines the number of users U2 to Un (second users) lined up in the queue Q from the captured image of the queue Q acquired in step S106. (S108).

Subsequently, the support determination unit 512 of the user support device 500 selects the support method by the determination using the user support determination table shown in FIG. 10 (S110). In step S110, the support determination unit 512 includes a first mental and physical information including information on the degree of confusion of the user U1 determined in step S104, and a queue including information on the number of people in the queue Q identified in step S108. Make a judgment based on the information. In step S110 or before, the communication unit 530 of the user support device 500 receives (acquires) the first mental and physical state information from the information processing device 100, and receives the queue information from the queue recognition device 200. Receive (acquire).

Subsequently, under the control of the support control unit 514 of the user support device 500, the operation support of the user U1 is started according to the support method selected in step S110 (S112). A specific example of the process related to the operation support of the user U1 started in step S112 will be described later for each selected support method.

Before the operation support in step S112 is started, a screen for confirming whether or not the user U1 desires the operation support may be displayed on the display unit 150. FIG. 12 is a diagram showing an example of a screen displayed on the display unit 150 when the support method is selected in step S110. The screen shown in FIG. 12 includes a dialog V122 for confirming whether or not the user U1 desires operation support. Then, when the button V123 "Ask a question" included in the dialog V122 is selected, the operation support of step S112 is started, and when the button V124 "Do not ask a question" is selected, the operation support of step S112 is started. Operation support may be skipped.

If the operation of the user U1 is not completed even after the support of the user U1 is completed (NO in S114), the process returns to step S102. On the other hand, when the operation of the user U1 is completed (YES in S114), the process is completed.

<3-2. Specific examples of operation support>
The processing flow of this embodiment has been described above. Subsequently, an example of the process related to the operation support of the user U1 started in step S112 of FIG. 11 will be specifically described for each support method selected in step S110.

(Support by agents)
First, a specific example of processing when the selected support method is "support by an agent" will be described. When "support by the agent" is selected by the support determination unit 512 in step S110, the support control unit 514 generates support information including agent control information for displaying the agent on the information processing apparatus 100. Further, the support control unit 514 controls the communication unit 530, establishes a connection between the information processing device 100 and the user support device 500, and causes the information processing device 100 to transmit support information including agent control information.

When the information processing apparatus 100 receives the support information including the agent control information, the agent is displayed on the display unit 150 of the information processing apparatus 100. FIG. 13 is a diagram showing an example of a screen displayed on the display unit 150 when the support by the agent is performed.

The screen shown in FIG. 13 includes a dialogue dialog V141 with an agent. The dialogue dialog V141 includes agent outputs V142 and V144 from the agent, and user input V143 by the user U1, for example, as shown in FIG.

The agent output V142 may be displayed, for example, based on the text information included in the agent control information. Then, the voice input by the user U1 who sees the agent output V142 via the microphone 140 is converted into text information by the control unit 110 of the information processing device 100 and transmitted to the user support device 500. Based on such text information, the user input V143 shown in FIG. 13 is displayed. Further, the support control unit 514 refers to the information stored in the storage unit 520, and generates agent control information including the text information to be a reply based on the text information. For example, the support control unit 514 may extract an answer corresponding to the information matching the text information from the information stored in the storage unit 520. When the agent control information including the text information to be the answer can be generated, the support control unit 514 controls the communication unit 530 to transmit the support information including the generated agent control information to the information processing apparatus 100. Then, the agent output V144 is displayed based on the agent control information.

The text information included in the agent control information transmitted from the user support device 500 to the information processing device 100 is converted from text information into voice information by the control unit 110 of the information processing device 100 and displayed on the display unit 150. At the same time, audio may be output from the speaker 130.

Further, when it is not possible to find information that matches the text information based on the input of the user U1 transmitted from the information processing device 100 to the user support device 500, the support control unit 514 includes the text information to be the answer. It may be difficult to generate agent control information. In such a case, the support control unit 514 generates agent control information for displaying a screen to be connected to the operator, and transmits the support information including the agent control information generated by controlling the communication unit 530 to the information processing device 100. You may let me. FIG. 14 is a diagram showing an example of a screen connected to the operator.

The screen shown in FIG. 14 includes a button V145 for connecting to an operator. When the user U1 selects the button V145 shown in FIG. 14, the support control unit 514 generates support information (an example of support information indicating support by the operator) indicating that the operator will take over the support according to the selection. do. The generated support information is transmitted to the operator terminal 400.

When there is a reaction from the operator O1 of the operator terminal 400, the support control unit 514 further generates support information including text information indicating the interaction with the user U1 in the support by the agent, and controls the communication unit 530. To send to the operator terminal 400. With such a configuration, the operator O1 can take over the support provided by the agent and provide support.

On the other hand, when there is no input from the operator O1 for a certain period of time, the support control unit 514 generates support information (an example of support information indicating support by the station staff) indicating that the support is taken over by the station staff to the monitoring device 300. The generated support information may further include text information indicating interaction with the user U1 in support by the agent. The generated support information is transmitted to the monitoring device 300, and further, an instruction based on the support information is transmitted from the monitoring device 300 to the communication terminal 350 of the station staff E1. With such a configuration, the station staff E1 can take over the support provided by the agent and provide support.

(Support by operator)
Next, a specific example of processing when the selected support method is "support by the operator" will be described. When "support by the operator" is selected by the support determination unit 512 in step S110, the support control unit 514 generates support information indicating support by the operator. Further, the support control unit 514 controls the communication unit 530 to transmit the connection instruction and the generated support information to the information processing device 100 and the operator terminal 400.

When the connection is started between the information processing apparatus 100 and the operator terminal 400 that have received the connection instruction and the support information, the captured image of the operator O1 is displayed on the display unit 150 of the information processing apparatus 100, and the operator O1 is displayed from the speaker 130. The voice of is output.

FIG. 15 is a diagram showing an example of a screen displayed on the display unit 150 when the support by the operator is performed. The screen shown in FIG. 15 includes a region V146 for displaying an image captured by the operator O1 captured by the image pickup unit 439 of the operator terminal 400.

Further, although not shown, similarly, the captured image of the user U1 is displayed on the display unit 435 of the operator terminal 400. Further, the screen information displayed on the display unit 150 of the information processing apparatus 100 may be shared with the operator terminal 400. For example, the screen information displayed on the display unit 150 of the information processing apparatus 100 may be transmitted to the operator terminal 400 and displayed on the display unit 435. With such a configuration, the operator O1 can provide support while grasping the screen viewed by the user U1.

If there is a question from the user U1, the voice of the user U1 is output from the speaker 431 of the operator terminal 400 when the voice is input from the microphone 140 of the information processing apparatus 100.

When the operator O1 gives an instruction to the user U1 on the display unit 150, the input content is displayed on the display unit 150 of the information processing apparatus 100 by operating the operation unit 437 of the operator terminal 400 and inputting the instruction. Will be done. Further, although not shown in FIG. 8, it may be possible for the operator O1 to operate on behalf of the user U1 from the keyboard connected to the operator terminal 400.

(Support by station staff)

Next, a specific example of processing when the selected support method is "support by station staff" will be described. When "support by station staff" is selected by the support determination unit 512 in step S110, the support control unit 514 generates support information indicating support by station staff. Further, the support control unit 514 controls the communication unit 530 to transmit the generated support information to the monitoring device 300.

Upon receiving the support information, the monitoring device 300 transmits an instruction to support the user U1 to the communication terminal 350 possessed by the station employee E1. The communication terminal 350 that has received the instruction from the monitoring device 300 displays a screen that guides the station employee E1 to the information processing device 100 according to the instruction. Then, the station employee E1 heads for the information processing device 100 and supports the operation of the user U1.

<< 4. Effect >>
As described above, according to the embodiment of the present invention, in addition to the mental and physical state information (confusion level) of the user U1 operating the information processing device 100, the information processing device 100 is installed after the user U1. It is possible to determine the support method based on the length (number of people) of the queue Q formed by the operating users U2 to Un. As a result, it becomes possible to provide support to the user U1 operating the information processing apparatus 100 by a more appropriate support method according to the length of the queue Q.

<< 5. Modification example >>
The embodiment of the present invention has been described above. Hereinafter, some modifications of the above embodiment will be described. In addition, each modification described below may be applied to the said embodiment alone, or may be applied to the said embodiment in combination. Further, each modification may be applied in place of the configuration described in the above embodiment, or may be additionally applied to the configuration described in the above embodiment.

<5-1. Modification 1>
In the above embodiment, an example in which the queue information includes the number of users U2 to Un arranged in the queue Q of the information processing apparatus 100 has been described, but the present invention is not limited to such an example. For example, the queue information includes information indicating the mental and physical states of users U2 to Un (second user) lined up in the queue Q (sometimes referred to as second mental and physical state information). May be good. Hereinafter, such an example will be described as a modification 1. Since the configuration and operation of the information processing system according to this modification are the same as the configuration of the information processing system 1 described above except for a part, duplicate explanations are omitted and the configuration is mainly peculiar to this modification. , And the operation will be explained.

(composition)
First, the configuration according to the first modification will be described. In the information processing system according to this modification, a part of the configurations of the queue recognition device 200 and the user support device 500 is different from the above-mentioned information processing system 1.

FIG. 16 is a block diagram showing an example of the configuration of the queue recognition device 200 according to the modification 1. As shown in FIG. 16, the queue recognition device 200 according to this modification includes a control unit 210, a storage unit 220, and a communication unit 230. Since the configurations of the storage unit 220 and the communication unit 230 according to the present modification are substantially the same as the configurations of the storage unit 220 and the communication unit 230 according to the present modification described with reference to FIG. 6, they overlap. The explanation is omitted.

The control unit 210 according to this modification has a function of controlling each configuration of the queue recognition device 200. Further, the control unit 210 according to this modification functions not only as an image acquisition unit 212 and an image analysis unit 214, but also as a mental and physical condition determination unit 216.

The mental and physical condition determination unit 216 determines the facial expressions and actions of the users U2-Un from the captured image of the queue Q acquired by the image acquisition unit 212 based on the person area detected by the image analysis unit 214, and determines the facial expressions and actions of the users U2-Un. The mental and physical states of the users U2-Un are estimated from the determination result.

The mental and physical states of the users U2 and Un estimated by the mental and physical state determination unit 216 may include, for example, the degree of dissatisfaction of the users U2 and Un. For example, an action such as frowning, an angry face, or frequently turning the face toward the information processing device 100 to check the operation status of the user U1 was detected from the person area. In this case, the mental and physical condition determination unit 216 estimates that the degree of dissatisfaction is high.

In addition to the functions of the image analysis unit 214 described in the above embodiment, the image analysis unit 214 according to this modification is a user who is presumed to have a high degree of dissatisfaction by the mental and physical condition determination unit 216 among the users U2 to Un. It has a function to specify the number of people. Then, the image analysis unit 214 according to this modification is dissatisfied with the information on the number of users U2-Un (second user) lined up (included) in the queue Q and among the users U2-Un. Generates queue information including information on the number of people with a high degree (an example of the second mental and physical condition information). The queue information generated by the image analysis unit 214 is transmitted to the user support device 500 by the communication unit 230 as described above.

Subsequently, the user support device 500 according to this modification will be described with reference to FIG. The user support device 500 according to the present modification has been described in the above embodiment in that the support determination unit 512 makes a determination based on the number of people with a high degree of dissatisfaction included in the queue information and selects a support method. It is different from the user support device 500. Therefore, the user support determination table stored in the storage unit 520 of the present modification and used by the support determination unit 512 of the present modification may include information about the number of people with a high degree of dissatisfaction as queue information. good.

FIG. 17 is a table showing an example of a user support determination table stored in the storage unit 520 according to this modification. By referring to the user support determination table shown in FIG. 17, the support determination unit 512 provides information indicating the degree of confusion of the user U1, information on the number of users U2 to Un in the queue Q, and user U2. It is possible to select a support method from the information on the number of people who are highly dissatisfied with ~ Un.

(motion)
FIG. 18 is a flowchart showing a processing flow of this modification. Since the processing of steps S202 to S208 shown in FIG. 18 is the same as the processing of steps S102 to 108 described with reference to FIG. 11, the description thereof is omitted here.

In step S209, the mental and physical state determination unit 216 of the queue recognition device 200 estimates the degree of dissatisfaction of each user based on the person area detected from the captured image of the queue Q, and the image analysis unit 214 uses it. The number of users who are highly dissatisfied among the persons U2 and Un is specified (S209).

Subsequently, the support determination unit 512 of the user support device 500 selects the support method by the determination using the user support determination table shown in FIG. 17 (S210). In step S210, the support determination unit 512 has first mental and physical information including information on the degree of confusion of the user U1 determined in step S204, information on the number of people in the queue Q identified in step S208, and step. The determination is made based on the queuing information including the information of the number of people with a high degree of dissatisfaction specified in S209. In step S210 or before, the communication unit 530 of the user support device 500 receives (acquires) the first mental and physical state information from the information processing device 100, and receives the queue information from the queue recognition device 200. Receive (acquire).

Since the subsequent processes of steps S212 to S214 are the same as the processes of steps S112 to 114 described with reference to FIG. 11, the description thereof is omitted here.

(effect)
The modification 1 has been described above. According to this modification, in addition to the mental and physical state information (confusion level) of the user U1 operating the information processing device 100 and the length (number of people) of the queue Q, the dissatisfaction level of the users lining up in the queue. It is possible to determine the support method based on. As a result, in order to reduce the dissatisfaction level of the users U2 to Un in the queue Q, it is possible to provide support to the user U1 operating the information processing apparatus 100 by a more appropriate support method. Become.

<5-2. Modification 2>
In the above embodiment, an example in which the information processing device 100 is a ticket issuing machine installed at a station has been described, but the present invention is not limited to such an example. The information processing device 100 may be another device such as an ATM as long as it is an information processing device that is sequentially operated by a plurality of users.

<5-3. Modification 3>
In the above embodiment, a method of detecting the mental and physical state of the user U1 who operates the information processing apparatus 100 from facial expressions by feature points has been described, but the present invention is not limited thereto. Behaviors such as tilting the user's head or facial expressions and behaviors may be combined and detected.

<5-4. Modification 4>
Further, in the above embodiment, as a feature for recognizing facial expressions, the mental and physical condition determination unit 1163 uses 3 points each for the left eyebrow and the right eyebrow, and the left eye, the right eye, and the mouth as feature points from the face image of the user U1. Although it is assumed that each of the four points is detected, the number of feature points is not limited to this. For example, the mouth may be divided into an upper lip and a lower lip to detect feature points.

<5-5. Modification 5>
Further, in the above embodiment, as the support method of the user U1 of the information processing apparatus 100, three methods of support by an agent, support by an operator, and support by a station employee have been described as an example, but the present invention is limited to this. It is not something that will be done.

<5-6. Modification 6>
Further, the configuration of the information processing system according to the embodiment of the present invention is not limited to the above-mentioned example. For example, the functions of the above-mentioned plurality of devices may be provided in one device, or the functions of one of the above-mentioned devices may be provided by being divided into a plurality of devices.

For example, in the above embodiment, an example in which the information processing device 100 and the user support device 500, which are sequentially operated by a plurality of users, are different devices has been described, but the present invention is not limited to such an example. For example, some or all of the functions of the user support device 500 described above may be provided in the information processing device 100. When all the functions of the user support device 500 described above are provided in the information processing device 100, the information processing device 100 is an information processing device that is sequentially operated by a plurality of users and is a user support device. But it may be understood that it is.

<< 6. Hardware configuration >>
Each embodiment of the present invention has been described above. The above-mentioned information processing is realized by the cooperation of the software and the hardware of the information processing device 100, the queue recognition device 200, the monitoring device 300, the communication terminal 350, the operator terminal 400, and the user support device 500. In the following, as a hardware configuration example of the information processing device 100, the queue recognition device 200, the monitoring device 300, the communication terminal 350, the operator terminal 400, and the user support device 500, which are the information processing devices according to the embodiment of the present invention, The hardware configuration of the information processing apparatus 1000 will be described.

FIG. 19 is an explanatory diagram showing a hardware configuration of the information processing apparatus 1000 according to the embodiment of the present invention. As shown in FIG. 19, the information processing apparatus 1000 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an input device 1004, and an output device 1005. , A storage device 1006 and a communication device 1007.

The CPU 1001 functions as an arithmetic processing unit and a control device, and controls the overall operation in the information processing device 1000 according to various programs. Further, the CPU 1001 may be a microprocessor. The ROM 1002 stores programs, calculation parameters, and the like used by the CPU 1001. The RAM 1003 temporarily stores a program used in the execution of the CPU 1001 and parameters that appropriately change in the execution. These are connected to each other by a host bus composed of a CPU bus or the like. Mainly, the functions of the control unit 110, the control unit 210, the control unit 310, the control unit 410, the control unit 510, and the like are realized by the collaboration between the CPU 1001, the ROM 1002, the RAM 1003, and the software.

The input device 1004 includes an input means for the user to input information such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 1001. It is composed of etc. By operating the input device 1004, the user of the information processing device 1000 can input various data to the information processing device 1000 and instruct the processing operation. The input device 1004 corresponds to the operation unit 160 and the operation unit 437.

The output device 1005 includes, for example, a liquid crystal display (LCD) device, an OLED device, and a display device such as a lamp. Further, the output device 1005 includes an audio output device such as a speaker and headphones. For example, the display device displays an captured image, a generated image, or the like. On the other hand, the voice output device converts voice data and the like into voice and outputs the data. The output device 1005 corresponds to the display unit 150 and the display unit 435.

The storage device 1006 is a device for storing data. The storage device 1006 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like. The storage device 1006 stores programs and various data executed by the CPU 1001. The storage device 1006 corresponds to a storage unit 120, a storage unit 220, a storage unit 420, and a storage unit 520.

The communication device 1007 is, for example, a communication interface composed of a communication device or the like for connecting to a communication network. Further, the communication device 1007 may include a wireless LAN (Local Area Network) compatible communication device, an LTE (Long Term Evolution) compatible communication device, a wire communication device for wired communication, or a Bluetooth (registered trademark) communication device. The communication device 1007 corresponds to the communication unit 180, the communication unit 230, the communication unit 320, the communication unit 440, and the communication unit 530.

<< 7. Conclusion >>
As described above, according to the embodiment of the present invention, not only the mental and physical condition of one user operating the information processing apparatus but also the situation of the users in the queue are reflected in the user. It is possible to provide support.

Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

For example, each step in the above embodiment does not necessarily have to be processed in chronological order in the order described as a flowchart. For example, each step in the processing of the above embodiment may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

Further, according to the above embodiment, the hardware such as the CPU 1001, ROM 1002, and RAM 1003 is configured with the above-mentioned information processing device 100, queue recognition device 200, monitoring device 300, operator terminal 400, and user support device 500. It is also possible to provide a computer program for exerting the same function as the above. Also provided is a recording medium on which the computer program is recorded.

1 Information processing system 100 Information processing device 110 Control unit 112 Output control unit 114 Operation processing unit 116 Image processing unit 120 Storage unit 130 Speaker 140 Microphone 150 Display unit 160 Operation unit 170 Imaging unit 180 Communication unit 200 Matrix recognition device 210 Control unit 212 Image acquisition unit 214 Image analysis unit 216 Mental and physical condition judgment unit 220 Storage unit 230 Communication unit 250 Matrix imaging unit 300 Monitoring device 310 Control unit 400 Operator terminal 410 Control unit 500 User support device 510 Control unit 512 Support judgment unit 514 Support control unit 520 Storage unit 530 Communication unit 600 Network 1114 Operation processing unit 1161 Image acquisition unit 1162 Image analysis unit 1163 Mental and physical condition determination unit

Claims (15)

It is a user support device for supporting the operation of an information processing device that is sequentially operated by a plurality of users.
The first mental and physical state information indicating the mental and physical state of the first user who operates the information processing device, and the queue information regarding the second user who operates the information processing device after the first user. , And a support determination unit that selects a support method for the operation of the information processing apparatus by the first user based on the determination based on the first mental and physical condition information and the queue information.
User support device equipped with. The user support device according to claim 1, wherein the support determination unit selects the support method by using a user support determination table that associates the first mental / physical state information and queue information with the support method. The user support device according to claim 1 or 2, further comprising a support control unit that controls the support of the first user according to the selected support method. The user support device according to claim 3, wherein the support control unit generates support information regarding operation support of the information processing device by the first user according to the selected support method. The user support device according to claim 4, wherein the support information generated by the support control unit includes information indicating the selected support method. The user support device according to claim 4 or 5, wherein the support information generated by the support control unit includes information indicating past operations by the first user. The user support device according to any one of claims 4 to 6, wherein the support information generated by the support control unit includes information indicating past exchanges with the first user. When the selected support method is support by an agent, the support information generated by the support control unit is used to display the agent supporting the operation by the first user on the information processing apparatus. The user support device according to any one of claims 4 to 7, which includes agent control information. The user support device according to any one of claims 1 to 8, wherein the first mental and physical condition information includes information indicating the degree of confusion of the facial expression of the first user. The user support device according to any one of claims 1 to 9, wherein the queue information includes information on the number of the second users. The user support device according to any one of claims 1 to 10, wherein the queue information includes the second mental and physical condition information indicating the mental and physical condition of the second user. The user support device according to claim 11, wherein the second mental and physical condition information includes information on the number of people who are highly dissatisfied among the second users. It is a user support method for supporting the operation of information processing devices that are sequentially operated by multiple users.
The first mental and physical state information indicating the mental and physical state of the first user who operates the information processing device, and the queue information regarding the second user who operates the information processing device after the first user. , And selecting a support method for the operation of the information processing apparatus by the first user based on the determination based on the first mental and physical condition information and the queue information.
User support methods, including. For computers that support the operation of information processing devices that are sequentially operated by multiple users.
The first mental and physical state information indicating the mental and physical state of the first user who operates the information processing device, and the queue information regarding the second user who operates the information processing device after the first user. , And a function of selecting a support method for operating the information processing apparatus by the first user based on a determination based on the first mental and physical condition information and the queue information.
A program to realize. Information processing equipment that is sequentially operated by multiple users,
The queue formed by the second user who operates the information processing device is recognized after the first user who operates the information processing device, and the queue information about the second user is generated. Queue recognition device,
An information processing system including a user support device for supporting the operation of the information processing device, and a network for interconnecting the information processing device, the queue recognition device, and the user support device.
The user support device is
An acquisition unit for acquiring first mental and physical condition information indicating the mental and physical state of the first user and queue information regarding the second user, and the first mental and physical state information and the queue. A support determination unit that selects a support method for operating the information processing apparatus by the first user based on a determination based on information.
An information processing system equipped with.

Images