JP7817564B2 - Relay system, information recording method, and information recording program - Google Patents

Description

The present invention relates to a relay system, an information recording method, and an information recording program.

In recent years, the use of teleworking and working from home has increased, and technology that uses web portals is known as a way to send instructions from a client terminal operated by a user to office PCs, cloud servers, etc. (hereinafter referred to as various servers).

In this technology, the web portal relays between the client terminal and various servers via the Internet, delivering operation screens sent from the various servers to the client terminal. The user operating the client terminal views the operation screen on a browser and operates it using a keyboard and mouse.

In such systems, communication delays can occur between the web portal and various servers. When communication delays occur, a user may perform an operation but the operation is not reflected, leading to problems such as the user repeating the same operation incorrectly.

Japanese Patent Application Laid-Open No. 2018-195209 Japanese Patent Application Laid-Open No. 2021-56763

When a problem occurs, administrators often refer to the logs generated by the operating programs of various servers to identify the cause of the problem.

However, when a problem occurs due to an incorrect operation caused by a communication delay, as described above, the various servers normally process the corresponding operation, so information about the problem is often not recorded in the logs. For this reason, it may not be possible to identify the cause of the problem from the logs.

In one aspect, the present invention aims to provide a relay system, information recording method, and information recording program that make it easier to identify the cause of problems that occur when communication delays occur.

In one aspect, a relay system relays between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal. The relay system includes a recording device that records an operation screen displayed on the terminal while the terminal and the processing device are connected and accumulates first recorded data; and a transceiver that, when receiving a request from the terminal to the processing device, transmits the request to the processing device and, when receiving a response corresponding to the request from the processing device, transmits the response to the terminal. When the time difference between the time when the transceiver transmits a request and the time when the response corresponding to the request is received exceeds a reference value, the recording device generates second recorded data by duplicating recorded data from the first recorded data that was recorded a predetermined time before the time when the request was transmitted, and adds recorded data obtained by recording the operation screen while the time difference exceeds the reference value to the second recorded data.

This makes it easier to identify the cause of problems that occur when communication delays occur.

FIG. 1 is a diagram illustrating a schematic configuration of a business system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a web portal. FIG. 2 is a functional block diagram of a web portal. FIG. 10 is a diagram illustrating an example of a characteristic operation table. FIG. 5A is a diagram showing an example of an operation information table, and FIG. 5B is a diagram showing an example of a screen information table. FIG. 6 is a flowchart showing the processing of the user terminal. 7 is a flowchart showing the processing of the web portal (screen providing unit), server, and recording server after the processing of step S12 in FIG. 6 is executed. 10 is a flowchart showing the processing of the screen providing unit, delay monitoring unit, and server when a user operates the keyboard or mouse after remote connection and operation information is sent from the user terminal to the web portal. FIG. 10 is a diagram showing a state in which operation information data is registered in an operation information table. 9 is a flowchart showing a process subsequent to the process of the delay monitoring unit in FIG. 8 . FIG. 11(a) is a diagram showing the state in which screen information data is registered in the screen information table, and FIG. 11(b) is a diagram showing the state in which the ID in FIG. 11(a) is registered as the response ID in the operation information table. 10 is a flowchart showing the processing of a video extraction unit and a recording server. 13A and 13B are diagrams for explaining the processing of the recording server. 14(a) and 14(b) are diagrams for explaining problems caused by delays.

One embodiment of the business system is described in detail below with reference to Figures 1 to 14.

Figure 1 shows a schematic diagram of a business system 100 according to one embodiment. The business system 100 comprises a user terminal 10, which is a terminal used by a user at home, work, a satellite office, etc.; a server 70, which is a processing device to which the user terminal 10 is remotely connected; and a relay system 50. The relay system 50 comprises a web portal 20, which is a transmitting/receiving device that acts as an intermediary between the user terminal 10 and the server 70 via the Internet, and a recording server 60, which is a recording device that works in cooperation with the web portal 20 to perform processing.

The user terminal 10 is an information processing device such as a PC (Personal Computer) that performs various processes in response to user operations. The server 70 is also an information processing device that performs processes in response to requests from the user terminal 10.

When a user requests a remote connection with the server 70 from the user terminal 10, the web portal 20 establishes a remote connection between the user terminal 10 and the server 70. In this remotely connected state, a screen created by the server 70 is displayed on the display unit of the user terminal 10, and when an operation is performed on that screen, the web portal 20 sends operation information (operation request) corresponding to that operation to the server 70. In addition, upon receiving the operation request, the server 70 executes processing corresponding to the operation request and sends information about the updated screen to the web 20 as a screen response. The web portal 20 then sends the screen response to the user terminal 10 and displays the updated screen on the display unit of the user terminal 10 (performs a screen update).

The web portal 20 has the function of relaying between the user terminal 10 and the server 70, and includes one or more information processing devices. For simplicity's sake, this embodiment will describe the case where the web portal 20 is implemented by a single information processing device.

Figure 2 shows an example of the hardware configuration of the web portal 20. As shown in Figure 2, the web portal 20 includes a CPU (Central Processing Unit) 90, ROM (Read Only Memory) 92, RAM (Random Access Memory) 94, a storage unit (here, an SSD (Solid State Drive) or HDD (Hard Disk Drive)) 96, a network interface 97, and a portable storage medium drive 99. These components of the web portal 20 are connected to a bus 98. In the web portal 20, the CPU 90 executes programs (including information recording programs) stored in the ROM 92 or the storage unit 96, or programs read from the portable storage medium 91 by the portable storage medium drive 99, thereby realizing the functions of the components shown in Figure 3. Note that the functions of the components in Figure 3 may also be realized by integrated circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

As shown in Figure 3, the web portal 20 has the functions of a screen providing unit 22, a delay monitoring unit 24, and a video extraction unit 26.

The screen providing unit 22 transmits operation requests from the user terminal 10 to the server 70, and transmits a screen response to the user terminal 10 when a response is received from the server 70.

The delay monitoring unit 24 monitors the time difference between sending an operation request and receiving a screen response, and determines that a delay has occurred if the time difference exceeds a predetermined threshold. Furthermore, depending on the determination result, the delay monitoring unit 24 requests the video extraction unit 26 to record the screen during the period in which the delay is occurring (delay period).

The delay monitoring unit 24 monitors the time difference when the operation related to the operation request is a predetermined "characteristic operation." A characteristic operation is an operation that can be used to determine whether a communication delay is occurring, such as an operation that requires a screen update (an operation that updates the screen). Figure 4 shows an example of a characteristic operation table referenced by the delay monitoring unit 24. The characteristic operation table in Figure 4 associates the type of characteristic operation with a corresponding threshold value. As shown in Figure 4, characteristic operations include pressing a "character key on the keyboard," pressing the "Enter key on the keyboard," pressing a "modifier key such as Ctrl or Shift," and right-clicking or left-clicking a mouse. The threshold value is set for each operation based on 500 ms, taking into account the maximum delay tolerance humans have during a remote connection (e.g., 250 ms) and the interrupt processing speed of the delay monitoring unit 24. For example, in the case of pressing a "character key on the keyboard," a "right click," or a "left click," the processing on the side of the application running on the server 70 is almost always limited to processing to reflect the input content, so the threshold is set short. On the other hand, in the case of pressing the "Enter key on the keyboard," the threshold is set long because processing is often requested from the application on the server 70 along with the input. Also, in the case of pressing a "modifier key such as Ctrl or Shift," pressing the modifier key alone has no particular effect, but often requests processing from the server 70 in combination with other keys, so the threshold is set long. Note that the processing corresponding to characteristic operations may differ depending on the application running on the server 70, so the characteristic operation table of Figure 4 may be prepared for each application. For example, when a "left click" is performed on a dialog, a relatively high-load process may be executed in the application on the server 70. In this case, the threshold is set long (e.g., 2000 ms).

For details on the maximum latency that humans can tolerate when connecting remotely (e.g., 250 ms), see 'Pedro Casas, Michael Seufert, Sebastian Egger, and Raimund Schatz, "Quality of Experience in Remote Virtual Desktop Services", https://ieeexplore.ieee.org/document/6573191, (2021-12-21)'.

In addition, when monitoring the time difference, the delay monitoring unit 24 appropriately generates the operation information table shown in Figure 5(a) and the screen information table shown in Figure 5(b).

Specifically, the delay monitoring unit 24 generates the operation information table shown in FIG. 5(a) when a characteristic operation is performed on the user terminal 10. The operation information table in FIG. 5(a) stores the following information: "ID," "Connection ID," "Sent Time," "Operator," "Destination Server Information," "Response ID," and "Operation Content." "ID" is identification information for identifying the operation information. "Connection ID" is identification information for the remote connection from which the operation information was generated. "Sent Time" is the time the operation was recorded, "Operator" is the username of the user who performed the operation, and "Destination Server Information" is the server name of the server to which the user is remotely connected. "Response ID" is the ID of the screen information table corresponding to the operation. Note that when a characteristic operation is performed by the user, the "Response ID" column is empty. However, when a screen response corresponding to the characteristic operation is transmitted from the server 70, data is entered in the "Response ID" column. "Operation Content" is information indicating how the user operated the keyboard or mouse. For example, in the case of a remote connection, the operation details are recorded in a comma-separated text format that conforms to the X11 KeySym standard. For example, if you press the key identified as 'ff52', it will be recorded as 'key,ff52,1'. Also, if you left-click at coordinates (865,312) on the screen (pressing the button identified as 'fd11'), it will be recorded as 'mouse,865,312,fd11,1'.

Furthermore, the delay monitoring unit 24 generates the screen information table shown in Figure 5(b) upon receiving a response containing updated screen information (screen update information) from the remotely connected server 70. The screen information table in Figure 5(b) stores the following information: "ID," "received time," and "screen update content." "ID" is identification information for identifying the screen update information. "received time" is the time the screen update information was received. "screen update content" is the updated screen information, expressed in a comma-delimited text file format. The updated screen information replaces updated sections of the screen, and the previous screen information is duplicated for sections that have not been updated. For example, when defining the layer size at startup, it is recorded as '4, size, 1.0.4, 1024, 768', and when linking the image type or layer settings, it is recorded as '3, img, 1.3.2, 12, <meta>, image/png:'. Also, when duplicating an unupdated section, it will be recorded as '4, copy, 2.-2, 1.0.1.0.2,', and when replacing an updated section, it will be recorded as '4, arc, 1.3.2, 12, <meta>'. In reality, it will be recorded as a text file with entries such as '4, size, 1.0.4, 1024, 768' or '3, img, 1.3.2, 12, <meta>, image/png:' concatenated without line breaks.

Returning to Figure 3, in response to a request from the delay monitoring unit 24, the video extraction unit 26 instructs the recording server 60 to record the screen during the delay period.

While the user terminal 10 and server 70 are remotely connected, the recording server 60 records the screen displayed on the display unit of the user terminal 10 to generate normal recording data (first recording data). Furthermore, in response to instructions from the video extraction unit 26, the recording server 60 records the screen displayed on the user terminal 10 during the delay period in parallel with recording the normal recording data to generate delay period recording data (second recording data).

(Regarding the processing of each device)
The processing of each device will be described in detail below.

(Regarding processing of the user terminal 10)
First, a description will be given of the processing of the user terminal 10. The processing of the user terminal 10 is shown in a flowchart in FIG.

When the processing in FIG. 6 begins, first, in step S10, the user terminal 10 waits until the user inputs a remote connection request. When the user inputs a remote connection request by specifying one of the servers 70, the process proceeds to step S12, where the user terminal 10 requests a remote connection from the web portal 20.

Next, in step S14, the user terminal 10 waits until it receives screen change information from the web portal 20. The screen change information includes screen information sent from the server 70 after the remote connection is established, and screen information updated on the server 70 as a result of operations performed on the screen sent from the server 70. Once the screen change information is received from the web portal 20, it proceeds to step S16.

When proceeding to step S16, the user terminal 10 updates the screen based on the received screen change information.

Next, in step S18, the user terminal 10 waits until the user operates the keyboard or mouse. When the user operates the keyboard or mouse, the process proceeds to step S20, and the user terminal 10 transmits operation information to the web portal 20.

Next, in step S22, the user terminal 10 determines whether a request to terminate the remote connection has been input. That is, the user terminal 10 determines whether the operation in step S18 was an operation to input a request to terminate the remote connection. If the determination in step S22 is negative, the process returns to step S14, and the processes from step S14 onwards described above are repeated. On the other hand, if the determination in step S22 is positive, all of the processes in FIG. 6 are terminated.

(What to do when a remote connection request is made)
Next, the processing of the web portal 20 (screen providing unit 22), server 70, and recording server 60 after the processing of step S12 in FIG. 6 is executed will be described with reference to the flowchart in FIG.

In the process of FIG. 7, the screen providing unit 22 of the web portal 20 waits until it receives a request to start a remote connection from the user terminal 10 (step S102). When a request is sent from the user terminal 10, the screen providing unit 22 proceeds to step S104, creates a request to start a remote connection, and in step S106, sends the created request to the connection destination server 70 and recording server 60.

Meanwhile, the server 70 and recording server 60 wait until they receive a request (steps S502 and S602). Therefore, once the screen provider 22 executes step S106, the server 70 proceeds to step S504, and the recording server 60 proceeds to step S604.

In step S604, the recording server 60 starts recording the screen displayed on the display unit of the user terminal 10, i.e., recording normal recording data. Meanwhile, the server 70 generates screen change information (e.g., information about the initial screen after remote connection) and sends it to the screen provider 22.

Returning to the explanation of the screen providing unit 22, after step S106, the screen providing unit 22 waits until it receives screen change information in step S108. Therefore, the screen providing unit 22 proceeds to step S110 when the server 70 executes step S504.

When the process proceeds to step S110, the screen providing unit 22 transmits the screen change information received from the server 70 to the user terminal 10. Note that, as described above, when the user terminal 10 receives the screen change information (S14: Yes), it updates the screen with the received screen change information (S16).

(Processing when the user performs an operation)
Next, the processing of the web portal 20 and the server 70 when the user operates the keyboard or mouse after remote connection (when operation information is sent to the web portal 20) will be described with reference to the flowcharts of FIGS.

In the processing of FIG. 8, the screen providing unit 22 and delay monitoring unit 24 of the web portal 20 wait until they receive operation information from the user terminal 10 (steps S120 and S202). Therefore, when operation information is sent from the user terminal 10, the screen providing unit 22 proceeds to step S122 and creates an operation request. In step S124, the screen providing unit 22 also sends the created operation request to the connected server 70.

Furthermore, when operation information is transmitted from the user terminal 10, the delay monitoring unit 24 proceeds to step S204 and determines whether the user's operation is a characteristic operation. At this time, the delay monitoring unit 24 refers to the characteristic operation table in FIG. 4 and determines whether the operation related to the received operation information matches an operation listed in the characteristic operation table. If the determination in step S204 is negative, the delay monitoring unit 24 terminates the processing in FIG. 8, but if the determination in step S204 is positive, the delay monitoring unit 24 proceeds to step S206.

When the process proceeds to step S206, the delay monitoring unit 24 creates an operation information table (see Figure 5(a)), and in step S208, registers operation information data in the operation information table. For example, data is registered in the operation information table as shown in Figure 9. Note that the operation information table in Figure 9 registers operation information that user "user-001" left-clicked the mouse at coordinates (865, 312) on the screen provided by server "WorkServer-001" at 14:06:16:409 on Feb. 2, 2022. Note that at this stage, the "Response ID" column in the operation information table is empty. After step S208, the delay monitoring unit 24 proceeds to the processing in Figure 10.

Meanwhile, the server 70 waits until it receives an operation request (step S510). Therefore, once the screen provider 22 executes step S124, the server 70 proceeds to step S512 and executes processing in accordance with the operation request. In the next step S514, the server 70 transmits the screen change information generated as a result of the processing of step S512 to the web portal 20. After step S514, the server 70 terminates the processing of FIG. 8.

In response to this, after step S124, the screen providing unit 22 waits until it receives screen change information in step S126. Therefore, when the server 70 executes the processing of step S514, the screen providing unit 22 proceeds to step S128 and transmits the received screen change information to the user terminal 10. As a result, the display unit of the user terminal 10 displays a screen updated in response to the user's operation (step S16 in Figure 6).

After the processing of FIG. 8, the delay monitoring unit 24 waits in step S210 of FIG. 10 until it receives screen change information. Therefore, when the processing of step S514 is executed by the server 70, the delay monitoring unit 24 proceeds to step S212.

When the process proceeds to step S212, the delay monitoring unit 24 creates screen information data from the screen change information. Next, in step S214, the delay monitoring unit 24 registers the screen information data in the screen information table (Figure 5(b)). As a result, the screen information table becomes as shown in Figure 11(a). Note that Figure 11(a) is a screen information table of the screen information data obtained from the screen change information received at 14:06:19:343 on February 2, 2022.

Next, in step S216, the delay monitoring unit 24 updates the response ID in the operation information table. At this time, there may be multiple operation information tables with an empty response ID. In this case, the delay monitoring unit 24 stores the response ID registered in the screen information table in step S214 in the response ID column of the operation information table with the oldest transmission time among those operation information tables. For example, if the response ID in Figure 11(a) is registered in the operation information table in Figure 9, it will look like Figure 11(b).

Next, in step S218, the delay monitoring unit 24 calculates the time difference between the timestamps. Specifically, the delay monitoring unit 24 calculates the time difference between the reception time of the screen information table (Figure 11(a)) and the transmission time of the operation information table (Figure 11(b)) that corresponds to that screen information table (with a matching response ID). In the examples of Figures 11(a) and 11(b), the calculated time difference is 2934 ms.

Next, when the process proceeds to step S220, the delay monitoring unit 24 determines whether the time difference exceeds a threshold value. In this case, the delay monitoring unit 24 references the characteristic operation table in Figure 4 and determines whether the calculated time difference exceeds a threshold value (in this example, the threshold value corresponding to a left click (500 ms)).

If the determination in step S220 is negative, the delay monitoring unit 24 proceeds to step S230. In step S230, the delay monitoring unit 24 determines whether the delay flag is ON. The delay flag is a flag that indicates whether a processing delay is currently occurring between the web portal 20 and the server 70, and if it is ON, it means that a delay is occurring. In the initial stage, the delay flag is OFF. If the delay flag is OFF and the determination in step S232 is negative, all processing in Figure 10 ends.

On the other hand, if the determination in step S220 is positive, the delay monitoring unit 24 proceeds to step S222. In step S222, the delay monitoring unit 24 determines whether the delay flag is ON. If the determination here is negative, that is, if it is determined that processing has not been delayed up to now, but processing is now delayed, the unit proceeds to step S224.

When the process proceeds to step S224, the delay monitoring unit 24 creates a request for video extraction for the delay period. Then, in step S226, the delay monitoring unit 24 sends the created video extraction request to the video extraction unit 26. Then, in step S228, the delay monitoring unit 24 changes the delay flag from OFF to ON. After the process of step S228 is performed, the process of FIG. 10 ends.

On the other hand, if the time difference exceeds the threshold (S220: Yes) and the delay flag is ON, this means that the delay is continuing. In this case, the determination in step S222 is Yes, and the delay monitoring unit 24 terminates the processing of FIG. 10 without performing any further processing.

On the other hand, if the time difference does not exceed the threshold (S220: No) and the delay flag is ON, this means that the ongoing delay has been resolved. In this case, the determination in step S230 is positive, and the delay monitoring unit 24 proceeds to step S232.

When the process proceeds to step S232, the delay monitoring unit 24 creates a request to end recording for the delay period. In step S234, the delay monitoring unit 24 sends the created request to end recording for the delay period to the video extraction unit 26. In step S236, the delay monitoring unit 24 changes the delay flag from ON to OFF. After the process of step S236 is performed, the process of FIG. 10 ends.

(Processing of the video extraction unit 26 and the recording server 60)
Next, the processing of the video extraction unit 26 and the recording server 60 will be described with reference to the flowchart of FIG.

When the processing in FIG. 12 starts, the video extraction unit 26 first waits in step S302 until it receives a request to extract video for the delay period. That is, when the delay monitoring unit 24 executes the processing in step S226 in FIG. 10, the video extraction unit 26 proceeds to step S304.

When proceeding to step S304, the video extraction unit 26 calculates the time a predetermined time (e.g., three minutes) before the transmission time (timestamp) in the operation information table (Figure 11 (b)). In this embodiment, the time calculated here is set as the start of the delay period. The predetermined time may be any time sufficient to monitor the delay, and may be a time other than three minutes.

Next, in step S306, the video extraction unit 26 creates a copy request and a recording start request for the recorded data for the delay period. In step S308, the video extraction unit 26 sends the created copy request and recording request to the recording server 60.

In step S610, the recording server 60 waits until it receives a copy request or a recording start request. Therefore, once step S308 has been executed by the video extraction unit 26, the recording server 60 proceeds to step S612.

When proceeding to step S612, the recording server 60 copies the recording data for the delay period from the normal recording data. That is, the recording data from the start of the delay period to the present, as indicated by the double-headed arrow in Figure 13(a), is copied from the normal recording data.

Next, in step S614, the recording server 60 starts recording for the delay period in parallel with normal recording. That is, the newly recorded recording data is added to the copied recording data (second recording data).

Meanwhile, after step S308, the video extraction unit 26 waits in step S310 until it receives a request to end recording for the delay period. Therefore, when the delay monitoring unit 24 executes step S234 in FIG. 10, the video extraction unit 26 proceeds to step S312.

When the video extraction unit 26 proceeds to step S312, it creates a request to end recording. In step S314, the video extraction unit 26 also sends the request to end recording to the recording server 60. After step S314 is performed, the video extraction unit 26 ends the processing of FIG. 12.

In response to this, after step S614, the recording server 60 waits in step S616 until it receives a request to end recording. Therefore, once the video extraction unit 26 executes step S314, the recording server 60 proceeds to step S618. When proceeding to step S618, the recording server 60 ends and saves recording of the delay period. As a result of this processing, the recording server 60 is able to save recorded data of the screen from the start of the delay period until it detects that the delay has ended, as indicated by the dashed arrow in Figure 13(b).

After processing step S618, the recording server 60 terminates the processing shown in Figure 12.

In FIG. 6, if a request to terminate the remote connection is input from the user to the user terminal 10 (S22: Yes), the request to terminate the remote connection is sent to the web portal 20. Upon receiving the request, the web portal 20 terminates the remote connection between the user terminal 10 and the server 70 and instructs the recording server 60 to end normal recording.

If a delay occurs when the user terminal 10 is remotely connected to the server 70, the acceptance of user operations (input) and screen updates will stop. In such cases, the user cannot immediately determine whether the screen is not updating due to an operational error (such as a misclick) or because of a delay. In such cases, the user may end up repeating the same operation on a screen that is not updating.

If the user repeats the same operation in this way, after the delay is resolved, the user terminal 10 will transmit all of the operations entered up to that point to the server 70 at once. For example, suppose a delay occurs while the screen shown in FIG. 14(a) is displayed, and the user repeatedly left-clicks multiple times. Then, suppose the delay is subsequently resolved and the screen shown in FIG. 14(b) is displayed. In this case, operation information indicating that the Skip button in FIG. 14(b) has been clicked is sent from the user terminal 10 to the server 70, which may result in the processing intended by the user not being performed and causing a problem. In this embodiment, the screen during the period in which such a processing delay occurs (the delay period) can be recorded, and the cause of the problem can be easily identified by referring to the recorded data during the delay period. Furthermore, similar problems may occur due to delays regardless of the click operation, such as entering text or selecting a checkbox. Therefore, recording the recorded data during the delay period makes it possible to easily identify the cause of the problem.

As described above in detail, in this embodiment, the business system 100 includes a recording server 60 that records the screen displayed on the user terminal 10 while the user terminal 10 and the server 70 are remotely connected. The business system 100 also includes a web portal 20 that relays the remote connection between the user terminal 10 and the server 70. The recording server 60 accumulates recording data for a delay period when the time difference between the transmission time of a request (operation information) and the reception time of a response (screen change information) exceeds a threshold. The recording data for a delay period includes data recorded a predetermined time (e.g., three minutes) before the transmission time of the request (operation information) and data recorded subsequent to that data until the delay is resolved. By recording data for a delay period in this way, if a problem occurs due to a communication delay between the server 70 and the web portal 20, the administrator can refer to the recording data for the delay period. This allows the cause of the problem to be easily identified. Furthermore, in this embodiment, the recording data from before the delay was detected is included in the recording data for the delay period, so by checking this recording data, it is possible to trace back to before the delay was detected and investigate the cause of the problem.

In addition, in this embodiment, an operation information table (Figure 9) is created when the user's operation is one of the characteristic operations shown in Figure 4. This limits the operation information table to operations that can be used to determine whether a communication delay has occurred, preventing the creation of an excessively large operation information table. Furthermore, the amount of processing can be reduced compared to when determining whether a delay has occurred each time a user operation is performed.

In the above embodiment, as an example, the web portal 20 is described as being composed of one information processing device, but as mentioned above, it may also include multiple information processing devices. In this case, each of the multiple information processing devices may have the screen providing unit 22, delay monitoring unit 24, and video extraction unit 26 of FIG. 3. Furthermore, the functions of the screen providing unit 22, delay monitoring unit 24, and video extraction unit 26 of FIG. 3 may be performed by separate information processing devices.

Note that the server 70 in the above embodiment may also be another information processing device, such as an office PC, that can be remotely connected from the user terminal 10.

The above processing functions can be realized by a computer. In this case, a program is provided that describes the processing details of the functions that the processing device should have. By executing the program on a computer, the above processing functions are realized on the computer. The program that describes the processing details can be recorded on a computer-readable storage medium (excluding carrier waves, however).

When distributing a program, it is sold, for example, in the form of a portable storage medium on which the program is recorded, such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory). The program can also be stored in the storage device of a server computer, and then transferred from the server computer to other computers via a network.

A computer that executes a program stores, for example, a program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable storage medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the received program each time a program is transferred from the server computer.

The above-described embodiment is a preferred example of the present invention. However, it is not limited to this, and various modifications are possible within the scope of the gist of the present invention.

In addition, the following supplementary notes are disclosed in relation to the above-described embodiments.
(Supplementary Note 1) A relay system that relays between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal,
a recording device that records an operation screen displayed on the terminal and stores first recording data while the terminal and the processing device are connected;
a transmitting/receiving device that, when receiving a request from the terminal to the processing device, transmits the request to the processing device, and, when receiving a response corresponding to the request from the processing device, transmits the response to the terminal;
The recording device is
A relay system characterized in that, when the time difference between the time when the transmitting/receiving device sends a request and the time when the transmitting/receiving device receives a response corresponding to the request exceeds a reference value, second recording data is generated by duplicating recording data from the first recording data that was recorded after a predetermined time before the time when the request was sent, and recording data obtained by recording the operation screen while the time difference exceeds the reference value is added to the second recording data.
(Appendix 2) The relay system described in Appendix 1 is characterized in that the recording device generates the second recording data when the time difference between the time when a request of a predetermined type is sent and the time when a response corresponding to the request is received exceeds a reference value.
(Supplementary Note 3) The relay system according to Supplementary Note 2, wherein the predetermined type of request is a request that causes a transition of the operation screen.
(Supplementary Note 4) The relay system according to any one of Supplementary Notes 1 to 3, wherein the reference value is set for each content of the request.
(Supplementary Note 5) A relay system that relays between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal,
While the terminal and the processing device are connected, an operation screen displayed on the terminal is recorded and first recorded data is stored;
When the time difference between the time when a request is transmitted from the terminal and the time when a response corresponding to the request is received from the processing device exceeds a reference value, second recorded data is generated by duplicating the first recorded data recorded after a predetermined time before the time when the request is transmitted, and the recorded data obtained by recording the operation screen while the time difference exceeds the reference value is added to the second recorded data.
2. An information recording method comprising:
(Appendix 6) An information recording method as described in Appendix 5, characterized in that the second recording data is generated when the time difference between the time when a request of a predetermined type is sent and the time when a response corresponding to the request is received exceeds a reference value.
(Supplementary Note 7) The information recording method according to Supplementary Note 6, wherein the predetermined type of request is a request that causes a transition of the operation screen.
(Supplementary Note 8) The information recording method according to any one of Supplementary Notes 5 to 7, wherein the reference value is set for each content of the request.
(Supplementary Note 9) A transmitting/receiving device that transmits and receives a request and a response between a terminal and a processing device that executes a process in response to a request from the terminal and transmits a response to the terminal,
While the terminal and the processing device are connected, an instruction is issued to a recording device connected to the transmitting/receiving device, an operation screen displayed on the terminal is recorded, and first recording data is accumulated;
when the time difference between the time when a request is transmitted from the terminal and the time when a response corresponding to the request is received from the processing device exceeds a reference value, an instruction is issued to the recording device to generate second recorded data by duplicating the first recorded data recorded after a predetermined time before the time when the request is transmitted, and to add to the second recorded data the recorded data obtained by recording the operation screen while the time difference exceeds the reference value;
An information recording program that causes a process to be executed.

10 User terminal (terminal)
20 Web portal (transmitting/receiving device)
50 Relay system 60 Recording server (recording device)
70 Server (processing device)

Claims (6)

A relay system that relays between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal,
a recording device that records an operation screen displayed on the terminal and stores first recording data while the terminal and the processing device are connected;
a transmitting/receiving device that, when receiving a request from the terminal to the processing device, transmits the request to the processing device, and, when receiving a response corresponding to the request from the processing device, transmits the response to the terminal;
The recording device is
A relay system characterized in that, when the time difference between the time when the transmitting/receiving device sends a request and the time when the transmitting/receiving device receives a response corresponding to the request exceeds a reference value, second recording data is generated by duplicating recording data from the first recording data that was recorded after a predetermined time before the time when the request was sent, and recording data obtained by recording the operation screen while the time difference exceeds the reference value is added to the second recording data. The relay system of claim 1, wherein the recording device generates the second recording data when the time difference between the time a request of a predetermined type is sent and the time a response corresponding to that request is received exceeds a reference value. The relay system described in claim 2, characterized in that the predetermined type of request is a request that causes a transition of the operation screen. The relay system described in any one of claims 1 to 3, characterized in that the reference value is set for each content of the request. a relay system that relays between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal;
While the terminal and the processing device are connected, an operation screen displayed on the terminal is recorded and first recorded data is stored;
When the time difference between the time when a request is transmitted from the terminal and the time when a response corresponding to the request is received from the processing device exceeds a reference value, second recorded data is generated by duplicating the first recorded data recorded after a predetermined time before the time when the request is transmitted, and the recorded data obtained by recording the operation screen while the time difference exceeds the reference value is added to the second recorded data.
2. An information recording method comprising: a transmitting/receiving device that transmits and receives the request and the response between a terminal and a processing device that executes processing in response to a request from the terminal and transmits a response to the terminal;
While the terminal and the processing device are connected, an instruction is issued to a recording device connected to the transmitting/receiving device, an operation screen displayed on the terminal is recorded, and first recording data is accumulated;
when the time difference between the time when a request is transmitted from the terminal and the time when a response corresponding to the request is received from the processing device exceeds a reference value, an instruction is issued to the recording device to generate second recorded data by duplicating the first recorded data recorded after a predetermined time before the time when the request is transmitted, and to add to the second recorded data the recorded data obtained by recording the operation screen while the time difference exceeds the reference value;
An information recording program that causes a process to be executed.