JP7762926B2 - Screen display system and screen display method - Google Patents

Description

This invention relates to a screen display system and a screen display method that transfers screen information generated by an apparatus to a terminal and displays it.

Using a GUI (Graphical User Interface) screen allows users to intuitively and easily operate devices. By transferring the display data of this GUI screen between devices, users can, for example, use a portable device to operate the GUI screen on the device, making it easy to operate the device.

Conventional technology includes a technique for reducing the amount of GUI data stored in the storage unit of a terminal with memory capacity constraints by storing only the glyph data used for GUI display in the storage unit (see, for example, Patent Document 1 below). In addition, in remote desktop, when a screen image on a remote terminal is changed, a technique exists for reconstructing the screen by sending the differential screen image, area information for the change, and drawing area information for the difference to the local terminal (see, for example, Patent Document 2 below).

JP 2008-117319 A JP 2010-146504 A

Embedded devices that perform predetermined controls can easily perform control operations by displaying the control content on a GUI screen. However, if the configuration also performs drawing processing in addition to generating the GUI screen, the number of hardware components in the embedded device increases, making it difficult to reduce costs. For example, in a system configuration in which a terminal is connected to the embedded device via communication, it is necessary to efficiently separate the GUI functions located on the embedded device from the GUI functions located on the terminal. Furthermore, with conventional technology, image data displayed on a remote terminal is sent to a local terminal, resulting in an increase in the amount of data transferred.

In consideration of the above problems, the present invention aims to enable a display screen showing the status of a device to be transferred to a terminal without increasing the amount of data.

To achieve the above object, the screen display system of the present invention is a screen display system that transfers screen information from a device to a terminal, wherein the device has a control unit that generates screen information indicating the control status of the device itself and a communication unit that transmits the screen information generated by the control unit to the terminal, and the terminal has a processing unit that generates a display screen based on the screen information received from the device and a display unit that displays the display screen generated by the processing unit, and the control unit of the device shares initial display list information, including text and images related to GUI (Graphical User Interface) screen construction and the placement of each component, with the terminal in an initial state, and after the device is started, selects screen information including the text and images corresponding to a specified state from the initial display list information and transmits it to the terminal, and the processing unit of the terminal references the shared initial display list information based on the received screen information and performs screen display processing of the device's operating status.

Furthermore, the device does not have a display unit for the screen information within itself, and the terminal refers to the shared initial display list information based on the received screen information, performs screen display processing corresponding to the state of the device, and displays it on the display unit.

Furthermore, the device assigns an identifier to each of the strings and images included in the initial display list information and transmits the identifier to the terminal as the screen information, and the terminal refers to the shared initial display list information based on the identifier of the received screen information and performs screen display processing corresponding to the state of the device.

The device is also characterized in that it transmits information regarding the shape, character string, display coordinates, and color of each component to the terminal as the initial display list information.

Furthermore, the device updates the screen information in response to changes in the state, generating only information regarding the shape, text, display coordinates, and color of the component to be updated as the screen information and transmitting it to the terminal.

Furthermore, the terminal has an operation unit, and transmits operation information from the operation unit to the device, and the device performs control based on the operation information received from the terminal, updates the screen information based on the control, and transmits the updated screen information to the terminal.

Furthermore, the terminal is communicatively connected to multiple devices, displays display information of the communicatively connected devices on the display unit, and controls the devices by operating the operation unit.

Furthermore, the device is characterized as being an embedded device that performs specified control with limited uses.

Furthermore, the screen display method of the present invention is a screen display method for transferring screen information from a device to a terminal, in which the device generates screen information indicating its own control status and transmits the generated screen information to the terminal, the terminal displays a display screen based on the screen information received from the device, the device shares initial display list information including text and images related to GUI (Graphical User Interface) screen construction and the layout of each component with the terminal in its initial state, and after the device is started, selects screen information including the text and images corresponding to a predetermined state from the initial display list information and transmits it to the terminal, and the terminal references the shared initial display list information based on the received screen information and performs screen display processing of the device's operating status.

According to the above configuration, the screen display system shares initial display list information with the terminal when the device is in its initial state. After the device is operational, the device selects screen information corresponding to a specific state from the initial display list information and sends it to the terminal, allowing the terminal to display a display screen indicating the device's state on the display unit. In this way, by sharing initial display list information between the device and the terminal in advance, the amount of data required for the screen information displayed on the terminal after the device is operational can be reduced.

The GUI screen information sent from the device to the terminal consists of the shape, text characters, coordinates, color, etc. of each object, and is a small amount of data, rather than the image data itself. This allows GUI screen data to be transferred from the device to the terminal with a small amount of data, reducing the processing load on the device. Furthermore, when updating the GUI screen, the device sends screen information such as the shape, text characters, coordinates, color, etc. of only the updated objects to the terminal. This further reduces the amount of data transferred from the device to the terminal when updating the GUI screen.

The device is, for example, an embedded device that performs specific controls for limited uses and does not have a display unit. The user can understand the device's control status on the display screen of the display unit on their terminal and control the device by operating the operation unit. This eliminates the need for screen drawing on the device, reducing the processing load and enabling lower device costs.

The present invention has the effect of allowing a display screen showing the status of the device to be transferred to a terminal without increasing the amount of data.

FIG. 1 is a diagram illustrating an example of a configuration of a screen display system according to an embodiment. FIG. 2 is a diagram illustrating functions of the screen display system according to the embodiment. FIG. 3 is a block diagram illustrating an example of a hardware configuration of the screen display system according to the embodiment. FIG. 4 is a diagram for explaining the transfer state of display data by the screen display system when the system is started up. FIG. 5 is a diagram for explaining the transfer state of a list of character strings and images by the screen display system at the time of system startup. FIG. 6 is a diagram for explaining the updated state of character strings and images by the screen display system after the system is started up. FIG. 7 is a diagram illustrating display and display update by the screen display system. (Part 1) FIG. 8 is a diagram illustrating display and display update by the screen display system. (Part 2) FIG. 9 is a flowchart showing the display and display update process by the screen display system (part 1). FIG. 10 is a flowchart showing the display and display update process by the screen display system (part 2). FIG. 11 is a flowchart showing a transfer state during display and display update according to the prior art. (Part 1) FIG. 12 is a flowchart showing the transfer state during display and display update according to the conventional technology (part 2). FIG. 13 is an explanatory diagram illustrating an example of a screen display on a terminal by the screen display system according to the embodiment (part 1). FIG. 14 is an explanatory diagram illustrating an example of a screen display on a terminal by the screen display system according to the embodiment (part 2). FIG. 15 is a diagram showing an example of the configuration of a conventional embedded device. FIG. 16 is a diagram illustrating another function of the screen display system according to the embodiment. FIG. 17 is a diagram for explaining the creation state of a conventional GUI application. FIG. 18 is a diagram illustrating an example of an update process of a GUI application in the screen display system according to the embodiment. FIG. 19 is an explanatory diagram for comparing development man-hours for different types of embedded devices. (Part 1) FIG. 20 is an explanatory diagram for comparing development man-hours for different types of embedded devices (part 2). FIG. 21 is an explanatory diagram for comparing development man-hours for different types of embedded devices (part 3).

(Embodiment)
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a screen display system and a screen display method according to the present invention will be described in detail below with reference to the accompanying drawings.

Figure 1 is a diagram showing an example configuration of a screen display system according to an embodiment. The screen display system 100 includes an embedded device (local device) 110 and a terminal 120 (remote device).

Embedded device 110 is a device for a specific purpose with a limited use (operation), such as a remote control, digital camera, IC recorder, in-vehicle device, medical device, surveillance camera, or intrusion detector. Terminal 120 is a device (smart device) that can be carried and moved by a user, such as a smartphone or tablet. Terminal 120 and embedded device 110 are connected for communication, for example, wirelessly via Wi-Fi (registered trademark).

Sensors A and B (111), LED1 (112), etc. are connected to the embedded device 110, and a control unit (not shown) generates GUI screen information. It also has a communication unit that transmits the GUI screen information generated by the control unit to the terminal 120. For example, the control unit of the embedded device 110 performs predetermined processing based on the detection of sensors A and B (111) connected to the embedded device 110, and displays the processing results on LED1 (112), etc. The control unit of the embedded device 110 also performs predetermined control operations to control the embedded device 110 based on operation information from the terminal 120.

In this control operation, the control unit of the embedded device 110 generates GUI screen information regarding the control status, for example, the on/off status of LED1 (112), based on the values detected by sensors A and B (111) and information on user operations.

Here, the embedded device 110 does not display the generated GUI screen information within its own device, but transmits it to the terminal 120 and displays it on the terminal 120. In other words, the embedded device 110 does not have a display unit.

The terminal 120 has a communication unit that receives GUI screen information transmitted from the embedded device 110, a control unit that controls the display of the GUI screen information, and a display unit 121 that displays the GUI screen information. The display unit 121 can also receive user operations via a touch panel or the like, and the control unit transmits operation information based on the user operation input to the embedded device 110.

The above-mentioned screen display system 100 transmits GUI screen information generated by the embedded device 110 to the terminal 120 and displays the GUI screen on the display unit 121 of the terminal 120. In other words, the GUI screen controlled and processed by the embedded device 110 is displayed on the display unit 121 of the terminal 120.

Figure 2 is a diagram showing the functions of the screen display system according to the embodiment. It shows the internal functions of the embedded device 110 and the terminal 120. Figure 2 mainly describes functions related to the display of the GUI screen.

The embedded device 110 has a GUI application 201 that is executed by the control unit. The GUI application 201 includes a screen construction processing unit 211 that constructs a GUI screen. The screen construction processing unit 211 generates component placement information k for components included in the constructed GUI screen and transmits it to the terminal 120 as GUI screen information K.

GUI screen information K (component placement information k) includes the placement positions of the GUI screen components displayed on the display unit 121 of the terminal 120, for example, the on/off positions of sensors A and B (111) and LED 1 (112) shown on the GUI screen displayed on the display unit 121 of the terminal 120 in Figure 1, and includes, for example, coordinate information on the GUI screen.

The terminal 120 includes a screen drawing processing unit 220. The screen drawing processing unit 220 performs screen drawing processing based on the GUI screen information K received from the embedded device 110 and displays and outputs the GUI screen on the display unit 121.

When the values of sensors A and B (111) connected to the embedded device 110 change, the displayed contents of the sensor values corresponding to sensors A and B (111) change on the GUI screen displayed on the display unit 121 of the terminal 120. Similarly, the on/off state of LED1 (112) on the embedded device 110 is displayed on the display unit 121 of the terminal 120 (see Figure 1 for the display state).

Furthermore, by operating the operation unit 221 provided on the terminal 120, operation information S can be sent to the embedded device 110. For example, the on/off state of LED1 (112) on the GUI screen displayed on the display unit 121 shown in Figure 1 can be changed by button operation. At this time, the GUI application 201 provided on the embedded device 110 performs display control to update the GUI screen based on the operation information S from the terminal 120.

Figure 3 is a block diagram showing an example of the hardware configuration of a screen display system according to an embodiment. It shows an example of the hardware configuration of each of the embedded device 110 and the terminal 120 described above.

The embedded device 110 includes a control unit (CPU) 301, a read-only memory (ROM) 302, a random access memory (RAM) 303, a storage unit 304 such as a semiconductor memory or a disk drive, and a communication interface (I/F) 305. The CPU 301 to the communication I/F 305 are connected to each other via a bus 308.

The CPU 301 is an arithmetic processing unit that controls the entire embedded device 110. The CPU 301 executes and processes the GUI application 201, generates the above-mentioned GUI screen, and transfers it to the terminal 120. The ROM 302 is a non-volatile memory that stores the program of the GUI application 201, etc. The RAM 303 is a volatile memory that is used as a work area when the CPU 301 executes the arithmetic processing of the program of the GUI application 201. The memory unit 304 can store and hold data for executing the functions of the embedded device 110, other applications, etc.

The communication interface 305 serves as the interface for network connection with the terminal 120 and controls the input and output of information to be communicated. Specifically, the communication interface (I/F) 305 is connected to a network 310 such as a wireless/wired local area network (LAN), a wide area network (WAN), or the Internet, and communicatively connects the embedded device 110 to the terminal 120.

The embedded device 110 has the function of generating GUI screen information K (component placement information k) for displaying a GUI screen by the CPU 301 executing the GUI application 201 program, and transmitting the generated GUI screen information K to the terminal 120. The function of transmitting the GUI screen information K to the terminal 120 and receiving operation information S from the terminal 120 can be realized by the above-mentioned communication I/F 305.

The terminal 120 also includes a control unit (CPU) 311, a ROM 312, a RAM 313, a storage unit 314 such as a semiconductor memory, a communication I/F 315, a touch panel 316, and a display 317. The CPU 311 to the display 317 are each connected by a bus 318.

The touch panel 316 also has buttons for operating the embedded device 110. The touch panel 316 corresponds to the operation unit 221 in Figure 2. The display 317 is a display device that displays a GUI screen based on program execution by the CPU 311, and corresponds to the display unit 121 in Figures 1 and 2. The display 317 can be, for example, a thin film transistor (TFT) liquid crystal display unit, a plasma display unit, or an organic EL display unit. The touch panel 316 is disposed on the display 317, and by operating the buttons on the GUI screen displayed on the display 317 on the touch panel 316, operation information S at the time of button operation can be output.

In this way, the terminal 120 can also be configured with hardware including the CPU 311, ROM 312, RAM 313, etc., as shown in Figure 3. The terminal 120 has the function of receiving GUI screen information K (component placement information k) from the embedded device 110 and drawing a GUI screen on the display 317 based on the received GUI screen information K. The screen drawing function can be realized by the execution of a program by the CPU 311. Furthermore, the function of receiving GUI screen information K from the embedded device 110 and transmitting operation information S of the touch panel 316 to the embedded device 110 can be realized by the communication I/F 315. The terminal 120 can be, for example, a general-purpose tablet, smartphone, portable PC, etc.

Figure 4 is a diagram illustrating the transfer state of display data when the screen display system is started up. In the embodiment, in the initial state of system startup, the local device (embedded device) 110 and the remote device (terminal) 120 share static data related to display (images, character strings, etc.).

For example, when the system is started, the local device (embedded device) 110 transfers information (initial display list information) of a list of strings and images (images, strings, etc.) as static data for display stored in the memory unit 304 to the remote device (terminal) 120 via the network 310 in one go.

In addition, before the system is started, information on a list of strings and images (images, strings, etc.) may be transferred from the cloud server 400, etc. to the terminal 120 as static data for display.

This static data for display (a list of text strings and images) is stored as the same data in the memory unit 304 of the embedded device 110 and the memory unit 314 of the terminal 120 at the time the system is started.

This allows the embedded device 110 to similarly retain the static display data that the terminal 120 has at system startup. The same static display data that both the terminal 120 and the embedded device 110 have at system startup is the GUI screen information at system startup, and is referred to as initial display list information K0.

Figure 5 is a diagram illustrating the transfer state of a list of character strings and images by the screen display system when the system is started up. Figure 5 shows an example of a list 500 of character strings and images corresponding to the initial display list information K0 when the system is started up. Figure 5 shows an example of a list 500 of character strings and images of the initial display list set in the initial display list information K0 being transferred in bulk from the embedded device 110 to the terminal 120 via the network when the system is started up.

The list of strings and images 500 includes a list of strings 510 and a list of images 520. The list of strings and images 500 is, for example, information on a list of strings and a list of images that can be set in the embedded device 110. It is not limited to this, and may also be information on a list of strings and a list of images that are set in the embedded device 110 at the time the system is started.

The example string and image list 500 shown in Figure 5 is a list of strings and images set in the embedded device 110 at system startup. The embedded device 110 shows the setting state of some of the strings and images. Here, since the embedded device 110 does not have a display unit, the string "ABC" and the image "star (★) shape" are not actually displayed on the embedded device 110 side in Figure 5.

The embedded device 110, for example, stores a list of various strings of characters and images required for operation in advance.

In the string list 510, the embedded device 110 stores string information by number (record). For example, number 1 is the string "aaa" and number 2 is the string "ABC".

In the image list table 520, the embedded device 110 stores image information by number (record). For example, number 1 is the image "star (★) shape", and number 2 is the image "triangle (▲) shape".

The terminal 120 can display the image information set on the embedded device 110 side on the display unit 121 of the terminal 120 by transferring initial display list information K0 consisting of information on a list 500 of character strings and images from the embedded device 110.

The numerical value "100" on the embedded device 110 is, for example, a sensor value detected by the embedded device 110 using a sensor or the like after the system is started. After the system is started as described below, the embedded device 110 sequentially transfers the sensor values detected over time to the terminal 120.

Furthermore, due to functional expansion of the embedded device 110, the character strings and images handled by the embedded device 110, i.e., the character strings and images stored in the embedded device 110, may be updated. In this case, the initial display list information K0 for the character strings and images is updated for the embedded device 110. Then, the updated initial display list information K0 can be transferred in bulk from the embedded device 110 or the cloud server 400 to the remote device (terminal) 120.

Figure 6 is a diagram illustrating the update state of strings and images by the screen display system after system startup. In the example of Figure 6, at a certain point after system startup, the embedded device 110's operational transition selects the string "ABC" and the image "star (★) shape" from the list of strings and images 500.

Based on the passage of time (operational transition) after system startup, the embedded device 110 selects from the acquired initial display list information K0 the strings of characters and images corresponding to the changing display content (changes in setting content, as this is not actually displayed) and notifies the terminal 120 of this as GUI screen information K (component placement information k).

Here, the number "100" on the embedded device 110 is the sensor value detected by the embedded device 110 using a sensor or the like after the system is started.

In the example shown in Figure 6, assume that the embedded device 110 selected the corresponding string "ABC" and image "star (★) shape" from the string list 510 at a certain point in time. In this case, the embedded device 110 transfers the string number "2" from the string list 510 and the display image number "1" from the image list 520 as GUI screen information K to the terminal 120. The embedded device 110 also includes the detected sensor value "100" in the GUI screen information K and transfers it to the terminal 120.

The embedded device 110 includes information about the placement of characters and components in GUI screen information K (e.g., the placement and area (size) of images and character strings, and the overlapping state of images and characters). In the example of Figure 6, GUI screen information K includes the position coordinates (x, y), size (w, h), and character string number "2" for the character string "ABC."

GUI screen information K also includes the position coordinates (x, y), size (w, h), and display image number "1" for the image "star (★) shape." GUI screen information K also includes the position coordinates (x, y), size (w, h), and display value "100" for the sensor value "100."

In this way, the embedded device 110 notifies the terminal 120 of GUI screen information K including the selected characters, images, and sensor values, so that the terminal 120 can display the information of the screen selected by the embedded device 110 on the display unit 121.

After this, the embedded device 110 notifies the terminal 120 of the updated GUI screen information K each time a change is made to a character, image, or sensor value. At this time, the embedded device 110 only needs to notify the terminal 120 of the number (identifier) that identifies the character string and image to be updated for the character string and image in the GUI screen information K, thereby reducing the amount of data to be transferred.

In addition, the terminal 120 has already acquired information about the images to be displayed by transferring initial display list information K0, which consists of information about the list of character strings and images 500, when the system is started up. Therefore, the terminal 120 can display the image using only the character string and image number to be updated, allowing for simple display processing. Thus, according to the embodiment, when updating an image, there is no need to transfer the entire data (pixel image) sequentially, which reduces the amount of data to be transferred.

(GUI screen transfer process)
7 and 8 are diagrams illustrating display and display update by the screen display system. Fig. 7 shows the state of the GUI screen update operation. The embedded device 110 updates the GUI screen based on, for example, a change in the detected state of the sensor value. However, the embedded device 110 may also update the GUI screen based on, for example, an operation by the terminal 120.

Figure 7 shows an example in which the embedded device 110 updates a GUI screen by operating the terminal 120 displaying the GUI screen. The area to be updated is touched on the touch panel of the display unit 121 of the terminal 120. For example, assume that an operation is performed to change the display coordinate position (x, y) of a square (□) shaped component. In this case, the terminal 120 touches the square (□) shaped component on the GUI screen and performs a drag operation or the like to change the display coordinate position (x, y), thereby transmitting operation information S including the updated coordinates (tx, ty) to the embedded device 110. The terminal 120 can similarly transmit operation information S to the embedded device 110 for updating the shape, text string, position (coordinates), color, etc. of other components by performing similar operations.

Figure 8 shows the transfer status of the updated GUI screen. On the embedded device 110 side, the updated GUI screen is transferred to the terminal 120 by operations on the terminal 120, etc.

The embedded device 110 updates the GUI screen based on operations on the terminal 120, etc. In the example shown in Figure 8, for a square (□) shaped component, information on the coordinate position (x', y') of the updated display is generated. Also, for a triangular (△) shaped component, information on the color being changed to yellow is generated. Also, information on the changed character string "aiu" is generated.

Then, the embedded device 110 transmits only the information about the updated (changed) component to the terminal 120 as component placement information k'. In the example of Figure 6, the embedded device 110 transmits to the terminal 120, as component placement information k', information about the coordinate position (x', y') of the updated display of the square (□) shaped component, information about the triangular (△) shaped component after its color has been changed to yellow, and information about the changed character string "aiu".

In this way, when the GUI screen is changed (updated), the embedded device 110 transmits only the component placement information k' updated on the GUI screen to the terminal 120. Here, the component placement information k' transmitted during an update simply transmits updated number information for the text string and image, as shown in Figure 6. This eliminates the need for the embedded device 110 to transmit image data for the entire GUI screen each time the GUI screen is updated, and instead only transmits the minimum amount of component placement information k' required to update the display screen, thereby reducing the amount of data transferred.

Figures 9 and 10 are flowcharts showing the display and display update processing by the screen display system. They mainly show the processing performed by the CPU 301 on the embedded device 110 side and the CPU 311 on the terminal 120 side, respectively.

Figure 9 shows the state at system startup. First, the embedded device 110 determines whether the system is starting up (step S901). For example, the embedded device 110 waits until it starts up or until it establishes a communication connection with the terminal 120 (step S901: No). If it starts up or establishes a communication connection with the terminal 120 (step S901: Yes), it performs the following processing.

Next, the embedded device 110 acquires a list of character strings and a list of images for image display (step S902). For example, the information for the list of character strings and the list of images is set by external input to the embedded device 110 via the communication I/F 305 or by operation on the device itself. Note that the embedded device 110 may previously store the information for the list of character strings and the list of images in a non-volatile state.

The list of text strings and images set in step S902 may be the minimum number of text strings and images used by the embedded device 110. For example, the text strings may be limited to text strings such as "Sensor A," "Sensor B," and "LED1" for identifying sensors A and B (111) and LED1 (112) connected to the embedded device 110 shown in Figure 1. Furthermore, the images may be limited to images such as "square (□) shape," "circle (◯) shape," and "star (★) shape" for identifying the state of the detected values of sensors A and B (111) and LED1 (112) connected to the embedded device 110 shown in Figure 1. Furthermore, color information may be assigned to each image.

Next, the embedded device 110 generates a list of the set strings and images as initial display list information K0 at system startup (step S903). The embedded device 110 stores and retains the generated initial display list information K0 at system startup in the memory unit 304. The embedded device 110 then transfers the generated initial display list information K0 at system startup to the terminal 120 via the communication I/F 305 (step S904).

The terminal 120 stores the initial display list information K0 transferred from the embedded device 110 in the memory unit 314 (step S905).

This allows the embedded device 110 and the terminal 120 to share the same initial display list information K0.

Figure 10 shows the state when the GUI screen is updated. When updating the GUI screen information K, the embedded device 110 transfers the updated GUI screen information K to the terminal 120. For example, when the embedded device 110 receives operation information S from the terminal 120, it updates the GUI screen information K based on the operation information S.

In the example shown in Figure 10, the coordinate position of the image of a triangular (△) shaped component and the information on the character string have been updated by operating the terminal 120. For the image of the triangular (△) shaped component, the coordinate position has been changed from (x, y) to (x', y'). Also, the character string has been changed from "ABCDE" to "XYZ."

In this case, the embedded device 110 generates only the image and text information of the updated component as component placement information k' (step S1001). In the example of Figure 10, the embedded device 110 generates triangular (△) shape information (unchanged), new coordinate position (x', y'), and color (e.g., blue) information (unchanged). Also, for the character string component, it generates new character string "XYZ" information, display coordinate position (x, y) information (unchanged), and color (e.g., white) information (unchanged).

Then, the CPU 301 of the embedded device 110 transmits component placement information k' for only the components that have been changed due to the update to the terminal 120 via the communication I/F 305 (step S1002).

The terminal 120 receives the component placement information k' sent from the embedded device 110 via the communication I/F 315 (step S1003). The terminal 120 then performs screen drawing processing for the received component placement information k (step S1004).

At this time, the terminal 120 retains the previous GUI screen information K and performs the drawing process by changing only the display state of the component parts that have been updated upon reception.

In other words, the terminal 120 draws unupdated circular (◯) and square (□) shaped components without making any changes. Then, for updated triangular (△) shaped components, the terminal 120 draws the same triangular (△) shape display position as before the update at the new coordinate position (x', y') and in the same color as before the update (e.g., blue). For updated character string components, the terminal 120 draws the new character string "XYZ" at the same coordinate position (x, y) as before the update and in the same color as before the update (e.g., white).

In this way, the embedded device 110 transfers GUI screen information K to the terminal 120 as part placement information k consisting of data for each updated component part, eliminating the need to send image data (pixel image) for the entire screen each time an update is made. In the example of Figure 10, only part placement information k' for part P that has been changed due to the update is generated and transferred from the embedded device 110 to the terminal 120, thereby reducing the amount of data to be transferred. Furthermore, as shown in Figure 5, for the GUI screen information K (updated part placement information k') to be transferred, only the updated character string and image number information (identifier information) need to be transferred, further reducing the amount of data to be transferred.

As a result, the embedded device 110 does not need to send image data for the entire screen each time the GUI screen is updated, but instead sends only the minimum amount of component placement information k' required to update the display screen, making the amount of data transferred between the embedded device 110 and the terminal 120 even smaller than the amount of component placement information k. Furthermore, the terminal 120 can also update the display unit 121 with a minimum amount of data, reducing the processing load for drawing.

(Example of comparison with conventional image data transfer)
11 and 12 are flowcharts showing the transfer state during display and display update according to the prior art.

Figure 11 shows the transfer status of a display screen from device 1101 to terminal 1111. Device 1101 creates display data G (step S1101) and performs a drawing process on the display screen of device 1101 (step S1102). It also transmits the created display data G to terminal 1111 (step S1103).

The terminal 1111 receives the display data G transmitted from the device 1101 (step S1104). The terminal 1111 then performs screen drawing processing of the received display data G (step S1105) and displays it on the display screen 1112.

Figure 12 shows the transfer status of the updated GUI screen. When updating the display screen, the device 1101 transfers the updated display data G' to the terminal 1111.

As shown in Figure 12, display data G' has updated coordinate positions of triangular (△)-shaped components and text information compared to display data G before the update. For the triangular (△)-shaped component, the coordinate position has changed from (x, y) to (x', y'). The text has also changed from "ABCDE" to "XYZ."

However, in conventional technology, even when device 1101 updates (changes) display data G, the screen drawing process is the same as in Figure 11, as shown in Figure 12. As a result, device 1101 creates updated display data G' for all components (step S1201) and performs drawing processing on the display screen of device 1101 (step S1202). The generated display data G' is also sent to terminal 1111 (step S1203).

Terminal 1111 receives display data G' transmitted from device 1101 (step S1204). Terminal 1111 then performs screen drawing processing of the received display data G' (step S1205) and displays it on display screen 1112.

Therefore, the display data G' generated when updating the display screen shown in Figure 12 has the same data size as the display data G before the update. In other words, display data G and display data G' have a data volume corresponding to the entire image (all pixels) of the display screen 1112.

As such, in conventional technology, image data for the entire screen is transmitted, so the data size of display data G transferred from device 1101 to terminal 1111 is larger than in the embodiment. Furthermore, even when updating the GUI screen, image data for the entire screen is transmitted, so the data size of display data G' transferred from device 1101 to terminal 1111 is the same as display data G, but is larger than in the embodiment.

(Example of remote display of GUI screen according to embodiment)
Various application examples in which the GUI screen of the above-described screen display system (embedded device 110) is displayed on the terminal 120 will be described below.

Figures 13 and 14 are explanatory diagrams showing examples of screen displays on a terminal using the screen display system of the embodiment.

Figure 13 shows an example in which GUI screen information K generated by an embedded device 110 in a factory 1300 is displayed on the display unit 121 of a terminal 120. The embedded device 110 is, for example, a part of a production line for parts produced in the factory 1300. By carrying the terminal 120, a user can view the display unit 121 and operate the embedded device 110 at a location away from the embedded device 110, without having to operate it directly at the embedded device 110's location.

For example, if the embedded device 110 is installed in a clean room, the terminal 120 can display the control status of the embedded device 110 even if it is located outside the clean room. This makes it possible to operate the embedded device 110 from a remote location, without being limited to the installation location of the embedded device 110.

In an embodiment, the embedded device 110 does not need to have a display screen; instead, the user can operate the embedded device 110 by looking at the display unit 121 of the terminal 120. In this case, since the embedded device 110 does not need to have a display unit, costs can be reduced. Furthermore, by communicatively connecting multiple embedded devices 110 to the terminal 120, the user can switch between embedded devices 110 while looking at the display unit 121 of one terminal 120, and monitor the operating status, etc., of the switched embedded device 110, displaying it on the display unit 121 of the terminal 120.

Figure 14 shows an example in which GUI screen information K generated by an embedded device 110 in a certain factory 1401 is displayed on the display unit 121 of a terminal 120 in another factory 1402. The embedded device 110 is, for example, a part of a production line for parts produced in the factory 1401. By operating the terminal 120 in the other factory 1402, a user can operate the embedded device 110 while viewing the display unit 121 at a location away from the embedded device 110, without having to operate it directly at the embedded device 110's location.

For example, if the embedded device 110 is installed in a factory 1401 in another country, the GUI screen information K of the embedded device 110 can be displayed by operating the terminal 120 located at the factory 1402 in the home country. This allows the embedded device 110 to be operated from a remote location, without being limited to the installation location of the embedded device 110, and enables, for example, remote maintenance of the embedded device 110 to be performed using the terminal 120.

(Comparison of the conventional technology and the embedded device of the embodiment)
15 is a diagram showing an example of the configuration of a conventional embedded device. The diagram shows a conventional embedded device 1501 having the same functions as the embodiment (see FIG. 2). The conventional embedded device 1501 includes a sensor 1511, an LED 1512, a GUI application 1521, a display unit 1541, and an operation unit 1542.

The GUI application 1521 includes a screen construction processing unit 1531 and a screen drawing processing unit 1532 to display a GUI screen on the display unit 1541 based on the component placement information k.

As such, conventional embedded devices 1501 have hardware such as an operation unit 1542, such as a touch panel, and a display unit 1541, such as an LCD screen, which increases the weight and size of the embedded device 1501 and increases component costs, making it difficult to reduce costs. Furthermore, the screen drawing process performed by the screen drawing processing unit 1532 places a heavy processing load on the device and requires a large amount of memory. Therefore, executing the GUI application 1521 requires a high-performance control unit (a microcomputer such as a CPU) and a large amount of memory, making it difficult to reduce the cost of the embedded device 1501.

In contrast, in an embodiment, as shown in FIG. 2, the screen drawing processing unit 220 and display unit 121 are provided on the terminal 120 side, not on the embedded device 110. The embedded device 110 transmits GUI screen information K (component placement information k) to the terminal 120. In this way, the embedded device 110 can reduce costs by providing the hardware required for screen drawing processing and display in a general-purpose smart device (terminal 120). Furthermore, the embedded device 110 can reduce the processing load and memory usage required for screen drawing processing and display.

In addition, according to the embodiment, it is possible to eliminate the waste of displaying the same GUI screen on the embedded device 110 and the terminal 120, and by displaying the GUI screen only on the terminal 120 carried by the user, it is possible to present the GUI screen only to the user who is viewing it, and it is also possible to eliminate waste such as power consumption associated with displaying the GUI screen on the embedded device 110.

Incidentally, since the terminal 120 is originally equipped with hardware for screen drawing processing and display, the processing load and memory usage for screen drawing processing and display on the terminal 120 do not increase, and costs do not increase. Furthermore, by providing the operation unit on the terminal 120 side rather than the embedded device 110, the cost of the embedded device 110 can be further reduced.

Figure 16 is a diagram showing other functions of the screen display system according to the embodiment. The example shown in Figure 16 shows a configuration example in which one terminal 120 is communicatively connected to multiple embedded devices 110 (110A to 110C) of different systems (types).

In the embodiment, as described above, the screen drawing process and display hardware are provided in the terminal 120, rather than in the embedded device 110. In this system configuration, as shown in FIG. 16, by communicatively connecting one terminal 120 to multiple embedded devices 110 (110A-110C), it becomes possible to display the GUI screens of multiple embedded devices 110 (110A-110C) on the single terminal 120. It also becomes possible to control multiple embedded devices 110 (110A-110C) with a single terminal 120.

For example, by operating the terminal 120 to switch to one embedded device 110A and establish a communication connection, the GUI screen being controlled by the connected embedded device 110A can be displayed on the display unit 121 of the terminal 120, and the connected embedded device 110A can be controlled by operating the operation unit 221 of the terminal 120.

According to the system configuration shown in Figure 16, it is possible to eliminate the screen drawing processing unit 1532, display unit 1541, and operation unit 1542 (see Figure 15) that were previously provided in each of, for example, three embedded devices 110 (110A to 110C), thereby eliminating unnecessary GUI displays at the installation locations of the embedded devices 110 and significantly reducing the costs of the entire system.

Figure 17 is a diagram illustrating the creation state of a conventional GUI application. Next, we will compare the conventional method with the embodiment from the perspective of GUI application creation. The embedded device 110 in Figure 17 shows an example in which there is no GUI screen display function.

In such conventional embedded devices 110, it was necessary to create a GUI application 1721 on the external computer 1710, such as a PC, for each configuration of the embedded device 110. As described above, the GUI application 1721 has a screen construction processing unit 1731 that generates component placement information k, and a screen drawing processing unit 1732 that draws the GUI screen to be displayed on the display unit 1741. It also transmits operation information from the operation unit 1742 to the embedded device 110.

In the above configuration, the number of types of GUI applications 1721 on the external computer 1710 increases with each different system configuration (type) of the embedded device 110, and it was necessary to create a GUI application 1721 for each system configuration. This results in the software structure and functionality of the GUI application 1721 becoming complex, increasing development man-hours and making it difficult to reduce development costs.

In contrast, according to the system configuration of the embodiment, as shown in Figure 2, only the minimum part of the GUI application processing that depends on the configuration of the embedded device 110 is placed within the embedded device 110 in each system configuration. In other words, the GUI application 201 is placed in the embedded device 110, and the screen drawing processing unit 220 and display unit 121 (and operation unit 221) are placed in the terminal 120.

As a result, on the smart device (terminal 120) side, the screen drawing processing unit 220 can process the GUI application as a general-purpose process of "performing screen drawing based on component placement information k." This allows a single type of GUI application for the terminal 120 to be compatible with embedded devices 110 of different configurations, reducing development man-hours. In other words, the screen drawing processing unit 220 of the terminal 120 only needs to have the single function of performing screen drawing based on component placement information k sent from the embedded device 110, allowing for generalization and reducing development costs compared to conventional technology.

Furthermore, in the conventional system configuration shown in Figure 17, a change to the GUI application 1721 (screen construction processing unit 1731, screen drawing processing unit 1732) is required due to an upgrade of the OS of the external computer 1710, etc. In this case, all individual GUI applications 1721 must be modified and retested, which poses the problem of increasing development man-hours.

In contrast, according to the system configuration of the embodiment (see Figure 2), a single type of GUI application (screen drawing processing unit 220, etc.) for the terminal 120 can be used for embedded devices 110 with different configurations. Therefore, in the embodiment, the GUI application 201 only needs to be changed and retested once, reducing development time compared to conventional technology.

In other words, since there is only one type of GUI application (screen drawing processing unit 220, etc.) on the terminal 120 regardless of the type of embedded device 110, changes and retesting of the GUI application 201 only need to be performed once, such as when upgrading the OS version, and changes and retesting can be completed in one go.

Figure 18 is a diagram showing an example of a GUI application update process for a screen display system according to an embodiment. When updating a GUI application, the modified GUI application 201B (screen drawing processing unit 220, etc.) on the terminal 120 side is uploaded from an external computer 1710, etc. to an application download server 1801. The terminal 120 can then download the GUI application 201B (screen drawing processing unit 220, etc.) from the application download server 1801 and install it on the terminal 120.

Next, Figures 19 to 21 are explanatory diagrams comparing the development man-hours for different types of embedded devices. Figure 19 shows the system configuration of different types of embedded devices using conventional technology, while Figure 20 shows the system configuration of different types of embedded devices using an embodiment. Figure 21 shows a chart comparing the development man-hours between conventional technology and an embodiment.

Conventionally, as shown in FIG. 19, three embedded devices 110 (110A to 110C) have, for example, embedded device 110A equipped with one sensor A (111) and one LED A (112), embedded device 110B equipped with two sensors B1 and B2 (111) and one LED B (112), and embedded device 110C equipped with one sensor C (111) and two LEDs C1 and C2 (112), and these different types perform different controls.

These multiple embedded devices 110 (110A-110C) of different types are each individually connected to an external computer 1710 such as a PC for communication. Each external computer 1710 has three GUI applications 1521A-1521C corresponding to each of the embedded devices 110A-110C.

In the system configuration of the embodiment shown in Figure 20, the three embedded devices 110 have three different GUI applications 201A to 201C, as in the conventional technology (Figure 19), but the GUI application on the terminal 120 is a single common GUI application (such as the screen drawing processing unit 220).

Using Figure 21, we compare the development man-hours for each system configuration of the conventional technology (Figure 19) and the embodiment (Figure 20). Comparing the conventional technology with the embodiment, the development man-hours required for the embedded devices 110A to 110C are the same for all three types.

In the conventional technology, on the external computer 1710 side, three different types of GUI applications 1721A to 1721C must be developed for each external computer 1710. Here, in the conventional technology, each of the GUI applications 1721A to 1721C requires the development of component placement and screen drawing processing.

In contrast, on the terminal 120 side of the embodiment, since the arrangement configurations of the sensors 111 and LEDs 112 differ for each of the three embedded devices 110, development man-hours are required for component placement processes k1 to k3 corresponding to each of the three embedded devices 110, but the screen drawing processing of the screen drawing processing unit 220 requires only a common set of development man-hours.

In this way, in the embodiment, the screen rendering process for the GUI application on the terminal 120 side can be made common to different types of embedded devices 110, allowing development to be completed in one go. Furthermore, by separating the component placement process and the screen rendering process, the parts affected by different system configurations can be minimized, reducing development man-hours compared to conventional technology.

As shown in Figure 21, when comparing the conventional technology with the embodiment, the difference (gap) in man-hours increases in proportion to the number of types of embedded devices 110. In other words, according to the embodiment, the more types of embedded devices 110 there are, the greater the effect of reducing development man-hours compared to the conventional technology can be achieved.

According to the embodiment described above, in the initial state, the device shares initial display list information, including text and images related to the GUI screen construction and the placement of each component, with the terminal. After the device is operational, it selects screen information, including text and images corresponding to a specified state, from the initial display list information and transmits it to the terminal. Based on the received screen information, the terminal references the shared initial display list information and processes the screen display of the device's operating status. In this way, by sharing initial display list information between the device and the terminal in advance, the terminal can display the device's status and reduce the amount of data required for the displayed screen information.

In addition, the device does not display a GUI screen, but displays the GUI screen only on the terminal. For example, the device is an embedded device with limited uses and no display unit. Furthermore, the device can perform corresponding control by operating the terminal's GUI screen. This eliminates the need for the device to perform drawing processes to display the GUI screen, reducing the processing load on the device and reducing the cost of the device. Devices that perform predetermined control can easily perform control operations by displaying the control content on a GUI screen, but in this embodiment, the device only generates and processes the GUI screen and does not perform drawing processes. In addition, the device is configured without a display unit. This simplifies the hardware configuration of the device and reduces costs.

In other words, in this embodiment, in a system configuration in which a device is connected to a terminal for communication, the GUI screen generation process is located on the device side, and the GUI screen drawing process is located on the terminal side. This efficiently separates the GUI functions between the device and the terminal, thereby reducing the cost of the entire system. Furthermore, since the terminal can perform the drawing process using a general-purpose smart device, this does not result in increased costs.

In addition, when updating the GUI screen, the device generates the shape, text characters, coordinates, color, etc. of only the updated objects and sends them to the terminal. This further reduces the amount of data required when updating the GUI screen, making it possible to transfer data from the device to the terminal with a smaller amount of data.

Furthermore, the device assigns an identifier to each of the text strings and images included in the initial display list information and transmits the identifiers as screen information to the terminal. The terminal then references the shared initial display list information based on the identifier of the received screen information and performs screen display processing that corresponds to the device's status. This means that when updating screen information due to a change in the device's status, only the identifier needs to be transmitted, further reducing the amount of data transferred.

In addition, a single terminal can connect to devices from multiple different systems (types), and, for example, the terminal can switch communications to display a GUI screen generated by one connected device and control the device. Furthermore, by switching communications, the same terminal can display and control the GUI screens of different devices, reducing the overall cost of a system that includes multiple devices.

Furthermore, according to the embodiment, of the GUI application for GUI screen display, the configuration mainly related to screen construction is stored on the device side, and the configuration related to screen drawing is stored on the terminal side, dividing and arranging the GUI application functions. As a result, even if the number of types of device increases to accommodate various systems, the terminal side can handle and perform drawing processing with a single-function GUI application.

As a result, even if a GUI application needs to be changed due to an OS upgrade or the like, according to the embodiment, since the GUI application on the terminal side is the same type regardless of the type of device, only one change and retest is required. In contrast, with conventional technology, it would be necessary to change and retest all GUI applications for each type of device, and the more types of devices there are, the more development work is required, which increases the development effort.

Furthermore, according to the embodiment, the GUI screen generated by the device can be displayed on the terminal, allowing the user to operate the device while viewing the terminal's display screen. In this case, the GUI screen sent from the device to the terminal consists of the shape of each object, text characters, coordinates, color, etc., and has a smaller data volume than image data. This allows the GUI screen data to be transferred from the device to the terminal with a smaller data volume. Furthermore, because the data transfer volume is small, transfer can be completed in a short time even if the transfer speed between the device and terminal is slow.

When a device and a terminal are connected via communication, the operation of the device can be controlled by operating the terminal. This allows users to monitor the device's operating status by looking at the terminal's display screen, and also control the device's operation, even if they are located far away from the device's installation location.

And because the amount of GUI screen data transferred from the device to the terminal can be reduced, the device can be controlled remotely by operating the terminal, without the user having to go to the device's location. It also makes it possible to perform remote maintenance from a remote location using the terminal in the event of a malfunction. Furthermore, because the device's GUI screen can be displayed on the terminal, costs can be reduced without having to set up a display screen for the device. Furthermore, by connecting the terminal to multiple devices, multiple devices can be monitored from a single terminal.

The screen display program described in this embodiment can be realized by executing a pre-prepared program on a computer such as a personal computer, workstation, or PC (including tablets and smartphones). This screen display method can also be recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM (Compact Disc-Read Only Memory), or DVD (Digital Versatile Disk), and executed by being read from the recording medium by the computer.

As described above, the present invention is useful when applied to technology for transferring display screens generated by devices performing specified controls to terminals.

100 Screen display system 110 Equipment (embedded equipment)
111 Sensor 112 LED
120 Terminal 121 Display unit 201 GUI application 211 Screen construction processing unit 220 Screen drawing processing unit 221 Operation unit 301, 311 CPU
302,312 ROM
303,313 RAM
304, 314 Storage unit 305, 315 Communication interface 308, 318 Bus 310 Network 316 Touch panel 317 Display 500 List of character strings and images (initial display list information K0)
510: Character string list 520: Image list K: GUI screen information k: Part placement information S: Operation information

Claims (8)

In a screen display system that transfers screen information from a device to a terminal,
The device comprises:
The device is composed of one or more devices, and does not have a display unit for the screen information within itself,
a control unit that generates screen information indicating the control status of the own device;
a communication unit that transmits screen information generated by the control unit to the terminal;
The terminal
a processing unit that generates a display screen based on the screen information received from one of the devices;
a display unit that displays the display screen generated by the processing unit,
The control unit of the device
In an initial state, initial display list information including character strings and images relating to screen construction of a GUI (Graphical User Interface) and the arrangement of each component is shared between the terminals;
After one of the devices is activated, screen information including the character string and the image corresponding to a predetermined state is selected from the initial display list information and transmitted to the terminal;
The processing unit of the terminal
A screen display system characterized by referring to the shared initial display list information based on the received screen information, performing screen display processing corresponding to the operating status of each device, and displaying it on the display unit. the device assigns an identifier to each of the character strings and images included in the initial display list information, and transmits the identifier to the terminal as the screen information;
The screen display system according to claim 1, characterized in that the terminal refers to the shared initial display list information based on the identifier of the received screen information and performs screen display processing corresponding to the state of the device. The screen display system described in claim 1, characterized in that the device transmits information regarding the shape, character string, display coordinates, and color of each component part to the terminal as the initial display list information. The screen display system described in claim 1, characterized in that the device updates the screen information in response to the change in state, generates only information regarding the shape, text, display coordinates, and color of the component to be updated as the screen information, and transmits it to the terminal. the terminal has an operation unit, and transmits operation information by the operation unit to the device;
The screen display system according to claim 1, characterized in that the device performs control based on the operation information received from the terminal, updates the screen information based on the control, and transmits the updated screen information to the terminal. The screen display system described in claim 5, characterized in that the terminal is communicatively connected to multiple devices, displays display information of the communicatively connected devices on the display unit, and controls the devices by operating the operation unit. The screen display system described in claim 1, characterized in that the device is an embedded device that performs specified control with limited uses. In a screen display method for transferring screen information from a device to a terminal,
The device comprises:
The display device may be one or more devices, may not have a display unit for the screen information within itself, and may generate screen information indicating the control status of the device itself;
Transmitting the generated screen information to the terminal;
The terminal
displaying a display screen based on the screen information received from one of the devices on a display unit within the terminal;
The device comprises:
In an initial state, initial display list information including character strings and images relating to screen construction of a GUI (Graphical User Interface) and the arrangement of each component is shared between the terminals;
After one of the devices is activated, screen information including the character string and the image corresponding to a predetermined state is selected from the initial display list information and transmitted to the terminal;
The terminal
A screen display method characterized by referring to the shared initial display list information based on the received screen information, performing screen display processing corresponding to the operating status of each device, and displaying it on the display unit.