JP7629477B2 - Information processing system, information processing device, and control method - Google Patents

Description

This application relates to an information processing system, an information processing device, and a control method, for example, for displaying drawn graphics.

Electrophoretic displays are characterized by being made of flexible materials that have plasticity, and are also called electronic paper displays (EPDs) or electronic ink displays (E-Ink displays). EPDs have the advantage of consuming less power than other types of display devices such as liquid crystal displays (LCDs) and organic electroluminescence displays (OLEDs). Patent Document 1 describes how adopting EPDs as display devices in information processing devices such as various portable information processing terminals, wearable information processing terminals, electronic POPs (Point of Purchase advertising), electronic shelf labels, and electronic posters makes it possible to reduce power consumption.

JP 2015-64421 A

However, EPDs are so-called "slow devices." The refresh rate of EPDs is typically about 8 Hz. In contrast, the refresh rate of LCDs and OLEDs is about 60 Hz to 120 Hz. When displaying a figure drawn arbitrarily using an input device, the visible rendering latency from operation to display may be significantly perceived by the user in EPDs. Such a latency may impair the user experience (UX) in handwritten input, particularly the sense of operation. In LCDs or OLEDs, it is conceivable to predict and display the input trajectory to compensate for the latency, and erase parts with significant errors, but this is not realistic for EPDs. As illustrated in FIG. 4, errors in the predicted trajectory tend to be large and are not immediately erased, so the prediction error is significantly visible, which may give the user a sense of discomfort.

The present application has been made to solve the above problems, and an information processing system according to one aspect of the present application includes a host system, a controller, a touch sensor, and a display, the touch sensor overlaps the display and detects a contact position with an input device, the controller samples the contact position, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample.

In the above information processing system, the controller may display the line segment with a width narrower than that of the trajectory.

In the above information processing system, the controller may display the line segment in the same line type as the trajectory.

In the above information processing system, the controller may erase any portion of the line segment that deviates from the trajectory.

In the above information processing system, the display may be an electrophoretic display.

The information processing system may further include the input device, which acquires drawing control information indicating whether drawing is required from the host system and outputs the drawing control information to the controller, and the controller may determine whether or not the line segment is required to be displayed based on the drawing control information.

In the above information processing system, the drawing control information may include drawing mode information indicating a drawing mode for representing the trajectory, and the controller may display the line segment according to the drawing mode.

The information processing device according to the second aspect of the present application includes a host system, a controller, a touch sensor, and a display, the touch sensor overlaps the display and detects a contact position with an input device, the controller samples the contact position from the touch sensor, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample.

A control method according to a third aspect of the present application is a control method for an information processing system including a host system, a controller, a touch sensor, and a display, the touch sensor overlapping the display, the touch sensor detects a contact position with an input device, the controller samples the contact position, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample.

According to the embodiment of the present application, the operational feel during drawing input is improved.

1 is a block diagram illustrating an example of a functional configuration of an information processing system according to a first embodiment. 1 is a block diagram showing a schematic configuration example of an information processing apparatus according to a first embodiment. FIG. 11 is an explanatory diagram showing an example of a display of a path created by handwriting input. FIG. 11 is an explanatory diagram showing an example of a predicted trajectory of handwritten input; 5 is a flowchart illustrating an example of a drawing process according to the first embodiment. FIG. 11 is a schematic block diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment.

First Embodiment
Hereinafter, embodiments of the present application will be described with reference to the drawings. First, a configuration example of an information processing system S1 according to a first embodiment of the present application will be described. Fig. 1 is a schematic block diagram showing an example of the functional configuration of the information processing system S1 according to the present embodiment.
The information processing system S1 includes an information processing device 10 and a display device 40. The information processing device 10 and the display device 40 are connected wirelessly or wired to be able to transmit and receive various data. The information processing device 10 is an example of an electronic device having a computer system. The information processing device 10 may be realized in any form, such as a personal computer, a tablet terminal, or a mobile phone. The information processing device 10 may be configured integrally with the display device 40, or may be configured as hardware separate from the display device 40.

The information processing device 10 acquires pen data from the display device 40. The pen data is data indicating a contact position sampled at a predetermined period. The contact position is a position on the display surface of the display 46 (described later) that is contacted by the input device 50. The information processing device 10 generates a trajectory of the contact position based on a time series consisting of the contact position for each sample. The information processing device 10 outputs display data representing the generated trajectory to the display device 40.

The display device 40 is an example of an electronic device equipped with a touch sensor 44 and a display 46. In this embodiment, an EPD is used as the display 46. The two overlap so that the detection surface of the touch sensor 44 is covered by the display surface of the display 46. The display device 40 can be configured as a so-called flexible display.

The touch sensor 44 detects (samples) the position on the detection surface where the input device 50 touches at a predetermined period, and generates pen data indicating the detected position as the touch position. The display device 40 outputs the generated pen data to the information processing device 10.
The display device 40 displays a line drawing based on the contact position of the previous sample and the contact position of the current sample on the display 46 for each sample (pre-rendering). After that, the display device 40 displays a trajectory indicated in the display data input from the information processing device 10 on the display 46 (actual rendering). Furthermore, the display device 40 erases the line drawing displayed on the display 46 when performing the pre-rendering. In this application, the "current sample" refers to the latest sample at that time, and the "previous sample" refers to the sample immediately before the current sample, that is, the next newest sample.

The input device 50 is held by a user and used to input or edit data such as characters, symbols, and figures. When one end of the input device 50 comes into contact with the detection surface of the touch sensor 44, a figure formed based on the contact position is detected. The input device 50 is not limited to electronic devices and may be any instrument as long as it can detect the contact position with the detection surface. The input device 50 is typically an operating medium having a pen shape, and may be configured as a writing device. The input device 50 may have any form, such as a stylus or a digital pen. The input device 50 may or may not have a haptic feedback function.

Next, an example of the hardware configuration of the information processing device 10 will be described.
The information processing device 10 includes a processor 11 , a main memory 12 , a flash memory 13 , a PCH 21 , an audio system 24 , a microphone 25 , a speaker 26 , a baseband chip 27 , a communication unit 28 , and an input/output unit 29 .

The processor 11 controls the overall functions of the information processing device 10. For example, one or more CPUs (Central Processing Units) are applied as the processor 11. The processor 11 executes a predetermined program, cooperates with the main memory 12 and other hardware, and performs the functions of a host system 100 (described later).
In this application, the processor 11 or other hardware executing processing instructed by instructions written in a program may be referred to as "executing a program" or "running a program."

The main memory 12 is a writable memory used as a working area for the processor 11, i.e., an area for reading programs to be executed and various setting data, and an area for writing processing data acquired by executing programs. The main memory 12 is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. The executable programs include the OS, various device drivers for controlling peripheral devices, various services/utilities, apps, etc.

The processor 11 and main memory 12 function as a system device that constitutes the computer system that is the core of the information processing device 10, that is, the host system. The computer system of the information processing device 10 is composed of a system device as hardware, and software such as an OS and schedule tasks.

The flash memory 13 stores various programs and data. The various programs include, for example, firmware, device drivers, services/utilities, and applications. These programs are executed by the processor 11. The stored data includes data to be processed by the processor 11 and data generated by the processing. The generated data may include not only final data, but also intermediate data used in a step subsequent to a certain step.

The PCH (Platform Controller Hub) 21 has one or more controllers and can be connected to multiple devices to input and output various data. The PCH 21 includes, for example, one or a combination of bus controllers such as USB, Serial ATA (Advanced Technology Attachment), SPI (Serial Peripheral Interface) bus, PCI (Peripheral Component Interconnect) bus, PCI-Express bus, and LPC (Low Pin Count). The multiple devices to which it can be connected include, for example, an audio system 24, a baseband chip 27, and an input/output unit 29, which will be described later.

The audio system 24 performs processes such as processing, input, output, recording, playback, encoding, and decoding of audio signals. The audio system 24 includes, for example, an audio IC (Integrated Circuit). A microphone 25 and a speaker 26 are connected to the audio system 24. Audio signals can be input to the audio system 24 from the processor 11, the microphone 25, or the baseband chip 27. The audio system 24 reads out audio signals recorded therein. The audio system 24 can output the acquired audio signals to the speaker 26 or the processor 11.

The microphone 25 picks up sounds arriving at the microphone 25 and outputs audio signals of the picked up sounds to the audio system 24 .
The speaker 26 presents sound based on the audio signal input from the audio system 24 .

The baseband chip 27 is a dedicated IC for controlling communication using the communication unit 28. The baseband chip 27 realizes communication using, for example, public wireless communication systems such as 4G (fourth generation wireless communication system) and 5G (fifth generation wireless communication system), and an in-house wireless communication network defined in IEEE802.11. The baseband chip 27, under the control of the processor 11, connects to other devices via a communication network using the communication unit 28 so that various data can be transmitted and received, and transmits and receives various data.

The communication unit 28 is a wireless communication module for connecting to a communication network wirelessly or via a wired connection. The communication unit 28 may include an antenna for transmitting and receiving radio waves used for wireless communication. The communication unit 28 can use, for example, a wireless communication method defined in IEEE802.11.

The input/output unit 29 has an interface that connects wirelessly or wired to other devices (mainly the display device 40) so that various data can be input and output using a predetermined input/output method. The input/output unit 29 can use input/output methods defined in, for example, USB (Universal Serial Bus) 3.0, 3.1, etc.

Next, an example of the hardware configuration of the display device 40 according to the present embodiment will be described.
The display device 40 includes a controller 42 , a touch sensor 44 , a display 46 and an input/output unit 48 .

The controller 42 executes a predetermined firmware to perform a process for implementing the functions of the display device 40 and a process for controlling the functions. The controller 42 includes, for example, a timing controller. The controller 42 displays on the display 46 a display screen indicated by display data input from the information processing device 10 via the input/output unit 48 (drawing process). The controller 42 specifies a signal value for each pixel specified by the display data, and causes the pixel to emit light at a luminance corresponding to the specified signal value.
The controller 42 samples the contact position indicated by the detection signal input from the touch sensor 44 at a predetermined sampling period, and acquires pen data indicating the contact position for each sample. The controller 42 outputs the acquired pen data to the information processing device 10 using the input/output unit 48.

The touch sensor 44 detects the position where another object has come into contact. The other object that comes into contact with the touch sensor is mainly the input device 50. The touch sensor 44 has a sheet-like shape that is much smaller than its width or length. A detection surface is set on most of the surface of the touch sensor 44. The touch sensor 44 has multiple detection elements, and each detection element is arranged at a different position within the detection surface. The touch sensor 44 detects an area of the detection surface that has come into contact with another object at each predetermined sampling period, and determines a representative point (e.g., reception) of the detected area as the contact position.

The touch sensor 44 generates pen data indicating the determined contact position, and outputs the generated pen data to the controller 42 using a predetermined input/output method. As the input/output method, for example, a serial communication method such as the I2C (Inter-Integrated Circuit) method can be used. In contrast, the controller 42 has a serial communication controller, obtains a detection signal indicating the presence or absence of contact for each pixel, determines the area in which the pixels determined to be in contact are distributed as the contact area, and determines the center of gravity of the contact area as the contact position. The touch sensor 44 may employ any operating principle, such as a resistive film method or a capacitive method.

The display 46 displays a pattern specified by the display data under the control of the controller 42. The display 46 has a sheet-like shape. The display 46 has a plurality of pixels, each of which is arranged at a different position on the display surface. Each of the pixels emits light at a brightness specified by the controller 42. The display 46 has, for example, a TTL (Transistor Transistor Logic) circuit. The TTL is used to supply drive power corresponding to a signal value to each of the pixels. Each of the pixels emits light at a brightness corresponding to the supplied drive power. In response to this, the controller 42 has a TTL controller, and applies an electrical signal having a potential corresponding to the signal value for each pixel to the TTL.

The input/output unit 48 has an interface that connects wirelessly or wired to other devices so that various data can be input and output using a specified input/output method. The other devices that are the input source and output destination are mainly the information processing device 10. The input/output unit 48 uses the same input/output method as the other devices that are the input source and output destination. In contrast, the controller 42 has a USB controller, and uses the input/output unit 48 to obtain input data from the input source in accordance with the USB regulations, generates output data in accordance with the regulations, and outputs the generated output data to the output destination.

Next, an example of the functional configuration of the information processing system S1 according to the present embodiment will be described below. Fig. 2 is a block diagram showing an example of the schematic configuration of the information processing system S1 according to the present embodiment.
The information processing apparatus 10 includes a host system 100 and an input/output unit 29. The host system 100 includes a pen data acquisition unit 102, a drawing processing unit 104, a post-processing unit 106, and a drawing setting unit .

The pen data acquisition unit 102 waits for pen data to be input from the display device 40 via the input/output unit 29. The pen data acquisition unit 102 outputs the acquired pen data to the drawing processing unit 104.
The drawing processing unit 104 receives pen data from the pen data acquisition unit 102. The drawing processing unit 104 identifies the contact position for each sample indicated by the pen data. The drawing processing unit 104 uses a predetermined function to generate a curve based on the contact positions for each sample up to the current sample as a trajectory. As the predetermined function, a function that passes through a plurality of contact positions and gives a curve that sequentially connects these contact positions can be used. For example, the pen data acquisition unit 102 generates a Bezier curve by interpolating the coordinates between the contact positions using a linear combination of Bernstein bases. The pen data acquisition unit 102 may generate a spline curve by interpolating the coordinates between the contact positions for each section between two contact positions that are in sequential contact using a polynomial.

The drawing processing unit 104 generates display data that expresses the generated trajectory using a predetermined drawing mode, and outputs the generated display data to the display device 40 via the input/output unit 29. This allows the display device 40 to display the generated trajectory (standard drawing). As the drawing mode, any one of the distinction between drawing and erasing, the line type, color, line width, etc., or any combination thereof, can be specified. The drawing mode may be set in advance in the drawing processing unit 104, or may be specified by the drawing setting unit 108. The display of the trajectory by standard drawing is delayed compared to the display of the line drawing by temporary drawing (described later). The drawing processing unit 104 outputs the generated display data to the post-processing unit 106.

Since the generated trajectory is a curved line, it does not necessarily overlap completely with the line segment connecting adjacent samples by the provisional drawing (see FIG. 3). Therefore, the post-processing unit 106 identifies the part of the line drawing up to the current sample that is not covered by the generated trajectory and is separated, that is, the part where the line segment does not overlap with the trajectory spatially and protrudes, as an unnecessary figure (post-processing). The post-processing unit 106 can identify the area occupied by the trajectory using the display data input from the drawing processing unit 104. The post-processing unit 106 can identify the line drawing displayed by the provisional drawing based on the pen data input from the pen data acquisition unit 102. From the contact position of each sample indicated in the pen data, a line segment with the two contact positions at both ends is identified for each pair of two contact positions that are adjacent in sequence, and the figure formed by connecting the individual line segments corresponds to the line drawing by the provisional drawing. The post-processing unit 106 may not generate a line drawing based on the contact position, but may acquire an existing line drawing. The post-processing unit 106 may, for example, obtain line drawing data indicating the line drawing displayed on the display 46 during temporary drawing from the display control unit 422 of the display device 40.

The post-processing unit 106 outputs unnecessary figure data indicating the erasure of the identified unnecessary figure to the display device 40 via the input/output unit 29. This makes it possible to erase unnecessary figures that deviate from the trajectory from among the provisionally drawn figures displayed on the display device 40 (unnecessary figure erasure). The period of the post-processing and unnecessary figure erasure may be longer than the sampling period of the contact position. The period of the post-processing is, for example, equal to the refresh rate of the display 46 (for example, about 60 Hz). However, it is preferable that the period of the post-processing is shorter than the presentation time required for visual recognition of the visual information.

The drawing setting unit 108 specifies a drawing mode based on operation data input from various input devices, and sets the specified drawing mode to the drawing processing unit 104. The drawing mode includes one or more element items, and is specified by one of a plurality of candidate items for each element item. The drawing setting unit 108, for example, displays a predetermined drawing setting screen on the display 46 in which screen parts representing each candidate item are arranged. The drawing setting unit 108 can select a drawing mode by specifying a candidate item corresponding to a screen part that includes a position designated by the operation data within the display area while the drawing setting screen is being displayed. The drawing setting unit 108 may apply pen data acquired from the display device 40 as the operation data. The element items include, for example, a distinction between drawing and erasing, a line type, a color, a line width, and the like. The drawing setting unit 108 sets drawing mode information indicating the specified drawing mode to the display device 40. The drawing mode set to the display device 40 can be applied to the display of a line segment in a temporary drawing. The drawing mode information is also called brush information. A drawing mode may be a characteristic of the writing displayed by a physical writing implement, and the characteristic or drawing mode may be distinguished by the type of writing implement (e.g., ballpoint pen, highlighter, brush, marker, etc.). Erasing may also be specified as an "eraser."

The controller 42 of the display device 40 , the touch sensor 44 , the display 46 , and the input/output unit 48 execute predetermined firmware and cooperate with memory and other hardware to perform the functions of a display control unit 422 .
The display control unit 422 executes processing related to drawing based on the pen data. The display control unit 422 samples the detection position where the input device 50 touches the detection surface of the touch sensor 44 at a constant sampling period, and acquires pen data indicating a time series of the contact position. The display control unit 422 generates a line drawing including a line segment from the contact position of the previous sample to the contact position of the current sample for each sample. The display control unit 422 displays the generated line drawing on the display surface of the display 46 (provisional drawing). In FIG. 3, x marks exemplify each sample of the contact position. The x mark closest to the tip T of the input device 50 indicates the current sample. A line drawing formed by sequentially connecting the line segments between adjacent samples is displayed by the provisional drawing.

2, the display control unit 422 transmits the acquired pen data to the information processing device 10 using the input/output unit 48 (pen data transfer). The transmitted pen data is used in the information processing device 10 mainly to generate a trajectory of the contact position.
Display data representing the generated trajectory is input to the display control unit 422 from the information processing device 10 using the input/output unit 48. The display control unit 422 displays the trajectory shown in the display data on the display surface of the display 46 (standard drawing).

The display control unit 422 receives input of unnecessary figure data indicating unnecessary figures that are not covered by the trajectory from the information processing device 10 using the input/output unit 48 from the information processing device 10. The display control unit 422 identifies the unnecessary figures indicated in the unnecessary figure data. The post-processing unit 106 erases the identified unnecessary figures from the line drawing displayed on the display 46 (unnecessary figure erasure). By erasing the unnecessary figures, it is possible to avoid causing discomfort to the user due to the temporary drawing. When erasing the unnecessary figures, the post-processing unit 106 may restore the display of a partial background, which is a part of the background image that was displayed before the line drawing was displayed, in the area occupied by the unnecessary figures.

The display control unit 422 may receive drawing mode information from the information processing device 10 using the input/output unit 48. The display control unit 422 sets the drawing mode indicated by the drawing mode information and applies it as the drawing mode for the line drawing in the temporary drawing. Since the line drawing in the temporary drawing is displayed in the same manner as the trajectory in the standard drawing, the two are no longer clearly distinguishable, and therefore the awkwardness caused by the juxtaposition of the differences between the two can be eliminated or reduced.

Next, an example of the display of a trajectory by handwriting input will be described. FIG. 3 is an explanatory diagram showing an example of the display of a trajectory by handwriting input. FIG. 3 illustrates a trajectory and a line segment displayed on the display surface of the display 46 while moving from left to right while the tip of the input device 50 is in contact with the detection surface of the touch sensor 44. In the example of FIG. 3, in order to distinguish between the trajectory by standard drawing and the line segment by provisional drawing, both are displayed with different line widths. The distance from the tip T of the input device 50 to the right end of the displayed trajectory is caused by a processing delay (standard drawing delay) from the detection of the contact position through the generation of the trajectory to the display of the generated trajectory. In this embodiment, the line segment Seg is displayed by provisional drawing before the trajectory by standard drawing is displayed. The processing delay (provisional drawing delay) from the detection of the contact position to the display of the line segment Seg by provisional drawing is shorter than the processing delay (standard drawing delay) from the detection of the contact position to the display of the trajectory by standard drawing, thereby shortening the distance between the tip T of the input device 50 and the end of the displayed line segment Seg, thereby improving the operability. Since both the trajectory displayed by the standard drawing and the line segment Seg displayed by the provisional drawing are based on a common contact position, most of the line segment Seg is overwritten by the trajectory. In the example of FIG. 3, the width of the line segment Seg by the provisional drawing is narrower than the width of the trajectory by the standard drawing. Since the trajectory is a curve, it is possible to avoid or suppress the generation of unnecessary figures that constitute the portion of the line segment Seg that protrudes from the trajectory, compared to when the width of the line segment Seg is equal to the trajectory. In addition, by making the line type of the line segment Seg by the provisional drawing and the trajectory by the standard drawing the same, it is possible to eliminate or reduce the sense of incongruity caused by the difference between the two.

Next, an example of the drawing process according to the present embodiment will be described with reference to a flowchart shown in FIG.
(Step S102) The display control unit 422 of the display device 40 detects the contact position of the touch sensor 44 of the input device 50 at every predetermined sampling period, and obtains pen data indicating the detected contact position.
(Step S104) The display control unit 422 causes the display 46 to display a line drawing (line segment Seg) including a line segment connecting the contact position of the previous sample to the contact position of the current sample (provisional drawing).

(Step S106) The display control unit 422 outputs the acquired pen data to the host system 100 of the information processing device 10.
(Step S108) The pen data acquisition unit 102 of the host system 100 acquires the pen data from the display device 40.
(Step S110) The drawing processing unit 104 generates a curve based on the contact position of each sample indicated in the acquired pen data as a trajectory, and generates display data that represents the generated trajectory in the set drawing mode (drawing process). The drawing processing unit 104 outputs the generated display data to the display device 40.
(Step S112) The display control unit 422 of the display device 40 receives display data from the information processing device 10 and causes the display control unit 422 to display the trajectory indicated in the received display data on the display 46 (standard drawing). That is, the display control unit 422 performs the provisional drawing in step S104 and also the standard drawing in step S112, based on the pen data indicating the contact position of the pen by the same sampling.

(Step S114) The post-processing unit 106 identifies, among the line segment Seg displayed in the provisional drawing, a portion that does not overlap with the locus displayed in the standard drawing as an unnecessary figure. The post-processing unit 106 may output unnecessary figure data indicating the erasure of the identified unnecessary figure to the display device 40 (post-processing). The processing in this step and the next step may be executed only when the line segment Seg in the provisional drawing is not completely covered by the locus in the standard drawing.
(Step S116) The display control unit 422 receives the unnecessary figure data from the information processing device 10 and erases the unnecessary figures from among the figures displayed on the display 46 (unnecessary figure erasure). After that, the process returns to step S102.

Second Embodiment
Next, a configuration example of an information processing system S1 according to a second embodiment of the present application will be described. In the following description, differences from the above embodiment will be mainly described. Configurations and processes common to the above embodiment will be denoted by the same reference numerals, and the description thereof will be used unless otherwise specified.

The display control unit 422 of the controller 42 causes the display 46 to provisionally draw a line drawing based on the contact position indicated by the pen data acquired from the touch sensor 44. If the contact position is included in a predetermined drawing area to be drawn, the display control unit 422 executes the provisional drawing, and if the contact position is not included in the drawing area, the display control unit 422 does not execute the provisional drawing. The drawing area may differ depending on the individual display screen. The drawing process may be instructed by an application executed by the host system 100 of the information processing device 10, but access to the controller 42 is usually not taken into consideration. If an appropriate drawing area is not set in the display control unit 42, it may not be possible to accurately determine whether or not provisional drawing is required.

In the information processing system S1 according to this embodiment, the controller 42 of the display device 40 acquires drawing control information indicating whether or not temporary drawing is required from the information processing device 10 via the input device 50. An input/output function (Haptic API) related to a haptic presentation function that vibrates the input device 50 during drawing may be used to transmit the drawing control information. The haptic presentation function is a function that vibrates in response to contact with the touch sensor 44. The input device 50 vibrates based on vibration control information input from the information processing device 10. The vibration control information includes at least information indicating whether or not vibration is required. Whether or not vibration is required corresponds to whether or not the contact position is included in the drawing area. In addition, the drawing mode may be selected based on a position (e.g., a screen part such as a button) indicated on a setting screen. The position setting may also differ depending on each application.

FIG. 6 is a schematic block diagram showing an example of the functional configuration of an information processing system S1 according to this embodiment. The information processing system S1 includes an information processing device 10, a display device 40, and an input device 50. In the description of this embodiment, the input device 50 is mainly a device pen having a tactile presentation function. The vibration control information may include information indicating the vibration characteristics of the input device 50. Each vibration characteristic may be associated with a drawing mode of a trajectory to be displayed on the display 46. The vibration characteristics may be specified by the type of physical writing implement to be simulated in that drawing mode.

The host system 100 of the information processing device 10 executes an application and determines whether the contact position indicated in the pen data acquired from the display device 40 is included in the drawing area set by the application. The application to be executed and the execution of the application are instructed according to the user's operation. The host system 100 performs drawing processing based on the drawing area contact position and does not perform drawing processing based on the contact position outside the drawing area. The host system 100 outputs drawing control information indicating whether drawing processing is required to the input device 50. The host system 100 may select one of a plurality of predetermined drawing modes according to the user's operation, and output drawing mode information indicating the selected drawing mode to the input device 50 by including it in the drawing control information. When performing drawing processing according to the application or selecting a drawing mode, the host system 100 may call the pen data acquisition unit 102, the drawing processing unit 104, the post-processing unit 106, and the drawing setting unit 108 and execute those processes.

The input device 50 includes a device control unit 52 , a vibration unit 55 , and an input/output unit 58 .
The device control unit 52 controls the functions of the input device 50. The device control unit 52 includes, for example, an MCU (Micro Controller Unit) and executes a predetermined program to realize its functions. The MCU includes a processor, a memory, and an input/output interface.

Drawing control information is input to the device control unit 52 from the information processing device 10 via the input/output unit 58 , and the device control unit 52 relays the input drawing control information and outputs it to the display device 40 .
The device control unit 52 acquires a drive signal for vibrating the vibration unit 55 based on the input drawing control information. The device control unit 52 outputs the acquired drive signal to the vibration unit 55.

For example, reference vibration waveform information is set in the device control unit 52. The reference vibration waveform information is information indicating a reference vibration waveform having vibration characteristics that are a predetermined standard. The device control unit 52 refers to the reference vibration waveform information, and when drawing control information (e.g., transmission API) indicating a drawing required (a vibration required) is input, the device control unit 52 outputs a drive signal indicating the reference vibration waveform to the vibration unit 55. When drawing control information (e.g., stop API) indicating a drawing not required (a vibration not required) is input, the device control unit 52 does not output a drive signal indicating the reference vibration waveform to the vibration unit 55.

The device control unit 52 may have pre-set vibration waveform information for each drawing mode. The vibration waveform information for each drawing mode is information indicating a vibration waveform having vibration characteristics corresponding to each drawing mode. The device control unit 52 refers to the vibration waveform information for each drawing mode, identifies the drawing mode specified by the drawing control information, and outputs a drive signal indicating the vibration waveform for each drawing mode related to the specified drawing mode to the vibration unit 55. The drawing mode may be specified using, for example, a waveform ID (wave ID). The waveform ID can also be considered as identification information indicating the vibration waveform for each drawing mode.

The vibration unit 55 vibrates according to the drive signal input from the device control unit 52. The vibration unit 55 includes, for example, a digital-to-analog converter (DAC), an amplifier, and an actuator. The DAC converts the digital drive signal input from the device control unit 52 into an analog drive signal. The DAC outputs the converted drive signal to the amplifier. The amplifier adjusts the amplitude of the drive signal input from the DAC, and outputs the amplitude-adjusted drive signal to the actuator. The actuator generates vibrations according to the drive signal input from the amplifier. The actuator includes, for example, a piezoelectric vibrator such as a piezo element.

The input/output unit 58 has an input/output interface for wirelessly or wiredly inputting and outputting various data to and from other devices. The other devices are mainly the information processing device 10 and the display device 40. The input/output unit 58 enables communication with the information processing device 10 and the display device 40 using, for example, a wireless communication method defined in IEEE802.15.1.

The display control unit 422 of the display device 40 determines whether or not temporary drawing is required according to the drawing control information input via the input device 50. When the display control unit 422 determines that temporary drawing is required, it performs temporary drawing based on the contact position indicated in the pen data, and when it determines that temporary drawing is not required, it does not perform temporary drawing. When the drawing control information includes drawing mode information, the display control unit 422 performs temporary drawing according to the drawing mode indicated by the drawing mode information.

Therefore, even if a drawing area is not set on the display device 40 by an application, it is possible to accurately determine whether or not temporary drawing is required based on the drawing control information notified via the input device 50. In addition, temporary drawing can be performed according to the drawing mode included in the drawing control information. Note that the input device 50 may omit the vibration unit 55 and may not have a haptic feedback function. Even in this case, the device control unit 52 relays the drawing control information output from the information processing device 10 to the display device 40.

As described above, the information processing system S1 according to this embodiment includes the host system 100, the controller 42, the touch sensor 44, and the display 46. The touch sensor 44 overlaps with the display 46 and detects a contact position with the input device 50. The controller 42 samples the contact position, and the host system 100 generates a trajectory of the contact position. Before acquiring a portion of the generated trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample, the controller 42 causes the display 46 to display a line segment from the contact position of the previous sample to the contact position of the current sample.
The display 46 may be an electrophoretic display. The information processing system S1 may be configured as an information processing device 10 including the host system 100 and the display device 40.
According to this configuration, before the portion of the trajectory corresponding to the contact position of the current sample from the contact position of the previous sample is acquired, a line segment from the contact position of the previous sample to the contact position of the current sample is provisionally displayed on the display 46 (provisional drawing). The delay time required for displaying the line segment from the contact position of the previous sample to the contact position of the current sample (standard drawing) can be reduced compared to the delay time required for displaying the portion of the trajectory corresponding to the contact position of the current sample from the contact position of the previous sample. Therefore, the operability of drawing input is improved. In addition, since the line drawing related to the provisional drawing and the trajectory related to the standard drawing are based on a common contact position, the difference between them and the trajectory generated based on the existing contact position can be reduced more than the trajectory predicted based on the past contact position.

The controller 42 may display the line segments in the provisional drawing with a narrower width than the trajectory in the standard drawing.
Since the trajectory is generally a curved line, if the line segment of the temporary drawing has the same width, there will be areas where the two do not overlap. By making the line segment of the temporary drawing narrower in width than the trajectory, the area not covered by the trajectory can be eliminated or reduced. The discomfort felt by the user by the area not covered by the trajectory can be eliminated or reduced.

The controller 42 may display the line segments related to the provisional drawing in the same line type as the trajectory related to the standard drawing.
By making the display mode common to the trajectory, the line segments related to the temporary drawing become less noticeable, so that it is possible to reduce the sense of discomfort felt by the user by the line segments related to the temporary drawing.

The controller 42 may erase any portion of the line segment displayed on the display 46 that deviates from the displayed trajectory.
According to this configuration, by erasing the portion of the line segment related to the provisional drawing that is not covered by the trajectory related to the standard drawing and extends beyond the trajectory, it is possible to eliminate any sense of discomfort felt by the user by the display of that portion.

The input device 50 may obtain drawing control information indicating whether drawing is required from the host system 100, and output the drawing control information to the controller 42, and the controller 42 may determine whether or not it is required to display the line segment based on the drawing control information.
According to this configuration, the controller 42 can determine whether or not temporary drawing based on the pen data is necessary, based on the drawing control information notified from the host system 100 via the input device 50. Therefore, even if the program executed by the host system 100 cannot know whether or not the pen data is related to temporary drawing, the controller 42 can determine whether or not temporary drawing is necessary. In addition, in transmitting the drawing control information, a procedure similar to that for the control information related to the tactile presentation function can be used, thereby making it possible to realize the information economically.

The drawing control information may include drawing mode information indicating a drawing mode for expressing a trajectory, and the controller 42 may display the line segment according to that drawing mode.
According to this configuration, the controller 42 receives the drawing mode from the drawing control information notified from the host system 100 via the input device 50. This makes it possible to more reliably realize temporary drawing using the drawing mode instructed by the host system 100. In transmitting the drawing mode, a similar procedure to that for the control information related to the tactile presentation function can be used, thereby making it possible to realize the drawing mode economically.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configurations are not limited to the above-mentioned embodiments, and include designs within the scope of the gist of the present invention. The configurations described in the above-mentioned embodiments can be combined in any manner.

S1...information processing system, 10...information processing device, 29...input/output unit, 40...display device, 42...controller, 44...touch sensor, 46...display, 48...input/output unit, 52...device control unit, 55...vibration unit, 58...input/output unit, 100...host system, 102...pen data acquisition unit, 104...drawing processing unit, 106...post-processing unit, 108...drawing setting unit, 422...display control unit,

Claims (7)

A host system, a controller, a touch sensor, an electrophoretic display and an input device,
the touch sensor overlies the electrophoretic display and detects a position of contact with the input device;
The controller samples the touch position;
The host system generates a trajectory of the touch location;
determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device ;
The controller:
determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information;
when it is determined that display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display;
The input device vibrates when a drawing control signal indicating a drawing request is input.
Information processing system. The information processing system according to claim 1 , wherein the controller displays the line segment with a width narrower than that of the locus. The information processing system according to claim 1 , wherein the controller displays the line segment in the same line type as the trajectory. The information processing system according to claim 2 , wherein the controller erases any portion of the line segment displayed on the electrophoretic display that deviates from the locus. the drawing control information includes drawing mode information indicating a drawing mode for expressing the trajectory,
The information processing system according to claim 1 , wherein the controller causes the line segment to be displayed in accordance with the drawing mode. A host system, a controller, a touch sensor and an electrophoretic display,
the touch sensor overlies the electrophoretic display and detects a position of contact with an input device;
The controller samples the touch location from the touch sensor;
The host system generates a trajectory of the touch location;
determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device ;
The controller:
determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information;
when it is determined that display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display ;
The input device vibrates when a drawing control signal indicating a drawing request is input.
Information processing device. A host system, a controller, a touch sensor, an electrophoretic display and an input device,
1. A control method for an information processing system in which the touch sensor overlaps the electrophoretic display, comprising:
The touch sensor detects a contact position with the input device;
The controller samples the touch position;
The host system generates a trajectory of the touch location;
determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device ;
The controller:
determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information;
when it is determined that the display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display;
The input device vibrates when a drawing control signal indicating a drawing request is input.
Control methods.

Images