JP7564699B2 - Handwritten data generating device, handwritten data reproducing device, and digital ink data structure - Google Patents

Description

The present invention relates to a handwritten data generation device, a handwritten data reproduction device, and a digital ink data structure, and in particular to a handwritten data generation device, a handwritten data reproduction device, and a digital ink data structure that use tactile feedback.

There are known tablet terminals that are configured to provide tactile feedback when operations such as selecting text displayed on a touch screen, or inserting or positioning a cursor within the text are performed. Patent Document 1 discloses an example of such a tablet terminal.

JP 2012-118993 A

One of the objectives of the present invention is to provide a handwritten data generation device, a handwritten data reproduction device, and a digital ink data structure that generate tactile feedback according to stroke data.

The handwriting data generating device according to the present invention is a handwriting data generating device including a processor, the processor associating haptic feedback with at least a portion of stroke data generated in response to handwriting input, and generating digital ink including the stroke data and haptic data indicative of the haptic feedback.

The handwriting data reproducing device according to the present invention is a handwriting data reproducing device that includes a display and a processor, the processor acquires digital ink including stroke data and haptic data indicating haptic feedback, reproduces the stroke data to display the stroke data on the display, and controls a force-sense generating device to reproduce the haptic feedback indicated by the haptic data when the stroke data is reproduced.

The digital ink data structure according to the present invention is used in a computer including a display, a storage unit, and a processor, and is a digital ink data structure stored in the storage unit, and includes stroke data and haptic data that associates haptic feedback with target stroke data among the stroke data, and is used in a process in which the processor acquires the contents of the haptic feedback from the storage unit when the stroke data is reproduced to display the stroke data on the display and when the force-sense generating device is controlled to reproduce the haptic feedback indicated by the haptic data during the reproduction of the stroke data.

According to the present invention, it is possible to generate haptic feedback according to stroke data.

1 is a diagram showing the appearance of tablet terminals 1a and 1b according to a first embodiment of the present invention; 2 is a diagram showing the internal configuration of tablet terminals 1a and 1b shown in FIG. 1. 2 is a diagram showing the data structure of digital ink DINK generated by the tablet terminal 1a shown in FIG. 1. FIG. 5 is a diagram showing an example of the stroke data group 102 shown in FIG. 4 . 5 is a diagram showing an example of haptic effect definition data 121 shown in FIG. 4. FIG. 5 is a diagram showing an example of haptic data 105 shown in FIG. 4 . 1. FIG. 4 is a flowchart showing a haptic feedback providing process performed by the tablet terminal 1a shown in FIG. 1. FIG. 4 is a flowchart showing a haptic feedback reproduction process performed by the tablet terminal 1b shown in FIG. 9 is a flowchart showing details of the stroke data reproducing process shown in FIG. 8 . 13 is a diagram showing a structure of partial haptics data PHD according to a second embodiment of the present invention. FIG. 13A to 13C are diagrams illustrating details of a stroke data reproduction process according to the second embodiment of the present invention. FIG. 13 is a diagram showing an example of a stroke data group 102 according to a third embodiment of the present invention. FIG. 13 is a flowchart showing stroke data generation processing performed by a tablet terminal 1a according to a third embodiment of the present invention. FIG. 13 is a diagram showing a correspondence table that associates writing pressure values with haptic data. FIG. 13 is a flowchart showing a haptic feedback reproduction process performed by a tablet terminal 1b according to a third embodiment of the present invention. 16 is a diagram illustrating a haptic feedback reproduction area set in step S53 of FIG. 15.

The following describes in detail an embodiment of the present invention with reference to the attached drawings.

Figure 1 is a diagram showing the external appearance of tablet terminals 1a and 1b according to a first embodiment of the present invention. Also, Figure 2 is a diagram showing the internal configuration of tablet terminals 1a and 1b.

As can be seen from FIG. 2, tablet terminals 1a and 1b, which are computers, are devices having the same configuration and are connected via a communication network 2. In this embodiment, tablet terminal 1a is a "handwritten data generation device" for generating digital ink DINK, and is configured to supply the generated digital ink DINK to tablet terminal 1b via communication network 2. Furthermore, tablet terminal 1b is a "handwritten data reproduction device" for reproducing digital ink DINK, and is configured to reproduce the digital ink DINK supplied from tablet terminal 1a. Of course, each of tablet terminals 1a and 1b may have the functions of both a "handwritten data generation device" and a "handwritten data reproduction device." Furthermore, it is also possible to implement the functions of a "handwritten data generation device" and a "handwritten data reproduction device" in a computer other than a tablet terminal, such as a personal computer or a smartphone.

As shown in FIG. 2, each of the tablet terminals 1a and 1b is configured to have a host processor 10, a memory unit 11, a display 12, a sensor 13, a sensor controller 14, a force-sense generating device 15, and a communication unit 16. Below, a detailed explanation will be given focusing on the tablet terminal 1a, but the same is true for the tablet terminal 1b. Note that the tablet terminal 1a may be configured to have a touch surface that receives touch by a pointing object such as an electronic pen P or a finger, without having a display 12.

The host processor 10 is a central processing unit that controls the entire tablet terminal 1a, including the display 12 and the force-sense generating device 15, and serves to execute the operating system of the tablet terminal 1a and various applications such as drawing software by executing programs stored in the memory unit 11. The memory unit 11 is a storage device configured to be able to store any data, and includes a main storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage device such as a hard disk.

The display 12 is a display device that displays arbitrary data under the control of the host processor 10, and is, for example, a liquid crystal display or an organic EL display. The sensor 13 is a sensor pattern composed of multiple sensor electrodes each connected to a sensor controller 14, and is arranged inside the display surface of the display 12. As a result, in the tablet terminal 1a, the display surface of the display 12 becomes the touch surface. However, the touch surface may be provided at a position separate from the display surface of the display 12, in which case the sensor 13 is arranged inside the touch surface.

The sensor controller 14 is an integrated circuit that detects the position of the electronic pen P or a pointing object such as a finger shown in FIG. 1 on the touch surface via the sensor 13, and has the function of communicating with the electronic pen P. The sensor controller 14 detects the position of the finger, for example, by a capacitance method, and detects the position of the electronic pen P and communicates with the electronic pen P, for example, by an active electrostatic method. Note that the sensor controller 14 may also be configured to detect the position of the electronic pen P and communicate with the electronic pen P by an electromagnetic induction method. When the sensor controller 14 detects the position of the finger or electronic pen P, and when it receives data from the electronic pen P by communicating with the electronic pen P, it is configured to sequentially supply these positions or data to the host processor 10. Based on the positions and data thus supplied, the host processor 10 executes processes such as generating and drawing digital ink DINK and moving the cursor.

The active electrostatic method will be briefly explained. In the following, the signal transmitted from the sensor controller 14 to the electronic pen P will be referred to as an "uplink signal," and the signal transmitted from the electronic pen P to the sensor controller 14 will be referred to as a "downlink signal."

The sensor controller 14 uses the sensor 13 to transmit an uplink signal periodically, in principle at regular time intervals. The uplink signal thus transmitted contains a command that is an instruction to the electronic pen P.

The electronic pen P is composed of a pen tip electrode provided at the pen tip, a control unit that receives uplink signals and transmits downlink signals via this pen tip electrode, a pen pressure detection unit that detects the value of pressure applied to the pen tip (pen pressure value), a grip force detection unit that detects the value of pressure applied to the side of the housing of the electronic pen P (grip force value), a power source that supplies the power required for the operation of the electronic pen P, and a memory unit that stores various data such as a pen ID for uniquely identifying the electronic pen P.

The control unit of the electronic pen P is configured to transmit a downlink signal in response to receiving an uplink signal. If the sensor controller 14 and the electronic pen P have not yet detected each other, the downlink signal transmitted in this manner will be a signal that includes only a burst signal, which is an unmodulated carrier signal. On the other hand, if the sensor controller 14 and the electronic pen P have already detected each other, the downlink signal will be a signal that includes a shorter burst signal and a data signal modulated by data corresponding to the command included in the uplink signal. The data transmitted by the data signal may include the above-mentioned writing pressure value, grip force value, pen ID, etc.

When the sensor controller 14 has not yet detected the electronic pen P, it attempts to detect the position of the electronic pen P across the entire touch surface by receiving burst signals using all of the sensor electrodes included in the sensor 13 (global scan). After detecting the position of the electronic pen P as a result, it updates the position of the electronic pen P by receiving burst signals using only one or more sensor electrodes in the vicinity of the detected position (local scan), and receives data transmitted by the electronic pen P by receiving and demodulating the data signal.

Here, the electronic pen P may have a plurality of pen tip electrodes arranged in the pen axis direction, and may be configured to transmit a burst signal from each pen tip electrode. In this case, the sensor controller 14 is configured to derive the respective positions by distinguishing and receiving the burst signals transmitted from each pen tip electrode, and to obtain a tilt angle indicating the inclination of the electronic pen P based on the result. The tilt angle is also supplied to the host processor 10 like other data, and is used for processing such as generating and drawing digital ink DINK. Note that a gyro may be provided in the electronic pen P, and the tilt angle of the electronic pen P may be calculated from its output value. In this case, the output value of the gyro may be supplied from the electronic pen P to the sensor controller 14, and the tilt angle may be calculated in the sensor controller 14, or the tilt angle may be calculated in the electronic pen P, and the tilt angle calculation result may be supplied from the electronic pen P to the sensor controller 14.

Returning to the explanation of each component shown in FIG. 2, the force-sense generating device 15 is a device for providing tactile feedback to the user, and may be configured, for example, from a vibrator that vibrates in response to the waveform of the input current, or a magnetic fluid whose hardness can be controlled by the waveform of the input current. In a typical example, the force-sense generating device 15 is disposed inside the housing of the tablet terminal 1a, and is configured to apply vibrations to the surface of the housing, including the touch surface.

The communication unit 16 is a communication device for connecting the host processor 10 to the communication network 2. In this embodiment, the communication unit 16 is used to transmit and receive digital ink DINK between the tablet terminal 1a and the tablet terminal 1b.

Figure 3 is a diagram showing the data structure of digital ink DINK generated by tablet terminal 1a. As shown in the figure, digital ink DINK is composed of ink data 100 and metadata 120. Of these, ink data 100 is composed of stroke data group 102 including multiple stroke data ST (see Figure 4) each indicating the result of handwritten input (i.e., the trajectory of the pointer), and context data 101 used to draw each stroke data ST.

The context data 101 includes brush attribute data 103, a timestamp 104, and haptics data 105. The brush attribute data 103 stores data (brush attributes) required for drawing stroke data ST, such as drawing color 107, transparency 108, and tip shape 109 of electronic pen P, in association with a brush ID 106. The timestamp 104 represents the time when the input of the first stroke data ST included in the digital ink DINK was started, expressed as the elapsed time from a predetermined time (for example, 0:00:00 on January 1, 1970). The haptics data 105 is data that associates haptic feedback with one or more stroke data ST, and includes a timestamp 111, target stroke data specification data 112, and haptic feedback specification data 113 in association with an area ID 110. Each data constituting the haptics data 105 will be described in detail later.

Figure 4 is a diagram showing an example of the stroke data group 102. As shown in the figure, each stroke data ST included in the stroke data group 102 has a structure including multiple position data PD between the <trace> tag and the </trace> tag. As shown in Figure 4, each position data PD includes an X coordinate, a Y coordinate, a tilt angle Tilt, a pen pressure value F1, and a grip force value F2. In addition, the stroke data ST includes a brush attribute reference value brushRef and a time offset timeOffset as attributes. The brush attribute reference value brushRef is set to the brush ID 106 described above. Each stroke data ST is drawn according to each brush attribute stored in the brush attribute data 103 in association with this brush ID 106. In addition, data representing the input start time of the stroke data ST by the elapsed time from the time represented by the time stamp 104 in the context data 101 is set to the time offset timeOffset.

Here, position detection and data acquisition by the sensor controller 14 are typically performed at regular time intervals. Therefore, the host processor 10 can derive the movement speed of the indicator and the length of time the indicator is stopped from the amount of change in the coordinates contained in the stroke data ST. The movement speed or the length of time the indicator is stopped thus derived can be used to draw the stroke data ST.

Returning to FIG. 3, metadata 120 is information added to ink data 100, and includes haptic effect definition data 121. Haptic effect definition data 121 is data that defines the specific content of the haptic feedback (i.e., the content of the force generated by force-sense generating device 15), and includes a wave file 123 (waveform data) that specifies the input waveform to force-sense generating device 15, or its path information, in association with definition ID 122. When tablet terminal 1b reproduces haptic feedback, the reproduction of haptic feedback is achieved by supplying a current conforming to this wave file 123 to force-sense generating device 15.

Figure 5 is a diagram showing an example of haptic effect definition data 121. "HE1", "HE2", and "HE3" shown in the figure are each definition IDs 122. The haptic effect definition data 121 in the figure includes wave files 123 (wave-file 20, wave-file 30) in association with "HE2" and "HE3". "HE1" includes the description "No haptics", which indicates that no haptic feedback is to be provided. In this way, it is also possible to define in the haptic effect definition data 121 that no haptic feedback is to be provided.

Figure 6 is a diagram showing an example of haptic data 105. The haptic data 105 shown in this figure corresponds to the digital ink shown in Figure 1 (digital ink showing a letter from a father to a son).

Referring to FIG. 6, the haptic data 105 is composed of one or more partial haptic data PHDs each including a different region ID 110. Each partial haptic data PHD is composed of a time stamp 111, target stroke data specification data 112, and haptic feedback specification data 113 in addition to the region ID 110.

Area ID 110 is a serial number assigned to an area designated by the user as a target for haptic feedback (hereinafter referred to as a "haptics-imparting area"). Figures 1 and 6 show an example in which four haptics-imparting areas D1 to D4 have been designated by the user. As will be described in detail later, the user designates each haptics-imparting area by inputting stroke data ST (shown by dashed lines in Figure 1) that surrounds one or more stroke data ST using a pointer such as an electronic pen P.

The target stroke data identification data 112 is data that identifies one or more stroke data ST included in the corresponding haptics-imparted area, and is described by the serial number of the stroke data ST in the digital ink DINK. Hereinafter, each of the one or more stroke data ST identified by the target stroke data identification data 112 is referred to as "target stroke data ST." For example, in the haptics-imparted area D1 shown in FIG. 1, the target stroke data ST includes a series of stroke data ST indicating the character string "I love Emma-chan," a series of stroke data ST constituting a heart mark, and a series of stroke data ST indicating a colored circle (which is gray in the drawing but is actually written in chromatic colors) that forms the background of the heart mark. Also, in the haptics-imparted area D2 shown in FIG. 1, each of the series of stroke data ST constituting a cherry (including leaves and colored parts) is the target stroke data ST.

The haptic feedback specific data 113 is data that defines the content of the haptic feedback to be played when the target stroke data ST is played, and is described using the definition ID 122 shown in FIG. 3. For example, in the case of the haptics-imparted area D1, "HE3" is described in the haptic feedback specific data 113, so "wave-file 30" (see FIG. 5) corresponding to the definition ID 122 "HE3" becomes the content of the haptic feedback for the haptics-imparted area D1.

Timestamp 111 is timing information indicating the start timing of playback of the corresponding haptic feedback, and is described by the input start time of the earliest of the corresponding one or more target stroke data (i.e., the earliest time indicated by the time offset timeOffset shown in FIG. 4). In one example, the time offset timeOffset set for the earliest of the corresponding one or more target stroke data is set directly as timestamp 111. Timestamp 111 also plays a role in determining the order of each haptic-imparted area.

Below, we will explain in more detail the processing performed by tablet terminal 1a as a handwritten data generation device and the processing performed by tablet terminal 1b as a handwritten data reproduction device, with reference to a flow diagram showing the processing performed by tablet terminals 1a and 1b.

Figure 7 is a flow diagram showing the haptic feedback providing process performed by tablet terminal 1a. Note that the process shown in the figure is executed by the host processor 10 of tablet terminal 1a, and is executed while the digital ink DINK to which the haptic feedback is to be provided is being displayed.

As shown in FIG. 7, the tablet terminal 1a first enters a haptic feedback providing mode in response to a user operation (step S1). After entering the haptic feedback providing mode, the tablet terminal 1a waits for the user to specify the above-mentioned haptics providing area. This specification is performed by inputting stroke data ST that surrounds one or more stroke data ST using a pointer such as an electronic pen P, as described above.

The tablet terminal 1a accepts the designation of a haptics-imparting region when the stroke data ST input by the pointer forms a closed region (step S2). Then, based on the designated haptics-imparting region, one or more target stroke data ST to which haptic feedback is to be imparted are determined (step S3). Specifically, stroke data ST that is entirely contained within the designated haptics-imparting region may be determined as the target stroke data ST, or stroke data ST that is partially contained within the designated haptics-imparting region may be determined as the target stroke data ST.

Next, the tablet terminal 1a acquires the time offset timeOffset (see FIG. 4) of the earliest of the one or more determined target stroke data (step S4), and accepts the selection of the haptic feedback to be imparted (step S5). The selection in step S5 is performed by selecting one of the multiple definition IDs 122 stored in the haptic effect definition data 121 shown in FIG. 5. However, it is also possible to allow multiple definition IDs 122 to be selected here. This point will be explained in detail later in the second embodiment.

Next, tablet terminal 1a generates the partial haptics data PHD shown in FIG. 6. More specifically, tablet terminal 1a first assigns area ID 110 to the haptics-imparting area received in step S2 and sets it in the partial haptics data PHD. Then, in association with this area ID 110, it sets timestamp 111, target stroke data identification data 112, and haptic feedback identification data 113 in the partial haptics data PHD. The time offset timeOffset acquired in step S4 is set in timestamp 111. The serial number of each of the one or more target stroke data ST determined in step S3 is set in target stroke data identification data 112. The definition ID 122 received in step S5 is set in haptic feedback identification data 113.

After completing the generation of the partial haptics data PHD, the tablet terminal 1a determines whether or not the designation of the haptics-imparting area is complete based on the presence or absence of a user operation indicating completion (step S7). If it is determined that the designation is not complete, the tablet terminal 1a returns to step S2 and accepts the designation of the next haptics-imparting area. On the other hand, if it is determined that the designation is complete, the tablet terminal 1a generates digital ink DINK that includes one or more pieces of partial haptics data PHD that have been generated up to that point (step S8). With the above processing, the application of haptic feedback to the digital ink DINK is completed. As can be understood from the explanation so far, the above haptic feedback application process can be executed by the user who wrote the digital ink DINK.

Figure 8 is a flow diagram showing the haptic feedback reproduction process performed by tablet terminal 1b. Note that the process shown in the figure is executed by the host processor 10 of tablet terminal 1b.

First, tablet terminal 1b acquires digital ink DINK including haptic data 105 from tablet terminal 1a and displays it on its display 12 (step S10). Next, tablet terminal 1b determines whether the user has pressed the PLAY_ALL button or the digital ink display area (step S11).

Here, the PLAY_ALL button is a button for inputting a user instruction to tablet terminal 1b to play all of the stroke data ST in the digital ink DINK being displayed. In one example, tablet terminal 1b displays the PLAY_ALL button together with the digital ink DINK on display 12, and determines that the PLAY_ALL button has been pressed when this PLAY_ALL button has been tapped with a pointer. The same applies to pressing the display area of the digital ink DINK; tablet terminal 1b determines that the display area of the digital ink DINK has been pressed when a position within the display area of the digital ink DINK has been tapped with a pointer.

If it is determined in step S11 that the PLAY_ALL button has been pressed, the tablet terminal 1b first stores the current time in the storage unit 11 shown in FIG. 2 (step S12). Then, the stroke data playback process is started with all stroke data ST in the digital ink DINK as the playback target (step S13).

Figure 9 is a flow diagram showing the details of the stroke data playback process. Having started the stroke data playback process, the tablet terminal 1b determines whether the playback start timing for any of the stroke data ST has arrived (step S30). Specifically, the time offset timeOffset of each stroke data ST to be played back is compared with the elapsed time from the time stored in the memory unit 11 in step S12, and when stroke data ST is generated in which the latter exceeds the former, it is determined that the playback start timing for that stroke data ST has arrived. When the tablet terminal 1b determines that the playback start timing for certain stroke data ST has arrived, it starts playing that stroke data ST (step S31).

Here, the reproduction of each stroke data ST is performed by plotting each coordinate included in the stroke data ST at the above-mentioned constant time interval (the time interval of position detection by the sensor controller 14) and drawing a curve (for example, a Catmull-Rom curve) that interpolates between them. At this time, the tablet terminal 1b may use one or more of the above-mentioned tilt angle Tilt, pen pressure value F1, grip force value F2, moving speed of the indicator, length of time the indicator is stopped, and various brush attributes (drawing color 107 of the stroke data ST, drawing color 107 of the stroke data ST, transparency 108 of the stroke data ST, tip shape 109 of the electronic pen P, etc.) to draw the curve. For example, the curve may be drawn with a line width according to the pen pressure value F1 and the moving speed of the indicator, and a line color according to the drawing color 107. The reproduction of the stroke data ST thus performed ends when the plotting of all coordinates included in the stroke data ST and the drawing of the corresponding curves are completed.

If it is determined in step S30 that the playback start timing of any stroke data ST has not arrived, or if step S31 is completed, the tablet terminal 1b determines whether the playback start timing of any haptic feedback has arrived (step S32). Specifically, the timestamp 111 of each partial haptic data PHD contained in the digital ink DINK is compared with the elapsed time from the time stored in the storage unit 11 in step S12, and if partial haptic data PHD occurs in which the latter exceeds the former, it is determined that the playback start timing of the haptic feedback contained in that partial haptic data PHD has arrived. When the tablet terminal 1b determines that the playback start timing of a certain haptic feedback has arrived, it first stops the playback of the haptic feedback being played by stopping the supply of current to the force-sense generating device 15 (step S33). Then, the supply of current to the force-sense generating device 15 is started based on the haptic feedback (specifically, the wave file) for which the playback start timing has arrived, thereby starting playback of the haptic feedback (step S34).

If it is determined in step S31 that the playback start timing for any haptic feedback has not arrived, and if step S34 has ended, the tablet terminal 1b determines whether the playback end timing for the haptic feedback being played has arrived (step S35). Specifically, when playback of all stroke data ST included in the partial haptic data PHD corresponding to the haptic feedback being played has been completed, it is determined that the playback end timing for the haptic feedback being played has arrived. If the tablet terminal 1b determines that the playback end timing for the haptic feedback being played has arrived, it ends the playback of the haptic feedback by stopping the supply of current to the force-sense generating device 15 (step S36).

If it is determined in step S35 that the timing to end the playback of the haptic feedback being played has not yet arrived, and if step S36 has ended, the tablet device 1b determines whether playback of all stroke data ST to be played has been completed (step S37). If it is determined that playback has been completed, the stroke data playback process and the haptic feedback playback process are terminated, and if it is determined that playback has not been completed, the process returns to step S30 and is repeated.

Returning to FIG. 8, if it is determined in step S11 that the display area of the digital ink DINK has been pressed, the tablet terminal 1b first acquires stroke data ST corresponding to the pressed position (step S20). Specifically, stroke data ST passing through the pressed position may be acquired, or if there is no stroke data ST passing through the pressed position, stroke data ST closest to the pressed position may be acquired.

Next, the tablet terminal 1b determines whether there is partial haptic data PHD including target stroke data identification data 112 indicating the stroke data ST acquired in step S20 (step S21). If it is determined that there is no target stroke data, the tablet terminal 1b ends the haptic feedback reproduction process. On the other hand, if it is determined that there is target stroke data PHD, the tablet terminal 1b acquires the partial haptic data PHD (step S22) and stores in the storage unit 11 a time that is earlier than the current time by the time indicated by the time stamp 111 of the acquired partial haptic data PHD (step S23). This is to enable the playback of the corresponding stroke data ST and the corresponding haptic feedback to begin immediately after the user taps the display area of the digital ink DINK. Then, the stroke data reproduction process is started with all stroke data ST in the acquired partial haptic data PHD as the target for reproduction (step S24).

Details of the stroke data replay process started here are the same as those of the stroke data replay process described with reference to FIG. 9. However, the determinations in steps S30 and S32 are made based on the time stored in the storage unit 11 in step S23, instead of the time stored in the storage unit 11 in step S12. In addition, the stroke data ST for which the replay start timing is determined in step S30 and the haptic feedback for which the replay start timing is determined in step S32 are only those included in the partial haptic data PHD acquired in step S22.

As described above, with the tablet terminals 1a, 1b and digital ink DINK according to this embodiment, the haptic feedback to be applied to one or more target stroke data ST can be set within the digital ink DINK, allowing the person who wrote the text to intentionally set the haptic feedback to be applied during playback.

Next, tablet terminals 1a and 1b according to a second embodiment of the present invention will be described. Tablet terminals 1a and 1b according to this embodiment differ from tablet terminals 1a and 1b according to the first embodiment in terms of the structure of the partial haptics data PHD that is generated and the specific contents of the stroke data reproduction process, but are otherwise similar to tablet terminals 1a and 1b according to the first embodiment. The following description will focus on the differences from tablet terminals 1a and 1b according to the first embodiment.

Figure 10 is a diagram showing the structure of partial haptic data PHD according to this embodiment. As shown in the figure, the partial haptic data PHD according to this embodiment differs from the partial haptic data PHD according to the first embodiment shown in Figure 6 in that tactile feedback specific data 113 is stored in association with pen pressure value F1 (specifically, range 114 of pen pressure value F1). In order to generate such partial haptic data PHD, the tablet terminal 1a according to this embodiment is configured to allow the user to select a definition ID 122 for each range 114 of pen pressure value F1 in step S5 of Figure 7.

Figure 11 is a diagram showing details of the stroke data reproduction process according to this embodiment. As can be seen by comparing with Figure 9, in this embodiment, after a negative determination in step S32, a process (step S32a) is added to determine whether the timing for changing the haptic feedback has arrived.

To explain step S32a in more detail, after obtaining a negative determination result in step S32, the tablet terminal 1b according to this embodiment determines whether the timing to change the haptic feedback to be played has arrived based on the pen pressure value F1 included in the currently played position data PD and the range 114 of pen pressure values F1 included in the partial haptics data PHD corresponding to the currently played haptic feedback. If it is determined that the timing has arrived, steps S33 and S34 are executed, as in the case where a positive determination result is obtained in step S32. In step S34, the haptic feedback corresponding to the pen pressure value F1 included in the currently played position data PD is played. If it is determined that the timing has not arrived, the process proceeds to step S35.

As described above, according to the tablet terminals 1a, 1b and digital ink DINK of this embodiment, the pen pressure value F1 is associated with the tactile feedback specific data 113 in the partial haptics data PHD, and the timing for changing the tactile feedback is determined based on the pen pressure value F1, making it possible to change the tactile feedback in response to changes in the pen pressure value F1. Therefore, a user who plays back the digital ink can view the played back digital ink while realistically feeling the pen pressure of the user who wrote the digital ink.

In this embodiment, an example of changing the haptic feedback according to the pen pressure value F1 has been described, but it is of course possible to change the haptic feedback based on other criteria. For example, the haptic feedback may be changed according to a combination of two or more data selected from the above-mentioned tilt angle Tilt, grip force value F2, moving speed of the indicator, length of time the indicator is stopped, various brush attributes (drawing color 107 of stroke data ST, drawing color 107 of stroke data ST, transparency 108 of stroke data ST, tip shape 109 of electronic pen P, etc.), and a combination of these with pen pressure value F1. Even in this case, the specific structure of partial haptics data PHD and stroke data reproduction process may be similar to those described in this embodiment. However, in step S32a when the brush attribute is used as a criterion for changing the haptic feedback, a determination is made as to whether or not the timing to change the haptic feedback to be reproduced has arrived based on the brush attribute of the currently reproduced stroke data ST instead of the pen pressure value F1 included in the currently reproduced position data PD.

In the first and second embodiments, an example was described in which one force-sense generating device 15 is provided in the tablet terminal 1b, which is the handwritten data playback device. However, for example, the display 12 may be divided into multiple areas, with a force-sense generating device 15 provided for each area, and haptic feedback may be played by driving the force-sense generating device 15 in the area that contains the stroke data ST being played back. In this way, it becomes possible to provide localized haptic feedback to only a portion of the display surface, rather than the entire surface.

The force-sense generating device 15 may be provided in a device separate from the tablet terminal 1b. For example, when a microphone is connected to the tablet terminal 1b and the tablet terminal 1b is used as a karaoke device by playing music in sync with the playback of digital ink showing lyrics, the force-sense generating device 15 is placed inside the microphone, making it possible to provide haptic feedback to the hand holding the microphone in accordance with the progress of the music. In addition, the force-sense generating device 15 can be provided in various devices, such as a smart bracelet, a smart finger, a smart ring, smart glasses, a smart shirt with a built-in heart rate sensor and a respiration sensor, a smart watch, a Bluetooth (registered trademark) key tracker, smart shoes, smart socks, smart pants, a smart belt, and a baby controller using SGPS (Simultaneous Global Positioning System)/GPRS (General Packet Radio Service).

Coordinate data defining a haptics-imparting region may also be placed in the partial haptics data PHD. In this way, when the haptics-imparting region is pressed, it becomes possible to execute the processing from step S22 onward shown in FIG. 8.

In the first and second embodiments, one or more target stroke data ST to which haptic feedback is to be applied is determined based on a haptics-applied area specified by the user, but one or more target stroke data ST may be determined by other methods. For example, a semantic segmentation process may be performed on the digital ink DINK, and an area obtained based on the results (in one example, an area including one or more stroke data corresponding to a semantic paragraph) may be determined as a haptics-applied area, and one or more target stroke data ST may be determined based on the haptics-applied area.

Next, tablet terminals 1a and 1b according to a third embodiment of the present invention will be described. Tablet terminals 1a and 1b according to this embodiment differ from tablet terminals 1a and 1b according to the first embodiment in that haptic data is placed in the position data PD of each stroke data ST, not in the context data 101, and in that the communication unit 16 also supports short-range wireless communication such as Bluetooth (registered trademark) and is configured to be able to communicate with the electronic pen P via short-range wireless communication, but are otherwise similar to tablet terminals 1a and 1b according to the first embodiment. The following description will focus on the differences from tablet terminals 1a and 1b according to the first embodiment.

Figure 12 is a diagram showing an example of stroke data group 102 according to this embodiment. As can be seen by comparing with the example shown in Figure 4, the stroke data ST according to this embodiment differs from the stroke data ST according to the first embodiment in that haptic data HEP is included in each piece of position data PD. The haptic data HEP according to this embodiment is data indicating the content of the haptic feedback, and is described, for example, by the playback intensity of the haptic feedback (hereinafter referred to as "haptic intensity"). In the following, the explanation will continue assuming that the haptic data HEP is haptic intensity.

The communication unit 16 (see FIG. 2) of the tablet terminal 1b according to this embodiment is configured to be able to communicate by short-range wireless communication such as Bluetooth (registered trademark). Similarly, the electronic pen P is also configured to be able to communicate by short-range wireless communication. The electronic pen P according to this embodiment further includes a force-sense generating device similar to the force-sense generating device 15 described above. As will be described in more detail later, the tablet terminal 1b according to this embodiment is configured to control the electronic pen P using short-range wireless communication in order to generate tactile feedback in the force-sense generating device within the electronic pen P.

Figure 13 is a flow diagram showing the stroke data generation process performed by tablet terminal 1a according to this embodiment. Note that the process shown in the figure is executed by the host processor 10 of tablet terminal 1a.

As shown in FIG. 13, the tablet terminal 1a first determines whether the position and data of the electronic pen P have been provided from the sensor controller 14 (step S40). The tablet terminal 1a repeatedly executes step S40 until the position and data of the electronic pen P are provided.

After determining in step S40 that the position and data of the electronic pen P have been supplied, the tablet terminal 1a next determines whether pen down has occurred (step S41). In this determination, the tablet terminal 1a determines that pen down has occurred if the writing pressure value F1 included in the data supplied from the sensor controller 14 has changed from 0 to a value greater than 0. When it has determined that pen down has occurred, the tablet terminal 1a executes a process to start generating stroke data ST (step S42). Specifically, it generates the header portion (<trace> tag) of the stroke data ST shown in FIG. 12.

If it is determined in step S41 that pen down has not occurred, or if step S42 has been completed, the tablet terminal 1a determines whether pen up has occurred (step S43). In this determination, the tablet terminal 1a determines that pen up has occurred if the writing pressure value F1 included in the data supplied from the sensor controller 14 has changed from a value greater than 0 to 0. If the tablet terminal 1a determines that pen up has occurred, it executes a process to end the generation of stroke data ST (step S42). Specifically, it generates the footer portion (</trace> tag) of the stroke data ST shown in FIG. 12, and returns to step S40.

If the tablet terminal 1a determines in step S43 that pen-up has not occurred, it acquires haptic data HEP corresponding to the pen pressure value F1 contained in the supplied data (step S45).

Figure 14 shows a correspondence table that associates the pen pressure value F1 with the haptic data HEP, which is the haptic intensity. The tablet terminal 1a stores this correspondence table in advance, and in the process of step S45, the haptic data HEP is obtained according to this correspondence table.

Returning to FIG. 13, the tablet terminal 1a that has acquired the haptic data HEP adds the position data PD that includes the acquired haptic data HEP to the stroke data ST being generated, and returns the process to step S40. Through the above process, stroke data ST that includes the haptic data HEP in the position data PD is generated between pen-down and pen-up.

Figure 15 is a flow diagram showing the haptic feedback reproduction process performed by tablet terminal 1b according to this embodiment. Note that the process shown in the figure is executed by host processor 10 of tablet terminal 1b. In this example, host processor 10 is configured to use a force-sense generating device in electronic pen P or tablet terminal 1b to reproduce haptic feedback indicated by haptic data HEP included in the specified position data PD, in response to receiving a user's designation of position data PD.

To be more specific, the tablet terminal 1b first draws one or more pieces of stroke data ST (step S50). As a result, the one or more pieces of stroke data ST are displayed on the display 12 of the tablet terminal 1b.

Next, tablet terminal 1b determines whether or not the user has performed a mode setting operation to enter the haptic feedback playback mode (step S51). This operation is, for example, clicking or tapping a button displayed on display 12 by tablet terminal 1b. Tablet terminal 1b repeats the process of step S51 until it determines that a mode setting operation has been performed, and when it determines that a mode setting operation has been performed, it enters the haptic feedback playback mode and determines the playback method for the haptic feedback (step S52).

The method of reproducing the haptic feedback is, for example, either a method of causing the force-sense generating device in the electronic pen P to reproduce the haptic feedback, or a method of causing the force-sense generating device 15 in the tablet terminal 1b to reproduce the haptic feedback. For example, if the mode setting operation is performed by the electronic pen P, the tablet terminal 1b may determine whether or not the electronic pen P has a built-in force-sense generating device from the contents of the downlink signal transmitted by the electronic pen P, and if it is determined that the electronic pen P has a built-in force-sense generating device, it may decide to cause the force-sense generating device in the electronic pen P to reproduce the haptic feedback, and if it is determined that the electronic pen P does not have a built-in force-sense generating device, it may decide to cause the force-sense generating device 15 in the tablet terminal 1b to reproduce the haptic feedback. Also, for example, if the mode setting operation is performed by a finger, the tablet terminal 1b may decide to cause the force-sense generating device 15 in the tablet terminal 1b to reproduce the haptic feedback.

Next, the tablet device 1b sets a haptic feedback playback area around the drawing area of the stroke data ST drawn in step S50 (step S53).

Figure 16 is a diagram illustrating the haptic feedback reproduction area set in step S53. The curve 200 shown in Figure 16(a) represents an approximation curve (specifically, a Bézier curve, a Catmull-Rom curve, etc.) obtained from multiple coordinates included in the stroke data ST. The circle 201 shown in Figure 16(a) represents the position of each coordinate included in the stroke data ST and the magnitude of the pen pressure value F1 at that position.

When drawing the line stroke data ST in step S50, the tablet terminal 1b finds the envelopes 202, 203 of the circle 201 at each coordinate. Then, as shown in FIG. 16(b), the area between the envelopes 202, 203 is acquired as the drawing area 210 of the stroke data ST. The tablet terminal 1b draws the stroke data ST by filling this drawing area 210 with a given color, and as a result, the user can visually recognize the stroke data ST.

In step S53, the tablet terminal 1b, having thus drawn the stroke data ST, sets a haptic feedback reproduction area 211 that is invisible to the user in the drawing area 210 and its periphery (on both sides). As a specific example, the tablet terminal 1b virtually sets a circle obtained by increasing the radius of each circle 201 described above by a predetermined percentage (for example, 5%), obtains their envelopes 202a, 203a, and sets the area sandwiched between the envelopes 202a, 203a as the haptic feedback reproduction area 211. In this way, the tablet terminal 1b can reproduce haptic feedback in an area slightly larger than the drawing area 210. If such an effect is not required, the tablet terminal 1b can set the drawing area 210 as the haptic feedback reproduction area 211.

Returning to FIG. 15, the tablet terminal 1b that has set the haptic feedback playback area determines whether or not the user has performed a mode release operation to release the haptic feedback playback mode (step S54). This operation may also be, for example, a click or tap on a button that the tablet terminal 1b has displayed on the display 12. If the tablet terminal 1b determines in step S54 that a mode release operation has been performed, it returns the process to step S51. On the other hand, if it determines that a mode release operation has not been performed, the tablet terminal 1b next determines whether or not the position of the indicator has been supplied from the sensor controller 14 (step S55).

If the tablet terminal 1b determines in step S55 that no data has been supplied, it returns to step S54 and continues processing. On the other hand, if the tablet terminal 1b determines that data has been supplied, it determines whether the supplied position is within the haptic feedback reproduction area set in step S53 (step S56). If the tablet terminal 1b determines that the position is not within the haptic feedback reproduction area, it returns to step S54 and continues processing. On the other hand, if the tablet terminal 1b determines that the position is within the haptic feedback reproduction area, it determines the haptic data HEP to be reproduced based on the supplied position (step S57). Specifically, it determines the position data PD that includes the coordinates closest to the supplied position from among one or more position data PD included in the stroke data ST being displayed, and determines the haptic data HEP included therein as the data to be reproduced.

The tablet terminal 1b then determines whether the reproduction method determined in step S52 was a method of causing the force-sense generating device in the electronic pen P to reproduce haptic feedback, or a method of causing the force-sense generating device 15 in the tablet terminal 1b to reproduce haptic feedback (step S58). If it is determined to be the former, the haptic data HEP determined in step S57 is transmitted to the electronic pen P by short-range wireless communication (step S59), and if it is determined to be the latter, a current based on the haptic data HEP determined in step S57 is supplied to the force-sense generating device 15 (step S60). This causes haptic feedback corresponding to the position touched by the user with the electronic pen P or finger to be reproduced by either the electronic pen P or the tablet terminal 1b.

Here, the waveform of the current supplied to the force-sense generating device as a result of tablet terminal 1b executing step S59 or step S60 may be set in advance in electronic pen P or tablet terminal 1b. However, it is also possible to accept waveform data designated by the user in the stroke data generation process shown in FIG. 13, and include information indicating that waveform data in the haptics data HEP. In this case, in step S59, the corresponding wave file may be sent to electronic pen P.

As described above, according to the tablet terminals 1a, 1b, electronic pen P, and digital ink DINK of this embodiment, haptic data HEP is placed within the position data PD of each stroke data ST, and haptic feedback indicated by the haptic data HEP contained in the specified position data PD is reproduced in response to receiving a user's designation of position data PD. Therefore, when a user traces the displayed stroke data ST with the electronic pen P or a finger, it is possible to reproduce haptic feedback corresponding to the pen pressure value of the user when writing the part of the stroke data ST corresponding to the position where the electronic pen P or finger is touching.

In addition, according to the tablet terminals 1a and 1b, electronic pen P, and digital ink DINK of this embodiment, haptic data HEP is placed within the position data PD of stroke data ST, so that haptic feedback can be reproduced even with electronic pens and tablet terminals that do not support the haptic data 105 or haptic effect definition data 121 shown in FIG. 3.

In the present embodiment, an example has been described in which haptic feedback is reproduced when the user traces the displayed stroke data ST with the electronic pen P or a finger, but haptic feedback may also be reproduced when the stroke data ST is reproduced. Specifically, when reproducing a portion of the stroke data ST corresponding to each position data PD in the stroke data ST, the haptic feedback indicated by the haptic data HEP included in that position data PD may be reproduced. In this way, the user reproducing the digital ink can view the reproduced digital ink while realistically feeling the pen pressure of the user who wrote the digital ink, as in the second embodiment.

In addition, in this embodiment, in step S45 of FIG. 13, the haptic data HEP is obtained from the pen pressure value F1, but the tablet terminal 1a may obtain the haptic data HEP from other values such as the above-mentioned grip force value F2, the movement speed of the indicator, the tilt angle, and the gyro output value. Also, the haptic data HEP may be obtained from the heart rate and blood oxygen concentration by attaching a device that measures the heart rate and blood oxygen concentration to the user and supplying the measurement results to the tablet terminal 1a. In this case, the combination of values to be referenced in order to obtain the haptic data HEP may be placed in the left column of FIG. 14.

According to the process shown in FIG. 15, haptic data HEP is sent to the electronic pen P each time a position is supplied from the sensor controller 14 to the host processor 10, but the tablet terminal 1b may send haptic data HEP to the electronic pen P only if there is a change in the content of the haptic data HEP, and the electronic pen P may continue to reproduce tactile feedback based on the haptic data HEP received up to that point until new haptic data HEP is sent. In this way, the amount of communication between the tablet terminal 1b and the electronic pen P can be reduced.

In addition, in this embodiment, an example has been described in which values such as the pen pressure value F1 are converted into haptic data HEP within the tablet terminal 1b and the haptic data HEP is transmitted from the tablet terminal 1b to the electronic pen P, but it is also possible to convert values such as the pen pressure value F1 into haptic data HEP within the electronic pen P and transmit values such as the pen pressure value F1 from the tablet terminal 1b to the electronic pen P. In this case, it is preferable to place the correspondence table shown in FIG. 14 in advance within the electronic pen P.

Although the present embodiment has been described with reference to stroke data ST generated in accordance with input from an electronic pen P, the present invention is also applicable to stroke data ST generated in accordance with input from other types of pointers. For example, if the haptic data HEP is determined based on the movement speed of an air mouse operated in the air and the determined haptic data HEP is placed in the position data PD, a user playing haptic feedback can realistically experience the movement speed of the air mouse. Similarly, if the haptic data HEP is determined based on the grip force value of a spray-type electronic pen and the determined haptic data HEP is placed in the position data PD, a user playing haptic feedback can realistically experience the spray emission intensity.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and the present invention can be implemented in various forms without departing from the spirit of the invention.

Reference Signs List 1a, 1b Tablet terminal 2 Communication network 10 Host processor 11 Storage unit 12 Display 13 Sensor 14 Sensor controller 15 Force-sense generating device 16 Communication unit 100 Ink data 101 Context data 102 Stroke data group 103 Brush attribute data 104 Time stamp 105, HEP Haptic data 106 Brush ID
107 Drawing color 108 Transparency 109 Tip shape of electronic pen P 110 Area ID
111: time stamp 112: target stroke data specification data 113: haptic feedback specification data 114: range of pen pressure value F1 120: metadata 121: haptic effect definition data 122: definition ID
123 Wave file 200 Curve 201 representing a trajectory Circles 202, 203, 202a, 203a representing pen pressure values Envelope 210 Drawing area 211 Tactile feedback reproduction area brushRef Brush attribute reference values D1 to D4 Haptics imparting area DINK Digital ink F1 Pen pressure value F2 Grip force value P Electronic pen PD Position data PHD Partial haptics data ST Stroke data Tilt Tilt angle timeOffset Time offset

Claims (26)

A handwriting data generation device comprising a processor,
The processor,
determining target stroke data to which haptic feedback is to be associated from among stroke data generated in response to handwriting input;
generating haptic data indicative of a haptic feedback associated with the target stroke data;
generating digital ink that includes the stroke data and the haptic data;
Handwriting data generator. the processor determines the target stroke data based on a target haptic region to associate with haptic feedback.
2. The handwritten data generating device according to claim 1 . the processor accepts designation of the haptic region when the stroke data input by the pointing device forms a closed region;
3. The handwritten data generating device according to claim 2 . The processor determines the haptic region by performing a semantic segmentation process on the stroke data.
3. The handwritten data generating device according to claim 2 . The stroke data includes a writing pressure value.
The haptic data is data that associates the writing pressure value with waveform data for generating the haptic feedback.
2. The handwritten data generating device according to claim 1 . The stroke data includes a tilt angle.
The haptic data is data that associates the tilt angle with waveform data for generating the haptic feedback.
2. The handwritten data generating device according to claim 1 . The stroke data includes a grip force value.
The haptic data is data that associates the grip force value with waveform data for generating the haptic feedback.
2. The handwritten data generating device according to claim 1 . The stroke data includes a plurality of coordinates detected at a fixed time interval;
the haptic data is data that associates a moving speed of an indicator, which is indicated by the change amount of the coordinates, with waveform data for generating the haptic feedback;
2. The handwritten data generating device according to claim 1 . The stroke data includes a plurality of coordinates detected at a fixed time interval;
the haptic data is data that associates a length of time during which the indicator is stopped, which is indicated by the change in the coordinates, with waveform data for generating the haptic feedback;
2. The handwritten data generating device according to claim 1 . The stroke data includes brush attributes.
The haptic data is data that associates the brush attribute with waveform data for generating the haptic feedback.
2. The handwritten data generating device according to claim 1 . The stroke data includes a plurality of position data.
the processor associates haptic feedback with at least a portion of the stroke data by associating the haptic data with the position data.
2. The handwritten data generating device according to claim 1. The processor determines the haptic data each time a position is provided from a sensor controller that detects a position of an electronic pen, and generates the position data including the provided position and the determined haptic data.
The handwritten data generating device according to claim 11 . The processor determines the haptic data based on a pen pressure value provided by the sensor controller.
13. The handwriting data generating device according to claim 12 . A handwriting data reproduction device comprising a display and a processor,
The processor,
obtaining digital ink including stroke data and haptic data indicative of tactile feedback;
reproducing the stroke data for displaying the stroke data on the display;
controlling a force-sense generating device to reproduce the haptic feedback represented by the haptic data when the stroke data is reproduced ;
the haptic data is data indicating a haptic feedback associated with target stroke data among the stroke data,
the processor controls a force-sense generating device to reproduce, when the target stroke data is reproduced, the force-sense generating device reproducing the force-sense feedback associated with the target stroke data and indicated by the haptic data .
Handwritten data playback device. the haptic data includes timing information indicating a start timing of reproduction of the haptic feedback,
The processor,
determining, while the stroke data is being reproduced on the display, when a reproduction start timing for the haptic feedback has arrived based on the timing information;
when it is determined that the playback start timing has arrived, playback of the haptic feedback is started;
15. The handwritten data reproduction device according to claim 14 . the haptic data includes timing information indicating a start timing of reproduction of the haptic feedback,
The processor,
acquiring the haptic data in response to a user operation;
determining, while the stroke data corresponding to the acquired haptic data is being reproduced, when a reproduction start timing for the haptic feedback has arrived based on timing information included in the acquired haptic data;
when it is determined that the playback start timing has arrived, playback of the haptic feedback is started;
15. The handwritten data reproduction device according to claim 14 . the processor acquires the haptic data based on the stroke data corresponding to a position on the display pressed by a user;
17. The handwritten data reproduction device according to claim 16 . The stroke data includes position data including a writing pressure value,
the haptic data is data that associates the writing pressure value with waveform data for generating the haptic feedback;
the processor changes the content of the haptic feedback to be reproduced based on the pen pressure value included in the position data being reproduced and the pen pressure value included in the haptics data.
15. The handwritten data reproduction device according to claim 14 . The stroke data includes brush attributes.
the haptic data is data that associates the brush attribute with waveform data for generating the haptic feedback;
the processor changes the content of the haptic feedback to be reproduced based on the brush attribute of the stroke data being reproduced and the brush attribute included in the haptic data.
15. The handwritten data reproduction device according to claim 14 . The stroke data includes position data including coordinates,
The haptic data is data included in the position data.
15. The handwritten data reproduction device according to claim 14 . the processor controls a force-sense generating device to reproduce the haptic feedback indicated by haptic data included in the position data when reproducing a portion of the stroke data corresponding to the position data.
21. The handwritten data reproduction device of claim 20 . the processor controls a force-sense generating device to reproduce the haptic feedback represented by haptic data included in the specified position data in response to receiving the designation of the position data by a user;
21. The handwritten data reproduction device of claim 20 . The processor,
providing a drawing area of the stroke data displayed on the display and a haptic feedback reproduction area around the drawing area;
accepting designation of the position data by the user in response to the user designating a position within the haptic feedback reproduction area;
23. The handwritten data reproduction device of claim 22 . the force-sense generating device is provided within the handwritten data reproducing device;
24. A handwritten data reproduction device according to any one of claims 14 to 23 . the force-sense generating device is provided in a device separate from the handwritten data reproducing device;
24. A handwritten data reproduction device according to any one of claims 14 to 23 . The separate device is an electronic pen.
26. The handwritten data reproduction device of claim 25 .

Images