JP7769373B2 - Electronic style reading position notification device - Google Patents

Description

This invention relates to an electronic style reading position notification device that uses and processes audio readouts to recover the forgotten reading position of electronic style when reading an electronic style on an electronic display screen, often resulting from unintentional forgetting or forgetting due to falling asleep for a moment. This is particularly likely to occur when reading silently.

Looking back at prior art documents, the applicant's work can be found in Patent Documents 1 and 2. Patent Document 1 involves extracting the viewpoint by observing the face with a camera and displaying the viewpoint position on an electronic display screen. Patent Document 2 involves using a head-mounted light point applying means to display a light spot on an electronic display screen, which is then photographed with a camera to obtain data on the reading position. Both of these systems require at least a photographing means.
Patent Documents 3 and 4 do not recover the reading position of a forgotten electronic style.
In Patent Document 3, a gaze detection unit determines the order of parts referenced (seen) by the user, and when it matches a predetermined part order (e.g., parts proceeding in the order P, Q, R), it controls to give instructions necessary for the next action (e.g., displaying an arrow pointing to S). However, this has the disadvantage of requiring a complicated configuration, such as a gaze detection unit for a camera or the like and a parts display position database.
Patent Document 4 discloses a conference relay device that identifies an area (not the text information itself) in the conference materials indicated by text information converted from the voices spoken during the conference, and highlights that area. Specifically, the content of the text information and the intended meaning of the sentence are identified, for example,
When a voice utters a phrase such as "the second line" or "red letters," the sentence "second line" is highlighted on the display screen, and the portion written in "red" letters is also emphasized. In the present application, the written sentence itself is described as, for example, "the second line," but the position of the writing may be at the end of the sentence and not the "second line." Similarly, in the present application, the written sentence itself is merely described as "red letters," and the sentence itself is normally written in black letters, and the "area indicated by the text information" does not refer to the text itself. In the present application, the position of the electronic writing style being read is one location immediately before, but in Patent Document 4, it is not necessarily one location.
When viewing text, drawings, charts, or other displays on a display screen, it is common to lose track of where you were looking when you realized it, resulting in wasted time and effort in rereading the text. This often occurs when viewing difficult content or when fatigued. Even when not asleep, it is often the case that people lose track of where they were looking. Therefore, a simple device is needed to recover the reading position of an electronic text style that has been forgotten due to sudden sleepiness, etc., when reading an electronic text style on an electronic display screen. This is in response to the reality that people often lose track of where they were looking, regardless of whether they are asleep or not.

Patent application 2021-097677 Patent application 2021-101671 JP 2006-107048 Patent Publication No. 2011-65467

The objective of this application is to address the issue that occurs when reading an electronic style on an electronic display screen, where the reading position of the electronic style is often lost from the user's awareness due to unintentional forgetting or forgetting due to falling asleep for a moment. The objective is to provide a device with a simple configuration that uses and processes the audio of the reading aloud to restore the reading position of the forgotten electronic style and notify the user.

The following description is based on the claims.
The invention described in claim 1 is an electronic style reading position notification device,
A control means and a device controlled by the control means,
a sentence data storage means for storing sentence data and layout data of the sentence data; a display means for displaying the stored sentence data as a sentence on an electronic screen; a voice input means for inputting a voice of a user reading the displayed sentence; a voice recognition means for recognizing the input voice and converting it into character string data; and a character string temporary storage means for temporarily storing the converted character string data.
The system is provided with a reading position determination means for comparing the character string data with the sentence data in the sentence data storage means and determining a single reading position from the layout data, a matching result storage means for storing the matching results, a reading position display means for displaying the single reading position in association with the sentence in the display means, and a matching condition designation notification means for designating a matching condition, so that when the reading position becomes unknown, the user can easily find the reading position and resume reading,
The matching condition specifying notification means is characterized in that it is possible to specify whether or not to issue an alarm if the user does not read aloud for a predetermined period of time, if the user reads aloud in a stuttering manner, or if a matching reading position is not found even after the matching, and/or to specify conditions for issuing a notification urging the user to read aloud at appropriate time intervals.

As configured as described above, the electronic style reading position notification device of the present invention converts the text being read aloud into characters, compares the characters, and determines a single reading position, which is then displayed to the user, thereby avoiding the inconvenience of the reading position being unknown.

1 is a diagram showing an embodiment of the configuration of an electronic style reading position notification device according to the present invention; 1 is a diagram showing an embodiment of the operation flow of an electronic style reading position notification device according to the present invention; 1 is a diagram showing an embodiment of designating a matching condition for an electronic style reading position notification device according to the present invention; 10 is a diagram showing an embodiment of a function for single determination of a reading position of an electronic style reading position notification device according to the present invention; FIG. FIG. 10 is a diagram showing one embodiment of the collation function of the electronic style reading position notification device according to the present invention.

FIG. 1 is a diagram showing an embodiment of the configuration of an electronic style reading position notification device according to the present invention.
a sentence data storage means 110 for storing data on sentences and their layout, a display means 120 for displaying the stored sentence data as sentences on an electronic screen, a voice input means 130 for inputting the voice of a user reading the displayed sentence, a voice recognition means 140 for recognizing the input voice and converting it into character string data, and a character string temporary storage means 150 for temporarily storing the converted character string data.
The system is provided with a reading position determination means 160 that compares the character string data with the sentence data in the sentence data storage means 110 and determines a single reading position from the placement data, a matching result storage means 170 that stores the matching results, a reading position display means 180 that displays the single matching result as a reading position in association with the sentence in the display means, a matching condition designation notification means 190 that designates matching conditions or notifies the user, and a control means 200 that controls these.
When the user is reading a sentence on the display means 120, if the device is reading aloud, it uniquely identifies the position of the sentence being read and indicates that position on the display means 120. Therefore, even if the user loses track of where they were reading, they can easily find the reading position and return to where they were. This often happens when the user gets sleepy for a moment, or when they stop reading midway, and it is inconvenient to have to reread and find the reading position each time, but this device avoids this inconvenience.
The sentence and layout data in the sentence data storage means 110 is, for example, HTML-described data on the web, and is data in a description format appropriate for the application. There are many examples of the display means 120, including LCD displays and EL displays. The voice input means 130 is a so-called microphone. The voice recognition means 140 is a voice-to-text conversion technology that has become commonplace with the advent of deep learning and the development of artificial intelligence, to the point where it is installed in smartphones. Voice recognition programs are also commercially available and can be used. The reading position determination means 160 and the matching condition designation notification means 190 will be described in detail below. The control means 200 is a computer that controls the procedures for performing the operations of the above means. The storage means is a so-called memory.
It should be noted that devices that display sentences on the display means 120 generally have a means for specifying display conditions that change the display of sentences on the display means 120 in accordance with the user's preferences.

The operation of the reading position determination means 160 and the matching condition designation notification means 190 is described in Figures 2 to 5. The reading position determination means 160 compares the converted text of the read aloud sentence with the sentence data in the sentence data storage means 110 to determine the unique position of the sentence that was read. The matching condition designation notification means 190 specifies the various matching conditions in this case or serves as an interface for notifying the user, and details are provided below.

2 is a diagram showing an embodiment of the operation flow of the electronic style reading position notification device according to the present invention. A) The device is started. B) The collation conditions shown in 1) and 2) of FIG. 3 are specified.
C) Select the operation state shown in 3) of Figure 3. ・Select Match to start matching. ・You can also hide the display. ・Select End to end the process. ・Select Go to Matching Condition Specification to return to the matching condition specification in B) and change the conditions.

3 is a diagram showing an embodiment of the designation of the matching conditions of the electronic style reading position notification device according to the present invention.
1) Choose whether to use the same conditions as last time or specify them this time. If you select the same conditions as last time, matching will begin immediately. If you select specify them this time, you can change the parts you want to change. If this is your first time, you can specify all conditions or select the predetermined standard conditions. Examples of specified conditions are:
(1) For the matching time interval, you can specify whether to perform the matching at a predetermined interval or at a (specified) appropriate time interval.
(2) When displaying the collation results, the following can be specified: always, at appropriate time intervals, upon request, or no display.
(3) When matching characters in an inserted image, you can specify whether to match them or not.
(4) As an alarm condition, you can specify whether or not to issue an alarm when there is no reading aloud for a specified time, such as when the voice is interrupted and reading continues, when the reading aloud is disjointed, or when a matching reading position is not found even after verification.You can also specify a specified time.
In this way, an alarm can be issued in case of dozing off, etc. It is desirable that instead of an alarm, a notification such as "Please pronounce" to encourage reading aloud at appropriate time intervals can also be specified as a condition.
In such a case, the problem can be solved by restarting the matching process after the alarm.
Although it may be possible to specify whether to save or not save the state of the reading position etc. when the process is finished, it is usually sufficient to save the state when the process is finished.
If you select the end of the specified position, you will move to the start position specification field 2).

2) Specifying the start (including restart) position: You can specify it using one of the following.
(1) From the xth line of the sentence currently displayed or from the cursor selection position. (2) From the previous reading end position. If you specify this, the previous reading page will automatically open and display the position.
(3) From page y, line z of the electronic style currently displayed. (4) Specify details from the table of contents of the electronic style currently displayed. Select Table of Contents → Start of Part → Page Advance → Line Confirmation and End Specification to move to the 3) Operation Status Selection/Input specification field.

3) Operation state selection/input: Can be specified by any of the following.
(1) Start of matching - Matching in progress: Enter the input to start matching and display the status of matching.
(2) Specify matching conditions: Switch to specifying matching conditions.
(3) Matching Suspension/Resume, End: When the user leaves the seat, in order to avoid the influence of other voices, the user can input the commands to suspend, resume, or end the matching.
・Saved ・Not saved: Specify whether to save the status of the reading position, etc., and notify the current status

Incidentally, 1) whether the same as the previous specification or the current specification and 2) the specification of the start position may be displayed on the entire display means 120 since the matching has not started. Once the matching has started, they may not be displayed.
3) The operation status selection may be performed while verification has already begun, and in that case, it is preferable to display the status at all times, in a minimal manner, such as on the edge of the display screen, so as not to interfere with the display of the sentence, and to make it possible to terminate the operation midway or change the conditions.In addition, the display unit for operation status selection may display a selection key called "display request," so that verification is performed, but the display is only displayed when requested.
Even when the display is off, the display can be restored by a user's request at any time using the "request display" selection key. In particular, this can be done by pressing a switch or by speaking a keyword that is unlikely to appear in a sentence.
Furthermore, there may be a lamp or other indicator that indicates that verification is in progress.

It is preferable that the following specifications be possible when specifying the matching conditions.
a) Specifying the range of the reading position display width a-1, specifying one phrase Example: In the sentence "I went to the doctor,"
It is possible to specify each segment, such as [I], [I went to the doctor], and [I went to the doctor].
E-2, Multiple clause specification (punctuation clause specification)
For example, in the sentence, "I had a fever since yesterday, so I went to the doctor. I got an injection and my fever went down."
It is possible to specify multiple segments, such as [I've had a fever since yesterday], [I went to the doctor], and [My fever went down after I got an injection].
B) Setting the type of reading position display B-1, Highlight display: Shown in a bright color as if marked with a highlighter B-2, Whiteout: The background of the text becomes black and the text becomes white B-3, Underline: A line is drawn under the text B-4, Enclosure: The text is enclosed in a square C) How to move the reading position display C-1, Current position: Moves by phrase (or multiple phrases) C-2, Add without moving: Once the position display (marking) is displayed, it does not disappear but is added. (It disappears when you select Clear or when you change pages)

FIG. 4 is a diagram showing an embodiment of the function of single determination of the reading position of the electronic style reading position notification device according to the present invention.
Suppose a user is reading aloud a sentence on an electronic screen. In the example of 4-A, when the characters "PQR" are read aloud, there are three corresponding reading positions, and it is not clear which one was being read.
4-B and 4-C are examples of a function that determines a single pronunciation. In 4-B, the function waits for the next scheduled pronunciation and adds that part to the comparison, resulting in a single pronunciation. In this case, it is advisable to extend the sentences that were pronounced before or after, as in this example, and repeat the comparison until a single pronunciation is determined. The longer the sentence is compared, the better the comparison performance, and as a result, a single pronunciation can be determined with certainty. In 4-C, when the user can determine which is the reading position from their memory of the sentence, the position is determined as a single pronunciation by clicking with a mouse or using a specific keyword (e.g., select select 2 (meaning select the second) or by voice instruction).
In 4-B, as a means of change, it is also possible to determine the same character that follows the previous reading position as a single reading position. This is possible because people usually read while moving in a fixed direction. However, it is common for people to go back and reread even earlier than the previous position because they cannot understand the meaning, so this approach can sometimes be inconvenient, but this inconvenience is a matter of frequency and can sometimes be tolerated. Next, Figure 5 shows an example of an unusual case.

5 is a diagram showing one embodiment of the matching function of the electronic style reading position notification device according to the present invention. a) This is a normal case where the next match candidate is read aloud along the direction of the sentence after the current reading position. However, as shown in b) the non-normal case,
b-1) Returning to an already read portion and reading b-2) Jumping to an unread portion and reading sometimes occur.
In this case, there may be cases where a match does not occur even when the next normal matching candidate, the previous (or subsequent) matching data, and even the previous (or subsequent) matching data are entered and compared. For example, the entire sentence may not be read aloud continuously, but in a disjointed manner. In such cases, a message such as "Please read aloud as a long sentence including the context" can be displayed to prompt the user to match the long sentence including the context. If this still does not result in a match, the system will display a message saying "No match with next matching candidate, etc." and prompt the user to specify matching conditions,
- It is preferable to have the user specify a location such as page □□□, line △△, end of specification, page 〇 before line △△, end of specification, or ***** line before (after) the current page, end of specification.

Furthermore, if you return to a section that you have already read, you can immediately find it because it has been previously matched.
However, if you jump to an unread section, you can specify conditions to assist in matching.
This response can also be considered when the current page suddenly jumps to another page that is different from the usual page, but when reading aloud, since it is acceptable to assume that the display has changed to a different page, it is possible to start from the beginning and check the page that has changed, or if it does not match the reading position, it can instead prompt the user to "specify the reading position."
This situation also applies when reading while moving back and forth between multiple sentences at the same time: when you jump from one sentence to another, the reading position in that sentence is displayed and saved, and when you move to another sentence (from sentence to sentence), the process described in 2) Specifying the start position in Figure 3 is carried out. In particular, a convenient process in this case is the ability to instantly specify the start position by specifying the Xth line or by using the cursor.

Since reading aloud is a necessary requirement, it is also preferable to perform partial cancellation of the match, which cancels the match result only at that point, in case the user makes or has made a sound unrelated to the sentence to be read, or in case a sound different from the user's is input. Also, the user may be warned with a message such as "Another sound has been input. Please do not input any more."
Although this would be a heavy function, it would also be possible to use voiceprint detection to distinguish and remove voices of other people or to issue a warning.
The intensity of the voice used for reading aloud can be normal sound of around 60 dB, as well as soft sounds of 10 dB to 30 dB and extremely soft sounds of 10 dB or less. Incidentally, the intensity of breathing sounds is around 10 dB to 20 dB. The handling of soft and extremely soft sounds can be found in JP 2014-063018 and JP 2015-228002.

As described above, the electronic style reading position notification device of the present invention compares the sentence being read aloud to determine the reading position and display it to the user, thereby avoiding the inconvenience of the reading position being unknown and making it extremely convenient for industrial use.

110 Sentence data storage means 120 Display means 130 Voice input means 140 Voice recognition means 150 Character string temporary storage means 160 Reading position determination means 170 Matching result storage means 180 Reading position display means 190 Matching condition designation notification means 200 Control means

Claims (1)

A control means and a device controlled by the control means,
a sentence data storage means for storing sentence data and layout data of the sentence data; a display means for displaying the stored sentence data as a sentence on an electronic screen; a voice input means for inputting a voice of a user reading the displayed sentence; a voice recognition means for recognizing the input voice and converting it into character string data; and a character string temporary storage means for temporarily storing the converted character string data.
The system is provided with a reading position determination means for comparing the character string data with the sentence data in the sentence data storage means and determining a single reading position from the layout data, a matching result storage means for storing the matching results, a reading position display means for displaying the single reading position in association with the sentence in the display means, and a matching condition designation notification means for designating a matching condition, so that when the reading position becomes unknown, the user can easily find the reading position and resume reading,
The matching condition designation notification means is capable of specifying whether or not to issue an alarm if the user does not read aloud for a predetermined period of time, if the user reads aloud in a staggered manner, or if a matching reading position is not found even after the matching, and/or is capable of specifying conditions for issuing a notification to encourage reading aloud at appropriate time intervals.