JP7496699B2 - Display device - Google Patents

Description

The present invention relates to a display device.

In recent years, personal computers and tablets that support touch and pen input have become widespread, and various technologies related to handwriting input have been proposed.

For example, a technique has been proposed that responds to receiving multiple handwritten strokes in an input area by generating one or more handwritten word blocks when the multiple handwritten strokes are text input and positioning them according to a first predetermined layout criterion, and by generating a sketch content object when the multiple handwritten strokes are a sketch and positioning it according to a second predetermined layout criterion that is different from the first predetermined layout criterion (see Patent Document 1).

In addition, a technology has been proposed that allows image data to be deleted by inputting a specific command (gesture) for the image data that was input immediately before (see Patent Document 2).

JP 2018-530042 A Japanese Patent Application Laid-Open No. 6-309093

When polishing content such as document files, reviews by others are essential, and the use of applications that allow handwritten comments is increasing. Some such applications have a function that displays a list of comments added to content. However, there is an issue that simply displaying a list of comments does not accurately indicate the points or areas being discussed, and users cannot understand which parts of the content the handwritten comments were entered for.

In order to solve the above-mentioned problems, the display device of the present invention aims to provide a display device that outputs an image in which a handwritten figure and an image of content that includes the handwritten figure are superimposed as comment information.

A first embodiment for solving the above-mentioned problem is a display device including a display unit (e.g., display unit 110 in FIG. 2) for displaying content, an input unit (e.g., input unit 120 in FIG. 2), and a control unit (e.g., control unit 100 in FIG. 2), characterized in that the control unit accepts input of a handwritten figure for the displayed content via the input unit, and outputs an image of the content in the range including the handwritten figure and an image in which the handwritten figure is superimposed as comment information.

A second embodiment is a display device including a display unit (e.g., display unit 110 in FIG. 2) for displaying content, an input unit (e.g., input unit 120 in FIG. 2), and a control unit (e.g., control unit 100 in FIG. 2), characterized in that the control unit accepts input of a handwritten figure for the displayed content via the input unit, and when the accepted input of the handwritten figure is an arrow, outputs an image in which an image of the content within the area indicated by the arrow and the handwritten figure within the area indicated by the arrow are superimposed as comment information.

The display device of the present invention can output an image in which a handwritten figure and an image of content containing the handwritten figure are superimposed as comment information.

FIG. 1 is a diagram for explaining an overview of a system in a first embodiment. FIG. 2 is a diagram for explaining a functional configuration of the display device according to the first embodiment. 4 is a diagram for explaining content information and comment information in the first embodiment. FIG. FIG. 4 is a diagram for explaining processing in the first embodiment. FIG. 4 is a diagram for explaining an operation example in the first embodiment. FIG. 4 is a diagram for explaining an operation example in the first embodiment. FIG. 11 is a diagram for explaining a process in a second embodiment. FIG. 11 is a diagram for explaining an operation example in the second embodiment. FIG. 13 is a diagram for explaining a process in a third embodiment. FIG. 13 is a diagram for explaining an operation example in the third embodiment.

The following describes embodiments in which the display device of the present invention is applied. Note that each embodiment is described for the convenience of explaining the present invention, and the technical scope of the present invention is not limited to the following embodiments.

[1. First embodiment]
[1.1 Overall configuration]
1A is a diagram showing the appearance of a display device 10. The display device 10 may be, for example, a tablet-type terminal device, an information processing device such as a notebook computer, a large display device used in an electronic whiteboard or an interactive whiteboard (IWB), or a table-type display device. A user's operation on the display device 10 is input by, for example, detecting a touch on a touch panel or detecting an operation using an input device 15.

In this embodiment, the display device 10 is described as a single device, but it may be configured to include multiple display devices or to have a display device and a server work together.

For example, as shown in FIG. 1(b), the system may be configured to include a display device 10H used by an administrator user and multiple display devices used by other users. The following is an overview of the operation of the entire system.

(1) In the display device 10H used by user H, who is an administrator, user H selects content (F10). At this time, user H may add a comment to the content himself. The comment may be stored as a file (data) separate from the content.

(2) User A reads the content and, if necessary, comments from display device 10H on display device 10A used by user A (F12).

(3) User A performs a process to add a comment to the retrieved content (F14).

(4) The comment added by user A to the content is sent to display device 10H (F16). Display device 10H updates the comment corresponding to the content based on the received comment, so that the comment added by user A is reflected in the content.

(5) Similarly, users B and C also perform the process of adding comments to the content selected by user H.

In this way, by using this system, each user can add their own comments to the content.

The display device may also be a system that can be connected to a server device. For example, as shown in FIG. 1(c), a server device 20 with which the display device 10 can communicate is connected to the system. The server device 20 may be on the same network or in the cloud. As shown in FIG. 1(c), each display device 10 is configured to be able to communicate with the server device 20. That is, in the configuration of FIG. 1(b), content stored in the display device 10H and data related to comments are stored in the server device 20. The display device 10H, display device 10A, and display device 10B can each refer to the server device 20 to realize the same operation as the configuration described above.

The following explanation will be given for the case where one display device is used, but those skilled in the art will be able to apply this to the second and third configurations as well.

[1.2 Functional configuration]
Next, the functional configuration of the display device 10 will be described with reference to Fig. 2. Each component is not essential and it is sufficient to have what is necessary for the operation.

The control unit 100 is a functional unit for controlling the entire display device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 150, and is composed of one or more arithmetic units (e.g., CPUs (Central Processing Units)).

The control unit 100 also executes programs to operate as a comment processing unit 102, a UI (User Interface) processing unit 104, and a user authentication unit 106.

The comment processing unit 102 executes editing processes such as inputting, changing, and deleting comments in response to operations input by the user. The comment processing unit 102 adds necessary information to the comment input by the user and stores it in the comment information 156 of the storage unit 150. In addition, if comment information is stored in another display device 10 or the server device 20, the comment processing unit 102 may send and receive comments to and from other devices via the communication unit 160.

The comment processor 102 processes objects input by the user as comments. Objects input by the user include, for example, figures, text (characters), and symbols.

For example, the comment processing unit 102 accepts input of a text comment when the user inputs from a software keyboard or a hardware keyboard. Furthermore, when a figure is drawn using a touch panel or an operation pen, the comment processing unit 102 accepts input of a comment for the input figure. In the following description, a figure input by a touch operation on the touch panel or an operation with an operation pen is referred to as a handwritten figure. Handwritten figures include, for example, figures that make up letters or symbols, as well as figures such as points, lines, arrows, and rectangles. Furthermore, the comment processing unit 102 may accept input by switching between a text comment and a figure comment using a touch button or operation.

Furthermore, the comment processing unit 102 may convert handwritten figures into text by character recognition, or may convert the text into an image. In this embodiment, the user can add comments without being aware of whether they are text input or handwritten input.

Furthermore, the comment processing unit 102 may process images such as stamps as comments added by the user.

The comment processing unit 102 can also perform other common editing operations, such as inserting, deleting, modifying, replacing, and moving, as well as well-known editing operations such as changing the character type, moving the cursor, changing the font, changing the line color and thickness, and transforming the line.

The UI processing unit 104 generates a user interface screen and displays it on the display unit 110. In this embodiment, the UI processing unit 104 performs processing to display comments and a list of comments on the display unit 110. For example, as shown in FIG. 5A, the content is displayed on the entire display screen W100, and the comments are displayed vertically in the region R100. The comments may be displayed in chronological order, or may be displayed for each user. For example, a user icon is displayed in the region R102 in FIG. 5A. When a user icon is selected by a user, the UI processing unit 104 reads out the comment corresponding to the selected user from the comment information 156 and displays it. The UI processing unit 104 may display the list of comments superimposed on the content, or may divide the display region of the display unit 110 and display the list of comments in a region different from the region in which the content is displayed. The UI processing unit 104 may also switch between displaying and not displaying the list of comments.

The user authentication unit 106 authenticates the user. Specifically, the user authentication unit 106 refers to the user information 158 to, for example, authenticate the user who adds a comment or authenticate the owner (user) of the content. The user authentication unit 106 may use an external authentication server. When an authentication server is used, the user authentication unit 106 and the user information 158 may be stored in the authentication server.

The display unit 110 displays content and comments, as well as various states of the display device 10 and the state of operational input. For example, it may be configured with a liquid crystal display (LCD), an organic EL panel, or electronic paper that uses electrophoresis.

The input unit 120 accepts operational input from the user. For example, it is configured with a capacitive or pressure-sensitive touch panel. The input unit 120 may be a combination of a touch panel and an operational pen, or may be an input device such as a keyboard or a mouse. In addition, the input unit 120 may be configured to allow the user to input information, for example, by combining it with a microphone or the like to input voice information.

The input unit 120 also detects the position where the user performed an input operation, and outputs this to the control unit 100 as the input position. The input position is preferably, for example, a position detected on a touch panel, and is coordinates on the display screen displayed on the display unit 110. Alternatively, it may be a position such as the position of the cursor (the row or column position in a sentence) or layout information (for example, on a button or on a scroll bar).

The storage unit 150 is a functional unit that stores various programs and various data necessary for the operation of the display device 10. The storage unit 150 is configured, for example, with a semiconductor memory such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The storage unit 150 also stores content information 152, handwritten graphic information 154, comment information 156, and user information 158.

Content information 152 stores content and information related to the content. For example, as shown in FIG. 3(a), content information 152 includes the following information:

Content ID: Information for identifying content.
Content: Information related to content. Content is, for example, a text file, an image file, a portable document format (PDF) file, an office file, etc. As such content, actual data may be stored, or the location where the data is stored (for example, a folder location, a uniform resource locator (URL)) may be stored. Furthermore, the content may be composed of one or more pages.
Comment ID list: a list of comment IDs, including comment IDs corresponding to comments added to the content.

Handwritten graphic information 154 stores information about handwritten graphics input to content. A handwritten graphic input to content refers to a handwritten graphic directly input to the content by a user inputting the graphic by hand with a finger, an operation pen, or the like in the area where the content is displayed. Handwritten graphic information 154 stores the handwritten graphic and the content ID of the content to which the handwritten graphic was input. It may also store the position of the handwritten graphic on the content. The handwritten graphic may also be stored as a raster format image or a vector format image.

Comment information 156 stores comments added by users to content and information about the content. For example, as shown in FIG. 3(b), comment information 156 includes the following information:

Comment ID: Information for identifying a comment.
Content ID: Information for identifying the content corresponding to the comment.
Position: Information about the position where the comment is added in the content. This is the position in the content or the page of the content, and is indicated by, for example, coordinates (X and Y coordinates). If the content is text data or other sentence data, it may be indicated by rows and columns.
Author: Information about the author of the comment.
Reference ID: Information for identifying the comment (parent comment) that the comment refers to. When a user adds comments hierarchically, the parent comment ID is stored to identify the parent comment. Note that related comments, such as parent and child comments, are referred to as a comment group.
Comment: Information indicating the content of a comment actually input (added) by a user. The content may include, for example, text data, image data, data related to an attached file, etc.

User information 158 stores information about users. For example, as information for identifying a user, a login ID, password, name, terminal used, biometric information of the user, ID of the input pen used, etc. for each user may be stored. User information 158 also stores an icon associated with each user.

The communication unit 160 communicates with other devices. For example, it connects to a LAN (Local Area Network) to send and receive information about comments and documents with other devices. The communication method may be a LAN, which is a common Ethernet (registered trademark), or LTE/4G/5G communications.

It should be noted that the display device 10 only needs to have the functions of each of the above-mentioned configurations as needed. For example, in the case of a display device 10 used by a user who is not an administrator (a general user) or in the case of storing each data in the server device 20, the content information 152 and the comment information 156 do not need to be stored in the storage unit 150. In addition, various information may be stored in a server on the cloud as needed.

The configuration shown in FIG. 2 needs to include at least the control unit 100 and the memory unit 150, and needs to be capable of implementing the operations of this embodiment. Some functions may be implemented by an external device. For example, the display unit 110 may be configured as an external device such as a display. In other words, the display device 10 may be configured with a display device and a terminal device, and the control unit 100 and the memory unit 150 need only be provided in the terminal device.

[1.3 Processing flow]
A main process flow of this embodiment will be described with reference to the flow in Fig. 4. First, the control unit 100 displays the content on the display unit 110 (step S102). For example, the control unit 100 may read the content from the content information 152 or may receive the content via the communication unit 160.

The control unit 100 determines whether or not a touch input has occurred (step S104). Specifically, the control unit 100 determines that a touch input has occurred when it detects a touch-down or touch-up on the input unit 120 (touch panel) or when it detects an input by an operation pen on the input unit 120.

When there is a touch input, the control unit 100 transitions to a handwritten comment input state (mode) and executes handwritten graphic input processing (step S104; Yes -> step S106). The handwritten comment input state (mode) is a state (mode) in which handwritten graphics can be directly input to the displayed content. When the control unit 100 executes the handwritten graphic input processing, it outputs a handwritten graphic based on the trajectory of the input on the touch panel. The control unit 100 may temporarily output the handwritten graphic to the storage unit 150 until the input is completed, or may output it to the handwritten graphic information 154.

Next, the control unit 100 determines whether the input of the handwritten figure has ended (step S108). Here, examples of cases in which the input of the handwritten figure has ended include the following:

(1) When a Predetermined Time Has Elapsed Since Touch-Up When a predetermined time has elapsed since the input unit 120 (e.g., a touch panel) detected a touch-up for a user's touch input, the control unit 100 determines that the input of the handwritten figure has ended. The predetermined time may be set in advance or may be set by the user. The predetermined time is preferably about 100 milliseconds to 3 seconds, and more preferably about 1 second. As a result, even if a touch-down and a touch-up are detected multiple times due to the successive input of multiple handwritten figures, the control unit 100 can continue the input of the handwritten figures if it detects another touch-down before the predetermined time has elapsed.

(2) When an operation other than that for inputting handwritten figures is detected An operation for inputting handwritten figures is, for example, an operation of moving the touched position while touching the touch panel, or an operation of moving the operation pen while keeping it in contact (drag operation). Here, when an operation other than that for inputting normal handwritten figures (for example, double tapping or long press) is detected, the control unit 100 determines that the input of the handwritten figure has ended.

(3) When an operation is performed indicating that the input of the handwritten figure is completed When an operation is performed by the user indicating that the input of the handwritten figure is completed, the control unit 100 determines that the input of the handwritten figure is completed. For example, the control unit 100 causes the display unit 110 to select a button such as "end input," and when the button is selected by the user, the control unit 100 determines that the input of the handwritten figure is completed.

When the control unit 100 determines that the user has finished inputting the handwritten figure, it ends the handwritten comment input state (mode) and displays a menu on the display unit 110 (step S108; Yes -> step S110). Specifically, the control unit 100 displays the menu in the upper left or lower right of a rectangular area that is in contact with the periphery of the handwritten figure, or displays the menu around the rectangular area. In addition, the control unit 100 may display a colored thick frame around the menu or blink the menu after a predetermined time has elapsed after displaying the menu. By displaying the menu in this manner, the control unit 100 can display the menu in a way that makes it easy for the user to recognize it.

The displayed menu includes an option to add the input handwritten shapes to the content as a comment (commenting). By allowing the user to select whether or not to comment via the menu, the user can confirm whether or not to add the series of handwritten shapes input from touch-down to touch-up on the displayed content to the displayed content.

Next, when commenting is selected from the menu (step S112; Yes), the control unit 100 outputs the input handwritten figure as a comment (step S114). In this way, the control unit 100 can determine that the handwritten figure input while the handwritten figure input process was being executed (while the control unit 100 was in the handwritten comment input state (mode)) is the handwritten figure to be commented on, and can turn it into a comment.

When the control unit 100 comments out all handwritten figures input between the detection of a touch-down and the lapse of a predetermined time after the detection of a touch-up, and is unable to determine which handwritten figure to comment out, the control unit 100 may prompt the user to specify the handwritten figure to comment out. In this case, the control unit 100 displays on the display unit 110 a screen that prompts the user to specify the handwritten figure to comment out (for example, a screen that prompts the user to specify the range of the area that includes the handwritten figure), and determines the handwritten figure to comment out based on the user's specification.

In step S114, the control unit 100 first determines the handwritten shape to be turned into a comment. Then, among the content displayed on the display unit 110, a range that includes the handwritten shape to be turned into a comment is identified, and an image of the content in that range is obtained by capturing it. The image of the content obtained in this way becomes the background of the handwritten shape to be turned into a comment.

Here, the control unit 100, for example, identifies an area (e.g., a rectangular area or a circular area) that is in contact with the periphery of the handwritten figure as the range that includes the handwritten figure. Note that the control unit 100 may also identify an area that is expanded by a predetermined size (e.g., 10 pixels) from the area in contact with the periphery of the handwritten figure as the range that includes the handwritten figure.

The control unit 100 may specify an area to be captured based on the characteristics of the content image, thereby capturing an area in which the characteristics of the background content image can be grasped. For example, the control unit 100 specifies the background color of the content image, and divides the content image into multiple areas based on the specified background color. Furthermore, the control unit 100 specifies, among the multiple areas in the content image, one or more areas that contain handwritten shapes to be turned into comments, as areas to be captured. In this way, the control unit 100 can acquire the content image of an area of the content image that contains handwritten shapes and certain content included in the content image.

Note that if the area to be captured includes an object other than the handwritten shape to be commented (for example, a handwritten shape that has already been input), the control unit 100 may obtain an image in which the content image and the object are superimposed.

The control unit 100 may also specify the area to be captured based on the characteristics of the handwritten shape to be commented. For example, when the control unit 100 recognizes a handwritten shape that includes a line extended in a certain direction, such as an arrow or a line, the control unit 100 may capture from the direction indicated by the arrow or line (one end direction) so as to include an image of the content included in the area indicated by the arrow or line (target area). For example, the one end direction is the direction in which the arrowhead is located in the case of an arrow, and the direction in which the tip of a line (the position where touch-up is detected) is located in the case of a line. The method of specifying the target area will be described later.

Next, the control unit 100 outputs to the memory unit 150 an image in which the image of the acquired content is superimposed on the handwritten figure input by the user.

By the above-mentioned process, the control unit 100 superimposes the handwritten figure input by the user and the image of the content (i.e., the image that serves as the background of the handwritten figure) and outputs it as a comment.

At this time, the control unit 100 adds necessary information to the comment and outputs (stores) it in the comment information 156. For example, the control unit 100 stores the comment, the ID of the content to which the comment was added, and the position to which the comment was added in the comment information 156. Here, the control unit 100 sets the position to which the comment was added to, for example, a predetermined position on the boundary line of the range including the commented handwritten figure (one of the four corners in the case of a rectangular area). Note that the control unit 100 may set the position to which the comment was added to the center position of the range in which the content was captured, or the position where input of the handwritten figure started or a position in the vicinity thereof. Note that, if the control unit 100 has temporarily output the handwritten figure to the storage unit 150, the control unit 100 may output (store) the handwritten figure to the handwritten figure information 154.

If the user selects to perform another process instead of commenting, the control unit 100 executes the selected process (step S112; No → step S116; Yes).

Also, if none of the processes is selected, the control unit 100 may transition the process to step S106 to execute the handwritten graphic input process again (step S116; No → step S106).

[1.4 Operation example]
An operation example of this embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5(a), Fig. 5(b), and Fig. 5(c) are diagrams showing examples of display screens displayed on the display unit 110. The display screen W100 shown in Fig. 5(a) is an example of a display screen in which content is displayed across the entire screen, and comments are displayed in a list in an area R100. The area R100 may be located at any of the four positions on the edge of the screen: above, below, left, or right. Moreover, the area R100 does not have to be located at the edge of the screen, and may be displayed, for example, in a window format, superimposed on the content.

Comments added to the content are displayed in a list in area R100. For example, a comment from user A is displayed as comment C100, and a comment from user B is displayed as comment C102.

When displaying comments in a list, the comments may be displayed so that parent/child comments can be confirmed. For example, comment C102, which is a child comment of comment C100, may be displayed so that it is subordinate to comment C100, which is the parent comment (displayed so as to be nested).

In addition, handwritten shape C104 is displayed on the display screen W100 as a handwritten shape input by the user on the content.

After the input of the handwritten shape is completed, the display screen displayed on the display unit 110 transitions from display screen W100 to display screen W110 shown in FIG. 5(b). On display screen W110, menu M110 is displayed near the handwritten shape C114. The user can comment the handwritten shape by selecting an item indicating that the handwritten shape should be commented ("Comment" in the example of FIG. 5(b)) from menu M110. When commenting the handwritten shape is selected, an image in which the handwritten shape and an image of the content of the area including the handwritten shape are superimposed is output as a comment. After the comment is output, the display screen displayed on the display unit 110 transitions to display screen W120 shown in FIG. 5(c).

In the display screen W120 shown in FIG. 5(c), in an area R120 that displays a list of comments, an image that combines a handwritten figure with an image of a content that includes the handwritten figure, such as comment C120, is displayed as a comment. Therefore, the user can understand the comment that was entered and the image of the content that the comment is the subject of, simply by checking area R120.

Figure 6 is another diagram showing an example of operation. Figure 6(a) is an example of a display screen W130 that is displayed on the display unit 110 when a handwritten shape is input into a portion where the text "XX Steamer" is displayed as a content image. Figure 6(a) shows that a line has been input below the text "XX Steamer" as the handwritten shape, and further shows that an arrow and a shape constituting the characters indicating "Title Change!!" have been input. Furthermore, area E130 shows a rectangular range that is in contact with the periphery of the input handwritten shape.

In this case, an image of the content within the range indicated by area E130 is captured, and an image in which the captured image is superimposed on the handwritten graphic is output. Furthermore, comment information including the output image as a comment is output. As a result, in area R130 which displays the comments in a list, an image in which the image of the content (the text "XX Steamer") and the handwritten graphic are superimposed as a comment, as shown in comment C130. By simply looking at comment C130, the user can tell that the comment "Change the title!!" is in response to the text "XX Steamer".

Figure 6 (b) is an example of a display screen W140 that is displayed on the display unit 110 when an arrow is input as the handwritten figure to be commented. An area E140 included in the display screen W140 indicates a rectangular range that is bordered by the periphery of the arrow, which is the handwritten figure to be commented.

Display screen W140 shows that areas 142 and 148 are included as areas containing handwritten shapes (handwritten characters) that were input before the arrow, which is the handwritten shape to be turned into a comment, was input. Areas 142 and 148 are located in the direction indicated by the arrow that was input as the handwritten shape.

In this case, the display device 10 identifies, for example, area E142 as the target area out of areas 142 and 148, which are areas located in the direction indicated by the arrow.

Here, the display device 10 may determine the image (target image) indicated by the arrow and then specify the area including the target image as the target area. For example, the display device 10 may search for images present around the position of the arrow among the content images and determine the target image from the search results. The display device 10 may measure the distance between the character or image closest to the arrow and other characters or images adjacent to that character or image, and recognize the area up to where the distance becomes wider than a predetermined value as the target image. The characters searched for by the display device 10 may be text (characters) of an object, characters composed of handwritten figures, or text (characters) included in an image of the content. Furthermore, the display device 10 may determine a group image including an image (clump) adjacent to the target image as the background. Next, the display device 10 specifies the area including the target image or group image (for example, the area where the periphery of the target image or group image is in contact) as the target area.

The display device 10 may use a method of determining the spacing between objects and determining the target area based on that. If the display device 10 is unable to determine the target area based on the spacing between objects alone, it may use a method of performing character recognition, meaning recognition, and graphic recognition to determine the range in which the text is understandable as the target area. Furthermore, if the target area cannot be determined automatically, the display device 10 may display a GUI image (e.g., a screen on which the target area can be specified) to allow the user to specify the target area.

The method for identifying the target area may be a combination of two or more of the above-mentioned methods, or may be a method other than the above-mentioned methods. The method for identifying the target area may be predetermined or may be selected by the user.

A specific example of the operation will be described with reference to FIG. 6B. First, the display device 10 searches for one character (handwritten graphic object) present in the vicinity of the position of the arrow. As a result, for example, one character among the characters included in the region 142 is searched for by the display device 10. Next, the display device 10 searches for characters adjacent to the searched character. Here, since the characters included in the region 142 are not spaced apart from each other, an appropriate predetermined value is set, so that the display device 10 can recognize the characters included in the region E142 as the target image. On the other hand, the display device 10 can recognize up to the characters included in the region E142 as the target image by determining that the space between the characters included in the region E142 and the characters included in the region E148 is wider than the predetermined value. As a result, the display device 10 specifies the region 142 as the target region among the regions 142 and 148 located in the direction indicated by the arrow.

After identifying the target area, the display device 10 then identifies the area within which the content image will be captured by one of the following methods:

(a) Method for identifying an area including an area including an arrow and a target area The display device 10 identifies an area including an area including an arrow and an area in a direction indicated by the arrow, and sets the area as a range for capturing an image of the content. For example, the display device 10 identifies a rectangular area in which the peripheries of the area including the arrow and the area in the direction indicated by the arrow are in contact with each other.

In the example of FIG. 6(b), the display device 10 specifies area E144, which is a rectangular area where the peripheries of areas E140 and E142 meet, as the range for capturing an image of the content and as the handwritten figure to be turned into a comment.

(b) Method of Specifying Only the Target Region The display device 10 specifies only the region in the direction indicated by the arrow, and sets this as the range for capturing the image of the content.

In the example of FIG. 6(b), the display device 10 identifies only area E142 as the range for capturing an image of the content.

(c) Method of specifying the other end direction of the arrow The display device 10 specifies an area in the other end direction other than the direction indicated by the arrow, in addition to the area including the arrow and the target area. The display device 10 then sets an area including the three areas including the area including the arrow, the target area, and the area in the other end direction other than the direction indicated by the arrow, as the range for capturing the content image. Note that the display device 10 may specify the area in the other end direction using the same method as the method for specifying the target area.

The example in FIG. 6(b) shows a case where area E146 is specified by the display device 10 as the area in the other end direction. In this case, the display device 10 specifies a rectangular area in which area E140, which is the area including the arrow, area E142, which is the target area, and area 146, which is the area in the other end direction, are bordered by their peripheries as the range for capturing the content image. As a result, as shown in comment C140 in FIG. 6(b), an image in which a handwritten arrow, characters located in one end direction and the other end direction of the arrow, and an image of the content of the area including the handwritten arrow and characters are superimposed is output as a comment.

In this way, the display device 10 identifies the target area based on the content images and objects, and the target area includes a group of images and objects. Furthermore, by acquiring an image of a range that includes the target area, the display device 10 can acquire a target image that conveys to the user the meaning of the direction indicated by the arrow, making the meaning of the comment clearer.

Note that while FIG. 6(b) describes a case where an arrow is input as a handwritten shape, if a line is input as a handwritten shape, an image of the content in the direction indicated by the line may also be captured.

In this embodiment, the condition (output condition) for outputting a handwritten figure as a comment is that an item for turning a handwritten figure into a comment is selected from a menu that is displayed a predetermined time after touch-up, but other conditions may be used. For example, the condition may simply be that a predetermined time has passed since touch-up, without displaying a menu.

If the content is made up of multiple pages, comments are added for each page. In this case, the handwritten graphic information 154 and the comment information 156 store information on the corresponding page. The UI processing unit 104 also displays the comments added to the page of the displayed content on the display unit 110 by superimposing them on the content, or displays them as a list of comments.

According to this embodiment, when a handwritten figure input by a user is added to content as a comment, the display device can output, as a comment, an image in which the image of the content in the area including the handwritten figure is superimposed on the handwritten figure. Also, comment information including the image thus output can be output. This allows the user to easily understand which part of the content the handwritten comment was input for, simply by viewing the list of comments.

[2. Second embodiment]
[2.1 Processing flow]
The second embodiment is an embodiment in which a file can be attached to a comment. In addition to the first embodiment, in this embodiment, the control unit 100 executes a file attachment process shown in FIG. 7. In this embodiment, the UI processing unit 104 displays an indication (e.g., a clip button) for receiving an operation to attach a file from the user on each comment displayed in a list or on a comment selected by the user from among the comments displayed in the list. In the following, only the points in this embodiment that are different from the first embodiment will be described.

First, the control unit 100 determines whether or not an operation to attach a file has been performed (step S202). The operation to attach a file is, for example, an operation in which the user selects a clip button, which is a display for accepting an operation to attach a file displayed in a comment displayed in a list. In this case, the comment including the clip button selected by the user becomes the comment to which the file is to be attached. Note that the operation to attach a file may be an operation (drag and drop) in which the file to be attached is dragged and dropped onto the comment to which the file is to be attached.

If an operation to attach a file has been performed, the control unit 100 controls the display unit 110 to display a file selection screen for selecting a file to attach to the comment (step S202; Yes -> step S204).

The control unit 100 displays the file selection screen until the user selects a file (step S206; No → step S204). On the other hand, if the user selects a file, the control unit 100 attaches the file selected by the user to the comment (step S206; Yes → step S208).

Specifically, the control unit 100 associates comment information 156 corresponding to the comment to which the file is to be attached with information on the file selected by the user in step S206. For example, the control unit 100 stores the file to be attached or the storage location of the file (for example, file path or URL) in comment information 156 corresponding to the comment to which the file is to be attached. Note that any method may be used as long as it is possible to associate a comment with a file, and the control unit 100 may use a method of associating the comment ID of the comment to which the file is to be attached with the file selected by the user and storing them in the storage unit 150.

When a user performs an operation to display a file attached to a comment, the control unit 100 displays the file attached to the comment on the display unit 110 based on the user's operation.

[2.2 Operation example]
An operation example of this embodiment will be described with reference to Fig. 8. Fig. 8(a) is a diagram showing an example of a display screen W200 on which content is displayed. An area R200 for displaying a list of comments is provided at the right end of the display screen W200.

Comments added to the content are displayed in a list in area R200. Here, a clip button M200 is displayed for each comment displayed in the list in area R200, such as comment C200. Note that while FIG. 8(a) shows a case where the clip button M200 is displayed for a comment selected by the user, the clip button may also be displayed for each individual comment displayed in area R200.

When the user selects the clip button M200, a display screen W210 including a file selection screen R210 is displayed, as shown in FIG. 8(b). The user selects a file to attach to the comment from the file selection screen R210.

Figure 8 (c) is a diagram showing an example of the display screen W220 when a file to be attached to a comment is selected by the user. In an area R220 displaying a list of comments secured at the right edge of the display screen W220, a display indicating that a file is attached is displayed for comments to which a file is attached. For example, as in the case of comment C220 with a file attached, a clip button M220 is displayed as a display indicating that a file is attached to the comment. In this case, the clip button M220 may be a button that accepts an operation to display the file attached to the comment.

Here, if the user selects the Clip button M220, the file attached to the comment is opened in the default application, and a display screen W230 showing the contents of the file is displayed, as shown in FIG. 8(d).

According to this embodiment, a user can attach related files to a comment. In addition, the contents of the attached files can be displayed by performing an operation on the comment displayed as a list. In this way, a user can understand the comments added to the content and the files related to the comments by simply viewing the list of comments.

[3. Third embodiment]
The third embodiment is an embodiment in which the control unit 100 controls switching between display and non-display of a handwritten figure superimposed on content. In this embodiment, in addition to the first and second embodiments, the control unit 100 executes a display switching process shown in Fig. 9. The display switching process is executed by the control unit 100 after the handwritten figure is stored in the handwritten figure information 154 in the first and second embodiments.

In this embodiment, the display or non-display of the handwritten figure is switched for each user. When the control unit 100 outputs the handwritten figure to the handwritten figure information 154, the control unit 100 outputs the handwritten figure in association with the user who input the handwritten figure, for example, by executing a process to output handwritten figure information 154 for each user. The user who input the handwritten figure can be determined, for example, by using the input device 15, based on the display device on which the comment input operation was performed, or by performing a switching operation at the time of input.

In addition, in this embodiment, when the control unit 100 displays a handwritten figure superimposed on the content, it displays an identification indicator indicating the user who input the handwritten figure near the handwritten figure. The displayed identification indicator may be, for example, an icon indicating the user, or a label or button indicating the user name, but in this embodiment, it will be described as displaying an icon indicating the user as the identification indicator.

3.1 Processing flow
The display switching process will be described with reference to Fig. 9. First, when a handwritten figure is not displayed on the content, the control unit 100 reads out the handwritten figure information 154 and displays the read handwritten figure superimposed on the content (step S302; No -> step S304). The control unit 100 also displays an icon indicating the user who input the handwritten figure near the handwritten figure (for example, the position where a comment is added).

Next, the UI processing unit 104 determines whether or not a touch input has been made to an icon representing a user displayed on the content (step S306). If it is determined that a touch input has been made to an icon on the content, the control unit 100 makes the handwritten figure input by the user and represented by the selected icon invisible (step S306; Yes -> step S308).

Specifically, the control unit 100 identifies the user indicated by the icon on which the touch input was performed, and prevents the handwritten graphic information input by the identified user from being read from the handwritten graphic information 154. The control unit 100 then redisplays the handwritten graphic, thereby preventing the handwritten graphic input by the user indicated by the selected icon from being superimposed on the content. Note that the control unit 100 leaves the icon on the content displayed.

The control unit 100 then determines whether or not a touch input has been made to the icon on the content selected by the user in step S306 (step S310). If it is determined that a touch input has been made to select an icon on the content, the control unit 100 displays the handwritten figure input by the user indicated by the selected icon (step S310; Yes -> step S312). Specifically, the control unit 100 reads out the handwritten figure information 154 corresponding to the user indicated by the icon that has been touched, and displays the read handwritten figure superimposed on the content displayed on the display unit 110.

After executing the process of step S312, the control unit 100 may transition to step S306 so as to again execute the process of hiding the hand-drawn figure.

[3.2 Operation example]
An operation example of this embodiment will be described with reference to Fig. 10. Fig. 10(a) is a diagram showing an example of a display screen W300 in which a handwritten figure is displayed superimposed on content. An area R300 shown in Fig. 10(a) is an area including a handwritten figure input by a user C. An icon M300 indicating the user C is also displayed near the handwritten figure input by the user C as an identification display. For example, as shown in Fig. 10(a), the icon M300 is displayed around the area R300 including the handwritten figure input by the user C.

When a touch input that is an operation for selecting the icon M300 is detected, as shown in FIG. 10(b), a display screen W310 is displayed on which the handwritten figure input by the user C is hidden. Note that the icon M310 representing the user C is displayed.

Furthermore, when a touch input, which is an operation to select icon M310, is detected on display screen W310, the handwritten shape input by user C is displayed, and a display screen such as that shown on display screen W300 in FIG. 10(a) is displayed again. In this way, each time an icon representing user C is selected, the handwritten shape input by user C can be switched between being displayed and not displayed.

According to this embodiment, the user can easily switch between displaying and hiding handwritten shapes, which generally cannot be hidden or require complicated operations to hide, by selecting the distinguishable display. This allows the user to always have an area available for inputting handwritten shapes.

In the present embodiment, the display/hide of handwritten shapes is switched for each user, but the display/hide of handwritten shapes may be switched for each comment, or the display/hide of handwritten shapes may be switched for all comments at once. When the display/hide of handwritten shapes is switched for each comment, the storage unit 150 may store handwritten shape information 154 for each comment. The UI processing unit 104 may read out handwritten shape information 154 corresponding to comments other than the comment selected to be hidden, and display the handwritten shapes stored in the read handwritten shape information 154 by superimposing them on the content.

[4. Modifications]
The present invention is not limited to the above-described embodiments, and various modifications are possible. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the gist of the present invention.

Furthermore, each function may be configured as a program, or may be configured as hardware. If the function is realized as a program, it may be recorded on a recording medium and the program may be read out and executed from the recording medium, or it may be stored on a network and downloaded and executed.

Although the above-mentioned embodiments are described separately for ease of explanation, they may of course be combined to the extent technically possible.

In addition, in the embodiments, the programs that run on each device are programs that control the CPU and the like (programs that make the computer function) so as to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and is then stored in various ROM, HDD, and SSD storage devices, and is read, modified, and written by the CPU as necessary.

When distributing the program on the market, the program can be stored on a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is of course included in the present invention.

Also, this can be achieved by installing and executing an application with each function on various devices such as a smartphone, tablet, or image forming device.

REFERENCE SIGNS LIST 10 Display device 100 Control unit 102 Comment processing unit 104 UI processing unit 106 User authentication unit 110 Display unit 120 Input unit 150 Storage unit 152 Content information 154 Handwritten graphic information 156 Comment information 158 User information 160 Communication unit

Claims (7)

A display device including a display unit for displaying content, an input unit, and a control unit,
The control unit is
accepting a handwritten input of a graphic for the displayed content via the input unit;
When an arrow is input as the handwritten figure and a figure constituting a line in the direction indicated by the arrow is input, an image of a first region of the content, which is a region where the periphery of the handwritten figure contacts, is captured and used as an image of the content;
When an arrow is input as the handwritten figure and a figure constituting a line in the direction indicated by the arrow is not input, an image of a second region of the content, which is a region where the handwritten figure and the periphery of a region located in the direction indicated by the arrow are in contact with each other, is captured and used as an image of the content;
outputting an image obtained by superimposing the captured content image and the handwritten figure as comment information;
A display device comprising: The control unit is
2. The display device according to claim 1, wherein, when a predetermined output condition is met, an image of the content including the handwritten figure and an image in which the handwritten figure is superimposed are output as comment information. It is equipped with a touch panel as an input unit.
3. The display device according to claim 2, characterized in that the output condition is that a menu is displayed after a predetermined time has elapsed since touch-up after input of the handwritten figure, and output as comment information is selected from the menu. 4. The display device according to claim 1, wherein the control unit stores the comment information in association with a file. The display device according to any one of claims 1 to 4, characterized in that the control unit controls switching between displaying and not displaying the handwritten figure. The control unit is
5. The display device according to claim 1, further comprising an identification display in the vicinity of the handwritten figure that indicates a user who inputs the handwritten figure. The display device according to claim 6, characterized in that the control unit controls switching between displaying and not displaying the handwritten figure when the identification display is selected.

Images