JP5654932B2 - User interface device, operation reception method using display device, and program - Google Patents

Description

  The present invention relates to a user interface via a display device.

  A display device including a touch screen (also referred to as a touch panel) can arrange objects such as buttons and icons relatively freely, and is advantageous in this respect over a key whose position is fixed (a so-called physical key). It can be said. On the other hand, since the object as described above is not fixed in position, it can be said that an operation error is more likely to occur than a physical key. Patent Document 1 describes a technique for improving operability by enlarging and displaying an image of a region near a touch position in a display device with a touch panel.

JP 2004-152217 A

The occurrence rate of operation mistakes in operations on the touch screen changes first depending on the skill of the user (operator) operation, that is, the degree of proficiency. However, the occurrence rate of operation mistakes is not always the same for the same user, and actually varies depending on the status of the operation at that time.
Therefore, an object of the present invention is to change the size of the hit determination area according to the situation so that the user's intention is easily reflected.

  The user interface device according to the present invention includes a display unit that displays an image in a display area, a display control unit that displays a first image including a selectable object in the display area, and the display instructed by an operator. A detection unit for detecting a position on an area; and a proficiency level coefficient including a position detected in the first image by the detection unit and indicating the transition of the position detected by the detection unit and the proficiency level of the operator A determination unit that determines whether or not the object is within a hit determination region whose size is determined by the determination unit, and a specifying unit that identifies an operation of the operator based on an instruction from the operator, wherein the determination unit determines the hit determination A specification that specifies that the instruction in the first image is an operation for selecting the object when it is determined that there is one object in the area When the operation is specified by the specifying unit, the proficiency level coefficient is updated according to the distance between the position detected in the first image by the detecting unit and the object selected by the operation. And an update unit.

In a preferred aspect, when the determination unit determines that there are a plurality of the objects in the hit determination area, the display control unit determines any of the plurality of objects as the operator. A second image to be selected by the operator is displayed on the display area, and the specifying unit is an operation on the object selected by the operator in the second image with the instruction in the first image. And the updating unit determines the position according to a distance between a position detected in the first image by the detection unit and a position in the first image of the object selected in the second image. Update proficiency factor.
In another preferred aspect, the display control unit displays an image obtained by enlarging the first image at a predetermined magnification and including a position detected by the detection unit as the second image, The magnification is changed according to the size of the hit determination area or the proficiency level coefficient.
In still another preferred aspect, the update unit is configured such that when the object selected by the operator in the second image is not the object having the shortest distance from the position detected in the first image, The proficiency level coefficient is updated so as to increase the size of the hit determination area.
In still another preferred aspect, the determination unit includes a first calculation unit that calculates a speed of an instruction by the operator based on a transition of the position detected by the detection unit, and the speed and the proficiency level By calculating using a coefficient, the size is determined so that the hit determination area increases as the speed increases.
In still another preferred aspect, the detection unit includes a contact position where the indicator is in contact with the display area when the operator indicates the position with the indicator, and the indicator is in contact with the display area. One or a plurality of non-contact positions that are positions before reaching the position are detected, and the first calculation unit calculates the speed based on a transition from the non-contact position to the contact position.
In still another preferred aspect, the update unit detects the position detected by the detection unit at the time of determination by the determination unit, and detects the position detected immediately before the position by the detection unit. When the difference between the two is equal to or greater than a threshold value, a predetermined reference value is used for the calculation instead of the speed calculated by the first calculation unit.
In still another preferred embodiment, the updating unit varies the reference value according to the proficiency level coefficient.
In still another preferred aspect, the determination unit includes a second calculation unit that calculates an offset amount and an offset direction, and is calculated by the second calculation unit from a position detected in the first image by the detection unit. Determination is made with the offset amount and the position moved in the offset direction as the center of the hit determination area.
In still another preferred aspect, the determination unit performs the determination while limiting the size of the hit determination area so as not to exceed a predetermined maximum size.

  An operation reception method for a display device according to another aspect of the present invention includes a display step of displaying a first image including a selectable object in a display area, and detecting a position on the display area instructed by an operator. The detection step, the position detected in the first image in the detection step, the position, the transition of the position detected before the position, and the proficiency coefficient indicating the proficiency level of the operator When there is one object in the hit determination area where is determined, a specifying step that specifies that the instruction in the first image is an operation of selecting the object, a position detected in the detection step, and the An update step of updating the proficiency level coefficient according to the distance to the object selected by the operation.

  A program according to another aspect of the present invention includes a display control step of causing a computer of a display device that displays an image in a display area to display a first image including a selectable object in the display area, and an instruction from an operator A detection step for detecting a position on the display area, and a position detected by the detection unit in the display area displaying the first image, the position detected in the detection step. A step of determining whether or not the object is in a hit determination area whose size is determined by the transition of the transition and a proficiency factor indicating the proficiency level of the operator, and specifying the operation of the operator based on an instruction from the operator A specific step in which it is determined in the determination step that there is one object in the hit determination area. A specifying step for specifying the instruction in the first image as an operation for selecting the object, and a position detected in the first image in the detecting step when the operation is specified in the specifying step. And an update step of updating the proficiency level coefficient according to the distance from the object selected by the operation.

  According to the present invention, the size of the hit determination area can be changed according to the situation so that the user's intention is easily reflected.

Block diagram showing the configuration of a display device Functional block diagram showing the functional configuration of the display device Flow chart showing operation of display device Flow chart showing area calculation processing Flowchart showing enlargement mode processing The figure which illustrates the display mode of the image before and behind magnification The figure which illustrates the mode of operation of a user, and a hit judgment area Flow chart showing area calculation processing Diagram for explaining the difference between operation with the right hand and operation with the left hand The figure for demonstrating the positional relationship of a contact position and a non-contact position Another example of the second image

[Summary of the Invention]
According to the present invention, in a display device including means (such as a touch screen) for detecting an instruction for a display area by an indicator, an area for receiving a user operation (hereinafter referred to as a “hit determination area”) is located in the vicinity of the indicated position. The object is set and the object included in the hit determination area is regarded as the user's operation target. Here, the indicator refers to a body part or instrument for the user to give instructions, for example, the user's finger or stylus.

  When the hit judgment area is provided, the user himself / herself intends to indicate the object, but when the user actually indicates a position slightly deviated from the object, such (incorrect) instruction is given to the object. Can be regarded as an operation instructing. Therefore, it can be said that the hit determination area allows an operation error to be regarded as not an operation error.

  The size of the hit determination area of the present invention varies depending on the degree of user's proficiency in operation (hereinafter referred to as “learning level”). The hit determination area basically becomes smaller as the user becomes familiar with the operation of the display device. This is because it is estimated that such a user rarely indicates a position different from his / her intention. Note that the proficiency level of the present invention can be varied for each user, and can also be updated by accumulating user operations.

  By the way, the occurrence rate of operation mistakes is not always the same for the same user. For example, it can be said that an operation error is more likely to occur when the user moves the indicator quickly or when the indicator moves a longer distance than when the user does not. Therefore, it can be said that the operation error occurrence rate has a correlation with the mode (distance or speed) of the transition of the instruction by the user. In addition, when the hit determination area is provided, the possibility that an operation unintended by the user is performed varies depending on whether a plurality of objects are densely packed or not.

  Therefore, the present invention is configured such that the size of the hit determination area is determined according to the situation at that time by determining the size of the hit determination area by a combination of the proficiency level and the transition of the instruction. Further, in the present invention, the display mode is different depending on whether or not there are a plurality of objects in the hit determination area, so that the user's operation is not mistaken for an operation not intended by the person. And easy operation when there is little risk of misperception. Further, according to the present invention, the feature of the display mode and the feature of changing the size of the hit determination area are combined, and the method of accepting an operation when a plurality of objects are displayed depends on the accumulation of the user's operations. It is also possible to achieve the effect of changing to an optimum one.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of a display device 10 according to an embodiment of the present invention. The display device 10 is a communication terminal having a display area in which a touch screen is overlaid, and includes a configuration corresponding to the user interface device of the present invention. The display device 10 is, for example, a mobile phone or a smartphone, and has a size and a shape that can be operated with one hand while being held by a user. Here, the display device 10 is assumed to be operated by a user with a finger. That is, the indicator in this embodiment is a user's own finger.

  As shown in FIG. 1, the display device 10 includes at least a control unit 100, a storage unit 200, a communication unit 300, a touch screen unit 400, and a physical key unit 500. In addition to these components, the display device 10 includes means (speakers, microphones) and vibrators for inputting and outputting audio, and further includes an acceleration sensor and a gyro sensor for detecting the tilt of the display area. May be.

  The control unit 100 is means for controlling the overall operation of the display device 10. The control unit 100 includes an arithmetic processing device such as a CPU (Central Processing Unit) and a memory corresponding to a main storage device, and realizes a predetermined function by executing a program. The control unit 100 controls information exchange between components described later, and performs information processing (calculation, etc.) during the delivery.

  The storage unit 200 is a means for storing information. The storage unit 200 includes a storage unit corresponding to an auxiliary storage device such as a hard disk or a flash memory, and stores data and application programs used by the control unit 100 for control. The storage unit 200 may include a removable medium, that is, a removable storage unit, and identifies a user (or display device 10) such as a UIM (User Identity Module) card or a SIM (Subscriber Identity Module) card. It is also possible to include a storage means in which the data is recorded.

  The communication unit 300 is a means for communicating with an external device. In this embodiment, the communication unit 300 is a unit for connecting to a network such as a mobile communication network, but is a unit that directly communicates with an external device without going through a network such as NFC (Near Field Communication). There may be. Note that the communication unit 300 is not an essential component in the present invention.

  The touch screen unit 400 is a unit that displays an image and receives a user operation. The touch screen unit 400 includes a display unit 410 and a sensor unit 420 in more detail. The display unit 410 includes a display panel that displays an image using a liquid crystal element or an organic EL (electroluminescence) element, a drive circuit that drives the display panel, and the like. The sensor unit 420 includes a sensor provided so as to overlap the display area of the display unit 410, and supplies coordinate information representing a position (hereinafter referred to as “instructed position”) instructed by the user to the control unit 100. Here, the coordinate information is described in a two-dimensional orthogonal coordinate system with an appropriate position in the display area as the origin. The sensor of the sensor unit 420 is, for example, a capacitive sensor that senses a finger using electrostatic coupling with the finger.

  Similar to the sensor unit 420, the physical key unit 500 is a unit that receives a user operation. However, the physical key unit 500 is different from the sensor unit 420 in that it includes a so-called physical key. The physical key of the physical key unit 500 is provided at a position different from the display area of the display unit 410. The physical key unit 500 supplies key information representing the physical key selected by the user to the control unit 100.

  The configuration of the display device 10 is as described above. With this configuration, the display device 10 displays an image in the display area by executing the application program, and changes the image displayed in the display area in accordance with a user operation. The application program executed by the display device 10 is, for example, a program (so-called Web browser) that receives HTML (HyperText Markup Language) data via the communication unit 300 and displays a Web page, or displays an image such as a map by the user. It is a program that displays an image by enlarging, reducing, scrolling, etc. according to the operation. The program executed by the display device 10 is not necessarily a single program, and there may be a plurality of programs.

  When the display image includes a selectable object, the display device 10 specifies a user operation using the hit determination area. Here, the selectable object refers to an image that can accept a user operation among images displayed in the display area. For example, in addition to an image having a so-called button function, an icon or a hyperlink is included. This is the case. The selectable object may be an image such as a so-called avatar and may show some reaction in response to a user operation. Hereinafter, such selectable objects are referred to as “operation objects”.

  FIG. 2 is a functional block diagram illustrating a functional configuration of the display device 10, particularly related to a function for receiving an operation by the touch screen unit 400 (hereinafter referred to as “operation receiving function”). The control unit 100 of the display device 10 realizes functions corresponding to the display control unit 110, the detection unit 120, the determination unit 130, the specifying unit 140, and the update unit 150 illustrated in FIG. Accept the operation. The operation reception function may be a function provided in the operating system or a function provided in a specific application program.

  The display control unit 110 has a function of displaying an image in the display area of the display unit 410. The display control unit 110 can switch the image to be displayed according to the determination result by the determination unit 130. The image displayed on the display unit 410 by the display control unit 110 includes an image including one or a plurality of operation objects (first image), and an image for causing the user to select the operation objects (second image). Is included.

  The detection unit 120 has a function of detecting an operation related to a user operation. More specifically, the detection unit 120 is functionally classified into a first detection unit 121 and a second detection unit 122. The first detection unit 121 acquires coordinate information from the sensor unit 420 to detect a position (that is, a designated position) designated by the user on the display area. On the other hand, the second detection unit 122 acquires the key information from the physical key unit 500 to detect the physical key selected by the user. The detection unit 120 further has a function of recording the detected operation as a history. The history here is a record in which the detected operation is associated with the detection time.

  The detection unit 120 may further have a function of detecting the type of user operation. The types here are tap (operation to tap the surface of the display area once), double tap (operation to tap the surface of the display area twice in succession), drag (operation to move the indicator while touching the display area) ) For distinguishing operations.

  The determination unit 130 has a function of determining whether or not there is an operation object in the vicinity of the designated position when the first detection unit 121 detects the designated position. The vicinity here is a range determined by the hit determination area. In the present embodiment, it is assumed that the hit determination area is a circle centered on the designated position. The radius (that is, the size) of this circle is determined by a user's fingering speed, which will be described later, and a proficiency level coefficient indicating the user's proficiency level.

  The determination unit 130 may further include a first calculation unit 131 and a second calculation unit 132, and may perform determination using these calculation results. The first calculation unit 131 has a function of calculating the fingering speed of the user based on the position transition detected by the first detection unit 121. Here, fingering refers to how the user's finger is carried (moved), and fingering speed refers to the speed of instructions when instructions are continuously given by the user. The fingering speed is a scalar amount that does not include information on the fingering direction. In the present embodiment, when it is necessary to calculate the fingering speed, the first calculation unit 131 detects the latest indication position detected at that time and immediately before the indication position. The fingering speed is calculated using the detection time of the indicated position. However, the 1st calculation part 131 may calculate the speed (for example, average value) of the movement between three or more positions not only in the speed of the movement between two instruction | indication positions. In addition, the 2nd calculation part 132 shall not be used in this embodiment, and is demonstrated in the modification mentioned later.

  The specifying unit 140 has a function of specifying a user operation based on a user instruction. In other words, the specifying unit 140 has a function of specifying that an operation in which the user points to a certain position on the display area corresponds to an operation for selecting an operation object. Here, the user's designated position is not necessarily the position of the operation object itself. Therefore, the specification by the specifying unit 140 includes determining which of the instructions that are unclear whether the operation is an operation object selecting operation using various conditions such as a hit determination area.

  The updating unit 150 has a function of updating the proficiency level coefficient based on the identification result obtained by the identifying unit 140. The update unit 150 changes the proficiency level coefficient according to how accurate the operation specified by the specifying unit 140 is performed. The update unit 150 records the proficiency level coefficient in the memory of the control unit 100 or the storage unit 200. The determination unit 130 reflects the user's operation mode in the determination by performing a determination using the proficiency level coefficient updated by the update unit 150. As a result, it is expected that the proficiency level coefficient and the size of the hit determination area are changed to those suitable for the user as the user's operation is repeated.

  FIG. 3 is a flowchart showing a series of operations related to the operation reception function of the display device 100. First, the control unit 100 displays an image including a manipulation object (a first image described later) on the touch screen unit 400 (step S1), and detects an instruction to the touch screen unit 400 by the user (step S2). When detecting the designated position, the control unit 100 executes a process of calculating the size (here, radius) of the hit determination area based on the designated position (step S3). The process of step S3 is hereinafter referred to as “region calculation process”.

  FIG. 4 is a flowchart showing the area calculation process. In the area calculation process, the control unit 100 calculates a time difference between the detection time of the designated position detected in step S2 and the detection time of the designated position detected immediately before the designated position, and the time difference is a predetermined threshold value. It is judged whether it is less than (step S31a). In other words, the determination in step S31a is to determine the continuity of instructions from the user. In order to obtain the fingering speed by calculation, it is desirable that two instructions are performed successively in a series of operations. This is because if the user stops his finger between two instructions and is performing another action or thinking, the time difference between these instructions becomes very large. This is because it is not always appropriate to obtain the fingering speed. The control unit 100 determines whether or not to obtain the fingering speed by calculation using an appropriate threshold value set so as to exclude such a case.

  When the time difference between the detection times of the two designated positions is less than the threshold, the control unit 100 calculates the fingering speed (step S32a). The fingering speed is a value obtained by dividing the distance between the two designated positions by the time difference. On the other hand, when the time difference between the detection times of the two designated positions is greater than or equal to the threshold value, the control unit 100 acquires a reference value for the speed used in place of the fingering speed calculated in step S32a (step S33a). ). The reference value is a predetermined value and is stored in the memory of the control unit 100 or the storage unit 200.

Then, the control unit 100 acquires a proficiency level coefficient (step S34a), and calculates the size of the hit determination area (step S35a). In the present embodiment, the size of the hit determination area can be expressed by the radius of a circle. Here, if the radius of the hit determination area is r, the proficiency coefficient is d, the fingering speed is v, and the reference speed value is v _def , the radius of the hit determination area is expressed by the following formula (1) or (2) Represented by an expression. Here, it is assumed that the proficiency level coefficient d is a value that increases as the proficiency level of the user is high (that is, when the user is proficient).
r = v / d (1)
r = v _def / d (2)

  The control unit 100 calculates the hit determination region r using the formula (1) when the fingering speed is calculated, and the formula (2) otherwise. The radius r of the hit determination area only needs to increase as the fingering speed increases and decreases as the proficiency level coefficient increases, and the calculation formula is limited to the above formula (1) or (2). is not. Equation (1) is an example that most simply shows the relationship between the radius r, the proficiency level coefficient d, and the fingering speed v.

  Note that the size of the hit determination area is preferably limited so as not to exceed a predetermined maximum size. This is because if the size of the hit determination area becomes larger than necessary, the possibility that even an operation object not intended by the user will be included in the hit determination area increases. Therefore, the control unit 100 of the present embodiment sets an upper limit value for the radius r, and when the radius r calculated by the equation (1) or (2) exceeds the upper limit value, the radius r is used as the calculation result. Set the upper limit regardless.

  When the size of the hit determination area is obtained by the area calculation processing of FIG. 4, the control unit 100 subsequently determines whether there is an operation object in the hit determination area of the radius r centered on the indicated position detected in step S2. . Here, the state where the operation object is in the hit determination area does not require that the entire operation object is included in the hit determination area, and at least a part of the operation object is included in the hit determination area. What should be done.

  Specifically, in this determination, a first determination (step S4) for determining whether or not there is an operation object in the hit determination area, and whether or not there are one or more operation objects in the hit determination area. This is realized by the second determination (step S5). If there is no operation object in the hit determination area, the control unit 100 determines that the instruction in step S2 does not correspond to the operation for selecting the operation object, and ends the process. In addition, when there are a plurality of operation objects in the hit determination area, the control unit 100 executes an enlargement mode process described later (step S9).

  On the other hand, when there is one operation object in the hit determination area, the control unit 100 determines that the instruction in step S2 corresponds to an operation for selecting the operation object. In this case, the control unit 100 notifies the application program that is controlling the display of the operation object that the operation object has been selected (step S8). Further, prior to this, the control unit 100 calculates a deviation amount between the selected operation object and the designated position (step S6), and updates the proficiency coefficient according to the deviation amount (step S7).

  The deviation amount referred to here is a numerical value indicating how far the user's designated position is away from the actually selected operation object. This deviation amount may be, for example, the distance between the gravity center position of the operation object and the designated position, and if the designated position does not overlap the operation object, any one of the designated position and the operation object It may be the shortest distance when connecting the points. For example, when the calculated deviation amount is smaller than a predetermined threshold, the control unit 100 increases the proficiency level coefficient (in order to reduce the size of the hit determination area). Alternatively, the control unit 100 decreases the proficiency level coefficient (in order to increase the size of the hit determination area) when the calculated deviation amount is larger than a predetermined threshold value. Note that the gravity center position of the operation object is the position of the center of the object if the object is a symmetrical figure. The center-of-gravity position of the operation object may be the position of the center of mass when the entire object is regarded as an object having a uniform mass distribution.

  The control unit 100 does not necessarily need to update the proficiency level coefficient. For example, the control unit 100 increases the proficiency level coefficient when the calculated deviation amount is less than the first threshold, and the deviation amount is equal to or greater than the second threshold (greater than the first threshold). In some cases, the proficiency level coefficient is reduced, while if the amount of deviation is greater than or equal to the first threshold value and less than the second threshold value, the proficiency level coefficient may be left as it is without being updated. Further, the control unit 100 can increase or decrease the proficiency level coefficient step by step using a larger number of thresholds.

  The operations in steps S6 to S8 described above are operations performed when it is determined that the operation object that the user is trying to select can be uniquely identified. On the other hand, when a plurality of operation objects can be assumed as the operation objects to be selected by the user, the control unit 100 allows the user to select without uniquely determining as in steps S6 to S8. At this time, the control unit 100 facilitates selection by the user by displaying an enlarged image of the image that has been displayed. The enlargement mode process in step S9 is a process for enlarging and displaying the image in this way to prompt the user to select an operation object. In addition, a mode for enlarging and displaying an image in this way is hereinafter referred to as an “enlargement mode”.

  FIG. 5 is a flowchart showing the enlargement mode process. FIG. 6 is a diagram illustrating a display mode of images before and after enlargement. In FIG. 6, the hit determination area is a circle indicated by a broken line with the point C being the designated position as the center. Further, the operation objects are Oa and Ob in the drawing, and these are hereinafter referred to as “object a” and “object b”. Furthermore, in the following, the image before enlargement shown in FIG. 6A is referred to as a “first image”, and the image after enlargement shown in FIG. 6B is referred to as a “second image”.

  In the enlargement mode process, the control unit 100 first displays the second image on the display unit 410 (step S91). At this time, the control unit 100 may switch the image displayed on the display unit 410 from the first image to the second image, but displays the second image without hiding the first image. It is also possible to do. For example, the control unit may display the first image in half of the display area and display the second image in the remaining half, or pop up the second image so as to overlap the first image. You may let them. In addition, when the display unit 410 has a plurality of display areas independently, the first image and the second image may be displayed in different display areas.

In the present embodiment, the second image is an image obtained by enlarging the first image at a predetermined magnification. It is desirable that the magnification of the second image is such that the objects a and b can be easily selected. Note that the control unit 100 can change the magnification of the second image in accordance with the size of the hit determination area and the proficiency level coefficient. For example, the control unit 100 may increase the magnification as the hit determination area is larger, or increase the magnification as the proficiency level coefficient is smaller.
Further, as shown in FIG. 6B, the second image may include a message that prompts the user to select.

  In the state where the second image is displayed, the control unit 100 specifies the operation object selected by the user (step S92). At this time, the user selects the operation object by instructing the operation object using the touch screen unit 400. If the displayed operation object is associated with the physical key, the physical key unit 500 may be used for selecting the operation object (not the touch screen unit 400).

  When the operation object is uniquely identified in this way, the control unit 100 determines that the operation object selected by the user is the operation object having the shortest distance from the designated position among the plurality of operation objects, that is, in step S2. It is determined whether or not the operation object is closest to the detected indication position (step S93). In addition, when the operation object selected by the user is the operation object closest to the designated position, the control unit 100 determines a deviation amount from the designated position. Here, the control unit 100 determines the shift amount by determining whether or not the center-of-gravity position in the first image of the operation object selected by the user in the second image is included in the hit determination region ( Step S94).

  When the operation object selected by the user is the operation object nearest to the designated position and the deviation amount is small, the control unit 100 updates the proficiency level coefficient so that it is larger than before. (Step S95). On the other hand, when the operation object selected by the user is not the nearest operation object from the designated position, or when the deviation amount between the operation object selected by the user and the designated position is large, the control unit 100 determines the proficiency level coefficient. Update to a smaller value than before (step S96). Then, the control unit 100 notifies the application program that controls the display of the operation object that the operation object has been selected (step S97).

  Note that the control unit 100 may make the updated proficiency level coefficient different when the determination at step S93 is negative (NO) and when the determination at step S94 is negative. Moreover, the control part 100 may update in the aspect from which a proficiency level coefficient changes in steps according to deviation | shift amount similarly to the case of step S7.

  By operating as described above, the display device 100 can switch between the case of being displayed in the enlarged mode and the case of not being displayed depending on the state of the user's operation. According to the display device 100, when the user's proficiency level is low, or when the user's fingering speed is fast, the display device 100 is more easily displayed in the enlarged mode than when the user is not. Further, according to the display device 100, when there is little possibility of misidentifying the user's intention, a desired operation object can be selected without going through the enlargement mode.

  FIG. 7 is a diagram illustrating an example of a user operation. Here, it is assumed that the image (first image) displayed in the display area is the same as the image illustrated in FIG. The example shown in FIG. 7A is a case where there is less possibility of misidentifying the user's intention than the example shown in FIG. In this case, even if the fingering speed and the proficiency level coefficient are the same (that is, the size of the hit determination area is the same), the display device 100 transitions to the enlargement mode only in the example of FIG. In the example of a), no transition is made to the enlargement mode.

  Further, the example shown in FIG. 7B is a case where the fingering speed is slower or the proficiency level coefficient is larger than the example shown in FIG. That is, the user in this case is either performing the operation slowly or is familiar with the operation. In this case, the size of the hit determination area is smaller than in the case of FIG. Therefore, even if the user in this case indicates the same position as in FIG. 6A, the user can select a desired operation object without shifting to the enlargement mode.

[Second Embodiment]
This embodiment is an embodiment in which a part of the first embodiment described above is changed. The present embodiment is an example in the case where the touch screen unit 400 can detect not only the state in which the indicator is in contact but also the state in which the indicator is in close proximity without being in contact. The touch screen unit 400 can perform such detection when the indicator is detected by a capacitive sensor, for example. The “proximity” here is generally about 1 cm although it depends on the performance of the sensor.

  The main differences in configuration between the present embodiment and the first embodiment are the coordinate information output by the sensor unit 420 and the position where the first detection unit 121 can detect. The coordinate information of this embodiment is not two-dimensional but three-dimensional information. However, the coordinates in the normal direction of the display area do not need to be information indicating a specific distance, and may be information that can distinguish whether they are in contact or in proximity. Further, based on such coordinate information, the first detection unit 121 can detect not only the above-described designated position, that is, the position in a close state before the designated position is reached, that is, the position where the user's finger contacts. It is. That is, the first detection unit 121 can detect a trajectory of a process in which the user indicates the specified position. In the following description, the designated position among the positions detected by the first detection unit 121 is indicated by “contact position”, and the other positions are indicated by “non-contact position”. One or a plurality of non-contact positions can be detected with respect to one contact position. In addition, the detection part 120 records a log | history about both a contact position and a non-contact position.

  Further, the main difference in operation between the present embodiment and the first embodiment lies in the area calculation process. The display device 10 of the present embodiment can detect that the user is gradually approaching the display area by detecting the contact position and the non-contact position. Can detect position transitions. If the display device 10 can detect the transition of the finger position in this way, the display device 10 can calculate the fingering speed based on the transition method. In this case, the display device 10 does not need a plurality of designated positions (contact positions) in order to calculate the fingering speed, and does not need to perform the determination in step S31a described above or use the speed reference value. It is possible to calculate the fingering speed. The operation of the present embodiment is the same as that of the first embodiment (see FIG. 3) except for the area calculation process in step S3.

  FIG. 8 is a flowchart showing the area calculation processing of this embodiment. In the present embodiment, when detecting the designated position (contact position) in step S2, the control unit 100 acquires the coordinate information of the designated position and the coordinate information of the non-contact position detected before the designated position (step). S31b, S32b). The coordinate information of the non-contact position acquired here need only be the position detected within a predetermined time (for example, 1 to 2 seconds) from the detection time of the coordinate information of the designated position. At this time, the control unit 100 also acquires the detection time associated with the coordinate information.

  Next, the control unit 100 calculates the fingering speed based on the information acquired in steps S31b and S32b (step S33b). At this time, the control unit 100 calculates the distance of the trajectory connecting the contact position and one or a plurality of non-contact positions, and the time difference between the detection time of the end point of the trajectory and the detection time of the start point, and the calculated distance The value divided by the calculated time difference is defined as the fingering speed. After calculating the fingering speed in this way, the control unit 100 acquires a proficiency level coefficient (step S34b) and calculates the size of the hit determination area (step S35b). The operations in steps S34b and S35b are the same as the operations in steps S34a and S35a of the first embodiment.

According to the display device 10 of the present embodiment, it is possible to calculate the fingering speed based on an operation in which the user brings a finger close to the display area. Therefore, in this embodiment, when calculating the fingering speed, it is not necessary to detect the indicated position more than once within a predetermined time.
Note that the display device 10 may execute the operation of the present embodiment in combination with the operation of the first embodiment. For example, when the indicated position is detected twice or more within a predetermined time, the display device 10 calculates the fingering speed based on the transition between the two indicated positions, and the indicated position within the predetermined time. Can be calculated based on the transition from the non-contact position to the contact position.

[Modification]
The present invention is not limited to the above-described embodiments, and can be implemented according to the following modifications, for example. Note that these modifications may be implemented in appropriate combination as necessary.

(1) The second calculation unit 132 has a function of calculating an offset for the hit determination area. Here, the offset refers to a deviation set with respect to the center of the hit determination area and the designated position. When it is necessary to provide an offset in the hit determination area, the second calculation unit 132 calculates the direction of the deviation (hereinafter referred to as “offset direction”) and the amount of deviation (hereinafter referred to as “offset amount”). The determination unit 130 determines the position moved in the offset amount and the offset direction from the indicated position as the center of the hit determination area.

  For example, the offset amount changes according to the proficiency level coefficient, and decreases as the proficiency level increases. The offset amount may be a value that increases as the fingering speed increases (faster). On the other hand, the offset direction changes depending on the inclination of the display area and whether the indicator is on the right side or the left side of the user. Note that whether or not to provide an offset in the hit determination area may be set in advance by the user.

  The offset direction is determined by, for example, how the user holds the display device 10 and what position on the touch screen is designated. For example, when the user holds the lower part of the display device 10 (front side as viewed from the user) with the left hand and points to the upper right part (back side as viewed from the user) of the display device 10 with the thumb of the left hand, It is difficult for your finger to reach (the position you are trying to point to). Therefore, it is conceivable that the actual indicated position is closer to the front side (or the left side) than the target position with respect to the fingering direction. In this case, an upward (or rightward) offset is given. Good. Further, if the user points to the lower left part of the display device 10 with the thumb of the left hand (that is, the part close to the base of the finger indicating the display area), a downward (or leftward) offset may be given. . Note that a method for determining the right and left of the hand operated by the user is, for example, as follows.

  FIG. 9 is a diagram for explaining the difference between the operation with the right hand and the operation with the left hand. 9A shows a mode in which the display device 10 is operated with its thumb (mother finger, first finger) while holding the display device 10 with the left hand, while FIG. 9B shows a mode in which the display device 10 is held with the right hand. The mode operated with the thumb is shown. In the figure, reference numeral 10A denotes a display area on which a touch screen is overlaid. In such a gripping mode, when the user moves the thumb around the base of the thumb (second joint) as a rotation center, a drag operation or a flick operation is realized. These operations are operations for rubbing the touch screen in a straight line. However, even if the user intends to rub the touch screen in a straight line, the operation is actually an arc, and the right hand swells to the left and the left hand to the right. become. This is because the finger motion is a rotational motion around the joint with the finger.

Therefore, when the rotation centers are P ₁ and P ₂ , the locus of the pointing position by the thumb becomes a curve like L ₁ and L ₂ , respectively. L ₁ can be approximated to a circular arc centered on P ₁ , and L ₂ can be approximated to a circular arc centered on P ₂ . The second calculation unit 132 can determine the hand used by the user for the operation based on the difference in bulging when such a linear (intended) operation is performed with the left and right hands. . Further, if the curves L ₁ and L ₂ are detected as trajectories, the positions of the centers P ₁ and P ₂ can also be estimated. It is also possible to presumely determine whether the user is gripping or gripping at the back. These determinations can also be made from a casual operation of the user. Therefore, the user does not need to explicitly perform an operation for teaching which hand he / she is operating.

  In the case of this example, the second calculation unit 132 can execute the calculation of the offset direction at a timing independent of the determination using the hit determination area (see FIG. 3). That is, when a drag operation or a flick operation is performed, the second calculation unit 132 may calculate an offset direction in advance based on the locus of the designated position and store this in the memory.

  Further, the tendency that the locus of the indicated position becomes an arc shape is recognized not only by the operation by the hand holding the display device 10 but also by the operation by the opposite hand. The locus of the indicated position when the display device 10 is operated with the hand on the side not holding the display device 10 is a complex rotational movement including movements at other joints such as a wrist and an elbow. There is no change in the locus of the shape approximated to.

  In addition to the example in which the locus of the designated position is used, the second calculation unit 132, based on the positional relationship between the contact position and the non-contact position, when the first detection unit 121 can detect the non-contact position. It is also possible to calculate the offset direction.

FIG. 10 is a diagram illustrating a positional relationship between the contact position and the non-contact position. As shown in the figure, the user index finger (index finger, the second finger) when operating in, in addition to the contact position P _a, forefinger belly portion (P _b, P _c) and the other fingers (P _d , _Pe , _Pf , _Pg ) can be detected as non-contact positions. These non-contact positions are usually located more to the right than the contact position if the user's hand is the right hand, and to the left than the contact position if the user's hand is the left hand. Therefore, when the non-contact position is detected more on the right side than the contact position, the second calculation unit 132 determines that the user is operating with the right hand, and the non-contact position is greater on the left side than the contact position. If detected, it can be determined that the user is operating with the left hand.

  In the case of this example, the second calculation unit 132 can calculate not only the horizontal direction but also the vertical offset direction. This is because when the user operates as shown in FIG. 10, more non-contact positions are detected on the near side than the contact position. Therefore, the second calculation unit 132 sets the offset direction in such a case as the upward direction. Note that the second calculation unit 132 may calculate the offset direction by a finer type such as “upper right” and “lower left” as well as the four directions of up, down, left, and right.

(2) The hit determination area is a combination of a distance from one operation to the next operation (hereinafter referred to as “fingering distance”) and a proficiency coefficient, instead of a combination of a fingering speed and a proficiency level coefficient. It may be determined based on the combination. This is because it can be said that when the fingering distance is long, an operation error is more likely to occur than when the fingering distance is short. Therefore, if the size of the hit determination area is increased when the fingering distance is long, it is possible to expect an effect of reducing the number of re-operations and shifting to the enlargement mode.

(3) The hit determination area is not necessarily a circle as long as it includes the designated position, and may be, for example, an ellipse or a rectangle. When the hit determination area is an ellipse, the control unit 100 may calculate either the major axis or the minor axis of the ellipse based on the proficiency level coefficient, or may be calculated based on a combination of the fingering speed and the proficiency level coefficient. . Further, when the hit determination area is a square, the control unit 100 may calculate the length of either the long side or the short side of the square based on the fingering speed and the proficiency level coefficient. In these cases, the ratio of the long diameter (or long side) to the short diameter (or short side) may be determined in advance.

  Alternatively, the control unit 100 sets the long diameter (or long side) as the user's fingering direction (direction of travel of the indicator) and the short diameter (or short side) as the direction orthogonal to the fingering direction, and is independent of each other. It is also possible to prepare a proficiency level coefficient and calculate each value. In this way, it is possible to change the ratio of the long diameter (or long side) to the short diameter (or short side) depending on how the user operates. In general, operational errors are more likely to occur in the fingering direction than in the direction orthogonal to the fingering direction.

(4) Unlike the above-described embodiment, the proficiency level coefficient may be a value that decreases as the proficiency level increases. In this case, the size of the hit determination area is determined based on the product of the fingering speed and the proficiency level coefficient. The size of the hit determination area does not have to be the product of the fingering speed and the proficiency level coefficient itself, but may be a value obtained by adding or multiplying other appropriate coefficients. When the proficiency level coefficient is set to a value that decreases as the proficiency level increases, the control unit 100 increases the proficiency level coefficient when increasing the size of the hit determination area and decreases the size of the hit determination area. In this case, the proficiency coefficient is changed in the opposite direction to the above-described embodiment, for example, the proficiency coefficient is decreased.

(5) The reference speed value is not a fixed value determined in advance, and may be a different value depending on the proficiency level coefficient. For example, the control unit 100 may divide the proficiency level into several levels and store a plurality of reference values corresponding to the levels in advance. Alternatively, the control unit 100 may perform a calculation using a function having the proficiency level coefficient as a variable, and calculate a reference value corresponding to the proficiency level coefficient at that time.

(6) The second image may not be an image obtained by enlarging the first image. If the second image is an image that allows the user to select one of the operation object candidates that the user wants to select, any specific display mode may be used.
FIG. 11 is another example of the second image. The control unit 100 may display the image illustrated in FIG. 11A or 11B instead of the image illustrated in FIG. The image in FIG. 11A is displayed by adjusting the position without enlarging the objects a and b. The image in FIG. 11B displays names assigned to the objects a and b as options instead of the objects a and b. At this time, the user can select a desired operation object via the physical key unit 500 by associating the first option with a physical key and associating the second option with another physical key. it can.

(7) The display device 10 may have a function of identifying a user, and may use a proficiency coefficient corresponding to each user for processing. The function for identifying the user may be realized by, for example, causing the user to input an ID or a password, or may be realized by reading biometric information such as a fingerprint. Further, when the present invention is implemented in a communication terminal, it is possible to identify a user based on information recorded on a UIM card or a SIM card.

(8) The present invention is not limited to a display device having a touch screen, and can be implemented by various display devices capable of detecting the position of the indicator. For example, the position of the indicator may be detected optically by photographing means such as a camera, or may be detected by infrared rays or ultrasonic waves. Moreover, when using the instrument containing a magnetic body as a pointer, it is also possible to detect the position of a pointer magnetically.

(9) The present invention is not limited to a mobile phone or a smartphone, and can be implemented in various electronic devices such as a tablet PC (Personal Computer), an electronic dictionary, and a portable game machine. In addition, even in a processing terminal where a user performs processing using his / her own recording medium (credit card with IC chip, cash card, prepaid card, etc.), information for identifying the user and a proficiency coefficient are recorded on the recording medium. Thus, the present invention can be implemented. Such a processing terminal is, for example, an automatic ticket machine such as a ticket or an ATM (Automated Teller Machine).

(10) The user interface device according to the present invention is not limited to the form of the display device 10 described above, and may be configured separately from the display device. For example, the present invention includes a computer device having at least a configuration corresponding to the control unit 100 and a display device having at least a configuration corresponding to the touch screen unit 400 and connected to the computer device by wireless communication or the like. Can also be implemented. In this case, the computer device can function as the user interface device according to the present invention as long as it has means for exchanging information with the display device.

(11) The present invention can also be grasped as a method for receiving a user operation using a program executed by the computer device, a hit determination area, or the like. The program can be provided in a form recorded on a recording medium such as an optical disk, or in a form in which the program is downloaded to a computer via a network such as the Internet and can be installed and used.

DESCRIPTION OF SYMBOLS 10 ... Display apparatus, 100 ... Control part, 110 ... Display control part, 120 ... Detection part, 121 ... 1st detection part, 122 ... 2nd detection part, 130 ... Determination part, 131 ... 1st calculation part, 132 ... 1st 2 calculating unit, 140 ... specifying unit, 150 ... updating unit, 200 ... storage unit, 300 ... communication unit, 400 ... touch screen unit, 410 ... display unit, 420 ... sensor unit, 500 ... physical key unit

Claims (12)

A display unit for displaying an image in the display area;
A display control unit for displaying a first image including a selectable object in the display area;
A detection unit for detecting a position on the display area instructed by an operator;
The position detected in the first image by the detection unit, the position is determined by the transition of the position detected by the detection unit and the hit determination region where the size is determined by a proficiency level coefficient indicating the proficiency level of the operator A determination unit for determining whether there is an object;
A specifying unit that specifies an operation of the operator based on an instruction from the operator, and when the determination unit determines that there is one object in the hit determination area, the first image A specifying unit for specifying that the instruction is an operation for selecting the object;
An update unit that updates the proficiency coefficient according to the distance between the position detected in the first image by the detection unit and the object selected by the operation when an operation is specified by the specifying unit A user interface device comprising: When the determination unit determines that there are a plurality of the objects in the hit determination area,
The display control unit causes the display area to display a second image for allowing the operator to select any of the plurality of objects.
The specifying unit specifies the instruction in the first image as an operation on the object selected by the operator in the second image;
The update unit calculates the proficiency coefficient according to a distance between a position detected in the first image by the detection unit and a position in the first image of the object selected in the second image. The user interface device according to claim 1, wherein the user interface device is updated. The display control unit
An image obtained by enlarging the first image at a predetermined magnification and including a position detected by the detection unit is displayed as the second image,
The user interface device according to claim 2, wherein the magnification is changed in accordance with a size of the hit determination area or the proficiency level coefficient. The update unit determines the learning coefficient when the object selected by the operator in the second image is not the object having the shortest distance from the position detected in the first image. The user interface device according to claim 2, wherein updating is performed to increase the size of the area. The determination unit
A first calculation unit that calculates a speed of an instruction by the operator based on a transition of the position detected by the detection unit;
The user according to any one of claims 1 to 4, wherein the size is determined by calculating using the speed and the proficiency level coefficient so that the hit determination area increases as the speed increases. Interface device. When the operator instructs the position with an indicator, the detection unit includes a contact position where the indicator contacts the display area, and a position before the indicator reaches the contact position. Detecting one or a plurality of non-contact positions,
The user interface device according to claim 5, wherein the first calculation unit calculates the speed based on a transition from the non-contact position to the contact position. The update unit is configured such that a difference between a detection time of the position detected by the detection unit at the time of determination by the determination unit and a detection time of a position detected immediately before the position by the detection unit is equal to or greater than a threshold value. 6. The user interface device according to claim 5, wherein in some cases, a predetermined reference value is used for the calculation instead of the speed calculated by the first calculation unit. The user interface device according to claim 7, wherein the update unit changes the reference value according to the proficiency level coefficient. The determination unit
A second calculation unit for calculating an offset amount and an offset direction;
The determination is performed with the position moved in the offset amount and the offset direction calculated by the second calculation unit from the position detected in the first image by the detection unit as the center of the hit determination region. The user interface device according to claim 1. The user interface device according to any one of claims 1 to 9, wherein the determination unit performs the determination while limiting the size of the hit determination area so as not to exceed a predetermined maximum size. A display step of displaying a first image including a selectable object in a display area;
A detection step of detecting a position on the display area instructed by an operator;
The hit determination area that includes the position detected in the first image in the detection step and whose size is determined by the position, a transition of the position detected before the position, and a proficiency level coefficient indicating the proficiency level of the operator A specifying step for specifying that the instruction in the first image is an operation for selecting the object when there is one object in the object;
An update accepting method for a display device, comprising: an update step of updating the proficiency level coefficient according to a distance between the position detected in the detection step and the object selected by the operation. In the computer of the display device that displays the image in the display area,
A display control step of displaying a first image including a selectable object in the display area;
A detection step of detecting a position on the display area instructed by an operator;
In the display area displaying the first image, including the position detected in the detection step, a transition of the position detected by the detection unit, and a proficiency level coefficient indicating the proficiency level of the operator A determination step of determining whether or not the object is in a hit determination area whose size is determined by
A specifying step of specifying an operation of the operator based on an instruction from the operator, wherein the first image is determined when it is determined in the determination step that there is one object in the hit determination area; A specifying step of specifying that the instruction in is an operation of selecting the object;
An update step of updating the proficiency level coefficient according to the distance between the position detected in the first image in the detection step and the object selected by the operation when the operation is specified in the specifying step. A program to execute and.

Images