JP6149211B2 - Mobile terminal device, program, and camera shake correction method - Google Patents

Description

  The present invention relates to a technique for correcting camera shake that occurs when a walking operator uses a mobile terminal device.

  In recent years, computers (hereinafter collectively referred to as “portable terminal devices”) that are assumed to be carried by users, such as mobile phones, smartphones, and electronic book browsing terminals, have become widespread. The portable terminal device can be used while the operator is moving or walking, and has an advantage that time during movement or walking can be used effectively. On the other hand, when an operator who is walking uses the mobile terminal device, there is a problem in that hand shake occurs due to the walking motion of the operator, and the screen shakes and is difficult to see.

  Conventionally, there has been proposed a technique for making a screen easy to see when the screen is shaken. For example, Patent Document 1 describes an imaging apparatus that determines whether or not a subject is stable without shaking in captured image data, and expands the zoom lens by widening if the subject is not stable. ing.

Japanese Patent Laid-Open No. 06-6666

  However, in the conventional technology, when camera shake is detected, there is a problem that only the content displayed on the screen is enlarged and the vibration of the screen itself is not suppressed. If the screen vibrates, even if the contents of the screen can be grasped, the problem that eyes are tired still remains. Moreover, since the content (display portion) displayed on the screen is reduced by enlarging, there is a problem that the screen switching frequency increases.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for correcting camera shake that occurs when a walking operator uses a mobile terminal device.

In order to solve the above problems, the invention of claim 1 is a portable terminal device operated by an operator, wherein the display means for displaying an image and the imaging period are continuous, and each of the faces of the operator is displayed. Image acquisition means for acquiring first image data and second image data, and a first pixel group expressing the operator's face from the first image data acquired by the image acquisition means, and the image A second pixel group representing the operator's face is extracted from the second image data acquired by the acquisition means, and the position of the first pixel group in the first image data and the second pixel data in the second image data are extracted. obtains the number of pixels moved of the pixels representing the face of the operator based on the two pixel groups located, on the basis of the moved pixel number and the first conversion factor is displayed on the display means images Detection means for detecting a change amount in a change in position relative to the operator's face, determination means for determining whether or not the operator is walking, and the determination means for allowing the operator to walk Display correction means for controlling the display position of the image displayed on the display means in the display means so as to suppress the amount of change detected by the detection means when it is determined that Prepare.

Further, the invention of claim 2 is a portable terminal device operated by an operator, the display means for displaying an image, the first image data that is continuous in the imaging cycle, each including the face of the operator, and An image acquisition unit that acquires second image data, and a first pixel group that represents the face of the operator is extracted from the first image data acquired by the image acquisition unit, and is acquired by the image acquisition unit A second pixel group expressing the operator's face is extracted from the second image data, and the position of the first pixel group in the first image data and the position of the second pixel group in the second image data The number of moved pixels of the pixel representing the operator's face is obtained based on the relative position between the image displayed on the display means and the operator's face based on the number of moved pixels. Strange Detecting means for detecting the amount of change in the position, determining means for determining whether or not the operator is walking, and when the determining means determines that the operator is walking, the detecting means Display correction means for controlling the display position of the image displayed on the display means according to the change amount and the second conversion magnification so as to suppress the change amount detected by .

The invention of claim 3 is a computer-readable program, and the execution of the program by the computer is continued in the imaging cycle by display means for displaying an image, each of which is performed by an operator. Extracting a first pixel group expressing the operator's face from the first image data acquired by the image acquisition means for acquiring the first image data and the second image data including the face, and the image acquisition means; A second pixel group expressing the operator's face is extracted from the second image data acquired by the image acquisition means, and the position of the first pixel group in the first image data and the second image data Based on the position of the second pixel group, the number of moved pixels of the pixel representing the operator's face is obtained, and the number of moved pixels and the first conversion magnification are determined. And detecting means for detecting a change in relative position change between the image displayed on the display means and the operator's face, and determining whether or not the operator is walking. When the determination means determines that the operator is walking by the determination means, the image displayed on the display means is suppressed so as to suppress the amount of change detected by the detection means. It is made to function as a portable terminal device provided with the display correction means which controls the display position in a display means .
According to a fourth aspect of the present invention, there is provided a computer-readable program, and the execution of the program by the computer is performed continuously with display means for displaying an image and an imaging cycle, each of which is performed by an operator. Extracting a first pixel group expressing the operator's face from the first image data acquired by the image acquisition means for acquiring the first image data and the second image data including the face, and the image acquisition means; A second pixel group expressing the operator's face is extracted from the second image data acquired by the image acquisition means, and the position of the first pixel group in the first image data and the second image data Based on the position of the second pixel group, the number of moved pixels of the pixel representing the operator's face is obtained, and based on the number of moved pixels, the table Detecting means for detecting a change amount in a relative position change between the image displayed on the means and the face of the operator, determining means for determining whether or not the operator is walking, When the determination means determines that the operator is walking, the display means is controlled according to the change amount and the second conversion magnification so as to suppress the change amount detected by the detection means. It is made to function as a portable terminal device provided with the display correction means which controls the display position in the said display means of the image currently displayed .

Further, the invention of claim 5 is a camera shake correction method in a mobile terminal device, the step of displaying an image on the display means of the mobile terminal device and an imaging cycle, each of which is an operator of the mobile terminal device Obtaining the first image data and the second image data including the face, extracting a first pixel group representing the face of the operator from the obtained first image data, and obtaining the obtained second image A second pixel group representing the operator's face is extracted from the data, and based on the position of the first pixel group in the first image data and the position of the second pixel group in the second image data. The number of moved pixels of the pixel representing the operator's face is obtained, and based on the number of moved pixels and the first conversion magnification, the image displayed on the display means and the relative of the operator's face Change of position Detecting the amount of change in the step, determining whether or not the operator is walking, and suppressing the detected amount of change when the operator is determined to be walking As described above, the display position of the image displayed on the display means on the display means is controlled .

Further, the invention of claim 6 is a camera shake correction method in a mobile terminal device, the step of displaying an image on the display means of the mobile terminal device and an imaging cycle, each of which is an operator of the mobile terminal device Obtaining the first image data and the second image data including the face, extracting a first pixel group representing the face of the operator from the obtained first image data, and obtaining the obtained second image A second pixel group representing the operator's face is extracted from the data, and based on the position of the first pixel group in the first image data and the position of the second pixel group in the second image data. In the change in the relative position of the image displayed on the display means and the face of the operator based on the number of moved pixels, the number of moved pixels representing the operator's face is obtained. Change A step of detecting, the step of determining the operator whether walking, when the operator is determined to be walking, so as to suppress the detected amount of change, the And a step of controlling a display position on the display unit of an image displayed on the display unit in accordance with a change amount and a second conversion magnification .

According to the first to sixth aspects of the present invention, the change in the relative position change between the image displayed on the display means and the face of the operator based on the acquired first image data and second image data. When the amount is detected and it is determined that the operator is walking, the display position of the image displayed on the display unit is controlled so as to suppress the detected change amount . Thereby, the image which suppressed the camera shake during walking can be displayed.

It is a figure which illustrates the external appearance of the portable terminal device in 1st Embodiment. It is a block diagram of the portable terminal device in a 1st embodiment. It is a figure which shows the functional block with which the portable terminal device in 1st Embodiment is provided with the flow of data. It is a flowchart which shows operation | movement of the display correction mode of the portable terminal device in 1st Embodiment. It is a figure which shows the functional block with which the portable terminal device in 2nd Embodiment is provided with the flow of data.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. However, unless otherwise specified in the following description, descriptions of directions and orientations correspond to the drawings for the convenience of the description, and do not limit, for example, a product, a product, or a scope of rights.

<1. First Embodiment>
FIG. 1 is a diagram illustrating an appearance of a mobile terminal device 1 according to the first embodiment. FIG. 2 is a block diagram of the mobile terminal device 1 according to the first embodiment. As shown in FIG. 1, the mobile terminal device 1 is a device designed to be portable by an operator, and is operated by the operator.

  The mobile terminal device 1 processes various information and generates control signals to control each component included in the mobile terminal device 1, a ROM 11 that stores a program 110 executed by the CPU 10, and a temporary CPU 10 RAM 12 used as a typical working area. Thereby, the portable terminal device 1 has the structure and function as a general computer.

  The operation and function of the CPU 10 will be described later. The RAM 12 is a representative example of a storage device that can read and write data by the CPU 10, and the storage device included in the mobile terminal device 1 is a storage device that can be accessed at a relatively small capacity and at high speed (ie, the RAM 12). It is not limited to. In addition to the RAM 12, the portable terminal device 1 may include a nonvolatile storage device, a storage device that can be accessed only at a relatively low speed, a large-capacity storage device, or the like, or a removable portable storage device. You may be comprised so that a medium can be utilized.

  The mobile terminal device 1 includes an operation unit 13, a display unit 14, an imaging unit 15, an image signal processing unit 16, and a sensor group 17.

  The operation unit 13 includes buttons, keys, switches, and the like that are operated by an operator. For example, the operation unit 13 includes a power button for turning on the power of the mobile terminal device 1. The operation unit 13 includes a touch panel disposed on the surface of the display unit 14 to be described later, and an operator can input an instruction by touching the screen (that is, the touch panel) of the display unit 14. ing.

  The display unit 14 has a function of displaying various types of information to the operator. Examples of the display unit 14 include a liquid crystal display, a liquid crystal panel, a lamp, and an LED.

  Although not shown in detail, the imaging unit 15 includes an optical element such as a lens and a photoelectric conversion element such as a CCD. The imaging unit 15 has a function of converting light incident from the lens into a digital electrical signal and transmitting the digital electrical signal to the image signal processing unit 16. The imaging unit 15 in the present embodiment is provided, for example, at the position shown in FIG. 1, and is configured to include the face of the operator who is gazing (browsing) the display unit 14 in the imaging range. . In other words, in the mobile terminal device 1 according to the present embodiment, the operator's face is imaged as a subject in the image data 160 captured when the operator is gazing at the display unit 14 in a normal posture.

  The imaging unit 15 in the first embodiment constitutes a so-called digital camera widely distributed in the market and has a function of capturing a color photograph (video) at a certain moment. In order to carry out the invention, it is not necessary to have a function of capturing an image for recognizing a subject by human eyes like a color photograph. That is, it is sufficient to detect a change in the relative distance or orientation between the subject and the mobile terminal device 1 by capturing light (not necessarily visible light) emitted from the subject, for example, an infrared camera. It may be anything.

  The image signal processing unit 16 is hardware that processes an electrical signal representing an image input from the imaging unit 15 and converts it into image data 160 having a predetermined format. That is, the image data 160 is data created directly by the image signal processing unit 16, but is also data captured by the imaging unit 15. That is, in the mobile terminal device 1, the imaging unit 15 and the image signal processing unit 16 have a function as an image acquisition unit that acquires the image data 160 obtained by imaging the surroundings.

  In the following description, a portion other than the operator among the subjects included in the image data 160 is referred to as “background”. That is, the “background” may include a person other than the operator. The “operator” in the subject will be described as including not only the operator's own body but also an object worn by the operator unless otherwise specified. For example, when an operator wears a hat, the hat is an “operator” as a subject and not a background.

  The sensor group 17 includes an acceleration sensor, a vibration sensor, a gyro sensor, a speed sensor, and the like, and is a detection device group that detects the movement of the mobile terminal device 1. Each sensor included in the sensor group 17 outputs the measurement result as information indicating the movement of the mobile terminal device 1. Information output from the sensor group 17 is used in particular to detect whether or not the operator is walking.

  In the present embodiment, the sensor group 17 has been described as a configuration including a plurality of detection devices. However, instead of the sensor group 17, only a single detection device may be provided. In that case, although detection accuracy falls, cost can be suppressed. Further, the sensors listed as detection devices constituting the sensor group 17 in the present embodiment are merely examples, and it is not necessary to include all of them, and sensors other than these are included in the mobile terminal device 1. You may employ | adopt as a detection apparatus which detects this motion (especially motion which shows an operator's walking motion).

  FIG. 3 is a diagram illustrating functional blocks included in the mobile terminal device 1 according to the first embodiment, together with a data flow. The determination unit 100, the detection unit 101, and the display control unit 102 illustrated in FIG. 3 are functional blocks that are realized by the CPU 10 operating according to the program 110.

  The determination unit 100 determines whether the operator is walking based on the measurement data 120 acquired by the sensor group 17 and creates determination data 121. Note that a method for determining whether or not the operator carrying the mobile terminal device 1 is walking based on an output (measurement data 120) from a detection device (sensor group 17) such as an acceleration sensor or a vibration sensor. Since conventional techniques can be applied as appropriate, details are omitted.

  The detection unit 101 detects a change in the relative position between the image displayed on the display unit 14 and the operator's face based on the image data 160 acquired by the imaging unit 15 and the image signal processing unit 16. . The detected relative position change is created as change data 122 by the detection unit 101. The principle of detecting a relative position change between the image displayed on the display unit 14 and the operator's face will be described later.

  The display control unit 102 has a function of displaying the display data 123 at a predetermined display position on the display unit 14. The display data 123 is information stored in advance as part of an application program, information created by an application, or the like, and is information representing an image displayed on the display unit 14. In the display data 123, each pixel (pixel) representing an image has a display position on the screen of the display unit 14 (hereinafter referred to as “standard display position”). The standard display position of each pixel may be changed by the operator operating the operation unit 13 or the like as long as the application permits. For example, when the operation unit 13 is operated by the operator, an image expressed in the display data 123 may scroll or slide on the screen of the display unit 14.

  Further, the display control unit 102 refers to the change data 122, so that the image (display data 123) displayed on the display unit 14 is suppressed so as to suppress a change in the relative position detected by the detection unit 101. The display position on the display unit 14 is controlled. That is, the display control unit 102 in the present embodiment corresponds to the display correction unit in the present invention. In the following description, the display position of the display data 123 determined by the display control unit 102 based on the change data 122 is referred to as “corrected display position” and is distinguished from the “standard display position”.

  The above is description of the structure and function of the portable terminal device 1 in 1st Embodiment. Next, a camera shake correction method for correcting camera shake due to the walking motion of the operator in the mobile terminal device 1 will be described.

  FIG. 4 is a flowchart showing the operation of the display correction mode of the mobile terminal device 1 according to the first embodiment. The display correction mode is an operation mode in which the mobile terminal device 1 corrects and displays the standard display position of the display data 123 as the correction display position in accordance with the walking motion of the operator. In addition, it is assumed that the start of the display correction mode is instructed in the mobile terminal device 1 before each process illustrated in FIG. 4 is started. In addition, the display correction mode start instruction is instructed by, for example, a direct start instruction by an operator or activation of an application using the operation mode. Further, since the display correction mode is an operation mode for controlling the display position of the display data 123 on the display unit 14, it is naturally only active when the display unit 14 is in an active state (a state in which some display data 123 is displayed). In FIG. 4, the process of displaying the display data 123 at the standard display position is omitted.

  When the display correction mode is started, the mobile terminal device 1 determines whether the measurement timing has arrived (step S1), whether the imaging timing has arrived (step S3), and whether the measurement data 120 has been newly created. (Step S5) is monitored. In the following description, this state is referred to as a “monitoring state”.

  When the measurement timing comes in the monitoring state (Yes in step S1), the sensor group 17 performs measurement (step S2) and creates measurement data 120. The measurement timing is determined by a preset measurement cycle. That is, in the display correction mode in the mobile terminal device 1, the measurement process (step S <b> 2) by the sensor group 17 occurs every measurement cycle, and the measurement data 120 is created. Note that the detection devices included in the sensor group 17 do not necessarily have to measure simultaneously.

  When the imaging timing comes in the monitoring state (Yes in step S3), the imaging unit 15 performs imaging (step S4), and the image signal processing unit 16 creates the image data 160. The imaging timing is determined by a preset imaging cycle, similarly to the measurement timing. That is, in the display correction mode in the mobile terminal device 1, imaging processing (step S <b> 4) is generated by the imaging unit 15 and the image signal processing unit 16 for each imaging cycle, and image data 160 is created.

  In the monitoring state, when new measurement data 120 is created (Yes in step S5), the operator creates the data based on the newly created measurement data 120 (including past measurement data 120 as necessary). The determination unit 100 determines whether or not walking (step S6). In step S6, the determination unit 100 creates determination data 121 indicating the determination result.

  If it is determined that the operator carrying the mobile terminal device 1 is walking (Yes in step S6), the detection unit 101 displays the display data displayed on the display unit 14 based on the image data 160. A change in the relative position between the operator 123 and the operator's face is detected (step S7). That is, the detection unit 101 performs step S7 when the determination data 121 indicates that “the operator is walking” while referring to the determination data 121.

  Here, the principle of detecting a relative position change between the display data 123 displayed on the display unit 14 and the operator's face will be described.

  In the mobile terminal device 1, the standard display position of the display data 123 displayed on the display unit 14 is determined with respect to the display unit 14. Therefore, the relative positional relationship between the standard display position of the display data 123 and the display unit 14 is fixed. Since the display unit 14 is fixed in the mobile terminal device 1 while the operator is gazing at the display unit 14, the relative positional relationship between the mobile terminal device 1 and the display unit 14 is fixed. Yes. Furthermore, since the imaging range of the imaging unit 15 is fixed in the mobile terminal device 1 while the operator is gazing at the display unit 14, the relative range between the mobile terminal device 1 and the imaging range of the imaging unit 15 is fixed. The positional relationship is also fixed.

  Therefore, in order to detect a relative position change between the display data 123 displayed on the display unit 14 and the operator's face after all, the relative range between the imaging range of the imaging unit 15 and the operator's face is determined. It is only necessary to detect a change in position. That is, if a change in the position of the operator's face as the subject is detected in the image data 160 obtained by imaging the subject within the imaging range of the imaging unit 15, the display data 123 displayed on the display unit 14 and the operator's face are displayed. A change in position relative to the face can be detected.

The detection unit 101 extracts pixels each representing the face of the operator from two image data 160 continuous in the imaging cycle, and obtains a change in the position of the pixel in the two image data 160. If an X axis and a Y axis that are orthogonal to each other are defined in the image data 160, the change in the position is obtained as _a change “Δx _a ” in the X axis direction and a change “Δy _a ” in the Y axis direction. The unit of “Δx _a ” and “Δy _a ” at this time is “pixel”. That is, the number of pixels moved in the image data 160 is obtained.

Next, the detection unit 101 obtains actual movement distances (Δx _L and Δy _L ) of the mobile terminal device 1 due to camera shake during walking based on “Δx _a ” and “Δy _a ”. This is obtained by Equation 1 and Equation 2, respectively, using the conversion magnification α ₁ [m / pixel].

Δx _L = −α ₁ × Δx _a.

Δy _L = −α ₁ × Δy _a ...

However, the conversion magnification α ₁ is a value indicating how much the movement distance due to the hand movement of the mobile terminal device 1 is reflected when the pixel representing the face of the operator moves by one pixel in the image data 160. Thus, it varies depending on the imaging magnification of the imaging unit 15 and the distance L between the subject (the operator's face) and the imaging unit 15. Here, the imaging magnification of the imaging unit 15 is known, but strictly speaking, the distance L is an unknown number. In the portable terminal device 1 of the present embodiment, the distance L is (can be expected to example 20 centimeters approximately.) Constant determined according to the length of a human arm a conversion ratio alpha ₁ was determined as, previously stored ing. However, the distance L may be measured each time using a distance measuring sensor, a 3D camera, or the like to obtain the conversion magnification α ₁ .

The detection unit 101 creates the change data 122 based on the movement distances (Δx _L and Δy _L ) of the mobile terminal device 1 due to camera shake during walking, which are obtained by Equation 1 and Equation 2. That is, in step S <b> 7, change data 122 indicating (Δx _L , Δy _L ) is created by the detection unit 101.

When the change data 122 is created, the display control unit 102 displays the display data 123 displayed on the display unit 14 in the display unit 14 so as to suppress the relative position change detected by the detection unit 101. The display position is controlled (step S8). More specifically, the display control unit 102 displays the (n-1) th corrected display position (X _(n-1) , Y _(n-1) ) that is the current display position, and the newly created change. Based on the data 122 and the conversion magnification β, the nth correction display position (X _n , Y _n ) is determined by Equation 3 and Equation 4.

_{X n = X (n-1} ) -β × Δx L = X (n-1) + α 1 × β × Δx a ··· Formula 3

_{Y n = Y (n-1} ) -β × Δy L = Y (n-1) + α 1 × β × Δy a ··· Equation 4

However, n is a natural number of 1 or more, and the 0th corrected display position (X ₀ , Y ₀ ) is a standard display position. Further, the conversion magnification β is a known constant determined according to how much distance is moved in the real space when the display position is moved by one pixel in the display unit 14, and the unit is [pixel / m]. .

In step S8, the display control unit 102 controls the display unit 14 to display the display data 123 at the corrected display position (X _n , Y _n ) thus obtained. As a result, the movement distance due to camera shake of the display data 123 is offset by the change of the display position, and the shaking of the screen is suppressed.

  As described above, the mobile terminal device 1 according to the first embodiment includes the display unit 14 that displays the display data 123, the imaging unit 15 that acquires the image data 160 including the face of the operator, and the image signal processing unit 16. And a detection unit 101 that detects a change in the relative position between the display data 123 displayed on the display unit 14 and the face of the operator based on the acquired image data 160 and creates the change data 122; The determination unit 100 that determines whether or not the operator is walking, and the relative position change detected by the detection unit 101 when the determination unit 100 determines that the operator is walking. Display control unit 102 for controlling the display position of the display data 123 displayed on the display unit 14 in the display unit 14 so as to suppress the camera shake during walking. Data 123 can be displayed.

  Further, for example, since the amount of change due to camera shake is an amount that can be predicted to some extent, if at least the direction of camera shake can be detected, the display data 123 may be moved and displayed by the amount of prediction in the opposite direction to that direction. Image shaking due to camera shake can be suppressed. However, the detection unit 101 according to the present embodiment can accurately correct camera shake by detecting the amount of change in relative position change.

<2. Second Embodiment>
In the first embodiment, an example has been described in which camera shake due to walking of the operator is detected from image data 160 obtained by capturing the face of the operator based on the movement of pixels representing the face of the operator. However, of the operator's face, the eyeball has a characteristic of following the (relatively) moving object in a reflective manner. Therefore, the movement of the eyeball in the operator's face is particularly effective as a target for detecting the movement of the image (display data 123) that is a moving object.

  FIG. 5 is a diagram illustrating functional blocks included in the mobile terminal device 1a according to the second embodiment together with a data flow. The detection unit 101a illustrated in FIG. 5 is a functional block realized by the CPU 10 operating according to the program 110.

  The mobile terminal device 1a in the second embodiment is different from the mobile terminal device 1 in the first embodiment in that a detection unit 101a is provided instead of the detection unit 101. Below, about the portable terminal device 1a in 2nd Embodiment, the same code | symbol is attached | subjected about the structure similar to the portable terminal device 1 in 1st Embodiment, and description is abbreviate | omitted suitably.

The detection unit 101a according to the second embodiment is based on the movement distance (pupil movement distance) “Δx _b ” and “Δy _b ” of the eyeball of the operator in the image data 160, and the mobile terminal device due to hand movements during walking The actual travel distance (Δx _L and Δy _L ) of 1 is obtained. This is obtained by Expression 5 and Expression 6, respectively, using the conversion magnification α ₂ [m / pixel].

Δx _L = α ₂ × Δx _b ... Formula 5

Δy _L = α ₂ × Δy _b.

However, the conversion magnification α ₂ is a value indicating how much the movement distance due to camera shake of the mobile terminal device 1a is reflected when a pixel representing the eyeball of the operator moves by one pixel in the image data 160. Like the conversion magnification α ₁ in the first embodiment, it is an unknown number. However, also in the present embodiment, as in the first embodiment, the conversion magnification α ₂ is stored in advance as a constant.

The detection unit 101a creates the change data 122a based on the movement distances (Δx _L and Δy _L ) of the mobile terminal device 1a due to camera shake during walking, which are obtained by Equations 5 and 6.

When the change data 122a is created, the display control unit 102 in the display unit 14 of the display data 123 displayed on the display unit 14 so as to suppress a change in relative position detected by the detection unit 101a. Control the display position. More specifically, the display control unit 102 displays the (n-1) th corrected display position (X _(n-1) , Y _(n-1) ) that is the current display position, and the newly created change. Based on the data 122a and the conversion magnification β, the nth correction display position (X _n , Y _n ) is determined by Expression 7 and Expression 8.

_{X n = X (n-1} ) -β × Δx L = X (n-1) -α 2 × β × Δx b ··· Equation 7

_{Y n = Y (n-1} ) -β × Δy L = Y (n-1) -α 2 × β × Δy b ··· Equation 8

  As described above, the mobile terminal device 1a according to the second embodiment can also display the display data 123 in which camera shake during walking is suppressed, similarly to the mobile terminal device 1 according to the first embodiment.

  The imaging unit 15 and the image signal processing unit 16 acquire image data 160 including the eyeball of the operator, and the detection unit 101a detects the movement of the eyeball of the operator based on the acquired image data 160. Thus, a change in the relative position between the display data 123 displayed on the display unit 14 and the face of the operator is detected. Thereby, camera shake can be corrected accurately.

<3. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

  For example, each process shown in the above embodiment is merely an example, and the order and contents may be appropriately changed as long as the same effect can be obtained. For example, steps S6 to S8 may be executed subsequent to step S2 shown in FIG.

  Moreover, in the display correction mode in the above embodiment, when the imaging unit 15 performs imaging and the image signal processing unit 16 creates the image data 160 regardless of whether the operator is walking or not. explained. However, the creation of the image data 160 may be performed only during a period in which the determination unit 100 determines that the operator is walking.

  Further, depending on the application or situation, when the operator who is walking stops walking motion (that is, when it is determined No again in step S6), the display control unit 102 performs correction display. The position may be returned to the standard display position.

  Moreover, although the functional block shown in the said embodiment demonstrated that it was implement | achieved by software because CPU10 operate | moves according to the program 110, a part or all of these functional blocks is by a dedicated electronic circuit. It may be realized in hardware.

  Further, in the above-described embodiment, it has been described that by performing feature extraction on the image data 160, a part expressing the operator's face or eyeball is specified. However, such processing can be realized by a method other than feature extraction for the image data 160. For example, it is possible to specify using a neural network.

  In the above embodiment, it has been described that it is determined whether or not the operator is walking based on the output from the sensor group 17. However, for example, the walking motion of the operator may be detected by analyzing the image data 160 and detecting the movement of a specific subject during walking. That is, whether the operator is walking or not may be detected in any way.

  Further, although the display control unit 102 has been described as controlling the display position depending on whether or not camera shake occurs, the display data 123 may be enlarged and displayed together with the display position control. Thereby, not only shaking of the screen is suppressed, but the display data 123 is displayed in a large size, so that the display data 123 becomes easier to see. Or you may further determine whether it expands according to the grade (size of the change data 122) of camera shake.

1,1a Mobile terminal device 10 CPU
100 Determination Unit 101, 101a Detection Unit 102 Display Control Unit (Display Correction Unit)
11 ROM
110 program 12 RAM
120 Measurement data 121 Judgment data 122, 122a Change data 123 Display data (image)
DESCRIPTION OF SYMBOLS 13 Operation part 14 Display part 15 Imaging part 16 Image signal processing part 160 Image data 17 Sensor group

Claims (6)

A portable terminal device operated by an operator,
Display means for displaying an image;
Image acquisition means for acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including the face of the operator;
A first pixel group expressing the operator's face is extracted from the first image data acquired by the image acquisition means, and the operator's face is expressed from the second image data acquired by the image acquisition means A pixel representing the operator's face based on the extracted second pixel group and the position of the first pixel group in the first image data and the position of the second pixel group in the second image data The amount of change in the relative position change between the image displayed on the display means and the operator's face is calculated based on the number of moved pixels and the first conversion magnification. Detecting means for detecting;
Determining means for determining whether or not the operator is walking;
When the determination means determines that the operator is walking, the display on the display means displays the image displayed on the display means so as to suppress the amount of change detected by the detection means. Display correction means for controlling the position;
A mobile terminal device comprising: A portable terminal device operated by an operator ,
Display means for displaying an image;
Image acquisition means for acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including the face of the operator;
A first pixel group expressing the operator's face is extracted from the first image data acquired by the image acquisition means, and the operator's face is expressed from the second image data acquired by the image acquisition means A pixel representing the operator's face based on the extracted second pixel group and the position of the first pixel group in the first image data and the position of the second pixel group in the second image data Detecting means for detecting a change amount in a change in a relative position between the image displayed on the display means and the face of the operator, based on the number of moved pixels;
Determining means for determining whether or not the operator is walking;
The display means according to the change amount and the second conversion magnification so as to suppress the change amount detected by the detection means when the determination means determines that the operator is walking. Display correction means for controlling the display position of the image displayed on the display means;
A mobile terminal device comprising: A computer-readable program, wherein execution of the program by the computer causes the computer to
Display means for displaying an image;
Image acquisition means for acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including an operator's face;
A first pixel group expressing the operator's face is extracted from the first image data acquired by the image acquisition means, and the operator's face is expressed from the second image data acquired by the image acquisition means A pixel representing the operator's face based on the extracted second pixel group and the position of the first pixel group in the first image data and the position of the second pixel group in the second image data The amount of change in the relative position change between the image displayed on the display means and the operator's face is calculated based on the number of moved pixels and the first conversion magnification. Detecting means for detecting;
Determining means for determining whether or not the operator is walking;
When the determination means determines that the operator is walking, the display on the display means displays the image displayed on the display means so as to suppress the amount of change detected by the detection means. Display correction means for controlling the position;
A program that causes a mobile terminal device to function . A computer-readable program, wherein execution of the program by the computer causes the computer to
Display means for displaying an image;
Image acquisition means for acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including an operator's face;
A first pixel group expressing the operator's face is extracted from the first image data acquired by the image acquisition means, and the operator's face is expressed from the second image data acquired by the image acquisition means A pixel representing the operator's face based on the extracted second pixel group and the position of the first pixel group in the first image data and the position of the second pixel group in the second image data Detecting means for detecting a change amount in a change in a relative position between the image displayed on the display means and the face of the operator, based on the number of moved pixels;
Determining means for determining whether or not the operator is walking;
The display means according to the change amount and the second conversion magnification so as to suppress the change amount detected by the detection means when the determination means determines that the operator is walking. Display correction means for controlling the display position of the image displayed on the display means;
A program that causes a mobile terminal device to function . A camera shake correction method for a mobile terminal device,
Displaying an image on the display means of the portable terminal device;
Acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including a face of an operator of the mobile terminal device;
Extracting a first pixel group expressing the operator's face from the acquired first image data and extracting a second pixel group expressing the operator's face from the acquired second image data; Based on the position of the first pixel group in the first image data and the position of the second pixel group in the second image data, the number of moved pixels representing the operator's face is obtained, and the movement Detecting a change amount in a relative position change between the image displayed on the display unit and the face of the operator based on the number of pixels and the first conversion magnification;
Determining whether the operator is walking;
Controlling the display position on the display means of the image displayed on the display means so as to suppress the detected amount of change when it is determined that the operator is walking;
A method for correcting camera shake . A camera shake correction method for a mobile terminal device,
Displaying an image on the display means of the portable terminal device;
Acquiring first image data and second image data, each of which is continuous in an imaging cycle, each including a face of an operator of the mobile terminal device;
Extracting a first pixel group expressing the operator's face from the acquired first image data and extracting a second pixel group expressing the operator's face from the acquired second image data; Based on the position of the first pixel group in the first image data and the position of the second pixel group in the second image data, the number of moved pixels representing the operator's face is obtained, and the movement Detecting a change amount in a relative position change between the image displayed on the display unit and the face of the operator based on the number of pixels obtained;
Determining whether the operator is walking;
When it is determined that the operator is walking, the image of the image displayed on the display unit according to the change amount and the second conversion magnification so as to suppress the detected change amount. Controlling the display position on the display means;
A method for correcting camera shake.

Images