JP6973524B2 - program - Google Patents

Description

The present invention relates to a program of a mobile terminal that performs keystone correction of an image generated by a camera.

Cited Document 1 describes a program of a mobile terminal that performs keystone correction of a captured image generated by a camera. More specifically, the program described in Cited Document 1 displays a captured image generated by a camera that captures the subject on a display, and causes the user to specify the position of the apex of a substantially quadrangular region surrounding the subject in the captured image. , Trims the image in the specified approximately quadrilateral area and corrects the trimmed image to keystone.

Japanese Unexamined Patent Publication No. 2013-218547

However, when the subject is zoomed and photographed by the camera, a part of the subject may not be included in the captured image. Then, if the user is to specify the position of the apex of the substantially quadrangular region in such a captured image, the position inside the substantially quadrangular region may be designated as the apex from the original position. As a result, when the image in such a substantially quadrangular region is trimmed and keystone-corrected, a part of the subject included in the captured image may be further missing.

The present invention has been made in view of the above circumstances, and an object thereof is a program of a mobile terminal capable of appropriately designating a region to be keystone-corrected in a photographed image generated by a camera. Is to provide.

The programs described herein are readable by a mobile terminal comprising a camera, a display, and an input interface having a touch sensor. The program expands the captured image by acquiring the captured image data generated by the camera that captured the object and adding additional pixels to the outer edge of the captured image indicated by the captured image data. The shape of the trapezoidal image is displayed in the extended process for generating the extended image data indicating the expanded image, the first display process for displaying the expanded image on the display, and the expanded image displayed in the first display process. The mobile terminal is made to execute the first reception process of receiving the designated touch operation through the input interface. The trapezoidal image is an image showing the object represented in a trapezoid in the extended image by taking a rectangular object at an angle. The program performs correction processing for generating corrected image data indicating a rectangular corrected image by performing trapezoidal correction of the trapezoidal image having a shape specified in the first reception processing, and an output instruction for outputting the corrected image data. The process is executed by the mobile terminal.

According to the above configuration, the user can specify the trapezoidal image to be the keystone correction in the extended image in which additional pixels are added to the outer edge of the captured image. That is, even when the captured image does not include a part of the subject, the user can appropriately specify the area to be the keystone correction.

FIG. 1 is a schematic diagram of a system 100 according to the present embodiment. FIG. 2 is a flowchart of print instruction processing. FIG. 3 is a flowchart of the trapezoidal expansion process. 4A and 4B are examples of captured images, in which FIG. 4A shows an example in which four straight lines are detected, and FIG. 4B shows an example in which three straight lines are detected. 5A and 5B are examples of extended images, in which FIG. 5A shows an example in which an additional pixel 85 is added to a captured image 80, and FIG. 5B shows an example in which an additional pixel 95 is added to a captured image 90. 6A and 6B are display examples of the display 53, in which FIG. 6A shows a trapezoidal designation screen and FIG. 6B shows a preview screen.

Hereinafter, embodiments of the present invention will be described with reference to the drawings as appropriate. It is needless to say that the embodiments described below are merely examples of the present invention, and the embodiments of the present invention can be appropriately changed without changing the gist of the present invention. For example, the execution order of each process described later can be appropriately changed without changing the gist of the present invention.

FIG. 1 is a schematic diagram of a system 100 according to the present embodiment. The system 100 shown in FIG. 1 includes a multifunction device 10 and a mobile terminal 50. Further, the system 100 may include a printer single-purpose machine instead of the multifunction device 10. The multifunction device 10 and the mobile terminal 50 are configured to be able to communicate with each other through the communication network 101. The specific example of the communication network 101 is not particularly limited, but may be, for example, the Internet, a wired LAN, a wireless LAN, or a combination thereof.

As shown in FIG. 1, the multifunction device 10 mainly includes a printer 11, a scanner 12, a display 23, an input I / F 24, a communication I / F 25, a CPU 31, a memory 32, and a communication bus 33. Prepare for. Each component constituting the multifunction device 10 is connected to each other through a communication bus 33.

The printer 11 executes a printing operation of recording an image represented by image data on a sheet. As the recording method of the printer 11, a known method such as an inkjet method or an electrophotographic method can be adopted. The scanner 12 performs a scanning operation of reading an image recorded on a document and generating image data.

The display 23 is a liquid crystal display, an organic EL display, or the like, and includes a display surface for displaying various information.

The input I / F 24 is a user interface that accepts an input operation by the user. Specifically, the input I / F 24 has a button, and outputs various operation signals associated with the pressed button to the CPU 31. Further, the input I / F 24 may have a film-like touch sensor superimposed on the display surface of the display 23. The operation of designating an object displayed on the display surface of the display 23 and the operation of inputting a character string or a number string are examples of user operations. The "object" is, for example, a character string, an icon, a button, a link, or the like displayed on the display 23.

The input I / F 24 realized as a touch sensor outputs position information indicating a position on the display surface touched by the user. The term "touch" in the present specification includes all operations of bringing the input medium into contact with the display surface. Further, "hover" or "floating touch" that brings the input medium close to a position where the distance between the input medium and the display surface is very small even if the input medium does not touch the display surface is used as the above-mentioned concept of "touch". May be included. Further, the input medium may be a user's finger, a touch pen, or the like. The user operation of tapping the position of the icon displayed on the display 23 is an example of the designated operation for designating the icon.

The communication I / F 25 is an interface capable of communicating with an external device through the communication network 101. That is, the multifunction device 10 outputs various information to the external device through the communication I / F25, and receives various information from the external device through the communication I / F25. The specific communication procedure of the communication I / F25 is not particularly limited, but for example, Wi-Fi (registered trademark of Wi-Fi Alliance) can be adopted.

The CPU 31 controls the overall operation of the multifunction device 10. The CPU 31 acquires and executes various programs described later from the memory 32 based on various information output from the input I / F 24, various information acquired from the external device through the communication I / F 25, and the like. The CPU 31 and the memory 32 form an example of the control unit.

The memory 32 has a program storage area 32A and a data storage area 32B. The OS 34 and the device program 35 are stored in the area 32A. The device program 35 may be a single program or a collection of a plurality of programs. The area 32B stores data or information necessary for executing the device program 35. The memory 32 is composed of, for example, a RAM, a ROM, an EEPROM, an HDD, a portable storage medium such as a USB memory attached to and detached from the multifunction device 10, a buffer included in the CPU 31, or a combination thereof.

The memory 32 may be a computer-readable storage medium. A computer-readable storage medium is a non-transitional medium. In addition to the above examples, the non-transitory medium also includes recording media such as a CD-ROM and a DVD-ROM. The non-transitory medium is also a tangible medium. On the other hand, an electric signal that carries a program downloaded from a server on the Internet is a signal medium that can be read by a computer, which is a kind of medium that can be read by a computer, but is a storage that can be read by a non-transitory computer. Not included in the medium. The same applies to the memory 62 of the mobile terminal 50 described later.

As shown in FIG. 1, the mobile terminal 50 mainly includes a camera 52, a display 53, an input I / F 54, a communication I / F 55, a CPU 61, a memory 62, and a communication bus 63. The display 53, the input I / F 54, the communication I / F 55, the CPU 61, the memory 62, and the communication bus 63 included in the mobile terminal 50 are the display 23, the input I / F 24, the communication I / F 25, and the CPU 31 included in the multifunction device 10. , The memory 32, and the communication bus 33 have the same configuration, and thus the description thereof will be omitted. The CPU 61 and the memory 62 are examples of the control unit.

The mobile terminal 50 is, for example, a mobile phone, a smartphone, a tablet terminal, or the like. More specifically, the display 53 of the mobile terminal 50 preferably has a display screen size of 12 inches or less, more preferably 8 inches or less. Further, it is desirable that the input I / F 54 of the mobile terminal 50 is a touch sensor superimposed on the display screen of the display 53. The OS 64 and the terminal program 65 are stored in the area 62A of the memory 62.

The camera 52 captures a subject and generates image data (hereinafter, may be referred to as “captured image data”). The operation of the camera 52 is controlled by, for example, a camera program (not shown) stored in the area 62A. That is, the camera program displays the image indicated by the captured image data and the [photographed] icon on the display 53. Further, the image displayed on the display 53 is changed in accordance with the change in the orientation of the camera 52. Then, the camera program stores the photographed image data in the area 62B in response to the acceptance of the user operation for designating the [shooting] icon through the input I / F 54.

[Operation of system 100]
The operation of the system 100 according to the present embodiment will be described with reference to FIGS. 2 and 3.

The flowchart of the present specification basically shows the processing of the CPUs 31 and 61 according to the instructions described in the program. That is, the processes such as "determination", "extraction", "selection", and "control" in the following description represent the processes of the CPUs 31 and 61. The processing by the CPU 61 also includes hardware control via the OS 64. Further, the "data" in the present specification is represented by a computer-readable bit string. Data having the same substantial meaning and different formats are treated as the same data. The same applies to "information" in the present specification.

First, the terminal program 65 of the mobile terminal 50 activates a camera program (not shown) (S11). Next, the camera program stores the photographed image data in the area 62B in response to the reception of the user operation for designating the [shooting] icon through the input I / F 54, and stores the stored photographed image data in the terminal program 65. offer. The terminal program 65 may execute, for example, an API for starting the camera program as a so-called plug-in. Further, the camera program may include the path of the captured image data stored in the area 62B in the return value of the API. Then, the terminal program 65 may read the captured image data based on the path included in the return value (S12: Yes).

In the present embodiment, the user of the mobile terminal 50 assumes that the rectangular whiteboard is photographed from an angle by the camera 52. As a result, in S12, for example, the terminal program 65 has acquired the photographed image data showing the photographed image 80 shown in FIG. 4 (A) or the photographed image data showing the photographed image 90 shown in FIG. 4 (B). And. The whiteboard is an example of an object. However, the specific example of the object is not limited to the whiteboard, and may be a generally rectangular or square object such as a blackboard, a poster, or a display, and it is desirable that characters or patterns are drawn on the surface.

As shown in FIG. 4A, the captured image 80 is a rectangular image surrounded by contour lines 81A, 81B, 81C, and 81D. As shown in FIG. 4B, the captured image 90 is a rectangular image surrounded by contour lines 91A, 91B, 91C, and 91D. Further, the captured images 80 and 90 include at least a part of the trapezoidal images 84 and 94. The trapezoidal images 84 and 94 are images showing the whiteboard represented in a trapezoidal shape in the photographed images 80 and 90 because the whiteboard is photographed from an angle.

Next, the terminal program 65 executes the image expansion process (S13). The image expansion process is a process of generating an expanded image by adding pixels to the captured image indicated by the captured image data. That is, the extended image is an expanded image of the captured image. The details of the image expansion process will be described with reference to FIG.

First, the terminal program 65 detects a straight line having a predetermined length or longer included in the captured image by using a well-known algorithm such as Hough transform (S31). Further, the terminal program 65 determines the number of straight lines detected in S31 (S32). The process of S31 is an example of the detection process, and the process of S32 is an example of the first determination process.

As an example, the terminal program 65 detects four straight lines 82A, 82B, 82C, 82D in the captured image 80 shown in FIG. 4 (A). As another example, the terminal program 65 detects three straight lines 92A, 92B, 92C in the captured image 90 shown in FIG. 4 (B). However, the number of straight lines detected in S31 is not limited to the example of FIG. 4, and may be two or less.

Next, as shown in FIG. 4B, for example, the terminal program 65 determines that the three straight lines 92A to 92C have been detected (S32: 3), and the four contour lines of the captured image 90. One of 91A to 91D is determined as the fourth straight line (S33). The process of S33 is an example of the determination process. On the other hand, as shown in FIG. 4A, the terminal program 65 skips the process of S33 in response to the determination that the four straight lines 82A to 82D have been detected (S32: 4).

For example, in S33, the terminal program 65 determines that of the three straight lines 92A to 92C detected in S31, the two straight lines 92A and 92C that are close to parallel are paired straight lines. The paired straight lines 92A and 92C are straight lines that can form the opposite sides of the trapezoidal image 94. Then, the terminal program 65 determines the contour line 91D farthest from the straight line 92B in which the paired straight line does not exist among the four contour lines 91A to 91D of the captured image 90 as the fourth straight line.

Next, the terminal program 65 specifies the positions of the intersections of the four straight lines detected in S31, or the positions of the intersections of the three straight lines detected in S31 and one straight line determined in S33. Then, the terminal program 65 estimates the position of the specified intersection as the position of the apex of the trapezoidal images 84 and 94 (S34). The process of S34 is an example of the estimation process. As an example, the terminal program 65 estimates the positions of the intersections of the straight lines in the captured image 80 shown in FIG. 4A to be the vertices 83A, 83B, 83C, and 83D of the trapezoidal image 84. As another example, the terminal program 65 estimates the position of the intersection of the straight lines in the captured image 90 shown in FIG. 4B to be the vertices 93A, 93B, 93C, 93D of the trapezoidal image 94. As shown in FIG. 4, a part of the vertices estimated in S34 may be located outside the captured images 80 and 90.

Next, the terminal program 65 determines whether or not the positions of the four vertices estimated in S34 are within the captured image (S35). Then, the terminal program 65 generates extended image data indicating an extended image according to the determination that the position of at least one vertex is outside the captured image (S35: No) (S36). On the other hand, the terminal program 65 skips the process of S36 according to the determination that the positions of all the vertices are in the captured image (S35: Yes). The process of S35 is an example of the second determination process, and the process of S36 is an example of the extended process.

The terminal program 65 generates extended image data in S36 by adding additional pixels to the outer edge of the captured image. As an example, the terminal program 65 generates an extended image 86 in which the additional pixel 85 shown in FIG. 5A is added along the contour line 81C to the outer edge of the captured image 80 shown in FIG. 4A. As another example, the terminal program 65 adds the L-shaped additional pixel 95 shown in FIG. 5A to the outer edge of the captured image 90 shown in FIG. 4B along the contour lines 91C and 91D. The expanded image 96 is generated. The extended images 86, 96 include the entire trapezoidal images 84, 94.

That is, it is desirable that the additional pixels 85 and 95 are provided along the contour lines 81C and 91C closest to the vertices 83C and 93C located outside the captured images 80 and 90. Further, it is desirable that the additional pixel 95 is provided along the contour line 91D determined to be a straight line in S33. However, the shapes of the additional pixels 85 and 95 are not limited to the example of FIG. As an example, the additional pixels 85 and 95 may not be provided in the entire area of the contour lines 81C, 91C and 91D. That is, the extended images 86 and 96 do not have to be rectangular. As another example, the terminal program 65 may generate an extended image 88 to which a frame-shaped additional pixel 87 surrounding the captured image 80 is added, as shown in FIG. 6 (A).

Further, the widths of the additional pixels 85 and 95 are appropriately set depending on, for example, the positions of the vertices 83C and 93C located outside the captured images 80 and 90 and the number of straight lines detected in S31. First, the terminal program 65 sets the widths of the additional pixels 85 and 95 so that the vertices 83C and 93C located outside the captured images 80 and 90 are located inside the contour lines of the expanded images 86 and 96. good. Further, in the terminal program 65, the width of the additional pixels 85 and 95 may be increased as the number of straight lines detected in S31 is smaller.

On the other hand, in response to the determination that S31 has detected two or less straight lines (S32: 2 or less), the terminal program 65 replaces S33 to S36, for example, as shown in FIG. 6A. By adding a frame-shaped additional pixel 87 that surrounds the captured image 80, extended image data indicating the extended image 88 is generated (S37). The process of S37 is an example of the extended process. In the present embodiment, the following processing will be described assuming that the frame-shaped additional pixel 87 is added to the captured image 80 in S36. Further, when the processing of S36 is skipped (S35: Yes), the captured image 80 is used instead of the extended image 88 in the following processing.

Returning to FIG. 2, the terminal program 65 causes the display 53 to display the trapezoidal designation screen shown in FIG. 6 (A) (S14). The trapezoidal designation screen includes an extended image 88, guide images 111, 112, 113, 114, and a [correction] icon 115. The guide images 111 to 114 are images showing the positions of the vertices 83A to 83D estimated in S34, and are displayed superimposed on the extended image 88. The [correction] icon 115 corresponds to an instruction to correct the trapezoidal image. Then, the terminal program 65 accepts the user operation for the trapezoidal designation screen through the input I / F 54 (S15). The process of S14 is an example of the first display process, and the process of S15 is an example of the first reception process.

In S14, the terminal program 65 may display the extended image 88 in which the non-background pixel value is set in the additional pixel 87 on the trapezoidal designation screen. The terminal program 65 copies, for example, the extended image data generated in S36 and S37, sets a non-background pixel value in the additional pixel 87 included in the copied extended image data, and sets the non-background pixel value to the extended image 88 indicated by the extended image data. Should be displayed on the trapezoidal designation screen. The non-background pixel value is a pixel value different from the pixel value of the background pixel of the captured image. The pixel value of the background pixel refers to, for example, the mode of the histogram of the pixel value included in the captured image. Further, the terminal program 65 may display the guide images 111 to 114 at predetermined positions on the extended image 88 when two or less straight lines are detected in S31 (S32: 2 or less).

Next, the terminal program 65 changes the position of the guide image 113 on the trapezoidal designation screen in response to receiving, for example, a drag operation for the position of the guide image 113 through the input I / F 54 (S15: guide image) (S15: guide image). S16 & S14). More specifically, the terminal program 65 changes the position of the guide image 113 following the input medium in response to the movement of the input medium touched to the position of the guide image 113 on the display surface. Then, the terminal program 65 determines the position where the input medium is separated, that is, the end position of the drag operation, at a new position of the guide image 113.

The guide image 113 can be moved to an arbitrary position in the extended image 88 of the trapezoidal designation screen, that is, to an arbitrary position in the captured image 80 and the additional pixel 87. On the other hand, the guide image 113 cannot be moved outside the extended image 88 of the trapezoidal designation screen. The same applies to the drag operation for the guide images 111, 112, 114. The drag operation is an example of a touch operation for changing the position of the guide images 111 to 114 and a touch operation for designating the shape of the trapezoidal image 84. The touch operation for designating the shape of the trapezoidal image is not limited to the drag operation for the guide images 111 to 114. For example, the terminal program 65 may change the position and angle of the side according to the acceptance of the drag operation with respect to the position of the side constituting the trapezoidal image 84 through the input I / F 54.

Next, the terminal program 65 receives the designation of the [correction] icon 115 through the input I / F 54 (S15: correction icon), and receives the trapezoidal image 84 having the positions of the guide images 111 to 114 as vertices. Extracted from the extended image 88. Then, the terminal program 65 performs keystone correction on the extracted trapezoidal image 84 to generate corrected image data indicating a rectangular corrected image (S17). The process of S17 is an example of the correction process.

When at least one of the guide images 111 to 114 is arranged at the position of the additional pixel 87, the terminal program 65 extracts the trapezoidal image 84 including the pixel of the captured image 80 and the additional pixel 87. Further, the terminal program 65 generates corrected image data by interpolating pixels so that the upper side and the short side are parallel to the trapezoidal image 84 whose apex is the position of the guide images 111 to 114, for example. .. Keystone correction is realized by a well-known algorithm such as projective transformation.

Next, the terminal program 65 causes the preview screen shown in FIG. 6B to be displayed on the display 53 (S18). The preview screen includes the corrected image 121 generated in S17, the [print] icon 122, and the [save] icon 123. The [print] icon 122 corresponds to an instruction to cause the multifunction device 10 to perform a print operation on the corrected image data. The [Save] icon 123 corresponds to an instruction to store the corrected image data in the area 62B. Then, the terminal program 65 accepts the user operation for the preview screen through the input I / F 54 (S19). The process of S18 is an example of the second display process, and the process of S19 is an example of the second reception process.

Then, the terminal program 65 transmits the print instruction information to the multifunction device 10 through the communication I / F 55 in response to the reception of the designation of the [print] icon 122 through the input I / F 54 (S19: print icon) (S20: print icon). ). The print instruction information is information for instructing a print operation with respect to the corrected image data, and includes the corrected image data. On the other hand, although not shown, the device program 35 of the multifunction device 10 receives print instruction information from the mobile terminal 50 through the communication I / F 25. Then, the device program 35 causes the printer 11 to execute a print operation according to the received print instruction information.

On the other hand, the terminal program 65 stores the corrected image data in the area 62B in response to the reception of the designation of the [save] icon 123 through the input I / F 54 (S19: save icon) (S21). When the corrected image data output in S20 and S21 includes additional pixels, the terminal program 65 may set the background pixel value of the captured image in the additional pixels. The designation of the [print] icon 122 or the [save] icon 123 is an example of the instruction operation. The processing of S20 and S21 is an example of output instruction processing.

[Action and effect of the embodiment]
According to the above embodiment, the user can specify the trapezoidal image to be the keystone correction in the extended image in which additional pixels are added to the outer edge of the captured image. That is, even when the captured image does not include a part of the subject, the user can appropriately specify the area to be the keystone correction.

By selectively adding the additional pixels 85 and 95 to the required positions as in the example of FIG. 5, the captured images 80 and 90 can be displayed relatively large on the trapezoidal designation screen. On the other hand, as shown in FIG. 6A, by adding the frame-shaped additional pixel 87 to the captured image 80, the centers of the captured image 80 and the expanded image 88 coincide with each other, so that the expansion is close to the angle at the time of photographing. The image 88 can be displayed on the trapezoidal designation screen. Further, since the captured images 80 and 90 are expanded only when the vertices 83C and 93C of the trapezoidal images 84 and 94 are located outside the captured images 80 and 90, the addition of unnecessary additional pixels is suppressed.

Further, according to the above embodiment, the positions of the vertices 83A to 83D of the trapezoidal image 84 are estimated, and the guide images 111 to 114 showing the estimated positions of the vertices 83A to 83D are superimposed and displayed on the extended image 88. The user is made to change the positions of the displayed guide images 111 to 114 as necessary. This makes it possible to simplify the user operation for designating the trapezoidal image 84. Further, it is possible to output the corrected image data after having the user confirm whether or not the appropriate corrected image 121 has been generated through the preview screen.

Further, according to the above embodiment, by setting the non-background pixel value for the additional pixel 87 on the trapezoidal designation screen, the user can distinguish between the pixel included in the captured image 80 and the additional pixel 87. This makes it possible to prompt the user to confirm whether or not the guide image 113 superimposed on the additional pixel 87 is displayed at an appropriate position. On the other hand, by setting the background pixel value for the additional pixel included in the corrected image data, the additional pixel can be made inconspicuous in the corrected image indicated by the output corrected image data.

Further, in the multifunction device 10 and the mobile terminal 50 of the above embodiment, the control unit of the present invention executes various programs stored in the areas 32A and 62A of the memories 32 and 62 by the CPUs 31 and 61. An example in which each process is realized has been described. However, the configuration of the control unit is not limited to this, and a part or all of the control unit may be realized by hardware such as an integrated circuit.

Further, the present invention can be realized not only as a multifunction device 10 and a mobile terminal 50, but also as a program for causing the multifunction device 10 and the mobile terminal 50 to execute a process. Then, the program may be recorded and provided on a non-transitory recording medium. The non-transitory recording medium may include a storage unit mounted on a server that can be connected to the multifunction device 10 and the mobile terminal 50 through the communication network 101, in addition to a CD-ROM, a DVD-ROM, and the like. Then, the program stored in the storage unit of the server may be distributed through a communication network 101 such as the Internet as information or a signal indicating the program.

10 ... Multifunction device 11 ... Printer 23, 53 ... Display 24, 54 ... Input I / F
25, 55 ... Communication I / F
31,61 ... CPU
32, 62 ... Memory 35 ... Device program 50 ... Mobile terminal 65 ... Terminal program

Claims (9)

A program readable by a mobile terminal comprising a camera, a display, and an input interface having a touch sensor.
The program
An acquisition process for acquiring captured image data generated by the camera that captured an object, and
The captured image data acquired by the acquisition process is the captured image data generated by the camera that captures a rectangular object from an angle, and includes a part of a trapezoidal image showing the object. When the captured image data does not include the above, the apex of the trapezoidal image included in the portion and the trapezoidal image included in the other portion are based on a part of the trapezoidal image indicated by the captured image data. Estimating processing to estimate the top of the image and
A correction process for generating corrected image data indicating a rectangular corrected image by performing trapezoidal correction on the trapezoidal image consisting of the estimated vertices.
Output instruction processing to output the corrected image data and
It is an expanded image that is an extension of the captured image by adding an additional pixel that is a vertex included in the other portion and includes the estimated vertex to the captured image indicated by the captured image data, and includes the trapezoidal image. Extended processing to generate extended image data indicating the extended image, and
The first display process for displaying the extended image on the display,
In the extended image displayed in the first display process, the mobile terminal is made to execute the first reception process of accepting the touch operation for adjusting the shape of the trapezoidal image through the input interface.
The program
In the correction process, a program for generating the corrected image data by performing trapezoidal correction of the trapezoidal image reflecting the adjustment received in the first reception process. The program
In the first display process, a guide image showing the positions of the vertices estimated by the estimation process is superimposed on the extended image and displayed on the display.
In the first reception process, at least a touch operation for changing the position of the guide image displayed superimposed on the additional pixel can be received through the input interface.
Wherein the correction processing, the program according to claim 1, said trapezoidal image keystone correction to the vertex position of the modified the guide image in the first reception process. In the first reception process, the program changes the drag operation with respect to the position of the guide image displayed superimposed on the additional pixel to the position of the apex corresponding to the guide image to the end position of the drag operation. The program according to claim 2, which is accepted as an operation. The program
A detection process for detecting a straight line of a predetermined length or longer included in the captured image, and
The mobile terminal is made to execute the first determination process of determining the number of straight lines detected by the detection process.
2. The program described in 3. The program
In response to the determination in the first determination process that three straight lines have been detected, the mobile terminal is made to execute a determination process for determining one of the contour lines of the captured image as the fourth straight line.
The program according to claim 4 , wherein in the estimation process, the positions of the intersections of the straight lines detected by the detection process and the straight lines determined by the determination process are estimated as the positions of the vertices of the trapezoidal image. In response to the determination in the first determination process that two or less straight lines have been detected, the photographed image is described so as not to cause the mobile terminal to execute the estimation process and to surround the captured image. The program according to claim 4 or 5 , wherein the mobile terminal is made to execute the expansion process for adding additional pixels. In the first display process, the program displays the extended image in which the non-background pixel value is set for the additional pixel on the display, and the non-background pixel value is the pixel value of the background pixel of the captured image. The program according to any one of claims 1 to 6 , which has different pixel values. The program outputs the corrected image data in which the background pixel value is set for the additional pixel in the output instruction process, and the background pixel value is the pixel value of the background pixel of the captured image from claim 1. The program described in any of 7. The program
A second display process for displaying the corrected image on the display,
A second reception process that accepts an instruction operation for instructing the output of the corrected image data through the input interface, and
The program according to any one of claims 1 to 8 , wherein the mobile terminal executes the output instruction process in response to the reception of the instruction operation in the second reception process.

Images