JP6911326B2 - Information processing device, program, and control method of information processing device - Google Patents

Description

The technology disclosed in the present specification relates to an information processing device or the like that causes a plurality of applications to execute a process of displaying an image on a display unit.

A plurality of images may be displayed simultaneously on the display unit of the information processing device. Patent Document 1 discloses a technique for adjusting the display area and display position of each image so that the plurality of images do not overlap each other when a plurality of images are simultaneously displayed on the display screen of 1.

Japanese Unexamined Patent Publication No. 11-194867

In the technique of Patent Document 1, the display area of an image whose display area has been adjusted is reduced, and the visibility is deteriorated. Therefore, the convenience of the user is impaired.

In the information processing apparatus disclosed in the present specification, a display unit, a first program for displaying one or more first images on the display unit, and one or more first images are displayed on the display unit. An information processing device including a storage unit in which a second program for displaying a second image on the display unit is installed, and one or more first images are displayed in the image display area of the display unit. A first display means for displaying in the first program, and a determination means for causing the first program or the second program to determine an undisplayed area in which one or more first images are not displayed in the image display area. A second display means for displaying a second image in a second program in an undisplayed area is provided.

In the above information processing apparatus, the first program or the second program determines the undisplayed area, which is the area where the first image is not displayed, in the image display area. Then, the second image is displayed in the determined undisplayed area. Thereby, the second image can be displayed so as not to overlap with the first image regardless of the display area and the display position of the first image. Since it is not necessary to adjust the display area of the first image that has already been displayed, the visibility does not deteriorate.

The configuration of the communication system is shown. The operation flow diagram of the print application 21a is shown. The operation flow diagram of the floating application 21b is shown. The operation flow diagram of the floating application 21b is shown. The operation flow diagram of the floating application 21b is shown. An example of display in the image display area E1 is shown. An example of the second image R2 is shown. An example of display in the image display area E1 is shown (including the second image R2a in which the display area is changed). An example of the second image R2b in which the display area is changed is shown.

(Communication system configuration)
FIG. 1 shows a block diagram of a communication system 1 exemplified as an embodiment according to the present application. The communication system 1 includes a mobile phone 10, a printer 110, and an access point 62. The access point 62 is a known relay device.

(Configuration of mobile phone 10)
The mobile phone 10 is a mobile phone equipped with a platform of Android (registered trademark of Google Inc.). The mobile phone 10 mainly includes a CPU 11, a storage unit 12, a button input unit 17, a display unit 18, a wireless LAN interface 22, and a camera unit 27. The CPU 11 controls each function according to a program stored in the storage unit 12, various signals transmitted and received via the wireless LAN interface 22, and the like. Further, the CPU 11 functions as various means by reading the program. In this specification, the interface may be referred to as "I / F".

The wireless LAN_I / F22 is an I / F (that is, an IC chip, a communication circuit) for executing wireless communication according to the Wi-Fi system defined by the Wi-Fi Alliance. The Wi-Fi method is based on, for example, the IEEE (abbreviation of The Institute of Electrical and Electronics Engineers, Inc.) 802.11 standard and the equivalent standard (for example, 802.11a, 11b, 11g, 11n, etc.). It is a wireless communication method based on. The wireless LAN_I / F22 performs wireless communication 202 with the access point 62. The button input unit 17 is a key for executing each function of the mobile phone 10. The button input unit 17 may be integrally configured with the display unit 18 as a touch panel. The display unit 18 displays various functional information of the mobile phone 10 in the image display area E1 (see FIGS. 6 and 8). The camera unit 27 is a camera mounted on the mobile phone 10.

The storage unit 12 includes a program 21. The storage unit 12 may be configured by combining a RAM, a ROM, a flash memory, an HDD, an SD card (a registered trademark of the SD Association), and the like. The program 21 includes a printing application 21a, a floating application 21b, and an operating system 21g (hereinafter, may be referred to as OS 21g. OS 21g is an OS of the Android platform. Each of the printing application 21a and the floating application 21b is for Android. An application. In this specification, an application may be abbreviated as an "application". In the following, a CPU 11 that executes a program such as a printing application 21a or an operating system 21g is simply referred to. It may be described by the program name. For example, the description "the printing application 21a is" may mean "the CPU 11 that executes the printing application 21a".

The print application 21a is an application for causing the printer 110 to perform printing via the wireless LAN_I / F22. FIG. 6 shows an example of the home screen G1 displayed by the print application 21a in the image display area E1. The home screen G1 includes a first image R1 and a non-display area E2 in the image display area E1. The first image R1 is an image displayed on the display unit 18 by the print application 21a. As an example of the first image R1, FIG. 6 shows button images 115, 116, 117, 118. Each of the button images 115 to 118 corresponds to the first image R1. The button image 115 is an image for receiving an input of an instruction for causing the printer 110 to execute the print process. The button image 116 is an image for receiving an input of an instruction to cause the printer 110 to execute the scanning process. The button image 117 is an image for receiving an input of an instruction to cause the printer 110 to execute the FAX process. The button image 118 is an image for receiving an input of an instruction for displaying a screen showing the state of various functions of the printer 110. The character string of the "home screen" shown in FIG. 6 also corresponds to the first image R1. The undisplayed area E2 is an area in the image display area E1 in which the first image R1 is not displayed. The print application 21a has undisplayed area information indicating the display position and the display area of the undisplayed area E2. The print application 21a stores the undisplayed area information in the storage unit 12.

The floating application 21b is an application capable of further displaying the second image R2 on the screen on which the printing application 21a is displaying the first image R1. In the explanatory example of the present embodiment, the floating application 21b is an application that displays a second image R2 that shows the user an evaluation, a comment, or the like for various functions of the printing application 21a. An example of the second image R2 is shown in FIG. When the floating application 21b receives the tap input for the displayed second image R2, for example, the floating application 21b displays an image for writing an evaluation, a comment, or the like.

The print application 21a and the floating application 21b may be applications provided by the vendor of the printer 110. The print application 21a and the floating application 21b may be installed on the mobile phone 10 from a server (not shown) on the Internet, or may be installed on the mobile phone 10 from the media shipped with the printer 110.

The OS 21g is a program that provides basic functions commonly used in the printing application 21a and the floating application 21b. Further, the OS 21g has a function of controlling the activation of the printing application 21a and the floating application 21b. Further, the OS 21g has a function of controlling various components such as a storage unit 12, a button input unit 17, and a display unit 18. By using the API (Application Programming Interface) of the OS 21g by applications such as the printing application 21a and the floating application 21b, a sharing function for transferring data between applications is realized.

The intent table 14 is an area for storing intents. An intent is information that acts as a message exchanged between applications on the Android platform. The intent is a string. By sending an intent, it is possible to call other applications and perform screen transitions within the application. That is, the intent makes it possible to bridge between the sharing source application and the sharing destination application in Android.

(Configuration of printer 110)
The printer 110 is a peripheral device (that is, a peripheral device such as a PC) capable of executing a printing function. The printer 110 can communicate with the mobile phone 10 via wireless communications 201 and 202. As for the configuration of the printer 110, a conventionally known configuration can be used, and therefore detailed description thereof will be omitted here.

(Operation of print application 21a)
The operation of the printing application 21a will be described with reference to FIG. In S10, when the user inputs an operation to start the print application 21a via the button input unit 17, the CPU 11 starts the print application 21a based on the OS 21g.

In S20, the print application 21a determines whether or not it is possible to communicate with the floating application 21b. If a positive judgment is made (S20: YES), the process proceeds to S30, and if a negative judgment is made (S20: NO), the process proceeds to S90. If the affirmative judgment is made in S20, the communication between the print application 21a and the floating application 21b is performed via the intent. Specifically, the print application 21a sets the information in the intent, and uses the API of the OS 21g to call the floating application 21b. Then, the floating application 21b reads out the information set in the intent, so that communication between the printing application 21a and the floating application 21b is performed. The print application 21a may call the floating application 21b by using the sharing function of the OS 21g. In the following description as well, communication between the print application 21a and the floating application 21b shall be performed via the intent even if there is no particular description regarding the intent.

In S30, the print application 21a determines whether or not the floating application 21b is activated. If a positive judgment is made (S30: YES), the process proceeds to S50, and if a negative judgment is made (S30: NO), the process proceeds to S40.

In S40, the print application 21a instructs the OS 21g to start the floating application 21b. If the floating application 21b is not started, this process starts the floating application 21b.

In S50, the print application 21a reads out the non-display area information indicating the display position and the display area of the non-display area E2 from the storage unit 12.

In S60, the print application 21a transmits the undisplayed area information to the floating application 21b.

In S70, the print application 21a assigns the first to Mth priority to M first images. The priority here is a priority indicating the importance of the display contents indicated by the M first images R1, and the priority may be determined in advance by the creator of the print application 21a. For example, on the home screen G1, the print application 21a assigns the first to third priorities to the button images 115, 116, and 117. In the following, "Xth" may be used to mean "Xth priority has been assigned".

In S80, the print application 21a transmits the priority information indicating the priority assigned to the first image R1 in S70 to the floating application 21b.

In S90, the print application 21a displays the first image R1 in the image display area E1.

In S100, the print application 21a determines whether or not the page switching operation has been input. For example, in the home screen G1 shown in FIG. 6, when the input by the indicator is received in the display area of the button image 115, the screen transitions to the screen for executing the print process. In this case, the print application 21a determines that the page switching operation has been input. If a positive judgment is made (S100: YES), the process returns to S20, and if a negative judgment is made (S100: NO), the process of S100 is repeated.

(Operation of floating application 21b)
The operation of the floating application 21b will be described with reference to the flow charts of FIGS. 3 to 5. In S110, the CPU 11 activates the floating application 21b based on the OS 21g. When the print application 21a and the floating application 21b can communicate in S20 described above, the floating application 21b is activated by an instruction from the print application 21a to the OS 21g in S40. The floating application 21b may be started by the user inputting an operation for starting the floating application 21b via the button input unit 17.

In S120, the floating application 21b acquires a list of activated applications. Specifically, the floating application 21b acquires the startup application information that is operating when the floating application 21b is started from the OS 21g.

In S125, the floating application 21b determines whether or not another floating application is running. The OS 21g stores an ID that uniquely identifies the application installed on the mobile phone 10. The ID of the startup application is included in the startup application information. The floating application 21b determines whether or not the startup application is a floating application based on the ID of the startup application indicated by the startup application information. If a positive judgment is made (S125: YES), the process proceeds to S140, and if a negative judgment is made (S125: NO), the process proceeds to S130.

In S130, the floating application 21b determines whether or not it is possible to communicate with the printing application 21a. Since the communication between the floating application 21b and the printing application 21a is the same as the communication in S20, detailed description thereof will be omitted. If affirmative judgment is made (S130: YES), the process proceeds to S135.

In S135, the floating application 21b receives the undisplayed area information indicating the display position and the display area of the undisplayed area E2 and the priority information indicating the priority assigned to the first image R1 from the print application 21a. When the floating application 21b executes the process of S135, the floating application 21b proceeds to S170.

On the other hand, if a negative determination is made in S130 (S130: NO), the process proceeds to S140. In S140, the floating app 21b obtains a screenshot of the image display area E1. Specifically, the floating application 21b obtains screenshot data which is image data in a bitmap format obtained by capturing the display screen of the image display area E1 on which the first image R1 is displayed.

In S150, the floating application 21b recognizes the character string image T1 and the specific image T2. 6 and 8 show an example of the character string image T1 and the specific image T2 displayed on the home screen G1. The character string image T1 is an image represented by characters (text). The specific image T2 is an image that is distinguished from the background image by a closed edge line. For example, a rectangular or circular image can be mentioned. Since edge line detection is a known technique, detailed description thereof will be omitted.

In S160, the floating application 21b determines the undisplayed area E2. Specifically, the floating application 21b determines the area in which the character string image T1 and the specific image T2 are not displayed as the undisplayed area E2 based on the obtained screenshot data. That is, the floating application 21b recognizes the character string image T1 and the specific image T2 as the first image R1. The display position and display area of the character string image T1 and the specific image T2 can be determined by using the respective coordinate values in the screenshot data. When the floating application 21b executes the process of S160, the floating application 21b proceeds to S170.

In S170, the floating application 21b assigns the first to Nth priorities to N second images R2. The priority here is a order indicating the importance of the display contents indicated by the N second images R2, and the priority may be determined in advance by the creator of the floating application 21b.

In S180, the floating application 21b sets the value of K to "1". Here, K is a variable that takes a natural number of 1 or more and N or less.

In S185, the floating application 21b determines whether or not K is N or less. In other words, the floating application 21b determines whether or not the display process has been completed up to the Nth second image R2. If a positive judgment is made (S185: YES), the process proceeds to S190, and if a negative judgment is made (S185: NO), the process proceeds to S250. Immediately after setting K = 1 in S180, the process of S185 may be omitted and the process may proceed to S190.

In S190, the floating application 21b acquires the maximum width of the undisplayed region in the y direction orthogonal to the x direction and the maximum width of the Kth second image R2. Specifically, for example, as shown in FIGS. 6 and 7, the maximum width Lx1 in the x direction of the undisplayed area E2, the maximum width Ly1 in the y direction of the undisplayed area E2, and the x direction of the Kth second image R2. The maximum width Lx2 of the above and the maximum width Ly2 of the Kth second image R2 in the y direction are acquired.

In S200, the floating application 21b compares the lengths of the maximum width Lx1 and the maximum width Lx2, and also compares the lengths of the maximum width Ly1 and the maximum width Ly2.

In S210, whether or not the K-th second image R2 overlaps with the first image R1 displayed in the image display area E1 (button images 115 to 118 and the character string image of the "home screen" in FIG. 6). to decide. If a positive judgment is made (S210: YES), the process proceeds to S310, and if a negative judgment is made (S210: NO), the process proceeds to S220. Specifically, the case where the affirmative judgment is made is a case where the maximum width Lx1 is exceeded by the maximum width Lx2 and a case where the maximum width Ly1 is exceeded by the maximum width Ly2, whichever is at least one of the cases. In this case, since the K-th second image R2 cannot be completely displayed in the undisplayed area E2, an affirmative judgment is made in S210. When K is 2 or more, in S210, the Kth second image R2 is up to the first image R1 displayed in the image display area E1 or the K-1th displayed in the undisplayed area E2. It is determined whether or not it overlaps with the second image R2.

In S220, the floating application 21b displays the Kth second image R2 in the undisplayed area E2.

In S230, the floating application 21b deletes the displayed area of the displayed second image R2 from the undisplayed area E2. Specifically, the floating application 21b detects the coordinate value of the displayed second image R2. Then, assuming that the area indicated by the coordinate values is not the undisplayed area E2, the undisplayed area information indicating the display position and the display area of the undisplayed area E2 received in S135, or the undisplayed area determined in S160. Area E2 is updated.

In S240, the floating application 21b counts up the value of K by 1 and returns to the process of S185.

On the other hand, in S250, the floating application 21b determines whether or not a predetermined time has elapsed. If affirmative judgment is made (S250: YES), the process returns to S120 and a series of processes is repeated. If a negative judgment is made (S250: NO), the process proceeds to S260.

In S260, the floating application 21b determines whether or not the operation is input to an area other than the display area of the second image R2. The operation input to the area other than the display area of the second image R2 is, for example, the input of an instruction into the display area of the first image R1 or the non-display area E2. If a positive judgment is made (S260: YES), the process returns to S120, and if a negative judgment is made (S260: NO), the process returns to S250.

If a positive judgment is made in S210 (S210: YES), the process proceeds to S310. In S310, the floating application 21b determines whether or not the display content of the second image R2 can be changed. An example of changing the display content is a change from the second image R2 shown in FIG. 7 to the second image R2a shown in FIG. The second image R2 is an image indicating the evaluation by five star marks. The second image R2a is an image showing the evaluation numerically. That is, the floating application 21b reduces the area of the display area of the second image R2 by changing the display content without changing the information to be provided to the user. If a positive judgment is made (S310: YES), the process proceeds to S320, and if a negative judgment is made (S310: NO), the process proceeds to S350.

In S320, the floating application 21b determines whether or not the second image R2a whose display content has been changed can be displayed in the undisplayed area E2. Specifically, as shown in FIG. 8, the floating application 21b acquires the maximum width Lx2a in the x direction and the maximum width Ly2a in the y direction for the Kth second image R2a after the display content is changed. Then, the maximum widths Lx2a and Ly2a and the maximum widths Lx1 and Ly1 of the undisplayed area E2 are subjected to the same processing as in S200 and S210 to determine whether or not the second image R2a can be displayed in the undisplayed area E2. be able to. If a positive judgment is made (S320: YES), the process proceeds to S330, and if a negative judgment is made (S320: NO), the process proceeds to S350.

In S330, the floating application 21b changes the display content of the second image R2. In S340, the floating application 21b displays the second image R2a whose display content has been changed in the undisplayed area E2. FIG. 8 shows an example of the second image R2a displayed in the undisplayed area E2. When the floating application 21b performs the processing of S340, it returns to S230.

In S350, the floating application 21b determines whether or not the display arrangement of the second image can be changed. An example of changing the display arrangement is a change from the second image R2 shown in FIG. 7 to the second image R2b shown in FIG. That is, the floating application 21b changes the display arrangement while maintaining the area of the display area of the second image R2 by changing the maximum widths Lx2 and Ly2. If a positive judgment is made (S350: YES), the process proceeds to S360, and if a negative judgment is made (S350: NO), the process proceeds to S410.

In S360, the floating application 21b determines whether or not the second image R2b whose display arrangement has been changed can be displayed in the undisplayed area E2. Since the processing in S360 is the same processing as in S320, the description thereof will be omitted. If a positive judgment is made (S360: YES), the process proceeds to S370, and if a negative judgment is made (S360: NO), the process proceeds to S410.

In S370, the floating application 21b changes the display arrangement of the second image R2. In S380, the floating application 21b displays the second image R2b whose display arrangement has been changed in the undisplayed area. When the floating application 21b performs the processing of S380, it returns to S230.

In S410, the floating application 21b determines whether or not the printing application 21a has received the priority information indicating the first to Mth priorities assigned to the first image R1. If a positive judgment is made (S410: YES), the process proceeds to S420, and if a negative judgment is made (S410: NO), the process proceeds to S425.

In S420, the floating application 21b superimposes the K-th second image R2 on the M-th first image R1 and displays it in the image display area E1. The M-th first image R1 is the first image R1 having the lowest importance among the first images R1.

In S425, the floating application 21b counts up the value of K by 1 and returns to the process of S185.

<Effect of Examples>
The mobile phone 10 causes the print application 21a or the floating application 21b to determine the undisplayed area E2, which is the area in which the first image R1 is not displayed, in the image display area E1 (S50 or S160). Then, the second image R2 is displayed in the determined undisplayed area E2 (S220). As a result, the second image R2 can be displayed in the image display area E1 so as not to overlap with the first image R1 regardless of the display area and the display position of the first image R1. Since it is not necessary to adjust the display area of the first image R1 that has already been displayed, the visibility of the user does not deteriorate.

The print application 21a stores the undisplayed area information indicating the display position and the display area of the undisplayed area E2 in the storage unit 12 in advance, and can determine the undisplayed area E2 by reading this (S50). can.

The print application 21a transmits the undisplayed area information to the floating application 21b (S60). The floating application 21b can display the second image R2 in the undisplayed area E2 based on the undisplayed area information received from the print application 21a. In this way, when the print application 21a and the floating application 21b can communicate with each other (S20: YES, S130: YES), the print application 21a determines the undisplayed area E2, which is the area where the first image R1 is not displayed. By transmitting the undisplayed area information to the floating application 21b, the floating application 21b can display the second image R2 in the undisplayed area E2.

The floating application 21b obtains screenshot data of the display screen of the image display area E1 in which the first image R1 is displayed (S140). The floating application 21b can determine the undisplayed area E2 by analyzing the screenshot data (S160). Even when the print application 21a and the floating application 21b cannot communicate (S20: NO, S130: NO), the floating application 21b can determine the undisplayed area E2 based on the screenshot data.

The floating application 21b acquires a list of activated applications (S120) and determines whether or not another floating application is activated (S125). When another floating application is running (S125: YES), the undisplayed area E2 is determined based on the screenshot data (S160). In this way, even when the print application 21a cannot detect the position of another floating application, the floating application 21b can determine the undisplayed area E2. Therefore, the floating application 21b can display the second image R2 at an appropriate position.

The floating application 21b determines the undisplayed area E2 by recognizing the character string image T1 and the specific image T2 as the first image R1 from the screenshot data (S150). In this way, among the images displayed by the print application 21a, an image that is not an image for providing information to the user (for example, a background image or the like) is regarded as not the first image R1 and the undisplayed area E2 is set. Can be decided.

When it is determined that the second image R2 overlaps with the first image R1 displayed in the image display area E1 (S210: YES), the floating application 21b changes the display area of the second image R2 (S330). , S370). In this way, even if the second image R2 to be displayed cannot be completely displayed in the undisplayed area E2, by changing the display area of the second image R2, it does not overlap with the first image R1. Can be displayed in the undisplayed area E2.

The floating application 21b acquires the maximum widths Lx1 and Ly1 of the undisplayed area E2 and the maximum widths Lx2 and Ly2 of the second image R2 (S190). Then, by comparing these (S200), it is possible to determine whether or not the second image R2 overlaps with the displayed first image R1 (S210).

When the second image R2 overlaps with the first image R1 (S210: YES), the floating application 21b changes the display content of the second image R2 (S330). As a result, the area of the display area of the second image R2 can be reduced. Therefore, even when the undisplayed area E2 is insufficient with respect to the second image R2 to be displayed, the display content of the second image R2 can be changed and displayed in the undisplayed area E2.

When the second image R2 overlaps with the first image R1 (S210: YES), the floating application 21b changes the display arrangement of the second image R2 (S370). That is, the maximum widths Lx2 and Ly2 of the second image R2 are adjusted. As a result, the display arrangement of the second image R2 can be changed while maintaining the area of the display area of the second image R2. Therefore, even when the undisplayed area E2 is insufficient with respect to the second image R2 to be displayed, the display arrangement of the second image R2 can be changed and displayed in the undisplayed area E2.

When a plurality of (M) first images R1 exist, the print application 21a assigns the first to Mth priority to the M first images R1 (S70), and assigns this priority. The indicated priority information is transmitted to the floating application 21b (S80). On the other hand, when the second image R2 overlaps with the first image R1 (S210: YES) and the display content and display arrangement of the second image R2 cannot be changed (S310: NO, S350: NO), or when the second image R2 overlaps with the first image R1 (S210: YES). In the case where the two images R2 overlap with the first image R1 (S210: YES), even if the display content and display arrangement of the second image R2 are changed, the changed second images R2a and R2b are still undisplayed areas. When it cannot be displayed on E2 (S320: NO, S360: NO), the floating application 21b is the Mth first image R1 (that is, the M first images R1) based on the priority information received from the printing application 21a. Of these, the second image R2 is superimposed and displayed on the first image R1), which has the lowest importance (S420). In this way, even if the second image R2 or the changed second images R2a and R2b cannot be displayed in the undisplayed area E2, it is important to display the second image R2 in an overlapping manner with the less important first image R1. The second image R2 can be displayed while maintaining the high visibility of the first image R1.

When a plurality of (N) second images R2 exist, the floating application 21b assigns the first to Nth priorities to the N second images R2 (S170). Then, N second images R2 are displayed in the undisplayed area E2 in order from the first second image R2 (S180) (S220). In this way, even when a plurality of second images R2 exist, the second images R2 can be displayed in the undisplayed area in descending order of priority. Further, even when the K-th second image R2 overlaps with the first image R1 (S210: YES), whether or not the floating application 21b can display the second image R2 after the K + 1th (S425). To judge. In this way, even if there is a second image R2 that cannot be displayed in the non-display area E2, it is possible to determine whether or not all the second images R2 can be displayed in the non-display area E2.

The floating application 21b is the case where the display processing up to the Nth second image R2 is completed (S185: YES) and the predetermined time has elapsed (S250: YES), and the display area of the second image R2. When the operation is input to an area other than (S260: YES), the non-display area E2 is determined again, and the process of displaying the second image R2 in the non-display area E2 is repeated. As a result, when the first image R1 displayed in the image display area E1 is switched within a predetermined time, or when an instruction is input into the display area or the non-display area E2 of the first image R1, the image display area E1 Even when the first image R1 displayed on the screen is switched, the second image R2 can be accurately displayed in the undisplayed area E2.

Although specific examples of the present invention have been described in detail above, these are merely examples and do not limit the scope of claims. The techniques described in the claims include various modifications and modifications of the specific examples illustrated above. Modifications of the above embodiment are listed below.

<First modification>
If a negative determination is made in S350 and S360 (S350: NO, S360: NO), the floating application 21b may end the process of displaying the second image R2. That is, when it is determined that the K-th second image R2 overlaps with the first image R1 (S210: Y), and the display content and display arrangement of the K-th second image R2 cannot be changed (S310: NO, S350). : NO) and the like may end the process of displaying the second image R2 without determining whether or not the second image R2 after the K + 1th can be displayed in the undisplayed area E2. In this case, the floating application 21b may return to the processing of S250.

<Second modification>
The floating application 21b may be always running. In this case, instead of S110 in FIG. 3, the floating application 21b may be configured so as to be activated at the same time when the mobile phone 10 is activated. Further, in this case, in FIG. 2, the processes of S30 and S40 can be omitted.

<Other variants>
The processing of S350 and S360 may be performed before the processing of S310 and S340. That is, if a negative judgment is made in S350 or S360, the process may proceed to S310, and if a negative judgment is made in S310 or S320, the process may proceed to S410.

The floating application 21b may be various applications, for example, a help application. In this case, when the floating application 21b accepts the tap input for the displayed second image R2, for example, a help image for showing an operation method or the like may be displayed.

The case where the mobile phone 10 is equipped with the Android (registered trademark of Google Inc.) platform has been described, but the present invention is not limited to this mode. For example, the mobile phone 10 may be equipped with an iOS (registered trademark of Sisco) platform.

The display modes shown in FIGS. 6 to 9 are examples, and other display modes may be used.

The device on which the print application 21a, the floating application 21b, and the OS 21g operate does not have to be the mobile phone 10, and may be a stationary PC or another device (for example, a television or the like).

The means for assigning the priority to the first image R1 (S70 to S80) can be omitted. The means (S100) for determining whether or not the page switching operation has been input can be omitted. The means for assigning the priority to the second image R2 (S170 to S185) can be omitted. The means for acquiring the maximum width and comparing the maximum widths with each other (S190 to S210) can be omitted. The means (S250) for determining whether or not the predetermined time has elapsed can be omitted. The means (S260) for determining whether or not the operation has been input to an area other than the display area of the second image R2 can be omitted. The means (S230 to S240) for deleting the displayed area of the displayed second image R2 from the undisplayed area E2 can be omitted. The means for changing the display area of the second image R2 (S310 to S380) can be omitted. The means (S410 to S425) for displaying the first image R1 and the second image R2 in an overlapping manner can be omitted. From the above, generally speaking, the mobile phone 10 causes the print application 21a to execute the "first display means", causes the floating application 21b to execute the "second display means", and causes the "decision means" to be executed by the print application 21a. Alternatively, it may be executed by the floating application 21b. As a specific example, the mobile phone 10 causes the print application 21a to execute S50 and S90 at least and causes the floating application 21b to execute S220 at least, or causes the print application 21a to execute S90 at least and causes S160 and S220 to execute at least the floating application 21b. At least let it run.

In addition, the technical elements described in the present specification or the drawings exhibit technical usefulness alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the techniques illustrated in this specification or drawings achieve a plurality of objectives at the same time, and achieving one of the objectives itself has technical usefulness.

The print application 21a is an example of the first program. The floating application 21b is an example of the second program. The mobile phone 10 is an example of an information processing device. The x direction is an example of the first direction. The y direction is an example of the second direction. The maximum width Lx1 is an example of the first maximum width of the undisplayed area. The maximum width Ly1 is an example of the second maximum width of the undisplayed area. The maximum width Lx2 is an example of the first maximum width of the second image. The maximum width Ly2 is an example of the second maximum width of the second image. The CPU 11 that executes S90 is an example of the first display means. The CPU 11 that executes S50 and S160 is an example of the determination means. The CPU 11 that executes S220, S340, and S380 is an example of the second display means. The CPU 11 that executes S140 is an example of the acquisition means. The CPU 11 that executes S260 is an example of the detection means. The CPU 11 that executes S150 is an example of the recognition means. The CPU 11 that executes S60 is an example of the first transmission means. The CPU 11 that executes S70 is an example of the first allocation means. The CPU 11 that executes S80 is an example of the second transmission means. The CPU 11 that executes S210 is an example of the determination means. The CPU 11 that executes S330 and S370 is an example of the display area changing means. The CPU 11 that executes S190 is an example of acquisition means. The CPU 11 that executes S200 is an example of the comparison means. The CPU 11 that executes S170 is an example of the second allocation means.

1: Communication system, 10: Mobile phone, 11: CPU, 12: Storage unit, 18: Display unit, 21a: Printing application, 21b: Floating application, 21g: Operating system, E1: Image display area, E2: Undisplayed area , R1: 1st image, R2: 2nd image

Claims (15)

Display and
A first program for displaying one or more first images on the display unit, and displaying a second image on the display unit while the one or more first images are displayed on the display unit. The storage unit where the second program is installed and
It is an information processing device equipped with
A first display means for causing the first program to display the one or more first images in the image display area of the display unit.
A determination means for causing the first program or the second program to determine an undisplayed area in which the one or more first images are not displayed in the image display area.
A second display means for displaying the second image in the undisplayed area in the second program,
An acquisition means for causing the second program to obtain screenshot data obtained by converting the display screen of the image display area on which the one or more first images are displayed into data as an image.
Recognition that causes the second program to recognize a character string image indicating a character string and a specific image distinguished from a background image by a closed edge line as the first image in the display screen indicated by the screenshot data. Means and
Is equipped with
The determination means is characterized in that the second program determines an area in which the character string image and the specific image are not displayed as the undisplayed area based on the screenshot data obtained by the acquisition means. Information processing device. The information processing apparatus according to claim 1, wherein the obtaining means, the determining means, and the second display means are repeatedly executed at predetermined time intervals. The display unit is further provided with a detection means for causing the display unit to detect a first instruction position indicating a position where the indicator body is in contact with or close to the image display area of the display unit. Ori,
The obtaining means, the determining means, and the second display means
When the detection means detects the first designated position, the detection means is re-executed according to the fact that the detected first designated position is outside the display area of the second image on which the detected first designated position is displayed. The information processing apparatus according to claim 2. When a third program for displaying a third image on the display unit is further installed in the storage unit while the one or more first images are displayed on the display unit, the image display is performed. When the one or more first image and the third image are displayed in the area,
The determination means according to any one of claims 1 to 3, wherein the determination means causes the second program to determine the undisplayed area based on the screenshot data obtained by the acquisition means. Information processing device. When the first program and the second program cannot communicate via the intent,
The determination means according to any one of claims 1 to 4, wherein the determination means causes the second program to determine the undisplayed area based on the screenshot data obtained by the acquisition means. Information processing device. When the second image is displayed in the undisplayed area by the second display means in the second program, the second image is displayed in the image display area beyond the undisplayed area. A determination means for causing the second program to determine whether or not it overlaps with one or more first images.
When it is determined by the determination means that the second image overlaps with the one or more first images displayed in the image display area beyond the undisplayed area, the display of the second image is performed. A display area changing means for changing the area to the second program, and
The information processing apparatus according to any one of claims 1 to 5, further comprising the information processing apparatus. The first maximum width of the undisplayed area, which is the maximum width of the undisplayed area in the first direction, the first maximum width of the second image, which is the maximum width of the second image in the first direction, and the undisplayed area. The second maximum width of the undisplayed region, which is the maximum width in the second direction orthogonal to the first direction, and the second maximum width of the second image, which is the maximum width of the second image in the second direction, are said to be the first. The acquisition method to be acquired by the program of 2 and
The first maximum width of the undisplayed area and the first maximum width of the second image are compared with the second program, and the second maximum width of the undisplayed area and the second maximum width of the second image are compared with the second program. A comparison means to compare with 2 programs,
The information processing device is further equipped with
In the comparison means, of the case where the first maximum width of the second image exceeds the first maximum width of the undisplayed area and the case where the second maximum width of the second image exceeds the second maximum width of the undisplayed area. When at least one of them is applicable, if the second image overlaps with the one or more first images displayed in the image display area beyond the undisplayed area in the determination means, the second program is informed. The information processing apparatus according to claim 6 , further comprising making a judgment. The display area changing means is
The second image displayed in the non-display area in the second display means does not overlap with the one or more first images displayed in the image display area beyond the non-display area. The information processing apparatus according to claim 6 or 7 , wherein the area of the display area of the second image is reduced by changing the display content indicated by the second image. The display area changing means is
When it is determined by the comparison means that the first maximum width of the second image exceeds the first maximum width of the undisplayed area, the second image second image maintains the area of the display area of the second image. 1 By reducing the maximum width and expanding the second maximum width of the second image, when the second image is displayed in the undisplayed area, the second image exceeds the undisplayed area and the image is displayed. The information processing apparatus according to claim 7 , wherein the display area of the second image is changed so as not to overlap with the one or more first images displayed in the display area. When displaying N (N is a natural number of 1 or more) the second image in the undisplayed area, the second priority is given from the first to the Nth with respect to the N second images. It also has a second allocation means to assign to the program of
The second display means causes the second program to display the second image in the undisplayed area in order from the second image to which the first priority is assigned.
The determination means is a case where the second image to which the K-th (K is a natural number of 1 or more and N or less) is assigned the priority is displayed in the undisplayed area by the second display means in the second program. In addition, the second image exceeds the undisplayed area, the one or more first images displayed in the image display area, or the first to K-1 displayed in the undisplayed area. When it is determined that the second image to which the second priority is assigned overlaps with the second image to which the second priority has been assigned, the second image to which the Kth and subsequent priorities are assigned to the undisplayed area is assigned to the second image. The information processing apparatus according to any one of claims 6 to 9 , wherein the information processing apparatus is not displayed in the program of the above. Display and
A first program for displaying one or more first images on the display unit, and displaying a second image on the display unit while the one or more first images are displayed on the display unit. The storage unit where the second program is installed and
It is an information processing device equipped with
A first display means for causing the first program to display the one or more first images in the image display area of the display unit.
A determination means for causing the first program or the second program to determine an undisplayed area in which the one or more first images are not displayed in the image display area.
A second display means for displaying the second image in the undisplayed area in the second program,
When the second image is displayed in the undisplayed area by the second display means in the second program, the second image is displayed in the image display area beyond the undisplayed area. A determination means for causing the second program to determine whether or not it overlaps with one or more first images.
When it is determined by the determination means that the second image overlaps with the one or more first images displayed in the image display area beyond the undisplayed area, the display of the second image is performed. A display area changing means for changing the area to the second program, and
The first maximum width of the undisplayed area, which is the maximum width of the undisplayed area in the first direction, the first maximum width of the second image, which is the maximum width of the second image in the first direction, and the undisplayed area. The second maximum width of the undisplayed region, which is the maximum width in the second direction orthogonal to the first direction, and the second maximum width of the second image, which is the maximum width of the second image in the second direction, are said to be the first. The acquisition method to be acquired by the program of 2 and
The first maximum width of the undisplayed area and the first maximum width of the second image are compared with the second program, and the second maximum width of the undisplayed area and the second maximum width of the second image are compared with the second program. A comparison means to compare with 2 programs,
Is equipped with
In the comparison means, of the case where the first maximum width of the second image exceeds the first maximum width of the undisplayed area and the case where the second maximum width of the second image exceeds the second maximum width of the undisplayed area. When at least one of them is applicable, if the second image overlaps with the one or more first images displayed in the image display area beyond the undisplayed area in the determination means, the second program may be used. An information processing device characterized by making a judgment. The display area changing means is
When it is determined by the comparison means that the first maximum width of the second image exceeds the first maximum width of the undisplayed area, the second image second image maintains the area of the display area of the second image. 1 By reducing the maximum width and expanding the second maximum width of the second image, when the second image is displayed in the undisplayed area, the second image exceeds the undisplayed area and the image is displayed. The information processing apparatus according to claim 11 , wherein the display area of the second image is changed so as not to overlap with the one or more first images displayed in the display area. Display and
A first program for displaying one or more first images on the display unit, and displaying a second image on the display unit while the one or more first images are displayed on the display unit. The storage unit where the second program is installed and
It is an information processing device equipped with
A first display means for causing the first program to display the one or more first images in the image display area of the display unit.
A determination means for causing the first program or the second program to determine an undisplayed area in which the one or more first images are not displayed in the image display area.
A second display means for displaying the second image in the undisplayed area in the second program,
When the second image is displayed in the undisplayed area by the second display means in the second program, the second image is displayed in the image display area beyond the undisplayed area. A determination means for causing the second program to determine whether or not it overlaps with one or more first images.
When it is determined by the determination means that the second image overlaps with the one or more first images displayed in the image display area beyond the undisplayed area, the display of the second image is performed. A display area changing means for changing the area to the second program, and
When displaying N (N is a natural number of 1 or more) the second image in the undisplayed area, the second priority is given from the first to the Nth with respect to the N second images. The second allocation means to be assigned to the program of
Is equipped with
The second display means causes the second program to display the second image in the undisplayed area in order from the second image to which the first priority is assigned.
The determination means is a case where the second image to which the K-th (K is a natural number of 1 or more and N or less) is assigned the priority is displayed in the undisplayed area by the second display means in the second program. In addition, the second image exceeds the undisplayed area, the one or more first images displayed in the image display area, or the first to K-1 displayed in the undisplayed area. When it is determined that the second image to which the second priority is assigned overlaps with the second image to which the second priority has been assigned, the second image to which the Kth and subsequent priorities are assigned to the undisplayed area is assigned to the second image. An information processing device characterized in that it is not displayed in the program of. Display and
A storage unit in which a first program for displaying one or more first images on the display unit is installed, and a storage unit.
The information processing device is readable and
A second program for displaying a second image on the display unit while the one or more first images are displayed on the display unit by the first program.
In the image display area of the display unit, a determination process for determining an undisplayed area in which the one or more first images are not displayed, and a determination process.
A display process for displaying the second image in the undisplayed area, and
An acquisition process for obtaining screenshot data obtained by converting the display screen of the image display area on which one or more first images are displayed into data as an image, and
In the display screen indicated by the screenshot data, a recognition process for recognizing a character string image indicating a character string and a specific image distinguished from a background image by a closed edge line as the first image.
Is equipped with
The second determination process is characterized in that a region in which the character string image and the specific image are not displayed is determined as the undisplayed region based on the screenshot data obtained in the acquisition process. program. Display and
A first program for displaying one or more first images on the display unit, and displaying a second image on the display unit while the one or more first images are displayed on the display unit. The storage unit where the second program is installed and
It is a control method of an information processing device provided with
A first display step of displaying the one or more first images in the image display area of the display unit in the first program, and
A determination step of causing the first program or the second program to determine an undisplayed area in which the one or more first images are not displayed in the image display area.
A second display step of displaying the second image in the undisplayed area in the second program,
An acquisition step of causing the second program to obtain screenshot data obtained by converting the display screen of the image display area on which the one or more first images are displayed into data as an image.
Recognition that causes the second program to recognize a character string image indicating a character string and a specific image distinguished from a background image by a closed edge line as the first image in the display screen indicated by the screenshot data. Steps and
Is equipped with
The determination step is characterized in that the second program determines an area in which the character string image and the specific image are not displayed as the undisplayed area based on the screenshot data acquired in the acquisition step. And the control method.

Images