JP6136667B2 - Image processing system, image forming apparatus, display apparatus, and control program - Google Patents

Description

  The present invention relates to an image processing system, an image forming apparatus, a display apparatus, a display method, and a control program, and in particular, an image processing system, an image forming apparatus, a display apparatus, and a display for displaying information on the image forming apparatus on a portable display apparatus. The present invention relates to a method and a control program.

  An image forming apparatus such as a printer, a copier, a facsimile transmitter / receiver, or an MFP (Multi-Functional Peripheral) having a combination of these functions may display information related to documents and image forming processing on its own operation panel. it can. However, when a document or information to be displayed includes confidential information, the confidential information may be leaked depending on a situation where the image forming apparatus is installed or a usage situation of a user. Therefore, documents such as Japanese Patent Application Laid-Open No. 2006-277239 (hereinafter referred to as Patent Document 1), Japanese Patent Application Laid-Open No. 2011-134053 (hereinafter referred to as Patent Document 2), and Japanese Patent Application Laid-Open No. 2012-138011 (hereinafter referred to as Patent Document 3). Is a portable display device such as a portable terminal or a wearable computer called a so-called head-mounted display (HMD), which is prescribed by the image forming apparatus without displaying such information on its operation panel. And a technique for superimposing and displaying the image on the operation panel of the image forming apparatus.

JP 2006-277239 A JP 2011-134053 A JP 2012-138011 A

  However, in the method disclosed in the above document, since the user operates the operation panel of the actual image forming apparatus while looking at the operation panel displayed on the display device, it is difficult to operate the actual operation panel. There was a problem that the operability was inferior compared with the case of operating while watching.

  The present invention has been made in view of such problems, and an object thereof is to provide an image processing system, an image forming apparatus, a display apparatus, a display method, and a control program having good operability.

In order to achieve the above object, according to one aspect of the present invention, an image processing system includes an image forming apparatus having a touch panel, and a portable display device that can communicate with the image forming apparatus. The image processing system includes a first display processing unit that displays an operation screen on the touch panel of the image forming apparatus and updates the operation screen when a touch on the operation screen is received by the touch panel. The first display processing unit includes: When an image based on secret information defined in advance is included in the operation screen, the operation screen is displayed on the touch panel with the image based on the secret information hidden. The image processing system further includes a second display processing unit for displaying an image based on the secret information on the display device, and a position information indicating the touched position when the touch on the operation screen is received by the touch panel. And transfer means for transferring from the display to the display device. The second display processing means displays the pointer image stored in advance at a position corresponding to the position information of the display device so as to overlap the image based on the secret information and displayed on the touch panel of the image forming apparatus. An image based on confidential information is superimposed on a captured image obtained by capturing the operation screen from the display device and displayed on the display device, and the position information is previously defined from the captured image before being transferred to the display device. When the marker image is detected, the pointer image is displayed at the position of the detected marker image so as to overlap the image in front of the image based on the secret information.

  More preferably, when the second display processing unit superimposes the image based on the secret information on the captured image and displays the image on the display device, the transfer unit displays the position information when receiving a touch on the operation screen with the touch panel. Transfer to device.

  Preferably, the first display processing means displays a button for confirming an instruction by touching the operation screen on the operation screen while the image based on the secret information is not displayed, and in addition to the touch on the operation screen, the button When the is operated, the operation screen is updated according to the touch.

According to another aspect, an image processing system includes an image forming apparatus having a touch panel, and a portable display device that can communicate with the image forming apparatus. The image processing system includes a first display processing unit that displays an operation screen on the touch panel of the image forming apparatus and updates the operation screen when a touch on the operation screen is received by the touch panel. The first display processing unit includes: When an image based on secret information defined in advance is included in the operation screen, the operation screen is displayed on the touch panel with the image based on the secret information hidden. The image processing system further includes a second display processing unit for displaying an image based on the secret information on the display device, and a position information indicating the touched position when the touch on the operation screen is received by the touch panel. And transfer means for transferring from the display to the display device. The second display processing means displays a pointer image stored in advance at a position corresponding to the position information of the display device so as to be superimposed on the front surface of the image based on the secret information. The first display processing means determines whether or not to accept a multi-touch operation and a flick operation on the touch panel according to the attribute of the image based on the secret information included in the operation screen.

According to still another aspect, an image processing system includes an image forming apparatus having a touch panel and a portable display device that can communicate with the image forming apparatus. The image processing system includes a first display processing unit that displays an operation screen on the touch panel of the image forming apparatus and updates the operation screen when a touch on the operation screen is received by the touch panel. The first display processing unit includes: When an image based on secret information defined in advance is included in the operation screen, the operation screen is displayed on the touch panel with the image based on the secret information hidden. The image processing system further includes a second display processing unit for displaying an image based on the secret information on the display device, and a position information indicating the touched position when the touch on the operation screen is received by the touch panel. And transfer means for transferring from the display to the display device. The second display processing means displays a pointer image stored in advance at a position corresponding to the position information of the display device so as to be superimposed on the front surface of the image based on the secret information. The second display processing means sets the display interval of the plurality of buttons displayed on the display device when the image based on the secret information includes a plurality of selection buttons, than the interval when displayed on the touch panel. Enlarge.

According to still another aspect, an image processing system includes an image forming apparatus having a touch panel and a portable display device that can communicate with the image forming apparatus. The image processing system includes a first display processing unit that displays an operation screen on the touch panel of the image forming apparatus and updates the operation screen when a touch on the operation screen is received by the touch panel. The first display processing unit includes: When an image based on secret information defined in advance is included in the operation screen, the operation screen is displayed on the touch panel with the image based on the secret information hidden. The image processing system further includes a second display processing unit for displaying an image based on the secret information on the display device, and a position information indicating the touched position when the touch on the operation screen is received by the touch panel. And transfer means for transferring from the display to the display device. The second display processing means displays a pointer image stored in advance at a position corresponding to the position information of the display device so as to be superimposed on the front surface of the image based on the secret information. The first display processing means uses an operation screen that does not include an image based on the secret information as a threshold value used for determining whether or not the touch operation is performed on the touch panel when the image based on the secret information is not displayed. Make it longer than the threshold when it is displayed.

According to yet another aspect, the image forming apparatus includes a touch panel, displays an operation screen on the touch panel, and includes first display processing means for updating the operation screen when a touch on the operation screen is received on the touch panel, The first display processing unit displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. The image forming apparatus further includes: a first transmission unit configured to transmit an image based on the secret information to the display device when the operation screen includes an image based on the secret information; and touching the operation screen with the touch panel. When received, it includes second transmission means for transmitting position information representing the touched position to the display device. The first display processing means determines whether or not to accept a multi-touch operation and a flick operation on the touch panel according to the attribute of the image based on the secret information included in the operation screen.
According to yet another aspect, the image forming apparatus includes a touch panel, displays an operation screen on the touch panel, and includes first display processing means for updating the operation screen when a touch on the operation screen is received on the touch panel, The first display processing unit displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. The image forming apparatus further includes: a first transmission unit configured to transmit an image based on the secret information to the display device when the operation screen includes an image based on the secret information; and touching the operation screen with the touch panel. When received, it includes second transmission means for transmitting position information representing the touched position to the display device. The first display processing means uses an operation screen that does not include an image based on the secret information as a threshold value used for determining whether or not the touch operation is performed on the touch panel when the image based on the secret information is not displayed. Make it longer than the threshold when it is displayed.

According to yet another aspect, the display device is communicable with the image forming device, is portable, and receives a first image input from the image forming device based on confidential information included in the operation screen. Input means, a camera, display processing means for displaying an image based on secret information, and second input means for receiving input of position information representing a touch position on the operation screen from the image forming apparatus. The The display processing means displays a pointer image stored in advance in a position corresponding to the position information so as to overlap the image based on the secret information, and shoots an operation screen displayed on the image forming apparatus from the camera. An image based on the confidential information is superimposed on the obtained captured image and displayed on the display device. If a predefined marker image is detected from the captured image before the position information is input to the display device, detection is performed. A pointer image is displayed at the position of the marker image superimposed on the front of the image based on the secret information .
According to yet another aspect, the display device is communicable with the image forming device, is portable, and receives a first image input from the image forming device based on confidential information included in the operation screen. Input means, display processing means for displaying an image based on secret information, and second input means for receiving input of position information representing a touch position on the operation screen from the image forming apparatus. The display processing means displays a pointer image stored in advance at a position corresponding to the position information in front of the image based on the secret information, and the image based on the secret information includes a plurality of selection buttons. The display interval of the plurality of buttons displayed on the display device is set larger than the interval when displayed on the image forming device.

  According to still another aspect of the present invention, the display method is a method for displaying an operation screen in the image processing system. The image processing system includes an image forming apparatus having a touch panel, and a portable display device that can communicate with the image forming apparatus. The display method displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen to be displayed includes an image based on the secret information defined in advance on the touch panel of the image forming apparatus. A step of displaying an image based on the secret information on the display device, a step of transferring position information representing the touched position from the image forming device to the display device when a touch on the operation screen is received by the touch panel, and the display device And displaying a pointer image stored in advance at a position corresponding to the position information in front of the image based on the secret information.

According to yet another aspect, the control program is a control program for an image forming apparatus having a touch panel, and is based on secret information when the touch screen includes an image based on pre-defined secret information on an operation screen to be displayed. A step of displaying the operation screen on the touch panel in a state where the image is not displayed, a step of transmitting an image based on the secret information to the display device, and a touch position on the operation screen when the touch panel is received represents the touched position. Causing the image forming apparatus to execute a step of transmitting position information to the display device and a step of determining whether or not to accept a multi-touch operation and a flick operation on the touch panel according to an image attribute based on secret information included in the operation screen . .
According to yet another aspect, the control program is a control program for an image forming apparatus having a touch panel, and is based on secret information when the touch screen includes an image based on pre-defined secret information on an operation screen to be displayed. A step of displaying the operation screen on the touch panel in a state where the image is not displayed, a step of transmitting an image based on the secret information to the display device, and a touch position on the operation screen when the touch panel is received represents the touched position. The step of transmitting the position information to the display device and the threshold value used for determining whether or not the touch operation is performed on the touch panel when the image based on the secret information is not displayed include the image based on the secret information. Causing the image forming apparatus to execute a step longer than a threshold value when the operation screen is displayed

According to still another aspect, the control program is a control program for a portable display device, and displays an image based on the secret information upon receiving an image input based on the secret information included in the operation screen from the image forming device. When an input of position information representing the touch position on the operation screen is received from the step and the image forming apparatus, a pointer image stored in advance is superimposed on the front surface of the image based on the secret information at a position corresponding to the position information. A step of displaying, a step of superimposing an image based on secret information on a captured image obtained by photographing an operation screen displayed on the image forming apparatus from the display device, and displaying the image on the display device; and position information on the display device If a predefined marker image is detected from the captured image before the input, the position of the marker image is detected. Than image based ter image secret information to be executed by the display device and displaying superimposed on the front.
According to still another aspect, the control program is a control program for a portable display device, and displays an image based on the secret information upon receiving an image input based on the secret information included in the operation screen from the image forming device. When an input of position information representing the touch position on the operation screen is received from the step and the image forming apparatus, a pointer image stored in advance is superimposed on the front surface of the image based on the secret information at a position corresponding to the position information. When the display step and the image based on the confidential information include a plurality of buttons for selection, the display interval of the plurality of buttons displayed on the display device is larger than the interval when displayed on the image forming device. And causing the display device to execute.

  According to the present invention, an image processing system having good operability while ensuring security can be provided.

1 is a diagram illustrating a specific example of a configuration of an image processing system (hereinafter abbreviated as a system) according to an embodiment. It is a block diagram which shows the specific example of the apparatus structure of MFP (Multi-Functional Peripheral) contained in a system. It is a block diagram which shows the specific example of an apparatus structure of wearable PC (personal computer) (henceforth HMD) with a head mounted display contained in a system. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 1st Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 1st Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 1st Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 1st Embodiment. 2 is a block diagram illustrating a specific example of a functional configuration of an MFP. FIG. It is a block diagram showing the specific example of a function structure of HMD. It is a flowchart showing the flow of operation | movement at the time of displaying the operation screen in the system concerning 1st Embodiment. 11 is a flowchart showing a flow of an operation (# 3) for displaying the marker image of FIG. 10. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 2nd Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 2nd Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 2nd Embodiment. It is a figure for demonstrating the operation | movement outline | summary in the system concerning 2nd Embodiment. It is a flowchart showing the flow of operation | movement at the time of displaying the operation screen in the system concerning 2nd Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, these descriptions will not be repeated.

<System configuration>
FIG. 1 is a diagram showing a specific example of the configuration of an image processing system (hereinafter abbreviated as a system) according to the present embodiment. Referring to FIG. 1, the system according to the present embodiment includes an MFP (Multi-Functional Peripheral) 100 as an example of an image forming apparatus, and a wearable PC with a head-mounted display (personal) as an example of a portable display device. Computer) (hereinafter abbreviated as HMD) 300. The MFP 100 and the HMD 300 can wirelessly communicate with each other.

  The image forming apparatus is not limited to an MFP as long as it has an operation device and a touch panel as a display device, and may be any other image forming apparatus such as a copier, a printer, or a facsimile transceiver. The display device has a portable size, and is not limited to the HMD as long as the camera is mounted or can be connected to the camera, and may be a terminal device such as a so-called smartphone, a mobile phone, a digital camera, or the like.

<Device configuration>
FIG. 2 is a block diagram showing a specific example of the device configuration of MFP 100. Referring to FIG. 2, MFP 100 includes a CPU (Central Processing Unit) 10A, and includes a system controller 10 for controlling the entire MFP, and a ROM (Read that is a memory for storing a program executed by CPU 10A, and the like. Only Memory) 11, RAM (Random Access Memory) 12 that is a memory for storing image data or a work area when the CPU 10A executes a program, and a large-capacity memory such as a hard disk device. A storage device 13, an image capturing unit 14 for reading an image, an input image processing unit 15 for processing input image data read by the image capturing unit 14, and a print data by processing the image data The output image processing unit 16 and a printer error for printing the print data from the output image processing unit 16 on paper. It includes a gin 17, an operation unit 18 including a touch panel is used as an operation device and a display device, and a network interface 19 for controlling the radio communication between the HMD 300. FIG. 2 shows an example of the configuration of the MFP. Therefore, MFP 100 may include a facsimile unit and a voice recognition unit (not shown) in addition to the configuration shown in FIG.

  FIG. 3 is a block diagram illustrating a specific example of the device configuration of the HMD 300. Referring to FIG. 3, the HMD 300 stores a CPU 30 for controlling the entire apparatus, a ROM 31 which is a memory for storing a program executed by the CPU 30, and image data or the CPU 30 executes a program. A network 32 for controlling wireless communication between the RAM 32, which is a memory for serving as a work area, a display 33 that can be worn on the body or clothes such as the user's head, the camera 34, and the MFP 100. 35. FIG. 3 shows an example of the configuration of a wearable PC with a head-mounted display. Therefore, the HMD 300 may include an operation device, a voice input device (microphone), a voice output device (speaker), etc. (not shown) in addition to the configuration shown in FIG.

[First Embodiment]
<Overview of operation>
4 and 5 are diagrams for explaining the outline of the operation in the system according to the first embodiment. 4A shows an example of an operation screen displayed on the operation unit 18 that is a touch panel of the MFP 100, and FIGS. 4B and 5 show examples of screens displayed on the display 33 of the HMD 300. FIG. Specifically, in this system, a login request is made from the HMD 300 to the MFP 100 in the secure mode, and when the login is successful in the MFP 100, the MFP 100 also operates in the secure mode. Accordingly, a login request in secure mode from the HMD 300 can be said to be a request for secure mode to the MFP 100. In the following description, when the secure mode request is received from the HMD 300 and the MFP 100 is operating in the secure mode, the system performs an operation for displaying a screen as described later. However, the system is limited to the secure mode. Instead, an operation for screen display may be performed when an image based on secret information described later is included in the operation screen.

  In the secure mode, when the operation screen to be displayed includes an image based on secret information defined in advance, the MFP 100 displays the operation screen on the operation unit 18 with the image based on the secret information hidden. To do. The image based on the secret information corresponds to, for example, a preview image of a document file, a selection button displaying a transmission destination for selecting a transmission destination of the document file, and the like. FIG. 4 shows an example in which an operation screen including a preview image of a document file is an operation target. As shown in FIG. 4A, the operation unit 18 of the MFP 100 has a preview. The operation screen is displayed with no image displayed.

  MFP 100 transmits an image (preview image in the example of FIG. 4) based on the secret information to HMD 300 together with coordinate information representing the display position on the operation screen. Alternatively, the MFP 100 may transmit secret information (a document file in the example of FIG. 4) itself to the HMD 300. The HMD 300 displays an image based on the secret information on the display 33. In the example of FIG. 4, a preview image is displayed on the display 33 of the HMD 300 as shown in FIG.

  When MFP 100 receives a touch operation on an area where a preview image that is not displayed on the operation screen is displayed, MFP 100 transmits position information indicating the touched position to HMD 300. Preferably, MFP 100 hides the preview image on the operation screen, and transmits to HMD 300 position information regarding the touch operation on the region received during the period in which the preview image is displayed on HMD 300.

  When the HMD 300 receives the position information from the MFP 100, as shown in FIG. 4B, the HMD 300 displays a pointer image stored in advance at a position corresponding to the preview image displayed on the display 33 rather than the preview image. Display in front of it. For example, as shown in FIG. 4B, the pointer image may be a hand image, an image such as an arrow or a pen, or a combination of these images. Good. Thereby, the operability of the user who operates while looking at the display 33 can be improved.

  Preferably, when displaying the preview image, HMD 300 captures the operation screen of MFP 100 with camera 34 and displays the preview image superimposed on the captured image. At that time, as shown in FIG. 5, HMD 300 uses the coordinate information from MFP 100 to display a preview image in a non-display area of the operation screen included in the captured image. .

  Preferably, the HMD 300 analyzes the photographed image, and when a marker image that is defined in advance before the position information is transmitted from the MFP 100 is detected from the photographed image, the HMD 300 is located at the detected position. The pointer image is displayed in front of the preview image. As an example, the marker image corresponds to an image of an object used for the operation of the touch panel such as a hand or a touch pen. That is, as shown in FIG. 6A, when the operation screen is displayed with the preview image hidden on the operation unit 18 of the MFP 100, the user's hand has approached the operation screen. 6 is detected by analyzing the captured image, as shown in FIG. 6B or FIG. 7, the HMD 300 displays a pointer image such as a hand image on the display 33 in front of the preview image. Is displayed. Thereby, the operability of the user who operates while looking at the display 33 can be further improved.

  More preferably, when the HMD 300 displays the pointer image according to the marker image detected before the touch, the HMD 300 determines the distance from the operation screen based on the size of the marker image, and the size according to the distance. Displays the pointer image. Thereby, the operability of the user who operates while looking at the display 33 can be further improved.

<Functional configuration>
FIG. 8 is a block diagram showing a specific example of a functional configuration of MFP 100 for performing the above operation. Each function of FIG. 8 is realized mainly by the CPU 10 </ b> A when the CPU 10 </ b> A of the MFP 100 reads a program stored in the ROM 11 onto the RAM 12 and executes it. However, at least a part of the functions may be realized by a hardware configuration such as an electric circuit.

  Referring to FIG. 8, a memory such as ROM 11 is provided with an operation screen storage unit 111 which is a storage area for storing an operation screen. Further, referring to FIG. 8, CPU 10A accepts an input of an operation signal from display processing unit 101 for performing a process of displaying an operation screen on operation unit 18 that is a touch panel, and operation unit 18 that is a touch panel. An instruction input unit 102 for receiving an instruction (touch), a processing unit 103 for executing an image forming process instructed by specifying the instruction content from the touch position and the display screen, and updating the operation screen, and a secret A first transmission unit 104 for transmitting a preview image as an example of an image based on information to the HMD 300 via the network interface 19, and positional information indicating the touched position on the operation screen via the network interface 19. And a second transmitter 105 for transmitting to the HMD 300.

  The display processing unit 101 hides the preview image when the operation screen includes a preview image as an example of an image based on confidential information or when the operation screen includes a preview image in the secure mode. Then, a process for displaying the operation screen on the operation unit 18 which is a touch panel is executed.

  The first transmission unit 104 transmits the preview image to the HMD 300 via the network interface 19 when the operation screen includes a preview image or when the operation screen includes a preview image in the secure mode. Preferably, first transmission unit 104 transmits, to the HMD 300, coordinate information representing an area for displaying the preview image on the operation screen together with the preview image.

  When the second transmission unit 105 receives a touch on the operation screen, the second transmission unit 105 transmits position information indicating the touched position to the HMD 300 via the network interface 19. Preferably, the second transmission unit 105 transmits position information to the HMD 300 upon receiving a touch on a region for displaying a preview image in the operation screen. More preferably, the second transmission unit 105 transmits position information about the touch received while the display processing unit 101 displays the operation screen on the operation unit 18 in a state where the preview image is not displayed.

  FIG. 9 is a block diagram showing a specific example of a functional configuration of HMD 300 for performing the above-described operation. Each function of FIG. 9 is mainly realized by the CPU 30 when the CPU 30 of the HMD 300 reads a program stored in the ROM 31 onto the RAM 32 and executes it. However, at least a part of the functions may be realized by a hardware configuration such as an electric circuit.

  Referring to FIG. 9, CPU 30 receives secret information image input unit 301 for receiving an input of a preview image as an example of an image based on secret information from MFP 100 via network interface 35, and MFP 100 via network interface 35. A position information input unit 302 for receiving input of position information from the camera, a display processing unit 303 for executing processing for displaying a preview image or the like on the display 33, and a captured image for receiving input of a captured image from the camera 34. An input unit 304 and an analysis unit 305 for analyzing a captured image are included.

  Display processing unit 303 preferably uses coordinate information from MFP 100 to display the preview image on display 33 in a region where the preview image of the operation screen included in the captured image is hidden.

  The display processing unit 303 stores, for example, a hand image as a pointer image in advance. Then, when the position information input unit 302 receives the input of the position information, the display processing unit 303 places the pointer image at a position corresponding to the position information of the preview image displayed on the display 33 in front of the preview image. Overlapping display.

  Preferably, the analysis unit 305 stores a marker image template such as a hand or a touch pen in advance, and searches for a marker image in the captured image by performing a matching process on the captured image, for example. When the marker image is detected from the photographed image, the above pointer image such as a hand image is displayed in front of the preview image at the position of the detected marker image. More preferably, the analysis unit 305 determines the distance from the operation screen to the operation object such as a hand based on the size of the detected marker image, and displays a pointer image having a size corresponding to the distance. As an example, the analysis unit 305 stores the distance in association with the size ratio of the marker image with respect to the operation screen, and determines the distance according to the size ratio of the detected marker image with respect to the operation screen in the captured image. Good.

<Operation flow>
FIG. 10 is a flowchart showing an operation flow when the operation screen is displayed in the system according to the first embodiment. The left side shows the operation in the MFP 100 and the right side shows the operation flow in the HMD 300. Yes. This operation is realized by each CPU reading and executing a program stored in the ROM on the RAM and executing the functions shown in FIG. 8 or FIG.

  Referring to FIG. 10, when CPU 30 of HMD 300 receives a user operation from an operation device (not shown), it turns on secure mode according to the operation and activates camera 34 (step S101). Then, CPU 30 requests login in secure mode to MFP 100 specified in advance or designated by user operation (step S103). When the MFP 100 is successfully authenticated, the login of the HMD 300 to the MFP 100 is permitted.

  If the login request from the HMD 300 is not in the secure mode (NO in step S201), the CPU 10A of the MF 100 executes the normal process after permitting the login (step S203). Here, the processing for displaying the operation screen is not performed, and thus the detailed description below is not described.

  On the other hand, if the login request from the HMD 300 is in the secure mode (YES in step S201), login in the secure mode from the HMD 300 is permitted (step S205), and an operation menu screen is displayed on the operation unit 18 that is a touch panel. It is displayed (step S207). The operation menu includes a menu for instructing a preview display of a document file as an example of an operation for displaying an operation screen including an image based on confidential information. The operation from the beginning of this operation to this point is referred to as a login operation (# 1).

  When the menu for instructing the preview display of the document file is not selected in the displayed operation menu (NO in step S209), the CPU 10A executes normal processing according to the selected menu (step S211). . Here, the processing for displaying the operation screen is not performed, and thus the detailed description below is not described.

  On the other hand, when the menu for instructing the preview display of the document file is selected in the operation menu (YES in step S209), the CPU 10A displays on the touch panel an operation screen that hides the preview image that is an image based on the confidential information. The information is displayed on a certain operation unit 18 (step S213). Further, the CPU 10A transmits a preview image to the HMD 300 (step S215). At this time, preferably, CPU 10A also transmits coordinate information representing an area for displaying a preview image on the operation screen to HMD 300.

  CPU 30 of HMD 300 displays a preview image on display 33 based on information from MFP 100 (step S105). At this time, preferably, CPU 30 causes camera 34 to capture the operation screen to acquire a captured image, and uses the coordinate information from MFP 100 to display the preview image superimposed on the operation screen. The operation from the beginning of this operation to this point is referred to as an operation screen display operation (# 2).

  CPU 10A of MFP 100 executes an operation (# 3) for displaying a marker image, which will be described later, before accepting a touch operation on the operation screen in a state where the operation screen with the preview image hidden is displayed. . When a touch operation on the operation screen (preferably a display area of a preview image) is received (YES in step S217), CPU 10A transmits position information indicating the touched position to HMD 300 (step S219). Then, the CPU 10A executes an image forming process according to the user operation (step S221).

  Preferably, CPU 10A notifies HMD 300 that the operation screen has been updated in step S221. Upon receiving the notification, the CPU 30 of the HMD 300 ends the display of the preview image on the display 33 and the display of the captured image on the operation screen.

  When the CPU 30 of the HMD 300 receives the position information from the MFP 100, the hand image as a pointer image is displayed in front of the preview image at a position corresponding to the displayed preview image (step S107).

  FIG. 11 is a flowchart showing the flow of the operation (# 3) for displaying the marker image. Referring to FIG. 11, CPU 30 of HMD 300 detects a hand image, which is a marker image, from the captured image before receiving position information indicating the touch position from MFP 100 (YES in step S301). The distance from the operation screen to the hand is determined based on the size of the hand image (step S303). Then, the CPU 30 determines the size of the hand image, which is a pointer image, according to the determined distance (step S305), and displays the hand image of that size in front of the preview image (step S307).

<Effect of the first embodiment>
By performing the above operation, the present system can display a secure mode when displaying an operation screen including an image based on confidential information such as a selection button displaying a transmission destination for selecting a transmission destination of a preview image or a document file. Then, the image is displayed on the operation unit 18 of the main body of the MFP 100 in a non-displayed state, and the image based on the secret information is displayed on a display device such as the HMD 300 carried by the user. Thereby, it is possible to prevent the confidential information from leaking from the operation screen depending on the installation state of the MFP 100 and the use state of the user.

  At this time, by performing the above-described operation, the present system displays an image of a hand or the like operating on the display 33 of the HMD 300 so as to be superimposed on the front of the image based on the secret information. Thereby, when operating the MFP 100 while viewing the display on the HMD 300, the position of a hand or the like can be grasped from the display on the HMD 300, and the operability for the user can be improved. More preferably, an image of a hand or the like is displayed with a size corresponding to the distance from the operation screen of MFP 100 to the hand or the like. Accordingly, when operating the MFP 100 while viewing the display on the HMD 300, the position of the hand or the like can be grasped from the display on the HMD 300 to the operation screen displayed on the MFP 100, and the operability of the user can be improved. Can be improved.

[Second Embodiment]
<Overview of operation>
12 to 15 are diagrams for explaining the outline of the operation in the system according to the second embodiment. 12A illustrates an example of an operation screen displayed on the operation unit 18 that is a touch panel of the MFP 100. FIGS. 12B, 13, 14, and 15 illustrate screens displayed on the display 33 of the HMD 300. An example is shown.

  Even in the system according to the second embodiment, in the secure mode, the MFP 100 displays the operation screen on the operation unit 18 that is a touch panel with the image based on the secret information hidden, and the HMD 300 displays the image based on the secret information. Is displayed on the display 33. FIG. 12 shows an example in which the operation screen includes a selection button that displays a transmission destination for selecting a transmission destination of a document file as an image based on confidential information.

  In the system according to the second embodiment, as illustrated in FIG. 12A, in the secure mode, MFP 100 hides a destination selection button that is an example of an image based on secret information. At the same time, a button (confirmation button) for confirming an instruction by touching the operation screen is displayed on the operation screen. Until the confirmation button is pressed, for example, the same selection button is touched again, or no other operation is performed until a predetermined time elapses after the selection button is pressed. Can do. That is, using the operation screen, the user can select the transmission destination with the selection button, press the confirm button to confirm the selection, and then press the OK button to instruct the transmission of the document file. it can.

  MFP 100 transmits an image of the selection button as an image based on the secret information to HMD 300. Then, the HMD 300 displays an image of the selection button on the display 33 as shown in FIG. Preferably, when there are a plurality of selection buttons, MFP 100 generates an image in which the display interval of the selection buttons is wider than the interval based on the display position on MFP 100, and transmits the generated image to HMD 300. As a result, as shown in FIG. 13, the selection button is displayed on the HMD 300 wider than the interval based on the display position of the MFP 100. Similar to the first embodiment, as shown in FIG. 14 or FIG. 15, the HMD 300 superimposes the image of the selection button on the captured image of the operation screen of the MFP 100 where the selection button is hidden. Then, it may be displayed on the display 33.

<Functional configuration>
Functional configurations of MFP 100 and HMD 300 for performing the above operation may be the same as the configurations shown in FIGS. 8 and 9, respectively.

  In the second embodiment, display processing unit 101 of MFP 100 has a selection button as an example of an image based on confidential information included in the operation screen, or a selection button included in the operation screen in the secure mode. In addition, the selection button is not displayed, and a confirmation button is added to the operation screen storage unit 111 with respect to the stored operation screen and displayed on the operation unit 18 which is a touch panel. The processing unit 103 accepts a touch on the confirmation button by the instruction input unit 102 to confirm the selection state, and executes an image forming process according to the user operation.

  Preferably, when there are a plurality of selection buttons, display processing unit 101 generates an image in which the display interval of the selection buttons is wider than the interval based on the display position on MFP 100.

<Operation flow>
FIG. 16 is a flowchart showing an operation flow when the operation screen is displayed in the system according to the second embodiment. As in FIG. 10, the operation on the left side is on the MFP 100 and the operation on the right side is on the HMD 300. The flow of operation is shown. This operation is also realized by causing each CPU to read out and execute a program stored in the ROM on the RAM and execute the functions shown in FIG. 8 or FIG.

  Referring to FIG. 16, after the login operation (# 1) shown in FIG. 10, if the menu for instructing the display of the destination information is not selected in the displayed operation menu (in step S231). NO), CPU 10A executes normal processing according to the selected menu (step S233). Here, the processing for displaying the operation screen is not performed, and thus the detailed description below is not described.

  On the other hand, when the menu for instructing the display of the destination information (select button) is selected in the operation menu (YES in step S231), the CPU 10A displays an operation screen that hides the area for displaying the image of the select button. The information is displayed on the operation unit 18 that is a touch panel (step S235). At this time, the CPU 10A further displays a button (confirmation button) for confirming an instruction by touching the operation screen described above on the operation screen (step S237). Further, the CPU 10A transmits an image of the selection button to the HMD 300 (step S239). At this time, preferably, CPU 10A also transmits coordinate information representing an area for displaying an image of a selection button on the operation screen to HMD 300. In addition, preferably, when there are a plurality of selection buttons, the CPU 10A generates an image with a wide display interval and transmits the generated image to the HMD 300 as shown in FIGS.

  The CPU 30 of the HMD 300 displays an image of the selection button on the display 33 based on the information from the MFP 100 (step S111). At this time, preferably, CPU 30 causes camera 34 to capture the operation screen to acquire a captured image, and uses the coordinate information from MFP 100 to display the selection button image superimposed on the operation screen.

  CPU 10A of MFP 100 accepts a touch operation on the operation screen (preferably a display area of the image of the selection button) (YES in step S241), and further, when the confirm button is touched (YES in step S243), according to the user operation. An image forming process is executed (step S245).

  In the system according to the second embodiment, as in the system according to the first embodiment, when the CPU 10A receives a touch operation on the operation screen, the CPU 10A transmits the position information to the HMD 300. A hand image as a pointer image may be displayed in front of the selection button image. In addition, when a hand image that is a marker image is detected in the captured image before the touch operation, the CPU 30 of the HMD 300 may display the hand image in front of the selection button image. Good.

<Effects of Second Embodiment>
By performing the above operation, the present system is hidden on the MFP 100 main body, and when the image based on the secret information displayed on the HMD 300 includes an image to be operated such as a selection button, the confirmation button is displayed on the operation screen of the MFP 100 main body. Is displayed. Then, only after the confirm button is operated, MFP 100 confirms the previous operation and executes the image forming process. Normally, the operation of MFP 100 while looking at a display device carried by the user such as HMD 300 is more likely to be erroneous than when MFP 100 is directly operated. However, since the confirmation button is displayed as described above, it is possible to prevent a malfunction based on an erroneous operation and to improve the operability.

  Further, when a plurality of selection buttons are included, the HMD 300 can display a wider button interval than that displayed on the MFP 100 main body, thereby improving operability.

<Modification>
As another example, CPU 10A of MFP 100 accepts certain operations such as a multi-touch operation and a flick operation on the touch panel in a state where an image based on confidential information is not displayed on the display screen of operation unit 18 that is a touch panel. It may not be possible. Preferably, the CPU 10A may determine whether or not these operations can be performed according to the attribute of the image based on the secret information. For example, when the image based on the secret information is a preview image, the above operation may be accepted, and when the image is a selection button, the above operation may not be accepted.

  As yet another example, CPU 10A of MFP 100 determines a touch time threshold value for determining that the touch operation is performed on the touch panel when the image based on the secret information is not displayed on the display screen of operation unit 18 that is a touch panel. May be longer than the threshold value during normal operation screen display. In other words, CPU 10A of MFP 100 may recognize that a touch operation has been performed only after a touch for a longer time than when a normal operation screen is displayed.

  By doing so, it is possible to appropriately prevent an erroneous operation due to an unintended touch such as a user hitting the touch panel when operating the touch panel of the MFP 100 while looking at a display device carried by the user such as the HMD 300, The operability can be improved.

  Further, respective programs for causing the CPU 10A of the MFP 100 and the CPU 30 of the HMD 300 to perform the above-described operation can be provided. Such a program is recorded on a computer-readable recording medium such as a flexible disk attached to the computer, a CD-ROM (Compact Disk-Read Only Memory), a ROM, a RAM, and a memory card, and provided as a program product. You can also Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. A program can also be provided by downloading via a network.

  The program according to the present invention is a program module that is provided as a part of a computer operating system (OS) and calls necessary modules in a predetermined arrangement at a predetermined timing to execute processing. Also good. In that case, the program itself does not include the module, and the process is executed in cooperation with the OS. A program that does not include such a module can also be included in the program according to the present invention.

  The program according to the present invention may be provided by being incorporated in a part of another program. Even in this case, the program itself does not include the module included in the other program, and the process is executed in cooperation with the other program. Such a program incorporated in another program can also be included in the program according to the present invention.

  The provided program product is installed in a program storage unit such as a hard disk and executed. The program product includes the program itself and a recording medium on which the program is recorded.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  10 system controller, 10A, 30 CPU, 11, 31 ROM, 12, 32 RAM, 13 storage device, 14 imaging unit, 15 input image processing unit, 16 output image processing unit, 17 printer engine, 18 operation unit, 19, 35 Network interface, 33 display, 34 camera, 100 MFP, 101, 303 display processing unit, 102 instruction input unit, 103 processing unit, 104 first transmission unit, 105 second transmission unit, 111 operation screen storage unit, 300 HMD, 301 Secret information image input unit 302 Position information input unit 304 Captured image input unit 305 Analysis unit

Claims (14)

An image processing system comprising: an image forming apparatus having a touch panel; and a portable display device capable of communicating with the image forming apparatus,
Comprising: a first display processing unit configured to display an operation screen on the touch panel of the image forming apparatus and update the operation screen when a touch on the operation screen is received on the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
Second display processing means for displaying an image based on the secret information on the display device;
A transfer means for transferring position information representing the touched position from the image forming apparatus to the display device when a touch on the operation screen is received by the touch panel;
The second display processing means includes:
In a position corresponding to the position information of the display device, a pointer image stored in advance is displayed in front of the image based on the secret information,
An image based on the secret information is superimposed on a photographed image obtained by photographing the operation screen displayed on the touch panel of the image forming apparatus from the display device and displayed on the display device;
If a predefined marker image is detected from the captured image before the position information is transferred to the display device, the pointer image is based on the secret information at the detected position of the marker image. An image processing system that displays images in front of each other . When the second display processing unit receives a touch on the operation screen with the touch panel while the second display processing unit superimposes an image based on the secret information on the captured image and displays the image on the display device, The image processing system according to claim 1 , wherein position information is transferred to the display device. The first display processing means displays a button for confirming an instruction by touching the operation screen on the operation screen in a state where an image based on the secret information is not displayed, in addition to the touch on the operation screen. updates the operation screen in accordance with the touch by said button is operated Te, image processing system according to claim 1 or 2. An image processing system comprising: an image forming apparatus having a touch panel; and a portable display device capable of communicating with the image forming apparatus,
Comprising: a first display processing unit configured to display an operation screen on the touch panel of the image forming apparatus and update the operation screen when a touch on the operation screen is received on the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
Second display processing means for displaying an image based on the secret information on the display device;
A transfer means for transferring position information representing the touched position from the image forming apparatus to the display device when a touch on the operation screen is received by the touch panel;
Said second display processing unit, a position corresponding to the position information of the display device, displays superimposed in front of the image based pointer image stored in advance in the secret information,
The image processing system, wherein the first display processing unit determines whether or not to accept a multi-touch operation and a flick operation on the touch panel according to an image attribute based on the secret information included in the operation screen. An image processing system comprising: an image forming apparatus having a touch panel; and a portable display device capable of communicating with the image forming apparatus,
Comprising: a first display processing unit configured to display an operation screen on the touch panel of the image forming apparatus and update the operation screen when a touch on the operation screen is received on the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
Second display processing means for displaying an image based on the secret information on the display device;
A transfer means for transferring position information representing the touched position from the image forming apparatus to the display device when a touch on the operation screen is received by the touch panel;
The second display processing means includes:
In a position corresponding to the position information of the display device, a pointer image stored in advance is displayed in front of the image based on the secret information,
An image in which, when an image based on the secret information includes a plurality of buttons for selection, a display interval of the plurality of buttons displayed on the display device is larger than an interval when displayed on the touch panel. Processing system. An image processing system comprising: an image forming apparatus having a touch panel; and a portable display device capable of communicating with the image forming apparatus,
Comprising: a first display processing unit configured to display an operation screen on the touch panel of the image forming apparatus and update the operation screen when a touch on the operation screen is received on the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
Second display processing means for displaying an image based on the secret information on the display device;
A transfer means for transferring position information representing the touched position from the image forming apparatus to the display device when a touch on the operation screen is received by the touch panel;
Said second display processing unit, a position corresponding to the position information of the display device, displays superimposed in front of the image based pointer image stored in advance in the secret information,
The first display processing means includes an image based on the secret information that is used to determine whether or not the touch operation on the touch panel is performed when the image based on the secret information is not displayed. An image processing system that is longer than the threshold when no operation screen is displayed . A display device that can communicate with an image forming apparatus and is portable.
A first input means for receiving an input of an image based on confidential information included in the operation screen from the image forming apparatus;
A camera,
Display processing means for displaying an image based on the secret information;
A second input means for receiving input of position information representing a touch position on the operation screen from the image forming apparatus;
The display processing means includes
In a position corresponding to the position information, a pointer image stored in advance is displayed in front of the image based on the secret information,
An image based on the secret information is superimposed on a captured image obtained by capturing the operation screen displayed on the image forming apparatus from the camera and displayed on the display device;
If a predefined marker image is detected from the captured image before the position information is input to the display device, the pointer image is based on the secret information at the detected position of the marker image. A display device that displays images in front of each other . There is a control program for a portable display device,
When receiving an input of an image based on confidential information included in the operation screen from the image forming apparatus, displaying an image based on the confidential information;
When an input of position information indicating a touch position on the operation screen is received from the image forming apparatus, a pointer image stored in advance at a position corresponding to the position information is superimposed on the front surface of the image based on the secret information. Step to display ,
Superimposing an image based on the secret information on a captured image obtained by photographing the operation screen displayed on the image forming apparatus from the display device, and displaying the image on the display device;
If a predefined marker image is detected from the captured image before the position information is input to the display device, the pointer image is based on the secret information at the detected position of the marker image. A control program that causes the display device to execute a step of displaying the image superimposed on the front side of the image . An image forming apparatus having a touch panel,
Comprising a first display processing means for displaying an operation screen on the touch panel and updating the operation screen upon receiving a touch on the operation screen with the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
First transmission means for transmitting an image based on the secret information to a display device when the operation screen includes an image based on the secret information;
When receiving a touch on the operation screen with the touch panel, the touch panel further includes second transmission means for transmitting position information representing the touched position to the display device ,
It said first display processing means that determine whether the reception of the multi-touch operation and a flick operation on the touch panel according to an attribute of the image based on the secret information included in the operation screen, the image forming apparatus. A control program for an image forming apparatus having a touch panel,
A step of displaying the operation screen on the touch panel in a state in which the image based on the secret information is not displayed when the operation screen to be displayed includes an image based on pre-defined secret information on the touch panel;
Transmitting an image based on the secret information to a display device;
When receiving a touch on the operation screen with the touch panel, transmitting position information representing the touched position to the display device ;
A control program that causes the image forming apparatus to execute a step of determining whether to accept a multi-touch operation and a flick operation on the touch panel according to an image attribute based on the secret information included in the operation screen . A display device that can communicate with an image forming apparatus and is portable.
A first input means for receiving an input of an image based on confidential information included in the operation screen from the image forming apparatus;
Display processing means for displaying an image based on the secret information;
A second input means for receiving input of position information representing a touch position on the operation screen from the image forming apparatus;
The display processing means includes
In a position corresponding to the position information, a pointer image stored in advance is displayed in front of the image based on the secret information,
When a plurality of selection buttons are included in the image based on the secret information, the display interval of the plurality of buttons displayed on the display device is made larger than the interval when displayed on the image forming device. , Display device. There is a control program for a portable display device,
When receiving an input of an image based on confidential information included in the operation screen from the image forming apparatus, displaying an image based on the confidential information;
When an input of position information indicating a touch position on the operation screen is received from the image forming apparatus, a pointer image stored in advance at a position corresponding to the position information is superimposed on the front surface of the image based on the secret information. Step to display ,
When a plurality of selection buttons are included in the image based on the secret information, the display interval of the plurality of buttons displayed on the display device is made larger than the interval when displayed on the image forming device. A control program causing the display device to execute steps . An image forming apparatus having a touch panel,
Comprising a first display processing means for displaying an operation screen on the touch panel and updating the operation screen upon receiving a touch on the operation screen with the touch panel;
The first display processing means displays the operation screen on the touch panel in a state where the image based on the secret information is not displayed when the operation screen includes an image based on the secret information defined in advance. ,
First transmission means for transmitting an image based on the secret information to a display device when the operation screen includes an image based on the secret information;
When receiving a touch on the operation screen with the touch panel, the touch panel further includes second transmission means for transmitting position information representing the touched position to the display device ,
The first display processing means includes an image based on the secret information that is used to determine whether or not the touch operation on the touch panel is performed when the image based on the secret information is not displayed. you longer than the threshold value when displaying the no operation screen, the image forming apparatus. A control program for an image forming apparatus having a touch panel,
A step of displaying the operation screen on the touch panel in a state in which the image based on the secret information is not displayed when the operation screen to be displayed includes an image based on pre-defined secret information on the touch panel;
Transmitting an image based on the secret information to a display device;
When receiving a touch on the operation screen with the touch panel, transmitting position information representing the touched position to the display device ;
When an operation screen that does not include an image based on the secret information is used as a threshold for determining whether or not the touch operation is performed on the touch panel in a state where the image based on the secret information is not displayed. And a step of causing the image forming apparatus to execute a step of making the value longer than the threshold value .

Images