JP5560672B2 - Information processing apparatus, information processing method, and recording medium recording information processing program - Google Patents

Description

  The present invention relates to a technique for supporting image acquisition by an image acquisition device.

  A scanning device, which is one of image acquisition devices, irradiates a paper document placed on an image reading surface with light, reads the document itself as an image, or reads information such as characters and graphics in the document. In addition, when a sticky note is attached to the document, the image acquisition device can acquire not only the information of the original but also the information of the sticky note. For example, Patent Document 1 discloses a technique for detecting a sticky note when a sticky note is attached to a document.

  Here, the sticky note paper can be attached not only on a document such as paper but also on a display on which information is displayed. For example, it is assumed that a screen whose operation method is unknown is displayed on the display of an information processing apparatus such as a personal computer. The user wants to attach a sticky note with a note attached to an unknown part on the screen displayed on the display and request an instruction from a third party regarding the unknown part. In such a case, the user wants to send data to a third party by attaching the data of the screen showing the state where the sticky note is attached to the mail. Therefore, for example, the user reads a screen with a sticky note attached by the image acquisition apparatus and sends the read data to a third party. As a result, the user can accurately notify an unknown part on the screen to a third party and receive an instruction. Especially in today's aging society, elderly people who are unfamiliar with the operation of the information processing apparatus are operating the information processing apparatus in many cases, and it is considered that such a situation often occurs.

JP 2007-159065

  When the image acquisition device reads a paper document, the paper document itself does not emit light, and the image acquisition device emits irradiation light to the paper document. On the other hand, when the image acquisition device performs a reading process on a display attached with a sticky note, for example, the image acquisition device not only emits irradiation light toward the display but also the display itself emits display light. . Therefore, the irradiation light emitted from the image acquisition device interferes with the display light emitted from the display, and image acquisition is not effectively executed. That is, the image acquisition device cannot recognize the sticky note on the display, or cannot acquire information described on the sticky note.

  Therefore, an object of the present invention is to provide a technique for assisting an image acquisition apparatus so as to acquire an image of a physical object placed on a display.

  An information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light, and a physical display in which a display having a screen and the image acquisition apparatus are arranged on the screen An optical interference control unit that controls display of the screen in a uniform state as the image of the object is acquired, and an image receiving unit that receives the acquired image acquired by the image acquisition device from the image acquisition device An information processing apparatus is provided.

  In the image acquisition device, it is possible to provide a technology for assisting in acquiring an image of a physical object arranged on a display.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. The schematic diagram which shows an example of the usage method of a scanning apparatus and information processing apparatus. 1 is an example of a block diagram illustrating a hardware configuration of a scanning device and an information processing device according to a first embodiment. The example of a display of the screen which an information processing apparatus displays on a display. Explanatory drawing which shows the state with which the sticky note paper which is the object of reading is attached to the screen. Explanatory drawing which shows a mode that a scanning apparatus reads the image of a mouse pointer. Explanatory drawing which shows a mode that a scanning apparatus acquires the image of the sticky note which is a target object. Explanatory drawing which shows a mode that the display of a screen is controlled in a scan bar area | region. Explanatory drawing which shows the relationship between the irradiation light B radiate | emitted from a scanning apparatus, and the display light a radiate | emitted from the screen of information processing apparatus. Explanatory drawing which shows a mode that a countermeasure object image is extracted from the scanning image which the scanning apparatus read. The example of a screen which shows a mode that the screen and the target object image are synthesize | combined. 1 is an example of a block diagram illustrating functional configurations of an information storage device and a scanning device according to a first embodiment. Explanatory drawing which shows an example of the detection process of a scanning apparatus Explanatory drawing which shows an example of the detection method of a scan bar area | region. 6 is a flowchart showing an example of the flow of overall processing executed by the information processing apparatus according to the first embodiment. 5 is a flowchart showing an example of a detection process flow of the scanning apparatus according to the first embodiment. The flowchart which shows an example of the flow of optical interference control A1 concerning the 1st embodiment. The flowchart which shows an example of the flow of optical interference control A2 concerning 1st Embodiment. The table | surface which shows the combination of the optical interference control of display light and irradiation light. 9 is a flowchart showing an example of the flow of optical interference control B according to Modification 1. FIG. 10 is a schematic diagram showing a relationship between an information processing apparatus including a touch panel according to Modification 3 and a scanning apparatus. The schematic diagram explaining the method to detect the position and inclination of a scanning apparatus using a touch panel. Explanatory drawing explaining the outline | summary of the process by the information processing apparatus 100 in case a target object is a photograph. Screen example (1) which shows the usage example of a synthesized image. Screen example (2) which shows the usage example of a synthesized image. Screen example (3) which shows the usage example of a synthesized image. Explanatory drawing explaining the outline | summary of the process which concerns on 2nd Example by an information processing apparatus, when a target object is a cellophane. The flowchart which shows an example of the flow of the optical interference control C concerning 2nd Embodiment.

  Hereinafter, a scanning apparatus will be described as an example of an image acquisition apparatus that acquires an image. Note that the scanning device refers to a device that sequentially irradiates a reading target with irradiation light to acquire an image or the like of the reading target.

  In the first embodiment, a physical object having low transparency will be described as an example of a reading target. As a physical object with low transparency, a sticky note is taken as an example. In the second embodiment, a physical object with high transparency will be described as an example of a reading target. Cellophane paper is taken as an example of a highly transparent physical object. Note that a physical object means an object that actually exists.

<First embodiment>
(1) External Appearance Configuration FIG. 1 is an external explanatory diagram of a scanning device and an information processing device according to a first embodiment.

  The information processing apparatus 100 includes a housing 115 in which a display 101, a mouse 102, a keyboard 103, a substrate for performing various processes, and the like are stored. The display 101 has a screen 101a for displaying various information such as images. The display 101 uses illumination light when displaying various information on the screen 101a. The mouse 102 and keyboard 103 accept input from the user. In the housing 115, various configurations such as a CPU (Central Processing Unit) 105, a ROM (Read Only Memory) 106, a RAM (Random Access Memory) 107, and a communication I / F (InterFace) 108 that constitute hardware described later. Is incorporated. In addition, the information processing apparatus 100 may include a speaker, headphones, a camera, and the like.

  The scanning device 200 includes a scanning mechanism 201, operation buttons 202, and an operation display 203. The scan mechanism 201 reads the image by sequentially scanning the irradiation light. For example, the scanning mechanism 201 includes an irradiation mechanism that generates and emits irradiation light, a scanning mechanism that sequentially scans the irradiation light by moving the irradiation mechanism, and an image acquisition mechanism that acquires an image. Hereinafter, the mechanism including the irradiation mechanism and the scanning mechanism is referred to as a scan bar.

  The operation buttons 202 include a start button for accepting scan start, a cancel button for accepting scan abort, and a setting button for accepting various scan settings. The setting buttons include buttons for various modes such as a screen reading mode and a document reading mode. In the present embodiment, as an example, when the user selects the setting button for the screen reading mode, the following processing is started.

  The display 203 displays various information such as current setting information and notification information that informs a scanning procedure.

  FIG. 2 is a schematic diagram illustrating an example of how to use the scanning apparatus and the information processing apparatus. The user places and uses the scanning device 200 on the screen 101a of the display 101 of the information processing apparatus 100 as shown in FIG. The user acquires a desired image by driving the scanning apparatus 200 in this state.

(2) Hardware Configuration FIG. 3 is an example of a block diagram illustrating a hardware configuration of the scanning device and the information processing device according to the first embodiment.

(2-1) Information Processing Device The information processing device 100 includes, for example, an input / output device controller 104, a CPU 105, a ROM 106, a RAM 107, and a communication I / F 108. These are connected to each other via a bus 109.

  The input / output device controller 104 is connected to the display 101, the mouse 102, the keyboard 103, and the like.

  The ROM 106 stores various control programs related to various controls described later performed by the information processing apparatus 100.

  The RAM 107 temporarily stores various control programs in the ROM 106 and information such as images. The RAM 107 temporarily stores information such as various flags according to the execution of various control programs.

  The CPU 105 develops various control programs stored in the ROM 106 in the RAM 107 and performs various controls. For example, the CPU 105 displays information such as an image on the screen 101 a of the display 101 via the input / output device controller 104. In addition, the CPU 105 receives user input operations from the mouse 102 and the keyboard 103, and receives user inputs via the input / output device controller 104.

  The communication I / F 108 performs communication such as transmission / reception of commands or data with the scanning device 200 based on the control of the CPU 105.

(2-2) Scan Device The scan device 200 includes, for example, an input / output device controller 204, a CPU 205, a ROM 206, a RAM 207, and a communication I / F 208. These are connected to each other via a bus 209.

  The input / output device controller 204 is connected to the scan mechanism 201, the operation buttons 202, the operation display 203, and the like.

  The ROM 206 stores various control programs executed by the scanning device 200.

  The RAM 207 temporarily stores information such as various control programs, images, and various flags in the ROM 206.

  The communication I / F 208 transmits / receives data such as commands and images to / from the information processing apparatus 100 based on the control of the CPU 205, for example.

  The CPU 205 develops various control programs stored in the ROM 206 in the RAM 207 and performs various controls. For example, the CPU 205 controls the scan mechanism 201 in response to an input from the operation button 202. In addition, the CPU 205 receives a command from the information processing apparatus 100 via the communication I / F 208 and controls the scan mechanism 201 according to the command. In addition, the CPU 205 acquires a scan image, a scan position of irradiation light, and the like from the scan mechanism 201 and outputs them to the information processing apparatus 100 via the communication I / F 208.

(3) Overview of Processing An overview of processing by the information processing apparatus 100 will be described using FIGS. 4A to 4G.

  FIG. 4A is a display example of a screen displayed on the display by the information processing apparatus. 4A, on the display 101 of the information processing apparatus 100, a mouse pointer 110 and a window 111 are displayed on the screen 101a. Here, the mouse pointer is a mark for indicating the operation position of the mouse 102 on the screen. A window is an independent small screen in the screen 101a, and is a screen capable of displaying an image or a document in each window. In the window 11, a search result is displayed as an example, and a plurality of URLs (Uniform Resource Locators) related to character strings used for the search are displayed.

  Here, it is assumed that even if the user sees the search result displayed in the window 111, for example, which URL should be accessed, or whether there is no problem even if accessed. Therefore, as shown in FIG. 4B, the user attaches a sticky note 112 (112a, 112b) on the screen 101a so as to correspond to an unknown part of the display information in the window 111. FIG. 4B is an explanatory diagram illustrating a state in which a sticky note as an object to be read is attached to the screen. Here, a sticky note 112a with a comment “I want to see this” is attached near URL3-CCCC. Also, a sticky note 112b with a comment “I don't know” is attached in the vicinity of URL5-EEEE. In this embodiment, the sticky note 112 is an object for image reading.

  Next, as shown in FIG. 4C, the user places the scanning apparatus 200 on the screen 101 a so that the mouse pointer 110 can be read together with the sticky note 112. Further, the user selects a setting button for the screen reading mode among the operation buttons 202. FIG. 4C is an explanatory diagram illustrating how the scanning device reads an image of a mouse pointer. Here, first, it is necessary to acquire information such as which position and position of the scanning device 200 is arranged on the screen 101a. The reason for acquiring the information is that it is necessary to control the display of the screen in the scan bar area 200b on the screen, as will be described later. In addition, the object image data on the sticky note 112 and the data only on the screen when the sticky note 112 is attached are finally displayed on the screen in a superimposed manner. Here, the scan bar region 200b is a region where the scan bar 200a is scanned on the screen 101a, that is, an irradiation region where the scanning device 200 irradiates irradiation light on the screen 101a. As described above, the scan bar 200a is a mechanism including an irradiation mechanism that generates and emits irradiation light and a scanning mechanism that sequentially scans the irradiation light by moving the irradiation mechanism.

  In order to acquire the position and inclination of the scanning device 200 on the screen 101a, the information processing device 100 uses a mouse pointer 110 displayed on the screen 101a here. Therefore, it is necessary to position the mouse pointer 110 as well as the sticky note 112 that is the object in the scan area 200c of the scanning device 200. When the mouse pointer 110 is not located in the scan area 200c, the user moves the mouse pointer 110 into the scan area 200c. Note that the information processing apparatus 100 may instruct the user to move the mouse pointer 110 into the scan area 200c. Here, the scan area 200c is an area where the scan bar 200a is scanned in a window provided in the scan apparatus 200 in the screen 101a as shown in FIG. 4C.

  After the scanning apparatus 200 is arranged as shown in FIG. 4C, the scanning apparatus 200 scans the scan bar 200a based on the scanning instruction command (I) from the information processing apparatus 100. Thereby, the scanning apparatus 200 reads the screen 101a including the mouse pointer 110 as an image. The scan instruction command (I) refers to a scan instruction command that is output when the position and inclination of the scanning device 200 on the screen 101a are acquired. On the other hand, a scanning instruction command (II) described later refers to a scanning instruction command that is output when an image of an object is read.

  In order to read the mouse pointer 110 clearly, the information processing apparatus 100 may adjust the color such as making the mouse pointer black. The information processing apparatus 100 detects the mouse pointer 110 from the scan image acquired by the scan apparatus 200, and acquires the position and inclination of the scan apparatus 200 on the screen 101a. This will be described later.

  Next, as illustrated in FIGS. 4D and 4E, the scanning apparatus 200 scans the scan bar 200a again based on the scanning instruction command (II) from the information processing apparatus 100. FIG. 4D is an explanatory diagram illustrating a state where the scanning apparatus acquires an image of a sticky note that is an object. FIG. 4E is an explanatory diagram showing a state in which screen display is controlled in the scan bar area. The information processing apparatus 100 controls the display of the screen of the scan bar area 200b to be uniform as the scan apparatus 200 scans the scan bar 200a. Controlling the screen display in a uniform state means controlling the screen uniformly, for example, in a black state or a white state by uniformly controlling the hue, saturation, brightness, and / or brightness of the screen, for example. Say. As shown in FIG. 4E, the display of the area immediately below the scan bar 200a in the screen 101a, that is, the display of the scan bar area 200b, is controlled to a uniform black state. Thereby, interference of light between the display light from the scan bar area 200b and the irradiation light from the scan device 200 is suppressed, and the scan device 200 can clearly read the sticky note 112 that is the object. Become. The information processing apparatus 100 grasps the scan bar area 200b based on the position and inclination of the scan apparatus 200 acquired previously. Further, the information processing apparatus 100 outputs a scanning instruction command (II) to the scanning apparatus 200 and acquires display information displayed on the screen 101a at that time. The display information displayed on the screen 101a is display information shown on the screen 101a in FIG. 4A here.

  FIG. 4F is an explanatory diagram illustrating a relationship between the irradiation light B emitted from the scanning device and the display light a emitted from the screen of the information processing device. The display 101 of the information processing apparatus 100 includes a screen 101a that displays various types of information and an illumination light generation unit 124b that generates illumination light used to display the various types of information. The illumination light generation unit 124b generates illumination light A and provides it to the screen 101a. The illumination light A is controlled when various types of information are displayed on the screen 101a, and is emitted from the screen 101a as display light a. On the other hand, the scanning device 200 emits the irradiation light B when reading the countermeasure. Therefore, when the scanning apparatus 200 reads the screen 101a, the irradiation light B emitted from the scanning apparatus 200 interferes with the display light a emitted from the screen, and it becomes difficult to read the object on the screen 101a. . Therefore, as illustrated in FIG. 4D, the information processing apparatus 100 controls the display of the scan bar region 200b to be uniform, thereby suppressing interference between the irradiation light B and the display light a. As a result, the scanning apparatus 200 can read an object. In practice, the scanning apparatus 200 irradiates the object with the irradiation light B, and acquires the image of the object by reading the reflected light b from the object. This reflected light b can also interfere with the display light a. However, for the sake of simplicity, the term “irradiation light” includes irradiation light B and reflected light b.

  FIG. 4G is an explanatory diagram illustrating a state in which a countermeasure image is extracted from a scanned image read by the scanning device. The information processing apparatus 100 receives the scanned image 113 read by the scanning apparatus 200. As shown in FIG. 4G, the scanned image 113 has a black background other than the portion corresponding to the sticky note 112a. This is because the display of the scan bar area 200b is controlled to a uniform black state as shown in FIG. 4E. The information processing apparatus 100 extracts an object image that is an image of the object from the scan image 113. Here, the information processing apparatus 100 extracts the sticky note images 112a_image and 112b_image from the scanned image 113. For example, the information processing apparatus 100 extracts the sticky note images 112a_image and 112b_image based on the contrast ratio between the sticky note images 112a_image and 112b_image and the background image. Further, the information processing apparatus 100 acquires a relative position occupied by the sticky note images 112a_image and 112b_image in the scan image 113.

  FIG. 4H is a screen example showing a state where the screen and the object image are combined. The information processing apparatus 100 acquires display information displayed on the screen 101a in advance. The information processing apparatus 100 superimposes the display information on the screen 101a and the sticky note images 112a_image and 112b_image, which are object images, to generate a composite image. At this time, the information processing apparatus 100 performs superposition based on the position and inclination of the scanning apparatus 200 on the screen 101a and the relative positions of the sticky note images 112a_image and 112b_image in the scanned image 113. In addition, the information processing apparatus 100 may create the operation diary 114 using a composite image as illustrated in FIG. 4H based on an input from the user.

(4) Functional Configuration Next, functional configurations of the scanning apparatus and the information processing apparatus will be described.

  FIG. 5 is an example of a block diagram illustrating functional configurations of the information storage device and the scanning device according to the first embodiment. First, the functional configuration of the scanning apparatus 200 will be described.

(4-1) Scan Device Processing by each functional unit of the scan device 200 is executed by the input / output device controller 204, the CPU 205, the ROM 206, the RAM 207, the communication I / F 208, and the like of the scan device 200 being mutually linked.

  The functional units of the scanning device 200 include, for example, a scan execution unit 210, an irradiation light control unit 211, an input reception unit 212, a transmission / reception unit 213, and the like.

(4-1-1) Transmission / Reception Unit The transmission / reception unit 213 receives various commands input from the information processing apparatus 100 and transmits them to each functional unit. The transmission / reception unit 213 transmits / receives data to / from each functional unit with the scan mechanism 201, the operation button 202, the operation display 203, and the like.

(4-1-2) Input Accepting Unit The input accepting unit 212 accepts user input from the operation buttons 202, the operation display 203, and the like via the transmission / reception unit 213. The input reception unit 212 outputs the received input to the scan execution unit 210 and / or the information processing apparatus 100. For example, when the input receiving unit 212 receives an input from a setting button for the screen reading mode, the input receiving unit 212 outputs the input to the information processing apparatus 100. At this time, when receiving the input of the screen reading mode, the information processing apparatus 100 generates commands such as scanning instruction commands (I) and (II) described later and outputs them to the scan execution unit 210.

(4-1-3) Scan Execution Unit The scan execution unit 210 executes image reading based on an input from the input reception unit 212 and / or a command from the information processing apparatus 100. For example, the scan execution unit 210 receives the scan instruction commands (I) and (II) generated by the information processing apparatus 100, controls the scan mechanism 201, and starts reading on the screen 101a. Further, the scan execution unit 210 transmits the read scanned image to the information processing apparatus 100.

  The scan execution unit 210 outputs the scan position of the scan bar 200a to the information processing apparatus 100 so that the information processing apparatus 100 can acquire the position and inclination of the scan apparatus 200. The scan bar area 200b is an area below the scan bar 200a in the screen 101a.

  The scan position of the scan bar 200a is represented by, for example, any coordinate S (α1, β0) in the scan bar 200a shown in FIG. 4C. Here, as an example, the coordinates S (α1, β0) of the right corner of the scan bar 200a are used. Further, the coordinates S (α1, β0) are relative coordinates when the right corner in the scan bar 200a is set as the origin coordinates S (α0, β0), for example, as shown in FIG. 4C. In the following, for the scan region 200c, a coordinate system based on the scanning device 200 side is referred to as a scan coordinate system. S is a suffix indicating that the coordinates are based on the scan coordinate system. Note that the α direction of the coordinate axis of the scan coordinate system is a direction in which the scan bar 200a advances from the origin coordinate S (α0, β0) to the end point coordinate S (αE, βE), as shown in FIG. 4C. Further, the β direction of the coordinate axis of the scan coordinate system is a direction substantially orthogonal to the α direction.

(4-1-4) Irradiation Light Control Unit The irradiation light control unit 211 controls irradiation light emitted when reading an image based on the control of the scan execution unit 210. Or the irradiation light control part 211 receives the irradiation light control command which controls irradiation light, and controls irradiation light according to an irradiation light control command.

(4-2) Information processing apparatus The processing by each functional unit of the information processing apparatus 100 is executed by the input / output device controller 104, the CPU 105, the ROM 106, the RAM 107, the communication I / F 108, etc. of the information processing apparatus 100 being mutually linked. Is done.

  The functional unit of the information processing apparatus 100 includes the following configuration.

(4-2-1) Transmission / Reception Unit The transmission / reception unit 120 transmits / receives various data and commands between the scanning apparatus 200 and the information processing apparatus 100. For example, the transmission / reception unit 120 receives various data from the scanning device 200 and transmits the data to each functional unit in the information processing device 100. In addition, the transmission / reception unit 120 outputs various commands from the information processing apparatus 100 to the scanning apparatus 200.

(4-2-2) Input Reception Unit The input reception unit 121 receives user input from the mouse 102, the keyboard 103, and the like via the transmission / reception unit 120. For example, the input reception unit 121 receives the movement of the mouse 102 by the user and outputs it to the display management unit 124. Accordingly, the display management unit 124 displays the mouse pointer 110 on the screen 101a following the movement of the mouse 102. In addition, for example, the input reception unit 121 may receive activation of various processes from the user and cause the display management unit 124 to create the operation diary 114 as illustrated in FIG. 4H described above.

(4-2-3) Instruction Control Unit The instruction control unit 122 generates various commands for controlling the scanning apparatus 200 and the information processing apparatus 100, and controls the entire process.

  For example, when the input receiving unit 212 of the scanning apparatus 200 receives the screen reading mode, the instruction control unit 122 first outputs a scanning instruction command (I) to the scanning apparatus 200 in order to start detection processing of the scanning apparatus 200. In addition, the instruction control unit 122 outputs a scanning device detection command that instructs detection processing of the scanning device 200 to the scanning device detection unit 126. The detection processing of the scanning device 200 is processing for acquiring the position and inclination of the scanning device 200 with respect to the screen 101a. In the present embodiment, the scanning device 200 acquires an image including the mouse pointer 110 displayed on the screen 101a, as shown in FIG. 4C described above, based on the scanning instruction command (I). Note that the instruction control unit 122 may instruct the display management unit 124 to adjust the color such as blacking the mouse pointer in order to read the mouse pointer 110 clearly.

  Here, the instruction control unit 122 instructs the display management unit 124 to display a warning to the user on the screen 101a when the position and tilt of the scanning apparatus 200 cannot be detected. For example, the display management unit 124 displays a warning message on the screen 101a instructing to move the mouse pointer 110 into the scan area 200c.

  Next, after the detection process of the scanning device 200, the instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200 in order to acquire an image of the sticky note 112 that is the object. At this time, the instruction control unit 122 outputs a light interference control command to the light interference control unit 123, and controls the display to be uniform in the scan bar area 200b of the screen 101a. Accordingly, as illustrated in FIGS. 4D and 4E, when the scanning apparatus 200 reads the sticky note 112, the scan bar area 200b is controlled to be in a black state, for example.

  In addition, the instruction control unit 122 outputs a scanning instruction command (II), outputs a screen acquisition command to the screen acquisition unit 134, and outputs an extraction command to the extraction unit 132. The screen acquisition command is a command for instructing to acquire screen data of the screen 101a, that is, display information displayed on the screen 101a. Here, the screen data of the screen 101a is the display information shown in FIG. 4A. The extraction command is a command for instructing to extract an image of an object from a scanned image read by the scanning apparatus 200.

  In addition, when the instruction control unit 122 acquires the determination result of the scan image from the image determination unit 131, the instruction control unit 122 may output the scan instruction command (II) and the optical interference control command again according to the determination result. Thereby, the scanning apparatus 200 reads an image again in a state where the display of the screen 101a is controlled.

(4-2-4) Screen acquisition unit, screen DB
When receiving the screen acquisition command from the instruction control unit 122, the screen acquisition unit 134 acquires screen data constituting the screen 101a from the display management unit 124 at that time. Alternatively, the screen acquisition unit 134 may acquire the screen 101a by a so-called screen capture function.

  The screen acquisition unit 134 stores the acquired screen data in the screen DB 135.

(4-2-5) Scanned image receiving unit, scanned image DB
The scanned image receiving unit 129 receives the scanned image read by the scanning device 200 and stores it in the scanned image DB 130. Here, the scan image includes a scan image including the mouse pointer 110 corresponding to the scan instruction command (I) and a scan image including the object image of the sticky note 112 corresponding to the scan instruction command (II). It is.

(4-2-6) Mouse pointer information DB
The mouse pointer information DB 125 stores mouse pointer information such as the shape and color of the mouse pointer 110. In addition, the mouse pointer information DB 125 acquires and stores the current coordinates I (X1, Y1) of the mouse pointer 110 from the display management unit 124. The coordinates I (X1, Y1) are, for example, the coordinates of the tip of the mouse pointer 110 as shown in FIG. 4C. For example, as shown in FIG. 4C, the coordinates I (X1, Y1) are relative coordinates when the right corner in the screen 101a is the origin coordinate I (0, 0). Hereinafter, for the screen 101a, a coordinate system with the information processing apparatus 100 side as a reference is referred to as a screen coordinate system. I is a subscript indicating that the coordinates are based on the screen coordinate system. Note that the X direction of the coordinate axes of the screen coordinate system is a direction from the origin coordinate I (0, 0) to the left in FIG. 4C, and the Y direction is a direction from the origin coordinate I (0, 0) to the bottom.

(4-2-7) Scanning device information DB
The scan device information DB 128 stores scan area information such as the width W1 and length L1 of the scan area 200c and scan bar information such as the width W2 and length L2 of the scan bar 200a in advance. Further, the scan device information DB 128 acquires and stores coordinates indicating the current scan position of the scan bar 200a from the scan execution unit 210 of the scan device 200.

(4-2-8) Scan Device Detection Unit Upon receiving the scan device detection command from the instruction control unit 122, the scan device detection unit 126 detects the position and inclination of the scan device 200 on the screen 101a. That is, the position and inclination of the scanning device 200 in the screen coordinate system are detected. Therefore, the scanning device detection unit 126 accesses the scan image DB 130 and acquires a scan image including the mouse pointer 110 as the scan bar 200a is sequentially scanned. In parallel with this, the scanning device detection unit 126 sequentially acquires the scanning position of the scanning bar 200a specified by the scanning coordinate system from the scanning device information DB 128.

  Further, the scanning device detection unit 126 detects the mouse pointer 110 from the scanned image. For example, the scanning device detection unit 126 acquires mouse pointer information from the mouse pointer information DB 125, and detects the mouse pointer 110 by comparing the mouse pointer information with the scanned image. That is, the scanning device detection unit 126 detects a shape similar to the shape of the mouse pointer indicated by the mouse pointer information from the scan image. Here, the scanning device detection unit 126 detects the mouse pointer 110 from the scan image based on the scan image when the scan bar 200a is scanned from the scanning start point to the tip of the mouse pointer 110.

  Next, the scanning device detection unit 126 acquires the coordinates of the tip of the mouse pointer 110 on the screen 101a specified by the screen coordinate system from the mouse pointer information.

  Through the above processing, the scanning device detection unit 126 determines the scan position of the scan bar 200a in the scan coordinate system when the tip of the mouse pointer 110 is detected, and the coordinates of the tip of the mouse pointer 110 in the screen coordinate system. Have acquired. Based on these pieces of information, the scanning device detection unit 126 detects the position and inclination of the scanning device 200 in the screen coordinate system.

  An example of the detection process of the scanning apparatus 200 will be specifically described with reference to FIG. FIG. 6 is an explanatory diagram illustrating an example of detection processing of the scanning device. 6 corresponds to the state in which the scanning device 200 reads the mouse pointer 110 in FIG. 4C described above.

  In FIG. 6, the scan bar 200a has moved from the origin coordinate S (α0, β0) to the coordinate S (α1, β0) in the scan coordinate system. Therefore, the scan image DB 130 stores scan images from the origin coordinate S (α0, β0) to the coordinate S (α1, β0). It is assumed that the scan bar 200a specified by the coordinates S (α1, β0) is located on the tip of the mouse pointer 110.

  When detecting the tip of the mouse pointer 110 from the scan image, the scan device detection unit 126 acquires the coordinates S (α1, β0) of the scan bar 200a at that time from the scan device information DB 128. As described above, the coordinates S (α1, β0) are relative coordinates when the right corner of the scan coordinate system is the origin coordinate S (α0, β0). The scanning device detection unit 126 calculates the coordinates S (α1, β1) of the tip of the mouse pointer 110 in the scan coordinate system from the positional relationship in the scan image between the coordinates S (α1, β0) and the tip of the mouse pointer 110. To do. For example, the scanning device detection unit 126 calculates the coordinates S (α1, β1) based on the distance b between the coordinates S (α1, β0) and the tip of the mouse pointer 110. Here, the relationship is β1 = β0 + b.

  Further, when the tip of the mouse pointer 110 is detected from the scan image, the scanning device detection unit 126 acquires the coordinates I (X1, Y1) of the tip of the mouse pointer 110 in the screen coordinate system from the mouse pointer information DB 125. As described above, the coordinates I (X1, Y1) are relative coordinates when the right corner of the screen coordinate system is the origin coordinate I (0, 0). Thereby, the scanning device detection unit 126 corresponds the coordinates of the coordinates S (α1, β1) in the scan coordinate system and the coordinates I (X1, Y1) in the screen coordinate system for the tip coordinates of the mouse pointer 110. Get relationship. That is, the correspondence between the coordinates of the scan coordinate system and the coordinates of the screen coordinate system is acquired.

  Next, the scanning device detection unit 126 converts the previously acquired coordinates S (α1, β0) of the scan bar 200a into coordinates in the screen coordinate system based on the correspondence relationship of the coordinates. The coordinates S (α1, β0) are converted into coordinates I (X2, Y2). Similarly, the scanning device detection unit 126 converts the origin coordinate S (α0, β0) in the scan coordinate system into the coordinate I (X0, Y0) in the screen coordinate system. From the coordinates I (X0, Y0) and the coordinates I (X2, Y2), the scanning direction α with respect to the screen 101a of the scan bar 200a can be acquired. The scanning device detection unit 126 sets an angle Δθ formed by the scanning direction α and the horizontal direction on the screen 101 a as an inclination Δθ on the screen 101 a of the scanning device 200.

  Further, the scanning device detection unit 126 acquires the coordinates of the four corners of the scan region 200c based on the coordinates I (X0, Y0), the inclination Δθ, the width W1 and the length L1 of the scan region 200c, and the like. The coordinates of the four corners are coordinates I (X0, Y0), coordinates I (X3, Y3), coordinates I (X4, Y4), and coordinates I (X5, Y5). Thereby, the scanning device detection unit 126 grasps the position of the scanning region 200c with respect to the screen 101a, and grasps the position of the scanning device 200 with respect to the screen 101a.

  The scanning device detection unit 126 outputs the acquired position and inclination Δθ of the scanning device 200 or the corresponding relationship of the coordinates between the scan coordinate system and the screen coordinate system to the composite image generation unit 133 and the scan bar region detection unit 127. .

  Further, if the mouse pointer 110 cannot be detected from the scanned image even after the scanning by the scan bar 200a is completed, the scanning device detection unit 126 outputs that fact to the instruction control unit 122. Accordingly, the instruction control unit 122 causes the display management unit 124 to display a warning message to the user on the screen 101a.

  In the above description, the position and inclination Δθ of the scanning device 200 are calculated using the coordinates of the tip of the mouse pointer 110, but the present invention is not limited to this as long as the coordinates can identify the position of the mouse pointer 110.

(4-2-9) Scan Bar Area Detection Unit FIG. 7 is an explanatory diagram showing an example of a scan bar area detection method.

  The scan bar area 200b on the screen 101a also varies depending on which position and inclination Δθ the scanning apparatus 200 reads with respect to the screen 101a. Therefore, the scan bar area detection unit 127 acquires the scan bar area 200b as follows based on the position and inclination of the scan apparatus 200 with respect to the screen 101a and the scan position of the scan bar 200a that moves sequentially. .

  The scan bar area detection unit 127 acquires the current coordinates S (α2, β0) of the scan bar 200a and the width W2 and length L2 of the scan bar 200a from the scan device information DB 128. Further, the scan bar area detection unit 127 acquires the coordinates of the four corners of the scan bar area 200b in the scan coordinate system based on the coordinates S (α2, β0), the width W2, and the length L2. The coordinates of the four corners in the scan coordinate system are coordinates S (α2, β0), coordinates S (α2, β0 + L2), coordinates S (α2 + W2, β0), and coordinates S (α2 + W2, β0 + L2). Note that the length L1 of the scan area 200c and the length L2 of the scan bar area 200b are the same.

  Next, the coordinates of the four corners in the scan coordinate system of the scan bar 200a are converted into the coordinates of the screen coordinate system based on the correspondence relationship between the scan coordinate system and the screen coordinate system obtained by the scanning device detection unit 126. To do. Note that the scan bar area detection unit 127 can acquire the position and inclination Δθ of the scanning device 200 from the scanning device detection unit 126 and perform coordinate conversion based on the position and inclination Δθ. The coordinates of the four corners in the scan coordinate system are converted into coordinates I (X6, Y6), coordinates I (X7, Y7), coordinates I (X8, Y8), and coordinates I (X9, Y9), respectively, in the screen coordinate system. . Accordingly, the scan bar area detection unit 127 can detect the scan bar area 200b on the screen 101a and outputs the detection result to the optical interference control unit 123.

(4-2-10) Image Determination Unit The image determination unit 131 reads a scan image including an object image from the scan image DB 130 and determines whether or not the image of the object is clearly read. For example, the image determination unit 131 determines whether the contrast ratio between the sticky note 112, which is the object, and a portion other than the sticky note 112 is equal to or greater than a predetermined value. The image determination unit 131 determines that the edge of the sticky note 112 has been detected when the contrast ratio is equal to or greater than the predetermined value, and conversely, when the contrast ratio is less than the predetermined value, can detect the edge of the sticky note 112. Judge that there was no. The edge of the sticky note 112 is the outline of the sticky note 112, and is the boundary between the sticky note 112 and the others.

  The image determination unit 131 outputs the determination result to the instruction control unit 122 and the extraction unit 132.

(4-2-11) Optical Interference Control Unit The optical interference control unit 123 receives the optical interference control command from the instruction control unit 122, and determines the coordinates of the scan bar area 200b in the screen coordinate system from the scan bar area detection unit 127. get. Based on this, the optical interference control unit 123 instructs the display management unit 124 to control the display of the screen of the scan bar area 200b in the entire screen 101a to a uniform state.

  Controlling the screen display in a uniform state means controlling the screen uniformly, for example, in a black state or a white state by uniformly controlling the hue, saturation, brightness, and / or brightness of the screen, for example. Say. For example, when the sticky note paper is white paper and the screen is controlled to be in a black state, the background color of the sticky note paper is in a black state, so that the difference between the object and the screen can be clarified. The black state includes not only black but also colors close to black such as dark gray and light gray. Similarly, the white state includes not only white but also lightness or luminance close to bright white, such as yellow and bright green. The display on the screen can be controlled by adjusting the amount of display light emitted from the screen, for example. Alternatively, the display of the screen can be controlled uniformly by controlling the display color of the screen, such as controlling the screen to output white.

  Therefore, for example, the optical interference control unit 123 suppresses the amount of display light emitted from the scan bar area 200b so that the scan bar area 200b is in a black state, or outputs black to the scan bar area 200b. The display management unit 124 is instructed. Further, the light interference control unit 123 increases the amount of display light emitted from the scan bar area 200b so that the scan bar area 200b is in a white state, or performs display management so that white is output to the scan bar area 200b. The unit 124 is instructed.

  By such control, interference between the irradiation light emitted from the scanning apparatus 200 and the display light emitted from the screen 101a of the information processing apparatus 100 when reading the object can be suppressed. Therefore, even if the target object is arranged on the screen 101a of the display 101, the scanning apparatus 200 can read the target object.

  Note that if the display of the entire screen 101a is controlled from a normal display state to a uniform state, the screen state changes instantaneously, which gives a strong stimulus to the user of the information processing apparatus 100. Such a change gives a great surprise to the user, for example, and causes the user to mistakenly recognize that the information processing apparatus 100 has malfunctioned. By controlling the display only in the scan bar area 200b, it is possible to suppress the stimulus given to the user as compared with the case where the display of the entire screen 101a is controlled.

  The optical interference control unit 123 may acquire the position of the scan area 200c from the scan device detection unit 126 and control the display of the scan area 200c. Further, the optical interference control unit 123 may control the display of the entire screen 101a regardless of the position of the scanning device 200 or the like.

  Further, the optical interference control unit 123 may control the display of the screen of the scan bar area 200b according to the determination result of the image determination unit 131. For example, when the image determination unit 131 determines that the edge of the sticky note 112 cannot be detected in the scanned image, the optical interference control unit 123 receives a new optical interference control command from the instruction control unit 122. The optical interference control unit 123 controls the display management unit 124 according to a new optical interference control command, and changes the display state of the screen. For example, the display state of the screen is changed from a black state to a white state. In response to the scanning instruction command from the instruction control unit 122, the scanning apparatus 200 executes the reading of the object again with the screen display state changed. In this way, by adjusting the display of the screen 101a in various ways, the scanning apparatus 200 can be adjusted so that the image of the object can be acquired more clearly.

(4-2-12) Display Management Unit The display management unit 124 includes a display control unit 124a, an illumination light generation unit 124b, and an image signal generation unit 124c. The illumination light generation unit 124b generates illumination light that gives brightness or luminance to the screen 101a. The image signal generation unit 124c generates an image signal for displaying information such as an image on the screen 101a. The screen includes a plurality of pixels that determine colors such as RGB, and the image signal generation unit 124c generates an image signal for each pixel, for example. The display control unit 124a displays information such as an image on the screen 101a using illumination light and / or image signals. In addition, the display control unit 124a controls the illumination light and / or the image signal according to the control of the light interference control unit 123, and controls the display of the scan bar area 200b in the screen 101a to be uniform.

  Screen display can be controlled by adjusting the amount of display light emitted from the screen or by controlling the display color of the screen.

  First, as a method of controlling the display of the screen by adjusting the emission amount of the display light, for example, the following method can be cited. The display control unit 124 a adjusts the amount of illumination light blocked in the scan bar region 100 b in accordance with the control of the light interference control unit 123, and finally adjusts the amount emitted from the screen as display light.

  For example, when the display 101 is a liquid crystal display device, when the display control unit 124a applies an image signal to the pixel electrode of each pixel, the liquid crystal tilts according to the image signal. The amount of illumination light blocked is controlled by the degree of tilt of the liquid crystal, and consequently the amount of display light emitted from the screen 101a is controlled. Therefore, the display control unit 124a controls the image signal of each pixel in the scan bar region 100b in accordance with the control of the light interference control unit 123, and adjusts the amount of display light emitted from the scan bar region 100b. . For example, by reducing the amount of emitted illumination light, the display on the screen can be in a uniform black state. Further, by increasing the emission amount of illumination light, the display on the screen can be brought close to a uniform white state.

  In addition, as a method for controlling the emission amount of the display light, adjusting the generation amount of the illumination light in the illumination light generation unit 124b can be cited. This method is applicable when the display 101 is a CRT (Cathode Ray Tube) or the like.

  Next, examples of a method for controlling the display of the screen by adjusting the display color of the screen include the following methods. Here, by applying an image signal to each pixel, RGB of each pixel is determined, and a display color of the screen is determined. Therefore, the display control unit 124 a controls the image signal corresponding to the pixels in the scan bar region 100 b and the display color in the scan bar region 100 b in accordance with the control of the light interference control unit 123.

  In addition, the display management unit 124 displays the mouse pointer 110 on the screen 101a in accordance with the movement of the mouse 102, and stores the current coordinates I (X1, Y1) of the mouse pointer 110 in the mouse pointer information DB 125. Further, the display management unit 124 processes the composite image from the composite image generation unit 133 based on the execution of various processes received by the input reception unit 121, and creates the operation diary 114 as shown in FIG. 4H described above. May be.

(4-2-13) Extraction Unit When receiving the extraction command from the instruction control unit 122, the extraction unit 132 starts the next process.

  The extraction unit 132 acquires the edge determination result of the sticky note 112 from the image determination unit 131. Further, the extraction unit 132 reads a scan image including an object from the scan image DB 130, extracts an image of the sticky note 112 from the scan image based on the edge, and acquires a relative position of the sticky note 112 with respect to the scan image. The extraction unit 132 outputs the image of the sticky note 112, which is the object, and the relative position with respect to the scanned image to the composite image generation unit 133.

(4-2-14) Composite Image Generation Unit The composite image generation unit 133 acquires the position of the scan region 200c from the scan device detection unit 126. Note that the composite image generation unit 133 may acquire the correspondence relationship between the coordinates of the scan coordinate system and the screen coordinate system from the scan device detection unit 126, and may grasp the scan region 200c based on this.

  In addition, the composite image generation unit 133 acquires the relative position with respect to the image of the sticky note 112 and the scan image from the extraction unit 132, and acquires the screen data displayed on the screen 101a from the screen DB 135. Finally, the composite image generation unit 133 combines the image of the sticky note 112 with the screen data of the screen 101a based on the scan area 200c on the screen 101a and the relative position of the sticky note 112 with respect to the scan image. Generate an image.

  The composite image generation unit 133 outputs the composite image to the display management unit 124, and displays the composite image on the screen 101a.

(5) Process Flow (5-1) Overall Process FIG. 8 is a flowchart illustrating an example of the overall process flow executed by the information processing apparatus according to the first embodiment.

  Step S1: When receiving the screen reading mode from the scanning device 200, the instruction control unit 122 executes the following processing.

  Step S2: The instruction control unit 122 first outputs a scanning instruction command (I) to the scanning device 200 in order to start the detection process of the scanning device 200. In addition, the instruction control unit 122 outputs a scanning device detection command to the scanning device detection unit 126. The scan device detection unit 126 executes a detection process of the scan device 200.

  Step S3: Next, the instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200 in order to acquire an image of the sticky note 112 that is the object. In addition, the instruction control unit 122 outputs an optical interference control command to the optical interference control unit 123, and controls display of the screen 101a. The optical interference control unit 123 may control the display of the screen 101a according to the determination result of the scanned image in the image determination unit 131.

  At this time, the screen acquisition unit 134 receives the screen acquisition command from the instruction control unit 122, acquires the screen data of the screen 101a, and stores it in the screen DB 135.

  Step S4: When the extraction unit 132 receives the extraction command from the instruction control unit 122, the extraction unit 132 extracts an image of the target object from the scanned image acquired by the scanning device 200. Specifically, the extraction unit 132 acquires the determination result of the edge of the object from the image determination unit 131 and reads a scan image including the object from the scan image DB 130. Next, the extraction unit 132 extracts an image of the object from the scan image based on the edge, and acquires a relative position of the object image with respect to the scan image. The extraction unit 132 outputs the object image and the relative position of the object image with respect to the scan image to the composite image generation unit 133.

  Step S5: The composite image generation unit 133 acquires the position and inclination Δθ of the scanning device 200 from the scanning device detection unit 126. In addition, the composite image generation unit 133 acquires the target image and the relative position of the target image with respect to the scan image from the extraction unit 132, and acquires screen data of the screen 101a from the screen DB 135. Based on the acquired information, the composite image generation unit 133 combines the image of the object with the image on the screen 101a to generate a composite image. The display management unit 124 displays the composite image on the screen 101a.

(5-2) Scanning Device Detection Processing FIG. 9 is a flowchart illustrating an example of the flow of scanning device detection processing according to the first embodiment. Hereinafter, the flow of the detection process of the scanning device among the above-described overall processes will be described.

  Step S2a: The instruction control unit 122 first outputs a scanning instruction command (I) to the scanning apparatus 200 in order to start detection processing of the scanning apparatus 200. In addition, the instruction control unit 122 outputs a scanning device detection command to the scanning device detection unit 126.

  Step S2b: When the scanning device detection unit 126 detects the mouse pointer 110 from the scanned image, the scanning device detection unit 126 proceeds to step S2c. Here, the scanning device detection unit 126 detects the tip of the mouse pointer 110. On the other hand, if the scanning device detection unit 126 does not detect the mouse pointer 110, the scanning device detection unit 126 proceeds to step S2i.

  Step S2c: When detecting the tip of the mouse pointer 110 from the scanned image, the scanning device detection unit 126 acquires the coordinates S (α1, β0) of the scan bar 200a at that time from the scanning device information DB 128. Based on the coordinates S (α1, β0), the scanning device detection unit 126 obtains coordinates S (α1, β1) in the scan coordinate system at the tip of the mouse pointer 110.

  Step S2d: The scanning device detection unit 126 acquires the coordinate I (X1, Y1) of the screen coordinate system in the screen 101a from the mouse pointer information DB 125 for the tip of the mouse pointer 110.

  Step S2e: The scanning device detection unit 126 acquires the correspondence between the coordinates of the scan coordinate system and the coordinates of the screen coordinate system based on the coordinates S (α1, β1) and the coordinates I (X1, Y1).

  Step S2f: Next, the scanning device detection unit 126 acquires the scanning direction α of the scan bar 200a with respect to the screen 101a using the correspondence relationship of the coordinates.

  Step S2g: The scanning device detection unit 126 calculates an angle Δθ formed by the scanning direction α and the horizontal direction on the screen 101a as an inclination Δθ on the screen 101a of the scanning device 200.

  Step S2h: The scanning device detection unit 126 converts the origin coordinate S (α0, β0) of the scan coordinate system into the coordinate I (X0, Y0) of the screen coordinate system. Further, the scanning device detection unit 126 grasps the position of the scanning area 200c with respect to the screen 101a based on the coordinates I (X0, Y0), the inclination Δθ, the width W1 and the length L1 of the scanning area 200c, and the like. Thereby, the position with respect to the screen 101a of the scanning apparatus 200 can be grasped.

  Step S2i: The scanning device detection unit 126 determines whether or not the scan bar 200a has reached the end point of the scan area 200c. If the scan bar 200a has reached the end point, the process proceeds to step S2j. Otherwise, the process returns to step S2b, and the scanning device detection unit 126 detects the mouse pointer 110 from the scanned images sequentially transmitted from the scanning device 200 along with the scanning of the scanning bar 200a.

  Step S2j: The display management unit 124 displays a warning message for the user on the screen 101a.

(5-3) Optical Interference Control Hereinafter, the flow of optical interference control in the overall processing described above will be described. Below, each of optical interference control A1 and A2 is demonstrated.

  In the optical interference control A1, the display of the scan bar area 200b in the screen 101a is controlled to be in a black state or a white state.

  In the optical interference control A2, the display of the entire screen 101a is controlled to a black state or a white state.

(5-3-1) Optical interference control A1
FIG. 10 is a flowchart illustrating an example of the flow of the optical interference control A1 according to the first embodiment.

  Step S31a: The instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200 in order to acquire an image of the object. In addition, the instruction control unit 122 outputs a light interference control command to the light interference control unit 123, and starts display control of the scan bar area 200b in the screen 101a.

  Step S31b: The scan bar area detecting unit 127 acquires the coordinates S (α2, β0) of the current scan bar 200a from the scan device information DB 128.

  Step S31c: The scan bar area detection unit 127 acquires the coordinates of the four corners of the scan bar area 200b in the scan coordinate system based on the coordinates S (α2, β0) of the current scan bar 200a. Next, the scan bar area detection unit 127 converts the coordinates of the four corners into the coordinates of the screen coordinate system, and detects the scan bar area 200b on the screen 101a.

  Step S31d: The light interference controller 123 controls the display of the scan bar area 200b to a black state.

  Step S31e: The scanned image receiving unit 129 receives the scanned image read by the scanning device 200 and stores it in the scanned image DB 130.

  Step S31f: The information processing apparatus 100 proceeds to step 31g if reading by the scanning apparatus 200 is completed, and returns to step S31b if not completed.

  Step S31g: The image determination unit 131 determines whether or not the edge of the object can be detected in the scanned image. If the edge of the object cannot be detected, the process proceeds to step 31h. If the edge of the object is detected, the process is terminated.

  Steps S31h to 31j: The instruction control unit 122 outputs the scanning instruction command (II) and the optical interference control command again (S31h). The scan bar area detection unit 127 detects the scan bar area 200b in the same manner as steps S31b and S31c (S31i, S31j).

  Step S31k: The light interference controller 123 changes the display of the scan bar area 200b from a black state to a white state.

  Step S31l: The scanned image receiving unit 129 receives the scanned image and stores it in the scanned image DB 130.

  Step S31m: The information processing apparatus 100 returns to step S31i if reading by the scanning apparatus 200 is not completed. If the reading is finished, the process is finished.

(5-3-2) Optical interference control A2
FIG. 11 is a flowchart showing an example of the flow of the optical interference control A2 according to the first embodiment. It differs from the light interference control A1 in that the entire screen is controlled to a black state or a white state.

  Step S32a: The instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200. In addition, the instruction control unit 122 outputs an optical interference control command to the optical interference control unit 123, and starts display control of the entire screen 101a.

  Step S32b: The optical interference control unit 123 controls the display of the entire screen 101a to a black state.

  Steps S32c and S32d: The information processing apparatus 100 continues to acquire and store a scanned image until reading by the scanning apparatus 200 is completed.

  Steps S32e and 32f: When the image determination unit 131 cannot detect the edge of the object in the scanned image (S32e), the instruction control unit 122 outputs the scanning instruction command (II) and the optical interference control command again (S32e). S31f). If the edge of the object is detected, the process is terminated.

  Step S32g: The light interference control unit 123 changes the display of the entire screen 101a from the black state to the white state.

  Steps S32h and S32i: The information processing apparatus 100 continues to acquire and store a scanned image until reading by the scanning apparatus 200 is completed.

  In the optical interference controls A1 and A2, the display is changed from the black state to the white state. However, the display may be changed from the white state to the black state. Furthermore, it is only necessary to acquire the object image of the object arranged on the screen 101a, and the display control is not limited to the black state or the white state.

(6) Effects As described above, by uniformly controlling the display of the screen 101a, the irradiation light emitted from the scanning device 200 and the light emitted from the screen 101a of the information processing device 100 at the time of image acquisition. Interference with display light can be suppressed. Thereby, even if the target object is arranged on the screen 101 a of the display 101, the target object can be read by the scanning device 200.

  Further, by controlling the display of only the scan bar area 200b irradiated with the irradiation light, it is possible to suppress the stimulus given to the user as compared with the case where the entire display of the screen 101a is controlled.

(7) Modification (7-1) Modification 1
In the above-described embodiment, the interference between the display light from the screen 101a and the irradiation light from the scanning device 200 is suppressed by controlling the display of the screen 101a. In this modification, the information processing apparatus 100 may not only control the display of the screen 101a but also control the irradiation light of the scanning apparatus 200.

  FIG. 12 is a table showing combinations of optical interference control between display light and irradiation light. In FIG. 12, the display on the screen 101a is adjusted in a plurality of stages depending on the amount of display light emitted from the screen 101a. Furthermore, the irradiation light from the scanning device 200 is adjusted in a plurality of stages depending on the difference in the amount of emission. Here, as an example, nine levels of optical interference control levels from Control 1 to Control 9 are prepared.

  FIG. 13 is a flowchart illustrating an example of the flow of optical interference control B according to the first modification. Next, the optical interference control B of this modification will be described.

  Step S33a: The instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200. In addition, the instruction control unit 122 outputs an optical interference control command to the optical interference control unit 123 to start display control of the screen 101a. In the case of this modification, the instruction control unit 122 further outputs an optical interference control command to the scanning device 200 to start control of irradiation light. Here, it is assumed that the light control command includes, for example, information instructing which of control 1 to control 9 in FIG.

  Steps S33b and S33c: Similar to steps S31b and S31c of the optical interference control A1 described above, the scan bar area detection unit 127 detects the scan bar area 200b on the screen 101a.

  Step S33d: The optical interference control unit 123 controls the display of the scan bar area 200b according to the optical interference control command. On the other hand, when receiving the optical interference control command from the information processing apparatus 100, the irradiation light control unit 211 of the scanning device 200 adjusts the emission amount of irradiation light according to the optical interference control command.

  For example, when the control 6 is specified in the optical interference control command, the optical interference control unit 123 performs control so that the amount of display light emitted from the screen 101a becomes level 3. Further, the irradiation light control unit 211 of the scanning device 200 controls the illumination light emission amount to be level 2.

  Steps S33e, S33f: The information processing apparatus 100 continues to acquire and store a scanned image until reading by the scanning apparatus 200 is completed.

  Step S33g: The image determination unit 131 determines whether or not the edge of the object can be detected in the scanned image. If the edge of the object cannot be detected, the process proceeds to step 33h. If the edge of the object is detected, the process is terminated.

  Step S33h: The instruction control unit 122 determines whether or not the optical interference control has been completed for all combinations. If the optical interference control has not ended, the instruction control unit 122 returns to step S33a. In step S33a, the instruction control unit 122 generates an optical interference control command designating control that is not executed in FIG. 12, and outputs the command to each functional unit.

  In this modification, the display of the screen 101a is controlled by adjusting the emission amount of the display light. However, the display of the screen 101a may be controlled by controlling the display color of the screen 101a.

(7-2) Modification 2
The image determination unit 131 of the above-described embodiment determines whether or not the edge of the object has been detected in the scan image in order to determine whether the image of the object is clear. The image determination unit 131 may further determine whether the image of the object is clear based on the color, saturation, hue, and the like.

  When the image determination unit 131 determines that the object is not clear based on the color, saturation, hue, and the like, the information processing apparatus 100 changes the display state of the screen 101a as in the above-described embodiment. You may do it. Or similarly to the said modification 1, the information processing apparatus 100 may control display light and irradiation light until the hue, saturation, hue, etc. will be in the best state.

(7-3) Modification 3
In the above embodiment, the information processing apparatus 100 detects the mouse pointer 110 from the scanned image, and detects the position and tilt of the scanning apparatus 200 on the screen 101a. However, in this modification, the position and inclination of the scanning device 200 are detected by the touch panel 140 on the screen 101a.

  FIG. 14 is a schematic diagram illustrating a relationship between an information processing apparatus including a touch panel according to Modification 3 and a scanning apparatus. The display 101 of the information processing apparatus 100 includes a touch panel 140 on a screen 101a. The touch panel 140 is a device that arranges a capacitance sensor corresponding to each pixel, for example, and detects a pressed position based on a change in capacitance caused by pressing. On the other hand, in the scanning device 200, protrusions 220a and 220b are provided on the rear surface at, for example, two corners facing diagonally. When the scanning device 200 is disposed on the touch panel 140 and the protrusions 220a and 220b press the touch panel 140, the scanning device detection unit 126 detects the position and inclination of the scanning device 200 on the screen 101a. The detection method will be described next.

  FIG. 15 is a schematic diagram illustrating a method for detecting the position and tilt of the scanning device using the touch panel.

  The scanning device detection unit 126 detects the pressed position of the touch panel 140 by the protrusions 220a and 220b of the scanning device 200. For example, it is assumed that the pressing position by the protrusion 220a is I (X10, Y10) and the pressing position by the protrusion 220b is I (X11, Y11).

  Here, it is assumed that the scanning device detection unit 126 holds the basic vector V1 in the diagonal direction of the scan region 200c in advance. Note that, when the coordinate axis of the screen 101a and the coordinate axis of the scanning device 200 are arranged so as to substantially coincide with each other, the scan region 200c indicated by a broken line in FIG. 15 has a basic vector V1. The basic vector V1 is represented by coordinates I (X10, Y10) and coordinates I (X12, Y12). At this time, the length direction of the scan bar 200a is substantially parallel to the Y direction of the screen coordinate system, and the scan bar 200a is scanned toward the X direction of the screen coordinate system.

  Next, the scanning device detection unit 126 calculates an inclination vector V2 based on the pressed positions I (X10, Y10) and I (X11, Y11) by the protrusions 220a and 220b. The scanning device detection unit 126 calculates the inclination Δθ of the scanning device 200 by the following equation (1) based on the inner product of the vectors.

  The scanning device detection unit 126 obtains the coordinates of the remaining two corners based on the coordinates I (X10, Y10), the coordinates I (X11, Y11), and the inclination Δθ, whereby the position and inclination of the scanning device 200 on the screen 101a. Is detected.

  In addition, protrusions 220 may be provided at the four corners on the back of the scanning device 200 to detect the position and inclination of the scanning device 200 with respect to the screen 101a.

(7-4) Modification 4
In the above embodiment, in order to detect the position and inclination of the scanning device 200 on the screen 101a, a caret used to indicate an input position between texts can be used instead of the mouse pointer 110. The detection method is the same as that of the mouse pointer.

(7-5) Modification 5
In the embodiment described above, the information processing apparatus 100 acquires the position and inclination Δθ of the scanning apparatus 200 with respect to the screen 101a before the display control in order to control the display of the scan bar area 200b. However, for example, when controlling the entire display of the screen 101a, the position and inclination Δθ of the scanning device 200 may be acquired before the screen 101a and the object image are combined.

(7-6) Modification 6
In the above embodiment, a sticky note paper is used as an example of a physical object with low transparency, but the object may be a photograph or the like.

  FIG. 16 is an explanatory diagram for explaining an overview of processing performed by the information processing apparatus 100 when the object is a photograph. Since only the object is different and basically the same as the above processing, it will be briefly described below. Note that the user is about to generate a composite image from the photo A displayed on the screen 101a and the actual photo B and photo C.

  As shown in FIG. 6A, a mouse pointer 110 and a window 111 are displayed on the screen 101a. In the window 111, a photographic image A is displayed as an example.

  As shown in FIG. 4B, the user arranges a photograph B and a photograph C, which are objects, on the screen 101a with respect to a photograph image A in the window 111.

  As shown in FIG. 3C, in this state, the scanning apparatus 200 reads an image including the photograph B and the photograph C that are objects. At this time, the information processing apparatus 100 controls the display of the screen 101a as the scanning apparatus 200 performs reading. Further, the information processing apparatus 100 acquires the position and the inclination Δθ of the scanning apparatus 200 with respect to the screen 101a prior to reading the object, simultaneously with reading the object or after reading the object.

  As shown in FIG. 4D, the information processing apparatus 100 extracts the object images of the photos B and C based on the edges from the scan image acquired in FIG. Get the position of the object image.

  As shown in FIG. 4E, the information processing apparatus 100 combines the screen 101a in FIG. 4A with the extracted object images of the photos B and C and displays them on the screen 101a. At this time, the information processing apparatus 100 performs composition based on the position and inclination of the scanning apparatus 200 with respect to the screen 101a and the position of the target object image with respect to the scanned image.

  In addition, the post may be a postcard, for example.

(8) Usage Example A usage example of the composite image formed by the above embodiment and modification examples will be described below.

  17 to 19 are screen examples showing examples of using the composite image.

  FIG. 17 shows an example in which the composite image created in FIG. 4H is saved as a personal computer operation diary. The personal computer operation diary refers to, for example, an application that stores a personal computer operation as a record. By clicking each date, an operation diary is shown as shown in FIG. Also, by clicking the “Attach to Mail” button, the displayed operation diary can be attached to the mail and sent.

  FIG. 18 shows a state where the personal computer operation diary is attached to the mail and transmitted. By sending a personal computer operation diary attached to an e-mail in this way, the sender can immediately and accurately convey the details of the operation on the personal computer to the other party. Therefore, the sender can also extract a desired answer from the other party.

  FIG. 19 shows a state where a personal computer operation diary is posted on a bulletin board operated by a senior club, for example. By posting the personal computer operation diary on the bulletin board in this way, the operation contents on the personal computer can be immediately and accurately transmitted to a third party. Further, as shown in FIG. 19, the contributor can receive a desired comment from a third party.

  By generating a composite image according to the present embodiment and providing the use form as described above, the use of the information processing apparatus 100 can be promoted to, for example, an elderly person who is unfamiliar with the use of the information processing apparatus 100.

<Second Embodiment>
In the second embodiment described below, a physical object having high transparency will be described as an example of a reading target. Further, a cellophane paper will be described as an example of a highly transparent physical object. Cellophane paper refers to a highly permeable thin film sheet made of cellulose, for example.

  Since the external configuration and hardware configuration are the same as those in the first embodiment, description thereof will be omitted. The functional configuration and processing thereof are the same as those of the first embodiment except for the following points except for the optical interference control method.

(1) Outline of Processing FIG. 20 is an explanatory diagram for explaining the outline of the processing according to the second embodiment by the information processing device when the object is a cellophane. Note that the user is about to generate a composite image in which the photograph A displayed on the screen 101a and the actual cellophane paper 112c are superimposed.

  As shown in FIG. 6A, a mouse pointer 110 and a window 111 are displayed on the screen 101a. In the window 111, a photographic image A is displayed as an example.

  As shown in FIG. 5B, the user places the cellophane paper 112c as the object on the image A of the photograph in the window 111.

  As shown in FIG. 6C, in this state, the scanning device 200 reads an image including the cellophane paper 112c that is the object. At this time, the information processing apparatus 100 controls the display of the screen 101a as the scanning apparatus 200 reads the information. The scanning device 200 performs reading by irradiating the screen 101a and the cellophane paper 112c with irradiation light. Further, the information processing apparatus 100 acquires the position and the inclination Δθ of the scanning apparatus 200 with respect to the screen 101a prior to reading the object, simultaneously with reading the object or after reading the object.

  As shown in FIG. 4D, the information processing apparatus 100 detects the edge of the cellophane paper 112c from the scanned image acquired in FIG. 4C and acquires the relative position of the edge with respect to the scanned image.

  As shown in FIG. 5E, the information processing apparatus 100 controls the scanning apparatus 200 so as not to emit the irradiation light. The scanning device 200 performs reading using display light emitted from the screen 101a. At this time, display light from the screen 101a is emitted through the cellophane paper 112c in the area of the screen 101a on which the cellophane paper 112c is superimposed. Therefore, the scanning device 200 reads the light emitted through the cellophane paper 112c in the region where the cellophane paper 112c is superimposed.

  As shown in FIG. 5F, the information processing apparatus 100 acquires uniform color information in the cellophane paper 112c specified by the edge from the scanned image. Here, the uniform color information refers to the most dominant color in a predetermined area.

  As shown in FIG. 6G, the information processing apparatus 100 synthesizes the screen 101a shown in FIG. 5A with the uniform color information of the cellophane paper 112c and displays it on the screen 101a. At this time, the information processing apparatus 100 performs composition based on the position and inclination of the scanning apparatus 200 with respect to the screen 101a and the position of the edge with respect to the scanned image.

(2) Flow of processing The flow of optical interference control, which is a feature of the second embodiment, will be described below. FIG. 21 is a flowchart showing an example of the flow of optical interference control C according to the second embodiment.

  Since the cellophane paper 112c has high transparency, it is difficult for the information processing apparatus 100 to acquire the edge of the cellophane paper from the scanned image. Further, it is difficult for the information processing apparatus 100 to simultaneously acquire the uniform color information of the cellophane paper and the edge of the cellophane paper from the scanned image obtained by one reading. Therefore, the information processing apparatus 100 first focuses on edge detection and controls display of the screen 101a. After detecting the edge, the information processing apparatus 100 focuses on the uniform color information of the cellophane paper and controls the irradiation light emitted from the scanning apparatus 200. The flow of optical interference control C will be described below.

  Step S34a: The information processing apparatus 100 executes any one of the optical interference controls A1, A2, and B of the first embodiment, and acquires a scan image including an image of the cellophane paper 112c that is the object. This process is performed in order to detect the edge of the cellophane paper 112c in the scanned image.

  Step S34b: The image determination unit 131 detects the edge of the cellophane paper 112c, which is the object, from the scanned image. The image determination unit 131 outputs the edge position in the scan image to the extraction unit 132 and the composite image generation unit 133.

  Step S34c: The instruction control unit 122 outputs a scanning instruction command (II) to the scanning device 200.

  Step S34d: In addition, the instruction control unit 122 outputs an optical interference control command to the scanning apparatus 200, and starts control of irradiation light. Here, it is assumed that the light control command includes information instructing to stop the emission of irradiation light from the scanning device 200. As a result, the irradiation light control unit 211 of the scanning apparatus 200 stops emission of irradiation light during image reading. Note that it is only necessary to obtain uniform color information in the cellophane paper 112c, and it is not necessary to completely stop the emission of irradiation light.

  Steps S34e, S34f: The information processing apparatus 100 continues to acquire and store a scanned image until reading by the scanning apparatus 200 is completed.

  Step S34g: Based on the edge detected by the image determination unit 131, the extraction unit 132 acquires uniform color information in the cellophane paper 112c specified by the edge. For example, the extraction unit 132 detects the color of each pixel of the image in the edge, and calculates the number of pixels for each color. The extraction unit 132 acquires the color with the largest number of pixels as uniform color information. The extraction unit 132 outputs the detected uniform color information to the composite image generation unit 133.

  After the above processing, the composite image generation unit 133 adds the uniform color information of the cellophane paper 112c to the image of the screen 101a based on the scan area 200c on the screen 101a and the position of the edge of the cellophane paper 112c with respect to the scan image. Combine to generate a composite image.

  Note that if the edge cannot be detected in step S34b, the information processing apparatus 100 may perform various optical interference controls as in the first modification of the first embodiment.

  In addition, various modifications of the first embodiment can be incorporated into the second embodiment.

<Other embodiment examples>
In the above embodiment, when the user selects the setting button for the screen reading mode among the operation buttons 202 of the scanning apparatus 200, the selection information is output to the information processing apparatus 100. The information processing apparatus 100 performs the above-described various processes based on the selection of the screen reading mode. However, even if the user does not select the screen reading mode, when the information processing apparatus 100 determines that the reading target is the screen 101a based on the first acquired scan image, the above-described various processes may be performed. good. For example, when the information processing apparatus 100 determines that the scanned image is not clear, the information processing apparatus 100 determines that the reading target is the screen 101a.

  In the above-described embodiment, the scanning device is exemplified as the image acquisition device. However, the image acquisition device is not particularly limited as long as the image acquisition device acquires an image by irradiating the target with irradiation light. It may be a device.

  Further, a computer program that causes a computer to execute the above-described method and a computer-readable recording medium that records the program are included in the scope of the present invention. Here, as a computer-readable recording medium, for example, a flexible disk, a hard disk, a CD-ROM (Compact Disc-Read Only Memory), an MO (Magneto Optical disk), a DVD, a DVD-ROM, a DVD-RAM (DVD-RAM). Random Access Memory), BD (Blue-ray Disc), USB memory, semiconductor memory, and the like. The computer program is not limited to the one recorded on the recording medium, and may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, or the like.

  Regarding the above embodiment and other embodiments, the following additional notes are disclosed.

<Appendix>
(Appendix 1)
An information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light,
A display having a screen;
An optical interference controller that controls the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object placed on the screen;
An image receiving unit that receives the acquired image acquired by the image acquiring device from the image acquiring device;
Including an information processing apparatus.

  The physical object is, for example, a sticky note that can be attached, and the sticky note that has been written down is attached to a desired location on the screen of the display. When the image acquisition device acquires an image of a sticky note on the screen, the information processing device controls the display of the screen to be uniform. Controlling the screen display in a uniform state means controlling the screen uniformly, for example, in a black state or a white state by uniformly controlling the hue, saturation, brightness, and / or brightness of the screen, for example. Say. The display on the screen can be controlled by adjusting the amount of display light emitted from the screen, for example. For example, when the display is a liquid crystal display device, the information processing device controls the image signal applied to the pixel electrode to control the tilt of the liquid crystal, thereby adjusting the amount of display light emitted from the screen. Alternatively, the amount of display light emitted from the screen may be adjusted by controlling illumination light that gives brightness or brightness to the display, or backlight light in the case of a liquid crystal display device. For example, when the display is a CRT (Cathode Ray Tube), the amount of display light emitted from the screen is adjusted by controlling the amount of illumination light emitted from the back of the display. In addition, the information processing apparatus can also control the display color of the screen uniformly by controlling the display color of the screen, such as controlling to output white on the entire screen.

  By uniformly controlling the display of the screen in this way, interference between the irradiation light emitted from the image acquisition device and the display light emitted from the display of the information processing device at the time of image acquisition is suppressed. be able to. Thereby, even if the target object is arrange | positioned on the screen of a display, it becomes possible to read a target object with an image acquisition apparatus.

(Appendix 2)
The information processing apparatus according to appendix 1, wherein the optical interference control unit controls display of the screen to a black state or a white state.

  The optical interference control unit controls the screen, for example, in a uniform black state when the image acquisition device acquires an image of the object. When the object is white paper, the background color of the object is in a black state, so that the difference between the object and the screen becomes clear, and the object on the screen can be read by the image acquisition device. Thereby, the display light emitted from the screen of the information processing apparatus and the irradiation light emitted from the image acquisition apparatus interfere with each other, and adverse effects such as difficulty in acquiring an image of the object can be suppressed. .

  The black state includes not only black but also colors close to black such as dark gray and light gray. Similarly, the white state includes not only white but also lightness or luminance close to bright white, such as yellow and bright green.

(Appendix 3)
An illumination light generator that generates illumination light that gives brightness or brightness to the screen;
An image signal generator for generating an image signal for displaying an image on the screen;
A display control unit that displays an image on the screen of the display based on the illumination light and the image signal, and
3. The information processing apparatus according to claim 1, wherein the optical interference control unit controls the display of the screen to be in a uniform state by controlling the image signal.

  The screen is composed of a plurality of pixels that determine colors such as RGB, and the image signal generation unit generates an image signal for each pixel, for example. When the display is a liquid crystal display device, when the information processing device applies an image signal to the pixel electrode of each pixel, the liquid crystal is tilted according to the image signal. The optical interference controller controls the tilt of the liquid crystal by controlling the image signal, controls the amount of blocking the illumination light, and consequently adjusts the amount of display light emitted from the screen.

(Appendix 4)
An illumination light generator that generates illumination light that gives brightness or brightness to the screen;
A display control unit that uses the illumination light to display an image on the screen of the display; and
The information processing apparatus according to claim 1 or 2, wherein the optical interference control unit controls the display of the screen to be in a uniform state by controlling an emission amount of the illumination light used for displaying the image. .

  When the display is a liquid crystal display device, the information processing device controls backlight light as illumination light, adjusts the amount of display light emitted from the screen, and controls the screen display to be uniform. For example, when the display is a CRT, the amount of display light emitted from the screen is adjusted by controlling the amount of illumination light emitted from the back of the display.

  For example, the information processing apparatus can reduce the amount of illumination light to make the display on the screen a uniform black state. Further, the information processing apparatus can make the display on the screen close to a uniform white state by increasing the emission amount of the illumination light.

(Appendix 5)
In the acquired image from the image acquisition device, further comprising an image determination unit that detects an edge of the target object based on a contrast ratio between the target object and other than the target object being a predetermined value or more,
The optical interference control unit controls the uniform state of the screen to another uniform state when the image determination unit determines that the contrast ratio is less than the predetermined value. The information processing apparatus according to any one of the above.

  For example, the information processing apparatus first controls the screen to a uniform black state and receives an acquired image from the image acquisition apparatus. Here, the acquired image includes an object and a screen that is a background of the object. In this acquired image, when the contrast ratio between the object and the object other than the object is less than the predetermined value, the information processing apparatus changes the screen from the uniform black state to the uniform white state, and again returns to the image acquisition apparatus. Get an image. As the information processing apparatus adjusts the display on the screen in various ways, the image acquisition apparatus can acquire the image of the object more clearly.

(Appendix 6)
As the image acquisition device acquires an image by scanning the irradiation light, the optical interference control unit includes an irradiation area of the screen irradiated with the irradiation light according to the scanning. The information processing apparatus according to any one of appendices 1 to 5, which controls display.

  For example, when the information processing apparatus controls the display of the entire screen from the normal state to the uniform state, the screen state changes instantaneously, which gives a strong stimulus to the user of the information processing apparatus. Such a change gives a great surprise to the user, for example, and causes the user to mistakenly recognize that the information processing apparatus has malfunctioned. By controlling the display only in the limited irradiation region irradiated with the irradiation light, it is possible to suppress the stimulus given to the user as compared with the case where the display of the full screen is controlled. For example, it is assumed that the image acquisition device is a scan device that emits scan light below the scan bar area and sequentially scans the scan bar to acquire an image. In this case, the information processing apparatus controls the display of the screen to be uniform only under the scan bar area in the screen.

(Appendix 7)
An image acquisition device detection unit that detects the position and inclination of the image acquisition device with respect to the screen based on the acquired image;
The scanning position of the irradiation light in the image acquisition device is acquired from the image acquisition device, and based on the acquired scanning position of the irradiation light and the position and inclination of the image acquisition device with respect to the screen, An irradiation area detection unit for detecting the irradiation area;
The information processing apparatus according to appendix 6, wherein the optical interference control unit controls display of the irradiation region detected by the irradiation region detection unit.

  The irradiation area of the irradiation light on the screen varies depending on the position and the inclination at which the image acquisition apparatus reads with respect to the screen of the information processing apparatus. Therefore, the information processing apparatus needs to acquire the position and inclination of the image acquisition apparatus with respect to the screen. Furthermore, since the scanning position of the irradiation light moves sequentially, it is necessary to acquire the scanning position of the irradiation light in the image acquisition device. Based on the acquired information, the information processing apparatus acquires an irradiation area in which irradiation light is scanned on the screen.

  For example, the information processing apparatus acquires an image such as a pointer indicating an operation position by a mouse or a caret used for indicating an input position between texts from the image acquisition apparatus. If, for example, a pointer is included in the acquired image, the information processing apparatus determines where the image acquiring apparatus is in the screen based on the coordinates of the pointer in the acquired image and the coordinates of the pointer in the screen. I can grasp. However, since the information processing apparatus controls the display of the pointer, it recognizes the coordinates of the pointer in the screen. In addition, it is possible to grasp how much the image acquisition device is inclined with respect to the screen by an angle formed by the scanning direction of the irradiation light and the horizontal direction on the screen.

(Appendix 8)
The information processing according to any one of appendices 1 to 7, wherein the optical interference control unit further controls irradiation light emitted from the image acquisition device in accordance with acquisition of an image of the object by the image acquisition device. apparatus.

  When an image is acquired by the image acquisition device, the display light emitted from the screen and the irradiation light emitted from the image acquisition device interfere with each other. Therefore, the optical interference control unit not only controls the display on the screen to be uniform, but further controls the irradiation light emitted from the image acquisition device. Thereby, interference with display light and irradiation light can be suppressed, and bad influences, such as making the image acquisition of a subject difficult, can be controlled.

(Appendix 9)
An information processing method executed by an information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light and includes a display having a screen,
An optical interference control step for controlling the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen;
An image receiving step of receiving the acquired image acquired by the image acquiring device from the image acquiring device;
Including an information processing method.

(Appendix 10)
A computer-readable recording medium on which an information processing program for performing data transmission / reception with an image acquisition device that acquires an image together with emission of irradiation light and causing an information processing device including a display to be executed is recorded There,
An optical interference control unit that controls the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen, and the image acquisition device An image receiving unit for receiving the acquired acquired image from the image acquiring device;
A computer-readable recording medium on which an information processing program for causing the information processing apparatus to function is recorded.

100: Information processing device
101: Display
101a: Screen 110: Mouse pointer
112c: Cellophane paper
112a, 112b: sticky notes
122: Instruction control unit
123: Optical interference controller
124: Display management unit
124a: Display control unit
124b: Illumination light generation unit
124c: Image signal generation unit
125: Mouse pointer information DB
126: Scanning device detection unit
127: Scan bar area detection unit
128: Scan device information DB
129: Scanned image receiving unit
130: Scanned image DB
131: Image determination unit
132: Extraction unit 133: Composite image generation unit
134: Screen acquisition unit
135: Screen DB
140: Touch panel 200: Scanning device
200a: Scan bar
200b: Scan bar area
200c: Scan area
201: Scan mechanism
202: Operation button
203: Operation display
210: Scan execution unit
211: Irradiation light control unit
212: Input reception unit
220a, 220b: protrusion

Claims (14)

An information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light,
A display having a screen;
An optical interference controller that controls the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object placed on the screen;
An image receiving unit that receives the acquired image acquired by the image acquiring device from the image acquiring device;
A synthesized image generating unit that synthesizes the acquired image and the screen of the area where the image acquiring device is placed;
Including an information processing apparatus.   The information processing apparatus according to claim 1, wherein the optical interference control unit controls display of the screen to a black state or a white state. The information processing apparatus according to claim 1, wherein the optical interference control unit further controls irradiation light emitted from the image acquisition apparatus in accordance with acquisition of an image of the object by the image acquisition apparatus. An information processing method executed by an information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light and includes a display having a screen,
An optical interference control step for controlling the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen;
An image receiving step of receiving the acquired image acquired by the image acquiring device from the image acquiring device;
An image synthesis step of synthesizing the acquired image and the screen of the area where the image acquisition device is placed;
Including an information processing method. A computer-readable recording medium on which an information processing program for performing data transmission / reception with an image acquisition device that acquires an image together with emission of irradiation light and causing an information processing device including a display to be executed is recorded There,
An optical interference control function for controlling the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen ;
An image reception function for receiving an acquired image acquired by the image acquisition device from the image acquisition device; and
An image synthesis function for synthesizing the acquired image and the screen of the area where the image acquisition device is placed;
The computer-readable recording medium which recorded the information processing program which makes the said information processing apparatus implement | achieve . An information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light,
A display having a screen;
An optical interference controller that controls the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object placed on the screen;
An image receiving unit that receives the acquired image acquired by the image acquiring device from the image acquiring device;
An image acquisition device detection unit that detects the position and inclination of the image acquisition device with respect to the screen based on the acquired image;
The scanning position of the irradiation light in the image acquisition device is acquired from the image acquisition device, and based on the acquired scanning position of the irradiation light and the position and inclination of the image acquisition device with respect to the screen, An irradiation area detection unit for detecting the irradiation area;
Have
As the image acquisition device acquires an image by scanning the irradiation light, the optical interference control unit includes an irradiation area of the screen irradiated with the irradiation light according to the scanning. Control the display,
The optical interference control unit controls display of the irradiation region detected by the irradiation region detection unit,
Information processing device. An information processing method executed by an information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light and includes a display having a screen,
An optical interference control step for controlling the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen;
An image receiving step of receiving the acquired image acquired by the image acquiring device from the image acquiring device;
Detecting the position and tilt of the image acquisition device relative to the screen based on the acquired image;
The scanning position of the irradiation light in the image acquisition device is acquired from the image acquisition device, and based on the acquired scanning position of the irradiation light and the position and inclination of the image acquisition device with respect to the screen, Detecting the irradiation region;
Have
In the optical interference control step, as the image acquisition device acquires an image by scanning the irradiation light, an irradiation area of the screen irradiated with the irradiation light according to the scanning is acquired. Control the display, and control the display of the irradiation area detected by the irradiation area detection unit,
Information processing method. A computer-readable recording medium on which an information processing program for performing data transmission / reception with an image acquisition device that acquires an image together with emission of irradiation light and causing an information processing device including a display to be executed is recorded There,
An optical interference control function for controlling the display of the screen in a uniform state as the image acquisition device acquires an image of a physical object arranged on the screen;
An image receiving function for receiving the acquired image acquired by the image acquiring device from the image acquiring device;
A function of detecting the position and inclination of the image acquisition device with respect to the screen based on the acquired image;
The scanning position of the irradiation light in the image acquisition device is acquired from the image acquisition device, and based on the acquired scanning position of the irradiation light and the position and inclination of the image acquisition device with respect to the screen, A function of detecting the irradiation region;
Is a computer-readable recording medium on which an information processing program for causing the information processing apparatus to realize is recorded,
The optical interference control function is configured to detect an irradiation area of the screen irradiated with the irradiation light according to the scanning, as the image acquisition device acquires an image by scanning the irradiation light. A recording medium that controls display and controls display of the irradiation region detected by the irradiation region detection unit. An information processing device that transmits and receives data to and from an image acquisition device that acquires an image by scanning irradiation light,
A display having a screen;
As the image acquisition device acquires an image by scanning the irradiation light, the display of the irradiation area irradiated with the irradiation light in the screen is controlled to a black state in the screen. An optical interference control unit;
An image receiving unit that receives the acquired image acquired by the image acquiring device from the image acquiring device;
An information processing apparatus. An information processing method executed by an information processing apparatus that transmits and receives data to and from an image acquisition apparatus and includes a display having a screen,
Acquiring an image by scanning the irradiation light;
As the image acquisition device acquires an image by scanning the irradiation light, the step of controlling the display of the irradiation region irradiated with the irradiation light by the scanning to a black state in the screen; ,
Receiving an acquired image acquired by the image acquisition device from the image acquisition device;
An information processing method. A computer-readable recording medium in which an information processing program for performing transmission / reception of data with an image acquisition device and causing an information processing device including a display to be executed is recorded,
A function to acquire an image by scanning irradiation light,
A function of controlling the display of the irradiation area irradiated with the irradiation light by the scanning to the black state in the screen as the image acquisition device acquires the image by scanning the irradiation light,
A function of receiving an acquired image acquired by the image acquisition device from the image acquisition device;
The computer-readable recording medium which recorded the information processing program which makes the said information processing apparatus implement | achieve. An information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light,
A display having a screen;
As the image acquisition device acquires an image of a physical object placed on the screen, the display of a region of the screen from which the image acquisition device acquires an image is controlled to be uniform. An optical interference control unit,
An image receiving unit that receives the acquired image acquired by the image acquiring device from the image acquiring device;
Including an information processing apparatus. An information processing method executed by an information processing apparatus that transmits and receives data to and from an image acquisition apparatus that acquires an image together with emission of irradiation light and includes a display having a screen,
As the image acquisition device acquires an image of a physical object placed on the screen, the display of a region of the screen from which the image acquisition device acquires an image is controlled to be uniform. An optical interference control step to perform,
An image receiving step of receiving the acquired image acquired by the image acquiring device from the image acquiring device;
Including an information processing method. A computer-readable recording medium on which an information processing program for performing data transmission / reception with an image acquisition device that acquires an image together with emission of irradiation light and causing an information processing device including a display to be executed is recorded There,
As the image acquisition device acquires an image of a physical object placed on the screen, the display of a region of the screen from which the image acquisition device acquires an image is controlled to be uniform. Optical interference control function, and
An image receiving function for receiving the acquired image acquired by the image acquiring device from the image acquiring device;
The computer-readable recording medium which recorded the information processing program which makes the said information processing apparatus implement | achieve.