JP5369976B2 - Information processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing system capable of maintaining readability of synthesized projection data when an input image is picked up and synthesized with projection data in a state where a projection image is displayed on a body to be projected such as a whiteboard. <P>SOLUTION: By performing confirmation processing of a handwritten part table, a handwritten part is selected (step S56). Then, whether or not the selected handwritten part is related data is determined (step S58). When it is the related data, its position is set relating to the projection image (step S60). When determining that the selected handwritten part is not the related data, whether or not it has position dependency is determined (step S64). When determining that it has the position dependency, the handwritten part is reduced at a set reduction ratio (step S68). Then, its temporary position is set while maintaining relative relation to the handwritten part dependent positionally (step S70). <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an information processing system, and more particularly, to an information processing system that synthesizes an image projected onto a projection object or the like by a projection device and characters or the like written and input to the projection object or the like.

  In meetings and various presentations held in offices, etc., a projector is generally used to project onto a projection object such as a whiteboard, and explanations or discussions of the matters related to the meeting are made while referring to the projected images. Is made. In addition, for exchanging opinions during the meeting, a comment or the like is entered (written in) on the image projected and displayed on the whiteboard.

  And, it is convenient if it is possible to synthesize meeting materials reflecting the contents written and input such as comments described during the meeting as minutes data.

  For this reason, conventionally, a system has been proposed in which a comment or the like written together with an image projected and displayed on a whiteboard is captured and combined by an imaging device such as a camera (Patent Documents 1 to 3).

JP 2005-114996 A JP 2006-5545 A JP 2006-108813 A

  On the other hand, for example, when projection data is projected onto a whiteboard or the like using a projector and a projected image (hereinafter also referred to as a projection image) is displayed on the whiteboard or the like, the display of the projection image is performed. Although it is not difficult to capture an image written and input into the area and combine it with the projection data as it is, for example, an image written and input outside the display area of the projection image on the whiteboard In the case of taking an image and synthesizing it with projection data, it is necessary to consider the arrangement of the written input image, and it is necessary to maintain the legibility of the synthesized projection data.

This point is not described in any of the above Patent Documents 1 to 3.
The present invention has been made in order to solve the above-described problems. When a projection image is displayed on a projection object such as a whiteboard, the present invention captures an input image and obtains projection data. An object of the present invention is to provide an information processing system capable of maintaining the legibility of the combined projection data when combining.

  An information processing system according to the present invention includes a projection apparatus that projects projection data onto a projection object, an image acquisition apparatus that acquires an image displayed on the projection object as display image data, and a display acquired by the image acquisition apparatus And an image processing device that performs a combining process of the projection data projected from the projection device and the input image data included in the display image data acquired by the image acquisition device based on the image data. The image processing device determines the relationship between the input image data included in the image acquired by the image acquisition device and a part of the projection data, and determines that there is no relationship with the part of the data In this case, the input image is synthesized by being arranged in a predetermined area within the area where the projection data is displayed.

  Preferably, when the image processing apparatus determines that there is a relationship with a part of the data, the image processing apparatus arranges and synthesizes the data at a position that maintains the positional relationship with the part of the related data.

  Preferably, the projection data is composed of a plurality of projection page data, the projection device switches the plurality of projection page data to the projection object and projects the projection data, and the image acquisition device performs the switching of each projection page data. The image displayed on the projection object corresponding to each projection page data is acquired as display image data, and the image processing apparatus acquires each projection page data displayed on the projection object and the corresponding image acquisition apparatus. A synthesis process with the input image data included in the displayed image data is executed.

  Preferably, the image processing apparatus combines the layer in which the projection data is arranged with another layer in which the input image data is arranged.

  Preferably, the image processing apparatus determines whether or not the input image data can be arranged in a predetermined area in the projection data area, and when the input image data cannot be arranged in the predetermined area, Object image data capable of displaying image data input in accordance with an instruction is arranged and combined.

  Another image processing apparatus according to the present invention is acquired by a projection apparatus that projects projection data onto a projection object, an image acquisition apparatus that acquires an image displayed on the projection object as display image data, and an image acquisition apparatus. An image processing device that executes a composition process on the projection data based on the display image data, the projection data is composed of a plurality of projection page data, and the projection device projects the plurality of projection page data. The image acquisition device acquires an image displayed on the projection object corresponding to each projection page data as display image data when switching each projection page data. Determining whether the same input image data is included in the plurality of display image data respectively obtained corresponding to the plurality of projection page data displayed on the projection object When it is determined that the same image data is included, it is determined whether or not the same image data is included in the display image data corresponding to the first projection page data, and the display image data corresponding to the first projection page data When the same image data is included in the image data, a composition process for adding the input same image data to the projection data as the first projection page data is executed.

  Still another image processing apparatus according to the present invention is obtained by a projection apparatus that projects projection data onto a projection object, an image acquisition apparatus that acquires an image displayed on the projection object as display image data, and an image acquisition apparatus. And an image processing device that performs a composition process on the projection data based on the displayed display image data. The projection data includes a plurality of projection page data, and the projection device receives the plurality of projection page data. The image acquisition apparatus switches to the projection body and projects the image, and the image acquisition apparatus acquires an image displayed on the projection object corresponding to each projection page data as display image data when switching each projection page data. Determines whether the input image data is included in the display image data corresponding to the last projection page data displayed on the projection target, and determines the last projection page data. Corresponds to a case that contains the image data input to the display image data, it executes a synthesis process to be added to the projection data of the input image data as the last projecting page data.

  Preferably, the input image data is an image of at least one of a character, a figure, a photograph pasted on the projection object, paper, and an image projected from another projection apparatus displayed on the projection object. This corresponds to the image data acquired by the acquisition device.

  The information processing system according to the present invention determines the relevance between the input image data included in the image acquired by the image acquisition device and a part of the projection data, and the relevance to the part of the data is If it is determined that there is no image data, the input image data is arranged and synthesized in a predetermined area within the area where the projection data is displayed. Therefore, the input image data is arranged in the predetermined area based on the relationship between the input image data and the projection data. That is, since it can be arranged in a blank area as a predetermined area, it is possible to maintain the legibility of the synthesized projection data.

It is a figure explaining the relationship of the information processing system according to Embodiment 1 of this invention. 2 is a diagram illustrating a schematic block diagram of an MFP 15. FIG. It is a figure explaining the case where it writes and inputs into the projection image projected on the whiteboard. It is a figure explaining the synthetic | combination process based on the imaging data which imaged the image displayed on the whiteboard 5 acquired with the camera. It is a figure explaining the example which judges whether the data inputted by writing included in picked-up image data have relevance with projection page data. It is a figure explaining another synthetic | combination process based on the imaging data (display image data) which imaged the image displayed on the whiteboard 5 acquired with the camera. It is a figure explaining the synthetic | combination process with the process input data 246 and projection page data. It is a figure explaining the whole flow of the synthetic | combination system according to Embodiment 1 of this invention. It is a figure explaining the subroutine process of a handwritten character extraction process. It is a figure explaining the subroutine processing of handwritten attribute judgment processing. It is a figure explaining the handwritten parts table according to embodiment of this invention. It is a figure explaining the handwritten parts analysis process according to embodiment of this invention. It is a figure explaining another handwritten parts table according to an embodiment of the invention. It is a figure explaining a handwritten character synthetic | combination process. It is a figure explaining the imaging data which imaged the image displayed on the whiteboard 5 acquired with the camera 20 according to Embodiment 2 of this invention. It is a figure explaining the synthetic | combination process based on the imaging data according to Embodiment 2 of this invention. It is a figure explaining the whole flow of the synthetic | combination system according to Embodiment 2 of this invention. It is a figure explaining the subroutine process of the related data synthetic | combination process according to Embodiment 2 of this invention. It is a figure explaining another subroutine process of the related data synthetic | combination process according to Embodiment 2 of this invention. It is a figure explaining the system which synthesize | combines the data written and input into display bodies, such as a white board different from the to-be-projected body according to this Embodiment 3, with projection page data. It is a figure explaining the whole flow of the synthetic | combination system according to Embodiment 3 of this invention. It is a figure explaining the subroutine process of the handwriting extraction process according to Embodiment 3 of the present invention. It is a figure explaining the subroutine process of the handwriting attribute judgment process according to Embodiment 3 of this invention. It is a figure explaining the handwritten parts table according to Embodiment 3 of this invention. A handwritten character composition process according to the third embodiment of the present invention will be described.

  Embodiment 1 of the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same.

(Embodiment 1)
FIG. 1 is a diagram for explaining the relationship of the information processing system according to the first embodiment of the present invention.

  Referring to FIG. 1, in information processing system 1 according to the first embodiment of the present invention, a projector 10 that is a projection device and a projection object for displaying a projection image (projection image) projected from projector 10. , A white board 5, a camera 20, an MFP (Multi Function Peripheral) 15, and information processing terminals (hereinafter referred to as PC (Personal Computer)) 25 and 26. The MFP 10 is a multi-function digital multi-function peripheral having a copy function, a printer function, a fax function, and the like. These apparatuses are connected to each other via a network 30 and can exchange data between the apparatuses.

  In the conference system 1 according to the first embodiment of the present invention, the projector 10 and the PC 25 are connected to each other, the projection data held by the PC 25 is output to the projector 10, and the projector 10 performs white according to the projection data. Assume that a projected image is projected onto the board 5. In this example, the projection data is composed of a plurality of projection page data, and is collectively referred to as projection data.

  In this example, a case is described in which projection page data constituting projection data is output from the PC 25 to the projector 10 via the network 30. However, the present invention is not limited to this, and projection page data may be output from the PC 26. Alternatively, the projection page data stored in the hard disk of the MFP 15 can be output to the projector 10 via the network 30.

  The camera 20 acquires image data (display image data) obtained by capturing an image displayed on the whiteboard 5 and outputs the acquired image data to the MFP 15 via the network 30 for image composition processing described later. In this example, the case where the MFP 15 executes the image composition process will be described. However, the process may be executed by the PCs 25 and 26. In that case, the camera 20 outputs imaging data to the PCs 25 and 26.

  The timing at which the camera 20 captures an image displayed on the whiteboard 5 is that the image displayed on the whiteboard 5 is captured when the projection page data is switched.

  Specifically, when it is possible to switch the projection page data page by page and display the projection image on the whiteboard 5, the PC 25 issues a switching instruction (switching instruction to the next projection image) from the user. When the input is accepted, the camera 20 is instructed to take an image displayed on the whiteboard 5. Then, according to the instruction, the camera 20 acquires imaging data (display image data) obtained by imaging an image displayed on the whiteboard 5. Then, the PC 25 is notified that the acquisition has been completed. It is assumed that the PC 25 outputs the next projection page data to the projector 10 after receiving a notification that the imaging has been completed from the camera 20.

  In this example, the method of instructing the camera 20 to take an image when the PC 25 receives an input of a switching instruction from the user has been described. However, the present invention is not limited to this. The camera 20 may be instructed to take an image.

  Further, when executing an image composition process to be described later, the MFP 15 acquires projection data from the PC 25 and also acquires the image data from the camera 20 described above and executes the image composition process.

(MFP configuration)
Next, a schematic block diagram of the MFP 15 will be described with reference to FIG.

  Referring to FIG. 2, MFP 15 includes a central processing unit (CPU) 101, a network interface (I / F) 108 for executing data exchange via network 30, and a predetermined processing in MFP 15. ROM (Read Only Memory) 102 in which various software programs for executing the above functions are stored, RAM (Random Access Memory) 103 used as a work area of the CPU 101, and HDD (Hard as a mass storage device) disc drive) 104, an image reading unit 106 for reading an original, and a color image by using toners of four colors of cyan, magenta, yellow and black, or any one of cyan, magenta, yellow and black A printing unit 107 that forms a monochrome image using toner, and a liquid crystal display An operation panel unit 105 having a display such as a display (LCD), an organic ELD (Electro Luminescence Display), and a plurality of keys that accept data input such as various instructions, characters, numbers, etc. by user operation corresponding to key operations Including. Each unit can exchange data with each other via the internal bus 109, and the CPU 101 controls the entire MFP 15. Further, it is assumed that the image composition processing described later is realized by the CPU 101 executing a predetermined application program stored in the HDD 104. It is also possible to separately provide hardware that executes dedicated image composition processing.

  In the information processing system according to the first embodiment, the MFP 15 receives input of imaging data (display image data) obtained by capturing an image displayed on the whiteboard 5 acquired by the camera 20. Specifically, the MFP 15 receives image data (display image data) via the network interface 108 and stores the data in the RAM 103 or the HDD 104. Further, it is assumed that the MFP 15 receives the projection data of the PC 25 via the network interface 108 and stores the data in the RAM 103 or the HDD 104.

  Then, the CPU 101 executes a predetermined application program stored in the HDD 104 as described above, thereby executing an image composition process of projection data described later and imaging data (display image data).

  Next, a flow of a synthesis process between projection data and input display image data in the information processing system according to the first embodiment will be described.

  Specifically, with respect to projection data composed of a plurality of projection page data, each projection page data is combined with display image data input corresponding to each projection page data.

  There are several conditions as a composition processing method in the information processing system according to the first embodiment.

  (1) When, for example, data input and written included in the captured image data is related to the projected page data, it is determined whether the data is related to the projected page data. Then, it is synthesized in association with the projection page data.

  (2) When the input data is divided into a plurality of parts, it is determined whether or not they are related to each other, and if it is determined that they are related to each other, they are combined with each other. .

  (3) When the input data is not related to the projection page data, the data is arranged and synthesized in a predetermined area which is a blank area of the projection page data.

  (4) In (3), if the size cannot be arranged in a predetermined area, which is a blank area of the projection page data, at the time of synthesis, an icon is arranged and synthesized.

First, a synthesis process when the above conditions (1) and (2) are satisfied will be described.
With reference to FIG. 3, a case will be described in which a projection image projected on the whiteboard 5 is written and input.

  Referring to FIG. 3, in this example, the characters “Patent Treatment Rights” are displayed as projected images in the projected image display area 50 projected on the whiteboard 5 and the characters “right” are corrected. A case where a character “IN” is displayed is shown together with a line drawing indicating “”. Here, the line drawing for correcting the character and the character of “inspection” are those directly written on the whiteboard 5.

  Further, in this example, a part of the line drawing indicating the correction of the “right” character and the “check” character are written and input outside the projected image display area 50.

  Next, a synthesis process based on image data obtained by capturing an image displayed on the whiteboard 5 acquired by the camera 20 will be described with reference to FIG. In this example, it is assumed that the first page when the projection data is composed of a plurality of projection page data is displayed.

  Referring to FIG. 4A, here, a case where imaging projection data 210 is included in imaging data 200 is shown.

  Here, the imaged projection data refers to image data obtained by the camera 20 using a projection image displayed on the whiteboard 5.

  In this example, the imaging range of the camera 20 is described as being equal to the entire area of the whiteboard 5, but may be smaller than the whiteboard 5 as long as it can include a projected image. Further, when imaging a wider area than the whiteboard 5, the entire area of the whiteboard 5 may be extracted from the captured area. The extraction of the whiteboard 5 can be determined based on the contrast difference from the background color, and a known technique can be used.

  The acquired imaging data is transmitted from the camera 20 to the network interface 108 of the MFP 15 via the network 30. The data is acquired by the MFP 15 and stored in the RAM 103 or the HDD 104.

  Note that, when executing the image composition process, the projection page data output from the PC 25 to the projector 10 is acquired in the size of the data area expanded in the memory, and the projection image displayed on the projection object from the camera 20 is acquired. Although it is considered that the size of the data area developed in the memory of the imaged projection data is actually different, in this example, it is assumed that the size is the same for the sake of simplicity. If they are different from each other, the sizes of the two are compared, and the data area may be enlarged or reduced so as to match one of the sizes.

  FIG. 4B is a diagram for explaining imaging separation data 202 obtained by separating projection page data from imaging data.

  Here, imaging separation data 202 obtained by removing imaging projection data 210 from imaging data 200 is shown.

  Specifically, a line drawing for correcting a character and image data 220 in which “right” is written and input are shown.

FIG. 4C is a diagram illustrating a case where the image data 220 is divided into handwritten parts.
Here, the case where the image data 220 is divided into three handwritten parts 222, 224, and 226 is shown.

  Specifically, it is possible to classify characters or line drawings by using a so-called known pattern matching algorithm and classify them into handwritten parts. In this example, the handwritten parts 222 and 224 are line drawings, and the handwritten parts 226 are characters. In this example, a rectangular area having upper left coordinates and lower right coordinates as parameter values is extracted as handwritten parts as position coordinate data of the handwritten parts.

  Further, as shown in FIG. 4A, the handwritten part 222 is arranged at a position overlapping with the imaged projection data, so that it is determined that the handwritten part 222 is related to the projection page data. Based on the determination of the relevance, the composition method of the handwritten part 222 is determined as described later. That is, this is a case where the condition (1) is satisfied.

  In addition, as shown in FIG. 4A, the handwritten parts 224 and 226 are determined not to be related to the projection page data because they are not arranged at positions that overlap the imaging projection data.

  On the other hand, the handwritten parts 222, 224, and 226 are determined to be handwritten parts having a positional relationship related to each other based on the mutual positional relationship (distance). Based on the determination of the relevance, the composition method of the handwritten parts 222, 224, and 226 is determined as described later. That is, this is a case where the condition (2) is satisfied.

  FIG. 4D is a diagram for explaining processed input data 204 processed to synthesize three handwritten parts 222, 224, and 226 with projection page data.

  Here, as the processed input data 204, the handwritten part 222 that corrects the “right” is a handwritten part related to the projection page data, so that the positional relationship with the “right” in the imaging projection data 210 is maintained. Be placed.

  On the other hand, the handwritten part 224 that is a line drawing and the handwritten part 226 that is a character of “check” are reduced and arranged because they are not handwritten parts related to the projection page data.

  Specifically, the handwritten parts 224 and 226 extracted in the rectangular area are reduced at a reduction ratio that reduces the image data 200 so that the range of the image data 200 coincides with the range of the imaged projection data 210.

  Since the handwritten parts 224 and 226 are related to the handwritten part 222, the reduced handwritten parts 224 and 226 are arranged in the processing input data 204 while maintaining a relative positional relationship with the handwritten part 222. .

  FIG. 4E is a diagram for explaining a synthesis process of the processing input data 204 and the projection page data (imaging projection data 210).

  As shown in FIG. 4E, the processed input data 204 and the projection page data (imaging projection data 210) are superimposed to generate composite data 206.

  As shown here, the handwritten part 222 that corrects the “right” character data related to the projection page data is synthesized so as to maintain the positional relationship with the “right”, and is related to the handwritten part 222. The handwritten parts 224 and 226 are reduced and combined with the handwritten parts 222 while maintaining a relative positional relationship with each other.

  By this composition method, even when writing is input outside the projected image display area 50, it is possible to compose the projection page data.

  When it is determined that there is a relationship with the projection page data, it is synthesized in association with the projection page data. Further, when the input data is divided into a plurality of parts and it is determined that they are related to each other, they are combined in association with each other.

  Accordingly, when a projected image is displayed on a projection object such as a whiteboard, the readability of the synthesized projected page data can be maintained when the input image is captured and synthesized with the projected page data. Is possible.

  FIG. 5 is a diagram for explaining an example of determining whether or not the input data included in the captured image data is related to the projection page data.

  As shown in FIG. 5A, as shown in FIG. 4A, a handwritten part 230 that overlaps the imaged projection data is shown, and when it is arranged at the overlapping position, projection is performed. Judged to be related to page data.

  In addition, as shown in FIG. 5B, a handwritten part 232 surrounding a part of the imaging projection data is shown, and when the part of the imaging projection data is included in the handwriting part 232, Judged to be related to the projected page data.

Next, a synthesis process when the conditions (3) and (4) are satisfied will be described.
With reference to FIG. 6, another composition process based on imaging data (display image data) obtained by capturing an image displayed on the whiteboard 5 acquired by the camera 20 will be described. In this example, it is assumed that the projection data is the second page when it is composed of a plurality of projection page data.

  Referring to FIG. 6A, here, a case where imaging projection data 210 # is included in imaging data 200 # is shown.

  Here, the imaged projection data refers to image data obtained by the camera 20 using a projection image displayed on the whiteboard 5.

  FIG. 6B is a diagram for explaining imaging separation data 202 # obtained by separating projection page data from imaging data.

  Here, imaging separation data 202 # obtained by removing imaging projection data 210 # from imaging data 200 # is shown.

Specifically, comment data 240 is shown on the whiteboard 5.
FIG. 6C is a diagram illustrating a case where the image data 240 is divided into handwritten parts.

  Here, the case where the image data 240 is divided into handwritten parts 242 is shown.

  Specifically, it is possible to classify characters or line drawings by using a so-called known pattern matching algorithm and classify them into handwritten parts. In this example, it is assumed that the handwritten part 242 is a character. In this example, a rectangular area having upper left coordinates and lower right coordinates as parameter values is extracted as handwritten parts as position coordinate data of the handwritten parts.

  Further, since the handwritten part 242 is not arranged at a position that overlaps the imaged projection data, it is determined that there is no relevance to the projection page data. Based on the determination of no relevance, the composition method of the handwritten part 242 is determined as described later. That is, this is a case where the condition (3) is satisfied.

  FIG. 6D is a diagram for explaining the processing input data 246 processed to synthesize the handwritten part 242 with the projection page data.

  Specifically, the handwritten part 240 extracted in the rectangular area is reduced at a reduction ratio to reduce the range of the imaging data 200 # to the range of the imaging projection data 210 #.

  In addition, the handwritten part 240 is not related to the imaging projection data, and thus is placed in a predetermined area that is a blank area in the processed input data 246.

  A process of synthesizing the processing input data 246 and the projection page data (imaging projection data 210 #) will be described with reference to FIG.

  As shown in FIG. 7A, the composite input data 246 is generated by superimposing the processing input data 246 and the projection page data (imaging projection data 210 #).

  As shown here, the reduced handwritten part 242 that is not related to the projection page data is arranged in a predetermined area that is a blank area of the projection page data.

  By this composition method, even when writing is input outside the projected image display area 50, it is possible to compose the projection page data.

  If it is determined that there is no relevance to the projected page data, the projected page data is arranged and synthesized in a predetermined area that is a blank area of the projected page data.

  Accordingly, when a projected image is displayed on a projection object such as a whiteboard, the readability of the synthesized projected page data can be maintained when the input image is captured and synthesized with the projected page data. Is possible.

  FIG. 7B is a diagram for explaining a synthesis process of the processing input data 246 and the projection page data (imaging projection data 210 #).

  As shown in FIG. 7A, the processed input data 246 and the projection page data (imaging projection data 210 #) are superimposed to generate composite data 252.

  Here, it is assumed that the reduced handwritten part 242 cannot be arranged in a predetermined area that is a blank area of the projection page data. Therefore, the icon (object image data) 248 is synthesized, and the image data (comment data) of the handwritten part 242 is displayed by designating the icon.

  By this composition method, even when writing is input outside the projected image display area 50, it is possible to compose the projection page data.

  That is, even if the data written and input in the predetermined area which is the blank area of the projection page data cannot be arranged, it is possible to synthesize an icon capable of displaying the comment data by giving an instruction.

  Accordingly, when a projected image is displayed on a projection object such as a whiteboard, the readability of the synthesized projected page data can be maintained when the input image is captured and synthesized with the projected page data. Is possible.

Hereinafter, a flow for executing the above synthesis method will be described.
With reference to FIG. 8, the overall flow of the synthesis method according to the first embodiment of the present invention will be described.

  Referring to FIG. 8, it is first determined whether or not there is a photographing request (step S1). In this example, when switching the projection page data, the PC 25 instructs the camera 20 to take an image in accordance with the input of the switching instruction from the user. Even when the final projection page data ends, the PC 25 instructs the camera 20 to take an image in accordance with the input of the instruction.

  If there is a photographing request, an image is acquired (step S2). Specifically, the camera 20 acquires imaging data (display image data) obtained by imaging an image displayed on the whiteboard 5 in accordance with an imaging request from the PC 25.

  Then, the acquired imaging data (display image data) and projection page data are output to the MFP 15 (step S3).

  Then, the MFP 15 stores the image data (display image data) and the projection page data in the RAM 103 or the HDD 104 (step S4).

  Next, it is determined whether the MFP 15 has acquired image data corresponding to all projection page data (step S5). If the acquisition is not completed in step S5, the process returns to step S1, and the same processing is repeated until acquisition of imaging data corresponding to all projection page data is completed.

  If the acquisition is completed in step S5, the process proceeds to step S6.

  As a result, the MFP 15 is ready to execute the image composition process, and the CPU 101 executes a predetermined application to execute the following process in software. Note that the following processing is executed for each projection page data in this example.

Specifically, first, handwritten character extraction processing is executed (step S6).
Next, handwriting attribute determination processing for determining the attribute of the handwritten character for the extracted handwritten character is executed (step S7).

  Then, handwritten character synthesis processing for synthesizing handwritten characters is executed (step S8). Then, the process ends (END).

(Handwritten character extraction process)
The subroutine processing of handwritten character extraction processing will be described with reference to FIG.

Referring to FIG. 9, image comparison processing is first executed (step S12).
Specifically, the imaging projection data included in the imaging data (display image data) is specified by comparing the imaging data (display image data) and the projection page data.

  Then, it is determined whether or not there is a writing input (step S14). Specifically, it is determined whether there is data other than the imaging projection data in the imaging data (display image data). The imaging projection data is image data obtained by acquiring a projection image with the camera 20.

  Therefore, when it is determined that there is image data other than the imaging projection data in the imaging data, it is determined that there is a writing input, and when there is no imaging data, it is determined that there is no writing input.

  If it is determined in step S14 that there is no write input (NO in step S14), the process ends (return). This is because in this case, there is no need to execute image composition processing.

  On the other hand, if it is determined in step S14 that there is a writing input (YES in step S14), handwritten image separation processing is executed (step S16).

  Specifically, data other than the specified imaging projection data in the imaging data (display image data) is extracted. For example, in the example of FIG. 4, the imaging separation data 202 shown in FIG. 4B is acquired.

And it preserve | saves (step S18). Specifically, it is stored in the HDD 104.
Then, the process ends (return).

Note that this processing is executed for each projection page data.
(Handwriting attribute judgment process)
Next, the handwriting attribute determination process will be described.

The subroutine process of the handwriting attribute determination process will be described with reference to FIG.
Referring to FIG. 10, first, a handwritten image reading process is executed (step S20).

  Specifically, the handwritten image data separated by the handwritten image separation process is acquired from the HDD 104. For example, in the example of FIG. 4, the imaging separation data 202 stored in the HDD 104 shown in FIG. 4B is acquired.

And it isolate | separates into a handwritten part (step S22).
Specifically, using a so-called known pattern matching algorithm or the like, characters or line drawings are classified and divided into handwritten parts.

  For example, in the example of FIG. 4, the image data 220 of the imaging separation data 202 shown in FIG. 4C is divided into three handwritten parts 222, 224, and 226.

  Then, each separated handwritten part is stored in the handwritten parts table (step S23).

  Specifically, a handwritten part ID is issued to each separated handwritten part, and information such as attribute data of the handwritten part is stored corresponding to the corresponding handwritten part ID.

  In this example, depending on the issued handwritten part ID, it is dependent on handwritten part attribute data, related page data, position coordinate data, data indicating related / normal data, and related data. Each handwritten part ID data is registered.

A handwritten parts table according to the embodiment of the present invention will be described with reference to FIG.
Referring to FIG. 11, here, a case where a handwritten part is registered in the handwritten parts table is shown.

Specifically, a case where four handwritten parts are registered is shown as an example.
First, the handwritten part in the example shown in FIG. 4 will be described.

  In this example, the case where handwritten part ID is issued corresponding to each of the three separated handwritten parts 222, 224, and 226 is shown. Specifically, the ID “001” is issued corresponding to the handwritten part 222, and the attribute “line drawing”, the related page “1”, and the positions “(X1, Y1), (X2, Y2)” are registered. The case is shown. The position coordinate data indicates a rectangular area having upper left coordinates and lower right coordinates as parameter values.

  Further, there is a case where ID “002” is issued corresponding to the handwritten part 224, and the attribute “line drawing”, the related page “1”, and the positions “(X3, Y3), (X4, Y4)” are registered. It is shown.

  Further, there is a case where ID “003” is issued corresponding to the handwritten part 226 and the attribute “character”, the related page “1”, and the positions “(X5, Y5), (X6, Y6)” are registered. It is shown.

  Although not yet registered in the column of data indicating other related / normal data and the dependency ID, it is registered as necessary based on a handwritten part analysis process described later.

The handwritten part in the example shown in FIG. 6 will be described.
In this example, the case where a handwritten part is issued corresponding to the separated handwritten part 242 described above is shown. Specifically, an ID “004” is issued corresponding to the handwritten part 242, and an attribute “character”, a related page “2”, and a position “(X7, Y7), (X8, Y8)” are registered. The case is shown.

  Referring to FIG. 10 again, next, handwritten parts analysis processing is executed (step S24).

The handwritten part analysis process according to the embodiment of the present invention will be described using FIG.
Referring to FIG. 12, first, a handwritten part is selected (step S30).

  Next, it is determined whether or not there is a relationship with the projection image (step S32). Specifically, whether or not the projection page data (imaging projection data) and the selected handwritten part overlap, or whether or not a part of the projection page data (imaging projection data) is included in the handwritten part Is judged.

  If it is determined in step S32 that there is a relationship with the projection image (YES in step S32), then it is registered as related data (step S34). Then, the process proceeds to step S38.

  On the other hand, if it is determined in step S32 that there is no relevance with the projection image (NO in step S32), it is determined whether or not there is position dependency with other handwritten parts (step S42).

  Specifically, the position coordinate data of another handwritten part registered in the handwritten part ID is confirmed, and it is determined whether there is a handwritten part ID adjacent to the selected handwritten part. For example, when a rectangular area is enlarged by providing an offset value for the upper left coordinate of the position coordinate data corresponding to the selected handwritten part ID and the parameter value that is the lower right coordinate, the enlarged rectangular area has another It is determined whether or not a part of the rectangular area defined by the position coordinate data of the handwritten part is included. If it is included, it is determined that there is an adjacent handwritten part ID. On the other hand, if not included, it is determined that there is no adjacent handwritten part ID.

  Note that the offset value is set to an appropriate value for determining adjacent handwritten parts. In this example, the case where the offset value is provided has been described, but the offset value may be determined as 0. Moreover, you may make it judge adjacent handwritten parts ID based on another system.

  If it is determined that there is position dependency with other handwritten parts, it is registered as normal data with position dependency (step S44). Specifically, it is registered as normal data, and other adjacent handwritten part IDs that are determined to be position-dependent are registered as dependent handwritten part ID data. Then, the process proceeds to step S38.

  On the other hand, if it is determined in step S42 that there is no position dependency with other handwritten parts, it is registered as normal data without position dependency (step S46). Specifically, while registering as normal data, since there is no position dependency, nothing is registered for the dependent handwritten part ID data. Then, the process proceeds to step S38.

In step S38, it is determined whether or not all handwritten parts have been selected.
If all the handwritten parts are not selected in step S38, the process returns to step S30. Then, the handwritten part is selected again and the same process is repeated. If it is determined that all the handwritten parts have been selected (YES in step S38), the process ends (return).

  For example, the handwritten parts registered in the handwritten parts table of FIG. 11 will be described.

  The handwritten part 222 corresponding to the handwritten part ID “001” is registered as related data because it is arranged at a position overlapping the projection page data (imaging projection data) as described above.

  As described above, the handwritten part 224 corresponding to the handwritten part ID “002” is not arranged at a position overlapping the projection page data (imaging projection data).

  On the other hand, the handwritten part corresponding to the handwritten part ID “002” is determined to be position-dependent from the handwritten parts corresponding to the other handwritten parts ID “001” and “003”, and is registered.

  Further, as described above, the handwritten part 226 corresponding to the handwritten part ID “003” is not arranged at a position overlapping the projection page data (imaging projection data).

  On the other hand, the handwritten part corresponding to the handwritten part ID “003” is determined to be position-dependent from the handwritten part corresponding to the other handwritten part ID “002” and registered.

Another handwritten parts table according to the embodiment of the present invention will be described with reference to FIG.
Referring to FIG. 13, here, a case where data is registered in the column of data indicating the other related / normal data and the dependency ID as compared with the handwritten parts table of FIG. 11 is shown.

  Specifically, for the first page, the case where the handwritten part 222 corresponding to the handwritten part ID “001” is registered as related data is shown.

  Further, the case where the handwritten part 224 corresponding to the handwritten part ID “002” is registered as normal data and is registered as being position-dependent with the handwritten parts ID “001” and “003” is shown. .

  Further, a case is shown in which the handwritten part 226 corresponding to the handwritten part ID “003” is registered as normal data and registered as having position dependency with the handwritten part ID “002”.

  In this example, a case where a handwritten part is issued corresponding to the separated handwritten part 242 described above is shown. Specifically, a handwritten part ID “004” is issued corresponding to the handwritten part 242, and an attribute “character”, a related page “2”, and positions “(X7, Y7), (X8, Y8)” are registered. The case has been shown.

  Specifically, regarding the second page, the handwritten part 242 corresponding to the handwritten part ID “004” is registered as normal data and is registered as having no position dependency of other handwritten parts. Has been.

(Handwriting synthesis process)
Next, the handwritten character synthesis process will be described.

The handwritten character synthesis process will be described with reference to FIG.
Referring to FIG. 14, first, a projection image and handwritten image reading process is executed (step S50).

  Specifically, a process of reading projection page data that is a projection image and a separated handwritten image is executed.

Next, a reduction ratio is set (step S52).
Specifically, a reduction ratio is set to reduce the image data so that the range of the image data matches the range of the projection page data (image image projection data).

Next, the handwritten parts table confirmation process is executed (step S54).
Next, a handwritten part is selected (step S56).

  And it is judged whether it is related data (step S58). Specifically, the determination can be made based on the related data or normal data registered in the handwritten parts table.

  If it is related data in step S58, the position is set in association with the projection image (step S60).

  Specifically, the position is set so as to maintain the positional relationship with the projection page data (imaging projection data).

  And it is judged whether the position of all the handwritten parts was set (step S62).

  Specifically, it is determined whether all the positions of the handwritten parts registered in the handwritten parts table have been set.

  If it is determined in step S62 that the positions of all the handwritten parts have not been set (NO in step S62), the process returns to step S56 again to select another handwritten part.

  On the other hand, if it is determined in step S58 that the selected handwritten part is not related data (NO in step S58), it is determined whether there is position dependency (step S64).

  Specifically, when the selected handwritten part is normal data, it is possible to determine whether there is a handwritten part ID registered in the column of the dependent handwritten part ID data.

  If it is determined in step S64 that the position is dependent (YES in step S64), it is determined whether there is a position setting of the dependent handwritten part (step S66).

Specifically, it is determined whether or not the position of the dependent handwritten part is already set.
If it is determined in step S66 that the position of the dependent handwritten part is set (YES in step S66), the handwritten part is then reduced according to the set reduction ratio (step S68).

  Then, the temporary position is set while maintaining the relative relationship with the dependent handwritten part (step S70).

  Specifically, when the position where the dependent handwritten part is placed is used as a reference, the distance from the handwritten part is reduced at a set reduction ratio based on the position coordinate data registered in the handwritten part table. Set the position to the temporary position.

  Next, it is determined whether or not the arrangement is possible (step S72). Specifically, it is determined whether the handwritten part set at the temporary position is arranged at a position overlapping the projection page data (imaging projection data) or is within the size of the projection page data (imaging projection data). to decide. When the handwritten part is arranged at a position overlapping with the projection page data (imaging projection data), or when the temporary position is outside the size of the projection page data (imaging projection data), it is determined that the arrangement is not possible. On the other hand, when the temporary position is within the size of the projection page data (imaging projection data) and the handwritten part is not arranged at a position overlapping the projection page data (imaging projection data), it is determined that the arrangement is possible. .

  If it is determined in step S72 that the arrangement is possible, the position is set (step S74).

That is, the temporary position set in step S70 is set.
Then, the process again proceeds to step S62.

  On the other hand, if it is determined in step S72 that the placement is not possible (NO in step S72), then the position of the icon is set (step S76). Then, the process again proceeds to step S62.

  Note that the position of the icon is set to the temporary position. If it overlaps with projection page data (imaging projection data), it is set to the position of the empty space around it.

  On the other hand, if it is determined in step S66 that there is no dependent handwritten part position setting (NO in step S66), the process proceeds to step S62, and the process returns to step S56 again to select another handwritten part. Shall be executed.

  That is, when there is no position setting of the dependent handwritten part, the position setting of the other handwritten parts is prioritized, the position setting of the dependent handwritten parts is confirmed, and the position is set again.

  On the other hand, if it is determined in step S64 that there is no position dependency (NO in step S64), the handwritten part is reduced according to the set reduction rate (step S80).

Next, an empty space is confirmed (step S82).
Specifically, it is determined whether or not an empty space area exists within the entire size of the projection page data (imaging projection data). It is assumed that the empty space area is an area having a predetermined size or more.

  Next, it is determined whether or not handwritten parts can be arranged in the empty space area within the entire size of the projection page data (imaging projection data) (step S84).

  If it is determined that a handwritten part can be placed in the empty space area (YES in step S84), the position of the handwritten part is set in the empty space area (step S86).

  On the other hand, if it is determined in step S84 that it cannot be placed in the empty space area (NO in step S84), the icon is set in the empty space area.

Then, the process proceeds to step S62.
If it is determined in step S62 that the positions of all the handwritten parts have been set (YES in step S62), a composition process in which the positions of the handwritten parts are set for the projection page data is executed in step S63. (Step S63).

  As a specific example of the synthesis process, the projection page data, which is a projection image, is expanded in the memory area (first layer), and the processing input data shown in FIG. The projected page data is expanded at a set position in another memory area (second layer) having the same size as the expanded memory area, and is expanded in two memory areas (first and second layers). It is assumed that the data is synthesized by superimposing the data. Then, the process ends (return).

  For example, the handwritten parts registered in the handwritten parts table in FIG. 13 will be described.

In this example, a handwritten part associated with one page will be described.
Since the handwritten part 222 corresponding to the handwritten part ID “001” is registered as related data, the position is set in association with the projection image. That is, since it is a handwritten part related to the projection page data (imaging projection data), it is arranged so as to maintain the positional relationship with the “right” in the projection page data (imaging projection data). In this example, as an example, the handwritten part 222 is arranged while maintaining the positional relationship overlapping with “right” without being reduced.

  The handwritten part 224 corresponding to the handwritten part ID “002” is registered not as related data but as normal data, and is registered as having position dependency with the handwritten parts ID “001” and “003”. Assuming that the placement position is set for the handwritten part ID “001”, the handwritten part 224 is reduced at the set reduction rate. And it sets to the temporary position which maintained the relative positional relationship with the handwritten part corresponding to handwritten part ID "001".

  Specifically, the handwritten part corresponding to the handwritten part ID “001” is based on the position coordinate data registered in the handwritten part table with the position where the handwritten part corresponding to the handwritten part ID “001” is arranged as a reference. And the position where the distance between the handwritten part corresponding to the handwritten part ID “002” is reduced at the set reduction ratio is set as the temporary position.

  Then, it is determined whether or not the handwritten part can be placed at the temporary position, and when it is determined that the handwritten part can be placed, the position of the handwritten part is set.

In this example, it is determined that arrangement is possible.
In addition, the handwritten part 226 corresponding to the handwritten part ID “003” is registered not as related data but as normal data, and is registered as position-dependent with the handwritten part ID “002”. Assuming that an arrangement position is set for the part ID “002”, the handwritten part 226 is reduced at the set reduction rate. And it sets to the temporary position which maintained the relative relationship with the handwritten part corresponding to handwritten part ID "002".

  Specifically, the handwritten part corresponding to the handwritten part ID “002” is based on the position coordinate data registered in the handwritten parts table with the position where the handwritten part corresponding to the handwritten part ID “002” is arranged as a reference. And a position where the distance from the handwritten part corresponding to the handwritten part ID “003” is reduced at the set reduction ratio is set as a temporary position.

  Then, it is determined whether or not the handwritten part can be placed at the temporary position, and when it is determined that the handwritten part can be placed, the position of the handwritten part is set.

In this example, it is determined that arrangement is possible.
Then, according to the above process, the combined data described with reference to FIG. 4E can be generated by synthesizing the handwritten part whose position has been set.

  Next, another handwritten part registered in the handwritten parts table of FIG. 13 will be described.

In this example, a handwritten part associated with two pages will be described.
The handwritten part 242 corresponding to the handwritten part ID “004” is registered as normal data and registered as having no position dependency. Therefore, the handwritten part 242 is reduced at the set reduction rate. Then, the empty space is confirmed and placed in the empty space.

  Specifically, the empty space is confirmed, it is determined whether the placement is possible, and when it is determined that the handwritten part can be placed, the position of the handwritten part is set.

  On the other hand, if the empty space is confirmed and it is determined that the arrangement is not possible, the icon is arranged in the empty space.

  Then, it is possible to generate the synthesized data described with reference to FIG. 7A by synthesizing the handwritten part whose position has been set in accordance with the above processing.

  That is, as described above, when a projection image is displayed on a projection target such as a whiteboard by the synthesis method, when the input image is captured and synthesized with the projection page data, the projection page data If there is a relevance, it can be related and placed and combined, and if there is no relevance, it can be placed in an empty space and combined to maintain the readability of the combined projected page data. Is possible.

  Further, when the data written and input in the predetermined area which is the blank area of the projection page data cannot be arranged, it is possible to synthesize an icon capable of displaying the comment data. It is possible to maintain the legibility of the synthesized projected page data.

(Embodiment 2)
In the above-described first embodiment, the method of combining the written input data with the projection page data has been described.

  In the second embodiment, a method of adding (combining) projected page data that can be displayed or printed as a separate sheet to the projected page data based on the contents of the data that has been written and input will be described.

  Specifically, as an example, the contents to be written and input to the whiteboard 5 include the date and time of a meeting or the like for projecting projection page data, information on participants, and the like. The information is information related to the entire meeting or the like that projects the projection page data. When the first head projection page data is projected and written to the whiteboard 5, the next projection page is displayed. Even when data is projected, it is often maintained as it is without being erased.

  On the other hand, the information is information related to the entire meeting or the like for projecting the projection page data, but it is considered that the direct relationship with the projection page data itself is weak. Therefore, it is not always necessary to synthesize the information in each projection page data, and the synthesis may affect the readability of the projection page data.

  Therefore, in the second embodiment, when the same image data is included in a plurality of imaging projection data that is written and input from the beginning, the image data is the whole of a conference or the like that projects projection page data. Is added as the first projection page data as the related data related to.

  Further, the image data finally written and input is added as final projection page data as related data related to the entire meeting, such as a future schedule or a meeting item such as a meeting for projecting the projection page data.

  Imaging data obtained by capturing an image displayed on whiteboard 5 acquired by camera 20 according to the second embodiment of the present invention will be described using FIG.

  Referring to FIG. 15A, here, a case where imaging projection data 400 is included in imaging data 300 is shown.

  Here, the imaging projection data 400 is obtained by imaging the projection page data of the first page.

  Then, as shown in FIG. 15 (A), “2008/11/27”, “19: 00”, “Attendees”, “A, B, C,... "Is shown. The image data is the date and time of a meeting or the like for projecting projection page data, information on participants, and the like.

  Referring to FIG. 15B, here, a case where imaging projection data 402 is included in imaging data 310 is shown.

  Here, the imaging projection data 402 is obtained by imaging the projection page data of the second page.

  Then, as shown in FIG. 15 (B), as in FIG. 15 (A), “2008/11/27”, “19: 00” obtained by writing into the whiteboard 5 and acquired by the camera 20 , “Attendees”, “A, B, C,...” Image data 304 is also shown. Similarly, the subsequent image data includes the image data 304.

  Referring to FIG. 15C, here, a case where imaging projection data 404 is included in imaging data 320 is shown.

  Further, image data 306 newly written and inputted to the whiteboard 5 and acquired by the camera 20 is shown.

  Referring to FIG. 15D, here, a case where imaging projection data 406 is included in imaging data 330 is shown.

  Referring to FIG. 15E, here, a case where imaging projection data 408 is included in imaging data 340 is shown.

  Further, image data 308 newly written and input to the whiteboard 5 and acquired by the camera 20 is shown.

  Therefore, the case where the image data 304 includes the imaging data 300 corresponding to the first page to the imaging data 340 corresponding to the last page is shown.

  Further, the case where the image data 308 is included only in the imaging data 340 corresponding to the last page is shown.

  With reference to FIG. 16, a composition process based on the imaging data according to the second embodiment of the present invention will be described.

  In the second embodiment of the present invention, the projection page data is added (synthesized) to the projection page data based on the contents of the input data of the imaging data.

  Specifically, when the agenda date and time, information on participants, etc. are written and input from the beginning, and the same image data is included in a plurality of imaged projection data, the image data is stored in the projection page. As related data related to the whole meeting or the like to project data, it is added as the first projected page data.

  Further, the image data finally written and input is the final projected page as related data related to the entire meeting such as a meeting for projecting projection page data, such as a future schedule or a meeting item such as a meeting for projecting the projection page data. Add as data.

  Referring to FIG. 16A, here, a case is shown in which leading projection page data is added as composite data.

  Specifically, as described in FIGS. 15A to 15E, a plurality of the same image data 304 is included in the imaging data 300 to 340, and the image data 304 is included from the first imaging data 300. Therefore, the case where the projection page data 500 is added to the projection page data as the related data related to the entire meeting or the like for projecting the projection page data is shown. Here, the case where the image data 304 is enlarged and combined so as to have the same size as the projection page data is shown. In this case, in the projection page data, the projection page data 500 is the first page.

  Referring to FIG. 16B, here, the same projection page data 502 as the imaged projection data 400 is shown. The projection page data is the second page in the projection page data.

  Referring to FIG. 16C, here, projection page data 504 that is the same as the imaging projection data 402 is shown. The projection page data is the third page in the projection page data.

  Referring to FIG. 16D, here, projection page data 506 obtained by combining projection page data (captured projection data 404) and image data 306 by the method according to the first embodiment is shown. The projection page data is the fourth page in the projection page data.

  Referring to FIG. 16E, here, projection page data 508 that is the same as the imaging projection data 406 is shown. The projection page data is the fifth page in the projection page data.

  Referring to FIG. 16F, here, the projection page data 510 that is the same as the imaging projection data 408 is shown. The projection page data is the sixth page in the projection page data.

  Referring to FIG. 16G, here, a case where final projection page data is added as composite data is shown.

  Specifically, as described with reference to FIG. 15E, the image data 308 is included in the final imaging data 340, and thus is projected as related data related to the entire meeting or the like that projects the projection page data. The case where it adds to projection page data as page data 512 is shown. Here, the case where the image data 308 is enlarged and combined so as to have the same size as the projection page data is shown. The projection page data is the seventh page in the projection page data.

Hereinafter, a flow for executing the above synthesis method will be described.
The overall flow of the synthesis method according to the second embodiment of the present invention will be described using FIG.

  Referring to FIG. 17, the difference from the flow of FIG. 8 is that a related data synthesis process (step S10) is further provided between step S6 and step S7. The other points are the same.

  With reference to FIG. 18, the subroutine processing of the related data synthesis processing according to the second embodiment of the present invention will be described.

  Referring to FIG. 18, first, a process for confirming a handwritten image is executed (step S100). Specifically, it confirms about the handwritten image extracted by the handwritten character extraction process in step S6. When the projection data is composed of a plurality of projection page data as a result of the handwritten character extraction process, the imaging projection data (projection page) is used for the imaging data (display image data) input corresponding to each projection page data. Data other than (same as data) is extracted, and separated imaging separation data (only handwritten images) is stored in the HDD 104. Therefore, a plurality of image pickup separation data respectively corresponding to the plurality of projection page data are referred to.

And it is judged whether there exists the same handwritten image (step S102).
Specifically, it is determined whether or not the same image data (handwritten image) is present in at least two or more of the plurality of separated image data.

  If it is determined that there is the same handwritten image (YES in step S102), it is determined whether the same image data (handwritten image) exists corresponding to the first page (step S104).

  If it is determined in step S104 that the data exists corresponding to the first page (YES in step S104), the same image data (handwritten image) extraction process is executed (step S106).

Specifically, the same image data included in the plurality of imaging separation data is extracted.
Then, composition (addition) processing of the extracted image data (handwritten image) is executed as projection page data (that is, the first page in the projection data) that is the top separate sheet (step S108). Specifically, the image data is enlarged so that the size of the image data is the same as that of the projection page data, and the enlarged image data is synthesized (added) to the projection data as the first projection page data.

  And the process which updates a handwritten image is performed (step S110). Specifically, the same image data extracted from a plurality of image pickup separation data is deleted, the deleted image data is updated as a plurality of image pickup separation data, and stored in the HDD 104.

Then, the process ends (return).
Therefore, this processing makes it possible to synthesize (add) the head projection page data to the projection data.

  Considering the example described in FIG. 15, as described in FIGS. 15A to 15E, “2008/11/27”, “19: 00”, “Attendees”, “A, B, C,..., Image data 304 is included in the imaging data corresponding to each projection page data.

Further, the image data 304 exists corresponding to the first page.
Therefore, as shown in FIG. 16A, the image data 304 is extracted and enlarged so as to have the same size as the projection page data, and the enlarged image data is projected as the first projection page data. Synthesized (added) to data.

  In this example, the case where it is determined in step S104 whether or not the same image data exists corresponding to the first page has been described. However, even if it does not exist corresponding to the first page. For example, if it is in the first half of the total number of pages, the same processing may be executed.

  With reference to FIG. 19, another subroutine process of the related data synthesis process according to the second embodiment of the present invention will be described.

  Referring to FIG. 19, first, a process for confirming a handwritten image is executed (step S112). Specifically, a plurality of updated imaging separation data stored in the HDD 104 after executing the processing of FIG. 18 is referred to.

  Then, it is determined whether or not the handwritten image exists corresponding to only the final page (step S114). That is, it is determined whether the same image data does not exist corresponding to the other pages, but exists only corresponding to the last page among the plurality of image pickup separation data.

  If it is determined in step S114 that the image exists only corresponding to the last page (YES in step S114), image data (handwritten image) existing corresponding to the last page is extracted (step S116). ).

  Then, synthesis (addition) processing of the extracted image data (handwritten image) is executed as projection page data (that is, the final page in the projection data) to be the final separate sheet (step S118). Specifically, the image data is enlarged so that the size of the image data is the same as that of the projection page data, and the enlarged image data is combined (added) to the projection data as final projection page data.

  Next, a process for updating the handwritten image is executed (step S110). Specifically, the extracted image data is deleted, the imaging separation data is updated, and stored in the HDD 104.

Then, the process ends (return).
Therefore, the final projection page data can be synthesized (added) to the projection data by this processing.

  Considering the example described with reference to FIG. 15, as described with reference to FIG. 15E, the image data 308 of “B team”, “A-1 delivery date 12/28”, “C team”, and “No application”. Exists only in the imaging data corresponding to the final projection page data.

  Therefore, as shown in FIG. 16G, the image data 308 is extracted and enlarged so as to have the same size as the projection page data, and the enlarged image data is projected as final projection page data. Synthesized (added) to data.

  Then, the same handwritten attribute determination processing and handwritten character synthesis processing as those described in the first embodiment are performed on the updated imaging separation data. Since this process is the same as that described in the first embodiment, detailed description thereof will not be repeated.

  That is, for example, the image data 306 shown in FIG. 15C is combined with the projection page data as shown in FIG.

  Therefore, it is possible to improve the readability of the projection page data by adding (combining) projection page data that can be displayed or printed as a separate sheet to the projection data based on the written data content. is there.

  In this example, the case where it is determined in step S114 whether or not the image data exists corresponding to only the last page has been described. However, the present invention is not limited to this. If there are a plurality of the same image data in the latter half, similar processing may be executed.

  In this example, the case where image data such as characters and figures input by handwriting is directly combined on the whiteboard 5 has been described above. However, the present invention is not particularly limited to the case of combining image data input by handwriting. For example, a photograph, paper, or the like pasted on the whiteboard 5 may be captured by the camera 20 to acquire image data, and the image data may be combined.

  In addition, for example, an image projected from another projection device may be captured by the camera 20 to obtain image data, and the image data may be synthesized, even if the photograph is not attached to the whiteboard 5 or the like. good.

(Embodiment 3)
In the first embodiment described above, the method of combining the written and input data displayed on the projection object with the projection page data has been described.

  In the third embodiment, a method will be described in which data written and input to a display body such as a whiteboard other than the projection target is combined with projection page data.

  Referring to FIG. 20, in information processing system 1 # according to the third embodiment of the present invention, a screen that is a projection target for displaying a projected image (projected image) projected from projector 10 that is a projection device. 6 and a whiteboard 7 which is a display body capable of writing input or the like is provided separately. And the camera 20 shall acquire the imaging data (display image data) which imaged the image displayed on the whiteboard 7 which is a display body. Since the other points are the same as those described with reference to FIG. 1, detailed description thereof will not be repeated.

  Next, a flow of a synthesis process between projection data and input display image data in the information processing system according to the third embodiment will be described.

  Specifically, with respect to projection data composed of a plurality of projection page data, each projection page data is combined with display image data input corresponding to each projection page data.

  As a composition processing method in the information processing system according to the third embodiment, display image data input corresponding to each projection page data is arranged and synthesized in a predetermined area which is a blank area of each projection page data.

  Further, when the size is such that the size cannot be arranged in a predetermined area that is a blank area of the projection page data, an icon is arranged and synthesized.

Hereinafter, a flow for executing the above synthesis method will be described.
With reference to FIG. 21, the overall flow of the synthesis method according to the third embodiment of the present invention will be described.

  Referring to FIG. 21, first, it is determined whether there is a photographing request (step S1). In this example, when the projection page data is switched, the PC 25 instructs the camera 20 to take an image displayed on the whiteboard 7 in accordance with a switching instruction input from the user. In this example, as an initial state, it is necessary to capture an image before writing on the whiteboard 7, so that the first projection from the projector 10 as the projection device to the screen 6 as the projection target is performed. When the page data is projected, the PC 25 instructs the camera 20 to take an image according to the input of the instruction. Thereby, imaging data (initial display image data) in an initial state before the writing to the whiteboard 7 is acquired. Even when the final projection page data ends, the PC 25 instructs the camera 20 to take an image in accordance with the input of the instruction.

  If there is a photographing request, an image is acquired (step S2). Specifically, the camera 20 acquires imaging data (display image data) obtained by imaging an image displayed on the whiteboard 7 in accordance with an imaging request from the PC 25.

  Then, the acquired imaging data (display image data) and projection page data are output to the MFP 15 (step S3).

  Then, the MFP 15 stores the image data (display image data) and the projection page data in the RAM 103 or the HDD 104 (step S4).

  Next, it is determined whether the MFP 15 has acquired image data corresponding to all projection page data (step S5). If the acquisition is not completed in step S5, the process returns to step S1, and the same processing is repeated until acquisition of imaging data corresponding to all projection page data is completed.

  If the acquisition is completed in step S5, the process proceeds to step S6.

  As a result, the MFP 15 is ready to execute the image composition process, and the CPU 101 executes a predetermined application to execute the following process in software.

  Note that the following processing is executed for each projection page data in this example.

Specifically, first, handwritten character extraction processing is executed (step S6 #).
Next, handwriting attribute determination processing is performed for determining the attributes of the handwritten characters for the extracted handwritten characters (step S7 #).

  Then, a handwritten character synthesis process for synthesizing a handwritten character is executed (step S8 #). Then, the process ends (END).

(Handwritten character extraction process)
With reference to FIG. 22, a subroutine process of the handwriting extraction process according to the third embodiment of the present invention will be described.

First, an image comparison process is executed (step S12 #).
Specifically, the initial display image data obtained by capturing an image before being written on the whiteboard 7 is compared with the captured image data (display image data) acquired corresponding to, for example, the first page, and matched. Determine whether to do. Also, the acquired imaging data corresponding to the first page is used as initial display image data, and the acquired imaging data (display image data) corresponding to the second page is compared to determine whether they match. The same processing is performed for the following pages.

  Then, it is determined whether or not there is a writing input (step S14 #). Specifically, it is determined whether there is data changed from the initial display image data as compared with the initial display image data. The initial display image data corresponds to data obtained by imaging an image before writing on the whiteboard 7 or acquired image data corresponding to the previous page as an initial state.

  Therefore, based on the comparison, when it is determined that there is changed data, it is determined that there is a write input, and when there is no data, it is determined that there is no write input.

  If it is determined in step S14 # that there is no write input (NO in step S14 #), the process ends (return). This is because in this case, there is no need to execute image composition processing.

  On the other hand, when it is determined in step S14 # that there is a writing input (YES in step S14), handwritten image separation processing is executed (step S16 #).

  Specifically, data other than the initial display image data is extracted from the acquired imaging data. That is, the image data newly written and input this time is acquired as the imaging separation data 202.

And it preserve | saves (step S18 #). Specifically, it is stored in the HDD 104.
Then, the process ends (return).

Note that this processing is executed for each projection page data.
Therefore, according to this example, for example, a handwritten input image written and input on the whiteboard 7 is determined as image data (initial display image data) corresponding to the previous projection page data without being erased. Only the newly written input image data is acquired as the imaging separation data this time, not the input image data.

(Handwriting attribute judgment process)
Next, the handwriting attribute determination process will be described.

The subroutine process of the handwriting attribute determination process will be described with reference to FIG.
FIG. 23 is different from the handwriting attribute determination process described in FIG. 10 in that the handwritten part analysis process is not executed. The other points are the same. In this subroutine processing, a handwritten part table such as handwritten part attributes, related pages, and positions is created.

  Specifically, first, a handwritten image reading process is executed (step S20), and handwritten image data separated by the handwritten image separation process is acquired from the HDD 104. And it isolate | separates into a handwritten part (step S22), and stores each separated handwritten part in a handwritten part table next (step S23). Then, the process ends (return).

A handwritten parts table according to the third embodiment of the present invention will be described with reference to FIG.
Referring to FIG. 24, here, a case where a handwritten part is registered in the handwritten parts table is shown.

Specifically, a case where four handwritten parts are registered is shown as an example.
In this example, the case where IDs “011” to “014” of four handwritten parts separated according to the above-described method are issued is shown. As an example, the case where the attribute “character”, the related page “1”, and the position “(X11, Y11), (X12, Y12)” are registered corresponding to the ID “011” is shown. The position coordinate data indicates a rectangular area having upper left coordinates and lower right coordinates as parameter values.

  In addition, a case is shown where ID “012” is issued and attribute “character”, related page “1”, and position “(X13, Y13), (X14, Y14)” are registered.

  Further, a case is shown in which an ID “013” is issued and an attribute “character”, a related page “1”, and a position “(X15, Y15), (X16, Y16)” are registered.

  In addition, a case is shown in which an ID “014” is issued and an attribute “character”, a related page “2”, and positions “(X17, Y17), (X18, Y18)” are registered.

(Handwriting synthesis process)
Next, the handwritten character synthesis process will be described.

A handwritten character composition process according to the third embodiment of the present invention will be described with reference to FIG.
Referring to FIG. 25, first, a projection image and handwritten image reading process is executed (step S50).

  Specifically, a process of reading projection page data that is a projection image and a separated handwritten image is executed.

Next, a reduction ratio is set (step S52).
Specifically, in accordance with the same method as described above, a reduction ratio for reducing the image data range to match the projection page data range is set.

Next, the handwritten parts table confirmation process is executed (step S54).
Next, a handwritten part is selected (step S56).

  Next, the handwritten part is reduced according to the set reduction rate (step S180).

Next, an empty space is confirmed (step S182).
Specifically, it is determined whether or not an empty space area exists within the entire size of the projection page data. It is assumed that the empty space area is an area having a predetermined size or more.

  Next, it is determined whether or not handwritten parts can be placed in the empty space area within the entire size of the projection page data (step S184).

  If it is determined that a handwritten part can be placed in the empty space area (YES in step S184), the position of the handwritten part is set in the empty space area (step S186).

  On the other hand, if it is determined in step S184 that the icon cannot be placed in the empty space area (NO in step S184), the icon is set in the empty space area.

Then, the process proceeds to step S62.
If it is determined in step S62 that the positions of all the handwritten parts have been set (YES in step S62), a composition process in which the positions of the handwritten parts are set for the projection page data is executed in step S63. (Step S63).

  As a specific example of the synthesis process, the projection page data which is a projection image is expanded in the memory area (first layer), and the handwritten parts and the like are the same as the memory area where the projection page data is expanded. It is assumed that the data is expanded at a set position in a memory area (second layer) having a different size and synthesized by superimposing data expanded in two memory areas (first and second layers). Then, the process ends (return).

  In this method, when data written and input to a display body such as a whiteboard other than the projection object is combined with the projection page data, handwritten parts are arranged in the corresponding projection page data in an empty space. Is possible.

  Further, even when it is not possible to arrange the data written and input in a predetermined area which is a blank area of the projection page data, it is possible to synthesize an icon capable of displaying comment data by giving an instruction.

  Therefore, for example, when a projected image is displayed on the screen, when the input image is captured and combined with the projected page data, the readability of the combined projected page data can be maintained.

  In addition, in order to implement | achieve the information processing system of this invention mentioned above, the computer can be functioned and the program which performs control like the above-mentioned flow can also be provided. Such a program is stored in a computer-readable recording medium such as a flexible disk attached to the computer, a CD-ROM (Compact Disk-Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), and a memory card. And can be provided as a program product. Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. A program can also be provided by downloading via a network.

  Of the program modules provided as part of the computer operating system (OS), necessary modules may be called at a predetermined timing and executed in a predetermined arrangement. In that case, the program itself does not include the module, and the process is executed in cooperation with the OS. A program that does not include such a module can also be included in the program according to the present invention.

  The program may be provided by being incorporated in a part of another program. Even in this case, the program itself does not include the module included in the other program, and the process is executed in cooperation with the other program. Such a program incorporated in another program can also be included in the program according to the present invention.

  The provided program product is installed in a program storage unit such as a hard disk and executed. The program product includes the program itself and a recording medium on which the program is recorded.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  1 Information processing system, 10 projector, 15 MFP, 20 camera, 25, 26 PC, 30 network, 101 CPU, 102 ROM, 103 RAM, 104 HDD, 105 operation panel unit, 106 image reading unit, 107 printing unit, 108 network interface.

Claims (6)

A projection device for projecting projection data onto a projection object;
An image acquisition device for acquiring an image displayed on the projection object as display image data;
Based on the display image data acquired by the image acquisition device, the projection data projected from the projection device and the input image data included in the display image data acquired by the image acquisition device are combined. An image processing device
The image processing apparatus includes:
Determining the relationship between the input image data included in the image acquired by the image acquisition device and a part of the projection data;
An information processing system, in which, when it is determined that there is no relevance to the partial data, the input image data is arranged and synthesized in a predetermined area within the area where the projection data is displayed.   2. The image processing apparatus according to claim 1, wherein when the image processing apparatus determines that there is a relationship with the partial data, the image processing apparatus is arranged and synthesized at a position that maintains a positional relationship with the partial data. Information processing system. The projection data is composed of a plurality of projection page data,
The projection device projects the plurality of projection page data by switching to the projection object,
The image acquisition device acquires, as display image data, an image displayed on the projection object corresponding to each projection page data when switching each projection page data.
The image processing apparatus executes a composition process of each projection page data displayed on the projection object and input image data included in the display image data acquired by the corresponding image acquisition apparatus. The information processing system according to claim 1.   The information processing system according to claim 1, wherein the image processing apparatus superimposes and combines a layer in which the projection data is arranged and another layer in which the input image data is arranged. The image processing apparatus determines whether the input image data can be arranged in the predetermined area within the area of the projection data;
2. The information processing according to claim 1, wherein when the input image data cannot be arranged in the predetermined area, object image data capable of displaying the inputted image data is arranged and synthesized by an instruction. system. The input image data includes at least one of a character, a figure, a photograph pasted on the projection object, paper, and an image projected from another projection apparatus displayed on the projection object. The information processing system according to any one of claims 1 to 5, corresponding to image data acquired by an image acquisition device .

Images