JP6127465B2 - Information processing apparatus, information processing system, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing system, and a program that perform data communication with other apparatuses via a network.

  For example, there is a KJ method as a method of giving an idea to a conference participant. When thinking about ideas using the KJ method, participants write their ideas on sticky notes, paste them on a place where everyone can see them, and change the placement of sticky notes one after another while discussing. By organizing, ideas are organized.

  However, moving and organizing sticky notes is a barrier when implementing the KJ method, which takes considerable time and is considered useful. For this reason, there are some known techniques for reducing labor by digitizing idea idea meetings including the KJ method.

  For example, by writing on a sticky note using a special pen with a pen pressure sensor, handwriting is digitized from the pressure sensor information and camera information, the sticky note has a wireless tag, and a special dot pattern There is a technique for identifying the position from them (see, for example, Patent Document 1).

  In addition, in order to perform the KJ method conveniently without an expensive device such as a touch display, there is a technique for attaching a barcode to a sticky note, taking an entire image and an individual image, and automating the last digitization (for example, Patent Document 2).

  Also, there is a technology that sends a shared screen, a pen tablet, and the entire view of the participants to each other, and displays the materials input by the participants and the materials input by the pen tablet on the shared screen for the purpose of remote brainstorming. Yes (see Patent Document 3, for example).

  In addition, there is a technique in which data is displayed on the I / F of each user when data is input, and then the data is transferred to a common I / F portion that can be recognized by a plurality of users (see, for example, Patent Document 4).

  Each of the above conventional techniques has problems. For example, in Patent Document 1, a large number of special equipment and paper must be prepared, and the versatility is low. Also, in Patent Document 1, it is necessary to take pictures periodically, and there is a high possibility that the flow of discussion will be interrupted there. Further, in Patent Document 2, a sticky note with a barcode is necessary, and it takes time to prepare, or it is expensive to buy the sticky note. Patent Document 3 does not describe the rearrangement of sticky notes and is not suitable for the KJ method. In Patent Document 4, a special display is required, which is difficult to carry and expensive.

  Therefore, in the prior art, when an idea idea meeting is digitized, the introduction cost is high or the meeting cannot be executed smoothly.

  Therefore, an object of the present invention is to reduce the introduction cost when an idea idea meeting is digitized and to smoothly execute the meeting.

An information processing apparatus according to an aspect of the present invention includes a receiving unit that receives data input to one or a plurality of devices, a storage unit that stores the data in which a display position on a display screen is set, and a distance A measurement result acquisition means for acquiring a measurement result obtained by measuring a three-dimensional position of an object in a projection plane direction on which a display screen including each data is projected based on information acquired from a device capable of measuring In response to an instruction recognizing unit for recognizing an instruction by the indicator on the display screen based on the measurement result, and an instruction for an area for aligning the data included in the display screen , according to the color of the data Change control means for grouping the data .

  According to the present invention, when an idea idea meeting is digitized, the introduction cost can be reduced and the meeting can be executed smoothly.

1 is a diagram illustrating an example of an information processing system in Embodiment 1. FIG. 1 is a diagram for explaining a concept of an information processing system in Embodiment 1. FIG. FIG. 2 is a block diagram illustrating an example of a configuration of a server in the first embodiment. FIG. 3 is a block diagram illustrating an example of functions of each device according to the first embodiment. The figure which shows an example of a calibration pattern. The figure which shows an example of the feature point of a picked-up image. The figure explaining ranging with a stereo camera. The figure which shows the correspondence of the coordinate value of a projection original image, and the three-dimensional coordinate value of the projected image. The figure explaining block matching. The figure for demonstrating touch detection. The figure explaining the concept of movement of tag data. The figure which shows an example of the display screen of information processing apparatus. The figure which shows an example of data position information. The figure which shows an example of a canvas image. 5 is a flowchart illustrating an example of input processing of the information processing apparatus according to the first embodiment. 3 is a flowchart illustrating an example of information processing in the first embodiment. 5 is a flowchart illustrating an example of preparation processing for movement processing according to the first embodiment. FIG. 6 is a block diagram illustrating an example of functions of devices in Embodiment 2. 9 is a flowchart illustrating an example of information processing performed by a projection apparatus according to a second embodiment. FIG. 10 is a diagram illustrating a concept of an information processing system according to a third embodiment. FIG. 9 is a block diagram illustrating an example of functions of devices in Embodiment 3. The figure which shows an example of stroke data. 10 is a flowchart illustrating an example of stroke processing according to the third embodiment. The figure explaining one concept of color selection. The figure explaining the concept of alignment of tag data. The figure explaining one concept of the information processing system in the modification 2.

Embodiments of the present invention will be described below with reference to the drawings.
[Example 1]
<System>
FIG. 1 is a diagram illustrating an example of an information processing system 1 according to the first embodiment. An information processing system 1 illustrated in FIG. 1 is a system in which a projection device 10, a server 20, and a plurality of information processing devices 30 are connected via a wired or wireless network and perform data communication with each other. The server 20 is also referred to as an information processing device.

  The information processing system 1 is a system used for an idea idea meeting, for example. Hereinafter, the KJ method will be described as an example for an idea idea meeting. The server 20 may be incorporated in the projection device 10 or the information processing device 30.

  In the example illustrated in FIG. 1, in the information processing system 1, each information processing device 30-n (n = 1 to 3 and n may be 3 or more) is connected to the server 20 via a network. In addition, each information processing apparatus 30-n will be described as the information processing apparatus 30 if it is not necessary to distinguish between them.

  Each information processing device 30 transmits data such as an idea input to the device (for example, sticky note data describing the idea) to the server 20.

  When the server 20 receives each data from each information processing device 30, the server 20 sets an arrangement position in a predetermined display screen for the data and stores the data position information. The predetermined display screen is a screen for displaying each information, and the data position information is information including attributes of each data. Attributes of each data include an arrangement position, a size, a magnification, and the like. The server 20 transmits display screen information of a display screen in which each data is arranged based on the data position information to the projection device 10.

  When receiving the display screen information from the server 20, the projection device 10 projects a display screen based on the display screen information onto the projection plane. At this time, the projection apparatus 10 has a device capable of measuring the distance, and measures the position to the projection plane and the distance to the indicator between the projection plane and the projection apparatus 10. This device is also referred to as a distance measuring sensor.

  As the distance measuring sensor, there are a compound-eye camera (stereo camera), an infrared sensor, an ultrasonic sensor, and the like. Hereinafter, a stereo camera will be described as an example of the distance measuring sensor. The projection apparatus 10 transmits the measurement result (distance information) obtained from the distance measuring sensor to the server 20. Note that the distance measurement sensor may transmit the measurement result directly to the server 20.

  Based on the measurement result received from the projection apparatus 10, the server 20 determines an instruction by the indicator for the projection surface on which the display screen is projected. As this instruction, for example, there is a contact with the projection surface by an indicator. Examples of the pointing object include a hand and a pointing stick.

  When the server 20 recognizes the instruction, the server 20 moves the displayed data according to the instruction, changes the data color, or organizes the data. The server 20 transmits the updated display screen information to the projection device 10 so that the updated display screen is projected onto the projection surface.

<Concept>
FIG. 2 is a diagram illustrating a concept of the information processing system 1 according to the first embodiment. In the information processing system 1 illustrated in FIG. 2, an example in which the server 20 is incorporated in the projection device 10 is illustrated. In the example shown in FIG. 2, the KJ method is electronically performed using the information processing system 1.

  The information processing apparatus 30 illustrated in FIG. 2 is, for example, a tablet terminal, and is displayed on the display unit as if the data handwritten on the tablet terminal was written on a sticky note.

  Each information processing device 30 generates handwritten data as sticky note data, and transmits image information of the sticky note data (for example, information in a bitmap format) to the projection device 10. A processing unit having the function of the server 20 of the projection apparatus 10 determines the arrangement position of these sticky note data and generates display screen information of the display screen including the sticky note data.

  The projection device 10 projects a display screen including each data transmitted from each information processing device 30 onto a projection surface. A distance measuring sensor 40 mounted on the projection apparatus 10 measures a distance to an object existing in the projection plane direction. The projection device 10 detects a touch on the projection surface based on the measurement result of the distance measuring sensor 40. Hereinafter, as an instruction, a touch will be described as an example.

  When detecting a touch, the projection device 10 moves the sticky data, changes the color of the sticky data, aligns the sticky data, or outputs the final result according to the reference when the touch is detected.

<Configuration>
FIG. 3 is a block diagram illustrating an example of the configuration of the server 20 according to the first embodiment. The server 20 illustrated in FIG. 3 includes a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a drive device 104, and a network I / F unit 106. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 101 is a CPU that controls each device, calculates data, and processes in a computer. The control unit 101 is an arithmetic device that executes a KJ method support program stored in the main storage unit 102 or the auxiliary storage unit 103. After receiving, calculating, and processing data from the input unit 107 or the storage device, The data is output to the display unit 108 or a storage device.

  The main storage unit 102 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 101. It is.

  The auxiliary storage unit 103 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 104 reads a program from the recording medium 105, for example, an SD card, and installs it in the storage device. Further, for example, a KJ method support program is stored in the recording medium 105, and the KJ method support program is installed in the server 20 via the drive device 104. The installed KJ method support program can be executed by the server 20.

  The network I / F unit 106 is an interface between the server 20 and a peripheral device having a communication function connected via a network such as a LAN or a WAN constructed by a data transmission path such as a wired and / or wireless line.

  In addition to the above configuration, the information processing apparatus 30 further includes an input unit and a display unit. The input unit includes a keyboard having cursor keys, numeric input, various function keys, and the like, a mouse and a slide pad for selecting keys on the display screen of the display unit, and the like. The input unit may be a touch panel integrated with the display unit.

  The display unit includes an LCD (Liquid Crystal Display) or the like, and displays according to display data input from the control unit 101. In addition, the display unit may be separate from the information processing apparatus 10. In this case, the information processing apparatus 10 is provided with a display control unit that controls the display unit.

<Function>
FIG. 4 is a block diagram illustrating an example of the function of each device according to the first embodiment. First, the function of the information processing apparatus 30 will be described. The information processing apparatus 30 illustrated in FIG. 4 includes a creation unit 201 and a communication unit 202. For example, the creating unit 201 creates the input data as sticky note data by the KJ method. The sticky note data is generated as image data including handwritten data, for example.

  The communication unit 202 transmits the sticky note data created by the creation unit 201 to the server 20 via the network. The communication unit 202 can be realized by, for example, a network I / F unit, and the creation unit 201 can be realized by, for example, an input unit or a control unit.

  Next, functions of the server 20 will be described. The server 20 includes a communication unit 301, an arrangement unit 302, a storage unit 303, a display control unit 304, a measurement result acquisition unit 305, an instruction recognition unit 306, a change control unit 307, and an output unit 308.

  The communication unit 301 transmits / receives data to / from the information processing apparatus 30 and the projection apparatus 10. For example, the communication unit 301 functions as a receiving unit that receives each data (for example, tag data) input to one or a plurality of information processing apparatuses 30. Further, the communication unit 301 transmits display screen information to the projection apparatus 10 and receives a measurement result of the distance measuring sensor from the projection apparatus 10.

  The arrangement unit 302 sets a display position in the display screen for the received data and gives position information.

  The storage unit 303 stores each data (data position information) in which the display position on the display screen is set. The storage unit 303 holds the position, size, image data, and the like of each data as data position information.

  Based on the data position information stored in the storage unit 303, the display control unit 304 arranges the image information of each data on the display screen, and generates display screen information. In addition, when the data is changed by the change control unit 307 described later, the display control unit 304 updates the display screen according to the change.

  The measurement result acquisition unit 305 is a measurement in which the three-dimensional position of the object in the projection plane direction on which the display screen including each data is projected is measured based on the information acquired from the device capable of measuring the distance (ranging sensor 40). Get the result. Objects in the direction of the projection plane include projection planes (walls and whiteboards), hands and pointers.

  The instruction recognizing unit 306 recognizes an instruction from the pointing object on the projected display screen based on the measurement result of the measurement result acquiring unit 305. For example, the instruction recognizing unit 306 recognizes an instruction (for example, touch) if the distance between the projection surface whose plane is estimated and the indicator is within a threshold.

  When the instruction is recognized by the instruction recognizing unit 306, the change control unit 307 determines what the instruction is according to a predetermined standard, and adds a change corresponding to the instruction to each data. The change control unit 307 outputs the change contents to the display control unit 304.

  For example, when an instruction for a display area of one data included in the display screen is recognized and then an instruction for another area included in the display screen is recognized, the change control unit 307 stores one data in the other area. Move.

  When there is a result output instruction from the user, the output unit 308 stores or outputs the captured image of the display screen at the time of the result output or the tag data for each category in the storage unit 303.

  The communication unit 301 can be realized by the network I / F unit 106, for example. The storage unit 303 can be realized by the main storage unit 102 and / or the auxiliary storage unit 103, for example. The arrangement unit 302, the display control unit 304, the measurement result acquisition unit 305, the instruction recognition unit 306, the change control unit 307, and the output unit 308 can be realized by the control unit 101 and the main storage unit 102 as a work memory, for example.

  Next, functions of the projection apparatus 10 will be described. The projection apparatus 10 is, for example, a projector, and includes a communication unit 401, a projection unit 402, and a distance measurement unit 403.

  The communication unit 401 receives display screen information from the server 20 and outputs it to the projection unit 402. In addition, the communication unit 401 transmits the measurement result acquired from the distance measuring unit 403 to the server 20.

  The projection unit 402 projects a display screen based on the acquired display screen information onto the projection plane. The distance measuring means 403 is, for example, the distance measuring sensor 40, and measures the distance to an object existing in the projection plane direction. The distance measuring unit 403 outputs the measurement result to the communication unit 401.

<Each process>
Next, each process when an idea idea meeting by the KJ method is electronically performed using the information processing system 1 will be described.

≪Move processing preparation≫
The server 20 performs a preparation process for moving the projected data (for example, sticky note data). When the server 20 is incorporated in the projection apparatus 10, the projection apparatus 10 performs this preparation process.

  First, the projection device 10 projects the calibration pattern onto the projection plane. FIG. 5 is a diagram illustrating an example of a calibration pattern. As shown in FIG. 5, the calibration pattern p <b> 11 is projected on the projection plane by the projection device 10.

  The calibration pattern p11 is photographed by the distance measuring sensor 40 of the stereo camera. The distance measuring sensor 40 transmits an image obtained by photographing the calibration pattern p <b> 11 (hereinafter also referred to as a photographed image) to the server 20 via the projection device 10.

  The distance measuring means 403 extracts feature points from the captured image. The feature points are extracted by the left and right cameras using a technique such as SIFT (Scale Invariant Feature Transform). The distance measuring unit 403 may obtain a corner of the photographed image from the left and right binocular images by using the Harris and Stephens corner detection method or the like, and may use this as a feature point (http: //d.hatena. ne.jp/rebelwidow/20101108/1289240724, C. Harris and M. Stephens, A combined corner and edge detector, Proceedings of the 4th Alvey Vision Conference. (1988) 147-151).

  FIG. 6 is a diagram illustrating an example of feature points of a captured image. In the example illustrated in FIG. 6, the corner is a feature point with respect to the captured image.

  The distance measuring means 403 obtains the position coordinates of the feature points in the left and right images, and measures the distance based on the principle of triangulation. The distance measuring unit 403 measures a distance using, for example, an image photographed by a stereo camera.

  FIG. 7 is a diagram for explaining distance measurement by a stereo camera. In FIG. 7, the stereo camera includes a first imaging lens 41a and a second imaging lens 41b. In addition, the stereo camera 40 includes a first imaging element 42a and a second imaging element 42a that are arranged in the direction of the back side of the first imaging lens 41a and the second imaging lens 41b (the direction opposite to the direction of the projection target). Imaging element 42b. Here, an area sensor, a surface sensor, a two-dimensional sensor, or the like can be used as the imaging element.

  The first imaging lens 41a and the second imaging lens 41b are spaced apart by a predetermined distance D (hereinafter referred to as a baseline length). The optical axis 43ax of the first imaging lens 41a and the optical axis 43bx of the second imaging lens 41b are parallel. The first image sensor 42a and the like include a light receiving surface that forms an image of an object on the front side (opposite the back side) surface. The optical axes 43ax and the like of the first imaging lens 41a and the like are positioned so as to coincide with the diagonal center of the light receiving surface of the first imaging element 42a and the like.

  The distance measuring means 403 extracts a plurality of corresponding points by the above-described (operation for extracting corresponding points) based on the two pieces of image data captured by the stereo camera. Further, the distance measuring unit 403 calculates distance data regarding the distance from the image processing apparatus to the corresponding point (projection object) by the triangulation principle at the plurality of extracted corresponding points.

  Specifically, the first image of the projection object obtained through the first imaging lens 41a is formed on the light receiving surface on the first imaging element 42a. On the other hand, the second image of the projection object obtained through the second imaging lens 41b is formed on the light receiving surface on the second imaging element 42b.

  Here, the first image and the second image of the projection object are displaced by the parallax Δ and formed on the respective light receiving surfaces. At this time, the imaging elements 42a and 42b photoelectrically convert the light and darkness of the first image and the second image into the amount of charge, and output the result to the distance measuring unit 403 as a pixel output signal. The distance measuring means 403 compares the pixel output signals of the image sensors 42a and 42b, and detects the parallax Δ at the corresponding point.

  Here, assuming that the parallax Δ, the base line length D, the separation distance L between the image processing apparatus and the corresponding point, and the focal length f of the imaging lens 41a and the like, the expression (1) is established on the condition that L >> f ( The principle of triangulation).

In this case, D and f are known. The distance measuring means 403 calculates the separation distance L from the detected parallax Δ using the equation (1). Further, the distance measuring unit 403 acquires distance data regarding the distance of the three-dimensional point group using the calculated separation distance L as distance data corresponding to the corresponding point.

(Operation to estimate the plane)
A process in which the distance measuring unit 403 estimates a plane corresponding to the projection object will be described. The distance measuring means 403 calculates regression plane data relating to the plane corresponding to the projection target based on the calculated distance data of the three-dimensional point group (corresponding point).

The distance measuring means 403 _obtains n corresponding points (x _ABi , y _ABi , z _ABi ) (i = 1 to n) as a three-dimensional point group. Here, the y-axis direction is the optical axis direction of the stereo camera 40. Y _ABi is the y component of the separation distance L between the projection target (corresponding point (x _ABi , z _ABi )) measured by the distance measuring means 403 and the image processing apparatus.

The distance measuring means 403 defines the regression plane equation as y = ax + bz + c in order to calculate the regression plane from the three-dimensional point group by regression analysis. Here, Equation (2) holds for the regression plane and the three-dimensional point group.

Here, the variable of Formula (2) is Formula (3).

At this time, e _i indicates a residual.

  Next, the normal equation becomes Equation (4).

Therefore, β is expressed by Equation (5).

As described above, the regression plane (y = ax + bz + c) can be obtained by calculating the constants a, b, and c that minimize the sum of squares of the residuals e _i using the least square method. The distance measuring unit 403 acquires constants a, b, and c of a regression plane equation (y = ax + bz + c) as regression plane data.

  The distance measuring means 403 sets ax + by + cz + d = 0 for the obtained regression plane data, and records an expression representing this regression plane. The distance measuring unit 403 holds a correspondence relationship between the three-dimensional coordinates of the feature points and the coordinate values of the feature points of the captured image. The distance measuring means 403 may perform perspective projection conversion in order to convert three-dimensional coordinates into coordinate values on the projection target image from a large number of feature points. The distance measuring means 403 obtains the coordinate value of the projection target image using Expression (6) when the three-dimensional coordinates are (X, Y, Z) and the coordinate value of the projection target image is (u, v). Can do.

FIG. 8 is a diagram illustrating a correspondence relationship between the coordinate values of the projection source image and the three-dimensional coordinate values of the projected image. In the example shown in FIG. 8, the coordinate value (u12, v12) of the projection source image corresponds to the three-dimensional coordinate value (X12, Y12, Z12) of the projected image.

  The distance measuring unit 403 transmits the measurement result including the obtained coordinate value on the projection target image to the server 20. Note that the processing for obtaining the distance data of the distance measuring unit 403 and the plane estimation processing may be performed on the server 20 side.

≪Movement process≫
Next, the sticky note data moving process will be described. First, the server 20 establishes communication with the information processing apparatus 30. The server 20 transmits the sticky note data acquired from the information processing device 30 to the projection device 10. The projection device 10 projects the tag data on the projection plane.

  The distance measuring sensor 40 captures the projection plane and uses the captured image to measure the distance to the vicinity of the screen.

  The distance measuring unit 403 assumes a fixed-size block, and obtains where the same point photographed by the left and right cameras is on the image by matching on the left and right. The distance measuring means 403 sets the coordinates of the candidate points to the left (xl, yl) and right (xr, yr), finds the right and left sides of a block of several peripheral pixels, and maximizes the following SAD or SSD value. This corresponds to the corresponding pixel (see, for example, http://www.cvl.iis.u-tokyo.ac.jp/~vanno/Programming/stereo_program.pdf).

FIG. 9 is a diagram illustrating block matching. In the example illustrated in FIG. 9, it is assumed that the matching process between the feature point U _L = (x _L , y) of the first image and the feature point U _R = (x _R , y) of the second image is performed. .

The distance measuring means 403 obtains a corresponding point between the first image and the second image based on the SAD obtained by Expression (7) and the SSD obtained by Expression (8). Note that part of the processing of the distance measuring unit 403 may be performed on the server 20 side.

  The instruction recognition unit 306 performs a recognition process for determining whether there is an instruction for the projection plane. The instruction recognizing means 306 obtains the distance between the three-dimensional coordinate value obtained by measuring a number of points on the projection image and the regression plane using the equation (9).

  For example, the instruction recognition unit 306 can calculate the distance between the point (x0, y0, z0) and the plane ax + by + cz + d = 0 by Expression (9).

The instruction recognizing unit 306 obtains a point group in which the obtained distance is within a certain range. This is for touch detection on the projection surface. Note that the instruction recognition unit 306 may perform skin color detection and use the distance between the feature point (edge point) of the skin color portion and the regression plane. Further, the instruction recognizing unit 306 may detect a specific color at the tip of the pointer, use the color portion as a feature point, and use the distance between the feature point and the regression plane.

  FIG. 10 is a diagram for explaining touch detection. In the example shown in FIG. 10, the case where the projection surface is touched with a hand is shown. As shown in FIG. 10, if the distance to the projection plane is obtained, the instruction recognition unit 306 can recognize a touch if there is a feature point (black rhombus) of the hand within a predetermined distance from the projection plane. . As the predetermined distance, an appropriate value may be set in advance through experiments or the like.

  In addition, the instruction recognition unit 306 does not recognize a touch if there is a hand feature point (white triangle) at a distance from the projection plane beyond a predetermined distance.

  If there is sticky note data at a location corresponding to a point cloud within a predetermined distance from the projection plane, the change control means 307 detects that the sticky note data has been selected. The change control unit 307 refers to the data position information stored in the storage unit 303 and determines which tag data has been selected.

  Here, the position of the tag data is recorded in the storage unit 303 in association with the coordinate value of the projection source image. Further, from the perspective projection matrix P, the corresponding position of the projection source coordinates is obtained from the three-dimensional point.

  If there is a detected hand at a position corresponding to the tag data in the projection source image, the change control unit 307 can determine that the tag data has been touched. The change control means 307 determines that the place is a place where the sticky note data is moved when a touch is detected in a place where there is nothing in a state where a piece of sticky note data is selected.

  The change control unit 307 instructs the display control unit 304 to change the coordinate value of the tag data to the coordinate value of the movement destination. When the coordinate value of the tag data is changed, the display control unit 304 similarly changes the data position information in the storage unit 303.

  When the coordinate value of the tag data is changed, the display control unit 304 transmits the display screen information of the updated display screen to the projection device 10. Therefore, the projected display screen is updated to the latest image.

  FIG. 11 is a diagram for explaining the concept of movement of tag data. In the example shown in FIG. 11A, tag data s11 and tag data s12 are displayed on the projection screen. In addition, a hand exists near the projection surface, but the distance from the projection surface of the hand is greater than a threshold value (predetermined distance).

  In the example shown in FIG. 11B, the instruction recognition unit 306 recognizes the touch. This is because the hand is within a predetermined distance from the projection plane. The change control unit 307 recognizes that the touch is in the tag data s11 and determines that the tag data s11 is in a selected state.

  In the example shown in FIG. 11C, the hand is moved. In the example shown in FIG. 11C, no touch has been detected yet.

  In the example shown in FIG. 11D, the instruction recognition unit 306 recognizes a touch. Since the change control unit 307 is a touch in another area, the change control unit 307 determines that the tag data s11 in the selected state is moved.

  The change control unit 307 may change the position so that, for example, the center of the touched hand portion and the center of the tag data s11 coincide. Further, the placement position of the tag data is divided into sections on the display screen, the change control means 307 detects which section is touched, and the change control means 307 so as to place the tag data s11 in the section. May change.

≪ Sticky note creation process ≫
Next, the tag creation process will be described. FIG. 12 is a diagram illustrating an example of a display screen of the information processing apparatus 30. The user can draw handwritten characters with the electronic pen in the center of the screen of the information processing apparatus 30 shown in FIG.

  There are buttons for operations on the right edge of the screen. By using the operation buttons, the user can transmit image information of handwritten characters to the server 20, output (print, etc.) sticky note data stored in the server 20, and establish or disconnect communication. . The user can also select the color of the sticky note and erase (cancel) the written character.

  The information processing apparatus 30 establishes communication with the server 20 before starting use. For example, characters and figures are drawn on the touch panel, and the information processing apparatus 30 temporarily stores these as image data in the terminal.

  When the user presses a “Send” button prepared on the display unit of the information processing device 30, the information processing device 30 transmits image data of handwritten data drawn on the display unit to the server 20.

  When the server 20 acquires the image data from the information processing apparatus 30, the server 20 creates sticky note data based on the image data. The arrangement unit 302 assigns a predetermined initial position and magnification to the tag data. For example, when the example shown in FIG. 2 is used, the arrangement unit 302 sets the position to the “unclassified” position.

  The data position information is information including image data and the position, size, magnification, category, and the like of the image data.

≪ Sticky note display process≫
Next, sticky note display processing will be described. The sticky note is converted into digital data, and the data position information includes the following data.
-Upper left coordinates (x, y) of the display position
・ Display magnification (a)
・ Image width and height (w, h)
・ Image data (i)
FIG. 13 is a diagram illustrating an example of the data position information. In the example shown in FIG. 13, the position, magnification, size, and image data of the tag data are associated. As shown in FIG. 2, when there is a classification on the display screen, the data position information may be further associated with a category, and may hold which category the tag data belongs to. In the example shown in FIG. 2, the category can be determined by the position of the y coordinate.

  The display control unit 304 creates a raster image that can be projected by the projection apparatus 10, and transmits display screen information of the raster image to the projection apparatus 10. The projection device 10 performs display by projecting the raster image. A raster image is created as follows.

  First, the display control unit 304 creates an image in which nothing is drawn with the maximum number of pixels that the projection apparatus 10 can project. Hereinafter, this image is referred to as a canvas image.

  The display control unit 304 acquires one piece of tag data from the data arrangement information. The display control means 304 generates an image by multiplying the image data (i) by the display magnification (a) and sets it as a tag image. The display control unit 304 pastes the tag image on the canvas image according to the display position (x, y). The display control unit 304 performs this process for all the tag data.

  When all the sticky note data is attached to the canvas image, the display control unit 304 transmits the updated display screen information to the projection apparatus 10. The projection apparatus 10 replaces the projection image with the received canvas image and refreshes the display. Further, a part of the processing of the display control unit 304 may be performed by the projection apparatus 10.

  FIG. 14 is a diagram illustrating an example of a canvas image. The example shown in FIG. 14 is a canvas image generated based on the data position information shown in FIG.

≪Output processing≫
Next, output processing will be described. The output process is a process for outputting the final result of the electronic KJ method described above.

  The server 20 receives an operation from the projection device 10, touching a specific icon on the projection surface, or an instruction from the information processing device 30, and confirms whether there is an output instruction. When there is an output instruction, the server 20 generates an image from the data position information.

  The output unit 308 may generate an image having a higher resolution than that for display. The output unit 308 can generate a high-resolution image by taking a large canvas image and increasing the magnification.

  There are various output methods. For example, it is possible to send an image to a person concerned by e-mail, send the image to a printing machine to print the image, or save it as an image file in the storage unit 303 or the like. As an image file, tag data may be stored in association with each category so that it can be edited later.

≪End processing≫
Next, the termination process will be described. The server 20 accepts an operation on the projection apparatus 10, a touch of a specific icon on the projection plane, or an instruction from the information processing apparatus 30, and confirms whether there is an instruction to end.

  If there is no end instruction, the server 20 continues the projection / movement, and if there is an end instruction, the server 20 disconnects the communication and ends the above processing.

<Operation>
Next, the operation of the information processing system 1 in the first embodiment will be described. First, input processing of the information processing apparatus 30 will be described.

  FIG. 15 is a flowchart illustrating an example of input processing of the information processing apparatus 30 according to the first embodiment. In step S101 illustrated in FIG. 15, the communication unit 202 establishes communication with the server 20.

  In step S102, the creation unit 201 receives data from a user by handwriting input or key input, and creates sticky note data describing this data.

  In step S103, the information processing apparatus 30 determines whether there is a transmission instruction by pressing a transmission button or the like. If there is a transmission instruction (step S103-YES), the process proceeds to step S104, and if there is no transmission instruction (step S103-NO), the process proceeds to step S105.

  In step S <b> 104, the communication unit 202 transmits the created tag data to the server 20.

  In step S105, the information processing apparatus 30 determines whether or not there is an end instruction by pressing an end button or the like. If there is an end instruction (step S105-YES), the process proceeds to step S106, and if there is no end instruction (step S105-NO), the process returns to step S102.

  In step S <b> 106, the communication unit 202 transmits an end command to the server 20 and disconnects communication with the server 20.

  Next, operations of the server 20 and the projection apparatus 10 will be described. FIG. 16 is a flowchart illustrating an example of information processing in the first embodiment. In step S201 illustrated in FIG. 16, the server 20 prepares for the movement process described above. This process will be described later with reference to FIG.

  In step S <b> 202, the communication unit 301 establishes communication with the information processing apparatus 30. In step S <b> 203, the display control unit 304 generates an initial display screen and transmits it to the projection apparatus 10. In the projection apparatus 10, an initial display screen is projected and displayed.

  In step S <b> 204, the communication unit 301 determines whether data (sticky note data) has been received from each information processing apparatus 30. If data has been received (step S204—YES), the process proceeds to step S205, and if data has not been received (step S204—NO), the process proceeds to step S207.

  In step S205, the arrangement unit 302 creates sticky note data. In step S206, the arrangement unit 302 arranges the tag data at the initial display position and gives position information.

  In step S207, the distance measuring unit 403 measures the position to the object in the projection direction based on the information from the distance measuring sensor 40. The measurement result acquisition unit 305 acquires the measurement result from the distance measurement unit 403.

  In step S208, it is assumed that the instruction recognizing unit 306 recognizes an instruction from the pointing object on the projection surface. In this recognition, if the distance between the pointing object and the projection plane is within a threshold value, the instruction (touch on the projection plane) is recognized.

  In step S209, the change control unit 307 determines whether there is a movement instruction. If there is a movement instruction (step S209—YES), the process proceeds to step S210, and if there is no movement instruction (step S209—NO), the process proceeds to step S211.

  In step S210, the change control unit 307 updates the tag data. This update is a process of changing the coordinates of the tag data.

  In step S211, the display control unit 304 rearranges the display position of the sticky note data at the changed position.

  In step S212, the server 20 determines whether there is an output instruction. If there is an output instruction (step S212—YES), the process proceeds to step S213, and if there is no output instruction (step S212—NO), the process proceeds to step S215.

  In step S213, the output unit 308 creates output data. Which output data (image data, image file, print image, etc.) to create may be selected by the user or the like. In step S214, the output unit 308 outputs the result information.

  In step S <b> 215, the communication unit 301 determines whether there is an end instruction by acquiring an end instruction from the information processing apparatus 30. If there is an end instruction (step S215-YES), the process proceeds to step S216, and if there is no end instruction (step S215-NO), the process returns to step S204. In step S <b> 216, the communication unit 301 disconnects communication with the information processing apparatus 30.

  Next, the preparation process for the movement process will be described. FIG. 17 is a flowchart illustrating an example of preparation processing for movement processing according to the first embodiment.

  In step S301 shown in FIG. 17, the projection unit 402 projects a calibration pattern (see, for example, FIG. 5).

  In step S302, the distance measuring unit 403 captures the calibration pattern projected on the projection surface and generates a captured image.

  In step S303, the distance measuring unit 403 extracts feature points from the captured image (see, for example, FIG. 6).

  In step S304, the distance measuring means 403 calculates the distance to the feature point based on the principle of triangulation.

  In step S305, the distance measuring unit 403 obtains the three-dimensional coordinates of the feature points as described above.

  In step S306, the distance measuring unit 403 obtains a regression plane as described above. In step S307, the distance measuring unit 403 obtains a perspective projection matrix P using equation (6) and the like. Thereby, the preparation process of the movement process is completed.

  As described above, according to the first embodiment, when an idea idea meeting is digitized, the introduction cost can be reduced and the meeting can be executed smoothly.

  Further, according to the first embodiment, special hardware is not required by adding the distance measuring sensor 40 to the projection apparatus 10. As the distance measuring sensor 40, one already sold can be used, and the distance measuring sensor 40 is inexpensive. In addition, the projector as the projection device 10 is easy to carry and provides a large screen. In addition, by introducing the information processing system 1 in the first embodiment, no special work such as taking a photograph is required during the discussion, and the meeting can be smoothly executed without hindering the flow of the discussion.

[Example 2]
Next, an information processing system according to the second embodiment will be described. The second embodiment is an example in which the function of the server 20 is incorporated in the projection device 10, and the display magnification of the tag data to be projected can be changed. The projection apparatus in the second embodiment is denoted as a projection apparatus 10A.

<System and configuration>
The system and configuration in the second embodiment are different from the first embodiment in that the server 20 in the first embodiment is incorporated in the projection apparatus 10A.

<Function>
FIG. 18 is a block diagram illustrating an example of the function of each device according to the second embodiment. Since the function of the information processing apparatus 30 is the same as that of the first embodiment, the description thereof is omitted.

  The projection apparatus 10A having a server function includes an arrangement unit 302, a storage unit 303, a display control unit 304, an instruction recognition unit 306, a change control unit 307, an output unit 308, a projection unit 402, a distance measurement unit 403, a communication unit 501, and A magnification changing unit 502 is provided.

  Of the functions of the projection apparatus 10A shown in FIG. 18, the same functions as those shown in FIG. In the following, different functions in the second embodiment will be mainly described.

  The communication unit 501 receives each data (for example, tag data) from each information processing apparatus 30. Further, the communication unit 501 may transmit the output data from the output unit 308 to each information processing apparatus 30.

  The magnification changing unit 502 changes the display magnification of the tag data according to the number of tag data included in the projected display screen, for example. For example, when the number of sticky note data in the display screen is equal to or greater than a threshold, the magnification changing unit 502 rewrites the magnification (a) portion of the data position information to be smaller than the current value. Further, the magnification changing unit 502 may increase the magnification (a) of the data position information from the current value if the number of tag data on the display screen is less than the threshold value.

  The display control unit 304 may generate a display screen after the magnification is updated, and the projection unit 402 may refresh the projected display screen to the updated display screen. The magnification changing unit 502 may display a magnification change button on the projection screen, and may change the magnification when detecting that the magnification change button is touched.

  Thereby, since the magnification of the tag data can be changed according to the number of tag data in the display screen, the tag data can be displayed and projected in an appropriate size.

<Operation>
Next, the operation of the information processing system in the second embodiment will be described. FIG. 19 is a flowchart illustrating an example of information processing performed by the projection apparatus 10A according to the second embodiment. The process of steps S401 to S411 shown in FIG. 19 is the same as the process of steps S201 to S211 shown in FIG.

  In step S412, the magnification changing unit 502 determines whether there is an instruction to change the magnification. As for the magnification change instruction, the display control means 304 may issue a change instruction to the magnification changing means 502 according to the number of sticky note data in the display screen, or it is explicitly instructed by the user via an operation button or the like. Also good.

  If there is a magnification change instruction (step S412-YES), the process proceeds to step S413, and if there is no magnification change instruction (step S412-NO), the process proceeds to step S415.

  In step S413, the magnification changing unit 502 updates the tag data. This update is a process for changing the magnification of the tag data.

  In step S414, the display control unit 304 updates the display screen so that the tag data is displayed at the changed magnification.

  The processes in steps S415 to S419 are the same as the processes in steps S212 to S216 shown in FIG.

  As described above, according to the second embodiment, when an idea idea meeting is digitized, the introduction cost can be reduced and the meeting can be executed smoothly. Further, according to the second embodiment, even when the amount of sticky note data increases, many sticky notes can be displayed side by side. This effect is not found on paper sticky notes.

[Example 3]
Next, an information processing system according to the third embodiment will be described. In the third embodiment, the convenience can be further increased by introducing the electronic blackboard mechanism together. In the third embodiment, an example in which the server 20 is incorporated in the projection apparatus 10 will be described. The projection apparatus in the third embodiment is displayed as the projection apparatus 10B.

<Concept>
FIG. 20 is a diagram illustrating a concept of the information processing system according to the third embodiment. As shown in FIG. 20, in the third embodiment, a writing instrument (for example, a dedicated pen) 60 having an infrared light emitting means 602 is used. When the tip of the writing instrument 60 is pressed, the pressure is detected by the pressure-sensitive means 601, and the tip of the writing instrument 60 emits infrared light by the infrared light emitting means 602.

  A distance measuring sensor 40B (stereo camera) capable of detecting infrared light also detects a light emission position in the projection plane and detects that a handwritten stroke is drawn. The projection device 10B updates the handwritten stroke data while drawing a straight line connecting the position of the writing instrument 60 detected immediately before (the position of the infrared light emitting unit) and the newly detected position.

<System and configuration>
The system and configuration in the third embodiment are different from the first embodiment in that the server 20 in the first embodiment is incorporated in the projection apparatus 10B.

<Function>
FIG. 21 is a block diagram illustrating an example of the function of each device according to the third embodiment. Since the function of the information processing apparatus 30 is the same as that of the first embodiment, the description thereof is omitted.

  The writing instrument 60 can recognize a stroke with respect to the projection plane, and includes a pressure-sensitive unit 601 and an infrared light emitting unit 602. The pressure sensitive means 601 detects the pressure when the tip of the writing instrument 60 is pressed. The pressure sensing means 601 instructs the infrared light emitting means 602 to emit infrared light when pressure is detected.

  The infrared light emitting means 602 emits infrared light when receiving an instruction of infrared light emission from the pressure sensitive means 601.

  The projection apparatus 10B having a server function includes an arrangement unit 302, an instruction recognition unit 306, a change control unit 307, an output unit 308, a projection unit 402, a distance measurement unit 403, a communication unit 501, a stroke recognition unit 701, a storage unit 702, and Display control means 703 is provided.

  The functions of the projection apparatus 10B shown in FIG. 21 that are the same as the functions shown in FIG. 4 and FIG. In the following, different functions in the third embodiment will be described.

  When the stroke recognition means 701 detects the emission of infrared light received by the camera which is the distance measuring sensor 40B, the stroke recognition means 701 calculates the emission position. At this time, the camera which is the distance measuring sensor 40B is in a state where the infrared cut filter is removed and infrared light can be received.

  The stroke recognizing unit 701 can recognize stroke data by sequentially calculating and recording the light emission position. The stroke data is stored in the storage unit 702, for example. This stroke data is updated one after another.

  The storage means 702 records the coordinate values recognized by the stroke recognition means 701 one after another as stroke data.

  FIG. 22 is a diagram illustrating an example of stroke data. In the example shown in FIG. 22, the stroke color and thickness are stored at the head of the stroke data. In the stroke data, a large number of coordinate values are recorded.

  Returning to FIG. 21, the display control means 703 can display strokes by drawing the coordinate values in order of stroke color and thickness in the order of recording the stroke data. This stroke data is updated one after another until the writing instrument 60 detects the pressure until it stops detecting it, and coordinate values are added.

  Once the pressure sensitivity of the writing instrument 60 is lost, new stroke data is generated the next time pressure is sensed, and stroke data is added one after another.

  As the thickness and color, a thickness and color preset as initial values may be used. In the example shown in FIG. 22, “5” is set as the initial value of the thickness, and “red” is set as the initial value of the color.

  Further, the thickness and / or color may be changed by the user. Since the thickness and color can be changed, the stroke data can be changed from, for example, a red and thick stroke to a blue and thin stroke.

<Operation>
Next, the operation of the information processing system in the third embodiment will be described. Hereinafter, the stroke process will be mainly described.

  FIG. 23 is a flowchart illustrating an example of stroke processing according to the third embodiment. In step S501 shown in FIG. 23, the distance measuring sensor 40B determines whether or not infrared light is detected. If infrared light is received by the distance measuring sensor 40B (step S501-YES), the process proceeds to step S502. If infrared light is not received (step S501-NO), the process ends.

  In step S502, the stroke recognizing unit 701 calculates the coordinate value of the emission position of the infrared light received by the distance measuring sensor 40B. The stroke recognition unit 701 updates the stroke data by recording the recognized coordinate value in the stroke data stored in the storage unit 702.

  In step S503, when the stroke data stored in the storage unit 702 is updated, the display control unit 703 visualizes the stroke based on the updated stroke data. The stroke visualized by the display control unit 703 is projected onto the projection plane by the projection unit 402. The processes in steps S501 to S503 are inserted between steps S211 and S212 shown in FIG.

  In this way, when outputting the final result, if you include the stroke information, you can record the characters that you put in for emphasis, which helps to organize the meeting results .

  As described above, according to the third embodiment, when the idea idea meeting is digitized, the introduction cost can be reduced and the meeting can be executed smoothly. Further, according to the third embodiment, handwritten data can be drawn.

[Modification]
Next, modifications of the above embodiment will be described.

(Modification 1)
In the first modification, the color of the projected tag data can be selected. For example, a color selection button is displayed on the display screen, and the user is allowed to select the color. Next, when the tag data is touched with the color selected, the color of the tag data is set to the selected color.

  FIG. 24 is a diagram for explaining a concept of color selection. As shown in FIG. 24, color patches are arranged on the left side of the display screen. As shown in FIG. 24, when the user selects a color patch color (color c21), the change control unit identifies which color is selected from the touch position. For example, the storage unit may hold color information in which colors are associated with positions. The change control means can determine which color is selected by referring to the color information from the touched position.

  Furthermore, when the change control unit recognizes that the tag data s21 is touched (instructed) in a state where the color is selected, the change control unit changes the color of the image of the touched tag data s21 to the selected color c21. .

  That is, the change control means recognizes an instruction for one data in the color selection area included in the display screen, and then recognizes an instruction for one data included in the display screen. To the indicated color.

  As described above, the color of the tag data can be changed during the meeting. Also, if the displayed sticky note data is colored according to some classification, it can be automatically rearranged based on the color. This is an advantage when compared with the case of using paper sticky notes.

  FIG. 25 is a diagram illustrating a concept of sticky note data alignment. In the example shown in FIG. 25, when the button b11 indicating alignment is touched, the tag data is grouped according to the color of the tag data.

  Regarding the above processing, when an instruction for an area for arranging data (for example, the area of the button b11) included in the display screen is recognized, the change control unit groups each data according to the color of the data. In this case, the change control means determines the color of the image data (i) in the data position information and rearranges (groups) the tag data for each color. A display control means produces | generates the display screen for displaying the tag data rearranged for every color. This makes it easy to organize ideas by adding a mechanism for assisting grouping to the information processing system of the embodiment.

(Modification 2)
Next, an information processing system in Modification 2 will be described. In Modification 2, an information processing system is constructed using only the information processing device 30. For example, one information processing device (also referred to as a parent device) is provided with the function of the server 20. Instead of projecting the display screen, the parent device displays it on its display unit.

  FIG. 26 is a diagram illustrating a concept of the information processing system in the second modification. As illustrated in FIG. 26, the parent device 35 is the same device as the information processing device 30 and receives data from each information processing device 30. Since the parent device 35 has the functions that the server 20 has, the parent device 35 generates and displays a display screen including each data.

  Further, if the parent device 35 has a touch panel, the touch detection in the above-described process is replaced with touch detection on the touch panel. Accordingly, the parent device 35 can move the tag data by performing touch detection.

  By installing a program for executing the function of the parent device 35 in a predetermined device and installing a program for executing the function of the information processing device 30 in each device, the system in the second modification can be constructed.

  The program executed in each device of each embodiment or modification is a file in an installable or executable format, such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk). Or the like recorded on a computer-readable recording medium.

  Further, the program executed by each device of each embodiment or modification may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by each device of each embodiment or modification may be configured to be provided or distributed via a network such as the Internet.

  In addition, a program executed by each device of each embodiment or modification may be provided by being incorporated in advance in a ROM or the like.

  Note that the information processing apparatus, information processing system, and program of each embodiment or modification are not limited to the above-described embodiments as they are, and in the implementation stage, the constituent elements may be modified without departing from the gist thereof. Can be Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments.

DESCRIPTION OF SYMBOLS 1 Information processing system 10 Projection apparatus 20 Server (information processing apparatus)
30 Information processing apparatus 101 Control unit 102 Main storage unit 103 Auxiliary storage unit 301 Communication unit 302 Arrangement unit 303 Storage unit 304 Display control unit 305 Measurement result acquisition unit 306 Instruction recognition unit 307 Change control unit 309 Output unit 403 Distance measurement unit 502 Magnification Change means 701 Stroke recognition means

Japanese Patent No. 4753842 JP 2006-59020 A JP-A-6-311510 Special table 2011-523739 gazette

Claims (8)

Receiving means for receiving each data input to one or more devices;
Storage means for storing the data set with the display position on the display screen;
Measurement result acquisition means for acquiring a measurement result obtained by measuring a three-dimensional position of an object in a projection plane direction on which a display screen including each data is projected based on information acquired from a device capable of measuring a distance;
An instruction recognizing means for recognizing an instruction by an indicator for the projected display screen from the measurement result;
A change control means for grouping each data according to a color of the data when an instruction for an area for arranging the data included in the display screen is recognized ;
An information processing apparatus comprising: The instruction recognition means includes
The information processing apparatus according to claim 1, wherein an instruction is recognized if a distance between the projection plane estimated as a plane and the indicator is within a threshold.   The information processing apparatus according to claim 1, further comprising a magnification changing unit that changes a display magnification of the data according to the number of data included in the display screen. Recognizing stroke identification means the stroke data on the basis of the data from the recognizable writing instrument stroke with respect to the projection plane,
Display control means for visualizing the stroke data on the display screen ;
The information processing apparatus according to claim 1 , further comprising: The change control means includes
After an instruction for one color in the color selection area included in the display screen is recognized, when an instruction for one data included in the display screen is recognized, the color of the one data is changed to the one color. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing apparatus is changed to a color. The change control means includes
The first data is moved to the other area when the instruction for the other area included in the display screen is recognized after the instruction for the display area of the one data included in the display screen is recognized. Item 6. The information processing device according to any one of Items 1 to 5. An information processing system in which each information processing device and a projection device are connected via a network,
Receiving means for receiving each data input to one or more information processing devices;
Storage means for storing the data set with the display position on the display screen;
Measurement means for measuring a three-dimensional position of an object in a projection plane direction on which a display screen including each data is projected by the projection device based on a measurement result acquired from a device capable of measuring a distance;
An instruction recognizing means for recognizing an instruction by an indicator for the projected display screen from a measurement result of the measuring means;
A change control means for grouping each data according to a color of the data when an instruction for an area for arranging the data included in the display screen is recognized ;
An information processing system comprising: A receiving step of receiving each data input to one or more devices;
A storage step of storing in the storage unit each data set with a display position on the display screen;
A measurement step of measuring a three-dimensional position of an object in a projection plane direction on which a display screen including each data is projected, based on a measurement result acquired from a device capable of measuring a distance;
An instruction recognizing step for recognizing an instruction by an indicator for the projected display screen from a measurement result of the measuring step;
A change control step of grouping the data according to the color of the data when an instruction for an area for aligning the data included in the display screen is recognized ;
A program that causes a computer to execute.