JP7371382B2 - Position pointing device, display system, position pointing method, and program - Google Patents

Description

The present application relates to a position pointing device, a display system, a position pointing method, and a program.

2. Description of the Related Art Conventionally, image display devices such as touch displays that are used for presentations, conferences, and the like and are operated by touching a display surface are known.

On the other hand, in presentations using a touch display, if a position indicating device such as a laser pointer is used to indicate the position on the display screen, the position of the irradiated laser light can be seen by the light emitted from the screen of the touch display. The visibility of the position indicated by the laser beam may deteriorate.

Furthermore, a technique for capturing a display image and utilizing it for pointing has been disclosed (for example, see Patent Document 1).

Furthermore, a technique has been disclosed in which an infrared light emitter is installed and the light emitting position is sensed and utilized for pointing (for example, see Patent Document 2).

However, the technique described in Patent Document 1 requires a user to perform a calibration operation, which may be less convenient. In addition, although the technology described in Patent Document 2 does not require calibration, since a light emitting body is installed outside the display, when applied to an infrared touch display, infrared light from an infrared touch sensor may be erroneously emitted. There were cases where it was detected. Therefore, in the infrared touch display, it may not be possible to eliminate the need for calibration operations and prevent false detections from occurring.

The disclosed technology aims to eliminate the need for calibration operations and to prevent false detections during pointing on an infrared touch display.

A position pointing device according to one aspect of the disclosed technology includes: an imaging unit that captures an image for identifying the distribution of infrared rays emitted from an outer frame of a display screen provided in an electronic device; and a direction detection unit that detects a predetermined direction. , an output unit that outputs the indicated position information acquired based on the infrared distribution and the predetermined direction to the electronic device , the output unit outputting the infrared distribution to the image captured by the imaging unit. If the entire infrared distribution is included, the indicated position information acquired based on the barycenter position information of the infrared distribution included in the image is output to the electronic device, and the entire infrared distribution is included in the image captured by the imaging unit. is not included, the electronic device uses the indicated position information acquired based on the number of sides in the infrared distribution included in the image, the arrangement information of the sides, and the intersection position information of the sides. Output to .

According to the disclosed technology, pointing on an infrared touch display can eliminate the need for calibration operations and prevent false detections.

1 is a diagram showing an example of the overall configuration of a display system according to an embodiment. 2A and 2B are diagrams illustrating an example of a touch position detection method of a general infrared touch display, in which (a) is a diagram illustrating an infrared cut-off method, and (b) is a diagram illustrating an infrared camera method. FIG. 2 is a block diagram showing an example of a hardware configuration of a touch display according to an embodiment. FIG. 2 is a block diagram of an example hardware configuration of a pointing device according to an embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of a processing unit according to the first embodiment. FIG. 3 is a diagram illustrating an example of binarization processing according to the embodiment. 2A and 2B are diagrams illustrating an example of approximate line generation processing according to the first embodiment, in which (a) is a diagram showing an image before processing, (b) is a diagram showing an image in the processing process, and (c) is an image after processing. FIG. FIG. 7 is a diagram illustrating an example of rotation correction processing according to the embodiment. FIG. 3 is a diagram illustrating an example of initial position determination processing according to the first embodiment, in which (a) is a diagram showing after infrared distribution approximation and before initial position determination processing; (b) is a diagram showing a shift in the center of gravity position of infrared distribution; (c) is a diagram showing a pointer displayed on the display screen. FIG. 3 is a diagram illustrating an example of a movement amount acquisition process according to the first embodiment, in which (a) is a diagram showing an infrared distribution and a pointer before movement, and (b) is a diagram showing an infrared distribution and a pointer after movement. It is a flowchart which shows the processing example by the processing part of embodiment. FIG. 3 is a diagram illustrating operation sensitivity, in which (a) is a diagram illustrating a case of a pointing device of a comparative example, and (b) is a diagram illustrating a case of a pointing device of an embodiment. FIG. 3 is a block diagram illustrating an example of a functional configuration of a processing unit according to a second embodiment. FIG. 6 is a diagram illustrating an example of approximate line generation processing according to the second embodiment, in which (a) is a diagram showing an image before processing, (b) is a diagram showing an image in the processing process, and (c) is an image after processing. FIG. FIG. 7 is a diagram illustrating an example of initial position coordinate determination processing according to the second embodiment.

Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In each drawing, the same components are designated by the same reference numerals, and redundant explanations may be omitted.

The position pointing device according to the embodiment is used in presentations, conferences, etc. that use electronic devices such as touch displays, and by changing the position of a pointing figure such as a pointer displayed on the display screen of the electronic device, the position pointing device It indicates the position within.

Further, in the embodiment, by outputting the indicated position information acquired based on the infrared distribution emitted from the outer frame of the display screen of the electronic device and the detected predetermined direction to the electronic device, the infrared To eliminate false detection while eliminating the need for calibration operations in a touch display. Here, the designated position information is information for designating the position within the display screen of the electronic device.

Hereinafter, embodiments will be described using a display system including a touch display and a pointing device as an example. Here, the touch display is an example of an "electronic device," and the pointing device is an example of a "position pointing device." Furthermore, a pointer is an example of an "instruction figure."

<Overall configuration of display system 1>
First, the overall configuration of a display system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram illustrating an example of the overall configuration of a display system 1. As shown in FIG. As shown in FIG. 1, the display system 1 includes a touch display 2 and a pointing device 100.

The touch display 2 includes a display surface 20 and an outer frame 21 that supports the outer periphery of the display surface 20, and displays an image drawn by an event caused by the user's hand H (touching the display surface 20 with the hand H). , can be displayed on the display surface 20. Furthermore, the image displayed on the display surface 20 can be changed based on gestures such as enlargement, reduction, and page turning.

A user participating in a conference using the touch display 2 can input information into the touch display 2 using the hand H. The input information is displayed on the display screen 20 and is used for communication among multiple users participating in the conference.

Note that in the above example, the user's hand H is shown, but in addition to the user's hand H, an event caused by the stylus pen (touching the tip of the stylus pen or the butt of the stylus pen on the display surface 20) It is also possible to display an image drawn on the display surface 20.

The pointing device 100 is held by a user who is holding a presentation, a meeting, etc., and changes the position of the pointer 200 displayed on the display surface 20 in response to user operations such as tilting the pointing device 100. This is a device that indicates the location within the building. Note that the user holding the pointing device 100 may be the same as the user holding the hand H, or may be different.

<Touch position detection method in general infrared touch display>
Here, a method for detecting the touch position when the user's hand H touches the display surface of a general infrared touch display will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a touch position detection method on a display surface in an infrared touch display, in which (a) is a diagram illustrating an infrared cutoff method, and (b) is a diagram illustrating an infrared camera method. .

In FIG. 2A, the infrared touch display 3 includes an outer frame 31 that supports a display surface 30, and includes infrared light source arrays 32a and 32b and infrared sensor arrays 33a and 33b.

Each of the infrared light source arrays 32a and 32b includes a plurality of light sources that emit infrared rays. Such a light source is an LED (Light Emitting Diode) light source or the like. The infrared light source array 32a is arranged on one of the long sides of the outer frame 31, and the infrared light source array 32b is arranged on one of the short sides of the outer frame 31.

Each of the infrared sensor arrays 33a and 33b includes a plurality of light receiving elements that output electrical signals according to the received infrared rays. Such a light receiving element is a PD (Photo Diode) or the like. The infrared sensor array 33a is arranged on the other long side of the outer frame 31, and the infrared sensor array 33b is arranged on the other short side of the outer frame 31.

The infrared rays emitted by the respective light sources of the infrared light source arrays 32a and 32b propagate along the surface of the display surface 30 in the direction of the arrow shown by the broken line in FIG. The light is received by each light receiving element provided.

When the user's hand H touches the display surface 30, the propagating infrared rays are blocked by the hand H, and the blocked position is detected by the infrared sensor arrays 33a and 33b. The infrared sensor array 33a detects the position in the X direction, and the infrared sensor array 33b detects the position in the Y direction. In the infrared blocking method, the XY coordinates of the touch position are detected in this way.

On the other hand, in FIG. 2B, the infrared touch display 4 includes an outer frame 41 that supports the display surface 40, infrared touch sensor units 42a and 42b, and a retroreflective sheet 43.

The infrared touch sensor unit 42 includes an LED light source that emits infrared light and an infrared camera that captures infrared images. The infrared touch sensor unit 42a is provided near the end in the negative X direction of one of the long sides of the outer frame 41, and the infrared touch sensor unit 42b is provided near the end in the positive X direction.

The retroreflective sheet 43 is a sheet-like member that has a function of strongly reflecting incident light in the direction of incidence. The retroreflective sheet 43 is provided on each of the four sides.

The LED light sources in each of the infrared touch sensor units 42 emit a plurality of infrared rays that spread along the plane of the display surface 40 so as to illuminate substantially the entire display surface 40 . The emitted infrared rays are strongly reflected in the incident direction by the retroreflective sheet 43, and are imaged by an infrared camera in the infrared touch sensor unit 42.

When the user's hand H touches the display surface 40, the propagating infrared rays are blocked by the hand H, and an infrared image with the blocked position in the shadow is captured by the infrared camera. The XY coordinates of the touch position are detected from the position of the shadow in the captured infrared image.

The touch display 2 according to the embodiment may use either the infrared cutoff method or the infrared camera method described above. Note that touch displays other than infrared types, such as capacitive type and resistive type, are not included. In the following, the touch display 2 will be described as one that detects the touch position on the display surface 20 using an infrared camera method.

<Hardware configuration of touch display 2>
Next, the hardware configuration of the touch display 2 according to the embodiment will be described with reference to FIG. 3. FIG. 3 is a block diagram illustrating an example of the hardware configuration of the touch display 2. As shown in FIG.

As shown in FIG. 3, the touch display 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, and a network I/O. /F 205 and an external device connection I/F (Interface) 206.

Among these, the CPU 201 controls the operation of the touch display 2 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader).

RAM 203 is used as a work area for CPU 201. The SSD 204 stores various data such as programs for the touch display 2. Network I/F 205 controls communication with a communication network connected to touch display 2 .

External device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

The touch display 2 also includes a GPU (Graphics Processing Unit) 212, a display controller 213, an infrared touch sensor unit 22, a touch sensor controller 215, a communication I/F 216, a short-range communication circuit 219, and a short-range communication It includes an antenna 219a of the circuit 219, a power switch 222, and selection switches 223.

Among these, the capture device 211 displays video information corresponding to the display content of the display of an external PC (Personal Computer) 270 on the display screen 20 as a still image or a moving image. The GPU 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display in order to output the output image from the GPU 212 to the display screen 20 or the like.

The infrared touch sensor unit 22 detects the touch position of a stylus pen, the user's hand H, etc. on the display surface 20 using the above-described infrared camera method. An infrared touch sensor unit 22a included in the infrared touch sensor unit 22 is installed at the left end of the upper long side of the outer frame 21 (see FIG. 1), and an infrared touch sensor unit 22b included in the infrared touch sensor unit 22 is installed at the right end. is installed. Each of them emits a plurality of infrared rays in parallel to the display surface 20, is reflected by a retroreflective sheet provided around the display surface 20, and returns along the same optical path as the light path emitted by the light receiving element. The touch position of the stylus pen, the user's hand H, etc. is detected by receiving the light coming from the sensor.

The touch sensor controller 215 controls the processing of the infrared touch sensor unit 22. The infrared touch sensor unit 22 outputs the ID of the infrared rays blocked by the user's hand H etc. to the touch sensor controller 215, and the touch sensor controller 215 specifies the coordinate position that is the touch position of the hand H.

The communication I/F 216 is a communication circuit such as Bluetooth (registered trademark) using a frequency band of 2.4 GHz, and is an interface for communicating with the pointing device 100 described later.

The near field communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth. The power switch 222 is a switch for switching the power of the touch display 2 on and off. The selection switches 223 are, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display surface 10.

Furthermore, the touch display 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 3.

<Hardware configuration of pointing device 100>
Next, the hardware configuration of the pointing device 100 according to the embodiment will be described with reference to FIG. 4. FIG. 4 is a block diagram illustrating an example of the hardware configuration of the pointing device 100. As shown in FIG. 4, the pointing device 100 includes a lens 101, an optical filter 102, an image sensor 103, a direction sensor 104, and a processing section 110. These are installed inside the casing (not shown) of the pointing device 100. Here, the optical filter 102 is an example of an "optical member," the image sensor 103 is an example of an "imaging section," and the direction sensor 104 is an example of a "direction detection section."

The lens 101 is a lens that forms an image of the touch display 2, which is a subject, on the imaging surface of the image sensor 103 via the optical filter 102. The lens 101 is attached to an opening that communicates the inside and outside of the housing. The lens 101 is configured with one lens or a combination of multiple lenses made of resin, glass, or the like, and allows light in an arbitrary wavelength band, such as visible light and infrared light, to pass therethrough and forms an image.

The optical filter 102 is provided in the optical path between the lens 101 and the image sensor 103, and is an optical filter that selectively passes infrared rays in a predetermined wavelength band and reflects or absorbs light other than the infrared rays, such as visible light. It's a filter. The infrared rays in the predetermined wavelength band are infrared rays emitted by a light source included in the infrared touch sensor unit 22 (see FIG. 3) installed in the outer frame 21 of the touch display 2 (see FIG. 1). The optical filter 102 is composed of a transparent plate-like member such as resin or glass, and a multi-reflection film formed on the surface of the plate-like member.

The image sensor 103 is an image sensor that captures an image of the touch display 2 formed by infrared rays that have passed through the optical filter 102 on an imaging surface. As the image sensor 103, a MOS (Metal Oxide Semiconductor Device), a CMOS (Complementary Metal Oxide Semiconductor Device), a CCD (Charge Coupled Device), or the like can be used.

Here, the infrared rays emitted from the light source included in the infrared touch sensor unit 22 propagate substantially parallel to the surface of the display surface 20 and are diffusely reflected by the outer frame 21 . A portion of the diffusely reflected infrared light propagates in the direction of pointing device 100 held by the user.

Infrared rays propagating in the direction of pointing device 100 pass through lens 101 and optical filter 102 and reach the imaging surface of image sensor 103 . On the imaging surface, an image of the outer frame 21 that diffusely reflects infrared rays is formed by the lens 101. Since light such as visible light other than infrared rays is removed by the optical filter 102, the image sensor 103 can capture an image for specifying the infrared distribution due to the infrared rays emitted by the outer frame 21, and output it to the processing unit 110. .

In addition, in the example mentioned above, although the optical filter 102 arrange|positioned in the optical path between the lens 101 and the image sensor 103 was illustrated, it is not limited to this. The optical filter 102 may be arranged between the lens 101 and the touch display 2 that is the subject.

Further, although the optical filter 102 has been illustrated as an optical member that selectively passes infrared rays emitted from the light source included in the infrared touch sensor unit 22, the optical filter 102 is not limited thereto. The optical member may be a multi-reflection film that is formed on the surface of the lens 101, the imaging surface of the image sensor 103, or the protective glass surface of the imaging surface, and selectively passes infrared rays in a predetermined wavelength band.

Further, since an image is taken using infrared rays, it is preferable that the lens 101 is designed to reduce aberrations with respect to infrared rays.

Returning to FIG. 4, the explanation will be continued. The direction sensor 104 is an acceleration sensor such as a gyro sensor. Here, depending on how the user holds the pointing device 100, the pointing device 100 may rotate. As the pointing device 100 rotates, the image captured by the image sensor 103 rotates, and it may not be possible to accurately obtain indicated position information, which will be described later. In particular, rotation around the horizontal direction (hereinafter referred to as "rolling") causes the infrared distribution 61 to roll within the captured image, so that the influence on the indicated position information becomes significant.

Therefore, the direction sensor 104 detects the direction of gravity based on the detected attitude data of the direction sensor 104, and outputs direction data indicating the direction of gravity to the processing unit 110. Based on this direction data, rolling, which has a large effect on the rotation of the pointing device 100, is detected, and the rolling of the infrared distribution 61 is corrected in the captured image.

Here, the direction of gravity is an example of a "predetermined direction." However, the direction sensor 104 may detect a predetermined direction other than the direction of gravity. For example, rolling, pitching, and yawing of the pointing device 100 may be detected by detecting three axial directions: the gravitational direction, the horizontal direction, and a direction perpendicular to these directions. In this case, rolling, pitching, and yawing of the infrared distribution 61 can be corrected within the captured image. Furthermore, instead of the acceleration sensor, a geomagnetic sensor or the like may be used.

Next, the processing unit 110 is a processing device that inputs an image from the image sensor 103 and inputs direction data from the direction sensor 104 to realize various functions described below. The processing unit 110 includes a CPU 111, a ROM 112, a RAM 113, a memory 114, a transmission I/F 115, an image sensor controller 116, and a direction sensor controller 117. Each of these components is connected to each other via bus B. Note that the above components may be connected via either wired communication or wireless communication.

The CPU 111 is composed of a processor and the like, and controls the operation of each part of the pointing device 100 and the overall operation. The ROM 112 is composed of a nonvolatile semiconductor storage device or the like, and stores various programs and various parameters that operate on the pointing device 100. The RAM 113 is composed of a volatile semiconductor memory device, etc., and is used as a work area for the CPU 111. The RAM 113 provides a storage area for temporarily storing data when performing various signal processing and image processing.

The memory 114 stores various information such as data used in various programs and images captured by the image sensor 103. The memory 114 is configured with a storage device such as a volatile or nonvolatile semiconductor memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive). Note that the memory 114 may include a ROM 112 and/or a RAM 113.

The program is stored in advance in the ROM 112, memory 114, or the like. The program is read by the CPU 111 from the ROM 112 or the memory 114 or the like to the RAM 113 and expanded. The CPU 111 executes each coded instruction in the program developed in the RAM 113. Note that the program is not limited to the ROM 112 and the memory 114, but may be stored in a recording medium such as a recording disk. Further, the program may be transmitted via a wired network, a wireless network, broadcasting, etc., and may be loaded into the RAM 113.

The transmission I/F 115 is an interface for transmitting data and/or signals processed by the processing unit 110 to the touch display 2, and is a communication circuit such as Bluetooth.

The image sensor controller 116 is an electric circuit that controls the processing of the image sensor 103, and the direction sensor controller 117 is an electric circuit that controls the process of the direction sensor 104.

[First embodiment]
<Functional configuration of the processing unit 110 according to the first embodiment>
Next, the functional configuration of the processing section 110 included in the pointing device 100 will be described with reference to FIG. 5. FIG. 5 is a block diagram illustrating an example of the functional configuration of the processing unit 110. As shown in FIG. 5, the processing section 110 includes a binarization section 121, an approximate line generation section 122, a rotation correction section 123, an initial position determination section 124, a movement amount acquisition section 125, and an output section 126. Equipped with

Among these, the respective functions of the binarization section 121, the approximate line generation section 122, the rotation correction section 123, the initial position determination section 124, and the movement amount acquisition section 125 are performed by the CPU 111 in FIG. 4 executing a predetermined program. This is realized by such things as Further, the function of the output unit 126 is realized by the transmission I/F 115 and the like.

With the above configuration, the processing unit 110 acquires instruction position information for instructing a position within the display surface 20 based on the image input from the image sensor 103 and the direction data input from the direction sensor 104, Output to touch display 2.

The binarization unit 121 inputs an image captured by the image sensor 103 and binarizes the input image by comparing the brightness value of each pixel in the image with a predetermined threshold value. Thereby, pixels corresponding to the infrared distribution 61 in the input image can be made valid pixels, and pixels corresponding to other than the infrared distribution 61 can be made invalid pixels. The binarization unit 121 outputs the image after the binarization process to the approximate line generation unit 122.

The approximate line generation unit 122 linearly approximates the infrared distribution 61 in the binarized image, and generates an image that includes approximate lines corresponding to each side (two long sides and two short sides) of the infrared distribution 61. Note that since conventional image processing techniques such as Hough transform and LSD (Line Segment Detector) can be applied to the linear approximation process, further detailed explanation will be omitted. The approximate line generation unit 122 outputs the generated image to the rotation correction unit 123.

The rotation correction unit 123 performs in-plane rotation of the image input from the approximate line generation unit 122 based on the direction data input from the direction sensor 104, and corrects rolling. Then, the corrected image is output to either the initial position determination section 124 or the movement amount acquisition section 125.

The initial position determining unit 124 determines the initial position of the pointer 200 in the image displayed on the display surface 20 based on the image after the rolling correction. In other words, when displaying an image on the display surface 20 is started, the position of the pointer 200 to be displayed within the display image is determined.

More specifically, the initial position determining unit 124 calculates the barycenter position coordinates of the infrared distribution 61, and calculates a vector (deviation amount Δd and deviation direction φ) of the barycenter position coordinates with respect to the image center position coordinates. Then, according to the vector, a position shifted from the center of the display image on the display surface 20 is determined as the initial position of the pointer 200. For example, a position shifted from the center of the display image in the direction π-φ by a shift amount Δd' corresponding to the shift amount Δd is determined as the initial position of the pointer 200. Conventional techniques such as calculating the average value of the X coordinate and the average value of the Y coordinate of effective pixels can be applied to the method of calculating the barycenter position coordinates.

The initial position determination unit 124 outputs the determined initial position information to the output unit 126. Further, after determining the initial position, the initial position determining unit 124 stores a flag indicating that the initial position has been determined in the RAM 113 or the like. This flag is referred to when the rotation correction unit 123 determines the output destination of the corrected image to either the initial position determination unit 124 or the movement amount acquisition unit 125.

The movement amount obtaining unit 125 obtains the movement amount for moving the pointer 200 in response to a user's operation such as tilting the pointing device 100.

More specifically, the above-described processing by the binarization unit 121, the approximate line generation unit 122, and the rotation correction unit 123 is performed on images captured by the image sensor 103 in time series at a frame rate of 30 Hz, etc. The processed image is input to the movement amount acquisition unit 125. The movement amount acquisition unit 125 calculates the amount of change in the barycenter position coordinates of the infrared distribution 61 between time-series images as the movement amount, and outputs the movement amount information to the output unit 126. Here, the amount of change in the coordinates of the center of gravity is an example of "center of gravity position information."

Note that in the above example, the amount of movement is calculated at the frame rate and outputted to the output unit 126, but the invention is not limited to this, and even if the amount of movement is calculated at a frequency lower than the frame rate. good.

The output unit 126 outputs either the initial position information input from the initial position determination unit 124 or the movement amount information input from the movement amount acquisition unit 125 to the touch display 2. Here, each of the above initial position information and movement amount information corresponds to "indicated position information".

The touch display 2 causes the display surface 20 to display a display image including the pointer 200 at a position corresponding to the designated position information input from the processing unit 110.

<Example of image after processing>
Next, images processed by each functional unit of the processing unit 110 will be described with reference to FIGS. 6 to 10.

First, FIG. 6 is a diagram illustrating an example of processing by the binarization unit 121. In FIG. 6, an image 60 captured by the image sensor 103 includes an infrared distribution 61. The partial image 63 is an enlarged view of the portion 62 indicated by the broken line in the image 60. The grid included in the partial image 63 represents pixels. Furthermore, the portion shown with matte hatching in the partial image 63 corresponds to a part of the infrared distribution 61. Partial image 64 shows an image after binarization processing of partial image 63. Pixels forming the infrared distribution 61 are valid pixels, and pixels forming other than the infrared distribution 61 are invalid pixels.

Next, FIG. 7 is a diagram illustrating an example of approximate line generation processing, in which (a) shows an image before processing, (b) shows an image in the process, and (c) shows an image after processing. It is a figure showing an image.

The infrared distribution 71 included in the image 70 after the binarization process is linearly approximated, and approximation lines 72a to 72d corresponding to each side of the infrared distribution 71 are generated. The image is divided into regions using the intersections of the approximate lines 72a to 72d as boundaries, and the number of effective pixels is compared between adjacent regions. Then, the approximation lines included in the area with more valid pixels are left as approximation lines corresponding to the sides of the infrared distribution 61, and the approximation lines included in the area with fewer valid pixels are removed (as invalid pixels). As a result, an approximate line image is generated in which only approximate lines corresponding to the sides of the infrared distribution 61 are left as valid pixels.

Next, FIG. 8 is a diagram illustrating an example of rotation correction processing by the rotation correction unit 123. FIG. 8 shows an infrared distribution 81 included in an image 80 captured by the image sensor 103. Due to the rolling of the pointing device 100, the infrared distribution 81 is rotated within the plane within the image 80. Since the rotation angle with respect to the direction of gravity is known from the direction data detected by the direction sensor 104, by in-plane rotation of the image 80 at this rotation angle, an infrared distribution 81 in which rolling is corrected can be obtained.

Next, FIG. 9 is a diagram showing an example of initial position determination processing by the initial position determination unit 124, in which (a) is a diagram showing after infrared distribution approximation and before initial position determination processing, and (b) is a diagram showing infrared distribution approximation. A diagram showing the resulting center-of-gravity position shift, and (c) a diagram showing the pointer 200 displayed on the display surface 20.

In FIG. 9B, a center of gravity position 93 indicated by an "x" indicates the center of gravity of the infrared distribution approximation result 92 included in the image 90, and a center position 94 indicates the center position of the image 90. The center of gravity position 93 is shifted from the center position 94 by a shift amount Δd in the shift direction φ. According to the initial position determined by the initial position determining unit 124, the pointer 200 is displayed on the display surface 20 with the initial position being shifted from the center 95 of the display image in the direction π-φ by the amount of displacement Δd' ( (See FIG. 9(c)).

Next, FIG. 10 is a diagram illustrating an example of the movement amount acquisition process by the movement amount acquisition unit 125, in which (a) is a diagram showing the infrared distribution approximation result and the pointer before movement, and (b) is a diagram showing the pointer after movement. It is a figure which shows the infrared distribution approximation result and a pointer.

In FIG. 10A, an image 96a before movement includes an infrared distribution approximation result 97a, and a pointer 99a displayed at a position based on the infrared distribution approximation result 97a is displayed on the display surface 20 shown below the image 96a. It is included.

When the pointing device 100 is operated from this state, the infrared distribution approximation result 97b moves within the image 96b according to the operation, as shown in FIG. 10(b). This movement amount is acquired by the movement amount acquisition unit 125, and the pointer 99b moves in the direction of the arrow within the display surface 20 according to the acquired movement amount.

<Flow of processing by processing unit 110>
Next, the flow of processing by the processing unit 110 will be explained. FIG. 11 is a flowchart illustrating an example of processing by the processing unit 110.

First, in step S111, the binarization unit 121 performs binarization processing on the image input from the image sensor 103 via the image sensor controller 116, and outputs the processed image to the approximate line generation unit 122. do.

Subsequently, in step S112, the approximation line generation unit 122 linearly approximates the infrared distribution in the binarized image, generates an image including approximation lines corresponding to each side of the infrared distribution 61, and rotates the generated image. It is output to the correction section 123.

Subsequently, in step S113, the rotation correction unit 123 rotates the image input from the approximate line generation unit 122 in-plane based on the direction data input from the direction sensor 104 via the direction sensor controller 117, and corrects rolling. .

Subsequently, in step S114, the rotation correction unit 123 refers to a flag stored in the RAM 113 or the like and determines whether the initial position has been determined.

If it is determined in step S114 that the initial position has not been determined (step S114, No), the rotation correction unit 123 outputs the corrected image to the initial position determination unit 124.

Thereafter, in step S115, the initial position determining unit 124 determines the initial position of the pointer 200 in the image displayed on the display surface 20 based on the image after the rolling correction, and outputs the determined initial position information to the output unit 126. do.

On the other hand, if it is determined in step S114 that the initial position has been determined (step S114, Yes), the rotation correction unit 123 outputs the corrected image to the movement amount acquisition unit 125.

Subsequently, in step S116, the movement amount acquisition unit 125 acquires the movement amount for moving the pointer 200, and outputs the acquired movement amount information to the output unit 126.

Subsequently, in step S117, the output unit 126 outputs either the input initial position information or movement amount information to the touch display 2 as designated position information.

In this way, the processing unit 110 transmits instruction position information to the touch display 20 for instructing a position within the display surface 20 based on the image input from the image sensor 103 and the direction data input from the direction sensor 104. It can be output to

<Actions and effects of pointing device 100>
As described above, in this embodiment, the indicated position is acquired based on the image for specifying the infrared distribution captured by the image sensor 103 and the direction data indicating the direction of gravity detected by the direction sensor 104. Information is output to the touch display 2.

If the pointing device 100 is tilted with respect to the touch display 2 when starting to display an image on the display screen, the position of the infrared distribution will shift according to the tilt, and the indicated position information corresponding to this shift will be touched. It can be output to display 2. This allows the initial position of the pointer to be appropriately determined when starting to display an image on the display surface, and eliminates the need for a calibration operation when pointing on an infrared touch display. Further, since infrared rays used in touch position detection are used and no additional infrared light source is provided, false detection can be prevented.

Furthermore, in addition to the above-mentioned effects, the following effects regarding the operation sensitivity of the pointing device 100 can also be obtained. FIGS. 12A and 12B are diagrams illustrating such operation sensitivity, in which (a) is a diagram illustrating the case of the pointing device according to the comparative example, and (b) is a diagram illustrating the case of the pointing device 100 according to the embodiment. It is.

In FIG. 12A, when the pointing device 300 is tilted by an angle θ at a position close to the display surface 20, the pointer moves by a movement distance e1 on the display surface 20. On the other hand, if the pointing device 300 is similarly tilted by the angle θ at a position far from the display surface 20, the pointer moves by the movement distance e1 on the display surface 20, as in the case at a close position. Therefore, regardless of the distance between the display surface 20 and the pointing device 300, the operational sensitivity of the moving distance with respect to the angle θ is constant.

On the other hand, as shown in FIG. 12B, in this embodiment, when the pointing device 100 is tilted by an angle θ at a position close to the display surface 20, the pointer moves by a movement distance e1 on the display surface 20. Furthermore, if the pointing device 100 is similarly tilted by an angle θ at a position far from the display surface 20, the pointer moves by a larger movement distance e2 on the display surface 20 than when the pointer is at a closer position. In other words, as the distance between the display surface 20 and the pointing device 300 becomes longer, the operating sensitivity changes such that the moving distance relative to the angle θ becomes larger.

The operation sensitivity according to this embodiment is the same as that of a pointer instruction on a display surface using a laser pointer. Therefore, according to this embodiment, the pointing device can be operated with the same operating feeling as a laser pointer, which is often used in presentations and the like.

In this embodiment, an example is shown in which an optical member that transmits infrared rays is provided. It's okay. However, by using an optical member that selectively passes infrared rays, visible light that is not used to obtain indicated position information can be removed in advance, which simplifies the process of acquiring indicated position information and allows highly accurate indicated position information to be obtained. Can be obtained.

[Second embodiment]
Next, a display system 1a according to a second embodiment will be explained.

Here, in the first embodiment, a case has been described in which the image captured by the image sensor 103 includes all four sides of the infrared distribution 61, but for example, if the pointing device 100 is located close to the touch display 2, In some cases, the infrared distribution 61 becomes so large that the entire infrared distribution 61 may not be included in the field of view captured by the image sensor 103. In other words, the image captured by the image sensor 103 may not include four sides in the infrared distribution. In this embodiment, in such a case, the indicated position information acquired based on the side arrangement information and the side intersection position information is output to the touch display 2.

FIG. 13 is a block diagram illustrating an example of the functional configuration of the processing unit 110a included in the pointing device 100a according to the present embodiment. As shown in FIG. 13, the processing unit 110a includes an approximate line number detection unit 127, an initial position determination unit 124a, and a movement amount acquisition unit 125a.

Among these, the approximate line number detection unit 127 detects the number of approximate lines in the approximate line image generated by the approximate line generation unit 122, and stores the detection result in the RAM 113. Here, the number of approximate lines corresponds to "information on the number of sides in the infrared distribution".

If the number of detected approximate lines is four, the initial position determining unit 124a determines the initial position of the pointer 200 in the display image on the display surface 20 by the same process as the initial position determining unit 124 described above. decide.

On the other hand, if the number of detected approximate lines is not four, the initial position determination unit 124a identifies which part of the infrared distribution 61 the approximate line in the approximate line image corresponds to, and corresponds to the identified part. The position on the display screen where the pointer 200 is displayed is determined as the initial position where the pointer 200 is displayed. The initial position determination unit 124a outputs the determined initial position information to the output unit 126. Note that the details will be explained separately using FIG. 15.

If the number of detected approximate lines is four, the movement amount acquisition unit 125a acquires the movement amount for moving the pointer 200 by the same process as the movement amount acquisition unit 125 described above. On the other hand, if the number of detected approximate lines is not four, the amount of change in the coordinates of the intersection points of the approximate lines between time-series images is acquired as the amount of movement for moving the pointer 200. Here, the amount of change in the intersection position coordinates is an example of "intersection position information." The movement amount acquisition unit 125a outputs the acquired movement amount information to the output unit 126.

Next, FIG. 14 is a diagram illustrating an example of approximate line generation processing by the approximate line generation unit 122 when the entire infrared distribution 61 is not included in the imaging field of view, and (a) shows the image before processing. (b) is a diagram showing an image in the processing process, and (c) is a diagram showing an image after processing.

As shown in FIG. 14A, the image 140 after the binarization process includes an infrared distribution 141 that corresponds to two of the four sides of the infrared distribution 61. The approximate line generation unit 122 approximates the infrared distribution 141 to a straight line, and generates approximate lines 142a to 142b corresponding to each side of the infrared distribution 141. The approximate line generation unit 122 divides the image into regions using the intersections of the approximate lines 142a to 142b as boundaries, and compares the number of effective pixels between adjacent regions. Then, the approximation line included in the area with more effective pixels is left as an approximation line corresponding to the side of the infrared distribution 61, and the approximation line included in the area with fewer valid pixels is removed (regarded as invalid pixels). . As a result, an approximate line image is generated in which only approximate lines corresponding to the sides of the infrared distribution 61 are left as valid pixels.

In the case of the approximate line image in FIG. 14, the approximate line number detection unit 127 can detect the number of approximate lines as two.

Next, FIG. 15 is a diagram showing an example of initial position determination processing by the initial position determination unit 124a, in which (a) shows an image before processing, and (b) shows the pointer 200 displayed on the display surface 20. FIG.

As shown in FIG. 15A, the approximate line image 150 includes infrared distribution approximation results 151 corresponding to two of the four sides of the infrared distribution 61. The initial position determination unit 124a determines whether the infrared distribution approximation result 151 corresponds to the upper right, lower right, upper left, or lower left of the display surface 20. This determination can be made based on the relative positional relationship between the intersection of the approximate lines and the approximate lines. Here, the relative positional relationship between the intersection of the approximate lines and the approximate lines is an example of "side arrangement information."

For example, in the case of FIG. 15, the approximate lines are on the upper and left sides of the intersection 152. In this case, the infrared distribution approximation result 151 corresponds to the lower right of the display surface 20. In addition, if the approximation line is located above and to the right of the intersection, the infrared distribution approximation result 151 corresponds to the lower left of the display surface 20. Furthermore, when the approximation line is located below and to the left of the intersection, the infrared distribution approximation result 151 corresponds to the upper right of the display surface 20. Furthermore, if the approximation line is located below and to the right of the intersection, the infrared distribution approximation result 151 corresponds to the upper left of the display surface 20.

The initial position determination unit 124a determines the upper right, lower right, upper left, or lower left of the display surface 20 as the initial position to display the pointer 200, and identifies the upper right, lower right, upper left, or lower left of the display surface 20. The information is output to the touch display 2 via the output unit 126.

According to the initial position determined by the initial position determination unit 124a, a pointer 200 is displayed on the display surface 20 with the lower right of the display image as the initial position (see FIG. 15(b)).

<Actions and effects of pointing device 100a>
As explained above, in this embodiment, the number of approximate lines in the approximate line image is detected, and if the number of approximate lines is not 4, it is acquired based on the side arrangement information and the intersection position information of the sides. The indicated position information is output to the touch display 2.

Thereby, even if the entire infrared distribution 61 is not included in the field of view captured by the image sensor 103, the indicated position information can be acquired and output to the touch display 2.

Note that the present invention is not limited to the above-described specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims.

Embodiments also include a position pointing method. For example, the position indication method includes a step of detecting an infrared distribution emitted from an outer frame of a display screen of an electronic device, a step of detecting a predetermined direction, and an instruction obtained based on the infrared distribution and the predetermined direction. The method includes a step of outputting position information to the electronic device. With such a position pointing method, it is possible to obtain the same effects as the above-described position pointing device.

The embodiments also include programs. For example, the program detects the infrared distribution emitted from the outer frame of the display screen of the electronic device, detects a predetermined direction, and uses the indicated position information acquired based on the infrared distribution and the predetermined direction on the electronic device. Have the computer perform processing to output to the device. With such a program, it is possible to obtain the same effects as the above-mentioned position pointing device.

Moreover, each function of the embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

1 Display system 2 Touch display (an example of electronic equipment)
20 Display surface 21 Outer frame 22 Infrared touch sensor unit 215 Touch sensor controller 216 Communication I/F
100 Pointing device (an example of a position pointing device)
101 Lens 102 Optical filter (an example of an optical member)
103 Image sensor (an example of an image sensor)
104 Direction sensor (an example of a direction detection unit)
110 Processing unit 115 Transmission I/F
121 Binarization section 122 Approximate line generation section 123 Rotation correction section 124 Initial position determination section 125 Movement amount acquisition section 126 Output section 127 Approximate line number detection section 200 Pointer H User's hand Δd Displacement amount φ Displacement direction

Japanese Patent Application Publication No. 2014-59678 Patent No. 3422383

Claims (6)

an imaging unit that captures an image for identifying the distribution of infrared rays emitted from the outer frame of a display surface of the electronic device;
a direction detection unit that detects a predetermined direction;
an output unit that outputs the indicated position information acquired based on the infrared distribution and the predetermined direction to the electronic device ,
The output section is
When the image captured by the imaging unit includes the entire infrared distribution, outputting the indicated position information acquired based on the center of gravity position information of the infrared distribution included in the image to the electronic device;
When the image captured by the imaging unit does not include the entire infrared distribution, information on the number of sides in the infrared distribution included in the image, arrangement information on the sides, and information on the intersection position of the sides; outputting the indicated position information acquired based on to the electronic device;
Position indicating device. Equipped with an optical member that selectively passes infrared rays ,
The imaging unit includes:
The position pointing device according to claim 1, wherein the infrared distribution that has passed through the optical member is detected. 3. The position pointing device according to claim 1, wherein the predetermined direction is one of three axial directions: a gravitational direction, a horizontal direction, and a direction orthogonal to these directions . A display system comprising: an electronic device having a display surface; and a position indicating device for indicating the position of an instruction figure displayed on the display surface, the display system comprising:
an imaging unit that captures an image for identifying the distribution of infrared rays emitted from the outer frame of the display surface;
a direction detection unit that detects a predetermined direction;
an output unit that outputs the indicated position information acquired based on the infrared distribution and the predetermined direction to the electronic device ,
The output section is
When the image captured by the imaging unit includes the entire infrared distribution, outputting the indicated position information acquired based on the center of gravity position information of the infrared distribution included in the image to the electronic device;
When the image captured by the imaging unit does not include the entire infrared distribution, information on the number of sides in the infrared distribution included in the image, arrangement information on the sides, and information on the intersection position of the sides; outputting the indicated position information acquired based on to the electronic device;
display system. a step of using an imaging unit to capture an image for identifying the distribution of infrared rays emitted from an outer frame of a display surface of the electronic device;
a step of detecting a predetermined direction by a direction detection section ;
outputting, by an output unit, the indicated position information acquired based on the infrared distribution and the predetermined direction to the electronic device ,
The output section is
When the image captured by the imaging unit includes the entire infrared distribution, outputting the indicated position information acquired based on the center of gravity position information of the infrared distribution included in the image to the electronic device;
When the image captured by the imaging unit does not include the entire infrared distribution, information on the number of sides in the infrared distribution included in the image, arrangement information on the sides, and information on the intersection position of the sides; outputting the indicated position information acquired based on to the electronic device;
Position indication method. The imaging unit captures an image for identifying the distribution of infrared rays emitted from the outer frame of the display screen of the electronic device,
A direction detection unit detects a predetermined direction,
outputting the indicated position information acquired based on the infrared distribution and the predetermined direction by the output unit to the electronic device ;
The output section is
When the image captured by the imaging unit includes the entire infrared distribution, outputting the indicated position information acquired based on the center of gravity position information of the infrared distribution included in the image to the electronic device;
When the image captured by the imaging unit does not include the entire infrared distribution, information on the number of sides in the infrared distribution included in the image, arrangement information on the sides, and information on the intersection position of the sides; outputting the indicated position information acquired based on to the electronic device;
A program that causes a computer to perform a process.

Images