JP7112627B2 - Control device, control method and program - Google Patents

Description

The present disclosure relates to control devices, control methods, and programs.

Conventionally, a projector that projects an image onto a projection plane is known. In Patent Document 1, an image is simply projected onto a projection surface and shown to the user, but information written on the projection surface by the user is acquired through a camera, and the acquired information is linked to the projected image and recorded. I am proposing a method.

Japanese Patent Application Laid-Open No. 2003-044839

The inventors of the technique according to the present disclosure recognize that the method proposed by Cited Document 1 can also be applied to user interfaces via projection images. However, depending on where the image is projected, various types of user interfaces are desirable.

According to the present disclosure, a projection control unit that controls projection of an image onto a projection surface by a projector, and a user interface (UI) related to the projected image based on the positional relationship of the projector with respect to the projection surface. a UI control unit that switches a mode between two or more UI modes, wherein the projector detects a user's touch position using exploration light emitted toward an exploration area near the projector. wherein the UI control unit selects a first UI mode using the first user input detection system based on a measurement result of a sensor of the projector. is provided.

Further, according to the present disclosure, in a projection control device that controls projection of an image onto a projection surface by a projector, a user interface (UI) related to the image to be projected is based on the positional relationship of the projector with respect to the projection surface. ) between two or more UI modes, wherein the projector detects a touch position by a user using probing light emitted toward a probing area near the projector. A control method is provided that has a first user input detection system and selects a first UI mode that utilizes the first user input detection system based on measurement results of a sensor that the projector has.

Further, according to the present disclosure, a projection control unit that controls a processor of a projection control device to project an image onto a projection surface by a projector; and a UI controller that switches modes of a user interface (UI) associated with the projector between two or more UI modes, wherein the projector emits search light toward a search area near the projector. , and the UI control unit uses the first user input detection system based on the measurement result of the sensor of the projector. A program is provided to function to select one UI mode.

According to the technology according to the present disclosure, it is possible to provide the user with a user interface of a desired type that matches the positional relationship of the projector with respect to the projection plane.
In addition, the above effects are not necessarily limited, and together with the above effects or instead of the above effects, any of the effects shown in this specification, or other effects that can be grasped from this specification may be played.

1 is a schematic diagram showing the appearance of a projection control apparatus according to an embodiment; FIG. FIG. 4 is an explanatory diagram for explaining a first installation posture of the projection control apparatus; FIG. 10 is an explanatory diagram for explaining a second installation attitude of the projection control apparatus; FIG. 4 is an explanatory diagram for explaining an outline of a touch input detection method that can be employed in the first user input detection system; 3 is a block diagram showing an example of a detailed configuration of a first user input detection system; FIG. FIG. 5 is an explanatory diagram for explaining the action of the polarizing beam splitter shown in FIG. 4; FIG. 11 is an explanatory diagram for explaining detection of a touch position using a search image; 1 is a block diagram showing an example of the configuration of a projection control apparatus according to one embodiment; FIG. FIG. 10 is an explanatory diagram for explaining a first example of UI mode switching based on the positional relationship of the projector with respect to the projection plane; FIG. 11 is an explanatory diagram for describing a second example of UI mode switching based on the positional relationship of the projector with respect to the projection plane; FIG. 12 is an explanatory diagram for describing a third example of switching between UI modes based on the positional relationship of the projector with respect to the projection plane; FIG. 4 is an explanatory diagram showing an example of user input that can be detected by a first user input detection system; FIG. 4 is an explanatory diagram showing a first example of user input that can be detected by a second user input detection system; FIG. 9 is an explanatory diagram showing a second example of user input that can be detected by the second user input detection system; FIG. 10 is an explanatory diagram for explaining an example of a user interface for rotating a projected image; FIG. 10 is an explanatory diagram for explaining an example of a projected image that changes based on input conditions that do not depend on user input; 6 is a flowchart showing an example of the flow of user interface control processing according to one embodiment; FIG. 11 is a flowchart showing a first example of a detailed flow of UI mode selection processing; FIG. FIG. 11 is a flowchart showing a second example of a detailed flow of UI mode selection processing; FIG. FIG. 11 is a flowchart showing a third example of a detailed flow of UI mode selection processing; FIG. FIG. 13 is a flowchart showing a fourth example of the detailed flow of UI mode selection processing; FIG. FIG. 16 is a flowchart showing a fifth example of the detailed flow of UI mode selection processing; FIG.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Also, the description is given in the following order.
1. Outline of Apparatus 1-1. Appearance of device 1-2. Various installation postures 1-3. Mechanism of touch input detection system 2 . Device Configuration 2-1. Overall Configuration 2-2. Switching UI Modes 2-3. Example of user input 2-4. Input conditions that do not depend on user input;3. Flow of processing 3-1. UI control process 3-2. UI mode selection process4. summary

<1. Device Overview>
[1-1. Appearance of device]
FIG. 1 shows the appearance of a projection control apparatus 100 according to one embodiment. The projection control device 100 is a so-called projector that projects an image onto a projection plane that exists outside the device. In the example of FIG. 1, the projection control apparatus 100 has the shape of a rectangular parallelepiped, and can be placed on any plane with any of its surfaces as the bottom surface. Note that the shape of the projection control apparatus 100 shown in FIG. 1 is merely an example. The projection control apparatus 100 may have any shape as long as it can be stably installed in real space.

One surface of the housing of the projection control apparatus 100 is the exit surface 10a. The exit surface 10a is provided with an aperture 11 for projection light and an aperture 12 for search light. Projection control apparatus 100 further includes camera 13 , microphone 14 and speaker 15 . The projection control apparatus 100 forms a projection image on the projection plane by projecting the projection light onto the projection plane through the projection light aperture 11 . Further, the projection control apparatus 100 detects a user's touch input by emitting search light through the search light opening 12 toward a search area near the projector, as will be described in detail later.

In this embodiment, the projection control apparatus 100 comprises two user input detection systems 17,18. The first user input detection system 17 uses the probe light described above to detect a user's touch input with relatively high detection accuracy. The second user input detection system 18 may have a coarser touch detection accuracy than the first user input detection system, or may be incapable of detecting touches. The second user input detection system 18 may, for example, detect gesture input from the user based on an image captured by the camera 13, or detect voice input from the user based on sound collected by the microphone 14. You may

[1-2. Various installation postures]
The projection control apparatus 100 can be installed in real space in various installation postures. In this section, two typical installation postures of the projection control apparatus 100 will be described.

FIG. 2A is an explanatory diagram for explaining the first installation attitude of the projection control apparatus 100. FIG. Referring to FIG. 2A, the projection control apparatus 100 is placed on the table surface 3a in the first installation posture. In the first installation posture, the exit surface 10a of the projection control apparatus 100 is substantially orthogonal to the table surface 3a, which is the installation surface. That is, when the table surface 3a is horizontal, the exit surface 10a is vertical. In the first installation posture, the projection control device 100 can project the projection image 2a onto the table surface 3a. In this case, below the projection control apparatus 100, the table surface 3a serves as the projection plane. In the example of FIG. 2A, three users are sitting around a table and each user can see the projected image 2a on the table surface 3a.

FIG. 2B is an explanatory diagram for explaining the second installation attitude of the projection control apparatus 100. As shown in FIG. Referring to FIG. 2B, the projection control apparatus 100 is installed on the floor in the second installation posture. In the second installation posture, the exit surface 10a of the projection control device 100 becomes the upper surface of the projection control device 100. FIG. That is, when the floor surface is horizontal, the exit surface 10a is also horizontal. In the second installation posture, the projection control device 100 can project the projection image 2b onto the wall surface 3b. In the example of FIG. 2B, the user is positioned on the opposite side of the wall surface 3b with respect to the projection control device 100, and the user can see the projected image 2b on the wall surface 3b.

Comparing the first installation posture of FIG. 2A and the second installation posture of FIG. In the second installation posture, it is not so easy to touch the wall surface 3b that is the projection surface. As an example of another installation posture, when the projection control device 100 is installed upside down from the first installation posture, the projection control device 100 can project a projection image onto the ceiling surface. In this case as well, it is not so easy for the user to touch the ceiling surface, which is the projection surface.

In this way, depending on the positional relationship of the projector with respect to the projection surface, there are cases where touch input on the projection surface is suitable for user interfaces (UI), and such touch input is not suitable for UI. There is also a case. Therefore, the technology according to the present disclosure adopts a mechanism that adaptively changes the method of the user interface that utilizes the projected image depending on where the image is projected, as will be described in detail in this specification.

[1-3. Mechanism of touch input detection system]
Here, an example of a touch detection method for detecting touch input on the projection surface will be described. Note that the method described here is merely an example, and other touch detection methods may be used.

(1) Outline of Touch Input Detection Method FIG. 3 is an explanatory diagram for explaining an outline of a touch input detection method that can be employed in the first user input detection system 17. As shown in FIG. In FIG. 3, similarly to the example of FIG. 2A, the projection control apparatus 100 is placed on the table surface 3a in the first installation posture. The projection control device 100 projects the projection image 21 onto the table surface 3a by emitting the projection light through the projection light aperture 11 . Furthermore, the projection control apparatus 100 forms the search plane 22 by emitting the search light through the search light opening 12 toward the search area near the projector. The probing light may be, for example, non-visible light such as infrared light.

The search light opening 12 is provided, for example, slightly above the bottom surface of the projection control apparatus 100 in the first installation posture (for example, the height from the bottom surface may be several millimeters to several tens of millimeters). The search plane 22 formed by the trajectory of the search light may be, for example, substantially parallel to the bottom surface of the projection control apparatus 100. In this case, the search plane 22 is a short distance from the table surface 3a, which is the projection plane. are separated by approximately parallel planes. Such a probing surface 22 may be understood to be a barrier film consisting of trajectories of the probing light covering the projected image 21 . The projection plane is usually flat (however, the technology according to the present disclosure can also be applied to cases where the projection plane is not necessarily flat). If there is no shielding object on or near the projection plane, the search light passes through the projection image 21 without being shielded. However, when the user touches the projected image with a finger or other manipulator, the probe light is diffusely reflected by the manipulator and goes in all directions. The first user input detection system 17 of the projection control apparatus 100 receives such reflected interrogation light through the projection light aperture 11, which is positioned in optical conjugate with the projector light valve. Detected by a sensor. As a result, the projection control apparatus 100 can detect where on the projected image is touched (pointed) with accuracy close to that of a general type of touch sensor, such as a resistive film type or capacitive type. .

(2) Configuration of Detection System FIG. 4 is a block diagram showing an example of a detailed configuration of the first user input detection system 17. As shown in FIG. In the example of FIG. 4, first user input detection system 17 is part of projector module 110 . Projector module 110 includes image projection system 16 , first user input detection system 17 , polarizing beam splitter 46 , projection lens 47 , and projector controller 55 . The image projection system 16 includes a first light source 41 , a light valve 42 and a first signal processor 43 .
The first user input detection system 17 includes a second light source 31 , a detection lens unit 51 , an image sensor 52 and a second signal processing section 53 . The second light source 31 is arranged in the projector sub-module 110a, which is connected to the opening 12 for the search light, while the remaining components of the projector module 110 shown in FIG. 4 are connected to the opening 11 for the projection light. It is arranged in the projector sub-module 110b.

The first light source 41 is a light source that generates visible light, and irradiates the generated visible light toward the light valve 42 via the polarization beam splitter 46 . The visible light generated by the first light source 41 may be, for example, laser light containing three color components of red, green and blue. The light valve 42 may be, for example, a reflective liquid crystal device such as LCOS (Liquid Crystal On Silicon). The light valve 42 modulates, for example, a first polarized component (for example, an s-polarized component Ls1 described later) of visible light incident from the first light source 41 based on an image signal input from the first signal processing section 43. , to a second polarization component (eg, a p-polarization component Lp2, described below). Visible light (modulated light) having the second polarization component is emitted toward the projection lens 47 via the polarization beam splitter 46 . The light valve 42 can express black by reflecting the first polarization component of the incident visible light in the same polarization state.

The projection lens 47 forms a projection image 21 on the projection plane by projecting the modulated light 21a incident from the light valve 42 through the polarization beam splitter 46 onto the projection plane through the projection light aperture 11. FIG. Further, the exploration light 22b (reflected by some operator) is incident on the projection lens 47 through the opening 11 for projection light from the direction opposite to the traveling direction of the modulated light 21a. The probing light 22 b is further incident on the polarizing beam splitter 46 via the projection lens 47 .

The polarizing beam splitter 46 splits the incident probe light into a first polarization component (eg, s-polarization component) and a second polarization component (eg, p-polarization component) and directs them in different directions. For example, the first polarization component of the interrogation light is selectively reflected at the polarizing beam splitter 46 and enters the detection lens unit 51 . On the other hand, the second polarization component of the probe light is selectively transmitted by the polarization beam splitter 46 and enters the light valve 42 . It should be noted that instead of the polarizing beam splitter 46, other types of polarization separating elements such as wire grids may be used.

The operation of the polarizing beam splitter shown in FIG. 4 will be further described with reference to FIG. The polarizing beam splitter 46 has a first optical surface 46a, a second optical surface 46b, a third optical surface 46c and a fourth optical surface 46d, and a polarization splitting surface 46e. The first optical surface 46a and the third optical surface 46c are arranged back to back in the horizontal direction in the drawing, and the second optical surface 46b and the fourth optical surface 46d are arranged back to back in the vertical direction in the drawing. The first optical surface 46a faces the first light source 41 (not shown in FIG. 5). The second optical surface 46 b faces the light valve 42 . The third optical surface 46 c faces the detection lens unit 51 . The fourth optical surface 46d faces the projection lens 47 (not shown in FIG. 5).

The visible light L1 emitted from the first light source 41 is incident on the first optical surface 46a, and the first polarized component (s-polarized component Ls1) of the visible light L1 is reflected on the polarization splitting surface 46e and reflected on the second optical surface. 46b. The second polarized component (p polarized component Lp1) of the visible light L1 passes through the polarization separating surface 46e and emerges from the third optical surface 46c. The modulated light Lp2 from the light valve 42 enters the second optical surface 46b, and the second polarized component (p polarized component Lp2) of the modulated light Lp2 exits from the fourth optical surface 46d.

On the other hand, the exploration light L3 is incident on the fourth optical surface 46d, and the first polarized component (s-polarized component Ls3) of the exploration light L3 is reflected by the polarization separation surface 46e and emitted from the third optical surface 46c. The second polarized component (p-polarized component Lp3) of the probe light L3 passes through the polarization separating surface 46e and emerges from the second optical surface 46b. The first polarized component Ls3 of the search light L3 passes through the relay lenses 51a and 51b of the detection lens unit 51 and the visible light cut filter and bandpass filter (not shown) to reach the image sensor 52. A bandpass filter selectively passes light in a particular wavelength range (including the wavelength of the probing light).

Referring to FIG. 4 again, the second light source 31 is a light source that generates search light 22a for searching the controls used for touch input, and directs the generated search light to a search area near the projector. Aim at. As described above, the probing light 22a may be non-visible light such as infrared light (eg, near-infrared light). The search light 22a from the second light source 31 is emitted radially through the search light opening 12, for example, so as to form the search surface 22 substantially parallel to the bottom surface of the projection control apparatus 100 in the example of FIG. . Then, the first polarized component of the exploration light 22b reflected by the operator touching the projection image 21 is further reflected by the polarization beam splitter 46 and reaches the detection lens unit 51, as described above.

The detection lens unit 51 includes one or more relay lenses that form an image of the search light 22b at a position optically conjugated with the light valve 42 . For the purpose of downsizing the image sensor 52, the detection lens unit 51 may reduce the image of the searching light 22b to equivalently reduce the image of the searching light 22b. The image sensor 52 captures an image of the search light 22b formed at a position optically conjugate with the light valve 42 to generate a search image signal. The image sensor 52 may be any type of imaging device such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge Coupled Device).

The second signal processing unit 53 performs predetermined signal processing such as development and noise removal on the search image signal output from the image sensor 52 . The second signal processing unit 53 then outputs the processed search image signal to the projector controller 55 .

The first signal processing unit 43 causes the driver circuit of the light valve 42 to supply a voltage to each pixel of the light valve 42 according to the projection image signal input via the projector controller 55 . As a result, the above-described visible light (projection light) is modulated, and the projection image 21 is reproduced on the projection plane.

The projector controller 55 outputs a projection image signal supplied from the outside of the projector module 110 to the first signal processing section 43 to project the projection image 21 onto the projection plane through the image projection system 16 . Projector controller 55 may perform common projection control functions such as autofocus and keystone correction. Furthermore, in this embodiment, the projector controller 55 detects touch input on the projected image based on the exploration image represented by the exploration image signal input from the first user input detection system 17 .

FIG. 6 is an explanatory diagram for explaining touch position detection using a search image. In the example of FIG. 6 as well, the search surface 22 is formed so as to cover the projection image 21. As shown in FIG. The user touches the position P _prj on the projected image 21 with, for example, a finger. The probe light reflected by the finger touching the position P _prj is captured by the image sensor 52 and forms the manipulator image R _fin within the probe image 23 . The projector controller 55 performs, for example, a known feature point detection method on such manipulator image R _fin to detect the touch position P _tou in the search image 23 . The feature point position (that is, the touch position) may be, for example, the position of the tip of the finger or the position of the center of gravity of the manipulator image R _fin . As described above, the exploration image is captured by the image sensor 52 arranged at a position substantially optically conjugate with the light valve 42. Therefore, the coordinates of the touch position P _tou in the exploration image 23 are the coordinates of the projected image 21 match the coordinates of the touch position P _prj at .

By using the first user input detection system 17 as described above, touch input on the projection plane can be detected with high detection accuracy. In particular, in the first installation posture shown in FIG. 2A, since the operator usually reflects the probe light only in the immediate vicinity of the projection surface, the touch input detection accuracy is at a level close to that of a general type of touch sensor. can be achieved. On the other hand, in the second installation posture shown in FIG. 2B, the search plane is not necessarily in the vicinity of the projection plane. It is not easy to detect with the accuracy of An obstacle (for example, a part of the user's body) other than the operator may interfere with touch detection as a result of intervening between the projection surface and the projector. In the embodiment described in detail in the next section, considering such circumstances, the positional relationship of the projector with respect to the projection plane is dynamically determined, and based on the determination result, the user interface mode is set to two or more UI modes. can be switched between

<2. Device Configuration>
[2-1. Overall configuration]
FIG. 7 is a block diagram showing an example of the configuration of the projection control apparatus 100 according to one embodiment. 7, projection control apparatus 100 includes projector module 110, recognition module 120, sensor module 130, software module 140, communication interface 150 and integrated A control unit 160 is provided. Projector module 110 includes image projection system 16 and first user input detection system 17 as described above. Camera 13 , microphone 14 and speaker 15 together with recognition module 120 constitute a second user input detection system 18 .

(1) Projector Module The projector module 110 is a module that projects an image onto a projection surface and controls the projection. An example of the detailed configuration of the projector module 110 is as described with reference to FIGS. 4 and 5. FIG. The projector module 110 projects an image generated by a software module 140, which will be described later, onto a projection plane through the image projection system 16 under the control of the integrated control unit 160, for example. The projected image may be a still image or each of the frames that make up a moving image.

The first user input detection system 17 of the projector module 110 can detect the user's touch position using the search light emitted toward the search area near the projection control device 100 as described above. When the projection control apparatus 100 is installed in the above-described first installation attitude, the search area forms an investigation plane substantially parallel to the installation plane. On the other hand, when the projection control apparatus 100 is installed in the above-described second installation attitude, the investigation area forms an investigation plane substantially orthogonal to the installation plane. When the projection plane is substantially parallel to the exploration plane and the distance between the projection plane and the exploration plane is small, the first user input detection system 17 can achieve a sufficient level of touch detection accuracy. . When the projection plane is almost parallel to the search plane, it means that the planes are strictly parallel to each other, and that the touch detection error caused by the angle between the planes (for example, for the touch input adaptive software described later) ) is within acceptable limits. Upon detecting a touch input, the first user input detection system 17 notifies the integrated control unit 160 of the occurrence of a touch event. Touch event information may include an event type such as touch, tap, double-tap, flick, pinch-in or pinch-out, and one or more touch position coordinates associated with the event. In some UI modes, the first user input detection system 17 can be used as the main input interface. In other UI modes, the first user input detection system 17 may be used as an auxiliary input interface or disabled.

(2) Recognition Module The recognition module 120 is a module that executes recognition processing for detecting user input in the second user input detection system 18 . Recognition module 120 may detect user input based on image input from camera 13, for example. The camera 13 is typically arranged in the projection control apparatus 100 so as to have an angle of view in the direction in which the user is supposed to exist in the second installation posture of the projection control apparatus 100 . In the example of FIG. 2B, the camera 13 is arranged on the opposite side of the projection plane with respect to the projection control device 100 . In another example, the camera 13 may be arranged on the side perpendicular to the plane of projection. Also, the camera 13 may be an omnidirectional camera or a half-dome camera. The recognition module 120 may detect a gesture input as a user input, for example, by recognizing a user's gesture appearing in a captured image input from the camera 13 . Further, the recognition module 120 may detect gaze input as the user input by recognizing the movement of the user's eye appearing in the captured image input from the camera 13 . When user input is detected based on such image input, even if it is possible to virtually touch (or point) a point on the projected image by user input, the touch detection accuracy (pointing accuracy) is Compared to the first user input detection system 17, it is relatively rough. In some situations touch detection (pointing to the desired point) is not possible.

Additionally or alternatively, recognition module 120 may detect user input based on voice input from microphone 14, for example. Recognition module 120 may detect voice commands as user input, for example, by recognizing predetermined voice patterns contained in the sound picked up by microphone 14 . If user input is detected based on such voice input, pointing to a desired point on the projection plane is not possible.

When the recognition module 120 detects a user input that can correspond to gesture input, gaze input, or voice command, the recognition module 120 notifies the integrated control unit 160 of occurrence of a user input event. The types of user inputs that can be detected by the second user input detection system 18 are not limited to the examples described above. For example, the second user input detection system 18 may detect user input via a mechanical interface such as a button, switch or lever located on the device, or may detect vibration of the device or recognition of user exhalation, etc. may detect user input through

(3) Sensor Module Sensor module 130 includes one or more sensors used to determine the positional relationship of projection control apparatus 100 with respect to the projection plane. As an example, sensor module 130 may include an orientation sensor 131 that measures the orientation of projection control apparatus 100 . The orientation sensor 131 may be, for example, an acceleration sensor or a gyro sensor. The attitude sensor 131 measures the tilt angle of the projection control apparatus 100 at that time with respect to the reference attitude of the projection control apparatus 100 and outputs the measurement result to the integrated control section 160 .

As another example, the sensor module 130 includes a depth sensor 133 that measures the distance from the projection control apparatus 100 to the projection plane (hereinafter referred to as the depth of the projection plane) instead of or in addition to the orientation sensor 131 . can contain. The depth sensor 133 may be any type of depth sensor such as, for example, a stereo camera type or a TOF (Time of Flight) type. Depth sensor 133 typically measures the depth of the projection plane with reference to the position of projection control apparatus 100 and outputs the measurement result to integrated control section 160 .

(4) Software module The software module 140 has a processor such as a CPU (Central Processing Unit) and a memory that can include ROM (Read Only Memory) and RAM (Random Access Memory), and generates an image to be projected. run software that As an example, the software module 140 selectively executes touch input adaptive software 141 and touch input non-adaptive software 143 under the control of the integrated controller 160 . The touch-input adaptive software 141 may include touch-input adaptive base software (eg, a mobile operating system such as Android®, iOS, or Windows Mobile®). The touch-input adaptive software 141 may also include touch-input adaptive application software. Touch input adaptive application software includes various software such as web browsers, mailers, text editors or instant messengers designed to operate on mobile devices such as smartphones or tablet PCs (Personal Computers). good.

Non-touch input adaptive software 143 may be any software that does not require fine pointing to operate. Non-touch input adaptive software 143 may also include any combination of base software and application software. Some examples of touch input non-adaptive software 143 are further described below.

(5) Communication Interface The communication interface 150 mediates communication between the projection control apparatus 100 and other apparatuses. The communication interface 150 uses a cellular communication protocol such as WiMAX, LTE (Long Term Evolution) or LTE-A (LTE-Advanced), or a wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). Other wireless communication protocols may be implemented. Also, the communication interface 150 may implement a wired communication protocol such as USB (Universal Serial Bus) or HDMI (High-Definition Multimedia Interface). Projector module 110 may project images received from other devices via communication interface 150 onto the projection surface in place of images generated in software module 140 .

(6) Integrated Control Unit The integrated control unit 160 has a processor such as a CPU and a memory that can include ROM and RAM, and controls the overall operation of the projection control apparatus 100 . For example, when the projection control apparatus 100 is activated, the integrated control section 160 causes the software module 140 to execute any software. The integrated control unit 160 also causes the projector module 110 to project an image generated by the executed software (for example, an image of an operating system or an image of an application) onto the projection surface. Further, when receiving notification of some user input event from the first user input detection system 17 or the second user input detection system 18, the integrated control unit 160 transfers the user input event to the software being executed. When the notified user input event indicates a specific user input, the integrated control unit 160 may process the user input event itself instead of passing the user input event to software (for example, stop operation). turning off the power of the projection control apparatus 100 according to the instructed voice command, etc.).

In this embodiment, the integrated control unit 160 determines the positional relationship of the projection control apparatus 100 with respect to the projection plane, and based on the determination result, selects two user interface (UI) modes related to the image to be projected. Switch between the above UI modes. The first UI mode is a mode that uses the first user input detection system 17 as a main input interface. A second UI mode is a mode in which the first user input detection system 17 is not used as a main input interface. In the following description, the first UI mode is called "touch mode" and the second UI mode is called "non-touch mode".

Integrated control 160 may cause software module 140 to run different software depending on whether touch mode or non-touch mode is selected. For example, integrated controller 160 may cause software module 140 to execute touch-input adaptive software 141 when touch mode is selected. Also, when the non-touch mode is selected, the integrated control unit 160 may cause the software module 140 to execute the touch input non-adaptive software 143 .

[2-2. Switch UI mode]
The integrated control unit 160 typically selects the touch mode when it is determined that the first user input detection system 17 can achieve a certain level of touch detection accuracy from the positional relationship of the projection control device 100 with respect to the projection plane. select. For example, as illustrated in FIG. 2A, when the projection control apparatus 100 is placed on the table surface in the first installation posture and an image is projected onto the table surface, the search plane is near the projected image. Since it is formed to cover the projected image, the first user input detection system 17 can achieve a sufficient level of touch detection accuracy for the touch input adaptive software 141 . Therefore, the integrated control unit 160 can select the touch mode in this case. Such selection may be made based on a simple determination using the orientation sensor 131 or the depth sensor 133 that the projection plane is positioned substantially horizontally below the projection control apparatus 100 . On the other hand, as illustrated in FIG. 2B, when the projection control apparatus 100 is installed in the second installation attitude, there is a certain distance between the search plane and the projection plane of the projection control apparatus 100, and the first It can be assumed that the user input detection system 17 does not achieve a sufficient level of touch detection accuracy. Therefore, the integrated control unit 160 can select the non-touch mode in this case.

As a first example of determining the positional relationship, the integrated control unit 160 performs projection control when the measurement result of the orientation of the projection control apparatus 100 input from the orientation sensor 131 indicates that the investigation plane is substantially horizontal. It may be assumed that the device 100 is installed on the projection plane in the above-described first installation posture. FIG. 8A is an explanatory diagram for describing a first example of UI mode switching based on the positional relationship of the projector with respect to the projection plane. Here, of the surfaces of the housing of the projection control apparatus 100 orthogonal to the exit surface 10a described using FIG. shall be the reference plane for measurement. The reference plane 10b is substantially parallel to the investigation plane. On the left side of FIG. 8A, the measurement result input from the orientation sensor 131 indicates that the reference plane 10b is horizontal and that the normal to the reference plane 10b is directed in the direction of gravity. This measurement result can mean that the projection control apparatus 100 is installed in the first installation posture on a horizontal surface such as a table surface. Therefore, the integrated control section 160 can select the touch mode based on this measurement result. On the other hand, on the right side of FIG. 8A, the measurement result input from the orientation sensor 131 indicates that the reference plane 10b is perpendicular to the horizontal plane. This measurement result may mean that the projection control apparatus 100 is not installed in the first installation attitude. Therefore, the integrated control section 160 can select the non-touch mode based on this measurement result. Table 1 provides a framework for UI mode selection based on such projector poses.

As a second example of determination of the positional relationship, the integrated control unit 160 determines when the measurement result indicating the depth of the projection plane input from the depth sensor 133 indicates that the distance between the search plane and the projection plane is small. , it may be simply determined that the projection control apparatus 100 is installed on the projection plane in the above-described first installation attitude. FIG. 8B is an explanatory diagram for explaining a second example of UI mode switching based on the positional relationship of the projector with respect to the projection plane. Here, the depth sensor 133 measures the depth in the normal direction of the reference plane 10b. On the left of FIG. 8B, the measurement result input from the depth sensor 133 shows a depth less than a certain threshold, which means that there is an installation surface, such as a table surface, directly below the reference surface 10b. Therefore, the integrated control section 160 can select the touch mode based on this measurement result. On the other hand, on the right side of FIG. 8B, the measurement result input from the depth sensor 133 indicates a depth greater than a certain threshold because the reference plane 10b is not in contact with any object. Therefore, the integrated control section 160 can select the non-touch mode based on this measurement result. Table 2 provides a framework for such depth-based UI mode selection.

Note that, according to the second example described above, for example, in the situation shown in FIG. Depth may be indicated and the integrated control unit 160 may erroneously select the touch mode. However, when an unintended UI mode is selected, the user can easily cause the projection control apparatus 100 to reselect the UI mode by moving the obstructing object. Therefore, the possibility of such an erroneous selection does not impair the usefulness of the method described above.

As a third example, the integrated control unit 160 determines the positional relationship of the projection control apparatus 100 with respect to the projection plane based on both the first measurement result from the attitude sensor 131 and the second measurement result from the depth sensor 133. You can judge. FIG. 9 is an explanatory diagram for explaining a third example of UI mode switching based on the positional relationship of the projector with respect to the projection plane. On the left side of FIG. 9, the first measurement result input from the orientation sensor 131 indicates that the reference plane 10b is horizontal and that the normal to the reference plane 10b is directed in the direction of gravity. Therefore, the integrated control section 160 can select the touch mode based on this measurement result. In the upper right of FIG. 9, the measurement result input from the orientation sensor 131 indicates that the reference plane 10b is perpendicular to the horizontal plane. Therefore, the integrated control unit 160 further acquires the second measurement result from the depth sensor 133 indicating the depth of the projection plane. Then, since the second measurement result obtained by the integrated control unit 160 indicates a depth smaller than a certain threshold, the search plane of the projection control apparatus 100 is sufficiently close to the projection plane, and the first user input detection is performed. It may be determined that system 17 can achieve a sufficient level of touch detection accuracy. Therefore, the integrated control unit 160 can select the touch mode in this case. On the other hand, when the second measurement result obtained from the depth sensor 133 indicates a depth greater than the threshold, as in the lower right of FIG. It may be determined that the level of touch detection accuracy is not achieved. Therefore, the integrated control unit 160 can select the non-touch mode in this case. Table 3 provides a framework for such UI mode selection based on both projector pose and projection surface depth.

Note that the UI mode selection framework shown in Table 3 is merely an example. In another example, when the second measurement result from the depth sensor 133 indicating the depth of the projection plane indicates a depth larger than a certain threshold, the integrated control unit 160 determines whether the non-touching is performed regardless of the orientation of the reference plane. You can choose the mode. Then, when the second measurement result indicates a depth smaller than the threshold, the integrated control unit 160 acquires the first measurement result from the orientation sensor 131, and the normal line of the reference plane points in the direction of gravity. The touch mode may be selected if the first measurement indicates that the touch mode is present, and the non-touch mode otherwise. According to such an example, there is a step between the plane on which the projection control apparatus 100 is installed and the projection plane, and in a situation where the search plane is not formed in the vicinity of the projection plane, sufficient touch detection accuracy can be achieved. It is possible to avoid an inconvenient situation in which the touch mode is selected even though it is not achieved. In addition, even if the depth of the projection surface is small, there is also the advantage that the non-touch mode is selected over the touch mode in the situation where the shadow of the finger becomes large when projecting the image on the wall and the operability of the touch input deteriorates. obtain. Table 4 provides a framework for selecting such UI modes that differs from Table 3.

So far, an example has been described in which the mode is switched between two UI modes. can be switched. For example, there are three UI modes: "table mode" when the table surface is the projection surface, "wall mode" when the wall surface is the projection surface, and "ceiling mode" when the ceiling surface is the projection surface. You may Table mode and wall mode may correspond to the touch mode and non-touch mode described above, respectively. Ceiling mode may also correspond to non-touch mode. However, while rotation of the projected image based on user input is enabled in "Ceiling Mode", rotation of the projected image may be disabled in "Wall Mode". For example, when the measurement result input from the attitude sensor 131 indicates that the reference plane 10b is horizontal, but the normal line of the reference plane 10b faces the direction opposite to the direction of gravity, These ceiling modes may be selected.

[2-3. Example of user input]
As described above, the integrated control section 160 uses the first user input detection system 17 as the main input interface in the touch mode. That is, the integrated control unit 160 activates the first user input detection system 17 in the touch mode. Then, when the first user input detection system 17 detects a touch input, the integrated control unit 160 transfers the touch event notified from the first user input detection system 17 to the software in execution.

FIG. 10 is an explanatory diagram showing an example of user input that can be detected by the first user input detection system 17. As shown in FIG. Referring to FIG. 10, projection control apparatus 100 is placed on a table surrounded by three users. The projection control device 100 projects the projection image Im10 onto the table surface 3a. The projected image Im10 is an image of a touch-input adaptive web browser that runs on an operating system for mobiles. For example, the user Ua taps a position P10 on the projection image _Im10 in order to transition the page displayed by the web browser to a desired linked web page. Then, the first user input detection system 17 captures the image of the user Ua who tapped the position _P10 with the probe light, and notifies the integrated control unit 160 of a touch event indicating that the position _P10 has been tapped. The integrated control unit 160 passes the notified touch event to the running web browser (via the operating system), and as a result, the web page of the link destination desired by the user Ua is displayed by the web browser. In the touch mode, in order for the first user input detection system 17 to achieve sufficient touch detection accuracy, the user can operate the touch sensor of a smartphone or tablet PC through the projected image as if they were operating it. can freely interact with running software.

In the non-touch mode, the integrated control unit 160 selects the second UI mode using the second user input detection system 18 as the main input interface. Then, when the second user input detection system 18 detects some user input, the integrated control unit 160 transfers the user input event notified from the second user input detection system 18 to the running software.

FIG. 11 is an explanatory diagram showing a first example of user input that can be detected by the second user input detection system 18. As shown in FIG. Referring to FIG. 11, a projection control device 100 is installed between the wall surface 3b and the user Ub. The projection control device 100 projects the projection image Im11 onto the wall surface 3b. The projection image Im11 is an image of a content viewer that is not adapted to touch input. For example, user Ub performs a gesture G1 moving his hand to the right or left. Gesture G1 may be interpreted as a so-called swipe. Then, the recognition module 120 of the second user input detection system 18 recognizes the gesture G1 appearing in the captured image input from the camera 13 and notifies the integrated control unit 160 of the gesture input event. The integrated control unit 160 passes the notified gesture input event to the content viewer, and as a result, the display of the content in the projection image is scrolled in the direction corresponding to the gesture input.

Recognition module 120 may recognize any type of gesture in addition to (or in place of) the swipes described above, such as hands up, hands waving, hands together and hands apart. In such a gesture interface, the second user input detection system 18 may be able to detect touch (or pointing) based on, for example, changes in hand position and area within the captured image. However, its detection accuracy is coarser than the level of accuracy achievable by the first user input detection system 17 . Even if the user's line of sight recognized from the captured image is used as user input, touch (or pointing) is not possible, or if not, only coarse touch detection accuracy is achieved.

FIG. 12 is an explanatory diagram showing a second example of user input that can be detected by the second user input detection system 18. As shown in FIG. Also in FIG. 12, the projection control device 100 is installed between the wall surface 3b and the user Ub. For example, user Ub utters voice command C1 for calling up the calendar. Then, the recognition module 120 of the second user input detection system 18 recognizes the voice command C1 included in the voice input from the microphone 14 and notifies the integrated control unit 160 of the voice input event. The integrated control unit 160 causes the software module 140 to activate the calendar application in response to the notified voice input event. As a result, a projection image Im12 displaying a calendar generated by the calendar application is projected onto the wall surface 3b. The calendar application is touch input non-adaptive application software and can be operated by the user through gesture input or voice input, for example. Recognition module 120 may recognize any type of voice command. User interfaces that use voice commands are typically touch-insensitive.

11 and 12, the second user input detection system 18 may include any type of user input means. For example, a human sensor that detects the presence of a human body within a range of several tens of centimeters may be included in the second user input detection system 18 as user input means that detects user input with binary on/off values.

11 and 12 show images of a content viewer and a calendar application as images of touch input non-adaptive software. However, non-touch input adaptive software is not limited to such examples. For example, touch-input non-adaptive software with various purposes, such as a clock, calendar or home energy management, may be implemented and any image of such software may be projected in non-touch mode.

When the touch mode is selected, the integrated control unit 160 uses the second user input detection system 18 as described with reference to FIGS. 11 and 12 as an input interface in addition to the first user input detection system 17. You can On the other hand, when the non-touch mode is selected, the integrated control unit 160 may disable the first user input detection system 17 in order to prevent an erroneous touch operation not intended by the user (for example, the detection light is stop irradiation, etc.). Also, when the non-touch mode is selected, the integrated control unit 160 may use the first user input detection system 17 as an input interface in addition to the second user input detection system 18 . In non-touch mode, fine pointing through touch input can be difficult due to the magnitude of the distance between the projection plane and the search plane. However, it is still possible to use the first user input detection system 17 as a simple user input means (for example, detecting binary values of ON/OFF) such as whether or not the probe light is blocked by hand.

The integrated control unit 160 activates a user interface (hereinafter referred to as rotation UI) for rotating the projected image when the projection plane is determined to be substantially horizontal, and when the projection plane is determined to be substantially vertical. may disable the rotation UI. The rotation UI may include one or more of a GUI (Graphical User Interface) and an audio UI displayed with the projection image.

FIG. 13 is an explanatory diagram for explaining an example of the rotation UI. Referring to the left side of FIG. 13, projection control apparatus 100 is placed on a table surrounded by two users. The projection control device 100 projects the projection image Im13a onto the table surface 3a. The projection image Im13a is an image of a touch-input adaptive web browser that runs on an operating system for mobiles. Buttons J1a, J1b, J1c and J1d are displayed at the four corners of the projected image Im13a. These buttons J1a, J1b, J1c and J1d are GUI items for rotating the projected image. For example, when the user Uc taps the button J1b, the first user input detection system 17 notifies the integrated control unit 160 of a touch event indicating that the position of the button J1b has been tapped. The integrated control unit 160 rotates the display of the web browser clockwise by 90 degrees according to the notified touch event. As a result, as shown on the right side of FIG. 13, a projection image Im13b displaying the web page in an orientation suitable for viewing by the user Uc is projected.

If the rotating UI is an audio UI, the sound may be acoustic or may be the user's speech. For example, the sound produced when the user hits the table may be used as an audio input for rotating the projected image. Further, the second user input detection system 18 may determine in which direction the projected image should be rotated from the arrival direction of the sound recognized using the array type microphone.

[2-4. Input conditions that do not depend on user input]
In non-touch mode, only touches with coarse touch detection accuracy are possible, or touch input is not possible. Therefore, in order to make the user experience richer in the non-touch mode, the integrated control unit 160 causes the software module 140 to execute touch-input non-adaptive software that changes the projected image based on input conditions that do not depend on user input. You may let Input conditions that do not depend on user input here may be, for example, conditions related to at least one of the following: date, time, weather, projector location, and recognized user.

FIG. 14 is an explanatory diagram for explaining an example of a projected image that changes based on input conditions that do not depend on user input. Referring to FIG. 14, the projection control apparatus 100 is installed in the second installation posture in front of the wall surface 3b. On the left side of FIG. 14, it is assumed that the clock indicates noon time and the weather is fine. The integrated control unit 160 can obtain the weather from weather data acquired from an external server via the communication interface 150, for example. Based on these input conditions, the integrated control unit 160 projects onto the wall surface 3b a projection image that displays a background that expresses the appearance of a sunny outdoors in the daytime. On the right side of FIG. 14, it is assumed that the clock indicates the time of night and the weather is cloudy. Based on these input conditions indicated by the time and weather data, the integrated control unit 160 projects a projection image displaying a background representing a cloudy outdoor scene at night onto the wall surface 3b.

Recognition module 120 may use known personal recognition techniques such as facial recognition, fingerprint recognition, or iris recognition to identify users viewing projected images. When a calendar or clock application is run in non-touch mode, the application may overlay pre-registered appointment information for the identified user on the calendar or clock image. The recognition module 120 may also recognize the installation location of the projector based on the position measurement results input from a positioning sensor such as a GPS (Global Positioning System) sensor or PlaceEngine (registered trademark). In the non-touch mode, for example, when the projection control device 100 is located in the living room of the home, a calendar or clock image is projected, while when the projection control device 100 is located in the kitchen, a recommended dish is projected. Recipe information may be projected.

<3. Process Flow>
In this section, an example of the flow of processing that can be executed by the projection control apparatus 100 described above will be described using several flowcharts. Although a plurality of processing steps are described in the flowchart, these processing steps do not necessarily have to be executed in the order shown in the flowchart. Some processing steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted.

[3-1. UI control processing]
FIG. 15 is a flowchart illustrating an example of the flow of user interface control processing according to one embodiment. Here, it is assumed that the projector module 110 of the projection control apparatus 100 has already been activated.

Referring to FIG. 15, the integrated control unit 160 first executes UI mode selection processing to select a UI mode based on the positional relationship of the projection control apparatus 100 with respect to the projection plane (step S110). Some more detailed examples of the UI mode selection process performed here are further described below. Subsequent processing branches depending on whether the touch mode is selected in the UI mode selection processing (step S130).

When the touch mode is selected as the UI mode, the software module 140 executes the touch input adaptive software 141 to generate an image of the touch input adaptive software 141 (step S140). The projector module 110 projects the generated image of the touch input adaptive software 141 onto the projection surface (step S140).

Next, the integrated control unit 160 determines whether or not a touch input has been detected by the first user input detection system 17 of the projector module 110 (step S145). When a touch input is detected by the first user input detection system 17, the integrated control unit 160 transfers the touch event notified from the first user input detection system 17 to the touch input adaptive software 141 in execution ( step S150). As a result, the touch-input adaptive software 141 can perform processing (for example, reading a web page following a tapped link, etc.) in response to the touch event, and generate a new projected image. If no touch input is detected, processing may proceed to step S165.

If the non-touch mode is selected as the UI mode in step S110, the software module 140 executes the touch input non-adaptive software 143 to generate an image of the touch input non-adaptive software 143 (step S155). The projector module 110 projects the generated image of the non-touch-input adaptive software 143 onto the projection surface (step S160).

Next, the integrated control unit 160 determines whether or not any user input is detected by the second user input detection system 18 of the projector module 110 (step S165). When some user input is detected by the second user input detection system 18, the integrated control unit 160 passes the user input event notified from the second user input detection system 18 to the software in execution (step S170). . As a result, the executing software may perform processing in response to user input events to generate new projection images.

Next, the integrated control unit 160 determines whether or not to change the UI mode (step S180). For example, if the measurement result input from the sensor module 130 indicates that the orientation of the projection control apparatus 100 or the depth of the projection plane has changed, or if a user input instructing a change of the UI mode is detected, integration The control unit 160 may determine to change the UI mode. When the integrated control unit 160 determines to change the UI mode, the user interface control process of FIG. 15 returns to step S110. When the integrated control unit 160 determines not to change the UI mode, the user interface control process of FIG. 15 returns to step S130, and the above-described software image generation and projection and user input monitoring are repeated.

[3-2. UI mode selection process]
(1) First Example FIG. 16 is a flow chart showing a first example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 16, the integrated control unit 160 first acquires from the sensor module 130 the measurement result of the positional relationship of the projection control apparatus 100 with respect to the projection plane (step S113). Next, the integrated control unit 160 determines whether or not the projection plane is positioned substantially horizontally below the projection control apparatus 100 based on the acquired measurement results (step S115).

Then, when the integrated control unit 160 determines that the projection plane is positioned substantially horizontally below the projection control device 100, it selects the touch mode as the UI mode (step S120). On the other hand, if not, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

(2) Second Example FIG. 17A is a flowchart showing a second example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 17A, first, the integrated control unit 160 acquires the measurement result of the orientation of the projection control device 100 from the orientation sensor 131 (step S114). Next, the integrated control unit 160 determines whether or not the projection control apparatus 100 is installed on the projection plane based on the acquired measurement result (step S116).

For example, when the measurement result from the attitude sensor 131 indicates that the reference plane 10b is horizontal, the integrated control unit 160 determines that the projection control apparatus 100 is installed in the first installation attitude on the projection plane. obtain. When the integrated control unit 160 determines that the projection control device 100 is installed on the projection plane in this way, it selects the touch mode as the UI mode (step S120).

On the other hand, if the measurement result from the orientation sensor 131 indicates that the reference plane 10b is not horizontal, the integrated control unit 160 further acquires the measurement result indicating the depth of the projection plane from the depth sensor 133 (step S122). Next, the integrated control unit 160 determines whether the distance between the exploration plane and the projection plane of the first user input detection system 17 (that is, the depth of the projection plane) is below a threshold based on the acquired measurement result. is determined (step S124). Then, when determining that the depth of the projection surface is below the threshold, the integrated control unit 160 selects the touch mode as the UI mode (step S120). On the other hand, if the integrated control unit 160 determines that the depth of the projection plane is not below the threshold, it selects the non-touch mode (step S128).

(3) Third Example FIG. 17B is a flowchart showing a third example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 17B, first, the integrated control unit 160 acquires the measurement result indicating the depth of the projection plane from the depth sensor 133 (step S122). Next, the integrated control unit 160 determines whether or not the depth of the projection plane is below the threshold based on the obtained measurement result (step S124). When the integrated control unit 160 determines that the depth of the projection plane is not below the threshold, it selects the non-touch mode (step S128).

On the other hand, when the integrated control unit 160 determines that the depth of the projection plane is below the threshold, it further acquires the measurement result of the orientation of the projection control device 100 from the orientation sensor 131 (step S125). Next, the integrated control unit 160 determines whether or not the reference plane 10b of the projection control apparatus 100 is horizontal based on the measurement result from the attitude sensor 131 (step S126).

Then, when the measurement result from the attitude sensor 131 indicates that the reference surface 10b is horizontal, the integrated control unit 160 selects the touch mode as the UI mode (step S127). On the other hand, when the measurement result from the orientation sensor 131 indicates that the reference surface 10b is not horizontal, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

(4) Fourth Example FIG. 18 is a flowchart showing a fourth example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 18, the integrated control unit 160 first acquires the measurement result indicating the depth of the projection plane from the depth sensor 133 of the sensor module 130 (step S122). Next, the integrated control unit 160 determines whether or not a projection plane exists in a direction perpendicular to the search plane based on the acquired measurement results (step S123). The integrated control unit 160 selects the non-touch mode as the UI mode when the projection plane does not exist in the direction orthogonal to the search plane (step S128).

On the other hand, if the projection plane exists in the direction orthogonal to the exploration plane, the integrated control unit 160 determines whether the depth of the projection plane is below the threshold based on the measurement result obtained from the depth sensor 133 (step S124). Then, when determining that the depth of the projection plane is below the threshold, the integrated control unit 160 selects the touch mode as the UI mode (step S127). On the other hand, if the integrated control unit 160 determines that the depth of the projection plane is not below the threshold, it selects the non-touch mode (step S128).

(5) Fifth example All of the first, second, third, and fourth examples described so far assume that the integrated control unit 160 autonomously selects the UI mode using sensor data. and However, when the user explicitly specifies the UI mode, the UI mode specified by the user may be given priority over the autonomously selected UI mode.

19 is a flowchart showing a fifth example of the detailed flow of the UI mode selection process shown in FIG. 15. FIG. Referring to FIG. 19, first, the integrated control unit 160 determines whether or not to autonomously select the UI mode (step S111). For example, when no user input designating the UI mode is detected, the integrated control unit 160 can determine to autonomously select the UI mode.

When the integrated control unit 160 determines to autonomously select the UI mode, it acquires the measurement result of the orientation of the projection control apparatus 100 from the orientation sensor 131 of the sensor module 130 (step S114). Next, the integrated control unit 160 determines whether or not the projection control apparatus 100 is installed on the projection plane based on the acquired measurement result (step S116). When the integrated control unit 160 determines that the projection control device 100 is installed on the projection plane, it selects the touch mode as the UI mode (step S120). On the other hand, when the integrated control unit 160 determines that the projection control device 100 is not installed on the projection plane, it selects the non-touch mode as the UI mode (step S128).

Further, when the integrated control unit 160 determines not to autonomously select the UI mode in step S111, based on the user input that can be detected via the first user input detection system 17 or the second user input detection system 18, , the UI mode is selected (step S129). Selection of the UI mode based on such user input may be combined with any of the first to fourth examples of the UI mode selection process described above.

<4. Summary>
So far, the embodiments of the technology according to the present disclosure have been described in detail with reference to FIGS. 1 to 19. FIG. According to the above-described embodiments, when an image is projected onto the projection plane by the projector, the UI mode associated with the projected image is switched between two or more UI modes based on the positional relationship of the projector with respect to the projection plane. be done. Therefore, it is possible to provide the user with a user interface of a desired type that matches the positional relationship of the projector with respect to the projection plane. For example, if there is a high possibility that an obstacle that is not an operator will interfere with touch detection when the user tries to touch the projection surface, smooth user interaction is ensured by mainly using user input other than touch input. can do.

Also, according to an embodiment, the first user input detection system that detects touch input to the projection surface can be used as the primary input interface when the projection surface is positioned substantially horizontally below the projector. For example, when a portable projector is placed on a horizontal surface such as a table surface and projects an image onto the table surface, such a positional relationship can be easily estimated using an orientation sensor built into the projector. In this case, a mechanism for determining whether the projector and projection surface are in a positional relationship suitable for touch input can be constructed at low cost using a small orientation sensor.

Also, according to one embodiment, the projector includes a first user input detection system that detects a user's touch position using the probe light emitted toward the probe area near the projector. A method using such probe light can achieve high touch detection accuracy by forming an probe plane so as to cover the projected image in the vicinity of the projected image when the projector has a specific positional relationship with respect to the projection plane. Therefore, this method is suitable for the above-described method of switching the UI mode based on the positional relationship of the projector with respect to the projection plane. For example, by mainly using the first user input detection system only when it is determined from the positional relationship that the first user input detection system can achieve a sufficient level of touch detection accuracy, the convenience of the method using the probe light It is possible to avoid the deterioration of the user experience due to the deterioration of the touch detection accuracy while making the best use of the characteristics.

Further, according to an embodiment, when the search plane formed in the method using the search light described above is substantially orthogonal to the installation plane of the projector (in this case, if the installation plane is horizontal, the search plane is vertical). , and the projection plane is also vertical), a further determination can be made based on the depth of the projection plane relative to the projector. Then, when the depth of the projection plane is greater than the threshold, it is determined that the first user input detection system does not achieve a sufficient level of touch detection accuracy, and the second user input detection system is used instead of the first user input detection system. can be mainly used. Therefore, for example, in applications where a user views content by projecting the content on a large screen on the wall surface of a room, it is possible to accurately determine whether it is difficult for the user to touch the projection surface. A suitable user interface (eg, gesture UI, voice UI, gaze UI, etc.) can be provided to the user.

Also, according to some embodiments, the projector may selectively run touch-input adaptive software and touch-non-adaptive software depending on the UI mode selected. Therefore, when it is determined that the first user input detection system can achieve a sufficient level of touch detection accuracy, execution of software designed to operate on a mobile device such as a smartphone or tablet PC is recommended. While permissive, alternative software can be automatically executed otherwise.

It should be noted that the various processes described herein may be implemented using software, hardware, or a combination of software and hardware. Programs constituting software are stored in advance in a storage medium (non-transitory media) provided inside or outside the device, for example. Each program, for example, is read into a RAM at the time of execution and executed by a processor such as a CPU.

Also, some of the logical functionality of the devices described herein may be implemented on devices residing within the cloud computing environment instead of being implemented on the device itself. In that case, information exchanged between logical functions may be sent or received via the communication interface illustrated in FIG.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. is naturally within the technical scope of the present disclosure.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification in addition to or instead of the above effects.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(1)
a projection control unit that controls projection of an image onto a projection plane by the projector;
a UI control unit that switches a user interface (UI) mode associated with the image to be projected between two or more UI modes based on the positional relationship of the projector with respect to the projection surface;
A control device comprising:
(2)
The UI control unit is a first UI that uses, as a main input interface, a first user input detection system that detects a touch input to the projection surface when the projection surface is positioned substantially horizontally below the projector. The control device according to (1) above, which selects a mode.
(3)
(1) or (2) above, wherein the projector includes a first user input detection system that detects a user's touch position using exploration light emitted toward an exploration area near the projector. Control device as described.
(4)
The UI control unit uses the first user input detection system as a main input interface when it is determined from the positional relationship that the first user input detection system can achieve a certain level of touch detection accuracy. The control device according to (3) above, which selects the UI mode of
(5)
In the first installation posture of the projector, the investigation area forms an investigation surface substantially parallel to the installation surface of the projector,
The UI control unit determines that the first user input detection system can achieve the level of touch detection accuracy when the projector is installed on the projection surface in the first installation orientation.
The control device according to (4) above.
(6)
The UI control unit according to any one of (3) to (5) above, wherein the UI control unit determines the positional relationship of the projector with respect to the projection plane based on a first measurement result from an orientation sensor of the projector. Control device as described.
(7)
The UI control unit uses the first user input detection system as a main input interface when the first measurement result from the orientation sensor indicates that the search surface formed by the search area is substantially horizontal. The control device according to (6) above, which selects the first UI mode to be used.
(8)
In the second installation posture of the projector, the investigation area forms an investigation plane substantially perpendicular to the installation plane of the projector,
In the case where the projector is installed in the second installation attitude, the UI control unit causes the first user input detection system to achieve the level of touch detection accuracy when the depth of the projection plane is greater than a threshold. determine not to achieve
The control device according to (5) above.
(9)
The control device according to (8), wherein the UI control unit acquires the depth of the projection plane based on a second measurement result from a depth sensor of the projector.
(10)
The projector further comprises a second user input detection system that has rougher touch detection accuracy than the first user input detection system or is incapable of touch detection,
The UI control unit uses the second user input detection system as a main input interface in a second UI mode in which the first user input detection system is not mainly used.
The control device according to any one of (2) to (9).
(11)
The control device according to (10), wherein the second user input detection system detects one or more of a user's gesture, voice, and line of sight as the user input.
(12)
The projector further comprises a camera having an angle of view facing a direction in which the user is assumed to exist when the second UI mode is selected,
The second user input detection system detects user input based on image input from the camera.
The control device according to (11) above.
(13)
The projector projects the image of selectively executed touch-input adaptive software or touch-input non-adaptive software onto the projection surface;
The touch input adaptive software is executed in a first UI mode in which the first user input detection system is mainly used.
The control device according to any one of (2) to (12) above.
(14)
The control device according to (13), wherein the touch input adaptive software includes one or more of touch input adaptive basic software and touch input adaptive application software.
(15)
The control device according to (13) or (14), wherein the touch input non-adaptive software changes the projected image based on input conditions independent of user input.
(16)
16. The control device according to (15), wherein the input condition independent of user input is associated with at least one of date, time, weather, installation location of the projector, and recognized user.
(17)
The UI control unit according to any one of (1) to (16) above, wherein when the projection plane is determined to be substantially horizontal, the UI for rotating the projected image is enabled. Control device.
(18)
The control device according to (17), wherein the UI for rotating the projected image includes one or more of a GUI displayed together with the image and an audio UI.
(19)
In a projection control device that controls projection of an image onto a projection plane by a projector,
switching a user interface (UI) mode associated with the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface;
Control method including.
(20)
the processor of the projection control device,
a projection control unit that controls projection of an image onto a projection plane by the projector;
a UI control unit that switches a user interface (UI) mode associated with the image to be projected between two or more UI modes based on the positional relationship of the projector with respect to the projection plane;
A program to function as

2a, 2b projected images 3a, 3b projection plane 11 projection light aperture 12 search light aperture 13 camera 14 microphone 15 speaker 16 image projection system 17 first user input detection system 18 second user input detection system 21 projected image 21a Projection light 22 Search surface 22a, 22b Search light 23 Search image 100 Projection controller 110 Projector module 120 Recognition module 130 Sensor module 131 Orientation sensor 133 Depth sensor 140 Software module 141 Touch input adaptive software 143 Touch input non-adaptive software 150 Communication Interface 160 integrated control unit

Claims (13)

a projection control unit that controls projection of an image onto a projection plane by the projector;
a UI control unit that switches a user interface (UI) mode associated with the image to be projected between two or more UI modes based on the positional relationship of the projector with respect to the projection surface;
with
The projector detects a user's touch position using search light emitted toward an search area formed in the vicinity of the projected image so as to cover the projected image, which is an search area in the vicinity of the projector. having a first user input detection system;
The UI control unit detects a touch input to the projection surface when the projection surface is positioned substantially horizontally below the projector from a first measurement result from an orientation sensor of the projector. Select the first UI mode that uses the input detection system as the main input interface ,
The projector further comprises a second user input detection system that has rougher touch detection accuracy than the first user input detection system or is incapable of touch detection,
The UI control unit
using the second user input detection system as a main input interface in a second UI mode in which the first user input detection system is not mainly used;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection surface is less than a predetermined value, selecting the first UI mode;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection surface is greater than a predetermined value, selecting the second UI mode . in the first installation posture of the projector, the investigation area forms an investigation surface substantially parallel to the installation surface of the projector;
The UI control unit determines that the first user input detection system can achieve touch detection accuracy equal to or higher than a predetermined threshold when the projector is installed on the projection plane in the first installation orientation. ,
A control device according to claim 1 . in the second installation attitude of the projector, the investigation area forms an investigation plane substantially orthogonal to the installation plane of the projector;
In the case where the projector is installed in the second installation attitude, the UI control unit is configured such that when the depth of the projection plane is greater than a threshold, the first user input detection system detects a depth greater than or equal to the predetermined threshold. determine that the touch detection accuracy is not achieved,
3. A control device according to claim 2 . 2. The control device according to claim 1 , wherein said second user input detection system detects one or more of user's gesture, voice, and gaze as user input. The projector further comprises a camera having an angle of view facing a direction in which the user is assumed to exist when the second UI mode is selected,
The second user input detection system detects user input based on image input from the camera.
5. A control device according to claim 4 . The projector projects the image of selectively executed touch-input adaptive software or touch-input non-adaptive software onto the projection surface;
The touch input adaptive software is executed in a first UI mode in which the first user input detection system is mainly used.
A control device according to claim 1 . 7. The controller of claim 6 , wherein the touch-input adaptive software comprises one or more of touch-input adaptive base software and touch-input adaptive application software. 7. The controller of claim 6 , wherein the touch input non-adaptive software changes the projected image based on input conditions independent of user input. 9. The controller of claim 8 , wherein the input conditions independent of user input relate to at least one of date, time of day, weather, location of the projector, and recognized users. 2. The control device according to claim 1, wherein said UI control unit enables a UI for rotating a projected image when said projection plane is determined to be substantially horizontal. 11. The controller of claim 10 , wherein the UI for rotating the projected image includes one or more of a GUI displayed with the image and an audio UI. In a projection control device that controls projection of an image onto a projection plane by a projector,
switching a user interface (UI) mode associated with the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface;
including
The projector detects a user's touch position using search light emitted toward an search area formed in the vicinity of the projected image so as to cover the projected image, which is an search area in the vicinity of the projector. having a first user input detection system;
A first user input detection system for detecting a touch input to the projection surface when the projection surface is positioned substantially horizontally below the projector from a first measurement result from an orientation sensor of the projector as a main input . Select the first UI mode to use as an interface ,
The projector further comprises a second user input detection system that has rougher touch detection accuracy than the first user input detection system or is incapable of touch detection,
using the second user input detection system as a main input interface in a second UI mode in which the first user input detection system is not mainly used;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection surface is less than a predetermined value, selecting the first UI mode;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection plane is greater than a predetermined value, selecting the second UI mode . the processor of the projection control device,
a projection control unit that controls projection of an image onto a projection plane by the projector;
a UI control unit that switches a user interface (UI) mode associated with the image to be projected between two or more UI modes based on the positional relationship of the projector with respect to the projection surface;
function as
The projector detects a user's touch position using search light emitted toward an search area formed in the vicinity of the projected image so as to cover the projected image, which is an search area in the vicinity of the projector. having a first user input detection system;
The UI control unit detects a touch input to the projection surface when the projection surface is positioned substantially horizontally below the projector from a first measurement result from an orientation sensor of the projector. Select the first UI mode that uses the input detection system as the main input interface ,
The projector further comprises a second user input detection system that has rougher touch detection accuracy than the first user input detection system or is incapable of touch detection,
The UI control unit
using the second user input detection system as a main input interface in a second UI mode in which the first user input detection system is not mainly used;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection surface is less than a predetermined value, selecting the first UI mode;
From the first measurement result from the orientation sensor of the projector and the second measurement result from the depth sensor of the projector, when the projection surface is positioned substantially vertically below the projector and the projector to the projection plane is greater than a predetermined value, the second UI mode is selected .

Images