JP7633017B2 - Aerial control device - Google Patents

Description

This disclosure relates to an airborne operation device that allows operations to be performed on a display surface that is displayed as a virtual image in the air.

JP 2019-109637 A describes a non-contact input device that includes a device housing, a display arranged inside the device housing, an optical plate that forms a real image of the display in the air outside the device housing, and a non-contact position detection unit. The non-contact position detection unit includes an invisible light emitting element and a light receiving element that detects reflected light, and detects light emitted from the invisible light emitting element and reflected on the detection surface with the light receiving element. The detection surface is arranged approximately parallel to the image plane of the aerial image formed by the optical plate.

The non-contact input device has a first detection surface and a second detection surface located further back than the first detection surface. A finger is placed on each of the first detection surface and the second detection surface as a pointing object. When the finger touches the first detection surface, the XY position coordinates of the finger pointing are displayed on the aerial image as a pointer. This allows the user to recognize where on the aerial image the finger is pointing.

When a finger touches the second detection surface, the point at which the finger touches the second detection surface is regarded as the time of input, and the indicated position is then corrected based on the position coordinates detected on the first detection surface. In other words, the position of the finger thereafter is corrected based on the point at which the finger touches the second detection surface, allowing the user to perform input operations without worrying about misalignment of the finger.

JP 2019-109637 A

As mentioned above, when a detection plane is set approximately parallel to the image plane of the aerial image and an object such as a finger is detected on the detection plane, erroneous detection of the object can be a problem. When multiple buttons are displayed as multiple detection planes, a malfunction can occur when a detection plane corresponding to a button other than the one pressed detects an object. In other words, a button other than the one pressed may react. Furthermore, when a display surface with buttons is displayed as a virtual image in the air, there is a possibility that an unintended operation will occur if something other than a finger, such as the user's clothing, touches the button. In this way, there is a concern that a malfunction will occur or an unintended operation will be performed.

The present disclosure aims to provide an aerial operation device that can suppress malfunctions.

An aerial manipulation device according to one aspect of the present disclosure includes a display device that displays information by emitting light, an aerial imaging device that reflects the light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button, and includes a recognition unit that recognizes whether or not the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal corresponding to the button pressed by the user of the aerial manipulation device to an external device of the aerial manipulation device and controls the operation of the device, and when the recognition unit recognizes that the object detected by the sensor is at the position of a button, it disables the multiple buttons to suppress the output of a control signal by the signal output unit, and when it recognizes that the object detected by the sensor is moving away from the button, it enables the button to allow the output of a control signal by the signal output unit.

In this aerial manipulation device, an aerial imaging device displays a display surface with multiple buttons as a virtual image in the air, and a sensor detects the position of an object, such as a finger, approaching a button. The recognition unit recognizes whether a button has been pressed based on the position of the object detected by the sensor. The signal output unit outputs a control signal corresponding to the pressed button to an external device of the aerial manipulation device. The recognition unit disables the multiple buttons when the object is at the position of a button. Therefore, by disabling the multiple buttons at the timing when a button is pressed, it is possible to prevent an unintended button from reacting and causing an unintended operation. In addition, when the object leaves the button, the recognition unit enables the button to allow the signal output unit to output a control signal. Therefore, by enabling the button at the timing when the object leaves the button, only the intended button can be reacted to realize the intended operation. This makes it possible to suppress malfunctions and improve the operability of the buttons.

The multiple buttons may form a group including at least one button, and the recognition unit may disable all buttons in the group when it recognizes that an object detected by the sensor is at the position of a button in the group. In this case, the recognition unit disables all buttons in the group when an object is at the position of a button belonging to the group. By disabling all buttons in the group when a button in the group is pressed, it is possible to prevent other buttons in the group from reacting and causing an unintended operation.

An aerial manipulation device according to another aspect of the present disclosure includes a display device that displays information by emitting light, an aerial imaging device that reflects the light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button. The aerial manipulation device includes a recognition unit that recognizes whether the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal to an external device of the aerial manipulation device to control the operation of the device. The sensor detects the position of the object multiple times, and the signal output unit outputs a control signal to the device only when the sensor detects multiple times that the object is located between a first surface and a second surface that sandwich the display surface.

In this aerial manipulation device, the signal output unit outputs a control signal to the device when the sensor detects multiple times that an object is located between the first and second surfaces that sandwich the display surface on which the image is formed by the aerial imaging device. In other words, the signal output unit outputs a control signal to the device and the device operates only when the sensor detects multiple times that an object is located between the first and second surfaces. Therefore, if the sensor does not detect multiple times that an object is located between the first and second surfaces, the signal output unit does not output a control signal, thereby suppressing malfunction of the device.

The above-mentioned aerial manipulation device includes a notification unit that notifies a user of the aerial manipulation device of the recognition unit's recognition, and the notification unit may notify the user that a button has been pressed when the sensor detects multiple times that an object is located between the first and second surfaces that sandwich the display surface. In this case, the user is notified that the button has been pressed when the sensor detects multiple times that an object is located between the first and second surfaces. Therefore, the user can be notified that the button has been pressed correctly, thereby improving the operability of the button.

The sensor may measure the distance from the sensor to the object multiple times, and the signal output unit may output a control signal to the device only when the average of the measured distances from the sensor to the object is equal to or greater than the distance from the sensor to the second surface and equal to or less than the distance from the sensor to the first surface. In this case, the signal output unit outputs a control signal to the device only when the average of the measured distances is equal to or greater than the distance from the sensor to the second surface and equal to or less than the distance from the sensor to the first surface. Therefore, when the average distance is shorter than the distance from the sensor to the second surface or longer than the distance from the sensor to the first surface, the signal output unit does not output a control signal. Therefore, false detection of the object can be suppressed and malfunctions can be more reliably suppressed.

The above-mentioned aerial manipulation device includes multiple sensors, and the signal output unit does not need to output a control signal to the equipment when the multiple sensors detect an object. In this case, the output of the control signal is suppressed when the multiple sensors detect an object, so that erroneous detection and malfunction can be more reliably suppressed.

An aerial manipulation device according to another aspect of the present disclosure includes a display device that displays information by emitting light, an aerial imaging device that reflects the light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button, and includes a recognition unit that recognizes whether the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal to an external device of the aerial manipulation device to control the operation of the device, the sensor detects the position of the object multiple times, and the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the position of the object detected by the sensor is a first surface and then recognizes that the position of the object detected by the sensor is a second surface that is closer to the sensor than the first surface.

In this aerial manipulation device, the recognition unit recognizes that the position of the object is on the first surface, and then the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the position of the object is on the second surface, which is closer to the sensor than the first surface. In other words, a control signal is output to the device only when it is recognized that the object has moved from the first surface to the second surface, which is closer to the sensor. Therefore, a control signal is not output to the device unless the object moves from the first surface to the second surface, making it possible to suppress erroneous detection of the object and malfunction of the device.

The above-mentioned aerial manipulation device may be equipped with multiple sensors, and the distance from the first surface to the second surface may be half the distance between two adjacent sensors. In this case, a control signal to the device is output only when the object moves half the distance between the two sensors from the first surface and reaches the second surface. Therefore, the output of the control signal to the device can be limited, thereby suppressing malfunctions.

The above-mentioned aerial operation device includes a dummy sensor that detects the position of an object approaching a position adjacent to the button on the display surface, and the signal output unit does not need to output a control signal to the device when the position of the object is detected by the dummy sensor. In this case, the output of a control signal to the device is restricted when the dummy sensor detects an object adjacent to the button. Therefore, erroneous detection of an object and malfunction of the device can be more reliably suppressed.

This disclosure makes it possible to suppress malfunctions.

FIG. 1 is a perspective view showing an application example of an aerial operation device according to an embodiment. FIG. 1 is a perspective view showing a state in which the aerial operating device according to the embodiment displays a display surface with buttons. FIG. 2 is a vertical cross-sectional view of the aerial operation device according to the embodiment. FIG. 2 is a block diagram showing the functions of the aerial operation device according to the embodiment. 3A to 3C are diagrams illustrating detection of an object by a sensor according to an embodiment. 4 is a block diagram showing an example of a relationship between a button and a recognition unit according to the embodiment; FIG. 10 is a flowchart showing steps of a button recognition method in the aerial operating device according to the embodiment. FIG. 13 is a perspective view showing an aerial operation device according to a modified example. FIG. 11 is a vertical cross-sectional view showing a modified embodiment of the aerial operation device.

Various embodiments of the aerial manipulation device according to the present disclosure will be described below with reference to the drawings. In the description of the drawings, the same or corresponding elements are given the same reference numerals, and duplicate descriptions will be omitted as appropriate. In addition, the drawings may be partially simplified or exaggerated to facilitate understanding, and the dimensional ratios and angles, etc. are not limited to those shown in the drawings.

Figure 1 is a perspective view showing an example of an aerial manipulating device 1 according to this embodiment. Figure 2 is a perspective view showing a state in which the aerial manipulating device 1 displays a display surface 10 with buttons 11 in the air. For example, the aerial manipulating device 1 according to this embodiment is an operation input device for operating equipment.

As an example, the aerial operation device 1 is provided in a toilet T, and the buttons 11 on the display surface 10 are switches that operate various parts of the toilet T. The display surface 10 with the buttons 11 is a floating image that is displayed as if it is floating in the air. In other words, the display surface 10 with the buttons 11 is displayed as a virtual image in the air. Therefore, a user of the aerial operation device 1 can operate various parts of the toilet T in a hygienic manner, as they do not need to directly touch buttons, etc.

For example, toilet T includes a toilet bowl B and a wall W adjacent to toilet bowl B. For example, wall W is provided on the right side as viewed from a user seated on toilet bowl B. An aerial operation device 1 and toilet paper E are fixed to the exemplary wall W, and a washbasin M may also be provided. Toilet bowl B includes, for example, a Washlet (registered trademark), and the Washlet of toilet T can be operated from a button 11 on a display surface 10. For example, display surface 10 includes a plurality of buttons 11, and a user can operate each part of the Washlet by operating the plurality of buttons 11.

The aerial operation device 1 is fixed to, for example, a wall W. The aerial operation device 1 may be embedded in the wall W, or may be detachable from the wall W. The exemplary aerial operation device 1 includes a housing 2 and a protrusion 3 protruding from the bottom of the housing 2. A plurality of sensors 12 are embedded in the protrusion 3 to detect an object F approaching the button 11.

In this disclosure, the term "object" refers to an object that operates a device by operating a button 11, for example, a user's finger. The sensors 12 are provided in correspondence with the buttons 11, and each button 11 is displayed diagonally above the corresponding sensor 12. As an example, the number of buttons 11 and sensors 12 is seven.

The display surface 10 is displayed so as to be oriented diagonally upward with respect to both the up-down direction D1 of the aerial operation device 1 and the protruding direction D3 from the wall W. This makes it easier for the user to view the display surface 10. For example, the display surface 10 is displayed glowing, so the display surface 10 can be easily viewed regardless of the brightness of the surroundings. The display surface 10 has a shape that extends long in the left-right direction D2, for example.

As an example, the display surface 10 is displayed in a rectangular shape, and the vertical length L1 of the display surface 10 is shorter than the horizontal length L2 of the display surface 10. If the aspect ratio of the length L1 to the length L2 is 1:X (X is a real number), then, for example, the value of X is 2 or more and 10 or less. The lower limit of the value of X may be 3, and the upper limit of the value of X may be 5, 6, 7, 8, or 9.

As an example, the value of X above may be 4, and the aspect ratio of length L1 to length L2 may be 1:4. The button 11 may be, for example, rectangular. As an example, the button 11 may be square. The length of one side of the button 11 is, for example, 18 mm or more and 24 mm or less. However, the shape and size of the button 11 may be changed as appropriate.

The display surface 10 may display something other than the multiple buttons 11. For example, the display surface 10 may include the multiple buttons 11 and text information 13. As an example, the text information 13 may include at least one of date information, time information, temperature information, temperature information, and air pressure information.

On the display surface 10, for example, a plurality of buttons 11 may be displayed lined up horizontally on the lower side of the display surface 10, and text information 13 may be displayed above the plurality of buttons 11. The buttons 11 may be arranged such that their centers 11b are positioned at a position offset from the center 10b of the display surface 10. In the present embodiment, as an example, each of the plurality of buttons 11 is displayed at a position offset from a reference line V that passes through the center 10b and extends in the longitudinal direction (horizontal direction) of the display surface 10.

Figure 3 is a cross-sectional view showing a schematic internal structure of the aerial operation device 1. As shown in Figures 2 and 3, for example, the housing 2 has a horizontally elongated box shape. As an example, the housing 2 has a front surface 2b extending in the up-down direction D1 and the left-right direction D2 when attached to the wall W, a pair of left and right side surfaces 2c extending in the up-down direction D1 and in the protruding direction D3 from the wall W, a top surface 2d extending in the left-right direction D2 and the protruding direction D3, and a bottom surface 2f facing the opposite side to the top surface 2d.

The protrusion 3 has, for example, a rectangular plate shape. As an example, the protrusion 3 has a main surface 3b that protrudes from the underside of the front surface 2b and faces upward, an end surface 3c that faces the left-right direction D2 or the protruding direction D3, and a back surface 3d that faces the opposite side to the main surface 3b. As described above, the protrusion 3 has a built-in sensor 12. This makes it easier for the sensor 12 to detect the object F.

The aerial operation device 1 includes an aerial imaging device 14 that displays a display surface 10 with buttons 11, a display device 15 arranged inside the housing 2, and a control unit 20. The aerial imaging device 14 and the display device 15 correspond to a virtual image display unit that displays the display surface 10 in the air as a virtual image. The aerial imaging device 14 is, for example, a retroreflective member (retroreflector) fixed to an opening 2g located inside the front surface 2b of the housing 2.

The opening 2g is defined, for example, by a pair of first plate portions 2h constituting the front surface 2b of the housing 2, and a pair of second plate portions 2j fixed to the inside of the housing 2 of the first plate portions 2h. As an example, the first plate portion 2h and the second plate portion 2j are provided as an upper and lower pair. The aerial imaging device 14 is fixed to the housing 2 while being sandwiched between the first plate portion 2h and the second plate portion 2j.

The aerial imaging device 14 is, for example, rectangular in shape with long sides extending in the left-right direction D2 and short sides extending in the up-down direction D1. One long side and the other long side of the aerial imaging device 14 are sandwiched between the first plate portion 2h and the second plate portion 2j. The display device 15 is disposed obliquely with respect to the aerial imaging device 14.

As an example, the display device 15 is a liquid crystal display (LCD). The display device 15 is disposed diagonally above the aerial imaging device 14 inside the housing 2. The display device 15 has a screen for displaying an image. The screen of the display device 15, for example, irradiates light C as an image diagonally downward toward the aerial imaging device 14. The aerial imaging device 14 reflects the light C from the display device 15 multiple times (for example, twice) inside the aerial imaging device 14, and images the display surface 10 in the space in front of the aerial imaging device 14 as seen by the user of the aerial operation device 1.

The sensor 12 may be exposed on the main surface 3b of the protrusion 3, for example. As an example, the sensor 12 is a depth sensor. For example, the sensor 12 is provided on a virtual straight line extending from the button 11, i.e., in front of the button 11, which is a virtual image. The sensor 12 acquires distance image data including information on the position (two-dimensional position) of the object F on a plane perpendicular to the virtual straight line and information on the distance K from the sensor 12 to the object F. The sensor 12 acquires, for example, distance image data. The sensor 12 outputs the acquired distance image data to the control unit 20 at a predetermined period (for example, 1/30 seconds).

As a specific example, the sensor 12 irradiates light rays (or infrared rays) to each point on an object present in a shooting area including the target object F, and receives the light rays reflected from each point on the object. The sensor 12 then measures the distance between the sensor 12 and each point on the object based on the received light rays, and outputs the measured distance for each pixel. The distance between the sensor 12 and each point on the object may be measured, for example, by a Light Coding method.

In the Light Coding method, the sensor 12 irradiates light rays in a random dot pattern to each point on an object present in a shooting area including the target object F. The sensor 12 then receives the light rays reflected from each point on the object and measures the distance between the sensor 12 and each point on the object by detecting distortion in the pattern of the reflected light rays. The sensor 12 detects information on the two-dimensional position of each point on the object and information on the distance from the sensor 12 to each point on the object as multiple pixels, and outputs the detected multiple pixels to the control unit 20.

The control unit 20 is capable of communicating with each of the sensor 12 and the display device 15. The control unit 20 includes, for example, a CPU (Central Processing Unit) that executes a program, a storage unit including a ROM (Read Only Memory) and a RAM (Random Access Memory), an input/output unit, and a driver. Each function of the control unit 20 is realized by operating the input/output unit under the control of the CPU and reading or writing data in the storage unit. There are no particular limitations on the form and location of the control unit 20.

Figure 4 is a functional block diagram of the control unit 20. As shown in Figures 3 and 4, the control unit 20 has, as functional components, an image output unit 21, an object detection unit 22, a recognition unit 23, a signal output unit 24, and a notification unit 25. The image output unit 21 outputs a control signal for image data of an image to be displayed on the display device 15 to the display device 15. The display device 15 is capable of displaying various types of images based on the control signal for the image data from the image output unit 21.

The object detection unit 22 detects the object F based on the range image data output from the sensor 12. When the object detection unit 22 detects the object F, it outputs position data indicating the position of the object F to the recognition unit 23. The recognition unit 23 recognizes that the button 11 has been pressed by the object F based on the position data output from the object detection unit 22.

The notification unit 25 receives an operation signal from the recognition unit 23 when the recognition unit 23 recognizes that the button 11 has been pressed. The notification unit 25 is a notification means that notifies the user of the aerial operation device 1 that the button 11 has been operated when the recognition unit 23 recognizes that the button 11 has been operated. The notification unit 25 has, for example, an audio output unit 25b and a color change unit 25c.

As an example, the audio output unit 25b is a speaker, and outputs audio when an operation signal is received from the recognition unit 23. When the user hears the audio from the audio output unit 25b, the user can understand that the button 11 has been operated. The color change unit 25c, for example, generates a color change signal when an operation signal is received from the recognition unit 23, and outputs the color change signal to the display device 15.

When the display device 15 receives a color change signal from the color change unit 25c, it changes the color of the button 11 pressed by the user, for example. When the user visually confirms that the color of the button 11 has been changed, the user can understand that the button 11 has been operated. Note that the display device 15 may change the color of a part other than the button 11 when it receives a color change signal from the color change unit 25c.

By providing the above-described notification unit 25, the user can know that the button 11 has been operated. However, in the notification unit 25, at least one of the audio output unit 25b and the color change unit 25c may be omitted. In addition, the notification unit 25 may notify the user that the button 11 has been operated in a manner other than audio output by the audio output unit 25b or color change by the color change unit 25c.

When the recognition unit 23 recognizes that the button 11 has been pressed, the signal output unit 24 generates a control signal based on the pressing operation of the button 11. The signal output unit 24 transmits the generated control signal to a device 30 external to the aerial operation device 1, and operates the device 30 by transmitting the control signal to the device 30.

The device 30 may be, for example, a toilet B with a washlet (see FIG. 1), any of the components of the washlet, or a washbasin M. As an example, the device 30 may be a toilet B, which may operate to flush the toilet when it receives a control signal from the signal output unit 24. In this way, with the exemplary aerial operation device 1, it is possible to operate each component of the toilet T by operating the button 11 displayed as a virtual image. Therefore, since there is no need to directly (physically) press a button or the like to operate each component of the toilet T, operation can be achieved in a hygienic manner.

For example, the recognition unit 23 may disable the button 11 to suppress output of a control signal by the signal output unit 24 when it recognizes that the object F detected by the sensor 12 is at the position of the button 11. In addition, the recognition unit 23 may enable the button 11 to allow output of a control signal by the signal output unit 24 when it recognizes that the object F detected by the sensor 12 is moving away from the button 11.

The recognition unit 23 may also determine whether the distance K between the sensor 12 and the object F is equal to or less than a threshold Y1, and whether the distance K between the sensor 12 and the object F is equal to or less than a threshold Y2 that is smaller than the threshold Y1. In this case, the recognition unit 23 determines whether the object F has reached the first surface Z1, which is a pressing determination surface, and whose distance K from the sensor 12 is the threshold Y1, and whether the object F has reached the second surface Z2, which is a pressing determination surface, and whose distance K from the sensor 12 is the threshold Y2.

The first surface Z1 and the second surface Z2 are virtual surfaces formed at a location at a constant distance from the sensor 12. As shown in FIG. 5, the first surface Z1 and the second surface Z2 are, for example, circular. The first surface Z1 and the second surface Z2 are, for example, provided in a position close to the button 11. At least one of the first surface Z1 and the second surface Z2 may coincide with the position of the button 11, or both the first surface Z1 and the second surface Z2 may not coincide with the position of the button 11.

The distance from the first surface Z1 to the second surface Z2 is, for example, half the distance between two adjacent sensors 12. Note that FIG. 3 shows an example in which the first surface Z1 is provided closer to the user than the button 11, and the second surface Z2 is provided closer to the sensor 12 than the button 11.

As described above, the sensor 12 detects the position of the object F multiple times. As an example, the signal output unit 24 outputs a control signal to the device 30 only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 that sandwich the display surface 10. The sensor 12 also measures the distance K from the sensor 12 to the object F multiple times. For example, the notification unit 25 notifies the user of the aerial operation device 1 that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 that sandwich the display surface 10.

Then, the signal output unit 24 outputs a control signal to the device 30 only when the average value of the measured distance K from the sensor 12 to the object F is equal to or greater than the distance from the sensor 12 to the second surface Z2 (threshold Y2) and equal to or less than the distance from the sensor 12 to the first surface Z1 (threshold Y1). As a specific example, when the sensor 12 measures the distance K n times (n is a natural number), the signal output unit 24 outputs a control signal to the device 30 when the average value of the distance K is equal to or greater than the threshold Y1 and equal to or less than the threshold Y2. The value of n above is, for example, 3 or more and 10 or less (5 as an example).

The signal output unit 24 may output a control signal to the device 30 only when the recognition unit 23 recognizes that the position of the object F detected by the sensor 12 is the first surface Z1, and then the recognition unit 23 recognizes that the position of the object F detected by the sensor 12 is the second surface Z2, which is closer to the sensor 12 than the first surface Z1.

The signal output unit 24 may be configured not to output a control signal to the device 30 when the multiple sensors 12 simultaneously detect an object F. Furthermore, at least one of the multiple sensors 12 may be a dummy sensor, and the signal output unit 24 may be configured not to output a control signal to the device 30 when the position of the object F is detected by the dummy sensor. The dummy sensor detects, for example, the position of the object F approaching a position adjacent to the button 11 on the display surface 10. As described above, by configuring the signal output unit 24 not to output a control signal under a specified condition, malfunction of the device 30 can be suppressed.

FIG. 6 is a block diagram showing a schematic example of the relationship between multiple buttons 11 and the recognition unit 23. As shown in FIG. 6, multiple buttons 11 form a group including at least one button 11. The number of groups may be one or more. A group may be made up of multiple buttons 11 that are in a touchable relationship on the same line. In the example of FIG. 6, group G1 including buttons 11A and 11B, group G2 including buttons 11C, 11D, and 11E, and group GX (X is a natural number equal to or greater than 3) including button 11N are formed.

Figure 7 is a flowchart of a recognition method in which the recognition unit 23 recognizes the operation of the buttons 11 grouped as described above. As shown in Figure 7, first, the recognition unit 23 recognizes the group of touched buttons 11. For example, when the object F detected by the sensor 12 is at the position of button 11A (or button 11B), the recognition unit 23 recognizes the area of group G1. At this time, for example, the recognition unit 23 recognizes the area on the same touch line in group G1 (step S1).

When the recognition unit 23 recognizes that the object F detected by the sensor 12 is at the position of button 11A (or button 11B) of group G1, it invalidates all buttons 11 (button 11A and button 11B) in group G1. That is, it invalidates all buttons in the aforementioned area (step S2). Then, the recognition unit 23 sets all buttons (button 11A and button 11B) in group G1 (e.g., the aforementioned area) as possible candidates for activation (step S3).

For example, the user may first try to press button 11A, but may realize that he or she actually wanted to press button 11B. At that time, the user may not remove the object F, such as a finger, from the display surface 10, but may move the object F to button 11B while keeping the object F on the display surface 10, and may want to operate the button 11B when the object F is removed from the button 11B. At this time, the recognition unit 23 determines whether the object F detected by the sensor 12 is moving away from the button 11B, i.e., whether the position of the object F is at the touch release button coordinates (step S4). When the recognition unit 23 recognizes that the object F detected by the sensor 12 is moving away from the button 11B, i.e., when the position of the object F is at the touch release button coordinates, the recognition unit 23 enables the button 11B to allow the signal output unit 24 to output a control signal (step S5), and ends the series of steps. On the other hand, when the recognition unit 23 does not recognize that the object F detected by the sensor 12 is leaving the button 11B, that is, when the position of the object F is not at the touch-release button coordinates, the series of steps ends.

Next, the effects obtained from the aerial manipulation device 1 according to this embodiment will be described. In the aerial manipulation device 1, the aerial imaging device 14 displays a display surface 10 with multiple buttons 11 as a virtual image in the air, and the sensor 12 detects the position of an object F, such as a finger, approaching the button 11. The recognition unit 23 recognizes whether the button 11 has been pressed based on the position of the object F detected by the sensor 12. The signal output unit 24 outputs a control signal corresponding to the pressed button 11 to an external device 30 of the aerial manipulation device 1. The recognition unit 23 disables the operation of the multiple buttons 11 when the object F is at the position of the button 11. Therefore, by disabling the multiple buttons 11 at the timing when the button 11 is pressed, it is possible to prevent an unintended action from occurring due to an unintended button 11 reacting.

In addition, when the object F leaves the button 11, the recognition unit 23 enables the button 11 to allow the signal output unit 24 to output a control signal. Therefore, by enabling the button 11 at the timing when the object F leaves the button 11, it is possible to make only the intended button 11 react and realize the intended operation. This makes it possible to suppress erroneous operations and improve the operability of the button 11.

The multiple buttons 11 form groups G1, G2, and GX each including at least one button 11, and the recognition unit 23 may disable the operation of all the buttons 11 in the groups G1, G2, and GX when it recognizes that the object F detected by the sensor 12 is at the position of a button 11 in the groups G1, G2, and GX. For example, the recognition unit 23 disables all the buttons 11 (buttons 11A and 11B) in the group G1 when the object F is at the position of the button 11A belonging to the group G1. By disabling all the buttons 11 belonging to the group G1 when a button in the group G1 is pressed, it is possible to prevent other buttons 11 in the group G1 from reacting and causing an unintended operation. As a specific example, it is possible to avoid the user feeling that the button 11B reacted when they intended to press the button 11A.

3, in the aerial operation device 1, the signal output unit 24 outputs a control signal to the device 30 only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 that sandwich the display surface 10 on which the image is formed by the aerial imaging device 14. That is, only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2, the signal output unit 24 outputs a control signal to the device 30 and the device 30 operates. Therefore, if the sensor 12 does not detect the object F located between the first surface Z1 and the second surface Z2 multiple times, the signal output unit 24 does not output a control signal, so that erroneous detection of the object F can be suppressed. As a result, erroneous operation of the device 30 can be suppressed.

The aerial operation device 1 includes a notification unit 25 that notifies the user of the aerial operation device 1 that the recognition unit 23 has recognized something, and the notification unit 25 may notify the user that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 that sandwich the display surface 10. In this case, the user is notified that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2. Therefore, since the user can be notified that the button 11 has been pressed correctly, the operability of the button 11 can be improved.

The sensor 12 may measure the distance K from the sensor 12 to the object F multiple times, and the signal output unit 24 may output a control signal to the device 30 only when the average value of the measured distance K from the sensor 12 to the object F is equal to or greater than the distance from the sensor 12 to the second surface Z2 and equal to or less than the distance from the sensor 12 to the first surface Z1. In this case, the signal output unit 24 outputs a control signal to the device 30 only when the average value of the measured distance K is equal to or greater than the distance from the sensor 12 to the second surface Z2 and equal to or less than the distance from the sensor 12 to the first surface Z1. Therefore, when the average value of the distance K is shorter than the distance from the sensor 12 to the second surface Z2 or longer than the distance from the sensor 12 to the first surface Z1, the signal output unit 24 does not output a control signal. Therefore, it is possible to suppress erroneous detection of the object F and more reliably suppress malfunctions.

The aerial operation device 1 includes multiple sensors 12, and the signal output unit 24 does not need to output a control signal to the equipment 30 when the multiple sensors 12 detect an object F. In this case, the output of a control signal is suppressed when the multiple sensors 12 detect an object F, so that erroneous detection and malfunction can be more reliably suppressed.

In the aerial operation device 1, the signal output unit 24 may output a control signal to the device 30 only when the recognition unit 23 recognizes that the position of the object F is on the first surface Z1 and then recognizes that the position of the object F is on the second surface Z2 closer to the sensor 12 than the first surface Z1. In this case, a control signal is output to the device 30 only when it is recognized that the object F has moved from the first surface Z1 to the second surface Z2 closer to the sensor 12. Therefore, since a control signal is not output to the device 30 except when the object F moves from the first surface Z1 to the second surface Z2, erroneous detection of the object F and malfunction of the device can be suppressed. As a specific example, erroneous detection of the button 11 can be suppressed when the clothes of a user using the toilet T pass over the display surface 10, when the hand of a user trying to pick up toilet paper E passes over the display surface 10, when the hand passes over the display surface 10 when cleaning the toilet T, and when a small insect passes over the display surface 10.

The aerial operation device 1 may include a plurality of sensors 12, and the distance from the first surface Z1 to the second surface Z2 may be half the distance between two adjacent sensors 12. In this case, a control signal to the device 30 is output only when the object F moves half the distance between the two sensors 12 from the first surface Z1 to reach the second surface Z2. Therefore, the output of the control signal to the device 30 can be limited, thereby suppressing malfunctions.

The aerial operation device 1 is equipped with a dummy sensor that detects the position of an object F approaching a position adjacent to the button 11 on the display surface 10, and the signal output unit 24 does not need to output a control signal to the device 30 when the position of the object F is detected by the dummy sensor. In this case, the output of a control signal to the device 30 is restricted when the dummy sensor detects an object F adjacent to the button 11. Therefore, erroneous detection of the object F and malfunction of the device 30 can be more reliably suppressed.

The above describes an embodiment of the aerial manipulation device according to the present disclosure. However, the aerial manipulation device according to the present disclosure is not limited to the above-described embodiment, and may be modified or applied to other things without changing the gist of the claims. In other words, the configuration, shape, size, number, material, and arrangement of each part of the aerial manipulation device may be modified as appropriate without changing the gist of the above-described claims.

For example, in the above embodiment, the sensor 12 measures the distance between the sensor 12 and each point on the object by the Light Coding method, but the method is not limited to this method. For example, the sensor 12 may measure the distance between the sensor 12 and each point on the object by the TOF (Time Of Flight) method. In the TOF method, the sensor 12 calculates the flight time (delay time) of a light ray until it reflects at each point on the object and reaches the sensor 12, and measures the distance between the sensor 12 and each point on the object from the calculated flight time and the speed of light. Even in this form, the same effect as in the above embodiment is achieved. Furthermore, the type of the sensor is not limited to a depth sensor. In other words, instead of the sensor 12 which is a depth sensor, an infrared sensor or an ultrasonic sensor, etc. may be provided, and the type of the sensor can be changed as appropriate.

In the above embodiment, an example was described in which the aerial imaging device 14 and the display device 15 display a display surface 10 in which multiple buttons 11 are arranged horizontally at the bottom and text information 13 is displayed above the buttons 11. However, the layout of the buttons and the like on the display surface is not limited to the above example and can be changed as appropriate.

Next, an aerial manipulation device 41 according to a modified example will be described with reference to Figs. 8 and 9. Fig. 8 is a perspective view showing an aerial manipulation device 41 according to a modified example. Fig. 9 is a vertical cross-sectional view showing a schematic internal structure of the aerial manipulation device 41. As shown in Figs. 8 and 9, the aerial manipulation device 41 displays a display surface 50 having a plurality of buttons 51, which are a numeric keypad, as a virtual image in the air. The aerial manipulation device 41 has a columnar shape. The aerial manipulation device 41 extends in a first direction A1, a second direction A2, and a third direction A3.

The first direction A1 corresponds to the depth direction (front-to-back direction) of the aerial manipulating device 41, the second direction A2 corresponds to the width direction of the aerial manipulating device 41, and the third direction A3 corresponds to the height direction of the aerial manipulating device 41. The length (height) of the third direction A3 of the aerial manipulating device 41 is longer than the length (depth) of the first direction A1 of the aerial manipulating device 41 and the length (width) of the second direction A2 of the aerial manipulating device 41. This makes it possible to further improve the visibility of the display surface 50 to the user.

The aerial operation device 41 includes a housing 42 that constitutes the outer surface of the aerial operation device 41. The housing 42 has, for example, a top surface 42b facing vertically upward, a pair of side surfaces 42c aligned along the second direction A2, a front surface 42d facing one side of the first direction A1, a back surface 42f facing the other side of the first direction A1, and a bottom surface 42g facing vertically downward. Furthermore, the housing 42 has an inclined surface 42h located above the front surface 42d.

A rectangular opening is formed on the inclined surface 42h, through which the aerial imaging device 54 is exposed. The inclined surface 42h is oriented, for example, diagonally upward. The display surface 50 imaged by the aerial imaging device 54 is displayed at a position closer to the user than the inclined surface 42h, thereby improving the visibility of the display surface 50.

The buttons 51 on the display surface 50 are buttons for operating an automated teller machine (ATM). The display surface 50 with the buttons 51 is a floating image that appears to float in the air. In other words, the display surface 50 with the buttons 51 is displayed as a virtual image in the air. Therefore, a user of the aerial operation device 41 can operate the automated teller machine in a hygienic manner since he or she does not need to directly touch a touch panel or the like.

For example, the display surface 50 has a number of buttons 51, and a user can operate an automated teller machine by operating the number of buttons 51. A sensor 43 is embedded in the housing 42, for example, to detect an object F approaching the button 51. The configuration of the sensor 43 is similar to the configuration of the sensor 12 described above.

The aerial operation device 41 includes, for example, a plurality of sensors 43. The sensors 43 are provided corresponding to, for example, the buttons 51, and each button 51 is displayed diagonally above each sensor 43. As an example, the number of buttons 51 and sensors 43 is 12. The display surface 50 is displayed at a position closer to the user than the aerial imaging device 54, and is displayed diagonally above the aerial imaging device 54. This makes it easier for the user to view the display surface 50.

For example, the display surface 50 may be illuminated, in which case the display surface 50 can be easily viewed regardless of the brightness of the surroundings. The display surface 50 has, for example, a shape that is elongated vertically. As an example, the display surface 50 is displayed in a rectangular shape, and the vertical length of the display surface 50 is longer than the horizontal length of the display surface 50. However, the shape and size of the display surface 50 are not limited to the above example, and are not particularly limited. The display surface 50 may display something other than the button 51. For example, the display surface 50 may include multiple buttons 51 and text information. As an example, the text information may include at least one of date information and time information.

On the display surface 50, for example, a plurality of buttons 51 are displayed in a grid arrangement. As one example, the plurality of buttons 51 are displayed as a numeric keypad. In this case, by pressing the button 51 of the aerial operation device 41, it is possible to specify the amount to be debited, the amount to be deposited, the amount to be transferred, or a PIN number for the automated teller machine.

The shape of the button 51 is, for example, a square shape (one example is a rectangle with rounded corners). However, the shape of the button 51 may be a circle, an oval, or a polygon, and is not particularly limited. For example, the button 51 may include at least one of a confirm button and a back button. In this case, the confirm button corresponds to a button that confirms an operation, and the back button corresponds to a button that goes back to the state before the operation. However, the type of the button 51 is not limited to the above examples, and is not particularly limited.

The aerial operation device 41 includes an aerial imaging device 54 that displays a display surface 50 with buttons 51, a display device 55 arranged inside the housing 42, and a control unit 60. The aerial imaging device 54 and the display device 55 correspond to a virtual image display unit that displays the display surface 50 as a virtual image in the air. The aerial imaging device 54 is, for example, a retroreflective member (retroreflective mirror) fixed to an opening formed in the inclined surface 42h of the housing 42. The display device 55 is arranged obliquely with respect to the aerial imaging device 54. Note that the configurations of the aerial imaging device 54, the display device 55, and the control unit 60 are the same as the configurations of the aerial imaging device 14, the display device 15, and the control unit 20 described above.

The above describes a modified aerial operation device 41 that is installed in an automated teller machine, but the aerial operation device 41 also provides the same effects as the aerial operation device 1. The operation input device according to the present disclosure can be further modified with respect to the modified aerial operation device 41. For example, in the above-mentioned modified example, the aerial operation device 41 that is installed in an automated teller machine was described. However, the aerial operation device according to the present disclosure may also be installed in a kitchen, a bathroom, a hospital, a public institution, a station, an airport, or the like.

For example, in an aerial operation device installed in a kitchen, a display surface with buttons is displayed as a virtual image in the air, allowing a cooking app to be operated without directly touching the display with fingers or the like that have ingredients on them. In other words, in the past, in order to touch the display of a mobile device or the like, a user had to take the trouble of washing their hands before operating a cooking app displayed on the display, but in the aerial operation device according to the present disclosure, a display surface with buttons is displayed as a virtual image in the air, allowing the cooking app to be operated by operating buttons in the air. Therefore, since the cooking app can be operated without the need to take the trouble of washing one's hands, an aerial operation device with high operability can be provided.

For example, in an aerial operation device installed in an ATM, hospital, public institution, station, or airport, a display surface with buttons is displayed as a virtual image in the air, making it possible to operate an ATM, or a hospital or airport reception machine, without directly touching the shared display. This makes it unnecessary to touch the shared display with one's fingers, allowing for hygienic operation.

1, 41... Aerial operation device, 2, 42... Housing, 2b... Front surface, 2c... Side surface, 2d... Top surface, 2f... Bottom surface, 2g... Opening, 2h... First plate part, 2j... Second plate part, 3... Projection part, 3b... Main surface, 3c... End surface, 3d... Back surface, 10, 50... Display surface, 10b... Center, 11, 11A , 11B, 11C, 11D, 11E, 11N, 51... Button, 11b... Center, 12, 43... Sensor, 13... Character information, 14, 54... Aerial imaging device, 15, 55... Display device, 20, 60... Control section, 21... Image output section, 22... Target object detection section, 23... Recognition unit, 24...signal output unit, 25...notification unit, 25b...audio output unit, 25c...color change unit, 30...device, 42b...top surface, 42c...side surface, 42d...front surface, 42f...rear surface, 42g...bottom surface, 42h...inclined surface, A1...first direction, A2...second direction, A3...third direction, B...toilet bowl, C...light, D1...up/down direction, D2...left/right direction, D3...protruding direction, E...toilet paper, F...object, G1, G2, GX...group, K...distance, M...sink, T...toilet, V...reference line, W...wall, Y1, Y2...threshold value, Z1...first surface, Z2...second surface.

Claims (5)

A display device that displays information by emitting light;
an aerial imaging device that reflects light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air;
a sensor for detecting a position of an object approaching the button;
An aerial control device comprising:
a recognition unit that recognizes whether the button has been pressed or not based on the position of the object detected by the sensor;
A signal output unit that outputs a control signal corresponding to the button pressed by a user of the aerial operation device to an external device of the aerial operation device and controls the operation of the external device;
Equipped with
The recognition unit is
disabling the plurality of buttons so as to suppress output of the control signal by the signal output unit when it is recognized that the object detected by the sensor is located at the position of the button;
enabling the button to allow the signal output unit to output the control signal when it is recognized that the object detected by the sensor is moving away from the button;
The sensor detects the position of the object multiple times;
the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the object is removed from the button, enables the button, and the sensor detects a plurality of times that the object is located between a first surface and a second surface sandwiching the display surface.
Aerial control device. the plurality of buttons constitute a group including at least one of the buttons;
the recognition unit invalidates all of the buttons in the group when it recognizes that the object detected by the sensor is at the position of the button in the group.
The aerial control device according to claim 1 . A notification unit that notifies a user of the aerial operation device that the recognition unit has recognized,
the notification unit notifies the user that the button has been pressed when the sensor detects a plurality of times that the object is located between a first surface and a second surface sandwiching the display surface.
The aerial control device according to claim 1 or 2. The sensor measures a distance from the sensor to the object multiple times;
the signal output unit outputs a control signal to the device only when an average value of the measured distances from the sensor to the object is equal to or greater than a distance from the sensor to the second surface and equal to or less than a distance from the sensor to the first surface.
The aerial operation device according to any one of claims 1 to 3. A plurality of the sensors are provided,
The signal output unit does not output a control signal to the device when the plurality of sensors detects the object.
The aerial operation device according to any one of claims 1 to 4 .

Images