JP6294183B2 - Menu selection device and menu selection method - Google Patents

Description

  The present invention relates to a technique for selecting a menu item.

  A technique for realizing selection of a menu item using a virtual surround technique is known. For example, in Patent Document 1, a plurality of menu sound images corresponding to a plurality of menu items are localized in a virtual three-dimensional sound field, and an arbitrary menu sound image is selected by the wearer moving his / her head and selected. It describes that a menu item corresponding to a menu sound image is executed. Further, as disclosed in Patent Document 2, a technique for displaying a virtual object using AR (Augmented Reality) is known.

JP 2000-194460 A Japanese Patent Application Laid-Open No. 2014-2151569

However, in the invention described in Patent Document 1, since the placement of the virtual object indicating the menu item must be recognized only by sound, it may be difficult for a user who is not particularly used to recognize the placement of the virtual object.
An object of the present invention is to make it easy to recognize the arrangement of virtual objects indicating menu items.

  According to the present invention, a placement unit that places a virtual object indicating a menu item in a virtual space corresponding to a real space where a user exists, and a transfer characteristic of sound from the position where the virtual object is placed to the user A signal processing unit that performs signal processing on an audio signal indicating the menu item, a speaker that outputs sound according to the audio signal subjected to the signal processing, and the virtual object along the arrangement of the virtual object A selection unit that selects the menu item in response to detection of a movement of moving the body in the direction in which the virtual object is arranged, a detection unit that detects movement of the user's body And a menu selection device.

  The menu selection device may further include a display control unit that changes a display of the virtual object indicating the menu item when a sound indicating the menu item is output from the speaker.

  The arrangement unit may arrange a plurality of virtual objects indicating a plurality of menu items, and the display unit may display a part of the plurality of virtual objects.

  The speaker may output a sound indicating the menu item only among the menu items indicated by the virtual object displayed on the display unit among the plurality of menu items.

  The selection unit may select only the menu item indicated by the virtual object displayed on the display unit among the plurality of menu items.

  The menu selection device further includes a setting unit configured to set a plurality of detection areas of the same size in the plurality of virtual objects, and the selection unit is configured to move the detection area in the direction of one of the detection areas. The menu item indicated by the virtual object in which the detection area is set is selected in response to the detection of the movement of the body, and the setting unit detects the virtual object displayed on the display unit. The area may be enlarged.

  The arrangement unit arranges a plurality of virtual objects indicating a plurality of menu items, and the menu selection device further includes a setting unit that sets a plurality of detection areas of the same size to the plurality of virtual objects, The selection unit selects a menu item indicated by the virtual object in which the detection area is set in response to detection of an action of moving the body in the direction of the detection area of any of the plurality of detection areas, The detection unit may detect a direction of the user's face, and the setting unit may enlarge a detection region set in the first virtual object arranged in the detected direction.

  The display unit displays the plurality of virtual objects, and the menu selection device further includes a display control unit that enlarges the first virtual object among the plurality of virtual objects displayed on the display unit. Also good.

  The signal processing unit may subject the audio signal indicating the first menu item indicated by the first virtual object to signal processing different from the audio signal indicating another menu item of the plurality of menu items.

  Further, the present invention provides a step of arranging a virtual object indicating a menu item in a virtual space corresponding to a real space where a user exists, and a sound transmission characteristic from the position where the virtual object is arranged to the user. Applying the signal processing to the audio signal indicating the menu item, outputting the sound corresponding to the audio signal subjected to the signal processing, and arranging the virtual object along the arrangement of the virtual object. A step of displaying, a step of detecting a motion of the user's body, and a step of selecting the menu item in response to detecting a motion of moving the body in the direction in which the virtual object is arranged. Provides a menu selection method.

  ADVANTAGE OF THE INVENTION According to this invention, it can make it easy to recognize arrangement | positioning of the virtual object which shows a menu item.

The figure which shows the structure of a menu selection system. The figure which shows the hardware constitutions of an audio | voice reproduction terminal. The figure which shows the function structure of the control part of an audio | voice reproduction terminal. The figure which shows the hardware constitutions of a display terminal. The figure which shows the function structure of the control part of a display terminal. The sequence chart which shows operation | movement of a menu selection system. The figure which shows an example of arrangement | positioning of a virtual icon. The figure which shows the example of a display of a virtual icon. The figure which shows an example of a movement of a display terminal. The figure which shows the example of a display of a virtual icon. The figure which shows the function structure of the control part of the audio | voice reproduction terminal which concerns on 2nd Embodiment. The sequence chart which shows operation | movement of the menu selection system which concerns on 2nd Embodiment. The figure which shows an example of arrangement | positioning of the virtual icon which concerns on 2nd Embodiment. The figure which shows the example of a display of the virtual icon which concerns on 2nd Embodiment. The figure which shows an example of a user's head direction change. The figure which shows the example of a display of the virtual icon which concerns on 2nd Embodiment. The figure which shows an example of the change of the magnitude | size of a detection area. The figure which shows an example of the change of the magnitude | size of a detection area. The figure which shows an example of the detection area which concerns on a modification.

1. First Embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.

(1) Overall Configuration of System FIG. 1 is a diagram showing the configuration of the menu selection system 1. The menu selection system 1 includes an audio playback terminal 10 and a display terminal 20. The audio reproduction terminal 10 is an earphone type wearable computer. The audio reproduction terminal 10 is worn on the user's ear. The display terminal 20 is a wristwatch-type wearable computer. The display terminal 20 is worn on the user's arm. The audio reproduction terminal 10 and the display terminal 20 are paired in advance. The audio reproduction terminal 10 and the display terminal 20 cooperate to realize menu item selection by the user. That is, in this embodiment, the audio reproduction terminal 10 and the display terminal 20 cooperate to function as a menu selection device.

(2) Configuration of Audio Playback Terminal 10 The audio playback terminal 10 includes a right ear casing 101, a left ear casing 102, a cable 103, and a main body 104. The housing 101 is attached to the right ear of the user. The casing 101 accommodates a speaker 16-1. The housing 102 is attached to the user's left ear. The casing 102 accommodates a speaker 16-2. The cable 103 connects the housing 101 and the main body 104, and the housing 102 and the main body 104. The main body 104 reproduces sound in a stereo manner by supplying 2-channel audio signals to the speakers 16-1 and 16-2.

  FIG. 2 is a diagram illustrating a hardware configuration of the audio reproduction terminal 10. The audio reproduction terminal 10 has a function of arranging virtual icons indicating menu items in a virtual space corresponding to a real space in which the user exists, and allowing the user to recognize the arrangement of the virtual icons using virtual surround. The user can select a menu item indicated by the virtual icon by performing an operation of moving the body in the direction in which the virtual icon is arranged. The audio reproduction terminal 10 includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, a microphone 15, speakers 16-1 and 16-2, a positioning unit 17, a sensor unit 18, And a power source 19. These components are connected via a bus or cable 103. Note that components other than the speakers 16-1 and 16-2 are basically housed in the main body 104. However, the sensor unit 18 is accommodated in the housing 101 or 102.

  The control unit 11 executes a program stored in the storage unit 12 to perform calculation of data and control of other components. For example, a CPU is used as the control unit 11. The storage unit 12 stores programs executed by the control unit 11 and various data. The storage unit 12 includes a main storage device and an auxiliary storage device. For example, a RAM is used as the main storage device. For example, a flash memory is used as the auxiliary storage device.

  The communication unit 13 is a communication interface for performing data communication. The communication unit 13 communicates with the display terminal 20 in accordance with a near field communication standard such as Bluetooth (registered trademark). The operation unit 14 inputs a signal corresponding to a user operation to the control unit 11. For example, a touch sensor is used for the operation unit 14. The microphone 15 receives sound and inputs a signal corresponding to the received sound to the control unit 11. The speakers 16-1 and 16-2 output sound according to the audio signal.

  The positioning unit 17 measures the current position of the audio playback terminal 10. For the positioning unit 17, for example, a GPS (Global Positioning System) receiver is used. The power source 19 supplies power to each component of the audio playback terminal 10. For example, a battery is used as the power source 19.

  The sensor unit 18 is used to detect a user operation. The sensor unit 18 includes, for example, an acceleration sensor 181, an angular velocity sensor 182, and a geomagnetic sensor 183. The acceleration sensor 181 measures triaxial acceleration. The angular velocity sensor 182 measures the angular velocity in the triaxial direction. The geomagnetic sensor 183 measures the triaxial geomagnetism.

  FIG. 3 is a diagram illustrating a functional configuration of the control unit 11. The control unit 11 executes one or a plurality of programs stored in the storage unit 12, thereby arranging the arrangement unit 111, the signal processing unit 112, the output control unit 113, the transmission unit 114, the detection unit 115, the selection unit 116, and The function of the execution unit 117 is realized. Further, these functions may be realized by cooperation between the control unit 11 and other components.

  The arrangement unit 111 arranges a virtual object indicating a menu item in a virtual space corresponding to the real space where the user exists. The signal processing unit 112 performs signal processing corresponding to sound transmission characteristics from the position where the virtual object is arranged to the user on the audio signal indicating the menu item. The output control unit 113 outputs sound corresponding to the audio signal subjected to signal processing from the speakers 16-1 and 16-2. The transmission unit 114 transmits arrangement information indicating the arrangement of the virtual object to the display terminal 20. The detection unit 115 detects the movement of the user's body. The selection unit 116 selects a menu item in response to detection of an action of moving the body in the direction in which the virtual object is arranged. The execution unit 117 executes a process corresponding to the menu item selected by the selection unit 116.

(3) Configuration of Display Terminal 20 FIG. 4 is a diagram illustrating a hardware configuration of the display terminal 20. The display terminal 20 has a function of displaying a virtual icon. The display terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an operation unit 24, a display unit 25, a sensor unit 26, and a power source 27. These components are connected via a bus.

  The control unit 21 executes a program stored in the storage unit 22 to perform calculation of data and control of other components. For example, a CPU is used as the control unit 21. The storage unit 22 stores programs executed by the control unit 21 and various data. The storage unit 22 includes a main storage device and an auxiliary storage device. For example, a RAM is used as the main storage device. For example, a flash memory is used as the auxiliary storage device.

  The communication unit 23 is a communication interface for performing data communication. The communication unit 23 communicates with the audio reproduction terminal 10 in accordance with a near field communication standard such as Bluetooth (registered trademark). The operation unit 24 inputs a signal corresponding to a user operation to the control unit 21. For the operation unit 24, for example, a touch sensor is used. The display unit 25 displays information. As the display unit 25, for example, a liquid crystal display is used. The power supply 27 supplies power to each component of the display terminal 20. For example, a battery is used as the power source 27.

  The sensor unit 26 is used to detect a user's operation. The sensor unit 26 includes, for example, an acceleration sensor 261, an angular velocity sensor 262, and a geomagnetic sensor 263. The acceleration sensor 261 measures acceleration in three axis directions. The angular velocity sensor 262 measures the angular velocity in the triaxial direction. The geomagnetic sensor 263 measures the geomagnetism in the triaxial direction.

  FIG. 5 is a diagram illustrating a functional configuration of the control unit 21. The control unit 21 implements the functions of the display control unit 211 and the transmission unit 212 by executing one or more programs stored in the storage unit 22. Further, these functions may be realized by cooperation between the control unit 21 and other components.

  The display control unit 211 displays the virtual object along the arrangement of the virtual object. The transmission unit 212 transmits information for identifying the virtual object displayed on the display unit 25 to the audio reproduction terminal 10.

(4) Operation FIG. 6 is a sequence chart showing the operation of the menu selection system 1. This operation is started, for example, when the user performs an operation for starting the menu item using the audio reproduction terminal 10. This operation is, for example, a voice operation using voice recognition technology. For example, this operation is started when the word “menu” is issued by the user.

  In step S101, the arrangement unit 111 arranges a virtual icon (an example of a virtual object) indicating a menu item in a virtual space corresponding to a real space where the user exists. “Arranging virtual icons” specifically refers to determining the position of a virtual icon in the virtual space. The placement of the virtual icons is performed based on, for example, a terminal coordinate system indicating a position on the xy plane with the position of the user's head as the origin. FIG. 7 is a diagram illustrating an example of the arrangement of virtual icons. FIG. 7 is a view of the placement of virtual icons as seen from above. In FIG. 7, the user's front direction is indicated by the y-axis direction, the user's rear direction is indicated by the -y-axis direction, the user's right direction is indicated by the x-axis direction, and the user's left direction is indicated by the -x-axis direction. It is shown in

  In the example illustrated in FIG. 7, virtual icons I1 to I8 are arranged at an interval of 45 degrees on a virtual circle C1 centered on the position of the user's head on the xy plane. Virtual icons I1 to I8 indicate first to eighth menu items, respectively. The virtual icon I1 is arranged in front of the user. The virtual icon I2 is arranged on the right front side of the user. The virtual icon I3 is arranged on the right side of the user. The virtual icon I4 is arranged on the right rear side of the user. The virtual icon I5 is arranged behind the user. The virtual icon I6 is arranged on the left rear side of the user. The virtual icon I7 is arranged on the left side of the user. The virtual icon I8 is arranged on the left front side of the user.

  In step S <b> 102, the transmission unit 114 transmits arrangement information indicating the arrangement of virtual icons to the display terminal 20. This arrangement information includes position coordinates indicating the position where the virtual icon is arranged in the virtual space. In the example shown in FIG. 7, the arrangement information includes the position coordinates of the virtual icons I1 to I8 in the virtual space. Further, the arrangement information includes position coordinates indicating the position of the user's head in the virtual space and position coordinates indicating the position of the display terminal 20. The position of the display terminal 20 is, for example, the position of the display terminal 20 when the user takes a basic attitude of looking at the display unit 25 of the display terminal 20 and is registered in the audio reproduction terminal 10 in advance. When receiving the arrangement information from the audio reproduction terminal 10, the display terminal 20 proceeds to step S103.

  In step S103, the display control unit 211 displays virtual icons on the display unit 25 in accordance with the arrangement indicated by the received arrangement information. At this time, all the virtual icons may be displayed, or only a part of the virtual icons may be displayed. In the latter case, the display control unit 211 sets a selection range in the virtual space and displays only the virtual icons in the selection range on the display unit 25. The display of the virtual icon is performed based on a screen coordinate system that indicates a position on the XY plane with the upper left of the screen of the display unit 25 as the origin.

  FIG. 8 is a diagram illustrating a display example of virtual icons. In this example, the selection range R1 is first set so as to include the display terminal 20 and the user's head on the xy plane shown in FIG. The selection range R1 has a size and shape corresponding to the display unit 25. As shown in FIG. 7, the selection range R1 includes virtual icons I1 and I8. In this case, as shown in FIG. 8, virtual icons I1 and I8 are displayed on the display unit 25 based on the positions of the virtual icons I1 and I8 in the selection range R1.

  Specifically, the virtual icon I1 is displayed at a position on the screen corresponding to the position in the selection range R1. The virtual icon I8 is displayed at a position on the screen corresponding to the position in the selection range R1. The positions of the virtual icons I1 and I8 on the screen are obtained by converting the position coordinates of the terminal coordinates of the virtual icons I1 and I8 into the position coordinates of the screen coordinate system.

  Further, the display unit 25 displays marks W1 and H1 indicating the position of the display terminal 20 and the position of the head in the selection range R1. Specifically, the mark W1 is displayed at a position on the screen corresponding to the position of the display terminal 20 in the selection range R1. The mark H1 is displayed at a position on the screen corresponding to the position of the head in the selection range R1. The positions of the marks W1 and H1 on the screen are obtained by converting the position coordinates in the terminal coordinate system of the display terminal 20 and the position coordinates in the head terminal coordinate system into position coordinates in the screen coordinate system.

  When the user moves the display terminal 20, the selection range R1 moves according to the movement amount, and the display on the display unit 25 is updated. For example, as shown in FIG. 9, when the user moves the display terminal 20 to the right side, the selection range R1 also moves to the right side. As shown in FIG. 9, the selection range R1 after movement includes virtual icons I1 and I2. In this case, as shown in FIG. 10, virtual icons I1 and I2 are displayed on the display unit 25 based on the positions of the virtual icons I1 and I2 in the selected range R1 after movement. The display unit 25 displays marks W1 and H1 indicating the position of the display terminal 20 and the position of the head in the selected range R1 after movement.

  In step S104, the signal processing unit 112 responds to the sound transfer characteristics from the position of the virtual icon to the user based on the position in the virtual space where the virtual icon is arranged and the position of the user in the virtual space. Signal processing is performed on the audio signal indicating the menu item. Humans recognize the direction and distance of sound arrival based on the difference in the volume of sound transmitted to the left and right ears, the difference in time, and the difference in frequency characteristics. Therefore, by adding a sound pressure level difference, a time difference, and a frequency characteristic difference to the audio signals supplied to the left and right speakers 16-1 and 16-2 according to the position and direction in which the virtual icon is arranged, Sound that can be heard from the position where the virtual icon is pseudo-positioned can be created. In this case, there is a virtual sound source at the position of the virtual icon. Specifically, in the signal processing, a process of convolving a head-related transfer function corresponding to the position where the virtual icon is arranged and the position of the user into the audio signal indicating the menu item is performed.

  In step S105, the output control unit 113 performs a menu item reading process. In this reading process, sounds indicating menu items are output one by one from the speakers 16-1 and 16-2. The sound indicating the menu item is, for example, a sound that reads out the name of the menu item.

  Specifically, the output control unit 113 first determines the reading order of menu items. In the example shown in FIG. 7, when viewed clockwise from the virtual icon I1 arranged in front of the user, the virtual icons I1 to I8 are the virtual icons I1, I2, I3, I4, I5, I6, I7, I8. It is arranged in the order. In this case, the reading order of the first to eighth menu items is as follows: first menu item, second menu item, third menu item, fourth menu item, fifth menu item, sixth menu item, seventh menu item, The order is the eighth menu item.

  Subsequently, the output control unit 113 supplies audio signals indicating the menu items subjected to signal processing to the speakers 16-1 and 16-2 one by one in accordance with the reading order. At this time, the audio signal for the right ear is supplied to the speaker 16-1, and the audio signal for the left ear is supplied to the speaker 16-2. Thus, sounds indicating the first to eighth menu items are output one by one from the speakers 16-1 and 16-2.

  In the example shown in FIG. 7, the virtual icon I1 indicating the first menu item is arranged in front of the user. Therefore, the sound indicating the first menu item is heard from the front of the user in a pseudo manner. The virtual icon I2 indicating the second menu item is arranged on the right front side of the user. Therefore, the sound indicating the second menu item is heard from the front right of the user in a pseudo manner. The same applies to the sounds indicating the third to eighth menu items. Thereby, the user can recognize the arrangement of the virtual icons I1 to I8.

  As another example, when only a part of the virtual icon is displayed on the display unit 25, a sound indicating the menu item is output only for the menu item indicated by the virtual icon displayed on the display unit 25. Also good. In the example illustrated in FIG. 8, virtual icons I <b> 1 and I <b> 8 are displayed on the display unit 25. In this case, the transmission unit 212 transmits IDs for identifying the first and eighth menu items indicating the virtual icons I1 and I8 displayed on the display unit 25 to the audio reproduction terminal 10. When receiving the IDs of the first and eighth menu items from the display terminal 20, the output control unit 113 sequentially outputs the sound indicating the first menu item and the sound indicating the eighth menu item from the speakers 16-1 and 16-2. Output.

  The same processing may be performed when the display of the display unit 25 is updated by moving the display terminal 20 by the user. For example, when the display of the display unit 25 is updated as illustrated in FIG. 10, virtual icons I1 and I2 are displayed on the display unit 25. In this case, the transmission unit 212 transmits IDs for identifying the first and second menu items indicated by the virtual icons I1 and I2 displayed on the display unit 25 to the audio reproduction terminal 10. When receiving the IDs of the first and second menus from the display terminal 20, the output control unit 113 sequentially outputs sounds indicating the first menu and sounds indicating the second menu item from the speakers 16-1 and 16-2. .

  In step S106, each time a sound indicating each menu item is output from the speakers 16-1 and 16-2, the transmission unit 114 transmits an ID for identifying the menu item to the display terminal 20. For example, when a sound indicating the first menu item is output from the speakers 16-1 and 16-2, an ID for identifying the first menu item is transmitted to the display terminal 20. When a sound indicating the second menu item is output from the speakers 16-1 and 16-2, an ID for identifying the second menu item is transmitted to the display terminal 20. When the display terminal 20 receives the menu item ID from the audio reproduction terminal 10, the process proceeds to step S107.

  In step S107, the display control unit 211 changes the display of the virtual icon indicating the menu item identified by the received ID. This change in display includes, for example, changing the color, shape, or size of the virtual icon. In the example illustrated in FIG. 8, virtual icons I1 and I2 indicating the first and second menu items are displayed on the display unit 25. For example, when the ID of the first menu item is received, the virtual icon I1 indicating the first menu item is illuminated.

  The user recognizes the arrangement of the virtual icons based on the sound indicating the menu items output from the speakers 16-1 and 16-2 and the arrangement of the virtual icons displayed on the display unit 25. In the example shown in FIG. 7, since the sound of the first menu item can be heard from the front of the user, it can be seen that the virtual icon I1 indicating the first menu item is arranged in front of the user. In the example shown in FIG. 8, it can be seen that the virtual icon I1 is arranged in front of the user based on the relative positional relationship between the virtual icon I1 and the mark H1. For example, when selecting the first menu item, the user performs an operation of tilting the head forward or an operation of moving the arm on which the display terminal 20 is worn forward.

  In step S108, the detection unit 115 detects the movement of the user's body. For example, when the user performs an operation of tilting his / her head, this operation is detected based on the angular velocity measured by the angular velocity sensor 182. On the other hand, when the user performs an operation of moving the arm on which the display terminal 20 is worn, this operation is detected based on the acceleration measured by the acceleration sensor 261. Specifically, the transmission unit 212 transmits the acceleration measured by the acceleration sensor 261 to the voice reproduction terminal 10. When the acceleration is received from the display terminal 20, the detection unit 115 detects an operation of moving the arm on which the display terminal 20 is worn based on the position of the display terminal 20 registered in advance and the acceleration.

  In step S <b> 109, when an operation of moving the body in the direction in which the virtual icon is arranged is detected, the selection unit 116 selects the menu item indicated by the virtual icon. Note that the direction in which the virtual icon is arranged does not necessarily need to be a direction passing through the position in which the virtual icon is arranged. For example, the direction passing through a predetermined range including the position where the virtual icon is arranged may also be included in the direction where the virtual icon is arranged. For example, when an action of tilting the head forward or an action of moving the arm on which the display terminal 20 is worn forward is detected, the first menu item indicated by the virtual icon I1 arranged in front of the user is selected. The

  As another example, the selection unit 116 may select only the menu item indicated by the virtual icon displayed on the display unit 25. In the example illustrated in FIG. 8, virtual icons I <b> 1 and I <b> 8 are displayed on the display unit 25. In this case, as described above, the transmission unit 212 transmits the ID for identifying the first and eighth menu items indicated by the virtual icons I1 and I8 displayed on the display unit 25 to the audio reproduction terminal 10. When receiving the IDs of the first and eighth menu items from the display terminal 20, the selection unit 116 can select only the first and eighth menu items and cannot select the other second to seventh menu items. To be limited. In this case, when an operation of moving the body in the direction in which the virtual icon I1 or I8 is arranged is performed, the first or eighth menu item is selected. On the other hand, even if an operation of moving the body in the direction in which the virtual icons I2 to I7 are arranged, the second to seventh menu items are not selected.

  In step S110, the execution unit 117 executes a process corresponding to the menu item selected by the user. For example, when the first menu item is related to an application, when the first menu item is selected, the application corresponding to the first menu item is activated.

  According to the present embodiment, since the user can recognize the arrangement of the virtual icon indicating the menu item by the sound and the display, the arrangement of the virtual icon is easily recognized. If the number of virtual icons is large, displaying all the virtual icons makes the virtual icons small and difficult to distinguish from each other. However, when only a part of the virtual icon is displayed as in the present embodiment, it is possible to prevent the virtual icon from becoming difficult to visually recognize. Further, when the sound indicating the menu item is output from the speakers 16-1 and 16-2, the display of the virtual icon indicating the menu item changes, so that the sound indicating the menu item is displayed on the display unit 25. It becomes easy to recognize the correspondence with the virtual icon.

2. Second Embodiment Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. In the following description, differences from the first embodiment will be mainly described.

  The configuration of the menu selection system 1 according to the second embodiment is basically the same as the configuration of the menu selection system 1 according to the first embodiment. However, the functional configuration of the control unit 11 is different from that of the first embodiment.

  FIG. 11 is a diagram illustrating a functional configuration of the control unit 11 according to the second embodiment. In the second embodiment, the control unit 11 realizes the function of the setting unit 118 in addition to the function described in the first embodiment. The setting unit 118 sets a detection area in the menu item. The selection unit 116 selects a menu item in which the detection area is set in response to detection of an action of moving the body in the direction of the detection area.

  FIG. 12 is a sequence chart showing the operation of the menu selection system 1 according to the second embodiment. This operation is started, for example, when the user performs an operation of starting the menu item using the audio reproduction terminal 10 as in the first embodiment.

  In step S201, the arrangement unit 111 arranges virtual icons indicating menu items in the virtual space corresponding to the real space where the user exists, as in step S101 described in the first embodiment. FIG. 13 is a diagram showing an example of the arrangement of menu items according to the second embodiment. FIG. 13 is a view of the placement of virtual icons as seen from above. In FIG. 13, the user's front direction is indicated by the y-axis direction, the user's rear direction is indicated by the −y-axis direction, the user's right direction is indicated by the x-axis direction, and the user's left direction is indicated by the −x-axis direction. It is shown in

  In the example shown in FIG. 13, virtual icons I1 to I5 are arranged at equal intervals on a virtual straight line L1 extending in the x direction in front of the user. Virtual icons I1 to I5 indicate first to fifth menu items, respectively. The virtual icons I1 to I5 are arranged in the order of virtual icons I1, I2, I3, I4, and I5 from the left side of the user.

  In step S202, the detection unit 115 detects the orientation of the user's head based on the angular velocity measured by the angular velocity sensor 182.

  In step S203, the transmission unit 114 transmits the arrangement information indicating the arrangement of the virtual icons and the direction information indicating the head direction detected in step S202 to the display terminal 20. When the display terminal 20 receives the arrangement information and the orientation information from the audio reproduction terminal 10, the display terminal 20 proceeds to step S204.

  In step S204, the display control unit 211 displays virtual icons on the display unit 25 in accordance with the arrangement indicated by the received arrangement information. In the second embodiment, all virtual icons are displayed.

  FIG. 14 is a diagram illustrating a display example of virtual icons according to the second embodiment. In the example illustrated in FIG. 14, virtual icons I1 to I5 are displayed on the display unit 25 along the arrangement of the virtual icons I1 to I5. Further, the display unit 25 displays marks W1 and H1 indicating the position of the display terminal 20 and the position of the head.

  Subsequently, the display control unit 211 changes the display size of the virtual icon based on the received orientation information. For example, the virtual icon may be enlarged so that the menu icon is closer to the menu item closer to the intersection of the head direction indicated by the orientation information and the virtual straight line L1. In the example shown in FIG. 13, a virtual icon I3 (an example of a first virtual object) is arranged in the direction of the user's face. In this case, as shown in FIG. 14, the virtual icon I3 is enlarged so as to become the largest. Also, the virtual icons I2 and I4 arranged next to the virtual icon I3 are enlarged so as to be the second largest.

  When the user changes the direction of the head, the display on the display unit 25 is updated. For example, as illustrated in FIG. 15, when the user turns his head to the left, the detection unit 115 detects the orientation of the user's head based on the angular velocity measured by the angular velocity sensor 182. The transmission unit 114 transmits direction information indicating the direction of the newly detected head to the display terminal 20. When the orientation information is received from the audio reproduction terminal 10, the display control unit 211 changes the display size of the virtual icon based on the received orientation information. In the example shown in FIG. 15, the virtual icon I1 is arranged in the direction of the user's face. In this case, as shown in FIG. 16, the virtual icon I1 is enlarged so as to become the largest. Further, the virtual icon I2 arranged next to the virtual icon I1 is enlarged so as to be the second largest.

  In step S205, the setting unit 118 sets a detection area for the virtual icon. This detection area is an area including the position where the virtual icon is arranged. In the example shown in FIG. 13, a triangular area having the virtual straight line L1 as the base and the user's position P0 as the apex is defined as detection areas D1 to D5 by lines connecting the position P0 and the points P1 to P6 on the virtual straight line L1. Divided. The detection areas D1 to D5 include the positions of the virtual icons I1 to I5, respectively. The lengths 11 to 15 of the bottom sides of the detection areas D1 to D5 are equal to each other. That is, the detection areas D1 to D5 have the same size.

  The setting unit 118 changes the size of the detection area based on the head orientation detected in step S202. FIG. 17 is a diagram illustrating an example of changing the size of the detection area. In this example, the virtual icon I3 is arranged in the direction of the user's face. In this case, the bottom length l3 of the detection region D3 is the largest, the bottom lengths l2 and l4 of the detection regions D2 and D4 adjacent to the detection region D3 are the second largest, and the bottom lengths of the detection regions D1 and D5 The sizes of the detection areas D1 to D5 are changed so that the lengths l1 and l5 are minimized.

  When the user changes the direction of the head, the size of the detection area is further changed. For example, as illustrated in FIG. 15, when the user turns his head to the left, the detection unit 115 detects the orientation of the user's head based on the angular velocity measured by the angular velocity sensor 182. The setting unit 118 changes the size of the detection area based on the newly detected head orientation. In this example, the virtual icon I1 is arranged in the direction of the user's face. In this case, the length l1 of the bottom of the detection region D1 is the largest, the length l2 of the bottom of the detection region D2 adjacent to the detection region D1 is the second largest, and the lengths 13 to 15 of the bottom of the detection regions D3 to D5. The sizes of the detection areas D1 to D5 are changed so that the value becomes the smallest.

  In step S206, as in step S104 described in the first embodiment, the signal processing unit 112 performs signal processing in accordance with sound transfer characteristics from the position where the virtual icon is arranged to the user as menu items. To the audio signal shown. Further, the signal processing unit 112 may perform different signal processing on the audio signal indicating the menu item according to the size of the detection area. This signal processing is processing for changing the volume level and processing for applying an echo. For example, the volume level of the audio signal indicating the menu item corresponding to the detection area may be increased as the detection area is larger. As another example, an echo may be applied so that the reverberation time of the audio signal indicating the menu item corresponding to the detection area increases as the detection area increases.

  In step S207, the output control unit 113 performs a menu item reading process in the same manner as in step S105 described in the first embodiment. In this reading process, sounds indicating menu items are output one by one from the speakers 16-1 and 16-2. In the example shown in FIG. 17, sounds indicating the first to fifth menu items are output one by one from the speaker 16-1 and the speaker 16-2 in order. At this time, the larger the detection area, the larger the volume level or reverberation time of the sound indicating the menu item. In this example, the volume level or reverberation time of the sound indicating the third menu item is the largest, and the volume level or reverberation time of the sound indicating the second and fourth menu items is the second largest. Therefore, the user can see that the detection area D3 corresponding to the third menu item is the largest, and the detection areas D2 and D4 corresponding to the second and fourth menu items are the second largest.

  The processes in steps S208 and S209 are the same as steps S106 and S107 described in the first embodiment.

  In the example shown in FIG. 17, when the third menu item is selected, the user performs an operation of tilting his / her head toward the detection area D3 or an operation of moving the arm on which the display terminal 20 is worn. In step S210, the detection unit 115 detects the movement of the user's body, similarly to step S108 described in the first embodiment.

  In step S211, the selection part 116 will select the menu item, if the operation | movement which moves a body toward the detection area is detected. For example, when an operation of tilting the head in a direction intersecting the bottom of the detection region D3 or an operation of moving the arm on which the display terminal 20 is worn is detected, the third icon indicated by the virtual icon I3 in which the detection region D3 is set is displayed. A menu item is selected.

  In step S212, the execution unit 117 executes processing corresponding to the menu item selected by the user, similarly to step S110 described in the first embodiment.

  According to the second embodiment, since the user can recognize the arrangement of the virtual icon indicating the menu item by the sound and the display, the arrangement of the virtual icon is easily recognized. Further, since the detection area set in the virtual icon arranged in the direction of the user's face is enlarged, the menu item indicated by the virtual icon can be easily selected. Further, a virtual icon arranged in the direction of the user's face is displayed in an enlarged manner, and the menu item indicated by the virtual icon indicates a sound volume level or a reverberation time indicating the menu item as another menu item. Since it is different from sound, the user can easily recognize that the detection area set in the virtual icon has been enlarged.

3. Modifications The present invention is not limited to the first and second embodiments described above. For example, the first and second embodiments may be modified as follows. Further, the following modifications may be combined.

(1) In the first embodiment, a detection area may be set in the menu item. In this case, the control unit 11 realizes the function of the setting unit 118 described in the second embodiment. The setting unit 118 sets a detection area for the virtual icon. FIG. 19 is a diagram illustrating an example of a detection region according to a modification. In the example shown in FIG. 19, the virtual circle C1 is divided into detection regions D1 to D8 by lines connecting the center P10 and points P11 to P18 on the virtual circle C1. The detection areas D1 to D8 include the positions of the virtual icons I1 to I8, respectively. The center angles of the detection areas D1 to D8 are equal to each other. That is, the detection areas D1 to D8 have the same size.

  In addition, the setting unit 118 changes the detection area in accordance with the virtual icon displayed on the display unit 25. In the example illustrated in FIG. 8, virtual icons I <b> 1 and I <b> 8 are displayed on the display unit 25. In this case, first, the detection areas D2 to D7 set in the virtual icons I2 to I7 that are not displayed on the display unit 25 are deleted. Further, the detection areas D1 and D8 set in the virtual icons I1 and I8 displayed on the display unit 25 are enlarged so that the central angle thereof is increased. When the movement of moving the body in the direction of the detection area D1 or D8, for example, the direction intersecting the circumference of the detection area D1 or D8 is detected, the selection unit 116 selects the first or eighth menu item. According to this modification, the menu item indicated by the virtual icon displayed on the display unit 25 can be easily selected.

(2) In the first embodiment, for the menu item indicated by the virtual icon displayed on the display unit 25, the sound indicating the menu item may be changed. In the example illustrated in FIG. 8, virtual icons I <b> 1 and I <b> 8 are displayed on the display unit 25. In this case, the transmission unit 212 transmits IDs for identifying the first and eighth menu items indicating the virtual icons I1 and I8 displayed on the display unit 25 to the audio reproduction terminal 10. When receiving the IDs of the first and eighth menu items from the display terminal 20, the signal processing unit 112 is different from the audio signals indicating the second to seventh menu items in the audio signals indicating the first and eighth menu items. Apply signal processing. This signal processing is processing for changing the volume level, pitch, or tone color. The output control unit 113 outputs sounds indicating the first to eighth menu items one by one from the speakers 16-1 and 16-2. At this time, for the first and eighth menu items indicated by the virtual icons I1 and I8 displayed on the display unit 25, the other second to seventh menu items are the volume level of the sound indicating the menu item, The pitch or tone is different. Therefore, it becomes easy to recognize the correspondence between the virtual icon displayed on the display unit 25 and the sound indicating the menu item.

(3) In the first embodiment and the second embodiment, the display size of the virtual icon may be changed according to the distance between the display terminal 20 and the virtual icon. For example, when the user brings the display terminal 20 close to a virtual icon, the virtual icon may be enlarged and displayed.

(4) The audio reproduction terminal 10 and the display terminal 20 are not limited to the examples described in the first embodiment and the second embodiment. For example, the audio reproduction terminal 10 may be an ear hook type or overhead type wearable computer. The display terminal 20 may be a glasses-type wearable computer or a smartphone. Moreover, the audio | voice reproduction terminal 10 and the display terminal 20 may be implement | achieved by one apparatus. For example, a glasses-type wearable computer may realize the functions of both the audio playback terminal 10 and the display terminal 20.

(5) Menu items are not limited to the examples described in the first embodiment and the second embodiment. The menu item may relate to music or images downloaded to the audio playback terminal 10.

(6) In the first embodiment and the second embodiment, the example in which the function of the control unit 11 and the function of the control unit 21 are realized by software has been described. At least a part of the function of the control unit 11 and the function of the control unit 21 may be realized by hardware. For example, the signal processing unit 112 may be realized by hardware.

(7) The program executed by the control unit 11 of the audio reproduction terminal 10 and the control unit 21 of the display terminal 20 may be downloaded via a communication line such as the Internet. The program is provided in a state of being recorded on a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory. May be.

10: Audio playback terminal, 16-1, 16-2: Speaker, 20: Display terminal, 25: Display unit, 111: Arrangement unit, 112: Signal processing unit, 113: Output control unit, 114: Transmission unit, 115: Detection unit 116: Selection unit 117: Execution unit 118: Setting unit 211: Display control unit 212: Transmission unit

Claims (11)

A placement unit for placing a virtual object indicating a menu item in a virtual space corresponding to a real space where the user exists;
A signal processing unit that performs signal processing according to sound transfer characteristics from the position where the virtual object is arranged to the user to the audio signal indicating the menu item;
A speaker that outputs a sound corresponding to the audio signal subjected to the signal processing;
A selection range setting unit that sets a selection range in the virtual space so as to include at least part of the user's body;
A display unit configured to display the virtual object included in the set selection range and a mark indicating the position of the body at a position on the screen corresponding to the position in the selection range based on the placement of the virtual object; ,
A detector for detecting the operation before Kitai,
A menu selection device comprising: a selection unit that selects the menu item in response to detection of an action of moving the body in a direction in which the virtual object is arranged. The menu selection device according to claim 1, further comprising: a display control unit configured to change a display of the virtual object indicating the menu item when a sound indicating the menu item is output from the speaker. The arrangement unit arranges a plurality of virtual objects indicating a plurality of menu items,
The menu selection device according to claim 1, wherein the display unit displays a part of the plurality of virtual objects. The menu selection device according to claim 3, wherein the speaker outputs a sound indicating the menu item only in the menu item indicated by the virtual object displayed on the display unit among the plurality of menu items. 5. The menu selection device according to claim 3, wherein the selection unit selects only the menu item indicated by the virtual object displayed on the display unit among the plurality of menu items. A detection area setting unit configured to set a plurality of detection areas of the same size in the plurality of virtual objects;
The selection unit selects a menu item indicated by the virtual object in which the detection area is set, in response to detection of an action of moving the body in the direction of the detection area of any of the plurality of detection areas,
The menu selection device according to any one of claims 3 to 5, wherein the detection area setting unit enlarges a detection area set in the virtual object displayed on the display unit. The arrangement unit arranges a plurality of virtual objects indicating a plurality of menu items,
A detection area setting unit configured to set a plurality of detection areas of the same size in the plurality of virtual objects;
The selection unit selects a menu item indicated by the virtual object in which the detection area is set, in response to detection of an action of moving the body in the direction of the detection area of any of the plurality of detection areas,
The detection unit detects the orientation of the user's face,
The menu selection device according to claim 1, wherein the detection area setting unit enlarges a detection area set in the first virtual object arranged in the detected direction. The display unit displays the plurality of virtual objects,
The menu selection device according to claim 7, further comprising a display control unit that enlarges the first virtual object among the plurality of virtual objects displayed on the display unit. The signal processing unit performs signal processing different from the audio signal indicating the other menu items of the plurality of menu items on the audio signal indicating the first menu item indicated by the first virtual object. Menu selection device described in 1. The selection range setting unit sets the selection range on the basis of the position of the display unit,
The detection unit detects a movement amount and a movement direction of the display unit,
The selection range setting unit moves the selection range according to the detected movement amount and movement direction,
When the selection range is moved, the display unit updates the screen to correspond the virtual object and the mark included in the moved selection range to the positions in the moved selection range. Display at the position on the screen
The menu selection device according to any one of claims 1 to 9. Placing a virtual object indicating a menu item in a virtual space corresponding to a real space where the user exists;
Applying signal processing corresponding to a sound transmission characteristic from the position where the virtual object is arranged to the user to the audio signal indicating the menu item;
Outputting a sound corresponding to the audio signal subjected to the signal processing;
Setting a selection range in the virtual space to include at least part of the user's body;
Based on the placement of the virtual object, the virtual included in the set selection range
Displaying an object and a mark indicating the position of the body at a position on the screen corresponding to the position in the selection range ;
Detecting an operation before Kitai,
A menu selection method comprising: selecting the menu item in response to detecting an action of moving the body in a direction in which the virtual object is arranged.

Images