JP6907613B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to a technique for inputting a locus in which a virtual object moves in a virtual space.

A technique for inputting a trajectory of a virtual object such as a sound image in a virtual space is known. For example, Patent Document 1 describes a technique for designating a two-dimensional sound image position locus by selecting a sound image position change pattern from a plurality of preset patterns and further setting parameters. ing.

Japanese Patent No. 3525653

Patent Document 1 does not describe a technique for inputting a three-dimensional movement locus. Further, in order to input the three-dimensional movement locus, a more complicated process than in Patent Document 1 is required. Furthermore, it is conceivable to input the trajectory expressed by the function using the knowledge of mathematics, but in either case, the processing becomes complicated.

On the other hand, the present invention relates to a technique for more intuitively inputting a three-dimensional movement locus.

In the present invention, a virtual object uses a locus acquisition unit that acquires a locus instructed on the detection surface from an input device having a detection surface that detects three-dimensional coordinates, and a locus acquired by the locus acquisition unit. Provided is an information processing device having an output unit that outputs a locus of movement in a virtual space.

This information processing device has a locus calculation unit that calculates the locus of the virtual object moving in the virtual space by using the locus acquired by the locus acquisition unit, and the output unit is calculated by the locus calculation unit. The completed locus may be output.

It has a coordinate conversion unit that converts the coordinates included in the locus acquired by the locus acquisition unit into the coordinates on the virtual solid set in the virtual space, and the locus calculation unit is converted by the coordinate conversion unit. The locus of movement in the virtual space may be calculated using the coordinates.

The virtual space is an acoustic space, and the virtual object may be a sound image in the acoustic space.

This information processing device uses a position acquisition unit that acquires the position of the listener in the acoustic space, the position of the listener acquired by the position acquisition unit, and the locus output by the output unit to perform sound image localization. It may have a localization parameter calculation unit for calculating parameters for control.

Further, in the present invention, a virtual object moves in a virtual space by using a step of acquiring a locus instructed on the detection surface from an input device having a detection surface for detecting three-dimensional coordinates and the acquired locus. Provided is an information processing method having a step of outputting a trajectory to be performed.

According to the present invention, a three-dimensional movement locus can be input more intuitively.

The figure which illustrates the functional structure of the information processing system 1 which concerns on one Embodiment. The figure which illustrates the hardware configuration of the information processing apparatus 10. The figure which illustrates the structure of the input device 20. The flowchart which illustrates the operation of the information processing apparatus 10. The figure which illustrates the image displayed on the display part 16. The figure which shows another example of the structure of the input device 20.

1. 1. Configuration FIG. 1 is a diagram illustrating the functional configuration of the information processing system 1 according to the embodiment. The information processing system 1 is a system for inputting a locus of a virtual object in a virtual space. The information processing system 1 includes an information processing device 10 and an input device 20. The input device 20 is a device that receives input of a locus, and has a three-dimensional shape detection surface. The input device 20 is a so-called touch-type input device, and outputs a signal (hereinafter referred to as “position signal”) indicating a position touched by an indicator (for example, a user's finger) on the detection surface.

In this example, the locus of the virtual object moves on the surface of the virtual solid 19 set in the information processing device 10. The information processing device 10 stores data that identifies the virtual solid 19. The virtual solid 19 has a shape corresponding to the detection surface of the input device 20. For example, the virtual solid 19 and the detection surface of the input device 20 are similar figures. The information processing device 10 calculates the locus of the virtual object in the virtual space so that the virtual object moves on the surface of the virtual solid 19 according to the locus of the user touching the detection surface of the input device 20.

In this example, the virtual object is a sound image and the virtual space is an acoustic space. The acoustic space may be a real space such as a concert hall or a movie theater, or may be a (virtual) space around a viewer wearing headphones. The detection surface of the input device 20 is a spherical surface or a hemispherical surface, and the virtual solid 19 has a spherical surface or a hemispherical surface (virtual spherical surface or virtual hemispherical surface) similar to the detection surface. The touch position of the detection surface is represented using a polar coordinate system (r, θ, φ). Assuming that the radius of the detection surface of the input device 20 is r2 and the radius of the virtual solid 19 is r1, the touch position (r2, θ1, φ1), which is one point of the detection surface, is 1 on the surface of the virtual solid 19. It is mapped to two points, virtual positions (r1, θ1, φ1).

The information processing device 10 includes an operation unit 11, a locus control unit 12, a localization parameter calculation unit 13, a localization processing unit 14, a storage unit 15, and a display unit 16. The operation unit 11 accepts the user's operation. The operations accepted by the operation unit 11 are, for example, parameters for setting the virtual solid 19, a method of coordinate conversion (mapping) from the touch position to the virtual position, movement start, movement end, or movement speed of the sound image. It is an operation.

The locus control unit 12 outputs locus data according to the position signal input from the input device 20 and the information input from the operation unit 11. The locus data shows a locus in which a virtual object (sound image) moves in a virtual space. The locus data includes coordinate information indicating coordinates in the virtual solid 19 and time information corresponding to each coordinate information. The time information is information indicating at least the relative positional relationship of two or more coordinate information on the time axis, and does not necessarily have to indicate the absolute time. Specifically, the locus control unit 12 includes a locus acquisition unit 121, a coordinate conversion unit 122, a locus calculation unit 123, and an output unit 124. The locus acquisition unit 121 acquires a locus instructed on the detection surface of the input device 20 from the input device 20. The locus acquired here is the time change of the coordinates. Specifically, the locus acquisition unit 121 acquires the coordinates (coordinate information) of the touch position indicated by the position signal. The locus acquisition unit 121 may acquire the time information corresponding to the coordinate information from the input device 20 together with the acquisition of the coordinates of the touch position. Alternatively, the locus acquisition unit 121 may acquire the time information corresponding to the coordinate information from the internal timer (not shown) of the information processing apparatus 10. As will be described later, since the process of acquiring the coordinate information from the input device 20 is repeated in time series, it can be said that the locus acquisition unit 121 acquires the locus of the touch position. The coordinate conversion unit 122 performs a process of converting the touch position into a virtual position. The virtual position referred to here is a position on the virtual solid 19. The coordinate conversion unit 122 outputs information indicating the virtual position. The locus calculation unit 123 calculates a locus in a virtual space (this virtual space corresponds to, for example, an acoustic space) based on a virtual position. The output unit 124 outputs locus data indicating the calculated locus.

The localization parameter calculation unit 13 receives the input of the trajectory data and outputs the localization parameter corresponding to the trajectory data. The localization parameter is a parameter for designating the sound image position in the acoustic space. The localization processing unit 14 performs a process of localizing the sound image of the input sound signal (hereinafter referred to as “input sound signal”) at a position according to the localization parameter (hereinafter, this process is referred to as “sound image localization process”) and is processed. Outputs the sound signal. Hereinafter, this signal is referred to as an "output sound signal". The localization parameter calculation unit 13 calculates the localization parameter at a timing determined according to the time information included in the locus data or the time information acquired from the internal timer (not shown) of the information processing apparatus 10. For example, the localization parameter calculation unit 13 has elapsed a predetermined time since the last time the localization parameter was calculated based on the time information included in the trajectory data or the time information acquired from the internal timer (not shown) of the information processing device 10. The localization parameters are calculated at the timing when it is shown that the data has been processed.

The storage unit 15 stores locus data and various types of data. The display unit 16 visually displays the movement of the sound image in the acoustic space. The display unit 16 displays an image object representing an acoustic space and an image object representing a sound image.

FIG. 2 is a diagram illustrating a hardware configuration of the information processing device 10. The information processing device 10 is a computer device having a CPU 101, a memory 102, an input / output IF 104, a keyboard 105, and a display 106. The CPU 101 is a control device that controls other hardware elements of the information processing device 10. The memory 102 is a storage device and stores various data and programs. The memory 102 may be configured to include an HDD (Hard Disk Drive) or an SSD (Solid State Drive), but includes at least a RAM (Random Access Memory). The input / output IF 104 is an interface for inputting or outputting a signal (data) to and from an external device. In this example, the input / output IF 104 receives the input of the position signal from the input device 20. The keyboard 105 is an input device for the user to input instructions or information to the information processing device 10. The display 106 is a device for displaying information, and includes, for example, an LCD (Liquid Crystal Display). Each device (hardware element) is connected on the bus of CPU 101.

In this example, the memory 102 stores a program (hereinafter referred to as “trajectory processing program”) for causing the computer device to function as the information processing device 10 in the information processing system 1. When the CPU 101 executes the locus processing program, the functions of the locus control unit 12, the localization parameter calculation unit 13, and the localization processing unit 14 in FIG. 1 are realized. The display 106 corresponds to the display unit 16. The keyboard 105 corresponds to the operation unit 11. The memory 102 corresponds to the storage unit 15.

FIG. 3 is a diagram illustrating the configuration of the input device 20. The input device 20 includes a housing 21, a sensor film 22, and a controller 23. The housing 21 is a structure that defines the shape of the detection surface of the input device 20. In this example, the housing 21 forms a hemisphere with a radius r2. The sensor film 22 is a film-like touch sensor. As the sensor film 22, for example, a so-called cloth sensor in which a touch sensor is woven into a cloth, or a sensor film in which touch sensor elements are arranged in a grid pattern and attached to the film is used. The sensor film 22 is attached along the housing 21 to form a detection surface. That is, the detection surface is a hemisphere having a radius r2. The controller 23 processes the signal output from the sensor film 22 and outputs a position signal representing the touch position in a predetermined coordinate system. When outputting the position signal, the controller 23 may also output the time information. Further, the housing 21 in FIG. 3 is a structure that partially supports the sensor film 22, but the housing 21 is a hemispherical resin molded product (a structure in which a sphere whose inside is a space is cut through the center of the sphere). ), And the housing 21 may be covered with a sensor film. According to this configuration, the entire sensor film 22 can be supported.

The user can input the locus by tracing the detection surface of the input device 20 with, for example, a finger. To "trace" the detection surface with your finger, first touch one point on the detection surface with your finger, move the position of your finger while touching the detection surface, and then move one point on the detection surface (the same point as the first point is different). It is a continuous operation of releasing the finger with (may be). The point on the detection surface touched first is called the sensing start point, and the touched time is called the sensing start time. The point where the finger is released from the detection surface is called the sensing end point, and the time when the finger is released is called the sensing end time. The locus acquisition unit 121 observes which position of the detection surface is touched at regular time intervals from the sensing start time to the sensing end time, and obtains a locus on the detection surface. The indicator that indicates the position on the detection surface in the input device 20 is not limited to the user's finger, and for example, a so-called stylus pen or the like may be used.

2. Operation FIG. 4 is a flowchart illustrating the operation of the information processing device 10. In the following, functional elements such as the locus control unit 12 will be described as the main body of processing, in which hardware elements such as CPU 101 executing software such as a locus processing program cooperate with other hardware elements. It means to execute the process. In this example, the acoustic space is a real space for listening to sound, for example, an acoustic space such as a theater in which a plurality of speakers are arranged, and its shape, size, and speaker arrangement are known. Information indicating the shape, size, speaker arrangement, and the like of the theater is input to the information processing device 10 in advance, and is stored in the storage unit 15.

In step S101, the operation unit 11 receives an input of a parameter used for calculating the locus. In this example, the parameter used to calculate the locus includes the radius r1 of the virtual solid 19 (in this example, a hemisphere represented by a virtual sphere). That is, the user can set the radius of the virtual sphere to a desired value by operating the operation unit 11. The data indicating the radius r1 is stored in the storage unit 15.

In step S102, the locus acquisition unit 121 samples the position signal output from the input device 20 and obtains the coordinates (r2, θ1, φ1) of the touch position, which is one point. When the coordinates are acquired, the locus acquisition unit 121 acquires time information from the internal timer (not shown) of the information processing device 10. This time information indicates the time t at which the coordinates were acquired. In this way, the locus acquisition unit 121 acquires a set of coordinates and time information. In this example, the locus acquisition unit 121 repeatedly acquires the coordinates at a constant sampling cycle. In this way, the locus acquisition unit 121 acquires a plurality of sets of coordinate and time information, that is, loci of the touch position. In this example, the locus acquisition unit 121 acquires the time information from the internal timer (not shown) of the information processing device 10, so that the input device 20 does not have to output the time information. In another example, the input device 20 may output a set of coordinates and time information to the information processing device 10. In this case, the locus acquisition unit 121 does not have to acquire the time information from the internal timer (not shown).

In step S103, the coordinate conversion unit 122 converts the coordinates obtained from the position signal into the coordinates on the surface of the virtual sphere. In this example, since the detection surface and the virtual sphere have similar figures, the coordinate conversion process is simply a process of converting the radius r2 into r1. Prior to the coordinate conversion process, zero point alignment (calibration) is performed in advance between the coordinate system representing the position on the detection surface and the coordinate system representing the position in the virtual space. Further, in step S103, the time information is maintained as it is.

In step S104, the locus calculation unit 123 calculates the locus of the coordinates in the virtual space. Specifically, the locus calculation unit 123 corrects the locus input from the input device 20 to a smoother locus by interpolating the coordinates obtained in step S103, and creates locus data. The locus calculated here represents the time change of the coordinates (r1, θ, φ). The locus is calculated, for example, by linearly interpolating or spline-interpolating the discrete coordinates obtained by sampling (for example, the coordinates acquired at time t1 and the coordinates acquired at time t2), or by using a predetermined function. The interpolation point is calculated by approximating. Here, the following steps will be described with reference to an example in which the points on the locus calculated by the locus calculation unit 123 include the coordinates (r1, θ1, φ1).

It is desirable that the interpolation point of the time information is also calculated by the same method as the method of calculating the interpolation point of the coordinates. Further, when a smoother locus is required, the locus does not have to include the coordinates themselves obtained from step S103.

In step S105, the output unit 124 outputs the locus data to the localization parameter calculation unit 13. In step S106, the localization parameter calculation unit 13 processes the trajectory data and calculates the localization parameter for localizing the sound image at a specific position when the time information reaches the designated time. Then, the localization parameter calculation unit 13 outputs the calculated localization parameter to the localization processing unit 14. In this embodiment, for example, the localization parameters for localizing a predetermined sound image to the coordinates (r1, θ1, φ1) are calculated. The localization parameter is used in the localization processing unit 14. The localization processing unit 14 performs sound image localization processing for localizing the sound image of the input sound signal by a predetermined method. The sound image localization process includes, for example, processing using a head-related transfer function (HRTF), panning for controlling gain and delay, and the like. The localization parameters are parameters that specify the head-related transfer function, parameters that specify gain and delay, and the like. The localization parameter calculation unit 13 reads out information indicating the shape, size, and speaker arrangement of the theater stored in the storage unit 15, and calculates the localization parameters with the position of the listener as the center of the virtual sphere (listener). The position of is set to the center of the virtual sphere by default.)

In step S107, the localization processing unit 14 performs sound image localization processing on the input sound signal using the localization parameters output from the localization parameter calculation unit 13. The localization processing unit 14 outputs an output sound signal in which the sound image of the input sound signal is localized at a position corresponding to the position signal output from the input device 20. In step S108, the display unit 16 updates the display of the moved sound image.

FIG. 5 is a diagram illustrating an image displayed on the display unit 16. The image displayed by the display unit 16 includes an image object 91 showing an acoustic space and an image object 92 showing a sound image of an input sound signal. The image object 92 is displayed at a position corresponding to the sound image localization of the input sound signal in the acoustic space. As the sound image localization moves, the image object 92 moves on the screen in real time while drawing a locus T. With this image, the user can touch the detection surface of the input device 20 while visually confirming the sound image localization.

See FIG. 4 again. In step S109, the locus acquisition unit 121 determines whether the end condition of the locus input is satisfied. The locus input end condition is, for example, a condition that the user has explicitly instructed to end the locus input, or a condition that the touch position has not been input for a predetermined time or longer. When the end condition of the locus input is satisfied (S109: YES), the locus acquisition unit 121 shifts the process to step S110. If the end condition is not satisfied (S109: NO), the locus acquisition unit 121 shifts the process to step S102. The processes of steps S102 to S109 are repeatedly executed at a predetermined cycle. Since the position signal input from the input device 20 is immediately used for the sound image localization processing of the input sound signal, the position of the sound image of the input sound signal moves in real time according to the user's operation on the input device 20. According to the present embodiment, the sound image can be moved in the three-dimensional space by a more intuitive operation input of tracing the detection surface with a finger.

In step S110, the output unit 124 outputs the locus data. The storage unit 15 stores the locus data output from the output unit 124. The locus data includes time-series coordinates when the sound image moves on the virtual sphere and time information of each coordinate. For example, by setting the time information of the first coordinate to 0 and expressing the time information of other coordinates, it is possible to define the locus of sound image localization and the speed at which the locus changes.

The information processing device 10 performs various processes using the trajectory data obtained as described above. As an example, the information processing device 10 reproduces the locus of the sound image according to the locus data. The details are as follows. The user selects the locus data to be reproduced from the data stored in the storage unit 15. The locus control unit 12 reads the selected locus data from the storage unit 15. The locus control unit 12 outputs the read locus data to the localization parameter calculation unit 13. The localization parameter calculation unit 13 calculates the localization parameter and outputs the calculated localization parameter to the localization processing unit 14. The localization processing unit 14 performs processing for localizing the sound image of the input sound signal according to the input localization parameter. According to this example, the locus of the sound image (actually the locus data) input via the input device 20 is stored in the storage unit, and the locus data is read out when necessary, so that the sound image can be easily localized. can.

Here, an example has been described in which the information processing device 10 has both a function of moving the sound image in real time and a function of reading the trajectory data saved in advance, but one of these two functions is omitted. May be done. For example, the information processing device 10 may not have a function of moving a sound image in real time, or may not have a function of reading out previously saved locus data.

3. 3. Modifications The present invention is not limited to the above-described embodiment, and various modifications can be performed. Hereinafter, some modification examples will be described. Two or more of the following modifications may be used in combination.

3-1. Modification 1
In the above embodiment, the listener's position is fixed at the center of the virtual sphere. However, the listener's position may be moved according to the user's instructions. In this case, the information processing device 10 has a position acquisition unit that acquires the position of the listener. The position acquisition unit acquires information that specifies the position of the listener, which is input by the user via the operation unit 11. The localization parameter calculation unit 13 calculates the localization parameter so that a desired localization sound image can be obtained at the designated listener's position. It should be noted that the position of the listener does not have to be fixed during the execution of the process of localizing the sound image of the input sound signal, and the listener may move. If the listener's position is specified by the user, the locus data may include information indicating the listener's position. In this case, the display unit 16 may display an image object representing the position of the listener in addition to the image object 91 and the image object 92. Alternatively, the locus data does not include the position of the listener, and the user may specify the position of the listener again when the locus data is generated or when the locus data is read from the storage unit.

3-2. Modification 2
The shape of the detection surface of the input device 20 is not limited to the spherical surface illustrated in the embodiment or the shape having a part of the spherical surface. The detection surface may be, for example, a hexahedron arranged obliquely with respect to the listener. By tracing a straight line with a finger on one of the surfaces, it is possible to input the locus of the sound image moving on the inclined straight line to the listener. Alternatively, the detection surface may be conical. By tracing the surface of the cone spirally with a finger, it is possible to input a spiral trajectory whose diameter gradually increases (or decreases).

FIG. 6 is a diagram showing another example of the configuration of the input device 20. In this example, the input device 20 has a plurality of actuators 24 arranged in a matrix. The actuator 24 is displaced according to a user instruction under the control of the controller 23. The displacement of the actuator 24 is specified by data input from another device such as the information processing device 10. This data is described in a data format such as STL (Standard Triangulated Language). The sensor film 22 is made of an elastic material. When the actuator 24 is displaced according to the input data, the sensor film 22 expands and contracts along the actuator 24 to form a detection surface having a specified shape. After the shape of the detection surface is changed, the touch position and the coordinates of the detection surface are calibrated. As described above, the locus control unit 12 has a virtual solid 19, but the virtual solid 19 in this example has a similar shape to the detection surface of the changed shape. According to this example, the detection surface can be formed into a desired shape, and a more complicated locus can be input. For example, according to this example, the shape of the detection surface can be changed to the shape of a structure (for example, a castle) used in a video game, and the sound image can be localized along the shape of the structure.

3-3. Modification 3
The shape or size of the virtual solid 19 in the locus control unit 12 may be changed during the input of the locus. For example, in an example in which the virtual solid 19 has a fixed shape (for example, a hemisphere) as in the embodiment, the radius of the virtual solid 19 responds to a user's instruction input or an information processing device while inputting a locus. It may be changed automatically by 10. Alternatively, in the example in which the shape of the detection surface is variable as in the modified example 2, the shape of the virtual solid 19 may change while the locus is being input. In this way, by changing the shape or size of the virtual solid 19 while inputting the locus from the input device 20, it is possible to easily input a complicated locus.

3-4. Modification 4
When the information processing device 10 has a function of reproducing the locus, the information processing device 10 may further have a process of editing the locus. The locus control unit 12 edits the locus according to the user's operation input via the operation unit 11. Editing the trajectory includes, for example, changing the position on the time axis and changing the spatial position.

3-5. Modification 5
The method of inputting the locus via the input device 20 is not limited to the method of tracing the detection surface with a finger. For example, the user may input the locus by touching the detection surface discretely with a finger. The locus control unit 12 obtains a smooth locus by interpolating these discrete points (for example, by linear interpolation or spline interpolation). Further, the locus control unit 12 may generate a locus by interpolating so that the specified coordinates exist on the locus among the plurality of coordinates discretely touched by the finger.

3-6. Other Deformation Examples The detection surface and the virtual solid 19 do not have to be similar figures. The detection surface and the virtual solid 19 may have any shape as long as the position on the detection surface and the position on the surface of the virtual solid 19 can be associated one-to-one.

The virtual object whose locus is controlled by the operation input via the input device 20 is not limited to the sound image in the acoustic space. The trajectory of an object other than a sound image, such as an image, may be controlled.

Some of the functions of the information processing system 1 illustrated in FIG. 1 may be omitted. For example, the information processing system 1 does not have to have the display unit 16 or the storage unit 15.

The correspondence between the function of the information processing system 1 and the hardware is not limited to that illustrated in the embodiment. For example, the functions corresponding to the operation unit 11, the locus control unit 12, and the localization processing unit 14 may be mounted on the input device 20. Alternatively, some functions of the information processing apparatus 10 may be implemented in a server on the network. For example, the storage unit 15 may be on the server, and a plurality of information processing systems 1 may share the storage unit 15. Further, in the above-described embodiment, the functions of the locus control unit 12, the localization parameter calculation unit 13, and the localization processing unit 14 are implemented as software using a general-purpose processor, but may be implemented as hardware.

1 ... Information processing system, 10 ... Information processing device, 11 ... Operation unit, 12 ... Trajectory control unit, 13 ... Localization parameter calculation unit, 14 ... Localization processing unit, 15 ... Storage unit, 16 ... Display unit, 19 ... Virtual solid , 20 ... Input device, 21 ... Housing, 22 ... Sensor membrane, 23 ... Controller, 24 ... Actuator, 91 ... Image object, 92 ... Image object, 101 ... CPU, 102 ... Memory, 104 ... I / O IF, 105 ... Keyboard, 106 ... Display, 121 ... Trajectory acquisition unit, 122 ... Coordinate conversion unit, 123 ... Trajectory calculation unit, 124 ... Output unit

Claims (6)

A locus acquisition unit that acquires a locus instructed on the detection surface from an input device having a detection surface that detects three-dimensional coordinates, and a locus acquisition unit.
An information processing device having an output unit that outputs a locus in which the localization of a sound image moves in a virtual space using the locus acquired by the locus acquisition unit. It has a locus calculation unit that calculates a locus in which the localization of the sound image moves in the virtual space by using the locus acquired by the locus acquisition unit.
The information processing device according to claim 1, wherein the output unit outputs a locus calculated by the locus calculation unit. It has a coordinate conversion unit that converts the coordinates included in the trajectory acquired by the trajectory acquisition unit into the coordinates on the virtual solid set in the virtual space.
The information processing apparatus according to claim 2, wherein the locus calculation unit calculates a locus moving in the virtual space using the coordinates converted by the coordinate conversion unit. The virtual space is an acoustic space,
The information processing device according to any one of claims 1 to 3, wherein the sound image is a sound image in the acoustic space. A position acquisition unit that acquires the position of the listener in the acoustic space, and
The information according to claim 4, further comprising a localization parameter calculation unit that calculates parameters for controlling sound image localization using the position of the listener acquired by the position acquisition unit and the locus output by the output unit. Processing equipment. A step of acquiring a locus instructed on the detection surface from an input device having a detection surface for detecting three-dimensional coordinates, and
An information processing method including a step of outputting a locus in which the localization of a sound image moves in a virtual space using the acquired locus.

Images