JP7824173B2 - Tactile presentation device and program - Google Patents

Description

The present invention relates to a tactile presentation device and program that generates information for presenting tactile stimuli to a viewer of a video via a tactile device.

Traditionally, video has primarily been a visual and auditory medium, but technology that presents tactile stimuli as a third sense is attracting attention. For example, a body-sensing audio system is known in which a chair is equipped with a mechanism that presents tactile stimuli linked to music, and a tactile device that converts sound into vibrations is also known.

Specifically, this bodily sensation audio system incorporates a vibrator built into a chair, extracts low-frequency components from music, and converts the low-frequency components into vibrational tactile information using the vibrator, thereby presenting vibrational tactile stimulation to the listener (see, for example, Patent Document 1).

In addition to haptic audio systems that incorporate vibrators into chairs, technologies are also known that present tactile stimuli such as vibrations and somatosensory stimuli such as a sense of movement in addition to conventional audio-visual content at theme parks, movie theaters, etc. Furthermore, technology is known that uses broadcasting and communication integration services to transmit recorded haptic information from vibrations via communication in addition to the audio-visual content of television broadcasts.

In addition, technology is also known for game devices that use visual displays, which provide players with a bodily sensation in addition to the visual display (see, for example, Patent Document 2). Specifically, this game device outputs a signal amplified by a high-power amplifier to a low-frequency speaker at the timing of a specific visual display, thereby providing the player with the bodily sensation of a low-frequency sound source through the low-frequency speaker.

Furthermore, technology is known for a bodily sound system that does not cause discomfort or a feeling of oppression to the viewer even when used for long periods of time (see, for example, Patent Document 3). Specifically, this bodily sound system includes a seat with a backrest and a seat portion, a band splitting circuit that splits an input audio signal into bands and outputs a first audio signal and a second audio signal, a first vibration element disposed within the backrest that vibrates in response to the first audio signal and whose vibration direction is parallel to the user-facing surface of the backrest, and a second vibration element disposed within the seat that vibrates in response to the second audio signal and whose vibration direction is parallel to the user-facing surface of the seat.

In this way, by stimulating the third sense, touch, in addition to sight and hearing while watching a video, it is possible to achieve a more immersive and realistic viewing experience. In other words, by inputting audio signals, converting them into tactile information, and presenting continuous tactile stimuli, it is possible to enhance the sense of immersion and realism in the video content.

Attempts to convert such audio signals into tactile information and present tactile stimuli to viewers have been made for some time. Hereinafter, this method of inputting audio signals, converting them into tactile information, and presenting tactile stimuli will be referred to as the "audio input method."

An example of this "voice input method" is a chair-type tactile presentation system. This chair-type tactile presentation system is linked to an image of a vehicle such as a tram displayed at a 180-degree viewing angle on a flexible display, and the chair-type tactile device converts the audio signal into tactile information to present tactile stimuli. This allows for a high level of immersion through visual stimuli from the image displayed on the flexible display and tactile stimuli from vibrations from the seat and feet of the chair.

Japanese Patent Application Publication No. 3-102499 Japanese Patent Application Publication No. 6-339576 Japanese Patent Application Laid-Open No. 2008-141477

The audio input method using the aforementioned body-sensing audio system converts audio signals into tactile information and presents tactile stimuli. Because this audio input method is based on actual audio, it is possible to present tactile stimuli that do not feel unnatural in combination with video and audio.

However, actual audio often includes background sounds such as environmental sounds and background music, so even if actual audio is used, it is not necessarily possible to obtain appropriate vibrations.

Here, if it were possible to control vibrations in accordance with the movements of animals or other objects in the video, it would be possible to provide a powerful and immersive experience.

The present invention has been made to solve the above-mentioned problems, and its purpose is to provide a tactile presentation device and program that generates information for presenting tactile stimuli that contribute to improving the sense of immersion when a viewer watches a video.

To solve the above problem, the tactile presentation device of claim 1 extracts low-frequency audio signals from video and generates information for presenting tactile stimuli to a viewer of the video via a tactile device based on the low-frequency audio signals, and is characterized by comprising: a video analysis unit that detects objects using a predetermined neural network (NN) for each of a plurality of time-series frames included in the video, calculates at least one piece of data from the object's movement amount, area, and magnification rate, and calculates a volume amplification amount based on the at least one piece of data; a volume control unit that extracts the low-frequency audio signals from the video and amplifies the volume of the low-frequency audio signals corresponding to the frames based on the volume amplification amount for the frames calculated by the video analysis unit; and a tactile presentation unit that outputs the low-frequency audio signals, the volume of which has been amplified by the volume control unit, to the tactile device.

Furthermore, the tactile presentation device of claim 2 is the tactile presentation device of claim 1, characterized in that the video analysis unit detects the object using the specified neural network for each of the plurality of frames, calculates the amount of movement, area, and magnification rate of the object, and calculates the volume amplification amount based on the amount of movement, area, and magnification rate of the object.

Furthermore, the tactile presentation device of claim 3 is the tactile presentation device of claim 2, wherein the video analysis unit includes a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals; an object detection processing unit that detects one or more objects using the predetermined NN for each of the plurality of frames sampled by the frame sampling processing unit and estimates rectangular coordinates including the object for each object; and a processing unit that selects an object with the largest area from the one or more objects based on the rectangular coordinates of each object estimated by the object detection processing unit for each of the plurality of frames, and generates first rectangular data regarding the amount of movement of the object and second rectangular data regarding the area of the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame in question. a rectangle calculation unit that calculates shape data and third rectangle data related to the magnification ratio of the object; a maximum value extraction unit that extracts the maximum value of the first rectangle data, the maximum value of the second rectangle data, and the maximum value of the third rectangle data from the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit for the multiple frames, respectively; and a volume amplification calculation unit that normalizes the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by the maximum value of the first rectangle data, the maximum value of the second rectangle data, and the maximum value of the third rectangle data extracted by the maximum value extraction unit, respectively, and calculates the volume amplification amount by weighting the normalized first rectangle data, second rectangle data, and third rectangle data.

Furthermore, the tactile presentation device of claim 4 is the tactile presentation device of claim 2, wherein the video analysis unit includes a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals; an object detection processing unit that detects one or more objects for each of the plurality of frames sampled by the frame sampling processing unit using the predetermined NN and estimates rectangular coordinates including the object for each object; and a processing unit that selects the object with the largest area from the one or more objects based on the rectangular coordinates for each object estimated by the object detection processing unit for each of the plurality of frames, and for the largest object, selects a predetermined number of frames including the frame. a rectangle calculation unit that calculates first rectangle data relating to the amount of movement of the object, second rectangle data relating to the area of the object, and third rectangle data relating to the magnification rate of the object based on the rectangular coordinates of the object in the system; and a volume amplification amount calculation unit that normalizes the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by a predetermined maximum value of the first rectangle data, a predetermined maximum value of the second rectangle data, and a predetermined maximum value of the third rectangle data, respectively, and calculates the volume amplification amount by weighting the normalized first rectangle data, second rectangle data, and third rectangle data.

Furthermore, the tactile presentation device of claim 5 is the tactile presentation device of any one of claims 1 to 4, characterized in that the volume control unit includes a low-frequency audio extraction unit that extracts the low-frequency audio signal, the video signal, and audio signals other than the low-frequency audio signal from the video; a volume amplification control unit that multiplies the low-frequency audio signal extracted by the low-frequency audio extraction unit corresponding to the frame by the volume amplification amount calculated by the video analysis unit to generate a new low-frequency audio signal; a synthesis unit that synthesizes the new low-frequency audio signal generated by the volume amplification control unit, the video signal extracted by the low-frequency audio extraction unit, and audio signals other than the low-frequency audio signal to obtain a volume-controlled image; and the tactile presentation unit extracts the new low-frequency audio signal from the volume-controlled image obtained by the synthesis unit and outputs the new low-frequency audio signal to the tactile device.

Furthermore, the program of claim 6 is characterized in that the computer constituting a tactile presentation device that extracts low-frequency audio signals from video and generates information for presenting tactile stimuli to a viewer of the video via a tactile device based on the low-frequency audio signals functions as: a video analysis unit that detects objects using a predetermined neural network (NN) for each of a plurality of time-series frames included in the video, calculates at least one piece of data from the object's movement amount, area, and magnification rate, and calculates a volume amplification amount based on the at least one piece of data; a volume control unit that extracts the low-frequency audio signals from the video and amplifies the volume of the low-frequency audio signals corresponding to the frames based on the volume amplification amount for the frames calculated by the video analysis unit; and a tactile presentation unit that outputs the low-frequency audio signals, the volume of which has been amplified by the volume control unit, to the tactile device.

As described above, according to the present invention, it is possible to generate information for presenting tactile stimuli that contribute to improving the sense of immersion when a viewer watches a video.

1 is a block diagram illustrating an example of the configuration of a tactile presentation device according to an embodiment of the present invention. 2 is a flowchart illustrating an example of processing performed by the tactile presentation device illustrated in FIG. 1 . FIG. 2 is a block diagram showing an example of the configuration of a video analysis unit. 4 is a flowchart showing an example of processing by the video analysis unit shown in FIG. 3 . 10 is a block diagram showing an example of the configuration of a volume amplification amount calculation processing unit. FIG. FIG. 2 is a block diagram showing an example of the configuration of a volume control unit. 7 is a flowchart showing an example of processing by the volume control unit shown in FIG. 6 . FIG. 10 is a diagram illustrating an example of the configuration of a tactile sense providing unit when playing back video E in 5.1ch format. 10A and 10B are diagrams showing examples of frames of video E and examples of rectangular coordinates k1, k2, and k3. FIG. 10 is a diagram showing the calculation result of the volume amplification amount a(n).

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[Tactile presentation device]
FIG. 1 is a block diagram showing an example of the configuration of a tactile presentation device according to an embodiment of the present invention, and FIG. 2 is a flowchart showing an example of processing performed by the tactile presentation device shown in FIG.

This tactile presentation device 1 includes a video analysis unit 11, a volume control unit 12, and a tactile presentation unit 13. The tactile presentation device 1 calculates a volume amplification amount a from the video E that is proportional to the amount of movement of the object, etc., and controls the volume of the low-frequency audio signal S that is the source of the vibration based on the volume amplification amount a to generate a volume-controlled video E'. It then extracts the controlled low-frequency audio signal S' from the volume-controlled video E' and outputs it to the tactile device 7. This improves the sense of immersion felt by the viewer when watching the video E.

The video analysis unit 11 receives video E containing one or more objects (step S201). Then, the video analysis unit 11 samples video E, detects one or more objects in each of multiple time-series frames contained in video E, selects the object with the largest area, and calculates rectangular data v relating to the object's movement, area, and magnification (step S202).

The multiple time-series frames included in video E may be all of the frames that make up video E, or may be a group of frames sampled at a predetermined interval.

The video analysis unit 11 calculates a volume amplification amount a, which is proportional to the amount of movement, etc., for each of the multiple frames based on the rectangle data v (step S203). Then, the video analysis unit 11 outputs the volume amplification amount a to the volume control unit 12.

As a result, rectangle data v is calculated for each of the multiple frames in time series contained in video E, from the first frame to the last frame, and the volume amplification amount a is calculated. Details of the video analysis unit 11 will be described later.

The volume control unit 12 inputs the volume amplification amount a for each sampled frame from the video analysis unit 11 and stores the volume amplification amount a in the memory 41, which will be described later (step S204). As a result, the volume amplification amount a for each of the multiple frames included in the video E is stored in the memory 41.

The volume control unit 12 determines whether or not an operation to start viewing of video E (the same video as video E input by the video analysis unit 11) has been performed in accordance with the viewer's operation (step S205). If the volume control unit 12 determines in step S205 that an operation to start viewing has not been performed (step S205: N), it waits until such an operation is performed.

If the volume control unit 12 determines in step S205 that an operation to start viewing has been performed (step S205: Y), it inputs video E (step S206). Then, the volume control unit 12 extracts the low-frequency audio signal S, the video signal, and audio signals other than the low-frequency audio signal S from video E (step S207).

For example, if video E includes an audio signal of a low-frequency channel, the volume control unit 12 extracts the audio signal of that channel from video E, thereby extracting it as a low-frequency audio signal S. Also, if video E is made up of a video signal and an audio signal, and the audio signal is a signal including high-frequency and low-frequency components, the volume control unit 12 extracts the low-frequency components from the audio signal included in video E, thereby extracting them as a low-frequency audio signal S.

The volume control unit 12 reads out from the memory 41 (described later) the volume amplification amount a for the frame of the video signal corresponding to the low-frequency audio signal S of the video E extracted in step S207 (step S208). As a result, the volume amplification amounts a stored in the memory 41 corresponding to the low-frequency audio signal S are read out in order.

The volume control unit 12 generates a new low-frequency audio signal S' by amplifying the volume A of the low-frequency audio signal S corresponding to the frame with the volume amplification amount a based on the volume amplification amount a read from the memory 41 (step S209). Here, the low-frequency audio signal S corresponding to the volume amplification amount a is the audio signal from the frame with the volume amplification amount a to just before the next frame of that frame (the frame next to that frame among the multiple frames stored in the memory 41).

Here, the larger the value of the rectangle data v, the larger the value of the volume amplification amount a, and the smaller the value of the rectangle data v, the smaller the value of the volume amplification amount a. Because the rectangle data v is data related to the object's movement amount, area, and magnification rate, the larger these data are, the larger the value of the volume amplification amount a, and the smaller these data are, the smaller the value of the volume amplification amount a.

The larger the value of the volume amplification amount a, the larger the volume A _new of the amplified new low-frequency audio signal S' becomes than the volume A before amplification, and the smaller the value of the volume amplification amount a, the smaller the volume A _new of the amplified new low-frequency audio signal S' becomes than the volume A before amplification.

That is, the greater the amount of movement, area, and magnification rate of the object, the greater the volume A _new of the amplified new low-frequency audio signal S'. Conversely, the smaller the amount of movement, area, and magnification rate of the object, the smaller the volume A _new of the amplified new low-frequency audio signal S'.

The volume control unit 12 obtains a volume-controlled image E' by combining the low-frequency audio signal S' generated in step S209 with the video signal extracted in step S207 and audio signals other than the low-frequency audio signal S (step S210). The volume control unit 12 then outputs the volume-controlled image E' to the haptic presentation unit 13. Details of the volume control unit 12 will be described later.

The haptic presentation unit 13 receives the volume-controlled image E' from the volume control unit 12, extracts a low-frequency audio signal S' from the volume-controlled image E', and outputs the low-frequency audio signal S' to the haptic device 7 (step S211). Details of the haptic presentation unit 13 will be described later.

As a result, the greater the amount of movement, area, and magnification of the object, the greater the low-frequency audio signal S' with a higher volume A that is input to the haptic device 7, thereby increasing the vibration. On the other hand, the smaller the amount of movement, area, and magnification of the object, the greater the low-frequency audio signal S' with a lower volume A that is input to the haptic device 7, thereby decreasing the vibration. In other words, it is possible to control the vibration in accordance with the movement of an object (e.g., an animal) in the video E, providing the viewer with a powerful and immersive experience.

Therefore, when viewing video E, the tactile presentation device 1 can generate information for presenting tactile stimuli that contribute to an enhanced sense of immersion, allowing the viewer to receive vibration stimuli in accordance with the movement of objects within video E, thereby enhancing the sense of immersion. Note that while in Figures 1 and 2, the volume amplification amount a is output from the video analysis unit 11 to the volume control unit 12, in Figure 3 and other figures described below, it will be explained as if the volume amplification amount a(n) is output. n indicates the frame number.

(Video analysis unit 11)
Next, a detailed description will be given of the video analysis unit 11 shown in Fig. 1. Fig. 3 is a block diagram showing an example of the configuration of the video analysis unit 11, and Fig. 4 is a flowchart showing an example of processing by the video analysis unit 11 shown in Fig. 3.

This video analysis unit 11 includes a frame sampling processing unit 21, an object detection processing unit 22, and a volume amplification calculation processing unit 23.

The frame sampling processing unit 21 inputs video E (step S401) and samples video E into multiple frames in time series at predetermined intervals (step S402). Sampling at predetermined intervals reduces the load of subsequent processing, such as calculations. Here, the frame sampling processing unit 21 may sample all frames that make up video E.

The frame sampling processing unit 21 outputs each of the multiple frames (frames 0, ..., n, ..., N) after sampling to the object detection processing unit 22. N is an integer greater than or equal to 1, and n satisfies 0≦n≦N. Frame n indicates the frame with frame number n.

The object detection processing unit 22 inputs each of the multiple frames from the frame sampling processing unit 21. Then, for each of the multiple frames, the object detection processing unit 22 uses a predetermined NN (neural network) to detect objects included in that frame, and estimates the rectangular coordinates and identification class of each detected object or objects (step S403). Then, the object detection processing unit 22 outputs the rectangular coordinates of each object included in that frame to the volume amplification amount calculation processing unit 23.

Rectangle coordinates are used to identify a rectangle that maximally encompasses an object, and consist, for example, of the coordinate values of the upper left vertex and the lower right vertex. In this case, rectangle coordinates may consist of the coordinate value of the upper left vertex, the width in the x-axis direction (horizontal direction), and the height in the y-axis direction (vertical direction).

This gives the rectangular coordinates of each object for each of the N frames (rectangular coordinates of each object for frame 0, ..., rectangular coordinates of each object for frame n, ..., rectangular coordinates of each object for frame N).

The NN used by the object detection processing unit 22 is not limited to a specific network configuration. For example, a configuration similar to the object detection model shown in the following document or an improved version based on this architecture may be used.
[Non-patent literature] Ge Zheng, et al., “Yolox: Exceeding yolo series in 2021.”, arXiv preprint arXiv:2107.08430 (2021).

Figure 9 shows an example frame of video E and examples of rectangular coordinates k1, k2, and k3. The object detection processing unit 22 uses a neural network to detect three objects contained in the frame of video E, and estimates the rectangular coordinates k1, k2, and k3 containing each object and the object's identification class (in this example, the identification class for all objects is "elephant").

Returning to Figures 3 and 4, the volume amplification amount calculation processing unit 23 inputs the rectangular coordinates of each object for each of multiple frames. The volume amplification amount calculation processing unit 23 then performs the processes of steps S404 to S409, which will be described later.

Figure 5 is a block diagram showing an example configuration of the volume amplification amount calculation processing unit 23. This volume amplification amount calculation processing unit 23 includes an object selection unit 31, a rectangle calculation unit 32, a memory 33, a maximum value extraction unit 34, and a volume amplification amount calculation unit 35.

Referring to Figures 4 and 5, the object selection unit 31 inputs the rectangular coordinates of each object for each of the multiple frames, calculates the rectangular area of each object based on the rectangular coordinates of each object, and selects the object with the largest rectangular area from among one or more objects included in the frame (step S404). The object selection unit 31 outputs the rectangular coordinates of the object with the largest rectangular area for each of the multiple frames to the rectangle calculation unit 32.

In the example shown in Figure 9, the object selection unit 31 selects the object with rectangular coordinate k1 that has the largest rectangular area from among three objects having rectangular coordinates k1, k2, and k3, respectively.

Returning to Figures 4 and 5, the rectangle calculation unit 32 receives the rectangular coordinates of the object with the largest rectangular area for each of the multiple frames from the object selection unit 31. Then, for the object with the largest rectangular area for each of the multiple frames, the rectangle calculation unit 32 calculates rectangle data v(n) such as the amount of movement of the object based on the rectangular coordinates of the object (step S405).

Here, the rectangle data v(n) for frame n consists of the movement amount _v1 (n) of the rectangle center, the rectangle area _v2 (n), and the rectangle enlargement rate _v3 (n). The rectangle data v(n) is calculated from rectangle coordinates, which are coordinates for specifying the rectangle that contains the object to the maximum extent. Therefore, the movement amount _v1 (n) of the rectangle center, the rectangle area _v2 (n), and the rectangle enlargement rate _v3 (n) can be used to specify the movement amount, area, and enlargement rate of the object, respectively.

For example, the rectangle calculation unit 32 calculates the movement amount _v1 (n) of the rectangle center and the rectangle enlargement rate _v3 (n) in frame n using the rectangle coordinates of the object with the largest rectangle area in a predetermined number of frames before and after frame n (the same number of rectangle coordinates as the predetermined number). The rectangle calculation unit 32 also calculates the rectangle area _v2 (n) using the rectangle coordinates of the object with the largest rectangle area in frame n.

Here, if no object is detected by the object detection processing unit 22, the rectangle calculation unit 32 sets the rectangle data v(n)=0, that is, _v1 (n)= _v2 (n)= _v3 (n)=0.

The rectangle calculation unit 32 also calculates the overlap rate between the object with the largest rectangular area in frame n-1 (object α) and the object with the largest rectangular area in frame n (object β) based on the coordinates of each rectangle.

When a frame n-1 including an object α is superimposed on a frame n including an object β, if the area of the region where the objects α and β exist is S1 and the area of the region where the objects α and β overlap is S2, the overlap rate is expressed by the following formula.
[Equation 1]
Overlap rate=S2/S1 (1)

If the overlap rate is equal to or less than a predetermined value (e.g., 0.75), the rectangle calculation unit 32 determines that the objects α and β are different, and sets the movement amount _v1 (n) of the rectangle center of object β in frame n to 0, and also sets the rectangle expansion rate _v3 (n) to 0. In other words, if the object with the largest rectangle area changes (if it changes to a different object of the same identification class, or if it changes to an object of a different identification class), the movement amount _v1 (n) of the rectangle center and the rectangle expansion rate _v3 (n) are reset. On the other hand, if the overlap rate is greater than a predetermined value (e.g., 0.75), the rectangle calculation unit 32 determines that the objects α and β are the same.

The rectangle calculation unit 32 stores the rectangle data v(n), that is, the movement amount v ₁ (n) of the rectangle center, the rectangle area v ₂ (n), and the rectangle enlargement ratio v ₃ (n), for each of the multiple frames in the memory 33 (step S406).

As a result, memory 33 stores rectangular data v(n) of multiple frames (all frames) in time series sampled from video E. That is, memory 33 stores rectangular data v(0) of frame 0 ( _v1 (0), _v2 (0), _v3 (0)), ..., rectangular data v(n) of frame n ( _v1 (n), _v2 (n), _v3 (n)), ..., rectangular data v(N) of frame N ( _v1 (N), _v2 (N), _v3 (N)).

Once the rectangular data v(n) of all sampled frames has been stored in memory 33, maximum value extraction unit 34 reads the rectangular data v(n) of all sampled frames from memory 33. Then, maximum value extraction unit 34 extracts the maximum values for each of the rectangle center movement amount _v1 (n), rectangle area _v2 (n), and rectangle enlargement rate _v3 (n) included in this rectangular data v(n), and sets maximum rectangular data _vmax (step S407).

Specifically, the maximum value extraction unit 34 extracts the largest rectangle center movement amount _v1 (0), ..., _v1 (n), ..., _v1 (N) among the rectangle center movement amounts in all frames and designates this as _v1max . The maximum value extraction unit 34 also extracts the largest rectangle area among the rectangle areas _v2 (0), ..., _v2 (n), ..., _v2 (N) among all frames and designates this as _v2max . The maximum value extraction unit 34 also extracts the largest rectangle enlargement rate among the rectangle enlargement rates _v3 (0), ..., _v3 (n), ..., _v3 (N) among all frames and designates this as _v3max . The maximum value extraction unit 34 then sets maximum rectangle data _vmax consisting of the maximum rectangle center movement amount _v1max , the maximum rectangle area _v2max , and the maximum rectangle enlargement rate _v3max .

The maximum value extraction unit 34 outputs the maximum rectangle data v _max , that is, the maximum movement amount v _1max of the rectangle center, the maximum rectangle area v _2max , and the maximum rectangle expansion rate v _3max , to the volume amplification amount calculation unit 35 .

The volume amplification amount calculation unit 35 receives the maximum rectangular data _vmax from the maximum value extraction unit 34. The volume amplification amount calculation unit 35 also reads rectangular data v(n) from the memory 33 sequentially for all sampled frames, normalizes the rectangular data v(n) by the maximum rectangular data _vmax , and calculates the volume amplification amount a(n) based on the normalized rectangular data v(n) (step S408). The volume amplification amount calculation unit 35 then outputs the volume amplification amount a(n) to the volume control unit 12 (step S409).

Specifically, the volume amplification amount calculation unit 35 normalizes the rectangle center movement amount _v1 (n) read from the memory 33 by the maximum rectangle center movement amount _v1max to obtain the normalized rectangle center movement amount _v'1 (n). The volume amplification amount calculation unit 35 also normalizes the rectangle area _v2 (n) read from the memory 33 by the maximum rectangle area _v2max to obtain the normalized rectangle area _v'2 (n). The volume amplification amount calculation unit 35 also normalizes the rectangle expansion rate _v3 (n) read from the memory 33 by the maximum rectangle expansion rate _v3max to obtain the normalized rectangle expansion rate _v'3 (n).

The volume amplification amount calculation unit 35 calculates the volume amplification amount a(n) of frame n using the following formula.
[Equation 2]
a(n) = (w ₁ v' ₁ (n) + w ₂ v' ₂ (n) + w ₃ v' ₃ (n)) / (w ₁ + w ₂ + w ₃ )
...(2)
w ₁ , w ₂ , and w ₃ are weighting coefficients for the preset amount of movement v ₁ (n) of the rectangle center, the rectangle area v ₂ (n), and the rectangle enlargement rate v ₃ (n), respectively.

As a result, the video analysis unit 11 obtains the volume amplification amount a(n) for each of multiple frames in a time series sampled at predetermined intervals from video E.

Figure 10 is a diagram showing the calculation results of the volume amplification amount a(n), which shows the volume amplification amount a(n) calculated by the video analysis unit 11. The vertical axis represents the volume amplification amount a(n), and the horizontal axis represents time (frame number: n).

The video analysis unit 11 calculates the volume amplification amount a(n) shown in Figure 10, and outputs the volume amplification amount a(n) to the downstream volume control unit 12.

(Volume control unit 12)
Next, a detailed description will be given of the volume control unit 12 shown in Fig. 1. Fig. 6 is a block diagram showing an example of the configuration of the volume control unit 12, and Fig. 7 is a flowchart showing an example of processing by the volume control unit 12 shown in Fig. 6.

This volume control unit 12 includes a memory 41, a low-frequency audio extraction unit 42, a volume amplification control unit 43, and a synthesis unit 44.

The volume control unit 12 receives the volume amplification amount a(n) for each sampled frame from the video analysis unit 11 and stores the volume amplification amount a(n) in memory 41 (step S701). As a result, the volume amplification amounts a(n) for multiple frames (all frames) in a time series sampled from video E are stored in memory 41.

The volume control unit 12 determines whether or not an operation to start viewing of video E (the same video as video E input by the video analysis unit 11 shown in FIG. 1) has been performed in accordance with the viewer's operation (step S702). If the volume control unit 12 determines in step S702 that an operation to start viewing has not been performed (step S702: N), it waits until such an operation is performed.

If the volume control unit 12 determines in step S702 that an operation to start viewing has been performed (step S702: Y), the low-frequency audio extraction unit 42 inputs video E and extracts the low-frequency audio signal S, the video signal, and audio signals other than the low-frequency audio signal S from video E (step S703).

In this case, the low-frequency audio extraction unit 42 may extract an audio signal from a sound source that has previously recorded only low frequencies as the low-frequency audio signal S.

For example, in video E, which consists of an 8K video signal and a 22.2ch audio signal, a low-frequency audio signal of 120 Hz or less called an LFE (Low Frequency Effect) channel is used in 0.2ch. In this case, the low-frequency audio extraction unit 42 extracts this LFE audio signal as the low-frequency audio signal S, and by using the LFE audio signal in the downstream haptic device 7, the viewer can receive tactile stimulation that is appropriate for the vibration.

Also, if low-frequency audio such as LFE is not available, a mixed audio signal may be used. In other words, the low-frequency audio extraction unit 42 can generate a pseudo-low-frequency audio signal S by equalizing the mixed audio signal to emphasize the low-frequency components and suppress the high-frequency components.

The low-frequency audio extraction unit 42 outputs the low-frequency audio signal S to the volume amplification control unit 43, and outputs the video signal and audio signals other than the low-frequency audio signal S (other audio signals) to the synthesis unit 44.

The volume amplification control unit 43 inputs the low-frequency audio signal S from the low-frequency audio extraction unit 42 and reads the volume amplification amount a(n) for all sampled frames from the memory 41, in order from the first (step S704). Then, the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n) read from the memory 41, and generates a new low-frequency audio signal S' (step S705). The volume amplification control unit 43 outputs the new low-frequency audio signal S' to the synthesis unit 44.

For example, the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S of the frame corresponding to the volume amplification amount a(n) so that it is proportional to the volume amplification amount a(n) using the following equation, and generates a new low-frequency audio signal S' having a volume A _new .
[Equation 3]
A _new =a(n)A+c...(3)

Here, c is a constant (minimum value) for raising the volume _Anew . The low-frequency audio signal S of the frame corresponding to the volume amplification amount a(n) is the audio signal from the time point of the frame corresponding to the volume amplification amount a(n) to the time point immediately before the next frame.

As a result, the smaller the volume amplification amount a(n), i.e., the smaller the movement amount _v1 (n), rectangular area _v2 (n), and rectangular expansion rate _v3 (n) of the rectangular center of the object with the largest rectangular area, the smaller the volume A of the low-frequency audio signal S. On the other hand, the larger the volume amplification amount a(n), i.e., the larger the movement amount _v1 (n), rectangular area _v2 (n), and rectangular expansion rate _v3 (n) of the rectangular center of the object with the largest rectangular area, the larger the volume A of the low-frequency audio signal S.

The synthesis unit 44 inputs the low-frequency audio signal S' from the volume amplification control unit 43, and inputs the video signal and audio signals other than the low-frequency audio signal S from the low-frequency audio extraction unit 42. The synthesis unit 44 then synthesizes the low-frequency audio signal S', the video signal, and the audio signals other than the low-frequency audio signal S to obtain a volume-controlled image E' (step S706). The synthesis unit 44 outputs the volume-controlled image E' to the haptic presentation unit 13 (step S707).

As a result, when the volume amplification amount a(n) in a frame of video E is small, the volume control unit 12 reduces the volume A of the low-frequency audio signal S, thereby generating a volume-controlled video E' including a low-frequency audio signal S' with a new volume A _new . Also, when the volume amplification amount a(n) in a frame of video E is large, the volume A of the low-frequency audio signal S is increased, thereby generating a volume-controlled video E' including a low-frequency audio signal S' with a new volume A _new .

The volume control unit 12 shown in FIG. 7 may further include a smoothing unit before the memory 41. The smoothing unit receives the volume amplification amount a(n) for each sampled frame from the video analysis unit 11.

If the volume amplification amount a(n) is unstable (for example, if the rate of change of the volume amplification amount a(n) is greater than or equal to a predetermined value), the smoothing unit smoothes the volume amplification amount a(n) of the frame in question using a predetermined number of frames before and after (nearby). The smoothing unit then stores the smoothed volume amplification amount a(n) in memory 41.

(Tactile presentation unit 13)
Next, the tactile sense providing unit 13 shown in Fig. 1 will be described in detail. Fig. 8 is a diagram showing an example of the configuration of the tactile sense providing unit 13 when playing back video E in 5.1ch format. This example shows a case where audio signals of three channels, L, R, and LFE, of audio signals in 5.1ch format (L, R, C, SL, SR, LFE) are used.

The tactile sensation providing unit 13 includes an extraction unit 51 and an amplification unit 52. Note that in Figure 8, the configuration for amplifying the video signal and the L and R audio signals is omitted.

The extraction unit 51 inputs the volume-controlled video E' from the volume control unit 12 and extracts the LFE audio signal as a low-frequency audio signal S' from the volume-controlled video E', as well as the video signal and L (left) and R (right) audio signals. The extraction unit 51 outputs the LFE audio signal to the amplifier unit 52, which amplifies the LFE audio signal and outputs it to the haptic device 7 and speaker 9. The extraction unit 51 also outputs the video signal to the display 8 and the L and R audio signals to the speaker 9.

The haptic device 7 receives the LFE audio signal from the amplifier 52, and presents smaller vibrations to the viewer as the volume of the LFE audio signal decreases, and larger vibrations to the viewer as the volume of the LFE audio signal increases.

As a result, the viewer can view video E and receive tactile stimulation linked to video E via the haptic device 7, which receives the LFE audio signal from the haptic presentation unit 13, the display 8, which receives the video signal, and the speaker 9, which receives the L, R, and LFE audio signals.

In particular, in a frame of video E, when the movement amount _v1 (n) of the rectangular center of the object with the largest rectangular area, the rectangular area _v2 (n), and the rectangular expansion rate _v3 (n) are small, the volume A of the low-frequency audio signal S is reduced, allowing the viewer to receive a weaker tactile stimulus than usual. On the other hand, when the movement amount _v1 (n) of the rectangular center of the object with the largest rectangular area, the rectangular area _v2 (n), and the rectangular expansion rate _v3 (n) are large, the volume A of the low-frequency audio signal S is increased, allowing the viewer to receive a stronger tactile stimulus than usual.

Here, the low-frequency audio signal S', which is an LFE audio signal, is output to the haptic device 7 and converted into a tactile stimulus because humans can generally only receive tactile stimuli at low frequencies of around 200 Hz or less, and cannot receive appropriate tactile stimuli when the frequency of the audio signal is high.

The extraction unit 51 may extract the LFE audio signal as a low-frequency audio signal S' from the volume-controlled video E', as well as the video signal and L (left) and R (right) audio signals, and output the LFE audio signal to the haptic device 7 via the amplifier unit 52, output the video signal to the display 8, and output the L (left) and R (right) audio signals to the speaker 9.

As described above, according to the tactile presentation device 1 of the embodiment of the present invention, the video analysis unit 11 samples the video E to acquire multiple frames in time series, detects the rectangular coordinates of one or multiple objects for each of the multiple frames, selects the object with the largest area, and calculates the rectangular data v of that object (the movement amount _v1 (n) of the rectangle center, the rectangular area _v2 (n), and the rectangular magnification rate _v3 (n)).The video analysis unit 11 then extracts the maximum rectangular data _vmax from the rectangular data v of all the sampled frames, and calculates the volume amplification amount a(n) based on the rectangular data v and the maximum rectangular data _vmax .

When a viewing start operation is performed, the volume control unit 12 extracts the low-frequency audio signal S from the video E, and multiplies the volume A of the low-frequency audio signal S by the volume amplification amount a(n) to generate a new low-frequency audio signal S' by amplifying the volume A of the low-frequency audio signal S. The volume control unit 12 then synthesizes a volume-controlled video E' that includes the low-frequency audio signal S'.

The haptic presentation unit 13 extracts a low-frequency audio signal S' from the volume-controlled image E' and outputs the low-frequency audio signal S' to the haptic device 7.

As a result, the greater the movement amount _v1 (n) of the rectangular center of the object included in the frame, the greater the rectangular area _v2 (n), and the rectangular expansion rate _v3 (n), the greater the volume amplification amount a(n), and a low-frequency audio signal S' with a new volume A _new where the volume A is increased is generated, thereby increasing the vibration caused by the haptic device 7. On the other hand, the smaller the movement amount _v1 (n), the rectangular area _v2 (n), and the rectangular expansion rate _v3 (n) of the rectangular center of the object included in the frame, the smaller the volume amplification amount a(n), and a low-frequency audio signal S' with a new volume A _new where the volume A is decreased is generated, thereby reducing the vibration caused by the haptic device 7.

In other words, the viewer can receive vibration stimulation via the haptic device 7 in response to the movement of an object included in the video E. For example, the greater the movement of the object, the greater the vibration stimulation the viewer will receive, and the smaller the movement of the object, the smaller the vibration stimulation the viewer will receive.

Therefore, when viewing video E, the tactile presentation device 1 can generate information for presenting tactile stimuli that contribute to improving the sense of immersion, allowing the viewer to receive vibration stimuli in accordance with the movement of objects in video E, thereby improving the sense of immersion.

The present invention has been described above using embodiments, but the present invention is not limited to these embodiments and can be modified in various ways without departing from the technical concept thereof.

For example, in the above embodiment, the volume amplification amount calculation unit 35 of the volume amplification amount calculation processing unit 23 provided in the video analysis unit 11 normalizes the movement amount _v1 (n) of the object's rectangular center, the rectangular area _v2 (n), and the rectangular enlargement rate _v3 (n) by their respective maximum values, and calculates the volume amplification amount a(n) by weighting the normalized movement amount _v'1 (n), rectangular area _v'2 (n), and rectangular enlargement rate _v'3 (n). Alternatively, the volume amplification amount calculation unit 35 may calculate the volume amplification amount a(n) using data on at least one of the movement amount _v1 (n) of the rectangle center, the rectangular area _v2 (n), and the rectangular enlargement rate _v3 (n).

For example, the volume amplification amount calculation unit 35 normalizes two pieces of data from among the movement amount _v1 (n) of the rectangle center, the rectangle area _v2 (n), and the rectangle magnification rate _v3 (n) by their respective maximum values, and then calculates the volume amplification amount a(n) by weighting the two normalized pieces of data.

Furthermore, the volume amplification amount calculation unit 35 normalizes one of the data of the rectangle center movement amount _v1 (n), the rectangle area _v2 (n), and the rectangle expansion rate _v3 (n) by the maximum value thereof, and sets the normalized data as the volume amplification amount a(n). In this case, the volume amplification amount calculation unit 35 may calculate the volume amplification amount a(n) so that it is proportional to the value of the normalized data.

2 and 7, the rectangle calculation unit 32 of the volume amplification amount calculation processing unit 23 provided in the video analysis unit 11 of the tactile presentation device 1 stores the rectangle data v(n) of all sampled frames in the memory 33. The maximum value extraction unit 34 reads the rectangle data v(n) of all sampled frames from the memory 33 and extracts the maximum rectangle data v _max , and the volume amplification amount calculation unit 35 calculates the volume amplification amount a(n) based on the rectangle data v(n) and the maximum rectangle data v _max . When the viewer starts viewing, the volume amplification control unit 43 of the volume control unit 12 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n).

On the other hand, the volume amplification amount calculation unit 35 may calculate the volume amplification amount a(n) using preset maximum rectangular data v _max instead of using the maximum rectangular data v _max extracted by the maximum value extraction unit 34. In this case, the volume amplification amount calculation processing unit 23 of the video analysis unit 11 does not need to include the memory 33 and the maximum value extraction unit 34 in the configuration example shown in FIG.

In other words, the volume amplification control unit 43 amplifies the volume A of the low-frequency audio signal S based on the volume amplification amount a(n) calculated by the time interleaving processing unit 35, without the volume control unit 12 waiting for the viewer to start viewing.

This allows a series of processes to be performed in real time, from the video analysis unit 11 inputting the video E and calculating the volume amplification amount a(n), the volume control unit 12 amplifying the volume A of the low-frequency audio signal S to generate the volume-controlled video E', and the haptic presentation unit 13 outputting the low-frequency audio signal S' to the haptic device 7.

In the examples shown in Figures 2 and 7, when the volume control unit 12 of the tactile presentation device 1 determines that the viewer has performed an operation to start viewing, it extracts a low-frequency audio signal S, etc. from the video E and amplifies the volume A of the low-frequency audio signal S to generate a volume-controlled video E', and the tactile presentation unit 13 extracts the low-frequency audio signal S' from the volume-controlled video E' and outputs it to the tactile device 7.

In response to this, the volume control unit 12 may store the generated volume-controlled image E' in a memory not shown in Figure 6, and the tactile presentation unit 13 may repeatedly use the volume-controlled image E' stored in the memory each time the viewer starts viewing.

The hardware configuration of the tactile presentation device 1 can be a conventional computer. The tactile presentation device 1 is composed of a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

The functions of the video analysis unit 11, volume control unit 12, and tactile presentation unit 13 provided in the tactile presentation device 1 are each realized by having the CPU execute a program that describes these functions.

These programs are stored on the storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory, and can also be sent and received over a network.

REFERENCE SIGNS LIST 1 haptic presentation device 7 haptic device 8 display 9 speaker 11 video analysis unit 12 volume control unit 13 haptic presentation unit 21 frame sampling processing unit 22 object detection processing unit 23 volume amplification amount calculation processing unit 31 object selection unit 32 rectangle calculation unit 33, 41 memory 34 maximum value extraction unit 35 volume amplification amount calculation unit 42 low-frequency audio extraction unit 43 volume amplification control unit 44 synthesis unit 51 extraction unit 52 amplifier E video E' volume-controlled video k1, k2, k3 rectangle coordinate a volume amplification amount v rectangle data v _max maximum rectangle data _v1 (n) movement amount of rectangle center _v2 (n) rectangle area _v3 (n) rectangle enlargement rate S, S' low-frequency audio signal A, A _new volume α, β object

Claims (6)

1. A tactile presentation device that extracts a low-frequency audio signal from a video and generates information for presenting a tactile stimulus to a viewer of the video via a tactile device based on the low-frequency audio signal,
a video analysis unit that detects an object using a predetermined neural network (NN) for each of a plurality of time-series frames included in the video, calculates at least one data item among a movement amount, an area, and an enlargement rate of the object, and calculates a volume amplification amount based on the at least one data item;
a volume control unit that extracts the low-frequency audio signal from the video and amplifies the volume of the low-frequency audio signal corresponding to the frame based on the volume amplification amount for the frame calculated by the video analysis unit;
a haptic presentation unit that outputs the low-frequency audio signal, the volume of which has been amplified by the volume control unit, to the haptic device;
A tactile presentation device comprising: The tactile presentation device according to claim 1 ,
The video analysis unit
A tactile presentation device characterized in that, for each of the plurality of frames, the object is detected using the specified NN, the amount of movement, area and magnification rate of the object are calculated, and the amount of volume amplification is calculated based on the amount of movement, area and magnification rate of the object. The tactile presentation device according to claim 2,
The video analysis unit
a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals;
an object detection processing unit that detects one or more objects using the predetermined neural network for each of the plurality of frames sampled by the frame sampling processing unit, and estimates rectangular coordinates including each object;
a rectangle calculation unit that selects, for each of the plurality of frames, an object having a largest area from among the one or more objects based on the rectangular coordinates of each of the objects estimated by the object detection processing unit, and calculates, for the largest object, first rectangular data relating to an amount of movement of the object, second rectangular data relating to an area of the object, and third rectangular data relating to an enlargement rate of the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame;
a maximum value extraction unit that extracts a maximum value of the first rectangle data, a maximum value of the second rectangle data, and a maximum value of the third rectangle data from the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit in the plurality of frames, respectively;
a volume amplification amount calculation unit that normalizes the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by the maximum value of the first rectangle data, the maximum value of the second rectangle data, and the maximum value of the third rectangle data extracted by the maximum value extraction unit, respectively, and calculates the volume amplification amount by weighting the normalized first rectangle data, the second rectangle data, and the third rectangle data;
A tactile presentation device comprising: The tactile presentation device according to claim 2,
The video analysis unit
a frame sampling processing unit that samples the video into a plurality of frames in time series at predetermined intervals;
an object detection processing unit that detects one or more objects using the predetermined neural network for each of the plurality of frames sampled by the frame sampling processing unit, and estimates rectangular coordinates including each object;
a rectangle calculation unit that selects, for each of the plurality of frames, an object having a largest area from among the one or more objects based on the rectangular coordinates of each of the objects estimated by the object detection processing unit, and calculates, for the largest object, first rectangular data relating to an amount of movement of the object, second rectangular data relating to an area of the object, and third rectangular data relating to an enlargement rate of the object based on the rectangular coordinates of the object in a predetermined number of frames including the frame;
a volume amplification amount calculation unit that calculates the volume amplification amount by normalizing the first rectangle data, the second rectangle data, and the third rectangle data calculated by the rectangle calculation unit by a predetermined maximum value of the first rectangle data, a predetermined maximum value of the second rectangle data, and a predetermined maximum value of the third rectangle data, respectively, and weighting the normalized first rectangle data, second rectangle data, and third rectangle data;
A tactile presentation device comprising: The tactile presentation device according to any one of claims 1 to 4,
The volume control unit
a low-frequency audio extraction unit that extracts the low-frequency audio signal, the video signal, and an audio signal other than the low-frequency audio signal from the video;
a volume amplification control unit that multiplies the volume amplification amount for the frame calculated by the video analysis unit by the low-frequency audio signal extracted by the low-frequency audio extraction unit corresponding to the frame to generate a new low-frequency audio signal;
a synthesis unit that synthesizes the new low-frequency audio signal generated by the volume amplification control unit, the video signal extracted by the low-frequency audio extraction unit, and an audio signal other than the low-frequency audio signal to obtain a volume-controlled video;
The tactile sense providing unit is
A tactile presentation device characterized in that the new low-frequency audio signal is extracted from the volume-controlled image obtained by the synthesis unit, and the new low-frequency audio signal is output to the tactile device. a computer constituting a tactile presentation device that extracts a low-frequency audio signal from a video and generates information for presenting a tactile stimulus to a viewer of the video via a tactile device based on the low-frequency audio signal;
a video analysis unit that detects an object using a predetermined neural network for each of a plurality of time-series frames included in the video, calculates at least one piece of data among a movement amount, an area, and an enlargement rate of the object, and calculates a volume amplification amount based on the at least one piece of data;
a volume control unit that extracts the low-frequency audio signal from the video and amplifies the volume of the low-frequency audio signal corresponding to the frame based on the volume amplification amount for the frame calculated by the video analysis unit; and
a program for causing the low-frequency audio signal, the volume of which has been amplified by the volume control unit, to function as a tactile presentation unit that outputs the low-frequency audio signal to the tactile device;