JP7731082B2 - Server device for communication system in virtual space and client device for communication system in virtual space - Google Patents

Description

The present invention relates to a server device for a virtual space communication system and a client device for a virtual space communication system, and in particular to a server device for a virtual space communication system and a client device for a virtual space communication system that are executed using a web browser.

Patent Literature 1 discloses an audio processing device that enables a listener to distinguish audio corresponding to a desired sound source through natural movements. This audio processing device controls the output of audio signals corresponding to multiple virtual sound sources virtually arranged around the listener, and includes a state determination unit that determines whether movement in the listener's direction, which indicates the orientation of the listener, has become stationary, and an output control unit that controls the volume of audio signals corresponding to the virtual sound sources included in a listening range set so that the listener's direction is centered, to be relatively larger than the volume of audio signals corresponding to the virtual sound sources not included in the listening range, and that reduces the listening range when it is determined that the listening range has become stationary.
Patent No. 5929455

Incidentally, the emergence of COVID-19 has completely changed our lives, leading to an increase in working from home and remote meetings, and even the emergence of services such as virtual exhibition halls. When working from home, it is not uncommon for companies to lend their employees personal computers, tablet devices, and smartphones (hereinafter referred to as "PCs, etc.") to use.

However, corporations have taken measures such as restricting the launch of some application programs installed on PCs, etc., to prevent information leaks that may occur when employees use PCs, etc. outside the office.

As a result, in the case of application programs that provide communication systems, it is conceivable that the programs may not be able to be used outside the office due to usage restrictions. As it is expected that opportunities to carry out work outside the office will continue to increase in the future, it is not desirable to provide communication systems that use application programs.

Therefore, the objective of the present invention is to provide a communication system that does not use application programs.

In order to solve the above problems, the server device for a communication system in a virtual space of the present invention comprises:
a video signal control means for controlling a virtual space displayed using a web browser on a single client device, in which a plurality of avatar images movable by operations of other users of a plurality of other client devices are superimposed on the virtual space and based on a single user of the single client device, so that the farther the avatar image is displayed in the virtual space based on the single user, the smaller the size of the avatar image;
an audio signal control means for controlling display positions of a plurality of avatar images in the virtual space as virtual sound sources of sounds emitted by other users corresponding to the avatar images, and for outputting audio signals based on the sounds, such that the volume of the audio signal decreases as the virtual sound source is located farther away from the one user in the virtual space;
Equipped with.

In addition, a transmission amount control means may be provided that controls the amount of audio signal transmitted so that the farther a virtual sound source is located relative to the one user in the virtual space, the less audio signal is transmitted.

Furthermore, the video signal control means and the audio signal control means can be controlled independently.

Further, the client device for the communication system in the virtual space of the present invention comprises:
a video signal output means for displaying a plurality of avatar images, each movable by a plurality of other users, superimposed on a virtual space based on the user himself/herself, the farther the avatar image is displayed in the virtual space based on the user himself/herself, the smaller the avatar image is displayed;
an audio signal output means for outputting an audio signal based on the sounds produced by the users corresponding to the avatar images, the audio signal output means defining display positions of the avatar images in the virtual space as virtual sound sources of the sounds produced by the users corresponding to the avatar images, and for decreasing the volume of the audio signal as the virtual sound source is located farther away from the user in the virtual space;
Equipped with.

Embodiments of the present invention will be described below with reference to the drawings.

Figure 1 is a block diagram showing a schematic configuration of a communication system in a virtual space according to an embodiment of the present invention. Figure 1 shows a server device 100, first through nth client devices 200A through 200N (those not distinguished from one another will be referred to as "client devices 200." The same rules will be used for assigning reference numbers hereafter), and a network 300, which will be described below.

The server device 100 has a web service function and provides a service that allows each user operating a client device 200 to communicate with each other in a virtual space by transmitting the desired video and audio signals in response to a request from the client device 200.

The client devices 200 are PCs or the like that are operated by users when receiving services provided by the server device 100. A web browser is installed on each client device 200, and the web browser is used to display video and audio signals transmitted from the server device 100 on the display 510 (Figure 3) and output them from the speaker 520.

Network 300 is a general term for networks that require the Internet. Network 300 can also include networks such as local area networks (LANs) that are constructed downstream from the Internet.

Figure 2 is a block diagram showing a schematic configuration of the server device 100 shown in Figure 1. Figure 2 shows the video signal control means 110, audio signal control means 120, transmission amount control means 130, content storage means 140, calculation means 150, and database 160, which are described below.

The video signal control means 110 superimposes a background image corresponding to the virtual space and an avatar image 600 (Figure 4), which is an image of a character that can be considered an alter ego of each user of the client device 200, using sprite processing or the like. Specific video signal control processing by the video signal control means 110 will be described later using Figure 4.

The audio signal control means 120 transmits an audio signal from one client device 200 to the remaining client devices 200, for example, by transmitting an audio signal collected by a microphone 430 (Figure 3) attached to the first client device 200A to the second to nth client devices 200B to 200N. Specific audio signal control processing by the audio signal control means 120 will be described later using Figure 5.

The transmission volume control means 130 typically performs control such as thinning out transmitted audio signals so that communication traffic across the entire network 300 does not exceed a predetermined threshold. Specific transmission volume control processing by the transmission volume control means 130 will be described later using Figure 6.

The content storage means 140 stores video content, such as background images of the virtual space and each user's avatar image 600, processed by the video signal control means 110 in a storage medium (not shown).

For each client device 200 connected to the server device 100, the calculation means 150 calculates distance information to each displayed object in the virtual space based on the position information of each displayed object, including other avatar images 600, relative to the user's own avatar image 600 in the virtual space.

Database 160 stores position information for each avatar image 600 in the virtual space and distance information calculated by calculation means 150. The distance information in database 160 can be used when performing control processing in video signal control means 110, audio signal control means 120, and transmission amount control means 130.

Figure 3 is a block diagram showing a schematic configuration of the client device 200 shown in Figure 1. Figure 3 shows the video signal processing means 210, audio signal processing means 220, receiving means 230, and transmission means 240, which will be described below. Figure 3 also shows the input device 400 and output device 500 associated with the client device 200.

The video signal processing means 210 receives a video signal transmitted from the server device 100 via the network 300 in accordance with a signal based on a user action accepted by the accepting means 230, and outputs the video signal to the display 510, which is the output device 500, using a web browser.

The audio signal processing means 220 receives audio signals transmitted via the server device 100 based on the voices uttered by users of other client devices 200, and outputs the audio signals to the speaker 520, which is the output device 500.

The accepting means 230 accepts actions performed by the user of the client device 200 via the input device 400. Specifically, for example, when the user presses a cursor key on the keyboard 410, clicks after moving the cursor with the mouse 420, or speaks in a situation where the sound is picked up by the microphone 430, these actions are accepted.

The transmitting means 240 transmits a signal corresponding to the user's action accepted by the accepting means 230 to the server device 100 via the network 300. Each user is assigned a unique login ID and login password, and only users who are authenticated by these are able to receive services provided by the communication system of this embodiment. A unique login ID is attached when signals are sent and received between the server device 100 and each client device 200, so that video and audio signals are customized for each user. However, a unique URL (uniform resource locator) may be used instead of or in addition to the login ID.

In this embodiment, input devices 400 such as a keyboard 410, mouse 420, and microphone 430, and output devices 500 such as a display 510 and speakers 520 are used to enable users of client devices 200 to communicate with each other within the virtual space.

Figure 4 is an explanatory diagram of the video signal control process performed by the video signal control means 110 shown in Figure 3. In this embodiment, when multiple avatar images 600B-600D that can be moved by operations of users of second to fourth client devices 200B-200D are superimposed and displayed in a virtual space based on the user of first client device 200A and displayed using the web browser of first client device 200A, the avatar image 600A of the user of first client device 200A is used as the base in the virtual space, and objects located closer to the base are displayed larger, and objects located further away are displayed smaller.

Figure 4(a) shows an example in which avatar images 600A to 600D corresponding to the first to fourth client devices 200A to 200D, respectively, are displayed in front of a virtual exhibition booth on the display 510 of the first client device 200A. In the example shown in Figure 4(a), avatar image 600B, avatar image 600C, and avatar image 600D are displayed in decreasing size in that order.

Note that although avatar image 600A belongs to the user (own user) of first client device 200A, it is not necessary for it to be displayed on display 510, and it is not necessary for it to be displayed.

Figure 4(b) shows the state after a transition from the state shown in Figure 4(a) when, for example, the user of first client device 200A presses the cursor key "↑" on keyboard 410. In this example, avatar images 600C and 600D are displayed larger than in the case shown in Figure 4(a), but avatar image 600B is no longer displayed within the display area of display 510.

Next, the operation of the video signal control process shown in Figure 4 will be described. First, when the user performs an action of pressing the cursor key "↑" on the keyboard 410, the accepting means 230 of the first client device 200A accepts that action.

Next, in the first client device 200A, the transmitting means 240 transmits a signal indicating the action accepted by the accepting means 230, together with the unique login ID, to the server device 100 via the network 300.

Furthermore, in server device 100, upon receiving the signal transmitted from first client device 200A, calculation means 150 calculates position information (coordinate information) of avatar image 600A based on the cursor key to be pressed and the duration for which that cursor key was pressed, and calculates distance information for each of avatar images 600B to 600D relative to avatar image 600A. The calculated position information and distance information are sequentially stored in database 160.

Furthermore, in server device 100, video signal control means 110 controls the video signal so that the content displayed on display 510 is changed, such as by gradually enlarging avatar image 600C, based on the distance information stored in database 160, which changes according to the virtual direction of travel of avatar image 600A, and transmits the video signal to first client device 200A.

When the first client device 200A receives a video signal transmitted from the server device 100, the video signal processing means 210 outputs the video signal to the display 510 using a web browser. The reason for using a web browser to display the video signal is that a web browser is positioned as an application program whose launch is unlikely to be restricted even outside the office.

The video signal control processing by the video signal control means 110 makes it possible to provide the user with a realistic display that is in line with real space. This can be clearly understood by replacing the above processing in virtual space with the user's actions in real space.

In other words, in real space, when the user associated with avatar image 600A approaches the user associated with avatar image 600C, the proportion of the user associated with avatar image 600C in the field of view increases. Incidentally, the user also approaches the user associated with avatar image 600D, increasing the proportion of that user, but on the other hand, the user associated with avatar image 600B moves out of the field of view.

In other words, the content displayed in the virtual space on the display 510 of the first client device 200A corresponds to the field of view of the user of the first client device 200A in real space, so the user can be provided with a realistic display that is in line with real space.

Figure 5 is an explanatory diagram of the audio signal control process performed by audio signal control means 120 shown in Figure 3. In this embodiment, when outputting audio signals based on the sounds emitted by the users of second to fourth client devices 200B to 200D corresponding to avatar images 600B to 600D as virtual sound sources for the sounds, the display positions of avatar images 600B to 600D in virtual space are controlled so that, relative to the user of first client device 200A in the virtual space, the volume of the audio signal from the user associated with avatar image 600, which is the virtual sound source located closer to the user, is increased, and the volume of the audio signal from the user associated with avatar image 600, which is the virtual sound source located farther away, is decreased.

Figure 5(a) shows control signals for the volume of the audio signals from each user corresponding to avatar images 600B-600D in Figure 4(a), transmitted from server device 100 and received by first client device 200A. In the example shown in Figure 5(a), this means that the volume of the audio signals decreases in the order of avatar image 600A, avatar image 600B, avatar image 600C, and avatar image 600D.

Figure 5(b) shows control signals for the volume of audio signals from each user corresponding to avatar images 600B-600D in Figure 4(b), transmitted from server device 100 and received by first client device 200A. In the example shown in Figure 5(b), this means that the volume of the audio signals from the users corresponding to avatar images 600C and 600D is higher than that in Figure 4(a), and the volume of the audio signal from the user corresponding to avatar image 600B is lower than that in Figure 4(a).

This control process makes it possible to provide the user with an audio signal with a realistic volume that conforms to the nature of hearing, whereby the volume in real space is perceived as louder the closer the sound source is to the user and quieter the farther away it is.

Next, the operation of the audio signal control process shown in Figure 5 will be described. First, when the user performs an action of pressing the cursor key "↑" on the keyboard 410, the accepting means 230 of the first client device 200A accepts that action.

Next, in the first client device 200A, the transmitting means 240 transmits a signal indicating the action accepted by the accepting means 230, together with the unique login ID, to the server device 100 via the network 300.

Furthermore, in server device 100, upon receiving the signal transmitted from first client device 200A, calculation means 150 calculates position information (coordinate information) of avatar image 600A based on the cursor key to be pressed and the duration for which that cursor key was pressed, and calculates distance information for each of avatar images 600B to 600D relative to avatar image 600A. The calculated position information and distance information are sequentially stored in database 160.

Furthermore, in server device 100, audio signal control means 120 generates an instruction signal to increase the volume of the audio signal related to avatar image 600C based on the distance information stored in database 160, which changes according to the virtual direction of travel of avatar image 600A, and transmits this instruction signal together with the audio signal to first client device 200A.

When the first client device 200A receives an audio signal or the like transmitted from the server device 100, the audio signal processing means 220 outputs the audio signal to the speaker 520 at a volume according to the instruction signal.

In this embodiment, even if the user of the first client device 200A simply issues a rotation instruction to change the orientation of the avatar image 600A using the mouse 420 or the like, the instruction signal for the volume of the audio signal does not change. In this respect, the technical concept regarding control of the volume of the audio signal differs from that of Patent Document 1.

Figure 6 is an explanatory diagram of the control process for the transmission amount of audio signals by the transmission amount control means 130 shown in Figure 3. In this embodiment, the greater the number of client devices 200 connected to the network 300, the smaller the amount of audio signals transmitted from the server device 100 to each client device 200.

Figure 6(a) shows the amount of audio signals transmitted to each client device 200 when the number of client devices 200 connected to the server device 100 is "3," Figure 6(b) shows the amount of audio signals transmitted to each client device 200 when the number of client devices 200 is "4," and Figure 6(c) shows the amount of audio signals transmitted to each client device 200 when the number of client devices 200 is "6."

As shown in Figure 6(a), when first to third client devices 200A to 200C are connected to the server device 100, the amount of audio signals transmitted from the server device 100 to each of the first to third client devices 200A to 200C is approximately "A." In other words, the total amount of audio signals transmitted from the server device 100 to the client device 200 is approximately "3A."

As shown in Figure 6(b), when a fourth client device 200D is further connected to the server device 100, the transmission amount to each of the first to fourth client devices 200A to 200D is reduced to three-quarters, or approximately 0.75 A, so that the total amount of audio signals transmitted from the server device 100 to the client device 200 remains at approximately 3 A.

As shown in Figure 6(c), when fifth and sixth client devices 200E-200F are further connected to the server device 100, the transmission amount to each of the first to sixth client devices 200A-200F is reduced to 3/6, or 1/2, approximately "0.5 A", so that the total amount of audio signals transmitted from the server device 100 to the client device 200 remains at approximately "3 A".

This reduces the amount of audio signals sent from the server device 100 to the client device 200, thereby preventing an increase in communication traffic on the network 300 connecting them, and reduces the time lag between the transmission of an audio signal from one client device 200 and the reception of that audio signal by another client device 200.

When a relatively large number of client devices 200 are connected to a server device 100, the amount of audio signals transmitted from the server device 100 to a given client device 200 can be uniformly thinned out. Alternatively, the amount of audio signals can be weighted and thinned out among the other client devices 200 based on the distance information and/or position information of the avatar images 600 stored in the database 160.

Therefore, as an example of weighting based on distance information, in the case of Figure 4(a), the thinning rate of the audio signal related to avatar image 600B from speaker 520 of first client device 200A could be, for example, zero, the thinning rate of the audio signal related to avatar image 600C could be, for example, 50%, and the thinning rate of the audio signal related to avatar image 600D could be, for example, 100%.

As an example of weighting based on position information, in the case of Figure 4(b), the thinning rate of the audio signal related to avatar image 600B, which is outside the display area, from speaker 520 of first client device 200A could be, for example, 30%, the thinning rate of the audio signal related to avatar image 600C could be, for example, zero, and the thinning rate of the audio signal related to avatar image 600D could be, for example, 10%.

According to the communication system of this embodiment, from the perspective of the user of the first client device 200A, the user operates the input device 400 to move the avatar image 600A appropriately within the virtual space, and when an audio signal of content of interest is heard, the avatar image 600A moves in the direction in which the volume increases.

Performing such operations in a virtual space corresponds to the behavior of moving around an exhibition hall in real space, stopping at a virtual exhibition booth that interests you, and having a conversation with the virtual exhibition booth staff, giving the user a sense of realism as if they were actually in the exhibition hall in real space.

In addition, in this embodiment, the server device 100 performs control processing of the video signal and control processing of the audio signal independently. In a real-world exhibition hall, a user is unlikely to head to a virtual exhibition booth that does not interest them, and will likely pass by such a virtual exhibition booth. Therefore, by reducing the amount of audio signal transmission from locations far from the user's avatar image 600, there is no significant inconvenience even if the audio signal is difficult to hear. On the other hand, the amount of video signal transmission can be kept the same without being reduced, so the resolution of the video signal does not decrease and the sense of realism is not impaired.

In the above, in this embodiment, the communication system has been described primarily as being applied to a virtual exhibition hall, but it can also be applied to virtual urban spaces, including, for example, a virtual mall that brings together multiple virtual stores/virtual event venues that exist on the Internet, a virtual concert venue/virtual live performance venue, a virtual movie theater, a virtual art gallery/virtual museum/virtual zoo, a virtual amusement park, a virtual office, etc.

The virtual stores/virtual event spaces in a virtual mall are equivalent to virtual exhibition booths at an exhibition. Users can move their own avatar image 600 to the desired virtual store, etc., and, if necessary, purchase products after receiving product explanations from a salesperson at the virtual store, etc.

In the case of a virtual concert venue/virtual live performance venue, users will not likely converse with each other much during the performance, but in real space, friends may still go to a concert together and sit next to each other, communicating by talking between songs or during breaks. Therefore, with the communication system of this embodiment, communication can also take place in a virtual concert venue, etc.

However, unlike a virtual exhibition hall, a virtual concert venue will likely require high-quality audio signals. Therefore, of the audio signals transmitted to each client device 200, the transmission amount control means 130 will not thin out the audio signals emitted by the artists singing or playing. Instead, with regard to audio signals from other client devices 200, audio signals from virtual sound sources corresponding to avatar images 600 located, for example, more than 30 m away in real space may be thinned out by, for example, 70% or more.

In a virtual movie theater, high-quality movie video signals could be superimposed on a background image that mimics the inside of a movie theater. If a user is watching a movie alone, watching a movie DVD would be sufficient, but if two or more people want to watch a movie, it would be possible for friends to communicate with each other in a virtual concert hall or on Saturday. Additionally, even if a user is watching a movie alone, it has the advantage of being able to experience an atmosphere similar to that of a real movie theater, and for movie theater operators, it also has the advantage of helping to mitigate the decline in sales caused by the COVID-19 pandemic.

As with virtual movie theaters, providing high-quality video signals such as moving and still images of exhibits in virtual art galleries, museums, and zoos would have useful effects such as allowing friends to communicate with each other, allowing a single user to experience a sense of realism, and providing economic benefits to the operators of museums, etc.

In the case of a virtual office, even if employees are forced to carry out their work from home, a communication system can be adopted between employees by using, for example, a video signal that superimposes an avatar image 600 of each employee on a background image that mimics the inside of an office in real space, which may also facilitate smooth work and provide mental care for each employee.

Furthermore, for example, if you encounter an avatar image 600 of an acquaintance in the virtual city space while moving around a virtual mall, you can communicate with them as if you were having a casual conversation.

Furthermore, if we apply this to real space, exhibition booths, stores, movie theaters, etc. only need to ensure capacity according to the number of visitors/guests; in other words, if the venue is full of empty seats, the rent for the space would ultimately be wasted.

In this regard, virtual exhibition booths, virtual stores, virtual movie theaters, etc. are thought to avoid such problems, but as the number of users increases, communication traffic problems will arise, as mentioned above. Therefore, in this embodiment, to prevent communication traffic problems, the capacity of exhibition booths, etc. is allocated according to the number of users, and it is also possible to rent virtual areas such as virtual exhibition booths among exhibitors, etc.

1 is a block diagram showing a schematic configuration of a communication system in a virtual space according to an embodiment of the present invention. FIG. 2 is a block diagram showing a schematic configuration of a server device 100 shown in FIG. 1 . FIG. 2 is a block diagram showing a schematic configuration of a client device 200 shown in FIG. 1 . 4 is an explanatory diagram of a control process of a video signal by the video signal control means 110 shown in FIG. 3. FIG. 4 is an explanatory diagram of a control process of an audio signal by the audio signal control means 120 shown in FIG. 3. 4 is an explanatory diagram of a control process of the transmission amount of an audio signal by the transmission amount control means 130 shown in FIG. 3. FIG.

REFERENCE SIGNS LIST 100 Server device 110 Video signal control means 120 Audio signal control means 130 Transmission amount control means 140 Content storage means 150 Calculation means 160 Database 200 Client device 210 Video signal processing means 220 Audio signal processing means 230 Reception means 240 Transmission means 300 Network 400 Input device 410 Keyboard 420 Mouse 430 Microphone 500 Output device 510 Display 520 Speaker 600 Avatar image

Claims (4)

a video signal control means for controlling a virtual space displayed using a web browser on a single client device, in which a plurality of avatar images movable by operations of other users of a plurality of other client devices are superimposed on the virtual space and based on a single user of the single client device, so that the farther the avatar image is displayed in the virtual space based on the single user, the smaller the size of the avatar image;
an audio signal control means for controlling display positions of a plurality of avatar images in the virtual space as virtual sound sources of sounds emitted by other users corresponding to the avatar images, and for outputting audio signals based on the sounds, such that the volume of the audio signal decreases as the virtual sound source is located farther away from the one user in the virtual space;
a calculation means for calculating distance information between the avatars based on signals transmitted from the client devices and indicating a virtual moving direction and time of each avatar in the virtual space;
A server device for a communication system in a virtual space, comprising: A server device for a communication system in a virtual space as described in claim 1, further comprising a transmission amount control means for controlling the transmission amount of audio signals so that the more distant a virtual sound source is located relative to the one user in the virtual space, the less audio signals are transmitted. The server device for a virtual space communication system according to claim 1, wherein the video signal control means and the audio signal control means perform independent control. a video signal output means for displaying a plurality of avatar images, each movable by a plurality of other users, superimposed on a virtual space based on the user himself/herself, the farther the avatar image is displayed in the virtual space based on the user himself/herself, the smaller the avatar image is displayed;
an audio signal output means for outputting an audio signal based on the sounds produced by the users corresponding to the avatar images, the audio signal output means defining display positions of the avatar images in the virtual space as virtual sound sources of the sounds produced by the users corresponding to the avatar images, and for decreasing the volume of the audio signal as the virtual sound source is located farther away from the user in the virtual space;
a calculation means for calculating distance information between the avatars based on a signal indicating a virtual moving direction and time of the avatars in the virtual space;
A client device for a communication system in a virtual space, comprising: