JP7758129B2 - Communication terminal, communication system, communication method, and program - Google Patents

Description

The present invention relates to a communication terminal, a communication system, a communication method, and a program.

Conference systems are known in which multiple communication terminals share content data, such as video data, image data, and document data, via a communication network such as the Internet.

For example, a conference system is known in which an information processing terminal transmits a second image, which is a first image of shared material data and has address information for the shared material data embedded therein, to another information processing terminal, thereby sharing the shared material data with the other information processing terminal (see, for example, Patent Document 1).

There is a demand for using such a conference system to hold conferences where content data recorded from multiple angles of three-dimensional objects such as models and prototypes can be shared.

However, for example, when sharing video data (an example of content data) of a three-dimensional object captured from multiple directions between multiple communication devices, it may take some time before sharing can begin depending on the size of the video data, the communication environment, etc.

One embodiment of the present invention has been made in consideration of the above-mentioned problems, and enables a communication terminal that communicates with other communication terminals via a communication network to quickly share content data recorded from multiple directions of a three-dimensional object with the other communication terminals.

In order to solve the above problem, a communication terminal according to one embodiment of the present invention is a communication terminal that communicates with other communication terminals via a communication network, and has a content data acquisition unit that acquires content data recorded from multiple directions of a three-dimensional object, an operation acceptance unit that accepts a designation of a display position of the content data, an image sharing unit that shares an image corresponding to the display position of the content data with the other communication terminals, and a display control unit that causes a display screen including the image shared by the image sharing unit to be displayed on a display unit , wherein the display control unit displays a display element that designates the display position on the display screen, and when the display position is designated by the display element, the image sharing unit transmits an image corresponding to the display position of the content data to the other communication terminals.

According to one embodiment of the present invention, a communication terminal that communicates with other communication terminals via a communication network can quickly share content data recorded from multiple angles of a three-dimensional object with the other communication terminals.

FIG. 1 is a diagram illustrating an example of a system configuration of a communication system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal according to an embodiment. FIG. 10 is a diagram illustrating another example of a hardware configuration of a communication terminal according to an embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of a communication terminal according to the first embodiment. FIG. 10 is a sequence diagram illustrating an example of a sharing process according to the first embodiment. FIG. 2 is a diagram showing an image of a display screen according to the first embodiment. FIG. 11 is a sequence diagram illustrating an example of an operation authority acquisition process according to the second embodiment. FIG. 10 is a diagram showing an image of a display screen according to the second embodiment. FIG. 11 is a diagram illustrating an example of a functional configuration of a communication terminal according to a third embodiment. FIG. 13 is a sequence diagram illustrating an example of a sharing process according to the third embodiment. FIG. 11 is a diagram illustrating model data according to the third embodiment. FIG. 13 is a sequence diagram illustrating an example of a sharing process according to the fourth embodiment. FIG. 13 is a diagram showing an image of a display screen according to the fourth embodiment.

Embodiments of the present invention will be described below with reference to the accompanying drawings.

<System Configuration>
1 is a diagram illustrating an example of a system configuration of a communication system according to an embodiment. The communication system 100 includes a plurality of communication terminals 110 that are communicably connected to each other via a communication network 101 such as the Internet or a local area network (LAN).

The communication terminal 110 is used in video conferences to send and receive, for example, video data and audio data for calls, shared content data, and the like, with other communication terminals 110 via the communication network 101. Note that a "video conference" is also called a "teleconference," "remote conference," etc. Furthermore, the communication terminal 110 may include, for example, an electronic whiteboard 110a, a videoconference terminal 110b, an information terminal 110c, etc.

The communication terminal 110 may hold a video conference with one or more other communication terminals 110, for example, via a server such as the communication management system 120. However, this is not limited to this, and the communication terminal 110 may also hold a video conference with other communication terminals 110 without going through the communication management system 120.

The electronic whiteboard 110a is a communication terminal 110 that exchanges, for example, video and audio data for calls, shared content data, stroke data, etc., and is also called an IWB (Interactive Whiteboard). The video and audio data for calls includes, for example, video of the user captured by a camera and the user's voice picked up by a microphone.

Content data refers to data such as text, graphics, audio, images, and video, or a combination of at least one of these. As an example, content data may include document data, video data, and still image data shared among multiple communication terminals 110. In this embodiment, content data also includes video data of a three-dimensional object photographed from multiple directions, or model data (electronic data) that is a three-dimensional model of the video data.

Stroke data is electronic data (coordinate information, line type information, etc.) that represents an image (hereinafter referred to as a stroke image) drawn by a user by touching and moving an electronic pen, hand, etc. on the display of the electronic whiteboard 110a. For example, when a stroke image is drawn on one electronic whiteboard 110a, this stroke data is sent to the other electronic whiteboard 110a, and the same stroke image is drawn on the other electronic whiteboard 110a as well.

The video conference terminal 110b is a communications terminal 110 that exchanges, for example, video data and audio data for calls, shared content data, etc. The video conference terminal 110b may also have the function of connecting to an electronic whiteboard or other device that does not have video conference functionality, and sending and receiving stroke data. Furthermore, the video conference terminal 110b may also have the function of connecting to a PC (Personal Computer) or the like, and acquiring shared content data from the PC.

The information terminal 110c is, for example, a general-purpose information processing device such as a PC, tablet terminal, or smartphone. For example, the information terminal 110c executes an application program compatible with the communication system 100, thereby functioning as a communication terminal 110 that transmits and receives video data and audio data for calls, shared content data, stroke data, etc.

Note that the electronic whiteboard 110a, video conferencing terminal 110b, information terminal 110c, etc. are examples of communication terminals 110. For example, the communication terminals 110 may be various electronic devices with communication functions, display functions, etc., such as output devices such as PJ (Projector), digital signage, HUD (Head Up Display) devices, industrial machinery, medical equipment, etc.

The communication management system 120 is, for example, an information processing device having a computer configuration, or a system including multiple information processing devices, and performs login authentication for the communication terminal 110, communication (session) management, and communication data relay. Note that in this embodiment, the communication management system 120 is optional and not required. Also, since it is assumed that the communication management system 120 will utilize an existing communication management system, a detailed description of the communication management system 120 will be omitted here.

In a communication system 100 such as the one described above, there is a demand for multiple communication terminals 110 to hold a conference by sharing video data (an example of content data) captured from multiple angles of a three-dimensional object such as a model or prototype.

However, with conventional technology such as that shown in Patent Document 1, in order to share multiple pieces of information possessed by a three-dimensional object, such as top/bottom, left/right, front/back, etc., it is necessary to repeatedly capture and share information while changing the positional relationship between the camera capturing the three-dimensional object and the three-dimensional object.

Another possible method would be to share video data of a three-dimensional object and explain the object while playing back the shared video data, but with this method, the viewpoints (sides, etc.) of the person explaining and the person receiving the explanation may not match.

Furthermore, for example, when video data of a three-dimensional object captured from multiple directions is shared among multiple communication terminals 110, it may take some time before sharing can begin, depending on the size of the video data, the communication environment, etc. Therefore, for example, as the amount of video data increases, there is a problem in that the time it takes for each communication terminal 110 to finish sending/receiving the video data (e.g., a video file, etc.) varies more.

To solve this problem, the communication terminal 110 according to this embodiment does not transmit video data, for example, of a three-dimensional object continuously recorded from multiple directions, to other communication terminals 110, but instead shares an image (still image) corresponding to a specified display position with the other communication terminals 110.

Preferably, the communication terminal 110 displays a seek bar (an example of a display element) on the display screen that specifies the playback position (an example of a display position) of the above-mentioned video data, and shares the image corresponding to the playback position specified by the seek bar with the other communication terminals 110. This allows each communication terminal 110 participating in the same video conference to quickly display, for example, an image of a three-dimensional object viewed from the same angle.

In this way, according to this embodiment, a communication terminal 110 that communicates with other communication terminals 110 via a communication network 101 can quickly share content data recorded from multiple directions of a three-dimensional object with the other communication terminals 110.

<Hardware configuration>
Next, an example of the hardware configuration of each device according to this embodiment will be described.

(Hardware configuration of information terminal and communication management system)
The information terminal 110c and the communication management system 120 have, for example, the hardware configuration of a computer 200 as shown in Fig. 2. Alternatively, the communication management system 120 may be configured by a plurality of computers 200.

Figure 2 is a diagram showing an example of the hardware configuration of a computer according to one embodiment. The computer 200 includes, for example, a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, HD (Hard Disk) 204, HDD (Hard Disk Drive) controller 205, display 206, external device connection I/F (Interface) 207, network I/F 208, keyboard 209, pointing device 210, DVD-RW (Digital Versatile Disk Rewritable) drive 212, media I/F 214, and bus line 215.

Of these, CPU 201 is a computing device that controls the overall operation of computer 200. ROM 202 is non-volatile memory that stores programs used to start up computer 200, such as IPL (Initial Program Loader). RAM 203 is volatile memory used as a work area for CPU 201. HD 204 is a large-capacity storage device that stores programs such as the OS (Operating System) and applications, as well as various types of data. HDD controller 205 controls the reading and writing of various types of data from HD 204 under the control of CPU 201.

The display 206 displays various types of information, such as a cursor, menus, windows, text, or images. The external device connection I/F 207 is an interface for connecting various external devices. For example, if the computer 200 is an information terminal 110c, a camera, speaker, etc. may be connected to the external device connection I/F 207. Alternatively, the information terminal 110c may have a built-in camera or speaker.

The network I/F 208 is a communication interface for communicating data using the communication network 101. The keyboard 209 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 210 is a type of input means for selecting and executing various instructions, selecting processing targets, moving the cursor, etc.

The DVD-RW drive 212 controls the reading and writing (storage) of data from the DVD-RW 211. The DVD-RW drive 212 may also control the reading and writing (storage) of data from discs other than the DVD-RW 211. The media I/F 214 controls the reading and writing (storage) of data from the media 213, such as flash memory. The bus line 215 includes an address bus, a data bus, and various control signals for electrically connecting the various components, such as the CPU 201, shown in FIG. 2.

(Hardware configuration 1 of communication terminal)
3 is a diagram illustrating an example of the hardware configuration of a communication terminal according to an embodiment. Here, an example of the hardware configuration of a video conference terminal 110b, which is an example of the communication terminal 110, will be described.

The videoconference terminal 110b includes a computer configuration, such as a CPU 301, ROM 302, RAM 303, flash memory 304, and SSD (Solid State Drive) 305. The videoconference terminal 110b also includes a media I/F 307, operation buttons 308, a power switch 309, a network I/F 311, a camera 312, an image sensor I/F 313, a microphone 314, a speaker 315, and an audio input/output I/F 316. The videoconference terminal 110b also includes a display I/F 317, an external device connection I/F 318, a short-range communication circuit 319, an antenna 319a for the short-range communication circuit 319, and a bus 310. The camera 321, microphone 314, speaker 315, and the like may be provided externally or internally to the videoconference terminal 110b.

CPU 301 is a computing device that realizes the various functions of video conference terminal 110b by reading programs and data from, for example, ROM 302 or flash memory 304 and executing processes. ROM 302 is non-volatile memory that pre-stores programs used to start up video conference terminal 110b, such as IPL. RAM 303 is volatile memory used as a work area for CPU 301, etc.

Flash memory 304 is a storage device that stores, for example, an OS, application programs, and various data. SSD 305 controls the reading and writing of various data from flash memory 304 under the control of CPU 301. Media I/F 307 controls the reading and writing (storage) of data from media 306, such as a memory card.

The operation button 308 is an input device that accepts input operations from the user of the video conference terminal 110b. The power switch 309 is a switch for turning the power of the video conference terminal 110b on and off. The network I/F 311 is a communication interface for communication using the communication network 101.

The camera 312 is an imaging device for capturing images of a subject under the control of the CPU 301. The imaging element I/F 313 controls the imaging by the camera 312 and converts the captured data into predetermined image data (video data). The microphone 314 converts acquired sound into an electrical signal. The speaker 315 converts the sound signal into sound and outputs it. The sound input/output I/F 316 controls the input and output of sound by the microphone 314 and speaker 315.

The display I/F 317 outputs image data to the display 320 under the control of the CPU 301. The external device connection I/F 318 is an interface for connecting various external devices. For example, an electronic whiteboard (IWB) without video conferencing functionality may be connected to the display I/F 317 or the external device connection I/F 318. This allows, for example, an electronic whiteboard connected to the video conferencing terminal 110b to display video conferencing images, shared images, stroke images, etc., in the same way as an electronic whiteboard with video conferencing functionality.

The short-range communication circuit 319 is a communication circuit that performs short-range wireless communication via the antenna 319a. The bus 310 is commonly connected to each of the above components and transmits address signals, data signals, various control signals, etc.

(Hardware configuration 2 of communication terminal)
4 is a diagram illustrating another example of the hardware configuration of a communication terminal according to an embodiment. Here, an example of the hardware configuration of an electronic whiteboard 110a, which is another example of the communication terminal 110, will be described.

The electronic whiteboard 110a includes a CPU 401, ROM 402, RAM 403, SSD 404, network I/F 405, and external device connection I/F 406. Of these, the CPU 401 is a computing device that controls the overall operation of the electronic whiteboard 110a. The ROM 402 is non-volatile memory that stores programs used to start the electronic whiteboard 110a, such as IPL. The RAM 403 is volatile memory used as a work area for the CPU 401. The SSD 404 is a large-capacity storage device that stores programs and various data for the electronic whiteboard 110a.

The network I/F 405 is a communication interface that connects the electronic whiteboard 110a to the communication network 101 and enables communication. The external device connection I/F 406 is an interface that connects various external devices. In the example of Figure 4, the external devices include, for example, a USB (Universal Serial Bus) memory 431, a microphone 432, a speaker 433, a camera 434, etc.

The electronic whiteboard 110a also includes a capture device 411, a GPU (Graphics Processing Unit) 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a for the short-range communication circuit 419, a power switch 417, and selection switches 418.

Of these, the capture device 411 captures (acquires) the display screen, etc. displayed on the display, etc. of the external PC 450, as a still image or video. The capture device 411 also captures video data, etc., captured by the camera 434.

The GPU 412 is a semiconductor chip (processor) that specializes in graphics. The display controller 413 controls and manages the screen display in order to output images from the GPU 412 to the display 460, etc. The contact sensor 414 detects contact with the electronic pen 440, the user's hand H, etc. on the display 460. The sensor controller 415 controls the processing of the contact sensor 414.

The contact sensor 414 inputs and detects coordinates using, for example, an infrared blocking method. This coordinate input and detection method involves two light-receiving and light-emitting devices installed at both ends of the upper side of the display 460 emitting multiple infrared rays parallel to the display 460, which are reflected by reflective members installed around the periphery of the display 460 and receive the light returning along the same optical path as the light emitted by the light-receiving elements. The contact sensor 414 outputs the IDs of the infrared rays emitted by the two light-receiving and light-emitting devices that are blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position, which is the contact position of the object. The electronic pen controller 416 communicates with the electronic pen 440 to determine whether the pen tip or the pen tail is touching the display 460. The short-range communication circuit 419 is a communication circuit that performs short-range wireless communication via the antenna 419a.

The power switch 417 is a switch for turning the power of the electronic whiteboard 110a on and off. The selection switches 418 are a group of switches for adjusting the brightness and color of the display 460, for example.

Furthermore, the electronic whiteboard 110a is equipped with a bus line 420. The bus line 420 includes an address bus, a data bus, and various control signals for electrically connecting each component, such as the CPU 401 shown in FIG. 4.

The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance, or a resistive film touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may also use various detection means, such as an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. Furthermore, the electronic pen controller 416 may determine whether or not a touch has occurred not only on the tip and but also on the part of the electronic pen 440 that the user holds, or on other parts of the electronic pen.

[First embodiment]
<Functional configuration>
Fig. 5 is a diagram illustrating an example of the functional configuration of the communication terminal 110 according to the first embodiment. As illustrated in Fig. 5, the communication terminal 110 includes, for example, a communication unit 501, a conference control unit 502, a content data acquisition unit 503, an operation reception unit 504, an image sharing unit 505, a display control unit 506, and a storage unit 507.

For example, if the communication terminal 110 is an electronic whiteboard 110a, the electronic whiteboard 110a realizes the above-mentioned functional configuration by executing a predetermined program with the CPU 401 in FIG. 4. Furthermore, if the communication terminal 110 is a videoconferencing terminal 110b, the videoconferencing terminal 110b realizes the above-mentioned functional configuration by executing a predetermined program with the CPU 301 in FIG. 3. Furthermore, if the communication terminal 110 is an information terminal 110c, the information terminal 110c realizes the above-mentioned functional configuration by executing an application program corresponding to the communication system 100 with the CPU 201 in FIG. 2. Note that at least some of the above-mentioned functional configuration may be realized by hardware. Furthermore, the following description will be given assuming that the communication terminal 110 is a videoconferencing terminal 110b as an example, but this does not limit the scope of the present invention.

The communication unit 501 connects the communication terminal 110 to the communication network 101 using, for example, the network I/F 405 in Figure 4, and communicates with other communication terminals 110, the communication management system 120, etc.

The conference control unit 502 performs various controls related to conferences, such as video conferences, in response to, for example, user operations or request information received from other communication terminals 110. For example, the conference control unit 502 controls the start and end of conferences, the sending and receiving of video data and audio data for calls, the sending and receiving of stroke data, etc.

The content data acquisition unit 503 acquires video data of a three-dimensional object captured from multiple directions, for example, using the camera 434 in FIG. 4. Note that video data of a three-dimensional object captured from multiple directions is an example of content data of a three-dimensional object recorded from multiple directions. For example, content data recorded from multiple directions may be model data of a three-dimensional object. The content data acquisition unit 503 may also acquire content data (e.g., captured video data, modeled model data, etc.) from an external device other than the camera 312, for example, using the external device connection I/F 318, network I/F 311, etc. in FIG. 3.

The operation acceptance unit 504 accepts various operations by the user using, for example, the contact sensor 414, sensor controller 415, etc. shown in FIG. 4. For example, the operation acceptance unit 504 accepts a specification operation that specifies the display position of content data recorded from multiple directions of a three-dimensional object. As an example, if the content data is video data of a three-dimensional object captured from multiple directions, the operation acceptance unit 504 accepts a specification of the playback position of the video data specified by the user using display elements such as a seek bar displayed on the display screen that displays the captured video.

Preferably, the operation reception unit 504 also receives operations such as specification information that specifies the display position of content data recorded from multiple directions of a three-dimensional object, which the communication unit 501 receives from another communication terminal 110.

The image sharing unit 505 shares an image (still image) corresponding to the display position of the content data accepted by the operation acceptance unit 504 with other communication terminals 110 currently communicating with the communication terminal 110. For example, when the operation acceptance unit 504 accepts a specification of the display position of content data that records a three-dimensional object from multiple directions, the image sharing unit 505 transmits an image corresponding to the specified display position of the content data to the other communication terminals 110 currently communicating with the communication terminal 110.

Preferably, when the image sharing unit 505 receives an image corresponding to a specified display position of content data acquired by another communication terminal 110 from the other communication terminal 110 with which it is communicating, it shares the received image with the other communication terminal 110.

The display control unit 506 displays on a display screen an image that the image sharing unit 505 shares with other communication terminals 110 with which communication is in progress. For example, the display control unit 506 displays on a display unit such as the display in FIG. 4 a display screen including an image corresponding to the display position of the content data accepted by the operation acceptance unit 504.

The storage unit 507 is realized, for example, by a program executed by the CPU 401, the SSD 404, the RAM 403, etc., and stores various information and data including content data shared with other communication terminals 110, display position information, etc.

Note that the functional configuration of the communication terminal 110 shown in FIG. 5 is an example. For example, some or all of the operation reception unit 504, image sharing unit 505, and display control unit 506 may be included in the conference control unit 502.

<Processing flow>
Next, the processing flow of the communication method according to the first embodiment will be described.

Figure 6 is a sequence diagram showing an example of a sharing process according to the first embodiment. This figure shows an example of a sharing process in which, for example, when communication (conference) is taking place between communication terminals 110-1 and 110-2 having the functional configuration shown in Figure 5, user A using communication terminal 110-1 shares video of a three-dimensional object captured from multiple directions with other communication terminal 110-2.

At the start of the processing shown in FIG. 6, communication terminal 110-1 and communication terminal 110-2 are engaged in communication (conference) to send and receive video data and audio data via communication network 101. Communication terminal 110-1 and communication terminal 110-2 may communicate with each other via communication management system 120, or they may communicate with each other without going through communication management system 120.

In step S601, during communication between communication terminal 110-1 and communication terminal 110-2, when user A using communication terminal 110-1 performs a sharing start operation on communication terminal 110-1, the processing from step S602 onwards is executed.

In steps S602 and S603, when the operation reception unit 504 of communication terminal 110-1 receives a sharing start operation from user A, it requests the content data acquisition unit 503 to start and end capturing video data captured by camera 312 or the like. At this time, user A, for example, continuously moves the camera 312 or the three-dimensional object in question so that the three-dimensional object is captured from multiple directions.

Here, it is desirable that the time from the start to the end of capture be relatively short, for example, around 10 seconds, to make it easier to specify the display position, as described below. However, in this embodiment, since the captured video data is not sent or received, the shared image can be quickly displayed even if the time from the start to the end of capture is relatively long, at 10 seconds or more.

Note that the processing in steps S602 and S603 is an example. For example, the content data acquisition unit 503 may acquire video data stored in advance in the storage unit 507 or an external device.

In step S604, the operation reception unit 504 of the communication terminal 110-1 sends a completion notification to the display control unit 506 and other devices indicating that acquisition of the video data has been completed.

In step S605, the display control unit 506 of the communication terminal 110-1 uses the video data acquired by the content data acquisition unit 503 to display a shared screen, for example, as shown in FIG. 7.

Figure 7 is a diagram showing an image of a display screen according to the first embodiment. In step S605 of Figure 6, the display control unit 506 of communication terminal 110-1 displays, for example, a display screen 710 as shown in Figure 7 on a display unit such as the display 460. In the example of Figure 7, the display screen 710 displays a shared screen 711, an image 712 of a user of another communication terminal such as 110-2, and an image 713 of a user of communication terminal 110-1.

On the shared screen 711, for example, an image of a three-dimensional object 714 based on the video data captured in steps S602 and S603 of FIG. 6 and a seek bar 715 for specifying the playback position of the video data are displayed. Note that the seek bar 715 is an example of a display element for specifying the playback position (display position) of the video data (content data).

Now, let's return to Figure 6 and continue explaining the sequence diagram.

For example, in step S605, when the display control unit 506 of the communication terminal 110-1 displays the shared screen 711, the communication terminal 110-1 repeatedly executes the processes of steps S611 to S618 until it receives an operation to end sharing in step S621.

In step S611, when the operation acceptance unit 504 of communication terminal 110-1 accepts an operation by user A to specify a playback position, processing from step S612 onwards is executed. Note that user A can specify the playback position of the video data, for example, by moving the position of the seek bar 715 left or right on the shared screen 711 as shown in FIG. 7.

In step S612, the operation reception unit 504 of the communication terminal 110-1 notifies the image sharing unit 505 of the received playback position information.

In step S613, the image sharing unit 505 of communication terminal 110-1 acquires an image (still image) corresponding to the notified playback position from the video data acquired by the content data acquisition unit 503.

In step S614, the image sharing unit 505 of the communication terminal 110-1 notifies the display control unit 506 of the communication terminal 110-1 of the image acquired in step S613. In response, in step S615, the display control unit 506 displays the notified image on a shared screen 711 included in the display screen 710 of the communication terminal 110-1, for example, as shown in FIG. 7.

In steps S616 and S617, the image sharing unit 505 transmits the image acquired in step S613 to the other communication terminal 110-2 via the communication unit 501.

Preferably, the image sharing unit 505 transmits the acquired image along with information about the playback position (e.g., the time from the start point to the playback position, the number of frames, etc.), information about the playback range (e.g., the time from the start point to the end point, the number of frames, etc.), etc. to the other communication terminal 110-2.

In step S618, the display control unit 506 of the communication terminal 110-2 displays the image (still image) received from the communication terminal 110-1 on a shared screen 721 included in the display screen 720 of the communication terminal 110-2, for example, as shown in FIG. 7. As a result, the shared screen 711 of the communication terminal 110-1 and the shared screen 721 of the communication terminal 110-2 display the three-dimensional object 714 viewed from the same direction, as shown in FIG. 7.

If another communication terminal 110-3 is participating in a conference in which communication terminals 110-1 and 110-2 are participating, communication terminal 110-1 (or communication management system 120) will also transmit the images and other information sent to communication terminal 110-2 to the other communication terminal 110-3 participating in the conference. As a result, for example, as shown in FIG. 7, a shared screen 731 similar to the shared screen 721 displayed by communication terminal 110-2 will also be displayed on the display screen 730 displayed by communication terminal 110-3.

In step S621, for example, if the operation reception unit 504 of communication terminal 110-1 receives a sharing termination operation, communication terminal 110-1 ends the processing of steps S611 to S618.

Through the above processing, communication terminal 110-1 can share a still image displaying three-dimensional object 714 from a specified position with other communication terminals 110-2, etc., without having to transmit content data recording three-dimensional object 714 from multiple directions to other communication terminals 110-2, etc.

Furthermore, user A using communication terminal 110-1 can hold a conference while sharing multiple pieces of information about the three-dimensional object 714, such as top and bottom, left and right, and front and back, by moving the position of the seek bar 715 left and right, etc.

As described above, according to this embodiment, a communication terminal 110-1 that communicates with another communication terminal 110-2 via a communication network 101 can quickly share content data recorded from multiple directions of a three-dimensional object with the other communication terminal 110-2.

Second Embodiment
In the second embodiment, an example of processing will be described in which the playback position of video data acquired by communication terminal 110-1 is specified from communication terminal 110-2, another participant in the conference.

Figure 8 is a sequence diagram showing an example of the process of acquiring operation authority according to the second embodiment. Note that it is assumed that steps S601 to S605 and steps S611 to S616 in Figure 6 have already been executed when the process shown in Figure 8 starts.

In step S801, user B using communication terminal 110-2 performs an operation to obtain operation rights. For example, user B performs an operation to obtain operation rights by displaying menu screen 901 on display screen 720 displayed by communication terminal 110-2, as shown in FIG. 9(A), and selecting "Seek Operation" 902.

In step S802, when the operation reception unit 504 of communication terminal 110-2 receives an operation by user B to obtain operation rights, it sends a request to obtain operation rights to communication terminal 110-1 via the communication unit 501 of communication terminal 110-2.

In step S803, the communication unit 501 of communication terminal 110-1 notifies the image sharing unit 505 of the request to obtain operation rights received from communication terminal 110-2.

At this time, if the image sharing unit 505 of communication terminal 110-1 is currently processing a request from another communication terminal such as 110-3, steps S811 to S813 are executed. On the other hand, if the image sharing unit 505 of communication terminal 110-1 is not currently processing a request from another communication terminal such as 110-3, steps S821 and subsequent steps are executed.

For example, if a request from another communication terminal 110-3 or the like is currently being processed, the image sharing unit 505 of the communication terminal 110-1 sends a refusal notice to the requesting communication terminal 110-2 in steps S811 and S812, refusing to acquire operation rights. Furthermore, in step S813, upon receiving the refusal notice, the display control unit 506 of the communication terminal 110-2 displays a display screen indicating that acquisition of operation rights has failed.

On the other hand, if a request from another communication terminal (e.g., 110-3) is not currently being processed, the image sharing unit 505 of communication terminal 110-1 sends an acquisition request acceptance notification to the display control unit 506 in step S821, indicating that a request to acquire operation rights has been accepted.

In step S822, the display control unit 506 of communication terminal 110-1 displays, for example, an inquiry screen 903 as shown in FIG. 9(B) on the display screen 710. In the example of FIG. 9(B), the display screen 710 displayed by the display control unit 506 of communication terminal 110-1 displays an inquiry screen 903 that displays a selectable "OK" button to allow acquisition of operation rights and an "NG" button to deny acquisition of operation rights.

In step S823, if user A performs an approval operation (presses the "OK" button), the processing from step S824 onwards is executed. On the other hand, if user A performs a refusal operation (presses the "NG" button), a refusal notification is sent to the requesting communication terminal 110-2 through processing similar to steps S811 to S813.

In step S824, when the operation acceptance unit 504 of the communication terminal 110-1 accepts the permission operation, it sends a permission notification to the image sharing unit 505 indicating that the permission operation has been accepted.

In steps S825 and S826, the image sharing unit 505 of communication terminal 110-1 sends a permission notification to communication terminal 110-2 via the communication unit 501. This permission notification may include, for example, information indicating the current display position, playback range, etc.

In step S827, upon receiving the permission notification sent from communication terminal 110-1, the display control unit 506 of communication terminal 110-2 displays, for example, a display screen 720 including a shared screen 904 as shown in FIG. 9(C) on a display unit such as the display 460. This shared screen 904 displays, for example, a seek bar 905 for specifying the playback position of the video data, as shown in FIG. 9(C).

This seek bar 905 is displayed based on, for example, playback position information (time from the start point, number of frames, etc.) and playback range information (for example, time from the start point to the end point, number of frames, etc.) notified from communication terminal 110-1. Note that the playback position information and playback range information are assumed to be included, for example, in the data received in step S617 of FIG. 6 or the notification received in step S826 of FIG. 8.

In step S828, when user B uses the seek bar 905 on the shared screen 904 to specify a playback position, processing from step S829 onwards is executed.

In step S829, the operation reception unit 504 of communication terminal 110-2 transmits information about the playback position specified by user B's specifying operation to communication terminal 110-1 via the communication unit 501 of communication terminal 110-2.

In step S830, the communication unit 501 of communication terminal 110-1 notifies the image sharing unit 505 of communication terminal 110-1 of the playback position information received from communication terminal 110-2.

In step S831, the image sharing unit 505 of communication terminal 110-1 acquires an image (still image) corresponding to the notified playback position from the video data acquired by the content data acquisition unit 503.

In step S832, the image sharing unit 505 of the communication terminal 110-1 notifies the display control unit 506 of the communication terminal 110-1 of the image acquired in step S831. In response, in step S833, the display control unit 506 displays the notified image on the shared screen 906 included in the display screen 710 of the communication terminal 110-1, for example, as shown in FIG. 9(C).

In steps S834 and S835, the image sharing unit 505 notifies the other communication terminal 110-2 of the image acquired in step S831 via the communication unit 501, for example, in parallel with the processing of step S832.

Preferably, the image sharing unit 505 transmits the acquired image along with the playback position information, playback range information, etc., to other communication terminals such as the communication terminal 110-2.

In step S836, the display control unit 506 of communication terminal 110-2 updates the shared screen 721 of the display screen 720, for example, as shown in FIG. 9(C), with the image received from communication terminal 110-1. As a result, the shared screen 906 of communication terminal 110-1 and the shared screen 904 of communication terminal 110-2 display the three-dimensional object 714 viewed from the same direction, as shown in FIG. 9(C).

If communication terminal 110-3 is also participating in a conference in which communication terminals 110-1 and 110-2 are participating, communication terminal 110-1 (or communication management system 120) will also transmit the image sent to communication terminal 110-2 to the other communication terminal 110-3 participating in the conference. As a result, for example, as shown in FIG. 9(C), a shared screen 907 similar to the shared screen 721 displayed by communication terminals 110-1 and 110-2 will also be displayed on the display screen 730 displayed by communication terminal 110-3.

As described above, according to the second embodiment, the playback position of the video data acquired by communication terminal 110-1 can be specified from other communication terminals participating in the conference, such as communication terminal 110-2.

[Third embodiment]
In the third embodiment, an example will be described in which content data recorded from multiple directions of a three-dimensional object and shared by multiple communication terminals 110 is model data (electronic data) that is a three-dimensional model of the three-dimensional object.

<Functional configuration>
Fig. 10 is a diagram showing an example of the functional configuration of a communication terminal according to the third embodiment. As shown in Fig. 10, the communication terminal 110 according to the third embodiment has a model generation unit 1001 in addition to the functional configuration of the communication terminal 110 according to the first embodiment described in Fig. 5.

The model generation unit 1001 generates three-dimensional model data (electronic data) representing the shape of a three-dimensional object from, for example, video data of the three-dimensional object photographed from multiple directions, acquired by the content data acquisition unit 503. The three-dimensional model data created by the model generation unit 1001 may be an existing three-dimensional model such as a "wireframe," "surface," or "solid." The method for creating the model data may be the same as any existing three-dimensional modeling method.

In this embodiment, the image sharing unit 505 shares with other communication terminals 110 with which it is communicating an image viewed from a specified display position based on the model data of a three-dimensional model of a three-dimensional object generated by the model generation unit 1001. Furthermore, in this embodiment, information indicating the position of the viewpoint from which the modeled three-dimensional object is displayed (coordinate information, etc.) is used as information on the display position of the three-dimensional object.

Note that other functional configurations of the communication terminal 110 according to the third embodiment may be similar to the functional configuration of the communication terminal 110 according to the first embodiment described in FIG. 5.

<Processing flow>
Fig. 11 is a sequence diagram showing an example of a sharing process according to the third embodiment. This diagram shows an example of the sharing process when, for example, during communication (conference) between communication terminals 110-1 and 110-2 having the functional configurations shown in Fig. 10, user A using communication terminal 110-1 shares model data of a three-dimensional object.

It is assumed that at the start of the processing shown in FIG. 11, communication terminal 110-1 and communication terminal 110-2 are communicating (conference) to send and receive video data and audio data via communication network 101. A detailed description of processing similar to that in the first embodiment will be omitted here.

In step S1101, during communication between communication terminal 110-1 and communication terminal 110-2, when user A using communication terminal 110-1 performs a sharing start operation on communication terminal 110-1, the processing from step S1102 onwards is executed.

In steps S1102 and S1103, when the operation reception unit 504 of communication terminal 110-1 receives a sharing start operation from user A, it requests the content data acquisition unit 503 to start and end the capture of video data captured by the camera 312, etc. Note that the processing in steps S1101 to S1103 may be similar to the processing in steps S601 to S603 in the first embodiment.

Here, if the model sharing setting of the communication system 100 is set to enabled, processing from step S1111 onwards is executed. On the other hand, if the model sharing setting of the communication system 100 is set to disabled, processing from step S604 onwards in Figure 6 is executed.

In step S1111, the operation reception unit 504 of the communication terminal 110-1 uses the model generation unit 1001 to generate model data that is a 3D model of a three-dimensional object from the video data captured in steps S1102 and S1103.

Figure 12 (A) shows an image of three-dimensionally modeled model data. For example, if the three-dimensional object is a die 1201, the model generation unit 1001 creates a three-dimensional surface model of each face of the die 1201 to create model data 1202. However, this is not limited to this, and the model generation unit 1001 may also create a three-dimensional wireframe model or solid model of the die 1201 to create model data 1202.

In step S1116, the operation reception unit 504 of the communication terminal 110-1 sends a completion notification to the display control unit 506, etc., indicating that modeling of the three-dimensional object has been completed.

In step S1117, the display control unit 506 of the communication terminal 110-1 displays a shared screen on the display screen displayed by the communication terminal 110-1 based on the model data that is a 3D model of the three-dimensional object 714. Note that on the shared screen 711 according to the first embodiment, the playback position of the video data was specified using the seek bar 715. On the other hand, on the shared screen according to the third embodiment, the display position of the three-dimensional object is specified, for example, using coordinate information indicating the position of the viewpoint 1211 from which the modeled three-dimensional object is displayed in a spherical coordinate system such as that shown in FIG. 12(B).

Note that in step S1117, when the display control unit 506 of communication terminal 110-1 displays the shared screen, steps S1121 to S1128 are repeatedly executed until, for example, an operation to end sharing is received in step S1131.

In step S1121, the operation reception unit 504 of communication terminal 110-1 receives a display position specification operation from user A. User A, for example, rotates the three-dimensional object displayed on the shared screen up, down, left, or right using a drag operation or the like to specify the display position of the three-dimensional object.

In step S1122, the operation reception unit 504 of the communication terminal 110-1 notifies the image sharing unit 505 of the received coordinate information of the display position.

In step S1123, the image sharing unit 505 of communication terminal 110-1 acquires (creates) an image of the three-dimensional object as seen from the specified display position based on the model data generated by the model generation unit 1001 and the coordinate information of the display position notified in step S1122.

In step S1124, the image sharing unit 505 of communication terminal 110-1 notifies the display control unit 506 of communication terminal 110-1 of the image acquired in step S1123.

In step S1125, the display control unit 506 of the communication terminal 110-1 displays the image notified by the image sharing unit 505 on a shared screen included in the display screen of the communication terminal 110-1.

In steps S1126 and S1127, the image sharing unit 505 of communication terminal 110-1 transmits the image acquired in step S1123 to other communication terminals with which it is communicating, such as communication terminal 110-2, via the communication unit 501 of communication terminal 110-1.

In step S1128, the display control unit 506 of communication terminal 110-2 displays the image received from communication terminal 110-1 on a shared screen included in the display screen of communication terminal 110-2.

In this way, content data recorded from multiple angles of a three-dimensional object is not limited to video data of a three-dimensional object, but may also be model data of a three-dimensional model of the three-dimensional object.

Note that the processing shown in FIG. 11 is an example. For example, in steps S1102, S1103, and S1111 in FIG. 11, instead of generating model data of the three-dimensional object, the conference control unit 502 may acquire model data in which the three-dimensional object has been previously three-dimensionally modeled from the storage unit 507, an external device, etc.

[Fourth embodiment]
In the fourth embodiment, an example of processing will be described in which an area including a three-dimensional object is selected and shared from video captured by a camera with a wider angle of view than a general camera, such as a 360-degree camera. Here, the 360-degree camera is a camera that can capture video in 360 degrees up, down, left, and right, and is also called, for example, a spherical camera or an omnidirectional camera.

The functional configuration of the communication terminal 110 according to the fourth embodiment may be similar to the functional configuration of the communication terminal 110 according to the first embodiment shown in FIG. 5, for example.

<Processing flow>
Fig. 13 is a sequence diagram showing an example of a sharing process according to the fourth embodiment. Note that the basic flow of the process is similar to the sharing process according to the first embodiment described with reference to Fig. 1, and therefore differences from the first embodiment will be mainly described here.

In steps S601 to S603, when the operation reception unit 504 of communication terminal 110-1 receives a sharing start operation from user A, it uses the content data acquisition unit 503 to capture video data captured by camera 312, as in the first embodiment. However, in the fourth embodiment, it is assumed that camera 312 is a camera with a wider angle of view than a typical camera, such as a 360-degree camera.

In step S1301, the operation reception unit 504 of the communication terminal 110-1 sends a capture completion notification to the display control unit 506, indicating that the capture has been completed.

In step S1302, the display control unit 506 of communication terminal 110-1 displays a shared area selection screen, for example, as shown in FIG. 14.

Figure 14 is a diagram showing an example of a display screen according to the fourth embodiment. In the example of Figure 14, a display screen 1400 displayed by the display control unit 506 of communication terminal 110-1 displays a shared area selection screen 1401 that displays the images captured in steps S602 and S603. From this shared area selection screen 1401, user A can select, for example, by dragging, a shared area 1402 that includes a three-dimensional object 714 or the like to be shared.

In step S1303, when user A performs a selection operation to select shared area 1402 from the shared area selection screen 1401 as shown in FIG. 14, the processing from step S1304 onwards is executed.

In steps S1304 and S1305, the operation reception unit 504 of the communication terminal 110-1 notifies the image sharing unit 505 and the display control unit 506 of information about the selected area (e.g., coordinate range, etc.).

In step S1306, the display control unit 506 of the communication terminal 110-1 displays the shared screen using the video data in the selected shared area 1402. As a result, the shared screen 711 is displayed on the display screen 710 displayed by the communication terminal 110-1, as in the first embodiment.

Note that in step S1306, when the display control unit 506 of communication terminal 110-1 displays the shared screen 711, steps S1311 to S1318 are repeatedly executed until, for example, a sharing end operation is received in step S1321.

In step S1311, when the operation acceptance unit 504 of communication terminal 110-1 accepts an operation by user A to specify a playback position, the processing from step S1312 onwards is executed. As in the first embodiment, user A can specify the playback position of the video data by, for example, moving the seek bar 715 left or right on the shared screen 711 as shown in FIG. 7.

In step S1312, the operation reception unit 504 of the communication terminal 110-1 notifies the image sharing unit 505 of the received playback position information.

In step S1313, the image sharing unit 505 of communication terminal 110-1 acquires the image corresponding to the specified playback position within the selected area from the video data acquired by the content data acquisition unit 503.

In step S1314, the image sharing unit 505 of the communication terminal 110-1 notifies the display control unit 506 of the communication terminal 110-1 of the image acquired in step S1313. In response, in step S1315, the display control unit 506 displays the notified image on a shared screen 711 included in the display screen 710 of the communication terminal 110-1, for example, as shown in FIG. 7.

In steps S1316 and S1317, the image sharing unit 505 transmits the image acquired in step S1313 to the other communication terminal 110-2 with which it is communicating via the communication unit 501.

Preferably, the image sharing unit 505 transmits, along with the acquired image, information about the playback position (e.g., the time from the start point to the playback position, the number of frames, etc.), information about the playback range (e.g., the time from the start point to the end point, the number of frames, etc.), etc. to the communication terminal 110-2.

In step S1318, the display control unit 506 of the communication terminal 110-2 displays the image received from the communication terminal 110-1 on a shared screen 721 included in the display screen 720 of the communication terminal 110-2, for example, as shown in FIG. 7. As a result, the shared screen 711 of the communication terminal 110-1 and the shared screen 721 of the communication terminal 110-2 display the three-dimensional object 714 viewed from the same direction, as in the first embodiment.

In step S1321, for example, if the operation reception unit 504 of communication terminal 110-1 receives a sharing termination operation, communication terminal 110-1 ends the processing of steps S1311 to S1318.

In the above example, the camera 312 was described as a 360-degree camera, but this embodiment is not limited to video data from a 360-degree camera and can be widely applied to cases where only part of the capture range of video data is selectively shared.

As described above, according to each embodiment of the present invention, a communication terminal that communicates with other communication terminals via a communication network can quickly start sharing content data recorded from multiple directions of a three-dimensional object with other communication terminals.

<Additional Information>
Each function of each of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

Furthermore, the program for the communication terminal 110 may be distributed by recording it on a computer-readable recording medium, for example, as an installable or executable file. Furthermore, the program for the communication terminal 110 may not only be recorded on a recording medium, but may also be downloaded from a server (cloud or on-premise) and installed on the communication terminal 110, for example.

Furthermore, the communication management system 120 is not limited to the computer 400 shown in FIG. 4, but may also be implemented as electronic devices such as mobile phones, smartphones, tablet devices, wearable devices, PDAs (Personal Data Assistants), and image forming devices.

100 Communication system 101 Communication network 110 Communication terminal 503 Content data acquisition unit 504 Operation reception unit 505 Image sharing unit 506 Display control unit 710 Display screen 715 Seek bar (an example of a display element)

JP 2015-52896 A

Claims (9)

A communication terminal that communicates with other communication terminals via a communication network,
a content data acquisition unit that acquires content data recorded from a plurality of directions of a three-dimensional object;
an operation receiving unit that receives a designation of a display position of the content data;
an image sharing unit that shares an image corresponding to the display position of the content data with the other communication terminal;
a display control unit that causes a display screen including the image shared by the image sharing unit to be displayed on a display unit;
and
the display control unit displays a display element for specifying the display position on the display screen;
The communication terminal, wherein the image sharing unit transmits an image corresponding to the display position of the content data to the other communication terminal when the display position is specified by the display element. The image sharing unit
sharing an image of the display position of the content data and information about the display position with the other communication terminal;
2. The communication terminal according to claim 1 , wherein, when designation information designating the display position is received from the other communication terminal, an image of the display position of the content data and information about the display position are transmitted to the other communication terminal. 3. The communication terminal according to claim 2, wherein the image sharing unit permits the other communication terminal to specify the display position in response to a request from the other communication terminal, and accepts the specification information from the other communication terminal that has been permitted to specify the display position. the content data includes video data of the three-dimensional object photographed from a plurality of directions,
The communication terminal according to claim 1 , wherein the information on the display position includes information indicating a playback position of the video data. the content data includes electronic data that models the three-dimensional object,
The communication terminal according to claim 1 , wherein the information on the display position includes information indicating a position of a viewpoint from which the modeled three-dimensional object is displayed. when receiving, from the other communication terminal, an image corresponding to one display position of content data acquired by the other communication terminal, the image sharing unit shares the image corresponding to the one display position with the other communication terminal ;
The communication terminal according to claim 1 , wherein the display control unit displays an image corresponding to the one display position on the display screen. A communication system including a plurality of communication terminals that communicate with other communication terminals via a communication network,
The communication terminal
a content data acquisition unit that acquires content data recorded from a plurality of directions of a three-dimensional object;
an operation receiving unit that receives a designation of a display position of the content data;
an image sharing unit that shares an image of the display position of the content data with the other communication terminal;
a display control unit that causes a display screen including the image shared by the image sharing unit to be displayed on a display unit;
and
the display control unit displays a display element for specifying the display position on the display screen;
When the display position is specified by the display element, the image sharing unit transmits an image corresponding to the display position of the content data to the other communication terminal . A communication terminal that communicates with other communication terminals via a communication network,
A process of acquiring content data recorded from a plurality of directions of a three-dimensional object;
a process of receiving a designation of a display position of the content data;
an image sharing process for sharing an image of the display position of the content data with the other communication terminal;
a display control process for displaying a display screen including an image shared with the other communication terminal;
Run
the display control process displays a display element that specifies the display position on the display screen;
The image sharing process is a communication method in which, when the display position is specified by the display element, an image corresponding to the display position of the content data is transmitted to the other communication terminal . A communication terminal that communicates with other communication terminals via a communication network,
A process of acquiring content data recorded from a plurality of directions of a three-dimensional object;
a process of receiving a designation of a display position of the content data;
an image sharing process for sharing an image of the display position of the content data with the other communication terminal;
a display control process for displaying a display screen including an image shared with the other communication terminal;
Execute
the display control process displays a display element that specifies the display position on the display screen;
The image sharing process is a program for transmitting an image corresponding to the display position of the content data to the other communication terminal when the display position is specified by the display element .