JP7772042B2 - Communication terminal, program, display method, recording medium - Google Patents

Description

The present invention relates to a communication terminal, a program, a display method, and a recording medium.

Communication systems are becoming commonplace, allowing communication terminals to communicate between multiple locations via communication networks such as the Internet or LANs (Local Area Networks) and send and receive video and audio between them. Using a communication system, users in remote locations can communicate, for example, to hold video conferences.

Telepresence robots, in which communication systems are applied to robots, are also known. Telepresence robots are robots that combine "video conferencing" and "remote control technology." By combining remote control from a distance with robot technology, this technology allows an operator to operate a robot from a distance and make it behave as if the operator is actually present in a certain location.

Technology has also been devised for mounting multiple cameras on robots (see, for example, Patent Document 1). Patent Document 1 discloses a mobile object equipped with a front camera and a rear camera that capture images in front of and behind the traveling device, as well as an overhead camera that can capture the entire surrounding area near the robot.

However, with conventional technology, the operator can display any direction within a wide-angle image, such as a full-surround image, which poses a problem: when the operator is looking in a certain direction within the wide-angle image, it is difficult to determine which direction the mobile device is facing.

For example, if the operator moves the mobile device forward when the direction the operator is looking is different from the front direction of the mobile device, the mobile device will move in a direction that the operator cannot see, resulting in reduced operability.

In consideration of the above-mentioned problems, the present invention aims to provide a communication terminal that makes it easy for an operator to understand in which direction a mobile device is facing when looking in a certain direction of a wide-angle image.

In view of the above problems, the present invention provides a communication terminal that communicates with other communication terminals mounted on a mobile device via a network, the communication terminal comprising: a communication means for receiving wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal; and a display processing means for displaying at least a portion of the wide-angle image received by the communication means on a display unit of the communication terminal that is different from the other communication terminal, wherein the display processing means displays an image of the mobile device that allows the orientation of the mobile device to be determined on the display unit on which at least a portion of the wide-angle image is displayed; and the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be determined changes orientation in accordance with the change when at least a portion of the wide-angle image displayed on the display unit changes.

This provides a communication terminal that makes it easy for the operator to understand in which direction the mobile device is facing when looking in a certain direction of the wide-angle image.

FIG. 1 is a diagram illustrating an example of an outline of a communication system. FIG. 1 is a diagram illustrating a display example of a spherical image captured by a wide-angle imaging device. FIG. 1 is a diagram illustrating an example of the configuration of a communication system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication management system. FIG. 2 illustrates an example of a hardware configuration of a mobile device. FIG. 1 is a diagram illustrating a hardware configuration of an example of a wide-angle imaging device. FIG. 1 is a diagram illustrating an example of an image of a wide-angle imaging device in use. 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. FIG. 12 is a diagram showing an example of a three-dimensional perspective view of FIG. 11 . 10 is a diagram illustrating an example of a relationship between predetermined area information and an image of a predetermined area. 1 is a functional block diagram showing a communication management system included in a communication system, a communication terminal 10A, and functions of the communication terminal 10A in blocks. FIG. 10 is a sequence diagram illustrating an example of a process in a preparation stage for communication in the communication system. FIG. 10 is a diagram illustrating an example of a destination selection screen displayed on the communication terminal. FIG. 10 is a sequence diagram illustrating an example of communication processing in the communication system. 10A and 10B are diagrams illustrating an example of a reference imaging direction that serves as a reference for an imaging direction and the front direction of a mobile device. 10A and 10B are diagrams illustrating an example of the relationship between an arbitrary imaging direction displayed by an operator and a reference imaging direction. 1 is a diagram illustrating an example of a perspective projection transformation; FIG. 10 is a diagram showing an example of a spherical video display screen displayed in pattern 1. 10 is a flowchart illustrating an example of a procedure in which a communication terminal displays a predetermined area image on a display unit. FIG. 10 is a diagram showing an example of a polygon model of a telepresence robot and an object showing an imaging direction arrow. FIG. 10 is a diagram showing another example of display of the imaging direction arrow in pattern 2. FIG. 10 is a flowchart illustrating an example of a procedure in which a communication terminal displays a spherical video on a display unit in pattern 2. FIG. 10 is a flowchart illustrating an example of a procedure in which the communication terminal displays a spherical video on the display unit in Pattern 3. 10 is an example of a simultaneous video display screen that simultaneously displays a spherical video image and a front video image (or a rear video image).

Below, we will explain a communication device and a display method performed by the communication device as an example of an embodiment of the present invention.

<Outline of operation of communication system and communication terminal>
First, an overview of the communication system will be explained using Fig. 1. Fig. 1(a) shows an example of a schematic configuration diagram of a communication system using a telepresence robot. A telepresence robot is placed in a company 801, and an operator 810 is present at home 802. The telepresence robot has a communication terminal 10B and a wide-angle imaging device 9 mounted on a mobile device 20. The operator 810 also has a communication terminal 10A.

The wide-angle imaging device 9 transmits a celestial sphere image (an example of a second image) described below to the communication terminal 10B. The communication terminals 10A and 10B each have a planar imaging device and a microphone, and transmit and receive images and audio to and from each other. That is, the communication terminal 10B transmits the celestial sphere image and a front image 831 (or/and a rear image 832) to the communication terminal 10A, and the communication terminal 10A transmits the front image 831 to the communication terminal 10B.

In addition, application software (hereinafter referred to as the app) that accepts operations related to the movement of mobile device 20 is running on communication terminal 10A, and operator 810 inputs operational instructions while viewing the video transmitted by communication terminal 10B. The operational instructions input by operator 810 are transmitted from communication terminal 10A to communication terminal 10B.

The communication terminal 10B and the mobile device 20 can communicate via short-range wireless communication such as Bluetooth (registered trademark), and the communication terminal 10B controls the mobile device 20 based on operational instructions, allowing the operator 810 to move the mobile device 20 from a remote location (home).

By moving the mobile device 20 in this manner, the operator 810 can converse with the desired person 820, as shown in Figure 1(b). In Figure 1(b), the operator 810 moves the mobile device 20 to the location of person A 820a to converse with A, and then moves the mobile device 20 to the location of person B 820b to converse with B.

In addition to meeting people, the operator 810 can also do many things remotely without having to travel to the location, such as checking out products in remote stores, inspecting factories, and attending exhibitions.

Furthermore, if the planar imaging device built into communication terminal 10B could only capture images from the front and rear, it would be difficult for operator 810 to check the situation around mobile device 20 (for example, underfoot or to the left and right). Therefore, in this embodiment, communication terminal 10B is equipped with a wide-angle imaging device 9 that can capture images of a 360-degree surrounding area. The wide-angle imaging device 9 may be mounted on communication terminal 10B or on mobile device 20. Because the wide-angle imaging device 9 can capture images over a wide angle range, it is positioned, for example, above where the surroundings are less likely to be obstructed. In Figure 1, it is positioned on the top side of communication terminal 10B, but it may also be positioned on the left or right side, or on the pole part of mobile device 20. The wide-angle imaging device 9 may be built into communication terminal 10B or may be externally attached.

<Example of spherical video display>
2 will be used to describe a display example that allows the operator to understand the direction in which mobile device 20 is facing when the operator is looking in a certain direction of a celestial sphere image that captures a subject in a 360-degree surrounding area. Fig. 2 is a diagram that describes a display example of a predetermined area image that is a part of the celestial sphere image captured by wide-angle imaging device 9. In this embodiment, three patterns of display examples that allow the operator to understand the direction in which mobile device 20 is facing will be described.

Note that the orientation of the mobile device 20 is the same as that of the telepresence robot, and therefore the mobile device 20 and the telepresence robot may be considered to be the same in terms of orientation. Furthermore, the orientation of the mobile device 20 refers to the direction in which the mobile device 20 is facing, for example, the forward direction or the forward direction. As long as the operator can grasp the direction in which the mobile device 20 is facing, the orientation of the mobile device 20 may be sideways, upward, downward, etc.

FIG. 2(a) shows an example of a spherical image display screen 1010 that displays a spherical image using pattern 1. Pattern 1 allows the operator to understand the orientation of the mobile device 20 by superimposing an image 840 of a telepresence robot on the spherical image that the operator is currently viewing. The operator can see in which direction they are looking relative to the orientation of the mobile device 20. Note that the image 840 of the telepresence robot is displayed in a manner that allows the orientation of the telepresence robot to be understood.

The operator causes communication terminal 10A to display any predetermined area image. The image displayed on spherical image display screen 1010 is the predetermined area image. In FIG. 2(a), part of an open living room is displayed. The predetermined area image is an image of a part of the spherical image that is displayed on the display device (display unit 109, described later) of communication terminal 10A. Display unit 109 sets an area, and the user displays the predetermined area image within this area.

Therefore, this does not necessarily coincide with the front direction of the mobile device 20. The communication terminal 10A rotates the telepresence robot based on the difference between the front direction of the mobile device 20 and the imaging direction that defines the specified area image (the direction that indicates the center of the specified area image), and displays the rotated telepresence robot in the omnidirectional image. The displayed image 840 of the telepresence robot is not a telepresence robot in real space, but is created using CG (computer graphics). CG is an image drawn using a computer. CG is mainly available in two dimensions and three dimensions, but in this embodiment, three-dimensional CG is used.

Therefore, it is possible to see in which direction the telepresence robot is facing in the spherical image. In the example of Figure 2(a), the telepresence robot is facing slightly to the right of the operator. By looking at image 840 of the telepresence robot, the operator can easily determine in which direction the robot will move if it continues to move forward.

Note that for ease of explanation, Figure 2(a) shows the entire image of the telepresence robot, image 840. However, because the omnidirectional video is positioned above the telepresence robot, it is preferable to display image 840 of the telepresence robot closer to the viewpoint (closer). Therefore, it is not necessary for the entire image of image 840 of the telepresence robot to be shown.

Next, pattern 2 will be described. FIG. 2(b) shows an example of a spherical image display screen 1010 that displays a spherical image using pattern 2. Pattern 2 displays the direction in which the operator is looking relative to the telepresence robot, in addition to a predetermined area image 853 that is part of the spherical image that the operator is looking at. Specifically, an image 854 of the telepresence robot and an imaging direction arrow 8 are displayed.

As shown in FIG. 2(b), communication terminal 10A displays an arbitrary predetermined area image 853 in omnidirectional image field 1001 in response to an operation by the operator. Displayed products are shown in FIG. 2(b). Communication terminal 10A also has a telepresence robot display field 1002 separate from omnidirectional image field 1001. Telepresence robot display field 1002 displays an imaging direction arrow 8 indicating the direction in which the predetermined area image 853 is viewed from the telepresence robot. In other words, imaging direction arrow 8 three-dimensionally represents the imaging direction viewed by the operator. "Three-dimensionally" means that it contains information about azimuth and elevation angles. The displayed image 854 of the telepresence robot is not a telepresence robot in real space, but is created using CG. The same is true for imaging direction arrow 8. In FIG. 2(b), the imaging direction is toward the front of the telepresence robot.

In this way, the operator can see in which direction the spherical image is viewed from the telepresence robot. The operator can also see in which direction the telepresence robot is facing. Also, in Figure 2(b), it can be seen that the telepresence robot approaches the product by moving forward. Compared to pattern 1, this has the advantage that the telepresence robot does not overlap with the spherical image.

FIG. 2(c) shows an example of a spherical image display screen 1010 displaying a spherical image according to pattern 3. The image displayed on the spherical image display screen 1010 is a predetermined area image. In pattern 3, the communication terminal 10A displays the operation button 860 only when the center direction of the predetermined area image (the imaging direction described above) and the front direction of the telepresence robot approximately match. In the left diagram c-1 of FIG. 2(c), the imaging direction and the front direction of the telepresence robot do not approximately match, so the operation button 860 is not displayed. In the right diagram c-2 of FIG. 2(c), the imaging direction and the front direction of the telepresence robot approximately match, so the operation button 860 is displayed. "Approximate match" refers to a match that does not impede the movement of the mobile device 20. If perfect match were required, the operation button 860 would rarely be displayed and would be inoperable. Therefore, a margin of error that does not impair operability is determined. For example, a deviation of approximately ±10 to ±20 degrees in both the horizontal and elevation directions may be tolerated. It would also be good if the operator could set the amount of deviation.

In pattern 3, operation button 860 is displayed when the imaging direction and the front direction of the telepresence robot are approximately aligned, so the operator can determine that the imaging direction and the front direction of the telepresence robot are approximately aligned when the operation button 860 is displayed after changing the specified area image. This allows the imaging direction to be aligned with the direction of the omnidirectional image viewed by the operator. Furthermore, operation button 860 is displayed only when the front direction of the telepresence robot and the direction of the omnidirectional image viewed by the operator are aligned, so the operator can know where the robot will move if it is moved forward.

<Terminology>
The operator 810 is a person who operates the mobile device 20. The operator 810 is also a user of the communication terminal 10A, but will be referred to as the operator in this embodiment.

The conversation partner 820 is typically a person within the company, but may also be an animal such as a dog or cat. Furthermore, the mobile device 20 does not necessarily have to be used by the operator 810 to talk to the other party. For example, if the mobile device 20 is patrolling within the company, the communication terminal 10B may simply transmit video and audio to the communication terminal 10A, and the operator may not need to talk.

The location where the mobility device 20 moves may be any location where the mobility device 20 can move. Furthermore, the device is not limited to being powered by a motor or other means, and movement may be assisted by a human.

A frontal image refers to an image with a normal angle of view, while a spherical image refers to an image with a wider angle than a frontal image (wide-angle image). A normal angle of view is an angle of view where distortion is acceptable even when captured with a perspective projection lens, and can simply be referred to as an image captured with a perspective projection lens. A wide-angle image refers to an image captured with a wide-angle lens such as a fisheye lens. Projection methods include stereoscopic projection, equidistant projection, equisolid angle projection, and orthogonal projection. Wide-angle images are not limited to spherical images that capture a 360-degree surrounding area, but can also include images captured on a hemisphere or images captured horizontally from 180 to 360 degrees.

In this embodiment, the term "video" is used when referring to a moving image, and the term "image" is used when referring to a single still image; however, video may contain multiple images, and no strict distinction will be made between video and images.

<System configuration example>
FIG. 3 is a diagram illustrating an example of the configuration of a communication system according to an embodiment. The communication system 1 includes a plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E), a mobile device 20, a relay device 30, and a communication management system 50. In the following description, the term "communication terminal 10" will be used to refer to any one of the plurality of communication terminals (10A, 10B1, 10B2, 10B3, 10E). Furthermore, the term "communication terminal 10B" will be used to refer to any one of the communication terminals (10B1, 10B2, 10B3). The number of communication terminals 10 in FIG. 3 is merely an example.

The communication terminal 10, relay device 30, and communication management system 50 are each connected to other communication terminals 10, devices, and systems via a communication network 2 so that they can communicate with each other. The communication network 2 may include, for example, a local area network (LAN), the Internet, a mobile phone network, or a dedicated line.

The communication terminal 10 may be, for example, a general-purpose information processing device such as a tablet terminal, smartphone, or PC (Personal Computer), or a communication terminal such as a dedicated video conferencing device. The communication terminal 10 can, for example, conduct video conferences by transmitting and receiving, for example, image data, audio data, etc., between one or more other communication terminals 10.

By executing an app compatible with communication system 1, communication terminal 10A can hold a video conference with communication terminal 10B and remotely control mobile device 20 via communication terminal 10B. For example, communication terminal 10A can move mobile device 20 equipped with communication terminal 10B, for example, forward/backward or left/right, by operating operation buttons displayed on the video conference display screen. Communication terminal 10A can also run browser software, which can display images and accept operations from mobile device 20 on the browser software.

The mobile device 20 and communication terminal 10B (including the wide-angle imaging device 9) are referred to as a telepresence robot. The mobile device 20 is a device with a travel function that moves "forward," "backward," "right turn," "left turn," etc., by driving multiple wheels in response to control from the communication terminal 10B attached to the mobile device 20. When a user commands left or right movement, the mobile device moves forward after making a "right turn" or a "left turn." Note that the appearance of the mobile device 20 shown in Figure 3 is merely an example. It is sufficient for the mobile device 20 to be able to move together with the communication terminal 10B in response to operational commands from the communication terminal 10B mounted on the mobile device 20.

The relay device 30 is, for example, an information processing device or a system including one or more information processing devices, and relays content data (video, audio, operation instruction information) transmitted and received between multiple communication terminals 10. However, multiple communication terminals 10 may also transmit and receive content data directly without going through the relay device 30.

The communication management system 50 is, for example, an information processing device or a system including one or more information processing devices. The communication management system 50 performs, for example, login authentication from the communication terminal 10, management of the communication status of the communication terminal 10, management of the destination list, and control of sessions in which communication takes place between multiple communication terminals 10 via the relay device 30.

In one embodiment, a session is realized by a relay device 30 relaying content data including image data and sound data (voice and other sounds) between multiple communication terminals 10.

In the above configuration, for example, the operator of communication terminal 10A can communicate with communication terminal 10B and remotely move communication terminal 10B and mobile device 20. This allows the operator of communication terminal 10A to move communication terminal 10B and mobile device 20 near any desired party and hold a video conference, etc.

The communication system 1 also includes a data provision system that transmits content data unidirectionally from one communication terminal 10 to another communication terminal 10 via a communication management system 50, and a communication system that transmits information, emotions, etc. between multiple communication terminals 10 via the communication management system 50. This communication system is a system for transmitting information, emotions, etc. between multiple communication terminals via a relay device 30, and examples of this system include a television (or video) conferencing system and a videophone system.

<Hardware configuration>
<<Hardware configuration of communication terminal>>
4 is a diagram illustrating an example of the hardware configuration of a communication terminal according to an embodiment of the present invention. Communication terminal 10 includes a typical computer configuration, such as a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, a flash memory 104, and a solid-state drive (SSD) 105. Communication terminal 10 also includes a media interface (I/F) 107, an input unit 108, a display unit 109, a network interface (I/F) 111, a camera 112, an image sensor interface (I/F) 113, a microphone 114, a speaker 115, and an audio input/output interface (I/F) 116. Communication terminal 10 also includes an external device connection interface (I/F) 117, a short-range wireless communication unit 118, and a bus 119.

CPU 101 is a computing device that implements the various functions of communication terminal 10 by reading programs and data from, for example, ROM 102 or flash memory 104 and executing the processes. ROM 102 is non-volatile memory that pre-stores programs used to start CPU 101, such as IPL (Initial Program Loader). RAM 103 is volatile memory used as a work area for CPU 101, etc.

Flash memory 104 is a storage device that stores, for example, an operating system (OS), application programs, and various types of data. SSD 105 controls the reading and writing of various types of data from flash memory 104 under the control of CPU 101. Media I/F 107 controls the reading and writing (storage) of data from recording medium 106, which is a recording medium such as a memory card. The recording medium can store a computer-readable program that controls communication terminal 10.

The input unit 108 is an input device, such as a touch panel, keyboard, or pointing device, for accepting input operations from the operator. Voice input may also be possible. The display unit 109 is a display device that displays various information to the operator. Note that the input unit 108 and display unit 109 may be a display input unit 110, such as a touch panel display that integrates a touch panel and a display.

Network I/F 111 is a communications interface that enables communication terminal 10 to send and receive data via communications network 2. Camera 112 includes an image sensor for capturing an image of a subject under the control of CPU 101. Image sensor I/F 113 controls image capture by camera 112 and converts the captured data into specified image data. Camera 112 is a planar imaging device that captures images with a narrower angle of view than wide-angle imaging device 9, but has a higher resolution than wide-angle imaging device 9. Resolution refers to the degree of detail in a digital image, and is expressed by quantifying the detail of the individual unit points (dots, pixels) that make up a digital image. Resolution is generally expressed in units of "dots." In the case of displays, it is often expressed as a horizontal x vertical arrangement of numbers, such as "1024 x 768 dots." When compared over the same imaging range, the number of dots in wide-angle imaging device 9 is fewer than that of camera 112.

The microphone 114 converts recorded audio into an electrical signal. The speaker 115 converts the audio signal into audio and outputs it. The audio input/output I/F 116 controls the input and output of audio via the microphone 114 and speaker 115.

The external device connection I/F 117 is an interface for connecting external devices such as a USB (Universal Serial Bus). Examples of external devices include the mobile device 20 shown in Figure 3.

The short-range wireless communication unit 118 is a communication interface for communicating with an external device (e.g., a mobile device 20) via short-range wireless communication, such as Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The bus 119 is commonly connected to each of the above components and transmits address signals, data signals, various control signals, etc.

<<Hardware configuration of communication management system and relay device>>
5 is a diagram showing an example of the hardware configuration of a communication management system according to an embodiment. The communication management system 50 has the configuration of a typical computer, including, for example, a CPU 501, a ROM 502, a RAM 503, a HDD 504, a HDD (Hard Disk Drive) 505, a media drive 507, and a display 508. The communication management system 50 also includes a network I/F 509, a keyboard 511, a mouse 512, a CD-ROM drive 514, and a bus 510.

The CPU 501 is a computing device that realizes each function of the communication management system 50 by reading and processing programs and data stored, for example, in the ROM 502, HDD 504, etc. The ROM 502 is a non-volatile memory that pre-stores programs used to start the CPU 501, such as IPL. The RAM 503 is a volatile memory used as a work area for the CPU 501, etc.

The HDD 504 is a storage device that stores, for example, the OS, programs such as application programs, and various data. The HDD 505 controls the reading and writing of various data from the HDD 504 under the control of the CPU 501. The display 508 is a display device that displays various information such as a cursor, menus, windows, characters, or images.

Network I/F 509 is a communications interface for data communication using communications network 2. Keyboard 511 is an example of an input device for accepting input operations such as characters, numbers, and various instructions from the system administrator. Mouse 512 is an example of a pointing device for accepting operations such as selecting and executing various instructions, selecting processing targets, and moving the cursor from the system administrator.

Note that the communication management system 50 or relay device 30 does not normally need to have a display 508, keyboard 511, or mouse 512. In this case, they may be connected as needed.

The media drive 507 controls the reading and writing (storing) of data from and to a recording medium 506, such as a memory card. The CD-ROM drive 514 controls the reading and writing of data from and to a disk 513, an example of a removable recording medium. The bus 510 electrically connects the above components and transmits address signals, data signals, various control signals, etc.

Please note that the above computer hardware configuration is only an example.

The relay device 30 has the same hardware configuration as the communication management system 50. The programs for the communication terminal 10, relay device 30, and communication management system 50 may be recorded on computer-readable recording media in installable or executable file formats and distributed. Examples of such recording media include CD-Rs (Compact Disc Recordables), DVDs (Digital Versatile Disks), Blu-ray Discs, and USB memory. Recording media such as CD-ROMs on which the programs are stored, as well as HDDs 504 on which these programs are stored, can be provided domestically or internationally as program products.

<<Hardware configuration of mobile device>>
6 is a diagram showing an example of the hardware configuration of a mobile device 20 according to an embodiment. The mobile device 20 includes, for example, a CPU 401, a RAM 402, a ROM 403, an external device I/F 404, a short-range wireless communication unit 405, a wheel drive unit 406, a steering unit 407, and the like.

The CPU 401 is a computing device that realizes the various functions of the mobile device 20 by executing programs stored in the ROM 403 and the like. The RAM 402 is a volatile memory used as a work area for the CPU 401. The ROM 403 is a non-volatile memory that stores programs and the like for the mobile device 20. The ROM 403 may be, for example, a rewritable non-volatile memory such as a flash ROM.

The external device I/F 404 is a wired communication interface that establishes a wired connection with the external device connection I/F 117 of the communication terminal 10 and performs communication.

The short-range wireless communication unit 405 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the short-range wireless communication unit 118 of the communication terminal 10. The mobile device 20 may be capable of communicating with the communication terminal 10, for example, via the external device I/F 404 or the short-range wireless communication unit 405.

The wheel drive unit 406 is an example of a drive unit that drives wheels to move the moving device 20. The wheel drive unit 406 includes, for example, a motor.

The steering unit 407 is an example of a steering device that steers the moving device 20, which is moved by the wheel drive unit 406. The steering unit 407 may, for example, change the direction or inclination of the wheels, or may change the direction of the moving device 20 by controlling the rotation speed or velocity of the left and right wheels.

<<Example of hardware configuration of wide-angle imaging device>>
The hardware configuration of wide-angle imaging device 9 will be described using Figure 7. Figure 7 is a hardware configuration diagram of an example of wide-angle imaging device 9. In the following, wide-angle imaging device 9 is an omnidirectional (omnidirectional) wide-angle imaging device 9 that uses two imaging elements, but the number of imaging elements may be two or more. Furthermore, the device does not necessarily have to be dedicated to omnidirectional imaging; a typical digital camera, smartphone, or the like may be equipped with an omnidirectional imaging unit as a retrofit, thereby achieving substantially the same functions as wide-angle imaging device 9.

As shown in FIG. 7, the wide-angle imaging device 9 is also connected to an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a CPU (Central Processing Unit) 311, a ROM (Read Only Memory) 312, an SRAM (Static Random Access Memory) 313, a DRAM (Dynamic Random Access Memory) 314, an operation unit 315, a network I/F 316, a communication unit 317, an antenna 317a, an electronic compass 318, a gyro sensor 319, and an acceleration sensor 320.

Of these, imaging unit 301 is equipped with wide-angle lenses (so-called fisheye lenses) 302a and 302b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 303a and 303b corresponding to each wide-angle lens. Imaging elements 303a and 303b each include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by fisheye lenses 302a and 302b into electrical image data for output, a timing generation circuit that generates horizontal and vertical synchronization signals and pixel clocks for the image sensors, and a group of registers that set various commands and parameters required for the operation of the imaging elements.

The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 via a parallel I/F bus. Meanwhile, the imaging elements 303a and 303b of the imaging unit 301 are connected via a serial I/F bus (such as an I2C bus) separately from the imaging control unit 305. The image processing unit 304, imaging control unit 305, and sound processing unit 309 are connected to the CPU 311 via bus 310. Also connected to the bus 310 are ROM 312, SRAM 313, DRAM 314, operation unit 315, network I/F 316, communication unit 317, and electronic compass 318.

The image processing unit 304 takes in the image data output from the image sensors 303a and 303b via the parallel I/F bus, performs predetermined processing on each piece of image data, and then combines the image data to create equirectangular projection image data such as that shown in Figure 9(c).

The imaging control unit 305 generally sets commands and other data in the registers of the imaging elements 303a and 303b using the I2C bus, with the imaging control unit 305 acting as the master device and the imaging elements 303a and 303b acting as slave devices. Necessary commands and other data are received from the CPU 311. The imaging control unit 305 also uses the I2C bus to retrieve status data and other data from the registers of the imaging elements 303a and 303b and send it to the CPU 311.

In addition, the imaging control unit 305 instructs the image sensors 303a and 303b to output image data when the shutter button on the operation unit 315 is pressed. Depending on the wide-angle imaging device 9, it may have a preview display function or a function that supports video display on a display (for example, the display of the communication terminal 10B). In this case, the image data is output from the image sensors 303a and 303b continuously at a predetermined frame rate (frames per minute).

In addition, as described below, the imaging control unit 305 also functions as a synchronization control means that cooperates with the CPU 311 to synchronize the output timing of image data from the imaging elements 303a and 303b. In this embodiment, the wide-angle imaging device 9 is not provided with a display, but a display unit may be provided.

The microphone 308 converts sound into sound (signal) data. The sound processing unit 309 receives the sound data output from the microphone 308 via the I/F bus and performs predetermined processing on the sound data.

The CPU 311 controls the overall operation of the wide-angle imaging device 9 and executes necessary processing. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and DRAM 314 are work memories that store programs executed by the CPU 311 and data currently being processed. In particular, the DRAM 314 stores image data currently being processed by the image processing unit 304 and data of processed equirectangular projection images.

The operation unit 315 is a general term for operation buttons such as the shutter button. The operator operates the operation unit 315 to input various imaging modes and imaging conditions.

Network I/F 316 is a general term for interface circuits (such as USB I/F) with external media such as SD cards and personal computers. Network I/F 316 can be either wireless or wired. The equirectangular projection image data stored in DRAM 314 is recorded on external media via this network I/F 316, or, if necessary, transmitted via network I/F 316 to an external terminal (device) such as communication terminal 10B.

The communication unit 317 communicates with an external terminal (device) of the communication terminal 10B via an antenna 317a provided in the wide-angle imaging device 9 using short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth (registered trademark). This communication unit 317 can also transmit data of the equirectangular projection image to an external terminal (device) such as the communication terminal 10B.

The electronic compass 318 calculates the orientation of the wide-angle imaging device 9 from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with the Exchangeable Image File Format (Exif) and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time the image was captured and the data size of the image data.

The gyro sensor 319 detects changes in angle (roll angle, pitch angle, yaw angle) associated with the movement of the wide-angle imaging device 9. The changes in angle are an example of related information (metadata) in accordance with Exif, and are used for image processing such as image correction of captured images.

The acceleration sensor 320 detects acceleration in three axial directions. The attitude of the wide-angle imaging device 9 (angle relative to the direction of gravity) is detected based on the detected acceleration. Having both the gyro sensor 319 and the acceleration sensor 320 improves the accuracy of image correction.

<About spherical images>
Next, the usage of the wide-angle imaging device 9 will be described with reference to Fig. 8. Fig. 8 is an image diagram of the wide-angle imaging device 9 in use. As shown in Fig. 8, the wide-angle imaging device 9 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the imaging element 303a and the imaging element 303b shown in Fig. 7, respectively.

Next, an overview of the processing up to creating an equirectangular projection image EC and a celestial sphere image CE from an image captured by the wide-angle imaging device 9 will be described using Figures 9 and 10. Note that Figure 9(a) is a diagram showing a hemispherical image (front side) captured by the wide-angle imaging device 9, Figure 9(b) is a hemispherical image (rear side) captured by the wide-angle imaging device 9, and Figure 9(c) is a diagram showing an image expressed using equirectangular projection (hereinafter referred to as an "equirectangular projection image"). Figure 10(a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and Figure 10(b) is a diagram showing a celestial sphere image.

As shown in Figure 9(a), the image obtained by the image sensor 303a is a hemispherical image (front side) curved by the fisheye lens 302a (described below). Also, as shown in Figure 9(b), the image obtained by the image sensor 303b is a hemispherical image (rear side) curved by the fisheye lens 302b. The hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are then combined by the wide-angle imaging device 9 to create an equirectangular projection image EC, as shown in Figure 9(c).

By using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the spherical surface as shown in FIG. 10(a), and a celestial sphere image CE as shown in FIG. 10(b) is created. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The celestial sphere image CE may be a still image or a video.

As described above, the spherical image CE is an image pasted to cover the spherical surface, which can create an unnatural appearance to people. Therefore, by displaying a predetermined region of the spherical image CE (hereinafter referred to as the "predetermined region image" above) as a flat image with little curvature, it is possible to display an image that does not cause discomfort to people. This will be explained using Figures 11 and 12. The predetermined region image may be a video or a still image.

FIG. 11 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is represented as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the operator viewing the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 12(a) is a three-dimensional perspective view of FIG. 11, and FIG. 12(b) is a diagram showing an image of the predetermined area when displayed on a display. In FIG. 12(a), the omnidirectional image shown in FIG. 11 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this manner is represented as a three-dimensional sphere CS, then as shown in FIG. 12, the virtual camera IC is located inside the omnidirectional image CE. The predetermined area T in the omnidirectional image CE is the imaging area of the virtual camera IC and is specified by predetermined area information that indicates the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE. Zooming of the predetermined region T can also be achieved by moving the virtual camera IC closer to or further away from the spherical image CE. The predetermined region video Q is an image of the predetermined region T in the spherical image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the spherical image CE (see FIG. 13).

The predetermined area image Q shown in FIG. 12(a) is then displayed on a predetermined display as an image of the imaging area of the virtual camera IC, as shown in FIG. 12(b). The image shown in FIG. 12(b) is a predetermined area image represented by the initial (default) predetermined area information. The following explanation will use the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may also be represented by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, rather than the angle of view α and distance f.

Next, the relationship between the predetermined area information and the image of the predetermined area T will be explained using Figure 13. Figure 13 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in Figure 13, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). In other words, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the imaging direction (ea, aa), coincides with the center point CP of the predetermined area T, which is the imaging area of the virtual camera IC. As shown in Figure 13, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP becomes the (x, y) parameter of the predetermined area information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In Figure 13, the trigonometric function shown in Equation 1 below generally holds.

L/f=tan(α/2)...(Formula 1)
<About the function>
Next, the functions of the communication system will be described with reference to Fig. 14. Fig. 14 is a functional block diagram showing the functions of the communication management system 50, the communication terminal 10B, and the communication terminal 10A in the communication system 1 in block form.

<<Functional configuration of communication terminal 10A>>
The communication terminal 10A is a communication terminal 10 that accepts operational instructions for controlling devices such as the mobile device 20, but does not have the function of controlling devices such as the mobile device 20. The communication terminal 10A includes a transmission/reception unit 11, an operational input reception unit 12, a communication control unit 13, an imaging unit 14, an audio input unit 15a, an audio output unit 15b, a display control unit 16, and a storage/readout unit 17. Each of these units is a function or means realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 101 in accordance with a program for the communication terminal 10 that is loaded from the flash memory 104 onto the RAM 103. The communication terminal 10 also includes a storage unit 18 realized by the RAM 103 shown in FIG. 4 and the flash memory 104 shown in FIG. 4.

<<Functional configuration of communication terminal 10B>>
The communication terminal 10B is a communication terminal 10 mounted on the mobile device 20. The communication terminal 10B is an example of a communication terminal 10 having a function of controlling devices such as the mobile device 20. In addition to the functional configuration of the communication terminal 10A described above, the communication terminal 10B has an operation instruction receiving unit 19a, a celestial sphere video receiving unit 19b, a device control unit 19c, an inter-device communication unit 19d, and the like.

(Functional configuration of communication terminal 10)
Next, each functional configuration of the communication terminal 10 (communication terminal 10A and communication terminal 10B) will be described in detail.

The transmitter/receiver unit 11 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2. Before starting a call with the desired destination terminal, the transmitter/receiver unit 11 begins receiving status information indicating the status of each communication terminal as a destination candidate from the communication management system 50. This status information not only indicates the operating status of each communication terminal 10 (whether it is online or offline), but also, if it is online, indicates detailed status such as whether it is available for a call or whether it is currently in a call.

The operation input reception unit 12 receives various inputs from the operator to the communication terminal. For example, when the operator performs an operation to turn on the power of the communication terminal 10, the operation input reception unit 12 receives the operation and controls the power to be turned on.

For example, upon receiving the power-on command, the communication control unit 13 automatically transmits login request information indicating a login request and the current IP address of the requesting terminal from the transceiver unit 11 to the communication management system 50 via the communication network 2. Furthermore, when the operator performs an operation to power off the communication terminal 10, the transceiver unit 11 transmits status information indicating that the power will be turned off to the communication management system 50, and then the operation input reception unit 12 turns the power off. This allows the communication management system 50 to know that the communication terminal 10 has changed from power-on to power-off.

The communication control unit 13 also performs various communication controls, such as establishing and disconnecting sessions for sending and receiving content data with other communication terminals 10 via the relay device 30. The communication control unit 13 in this embodiment includes the communication ID (Identification) of the communication terminal 10 in the session control information (e.g., start request information, start response information, etc., described below) sent to the communication management system 50.

The communication ID is an example of identification information for an account that can participate in a session in which content data is sent and received using the communication terminal 10. The communication ID may be, for example, a user ID, which is identification information for the operator, an app ID, which is identification information for the app, or a contract ID, which is identification information for the subscriber of the communication terminal 10. The communication ID may also be information that combines at least two of letters, numbers, symbols, and various marks. It may also be an email address, etc.

The imaging unit 14 converts the imaging data obtained by imaging a subject into specified image (video) data and outputs it. There is one imaging unit 14 on the front side of the communication terminal (capturing the front video image) and one on the rear side (capturing the rear video image). There may be more imaging units than this.

After the operator's voice is converted into an audio signal by the microphone 114, the audio input unit 15a converts this audio signal into predetermined audio data and outputs it. The audio output unit 15b converts the audio data into an audio signal and outputs it to the speaker, causing the speaker 115 to output the audio.

The display control unit 16, for example, displays image data included in content data received by the communication terminal 10 on the display unit 109 or the display input unit 110. The display control unit 16 can also transmit information about the destination list received from the communication management system 50 to the display unit 109, causing the display unit 109 to display the destination list.

The storage/readout unit 17 stores various data in the storage unit 18 and reads out various data stored in the storage unit 18.

The memory unit 18 stores authentication information such as the aforementioned communication ID and a password corresponding to the communication ID. Image data and audio data received when making a call with a destination terminal are overwritten and stored in the memory unit 18 each time they are received. Images are displayed on the display unit 109, etc., using the image data before being overwritten, and audio is output from the speaker 115 using the audio data before being overwritten.

Next, we will explain each functional configuration included in communication terminal 10B.

The omnidirectional video receiving unit 19b receives equirectangular projection video from the wide-angle imaging device 9, for example, via wireless communication such as Bluetooth (registered trademark) or wired communication such as a USB cable. The equirectangular projection video is a video that is repeatedly transmitted frequently enough to be considered a video. However, it may also be a still image, or it may be possible to switch between video and still images.

The operation instruction receiving unit 19a receives operation instruction information from communication terminal 10A via the transmitting/receiving unit 11, requesting communication terminal 10B to control a device (mobile device 20). This operation instruction information includes, for example, the communication ID of the communication terminal 10 that sent the operation instruction information, an operation instruction indicating the requested control, etc.

The device control unit 19c controls the mobile device 20 based on the operation instructions included in the operation instruction information received by the operation instruction receiving unit 19a. The inter-device communication unit 19d communicates with the mobile device 20 using the short-range wireless communication unit 118.

<<Functional configuration of mobile device>>
The mobile device 20 includes, for example, an inter-device communication unit 21 and a travel control unit 22. The inter-device communication unit 21 is realized by, for example, instructions from the CPU 401 shown in Fig. 6, an external device I/F 404, or a short-range wireless communication unit 405. Here, it is assumed that the inter-device communication unit 21 communicates with the communication terminal 10B using the short-range wireless communication unit 405.

The driving control unit 22 controls the movement (driving) of the mobile device 20, such as forward, backward, left turn, right turn, etc., by controlling, for example, the wheel drive unit 406 and the steering unit 407 shown in Figure 6, depending on the control content acquired from the communication terminal 10A.

<<Functional configuration of wide-angle imaging device>>
The wide-angle imaging device 9 includes, for example, a wide-angle image transmission unit 31 and a wide-angle image capturing unit 32. The wide-angle image capturing unit 32 captures a 360-degree wide-angle equirectangular projection image at a predetermined frame rate. It may also capture still images. The wide-angle image transmission unit 31 is realized by the communication unit 317 shown in FIG. 7 and transmits a moving or still equirectangular projection image to the communication terminal 10B.

<<Functional configuration of the communication management system>>
The communication management system 50 includes a transmission/reception unit 51, a terminal authentication unit 52, a terminal management unit 53, a terminal extraction unit 54, a session management unit 55, and a storage/readout unit 57. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 501 in accordance with a program for the communication management system 50 that is loaded from the HD 504 onto the RAM 503. The communication management system 50 also includes a storage unit 5000 that is realized by the HD 504 etc. shown in Fig. 5.

(Functional configuration of the communication management system)
Next, a detailed description will be given of each functional configuration of the communication management system 50. The transmitting/receiving unit 51 transmits and receives various data (or information) to and from other communication terminals, devices, or systems via the communication network 2.

The terminal authentication unit 52 authenticates the communication terminal 10 by determining whether the combination of communication ID and password included in the login request information received via the transmission/reception unit 51 is included in the authentication management database (DB) 5002.

The terminal management unit 53 stores and manages, in the terminal management DB 5003, the destination name, operating status, date and time of receipt of request information, etc., and the IP address of the requesting terminal, in association with each communication ID. For example, when an operator turns the power of the communication terminal 10 from ON to OFF, the terminal management unit 53 changes the operating status in the terminal management DB 5003 from ON to OFF based on the status information sent from the communication terminal 10 indicating that the power is being turned OFF.

The terminal extraction unit 54 searches the destination list management DB 5004 using the communication ID of the requesting terminal that issued the login request as a key, and extracts the communication IDs of destination terminals that can communicate with the requesting terminal. Furthermore, the terminal extraction unit 54 searches the destination list management DB 5004 and extracts the communication IDs of other communication terminals that have the communication ID of the requesting terminal registered as a candidate destination terminal.

Furthermore, the terminal extraction unit 54 searches the aforementioned terminal management DB 5003 using the communication IDs of the extracted destination terminal candidates as search keys, and reads out the operating status for each extracted communication ID. This allows the terminal management unit 53 to obtain the operating status of destination terminal candidates that can communicate with the requesting terminal that has made the login request. The terminal management unit 53 also searches the aforementioned terminal management DB 5003 using the requesting terminal's communication ID as a search key, and obtains the operating status of the requesting terminal that has made the login request.

The session management unit 55 controls sessions managed by the communication management system 50. Session control includes, for example, control for establishing a session, control for allowing a communication terminal 10 to join an established session, control for disconnecting a session, and generation of a session ID. The session management unit 55 also stores and manages the communication ID of the communication terminal 10 that requested the start of a session and the communication ID of the destination terminal, etc., in association with the session ID, which is session identification information, in the session management DB 5005.

The storage/readout unit 57 is implemented by instructions from the CPU 501 and the HDD 505 shown in FIG. 5, or by instructions from the CPU 501. The storage/readout unit 57 stores various data in the storage unit 5000 and reads various data from the storage unit 5000.

<Examples of information managed by the communication management system>
Each management DB stored in the storage unit 5000 of the communication management system 50 will be described.

The authentication management DB 5002 stored in the storage unit 5000 of the communication management system 50 includes, for example, an authentication management table 602 as shown in Table 2. In this authentication management table 602, the communication IDs of the communication terminals 10 managed by the communication management system 50 are associated with the passwords corresponding to the communication IDs and are managed. For example, the authentication management table 602 shown in Table 2 indicates that the password for the communication terminal 10 with the communication ID "01aa" is "aaaa."

The terminal management DB 5003 stored in the storage unit 5000 of the communication management system 50 includes a terminal management table 603 such as that shown in Table 2. In this terminal management table 603, for each communication ID of a communication terminal 10, the destination name when each communication terminal 10 is the destination, the operating status of each communication terminal 10, the date and time when login request information (described later) was received by the communication management system 50, and the IP address of the communication terminal 10 are associated and managed. For example, the terminal management table 603 shown in Table 3 indicates that the communication terminal 10 with the communication ID "01aa" has the terminal name "Japan Headquarters" and the operating status "ONLINE (communication available)." It also indicates that the date and time when login request information was received by the communication management system 50 for the communication terminal 10 with the communication ID "01aa" was "April 10, 20xx, 1:40 PM" and the IP address was "1.2.1.3."

The destination list management DB 5004 stored in the storage unit 5000 of the communication management system 50 includes, for example, a destination list management table 701 as shown in Table 3. In this destination list management table 701, the communication IDs of all destination terminals registered as candidate destination terminals are associated with the communication ID of a requesting terminal requesting the start of communication in a video conference. For example, in the destination list management table 701 shown in Table 4, the candidate destination terminals to which a requesting terminal with a communication ID of "01aa" can request the start of communication are the communication terminals with communication IDs of "01b1,""01b2," and "01b3." These candidate destination terminals are updated by being added or deleted by the communication management system 50 in response to an addition or deletion request from any requesting terminal to the communication management system 50.

With this configuration, the requesting terminal (e.g., 01aa) can only initiate communication with a pre-registered candidate destination terminal (e.g., 01b1). This destination terminal (e.g., 01b1) cannot communicate with the requesting terminal (e.g., 01aa) unless it has also registered the requesting terminal (e.g., 01aa) as a destination terminal in the destination list management table 701. This mechanism is preferable in that it reduces the possibility of unexpected communication terminals 10 communicating with each other. However, it is also possible to eliminate the need for registration in the destination list management table 701 and allow communication terminals 10 to communicate with each other arbitrarily.

The session management DB 5005 stored in the storage unit 5000 of the communication management system 50 includes a session management table 702 such as that shown in Table 4. This session management table 702 manages, for each session ID, which is session identification information, information such as the relay device ID of the relay device 30 used for relay, the communication ID of the requesting terminal, the communication ID of the destination terminal, and the date and time of session participation. For example, the session management table 702 shown in Table 5 indicates that a session with session ID "se2" is being conducted between the communication ID "01ad" of the requesting terminal and the communication ID "01ca" of the destination terminal. It also indicates that the session with session ID "se2" was started at "20xx/4/10 13:11:11" via the relay device 30 with relay device ID "111b."

<Processing flow>
Next, the processing flow of the communication system 1 will be described.

(Preparatory processing)
15 is a sequence diagram showing an example of processing in a preparation stage for communication in communication system 1. Here, as an example, processing in a preparation stage before starting a session between communication terminal 10A and communication terminal 10B1 will be described. In the following description, it is assumed that the communication ID of communication terminal 10A is "01aa" and the communication ID of communication terminal 10B1 is "01b."

First, when the operator of the communication terminal 10A, which is the requesting terminal, performs an operation to turn on the power of the communication terminal 10A, for example, the operation input reception unit 12 receives the power-on operation and turns on the power of the communication terminal 10A (step S21).

Then, in response to the power-on, the communication control unit 13 transmits login request information requesting login from the transceiver unit 11 via the communication network 2 to the communication management system 50 (step S22). Note that the login request information being transmitted by turning on the power of the communication terminal 10A is just one example, and may also be transmitted by, for example, an operator operating the input unit 108 or launching an app.

The login request information also includes a communication ID (the communication ID of the requesting terminal) and password for identifying the communication terminal 10A that is the requesting terminal. This communication ID and password are, for example, information read from the memory unit 18 via the memory/read unit 17. When the login request information is sent from the communication terminal 10A to the communication management system 50, the communication management system 50, which is the receiving side, can determine the IP address of the communication terminal 10A, which is the sending side.

Next, the terminal authentication unit 52 of the communication management system 50 searches the authentication management table 602 described above using the communication ID and password included in the login request information received via the transmission/reception unit 51 as search keys. The terminal authentication unit 52 performs authentication based on whether the combination of communication ID and password included in the login request information received from the communication terminal 10A is included in the authentication management table 602 (step S23).

If the terminal authentication unit 52 determines that the login request is from a communication terminal 10A with valid usage rights, the terminal management unit 53 changes the operating status of communication terminal 10A's communication ID "01aa" recorded in the terminal management table 603 to "ONLINE (communication available)." At this time, the terminal management unit 53 updates the reception date and time and, if necessary, updates the IP address of communication terminal 10A (step S24). As a result, the terminal management table 603 manages the communication ID "01aa" of communication terminal 10A in association with the operating status "ONLINE (communication available)," the reception date and time "20xx.4.10.13:40," and the IP address of communication terminal 10A "1.2.1.3."

Then, the transmitter/receiver unit 51 of the communication management system 50 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the communication terminal 10A, which is the requesting terminal that made the login request, via the communication network 2 (step S25). Here, we will explain what happens when the terminal authentication unit 52 determines that the communication terminal has valid usage rights.

The terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the requestor terminal (communication terminal 10A) that made the login request as a search key. As a result, the terminal management unit 53 extracts the communication IDs of candidate destination terminals that can communicate with the requestor terminal (communication terminal 10A) (step S26). Here, as an example, it is assumed that "01b1," "01b2," and "01b3" are extracted as the communication IDs of destination terminals that correspond to the communication ID "01aa" of the requestor terminal (communication terminal 10A).

Next, the terminal extraction unit 54 searches the terminal management table 603 using the communication IDs ("01b1", "01b2", "01b3") of the extracted destination terminal candidates as search keys. This reads the operating status for each extracted communication ID, thereby obtaining the operating status for each communication ID ("01b1", "01b2", "01b3") (step S27).

Next, the transmitter/receiver 51 transmits destination status information including the operating status of each of the communication IDs ("01b1", "01b2", "01b3") of the destination terminal candidates to the requestor terminal (communication terminal 10A) (step S28). This allows the requestor terminal (communication terminal 10A) to grasp the current operating status of the communication IDs ("01b1", "01b2", "01b3") that are candidate destination terminals for this requestor terminal (communication terminal 10A). Communication terminal 10A displays the destination selection screen shown in FIG. 16.

Furthermore, the terminal extraction unit 54 of the communication management system 50 searches the destination list management table 701 using the communication ID "01aa" of the request source terminal (communication terminal 10A) that made the login request as a search key. As a result, the terminal extraction unit 54 extracts the communication IDs of other request source terminals that have registered the communication ID "01aa" of the request source terminal (communication terminal 10A) as a destination terminal candidate (step S29). In the destination list management table 701 shown in Table 3, the communication IDs of the other extracted request source terminals are "01b1," "01b2," and "01b3."

Next, the terminal management unit 53 of the communication management system 50 searches the terminal management table 603 using the communication ID "01aa" of the requesting terminal (communication terminal 10A) that made the login request as a search key. As a result, the terminal management unit 53 obtains the operating status of the requesting terminal (communication terminal 10A) that made the login request (step S30).

Then, the terminal management unit 53 of the communication management system 50 extracts the communication IDs ("01b1", "01b2", "01b3") that are listed in the terminal management table 603 as having an operating status of "ONLINE (communication available)" from among the communication IDs ("01b1", "01b2", "01b3") extracted in step S29 above.

The transmitter/receiver 51 also transmits destination status information, including the communication ID "01aa" of the requesting terminal (communication terminal 10A) and the operating status "ONLINE (communication available)," to the communication terminal 10B1 corresponding to the extracted communication IDs ("01b1," "01b2," "01b3") (step S31).

Meanwhile, the other communication terminal 10B1 also performs the same processes as steps S22 to S32 above, for example, in response to a power-on operation. The communication terminal 10B1 is powered on by, for example, an administrator of the mobile device 20.

<Destination selection screen>
16 is a diagram showing an example of a destination selection screen displayed on communication terminal 10A. Destination selection screen 1201 shown in FIG. 16 displays a message 1202 prompting the operator to select a destination communication terminal 10 and multiple buttons 1203 for selecting destination communication terminals 10. The operator of communication terminal 10A selects a destination terminal to request participation in a session by, for example, selecting one button 1203 from the multiple displayed buttons 1203. In other words, the operator selects which mobile device 20 to control. As shown in FIG. 16, the operator can select multiple communication terminals 10B, allowing the operator to grasp the status of different locations from the same location.

<Communication processing>
17 is a sequence diagram showing an example of communication processing in communication system 1. Here, an example of a communication management method for starting communication between communication terminal 10A and communication terminal 10B1, which is a device control terminal capable of controlling mobile device 20, will be described.

In step S901, the operation input reception unit 12 of communication terminal 10A receives a selection operation of a destination terminal (communication terminal 10B1) by the operator of communication terminal 10A.

The transmitter/receiver 11 of communication terminal 10A transmits start request information to the communication management system 50, requesting the start of a session (step S902). This start request information includes, for example, the communication ID of the requesting terminal, which is the communication ID of communication terminal 10A, which is the requesting terminal, and the communication ID of the destination terminal, which is the communication ID of communication terminal 10B1, which is the destination terminal. The start request information also includes information such as the IP address of communication terminal 10A (requesting IP address).

In step S903, the terminal management unit 53 of the communication management system 50, which has received the start request information from the communication terminal 10A, updates the terminal management DB 5003 based on the communication ID "01aa" of the requesting terminal (communication terminal 10A) included in the start request information. For example, the terminal management unit 53 changes the operating status information corresponding to the communication ID "01aa" of the communication terminal 10A to "ONLINE (communicating)" and also updates the reception date and time information.

In step S904, the session management unit 55 of the communication management system 50 sends start request information to the destination terminal, communication terminal 10B1, requesting the start of a session. This start request information includes, for example, the communication ID of the requesting terminal, communication terminal 10A.

In step S905, communication terminal 10B1, which has received the start request information from communication management system 50, transmits start response information to communication management system 50. This start response information includes, for example, the communication ID of the destination terminal of communication terminal 10B1. In this embodiment, the start response information is transmitted without requiring any operation on the communication terminal 10B1 side, but it may also be possible to require an administrator to operate communication terminal 10B1.

In step S906, the terminal management unit 53 of the communication management system 50, which has received the start response information from communication terminal 10B1, updates the terminal management DB 5003 based on the communication ID "01b1" of communication terminal 10B1 included in the start response information. For example, the terminal management unit 53 changes the operating status information corresponding to communication ID "01b1" of communication terminal 10B1 to "ONLINE (communicating)" and updates the reception date and time information.

In step S907, the session management unit 55 of the communication management system 50 assigns a session ID, which is identification information for identifying the session. The session management unit 55 also stores the created session ID in the session management DB 5005 in association with the communication ID of the requesting terminal (communication ID of communication terminal 10A) and the communication ID of the destination terminal (communication ID of communication terminal 10B1).

In step S909, the session management unit 55 of the communication management system 50 transmits session information to the relay device 30. This session information includes, for example, information such as the session ID created in step S907. The relay device 30 can obtain the session information from the session management DB 5005 based on the session ID.

In step S910a, the session management unit 55 of the communication management system 50 sends start instruction information to the communication terminal 10A, instructing it to start a session. Similarly, in step S910b, the session management unit 55 of the communication management system 50 sends start instruction information to the communication terminal 10B1, instructing it to start a session. The start instruction information includes a session ID, and the communication terminal 10 can obtain session information from the session management DB 5005 based on the session ID.

In step S911a, communication terminal 10A establishes a session with relay device 30 based on the received start instruction information. Similarly, in step 911b, communication terminal 10B1 establishes a session with relay device 30 based on the received start instruction information. This allows communication terminals 10A and 10B1 to participate in the same session.

In step S911c, the wide-angle image transmission unit 31 of the wide-angle imaging device 9 transmits the equirectangular projection image to the communication terminal 10B1. For example, when the communication terminal 10B1 establishes a session, the communication terminal 10B1 instructs the wide-angle imaging device 9 to begin capturing and transmitting the equirectangular projection image. Alternatively, the wide-angle imaging device 9 may constantly transmit the equirectangular projection image to the communication terminal 10B1 while the power of the wide-angle imaging device 9 is ON. Alternatively, the equirectangular projection image may be transmitted before the session is established.

Communication terminal 10A and communication terminal 10B1 can participate in a session with the same session ID and transmit and receive content data such as image data and audio data, thereby conducting, for example, a video conference (S912).

In addition, communication terminal 10A can use the established session to send and receive information regarding the operation of a device (e.g., mobile device 20) between communication terminal 10B1.

In addition, information regarding device operation may be transmitted and received using a control session via the communication management system 50 without using a session, or may be transmitted and received between communication terminal 10A and communication terminal 10B1 via communication network 2, etc.

The following explanation assumes that communication terminal 10A uses the established session to send and receive information about the operation of a device (e.g., mobile device 20) between communication terminal 10B1.

In step S913, when the operator of communication terminal 10A inputs information on the device's operation screen, operation instruction information corresponding to the input operation is transmitted using the session. This operation instruction information includes, for example, the communication ID of communication terminal 10A and an operation instruction corresponding to the operation performed by the operator.

In step S914, the device control unit 19c of the communication terminal 10B1 controls the mobile device 20 via the device-to-device communication unit 19d based on the operation instruction included in the notified operation instruction information.

<Pattern 1>
Next, for each of patterns 1 to 3, a display example will be described that allows the operator to know in which direction the mobile device 20 is facing when the operator is looking in a certain direction.

First, pattern 1 will be explained using Figures 18 to 21. Figure 18 is an example diagram illustrating the reference imaging direction that serves as the basis for the imaging direction and the front direction of the mobile device 20. Because the wide-angle imaging device 9 is fixed to the mobile device 20, the front direction of the mobile device 20 can be expressed as the imaging direction of the wide-angle imaging device 9. The imaging direction that is the same as the front direction of the mobile device 20 is defined as the reference imaging direction (ea0, aa0).

If the imaging direction that the operator is looking at is the standard imaging direction, the operator will be looking at the front of the telepresence robot from behind it.

19 is a diagram showing the relationship between an arbitrary imaging direction (ea, aa) displayed by the operator and the reference imaging direction (ea0, aa0). Since the reference imaging direction (ea0, aa0) is fixed, the difference (Δea, Δaa) between the imaging direction (ea, aa) and the reference imaging direction (ea0, aa0) is as follows:
Δea＝ea0－ea
Δaa＝aa0－aa
To simplify the explanation, only the horizontal direction will be considered. When the imaging direction changes by Δaa from the reference imaging direction, it means that the telepresence robot has rotated in the opposite direction by Δaa from the perspective of the telepresence robot. For example, if the imaging direction increases by 90 degrees horizontally, the telepresence robot will rotate in a direction that decreases by 90 degrees horizontally. If the imaging direction decreases by 90 degrees horizontally, the telepresence robot will rotate in a direction that increases by 90 degrees horizontally. The center of rotation can be the center of gravity of the telepresence robot.

Therefore, the display control unit 16 rotates the telepresence robot by (-Δea, -Δaa) and displays it. (-Δea, -Δaa) may be calculated as the difference from the immediately preceding imaging direction (ea, aa) rather than the difference from the reference imaging direction (ea0, aa0).

The display control unit 16 rotates a pre-prepared polygonal model of the telepresence robot in the horizontal and elevation directions. It performs coordinate transformation to rotate the model around two axes of a Cartesian coordinate system that pass through the center of the polygonal model of the telepresence robot.

Figure 20 is an example diagram illustrating perspective projection transformation. Perspective projection transformation is a transformation in which the viewpoint is on the Z axis and an object is projected onto a projection plane 830 perpendicular to the Z axis. In this embodiment, the objects are a polygonal model of a telepresence robot and a predetermined area image Q of the omnidirectional video. The origin O is the center of the three-dimensional sphere as shown in Figure 10. The objects closest to the origin O are the projection plane 830, the polygonal model 839 of the telepresence robot, and the predetermined area image Q. In normal perspective projection transformation for displaying a omnidirectional video on the display unit 109, the polygonal model 839 of the telepresence robot is not positioned, but in pattern 1, the polygonal model 839 of the telepresence robot is positioned on the Z axis. The distance from the projection plane 830 may be determined experimentally. A wide-angle imaging device 9 is mounted on the top of communication terminal 10A, and the omnidirectional image is captured by wide-angle imaging device 9, so the closer the polygon model 839 is to projection surface 830, the less misalignment there is between the viewpoint and wide-angle imaging device 9.

Note that the polygon model 839 of the telepresence robot does not have to be on the Z axis. For example, it may be positioned so that it is projected onto one of the four corners of the specified area image Q.

As described in FIG. 19, the polygon model 839 of the telepresence robot is rotated in the local coordinate system due to the difference (Δea, Δaa) between the imaging direction (ea, aa) and the reference imaging direction (ea0, aa0). It is then placed in a coordinate system for projecting the omnidirectional video as shown in FIG. 20. This allows the image of the telepresence robot to display the direction in which the mobile device 20 is facing in the omnidirectional video.

Figure 21 shows an example of a spherical image display screen 1010 displayed in pattern 1. On the spherical image display screen 1010 in Figure 21 (a), communication terminal 10A displays a spherical image in the reference imaging direction. Therefore, the image 840 of the telepresence robot faces away from the operator. Next, the operator rotates the spherical image to the right. For ease of explanation, it is assumed that the image is rotated 90 degrees horizontally. During this time, mobile device 20 does not move or turn.

Figure 21(b) shows image 840 of the telepresence robot after the operator rotates the omnidirectional image to the right. Since the telepresence robot in the real space is facing 90 degrees to the left with respect to the imaging direction of the omnidirectional image in Figure 21(b), image 840 of the telepresence robot also faces 90 degrees to the left. When the predetermined area image Q changes, the orientation of image 840 of the telepresence robot changes according to the amount of change.

Therefore, even if the operator rotates the spherical image as desired, they can easily understand which direction the telepresence robot in real space is facing.

Figure 22 is an example flowchart showing the procedure by which communication terminal 10A displays a predetermined area image on display unit 109. The process in Figure 22 is executed each time the user changes the imaging direction.

First, the operation input reception unit 12 receives an operation on the imaging direction by the operator (S1001).

Next, the display control unit 16 calculates the difference (Δea, Δaa) between the imaging direction and the reference imaging direction (S1002). Instead of using the reference imaging direction as a reference, the difference from the immediately preceding imaging direction may be calculated.

Next, the display control unit 16 rotates the polygon model of the telepresence robot by (-Δea, -Δaa) in the opposite direction to the difference (S1003).

Then, the display control unit 16 performs perspective projection conversion on the specified area image and the polygon model and displays them (S1004).

It is preferable that the image of the telepresence robot is semi-transparent. It is also preferable that the degree of semi-transparency can be determined by the operator. Furthermore, it is preferable that the operator can set whether to display the image of the telepresence robot. This prevents the image of the telepresence robot from interfering with the viewing of the spherical video.

<Pattern 2>
Next, a description will be given of a method for displaying the orientation of the telepresence robot in Pattern 2. Pattern 2 is a display method in which, separate from the omnidirectional image viewed by the operator, the direction in which the omnidirectional image is displayed as seen from the telepresence robot is displayed by the telepresence robot and an imaging direction arrow 8.

Figure 23 shows an example of a polygon model of a telepresence robot and an imaging direction arrow object. A polygon model 850 of the telepresence robot is prepared in advance. The polygon model 850 of the telepresence robot is placed in a predetermined coordinate system. The imaging direction arrow object 851 is also formed from polygons. The imaging direction determined by the operator is represented by ea, aa, so the display control unit 16 generates an imaging direction arrow object 851 of a predetermined length indicating the imaging direction ea, aa. For example, the prepared imaging direction arrow object 851 is placed facing the imaging direction ea, aa.

As shown in the figure, the display control unit 16 aligns the arrowhead of the imaging direction arrow object 851 with the wide-angle imaging device 9 and places it in a predetermined coordinate system. It then performs perspective projection transformation and places it on the display unit 109. This allows the imaging direction arrow 8 to be displayed three-dimensionally. The imaging direction arrow 8 is information indicating the direction of the image of the specified area image Q as seen from the mobile device 20. Therefore, the imaging direction arrow 8 is an arrow image with its origin at the telepresence robot image 854 (mobile device 20).

As shown in Figure 2(b), the telepresence robot image 854 and the imaging direction arrow 8 are displayed alongside the predetermined area image 853. However, to accommodate cases where the screen size of the display unit 109 is small, the display control unit 16 may display only one of them. For example, when the operator drags the predetermined area image 853 with the mouse or swipes with their finger, the telepresence robot image 854 and the imaging direction arrow 8 appear.

Also, in Figure 23, perspective projection can be performed after rotating the telepresence robot object 851 according to the imaging direction. This is because the viewpoint from which the imaging direction arrow 8 is easily seen varies depending on the imaging direction. For example, the telepresence robot object 851 in the standard imaging direction (where the imaging direction arrow 8 facing forward is at a horizontal angle that makes it easy to see) is rotated horizontally by Δea. This makes it easier for the operator to understand in which direction the imaging direction arrow 8 is pointing, regardless of the imaging direction.

As shown in FIG. 24, in pattern 2, the imaging direction arrow 8 may be displayed separately for the horizontal direction and the elevation angle direction. FIG. 24 shows another display example of the imaging direction arrow 8 in pattern 2. The omnidirectional image display screen 1010 in FIG. 24(a) has a omnidirectional image field 1001, a horizontal direction instruction field 1003, and an elevation angle direction instruction field 1004. The horizontal direction instruction field 1003 has an image 855 of a telepresence robot viewed from above, and indicates only the horizontal direction of the imaging direction with a horizontal arrow 8a. The elevation angle direction instruction field 1004 has an image 854 of the telepresence robot, and indicates only the elevation angle direction of the imaging direction with an elevation angle arrow 8b.

By showing the horizontal and elevation directions separately, the operator can more easily determine how much of the imaging direction is horizontal and how much is elevation.

In Figure 24(a), the horizontal direction of the imaging direction is included in the forward 180 degrees, so the elevation angle direction arrow 8b in the elevation angle direction indication field 1004 is displayed in front of the telepresence robot. On the other hand, in Figure 24(b), the horizontal direction of the imaging direction is included in the backward 180 degrees, so the elevation angle direction arrow 8b in the elevation angle direction indication field 1004 is displayed behind the telepresence robot. By switching the display position of the elevation angle direction arrow in this way, the operator can intuitively understand whether the horizontal direction is forward or backward.

Note that the telepresence robot images 854, 855, horizontal arrow 8a, and elevation arrow 8b in Figure 24 represent directions in two dimensions, eliminating the need to prepare a polygon model and perform perspective projection conversion, thereby reducing the processing load.

Figure 25 is an example flowchart showing the procedure by which communication terminal 10A displays a spherical image on display unit 109 in pattern 2. The process in Figure 25 is executed each time the user changes the imaging direction.

First, the operation input receiving unit 12 receives an operation on the imaging direction by the operator (S2001).

Next, the display control unit 16 performs perspective projection conversion on the image of the specified area and displays it (S2002).

Next, the display control unit 16 determines the viewpoint for the polygon model 850 of the telepresence robot based on the imaging direction (S2003).

Next, the display control unit 16 places the imaging direction arrow object 851 in a predetermined coordinate system (S2004). That is, the display control unit 16 aligns the arrowhead of the imaging direction arrow object 851 with the wide-angle imaging device 9, and places it in the coordinate system in which the telepresence robot object is placed.

The display control unit 16 performs perspective projection transformation on the polygon model 850 of the telepresence robot and the imaging direction arrow object 851 and displays them on the display unit 109 (S2005).

<Pattern 3>
In pattern 3, the operation button is displayed only when the imaging direction substantially coincides with the front direction of mobile device 20. That is, display control unit 16 displays operation button 860 only when imaging direction (ea, aa)≈(ea0, aa0). See FIG. 2( c) for spherical image display screen 1010 in pattern 3.

Figure 26 is an example flowchart showing the procedure by which communication terminal 10A displays a spherical image on display unit 109 in pattern 3. The process in Figure 26 is executed each time the user changes the imaging direction.

First, the operation input receiving unit 12 receives an operation on the imaging direction by the operator (S3001).

Next, the display control unit 16 performs perspective projection conversion on the specified area image Q and displays it (S3002).

Next, the display control unit 16 determines whether the imaging direction and the reference imaging direction approximately match (S3003).

If the determination in step S3003 is Yes, the display control unit 16 displays the operation button 860 superimposed on the spherical image (S3004).

If the determination in step S3003 is No, the display control unit 16 does not display the operation button 860 on the spherical image.

Therefore, the operation button 860 is displayed only when the front direction of the mobile device 20 and the imaging direction of the omnidirectional video viewed by the operator are approximately aligned, allowing the operator to know where the mobile device 20 will move if it is moved forward. Furthermore, when the operation button 860 is displayed, the operator can know the orientation of the mobile device 20.

In the left diagram c-1 in FIG. 2(c), the operation button 860 is not displayed at all. However, if the front direction of the mobile device 20 and the imaging direction of the omnidirectional video viewed by the operator do not substantially match, the operation button 860 may be displayed in a manner that makes it inoperable by the operator. An inoperable manner means, for example, displaying the operation button 860 at low brightness, in gray, or semi-transparently (increasing transparency). Even if the operator presses the operation button 860, the operation input receiving unit 12 does not receive the operation.

<Other application examples>
The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

For example, although this embodiment does not describe the front image (or rear image) captured by the front imaging device, the front image (or rear image) is displayed as appropriate in each of patterns 1 to 3.

Figure 27 shows an example of a simultaneous video display screen 2001 that simultaneously displays a spherical video and a front video (or rear video). The simultaneous video display screen 2001 has an overall display field 2002 and a front display field 2003. An image from one of patterns 1 to 3 is displayed in the overall display field 2002. The front display field 2003 displays the front video (or rear video) captured by the front imaging device. Therefore, the operator can display any direction in the overall display field 2002 while always checking the front or rear.

In addition, in this embodiment, the wide-angle imaging device 9 transmits equirectangular projection images to communication terminal 10B, and communication terminal 10B transmits the equirectangular projection images to communication terminal 10A. However, the wide-angle imaging device 9 may directly connect a session with communication terminal 10A and transmit the equirectangular projection images. In this case, the wide-angle imaging device 9 becomes one form of communication terminal 10, and becomes a communication terminal 10 that transmits images but does not receive them.

In addition, although the present embodiment has been described as transmitting a rear view image, communication terminal 10B does not have to transmit a rear view image. Furthermore, communication terminal 10B may transmit a normal view angle image separately from the front view image and the rear view image.

In addition, while this embodiment has been described using an example of a mobile device 20 that travels on land, the mobile device 20 may also fly in the air, such as a drone. It may also sail on the sea or travel underwater. It may also travel underground, through narrow passages, or underground. When traveling on land, it may move on wheels, or it may move on multiple legs such as two, three, or four legs, or it may move on caterpillars (registered trademark).

Furthermore, the display method of this embodiment may be applied not only to telepresence robots, but also to images from a wide-angle imaging device 9 and a planar imaging device mounted on a device that is moved by a person.

In addition, while this embodiment has mainly described an example in which one operator operates one mobile device 20, it is also possible for one operator to operate multiple mobile devices 20. In this case, communication terminal 10A establishes sessions with multiple communication terminals 10B. Conversely, it is also possible for multiple operators to operate one mobile device 20. In this case, for example, the operation instruction information last sent to communication terminal 10B may be valid, or the concept of operation authority may be introduced so that only communication terminal 10A with operation authority is able to send operation instructions. Operation authority can be transferred by operation by the operator, etc.

Furthermore, although this embodiment describes operation instruction information being transmitted in a communication session, it may also be transmitted in a content data session.

In addition, in this embodiment, the communication terminal 10 communicates via the relay device 30, but the communication terminal 10 may also communicate without going through the relay device. A known communication protocol for such communication is, for example, WebRTC (Web Real-Time Communication).

Furthermore, the configuration examples shown in Figure 14 and other figures are divided according to main functions to make it easier to understand the processing by the communication management system 50 and communication terminal 10. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by the communication management system 50 and communication terminal 10 can also be divided into even more processing units depending on the processing content. Furthermore, it is also possible to divide the processing units so that one processing unit includes even more processes.

Furthermore, for the sake of convenience, in this embodiment the communication management system 50 and relay device 30 are described as separate devices, but a device that integrates the functions of both the communication management system 50 and relay device 30 may also provide the functions of both.

Furthermore, the communication system 1 may have multiple communication management systems 50, and the functions of the communication management systems 50 may be distributed across multiple servers.

Furthermore, one or more of the databases that the communication management system 50 has in the memory unit 5000 may be located on the communication network 2.

Note that the transmitter/receiver 11 is an example of a communication means, the display control unit 16 is an example of a display processing means, and the operation input reception unit 12 is an example of a reception means.

1 Communication system 9 Wide-angle imaging device 10 Communication terminal 20 Mobile device 50 Communication management system

JP 2013-112030 A

Claims (17)

A communication terminal that communicates with another communication terminal mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
and
the display processing means displays an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image allowing the orientation of the mobile device to be grasped;
When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change .
A communication terminal characterized by: A communication terminal that communicates with another communication terminal mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
and
The display processing means
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device.
A communication terminal characterized by: the communication means transmits operation instruction information for the mobile device to the other communication terminal;
3. The communication terminal according to claim 1 or 2. The communication terminal described in claim 1 or 2, characterized in that the display processing means superimposes an image of the mobile device, displayed in a manner that makes the orientation of the mobile device clear, on at least a portion of the wide-angle image displayed on the display unit. The communication terminal described in claim 1, characterized in that the display processing means performs perspective projection conversion between the wide-angle image attached to the 3D sphere and a model of the mobile device displayed in a manner that allows the orientation of the mobile device to be seen, thereby superimposing an image of the mobile device displayed in a manner that allows the orientation of the mobile device to be seen on at least a portion of the wide-angle image displayed on the display unit. The communication terminal described in claim 2, characterized in that the information indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is an image of an arrow starting from the image of the mobile device. The communication terminal described in claim 6, characterized in that the display processing means displays information indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is viewed from the image of the mobile device by performing perspective projection transformation on a model of the mobile device and a model of an arrow indicating the direction in which at least a portion of the wide-angle image displayed on the display unit is viewed from the image of the mobile device. The communication terminal described in claim 2, characterized in that the information indicating the direction of at least a portion of the wide-angle image displayed on the display unit relative to the image of the mobile device is an image of arrows pointing in the horizontal and elevation directions relative to the image of the mobile device. The communication terminal described in claim 1 or 2, characterized in that the information regarding the orientation of the mobile device is an operation button for receiving operation instruction information for the mobile device, which is displayed when the center direction of at least a portion of the wide-angle image displayed on the display unit and the front direction of the mobile device are approximately aligned. A communication terminal that communicates with a telepresence robot via a network, the communication terminal comprising: a mobile device; another communication terminal mounted on the mobile device; and a wide-angle imaging device mounted on the mobile device or the other communication terminal and configured to capture wide-angle images,
a communication means for receiving the wide-angle image captured by the wide-angle imaging device;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
and
the display processing means displays an image of the telepresence robot on the display unit on which at least a part of the wide-angle image is displayed, the image enabling the orientation of the telepresence robot to be grasped;
When at least a part of the wide-angle image displayed on the display unit changes, the image of the telepresence robot displayed in a manner that allows the orientation of the telepresence robot to be known changes orientation in accordance with the change .
A communication terminal characterized by: A communication terminal that communicates with a telepresence robot via a network, the communication terminal comprising: a mobile device; another communication terminal mounted on the mobile device; and a wide-angle imaging device mounted on the mobile device or the other communication terminal and configured to capture wide-angle images,
a communication means for receiving the wide-angle image captured by the wide-angle imaging device;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
and
The display processing means
displaying an image of the telepresence robot on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the telepresence robot to be grasped;
In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the telepresence robot is displayed together with the image of the telepresence robot.
A communication terminal characterized by: A communication terminal that communicates with other communication terminals mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device connected to the mobile device or the other communication terminal;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
It functions as
The display processing means
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change .
program. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
It functions as
The display processing means
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device.
program. A display method performed by a communication terminal that is mounted on a mobile device and communicates with another communication terminal via a network, comprising:
a communication step in which a communication means receives a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing step in which a display processing means displays at least a part of the wide-angle image received by the communication means in the communication step on a display unit of the communication terminal different from the other communication terminal;
Including,
The display processing step includes:
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change .
A display method characterized by: A display method performed by a communication terminal that is mounted on a mobile device and communicates with another communication terminal via a network, comprising:
a communication step in which a communication means receives a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing step in which a display processing means displays at least a part of the wide-angle image received by the communication means in the communication step on a display unit of the communication terminal different from the other communication terminal;
Including,
The display processing step includes:
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device.
A display method characterized by: A communication terminal that communicates with other communication terminals mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing unit that displays at least a part of the wide-angle image received by the communication unit on a display unit of the communication terminal different from the other communication terminal;
It functions as
The display processing means
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
When at least a part of the wide-angle image displayed on the display unit changes, the image of the mobile device displayed in a manner that allows the orientation of the mobile device to be known changes orientation in accordance with the change .
A computer-readable recording medium on which a program is recorded. A communication terminal that communicates with other communication terminals mounted on a mobile device via a network,
a communication means for receiving a wide-angle image captured by a wide-angle imaging device mounted on the mobile device or the other communication terminal;
a display processing means for displaying at least a part of the wide-angle image received by the communication means on a display unit of a communication terminal different from the other communication terminal;
It functions as
The display processing means
displaying an image of the mobile device on the display unit on which at least a portion of the wide-angle image is displayed, the image enabling the orientation of the mobile device to be grasped;
In addition to at least a portion of the wide-angle image displayed on the display unit, information indicating in which direction the at least a portion of the wide-angle image displayed on the display unit is an image starting from an image of the mobile device is displayed together with the image of the mobile device.
A computer-readable recording medium on which a program is recorded.