JP6568981B2 - Image display device, image display system, image display method, and program - Google Patents

Description

  The present invention relates to a technique for displaying an image showing the periphery of a vehicle.

  2. Description of the Related Art Conventionally, there has been known a technique for synthesizing images obtained by capturing the periphery of a vehicle such as an automobile and displaying the appearance of the vehicle periphery viewed from a virtual viewpoint. With such a technique, a user (typically a driver) can check the situation around the vehicle while in the vehicle.

  In recent years, a technique for transmitting such an image to a portable terminal carried by the user has been proposed (for example, Patent Document 1). By visually recognizing such an image, the user can grasp the state of the vehicle and the risk of theft even if the user is located remotely from the vehicle.

JP2012-121384A

  However, with the proposed technique, the viewpoints of viewing the periphery of the vehicle are limited, and the peripheral image cannot always be viewed from the angle desired by the user. For this reason, for example, even if an image of a suspicious person in the vicinity of the vehicle is confirmed on a mobile terminal, it is not clear what kind of person the suspicious person is or what action is being taken. On the other hand, there is a fear of promoting user's anxiety.

  In view of the above problems, an object of the present invention is to provide a technique capable of changing a virtual viewpoint to a viewpoint desired by a user when an image showing the periphery of the vehicle is displayed from a virtual viewpoint.

In order to solve the above-mentioned problem, the invention of claim 1 is an image display device that displays an image, and uses an image captured by a camera to generate a composite image viewed from a virtual viewpoint; Operation detection means for detecting a user operation for the change of the viewpoint of the composite image, the operation detection means corresponding to an image area in which the composite image is displayed and the predetermined viewpoint. A button area in which a plurality of buttons for selecting display of the composite image are displayed, and the composite image corresponding to the viewpoint selected by the button is displayed in the image area. , when a touch operation is performed in the image area, said composite image to the image area in the viewpoint in accordance with the touch operation is displayed, the touch operation, flicking operation or pinch Misao It is.

Further, the invention of claim 2 is an image display device for displaying an image, wherein an image captured by a camera is used to generate a composite image viewed from a virtual viewpoint, and the composite image Operation detecting means for detecting a user operation with respect to the change of the viewpoint, the operation detecting means of the composite image corresponding to the image area in which the composite image is displayed and the predetermined viewpoint. includes a button area in which a plurality of buttons for selecting the display is displayed, and the button, the vertical and horizontal directions of the button regions corresponding to the longitudinal and lateral direction of the vehicle, are each of the button arrangement, the button the alloy in the region, buttons for selecting the display in the viewpoint of the initial display in the wide surroundings vehicle area is further displayed in the image area, corresponding to the view point that is selected in the button Image that is displayed.

According to a third aspect of the present invention, there is provided communication means for obtaining image information about a composite image generated from a virtual viewpoint using a photographed image photographed by a camera, and based on the obtained image information. Display means for displaying an image; and operation detection means for giving a user instruction to change the viewpoint of the composite image displayed on the display means. The operation detection means is a user touch on the display means. has a touch panel for detecting an operation, the display means, button plurality of buttons for instructing an image region where the composite image is displayed, the display of the composite image corresponding to the viewpoint predetermined is displayed has a region, a, in the image area, the with the synthetic image is displayed, the touch operation is the row in the image region corresponding to the viewpoint is selected by the button The when, the said composite image of the viewpoint in accordance with the touch operation is displayed on the image area, the touch operation is a terminal device which is a flick operation or pinch-in operation.

According to a fourth aspect of the present invention, there is provided communication means for obtaining image information about a composite image generated in a form viewed from a virtual viewpoint using a photographed image photographed by a camera, and based on the obtained image information. Display means for displaying an image; and operation detection means for giving a user instruction to change the viewpoint of the composite image displayed on the display means. The operation detection means is a user touch on the display means. has a touch panel for detecting an operation, the display means, button plurality of buttons for instructing an image region where the composite image is displayed, the display of the composite image corresponding to the viewpoint predetermined is displayed has a region, wherein the button corresponds to the front, rear, left and right directions of the vehicle, each button in the vertical and horizontal directions of said button region is disposed in the button area of the vehicle surroundings Is initially displayed the viewpoint display displaying buttons for selecting the further in the in a wide area, the image area, characterized in that the synthesized image corresponding to the viewpoint is selected in the button is displayed It is a terminal device.

According to a fifth aspect of the present invention, in the terminal device according to the third or fourth aspect , the communication means receives a notification when the security device detects an abnormality.

According to the first to fifth aspects of the invention, since the position of the virtual viewpoint of the composite image is changed based on the user's operation, the subject can be easily displayed from the viewpoint desired by the user.

  In addition, since the direction of the virtual viewpoint is determined toward a specific point set at a specific position of the vehicle and the position of the virtual viewpoint is changed based on the world coordinate system and the viewpoint coordinate system, the subject can be displayed from various directions and positions. .

In particular, according to the invention of claim 5 , even if the user of the image display device is located remotely from the vehicle, the user can recognize that a pre-phenomenon that leads to theft of the vehicle has occurred, and quickly see the situation around the vehicle. You can check on the screen.

FIG. 1 is a diagram showing an overview of an image display system. FIG. 2 is a diagram showing an outline of the image display system. FIG. 3 is a block diagram illustrating a configuration of the image acquisition apparatus according to the first embodiment. FIG. 4 is a diagram for explaining the arrangement of the cameras. FIG. 5 is a block diagram illustrating a configuration of the mobile terminal according to the first embodiment. FIG. 6 is a diagram illustrating a method for generating a peripheral image and a composite image. FIG. 7 is a diagram illustrating the position of the virtual viewpoint. FIG. 8 is a diagram illustrating the position of the virtual viewpoint. FIG. 9 is a diagram for explaining the movement of the virtual viewpoint. FIG. 10 is a diagram for explaining the movement of the virtual viewpoint. FIG. 11 is a diagram for explaining the movement of the virtual viewpoint. FIG. 12 is a diagram for explaining the movement of the virtual viewpoint. FIG. 13 is a diagram illustrating a display image. FIG. 14 is a diagram illustrating a display image. FIG. 15 is a diagram illustrating a display image. FIG. 16 is a diagram illustrating a display image. FIG. 17 is a diagram illustrating a display image. FIG. 18 is a diagram illustrating a display image. FIG. 19 is a diagram illustrating a display image. FIG. 20 is a diagram illustrating a display image. FIG. 21 is a diagram illustrating a display image. FIG. 22 is a diagram illustrating a display image. FIG. 23 is a diagram illustrating a display image. FIG. 24 is a diagram illustrating a display image. FIG. 25 is a flowchart showing the processing procedure of the first embodiment. FIG. 26 is a flowchart showing the processing procedure of the first embodiment. FIG. 27 is a block diagram illustrating a configuration of the mobile terminal according to the first embodiment. FIG. 28 is a diagram for explaining the movement of the virtual viewpoint. FIG. 29 is a diagram for explaining the movement of the virtual viewpoint. FIG. 30 is a diagram illustrating a display image. FIG. 31 is a flowchart illustrating a processing procedure according to the second embodiment. FIG. 32 is a flowchart illustrating a processing procedure according to the second embodiment. FIG. 33 is a diagram illustrating a modification of the image display system.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Overview>
FIG. 1 shows an outline of an image display system 1 according to an embodiment of the present invention. The image display system 1 includes a camera 4 (4F, 4B, 4B) in which an image acquisition device 3 provided in the vehicle 2 is disposed in the vehicle 2.
L, 4R) is a system for displaying an image around the vehicle 2 acquired from the mobile terminal 8 of a user located remotely from the vehicle 2.

In such a system, a security device 5 having a monitoring sensor is installed in the vehicle 2. The monitoring sensor is activated when the suspicious person SI approaches an abnormality to the vehicle 2 or when a physical hazard is applied to the vehicle 2, and the security device 5 notifies the user that the monitoring sensor has been activated. To do. The user who receives the notification from the security device 5 requests the image acquisition device 3 to transmit an image and causes the portable terminal 8 to display an image around the vehicle 2. At this time, the user freely changes the display range of the image and refers to the situation around the vehicle 2 from a desired angle.

FIG. 2 is an example of the display 83 that displays an image around the vehicle 2 on the mobile terminal 8. By operating the touch panel 83a of the portable terminal 8, the user can change the display range of the image and check in detail the area of concern. The user can remotely activate the alarm of the vehicle 2 or report it to the security company as needed by referring to the image around the vehicle 2. By using such an image display system 1, even when the user is located remotely from the vehicle 2, the vehicle 2 owned by the user can be always maintained safely.

<1-2. Configuration>
Next, the configuration of each device included in the vehicle 2 used in the image display system 1 will be described. FIG. 3 shows the configuration of each device provided in the vehicle 2. The vehicle 2 includes an image acquisition device 3, a camera 4,
A security device 5 and an alarm device 6 are provided.

The image acquisition device 3 is an electronic control device that receives image data from the camera 4 installed in the vehicle and transmits the received image data to the portable terminal 8. The image acquisition device 3 includes a control unit 31, a communication unit 32, and a storage unit 33.

The control unit 31 is a microcomputer including a CPU, a RAM, and a ROM. The control unit 31 is connected to other components included in the image acquisition device 3 and controls the entire device. The control unit 31 includes a request reception unit 31a, a camera control unit 31b, an image acquisition unit 31c, and an image transmission unit 31d.
And an anti-theft unit 31e.

The request receiving unit 31 a receives a request signal for operating the camera 4 and starting photographing from a user who possesses the mobile terminal 8.

When the request receiving unit 31a receives a request signal for starting shooting from the user, the camera control unit 31b transmits an operation signal to the camera 4 to cause the camera 4 to start shooting.

The image acquisition unit 31 c acquires image data transmitted from the camera 4, and converts the acquired image data into a format that can be processed by the control unit 31.

The image transmission unit 31d transmits the image data acquired by the image acquisition unit 31c from the camera 4 to the communication unit 3.
2 to the mobile terminal 8.

The anti-theft unit 31e transmits an operation signal to the alarm device 6 to cause the alarm device 6 to perform an alarm. This prevents the vehicle 2 from being stolen. The anti-theft unit 31e may cause the alarm device 6 to issue an alarm and control an engine control device (not shown) so that the engine does not operate. Moreover, you may report to a security company via the communication part 32. FIG. In short, the anti-theft unit 31e only needs to function so as to prevent the theft of the vehicle 2 by using a device provided in the vehicle 2.

The communication unit 32 includes a communication function using wireless communication, and performs information communication with the mobile terminal 8 via the network 7. For example, WiMAX (Worldwide Interoperability for Microwave
Access) and information communication using wireless communication technologies such as LTE (Long Term Evolution).

The storage unit 33 is a memory that stores data. For example, EEPROM (Electrical E
rasable Programmable Read-Only Memory), flash memory, and hard disk drive equipped with a magnetic disk. The storage unit 33 stores a program 33a.

The program 33 a is firmware that is read by the control unit 31 and executed for the control unit 31 to control the image acquisition device 3.

Next, the camera 4 will be described. The camera 4 includes a front camera 4F, a rear camera 4B, a left side camera 4L, and a right side camera 4R. Each camera (4F, 4B, 4L, 4R
) Includes a lens and an image sensor, and is disposed at different positions on the vehicle 2.

FIG. 4 shows a position where the camera 4 (4F, 4B, 4L, 4R) is arranged in the vehicle 2 and a direction in which the optical axis of each camera is directed. The front camera 4F is installed at the front end of the vehicle 2, and the optical axis 4Fa is directed in the straight traveling direction of the vehicle 2. The rear camera 4B is installed at the rear end of the vehicle 2, and the optical axis 4Ba is directed in the direction opposite to the straight traveling direction of the vehicle 2, that is, in the backward traveling direction. The left side camera 4L is installed on the left side mirror ML, and the optical axis 4La is directed to the left direction of the vehicle 2 (a direction orthogonal to the straight traveling direction). The right side camera 4R is installed on the right side mirror MR, and its optical axis 4Ra is directed in the right direction of the vehicle 2 (a direction orthogonal to the straight traveling direction).

Each camera (4F, 4B, 4L, 4R) of the camera 4 captures a different direction around the vehicle 2 and electronically acquires a captured image. Further, the lens included in the camera 4 has a relatively shorter focal length than a standard lens and has an angle of view θ of 180 degrees or more. For this reason, the entire periphery of the vehicle 2 can be photographed by using the four cameras 4.

Please refer to FIG. 3 again. The security device 5 is a device that detects an advance phenomenon that leads to theft of the vehicle 2 or an article in the vehicle 2 and notifies the portable terminal 8 possessed by the user of the vehicle 2 of the possibility of theft by e-mail. . The security device 5 includes a monitoring sensor 5a and a mail notification unit 5b.

The monitoring sensor 5a is a sensor that detects a prior phenomenon that leads to theft of the vehicle 2 or an article in the vehicle 2. For example, a vibration sensor that detects vibration generated in the vehicle 2, a glass break sensor that detects glass breakage of the vehicle 2, a tilt sensor that detects the tilt of the vehicle 2 by a jack or a crane, and an intruder into the vehicle 2 An intrusion sensor to detect.

When the monitoring sensor 5a detects a prior phenomenon that leads to theft, the mail notification unit 5b notifies the user that the theft has occurred. Specifically, the mail notification unit 5b generates an e-mail with the content of the fear of theft and transmits the e-mail to the mobile terminal 8 carried by the user. In addition, you may notify by audio | voice information instead of the character information using an email. In this case, the voice information may be read out by the portable terminal 8. In addition, when the security device 5 includes a plurality of monitoring sensors, the mail notification unit 5b preferably includes in the electronic mail which of the monitoring sensors has detected a prior phenomenon that leads to theft. This is to make it easier for the user to grasp the situation of the vehicle 2.

The alarm device 6 is a device that emits a sound to the surroundings and gives a warning. A horn provided in a speaker or the vehicle 2. The alarm device 6 may emit light in addition to sound. For example, a warning light or a light provided in the vehicle 2. In short, anything that alerts the surroundings and gives a warning is sufficient.

Next, the portable terminal 8 will be described. The portable terminal 8 is an information terminal that is carried by a user and has a function of displaying an image, a function of connecting to an information network, and the like. For example, a mobile phone or a smartphone. FIG. 5 shows the configuration of the mobile terminal 8. The portable terminal 8 includes a control unit 81, a communication unit 82, a display 83, and a storage unit 84.

The control unit 81 is a microcomputer including a CPU, a RAM, and a ROM. The control unit 81 is connected to another configuration included in the mobile terminal 8 and controls the entire terminal. Control unit 81
Each function provided by will be described later.

The communication unit 82 has a communication function using wireless communication, and performs information communication with the image acquisition device 3 and the security device 5 via the network 7. For example, WiMAX (Worldwide Inte
Information communication using wireless communication technologies such as roperability for microwave access (LTE) and long term evolution (LTE).

The display 83 displays various information such as characters and graphics, and visually presents the information to the user. For example, a display device such as a liquid crystal display, a plasma display, or an organic EL display. The display 83 includes a touch panel 83a.

The touch panel 83 a senses the user's contact with the button area displayed on the display 83 and transmits the sensed position information to the control unit 81.

The storage unit 84 is a memory that stores data. For example, EEPROM (Electrical E
rasable Programmable Read-Only Memory), flash memory, and hard disk drive equipped with a magnetic disk. The storage unit 84 stores vehicle body image data 84a and a program 84b.

The vehicle body image data 84a is image data indicating the appearance of the vehicle 2. Body image data 84
a includes image data when the vehicle 2 is viewed from any external angle. The vehicle body image data 84a may be acquired from an external server via the network 7 after the image acquisition device 3 is attached to the vehicle 2 without being stored in the mobile terminal 8 in advance. In this case, vehicle body image data 84a that matches the appearance of the vehicle 2 to which the image acquisition device 3 is attached can be acquired. The user can use the mobile terminal 8
The vehicle model name and the like of the vehicle 2 may be transmitted to the external server, and the vehicle body image data 84a may be requested.

The program 84 b is firmware that is read by the control unit 81 and executed for the control unit 81 to control the mobile terminal 8.

Each function with which the above-mentioned control part 81 is provided is explained. The control unit 81 includes an image acquisition unit 81a, an image generation unit 81b, a display control unit 81c, and an operation detection unit 81d.

The image acquisition unit 81 a acquires image data transmitted from the image acquisition device 3, and converts the acquired image data into a format that can be processed by the control unit 81. That is, the image acquisition unit 81a acquires a plurality of images obtained by shooting the subject with the plurality of cameras 4, and the image generation unit 81b combines the plurality of captured images acquired with the camera 4, and the vehicle 2 viewed from a virtual viewpoint. A peripheral image showing the surrounding state is generated. The virtual viewpoint is a viewpoint in which the vehicle 2 is viewed from a position outside the vehicle 2. In addition, the image generation unit 81b superimposes a vehicle body image showing the vehicle 2 seen from the virtual viewpoint on the peripheral image. A peripheral image generation method and a virtual viewpoint setting method by the image generation unit 81b will be described later.

The display control unit 81 c displays data such as images and characters on the display 83. Further, the display control unit 81c changes an image or the like displayed on the display 83 based on the touch position input to the touch panel 83a.

The operation detection unit 81d detects a user operation on the touch panel 83a. Specifically, the operation detection unit 81d detects in which direction on the touch panel 83a the user's fingertip moves after touching the touch panel 83a based on the touch position information transmitted from the touch panel 83a.

Note that the user operation on the touch panel 83a is, for example, touching the fingertip with the touch panel 8.
This is a so-called flick operation in which the slide is performed while being in contact with 3a. Also, this is a so-called pinch-in operation in which the distance between the fingertips is reduced while the two fingertips are in contact with the touch panel 83a. Further, this is a so-called pinch-out operation in which the distance between the fingertips is increased while the two fingertips are in contact with the touch panel 83a.

The notification unit 81e transmits a predetermined signal to a device outside the mobile terminal 8 via the communication unit 82. The signal transmitted by the notification unit 81e is, for example, a signal requesting the image acquisition apparatus to transmit image data.

<1-3. Generation of peripheral images and composite images>
Next, the image generation unit 81b includes a peripheral image AP indicating a peripheral region of the vehicle 2, and a peripheral image AP.
A method for generating the composite image CP in which the vehicle body image 10 is superimposed on the vehicle body will be described. FIG. 6 shows a method in which the image generation unit 81b generates the peripheral image AP.

First, when each of the front camera 4F, the rear camera 4B, the left side camera 4L, and the right side camera 4R captures the vicinity of the vehicle 2, four images AP respectively showing the front, rear, left side, and right side of the vehicle 2 are displayed. (F), AP (B), AP (L), and AP (R) are acquired. These four images include image data indicating the entire periphery of the vehicle 2.

The image generation unit 81b uses these four images AP (F), AP (B), AP (L), AP (
Data (value of each pixel) included in R) is projected onto a projection surface TS that is a three-dimensional curved surface in a virtual three-dimensional space. The projection surface TS is, for example, a substantially hemispherical shape (a bowl shape). The center part (bottom part of the bowl) of the projection surface TS is the position where the vehicle 2 is present. Further, the portion other than the central portion of the projection surface TS is associated with any one of the images AP (F), AP (B), AP (L), and AP (R).

The image generation unit 81b has peripheral images AP (F), AP (
B), AP (L), AP (R) are projected. The image generation unit 81b projects the image AP (F) of the front camera 4F on an area corresponding to the front of the vehicle 2 on the projection surface TS, and projects the image AP (B) of the rear camera 4B on an area corresponding to the rear of the vehicle 2. To do. Further, the image generation unit 81b
Is the image AP (left side camera 4L image AP (
L) is projected, and the image AP (R) of the right side camera 4R is projected onto a region corresponding to the right side of the vehicle 2. Thereby, the surrounding image AP which shows the area | region of the vehicle 2 whole periphery is produced | generated.

Next, the image generation unit 81b sets a virtual viewpoint VP directed from an arbitrary viewpoint position in this three-dimensional space to an arbitrary viewpoint direction. Of the projection plane TS, the set virtual viewpoint VP
The image data projected on the area included in the viewing angle viewed from the side is cut out. A peripheral image AP indicating a region around the vehicle 2 viewed from the virtual viewpoint VP is generated from the cut-out image data.

The image generation unit 81b reads the vehicle body image data from the storage unit 84, and generates the vehicle body image 10 of the vehicle 2 viewed from the virtual viewpoint VP. Then, the vehicle body image 10 is synthesized with the peripheral image AP indicating the area around the vehicle 2 viewed from the virtual viewpoint VP to generate a composite image CP.

The image generation unit 81b generates a virtual viewpoint VPt with the viewpoint position directly above the vehicle 2 and the visual field direction immediately below.
Is used to generate the composite image CPt overlooking the vehicle 2 and the peripheral region of the vehicle 2 using the peripheral images AP (F), AP (B), AP (L), AP (R) and the vehicle body image 10. To do. The composite image CPt is an image as if the vehicle 2 was looked down from directly above, and shows an area around the vehicle 2.

Further, when the virtual viewpoint VPb is set with the viewpoint position at the front upper side of the vehicle 2 and the visual field direction at the rear lower side of the vehicle 2, the peripheral image AP (B), AP (L), AP (R
) And the vehicle body image 10 are used to generate a composite image CPb overlooking the vehicle 2 and the surrounding area of the vehicle 2. The composite image CPb is an image as if the vehicle 2 is looked down from the upper front side and shows the area behind the vehicle 2.

In addition, when the virtual viewpoint VP1 is set with the viewpoint position obliquely left rearward and upward of the vehicle 2 and the visual field direction facing forward and downward of the vehicle 2, the image generation unit 81b sets the peripheral images AP (F) and AP (L).
, AP (R) and the vehicle body image 10 are used to generate a composite image CP1 overlooking the vehicle 2 and the surrounding area of the vehicle 2. The composite image CPI is an image as if the vehicle 2 is looked left diagonally backward and from above, and shows a region on the left side of the vehicle 2.

<1-4. Virtual viewpoint settings>
Next, a method in which the image generation unit 81b sets the virtual viewpoint VP will be described in detail. 7 to 12 show a method in which the image generation unit 81b sets the virtual viewpoint VP. In each figure, a three-dimensional XYZ orthogonal coordinate system cc is used, and directions and directions are shown as appropriate. Cartesian coordinate system cc
These coordinate axes are fixed relative to the vehicle 2. That is, the X-axis direction is the left-right direction of the vehicle 2, the Y-axis direction is the front-rear direction of the vehicle 2, and the Z-axis direction is the vertical direction. In addition, the + X side is the vehicle 2
The right side, + Y side is the front side of the vehicle 2, and + Z side is the vertical upper side. Therefore, the -X side is the left side of the vehicle 2, the -Y side is the rear side of the vehicle 2, and the -Z side is the vertically lower side.

First, referring to FIG. FIG. 7 is a diagram of the vehicle 2 as viewed from the vertically upward direction (+ Z side). Among the virtual viewpoints VP, five reference virtual viewpoints VPa (VPat, VPaf, VPab) are used as references.
, VPal, VPar).

Each of the five reference virtual viewpoints VPa has five reference positions (VLt, VLf, VLb, VLl
, VLr), five movement center points RC (RCt, RCf, RCb, RCl, RCr), and five reference directions (VDt, VDf, VDb, VDtl, VDr).

The reference position VLt is located immediately above the vehicle 2. The reference position VLf is located slightly in front of the vehicle 2 (+ Y side). The reference position VLb is located slightly above (−Y side) immediately above the vehicle 2. The reference position VLl is located diagonally to the left of the vehicle 2 (−Y side and −X side). The reference position VLr is located diagonally to the right of the vehicle 2 (+ X side and −Y side). The height (+ Z side) of each reference position VL is the same height from the ground contact surface GR with which the vehicle 2 is in contact. For example, vehicle height 2
Double the height.

FIG. 8 is a diagram when the vehicle 2 is viewed from the right side (+ X side) in the horizontal direction, and is a diagram illustrating the movement center point RC and the reference direction VD. The movement center point RC is a reference point that becomes the center when the virtual viewpoint VP moves. Therefore, even if the viewpoint position of the virtual viewpoint VP moves, the viewpoint direction always faces the movement center point RC. A method of moving the virtual viewpoint VP viewpoint position using the movement center point RC will be described later.

The movement center point RCt is a specific point set at a specific position of the vehicle. The movement center point RCt is located at the center of the vehicle 2. The movement center point RCf is located at the left-right center of the vehicle 2 and slightly forward (+ Y side) of the front end of the vehicle 2. The movement center point RCb is located at the left and right center of the vehicle 2 and slightly rearward (−Y side) of the rear end of the vehicle 2. The movement center point RCl is located at the front-rear center of the vehicle 2 and slightly left outside (−X side) of the vehicle 2. The movement center point RCr is located at the front-rear center of the vehicle 2 and slightly outside (+ X side) of the vehicle 2. The height (+ Z side) of each moving center point RC is the vehicle 2
Is the height h of the user's line of sight.

The reference direction VDt is directed from the reference position VLt to the movement center point RCt (−Z side). The reference direction VDf is directed from the reference position VLf to the movement center point RCf (−Z side and + Y side). The reference direction VDb is directed from the reference position VLb to the movement center point RCb (−Z side and −Y side). The reference direction VDl is directed from the reference position VLl to the movement center point RCl (−Z side and + Y side). The reference direction VDr is the direction from the reference position VLr to the movement center point RCr (−Z
Side and + Y side).

Thus, the virtual viewpoint VP is a viewpoint when the movement center point RC is viewed from the viewpoint position VD. Further, the reference virtual viewpoint VPat is a viewpoint (top view) in which the vehicle 2 is viewed from directly above.
The reference virtual viewpoint VPaf is a viewpoint in which the vehicle 2 is seen from the front and the front of the vehicle 2 from above (front view). The reference virtual viewpoint VPab is a viewpoint (back view) for viewing the vehicle 2 from behind and from above. The reference virtual viewpoint VPal is a viewpoint (left side view) in which the left side region of the vehicle 2 is viewed from the diagonally left rear and above the vehicle 2. The reference virtual viewpoint VPar is a viewpoint (right side view) in which the vehicle 2 is viewed from the upper right and obliquely rearward to the right side area of the vehicle 2.

Such a reference virtual viewpoint VPa is selected by the user. The selection by the user is selected by a touch operation on a touch panel button associated with each reference virtual viewpoint VP. That is, when one of the reference virtual viewpoints VPa is selected by the user while the composite image CP is displayed on the display, the image generation unit 81b generates a composite image viewed from the reference virtual viewpoint VPa. Thus, even when the user cannot change the virtual viewpoint back to the reference virtual viewpoint VP after changing the position of the virtual viewpoint, the user can easily set the virtual viewpoint to the reference virtual viewpoint VP by touching the touch panel button. can do.

In the initial stage of image display before the user's viewpoint selection, one of the viewpoints of the reference virtual viewpoint VPa is set in advance. At this time, a reference virtual viewpoint VPat overlooking the vehicle 2 from directly above.
It is preferable to generate the composite image CP. A wide area around the vehicle 2 can be presented to the user,
This is because it is suitable as an initial display.

Next, a method for moving the viewpoint position VL using the movement center point RC will be described. First, a method of moving the viewpoint position VL in the left-right direction (horizontal direction) when viewing the display from the user will be described with reference to FIGS. 9 and 10. The movement of the viewpoint position VL described below is as follows:
This is performed in response to a left / right (horizontal) flick operation by the user on the touch panel.

FIG. 9 is a diagram when the vehicle 2 is viewed from the vertically upward direction (+ Z side), and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves. The reference position VLl moves along the movement locus OBlh around the movement center point RCl. The movement of the viewpoint position VL is based on an orthogonal coordinate system cc (world coordinate system) that is relatively fixed to the vehicle 2 with the position of the movement center point RCl as the origin. The coordinate axis that is the reference for the movement of the viewpoint position VLl is the orthogonal coordinate system cc at the position of the movement center point RCl.
Z axis.

FIG. 10 is a view of the vehicle 2 as viewed from the right side in the horizontal direction (+ X side), and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves as in FIG. The reference position VLl is the movement center point R
The movement locus OBlh is moved around Cl. The coordinate axis that is the reference for the movement of the viewpoint position VLl is the Z axis (AXl) of the orthogonal coordinate system cc at the position of the movement center point RC.

9 and 10, the viewpoint direction VDl is directed from the viewpoint position VLl to the movement center point RCl regardless of where the viewpoint position VLl is located on the movement locus OBlh. Therefore, when the reference position VLl moves on the movement locus OBlh, the user looking at the display 83 seems to move the image in the horizontal direction.

In addition, when the user performs a flick operation from the left direction to the right direction (horizontal direction) on the touch panel 83a, VLl moves counterclockwise on the movement locus OBlh. On the other hand, when the user performs a flick operation from the right direction to the left direction (horizontal direction) on the touch panel 83a, VLl moves clockwise on the movement locus OBlh. As a result, the direction of the user's flick operation matches the direction in which the image moves, and the user can intuitively perform the touch panel operation.

Next, a method of moving the viewpoint position VL in the vertical direction (vertical direction) when viewing the display from the user will be described with reference to FIGS. 11 and 12. The movement of the viewpoint position VL described below is performed in response to a flick operation in the vertical direction (vertical direction) on the touch panel by the user.

11 and 12, a three-dimensional XYZ orthogonal coordinate system VPcc (viewpoint coordinate system) is further used in addition to the three-dimensional XYZ orthogonal coordinate system cc, and directions and directions are shown as appropriate. Each coordinate axis of the orthogonal coordinate system VPcc is fixed relative to the virtual viewpoint VP. That is, X
The axis (VPx) direction is the left-right direction of the virtual viewpoint VP, the Y-axis (VPy) direction is the front-rear direction of the virtual viewpoint VP, and the Z-axis (VPz) direction is the vertical direction of the virtual viewpoint VP. Also, + X side is the virtual viewpoint V
The right side of P, + Y side is the front side of the virtual viewpoint VP, and + Z side is the upper side of the virtual viewpoint VP.
Accordingly, the −X side is the left side of the virtual viewpoint VP, the −Y side is the rear side of the virtual viewpoint VP, and the −Z side is the vertically lower side.

FIG. 11 is a diagram when the vehicle 2 is viewed from the vertically upward direction (+ Z side), and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves. The reference position VLl moves along the movement locus OBlv around the movement center point RCl. The movement of the viewpoint position VL is based on the orthogonal coordinate system VPcc that is relatively fixed to the virtual viewpoint VP with the position of the movement center point RCl as the origin. Viewpoint position VLl
The coordinate axis that serves as a reference for the movement is the X axis (AXlv) of the orthogonal coordinate system VPcc at the position of the movement center point RCl.

FIG. 12 is a view of the vehicle 2 viewed from the right side (−X side) in the horizontal direction, and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves, as in FIG. 11. The reference position VLl moves along the movement locus OBlv around the movement center point RCl. The coordinate axis that is the reference for the movement of the viewpoint position VLl is the X axis of the orthogonal coordinate system cc at the position of the movement center point RC.

11 and 12, the viewpoint direction VDl is directed from the viewpoint position VLl to the movement center point RCl regardless of where the viewpoint position VLl is located on the movement locus OBlv. Therefore, when the reference position VLl moves on the movement locus OBlv, the user looking at the display 83 seems to move the image in the vertical direction.

The movement locus OBlv is set above the ground contact surface GR of the vehicle 2. Therefore,
The reference position VLl moves along the movement trajectory OBlv indicated by a broken line, and the movement trajectory OB indicated by a solid line.
lx does not move. As a result, it is possible to prevent image display from a viewpoint position that cannot normally occur when the vehicle 2 is viewed from below the ground contact surface. The user can move the viewpoint position without a sense of incongruity.

In addition, when the user performs a flick operation from the upper direction to the lower direction (vertical direction) on the touch panel 83a, VLl moves clockwise on the movement locus OBlv. On the other hand, when the user performs a flick operation from the downward direction to the upward direction (vertical direction) on the touch panel 83a, VLl moves counterclockwise on the movement locus OBlv. As a result, the direction of the user's flick operation matches the direction in which the image moves, and the user can intuitively perform the touch panel operation.

<1-5. Example of composite image display>
Next, examples of the composite image CP displayed on the display 83 will be described with reference to FIGS.
Will be described with reference to FIG.

FIG. 13 is an example in which the reference virtual viewpoint VPat is set as the virtual viewpoint VP, and the composite image CPt1 when the vehicle 2 is viewed from the reference virtual viewpoint VPat is displayed on the display 83. The composite image CPt1 displays the vehicle body image 10 in addition to the peripheral image showing the periphery of the vehicle 2. Composite image CP
t1 is displayed when the user performs a touch operation on the touch panel button TB13 indicating the periphery of the vehicle 2 and when the composite image CP is initially displayed. By referring to the composite image CPt1, the user can check the situation around the vehicle 2 in a list.

FIG. 14 shows a composite image in which the reference virtual viewpoint VPat is set by moving clockwise on OBlh in FIG. 9 after the composite image CPt1 (FIG. 13) is displayed, and the vehicle 2 is viewed from the set reference virtual viewpoint VPat. In this example, CPt2 is displayed on the display 83. The composite image CPt2 is
When the user views the display 83, the vehicle body image 10 is displayed in addition to the peripheral image showing the periphery of the vehicle 2 displayed by rotating to the left. The composite image CPt2 is a composite image CPt1 (FIG. 13).
) Is displayed, and a flick operation FH is performed on the touch panel 83a from the left direction to the right direction (horizontal direction). By referring to the composite image CPt2, the user can check the situation around the vehicle 2 from an angle different from that of the composite image CPt1.

FIG. 15 shows that after the composite image CPt1 (FIG. 13) is displayed, a reference virtual viewpoint VPat is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is set from the set reference virtual viewpoint VPat.
Is an example in which a composite image CPt3 is displayed on the display 83. Composite image CPt3
Is displayed including directly under the front end of the vehicle 2. The composite image CPt2 is displayed when a flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a in a state where the composite image CPt1 (FIG. 13) is displayed. By referring to the composite image CPt2,
The user can check the situation around the vehicle 2 including immediately under the front end.

FIG. 16 is an example in which the reference virtual viewpoint VPaf is set to the virtual viewpoint VP, and the composite image CPf1 when the vehicle 2 is viewed from the reference virtual viewpoint VPaf is displayed on the display 83. The composite image CPf1 displays the vehicle body image 10 in addition to the peripheral image showing the front of the vehicle 2. Composite image CP
f1 is displayed when the user touches the touch panel button TB16 indicating the front of the vehicle 2. The user can confirm the situation ahead of the vehicle 2 by referring to the composite image CPf1.

FIG. 17 shows a composite image in which the reference virtual viewpoint VPaf is set by moving clockwise on OBlh in FIG. 9 after the composite image CPf1 (FIG. 16) is displayed, and the vehicle 2 is viewed from the set reference virtual viewpoint VPaf. In this example, CPf2 is displayed on the display 83. The composite image CPf2 is
When the user views the display 83, the vehicle body image 10 is displayed in addition to the peripheral image showing the periphery of the vehicle 2 displayed by rotating to the left. The composite image CPf2 is a composite image CPf1 (FIG. 16).
) Is displayed, and a flick operation FH is performed on the touch panel 83a from the left direction to the right direction (horizontal direction). By referring to the composite image CPf2, the user can confirm the situation ahead of the vehicle 2 from an angle different from that of the composite image CPf1.

18, after displaying the composite image CPf1 (FIG. 16), the reference virtual viewpoint VPaf is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 from the set reference virtual viewpoint VPaf.
Is an example in which the composite image CPf3 is displayed on the display 83. Composite image CPf3
Displays the front area of the vehicle 2 in more detail than the composite image CPf1. The composite image CPt3 is displayed when a flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a in a state where the composite image CPf1 (FIG. 16) is displayed. Composite image CPt3
The user can confirm the situation ahead of the vehicle 2 in detail.

FIG. 19 is an example in which the reference virtual viewpoint VPab is set to the virtual viewpoint VP, and the composite image CPb1 when the vehicle 2 is viewed from the reference virtual viewpoint VPab is displayed on the display 83. The composite image CPb1 displays the vehicle body image 10 in addition to the peripheral image showing the rear of the vehicle 2. Composite image CP
b <b> 1 is displayed when the user performs a touch operation on the touch panel button TB <b> 19 indicating the rear of the vehicle 2. The user can check the situation behind the vehicle 2 by referring to the composite image CPb1.

FIG. 20 shows a composite image in which the reference virtual viewpoint VPab is set by moving clockwise on OBlh in FIG. 9 after the composite image CPb1 (FIG. 19) is displayed, and the vehicle 2 is viewed from the set reference virtual viewpoint VPab. In this example, CPb2 is displayed on the display 83. The composite image CPb2 is
The user looks at the display 83 and displays the vehicle body image 10 in addition to the peripheral image showing the rear of the vehicle 2 displayed by moving leftward. The composite image CPb2 is composed of the composite image CPb1 (FIG. 19).
) Is displayed, and a flick operation FH is performed on the touch panel 83a from the left direction to the right direction (horizontal direction). By referring to the composite image CPb2, the user can check the situation behind the vehicle 2 from an angle different from that of the composite image CPb1.

In FIG. 21, after the composite image CPb1 (FIG. 19) is displayed, the reference virtual viewpoint VPab is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is set from the set reference virtual viewpoint VPab.
Is an example in which the synthesized image CPb3 is displayed on the display 83. Composite image CPb3
Displays the rear region of the vehicle 2 in more detail than the composite image CPb1. The composite image CPb3 is displayed when a flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a in a state where the composite image CPb1 (FIG. 16) is displayed. Composite image CPb3
By referring to, the user can confirm the situation behind the vehicle 2 in detail.

FIG. 22 is an example in which the reference virtual viewpoint VPal is set as the virtual viewpoint VP, and the composite image CPL1 when the vehicle 2 is viewed from the reference virtual viewpoint VPal is displayed on the display 83. The composite image CPL1 displays the vehicle body image 10 in addition to the peripheral image indicating the left side of the vehicle 2. Composite image CP
l1 is displayed when the user performs a touch operation on the touch panel button TB22 indicating the left side of the vehicle 2. The user can check the situation on the left side of the vehicle 2 by referring to the composite image CPL1.

FIG. 23 shows a composite image in which the reference virtual viewpoint VPal is set by moving clockwise on OBlh in FIG. 9 after the composite image CP11 (FIG. 22) is displayed, and the vehicle 2 is viewed from the set reference virtual viewpoint VPal. In this example, CPb2 is displayed on the display 83. The composite image CPL2 is
When viewing the display 83 from the user, the vehicle body image 10 is displayed in addition to the peripheral image showing the left side of the vehicle 2 displayed by moving in the left direction. The composite image CPl2 is composed of the composite image CPl1 (FIG. 22).
) Is displayed, and a flick operation FH is performed on the touch panel 83a from the left direction to the right direction (horizontal direction). By referring to the composite image CP12, the user can confirm the situation on the left side of the vehicle 2 from an angle different from that of the composite image CP11.

24, after displaying the composite image CP11 (FIG. 22), the reference virtual viewpoint VPal is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is set from the set reference virtual viewpoint VPal.
Is an example in which the composite image CPL3 is displayed on the display 83. Composite image CP13
Displays the left region of the vehicle 2 in more detail than the composite image CP11. The composite image CPl3 is displayed when the flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a in a state where the composite image CPl1 (FIG. 22) is displayed. Composite image CP13
The user can confirm the situation on the left side of the vehicle 2 in detail.

Note that the composite image CP showing the right side of the vehicle 2 may be operated and controlled in a left-right contrast to the case where the left side of the vehicle 2 described with reference to FIGS. 22 to 24 is shown. By referring to the composite image CP showing the right side of the vehicle 2, the user can check the situation on the right side of the vehicle 2.

<1-6. Processing>
Next, the processing procedure of the image display system 1 will be described with reference to FIGS. FIG.
Reference numeral 5 denotes a processing procedure of the image acquisition device 3, the security device 5, and the mobile terminal 8. FIG.
These show the processing procedure of the alarm device 6 and the portable terminal 8. This process is repeatedly executed at a predetermined cycle.

First, the security device 5 determines whether or not the monitoring sensor 5a is activated (step S101 in FIG. 25).

When the security device 5 determines that the monitoring sensor 5a has been activated (Yes in step S101), the security device 5 controls the mail notification unit 5b to indicate to the user's portable terminal 8 that the monitoring sensor 5a has been activated, that is, the vehicle 2 is stolen. An e-mail is transmitted with the content that a prior phenomenon that leads to (2) has occurred (step S102). Note that the case where the monitoring sensor 5a is activated means that the monitoring sensor 5a detects the occurrence of vibration or tilt in the vehicle 2, the case where glass breakage is detected, the case where an intruder into the vehicle 2 is detected, etc. It is.

When the security device 5 determines that the monitoring sensor 5a is not operating (No in step S101), and when the mail notification unit 5b transmits an e-mail, the processing of the security device 5 ends, and a predetermined time The process starts again after the elapse.

When the communication unit 82 of the mobile terminal 8 receives the email transmitted from the email notification unit 5b,
The display control unit 81c displays the contents of the e-mail on the display 83 (step S201).
). The user confirms the content of the e-mail displayed on the display and determines whether or not to display a surrounding image of the vehicle 2. When the user desires to display a surrounding image of the vehicle 2, the user may perform a touch operation on a predetermined position of the touch panel 83a.

The operation detection unit 81d determines whether or not the user has performed a touch operation on a predetermined position on the touch panel 83a where a peripheral image should be displayed (step S202).

When the operation detection unit 81d determines that the user performs a touch operation on a predetermined position on the touch panel 83a where the peripheral image should be displayed and desires to display the image (Yes in step S202), the notification unit 81e notifies the image acquisition device 3. A signal requesting image transmission is transmitted (step S203).

On the other hand, when the operation detection unit 81d performs a touch operation on a predetermined position indicating that the user does not display the peripheral image on the touch panel 83a, that is, when it is determined that the image is not desired (No in step S202), the process ends. To do. This is because it is not necessary to continue the process as long as the user does not desire to display an image.

When the request receiving unit 31a of the image acquisition device 3 receives a signal requesting transmission of an image from the portable terminal 8, the camera control unit 31b controls the camera 4 (4F, 4B, 4L, 4R) to start shooting. (Step S301).

When the image acquisition unit 31c receives the image data captured by the camera 4 (4F, 4B, 4L, 4R), the image transmission unit 31d transmits the image data to the mobile terminal 8 via the communication unit 32 (
Step S302).

When the image acquisition unit 81a of the portable terminal 8 receives the image data transmitted from the image acquisition device 3, the image generation unit 81b sets the position and direction of the virtual viewpoint for generating the peripheral image. (Step S204). When first generating a peripheral image, the image generation unit 81b sets the position of the virtual viewpoint to be directly above the vehicle 2 and the direction of the virtual viewpoint to be downward (top view). This is a viewpoint in which the user looks down from a position directly above the vehicle 2. Such a viewpoint is preferable as the position and direction of the virtual viewpoint to be displayed to the user first. This is because the entire periphery of the vehicle is displayed, so that a wide range of situations can be communicated to the user.

When the virtual viewpoint is set, the image generation unit 81b generates a peripheral image by the above-described method.
Then, the vehicle body image data 84a is read from the storage unit 84, and a composite image is generated by combining the vehicle body image with the generated peripheral image (step S205).

When the image generation unit 81b generates a composite image, the display control unit 81c displays the composite image on the display 83 (step S206). Thereby, the user can confirm the state around the vehicle 2.

When the display control unit 81c displays the composite image on the display 83, the operation detection unit 81d
It is determined whether or not the user has flicked the display 83 (step S).
207).

If the operation detection unit 81d determines that a flick operation has been performed by the user (step S2).
In step 07, the image generation unit 81b sets the virtual viewpoint again (step S204).

At this time, if the user flicks the display 83 in the left-right direction (horizontal direction), as described above, the position of the virtual viewpoint is set by rotating the virtual viewpoint based on the vertical axis of the world coordinate system. To do.

Further, when the flick operation is an operation from the right side to the left side with respect to the display 83, the position of the virtual viewpoint so that the virtual viewpoint is counterclockwise when viewed from above the vertical axis of the world coordinate system. Rotate. When the flick operation is an operation from the left side to the right side with respect to the display 83, the position of the virtual viewpoint is rotated so as to be clockwise when the vertical axis of the world coordinate system is viewed from above.

On the other hand, when the user performs a flick operation in the vertical direction (vertical direction) on the display 83, the position of the virtual viewpoint is set by rotating the virtual viewpoint based on the horizontal axis of the viewpoint coordinate system as described above. .

Further, when the flick operation is an operation from the upper side to the lower side with respect to the display 83, the virtual viewpoint is virtual so as to be counterclockwise when the horizontal axis of the viewpoint coordinate system is viewed from the right of the viewpoint. Rotate the viewpoint position. In addition, when the flick operation is an operation from the lower side to the upper side with respect to the display 83, the virtual viewpoint position is set so as to be clockwise when the horizontal axis of the viewpoint coordinate system is viewed from the right side of the viewpoint. Rotate.

By such a method of rotating the position of the virtual viewpoint, the direction in which the user wants to move the viewpoint by the flick operation matches the direction in which the displayed image moves, and the user intuitively
An image can be moved by an operation on the touch panel 83a.

On the other hand, when the operation detection unit 81d determines that the flick operation is not performed by the user (No in step S207), the operation detection unit 81d determines whether there is an operation for changing the virtual viewpoint to the reference viewpoint (step S208). The operation detection unit 81d determines whether or not the user has performed a touch operation on any touch panel button indicating the reference viewpoint.

When the operation detection unit 81d determines that there is an operation to change the virtual viewpoint to the reference viewpoint (Yes in step S208), the image generation unit 81b sets the position and direction of the virtual viewpoint to the reference viewpoint desired by the user by the touch operation. (Step S204). For example, the image generation unit 81b
When the user desires a reference viewpoint for displaying the left side area of the vehicle 2, the position and direction of the virtual viewpoint as shown in FIGS. 9 and 10 are set. The reference viewpoints are five viewpoints immediately above the vehicle, the left side of the vehicle 2, the right side of the vehicle 2, the front side of the vehicle 2, and the rear side of the vehicle 2. The position of the movement center point RC, which is a specific point set at the specific position of the vehicle, is set at a position corresponding to the reference viewpoint desired by the user.

On the other hand, if the operation detection unit 81d determines that there is no operation to change the virtual viewpoint to the reference viewpoint (
No in step S208), it is determined whether or not the user intends to activate the alarm device 6 provided in the vehicle 2 (step S209). The operation detection unit 81d is displayed on the touch panel 83 by the user.
Judgment is made based on whether or not a touch operation has been performed on a predetermined position indicating that the alarm device in a is to be activated.

If the operation detection unit 81d determines that the user is about to activate the alarm device 6 (Yes in step S209), the notification unit 81e transmits a signal requesting the image acquisition device 3 to activate the alarm device 6 ( Step S210).

When the notification unit 81e transmits a signal requesting to operate the alarm device 6, the anti-theft unit 31e of the image acquisition device 3 causes the alarm device 6 to perform an alarm (step S401). Alarm 6
The alarm is terminated after a predetermined time has elapsed. The predetermined time may be sufficient to give a warning to the suspicious person. For example, 5 seconds. From the beginning, the user may operate when the alarm ends.

When the operation detection unit 81d determines that the user is not trying to activate the alarm device 6 (
When the anti-theft unit 31e gives an alarm to the alarm device 6 (step S210), the operation detection unit 81d determines whether or not the user is about to end the image display (step S211). ).

When the operation detection unit 81d determines that the user is about to end the image display (Yes in step S211), the notification unit 81e transmits a signal requesting the image acquisition device 3 to end the shooting (step S212). ). When the notification unit 81e transmits a signal requesting the image acquisition device 3 to end photographing, the processing of the portable terminal 8 ends.

On the other hand, when the operation detection unit 81d determines that the user does not end the image display (No in step S211), the display control unit 81c continues to generate and display the image, and the process executes step S205 and the subsequent steps again. To do.

When the image transmission unit 31d transmits image data in step S302, the image acquisition device 3 determines whether or not an end request signal for requesting the request reception unit 31a to end shooting is transmitted from the mobile terminal 8 (step S303). ). The image acquisition device 3 repeatedly transmits the image data obtained most recently in the camera 4 (4F, 4B, 4L, 4R) to the portable terminal 8 until receiving the end request signal. Thereby, the image generation part 81b of the portable terminal 8 can generate a peripheral image showing the state of the periphery of the vehicle 2 in substantially real time based on the most recently obtained image data.

When the request receiving unit 31a determines that an end request signal for requesting to end shooting is transmitted from the mobile terminal 8 (Yes in step S303), the camera control unit 31b determines that the camera 4 (4
F, 4B, 4L, 4R) are controlled to stop photographing (step S304). When the camera control unit 31b controls the camera 4 (4F, 4B, 4L, 4R) and stops photographing, the processing of the image acquisition device 3 ends.

As described above, in the first embodiment, based on the user's operation, the position of the virtual viewpoint is changed and the composite image is displayed. Thereby, the subject can be easily displayed from the viewpoint desired by the user.

<2. Second Embodiment>
<2-1. Overview>
Next, a second embodiment will be described. In the first embodiment described above, the viewpoint position VL
When the vehicle is directly above the vehicle 2 and the viewpoint direction VD is downward (−Z side) (that is, the top view), the viewpoint position VL with respect to the flick operation in the left-right direction (horizontal direction) of the user
Is based on the vertical axis (Z axis) of the orthogonal coordinate system cc. As a result, the composite image CP is displayed rotated to the left or right.

However, even if the synthesized image CP is rotated and displayed, a new peripheral area is not presented to the user.

Therefore, in the second embodiment, in the top view, the movement of the viewpoint position VL is based on the axis (Y axis) that is the front and rear of the vehicle 2 in the orthogonal coordinate system cc. Thereby, in the top view, when the user performs a flick operation in the left-right direction (horizontal direction) on the touch panel 83a, the left and right areas of the vehicle 2 including the side surface of the vehicle 2 are displayed. The left and right areas can be confirmed in detail.

Since the second embodiment includes the same configuration and processing as the first embodiment, the first embodiment will be described below.
Differences from the embodiment will be mainly described.

<2-2. Configuration>
First, the structure of the portable terminal 8 in 2nd Embodiment is demonstrated. FIG. 27 shows the second
The structure of the portable terminal 8 of embodiment of this is shown. The main difference from the first embodiment is that the control unit 81 of the mobile terminal 8 includes a coordinate axis conversion unit 81f. Other configurations are the same as those in the first embodiment and function in the same manner.

When the virtual viewpoint is the top view, the coordinate axis conversion unit 81f converts the coordinate axis serving as a reference for moving the viewpoint position VL from the vertical axis (Z axis) of the orthogonal coordinate system cc to the vehicle longitudinal axis (Y axis).

<2-3. Virtual viewpoint settings>
Next, a method of moving the viewpoint position VL in the left-right direction (horizontal direction) when viewing the display from the user in the top view display will be described with reference to FIGS. 28 and 29. FIG. The movement of the viewpoint position VL described below is performed in response to a left / right (horizontal) flick operation on the touch panel by the user in the top view display.

FIG. 28 is a view of the vehicle 2 as viewed from the vertically upward direction (+ Z side), and shows a trajectory along which the reference position VLt of the reference virtual viewpoint VPat moves. The reference position VLt moves on the movement locus OBth around the movement center point RCt. The movement of the viewpoint position VLt is based on the orthogonal coordinate system cc that is relatively fixed to the virtual viewpoint VP with the position of the movement center point RCt as the origin. The coordinate axis that serves as a reference for the movement of the viewpoint position VLt is the Y axis of the orthogonal coordinate system cc at the position of the movement center point RCt (
AXth). That is, it is an axis along the longitudinal direction of the vehicle 2.

FIG. 29 is a view of the vehicle 2 as viewed from the rear side in the horizontal direction (−Y side), and shows a trajectory along which the reference position VLt of the reference virtual viewpoint VPat moves as in FIG. 28. The reference position VLt moves on the movement locus OBth around the movement center point RCt. The movement locus OBth is the horizontal rear side (−Y
When viewed from the side, it is circular. The coordinate axis serving as the reference for movement of the viewpoint position VLl is the Y axis of the orthogonal coordinate system cc at the position of the movement center point RCt.

28 and 29, the viewpoint direction VDt is directed from the viewpoint position VLt to the movement center point RCt regardless of where the viewpoint position VLt is located on the movement locus OBth. Therefore, when the reference position VLt moves on the movement locus OBth, the user looking at the display 83 seems to move the image in the left-right direction (horizontal direction). In particular, since the coordinate axis that is the reference for the movement of the viewpoint position VLl is the Y axis of the orthogonal coordinate system cc at the position of the movement center point RCt, the user refers to the image around the vehicle 2 so as to go around the side surface of the vehicle 2 can do.

The movement locus OBth is not set below the ground contact surface GR of the vehicle 2. Accordingly, the reference position VLt moves along the movement locus OBth indicated by a broken line, and the movement locus OBtx indicated by a solid line does not move.

When the user performs a flick operation from the left direction to the right direction (horizontal direction) on the touch panel 83a, the VLt moves counterclockwise on the movement locus OBth. On the other hand, when the user performs a flick operation from left to right (horizontal direction) on touch panel 83a, VLt moves clockwise on movement locus OBth. As a result, the direction of the user's flick operation matches the direction in which the image moves, and the user can intuitively perform the touch panel operation.

<2-4. Example of composite image display>
In the upper part of FIG. 30, the reference virtual viewpoint VPat is set as the virtual viewpoint VP, and the reference virtual viewpoint V
This is an example in which the composite image CPt <b> 1 when the vehicle 2 is viewed from Pat is displayed on the display 83.
That is, the composite image CPt1 is a composite image when the virtual viewpoint is the top view.

The lower part of FIG. 30 shows a composite image CPt when the coordinate axis for moving the viewpoint position VL is converted from the vertical axis (Z axis) of the orthogonal coordinate system cc to the vehicle longitudinal axis (Y axis) after displaying the composite image CPt1.
4. That is, the reference virtual viewpoint VPa is moved counterclockwise on OBth in FIG.
In this example, t is set, and a composite image CPt4 in which the vehicle 2 is viewed from the set reference virtual viewpoint VPat is displayed on the display 83. The composite image CPt4 is the vehicle 2 including the side of the vehicle 2
A vehicle body image 10 is displayed in addition to the surrounding image of the vehicle. For this reason, the user can view the vehicle 2 from the top view.
The left and right areas can be confirmed in detail. The composite image CPt4 is displayed when a flick operation FH is performed from left to right (horizontal direction) on the touch panel 83a in a state where the composite image CPt1 is displayed. By referring to the composite image CPt2, the user can confirm in detail the left and right regions of the vehicle 2 including the side surface of the vehicle 2.

<2-5. Processing>
Next, a processing procedure in the second embodiment will be described with reference to FIGS. FIG. 31 shows a processing procedure of the image acquisition device 3, the security device 5, and the mobile terminal 8 in the second embodiment. The difference from the processing procedure in the first embodiment shown in FIG.
It is a point provided with the process of step S209.

First, the image generation unit 81b sets the position and direction of a virtual viewpoint for generating a peripheral image. (Step S204). When the image generating unit 81b first generates a peripheral image,
The position of the virtual viewpoint is set directly above the vehicle 2 and the direction of the virtual viewpoint is set downward (top view). In addition, the coordinate axis conversion unit 81f sets a reference axis for moving the viewpoint position VL as the vehicle longitudinal axis (Y axis).

When step S205, step S206, and step S207 are executed, the operation detection unit 81d determines whether there is an operation for changing the virtual viewpoint to the reference viewpoint (step S2).
08). The operation detection unit 81d determines whether or not the user has performed a touch operation on any touch panel button indicating the reference viewpoint.

When the operation detection unit 81d determines that there is an operation for changing the virtual viewpoint to the reference viewpoint (Yes in step S208), the image generation unit 81b executes a coordinate axis setting process (step S209).

FIG. 32 shows details of the coordinate axis setting process (step S209). Coordinate axis setting process (
When the process proceeds to step S209), the operation detection unit 81d determines whether or not the virtual viewpoint has been changed to the top view viewpoint (step S401).

If it is determined that the virtual viewpoint has been changed to the top view viewpoint (Yes in step S401).
), The coordinate axis conversion unit 81f uses the coordinate axis serving as a reference for moving the viewpoint position VL as the vehicle longitudinal axis (
Y axis) is set (step S402).

On the other hand, if it is determined that the virtual viewpoint has been changed to a viewpoint other than the top view (step S40).
1), the coordinate axis conversion unit 81f sets the coordinate axis serving as a reference for moving the viewpoint position VL to the vertical axis (Z axis) (step S403).

When step S402 or step S403 is executed, the process returns to FIG. 31, and the processes after step S204 are executed again.

As described above, in the second embodiment, in the top view, the movement of the viewpoint position VL is based on the axis (Y axis) that is the front and rear of the vehicle 2 in the orthogonal coordinate system cc. Thereby, in the top view, when the user performs a flick operation in the left-right direction (horizontal direction) on the touch panel 83a,
Since the left and right areas of the vehicle 2 including the side surfaces of the vehicle 2 are displayed, the user can check the left and right areas of the vehicle 2 in detail.

<3. Modification>
The embodiment of the present invention has been described above. However, the present invention is not limited to the above embodiment. Various modifications are possible. Hereinafter, modified examples will be described. The above and following embodiments can be combined as appropriate.

In the embodiment described above, the image acquisition device 3 starts shooting when receiving a request for image transmission from the user. However, photographing may be started before receiving a request for image transmission from the user. That is, photographing may be started when the monitoring sensor 5a is activated. In this case, the user can refer to the situation around the vehicle 2 from when the abnormality of the vehicle 2 occurs such that the monitoring sensor 5a operates.

FIG. 33 shows an outline of the image display system 1a in which the image acquisition device 3 starts photographing when the monitoring sensor 5a is operated. When the monitoring sensor 5a detects an abnormality in the vehicle 2, the security device 5
Transmits a signal requesting the image acquisition device 3 to start photographing. When receiving the signal, the image acquisition device 3 operates the camera 4 to start photographing. That is, the image acquisition device 3 does not wait for an image transmission request from the user, and starts shooting when an abnormality occurs in the vehicle 2. When the image acquisition device 3 starts photographing, the image acquisition device 3 transmits the image data to the server SV installed outside. Thereby, image data is preserve | saved at server SV from the time of abnormality of the vehicle 2. FIG. Therefore, when the user receives an email notifying the occurrence of an abnormality in the vehicle 2 from the security device 5, the user makes a request for image transmission to the server SV. When the server SV receives the request for image transmission, the server SV transmits image data from when the abnormality of the vehicle 2 occurs to the portable terminal 8 possessed by the user. Thereby, the user can refer to the situation around the vehicle 2 when the monitoring sensor 5a is operated, that is, when the abnormality of the vehicle 2 occurs. Further, if the server SV is a dedicated server that transmits and receives image data, there is no possibility that the image data is intercepted by others, and the confidentiality of the image data can be improved. In addition, what is necessary is just to request | require transmission of the present image data to the server SV, when a user wants to confirm the mode around the present vehicle 2. FIG. In this case, the server SV omits transmission of the image data from the time of occurrence of the abnormality to the present, and transmits the current image data.

Another modification will be described. In the above embodiment, it has been described that the image acquisition device 3, the camera 4, the security device 5, and the alarm device 6 are mounted on the vehicle 2. However, it may not be the vehicle 2. It may be a house or a building. In short, it may be any land, building, or article to be monitored.

Moreover, in the said embodiment, the touch panel was demonstrated as what a user performs input operation. However, it may not be a touch panel. A push button type switch may be used. For example,
This is a so-called cross key. In short, it is sufficient that the user's input operation can be determined.

Moreover, in the said embodiment, the image acquisition apparatus 3 and the portable terminal 8 were demonstrated as a separate apparatus. However, the image acquisition device 3 and the mobile terminal 8 may be configured as an integrated device.

Further, in the above embodiment, each function has been described as being realized as software according to a program. However, it does not have to be software. It may be realized as an electrical hardware circuit.

DESCRIPTION OF SYMBOLS 1 Image display system 2 Vehicle 3 Image acquisition device 4 Camera 5 Security device 6 Alarm device 7 Network 8 Mobile terminal

Claims (5)

An image display device for displaying an image,
Generating means for generating a composite image viewed from a virtual viewpoint using an image captured by a camera;
Operation detecting means for detecting a user operation for the change of the viewpoint of the composite image,
The operation detecting means includes
An image area in which the composite image is displayed;
A button area on which a plurality of buttons for selecting display of the composite image corresponding to the predetermined viewpoint is displayed;
In the image area, the composite image corresponding to the viewpoint selected by the button is displayed,
When a touch operation is performed in the image area, the composite image is displayed in the image area at the viewpoint according to the touch operation ,
The image display device , wherein the touch operation is a flick operation or a pinch-in operation . An image display device for displaying an image,
Generating means for generating a composite image viewed from a virtual viewpoint using an image captured by a camera;
Operation detecting means for detecting a user operation for the change of the viewpoint of the composite image,
The operation detecting means includes
An image area in which the composite image is displayed;
A button area on which a plurality of buttons for selecting display of the composite image corresponding to the predetermined viewpoint is displayed;
The buttons are arranged in the up / down / left / right direction of the button area corresponding to the front / rear / left / right direction of the vehicle,
In the button area, a button for selecting display at the viewpoint of the initial display in a wide area around the vehicle is further displayed.
The image display device, wherein the composite image corresponding to the viewpoint selected by the button is displayed in the image area. Communication means for obtaining image information about a composite image generated in a form viewed from a virtual viewpoint using a photographed image photographed by a camera;
Display means for displaying an image based on the obtained image information;
Operation detecting means for giving a user instruction to change the viewpoint of the composite image displayed on the display means,
The operation detection unit includes a touch panel that detects a user's touch operation on the display unit,
The display means includes
An image area in which the composite image is displayed;
A button area in which a plurality of buttons for instructing display of the composite image corresponding to the predetermined viewpoint is displayed;
In the image area, the composite image corresponding to the viewpoint selected by the button is displayed,
When a touch operation is performed in the image area, the composite image of the viewpoint corresponding to the touch operation is displayed in the image area ,
The terminal device , wherein the touch operation is a flick operation or a pinch-in operation . Communication means for obtaining image information about a composite image generated in a form viewed from a virtual viewpoint using a photographed image photographed by a camera;
Display means for displaying an image based on the obtained image information;
Operation detecting means for giving a user instruction to change the viewpoint of the composite image displayed on the display means,
The operation detection unit includes a touch panel that detects a user's touch operation on the display unit,
The display means includes
An image area in which the composite image is displayed;
A button area in which a plurality of buttons for instructing display of the composite image corresponding to the predetermined viewpoint is displayed;
The buttons are arranged in the up / down / left / right direction of the button area corresponding to the front / rear / left / right direction of the vehicle,
In the button area, a button for selecting display at the viewpoint of the initial display in a wide area around the vehicle is further displayed.
The terminal device, wherein the composite image corresponding to the viewpoint selected by the button is displayed in the image area. It said communication means when the security device detects an abnormality, the terminal apparatus according to claim 3 or claim 4, wherein the receiving a notification.

Images