JP7542420B2 - Camera position adjustment information generating device and program - Google Patents

Description

The present invention relates to a camera position adjustment information generating device and program that generates information for adjusting the position of a camera.

Currently, there has been progress in the development of a device (360-degree video player) that allows the viewer to operate a controller to wipe between video footage shot in the present and video footage shot in the past (see non-patent literature 1 and 2).

Figure 19 is a diagram illustrating a state in which a 360-degree video player wipes and switches between a video currently shot (After video, hereafter referred to as "Current Video AA") and a video shot in the past (Before video, hereafter referred to as "Past Video BB"). A viewer 100 is wearing an HMD (Head Mounted Display) 101 and is currently viewing video AA using VR (Virtual Reality) technology.

The viewer 100 operates the controller 103 to move the boundary line 102 from left to right, causing the past video BB to appear to the left of the boundary line 102. The viewer 100 then watches the past video BB in the left area, and the current video AA in the right area. In other words, the viewer 100 can wipe and switch between the current video AA and the past video BB by operating the controller 103.

In this type of 360-degree video player, in order to align the spatial relationship between the past video BB and the current video AA, when shooting the current video AA, the camera needs to be installed in the same position as when the past video BB was shot.

If the camera for the current video AA is not installed in the same position as the camera for the past video BB, the current video AA will be misaligned with respect to the past video BB, and the viewer 100 will feel uncomfortable when the two videos are wiped to switch between them.

Conventionally, adjustments to align the camera position of the current video AA with the camera position of the past video BB have been performed manually by the photographer (user) while visually viewing the past video BB and the current video AA when shooting the current video AA. Specifically, the user adjusted the camera position while comparing a one-frame equirectangular image extracted from the past video BB with the equirectangular video output live by the camera shooting the current video AA.

The equirectangular format is an equirectangular projection format, and is a format that conforms to the Mercator projection for representing maps. Like Mercator projection images, equirectangular images represent a subject in three-dimensional space using the coordinate values of the longitude and latitude components of a sphere, but do not represent the distance from the camera.

"Diverse Vision "Before/After" VR 360 video comparison beyond time", [online], Japan Broadcasting Corporation, [Retrieved October 12, 2020], Internet <URL: https://www.nhk.or.jp/strl/english/ibc2020/b5.html> H.Kawakita, T.Niida, and K.Hisatomi, “Development of a 360 Video Player for Head-Mounted Display to Enable Comparison Between Before and After Incident”, Forum on Information Technology 2020 (FIT2020), Part III, K-010 ,pp.361-363,2020

However, the method of manually adjusting the camera position while comparing the equirectangular image extracted from the past video BB with the equirectangular image of the current video AA had the problem that the adjustments were difficult.

It is difficult to intuitively recognize the positional relationship in equirectangular images and videos, and even if it is found that there is a positional misalignment, it is difficult to know in which direction and how much to move the camera. For this reason, users have to make adjustments by trial and error, and often end up compromising on the adjustments when a positional misalignment occurs. Here, the equirectangular image extracted from the past video BB is called the past image, and the equirectangular image extracted from the current video AA is called the current image.

The present invention has been made to solve the above problems, and its purpose is to provide a camera position adjustment information generating device and program capable of generating camera position adjustment information used to adjust the position of the camera capturing the current image in order to align the spatial positional relationship between the past image and the current image.

In order to solve the above problem, the camera position adjustment information generating device of claim 1 generates adjustment information for aligning the position of a camera currently capturing an image with the position of the camera when the image was captured in the past, and the image represents the coordinate values of a subject to be captured using longitude and latitude components, the image captured in the past is a past image, the image captured currently is a current image, the camera that captured the past image is a past camera, the camera that captured the current image is a current camera, and the positions of the past camera and the current camera in three-dimensional space are respectively a past camera position and a current camera position,
an acquisition unit that acquires the current image captured by the current camera, where, when the three-dimensional space positions corresponding to the past camera position, the current camera position, and the positions of landmarks at K points (k=1, 2, ..., K, K is an integer equal to or greater than 3) set commonly to the past image and the current image are projected onto a predetermined horizontal plane, the past camera position is set to a previously set past camera horizontal plane coordinate O, the current camera position is set to a current camera horizontal plane coordinate P, the position of the landmark at the K point is set to a landmark horizontal plane coordinate qk (=q1, q2, ..., qK), and the distance between the current camera horizontal plane coordinate P and the landmark horizontal plane coordinate qk is set to a landmark horizontal distance (P-qk) (=P-q1, P-q2, ..., P-qK); and, by specifying the landmarks in the past image as past landmarks in accordance with an operation by a user, outputs each of the cursor positions of the past image at the K points as past landmark cursor positions of the K points, and by specifying the landmarks in the current image as current landmarks, a landmark distance adjustment unit which calculates the distance between the past camera horizontal plane coordinate O and the landmark horizontal plane coordinate qk by adjusting the landmark horizontal distance (P-qk) to match the horizontal distance between the current camera position measured by a predetermined sensor and a position in three-dimensional space corresponding to the current landmark according to an operation by the user on each position in the landmark horizontal plane coordinate qk so that the distance matches the horizontal distance between the current camera position measured by a predetermined sensor and a position in three-dimensional space corresponding to the current landmark, and outputs the distance as the past camera/landmark horizontal distance of the K point; and a landmark distance adjustment unit which calculates the image coordinates corresponding to the past landmark cursor position of the K point and the current landmark cursor position of the K point outputted by the landmark designation unit as past landmark image coordinates Qk'(=Q1',Q2', ..., QK') and current landmark image coordinates Qk (=Q1, Q2, ..., QK), respectively, and adjusts the past landmark image coordinates Qk' and the past landmark image coordinates Qk' of the K point outputted by the landmark distance adjustment unit. a current camera horizontal position estimating unit which calculates the landmark horizontal plane coordinate qk based on the past camera/landmark horizontal distance, calculates an angle between the two landmark horizontal plane coordinates qk with the current camera horizontal plane coordinate P as a reference based on a coordinate value of a longitude component of the current landmark image coordinate Qk, calculates the current camera horizontal plane coordinate P based on the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk and the angle, calculates the landmark horizontal distance (P-qk) based on the current camera horizontal plane coordinate P and the landmark horizontal plane coordinate qk, and calculates horizontal position adjustment information consisting of a horizontal direction and distance for aligning the current camera position with the past camera position based on the past camera horizontal plane coordinate O and the current camera horizontal plane coordinate P; and and a current camera vertical position estimation unit that calculates angles αk'(α1',α2', ..., αK') and angles αk (α1, α2, ..., αK) between the past and current camera positions, and calculates vertical position adjustment information consisting of a vertical height for aligning the current camera position with the past camera position, based on the angles αk' and αk, and the landmark horizontal distance (P-qk) calculated by the current camera horizontal position estimation unit; and a display unit that displays the past image and the current image acquired by the acquisition unit, and displays a mark of the past landmark cursor position of the K point and a mark of the current landmark cursor position of the K point output by the landmark designation unit on the past image and the current image, respectively, and displays the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk calculated by the current camera horizontal position estimation unit, the current camera horizontal plane coordinate P, the landmark horizontal distance (P-qk), and the horizontal position adjustment information, as well as the vertical position adjustment information calculated by the current camera vertical position estimation unit.

The camera position adjustment information generating device of claim 2 is the camera position adjustment information generating device of claim 1, wherein the current camera horizontal position estimation unit includes a first landmark image coordinate conversion unit that converts the past landmark cursor position of the K point into the past landmark image coordinate Qk' and converts the current landmark cursor position of the K point into the current landmark image coordinate Qk, a landmark horizontal plane coordinate conversion unit that converts the past landmark image coordinate Qk' converted by the first landmark image coordinate conversion unit into the landmark horizontal plane coordinate qk based on a previously set past camera/landmark tentative horizontal distance of the K point between the past camera horizontal plane coordinate O and the landmark horizontal plane coordinate qk, or a past camera/landmark horizontal distance of the K point, and The present invention is characterized by comprising: a current camera horizontal plane coordinate estimation unit that calculates the angle between the two landmark horizontal plane coordinates qk based on the current camera horizontal plane coordinate P based on the coordinate value of the longitude component of the landmark image coordinate Qk, and estimates the current camera horizontal plane coordinate P based on the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk, and the angle; a landmark horizontal distance calculation unit that obtains the distance between the current camera horizontal plane coordinate P estimated by the current camera horizontal plane coordinate estimation unit and the landmark horizontal plane coordinate qk converted by the landmark horizontal plane coordinate conversion unit as the landmark horizontal distance (P-qk); and a horizontal position adjustment information processing unit that obtains the horizontal position adjustment information based on the past camera horizontal plane coordinate O and the current camera horizontal plane coordinate P estimated by the current camera horizontal plane coordinate estimation unit.

The camera position adjustment information generating device of claim 3 is the camera position adjustment information generating device of claim 2, characterized in that the current camera horizontal plane coordinate estimation unit estimates the current camera horizontal plane coordinate P based on the equation of a circle passing through the landmark horizontal plane coordinate qk of a combination of two predetermined points out of K points and the current camera horizontal plane coordinate P, the sine theorem equation using the angle between the landmark horizontal plane coordinate qk of the combination of the two points based on the current camera horizontal plane coordinate P, the equation of a circle passing through the landmark horizontal plane coordinate qk of a combination of other two points out of K points and the current camera horizontal plane coordinate P, and the sine theorem equation using the angle between the landmark horizontal plane coordinate qk of the combination of the other two points based on the current camera horizontal plane coordinate P.

The camera position adjustment information generating device of claim 4 is the camera position adjustment information generating device of any one of claims 1 to 3, wherein the current camera vertical position estimation unit includes a second landmark image coordinate conversion unit that converts the past landmark cursor position of the K point into the past landmark image coordinate Qk' and converts the current landmark cursor position of the K point into the current landmark image coordinate Qk, an angle processing unit that calculates the angle αk' based on the coordinate value of the latitude component of the past landmark image coordinate Qk' converted by the second landmark image coordinate conversion unit and calculates the angle αk based on the coordinate value of the latitude component of the current landmark image coordinate Qk, and a second landmark image coordinate conversion unit that converts the past landmark cursor position of the K point into the past landmark image coordinate Qk' and converts the current landmark image coordinate Qk into the current landmark image coordinate Qk. The system is characterized by having a height calculation unit that calculates the height hk using the formula: h = d (tan(α) - tan(α')), where the angles αk' and αk calculated by the height calculation unit are angles α' and α, the landmark horizontal distance (P-qk) is distance d, and height hk (h1, h2, ..., hK) indicating the distance between the past camera position and the current camera position projected on the vertical plane is height h; and a vertical position adjustment information processing unit that identifies the height hk with the smallest absolute value when all the heights hk calculated by the height calculation unit are positive values or when all the heights hk are negative values, and determines the identified hk as the vertical position adjustment information.

The camera position adjustment information generating device of claim 5 is the camera position adjustment information generating device according to any one of claims 1 to 4, further comprising a tracking processing unit that tracks the current landmark cursor positions of the K points output by the landmark designation unit as the user changes the current camera position, and the tracking processing unit performs a matching process between a peripheral image of the current landmark cursor position of the current image acquired by the acquisition unit and a predetermined template image for each of the current landmark cursor positions of the K points, cuts out the matched peripheral image as a landmark image, sets the landmark image as the new template image, and sets a predetermined position within the landmark image as a new current landmark cursor position; and a tracking unit that outputs the new current landmark cursor position of the K point set by the image cutout unit as the current landmark cursor position of the K point after tracking, and the current camera horizontal position estimation unit sets the image coordinates corresponding to the current landmark cursor position of the K point after tracking output by the tracking unit as the current landmark image coordinates Qk.

The camera position adjustment information generating device of claim 6 is the camera position adjustment information generating device of any one of claims 1 to 5, further comprising a snap processing unit that, in accordance with an operation by the user, snaps the cursor position of the past image to a feature point position in the past image by designating the landmark of the past image as the past landmark, and snaps the cursor position of the current image to a feature point position in the current image by designating the landmark of the current image as the current landmark, and the snap processing unit includes a feature point extraction unit that extracts a plurality of feature point positions from the past image and the current image, and, when the cursor position is within an area of the past image, identifies a feature point position that is closest to the cursor position among the plurality of feature point positions extracted from the past image by the feature point extraction unit, and snaps the cursor position to the identified feature point position. a snapping unit that snaps to a feature point position, outputs the cursor position after snapping to the landmark designation unit as the cursor position of the past image, and if the cursor position is within the area of the current image, identifies the feature point position that is closest to the cursor position among the multiple feature point positions extracted from the current image, snaps the cursor position to the identified feature point position, and outputs the cursor position after snapping to the landmark designation unit as the cursor position of the current image; and the landmark designation unit inputs the cursor position of the past image and the cursor position of the current image from the snapping unit, respectively, outputs each of the cursor positions of the past image at K points as the past landmark cursor positions of the K points, and outputs each of the cursor positions of the current image at K points as the current landmark cursor positions of the K points.

Furthermore, the program of claim 7 causes a computer to function as a camera position adjustment information generating device according to any one of claims 1 to 6.

As described above, according to the present invention, it is possible to generate camera position adjustment information that is used when adjusting the position of the camera capturing the current image in order to align the spatial positional relationship between the previous image and the current image. Then, by using the camera position adjustment information, the user can easily and quickly adjust the position of the camera capturing the current image to the position where the previous image was captured.

1 is a schematic diagram showing an entire system including a camera position adjustment information generating device according to a first, second, and third embodiments. 1 is a block diagram showing an example of the configuration of a camera position adjustment information generating device according to a first embodiment. FIG. 2 is a diagram showing an example of a display screen. FIG. 13 is a diagram illustrating an example of designating landmarks LM(1), LM(2), and LM(3). 11 is a flowchart illustrating a user operation procedure. 13 is a block diagram showing an example of the configuration of a current camera horizontal position estimation unit. FIG. 13 is a flowchart illustrating an example of processing by a current camera horizontal position estimation unit. 11A to 11C are diagrams illustrating an example of processing by a current camera horizontal position estimation unit. 7A and 7B are diagrams illustrating the estimation process of the current camera horizontal plane coordinate P by the current camera horizontal plane coordinate estimation unit (step S704). 13 is a block diagram showing an example of the configuration of a current camera vertical position estimation unit. FIG. 13 is a flowchart illustrating an example of processing by a current camera vertical position estimation unit. 11 is a diagram for explaining the angle calculation process (step S1103) performed by the angle processing unit. FIG. FIG. 11 is a block diagram showing an example of the configuration of a camera position adjustment information generating device according to a second embodiment. 4 is a block diagram showing a configuration example of a tracking processing unit. FIG. 13 is a flowchart illustrating an example of processing by a tracking processing unit. FIG. 13 is a block diagram showing an example of the configuration of a camera position adjustment information generating device according to a third embodiment. 11 is a block diagram showing an example of the configuration of a snap processing unit. FIG. 13 is a flowchart showing an example of processing by a snap processing unit. FIG. 13 is a diagram illustrating a state in which a currently shot video and a previously shot video are switched by wiping using a 360-degree video player.

The following describes in detail the embodiments of the present invention with reference to the drawings. The present invention is characterized in that, by user operation, the current camera position is estimated from the past camera position using the coordinates on the image of K landmark points (K is an integer of 3 or more) common to equirectangular images captured in the past and live equirectangular images captured currently, and the horizontal distance between the current camera and the landmark measured using a laser range finder or the like, so that the angle at which any two landmarks are viewed is consistent on the map, thereby generating camera position adjustment information for aligning the current camera position with the past camera position.

This makes it possible to generate camera position adjustment information that is used to adjust the current camera position in order to match the spatial positional relationship of the current image with the image captured in the past. Then, by using the camera position adjustment information, the user can easily and quickly adjust the current camera position.

In the following, the equirectangular image in three-dimensional space captured in the past will be referred to as "past image B," and the equirectangular live video image in three-dimensional space captured currently will be referred to as "current image A." Furthermore, the camera that captured past image B will be referred to as the "past camera," and the camera that captures current image A will be referred to as the "current camera." Furthermore, the position of the past camera in three-dimensional space will be referred to as the "past camera position," and the position of the current camera in three-dimensional space will be referred to as the "current camera position."

[Overall system]
1 is a schematic diagram showing an entire system including the camera position adjustment information generating devices of Examples 1, 2, and 3. This system is configured with the camera position adjustment information generating devices 1, 2, and 3 of Examples 1, 2, and 3, a camera (current camera) 4, a capture device 5, and a laser range finder 6. The camera position adjustment information generating devices 1, 2, and 3 and the capture device 5 are connected by a Universal Serial Bus (USB). Furthermore, the current camera 4 and the capture device 5 are connected by a High Definition Multimedia Interface (HDMI (registered trademark)).

The camera position adjustment information generating devices 1, 2, and 3 input live video from the current camera 4 via the capture device 5, and display the pre-stored past image B and the live video. The camera position adjustment information generating devices 1, 2, and 3 then generate camera position adjustment information by estimating the position of the current camera 4 on the horizontal plane from the position of the past camera, using the coordinates on the image of the landmark at point K that is specified commonly in the past image B and the corresponding current image A by user operation, and the horizontal distance measured using the laser rangefinder 6, so that the angle at which any two landmarks are viewed is consistent on the map.

Here, landmarks are reference points required to generate camera position adjustment information in past image B and current image A, and are specified by the user.

The current camera 4 is a 360-degree camera that captures and outputs live video, which is current video AA. The past camera (not shown) is a 360-degree camera that captures and outputs past video BB. The live video is made up of multiple current images A in chronological order, and the past video BB is made up of multiple past images B in chronological order.

The capture device 5 is a converter that converts the live video of the HDMI (registered trademark) standard currently output from the camera 4 into live video of the USB standard, and the converted live video is output to the camera position adjustment information generating devices 1, 2, and 3.

The laser range finder 6 is a sensor that measures the horizontal distance between the current camera 4 and a position in three-dimensional space corresponding to the landmark point K specified in the current image A by the user's operation.

As mentioned above, K, which indicates the number of landmark points, is an integer equal to or greater than 3, but for simplicity, we will assume that the number of landmark points is 3 (K=3).

Example 1
First, a camera position adjustment information generating device 1 of Example 1 will be described. Fig. 2 is a block diagram showing an example of the configuration of the camera position adjustment information generating device 1 of Example 1. This camera position adjustment information generating device 1 includes a past image storage unit 10, a camera video acquisition unit 11, a video and image display unit 12, a landmark designation unit 13, a landmark distance adjustment unit 14, a current camera horizontal position estimation unit 15, a current camera vertical position estimation unit 16, and a map display unit 17.

The past image storage unit 10 stores past image B of past video BB in equirectangular format captured by a past camera. The past camera is assumed to have been installed at a predetermined fixed position in three-dimensional space (hereinafter referred to as the "past camera position") when the past video BB was captured.

The camera image acquisition unit 11 acquires the live image in equirectangular format currently captured by the camera 4, and outputs the live image to the image and video display unit 12.

The video and image display unit 12 reads out past image B from the past image storage unit 10, inputs live video from the camera video acquisition unit 11, and outputs past image B and the live video to a display device (not shown) to display them at a specified position on the screen.

Figure 3 is a diagram showing an example of a display screen. This display screen displays live video, a landmark image, a past image B, and a horizontal map. Displaying live video means that current image A is also displayed.

As shown in FIG. 3, the video and image display unit 12 displays the live video in the upper left area of the display screen, and the past image B in the lower left area of the display screen.

Returning to FIG. 2, the user uses the pointing device 7, not shown in FIG. 1, to click on past image B displayed on the screen to designate three landmarks as past landmarks.

The landmark designation unit 13 inputs the cursor position in past image B according to the user's operation of the pointing device 7. Then, the landmark designation unit 13 sets the input cursor position as the past landmark cursor position according to the user's click operation, thereby designating three past landmarks in past image B as landmarks LM common to past image B and current image A. The landmark designation unit 13 outputs the three past landmark cursor positions to the current camera horizontal position estimation unit 15 and the current camera vertical position estimation unit 16.

The user also uses the pointing device 7 to perform a click operation on the current image A of the live video image displayed on the screen to specify three current landmarks that correspond (are in the same positions) to the three past landmarks.

The landmark designation unit 13 inputs the cursor position in the current image A according to the user's operation of the pointing device 7. Then, the landmark designation unit 13 sets the input cursor position as the current landmark cursor position according to the user's click operation, thereby designating three current landmarks in the current image A corresponding to the three past landmarks as landmarks LM common to the past image B and the current image A. The landmark designation unit 13 outputs the three current landmark cursor positions to the current camera horizontal position estimation unit 15 and the current camera vertical position estimation unit 16.

Although not shown in FIG. 2, the landmark designation unit 13 outputs the three past landmark cursor positions and the three current landmark cursor positions to the video and image display unit 12. As a result, the video and image display unit 12 displays a past image B including marks at the three past landmark cursor positions, and displays a current image A including marks at the three current landmark cursor positions.

In the example of FIG. 3, the display of past image B includes landmark images LM-kB (k=1, 2, 3, i.e. LM-1B, LM-2B, LM-3B), which are peripheral images for each of the three past landmark cursor positions. In addition, the display of live video includes landmark images LM-kA (k=1, 2, 3, i.e. LM-1A, LM-2A, LM-3A), which are peripheral images for each of the three current landmark cursor positions.

Referring to FIG. 3, the horizontal plane map displayed on the right side of the display screen is a map showing the positions in three-dimensional space corresponding to the positions of the three past landmarks in past image B of the past camera (and the three current landmarks in the corresponding current image A of the live video), as well as the past camera positions and the current camera positions in the three-dimensional space, projected onto a specified horizontal plane.

The coordinates of the past camera position in three-dimensional space projected onto a horizontal plane are called "past camera horizontal plane coordinates O" and are set in advance. The coordinates of the current camera position in three-dimensional space projected onto a horizontal plane are called "current camera horizontal plane coordinates P". The coordinates of the three-dimensional space positions corresponding to the positions of the three past landmarks in past image B of the past camera (and the corresponding three current landmarks in current image A of the live video) projected onto a horizontal plane are called "landmark horizontal plane coordinates qk (k = 1, 2, 3, i.e. q1, q2, q3)". The points of landmark horizontal plane coordinates q1, q2, q3 are called "landmark LM-k (k = 1, 2, 3, i.e. LM-1, LM-2, LM-3)".

In the horizontal plane, the distance between the current camera horizontal coordinate point P and the landmark horizontal coordinate points q1, q2, and q3 (landmarks LM-1, LM-2, and LM-3) is defined as the "landmark horizontal distance (P-qk) (k = 1, 2, and 3, i.e. P-q1, P-q2, and P-q3)."

At landmark LM-1, Up: 0.10 m indicates a height of 0.10 m, which corresponds to the height h1 calculated by the current camera vertical position estimation unit 16 described below. At landmark LM-2, Up: 0.03 m indicates a height of 0.03 m, which corresponds to the height h2 calculated by the current camera vertical position estimation unit 16 described below. At landmark LM-3, Up: 0.06 m indicates a height of 0.06 m, which corresponds to the height h3 calculated by the current camera vertical position estimation unit 16 described below.

Horizontal position adjustment information and vertical position adjustment information are displayed at the top of the horizontal plane map. Horizontal position adjustment information consists of the horizontal direction (direction of the arrow) and the horizontal distance. In this example, the horizontal direction is in the direction of landmark LM-2 based on the point of the current camera horizontal plane coordinate P, and the horizontal distance is 0.04 m. In other words, this horizontal position adjustment information means that, on the horizontal plane of three-dimensional space, the horizontal position of the current camera 4 can be aligned with the previous camera position by moving the current camera 4 0.04 m in the direction of approximately 45 degrees where landmark LM-2 is located.

The vertical position adjustment information also consists of the vertical height, which in this example is 0.03 m. In other words, this vertical position adjustment information means that by moving the current camera 4 vertically upward by 0.03 m on the vertical plane of the three-dimensional space, the vertical position of the current camera 4 can be aligned with the previous camera position. The horizontal position adjustment information is estimated by the current camera horizontal position estimation unit 15, which will be described later, and the vertical position adjustment information is estimated by the current camera vertical position estimation unit 16, which will be described later.

Returning to FIG. 2, the user performs a drag operation using a pointing device 7 (not shown in FIG. 1) for each of the landmarks LM-1, LM-2, and LM-3 on the horizontal map displayed on the screen to move the landmarks LM-1, LM-2, and LM-3 (the points at landmark horizontal coordinates q1, q2, and q3) closer to or farther away from the point at the past camera horizontal coordinate O, thereby adjusting the landmark horizontal distance (P-q1, P-q2, P-q3) between the point at the current camera horizontal coordinate P and the points at the landmark horizontal coordinates q1, q2, and q3.

Specifically, the user uses a laser range finder 6 to measure the horizontal distance (hereinafter referred to as the "distance measurement value") between the point of the current camera horizontal plane coordinate P and the positions in three-dimensional space corresponding to each of the three current landmarks. The user then performs a drag operation by moving each of the landmarks LM-1, LM-2, and LM-3 on the horizontal map displayed on the screen closer to or farther away from the point of the past camera horizontal plane coordinate O, so that the landmark horizontal distances (P-q1, P-q2, P-q3) match the distance measurement values.

The landmark distance adjustment unit 14 inputs the initial landmark horizontal plane coordinates q1, q2, q3 (landmark horizontal plane coordinates q1, q2, q3 calculated from the past camera/landmark tentative horizontal distance described later) from the current camera horizontal position estimation unit 15 described later, and inputs the cursor position according to the drag operation performed by the user.

The landmark distance adjustment unit 14 here restricts the cursor position input in accordance with the drag operation to a straight line (passing through the points) between the points of the initial landmark horizontal plane coordinates q1, q2, and q3 and the point of the previously set previous camera horizontal plane coordinate O. That is, in the horizontal plane map before the drag operation in FIG. 8(1) described later, the drag operation is restricted to a straight line (passing through the points) between the landmarks LM-1, LM-2, and LM-3 and the point of the previous camera horizontal plane coordinate O.

The landmark distance adjustment unit 14 adjusts the landmark horizontal distance (P-q1, P-q2, P-q3) according to the drag operation so that it matches the distance measurement value, thereby calculating the distance between the cursor position (point in horizontal plane coordinates) at the time of the drag operation and a point in the previously set past camera horizontal plane coordinates O (hereinafter referred to as the "past camera/landmark horizontal distance"), and outputs the past camera/landmark horizontal distance to the current camera horizontal position estimation unit 15.

The current camera horizontal position estimation unit 15 inputs the three current landmark cursor positions and the three past landmark cursor positions from the landmark designation unit 13, and inputs the preset fixed distance between the landmarks LM-1, LM-2, LM-3 and the point of the past camera horizontal plane coordinate O (hereinafter referred to as the "past camera/landmark tentative horizontal distance"). The current camera horizontal position estimation unit 15 also inputs the past camera/landmark horizontal distance at the time of the drag operation from the landmark distance adjustment unit 14.

The current camera horizontal position estimation unit 15 converts each of the three current landmark cursor positions into landmark equirectangular coordinates (current landmark image coordinates) Qk (k = 1, 2, 3, i.e. Q1, Q2, Q3), which are the coordinates of an equirectangular image. The current camera horizontal position estimation unit 15 also converts each of the three past landmark cursor positions into landmark equirectangular coordinates (past landmark image coordinates) Qk' (k = 1, 2, 3, i.e. Q1', Q2', Q3').

The current camera horizontal position estimation unit 15 calculates the initial landmark horizontal plane coordinates q1, q2, q3 based on the landmark equirectangular coordinates Q1', Q2', Q3' and the previous camera/landmark tentative horizontal distance. The current camera horizontal position estimation unit 15 also calculates the landmark horizontal plane coordinates q1, q2, q3 at the time of the drag operation based on the landmark equirectangular coordinates Q1', Q2', Q3' and the previous camera/landmark horizontal distance.

The current camera horizontal position estimation unit 15 estimates the current camera horizontal coordinate P based on the landmark horizontal coordinates q1, q2, q3, etc. That is, the current camera horizontal position estimation unit 15 determines the angle at which any two landmarks (for example, LM-1, LM-2) are viewed from the landmark equirectangular coordinates Q1, Q2, Q3. Then, since the candidates for the horizontal position of the current camera 4 are drawn on a horizontal map in the shape of an arc, the current camera horizontal position estimation unit 15 estimates the intersection of two arcs obtained from two sets of landmarks as the horizontal position of the current camera 4 (current camera horizontal coordinate P). Details will be described later. The current camera horizontal position estimation unit 15 also determines the landmark horizontal distance (P-q1, P-q2, P-q3) and horizontal position adjustment information.

The current camera horizontal position estimation unit 15 outputs the initial landmark horizontal coordinates q1, q2, q3 to the landmark distance adjustment unit 14. The current camera horizontal position estimation unit 15 also outputs the landmark horizontal distance (P-q1, P-q2, P-q3) to the current camera vertical position estimation unit 16. The current camera horizontal position estimation unit 15 also outputs the landmark horizontal coordinates q1, q2, q3, the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3) and the horizontal position adjustment information to the map display unit 17. Details of the current camera horizontal position estimation unit 15 will be described later.

The current camera vertical position estimation unit 16 inputs three past landmark cursor positions and three current landmark cursor positions from the landmark designation unit 13. In addition, the current camera vertical position estimation unit 16 inputs the landmark horizontal distances (P-q1, P-q2, P-q3) from the current camera horizontal position estimation unit 15.

The current camera vertical position estimation unit 16 converts the three past landmark cursor positions into landmark equirectangular coordinates Q1', Q2', Q3', respectively, and determines the angle αk' (k = 1, 2, 3, i.e. α1', α2', α3') between the position corresponding to the landmark horizontal plane coordinates q1, q2, q3 projected on the vertical plane and the horizontal direction based on the past camera position projected on a vertical plane perpendicular to the horizontal plane in the three-dimensional space (see Figure 12 described later).

The current camera vertical position estimation unit 16 also converts the three current landmark cursor positions into landmark equirectangular coordinates Q1, Q2, and Q3, respectively, and determines the angle αk (k=1, 2, 3, i.e. α1, α2, α3) between the horizontal direction and the positions corresponding to the landmark horizontal plane coordinates q1, q2, and q3 projected onto the vertical plane, based on the current camera position projected onto a vertical plane perpendicular to the horizontal plane in the three-dimensional space (see FIG. 12, described later).

The current camera vertical position estimation unit 16 calculates the distance between the previous camera position and the current camera position projected on a vertical plane as height hk (k = 1, 2, 3, i.e. h1, h2, h3) based on the landmark horizontal distance (P-q1, P-q2, P-q3) and the angles α1, α2, α3, α1', α2', α3', and calculates vertical position adjustment information (see FIG. 12 described later). Note that in FIG. 12 described later, the previous camera position and the current camera position are on the same vertical axis because they have been adjusted to the same position on the horizontal plane by the user's operation of horizontally moving the current camera 4 in accordance with the instructions of the horizontal position adjustment information calculated by the current camera horizontal position estimation unit 15.

The current camera vertical position estimation unit 16 outputs the heights h1, h2, h3 and the vertical position adjustment information to the map display unit 17. Details of the current camera vertical position estimation unit 16 will be described later.

The map display unit 17 inputs the landmark horizontal coordinates q1, q2, q3, the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3) and horizontal position adjustment information from the current camera horizontal position estimation unit 15. The map display unit 17 also inputs the heights h1, h2, h3 and vertical position adjustment information from the current camera vertical position estimation unit 16.

The map display unit 17 outputs the landmark horizontal coordinates q1, q2, q3, the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3), horizontal position adjustment information, heights h1, h2, h3 and vertical position adjustment information to a display device (not shown), and displays them at a specified position on the screen as a horizontal map, as shown in FIG. 3.

As shown in FIG. 3, the map display unit 17 displays the landmark horizontal coordinates q1, q2, q3, the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3), horizontal position adjustment information, heights h1, h2, h3, and vertical position adjustment information in the right area of the display screen.

The map display unit 17 also determines and displays the angle xn_1,2 angle between the landmark horizontal coordinates q1, q2 based on the landmark horizontal coordinates q1, q2 and the current camera horizontal coordinate P, with the point at the current camera horizontal coordinate P as the reference point. The map display unit 17 also determines and displays the angle xn_2,3 angle between the landmark horizontal coordinates q2, q3 based on the landmark horizontal coordinates q2, q3 and the current camera horizontal coordinate P, with the point at the current camera horizontal coordinate P as the reference point.

This allows the user to recognize the horizontal position adjustment information after performing a drag operation so that the landmark horizontal distance (P-q1, P-q2, P-q3) matches the measured distance value, and can move the current camera 4 horizontally accordingly. In addition, the user can recognize the vertical position adjustment information after moving the current camera 4 horizontally, and can move the current camera 4 vertically accordingly.

[User operation procedure]
Next, the operation procedure of the user who uses the camera position adjustment information generating device 1 and the laser range finder 6 will be described.

First, the following prerequisites are required:
<a> The current camera 4 is always kept horizontal using a spirit level or the like (the optical axis of the lens of the current camera 4 is aligned with the horizontal direction).
<b> Three or more corresponding landmarks LM can be specified in the past image B and the current image A of the live video. In addition, the three or more landmarks LM have different x values (coordinate values of the longitude component) when the horizontal axis in the east-west direction is the x axis representing the longitude component and the vertical axis in the north-south direction is the y axis representing the latitude component in each of the past image B and the current image A.
<c> The horizontal distance between the current camera position and positions in three-dimensional space corresponding to three or more landmarks LM in the current image A of the live video can be measured using a laser range finder 6 or the like.

Figure 4 is a diagram for explaining an example of designating landmarks LM(1), LM(2), and LM(3), and shows the case where K=3 in <b> above. As shown on the left side of Figure 4, if the coordinate values of the longitude component, which is the x value on the xy axis, are the same for at least two of the landmarks LM(1), LM(2), and LM(3) in each of past image B and current image A, for example, (landmarks LM(2) and LM(3)), the condition of <b> above is not met.

Also, as shown on the right side of Figure 4, in each of the past image B and the current image A, the landmarks LM(1), LM(2), and LM(3) may have the same coordinate value of the latitude component (y value) as long as the coordinate value of the longitude component (x value) on the xy axis is different.

Figure 5 is a flowchart explaining the user's operation procedure. The user clicks on three corresponding landmarks LM for the past image B and the current image A of the live video displayed on the screen (step S501).

Specifically, the user performs the following steps S501-1 to S501-6.
(Step S501-1): Click the first landmark LM in past image B. (Step S501-2): Click the corresponding first landmark LM in current image A of the live video. (Step S501-3): Click the second landmark LM in past image B. (Step S501-4): Click the corresponding second landmark LM in current image A of the live video. (Step S501-5): Click the third landmark LM in past image B. (Step S501-6): Click the corresponding third landmark LM in current image A of the live video.

Three past landmark cursor positions and three current landmark cursor positions are specified by the landmark specification unit 13 shown in FIG. 2. Then, the current camera horizontal position estimation unit 15 uses the previously set past camera/landmark tentative horizontal distance to determine the landmark horizontal coordinates q1, q2, q3, and determines the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3), and horizontal position adjustment information. The current camera vertical position estimation unit 16 determines the heights h1, h2, h3, and vertical position adjustment information. Then, a horizontal map including this information is displayed on the screen.

The user uses the laser range finder 6 to measure the horizontal distance (distance measurement value) from the laser range finder 6, which is currently in the same position as the camera 4, to positions in three-dimensional space corresponding to the three landmarks LM (step S502).

The user refers to the horizontal map displayed on the screen and drags the three landmarks LM-1, LM-2, and LM-3 in order on the horizontal map so that the landmark horizontal distances (P-q1, P-q2, P-q3) match the corresponding distance measurements (step S503).

Specifically, the user performs the following steps S503-1 to S503-4.
(Step S503-1): By dragging the landmark LM-1 on the horizontal map, the landmark horizontal distance (P-q1) is adjusted to the corresponding measured distance value. (Step S503-2): By dragging the landmark LM-2 on the horizontal map, the landmark horizontal distance (P-q2) is adjusted to the corresponding measured distance value. (Step S503-3): By dragging the landmark LM-3 on the horizontal map, the landmark horizontal distance (P-q3) is adjusted to the corresponding measured distance value. (Step S503-4): The landmark horizontal distances (P-q1, P-q2, P-q3) are rechecked, and if they deviate by more than a predetermined threshold, the operations of steps S503-1 to S503-3 are performed again.

As a result, for each drag operation in steps S503-1 to S503-3, the landmark distance adjustment unit 14 determines the past camera/landmark horizontal distance. Then, the current camera horizontal position estimation unit 15 determines the landmark horizontal plane coordinates q1, q2, q3 using the past camera/landmark horizontal distance, and determines the current camera horizontal plane coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3), and horizontal position adjustment information. In addition, the current camera vertical position estimation unit 16 determines the heights h1, h2, h3, and vertical position adjustment information.

Then, the screen display of the horizontal map containing this information is updated, and the horizontal position adjustment information is determined when the landmark horizontal distances (P-q1, P-q2, P-q3) match the corresponding measured distances (when they match the corresponding measured distances within a specified error).

Assuming that the horizontal movement operation of the current camera 4 (step S505 described later) has not been completed (step S504: N), the user refers to the horizontal map displayed on the screen and moves the current camera 4 horizontally in the direction and distance indicated by the confirmed horizontal position adjustment information (step S505). The user then redoes the operations of steps S501 to S503. As a result, the current camera horizontal position estimation unit 15 determines the current camera horizontal coordinate P, etc., and the current camera vertical position estimation unit 16 determines the heights h1, h2, h3 and vertical position adjustment information, and the vertical position adjustment information is confirmed.

After completing steps S501 to S503, the user determines that the horizontal movement operation of the current camera 4 (step S504) is complete (step S504: Y), and moves the current camera 4 vertically to the height indicated by the confirmed vertical position adjustment information (step S506).

In this way, the user moves the current camera 4 a predetermined distance in the horizontal direction according to the horizontal position adjustment information displayed on the screen after the drag operation, and moves the current camera 4 a predetermined distance in the vertical direction according to the vertical position adjustment information displayed on the screen after the horizontal movement operation. This allows the user to easily and quickly adjust the current camera position to the previous camera position.

(Current camera horizontal position estimation unit 15)
Next, the current camera horizontal position estimating unit 15 shown in Fig. 2 will be described in detail. Fig. 6 is a block diagram showing an example of the configuration of the current camera horizontal position estimating unit 15, Fig. 7 is a flowchart showing an example of the processing of the current camera horizontal position estimating unit 15, and Fig. 8 is a diagram explaining an example of the processing of the current camera horizontal position estimating unit 15.

This current camera horizontal position estimation unit 15 includes a landmark equirectangular coordinate conversion unit (landmark image coordinate conversion unit) 20, a landmark horizontal plane coordinate conversion unit 21, a current camera horizontal plane coordinate estimation unit 22, a landmark horizontal distance calculation unit 23, and a horizontal position adjustment information processing unit 24.

The current camera horizontal position estimation unit 15 inputs the three current landmark cursor positions and the three past landmark cursor positions from the landmark designation unit 13, inputs the previously set past camera/landmark tentative horizontal distance, and inputs the past camera/landmark horizontal distance at the time of the drag operation from the landmark distance adjustment unit 14 (step S701).

The landmark equirectangular coordinate conversion unit 20 converts the three current landmark cursor positions into the landmark equirectangular coordinates Q1, Q2, and Q3 of the current image A, respectively, and converts the three past landmark cursor positions into the landmark equirectangular coordinates Q1', Q2', and Q3' of the past image B, respectively (step S702). As a result, the coordinates of the cursor positions on the screen are converted into coordinates that indicate the longitude and latitude components of the equirectangular image.

The landmark equirectangular coordinate transformation unit 20 outputs the landmark equirectangular coordinates Q1, Q2, and Q3 to the current camera horizontal plane coordinate estimation unit 22, and outputs the landmark equirectangular coordinates Q1', Q2', and Q3' to the landmark horizontal plane coordinate transformation unit 21.

The landmark horizontal plane coordinate conversion unit 21 inputs the landmark equirectangular coordinates Q1', Q2', and Q3' from the landmark equirectangular coordinate conversion unit 20. In addition, the landmark horizontal plane coordinate conversion unit 21 inputs the previously set previous camera/landmark tentative horizontal distance during the initial processing before the drag operation, and inputs the previous camera/landmark horizontal distance at the time of the drag operation from the landmark distance adjustment unit 14 during the drag operation.

The landmark horizontal plane coordinate conversion unit 21 generates landmark horizontal plane coordinates q1, q2, q3 from the landmark equirectangular coordinates Q1', Q2', Q3' and the past camera/landmark tentative horizontal distance or the past camera/landmark horizontal distance, and outputs them to the current camera horizontal plane coordinate estimation unit 22, the landmark horizontal distance calculation unit 23, and the map display unit 17 (step S703).

In addition, the landmark horizontal plane coordinate conversion unit 21 also outputs the initial landmark horizontal plane coordinates q1, q2, and q3 generated during the initial processing to the landmark distance adjustment unit 14.

In equirectangular images, the direction from the previous camera position to the landmark LM is known, but the distance is unknown. For this reason, the landmark horizontal plane coordinates q1, q2, q3 cannot be determined using only the landmark equirectangular coordinates Q1', Q2', Q3'. Therefore, before the drag operation, the previous camera/landmark tentative horizontal distance, which is a provisional fixed value set in advance, is used, and during the drag operation, the previous camera/landmark horizontal distance input from the landmark distance adjustment unit 14 is used to determine the landmark horizontal plane coordinates q1, q2, q3.

Specifically, before the drag operation, the landmark horizontal plane coordinate conversion unit 21 converts the landmark equirectangular coordinates Q1', Q2', Q3' into the landmark horizontal plane coordinates q1, q2, q3 using the previous camera/landmark tentative horizontal distance that is preset. Also, during the drag operation, the landmark horizontal plane coordinate conversion unit 21 converts the landmark equirectangular coordinates Q1', Q2', Q3' into the landmark horizontal plane coordinates q1, q2, q3 using the camera/landmark horizontal distance at the time of the drag operation.

In the example before the drag operation in Figure 8 (1), the horizontal map shows the initial landmark horizontal coordinates q1, q2, and q3 when the past camera/landmark tentative horizontal distances of landmarks LM-1, LM-2, and LM-3 are each set to 2.0 m. The past camera horizontal coordinate O is set in advance and its coordinates are (0.0, 0.0).

In addition, in the example after the drag operation in Figure 8 (2), the horizontal plane map shows the landmark horizontal coordinates q1, q2, and q3 of landmarks LM-1, LM-2, and LM-3 at the past camera/landmark horizontal distance when the drag operation is completed.

This horizontal map after the drag operation is a map after the user has dragged the landmarks LM-1, LM-2, and LM-3 on the horizontal map before the drag operation, and the landmark horizontal distances (P-q1, P-q2, P-q3) calculated by the landmark horizontal distance calculation unit 23 (described later) have been adjusted to match the distance values measured using the laser range finder 6. Note that the drag operation is limited to positions on a straight line (passing through the points) between the landmarks LM-1, LM-2, and LM-3 and the point on the past camera horizontal coordinate O, as shown in the horizontal map before the drag operation in Figure 8 (1).

The current camera horizontal plane coordinate estimation unit 22 inputs the landmark equirectangular coordinates Q1, Q2, and Q3 from the landmark equirectangular coordinate transformation unit 20, and inputs the landmark horizontal plane coordinates q1, q2, and q3 from the landmark horizontal plane coordinate transformation unit 21. Then, the current camera horizontal plane coordinate estimation unit 22 calculates the angle between the landmark horizontal plane coordinates q1, q2, and q3 based on the current camera horizontal plane coordinate P, based on the coordinate values of the longitude components of the landmark equirectangular coordinates Q1, Q2, and Q3.

The current camera horizontal plane coordinate estimation unit 22 estimates the current camera horizontal plane coordinate P based on the landmark horizontal plane coordinates q1, q2, q3, the angle, and the previously set previous camera horizontal plane coordinate O. The current camera horizontal plane coordinate estimation unit 22 outputs the current camera horizontal plane coordinate P to the landmark horizontal distance calculation unit 23, the horizontal position adjustment information processing unit 24, and the map display unit 17 (step S704).

The horizontal plane map before the drag operation in FIG. 8 (1) shows the current camera horizontal plane coordinate P (0.1, 0.4) estimated based on the landmark horizontal plane coordinates q1, q2, q3 before adjustment and the previous camera horizontal plane coordinate O (0.0, 0.0), etc. When the user drags the landmarks LM-1, LM-2, LM-3, the landmark horizontal plane coordinates q1, q2, q3 change, and the current camera horizontal plane coordinate P also changes. In addition, this horizontal plane map shows the angle of 263° between the landmark horizontal plane coordinates q1, q2 and the angle of 41° between the landmark horizontal plane coordinates q2, q3 based on the current camera horizontal plane coordinate P.

The horizontal plane map after the drag operation in FIG. 8(2) shows the landmark horizontal plane coordinates q1, q2, q3 at the time the drag operation is completed and the current camera horizontal plane coordinate P (0.1, 0.2) estimated based on the previous camera horizontal plane coordinate O (0.0, 0.0), etc. This horizontal plane map also shows the angle of 263° between the landmark horizontal plane coordinates q1, q2 and the angle of 41° between the landmark horizontal plane coordinates q2, q3 based on the current camera horizontal plane coordinate P.

Figure 9 is a diagram explaining the process of estimating the current camera horizontal plane coordinate P by the current camera horizontal plane coordinate estimation unit 22 (step S704). A horizontal plane map is shown on the left side of Figure 9.

It is assumed that the past camera horizontal plane coordinate O ( _xO , _yO ) is known, and that the landmark horizontal plane coordinates q1 ( _xq1 , _yq1 ) and landmark horizontal plane coordinates q2 ( _xq2 , _yq2 ) are known. In addition, although not shown in the horizontal map of FIG. 9, it is also assumed that the landmark horizontal plane coordinate q3 ( _xq3 , _yq3 ) is known.

Also known are the angle θ _O1,2 between the point at the landmark horizontal plane coordinate q1 and the point at the landmark horizontal plane coordinate q2 based on the past camera horizontal plane coordinate O, the angle θ _{O2,3 (not shown) between the point at the landmark horizontal plane coordinate q2 and the point at the landmark horizontal plane coordinate q3 based on the past camera horizontal plane coordinate O, the angle θ P1,2 between} the point at the landmark horizontal plane coordinate q1 and the point at the landmark horizontal plane coordinate q2 based on the current camera horizontal plane coordinate P, and the angle θ _P2,3 (not shown) between the point at the landmark horizontal plane coordinate q2 and the point at the landmark horizontal plane coordinate q3 based on the current camera horizontal plane coordinate P. The current camera horizontal plane coordinate P to be estimated is (x _p , y _p ).

The angles θ _O1,2 and θ _O2,3 are calculated based on the coordinate values of the longitude components of the landmark equirectangular coordinates Q1', Q2', and Q3', and the angles θ _P1,2 and θ _P2,3 are calculated based on the coordinate values of the longitude components of the landmark equirectangular coordinates Q1, Q2, and Q3. For example, the angle θ _P1,2 corresponds to the difference between the coordinate value of the longitude component of the landmark equirectangular coordinate Q1 and the coordinate value of the longitude component of the landmark equirectangular coordinate Q2.

If a circle passing through the landmark horizontal plane coordinates q1, q2 and the current camera horizontal plane coordinate P is denoted by β1, the equation of circle β1 is as follows: The coordinates of the center of circle β1 are denoted by _EO1,2 ( _xEO1,2 , _yEO1,2 ).
[Equation 1]
(x _p -x _EO1,2 ) ² + (y _p -y _EO1,2 ) ² = r _1,2 ²
(x _q1 - _{x EO1,2} ) ² + (y _q1 - y _EO1,2 ) ² = r _1,2 ² ...(1)
(x _q2 - x _EO1,2 ) ² + (y _q2 - y _EO1,2 ) ² = r _1,2 ²

Furthermore, the sine law gives us the following equation:
[Equation 2]
[√{(x _q2 - x _q1 ) ² + (y _q2 - y _q1 ) ² }]/sinθ _P1,2 =2r _1,2 ...(2)

In the above formulas (1) and (2), _xq1 , _yq1 , _xq2 , _yq2 , and _θP1,2 are known, the number of variables is 5 ( _xp , _yp , _r1,2 , _xEO1,2 , _yEO1,2 ), and the number of formulas is 4. Therefore, the relational expression of the coordinate values _xp and _yp of the current camera horizontal plane coordinate P to be estimated is obtained. Specifically, the current camera horizontal plane coordinate P ( _xp , _yp ) is on a circular arc between the point of the landmark horizontal plane coordinate q1 ( _xq1 , _yq1 ) and the point of the landmark horizontal plane coordinate q2 ( _xq2 , _yq2 ). The angle _θP1,2 takes the same value as long as the current camera horizontal plane coordinate P ( _xp , _yp ) is on this circular arc.

Although not shown, if a circle passing through each point of the landmark horizontal plane coordinates q2, q3 and the point of the current camera horizontal plane coordinate P is denoted as β2, an equation corresponding to the above equations (1) and (2) can be obtained. Then, a relational equation of the coordinate values _xp , _yp of the current camera horizontal plane coordinate P of the estimation target is obtained, and the current camera horizontal plane coordinate P ( _xp , _yp ) is on the arc between the point of the landmark horizontal plane coordinate q2 ( _xq2 , _yq2 ) and the point of the landmark horizontal plane coordinate q3 ( _xq3 , _yq3 ).

Then, by solving two relational expressions of the coordinate values xp, _yp of the current camera horizontal plane coordinate _P to be estimated, the current camera horizontal plane coordinate P ( _xp , _yp ) can be identified as the intersection of the two arcs. Therefore, the current camera horizontal plane coordinate P can be estimated based on the landmark horizontal plane coordinates q1, q2, q3, the angles _θP1,2 , _θP2,3 , and the previously set previous camera horizontal plane coordinate O.

Returning to Figures 6, 7 and 8, the landmark horizontal distance calculation unit 23 inputs the landmark horizontal coordinates q1, q2, q3 from the landmark horizontal coordinate conversion unit 21, and also inputs the current camera horizontal coordinate P from the current camera horizontal coordinate estimation unit 22. The landmark horizontal distance calculation unit 23 then calculates the distances between the point at the current camera horizontal coordinate P and the points at the landmark horizontal coordinates q1, q2, q3 as the landmark horizontal distances (P-q1, P-q2, P-q3), respectively. The landmark horizontal distance calculation unit 23 outputs the landmark horizontal distances (P-q1, P-q2, P-q3) to the current camera vertical position estimation unit 16 and the map display unit 17 (step S705).

In the horizontal map before the drag operation in FIG. 8 (1), the previous camera/landmark horizontal distances (O-q1, O-q2, O-q3) are 2.00 m, 2.00 m, and 2.00 m, respectively. In addition, the landmark horizontal distances (P-q1, P-q2, P-q3) are 1.65 m, 2.16 m, and 1.89 m, respectively. Due to the drag operation described above, the landmark horizontal distances (P-q1, P-q2, P-q3) approach the distance values measured using the laser range finder 6. Then, in the horizontal map after the drag operation in FIG. 8 (2), the previous camera/landmark horizontal distances (O-q1, O-q2, O-q3) are 2.21 m, 1.54 m, and 1.52 m, respectively. Additionally, the landmark horizontal distances (P-q1, P-q2, P-q3) are 1.98 m, 1.65 m, and 1.46 m, respectively.

The horizontal position adjustment information processing unit 24 inputs the current camera horizontal plane coordinate P from the current camera horizontal plane coordinate estimation unit 22. Then, based on the current camera horizontal plane coordinate P and the previously set previous camera horizontal plane coordinate O, the horizontal position adjustment information processing unit 24 calculates the direction of the point of the previous camera horizontal plane coordinate O as seen from the point of the current camera horizontal plane coordinate P, and calculates the distance between the point of the current camera horizontal plane coordinate P and the point of the previous camera horizontal plane coordinate O. The horizontal position adjustment information processing unit 24 outputs horizontal position adjustment information consisting of the direction and distance to the map display unit 17 (step S706).

As a result, the landmark horizontal coordinates q1, q2, q3, the current camera horizontal coordinate P, the landmark horizontal distance (P-q1, P-q2, P-q3), horizontal position adjustment information, etc. are generated, and a horizontal map including this information is displayed. Then, the user moves the current camera 4 horizontally according to the horizontal position adjustment information after the drag operation, as shown in step S505 of FIG. 5.

In addition, in the horizontal plane map before the drag operation in FIG. 8(1), the heights h1, h2, h3 estimated by the current camera vertical position estimation unit 16, which will be described later, are also displayed as UP: -0.07m, UP: 0.37m, and UP: 0.15m. Also, in the horizontal plane map after the drag operation in FIG. 8(2), the heights h1, h2, h3 estimated by the current camera vertical position estimation unit 16, which will be described later, are also displayed as UP: -0.08m, UP: 0.29m, and UP: 0.11m.

(Current camera vertical position estimation unit 16)
Next, a detailed description will be given of current camera vertical position estimating unit 16 shown in Fig. 2. Fig. 10 is a block diagram showing an example of the configuration of current camera vertical position estimating unit 16, and Fig. 11 is a flowchart showing an example of the processing performed by current camera vertical position estimating unit 16.

This current camera vertical position estimation unit 16 includes a landmark equirectangular coordinate conversion unit (landmark image coordinate conversion unit) 30, an angle processing unit 31, a height calculation unit 32, and a vertical position adjustment information processing unit 33.

The current camera vertical position estimation unit 16 inputs the three current landmark cursor positions and the three past landmark cursor positions from the landmark designation unit 13, and also inputs the landmark horizontal distances (P-q1, P-q2, P-q3) from the current camera horizontal position estimation unit 15 (step S1101).

The landmark equirectangular coordinate conversion unit 30 converts the three current landmark cursor positions into landmark equirectangular coordinates Q1, Q2, and Q3, respectively, and converts the three past landmark cursor positions into landmark equirectangular coordinates Q1', Q2', and Q3', respectively (step S1102). The landmark equirectangular coordinate conversion unit 30 then outputs the landmark equirectangular coordinates Q1, Q2, Q3, Q1', Q2', and Q3' to the angle processing unit 31.

The angle processing unit 31 inputs the landmark equirectangular coordinates Q1, Q2, Q3, Q1', Q2', and Q3' from the landmark equirectangular coordinate conversion unit 30.

The angle processing unit 31 determines angles α1, α2, α3 between the horizontal direction and the positions corresponding to the landmark horizontal coordinates q1, q2, q3 projected on the vertical plane from the coordinate values of the latitude components of the landmark equirectangular coordinates Q1, Q2, Q3 of the current image A, with the current camera position projected on the vertical plane of the three-dimensional space as the reference. The angle processing unit 31 also determines angles α1', α2', α3' between the horizontal direction and the positions corresponding to the landmark horizontal coordinates q1, q2, q3 projected on the vertical plane from the coordinate values of the latitude components of the landmark equirectangular coordinates Q1', Q2', Q3' of the past image B, with the past camera position projected on the vertical plane of the three-dimensional space as the reference (step S1103).

The angle processing unit 31 outputs the angles α1, α2, α3, α1', α2', and α3' to the height calculation unit 32.

Figure 12 is a diagram explaining the angle calculation process (step S1103) by the angle processing unit 31. As shown in Figure 12 (1), in an equirectangular image that represents three-dimensional space with the longitude and latitude components of a sphere, the latitude component α is in the range of -π/2 to +π/2. In other words, as shown in Figure 12 (2), the latitude component α is the angle between the position of landmark LM-1 projected onto the vertical plane and the horizontal direction, when the current camera position or the previous camera position projected onto the vertical plane of three-dimensional space is used as a reference.

Therefore, the angle processing unit 31 can set the coordinate values of the latitude components of the landmark equirectangular coordinates Q1, Q2, and Q3 in the current image A as angles α1, α2, and α3, respectively. Also, the angle processing unit 31 can set the coordinate values of the latitude components of the landmark equirectangular coordinates Q1', Q2', and Q3' in the past image B as angles α1', α2', and α3', respectively.

Returning to FIG. 10 and FIG. 11, the height calculation unit 32 inputs angles α1, α2, α3, α1', α2', α3' from the angle processing unit 31. The height calculation unit 32 calculates heights h1, h2, h3, which are the distances between the previous camera position projected on the vertical plane and the current camera position, based on angles α1, α2, α3, α1', α2', α3' and the landmark horizontal distances (P-q1, P-q2, P-q3) input from the current camera horizontal position estimation unit 15 (step S1104). The height calculation unit 32 then outputs the heights h1, h2, h3 to the vertical position adjustment information processing unit 33. These heights h1, h2, h3 mean the heights to which the current camera 4 should be moved in the vertical direction.

Specifically, referring to FIG. 12 (2), the landmark horizontal distances (P-q1, P-q2, P-q3) correspond to the horizontal distances d1, d2, d3 from the previous and current camera positions to the landmarks LM-1, LM-2, LM-3 in the vertical plane. The current camera position here is the position when the user completes the horizontal movement operation of the current camera 4 (step S505 in FIG. 5), i.e., the same position in the horizontal plane as the previous camera position, and the landmark horizontal distances (P-q1, P-q2, P-q3) are the distances when the user completes the horizontal movement operation of the current camera 4. For this reason, the distances d1, d2, d3 are the same values for the previous and current camera positions.

The height calculation unit 32 calculates heights h1, h2, and h3 using the landmark horizontal distance (P-q1, P-q2, P-q3) as distances d1, d2, and d3 according to the following formula.
[Equation 3]
h1=d1(tan(α1)-tan(α1'))
h2=d2(tan(α2)-tan(α2'))...(3)
h3=d3(tan(α3)-tan(α3'))

As shown in FIG. 12 (2), the height h1, which is the distance between the previous camera position and the current camera position projected onto a vertical plane, can be calculated from the distance d1, which is the landmark horizontal distance (P-q1), the tangent of the angle α1' of the previous camera position, and the tangent of the angle α1 of the current camera position.

The vertical position adjustment information processing unit 33 inputs the heights h1, h2, and h3 from the height calculation unit 32, and determines whether the orientations indicated by the heights h1, h2, and h3 are the same (step S1105). If the signs of all of the heights h1, h2, and h3 are positive or negative, it is determined that the orientations indicated by the heights h1, h2, and h3 are the same.

If the vertical position adjustment information processing unit 33 determines in step S1105 that the orientations are the same (step S1105: Y), it identifies the minimum value among the absolute values of the heights h1, h2, and h3, and sets the height corresponding to the minimum value as the vertical position adjustment information. Then, the vertical position adjustment information processing unit 33 outputs the vertical position adjustment information to the map display unit 17 (step S1106). As a result, the vertical position adjustment information is displayed at the top of the horizontal map, as shown in the example of the display screen in Figure 3.

In general, when the heights h1, h2, and h3 are the same value, the camera 4 is currently kept horizontal, and when the heights h1, h2, and h3 are different, the camera 4 is currently not kept horizontal. Therefore, when the heights h1, h2, and h3 are all different values with the same sign, highly reliable vertical position adjustment information can be obtained by setting the minimum value of each absolute value as the vertical position adjustment information.

On the other hand, if the vertical position adjustment information processing unit 33 determines in step S1105 that the orientations are not the same (step S1105: N), it does not set the vertical position adjustment information. In this case, in the example display screen of FIG. 3, the vertical position adjustment information is not displayed at the top of the horizontal map.

If the directions indicated by the heights h1, h2, and h3 are not the same, the above-mentioned precondition <a> is not met, i.e., the current level of the camera 4 may not be maintained. Alternatively, the previous camera may not have been kept level when the previous image B was taken. In the former case, the user should use a spirit level or the like to keep the current camera 4 level so that the directions indicated by the heights h1, h2, and h3 are the same.

As described above, according to the camera position adjustment information generating device 1 of the first embodiment, the camera image acquiring unit 11 acquires live video currently captured by the camera 4. Then, the video and image display unit 12 displays the current image A of the live video on the screen, and also displays the past image B read from the past image storage unit 10 on the screen.

The landmark designation unit 13 designates three past landmark cursor positions in past image B and three corresponding current landmark cursor positions in current image A in accordance with a click operation by the user.

The landmark distance adjustment unit 14 calculates the past camera/landmark horizontal distance between the cursor position, which is limited to the straight line (passing through the points) between the points of the initial landmark horizontal plane coordinates q1, q2, and q3 and the point of the past camera horizontal plane coordinate O, and the point of the past camera horizontal plane coordinate O, in accordance with the drag operation by the user.

The current camera horizontal position estimation unit 15 calculates the landmark horizontal plane coordinates q1, q2, q3 at the initial or drag operation based on the landmark equirectangular coordinates Q1', Q2', Q3' corresponding to the three past landmark cursor positions and the past camera/landmark tentative horizontal distance or the past camera/landmark horizontal distance.

The current camera horizontal position estimation unit 15 estimates the current camera horizontal plane coordinate P based on the landmark horizontal plane coordinates q1, q2, q3, etc., and calculates the landmark horizontal distance (P-q1, P-q2, P-q3) and horizontal position adjustment information.

The current camera vertical position estimation unit 16 uses the landmark equirectangular coordinates Q1', Q2', Q3' corresponding to the three past landmark cursor positions to determine angles α1', α2', α3' of the landmark horizontal plane coordinates q1, q2, q3 in a vertical plane based on the past camera position. Similarly, the current camera vertical position estimation unit 16 uses the landmark equirectangular coordinates Q1, Q2, Q3 corresponding to the three current landmark cursor positions to determine angles α1, α2, α3 based on the current camera position.

The current camera vertical position estimation unit 16 calculates the heights h1, h2, and h3 between the previous camera position and the current camera position projected onto the vertical plane based on the landmark horizontal distances (P-q1, P-q2, P-q3) and angles α1, α2, α3, α1', α2', and α3', and calculates vertical position adjustment information.

The map display unit 17 displays the landmark horizontal coordinates q1, q2, q3, horizontal position adjustment information, vertical position adjustment information, etc. on the screen as a horizontal map.

The user then performs a drag operation so that the landmark horizontal distance (P-q1, P-q2, P-q3) matches the horizontal distance measurement value between the current camera position measured using the laser range finder 6 and the positions in three-dimensional space corresponding to the three landmarks LM. After the drag operation, the user then refers to the horizontal position adjustment information determined by the current camera horizontal position estimation unit 15 and moves the current camera 4 horizontally. After that, after moving the current camera 4 horizontally, the user refers to the vertical position adjustment information determined by the current camera vertical position estimation unit 16 and moves the current camera 4 vertically.

This makes it possible to generate horizontal position adjustment information and vertical position adjustment information that are used when adjusting the current camera position to match the spatial positional relationship of current image A to previous image B. Then, by using this horizontal position adjustment information and vertical position adjustment information, the user can easily and quickly adjust the current camera position to the previous camera position.

Example 2
Next, a description will be given of a camera position adjustment information generating device 2 of Example 2. The camera position adjustment information generating device 2 of Example 2 is a device having a tracking function of a landmark image and a current landmark cursor position in a current image A of a live video.

In the camera position adjustment information generating device 1 of the first embodiment, angles θ _P1,2 and θ _P2,3 are calculated based on the current landmark cursor position specified by a click operation, and the current camera horizontal plane coordinate P is calculated based on the landmark horizontal plane coordinates q1, q2, q3 and angles θ _P1,2 and θ _P2,3 , etc. In this case, since the current landmark cursor position is not changed, the angles θ _P1,2 and θ _P2,3 used in determining the current camera horizontal plane coordinate P are fixed by the first calculation.

However, although the current camera horizontal plane coordinate P is calculated using the fixed angles θ _P1,2 and θ _P2,3 in the first calculation, and the horizontal position adjustment information is obtained, it is difficult to accurately adjust the current camera position. This is because the user moves the current camera 4 in the air, and it is difficult to adjust the current camera position to centimeters. The same is true for the vertical position adjustment information. The longer the distance indicated by the horizontal position adjustment information and the higher the height indicated by the vertical position adjustment information, i.e., the longer the distance the current camera 4 is moved, the larger the adjustment error will be.

Therefore, in the camera position adjustment information generating device 2 of the second embodiment, a tracking function is realized in which the image surrounding the specified current landmark cursor position is cut out as a landmark image from the current image A, and the current landmark cursor position and the landmark image are tracked in real time in the current image A as the current camera position changes. With this tracking function, the current camera horizontal plane coordinate P is estimated in real time as the current camera position changes, even if the current camera position is moving, and the horizontal position adjustment information and vertical position adjustment information are updated in real time.

Figure 13 is a block diagram showing an example of the configuration of a camera position adjustment information generating device 2 of Example 2. This camera position adjustment information generating device 2 includes a past image storage unit 10, a camera video acquisition unit 11, a video and image display unit 12, a landmark designation unit 13, a landmark distance adjustment unit 14, a current camera horizontal position estimation unit 15, a current camera vertical position estimation unit 16, and a map display unit 17, as well as a tracking processing unit 18.

Comparing the camera position adjustment information generation device 1 of Example 1 shown in FIG. 2 with the camera position adjustment information generation device 2 of Example 2 shown in FIG. 13, both camera position adjustment information generation devices 1 and 2 have in common the fact that they include a past image storage unit 10, a camera video acquisition unit 11, a video and image display unit 12, a landmark designation unit 13, a landmark distance adjustment unit 14, a current camera horizontal position estimation unit 15, a current camera vertical position estimation unit 16, and a map display unit 17. On the other hand, the camera position adjustment information generation device 2 differs from the camera position adjustment information generation device 1 in that, in addition to the components of the camera position adjustment information generation device 1, it also includes a tracking processing unit 18. In FIG. 13, parts that are common to FIG. 2 are given the same reference numerals as in FIG. 2, and detailed explanations thereof will be omitted.

The camera image acquisition unit 11 outputs a current image A of the live image to the image and video display unit 12 and the tracking processing unit 18, and the landmark designation unit 13 outputs the three current landmark cursor positions and the three past landmark cursor positions to the tracking processing unit 18.

The tracking processing unit 18 inputs current image A of the live video from the camera image acquisition unit 11, as well as three current landmark cursor positions and three past landmark cursor positions from the landmark designation unit 13. The tracking processing unit 18 then cuts out each landmark image in current image A by matching the surrounding image of the three current landmark cursor positions with a template image. The tracking processing unit 18 tracks the three current landmark cursor positions and the three landmark images in current image A as the current camera position changes.

The tracking processing unit 18 outputs the three current landmark cursor positions after tracking and the three input past landmark cursor positions to the current camera horizontal position estimation unit 15 and the current camera vertical position estimation unit 16. The tracking processing unit 18 also outputs the three landmark images after tracking to the video and image display unit 12. This causes the landmark images to be displayed on the screen.

The tracking processing unit 18 also reads out past image B from the past image storage unit 10, cuts out the surrounding images of the three past landmark cursor positions from past image B as landmark images, and outputs the three landmark images of past image B to the video and image display unit 12.

As a result, as shown in Figure 3, the landmark images LM-1A, LM-1B, and LM-1A of the current image A and the landmark images LM-1B, LM-2B, and LM-3B of the past image B are displayed on the screen as thumbnail images along with the current image A and past image B of the live video.

In the area where the landmark images in FIG. 3 are displayed, the landmark images LM-2A, LM-3A, and LM-1A when the current landmark cursor position is specified by a user click operation are displayed on the left side of the top row, with three images as a set. The tracked landmark images LM-2A, LM-3A, and LM-1A are displayed in real time on the right side of the top row. The landmark images LM-2B, LM-3B, and LM-1B when the past landmark cursor position was specified by a user click operation are displayed on the bottom row. The numerical values (0.98, 0.97, 0.83) displayed near the landmark images indicate the template matching accuracy between the landmark images LM-2A, LM-3A, and LM-1A and the landmark images LM-2B, LM-3B, and LM-1B, respectively.

(Tracking Processing Unit 18)
Next, a detailed description will be given of the tracking processing unit 18 shown in Fig. 13. Fig. 14 is a block diagram showing an example of the configuration of the tracking processing unit 18, and Fig. 15 is a flowchart showing an example of processing by the tracking processing unit 18.

The tracking processing unit 18 includes an image cropping unit 40, a landmark image storage unit 41, a landmark image tracking unit 42, and a cursor position tracking unit 43.

The image cropping unit 40 inputs the three current landmark cursor positions and the three past landmark cursor positions from the landmark designation unit 13 (step S1501), and inputs the current image A of the live video from the camera image acquisition unit 11 (step S1502). The processes of steps S1503 to S1507 shown below are performed for each of the three current landmark cursor positions.

For each of the three current landmark cursor positions, the image cropping unit 40 performs a matching process (template matching) between the surrounding image of the current landmark cursor position and the template image set in step S1504 described below (step S1503). Then, the image cropping unit 40 crops out the matched surrounding image as a landmark image, stores it in the landmark image storage unit 41, and sets the landmark image as the template image (step S1504).

This updates the template image and prevents tracking deviations of the current landmark cursor position and landmark image being tracked.

Here, the image cropping unit 40 does not perform the matching process of step S1503 during initial processing. In step S1503, the image cropping unit 40 crops out a predetermined peripheral image of the current landmark cursor position input in step S1501 as a landmark image, and stores it in the landmark image storage unit 41. The image cropping unit 40 then sets the landmark image as a template image.

The image cropping unit 40 sets a predetermined position (e.g., the center position) within the cropped landmark image as the new current landmark cursor position, and stores the new current landmark cursor position in the landmark image storage unit 41 (step S1505).

The landmark image tracking unit 42 updates the landmark image by reading the landmark image from the landmark image storage unit 41, and outputs it to the video and image display unit 12 as the tracked landmark image (step S1506). As a result, the tracked landmark image is displayed on the screen.

The cursor position tracking unit 43 also updates the current landmark cursor position by reading out the current landmark cursor position from the landmark image storage unit 41, and outputs it to the current camera horizontal position estimation unit 15 and the current camera vertical position estimation unit 16 as the current landmark cursor position after tracking (step S1507). As a result, the current camera horizontal position estimation unit 15 estimates the current camera horizontal plane coordinate P using the current landmark cursor position after tracking, and determines horizontal position adjustment information, etc. Furthermore, the current camera vertical position estimation unit 16 determines vertical position adjustment information using the current landmark cursor position after tracking.

The image cropping unit 40 also outputs the past landmark cursor position to the cursor position tracking unit 43, and the cursor position tracking unit 43 outputs the past landmark cursor position to the current camera horizontal position estimation unit 15 and the current camera vertical position estimation unit 16 (step S1508).

The tracking processing unit 18 determines whether the processing of the tracking processing unit 18 is completed (step S1509). Unless the processing is completed, the tracking processing unit 18 proceeds to step S1502, inputs a new current image A of the live video in the image cropping unit 40, and repeats the processing of steps S1503 to S1507.

This outputs the landmark image and current landmark cursor position tracked as the current camera position changes.

In addition, in the matching process of step S1503 by the image cropping unit 40, it is effective to limit the search range of the surrounding images to a specified range centered on the current landmark cursor position of the previous frame of the current image A, taking into account the calculation time. However, if the search range is narrowed, there is a possibility that tracking will not be possible if the current camera 4 moves a long distance. For this reason, it is desirable to set the search range while taking into account the trade-off between calculation resources and convenience.

As described above, according to the camera position adjustment information generating device 2 of the second embodiment, the tracking processing unit 18, in response to a change in the current camera position, performs a matching process between the peripheral image of the current landmark cursor position in the current image A and a template image, which is a previously cut-out landmark image, and tracks the landmark image by cutting out the matched peripheral image as the landmark image. Then, the tracking processing unit 18 sets the landmark image as the template image.

The tracking processing unit 18 also tracks the current landmark cursor position by setting the current landmark cursor position from the cut-out landmark image.

As a result, the current camera horizontal position estimation unit 15 in the subsequent stage estimates the current camera horizontal plane coordinate P using the current landmark cursor position after tracking, and horizontal position adjustment information and the like are obtained. Also, the current camera vertical position estimation unit 16 in the subsequent stage uses the current landmark cursor position after tracking to obtain vertical position adjustment information.

Therefore, the same effect as in Example 1 is achieved, and the tracking function allows the current camera horizontal position estimation unit 15 to estimate the current camera horizontal plane coordinate P in real time in response to a change in the current camera position, even if the current camera position is moving. As a result, the horizontal position adjustment information and vertical position adjustment information can be updated in real time.

[Example 3/Snap function]
Next, a description will be given of a camera position adjustment information generating device 3 according to an embodiment 3. The camera position adjustment information generating device 3 according to the embodiment 3 is a device having a cursor position snap function.

In the user operation procedure shown in FIG. 5, the user sets the position of the landmark LM by clicking in step S501. However, if the click position of the landmark LM is shifted, the estimation accuracy of the current camera horizontal plane coordinate P decreases, and as a result, the accuracy of the horizontal position adjustment information and vertical position adjustment information used when adjusting the current camera position decreases.

Therefore, the camera position adjustment information generating device 3 of the third embodiment realizes a snap function that extracts local features of the image near the cursor position and moves the cursor position to a feature point position such as the corner of an object. This snap function frees the user from having to be careful when clicking a landmark LM, and allows the user to adjust the current camera position using horizontal position adjustment information with stable accuracy.

Figure 16 is a block diagram showing an example of the configuration of a camera position adjustment information generating device 3 of Example 3. This camera position adjustment information generating device 3 includes a past image storage unit 10, a camera video acquisition unit 11, a video and image display unit 12, a landmark designation unit 13, a landmark distance adjustment unit 14, a current camera horizontal position estimation unit 15, a current camera vertical position estimation unit 16, and a map display unit 17, as well as a snap processing unit 19.

Comparing the camera position adjustment information generating device 1 of Example 1 shown in Figure 2 with the camera position adjustment information generating device 3 of Example 3 shown in Figure 16, both camera position adjustment information generating devices 1 and 3 have in common the fact that they include a past image storage unit 10, a camera video acquisition unit 11, a video and image display unit 12, a landmark designation unit 13, a landmark distance adjustment unit 14, a current camera horizontal position estimation unit 15, a current camera vertical position estimation unit 16, and a map display unit 17. On the other hand, the camera position adjustment information generating device 3 differs from the camera position adjustment information generating device 1 in that, in addition to the components of the camera position adjustment information generating device 1, it also includes a snap processing unit 19. In Figure 16, parts that are common to Figure 2 are given the same reference numerals as in Figure 2, and detailed explanations thereof will be omitted.

The snap processing unit 19 reads out a past image B from the past image storage unit 10, inputs a current image A of the live video from the camera image acquisition unit 11, and inputs the cursor position from the pointing device 7 according to the user's operation.

The snap processing unit 19 extracts the feature points of the previous image B and the feature points of the current image A, snaps the cursor position to the position of the feature point of the previous image B or the position of the feature point of the current image A, and outputs the cursor position after snapping to the landmark designation unit 13.

(Snap Processing Unit 19)
Next, the snap processor 19 shown in Fig. 16 will be described in detail. Fig. 17 is a block diagram showing an example of the configuration of the snap processor 19, and Fig. 18 is a flowchart showing an example of processing by the snap processor 19.

The snap processing unit 19 includes a feature point extraction unit 50 and a snap unit 51. The snap processing unit 19 reads out a past image B from the past image storage unit 10, inputs a current image A of the live video from the camera image acquisition unit 11, and inputs the cursor position according to the user's operation of the pointing device 7 (step S1801).

The user operation here is an operation to specify the cursor positions of past image B and current image A as the past landmark cursor position and the current landmark cursor position, respectively.

The feature point extraction unit 50 extracts multiple feature point positions from the previous image B, for example by Harris corner detection processing, and extracts multiple feature point positions from the current image A (step S1802), and outputs these feature point positions to the snap unit 51.

By using the Harris corner detection process, corners of the object's edges that have a clear contrast with the background, for example, are extracted as feature point positions. The Harris corner detection process is one example, and other processes may be used.

For details of the Harris corner detection process, see, for example, the following documents:
[Non-Patent Literature]
"Harris corner detection", [online], [searched October 20, 2020], Internet <URL: http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_feature2d/py_features_harris/py_features_harris.html>

The snapping unit 51 inputs the feature point position from the feature point extraction unit 50 and also inputs the cursor position from the pointing device 7, identifies the feature point position closest to the cursor position, and snaps the cursor position to the identified feature point position (step S1803).

Specifically, when the cursor position is within the area of past image B, the snapping unit 51 identifies the feature point position closest to the cursor position among the multiple feature point positions extracted from past image B, and snaps the cursor position to the identified feature point position. When the cursor position is within the area of current image A, the snapping unit 51 identifies the feature point position closest to the cursor position among the multiple feature point positions extracted from current image A, and snaps the cursor position to the identified feature point position.

The snap unit 51 outputs the cursor position after snapping to the landmark designation unit 13 (step S1804).

As a result, the snap processing unit 19 outputs the feature point positions of the past image B and the current image A to the landmark designation unit 13 as the cursor position after snapping.

As described above, according to the camera position adjustment information generating device 3 of the third embodiment, the snap processing unit 19 extracts multiple feature point positions for each of the current image A and the past image B, identifies the feature point position closest to the input cursor position, and snaps the cursor position to the identified feature point position.

As a result, the feature point positions of the current image A and the past image B are input to the subsequent landmark designation unit 13 as the cursor positions after snapping, and the current landmark cursor position and the past landmark cursor position of the feature point position are designated. Therefore, the current landmark cursor position and the past landmark cursor position are more likely to be designated as the same feature point position, and can be accurately associated.

Therefore, the same effect as in Example 1 is achieved, and the snap function frees the user from having to be careful when clicking the landmark LM, and the downstream current camera horizontal position estimation unit 15 or the like can always obtain horizontal position adjustment information with stable accuracy. As a result, the user can accurately adjust the current camera position using highly accurate horizontal position adjustment information, etc.

The camera position adjustment information generating device 3 may further include a tracking processing unit 18 provided in the camera position adjustment information generating device 2 of Example 2 in FIG. 13. This allows the tracking processing unit 18 to extract landmark images that are easy to match with using templates from the current image A. Also, it is possible to obtain characteristic landmark images in the current image A that do not appear frequently in other areas.

The present invention has been described above using Examples 1, 2, and 3, but the present invention is not limited to the above Examples 1, 2, and 3, and various modifications are possible without departing from the technical concept thereof.

For example, in the first, second, and third embodiments, the number of landmarks LM is described as K=3, but the present invention does not limit the number of landmarks LM to K=3, and K may be a predetermined number (integer) equal to or greater than 3. Furthermore, by setting the number of landmarks to K=3, the processing load can be reduced.

As described above, when K=3, the current camera horizontal plane coordinate estimating section 22 of the current camera horizontal position estimating section 15 can calculate the current camera horizontal plane coordinate P using the landmark horizontal plane coordinates q1, q2, q3, angles θ _P1,2 , θ _P2,3 and the previously set previous camera horizontal plane coordinate O. On the other hand, when K≧4, the current camera horizontal plane coordinate estimating section 22 calculates a plurality of current camera horizontal plane coordinates P using the landmark horizontal plane coordinates q1, q2, q3, q4, ..., angles θ _P1,2 , θ _P2,3 , θ _P3,4 , ... and the previously set previous camera horizontal plane coordinate O, and is unable to identify a single current camera horizontal plane coordinate P. In this case, the current camera horizontal plane coordinate estimating section 22 identifies a single current camera horizontal plane coordinate P from a plurality of current camera horizontal plane coordinates P using the least squares method or the like.

In addition, in the first, second, and third embodiments, the current image A and the previous image B of the live image are described as equirectangular images, but the present invention does not limit the image of the subject to be captured to an equirectangular image. The current image A and the previous image B of the live image may be in any format that allows the coordinate values of the subject to be expressed by the longitude and latitude components of a sphere.

In addition, in Examples 1, 2, and 3, the user manually performs a drag operation so that the landmark horizontal distances (P-q1, P-q2, P-q3) match the measured distances. Then, the landmark distance adjustment unit 14 of the camera position adjustment information generation device 1, 2, and 3 calculates the past camera/landmark horizontal distance at that time, and the current camera horizontal position estimation unit 15 uses the past camera/landmark horizontal distance at that time to determine the landmark horizontal plane coordinates q1, q2, and q3, estimates the current camera horizontal plane coordinate P, and determines the horizontal position adjustment information. Then, the user moves the current camera 4 horizontally according to the horizontal position adjustment information after the drag operation.

Here, instead of a manual drag operation by the user, the landmark distance adjustment unit 14 may perform the drag operation automatically.

In this case, the landmark distance adjustment unit 14 inputs the measured distance values corresponding to the three landmarks LM from the laser range finder 6. The landmark distance adjustment unit 14 then moves the cursor position while limiting the cursor position to a position on the straight line (passing through the points) between the points of the initial landmark horizontal coordinates q1, q2, q3 and the point of the previous camera horizontal coordinate O, so that the landmark horizontal distances (P-q1, P-q2, P-q3) match the measured distance values.

The landmark distance adjustment unit 14 calculates the past camera/landmark horizontal distance when the landmark horizontal distance (P-q1, P-q2, P-q3) matches a value within a predetermined range centered on the distance measurement value, and outputs the past camera/landmark horizontal distance to the current camera horizontal position estimation unit 15.

In addition, a normal computer can be used as the hardware configuration of the camera position adjustment information generating devices 1, 2, and 3 according to the embodiments of the present invention. The camera position adjustment information generating devices 1, 2, and 3 are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface, etc.

The functions of the past image storage unit 10, camera video acquisition unit 11, video and image display unit 12, landmark designation unit 13, landmark distance adjustment unit 14, current camera horizontal position estimation unit 15, current camera vertical position estimation unit 16 and map display unit 17 provided in the camera position adjustment information generation device 1 are each realized by having the CPU execute a program that describes these functions.

Furthermore, the functions of the past image storage unit 10, camera video acquisition unit 11, video and image display unit 12, landmark designation unit 13, landmark distance adjustment unit 14, current camera horizontal position estimation unit 15, current camera vertical position estimation unit 16, map display unit 17 and tracking processing unit 18 provided in the camera position adjustment information generation device 2 are each realized by having the CPU execute a program that describes these functions.

Furthermore, the functions of the past image storage unit 10, camera video acquisition unit 11, video and image display unit 12, landmark designation unit 13, landmark distance adjustment unit 14, current camera horizontal position estimation unit 15, current camera vertical position estimation unit 16, map display unit 17 and snap processing unit 19 provided in the camera position adjustment information generation device 3 are each realized by having the CPU execute a program that describes these functions.

These programs are stored in the storage medium and are read and executed by the CPU. In addition, these programs can be distributed by storing them on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network.

1, 2, 3 Camera position adjustment information generating device 4 Camera (current camera)
5 Capture device 6 Laser range finder 7 Pointing device 10 Past image storage unit 11 Camera image acquisition unit 12 Image and image display unit 13 Landmark designation unit 14 Landmark distance adjustment unit 15 Current camera horizontal position estimation unit 16 Current camera vertical position estimation unit 17 Map display unit 18 Tracking processing unit 19 Snap processing units 20, 30 Landmark equirectangular coordinate conversion unit 21 Landmark horizontal plane coordinate conversion unit 22 Current camera horizontal plane coordinate estimation unit 23 Landmark horizontal distance calculation unit 24 Horizontal position adjustment information processing unit 31 Angle processing unit 32 Height calculation unit 33 Vertical position adjustment information processing unit 40 Image extraction unit 41 Landmark image storage unit 42 Landmark image tracking unit 43 Cursor position tracking unit 50 Feature point extraction unit 51 Snap unit 100 Viewer 101 HMD (Head Mounted Display)
102 Boundary 103 Controller AA Current image BB Past image A Current image B Past image O Past camera horizontal plane coordinate P Current camera horizontal plane coordinate qk, q1, q2, q3 Landmark horizontal plane coordinate Qk, Q1, Q2, Q3, Qk', Q1', Q2', Q3' Landmark equirectangular coordinate P-qk, P-q1, P-q2, P-q3 Landmark horizontal distance LM, LM-k, LM-1, LM-2, LM-3 Landmark LM-kA, LM-1A, LM-2A, LM-3A, LM-kB, LM-1B, LM-2B, LM-3B Landmark image αk, α1, α2, α3, αk', α1', α2', α3' Angle hk, h1, h2, h3 Height

Claims (7)

A camera position adjustment information generating device that generates adjustment information for adjusting a position of a camera that currently captures an image to a position of the camera that captured an image in the past, comprising:
an image expressing coordinate values of a subject to be photographed by longitude and latitude components, the image photographed in the past being a past image, the image photographed currently being a current image, the camera that photographed the past image being a past camera, the camera that photographed the current image being a current camera, and the positions of the past camera and the current camera in three-dimensional space being the past camera position and the current camera position, respectively;
When the three-dimensional space positions corresponding to the past camera position, the current camera position, and the positions of landmarks at K points (k=1, 2, ..., K, K is an integer equal to or greater than 3) set in common to the past image and the current image are projected onto a predetermined horizontal plane, the past camera position is defined as a previously set past camera horizontal plane coordinate O, the current camera position is defined as a current camera horizontal plane coordinate P, the position of the landmark at the K point is defined as a landmark horizontal plane coordinate qk (=q1, q2, ..., qK), and the distance between the current camera horizontal plane coordinate P and the landmark horizontal plane coordinate qk is defined as a landmark horizontal distance (P-qk) (=P-q1, P-q2, ..., P-qK),
an acquisition unit that acquires the current image captured by the current camera;
a landmark designation unit that, in accordance with a user's operation, designates the landmarks in the past image as past landmarks, thereby outputting each of the cursor positions of the past image at the K points as past landmark cursor positions of the K points, and designates the landmarks in the current image as current landmarks, thereby outputting each of the cursor positions of the current image at the K points as current landmark cursor positions of the K points;
a landmark distance adjustment unit which adjusts the landmark horizontal distance (P-qk) to coincide with a horizontal distance between the current camera position measured by a predetermined sensor and a position in a three-dimensional space corresponding to the current landmark, according to an operation performed by the user on each position in the landmark horizontal coordinate qk, thereby calculating a distance between the past camera horizontal coordinate O and the landmark horizontal coordinate qk, and outputs the distance as a past camera/landmark horizontal distance of point K;
The image coordinates corresponding to the past landmark cursor position of the K point and the current landmark cursor position of the K point output by the landmark designation unit are respectively defined as past landmark image coordinates Qk'(=Q1',Q2', ..., QK') and current landmark image coordinates Qk (=Q1, Q2, ..., QK),
The landmark horizontal plane coordinate qk is calculated based on the past landmark image coordinate Qk′ and the past camera/landmark horizontal distance of the K point output by the landmark distance adjustment unit;
Calculate an angle between the two landmark horizontal plane coordinates qk based on the current camera horizontal plane coordinate P based on the coordinate value of the longitude component of the current landmark image coordinate Qk, and determine the current camera horizontal plane coordinate P based on the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk, and the angle;
Calculate the landmark horizontal distance (P-qk) based on the current camera horizontal plane coordinate P and the landmark horizontal plane coordinate qk,
a current camera horizontal position estimating unit that determines horizontal position adjustment information including a horizontal direction and a distance for aligning the current camera position with the previous camera position, based on the previous camera horizontal plane coordinate O and the current camera horizontal plane coordinate P;
based on the past landmark image coordinate Qk' and the current landmark image coordinate Qk, and using the past camera position and the current camera position projected on a vertical plane perpendicular to the horizontal plane as references, calculate angles αk'(α1',α2', ..., αK') and angles αk (α1, α2, ..., αK) between the position corresponding to the landmark horizontal plane coordinate qk projected on the vertical plane and the horizontal direction, respectively;
a current camera vertical position estimating unit that calculates vertical position adjustment information consisting of a vertical height for aligning the current camera position with the past camera position, based on the angles αk′ and αk, and the landmark horizontal distance (P−qk) calculated by the current camera horizontal position estimating unit;
displaying the past image and the current image acquired by the acquisition unit, and displaying a mark of the past landmark cursor position of the K point and a mark of the current landmark cursor position of the K point output by the landmark designation unit on the past image and the current image, respectively;
a display unit that displays the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk calculated by the current camera horizontal position estimation unit, the current camera horizontal plane coordinate P, the landmark horizontal distance (P-qk), the horizontal position adjustment information, and the vertical position adjustment information calculated by the current camera vertical position estimation unit;
A camera position adjustment information generating device comprising: 2. The camera position adjustment information generating device according to claim 1,
The current camera horizontal position estimation unit
a first landmark image coordinate conversion unit that converts a past landmark cursor position of the K point into the past landmark image coordinate Qk′ and converts a current landmark cursor position of the K point into the current landmark image coordinate Qk;
a landmark horizontal plane coordinate conversion unit that converts the past landmark image coordinate Qk′ converted by the first landmark image coordinate conversion unit into the landmark horizontal plane coordinate qk based on a past camera/landmark tentative horizontal distance of a preset K point between the past camera horizontal plane coordinate O and the landmark horizontal plane coordinate qk, or the past camera/landmark horizontal distance of the K point;
a current camera horizontal plane coordinate estimation unit that calculates the angle between the two landmark horizontal plane coordinates qk based on the current camera horizontal plane coordinate P based on the coordinate value of the longitude component of the current landmark image coordinate Qk converted by the first landmark image coordinate conversion unit, and estimates the current camera horizontal plane coordinate P based on the past camera horizontal plane coordinate O, the landmark horizontal plane coordinate qk, and the angle;
a landmark horizontal distance calculation unit that calculates, as the landmark horizontal distance (P-qk), a distance between the current camera horizontal plane coordinate P estimated by the current camera horizontal plane coordinate estimation unit and the landmark horizontal plane coordinate qk converted by the landmark horizontal plane coordinate conversion unit;
a horizontal position adjustment information processing unit that calculates the horizontal position adjustment information based on the past camera horizontal plane coordinate O and the current camera horizontal plane coordinate P estimated by the current camera horizontal plane coordinate estimation unit;
A camera position adjustment information generating device comprising: 3. The camera position adjustment information generating device according to claim 2,
The current camera horizontal plane coordinate estimation unit
a camera position adjustment information generating device that estimates the current camera horizontal plane coordinate P based on an equation of a circle that passes through the landmark horizontal plane coordinate qk of a combination of predetermined two points out of K points and the current camera horizontal plane coordinate P, a sine law equation using the angle between the landmark horizontal plane coordinate qk of the combination of the two points based on the current camera horizontal plane coordinate P, an equation of a circle that passes through the landmark horizontal plane coordinate qk of a combination of other two points out of K points and the current camera horizontal plane coordinate P, and a sine law equation using the angle between the landmark horizontal plane coordinate qk of the combination of the other two points based on the current camera horizontal plane coordinate P. 4. The camera position adjustment information generating device according to claim 1,
The current camera vertical position estimation unit
a second landmark image coordinate conversion unit that converts a past landmark cursor position of the K point into the past landmark image coordinate Qk′ and converts a current landmark cursor position of the K point into the current landmark image coordinate Qk;
an angle processing unit that calculates the angle α′ based on the coordinate value of a latitude component of the past landmark image coordinate Q′ converted by the second landmark image coordinate conversion unit, and calculates the angle α based on the coordinate value of a latitude component of the current landmark image coordinate Q;
a height calculation unit that calculates the height hk by the formula: h=d(tan(α)-tan(α')), where the angles αk' and αk found by the angle processing unit are angles α' and α, the landmark horizontal distance (P-qk) is distance d, and height hk (h1, h2, ..., hK) indicating the distance between the past camera position and the current camera position projected on the vertical plane is height h; and
a vertical position adjustment information processing unit that, when all of the heights hk calculated by the height calculation unit are positive values or when all of the heights hk are negative values, identifies the height hk with the smallest absolute value and determines the identified hk as the vertical position adjustment information;
A camera position adjustment information generating device comprising: 5. The camera position adjustment information generating device according to claim 1,
a tracking processing unit that tracks the current landmark cursor positions of the K points output by the landmark designation unit in response to a change in the current camera position by the user,
The tracking processing unit is
an image cropping unit that performs a matching process for each of the K points of the current landmark cursor position between a peripheral image of the current landmark cursor position of the current image acquired by the acquisition unit and a predetermined template image, crops out the matched peripheral image as a landmark image, sets the landmark image as a new template image, and sets a predetermined position within the landmark image as a new current landmark cursor position;
a tracking unit that outputs the new current landmark cursor position of the K point set by the image cropping unit as a current landmark cursor position of the K point after tracking,
The current camera horizontal position estimation unit
a current landmark image coordinate Qk being the image coordinate corresponding to the current landmark cursor position of the K point after the tracking output by the tracking unit. 6. The camera position adjustment information generating device according to claim 1,
a snap processing unit that, in accordance with an operation by the user, designates the landmark of the past image as the past landmark, thereby snapping a cursor position of the past image to a feature point position in the past image, and designates the landmark of the current image as the current landmark, thereby snapping a cursor position of the current image to the feature point position in the current image,
The snap processing unit includes:
a feature point extraction unit that extracts a plurality of feature point positions from the past image and the current image;
if the cursor position is within an area of the previous image, identifying a feature point position that is closest to the cursor position among the plurality of feature point positions extracted from the previous image by the feature point extraction unit, snapping the cursor position to the identified feature point position, and outputting the cursor position after snapping to the landmark designation unit as a cursor position of the previous image;
a snapping unit that, when the cursor position exists within an area of the current image, identifies a feature point position that is closest to the cursor position among the plurality of feature point positions extracted from the current image, snaps the cursor position to the identified feature point position, and outputs the cursor position after snapping to the landmark designation unit as a cursor position of the current image;
The landmark designation unit is
A camera position adjustment information generating device characterized in that the cursor position of the past image and the cursor position of the current image are input from the snap unit, each of the cursor positions of the past image at K points is output as a past landmark cursor position of the K point, and each of the cursor positions of the current image at K points is output as a current landmark cursor position of the K point. A program for causing a computer to function as a camera position adjustment information generating device according to any one of claims 1 to 6.

Images