JP7031489B2 - Compensation program, compensation method and compensation device - Google Patents

Description

The present invention relates to a compensation program and the like.

In recent years, a camera using a plurality of circumferential fisheye lenses can shoot 360-degree omnidirectional images, and is considered to be applied to VR (Virtual Reality), AR (Augmented Reality), and the like. Here, since the image taken by the camera of the circumferential fisheye lens contains distortion, the camera parameter is estimated by performing calibration, and the correction for removing the distortion is performed based on the camera parameter. In the following description, a camera using a circumferential fisheye lens is simply referred to as a "camera".

13 and 14 are diagrams for explaining the conventional calibration. In the prior art, a grid pattern 10 having a known grid length is photographed by a plurality of cameras. For example, as shown in FIG. 13, when the grid pattern 10 is photographed by two cameras, the images 11a and 11b are photographed. Images 11a and 11b are collectively referred to as image 11.

In the calibration of a single camera, the grid patterns 10 whose grid lengths are known are arranged at a plurality of positions, and the internal parameters and the grid external parameters are obtained from the plurality of images. presume. Further, for example, the relative position is estimated from the result of photographing the grid pattern in which one camera and the other camera are arranged at the same position.

In the prior art, when calibrating, a grid point is detected from the image 11 and the relationship between the incident angle of the grid point and the image height is estimated. In the circumferential fisheye lens, the grid points are mapped to the image plane on the assumption of a virtual spherical surface. In the example shown in FIG. 14, the grid point 10a is incident on the virtual spherical surface 5 at an incident angle θa and is mapped to the point 20a on the image plane 20. The image height of the incident angle θa is the image height 21a.

Here, in the model of the virtual sphere described with reference to FIG. 14, it is assumed that the incident angle does not depend on the distance of the object and the height of the center of the optical axis is always the same. FIG. 15 is a diagram for explaining a conventional model. As shown in FIG. 15, the grid points 10b and 10c are incident at the incident angle θb, pass through the center 5a of the optical axis, and are mapped to the image height corresponding to the incident angle θb. The grid points 10d and 10e are incident at an incident angle θd, pass through the center 5a of the optical axis, and are mapped to an image height corresponding to the incident angle θd.

In the prior art, the internal and external parameters of the camera are estimated based on the relationship between the three-dimensional coordinates of each lattice point and the image height when each lattice point is incident on the virtual spherical model. In the calibration for the first camera and the second camera, the internal parameters of the first camera, the internal parameters of the second camera, and the relative external parameters of the first camera and the second camera are estimated as the camera parameters. ..

Next, a conventional technique for synthesizing images taken by a plurality of cameras using estimated camera parameters will be described. FIG. 16 is a diagram for explaining a conventional image composition process. In the example shown in FIG. 16, it is assumed that the cameras 15a and 15b are used to photograph the objects 16a, 16b and 16c. The cameras 15a and 15b are cameras using a circumferential fisheye lens.

The prior art develops a first equirectangular projection from an image taken by the camera 15a based on the internal parameters of the camera 15a and the relative external parameters. The prior art develops a second equirectangular projection from an image taken by the camera 15b based on the internal parameters of the camera 15b and the relative external parameters. In the prior art, an omnidirectional image is generated by synthesizing a first equirectangular projection and a second equirectangular projection. In the prior art, parallax correction may be performed for objects having a short distance, such as objects 16a and 16b, in consideration of the influence of parallax. It is assumed that there is no synthetic deviation for the object 16c, which is sufficiently far from the objects 16a and 16b.

Japanese Unexamined Patent Publication No. 2013-25255 Japanese Unexamined Patent Publication No. 2013-187860 Japanese Unexamined Patent Publication No. 2005-244861

However, the above-mentioned conventional technique has a problem that the image composition deviation cannot be compensated.

As described with reference to FIG. 15, in the model of the virtual spherical surface used in the prior art, it is assumed that the height of the center of the optical axis is the same regardless of the distance and the angle of incidence of the object. However, in an actual optical system, the height of the center of the optical axis differs depending on the angle of incidence. Further, assuming that the height of the center of the optical axis is the same, the incident angle is not uniquely determined by the distance of the object.

FIG. 17 is a diagram for explaining the dissociation between the model of the virtual sphere and the optical system. The grid points 10d and 10e are incident at the incident angle θd, pass through the optical axis center 6a of the spherical surface 6, and are mapped to the image height corresponding to the incident angle θb. On the other hand, the grid points 10b and 10c are incident at the incident angle θb', pass through the optical axis center 6b of the spherical surface 6, and are mapped to the image height corresponding to the incident angle θb'. Assuming that the heights of the optical centers are the same, the incident angle of the grid point 10b is θB, and the incident angle of the grid point 10c is θC, which is not uniquely determined. The circumferential fisheye lens has the characteristic that the incident angle is not uniquely determined by the distance of the object as it is closer to the horizontal.

As described above, the model of the virtual sphere and the optical system are separated from each other. Therefore, when the camera parameters are obtained by performing calibration using the model of the virtual sphere, the first equirectangular projection and the second equirectangular projection are obtained. When combined with a cylindrical diagram, a composition shift occurs for a distant object. FIG. 18 is a diagram showing an example of synthetic deviation. In the example shown in FIG. 18, the composition shift occurs in the region 7.

In one aspect, it is an object of the present invention to provide a compensating program, compensating method and compensating device capable of compensating for image composition misalignment.

In the first plan, the compensation program causes the computer to perform the following processing. The computer estimates the first line segment that indicates the relationship between the incident angle defined by parameters including the image height and focal length of the camera, and the image height and incident angle when the focal length is changed. The second line segment showing the relationship between is calculated. The computer converts the image height deviation and the incident angle deviation specified based on the first line segment, the second line segment, and the line segment of the reference incident angle into pixel deviation, thereby determining the amount of change in the focal length and the amount of change in the focal length. Generate a table associated with the pixel shift. The computer identifies the pixel shift based on the first developed image obtained by expanding the image taken by the first camera with parameters and the second developed image obtained by expanding the image taken by the second camera with parameters. The amount of focal length deviation is calculated based on the specified pixel deviation and the table.

It is possible to compensate for image composition deviation.

FIG. 1 is a functional block diagram showing a configuration of a compensation device according to the first embodiment. FIG. 2 is a diagram showing an example of the data structure of the look-up table according to the first embodiment. FIG. 3 is a diagram for explaining the processing of the generation unit according to the first embodiment. FIG. 4 is a diagram for explaining the processing of the calculation unit according to the first embodiment. FIG. 5 is a diagram showing the relationship between the incident angle and the image height before and after the correction. FIG. 6 is a diagram showing the relationship between the focal length and the image height. FIG. 7 is a diagram showing an example of composite image information before and after correction. FIG. 8 is a flowchart showing a processing procedure of the compensation device according to the first embodiment. FIG. 9 is a flowchart showing a processing procedure of the lookup table generation processing. FIG. 10 is a functional block diagram showing the configuration of the compensation device according to the second embodiment. FIG. 11 is a flowchart showing a processing procedure of the compensation device according to the second embodiment. FIG. 12 is a diagram showing an example of a hardware configuration of a computer that realizes a function similar to that of a compensator. FIG. 13 is a diagram (1) for explaining the conventional calibration. FIG. 14 is a diagram (2) for explaining the conventional calibration. FIG. 15 is a diagram for explaining a conventional model. FIG. 16 is a diagram for explaining a conventional image composition process. FIG. 17 is a diagram for explaining the dissociation between the model of the virtual sphere and the optical system. FIG. 18 is a diagram showing an example of synthetic deviation.

Hereinafter, examples of the compensation program, compensation method, and compensation device disclosed in the present application will be described in detail with reference to the drawings. The present invention is not limited to this embodiment.

FIG. 1 is a functional block diagram showing a configuration of a compensation device according to the first embodiment. As shown in FIG. 1, the compensating device 100 includes cameras 110a and 110b, a communication unit 120, an input unit 130, a display unit 140, a storage unit 150, and a control unit 160.

The cameras 110a and 110b are cameras that capture images with a circumferential fisheye lens. The cameras 110a and 110b output information (image information) of the captured image to the control unit 160. The cameras 110a and 110b may be connected to the network. In the following description, the cameras 110a and 110b are collectively referred to as "camera 110" unless otherwise specified.

The communication unit 120 is a processing unit that executes data communication with another device via a network. For example, when the camera 110 is connected on the network, the communication unit 120 receives the image information from the camera 110 via the network and outputs the received image information to the control unit 160.

The input unit 130 is an input device for the user to input various information. The input unit 130 corresponds to a keyboard, a mouse, a touch panel, and the like.

For example, when the compensator 100 is made to perform calibration, the user arranges the grid pattern 10 at a predetermined position and causes the camera 110 to take an image. By arranging the grid pattern 10 at a predetermined position, the compensator 100 can treat the three-dimensional coordinates of the grid points of the grid pattern 10 as known ones. For example, when the grid pattern 10 is arranged, the user operates the input unit 130 to input a calibration execution request to the compensation device 100.

The display unit 140 is a display device that displays information output from the control unit 160. The display unit 140 corresponds to a liquid crystal display, a touch panel, or the like. For example, the display unit 140 displays the composite image information 150e generated by the control unit 160. The composite image information 150e is information obtained by synthesizing the image information captured by the cameras 110a and 110b.

The storage unit 150 has a buffer 150a, a camera parameter information 150b, a look-up table 150c, a development drawing information 150d, and a composite image information 150e. The storage unit 150 corresponds to semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory (Flash Memory), and storage devices such as HDD (Hard Disk Drive).

The buffer 150a is a buffer for storing each image information taken by the camera 110. For example, the buffer 150a separately stores the image information captured by the camera 110a and the image information captured by the camera 110b.

The camera parameter information 150b is information for holding each camera parameter of the camera 110. For example, the camera parameter information 150b includes an internal parameter of the camera 110a, an internal parameter of the camera 110b, and an external parameter relative to the camera 110a and the camera 110b.

The internal parameters include the focal length, the coordinates of the optical axis point, the distortion coefficient, the coefficient described later, and the like. Relative external parameters include the relative orientation of the camera 110a and the camera 110b (rotation vector) and the relative camera position (translation vector). The coordinates in three dimensions are associated with the coordinates in two dimensions based on the internal and external parameters.

Here, in the first embodiment, the incident angle θ ₁ is defined by the ratio between the image height ρ ₁ of the camera 110a and the focal length f ₁ , and is shown in the equation (1). Further, the incident angle θ ₂ is defined by the ratio between the image height ρ ₂ of the camera 110b and the focal length f ₂ , and is shown in the equation (2). The k ₁₁ , k ₁₂ , k ₁₃ , and k ₁₄ included in the equation (1) are some of the internal parameters of the camera 110a. The k ₂₁ , k ₂₂ , k ₂₃ , and k ₂₄ included in the equation (2) are some of the internal parameters of the camera 110b.

The look-up table 150c is a table showing the relationship between the pixel shift and the focal length error. The pixel shift indicates the pixel shift between the equirectangular view of the image information taken by the camera 110 and the equirectangular view of the image information taken by the camera 110b. The look-up table 150c has a look-up table corresponding to the camera 110a and a look-up table corresponding to the camera 110b. Here, as an example, a look-up table corresponding to the camera 110a will be described. The description of the look-up table corresponding to the camera 110b will be omitted.

FIG. 2 is a diagram showing an example of the data structure of the look-up table according to the first embodiment. As shown in FIG. 2, in the look-up table 150c, the vertical axis indicates the magnitude of pixel shift (for example, pixels). The horizontal axis shows the focal length error (df). When the pixel shift is determined by the look-up table 150c, the focal length error df is uniquely determined. This look-up table 150c is generated by the generation unit 160d described later.

The development view information 150d includes a first equirectangular projection and a second equirectangular projection. The first equirectangular projection is an equirectangular projection generated based on the image information taken by the camera 110a. The second equirectangular projection is an equirectangular projection generated based on the image information taken by the camera 110b.

The composite image information 150e is image information obtained by synthesizing (blending) the first equirectangular projection and the second equirectangular projection.

The control unit 160 includes an acquisition unit 160a, a calibration execution unit 160b, an estimation unit 160c, a generation unit 160d, a development unit 160e, a synthesis unit 160f, and a calculation unit 160g. The control unit 160 can be realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Further, the control unit 160 can also be realized by hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The acquisition unit 160a is a processing unit that acquires image information from the camera 110 and stores the acquired image information in the buffer 150a.

The calibration execution unit 160b is a processing unit that executes calibration based on the image information stored in the buffer 150a and generates camera parameter information 150b. The calibration execution unit 160b executes a process when it receives a calibration execution request from the input unit 130.

The calibration execution unit 160b may use any conventional technique to generate the camera parameter information 150b. When the calibration execution unit 160b executes the calibration, the model of the virtual spherical surface described with reference to FIG. 15 is used. This model of the virtual sphere is a model assuming that the height of the center of the optical axis is the same regardless of the distance of the object (lattice point) and the angle of incidence.

For example, the calibration execution unit 160b detects each grid point from the image information stored in the buffer 150a. The calibration execution unit 160b maps each grid point from the relationship between the incident angle and the image height according to the virtual spherical model, and the line segment connecting the mapped grid points becomes a grid-like straight line. Search for internal parameters. The calibration execution unit 160b searches for internal parameters of the camera 110a and the camera 110b and sets them in the camera parameter information 150b.

The calibration execution unit 160b searches for an external parameter from the correspondence between the two-dimensional coordinates of each grid point on the image information and the three-dimensional coordinates of each known grid point. The calibration execution unit 160b searches for external parameters for the camera 110a and the camera 110b. The calibration execution unit 160b calculates the relative value of each external parameter as a relative external parameter and sets it in the camera parameter information 150b.

The estimation unit 160c is a processing unit that estimates the relationship between the incident angle of the optical system and the image height of the circumferential fisheye lens used in the camera 110. The estimation unit 160c outputs the estimation result to the generation unit 160d.

For example, the estimation unit 160c uses a polynomial of the image height ρ and the focal length f to determine the incident angle based on the correspondence between the intersection of each grid of the known grid pattern 10 and the intersection of each grid detected from the image information. And the image height are estimated. The polynomial of the image height ρ and the focal length f is the polynomial shown in the equations (1) and (2). Here, the case where the estimation unit 160c automatically estimates the relationship between the incident angle of the optical system and the image height has been described, but the user manually estimates the relationship between the incident angle of the optical system and the image height. , The estimation result may be input to the compensation device 100. The estimation unit 160c estimates the relationship between the incident angle of the optical system and the image height in the camera 110a and the relationship between the incident angle and the image height of the optical system in the camera 110b, respectively.

The generation unit 160d is a processing unit that generates a look-up table 150c by using the relationship between the incident angle of the optical system and the image height. FIG. 3 is a diagram for explaining the processing of the generation unit according to the first embodiment. The horizontal axis of the graph of FIG. 3 corresponds to the image height, and the vertical axis corresponds to the incident angle. The line segment 25a is a line segment acquired from the estimation unit 160c and showing the relationship between the incident angle of the optical system and the image height.

The generation unit 160d changes the focal length f by changing the focal length error df shown in the equation (3). The generation unit 160d changes the error df, changes the relationship between the incident angle and the image height, and repeatedly executes the process of converting the image height deviation and the incident angle deviation obtained from the change result into the pixel deviation, thereby focusing. The relationship between the distance error and the pixel shift is obtained, and the look-up table 150c is generated.

f = f (1 + df) ... (3)

The line segment 25a shown in FIG. 3 is a diagram showing the relationship between the incident angle and the image height when “df = 0”. For example, the generation unit 160d changes the focal length f by setting “df = 0.015”, and the incident angle and the image height are determined based on the changed focal length f and the equation (1). A line segment 25b showing the relationship between the above is obtained. The generation unit 160d sets a line segment 30 passing through the incident angle 0 and a line segment 27b passing through the intersection 26b with the line segment 26b. The generation unit 160d calculates the distance between the intersection 26a and the intersection 26b as "image height deviation (image height radius deviation)". The intersection 26a is the intersection of the line segment 30 and the line segment 25a. The generation unit 160d calculates the incident angle (rad) between the intersection 28a and the intersection 26b as the “incident angle deviation”. The intersection 28a is the intersection of the line segment 25a and the line segment 27b.

The generation unit 160d holds a pixel shift conversion table (not shown) that defines the relationship between the incident angle shift and the pixel shift. The generation unit 160d compares the incident angle deviation between the intersection 28a and the intersection 26b with the pixel deviation conversion table to obtain the pixel deviation. Thereby, the generation unit 160d can specify the relationship between the focal length error df = 0.015 and the pixel shift.

Further, the generation unit 160d changes the focal length f by setting “df = −0.015”, and the incident angle and the image height are based on the changed focal length f and the equation (1). A line segment 25c showing the relationship with is obtained. The generation unit 160d sets a line segment 30 passing through the incident angle 0 and a line segment 27c passing through the intersection 26c with the line segment 26c. The generation unit 160d calculates the distance between the intersection 26a and the intersection 26c as "image height deviation (image height radius deviation)". The generation unit 160d calculates the incident angle (rad) between the intersection 28b and the intersection 26c as the “incident angle deviation”. The intersection 28b is the intersection of the line segment 25a and the line segment 27c.

The generation unit 160d compares the incident angle deviation between the intersection 28b and the intersection 26c with the pixel deviation conversion table to obtain the pixel deviation. As a result, the generation unit 160d can specify the relationship between the focal length error df = −0.015 and the pixel shift.

The generation unit 160d repeatedly executes a process of calculating the relationship between the error df and the pixel shift while changing the focal length error df, thereby generating information on the look-up table 150c corresponding to the camera 110a. Similarly, the generation unit 160d corresponds to the camera 110b by repeatedly executing a process of calculating the relationship between the error df and the pixel shift while changing the focal length error df based on the equation (2). Generates information for the lookup table 150c.

Returning to the description of FIG. The expansion unit 160e, the synthesis unit 160f, and the calculation unit 160g, which will be described later, perform processing after the above-mentioned calibration execution. Further, when the developing unit 160e, the synthesizing unit 160f, and the calculating unit 160g perform processing, the grid pattern 10 may not be arranged.

The expansion unit 160e is a processing unit that generates expansion diagram information 150d based on the camera parameter information 150b and the image information stored in the buffer 150a. For example, the expansion unit 160e generates information of the first equirectangular projection based on the image information taken by the camera 110a, the internal parameters of the camera 110a, and the relative external parameters. The expansion unit 160e generates information of the second equirectangular projection based on the image information captured by the camera 110b, the internal parameters of the camera 110b, and the relative external parameters.

When generating the information of the first equirectangular projection, the expansion unit 160e identifies the equator center position on the first equirectangular projection based on the internal parameters and the relative external parameters, and the specified equator. Information on the center position is added to the information on the first equirectangular projection. When generating the information of the second equirectangular projection, the expansion unit 160e identifies the equator center position on the second equirectangular projection based on the internal parameters and the relative external parameters, and the specified equator. Information on the center position is added to the information on the second equirectangular projection.

The compositing unit 160f is a processing unit that synthesizes (blends) the first equirectangular projection and the second equirectangular projection included in the development map information 150d to generate the composite image information 150e. For example, the compositing unit 160f overlaps the equatorial center position of the first equirectangular projection with the equatorial center position of the second equirectangular projection to generate the composite image information 150e.

The calculation unit 160g is a processing unit that calculates pixel deviation based on the composite image information 150e. For example, the calculation unit 160g scans a region near the joint between the first equirectangular projection and the second equirectangular projection included in the composite image information 150e to calculate the pixel deviation.

FIG. 4 is a diagram for explaining the processing of the calculation unit according to the first embodiment. In the composite image information 150e shown in FIG. 4, it is assumed that the first equirectangular projection 151a and the second equirectangular projection 151b are synthesized by the equator 152a (joint). The calculation unit 160g performs shape detection (edge detection, etc.) on a predetermined range 152b with reference to the equator 152a, and estimates pixel deviation.

In the example shown in FIG. 4, the shapes 40a and 40b and the shapes 41a and 41b are detected from the predetermined range 152b. The calculation unit 160g calculates the degree of similarity based on the matching rate of pixel values and the like, and identifies a set of similar shapes based on the degree of similarity. For example, the shape 40a corresponds to the shape 40b, and the shape 41a corresponds to the shape 41b.

The calculation unit 160g calculates the pixel deviation 40c between the shape 40a and the shape 40b. The calculation unit 160g calculates the pixel deviation 41c between the shape 41a and the shape 41b. When a plurality of sets having similar shapes exist, the calculation unit 160g calculates the average value, the maximum value, or the minimum value of each pixel deviation as the final pixel deviation. In the following description, the final pixel shift is simply referred to as "pixel shift".

Subsequently, the calculation unit 160g calculates the pixel shift, compares the pixel shift with the look-up table 150c, specifies the shift compensation parameter, and executes a process of correcting the camera parameter information 150b by the shift compensation parameter. ..

The calculation unit 160g compares the pixel shift with the information in the look-up table 150c corresponding to the camera 110a to specify the focal length error df. The calculation unit 160g sets the specified error df as the deviation compensation parameter df ₁ corresponding to the camera 110a. The calculation unit 160g corrects the camera parameter information 150b by the deviation compensation parameter df ₁ . Specifically, the calculation unit 160g corrects the focal length of the internal parameter corresponding to the camera 110a from the focal length f ₁ to the focal length f ₁ (1 + df ₁ ).

By making the above corrections by the calculation unit 160g, the relationship between the incident angle and the image height is shown in the equation (4).

The calculation unit 160g compares the pixel shift with the information in the look-up table 150c corresponding to the camera 110b to specify the focal length error df. The calculation unit 160g sets the specified error df as the deviation compensation parameter df ₂ corresponding to the camera 110b. The calculation unit 160g corrects the focal length of the internal parameter corresponding to the camera 110b from the focal length f ₂ to the focal length f ₂ (1 + df ₂ ) by the deviation compensation parameter df ₂ .

By making the above corrections by the calculation unit 160g, the relationship between the incident angle and the image height is shown in the equation (5).

FIG. 5 is a diagram showing the relationship between the incident angle and the image height before and after the correction. Here, as an example, the relationship between the incident angle and the image height of the circumferential fisheye lens of the camera 110a will be described. The vertical axis of FIG. 5 corresponds to the incident angle, and the horizontal axis corresponds to the image height. The line segment 25d shows the relationship between the incident angle and the image height before the focal length is corrected. The line segment 25e shows the relationship between the incident angle and the image height after the focal length is corrected. Since the line segment 25e is matched to the relationship between the incident angle of the optical system and the image height, it is possible to compensate for the pixel shift that occurs when the composite image information 150e is created in relation to the line segment 25d. Become.

As described with reference to FIGS. 15 and 17, the incident angle is different between the actual optical system and the virtual spherical model, and it can be said that the difference in the incident angle corresponds to the difference in the focal length f. .. Therefore, by correcting the error of the focal length f with the deviation compensation parameter, the deviation of the image height is compensated and the pixel deviation is compensated.

FIG. 6 is a diagram showing the relationship between the focal length and the image height. For example, the image height corresponding to the point incident at a certain angle of incidence is actually ρ (1 + df), but using the virtual spherical model, the image height is ρ. Pixel shift occurs due to the difference between the image height ρ and the image height ρ (1 + df). Therefore, the compensation device 100 corrects the image height ρ to the image height ρ (1 + df) by correcting the focal length from the focal length f to the focal length f (1 + df) according to the shift compensation parameter, and compensates for the pixel shift. do.

When the camera parameter information 150b is modified, the above-mentioned expansion unit 160e regenerates the first equirectangular projection and the second equirectangular projection based on the camera parameter information 150b, and outputs them to the synthesis unit 160f. ..

The compositing unit 160f regenerates the composite image information 150e by synthesizing the regenerated first equirectangular projection and the second equirectangular projection. The regenerated composite image information 150e is image information compensated for the influence of image misalignment. The compositing unit 160f may output the composite image information 150e to the display unit 140 for display, or may transmit the composite image information 150e to an external device on the network for display.

FIG. 7 is a diagram showing an example of composite image information before and after correction. In FIG. 7, the composite image information _150ea indicates the composite image information before modification. The composite image information _150eb indicates the corrected composite image information. In the composite image information 150e _a , pixel shift occurs in the region 7a. On the other hand, in the composite image information _150eb , no pixel shift occurs in the region 7b. That is, the pixel deviation can be compensated by correcting the camera parameter information 150b with the deviation compensation parameter.

Next, an example of the processing procedure of the compensation device 100 according to the first embodiment will be described. FIG. 8 is a flowchart showing a processing procedure of the compensation device according to the first embodiment. As shown in FIG. 8, the calibration execution unit 160b of the compensation device 100 identifies the internal parameters and the relative external parameters by executing the calibration of each camera 110, and generates the camera parameter information 150b. (Step S101).

The estimation unit 160c and the generation unit 160d of the compensation device 100 execute the look-up table generation process (step S102). The expansion unit 160e of the compensation device 100 expands the image information of each camera into an equirectangular projection (step S103).

The compositing unit 160f of the compensator 100 generates composite image information 150e by compositing each equirectangular projection (step S104). The calculation unit 160g of the compensation device 100 calculates the pixel deviation based on the composite image information 150e (step S105). The calculation unit 160g calculates the deviation compensation parameter based on the look-up table 150c and the pixel deviation (step S106).

The calculation unit 160g corrects the internal parameters of the camera parameter information 150b based on the deviation compensation parameters (step S107). The expansion unit 160e expands the image information of each camera into a circular cylinder diagram based on the modified internal parameters (step S108).

The synthesizing unit 160f generates composite image information 150e by synthesizing each circular cylinder diagram (step S109). The compositing unit 160f outputs the composite image information 150e to the display unit 140 and displays it (step S110).

Next, an example of the processing procedure of the lookup table generation processing shown in step S201 of FIG. 8 will be described. FIG. 9 is a flowchart showing a processing procedure of the lookup table generation processing. As shown in FIG. 9, the estimation unit 160c of the compensation device 100 estimates the relationship between the image height of the optical system and the incident angle (step S201).

The generation unit 160d of the compensation device 100 changes the focal length df (step S202). The generation unit 160d changes the relationship between the image height and the incident angle according to the changed focal length df (step S203). The generation unit 160d calculates the image height deviation at the incident angle 0 (step S204).

The generation unit 160d calculates the incident angle deviation corresponding to the image height deviation (step S205). The generation unit 160d converts the incident angle deviation into a pixel deviation (step S206). The generation unit 160d registers the relationship between the focal length df and the pixel shift in the look-up table 150c (step S207).

When the process of generating the look-up table 150c is terminated (step S208, Yes), the generation unit 160d terminates the process. On the other hand, if the generation unit 160d does not end the process of generating the look-up table 150c (steps S208, No), the generation unit 160d proceeds to step S202.

Next, the effect of the compensation device 100 according to the first embodiment will be described. The compensator 100 estimates the relationship between the incident angle and the image height of the optical system of the circumferential fisheye lens, changes the focal length df, and changes the relationship between the incident angle and the image height to obtain the focal length df. The corresponding pixel shift is specified and the lookup table 150c is generated. The compensation device 100 specifies the deviation compensation parameter based on the pixel deviation of the composite image information generated based on the camera parameter information 150b before correction and the look-up table 150c, and the camera is based on the deviation compensation parameter. Correct the internal parameters of the parameter information 150b. As a result, the image height deviation due to the incident angle can be compensated by the deviation compensation parameter, and the occurrence of pixel deviation can be suppressed.

For example, as described with reference to FIGS. 5 and 6, the deviation compensation parameter allows the relationship between the incident angle and the image height of the virtual spherical model to be matched with the relationship between the incident angle and the image height of the optical system, and the pixel shift can be adjusted. Can be compensated.

Next, the compensating device according to the second embodiment will be described. The compensation device according to the second embodiment generates a look-up table in the same manner as the compensation device 100 of the first embodiment, and based on the pixel shift of the composite image information and the lookup table 150c, the shift compensation parameter. To identify. The compensation device according to the second embodiment corrects the image height by the deviation compensation parameter and compensates for the pixel deviation.

For example, in the compensation device 100 of the first embodiment, as described with reference to FIG. 6, the focal length of the internal parameter is corrected from the focal length f to the focal length f (1 + df) to obtain the actual image height ρ (1 + df). It was matched. On the other hand, the compensator of the second embodiment directly corrects the image height from the image height ρ to the image height ρ (1 + df) without modifying the focal length.

FIG. 10 is a functional block diagram showing the configuration of the compensation device according to the second embodiment. As shown in FIG. 10, the compensating device 200 includes cameras 210 and 210b, a communication unit 220, an input unit 230, a display unit 240, a storage unit 250, and a control unit 260.

The cameras 210a and 210b are cameras that capture images with a circumferential fisheye lens. The cameras 210a and 210b output the captured image information (image information) to the control unit 260. The cameras 210a and 210b may be connected to the network. In the following description, the cameras 210a and 210b are collectively referred to as "camera 210" unless otherwise specified.

The communication unit 220 is a processing unit that executes data communication with other devices via a network. For example, when the camera 210 is connected to the network, the communication unit 220 receives the image information from the camera 210 via the network and outputs the received image information to the control unit 260.

The input unit 230 is an input device for the user to input various information. The input unit 230 corresponds to a keyboard, a mouse, a touch panel, and the like. The user arranges the grid pattern 10 at a predetermined position in the same manner as in the first embodiment, operates the input unit 230, and inputs the calibration execution request to the compensation device 200.

The display unit 240 is a display device that displays information output from the control unit 260. The display unit 240 corresponds to a liquid crystal display, a touch panel, or the like. For example, the display unit 240 displays the composite image information 250e generated by the control unit 260. The composite image information 250e is information obtained by synthesizing the image information captured by the cameras 210a and 210b.

The storage unit 250 has a buffer 250a, a camera parameter information 250b, a look-up table 250c, a development drawing information 250d, and a composite image information 250e. The storage unit 250 corresponds to a semiconductor memory element such as a RAM, ROM, or a flash memory, or a storage device such as an HDD.

The buffer 250a is a buffer for storing each image information taken by the camera 210. For example, the buffer 250a separately stores the image information captured by the camera 210a and the image information captured by the camera 210b.

The camera parameter information 250b is information for holding each camera parameter of the camera 110. For example, the camera parameter information 250b includes an internal parameter of the camera 210a, an internal parameter of the camera 210b, and an external parameter relative to the camera 210a and the camera 210b.

The internal parameters include the focal length, the coordinates of the optical axis point, the distortion coefficient, the coefficient described later, and the like. Relative external parameters include the relative orientation of the camera 210a and the camera 210b (rotation vector) and the relative camera position (translation vector). The coordinates in three dimensions are associated with the coordinates in two dimensions based on the internal and external parameters.

The look-up table 250c is a table showing the relationship between the pixel shift and the focal length error. The data structure of the look-up table 250c is the same as the data structure of the look-up table 150c described with reference to FIG.

The development view information 250d includes a first equirectangular projection and a second equirectangular projection. The first equirectangular projection is an equirectangular projection generated based on the image information taken by the camera 210a. The second equirectangular projection is an equirectangular projection generated based on the image information taken by the camera 210b.

The composite image information 250e is image information obtained by synthesizing (blending) the first equirectangular projection and the second equirectangular projection.

The control unit 260 includes an acquisition unit 260a, a calibration execution unit 260b, an estimation unit 260c, a generation unit 260d, a development unit 260e, a synthesis unit 260f, and a calculation unit 260g. The control unit 260 can be realized by a CPU, an MPU, or the like. Further, the control unit 260 can also be realized by hard-wired logic such as ASIC or FPGA.

The acquisition unit 260a is a processing unit that acquires image information from the camera 210 and stores the acquired image information in the buffer 250a.

The calibration execution unit 260b is a processing unit that executes calibration based on the image information stored in the buffer 250a and generates camera parameter information 250b. The processing of the calibration execution unit 260b is the same as the processing of the calibration execution unit 160b described in the first embodiment.

The estimation unit 260c is a processing unit that estimates the relationship between the incident angle of the optical system and the image height of the circumferential fisheye lens used in the camera 210. The estimation unit 260c outputs the estimation result to the generation unit 260d. The processing of the estimation unit 260c is the same as the processing of the estimation unit 160c described in the first embodiment.

The generation unit 260d is a processing unit that generates a look-up table 150c by using the relationship between the incident angle of the optical system and the image height. The processing of the generation unit 260d is the same as the processing of the generation unit 160d described in the first embodiment.

The expansion unit 260e, the synthesis unit 260f, and the calculation unit 260g, which will be described later, perform processing after the above-mentioned calibration execution. Further, when the developing unit 260e, the synthesizing unit 260f, and the calculating unit 260g perform processing, the grid pattern 10 may not be arranged.

The expansion unit 260e is a processing unit that generates expansion diagram information 250d based on the camera parameter information 250b and the image information stored in the buffer 250a. The processing of the expanding unit 260e is the same as the processing of the expanding unit 160e described in the first embodiment.

The compositing unit 260f is a processing unit that synthesizes (blends) the first equirectangular projection and the second equirectangular projection included in the development map information 250d to generate the composite image information 250e. For example, the compositing unit 260f overlaps the equatorial center position of the first equirectangular projection with the equatorial center position of the second equirectangular projection to generate the composite image information 250e.

The calculation unit 260g is a processing unit that calculates pixel deviation based on the composite image information 250e. Further, the calculation unit 260g specifies the deviation compensation parameter based on the pixel deviation and the look-up table 250c. The process of calculating the pixel shift by the calculation unit 260 g and the process of specifying the shift compensation parameter are the same as the process of the calculation unit 160 g described in the first embodiment.

The calculation unit 260g outputs the information of the deviation compensation parameter to the expansion unit 260e. Here, the deviation compensation parameter corresponding to the camera 210a is defined as the deviation compensation parameter df ₁ . The deviation compensation parameter corresponding to the camera 210b is defined as the deviation compensation parameter df ₂ .

Here, when the expansion unit 260e acquires the deviation compensation parameter information, the expansion unit 260e generates the development diagram information 150d based on the camera parameter information 150b and the image information stored in the buffer 150a. In addition, the image height is corrected. For example, the development unit 260e modifies the image height when generating the first equirectangular projection as shown in the equation (6). The unfolding section 260e modifies the image height when generating the second equirectangular projection as shown in the equation (7). Thereby, the position (image height) of each point of the image information mapped based on the camera parameter information 250b can be corrected, and the pixel shift can be compensated.

ρ ₁ = ρ ₁ × (1 + df) ・ ・ ・ (6)
ρ ₂ = ρ ₂ × (1 + df) ・ ・ ・ (7)

Next, an example of the processing procedure of the compensation device 200 according to the second embodiment will be described. FIG. 11 is a flowchart showing a processing procedure of the compensation device according to the second embodiment. As shown in FIG. 11, the calibration execution unit 260b of the compensation device 200 identifies the internal parameters and the relative external parameters by executing the calibration of each camera 210, and generates the camera parameter information 250b. (Step S301).

The estimation unit 260c and the generation unit 260d of the compensation device 200 execute the look-up table generation process (step S302). The expansion unit 260e of the compensation device 200 expands the image information of each camera into an equirectangular projection (step S303).

The synthesis unit 260f of the compensator 100 generates composite image information 250e by synthesizing each equirectangular projection (step S304). The calculation unit 260g of the compensation device 200 calculates the pixel deviation based on the composite image information 250e (step S305). The calculation unit 260g calculates the deviation compensation parameter based on the look-up table 150c and the pixel deviation (step S306).

The expansion unit 260e expands the image oz information of each camera 210 into an equirectangular projection based on the camera parameter information 250b while correcting the image height by the deviation compensation parameter (step S307). The synthesizing unit 260f generates composite image information 250e by synthesizing each circular cylinder diagram (step S308). The compositing unit 260f outputs the composite image information 250e to the display unit 240 and displays it (step S309).

Next, the effect of the compensation device 200 according to the second embodiment will be described. The compensation device 200 corrects the image height by using the deviation compensation parameter when generating the development view information 250d based on the camera parameter information 250b and the image information stored in the buffer 250a. As a result, even if the image height specified by the camera parameter information 250b is different, the position of the image height is corrected by the shift compensation parameter, so that the pixel shift of the composite image information 250e can be compensated.

Next, an example of the hardware configuration of the computer that realizes the same functions as the compensation devices 100 and 200 shown in the first and second embodiments will be described. FIG. 12 is a diagram showing an example of a hardware configuration of a computer that realizes a function similar to that of a compensator.

As shown in FIG. 12, the computer 300 includes a CPU 301 that executes various arithmetic processes, an input device 302 that receives data input from a user, and a display 303. Further, the computer 300 has a reading device 304 that reads a program or the like from a storage medium, and an interface device 305 that exchanges data with another device via a wired or wireless network. Further, the computer 300 has a RAM 306 for temporarily storing various information and a hard disk device 307. Then, each of the devices 301 to 307 is connected to the bus 308.

The hard disk apparatus 307 includes an acquisition program 307a, a calibration execution program 307b, an estimation program 307c, a generation program 307d, a development program 307e, a synthesis program 307f, and a calculation program 307g. Further, the hard disk device 307 reads out each program 307a to 307g and develops it in the RAM 306.

The acquisition program 307a functions as the acquisition process 306a. The calibration execution program 307b functions as the calibration execution process 306b. The estimation program 307c functions as an estimation process 306c. The generation program 307d functions as the generation process 306d. The deployment program 307e functions as the deployment process 306e. The synthesis program 307f functions as a synthesis process 306f. The calculation program 307g functions as a calculation process 306g.

The processing of the acquisition process 306a corresponds to the processing of the acquisition units 160a and 260a. The processing of the calibration execution process 306 corresponds to the processing of the calibration execution units 160b and 260b. The processing of the estimation process 306c corresponds to the processing of the estimation units 160c and 260c. The processing of the generation process 306d corresponds to the processing of the generation units 160d and 260d. The processing of the unfolding process 306e corresponds to the processing of the unfolding unit 360e. The processing of the synthesis process 306f corresponds to the processing of the synthesis units 160f and 260f. The processing of the calculation process 306g corresponds to the processing of the calculation units 160g and 260g.

The programs 307a to 307e do not necessarily have to be stored in the hard disk device 307 from the beginning. For example, each program is stored in a "portable physical medium" such as a flexible disk (FD), a CD-ROM, a DVD, a magneto-optical disk, or an IC card inserted in the computer 300. Then, the computer 300 may read and execute each program 307a to 307e.

The following additional notes will be further disclosed with respect to the embodiments including each of the above embodiments.

(Appendix 1) Estimate the first line segment showing the relationship between the incident angle defined by the parameters including the image height and focal length of the camera and the image height.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated.
By converting the image height deviation and the incident angle deviation specified based on the first line segment, the second line segment, and the line segment of the reference incident angle into pixel deviation, the amount of change in the focal length and the amount of change in the focal length can be obtained. A table associated with the pixel shift is generated, and the table is generated.
Pixel deviation is specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by developing the image taken by the second camera with the above parameters.
A compensation program characterized by having a computer execute a process of calculating the amount of focal length deviation based on the specified pixel deviation and the table.

(Appendix 2) The compensation program according to Appendix 1, wherein a computer is further executed to correct the focal length included in the parameter based on the amount of deviation of the focal length.

(Appendix 3) A computer is further executed to correct the image height when the first developed image is developed and the image height when the second developed image is developed based on the amount of deviation of the focal length. The compensation program according to Appendix 1, characterized in that.

(Supplementary Note 4) The compensation program according to Supplementary note 2 or 3, wherein a computer is further executed to perform a process of synthesizing the first developed image and the second developed image.

(Appendix 5) This is a compensation method executed by a computer.
The first line segment showing the relationship between the incident angle defined by the parameters including the image height and focal length of the camera and the image height is estimated.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated, and specified based on the first line segment, the second line segment, and the reference incident angle. By converting the image height shift and the incident angle shift to be pixel shifts, a table associated with the change amount of the focal length and the pixel shifts is generated.
Pixel deviation is specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by developing the image taken by the second camera with the above parameters.
A compensation method characterized by executing a process of calculating the amount of focal length deviation based on the specified pixel deviation and the table.

(Supplementary Note 6) The compensation method according to Supplementary note 5, wherein the process of correcting the focal length included in the parameter is further executed based on the amount of deviation of the focal length.

(Appendix 7) A computer is further executed to correct the image height when the first developed image is developed and the image height when the second developed image is developed based on the amount of deviation of the focal length. The compensation method according to Appendix 5, characterized in that.

(Supplementary Note 8) The compensation method according to Supplementary note 6 or 7, wherein the process of synthesizing the first developed image and the second developed image is further executed.

(Appendix 9) An estimation unit that estimates a first line segment that indicates the relationship between the incident angle defined by parameters including the image height and focal length of the camera and the image height.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated, and specified based on the first line segment, the second line segment, and the reference incident angle. By converting the image height shift and the incident angle shift to be pixel shifts, a generation unit that generates a table associated with the change amount of the focal length and the pixel shifts.
Pixel shift is specified and specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by developing the image taken by the second camera with the above parameters. A compensator having a calculation unit for calculating the amount of focal length deviation based on the pixel deviation and the table.

(Appendix 10) The compensator according to Appendix 9, wherein the calculation unit corrects the focal length included in the parameter based on the amount of deviation of the focal length.

(Appendix 11) The calculation unit corrects the image height when the first developed image is developed and the image height when the second developed image is developed based on the amount of deviation of the focal length. The compensation device according to Appendix 9, which is a feature.

(Supplementary Note 12) The compensator according to Supplementary Note 10 or 11, further comprising a compositing unit for synthesizing the first developed image and the second developed image.

110a, 110b, 210a, 210b Camera 120, 220 Communication unit 130, 230 Input unit 140, 240 Display unit 150, 250 Storage unit 150a, 250a Buffer 150b, 250b Camera parameter information 150c, 250c Look-up table 150d, 250d Expansion drawing information 150e, 250e Composite image information 160, 260 Control unit 160a, 260a Acquisition unit 160b, 260b Calibration execution unit 160c, 260c Estimate unit 160d, 260d Generation unit 160e, 260e Deployment unit 160f, 260f Synthesis unit 160g, 260g Calculation unit

Claims (6)

The first line segment showing the relationship between the incident angle defined by the parameters including the image height and focal length of the camera and the image height is estimated.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated.
By converting the image height deviation and the incident angle deviation specified based on the first line segment, the second line segment, and the line segment of the reference incident angle into pixel deviation, the amount of change in the focal length and the amount of change in the focal length can be obtained. A table associated with the pixel shift is generated, and the table is generated.
Pixel deviation is specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by developing the image taken by the second camera with the above parameters.
A compensation program characterized by having a computer execute a process of calculating the amount of focal length deviation based on the specified pixel deviation and the table. The compensation program according to claim 1, further comprising causing a computer to perform a process of correcting the focal length included in the parameter based on the amount of deviation of the focal length. Based on the amount of deviation of the focal length, the computer is further subjected to a process of correcting the image height when the first developed image is developed and the image height when the second developed image is developed. The compensation program according to claim 1. The compensation program according to claim 2 or 3, wherein a computer is further executed to perform a process of synthesizing the first developed image and the second developed image. It ’s a computer-executed compensation method.
The first line segment showing the relationship between the incident angle defined by the parameters including the image height and focal length of the camera and the image height is estimated.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated, and specified based on the first line segment, the second line segment, and the reference incident angle. By converting the image height shift and the incident angle shift to be pixel shifts, a table associated with the change amount of the focal length and the pixel shifts is generated.
Pixel deviation is specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by developing the image taken by the second camera with the above parameters.
A compensation method characterized by executing a process of calculating the amount of focal length deviation based on the specified pixel deviation and the table. An estimation unit that estimates a first line segment that indicates the relationship between the incident angle defined by parameters including the image height and focal length of the camera and the image height.
A second line segment showing the relationship between the image height and the incident angle when the focal length is changed is calculated, and specified based on the first line segment, the second line segment, and the reference incident angle. By converting the image height shift and the incident angle shift to be pixel shifts, a generation unit that generates a table associated with the change amount of the focal length and the pixel shifts.
Pixel shift is specified and specified based on the first developed image obtained by expanding the image taken by the first camera with the above parameters and the second developed image obtained by expanding the image taken by the second camera with the above parameters. A compensator having a calculation unit for calculating the amount of focal length deviation based on the pixel deviation and the table.

Images