JP6234859B2 - Image processing system - Google Patents

Description

  The present invention relates to an image processing system that forms a composite image including a captured image.

The captured image captured by the camera is not always easy for the user to see.
For example, individual captured images output from a plurality of cameras installed in a vehicle are not easy to see for the driver, and the situation of a monitoring object (for example, a road behind the vehicle that becomes a blind spot of the vehicle) is grasped. It may be difficult. Therefore, in the technique described in Patent Document 1, if the monitoring object is photographed from a specific viewpoint (virtual viewpoint) that is most easily understood by the driver based on the captured image obtained by these cameras. Then, the image that would have been formed is synthesized. For example, by combining images taken by a plurality of cameras arranged on the ceiling of the vehicle, a combined image that would have been obtained if the vehicle was viewed from above is formed.

International Publication No. 00/07373

  However, the image processing system in which the monitoring target is set according to the user's request has the following problems. That is, since the most easily viewable angle with respect to the monitoring target differs depending on the monitoring target specified by the user, only a composite image from a preset predetermined viewpoint (for example, a viewpoint capable of overlooking the shooting target) is used. In this case, the user's request may not be sufficiently satisfied.

  For example, in the case of a monitoring system that monitors the traffic state of a city area and the crowded state of a person, a composite image showing the traffic condition of the city area is formed based on an aerial photograph image of the monitoring area and a captured image taken by a camera. However, since the streets to be photographed are not uniform, the monitoring target may be difficult to visually recognize depending on the streets only with a composite image from a predetermined viewpoint set in advance.

  In addition, the above-described technique, that is, the technique for synthesizing the image captured by the camera and the image (area image) as the background of the captured image is a technique that is also used in the video industry field such as a game. There is a problem. For example, there is a technology that combines a captured image captured by a camera with a person as an imaging target and an area image, but only a composite image from a predetermined viewpoint that does not satisfy the request of the viewer There is.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an image processing system capable of outputting a composite image of a viewpoint that meets a user's request.

An image processing system that solves the above-described problem is obtained by combining a converted photographed image obtained by projective transformation of at least a part of a photographed image output from a camera and an area image serving as a background of the photographed image. A combining unit that forms a combined model in space; and a converting unit that forms a combined image by converting the combined model into an image viewed from a preset first viewpoint or a second viewpoint set by an operator; And a selection unit that selects which image is to be displayed on the display unit from among the composite image related to the first viewpoint and the composite image related to the second viewpoint.
For example, the camera is for monitoring installed around a monitoring place. Alternatively, the camera is a mobile phone camera that captures a scene around the current position of the user.

  According to this configuration, an image viewed from the preset first viewpoint (synthesized image) or an image viewed from the second viewpoint set by the operator (synthesized image) can be formed based on the synthesized model. . In addition, an image to be displayed on the display unit can be selected via the selection unit. That is, this image processing system can output a composite image of viewpoints that meet the user's request.

  The operator means a person who operates the image processing system. The operator includes a user of the image processing system and a person who receives a request from the user and operates the image processing system to set the second viewpoint.

  The image processing system further includes a setting unit for setting the second viewpoint, and the setting unit receives viewpoint information related to the second viewpoint, and forms the composite image related to the viewpoint information in the conversion unit. It is preferable to output the composite image to the display unit.

  When the image processing system does not have such a setting unit, it is necessary to incorporate a viewpoint that meets the user's request in the program of the image processing system in advance. According to the above configuration, the viewpoint is used via the setting unit. It is possible to easily set the viewpoint according to the request of the person.

  In the image processing system, it is preferable that the setting unit associates the viewpoint information with a switch of the selection unit in accordance with a save command based on an operation by an operator. According to this configuration, it is possible to display a composite image from a viewpoint set in the past by a simple operation.

  In the image processing system, the setting unit is configured to output one of the photographed images output from the camera to the combining unit while an operation for setting the second viewpoint is being performed. It is preferable to form the synthetic model using an image.

While the operation for setting the second viewpoint is being performed, the number of arithmetic processes executed by the image processing system increases, and thus there is a possibility that a delay may occur in the formation of the composite image.
In this regard, in the above configuration, a composite model is formed using one image of the captured images output from the camera. Thus, the number of computation processes related to the formation of the composite model and the composite image within a predetermined period is smaller than when a composite model is formed each time a captured image is updated and a composite image is formed based on the composite model. Become. As a result, the occurrence of the delay in the formation of the composite image is suppressed.

  In the image processing system, the area image is an aerial photograph map image including a photographing location photographed by the camera, and the combining unit includes a setting region having a geometric figure set on the photographed image and the setting region. It is preferable that the converted photographed image is formed by projective transformation of the photographed image so that a portion corresponding to the setting region on the area image matches on the outer shape.

  According to this configuration, since the captured image is projectively transformed so that the setting area on the captured image and the portion corresponding to the setting area on the area image coincide with each other, at the boundary between the converted captured image and the area image Image shift is reduced.

  The image processing system can output a composite image of a viewpoint that meets a user's request.

The schematic diagram of a monitoring system. 1 is a block diagram of an image processing system. The block diagram which shows the flow about the synthesis | combination of a synthesized image. It is a figure for demonstrating projective transformation, (a) is a figure which shows the picked-up image of a monitoring place, (b) is a figure which shows the map image of the monitoring vicinity whole area. It is a figure for demonstrating projective transformation, (a) is a coordinate diagram which shows the coordinate of each point of the geometric figure preset in the picked-up image, (b) is preset in a map image. The coordinate diagram which shows the coordinate of each point of the geometric figure. The figure which shows a synthetic | combination model. The figure which shows the synthesized image which concerns on a 1st viewpoint. The figure which shows the synthesized image which concerns on a 2nd viewpoint. The figure which shows the synthesized image which concerns on a 1st viewpoint. The figure which shows the synthesized image which concerns on a 2nd viewpoint.

With reference to FIGS. 1-10, the monitoring system which monitors the corner of a city is demonstrated as an example of embodiment of this technique.
As shown in FIG. 1, the monitoring system 1 includes an image processing system 10 and one or a plurality of cameras 20.

  The camera 20 is connected to the image processing system 10 via the communication network 100. Examples of the communication network 100 include a LAN (Local Area Network), the Internet, and a wireless network LAN. Note that the camera 20 may be directly connected to the image processing system 10.

  The camera 20 photographs a preset monitoring place from above. For example, the camera 20 is installed outdoors or indoors. Outdoors, the camera 20 is installed on the roof of a building, the outer wall of the building, a power pole, or the like. Indoors, the camera 20 is installed on the ceiling surface, wall surface, pillar, etc. of the building.

Further, when a plurality of cameras 20 are used to monitor different monitoring locations, the cameras 20 are installed at different locations.
In addition, when the monitoring place is wide, the camera 20 is installed so that the ranges captured by the camera 20 overlap each other at the boundary of the range.

  The camera 20 continuously shoots the monitoring place periodically or at a predetermined (or arbitrary) timing. Then, the camera 20 packetizes the captured image FI and outputs it. When the monitoring location is monitored using a plurality of cameras 20, each camera 20 is controlled so that the relevance of the captured image FI captured by each camera 20 is maintained. For example, the shooting timing is set in each camera 20 so that each camera 20 performs shooting at the same timing (for example, at the same time).

  The relevance of the captured image FI indicates the relationship between the captured images FI captured at the same time. When tracking a person or a vehicle based on a plurality of captured images FI, it is preferable that the relevance of the captured images FI is maintained, so that the capturing timing of each camera 20 is controlled.

  The image processing system 10 displays a composite model SM (described later) constructed on a three-dimensional space (on a three-dimensional data space) based on a captured image FI and a map image MI (area image) output from the camera 20. A composite image SI (described later) to be displayed on the unit 17 is formed.

The configuration of the image processing system 10 will be described with reference to FIG.
The image processing system 10 includes a main device 10A, a display unit 17 connected to the main device 10A, and an operation unit 18.

  The display unit 17 is configured by a liquid crystal monitor, for example. The operation unit 18 includes a keyboard and a mouse (pointing device). The display unit 17 can also be configured with a touch panel. In this case, the operation unit 18 can be omitted.

  The main body device 10A includes an accumulation unit 11 that accumulates captured images FI and the like, an image processing unit 12 that performs image processing on the captured images FI, a storage unit 13 that stores a map image MI, and a combining unit 14 that forms a combined model SM. And a conversion unit 15 that converts the composite model SM into an image viewed from a predetermined viewpoint (hereinafter referred to as “composite image SI”). Moreover, it is preferable that 10 A of main body apparatuses are provided with the setting part 16 for setting the viewpoint which sees the synthetic model SM.

The image processing unit 12 includes a distortion correction unit 12A and a projective conversion unit 12B.
The distortion correction unit 12 </ b> A corrects distortion of the captured image FI generated by the lens of the camera 20. Specifically, since a distortion caused by the lens occurs in a portion away from the optical center axis in the captured image FI, the entire captured image FI is corrected so that the distortion in this portion is eliminated. According to this correction, when the boundary line of a road or the ridge line of a building that should be a straight line is shown in a distorted state, the boundary line or the ridge line is corrected to a straight line.

  The projective conversion unit 12B performs projective conversion of the captured image FI so that a setting area having a geometric figure on the ground in the captured image FI matches a portion corresponding to the setting area on the map on the outer shape. Further, the projective conversion unit 12B removes unnecessary portions from the captured image FI subjected to the projective conversion. For example, the other parts are removed so that the monitoring target area remains. When there is no unnecessary portion in the projected image FI subjected to the projective transformation, such a removal process is not executed. The image formed as described above is referred to as “converted photographed image CFI” in the following description.

The storage unit 13 stores a map image MI and the like. The map image MI is an image including a shooting location taken by the camera 20.
Examples of the map image MI include a road map image, an aerial photograph map image, and an architectural drawing image. The road map image indicates a map on which at least roads are described. The aerial photograph map image indicates a map created based on a photograph obtained by photographing the ground from above. For example, examples of the aerial photograph map image include an aerial photograph map image created based on a photograph taken by an aircraft, and a satellite photograph map image created based on a photograph taken by an artificial satellite.

The synthesizing unit 14 forms a two-dimensional synthetic model SM in the three-dimensional space based on the map image MI and the converted captured image CFI (see FIG. 6).
Specifically, the composition unit 14 arranges the map image MI in the three-dimensional space, and the four set points set in the converted photographed image CFI and the four corresponding points set in the map image MI are respectively set. The converted captured image CFI is superimposed on the map image MI so as to match (see FIGS. 4 to 6).

  “Superimpose” indicates that another image is superimposed on the original image. In other words, in the image obtained by superimposing, in the portion where another image is superimposed on the original image, the original image (the map image MI) is not seen so that the original image is not seen through. Another image (converted image CFI) is superimposed on the image.

  The conversion unit 15 performs perspective projection conversion of the composite model SM on the projection plane PPP. Specifically, the conversion unit 15 forms a perspective projection image (synthesized image SI) by perspectively projecting the synthesized model SM on a projection plane PPP arranged between the synthesized model SM and a predetermined viewpoint. . This conversion is performed using, for example, a graphics interface program or a predetermined conversion function.

The viewpoint is composed of at least the following three parameters (hereinafter, each parameter is referred to as “first to third parameters”).
These three parameters are also referred to as viewpoint information.
The first parameter is a coordinate point (viewpoint position VP) indicating the position of the viewpoint.
The second parameter is a point (gaze point LAP) at which the virtual camera placed at the viewpoint is gaze, and is a coordinate point (x1, y1, z1) set in the three-dimensional space.
The third parameter is a unit vector EV indicating the vertical direction of the projection plane PPP on which the composite model SM is projected.

These three parameters are configured as parameters that can be set by the operator.
Parameters relating to the viewpoint include parameters such as the position and size of the projection plane PPP and the viewing angle of the virtual camera, and default values set in the graphics interface program are used as these parameters. These parameters can also be configured as parameters that can be set by the operator.

  At least one viewpoint is stored in the storage unit 13 as a representative viewpoint. For example, a viewpoint overlooking the synthesized model SM is stored in advance in the storage unit 13 as the first viewpoint FFP (see FIG. 6). If the operator does not change the viewpoint, an image (synthesized image SI) when the synthesized model SM is viewed from the first viewpoint FFP is formed.

  The setting unit 16 is a user interface for setting a viewpoint (hereinafter, “second viewpoint SFP”) different from the preset first viewpoint FFP. For example, the setting unit 16 is configured as follows.

  The setting unit 16 includes a first display instruction function for displaying the composite model SM on the display unit 17, a reception function for receiving viewpoint information input via the operation unit 18, and a viewpoint (hereinafter referred to as “input viewpoint”) related to viewpoint information. ”) On the display unit 17 and a conversion instruction function that causes the conversion unit 15 to convert the composite model SM into an image from the input viewpoint. Furthermore, the setting unit 16 has a storage function for storing viewpoint information in the storage unit 13 based on an operator's storage command input via the operation unit 18.

The setting unit 16 operates as follows.
The setting unit 16 causes the display unit 17 to display the three-dimensional coordinate axis and the synthesized model SM.
In addition, the setting unit 16 receives viewpoint information input via the operation unit 18, and when the viewpoint information is received, the input viewpoint based on the viewpoint information is displayed on the three-dimensional coordinates on the display unit 17. Let Furthermore, the setting unit 16 causes the conversion unit 15 to execute conversion processing for converting the composite model SM into a composite image SI viewed from the input viewpoint. Then, the setting unit 16 displays the composite image SI on the display unit 17. Thereby, the user (or operator) can confirm the image (synthesized image SI) that would have been obtained if the synthesized model SM was viewed from the input viewpoint.

  In addition, when receiving a save command for saving viewpoint information from the operation unit 18, the setting unit 16 stores the viewpoint information related to the input viewpoint in the storage unit 13 as the second viewpoint SFP, and the user (or operator) ) Is associated with each of the two buttons displayed on the display unit 17 so that the user can select one of the first viewpoint FFP and the second viewpoint SFP. These buttons can be selected by a user (or an operator) via a pointer that moves on the screen of the display unit 17 by operating the operation unit 18. In other words, the display unit 17 and the operation unit 18 function as a selection unit 19 for selecting which viewpoint to use from the first viewpoint FFP and the second viewpoint SFP.

  For this reason, the user (or operator) selects one of the two buttons shown on the display unit 17 via the operation unit 18 to view the composite model SM from the first viewpoint FFP. Any one of the synthesized images SI obtained by viewing the obtained synthesized image SI and the synthesized model SM from the second viewpoint SFP can be displayed on the display unit 17.

Furthermore, the setting unit 16 preferably has the following configuration.
When the image processing system 10 is in a state of receiving viewpoint information input by the operator (hereinafter, this state is referred to as “viewpoint setting mode”) (for example, when displaying the composite model SM on the display unit 17). The setting unit 16 causes the combining unit 14 to form the combined model SM using only one image of the captured images FI output from the camera 20.

This is due to the following reason.
In the viewpoint setting mode, the main body apparatus 10A executes various arithmetic processes such as displaying the synthesized model SM together with the three-dimensional coordinate axis, displaying the viewpoint on the three-dimensional coordinates, and displaying the synthesized image SI. These increase the number of arithmetic processes of the main device 10A. Therefore, each time a new converted captured image CFI is stored in the storage unit 11, the combined unit SM is formed based on the converted captured image CFI and the map image MI. There is a risk of exceeding its computing power. On the other hand, in the viewpoint setting mode, since the purpose is only to set the viewpoint, the latest information (for example, the degree of congestion at the shooting location) of the captured image FI output from the camera 20 is not required. For this reason, in the viewpoint setting mode, the setting unit 16 causes the combining unit 14 to form the combined model SM using only one image of the captured image FI output from the camera 20. As a result, the calculation burden on the main unit 10A is reduced, and the occurrence of a situation in which the number of calculation processes of the main unit 10A exceeds the calculation capability is suppressed.

With reference to FIG. 3, the flow of synthesis of the composite image SI will be described.
The synthesizing unit 14 acquires the map image MI from the storage unit 13 and superimposes the converted captured image CFI on the map image MI. When there are a plurality of converted captured images CFI, the converted captured images CFI are sequentially superimposed as shown in FIG. In this way, the synthesis unit 14 forms a synthesis model SM.

  Also, a new converted captured image CFI is formed based on the captured image FI output from the camera 20 over time, and this converted captured image CFI is stored in the storage unit 11 (or the storage unit 13). For this reason, each time a new converted captured image CFI is accumulated (or at a predetermined timing different from the timing at which the converted captured image CFI is accumulated), the above process (a process of superimposing the converted captured image CFI on the map image MI). Is executed. Each time a new composite model SM is formed, the conversion unit 15 performs perspective projection conversion on the composite model SM to form a composite image SI. The display unit 17 displays a new composite image SI on the screen every time the composite image SI is formed. Thereby, a part of the composite image SI (a part corresponding to the converted photographed image CFI) is displayed on the screen of the display unit 17 like a moving image.

Projective transformation of the captured image FI will be described with reference to FIGS.
FIG. 4A shows a captured image FI captured by the camera 20. The captured image FI is an image after distortion correction. FIG. 4B shows a map image MI including a shooting location. Points p1 to p4 shown in FIG. 4A correspond to points P1 to P4 (points on FIG. 4B) of the map image MI, respectively. A region surrounded by the points p1 to p4 shown in FIG. 4A is a setting region.

  A captured image FI to be captured, which is captured through a lens such as the camera 20, can be regarded as an image formed by the central projection method. In such a photographed image FI, the geometric figure element on the plane included in the photographed image FI and the figure element on the plane corresponding to the geometric figure element on the map image MI are matched by projective transformation. Can be made.

The projective transformation of the image is expressed by the following equation.
X = (a1 · x + b1 · y + c1) / (a3 · x + b3 · y + c3) (1)
Y = (a2 x x b2 y y c2) / (a3 x x b3 y y c3) (2)
Here, x and y indicate the coordinates of the captured image FI (see FIG. 5A). X and Y indicate the coordinates of the map image MI (see FIG. 5B). a1, b1, c1, a2, b2, c2, a3, b3, and c3 are constants. c3 can be set arbitrarily. The remaining constants (eight parameters) excluding c3 are the four coordinate points p1 to p4 set in the captured image FI and the corresponding four coordinate points P1 to P4 on the map image MI (1). ) And the simultaneous equations of the equations obtained by substituting into the equations (2) are derived.

FIG. 5A shows an example of setting four coordinate points p1 to p4 on the captured image FI.
FIG. 5B shows four coordinate points P1 to P4 corresponding to the four coordinate points p1 to p4 on the captured image FI in the map image MI.

  As shown in FIG. 5A, the four coordinate points on the captured image FI are set so that none of the three points are aligned on a straight line. In addition, the four coordinate points on the captured image FI are set so as to correspond to points (for example, road corners or line ends drawn on the road) represented on the map image MI (FIG. 5). (See (b)). Based on the set point set on the captured image FI in FIG. 5A and the corresponding point on the map image MI in FIG. 1) and equation (2) are derived. These equations are derived in advance, stored in the storage unit 13, and used in the projective transformation unit 12B.

FIG. 6 is a schematic diagram of the synthesis model SM formed by the synthesis unit 14. FIG. 6 shows a first viewpoint FFP and a second viewpoint SFP.
As described above, the composite model SM is configured as a two-dimensional image in a three-dimensional space. The composite model SM shown in FIG. 6 superimposes the two converted photographed images CFI formed based on the photographed images FI output from the two cameras 20 on the two-dimensional map image MI. Is obtained.

  In FIG. 6, the first viewpoint FFP is set as a viewpoint overlooking the composite model SM. The first viewpoint FFP includes a viewpoint position VP (0, 0, α) on the Z axis and away from the origin, a gazing point LAP (0, 0, 0) set at the origin of the three-dimensional coordinate, and the Y axis. It consists of parallel unit vectors EV (0, 1, 0).

  The second viewpoint SFP is set as a viewpoint for viewing the composite model SM from an oblique direction. The second viewpoint SFP includes a viewpoint position VP (β1, β2, β3) set above the portion where the converted photographed image CFI is arranged, and a gazing point set near the portion where the converted photographed image CFI is arranged. It is composed of LAP (γ1, γ2, 0) and unit vector EV (ε1, ε2, ε3).

  FIG. 7 shows the composite image SI viewed from the first viewpoint FFP. The composite image SI is a perspective projection image obtained by viewing the composite model SM formed based on the captured image FI in the urban area from the first viewpoint FFP.

  Since the composite image SI is displayed like a bird's-eye view image, it is possible to grasp the status of the monitoring place (portion taken by the camera 20) on the aerial photograph map. For example, it can be easily grasped from the composite image SI where traffic congestion has occurred.

FIG. 8 shows the composite image SI viewed from the second viewpoint SFP.
This composite image SI is a perspective projection image obtained by viewing the composite model SM that is the basis of the composite image SI in FIG. 7 from the second viewpoint SFP.

  In the bird's-eye view image shown in FIG. 7, it is difficult to grasp details of the monitoring place. This is because the first viewpoint FFP is set at a position far away from the synthesized model SM, so that the portion corresponding to the converted photographed image CFI is displayed in a compressed state in the synthesized image SI. That is, the image information of the converted captured image CFI is not sufficiently displayed in the overhead view image shown in FIG.

  On the other hand, in FIG. 8, since the second viewpoint SFP is set at a position close to the portion where the converted photographed image CFI is arranged, the converted photographed image CFI is compared with the synthesized image SI related to the first viewpoint FFP. The part corresponding to is enlarged and displayed. Thereby, it becomes easy to grasp the details of the monitoring place.

  FIG. 9 shows a composite image SI as seen from the first viewpoint FFP in a place different from the place shown in FIG. This composite image SI is a perspective projection image obtained by viewing the composite model SM formed based on the captured image FI around the building from the first viewpoint FFP.

  The first viewpoint FFP for forming the composite image SI is set above the building. The composite image SI having such settings is configured as an overhead image. For this reason, the user can easily grasp the situation of the monitoring place (portion taken by the camera 20). Conventionally, it is necessary to compare the captured images FI displayed on a plurality of monitors to grasp the situation, and when a plurality of vehicles are moving at the same time, it is difficult to grasp the situation as a whole. However, with this composite image SI, it is easy to grasp the situation as a whole. In addition, it is possible to easily grasp the movement of a vehicle parked far away from the camera 20.

FIG. 10 shows the composite image SI viewed from the second viewpoint SFP.
This composite image SI is a perspective projection image obtained by viewing the composite model SM that is the basis of the composite image SI in FIG. 9 from the second viewpoint SFP. The second viewpoint SFP for forming the composite image SI is set at a position corresponding to the installation height of the camera 20.

  In the composite image SI of FIG. 9, the portion away from the installation position of the camera 20 is greatly distorted, and it is difficult to understand the object to be photographed (vehicle or the like). Therefore, the composite image SI gives the user a sense of incongruity. In this regard, in the composite image SI shown in FIG. 10, since the second viewpoint SFP is set at a position corresponding to the installation height of the camera 20, such distortion is suppressed. For this reason, according to the composite image SI from the second viewpoint SFP, the uncomfortable feeling given to the user can be alleviated.

  Thus, since the image processing system 10 has the second viewpoint SFP that can be set as a viewpoint different from the preset first viewpoint FFP, according to the image processing system 10, the user's request can be made. Can be satisfied. Further, since the composite image SI of the first viewpoint FFP and the composite image SI of the second viewpoint SFP can be selected by the user, the user selects the composite image SI according to the occasional request. Can do. For example, when the user wants to grasp the movement of the vehicle as a whole including the movement of the vehicle parked far away from the camera 20, the first viewpoint FFP can be selected. On the other hand, the second viewpoint SFP can be selected when it is desired to monitor the monitoring location with a natural image without a sense of incongruity.

As described above, the image processing system 10 according to the present embodiment has the following effects.
(1) The image processing system 10 according to the present embodiment includes a conversion unit 15. The conversion unit 15 forms the composite image SI by converting the composite model SM into an image viewed from the first viewpoint FFP set in advance or the second viewpoint SFP set by the operator. In addition, the image processing system 10 includes a selection unit 19 that selects which image is to be displayed on the display unit 17 from the composite image SI related to the first viewpoint FFP and the composite image SI related to the second viewpoint SFP. With this configuration, the image processing system 10 can output the composite image SI of the second viewpoint SFP that meets the user's request.

  (2) In the above embodiment, the image processing system 10 further includes the setting unit 16 for setting the second viewpoint SFP. The setting unit 16 receives viewpoint information regarding the second viewpoint SFP, causes the conversion unit 15 to form a composite image SI that will be obtained when the composite model SM is viewed from the second viewpoint SFP, and generates the composite image SI. The data is output to the display unit 17.

  When the image processing system 10 does not have such a setting unit 16, it is necessary to incorporate a viewpoint that meets the user's request in the image processing system 10 in advance. The viewpoint according to the user's request can be easily set.

  (3) In the above embodiment, in the image processing system 10, the setting unit 16 may associate the viewpoint information related to the second viewpoint SFP with the button (switch) of the selection unit 19 in accordance with a save command based on the operation of the operator. preferable. According to this configuration, it is possible to display the composite image SI from the second viewpoint SFP set in the past by a simple operation.

  (4) In the above-described embodiment, in the image processing system 10, the setting unit 16 captures an image output from the camera 20 to the combining unit 14 while an operation for setting the second viewpoint SFP is being performed. A composite model SM is formed using one image of the image FI.

  While the operation for setting the second viewpoint SFP is being performed, the number of calculation processes executed by the image processing system 10 increases, and thus there is a possibility that a delay may occur in the formation of the composite image SI. In this regard, in the above configuration, the composite model SM is formed using one image of the captured image FI output from the camera 20. As a result, the number of computation processes related to the formation of the composite model SM and the composite image SI within a predetermined period is reduced. As a result, the occurrence of the delay in the formation of the composite image SI is suppressed.

  (5) In the above embodiment, an aerial photograph map image is used as the map image MI (area image) in the image processing system 10. The synthesizing unit 14 performs projective transformation on the photographed image FI so that the setting area having the geometric figure set on the photographed image FI and the part corresponding to the setting area on the map image MI coincide on the outer shape. Thus, the converted photographed image CFI is formed. For this reason, the image shift at the boundary between the converted captured image CFI and the map image MI is reduced.

(Modification)
In addition, the said embodiment can be changed as follows.
-Although the setting part 16 of the said embodiment sets the 2nd viewpoint SFP different from the preset 1st viewpoint FFP, several 2nd viewpoints SFP (viewpoints of a user setting mutually different) are set. You may comprise the setting part 16 as what can be set.

  In the above embodiment, the setting unit 16 sets the second viewpoint SFP to a viewpoint position different from the first viewpoint FFP, but the configuration of the second viewpoint SFP is not limited to this. For example, the second viewpoint SFP according to another embodiment includes the same viewpoint position VP as the first viewpoint FFP, the same gazing point LAP as the first viewpoint FFP, the same unit vector as the first viewpoint FFP, and the display unit. It consists of a clipping range that specifies the part to be displayed. The clipping range is a range set on the composite model with the gazing point LAP as a reference point. In this case, only the viewpoint information different from the first viewpoint FFP is set by the operator. That is, a clipping range can be set.

  In the case of this configuration, the synthesis unit 14 forms a synthesis model SM as in the above embodiment. Then, the conversion unit 15 performs perspective projection conversion of the combined model SM on the projection plane PPP of the second viewpoint SFP according to another form, extracts only the clipping range, and sets the image thus obtained as the combined image SI.

  According to such setting of the second viewpoint SFP, the user can easily expand the predetermined range (clipping range) of the image viewed from the first viewpoint FFP by selecting the second viewpoint SFP. That is, the user can enlarge and visually recognize the main monitoring part (a predetermined range set in advance by the user's designation) without changing the viewpoint direction.

  In the above-described embodiment, the image processing system 10 includes the setting unit 16 for setting a viewpoint for viewing the composite model SM as a preferable example, but the setting unit 16 is different from the image processing system 10 (for example, PC (personal computer)). In this case, this apparatus forms a two-dimensional composite model SM in a three-dimensional space based on the map image MI and the converted photographed image CFI, and displays this on the screen. And this apparatus receives the viewpoint information which an operator inputs similarly to the setting part 16 which concerns on embodiment, displays the input viewpoint which concerns on this viewpoint information on a screen, and also saw from this input viewpoint The composite image SI is displayed. In addition, this apparatus stores viewpoint information by an operation of the operator.

  In this configuration, the image processing system 10 has a function of incorporating viewpoint information as the second viewpoint SFP. The built-in function is a function for associating the second viewpoint SFP with a button or the like to make it available to the user. The viewpoint information for setting the second viewpoint SFP is input directly to the image processing system 10 or by remote operation via the communication network 100. In the case of such a configuration, the image processing system 10 is simplified because the setting unit 16 is not included.

  In the above embodiment, the two buttons displayed on the display unit 17 are used as buttons for switching the viewpoint of the composite model SM, but instead of these buttons, the two keys of the operation unit 18 are used. You can also.

  In the above embodiment, the composite captured image CFI is superimposed on the map image MI in the combining unit 14, but before this superimposition, the map image MI and the converted captured image CFI are visually processed as preprocessing of the map image MI. The map image MI may be subjected to image processing so that it can be identified. For example, the map image MI is converted into an image having a color tone different from that of the converted captured image CFI or an image having a lower resolution than the converted captured image CFI. Thereby, in the composite model SM or the composite image SI formed by performing such preprocessing, the portion of the converted photographed image CFI (that is, the portion monitored by the user or the portion watched by the user) is easily discriminated. It becomes like this.

  In the above embodiment, the synthesis unit 14 forms the two-dimensional synthesis model SM based on the converted photographed image CFI obtained by projective transformation of the photographed image FI and the map image MI (area image). An SM may be formed. For example, an object that models a subject to be photographed is formed in a three-dimensional space. Further, the captured image FI is divided so as to correspond to the polygons constituting the object, and each is subjected to projective transformation to form a converted captured image CFI. On the other hand, the aerial photograph map image (map image MI) is divided so as to correspond to the polygons constituting the object, and the divided pieces of the aerial photograph map image are pasted on predetermined polygons. In this way, a three-dimensional synthesis model SM is configured. The three-dimensional composite model SM is converted into a composite image SI by the converter 15.

  In the present technology, the composite image SI is formed using the captured image FI obtained by the camera 20, but the structure of the camera 20 that forms the captured image FI is not particularly limited. For example, as the camera 20 for forming the captured image FI, there are a monitoring camera, a wide-angle camera, a panoramic camera (for example, a camera having a fisheye lens, a camera that captures an object to be photographed through a curved mirror, and the like). Can be mentioned.

  -The use of this technique is not limited to the application to the monitoring system 1 only. The present technology can be applied to a technology for forming a composite image SI by superimposing a captured image FI captured by the camera 20 on an area image serving as a background. For example, in a game machine, when a landscape image similar to the captured image FI is used as a background image (area image), a problem similar to the problem in the present technology may occur. Therefore, the present technology can be applied in such a video industry field. Even in this case, an effect according to the effect shown in the embodiment can be obtained.

-This technology can also be applied to map search technology in mobile phones and the like.
For example, while displaying an aerial photograph map image (area image) on a display device of a mobile phone, the surroundings are photographed by the camera of the mobile phone, and the surroundings are photographed and displayed on the display device. From the image FI, the user can easily grasp the current position (in particular, his / her own position with respect to the building). In this technique, the composite image SI displayed on the display unit is preferably an image that matches the actual landscape (or cityscape) seen by the user rather than the overhead image. Therefore, it is preferable that there is a display of the composite image SI by the second viewpoint SFP that matches the line of sight of the user looking at the landscape (or the cityscape) separately from the first viewpoint FFP indicating the overhead image. For this reason, the technology according to the present embodiment is preferably applied to the map search technology in a mobile phone or the like.

  DESCRIPTION OF SYMBOLS 1 ... Monitoring system, 10 ... Image processing system, 11 ... Accumulation part, 12 ... Image processing part, 12A ... Distortion correction part, 12B ... Projection conversion part, 13 ... Memory | storage part, 14 ... Composition part, 15 ... Conversion part, 16 ... setting part, 17 ... display part, 18 ... operation part, 19 ... selection part, 20 ... camera, 100 ... communication network.

Claims (5)

By combining the converted photographed image obtained by projective transformation of at least a part of the photographed image output from the surveillance camera installed around the surveillance place and the area image as the background of the photographed image, 3 A synthesis unit that forms a synthesis model corresponding to the monitoring location in a dimensional space;
A conversion unit that forms a composite image by converting the composite model into an image viewed from a preset first viewpoint or a second viewpoint set by an operator;
An image processing system comprising: a selection unit that selects which image is to be displayed on a display unit from among the composite image related to the first viewpoint and the composite image related to the second viewpoint. The converted image obtained by projective transformation of at least a part of the captured image output from the camera of the mobile phone that captures the scenery around the current position of the user and the area image that is the background of the captured image are combined. A synthesis unit that forms a synthesis model corresponding to the scenery at the current position in a three-dimensional space;
A conversion unit that forms a composite image by converting the composite model into an image viewed from a preset first viewpoint or a second viewpoint set by an operator;
An image processing system comprising: a selection unit that selects which image is to be displayed on a display unit from among the composite image related to the first viewpoint and the composite image related to the second viewpoint. A setting unit for setting the second viewpoint;
The setting unit receives the viewpoint information on the second viewpoint, the composite image according to the viewpoint information by forming the conversion unit, according to claim 1 or 2 and outputs the composite image on the display unit Image processing system. The setting unit
Associating the viewpoint information with the switch of the selection unit according to a save command based on the operation of the operator
The image processing system according to claim 3 . The setting unit
While the operation for setting the second viewpoint is being performed, the synthesis unit is configured to form the synthesis model using one of the captured images output from the camera.
The image processing system according to claim 3 or 4 .