JP7770656B2 - Method and system for estimating CCTV camera attitude and object coordinates based on precise road map - Google Patents

Description

The present invention relates to a method and system for estimating CCTV camera pose and object coordinates based on a precise road map.

Currently, thousands of CCTVs have been installed along expressways to monitor traffic conditions, and even in urban areas, CCTVs are installed at many intersections and used for traffic monitoring. In addition, recent advances in AI technology have made it possible to recognize vehicles and people, and to use this information to recognize stopped vehicles, pedestrians, and vehicles driving in the wrong direction.

However, even when dangerous situations are automatically recognized, the operator must make a rough estimate of the location through visual judgment, or in most cases the location cannot be estimated using a global coordinate system of latitude, longitude, and altitude. Even when an estimate is made, there are problems such as large errors, making it difficult to share information.

In particular, until now, most CCTVs have required operators to recognize dangerous situations through visual monitoring, and then share information after roughly estimating the location through visual judgment. This takes an excessive amount of time to generate information, and the accuracy is significantly reduced, making it difficult to share information through mapping.

Korean Patent Publication No. 10-2021-0050997

In order to solve problems such as the difficulty of rough position estimation using the naked eye, the difficulty of estimating global coordinates, and the significantly low accuracy of estimation even if it is done, the present invention aims to provide an automated position coordinate estimation technology for objects and areas in images captured by CCTV cameras and a technology related to automatic mapping using this technology.

In addition to fixed cameras, we will also develop technology that can be applied to rotating PTZ cameras, maximizing the range of applications.

In particular, we propose a method to automatically estimate the camera orientation of CCTV cameras using the precision road map (HD Map) that the government is currently constructing based on major roads, and then use the estimated camera orientation to propose coordinate estimation and mapping technology. In particular, we provide a method to check errors in the coordinate estimation results and use the results to automatically correct position errors and camera orientation.

Therefore, once the construction of precise road maps is completed nationwide, this invention can be applied to automatically estimate the precise location of objects and dangerous areas recognized by AI, automatically map them, and share danger information.

To solve the above problem, one embodiment of the present invention provides a method for estimating CCTV camera pose and object coordinates based on a precise road map, including: (a) matching road objects in an image captured by a CCTV camera with a precise road map to estimate the camera pose; (b) selecting an object to be mapped within the captured image; and (c) estimating the coordinates of the object to be mapped selected within the captured image based on the estimated camera pose, wherein the camera pose includes information on the position (X, Y, Z), pan, and tilt of the CCTV camera.

Step (a) may include extracting pixel coordinates of feature points of the road surface object from the captured image; matching the road surface object with a precise road map to extract spatial coordinates of the feature points of the road surface object; and estimating the camera pose using Perspective-n-Point (PnP) from the relationship between the pixel coordinates and spatial coordinates of the extracted feature points of the road surface object.

If the CCTV camera is a PTZ camera, step (a) may further include converting the pan and tilt values provided by the PTZ camera into absolute values based on true north and the vertical direction based on the difference between the pan and tilt values of the estimated camera attitude and the pan and tilt values provided by the PTZ camera; and reducing the captured image by the reciprocal of the zoom value provided by the PTZ camera to convert it into a captured image with a zoom value of "0".

Step (a) may further include: (a1) projecting the precise road map onto the captured image using the estimated camera pose, or back-projecting an object in the captured image onto the precise road map using the estimated camera pose; (a2) calculating pixel errors of corresponding feature points of the object in the captured image and the object in the precise road map; and (a3) correcting the estimated camera pose to reduce the calculated pixel error to less than the critical value if the calculated pixel error is greater than or equal to a critical value.

In step (b), the mapping target object may be an area or object automatically recognized and extracted from the captured image according to pre-set rules based on AI (artificial intelligence), or an area or object manually selected from the captured image by a user.

Step (b) may include back-projecting each feature point of the mapping target object onto the precise road map using the estimated camera pose; and calculating the spatial coordinates of each back-projected feature point of the mapping target object from the precise road map.

Therefore, the CCTV camera pose and object coordinate estimation method and system according to one embodiment of the present invention provides a method for estimating the CCTV camera pose through a simple method using a precision road map without on-site work such as surveying or setting control points, thereby automating coordinate estimation and mapping, and achieving benefits such as higher accuracy and time and cost savings compared to existing methods.

It can also be applied to fixed CCTVs and PTZ cameras, automating the coordinate estimation and mapping process and estimating coordinates on a precision road map, simplifying and improving accuracy. It can be applied not only when AI is available to recognize road objects such as lanes, but also when such functionality is not available.

In addition, errors in coordinate estimation results and camera attitude are checked and automatically corrected during the estimation process, enabling highly accurate estimation. If road surface objects can be recognized, the entire process can be automated, increasing convenience.

1 is a network configuration diagram of a method and system for estimating CCTV camera pose and object coordinates based on a precise road map according to an embodiment of the present invention; FIG. 2 is a detailed configuration diagram of the information processing server shown in FIG. 1; 1 is a flowchart illustrating a method for estimating a CCTV camera pose and object coordinates based on a precise road map according to an embodiment of the present invention. 4 is a detailed flowchart of the camera pose estimation step shown in FIG. 3. 10 is an exemplary diagram illustrating a fine adjustment procedure after projecting a precise road map onto a captured image. FIG. 4 is a flowchart illustrating steps from a coordinate estimation step to a message generation and distribution step shown in FIG. 3; 10 is an exemplary diagram illustrating the concept of posture correction and mapping error correction through lane designation adjacent to a dangerous area; FIG.

The description of the present invention is merely an example for the purpose of structural or functional explanation, and therefore the scope of the present invention should not be construed as being limited by the example described herein. In other words, since the example may be modified in various ways and have various forms, the scope of the present invention includes equivalents that embody the technical concept. Furthermore, the objectives or effects presented in the present invention do not mean that a particular example must include all of these or only these effects, and therefore the scope of the present invention should not be construed as being limited thereby.

Meanwhile, the meanings of the terms used in this invention shall be understood as follows: Terms such as "first" and "second" are used to distinguish one component from another, and shall not limit the scope of rights. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component. When a component is referred to as being "connected" to another component, it is understood that it may be directly connected to the other component, but that there may be other components between them. Conversely, when a component is referred to as being "directly connected" to another component, it is understood that there are no other components between them. Meanwhile, other expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," shall be interpreted similarly. Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the presence of stated features, numbers, steps, operations, components, parts, or combinations thereof, and are understood not to preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, a system and method for estimating roadside CCTV posture based on a precise road map, estimating AI-recognized object coordinates, and automating dangerous area mapping based on a precise road map, according to one embodiment of the present invention, will be described in more detail with reference to the accompanying drawings.

Figure 1 is a network configuration diagram of a system for estimating CCTV camera pose and object coordinates based on a precise road map according to one embodiment of the present invention. Figure 2 is a detailed configuration diagram of the information processing server shown in Figure 1.

First, as shown in FIG. 1, a precise road map-based CCTV camera pose and object coordinate estimation system 100 according to one embodiment of the present invention includes a CCTV camera 10 and an information processing server 20, and may further include an operator terminal 30.

When each component communicates over a network, the network refers to a connection structure that allows information exchange between nodes such as multiple terminals and servers. Examples of such networks include RF, 3GPP (registered trademark) (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, 5GPP (5th Generation Partnership Project) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), WirelessLAN (Wireless Local Area Network), and WAN (Wide Area Network). Network), PAN (Personal Area Network), Bluetooth (registered trademark) network, NFC network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but are not limited to these.

In the following, the term "at least one" is defined as a term that includes both singular and plural, and it is clear that even if the term "at least one" is not present, each component can exist in singular or plural, and can mean singular or plural. Furthermore, it can be said that whether each component is provided in singular or plural can vary depending on the embodiment.

The CCTV camera 10 may be a PTZ (pan tilt zoom) camera or a fixed camera installed on the side of the road. Here, if the CCTV camera is a PTZ camera, it may be a device that provides information on the camera's position (X, Y, Z), pan, tilt, and zoom.

Next, the information processing server 20 estimates the camera pose (X, Y, Z, Pan, Tilt, Zoom) of the CCTV camera using the image captured by the CCTV camera 10 and a high definition road map (High Definition Map, HD Map). If the CCTV camera is fixed, the estimated camera pose may be used until an error exceeding a critical value is detected. If the CCTV camera is a PTZ camera, the camera pose may be estimated each time the pan, tilt, or zoom is changed.

In addition, the information processing server 20 sets as mapping target objects objects or areas automatically recognized and extracted from the captured image captured by the CCTV camera according to pre-set rules based on AI (artificial intelligence), or areas or objects manually selected by the user from the captured image.

Mapping targets may be moving objects or dangerous areas that conform to rules based on AI (Artificial Intelligence), or they may be objects or dangerous areas that an operator recognizes with the naked eye and are selected as mapping targets.

The information processing server 20 may be configured to automatically estimate the coordinates of characteristic points of mapping targets, such as moving objects and dangerous areas, that are automatically recognized by AI or selected by an operator, based on a precise road map. In the process, the information processing server 20 may correct pre-estimated position errors and camera attitudes using the precise road map, and then generate a message containing surface-level information on the type of danger and prohibited areas, and distribute the message using V2X, mobile communications, etc.

Next, the operator terminal 30 may be configured to provide the information processing server 20 with setting information that sets dangerous areas within the precise road map.

More specifically, the information processing server 20 may include at least one of a camera pose estimation unit 21, a mapping target object setting unit 22, a coordinate estimation unit 23, a correction unit 24, and a message generation and distribution unit 25.

The camera posture estimation unit 21 performs the role of estimating the camera posture by matching road surface objects in the image captured by the CCTV camera with the corresponding objects in the precise road map. Here, the camera posture includes information on the CCTV camera's position (X, Y, Z), pan, and tilt, and may further include information on zoom.

The camera posture estimation unit 21 extracts pixel coordinates of feature points of road surface objects in the captured image, matches the road surface objects with a precise road map to extract spatial coordinates of each feature point of the road surface objects, and estimates the camera posture using PnP (Perspective-n-Point) based on the relationship between the pixel coordinates and spatial coordinates of each extracted feature point of the road surface objects.

PnP (Perspective-n-Point) is a method of estimating camera posture based on the relative positions of three or more 3D points projected onto a 2D plane. In this invention, three or more feature points are extracted from road surface objects, and the camera posture can be estimated using PnP (Perspective-n-Point) based on the relationship between the pixel coordinates of the feature points and the spatial coordinates.

Meanwhile, in the present invention, a feature point refers to a point that can exhibit characteristics that can distinguish objects present in an image, and can be a corner point or a point that can be distinguished from its surroundings by hue or brightness.

If the CCTV camera is fixed, the camera position (X, Y, Z), pan, tilt, and zoom do not change, so the initially estimated camera posture can continue to be used. Of course, even in the case of a fixed camera, the camera may shake for various reasons or there may be errors in the initially estimated camera posture, so the correction unit 24, described below, performs corrections to the camera posture through various processes.

If the CCTV camera is a PTZ camera, the initial estimated camera posture can continue to be used because the posture can be maintained unless pan, tilt, or zoom is performed. If at least one of pan, tilt, and zoom operations is performed, the corresponding operation value can be obtained, so the camera posture estimation unit 21 can obtain a modified camera posture by considering the PTZ (PanTilt Zoom) value provided by the camera in addition to the initial camera posture. Since the camera installation position is fixed, the camera position (X, Y, Z) continues to be maintained unless correction is required.

On the other hand, if the CCTV camera is a PTZ camera, the camera attitude estimation unit 21 can easily obtain the PT value from the difference between the pan and tilt values of the estimated camera attitude and the pan and tilt values provided by the PTZ camera by converting the pan and tilt values provided by the PTZ camera into absolute values based on true north and the vertical direction. Also, if the CCTV camera is a PTZ camera, the camera attitude estimation unit 21 can perform a process of reducing the captured image by the reciprocal of the zoom value provided by the PTZ camera to convert it into a captured image with a zoom value of "0".

Meanwhile, the correction unit 24 can correct information regarding the camera pose and/or information regarding the coordinates of the object to be mapped.

The correction unit 24 projects a precise road map onto the captured image using the estimated camera pose, calculates the pixel error of each feature point of the object in the captured image and the corresponding object in the precise road map, and if the calculated pixel error is above a critical value, corrects the camera pose and repeats the above process to reduce the pixel error to below the critical value.

Of course, the correction unit 24 can correct the pose by reversing the above process, but it can also back-project the objects in the captured image onto a precise road map using the estimated camera pose, calculate the position error (coordinate error) of each feature point of the back-projected objects in the captured image and the corresponding objects in the precise road map, and if the calculated position error is greater than or equal to a critical value, correct the camera pose and repeat the above process to reduce the pixel error to less than the critical value.

The correction unit 24 can minimize camera attitude and position errors by calculating at least one of pixel error and position error (coordinate error) and repeating the process of finely correcting the camera attitude to reduce the corresponding error below a critical value.

In the present invention as a whole, the correction process of the correction unit 24 may be one of the following: i) a method of projecting a precise road map onto the captured image using the estimated camera attitude, and then correcting using pixel errors between objects on the precise road map and corresponding image objects; or ii) a method of back-projecting the captured image onto the precise road map using the estimated camera attitude, and then correcting using coordinate errors between objects on the precise road map and corresponding image objects. The correction processes i) and ii) are inversely related, and the correction methods and results are the same, so either one of the two methods or both of them may be used.

The mapping target object setting unit 22 sets, as mapping target objects, objects or areas that are automatically recognized and extracted from the captured image based on AI (artificial intelligence) in accordance with pre-set rules, or objects or areas that are manually selected from the captured image by a user (operator).

The mapping target object setting unit 22 may be configured to automatically estimate the coordinates of objects such as vehicles and people recognized based on AI such as deep learning, in addition to road surface objects such as lanes, based on a precise road map, or to automatically map automatically recognized hazardous elements or areas on the road occupied by the hazardous elements using AID (Automatic Incident Detection) as mapping target objects.

The operation of mapping recognized objects such as vehicles and people, or dangerous areas recognized by AID, is as follows:

The system extracts pixel coordinates of feature points (three or more feature points) of road surface objects such as lanes and recognizes mapping targets other than road surface objects (including AID-based hazardous areas). Then, it back-projects the mapping target objects and feature points of the road surface objects onto a precise road map, or calculates the position error between the back-projected road surface objects and the corresponding objects on the precise road map. It then calculates the spatial coordinates of the mapping target feature points using the precise road map and corrects the spatial coordinates of the mapping target feature points. Then, it re-projects the mapping target feature points onto the image, and calculates the pixel error for each re-projected feature point based on the initially set point. If the pixel error is smaller than a critical value, it displays the mapping target (position, attributes) on the precise road map, and automatically generates and displays a message.

Here, if the pixel error is above a critical value, the pose (pan, tilt) can be corrected using the position error for each feature point.

In addition, the mapping target object setting unit 22 can set objects or dangerous areas set or designated by the operator as mapping target objects. To this end, the mapping target object setting unit 22 recognizes road surface objects on the precision road map, such as lanes and arrows, and extracts pixel coordinates of feature points (three or more feature points) of the recognized road surface objects.

When the operator inputs the settings and attributes of a danger zone (three or more feature points) in the captured image, the coordinate estimation unit 23, described below, back-projects the danger zone and feature points of road surface objects onto a precise road map and can calculate the spatial coordinates of the feature points of the danger zone using the precise road map.

Meanwhile, the correction unit 24 calculates the position error between the back-projected road surface object and the same object on the precise road map, then corrects the spatial coordinates of the feature points in the dangerous area to reflect the position error, re-projects the feature points in the dangerous area onto the captured image, and calculates the pixel error for each re-projected feature point based on the minimum set point. Once the calculation is complete, if the pixel error is smaller than a critical value, the message generation and distribution unit 25 may be configured to mark the dangerous area (location, attributes) on the precise road map and automatically generate and display a message.

Here, if the pixel error is greater than or equal to a critical value, the correction unit 24 can correct the posture (pan, tilt) using the position error for each feature point.

The coordinate estimation unit 23 can automatically estimate the coordinates of feature points for mapping target objects (objects recognized by AI, danger areas recognized by AID, objects or danger areas selected by the operator) based on a precise road map.

The correction unit 24 may be configured to correct at least one of pixel error, position error, and camera attitude that are pre-estimated using a precise road map during the camera attitude estimation process or the coordinate estimation process of the mapping target object.

The correction unit 24 may be configured to correct camera attitude and position errors in the process of estimating or determining objects recognized by AI, such as vehicles, people, and signs, danger areas recognized by AID, or danger areas set by an operator.

The correction unit 24 projects the precise road map onto the captured image and then automatically determines whether to perform correction based on a preset criterion, which is based on the positional error between road surface objects such as lanes in the captured image and the road surface objects on the projected precise road map. At this time, the recognized road surface objects are back-projected onto the precise road map, or the same objects on the precise road map are projected onto the captured image, and then the coordinates of the objects to be mapped and dangerous areas are automatically corrected through position comparison (coordinate comparison), and the posture is also corrected.

Even in the case of a fixed system, there is a high possibility that rotation errors will occur due to displacement over time, so accuracy can be maintained through correction at each mapping. Once the initial camera posture estimation is complete, the position (X, Y, Z) is unlikely to change, so it is assumed that there is no error, and the error types can be classified as Pan and Tilt.

Left-right errors can be corrected by fine-tuning Pan, and up-down errors can be corrected by fine-tuning Tilt. After calculating the magnitude of the error and determining the adjustment unit, this can be repeated until an error in the opposite direction occurs, and then fine-tuning the opposite direction to 1/10 of the original unit, and then repeating this process until an error in the opposite direction occurs again. At each stage, a comparison with a threshold is performed, and if satisfied, the process can be terminated.

However, if there is an error in the PT value when calculating coordinates on the precise road map after back-projecting the mapping object or danger zone designated by the operator using only the PTZ value provided by the camera, there is a possibility that the coordinate estimation results will have a large error.

These errors can be checked by estimating the coordinates of the dangerous area on a precise road map, projecting it onto the captured image along with surrounding road objects, and then visually comparing the position of the road object on the precise road map projected onto the captured image with the position of the corresponding road object in the existing captured image. If the operator determines that a significant error has occurred after visually inspecting it, they can mark the start and end points of the road object in the captured image that match the projected road object. This can then be used to automatically correct the camera posture, and the coordinate estimation process can be performed again to further correct the error.

Even with fixed systems, there is a high possibility of rotational errors occurring due to displacement over time, so accuracy can be maintained through corrections made each time mapping is performed.

In other words, once initial camera pose estimation is complete, the position (X, Y, Z) is unlikely to change, so we can assume there is no error, and classify the types of error into Pan and Tilt.

Left-right errors can be corrected by fine-tuning Pan, and up-down errors by fine-tuning Tilt. The correction procedure involves calculating the magnitude of the error, then determining the adjustment unit → repeating until an error in the opposite direction occurs → fine-tuning in the opposite direction to 1/10 of the original unit → repeating until an error in the opposite direction occurs again. At each stage, a comparison with a threshold is made, and if satisfied, the process is terminated.

The message generation and distribution unit 26 may be configured to generate messages containing information such as the type of danger and surface-level information on prohibited areas, and distribute these messages using V2X, mobile communications, etc.

Figure 3 is a flowchart illustrating a method for estimating CCTV camera pose and object coordinates based on a precise road map according to one embodiment of the present invention, Figure 4 is a detailed flowchart of the camera pose estimation step shown in Figure 3, Figure 5 is an example diagram illustrating a fine adjustment procedure after projecting a precise road map onto a captured image, and Figure 6 is a flowchart illustrating the coordinate estimation step through the message generation and distribution step shown in Figure 3. Figure 7 is an example diagram illustrating the concept of pose correction and mapping error correction through lane designation adjacent to a dangerous area.

First, as shown in FIG. 3, a method for estimating CCTV camera pose and object coordinates based on a precise road map according to one embodiment of the present invention (S700) includes at least one of a camera pose estimation step (S710), a mapping target object setting step (S720), a coordinate estimation step (S730), an error correction step (S740), and a message generation and distribution step (S750).

The camera attitude estimation step (S710) may be a step of estimating the camera attitude (X, Y, Z, Pan, Tilt) of the CCTV camera, determining the position (X, Y, Z) of the PTZ camera, and determining a relational equation (difference) for converting the Pan/Tilt values provided by the PTZ camera into absolute values (true north direction, horizontal reference).

The camera pose estimation referred to in this application is a technology that attempts to estimate the CCTV camera pose through a simple process using a precision road map without on-site work such as surveying, setting control points, and marking. It provides a foundation for automating coordinate estimation and mapping, which will be described later, and can be a technology that offers benefits such as high accuracy and time and cost savings compared to existing methods. Therefore, in the case of a PTZ camera, since the device provides relative values based on the initial setting position, it is not possible to immediately convert them to the camera pose (Pose). Taking this into consideration, this technology can convert the PTZ values provided subsequently and use them as the camera pose (Pose) by calculating and reflecting the difference between the value provided when estimating the initial camera pose (Pose) and the true value.

Next, the mapping target object setting step (S720) may include at least one of AI-based mapping target object recognition, AID (Automatic Incident Detection)-based object or danger area recognition, and a step of designating an object or area that an operator recognizes as a dangerous situation with the naked eye as a danger area.

Next, the coordinate estimation step (S730) may be a step of automatically estimating and mapping the coordinates of feature points based on a precise road map for the object to be mapped (objects or areas recognized by AI, objects or dangerous areas recognized by AID, objects or dangerous areas manually set by an operator, etc.).

The coordinate estimation of the object to be mapped can be applied when using not only fixed CCTV but also PTZ cameras. It is a method that can automate the coordinate estimation and mapping process, and by estimating coordinates on a precise road map, it is simple and highly accurate. In particular, it can be applied not only when it is possible to recognize road surface objects such as lanes using AI (Artificial Intelligence), but also when such functionality is not available.

Next, the error correction step (S740) may be a step of correcting the position, pixel error, and camera posture previously estimated using a precise road map in the camera posture estimation step (S710) and the coordinate estimation step (S730) of the mapping target object.

The position, pixel error, and/or attitude correction step is a method that enables highly accurate estimation by checking and automatically correcting errors in the coordinate estimation results and the camera attitude applied at that time throughout the entire estimation process. In particular, if it is possible to recognize road objects such as lanes, automating the entire process can increase convenience. Even if such functionality is not available, this technology can achieve the same effect with just a simple operation by the operator.

Next, the message generation and distribution step (S750) may be a step of automatically generating and distributing risk-related messages.

When sharing information about the mapping target object created based on the above process, messages are automatically generated to eliminate additional manual work by the operator, and the results are displayed on a monitor managed by the operator so that they can be checked. This technology also allows for automatic distribution.

Below, we'll explain each of the above steps in more detail.

First, referring to FIG. 4, the initial camera pose estimation step (S710) extracts pixel coordinates of feature points of road objects in CCTV-captured images, matches them with a precise road map to extract spatial coordinates of the feature points of the road objects, estimates the camera pose through Perspective-n-Point (PnP), and then projects the precise road map onto the captured image using the estimated camera pose. Then, pixel errors within the image are calculated for each feature point, and the pixel errors are compared with a threshold.

As mentioned above, PnP (Perspective-n-Point) refers to a technique for obtaining the exact position and orientation of a camera that captured an image relative to a 3D coordinate system based on the coordinates of 3D feature points whose coordinate positions are known and the 2D coordinates of feature points on the image onto which the corresponding feature points are projected.

If the error is less than the threshold, the process ends. However, if the error is greater than the threshold, the camera posture is finely corrected taking into account the error characteristics, and a precise road map is projected onto the captured image using the corrected camera posture. The process can then be repeated until the pixel error in the image for each feature point falls below the threshold.

Meanwhile, the camera pose estimation step (S710) can be performed as follows after estimating the primary pose through PnP.

For example, the method may include back-projecting the feature points of the road surface object onto a precise road map based on the estimated camera pose, calculating the position error between the spatial coordinates of the back-projected feature points of the road surface object and the corresponding spatial coordinates of the object on the precise road map for each feature point, and terminating the process if the error is less than a threshold. If the error is greater than or equal to the threshold, finely correcting the estimated camera pose in consideration of the error characteristics, and then back-projecting onto the precise road map using the corrected pose. Again, the process may be repeated until the position error in the space for each feature point becomes less than the threshold.

Here, in PnP-based primary posture estimation, if a rotating PTZ camera is used, it is efficient to set the camera roughly perpendicular to the road.

Based on the roughly known position information of the CCTV camera and information on objects on the road surface or roadside (shown on a precise road map), the process involves comparing the precise road map with road surface objects such as lanes, searching for matching pairs between the image and the precise road map with the naked eye using the rough position information and surrounding objects (surrounding objects: traffic lights, signs, road markings other than lanes, start and end points of bridges, start and end points of protective facilities, etc.), extracting pixel coordinates and spatial coordinates for three or more matching pairs, and then performing PnP.

In addition, the process of fine-tuning the pose taking into account error characteristics may involve estimating the primary camera pose based on PnP, projecting a precise road map onto the captured image using this camera pose, calculating the pixel error between object feature points to confirm the degree of error, and finally correcting the camera pose through fine adjustments that reflect the error characteristics.

Alternatively, it may be a method of back-projecting feature points of specific objects in the captured image onto a precise road map using the camera pose, and then correcting the camera pose based on the positional error between the feature points of the corresponding objects.

In the camera posture estimation step (S710), if the camera is a PTZ camera, camera calibration is performed to determine true values, and then the difference between the provided Pan and Tilt values and the true values is converted into absolute values. In the case of zoom, the provided values are used to determine the enlargement ratio, and the image is reduced by the reciprocal of this ratio, thereby converting it into an image with a zoom value of '0'. This can include a process of continuously correcting the difference using the corrected camera posture values during the automated mapping process.

Meanwhile, the mapping target object setting step (S720) may vary depending on the method of setting recognized objects such as vehicles and people recognized based on AI, objects and danger areas recognized by AID, objects and danger areas set by an operator, etc.

First, referring to Figure 6, pixel coordinates of characteristic points of road surface objects such as lanes are extracted, and mapping targets other than road surface objects (including AID-based danger areas) are recognized.

Then, the feature points of the object to be mapped and the road surface object are back-projected onto the precise road map to calculate the spatial coordinates of the feature points of the object to be mapped, and the position error between the back-projected object and the same object on the precise road map is calculated, and the spatial coordinates of the feature points of the object to be mapped are corrected. Then, the feature points of the object to be mapped are re-projected onto the image, and the pixel error for each re-projected feature point is calculated based on the initially set point. If the pixel error is smaller than a threshold, the method may include the steps of notating the object to be mapped (position, attributes) on the precise road map and automatically generating and displaying a message.

Here, if the pixel error is greater than or equal to a critical value, the method may further include a process of correcting the camera pose (pan, tilt) using the position error for each feature point.

When an object such as a vehicle, person, fallen object, or animal is recognized, the coordinates of the object are automatically estimated. When a dangerous area such as an accident, wrong-way driving, or traffic jam is automatically recognized using AID (Automatic Incident Detection), the coordinates of the dangerous area are automatically estimated and the results are reflected in the automatic mapping.

Here, AI-based recognition targets include objects requiring coordinate estimation, moving objects such as vehicles, people, and animals, atypical obstacles such as wrong-way vehicles and fallen objects, the tail of traffic jams, traffic cones (registered trademark), precast barriers, and other construction-related objects.

Dangerous areas requiring coordinate estimation may be accident areas identified by AID, construction zones identified using a series of traffic cones (registered trademark) or precast barriers as reference points, or other dangerous areas identified by surface features.

Furthermore, road surface objects for error detection and posture correction may be road surface objects included in a precision road map, such as lanes, arrows, and stop lines (including signboards and traffic lights, if necessary).

Here, if the initial camera posture estimation state is a PTZ camera, the difference between the provided value and the true value is known, so it may be possible to estimate the absolute values (true values) of Pan and Tilt in real time even during rotation.

In addition, position and orientation error correction is performed by projecting a precise road map onto the captured image, and then automatically determining whether or not to correct it based on pre-set criteria based on the position error between road objects in the captured image and those on the projected precise road map. This includes a function to back-project road objects recognized in the image onto the precise road map, or to project the same objects on the precise road map onto the captured image, and then automatically correct the coordinates of the objects being mapped and dangerous areas by comparing the positions of the corresponding objects, as well as correcting their orientation.

Even in the case of a fixed system, there is a high possibility that rotation errors will occur due to displacement over time, so accuracy can be maintained through corrections at each mapping. Once the initial camera posture estimation is complete, there is little possibility of position (X, Y, Z) changes, so it is assumed that there are no errors, but the types of errors can be classified as Pan and Tilt.

Left-right errors can be corrected by fine-tuning Pan, and up-down errors by fine-tuning Tilt. The correction procedure involves calculating the magnitude of the error and determining the adjustment unit, then repeating the process until an error in the opposite direction occurs, fine-tuning the adjustment in the opposite direction to 1/10 of the original unit, and then repeating the process again until an error in the opposite direction occurs. At each stage, a comparison with a threshold is performed and the process ends when a satisfactory result is reached.

When an object such as a vehicle, person, fallen object, or animal is recognized, the coordinates of the object are automatically estimated. When a dangerous area such as an accident, wrong-way driving, or traffic jam is automatically recognized using AID (Automatic Incident Detection), the coordinates of the dangerous area are automatically estimated and the results are reflected in the automatic mapping.

AI-based recognition targets are objects that require coordinate estimation, and include moving objects such as vehicles, people, and animals (including atypical obstacles such as wrong-way vehicles and fallen objects, the tail of traffic jams, traffic cones (registered trademark), precast barriers, and other construction-related objects).

In addition, danger areas that require coordinate estimation may be accident areas recognized by AID, construction areas recognized using a series of traffic cones (registered trademark) or precast barriers as reference points, or other danger areas recognized by surface.

On the other hand, if a PTZ camera is used in the camera posture estimation process, the difference between the provided value and the true value is known, so it may be possible to estimate the absolute values (true values) of Pan and Tilt in real time even during rotation.

In addition, during the error correction process, the mapping object or danger zone designated by the operator is back-projected onto a precise road map using only the provided PTZ value, and then when calculating coordinates on the precise road map, if there is an error in the PT value, there is a possibility that a large error will occur in the coordinate estimation results.

Such errors can be corrected by estimating the coordinates of the danger zone on a precise road map, then re-projecting the surrounding road objects onto the captured image using the estimated camera attitude. The operator can then visually compare the position of the re-projected road object in the image with the position of the corresponding identical road object in the existing captured image to check the magnitude of the error. If the operator determines that the error is significant after visually inspecting it, they can mark the start and end points of the road object in the captured image that correspond to the re-projected road object, which can then be used to automatically correct the camera attitude. Using this, the coordinate estimation process can be performed again to correct the coordinate estimation error.

Even with fixed systems, there is a high possibility of rotational errors occurring due to displacement over time, so accuracy can be maintained through corrections made each time mapping is performed.

In other words, once the initial camera pose estimation is complete, there is almost no possibility of the position (X, Y, Z) changing, so it can be assumed that there is no error, and the error type can be classified as Pan and Tilt.

Left-right errors can be corrected by fine-tuning Pan, and up-down errors by fine-tuning Tilt. The correction procedure involves calculating the magnitude of the error and determining the adjustment unit → repeating until an error in the opposite direction occurs → fine-tuning in the opposite direction to 1/10 of the original unit → repeating until an error in the opposite direction occurs again. At each stage, a comparison with a threshold is performed, and if satisfied, the process is terminated.

Meanwhile, an operator can set a danger zone (three or more feature points) within the captured image or input attributes (operator). In this case, the start and end points of adjacent road objects adjacent to the danger zone can be set, and the feature points of the danger zone and adjacent road objects can be back-projected onto a precise road map. The spatial coordinates of the feature points of the danger zone and adjacent road objects can then be calculated using the precise road map. At this time, the position error between the back-projected adjacent road objects and their corresponding objects on the precise road map can also be calculated. The spatial coordinates of the feature points of the danger zone are then corrected using the position error, and the feature points of the danger zone are re-projected onto the captured image. The pixel error for each re-projected feature point is calculated based on the initially set point. If the error is smaller than a critical value, the system proceeds to the process of notating the mapping target's danger zone (location, attributes) on the precise road map and automatically generating and displaying a message.

Here, if the error is greater than a critical value, the method may further include a process of correcting the camera pose (pan, tilt) using the position error for each feature point.

A method for estimating CCTV camera pose and object coordinates based on a precise road map according to one embodiment of the present invention provides a method for estimating the camera pose of a CCTV camera through a simple method using a precise road map without on-site work such as surveying, setting control points, and marking. This provides a foundation for automating coordinate estimation and mapping, and has the advantages of achieving higher accuracy and time and cost savings compared to existing methods.

In addition, in the case of a PTZ camera, since the device provides relative values based on the initial setting position, it is not possible to immediately convert them to camera posture. Taking this into consideration, the difference between the value provided during initial posture estimation and the true value is calculated and reflected, which has the advantage that PTZ values provided thereafter can be converted and immediately used as camera posture.

It can also be applied to cases where not only fixed CCTVs but also PTZ cameras are used, and it presents a method that can automate the coordinate estimation and mapping process, estimating coordinates on a precision road map, which is simple and improves accuracy. It has the advantage of being applicable not only when it is possible to recognize road objects such as lanes using AI, but also when such functionality is not available.

Another advantage is the ability to automate error and attitude correction. For example, errors in the coordinate estimation results and the camera attitude applied at that time are checked and automatically corrected throughout the entire estimation process, enabling highly accurate estimation. In particular, when road surface objects such as lanes can be recognized, the entire process can be automated, increasing convenience. Even if such functionality is not available, an algorithm can be implemented to achieve the same effect with simple operator operations.

Another advantage is that messages can be automatically generated and distributed. For example, messages can be generated automatically to share mapping information created based on the above process, eliminating the need for additional manual work by the operator. The results can be displayed on a monitor managed by the operator, allowing them to be checked, and distribution is also automatic.

Meanwhile, the components disclosed herein may be implemented using one or more general-purpose or special-purpose computers, including, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the description may assume that a single processing device is used. However, those skilled in the art will recognize that a processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. Software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, to be interpreted by the processing device or to provide instructions or data to the processing device. Software may also be distributed across computer systems connected to a network, where it is stored and executed in a distributed manner. Software and data may be stored on one or more computer-readable storage media.

Methods according to embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments, or may be readily available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform the operations of an embodiment, and vice versa.

The present invention has been described in detail above with reference to examples. However, it will be apparent to those skilled in the art that various substitutions, additions, and modifications may be made without departing from the technical concept described above, and such modified embodiments also fall within the scope of the present invention as defined by the following claims.

100: System 10: CCTV
20: Information processing server 21: Camera posture estimation unit 22: Mapping target object setting unit 23: Coordinate estimation unit 24: Correction unit 25: Message generation and distribution unit 30: Operator terminal

Claims (4)

(a) estimating the camera posture by matching road objects in an image captured by a CCTV camera with a precise road map;
(b) selecting an object to be mapped within the captured image; and
(c) estimating coordinates in the precise road map of a mapping target object selected in the captured image based on the estimated camera pose;
The camera posture includes information on the position (X, Y, Z), pan, and tilt of the CCTV camera,
The step (a) comprises:
(a1) projecting the precise road map onto the photographed image using the estimated camera pose, or back-projecting an object in the photographed image onto the precise road map using the estimated camera pose;
(a2) calculating pixel errors of corresponding feature points of the object in the captured image and the object in the precise road map; and
(a3) if the calculated pixel error is equal to or greater than a critical value, correcting the estimated camera pose to reduce the calculated pixel error to less than the critical value;
The step (c)
back-projecting each feature point of the mapping target object onto the precise road map according to the estimated camera pose; and
and calculating spatial coordinates of each feature point of the back-projected mapping target object from the precise road map.
A method for estimating CCTV camera attitude and object coordinates based on a precise road map. The step (a) comprises:
extracting pixel coordinates of feature points of the road surface object from the captured image;
extracting spatial coordinates of feature points of the road surface object by matching the road surface object with a precise road map; and
2. The method of claim 1, further comprising: estimating the camera pose using Perspective-n-Point (PnP) from a relationship between pixel coordinates and spatial coordinates of the extracted road surface object feature points. If the CCTV camera is a PTZ camera, step (a) includes converting the pan and tilt values provided by the PTZ camera into absolute values based on true north and a vertical direction from the difference between the pan and tilt values of the estimated camera attitude and the pan and tilt values provided by the PTZ camera; and
3. The method of claim 2, further comprising: reducing the captured image by an inverse of the zoom value provided by the PTZ camera to convert it into a captured image with a zoom value of '0'. In the step (b),
2. The method of claim 1, wherein the mapping target object is an area or object that is automatically recognized and extracted from the captured image according to a predetermined rule based on artificial intelligence (AI), or an area or object that is manually selected from the captured image by a user.