JP4876117B2 - Image information processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system or the like for enabling an image display device to stably display thereon image information of an object to be recognized through an infrared camera even if an image region of the relevant object is unstable. <P>SOLUTION: In an image information processing system 100 of the present invention, on the basis of a second parameter X2 corresponding to a position of an object measured through an infrared camera 210, a first image region recognizing unit 121 recognizes a first image region A1 corresponding to the relevant object. Furthermore, on the basis of a first parameter X1 corresponding to a traveling state of an automobile 200 and the second parameter X2, a second image region recognizing unit 122 estimates the first image region A1 and then recognizes this as a second image region A2. An image information control unit 140 then displays on a head-up display (HUD) 220 combined image information corresponding to a combined image region as a combination result of the first image region A1 and the second image region A2. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a system for processing an image of an object photographed by an infrared camera mounted on a moving body and displaying image information of the object on an image display device mounted on the moving body.

2. Description of the Related Art Conventionally, a technique has been proposed in which an image area corresponding to an object is displayed on an image display device mounted on the vehicle based on an image of the object (human) taken by an infrared camera mounted on the vehicle. (For example, see Patent Document 1).
JP, 2003-134508, A Claim 1

  However, it is displayed on the image display device due to the characteristics of the infrared camera, the relative movement of the car and the object being relatively steep, and the object being hidden behind other objects. In some cases, the image area becomes unstable, such as its position, size, and number fluctuate rapidly.

  For example, as shown in FIG. 5A, in the state where the rectangular image area A1 corresponding to the human O1 that is the object is highlighted on the image display device, the automatic O2 near the human O1. When the light from the headlamp intensifies, the image area A1 corresponding to the headlamp may be highlighted as shown in FIGS. 5B and 5C. Further, when the relative movement between the automobile and the human O1 is abrupt, such as when the automobile is running while turning left relatively quickly in the state shown in FIG. 5A, the relative movement of the human O1 is included in this relative movement. The image information A1 corresponding to the human O1 may disappear from the image display device without being able to follow up the recognition. Further, since a part of the body of the human O1 is hidden behind the automobile O2, the frame A1 highlighted on the image display device as shown in FIG. 5D includes a part of the human body. The image area A1 may disappear from the image display device as shown in FIG. 5 (e). If the image area A1 highlighted on the image display device is unstable in this way, there is a risk that the driver, such as a car driver, may feel anxiety.

  Accordingly, the present invention provides a system that can stably display image information of an object on an image display device even if the image area of the object recognized through the infrared camera is unstable. Let it be an issue.

  An image information processing system according to a first aspect of the present invention processes an image of an object photographed by an imaging means mounted on a moving body and displays image information of the object on an image display device mounted on the moving body. A first variable measuring unit that measures a first variable according to a traveling state of the moving body, and a second variable measurement that measures a second variable according to the position of the object through the imaging unit. And first image area recognition means for recognizing an image area corresponding to the object as a first image area based on the second variable measured by the second variable measurement means, and the first variable measurement means Based on the first variable measured by the second variable measurement unit and the second variable measured by the second variable measurement unit, the image region recognized by the first image region recognition unit is estimated, and the second image region is determined. recognize A composite image obtained by synthesizing the two image area recognition means, the first image area recognized by the first image area recognition means, and the second image area recognized by the second image area recognition means; And a composite image region recognition unit that recognizes the region.

  The image information processing system of the second invention is the image information processing system of the first invention, wherein the first image information and the second image region recognition according to the first image region recognized by the first image region recognition means. At least one of the second image information corresponding to the second image area recognized by the means, or the composite image information corresponding to the composite image area recognized by the composite image area recognition means is stored in the image display device. An image information control means for displaying is further provided.

  The image information processing system according to a third aspect of the present invention is the image information processing system according to the second aspect, wherein the image information control means displays the composite image information when the first image area and the second image area overlap. The image is displayed on the image display device.

  The image information processing system according to a fourth aspect of the present invention is the image information processing system according to the first aspect, wherein the composite image area recognition means reflects the first image area to a degree corresponding to a first weighting factor, and An image area synthesized so that the two image areas are reflected to the extent corresponding to the second coefficient is recognized as the synthesized image area.

  The image information processing system according to a fifth aspect of the present invention is the image information processing system according to the fourth aspect, wherein the composite image area recognition means is configured to determine the difference between the previous first image area and the current first image area. A first weighting factor is determined, and the second weighting factor is determined based on a deviation between the previous first image region and the current second image region.

  An image information processing system according to a sixth aspect of the present invention is the image information processing system according to the second aspect, wherein the image control means causes the image display device to display the first image information when the first image area is stable. On the other hand, when the first image area is unstable, the second image information or the composite image information is displayed on the image display device.

  The image information processing system according to a seventh aspect of the present invention is the image information processing system according to the sixth aspect, wherein the image information control means determines whether the instability of the first image area is large or small when the first image area is unstable. If the instability is determined to be small, the composite image information is displayed on the image display device. On the other hand, if the instability is determined to be large, the second image information is displayed as the image. It is displayed on a display device.

  An embodiment of an image information processing system of the present invention will be described with reference to the drawings.

  An information providing system 100 shown in FIG. 1 is mounted at an appropriate position of an automobile (mobile body) 200. The automobile 200 includes a vehicle speed sensor 201 that outputs a signal corresponding to the vehicle speed v, and a yaw rate sensor 202 that outputs a signal corresponding to the angular velocity ω of the automobile 200 around the vertical axis passing through the rear wheel axis. In addition, two left and right infrared cameras (imaging means) 210 are mounted on the front portion of the automobile 200. Further, an HUD (Head Up Display) (image display device) 220 is mounted on the front window of the automobile 200 at a position that does not hinder the driver's forward view.

  The image information processing system 100 includes a first variable measurement unit 111, a second variable measurement unit 112, a first image region recognition unit 121, a second image region recognition unit 122, a composite image region recognition unit 130, an image And an information control unit 140. Each unit includes a CPU, ROM, RAM, and the like as hardware, and an “image information processing program” as software that gives an execution function of an image information processing method to be described later to the hardware.

  The first variable measuring unit 111 is a first variable according to the running state of the vehicle 200 based on the outputs of the vehicle speed sensor 201 and the yaw rate sensor 202, and the vehicle speed v and the angular velocity ω of the automobile 200 around the vertical axis passing through the rear wheel axis. Measure as X1. The second variable measurement unit 112 determines the distance to the object relative to the infrared camera 210 as well as the vertical and horizontal position and size of the object through one or both of the two infrared cameras 210 according to the position of the object. Measured as the second variable X2. The first image area recognition unit 121 recognizes the image area corresponding to the object as the first image area A1 based on the second variable X2 measured by the second variable measurement unit 112. The second image area recognition unit 122 is based on the first variable X1 measured by the first variable measurement unit 111 and the second variable X2 measured by the second variable measurement unit 112. Is estimated as the second image area A2. The composite image area recognition unit 130 combines the first image area A1 recognized by the first image area recognition unit 121 and the second image area A2 recognized by the second image area recognition unit 122, and then combines them. Is recognized as a composite image area A.

  The image information control unit 140 causes the HUD 220 to display the first frame surrounding the first image area A1 recognized by the first image area recognition unit 121 as “first image information”. Further, the image information control unit 140 causes the HUD 220 to display the second frame surrounding the second image area A2 recognized by the second image area recognition unit 122 as “second image information”. Further, the image information control unit 140 causes the HUD 220 to display a frame surrounding the composite image area A recognized by the composite image area recognition unit 130 as “composite image information”.

  An image information processing method according to the first embodiment of the present invention executed by the image information processing system having the above-described configuration will be described with reference to FIGS.

  First, the number n of control cycles is reset to “1” at a predetermined timing (S101), and the cumulative number m determined that the second variable X2 is unstable is reset to “0” (S102). As the predetermined timing, for example, when it is recognized that the first image area A1 recognized by the first image area recognition unit 121 corresponds to a predetermined object such as a person or a car is adopted.

  The first variable measuring unit 111 measures the vehicle speed v (n) and the angular speed ω (n) as the first variable X1 (n) (S111). In parallel with this, the second variable measurement unit 112 measures the distance to the object, the position in the vertical and horizontal directions, and the size through the infrared camera 210 as the second variable X2 (n) (S112). As for the measuring method of the second variable X2 (n), it is sufficient that the one disclosed in Japanese Patent Application Laid-Open No. 2002-298298 or the like is adopted, and the detailed description thereof is omitted in this specification.

  Further, the first image area recognition unit 121 recognizes the image area corresponding to the object as the first image area A1 based on the second variable X2 (n) measured by the second variable measurement unit 112 (S121). As a result, for example, the first image area A1 having a rectangular shape that varies as shown in FIGS. 5A to 5E is recognized. As for the method for recognizing the first image area A1, a general method in the field of three-dimensional image processing, such as that disclosed in Japanese Patent Application Laid-Open No. 2002-2982982, etc. is sufficient. The detailed explanation is omitted.

  In addition, the second image area recognition unit 122 determines whether the second variable X2 (n) is “stable” or “unstable” prior to recognition (S122) of a second image area A2 (n) described later. Determine (S113). Specifically, when the second variable X2 (n) is measurable or when its variation (δ = | X2 (n) −X2 (n−1) |) is less than or equal to the threshold ε (> 0). The second variable X2 (n) is determined to be “stable” (S113... YES). On the other hand, when the second variable X2 (n) is not measurable (when it is not measured) or when the variation δ exceeds the threshold ε, it is determined that the second variable X2 (n) is “unstable”. (S113... NO).

  When the second image area recognition unit 122 determines that the second variable X2 (n) is “stable” (S113... YES), the second variable X2 (n) (measured value) is determined as the current control cycle. After setting the second variable X2 (n) at n (S114), the second image area A2 (n) is recognized (S122). On the other hand, when the second image area recognition unit 122 determines that the second variable X2 (n) is “unstable” (S113... NO), the second image area recognition unit 122 increases the determination count m by “1” (S115). The second variable X2 (n) (estimated value) at time n estimated by the second image area recognition unit 122 in accordance with the procedure described later in the previous control cycle n-1 is set as the current second variable X2 (n). After that (S116), the second image area A2 (n) is recognized (S122).

  When the second image area recognition unit 122 recognizes the second image area A2 (n) (S122), the vehicle speed v (n-1) and the angular speed ω at time (represented by the number of control cycles) n-1 (past). (n-1) and the object position p2 (n-1) with respect to the camera position pc (n-1), the vehicle speed v (n), the angular velocity ω (n), and the camera position pc ( Based on the object position p2 (n) with respect to n), the object position p2 (n + 1) with respect to the camera position pc (n + 1) at time n + 1 (future) is estimated as a part of the second variable X2 (n + 1). When estimating the second variable X2 (n + 1), as shown in FIG. 1, the X axis is a three-dimensional system in which the X axis, the Y axis, and the Z axis are defined in the left-right direction, the up-down direction, and the front-rear direction of the automobile 200, respectively. The vehicle position p1 (i), camera position pc (i), and object position p2 (i) in space are each expressed by a quaternary (quaternion).

  A method of estimating the object position p2 (n + 1) with respect to the camera position pc (n + 1) will be described in detail with reference to FIG.

  A vector (see arrow c (n)-in FIG. 3) extending from the camera position pc (n) to the object position p2 (n-1) extends from (1) the camera position pc (n) to the vehicle position p1 (n). Vector (see broken line b (n) in FIG. 3), (2) vector extending from vehicle position p1 (n) to vehicle position p1 (n-1) (see broken line A (n) in FIG. 3), (3) vehicle position p1 A vector extending from (n-1) to the camera position pc (n-1) (see line segment b (n-1) in FIG. 3) and (4) the object position p2 (n-) from the camera position pc (n-1). The object position p2 (n-1) with respect to the camera position pc (n) is determined based on the fact that it is represented by a combined vector of vectors extending to 1) (see arrow c (n-1) in FIG. 3).

  (1) A vector extending from the camera position pc (n) to the vehicle position p1 (n) (see the broken line b (n) in FIG. 3) relates to the positional relationship between the rear wheel axle central portion stored in the memory and the infrared camera 210. It is determined based on the data.

  (2) A vector (see broken line A (n) in FIG. 3) extending from the vehicle position p1 (n) to the vehicle position p1 (n-1) is generated by the automobile 200 over a minute time δt from time n-1 to time n. It is determined on the assumption that it was swiveling. Specifically, based on the vehicle speed v (n−1) and the angular velocity ω (n−1) measured at the time n−1 by the first variable measurement unit 111, the turning axis O ( n-1) and turning radius (distance from the center of the rear wheel axis of the automobile 200 to the turning axis O (n-1)) R (n-1) (= v (n-1) / ω (n-1)). Determined. Further, based on the vehicle position p1 (n), the vehicle according to the following equation (1) including a quaternion Qn− and its conjugate quaternion Qn− * for representing a turn around the turning axis O (n−1). A position p1 (n-1) is determined. Thus, a vector extending from the vehicle position p1 (n) to the vehicle position p1 (n-1) is determined (see broken line A (n) in FIG. 3).

p1 (n-1) = Qn- * p1 (n) Qn-
Qn − = [0, −sin {ω (n−1) δt / 2}, 0, cos {ω (n−1) δt / 2}],
Qn − * = [0, sin {ω (n−1) δt / 2}, 0, cos {ω (n−1) δt / 2}] (1)
(3) In the vector extending from the vehicle position p1 (n-1) to the camera position pc (n-1) (see b (n-1) in FIG. 3), the extension line of the rear wheel axis is the turning axis O (n- It is determined based on the attitude of the vehicle at time n-1 determined by the condition of 1) and data relating to the positional relationship between the rear wheel axle central portion and the infrared camera 210 stored and held in the memory.

  (4) A vector (see arrow c (n-1) in FIG. 3) extending from the camera position pc (n-1) to the object position p2 (n-1) was captured by the infrared camera 210 at time n-1. Based on the image of the object, it is determined by the second variable according to the position of the object measured by the second variable measurement unit 122.

  As a result, a vector extending from the camera position pc (n) to the object position p2 (n-1) (see arrow c (n)-in FIG. 3), and consequently the object position p2 (n-1) relative to the camera position pc (n). ) Is required.

  A vector (see arrow c (n) + in FIG. 3) extending from the camera position pc (n) to the object position p2 (n + 1) is (5) from the camera position pc (n) to the object position p2 (n). And (6) a vector that extends from the object position p2 (n) to the object position p2 (n + 1) (see the arrow d (n + 1) in FIG. 3). The object position p2 (n + 1) with respect to the camera position pc (n) is estimated.

  (5) A vector (see arrow c (n) in FIG. 3) extending from the camera position pc (n) to the object position p2 (n) is based on the image of the object photographed by the infrared camera 210 at time n. It is determined by the second variable measured by the two-variable measuring unit 122 at time n.

  (6) A vector extending from the object position p2 (n) to the object position p2 (n + 1) (see the arrow d (n + 1) in FIG. 3) indicates that the object is at a time n over a minute time δt from time n to time n + 1. It is assumed that the movement is the same as the minute time δt from −1 to time n, and the object position p2 (n) is the starting point, and the object position p2 (n−1) to the object position p2 (n). Is defined as a vector having the same size and direction as the vector extending to (arrow d (n) in FIG. 3).

  As a result, a vector (see arrow c (n) + in FIG. 3) extending from the camera position pc (n) to the object position p2 (n + 1), and thus the object position p2 (n + 1) with respect to the camera position pc (n) is determined. .

  Further, a vector (see arrow c (n + 1) in FIG. 3) extending from the camera position pc (n + 1) to the object position p2 (n + 1) is (7) a vector extending from the camera position pc (n + 1) to the vehicle position p1 (n + 1) ( (See broken line b (n + 1) in FIG. 3), (8) vector extending from vehicle position p1 (n + 1) to vehicle position p1 (n) (see broken line A (n + 1) in FIG. 3), (9) from vehicle position p1 (n) A vector extending to the camera position pc (n) (see line b (n) in FIG. 3) and (10) a vector extending from the camera position pc (n) to the object position p2 (n + 1) (see arrow c (n + 1) in FIG. 3). ), The object position p2 (n + 1) with respect to the camera position pc (n + 1) is determined.

  (7) The vector extending from the camera position pc (n) to the vehicle position p1 (n) (see the broken line b (n + 1) in FIG. 3) is a condition that the extension line of the rear wheel axis is on the turning axis O (n) (described later) Is determined based on the vehicle attitude at time n + 1 determined by the above and data on the positional relationship between the rear wheel axle central portion and the infrared camera 210 stored and held in the memory. This vector (7) is determined after the vector (8) described below is determined.

  (8) A vector extending from the vehicle position p1 (n + 1) to the vehicle position p1 (n) (see the broken line A (n + 1) in FIG. 3) assumes that the automobile 200 makes a turning motion over a minute time δt from time n to time n + 1. It is determined after being done. Specifically, first, based on the vehicle speed v (n) and the angular velocity ω (n) measured at the time n by the first variable measurement unit 111, the turning axis O (n) and the turning radius located on the extension line of the rear wheel axis. (Distance from the center of the rear wheel axis of the automobile 200 to the turning axis O (n)) R (n) (= v (n) / ω (n)) is determined. Further, based on the vehicle position p1 (n), the vehicle position p1 (n + 1) according to the following equation (2) including the quaternion Qn + and its conjugate quaternion Qn + * for representing a turn around the turning axis O (n). ) Is determined. Thus, a vector extending from the vehicle position p1 (n + 1) to the vehicle position p1 (n) is determined (see broken line A (n + 1) in FIG. 3).

p1 (n + 1) = Qn + * p1 (n) Qn +
Qn + = [0, sin {ω (n) δt / 2}, 0, cos {ω (n) δt / 2}],
Qn + * = [0, −sin {ω (n) δt / 2}, 0, cos {ω (n) δt / 2}] (2)
(9) A vector extending from the vehicle position p1 (n) to the camera position pc (n) (refer to the line segment b (n) in FIG. 3) is a positional relationship between the rear wheel axle central portion stored in the memory and the infrared camera 210. Data based on the data.

  (10) As the vector extending from the camera position pc (n) to the object position p2 (n + 1) (see the arrow c (n + 1) in FIG. 3), the previously determined vector is used.

  As a result, a vector extending from the camera position pc (n + 1) to the object position p2 (n + 1) (see the arrow c (n + 1) in FIG. 3), and the object position p2 (n + 1) with respect to the camera position pc (n + 1) is obtained. Then, the second image area recognition unit 122 performs time according to the same method as the first image area recognition unit 121 based on the object position p2 (n + 1) with respect to the camera position pc (n + 1) at time n + 1 (future). The first image area A (n + 1) recognized by the first image area recognition unit 121 at n + 1 (future) is estimated and recognized as the second image area A2. Thereby, for example, a rectangular second image region A2 that changes as shown in FIGS. 6A to 6E is recognized.

  Subsequently, the image information control unit 140 includes a first image area A1 (n) recognized by the first image area recognition unit 121, and a second image area A2 (n) recognized by the second image area recognition unit 122. It is determined whether or not there is an overlapping portion (S123).

  If the image information control unit 140 determines that there is an overlapping portion (S123... YES), the composite image region recognition unit 130 combines the first image region A1 (n) and the second image region A2 (n). Then, this is recognized as a composite image area A (S130). Specifically, an area having the coordinates of the representative points of each of the first image area A1 and the second image area A2 as the representative point coordinates is recognized as a composite image area A of both. Is done. Assuming that the first image area A1 and the second image area A2 are expressed by vectors, the composite image area A is recognized according to the following equation (3).

A = C1 · A1 + C2 · A2 (C1> 0, C2> 0, C1 + C2 = 0) (3)
The first weighting factor C1 and the second weighting factor C2, which indicate the extent to which the first image region A1 and the second image region A2 are reflected in the composite image region A, are the first image region A1 in the previous control cycle n-1. It may be variably set based on various factors such as deviations of the first image area A1 (n) and the second image area A2 (n) in the current control cycle n with respect to (n-1). .

  Further, the image information control unit 140 displays a composite frame (composite image information) surrounding the composite image area A on the HUD 220 (S140). For example, as shown in FIG. 4A, a rectangular first image area A1 recognized by the first image area recognition unit 121 and a rectangular second image recognized by the second image area recognition unit 122. When the image area A2 overlaps, a rectangular composite image as shown in FIG. 4 (b) having the coordinates of the four corners (representative points) or the weighted average of the two image areas as the coordinates of the four corners. A composite frame surrounding the area A is displayed on the HUD 220 (see S130 and S140).

  On the other hand, when the image information control unit 140 determines that there is no overlapping portion (S123... NO), the first frame (first image information) surrounding the first image area A1 is displayed on the HUD 220 (S141). Then, the second frame (second image information) surrounding the second image region A2 is displayed on the HUD 220 (S142). For example, as shown in FIG. 4C, a rectangular first image area A1 recognized by the first image area recognition unit 121 and a rectangular second image recognized by the second image area recognition unit 122. When the image area A2 does not overlap, the first frame surrounding the first image area A1 and the second frame surrounding the second image area A2 as shown in FIG. Is displayed (see S141, S142). If the first image area A1 is not recognized, the first frame is not displayed on the HUD 220. Similarly, if the second image area A2 is not recognized, the second frame is not displayed on the HUD 220.

  Thereafter, when the determination number m that the second variable X2 (n) is unstable is less than the predetermined value M (S190... NO), the control cycle number n is increased by “1” (S191). The above processes such as the measurement of the first parameter X1 (n) (S111) and the measurement of the second parameter X2 (n) (S112) are repeated. On the other hand, when the determination number m is equal to or greater than the predetermined value M (S190... YES), the image information processing method for the target object ends.

  According to the image information processing method of the first embodiment of the present invention, the first image area A1 that changes as shown in FIG. 5A to FIG. When the second image area recognition unit 122 recognizes and changes the second image area A2 as shown in FIGS. 6A to 6E, the frame (image) displayed on the HUD 220 is recognized. Information) changes as shown in FIGS. 7 (a) to 7 (e).

  That is, when the first image area A1 shown in FIG. 5 (a) and the second image area shown in FIG. 6 (a) are the same (when all are overlapping portions), As shown in FIG. 7 (a), a composite frame surrounding the same composite image area A as the first image area A1 and the second image area A2 is displayed on the HUD 220.

  When the first image area A1 shown in FIG. 5 (b) and the second image area A2 shown in FIG. 6 (b) have no overlapping portion, they are shown in FIG. 7 (b). As shown, the first frame surrounding the first image area A1 and the second frame surrounding the second image area A2 are separately displayed on the HUD 220. Similarly, when there is no overlap between the first image area A1 shown in FIG. 5C and the second image area A2 shown in FIG. 6C, the first image area A1 shown in FIG. As described above, the first frame surrounding the first image area A1 and the second frame surrounding the second image area A2 are separately displayed on the HUD 220.

  Further, when the first image area A1 shown in FIG. 5 (d) and the second image area shown in FIG. 6 (d) have an overlapping portion, it is shown in FIG. 7 (d). Thus, a composite frame surrounding the composite image area A of the first image area A1 and the second image area A2 is displayed on the HUD 220.

  Further, when the first image area A1 is not recognized as shown in FIG. 5E, but the second image area A2 is recognized as shown in FIG. Only the second frame surrounding the second image area A2 is displayed on the HUD 220 as shown in 7 (e).

  According to the image information processing method of the first embodiment of the present invention executed by the image information processing system 100 of the present invention, the first image area A1 corresponding to the object recognized through the infrared camera 210 is unstable. Even so (see FIGS. 5A to 5E), the frame of the image area corresponding to the object can be stably displayed on the HUD 220 (FIGS. 7A to 7E). )reference).

  Next, an image information processing method according to the second embodiment of the present invention executed by the image information processing system having the above-described configuration will be described with reference to FIGS. 5, 6, and 8 to 9.

  The image information processing method according to the second embodiment of the present invention is a process for determining whether or not there is an overlap between the first image area A1 and the second image area A2 in the image information processing method according to the first embodiment of the present invention (see S123) Since common processing is executed with the image information processing method according to the first embodiment of the present invention except that the image information processing (see S130, S140 to S142) according to the determination result is different, the common processing will be described. Is omitted.

  According to the image information processing method of the second embodiment of the present invention, the first image region A1 (n) recognition process (S121) by the first image region recognition unit 121 and the second image region recognition unit 122 perform the second processing. After the recognition process (S122) of the image area A2 (n), the image information control unit 140 determines whether the first image area A1 (n) is “stable” or “unstable” ( S124). Specifically, the deviation δA (n) between the first image area A1 (n) and the first image area A2 (n), specifically, the first image area A1 (n) and the second image area A2 (n). The first image area A1 (n) is “stable” when the absolute value such as deviation (distance), area deviation, average of both deviations, or weighted average is equal to or less than the first threshold value ε1. It is determined that On the other hand, when the deviation δA (n) exceeds the first threshold value ε1, it is determined that the first image area A1 (n) is “unstable”.

  When the image information control unit 140 determines that the first image area A1 (n) is “stable” (S124... YES), the first frame (first image information) surrounding the first image area A1 (n). ) Is displayed on the HUD 220 (S141).

  If the image information control unit 140 determines that the first image area A1 (n) is unstable (S124... NO), the image information control unit 140 determines the degree of the instability (S125). Specifically, when the deviation δA (n) is equal to or smaller than the second threshold value ε2 greater than the first threshold value ε1, it is determined that the degree of instability is “small”, and the deviation δA (n) is equal to the second threshold value εA (n). When the threshold value ε2 is exceeded, it is determined that the instability is “high”.

  When the image information control unit 140 determines that the instability of the first image area A1 (n) is “small” (S125... YES), the composite image area recognition unit 130 determines the composite image area A (n). It is recognized (S130), and the image information control unit 140 displays a composite frame surrounding the composite image area A (n) on the HUD 220 (S140). On the other hand, when the image information control unit 140 determines that the instability of the first image area A1 (n) is “large” (S125... NO), the image information control unit 140 determines that the second image area A2 (n ) Is displayed on the HUD 220 (S142).

  According to the image information processing method of the second embodiment of the present invention, the first image area A1 that changes as shown in FIG. 5A to FIG. When the second image area recognition unit 122 recognizes and changes the second image area A2 as shown in FIGS. 6A to 6E, the frame (image) displayed on the HUD 220 is recognized. Information) changes as shown in FIGS. 9 (a) to 9 (e).

  That is, the first image area A1 shown in FIG. 5A and the second image area A2 (n) shown in FIG. 6A are the same, and the first image area A1 is “ If it is determined as “stable”, a first frame surrounding the first image area A1 is displayed on the HUD 220 as shown in FIG.

  Further, the first image area A1 shown in FIG. 5B and the second image area A2 (n) shown in FIG. 6B are remarkably different, and the first image area A1. Is “unstable” and the degree of instability is determined to be “large”, the second frame surrounding the second image region A2 is displayed on the HUD 220 as shown in FIG. Is displayed. Similarly, the first image area A1 shown in FIG. 5C and the second image area A2 shown in FIG. 6C are separated from each other. If it is determined that the degree of instability is “large”, the second frame surrounding the second image area A2 is displayed on the HUD 220 as shown in FIG. 9C. The

  Further, the first image area A1 shown in FIG. 5 (d) is different from the second image area A2 (n) shown in FIG. 6 (d). If it is determined that the degree of instability is “small” and the degree of instability is “small”, a composite frame that surrounds the composite image area A is displayed on the HUD 220 as shown in FIG.

  Further, the first image area A1 is not recognized as shown in FIG. 5E, while the second image area A2 is recognized as shown in FIG. When it is determined that the area A1 is “unstable” and the degree of instability is “large”, a second frame surrounding the second image area A2 is displayed as shown in FIG. Displayed on the HUD 220.

  According to the image information processing method of the second embodiment of the present invention executed by the image information processing system 100 of the present invention, the first image area A1 corresponding to the object recognized through the infrared camera 210 is unstable. Even so (see FIGS. 5A to 5E), the frame of the image area corresponding to the object can be stably displayed on the HUD 220 (FIGS. 9A to 9E). )reference).

  In the above embodiment, image information is displayed on the HUD 220 mounted on the front window of the automobile 200. However, as another embodiment, a meter-integrated display mounted on the automobile 200 or a display of a navigation device (not shown). ) May display image information.

  In the embodiment described above, the position and rotation (turning) of the vehicle body position p1 (i) and the like in the three-dimensional space are represented by the quaternion in the recognition process of the image area including the object (see S121 and S122) As another embodiment, the position of the vehicle body position p1 (i) or the like in the three-dimensional space may be represented by a three-dimensional vector, and the rotation may be represented by a three-dimensional matrix.

  In the embodiment, the first, second, and composite frames surrounding the first image area A1, the second image area A2, and the composite image area A are displayed on the HUD 220 as the first, second, and composite image information. As another embodiment, each image area may be displayed on the HUD 220 so as to be distinguishable from other image areas by its color, pattern, brightness, etc., and the brightness in the grayscale or binarized image included in each image area A frame that surrounds the target object of a certain level or more may be displayed on the HUD 220, and a part or all of the target object that has a certain level or more of brightness included in each image area may be another image area depending on its color, pattern, brightness, etc. Any form that allows each image area or an object having a certain level of brightness or more to be identified from other image areas, such as being displayed on the HUD 220 so that it can be identified from the other. In image information representing the image areas may be displayed on the HUD220.

  In the second embodiment, the degree of instability of the first image area A1 (n) is determined (see FIG. 8 / S125). However, as another embodiment, the determination step is omitted, and the first image area A1 (n) is determined. When the area A1 (n) is unstable, the second image information corresponding to the second image area A2 (n) or the combined image information corresponding to the combined image area A (n) is uniformly displayed in the image information control unit 140. May be displayed on the HUD 220.

Configuration explanatory diagram of the image information processing system of the present invention Explanatory drawing of the image information processing method of 1st Embodiment of this invention. Explanation supplementary diagram for recognition processing of second image area Supplementary explanation of processing according to presence / absence of overlapping portion of first and second image regions Example of first image area Illustration of second image area FIG. 4 is a view showing an example of a composite image area by the image information processing method according to the first embodiment of the present invention. Explanatory drawing of the image information processing method of 2nd Embodiment of this invention. Illustration of composite image area by image information processing method of second embodiment of the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Image information processing system, 111 ... 1st variable measurement unit, 112 ... 2nd variable measurement unit, 121 ... 1st image area recognition unit, 122 ... 2nd image area recognition unit, 130 ... Composite image area recognition unit, 140 Image information control unit 200 Automobile (moving body) 201 Vehicle speed sensor 202 Yaw rate sensor 210 Infrared camera 220 HUD (image display device)

Claims (7)

A system for processing an image of an object photographed by an imaging unit mounted on a moving body and displaying image information of the object on an image display device mounted on the moving body,
First variable measuring means for measuring a first variable according to the traveling state of the moving body;
Second variable measuring means for measuring a second variable according to the position of the object through the imaging means;
First image area recognition means for recognizing an image area corresponding to the object as a first image area based on the second variable measured by the second variable measurement means;
Based on the first variable measured by the first variable measuring means and the second variable measured by the second variable measuring means, the image area recognized by the first image area recognizing means is estimated. A second image area recognition means for recognizing as a second image area,
A composite image area that is recognized as a composite image area after combining the first image area recognized by the first image area recognition means and the second image area recognized by the second image area recognition means. An image information processing system comprising a recognition unit. The image information processing system according to claim 1,
At least of the first image information corresponding to the first image area recognized by the first image area recognition means and the second image information corresponding to the second image area recognized by the second image area recognition means. On the other hand, the image information processing system further comprises image information control means for causing the image display device to display composite image information corresponding to the composite image area recognized by the composite image area recognition means. . The image information processing system according to claim 2.
An image information processing system in which the image information control means displays the composite image information on the image display device when there is an overlapping portion in the first image region and the second image region. The image information processing system according to claim 1,
The synthesized image area recognizing means is synthesized so that the first image area is reflected to the extent corresponding to the first weighting factor, and the second image area is reflected to the extent corresponding to the second coefficient. An image information processing system that recognizes an image area as the composite image area. The image information processing system according to claim 4.
The composite image area recognition unit determines the first weighting factor based on a deviation between the previous first image area and the current first image area, and the previous first image area and the current first image area. An image information processing system, wherein the second weighting factor is determined based on a deviation from two image regions. The image information processing system according to claim 2.
When the first image area is stable, the image control means causes the first image information to be displayed on the image display device, and when the first image area is unstable, the second image information is displayed. Or an image information processing system for displaying the composite image information on the image display device. The image information processing system according to claim 6.
When the image information control means determines that the degree of instability of the first image area is small when the first image area is unstable, and determines that the degree of instability is small, the composite image information Is displayed on the image display device, and when it is determined that the degree of instability is large, the second image information is displayed on the image display device.

Images