JP4264944B2 - Human state estimation system, human state estimation method, and human state estimation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate the intention of a human being in order to properly support the action of the human being. <P>SOLUTION: The state of a human being is estimated by using a plurality of dispersed feeling intelligences comprising a sensor and a process operation unit. That is, as the sensor for acquiring the action result of the human being, cameras 21, 24, 31 and 32, and a vehicle velocity detecting device 25 are arranged. As the process operation unit, a vehicle position information calculating unit 202, a first face direction information calculating unit 203 and a second face direction information calculating unit 204 are arranged. An intention recognizing unit 207 performs an intention inference based on environmental information such as vehicle position information V<SB>L</SB>, face direction information &theta;<SB>D</SB>, and vehicle velocity information V<SB>v</SB>acquired from a plurality of the dispersed feeling intelligences, and an intention recognition result is acquired. <P>COPYRIGHT: (C)2005,JPO&amp;NCIPI

Description

  The present invention relates to a human state estimation system, a human state estimation method, and a human state estimation program for grasping a human intention in order to support human behavior.

  With the rapid development of information and communication technology, the development of information infrastructure is being promoted, and networks and information systems are spreading to ordinary homes and mobiles. On the other hand, as far as these network information systems are viewed from the user side, even though they are users who live in the space formed by the network, the formed spatial system effectively acts on the user and supports humans. It is hard to say that it is.

  In addition, while the number of people who benefit from daily life increases, information weak people who are alienated from information technology, such as the elderly and functionally disabled people in hospitals and households, and are not fully receiving the benefits It is the current situation that the disparity with the called people is relatively advanced.

  On the other hand, the Ministry of Transport's Advanced Safety Vehicle Promotion Study Group has proposed preventive safety technology for improving the safety of automobiles by using new technologies such as electronics technology and using this information processing technology. (Non-Patent Document 1). This preventive safety technology detects the driver's dangerous condition during normal driving, the dangerous situation of the driving environment, etc. with various sensors and alerts the driver, reducing the driver's burden, etc. It is designed for illustration purposes. However, even if the danger can be notified immediately before the dangerous situation, the situation before the dangerous situation cannot be notified. Further, it is known that when a danger is suddenly displayed immediately before a dangerous situation, the display immediately before that enters the field of view, so that the peripheral visual field becomes extremely difficult to see.

In this manner, for example, the presence of pedestrians and vehicles can be detected and presented in an urban space, but the information becomes excessive, which causes confusion and hinders the driver.
Secretariat for the Promotion of Advanced Safety Vehicles Promotion Ministry of Transportation

  In this manner, for example, the presence of pedestrians and vehicles can be detected and presented in an urban space, but the information becomes excessive, which causes confusion and hinders the driver.

  The present invention has been made to solve the above-described problems, and the object of the present invention is to provide a human state estimation system, a human state estimation method, and a human state that infer human intentions in order to appropriately support human behavior. To provide an estimation program.

  In order to achieve the above object, one aspect of the present invention is to estimate the driving behavior that the driver of the vehicle will perform when the vehicle reaches a preset target point. The vehicle position information is calculated, the vehicle speed information is detected, the driver's face image data is obtained by a camera mounted on the vehicle, and a face identification area is obtained from the face image data by image recognition processing. By detecting the center of gravity of the vehicle, information indicating the direction of the driver's face is detected. An input membership function preset for each of the vehicle position, speed, and driver's face direction, an output membership function preset for each of a plurality of assumed driving intentions, and each of the above inputs An associative memory for storing a plurality of individuality data for associating a membership function and each of the output membership functions in a matrix is prepared, and the calculated vehicle position information and the detected vehicle position are detected in the associative memory. Speed information and information indicating the detected driver's face direction are input as input parameters, and forward and backward matrix operations are repeatedly executed a predetermined number of times using this associative memory. To obtain a plurality of output membership functions that have converged, and respectively correspond to the output membership functions that have been output. Select the one value from the driver's intention grade greatest, the intention corresponding to the selected driving intention grade is obtained so as to output as intended result of recognition of the driver.

  Another aspect of the present invention is that, when calculating vehicle position information, a plurality of cameras are arranged at the target point, the vehicles are picked up from different directions by these cameras, and the image data is transferred to the vehicle. Send to. And in the vehicle position information calculation part provided in the vehicle, a plurality of vehicle image data sent from the plurality of cameras is received, and a tertiary view function is used using the stereo vision function based on the received plurality of vehicle image data. The position information of the vehicle represented by the original coordinates is calculated.

According to still another aspect of the present invention, when calculating the face direction, the center-of-gravity coordinates of the driver's mouth area are detected from the face image data obtained by the camera by image recognition processing. An angle formed by a straight line Lo connecting the camera and the center of the driver's head and a straight line Lx connecting the camera and the center of gravity coordinates of the detected mouth area is θ _A , An angle formed by a straight line connecting the center and the center of gravity coordinates of the detected mouth area and the straight line Lx is θ _B , and a distance from the camera to the center of the driver's head is d, and the driver's head When the head radius determined by the part model is r and the angle of view of the camera is φ, the direction θ _D2 of the driver's face is
r / sin θ _A = d / sin θ _B
sin θ _B = (d / r) · sin θ _A
θ _B = 180−arcsin (sin θ _B ) · 180 / π
θ _D2 = 180− (θ _A + θ _B )
It is made to calculate by.

  According to the present invention, it is possible to appropriately infer human intentions.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating an example of a distributed sensory intelligence space in an urban space to which a human state estimation system according to an embodiment of the present invention is applied.

FIG. 1 shows a situation where the vehicle 2 is traveling on a road 41. Two cameras 31 and 32 are arranged at the intersection of the road 41 and the road 42. These cameras 31 and 32 image the vehicle 2 passing through the road 41. An example of images captured by the cameras 31 and 32 is indicated by 102. Further, the vehicle 2 is equipped with a vehicle-mounted camera 24 for acquiring the operation of the driver 10 and images the driver 10. An example of this driver captured image is indicated by 101. Furthermore, a vehicle speed detection device 25 is mounted on the vehicle 2. Detected vehicle speed information V _v is displayed as a captured image indicated by the example 103.

Thus, the operation of the driver 10 calculated based on the driver image captured by the vehicle-mounted camera 24, the vehicle position information V _L calculated based on the vehicle images acquired by the cameras 31 and 32, and the vehicle based on the vehicle speed information V _v obtained from the speed detector 25, to infer the driver's intention 10.

  In this distributed sensory intelligence space, the distributed sensory intelligence is a vehicle position detection mechanism including each vehicle 2 and cameras 31 and 32. These distributed sensory intelligence consists of a sensory part and an intelligent part. The sensory part is implemented by a sensor such as a CCD camera. The intelligence unit is implemented by a processing arithmetic device that executes processing such as a CPU. The sensory part of the distributed sensory intelligence model acquires physical quantities such as color information, motion information, and shape information in the environment and performs preprocessing. This physical quantity includes not only physical values such as speed and position but also complex physical quantities. For example, the image information is also a physical quantity indicating the color, movement, and shape of the environment. Based on the physical quantity obtained by the sensory part, the intelligence part performs environmental processing such as generating environmental information representing a human action result and outputting the information. In the distributed sensory intelligence space, multiple sensory intelligences are distributed throughout the space to share and supplement information as if the entire space had high intelligence.

  In the present embodiment, software called I-Space is used as an example for obtaining physical quantities sent from each camera sensor in the distributed sensory intelligence section. In this I-Space, coordinate data is extracted by a method in which a portion having a color desired to be detected from an image sent from a camera is held by a window, and the center of gravity of the detected pixel is calculated therefrom. In addition, since the window tracks the movement of a group of pixels of a color that is held at regular intervals, it is possible to track a portion that is desired to be detected.

FIG. 2 is a diagram showing an example of each configuration of the human state estimation system. As shown in FIG. 2, two vehicle image data captured from the points A and B are received from the cameras 31 and 32 via the communication control means 201 of the vehicle 2. The vehicle position information calculation unit 202 calculates vehicle position information _VL based on the two vehicle image data captured from these points A and B. The calculation of the vehicle position information V _L is obtained by a calculation method based on stereo vision.

Further, the vehicle speed detecting device 25 provided in the vehicle 2 calculates the vehicle speed information V _v from the number of revolutions of the vehicle 2 motor 2c. The first face direction information calculation unit 203 calculates first face direction information θ _D1 based on in-vehicle camera image data including the driving action of the driver 10 obtained by the vehicle-mounted camera 24. The second face direction information calculation unit 204 calculates the second face direction information θ _D2 based on the image data of the driver's 10 visual field obtained by the visual field image acquisition camera 21. The information reconstruction unit 205 performs information reconstruction processing based on the obtained face direction information θ _D1 and θ _D2 to obtain temporary face direction information θ _P. Newton's method correction section 206, a face direction provisional information theta _P corrected by Newton's method to obtain the face direction information theta _D. The intention recognition unit 207 generates an intention recognition result indicating the intention of the driver 10 by performing an intention recognition process based on the vehicle position information V _L , the vehicle speed information V _v, and the face direction information θ _D. The obtained intention recognition result is displayed on the display unit 22 provided in the vehicle 2. Of course, the display unit 22 is provided in the distributed sensory intelligence arranged separately from the vehicle 2, and the intention recognition result may be transmitted to the distributed sensory intelligence wirelessly and displayed on the display unit 22.

  In FIG. 2, the distributed sensory intelligence is configured by the cameras 31 and 32 and the vehicle position information calculation unit 202. The cameras 31 and 32 function as a sensor unit, and the vehicle position information calculation unit 202 functions as a processing calculation unit. Further, the vehicle speed detection device 25 functions as one distributed sensory intelligence. Further, the visual field image acquisition camera 21, the vehicle mounted camera 24, the first face direction information calculation unit 203, the second face direction information calculation unit 204, the information reconstruction unit 205, and the Newton method correction unit 206 constitute a distributed sensory intelligence. . The field-of-view image acquisition camera 21 and the vehicle-mounted camera 24 function as a sensor unit, and the first face direction information calculation unit 203, the second face direction information calculation unit 204, the information reconstruction unit 205, and the Newton method correction unit 206 are processing calculation units. Function as.

  Further, processing data in each configuration and a program for executing each configuration are stored in the storage device 39 provided in the vehicle 2 and read from each configuration as necessary.

Next, a method for calculating the vehicle position information _VL using the cameras 31 and 32 will be described with reference to FIG.

FIG. 3 uses the stereo view function of I-Space to detect the vehicle position information _VL . In this method, two I-Spaces are used, and the coordinate data obtained by each is calculated to obtain the position of the three-dimensional coordinate. In FIG. 3, the coordinates of the target vehicle 2 is P (x, y, z), the coordinates of the left camera 32 with respect to the vehicle 2 are OL, and the coordinates of the right camera 31 with respect to the vehicle 2 are OR. The coordinates of the image plane of the vehicle 2 in the image captured by the camera 32 are P _L (X _L , Y _L ), and the coordinates of the image plane of the vehicle 2 in the image captured by the camera 31 are P _R (X _{R 1} , Y _R ). These coordinates P _L and P _R are the coordinates in the image on the imaging surface.

  When the center of the image of the left camera 32 is OL and the center of the image of the right camera 31 is OR, d is the distance between these centers OL and OR. f is the focal length of the cameras 31 and 32.

The target P (x, y, z) is calculated by the following equations (1) to (3).
x = d · X _L / (X _L −X _R ) (1)
y = d · Y _L / (X _L −X _R ) (2)
z = d · f / (X _L −X _R ) (3)
FIG. 4 is an overview diagram of the vehicle 2, and FIG. 5 is a diagram schematically illustrating functions of the vehicle 2.

  As shown in FIG. 4, a vehicle speed detection device 25 that detects the rotation speed of the wheel 2 a of the vehicle 2 and the rotation angle of the handle 2 b is provided. As shown in FIG. 5, the control computer 2e is provided in the vehicle 2 so that it can communicate with the external terminals 2f and 2g via the wireless LAN 2m and the wireless LAN 2n. The external terminals 2f and 2g are implemented in other distributed sensory intelligence, and correspond to, for example, a control computer provided in another vehicle, a portable terminal held by a pedestrian, or the like.

  The control unit 2d such as the vehicle control microcomputer PP7 receives the speed command and the handle angle command from the control computer 2e, for example, by wire, and the control unit 2d controls the motor 2c and the handle 2b provided in the vehicle 2. Thereby, the driving | operation of the vehicle 2 by rotation of a desired vehicle speed and a handle | steering-wheel is attained.

On the other hand, the control unit 2d detects the rotation speed of the motor 2c and the rotation angle of the handle 2b. The wheel speed and the steering wheel rotation angle obtained from the motor rotation speed detected by the control unit 2d are transmitted to the control computer 2e. Control computer 2e vehicle speed detecting device 25 provided calculates the vehicle speed information V _v based on the wheel speed received and transmitted with the wheel turning angle to the external terminal 2f and 2g.

  FIG. 6 is a diagram for explaining acquisition of face direction information. As shown in FIG. 6, in the vehicle 2, a field-of-view image acquisition camera 21 composed of a CCD camera or the like for acquiring a field-of-view image fixed above the head, and a control computer 2 h connected to the field-of-view image acquisition camera 21. Is provided. In addition, the vehicle 2 includes a vehicle-mounted camera 24 made up of a CCD camera or the like installed in the vehicle 2 in front of the driver 10, and a control computer 2 i connected to the vehicle-mounted camera 24. Further, three color markers 2j to 2l such as color balls are arranged in the visual field of the driver 10, for example, in the vicinity of the windshield.

  The visual field image acquisition camera 21 acquires a driver's visual field image and outputs it to the control computer 2h. The control computer 2h has an I-Space function, and detects three color markers 2j to 2l from the visual field image data.

  On the other hand, the vehicle-mounted camera 24 acquires an in-vehicle camera image (driver image) and outputs it to the control computer 2i. The control computer 2i detects the face and lips of the driver 10 from the in-vehicle camera image data.

Based on the detection data obtained by these control computers 2h and 2i, face direction information θ _D1 and θ _D2 are calculated using a calculation formula described later, and the information reconstruction unit 205 having an attention function reconstructs the information. to calculate the conduct face direction provisional information θ _P. Moreover, to calculate the face direction information theta _D corrects the Newton method correction section 206. Thereby, detection with high robustness and high accuracy is performed. The calculation of the face direction information θ _D1 and θ _D2 , the calculation of the face direction provisional information θ _P , and the calculation of the face direction information θ _D may be performed by either the control computer 2h or 2i, or executed by another computer. May be. Specifically, the control data obtained by a computer 2h transmits to the control computer 2i, the control computer also may perform the processing up to the calculation of the face direction information theta _D within 2i, contrary to the control computer to this Data may be aggregated on the 2i side and performed on the control computer 2i side.

FIG. 7 is a schematic diagram for explaining an example of calculating the first face direction information θ _D1 . FIG. 7 shows a case where 2j of the markers 2j to 2l fixed to the vehicle 2 is included in the image acquisition range of the angle 2φ of the visual field image acquisition camera 21 fixed to the head of the driver 10. ing. The markers 2j to 2l may be portions formed by members or the like different from other members of the vehicle 2.

Field image obtaining camera 21 and the image acquisition range parallel to the z-axis to the straight line l _o that connects the center of is shown, x-axis and y-axis is shown in a direction perpendicular to the z-axis. Distance from view the image acquisition camera 21 to the marker 2j is _{d z.} The position of the marker 2j is detected by a known image recognition technique. In this case, when the x coordinate on the image plane 71 of the marker 2j in the field image is measured, if the distance d _x from the line-of-sight direction of the marker 2j in the vehicle 2 and the field angle φ are known, the driver 10 The first face direction information θ _D1 is obtained by the following equations (4) to (6). Here, φ is obtained from the angle of view.

d _x = d _z tanφ | x−x ₀ | / x ₀ (4)
tan θ _D1 = d _x / d _z (5)
θ _D1 = arctan (tan θ _D1 ) · 180 / π (6)
x ₀ is an x coordinate on the image plane 71 of the straight line l _o .

Next, a method for calculating the second face direction information θ _D2 using the vehicle-mounted camera 24 will be described with reference to FIGS. 8 and 9.

  FIG. 8 is a diagram for explaining a method for calculating the distance d between the driver 10 and the vehicle-mounted camera 24 obtained from the y-coordinate.

In FIG. 8, h is the height of the vehicle-mounted camera 24 from the ground 81, and H is the height from the ground 81 to the top of the driver 10. If L ₀ is a straight line connecting the center positions of the images taken by the vehicle-mounted camera 24, the angle between the straight line L ₀ and the straight line L ₁ connecting the top of the driver 10 is represented by θ _y . Φ is the viewing angle of the on-vehicle camera 24. When the position of the parietal region of the driver 10 at a distance Z _y away the image plane from the vehicle mounted camera 24 y, and also the intersection of the boundary line L ₂ and the image plane of the viewing angle φ and y _0, the distance d is less It calculates | requires by Formula (7)-(9).

Z _y = y ₀ tanφ (7)
θ _y = arctan (Z _y / (y−y ₀ )) (8)
d = (H−h) / tan θ _y (9)
FIG. 9 is a diagram for explaining a method of calculating the second face direction information θ _D2 from the x coordinate using the distance d obtained based on FIG. FIG. 9A is a schematic view of the driver 10 and the vehicle-mounted camera 24 as viewed from above, and FIG. 9B is a diagram illustrating an example of the image plane.

FIG. 9A shows a case where the head 10 a of the driver 10 to be detected is included in the image acquisition range of the angle 2φ of the vehicle-mounted camera 24. As an example, among the head 10a of the driver 10, the mouth of the driver 10, that is, the mouth region 10b of FIG. 9A is used as a marker. The driver's head 10a is approximated by a circular user head model having a radius r, but may be approximated by other shapes. A Z axis is shown parallel to a straight line connecting the vehicle-mounted camera 24 and the center C of the driver's head 10a, and an X axis and a Y axis are shown in a direction perpendicular to the Z axis. The distance from the vehicle-mounted camera 24 to the center C of the driver's head 10a is d. The center of gravity of the mouth region 10b is detected by a known image recognition technique. In this case, when the position x on the image plane 91 is measured and the d, r, and φ are known, the second face direction information θ _D2 of the driver 10 is expressed by the following equations (10) to (13). Desired. Here, φ is obtained from the angle of view, and r is obtained from the user's head model.

r / sin θ _A = d / sin θ _B (10)
sin θ _B = (d / r) · sin θ _A (11)
θ _B = 180−arcsin (sin θ _B ) · 180 / π (12)
θ _D2 = 180− (θ _A + θ _B ) (13)
As shown in FIG. 9B, x ₀ is a position where a straight line _Lo connecting the vehicle-mounted camera 24 and the center C of the driver's head 10a intersects the image plane 91, that is, the center of the image plane 91. X coordinate. Incidentally, y coordinates of the center of the image plane 91 is represented by _{y 0.} θ _A is an angle formed by a straight line L _x connecting the center of gravity of the mouth area 10b and the vehicle-mounted camera 24 with the straight line L _o . θ _B is an angle formed by a straight line connecting the center C and the center of gravity of the mouth region 10b with the straight line L _x, and θ _B ≧ 90.

  FIG. 10 is a conceptual diagram for explaining the reconstruction process of the information reconstruction unit 205.

  A model of the hippocampal system modeled as one of the brain memory systems related to human recognition and consciousness is the reconstruction model based on the attention function shown in FIG. This information reconstruction model is very effective for recognition and learning from a plurality of attribute patterns (n in FIG. 10). This model consists of two layers, a pattern layer 101 and a symbol layer 102. The pattern layer 101 expresses attributes such as a color attribute, a motion attribute, and a shape attribute, and each attribute is expressed by attribute patterns 101a to 101n. The symbol layer 102 represents independent symbols such as a human, a ball, and a car. When a certain thing or concept is recalled, the cell groups of both the pattern layer 101 and the symbol layer 102 are excited. At this time, the attention function controls the pattern layer 101 so as to use only a part of the attribute patterns 101a to 101n, and inhibits a part of the symbol layer 102 from being recalled. There are various other human attention functions. In the case of the attention function according to the present embodiment, the degree of attention to which attribute of many attribute patterns 101a to 101n is determined when information is reconstructed. Control.

  Information is reconstructed by the attention function when the number of symbol elements constituting the symbol layer 102 in FIG. 10 is 1, the number of internal states of each attribute is m, and the number of attribute patterns is n. It is expressed by

M _e = attn ₁ M _1e + attn ₂ M _2e +... + Attn _n M _ne (14)
Here, attn _i ε [0, 1]. Here, M _1e , M _2e ,..., M _ne ∈R ^{1 × mn} is a normalized association matrix between attribute symbols, and M _e ∈R ^{1 × mn} is a normalized association matrix reconstructed by the attention function. is there. Also, attn ₁ , attn ₂ ,... Attn _n ∈ [0, 1] are variables for controlling attention for each attribute.

In FIG. 10, by reconstructing information from a plurality of attribute patterns by the attention function, one symbol is recalled, and the resulting information can be obtained. For example, the attribute pattern 101a is a color attribute, the attribute pattern 101b is a motion attribute, and the attribute pattern 101c is a shape attribute. At this time, if it is desired to focus only on the color attribute, the value of attn ₁ may be brought close to 1, and the values of other attributes attn _j (j ≠ 1) may be brought closer to 0. As described above, the information of each distributed sensory intelligence can be reconfigured by appropriately setting the value of attni _{i to} a value between 0 and 1 by the attention function and controlling the degree of attention to each attribute. it can.

Using this technique, the information of the field-of-view image acquisition camera 21, that is, the first face direction information θ _D1 is attribute pattern 101a, and the information of the vehicle-mounted camera 24, that is, the second face direction information θ _D2 is the attribute pattern 101b. I do. In this case, the association matrix M ₁ for correlating the advance attribute pattern 101a and the symbol layer 102, stores the association matrix M ₂ that associates the attribute pattern 101b and the symbol layer 102 in the storage device through an experiment or the like, reads out this by calculating the value of the symbol layer 102, that is, the face direction provisional information theta _P obtained. The calculation method of the association matrix is the same as the calculation of the association matrix in the intention recognition unit 207 described later. More accurate angle detection can be performed by changing the value of attn when any information fails to be tracked. The resulting information is referred to as face direction provisional information θ _P. The details of processing of the associative storage system using such an associative matrix are the same as those of the intention recognition unit 207 described later.

Newton correction unit 206 generates the face direction information theta _D corrects by Newton's method with respect to the face direction provisional information theta _P. As a case where correction is necessary, for example, the first face direction information θ _D1 calculated based on the captured image of the field-of-view image acquisition camera 21 is calculated based on the captured image of the vehicle-mounted camera 24. Assume that the face direction is found to be more accurate than _θD2 . In this case, it is given when calculating the first face direction information θ _D1 obtained based on the image of the on-vehicle camera 24 so as to approach the value of the second face direction information θ _D2 obtained based on the image of the visual field image acquisition camera 21. The distance d between the vehicle-mounted camera 24 and the driver 10 is corrected by the Newton method to obtain a corrected distance value d ′. The corrected face direction information θ _D1 ′ is obtained by replacing the distance d with the obtained correction distance value d ′. Based on the obtained corrected face direction information θ _D1 ′ and the already acquired second face direction information θ _D2 , the information reconstruction unit 205 reconstructs the information again, and the obtained value is used as the face direction information θ _D. And

Intention recognizing section 207, the face direction information theta _D, and the vehicle speed information V _v, based on the vehicle position information V _L, to generate the intended recognition result performed intention recognizing process. The intention recognition result generation process will be described below. In the following processing in the intention recognition unit 207, intention recognition processing is executed based on vehicle position information V _L , speed information V _v , and face direction information θ _D that are read out and input in advance, and an intention recognition result is generated. To do.

When attempting to take a certain driving action, it is generally confirmed that the driver operates in a certain order. Detecting the operation of the face of the driving operation, it recognizes its face direction information theta _D, the driver's intention from the vehicle speed information V _v and the vehicle position information V _L. Driving intention is classified into, for example, right turn, left turn, and straight ahead.

In the present embodiment, a knowledge representation of a concept fuzzy set (Conceptual Fuzzy Sets: hereinafter referred to as CFS) is used as a technique for constructing knowledge about a human intention that is an ambiguous concept. CFS has the following two properties.
a) The real-world numerical data and human-thinking symbols can be fused.
b) It is possible to fuse bottom-up processing from a physical quantity to a higher-order concept and top-down processing from a higher-order concept to a physical quantity.

Hereinafter, an intention recognition model for recognizing three basic driving action intentions of straight driving, right turn, and left turn of driving intention from face direction information θ _D , vehicle speed information V _v, and vehicle position information V _L will be described.

  FIG. 11 is a conceptual diagram of the intention recognition model. For example, the intention recognition model is represented by a lower layer, a middle layer 53, and an upper layer 54 indicated by a membership function 51 and a fuzzy label 52. The lower layer fuzzy label 52 (fuzzy set) includes an angle label 521, a position label 522, and a velocity label 523. The angle label 521 includes three types of labels: negative 521a “N”, zero 521b “Z”, and positive 521c “P”. The position label 522 is composed of three types of labels: negative 522a “N”, zero 522b “Z”, and positive 522c “P”. The speed label 523 includes three types of labels: negative 523a “N”, zero 523b “Z”, and positive 523c “P”.

  These nine types of fuzzy labels are represented by membership functions 51, respectively. Reference numerals 511a to 511c denote functions corresponding to three types of labels of the negative label 521a “N”, zero 521b “Z”, and positive 521c “P” of the angle label 521, respectively. Reference numerals 512a to 512c respectively denote functions corresponding to three types of labels of the position label 522: negative 522a “N”, zero 522b “Z”, and positive 522c “P”. Reference numerals 513a to 513c denote functions corresponding to three types of labels of negative 523a “N”, zero 523b “Z”, and positive 523c “P” of the speed label 523, respectively.

  The middle layer 53 represents a case node which is a case used in the learning process. For example, since three basic driving behaviors are learned for N cases, a total of 3N patterns are learned. It is desirable that the N cases are not limited to the case of one driver, but, for example, the case of separate N drivers is adopted. Thereby, for example, it can be registered as data reflecting the habit of driving for N people, that is, individuality. Since this case node is data reflecting individuality in this way, it should also be referred to as individuality data. Driving habits vary from person to person. Therefore, by using a case node reflecting this individuality in driving support and referring to the data of the case node with the most similar wrinkles, it is possible to provide driving support that properly identifies the person's wrinkles. Furthermore, it is desirable to register a single driver's behavior separately for a plurality of cases. Even the driving behavior of the same person varies depending on the person's mood, for example, when he / she is irritated or relaxed. Therefore, a plurality of driving actions of the same person are registered according to the state of the driver when driving. This makes it possible to provide more detailed driving assistance including the person's mood.

  In this way, instead of averaging the cases and comparing them with the average value, the individuality data reflecting each individuality is left as it is, and the individuality data and the driving behavior are compared, thereby achieving high accuracy. Driving assistance is possible.

  As for the case nodes, three types of 531 to 533 are given to the basic driving behaviors of left turn, straight ahead, and right turn with respect to the angles, and each of these case nodes 531 to 533 is associated with the nodes A, 521a to 521c, B, C, and the nodes A, B, C and the nodes 541, 542, and 543 of the upper layer 54 are represented by a coupling relationship.

  As for the position, three types of 534 to 536 are given to the basic driving behaviors of left turn, straight ahead, and right turn, and each of these case nodes 534 to 536 are nodes A, B, and C associated with labels 522a to 522c. These nodes A, B, and C are expressed by a connection relationship between the nodes 541, 542, and 543 of the upper layer 54.

  As for speed, three types of 537 to 539 are given to the basic driving behaviors of left turn, straight ahead, and right turn. Each of these case nodes 537 to 539 is associated with nodes A, B, and C associated with labels 523a to 523c. These nodes A, B, and C are expressed by a connection relationship between the nodes 541, 542, and 543 of the upper layer 54.

  The reason why the case nodes of the middle layer 53 are arranged by dividing into angles, positions, and speeds like 531 to 533, 534 to 536, and 537 to 539 is that the intention of driving behavior is not expressed as a fixed pattern, This is because it is composed of a combination of various patterns of angle, position and speed.

  The upper layer 54 is a layer indicating driving intention, and is composed of three types of nodes: a left turn 541, a straight advance 542, and a right turn 543.

  When angle, position, and velocity feature values are input, the node having the closest feature in the middle layer 53 is most strongly activated. When the degree of activation is obtained as an activity value distribution, it can be seen which case is close for each pattern.

  However, the driving behavior cannot be specified only by the activity value distribution appearing in the middle layer 53. This problem is solved by introducing a context. The upper layer 54 is a layer representing driving behavior, and is coupled to all the case nodes 531 to 539 of the middle layer 53.

  The learning data is given to the lower layer indicated by the fuzzy label 52 and the membership function 51. As a result, the middle layer 53 obtains the active value distribution of the feature values and the case nodes 531 to 539. Further, on the basis of the activity value distribution of the middle layer 53, the upper layer 54 is given a driving intention of “right turn”, “straight forward”, and “left turn” as a teacher. Based on these, a connection relationship between the middle layer 53 and the upper layer 54 is obtained by Heb learning. In this intention recognition model, driving behavior is stored as a fuzzy set from an example, and when an input is given, echo action is repeated between the lower layer and the middle layer 53, and between the middle layer 53 and the upper layer 54. The recognition result is obtained by the convergence of the activity value distribution to a state consistent with the context. Here, the context is a standard pattern obtained from each driving action example, and a combination of feature values of angles, positions, and speeds similar to the pattern is promoted, while feature values that do not match the pattern are The combination is suppressed.

Next, the concept of CFS (concept fuzzy set) will be described with reference to FIG.
The concept fuzzy 61 not only has fuzzy knowledge expressed in a distributed manner on an associative memory, but also has a mechanism for performing fuzzy knowledge processing using an associative memory recall process. Further, since the concept fuzzy 61 is realized by an associative memory, it is possible to obtain merits such as handling of situation dependency and complicated processing by multi-layer coupling.

  The fuzzy labels 62 and 63 of the fuzzy set express the name of the concept, and the form of the fuzzy set expresses the meaning of the concept. According to the meaning usage theory, the meaning of a word is expressed by other words, but its meaning changes depending on the time of use. Therefore, if the attention range at the time of use is expressed by an active value associated with a word, the meaning of an ambiguous word can be expressed by a fuzzy set formed by another word. This distributed knowledge representation is called a concept fuzzy 61.

  Since the distribution of the activity values changes depending on various situations, it is possible to express a change in meaning depending on the situation. The concept fuzzy 61 is not limited to a logical expression, and can represent a knowledge. Therefore, what is logically impossible to express explicitly can be expressed explicitly.

  The activation value of the concept fuzzy 61, that is, the membership value is controlled by the associative memory. Nodes represent concepts, and links correspond to the strength of relationships between concepts. In FIG. 12, nodes are represented by fuzzy labels 62 and 63, and links are represented by arrows. For example, when the fuzzy label 62 in FIG. 12 is activated as a concept node to be expressed semantically, an activation value is propagated through a network constructed in advance, and an echo operation occurs. This reverberation stops when the system energy is at a minimum, and as a result, the conceptual node on the side that expresses the meaning is recalled in a form that is activated with a certain grade. The distribution of activity values appearing here is a fuzzy set.

The concept fuzzy 61 is functionally constructed using a neural network learning rule. That is, according to the concept expression example, the link between nodes is corrected using the Hebb learning rule of the following equation (21).

Here, m _ij denotes the matrix element of the associative matrix M of i rows and j row below, a _i is the respective nodes of the layer L _A, b _j corresponds to each node of layer L _B.

  In the concept fuzzy set, other concepts that are not necessary to explain the concept are not activated. Therefore, structural learning including node selection is performed even from redundant explanation elements. Moreover, in the concept fuzzy set, the general concept is formed by sequentially combining the individual concepts formed separately, as in the case where a person usually forms a concept. That is, a plurality of conceptual fuzzy sets can be synthesized according to the following equation (22).

M = Norm (M ₁ + M ₂ +... + M _n ) (22)
M _{1 to} M _{n denote} individual concept fuzzy, and M denotes a synthesized concept fuzzy set. Norm indicates normalization. In a general neural network, an explicit knowledge expression is not performed on the learning result, but an explicit expression is obtained by the method of the present embodiment.

  Next, fuzzy associative reasoning that is the basis of the fuzzy associative memory system used in the intention recognition unit 207 will be described.

  Fuzzy associative reasoning is a network that uses multiple bi-directional associative memories (BAMs), which is a type of associative memory neural network, and infers by echoing that repeats the propagation of active values. is there. Here, the bidirectional associative memory, which is the basis of associative reasoning, will be described first, and then the fuzzy associative memory system in which this is systemized will be described.

FIG. 13 is a diagram for explaining the concept of bidirectional associative memory (BAM). As shown in FIG. 13, the layer _{L A} and layer _{L B} is has a network of the layer _L plurality of nodes to _{A a 1, a 2, ...} , there is _{a n,} a plurality in layers _{L B} Nodes b ₁ , b ₂ ,..., B _p exist. The connection weight between nodes is expressed by an association matrix MεR ^{n × p} . Pattern A∈R ⁿ for this association matrix M Layer _{L A} and layer _{L B,} when to be noted the pair B∈R ^p, given A'∈R ⁿ containing noise to the layer _{L A} as also, by the echo operation between layers L _a and layer L _B, the pattern B is recalled. Conversely, the pattern A will occur can give a pattern B'∈R ^p containing noise to the layer L _B. Association in this bidirectional associative memory is performed by the following equations (23.1) and (23.2).

Here, a _i (t) and b _j (t) are active value vectors of the respective nodes in step t and take values belonging to the section [0, 1]. It may be a value of {0, 1}. φ represents each node A (t) = (a ₁ (t), a ₂ (t),..., a _n (t)), B (t) = (b ₁ (t), b ₂ (t),. , B _p (t)), for example, a sigmoid function is used. The association matrix M is obtained by the following equations (24.1) and (24.2) when the pattern pairs to be recorded are (A ₁ , B ₁ ),..., (A _m , B _m ). .

Here, β _i is an associative coefficient. In the Kosko method, when an associative matrix M is obtained, a pattern pair obtained by bipolar conversion of a pattern pair of {0, 1} is used. However, if the associative matrix M obtained by the Kosko method is used, the network may cause saturation, and adjustment is necessary. In order to solve this problem, a method for normalizing the associative matrix M has been proposed. Therefore, in the fuzzy associative memory system used in the present embodiment, a normalized associative matrix obtained by this normalization method is used.

There are restrictions on the pattern pairs that can be memorized by bidirectional associative memory (BAM). One is that the orthogonality between pattern pairs is strong and the Hamming distance between patterns and pattern complements must be sufficiently separated within each layer. Another is that the continuity assumption needs to hold. The continuity assumption is expressed by the following equation (25).

  Here, H is a Hamming distance. That is, it can be said that it is difficult to record unless the pattern to be recorded in each layer is as far as the Hamming distance.

  Next, a method for constructing a fuzzy associative memory system will be described.

FIG. 14 is a schematic diagram expressing fuzzy rules using an associative memory network. In setting the layers, the layer representing the antecedent part of the fuzzy rule is the If (X) layer, and the layer representing the consequent part is the Then (Y) layer. A node in the If (X) layer represents a fuzzy set (membership function) of the antecedent part, and a node in the Then (Y) layer represents a fuzzy set (membership function) of the consequent part or an input / output function. . Further, since the correlation between the antecedent part and the consequent part exceeds the limit that BAM can store, one node sets a Rule (R) layer representing one rule. Therefore, the fuzzy rule can be expressed by configuring a BAM between the If (X) layer and the Rule (R) layer (associative matrix M _xr ). Furthermore, in the Rule (R) layer, in order to prevent the explosion of ambiguity, a coordinator combination (M _rr ) having a function of suppressing activation of other nodes is set. Coordinator coupling means coupling to a node with a positive weight at an arbitrary node in the same layer and coupling to another node with a negative weight.

Next, the concept of construction on the memory will be described by taking the network configuration of FIG. 14 as an example. FIG. 15 shows knowledge necessary for constructing a network of associative memory. First, in order to obtain an association matrix between each layer, the correlation of fuzzy rules is replaced with a value belonging to the interval [0, 1]. In some cases, a binary value of 0 or 1 indicated by {0, 1} may be substituted. The result is as shown in the following equation (26).

  In this equation (26), as an example, the correlation is represented by binary values of 0 and 1 represented by {0, 1}, but it is needless to say that the present invention is not limited to this.

  Using this calculation result as a pattern pair to be recorded, an associative matrix between each layer is obtained by Expression (24) -1 and Expression (24) -2. However, the Kosko method obtained using a pattern pair obtained by bipolar conversion is not used.

As described above, the fuzzy associative memory system used in the present embodiment uses the normalized associative matrix at the time of inference. Thus, where an associative matrix obtained saves obtains the normalized associative matrix M _e by the following equation (27).

M _e = a (M + B) (27)
Here, each parameter is as follows.

Here, M, M _e , BεR ^{m × n} , and m _ij are M (i, j) elements.

  Here, when a plurality of rule sets are defined, the above processing is repeated by the number of rule sets to propose a plurality of networks. Then, information relating to the connection between the rule sets is read from the text file, and the respective networks are connected. At the time of inference, the weight (association matrix) of the connection between defined rule sets is also normalized. From here, the network constructed by the fuzzy rule R is referred to as a network R.

Next, the operation process of the fuzzy associative memory system will be described.
Inference is performed by repeating the propagation of active values while maintaining synchronization throughout the network. First, a value obtained by evaluating the input with the membership function is input to the If (X) layer. Then, by applying the associative matrices M _xr , M _rr , and M _ry to the expressions (23.1) and (23.2), the propagation of the active value is repeated for the set number of associative steps. For example, the output out _j of each node OUT layers connected by associative matrix _{M r} multiple INr layers is calculated by equation (31).

The proposed initial associative reasoning does not integrate reasoning results and can cause excessive ambiguity. In order to solve this, the active values of the Then (Y) layer are normalized using equations (32.1) to (32.3).

Here, n is the number of nodes in the Then (Y) layer, γf _i is the activation value of node i, and γ _i is the contribution.

Thereby, the contribution degree γ _i is obtained. An inference result is obtained by interpolating the output value of the membership function of the consequent part or the input / output function with this contribution, and MAX functioning (summing) the obtained function (value).

  Next, intention recognition determination processing by the intention recognition unit 207 based on the actually constructed fuzzy associative memory system will be described with reference to the data flow diagrams of FIGS. 21 and 22.

  First, before executing the intention recognition determination process, a membership function used for the intention recognition determination process and N individuality data are registered in the storage device 39 in advance.

More specifically, the speed membership function mu _vS, mu _vB, face direction membership functions mu _.theta.S, mu _.theta.B, stored in the storage device 39 located membership function _{mu LS,} a mu _LB as input membership functions. Also, as the N personality data, associative matrix ^{_{M xr1, M T xr1 ~M xrN}} , M T xrN, normalized matrix _{M rr,} associative matrix ^{_{M ry1, M T ry1 ~M ryN}} , the _{M T RYn} as personality data It is stored in the storage device 39. Further, the driving intention membership functions μ _LS , μ _LB , μ _SS , μ _SB , μ _RS , μ _{RB to} μ _LS , μ _LB , μ _SS , μ _SB , μ _RS , μ _RB are stored as output membership functions. 39.

  N individuality data can be registered, for example, by the following processing.

For driving behavior for N persons, vehicle speed information V _v , face direction information θ _D and vehicle position information V _L are calculated, and the calculated values are set as input nodes. Then, the left turn, straight ahead, and right turn, which are the results of the driving action, are set as output nodes. An intermediate node is set as a buffer for these input nodes and output nodes. Then, by applying the expressions (23.1), (23.2), (24.1), and (24.2) to the input node and the intermediate node, the associative matrix M _xr1, M ^T _xr1 ~M _xrN, to calculate the _{^M T} ^xrN. On the other hand, by applying the expressions (23.1), (23.2), (24.1), and (24.2) to the intermediate node and the output node, the associative matrix M _{ry1, M} ^T _ry1 ~M ryN, to calculate the _{^M T} ^ryN.
As described above, N individuality data are obtained as an association matrix.

Next, intention recognition determination processing is performed as follows using the registration data obtained as described above.
First, the intention recognition unit 207, based on the input vehicle velocity V _v , face direction θ _p , and position V _L , respectively, velocity membership functions μ _vs , μ _vB , face direction membership functions μ _θs , μ _θB , The function values (velocity grade μ _vs (V _v ), μ _vB (V _v ), face direction grade μ _θs (θ _p ), μ _θB (θ _p ), position grade μ _{LS are obtained from the} position membership functions μ _LS and μ _LB. (V _L ), μ _LB (V _L )) are calculated.

Next, matrix calculation in the positive direction using associative memory is performed.
Specifically, first, μ _vs (V _v ), μ _vB (V _v ), face direction grades μ _θs (θ _p ), μ _θB (θ _p ), position grades μ _LS (V _L ), μ _LB ( Let V _L ) be associative memory inputs IN _vs , IN _vB , IN _ps , IN _pB , IN _Ls , IN _LB. Then, to these associative memory input, performs N number of matrix calculation using the matrix _M xr1 _{~M XRN} corresponding to the N personality data, respectively obtained an associative memory intermediate output _R 1 to R _N. The associative memory intermediate outputs R _{1 to} R _N are normalized by the matrix M _rr every time they are calculated. Further, the obtained associative memory intermediate outputs R _{1 to} R _N are further _subjected to N matrix calculations using the matrices M _{ry1 to} M _ryN , respectively, and associative memory outputs OUT _L1 , OUT _L expressed as left turn, straight _advance , and right turn _S1 , OUT _{R1 to} OUT _LN , OUT _SN , and OUT _RN are obtained. The associative memory outputs OUT _{L1 to} OUT _LN are output values corresponding to the driving intention “left turn”, the associative memory outputs OUT _{s1 to} OUT _sN are output values corresponding to the driving intention “straight ahead”, and the associative memory outputs OUT _{R1 to} OUT _RN. Is an output value corresponding to the driving intention “right turn”.

Next, reverse matrix calculation using an associative memory is performed.
Based on the obtained associative memory outputs OUT _L1 , OUT _S1 , OUT _{R1 to} OUT _LN , OUT _SN , and OUT _RN , N matrix calculations are performed using the associative matrix M ^T _{ry1 to} M ^T _ryN , and the associative memory intermediate output R obtain ₁ to R _N. Each of the associative memory intermediate outputs R _{1 to} R _N is obtained for a plurality of nodes. Then, N matrix calculations are performed on the obtained associative memory intermediate outputs R _{1 to} R _N using the matrices M ^T _{xr1 to} M ^T _xrN , and associative memory inputs IN _vs , IN _vB , IN _ps , IN _pB , IN _Ls , IN _LB are obtained.

A reverberation operation is performed in which forward and backward calculations using these associative memories are repeated a predetermined number of times, and associative memory outputs OUT _L1 , OUT _S1 , OUT _{R1 to} OUT _LN , OUT _SN , Let OUT _RN converge. Then, the converged associative memory outputs OUT _L1 , OUT _S1 , OUT _{R1 to} OUT _LN , OUT _SN , OUT _RN are normalized to normalize associative memory outputs OUT ^* _{L 1} , OUT ^* _{S 1} , OUT ^* _{R 1} to OUT ^* _LN , OUT ^* _SN and OUT ^* _RN are obtained. Driving intention grades μ _LS1 (OUT ^* _L1 ), μ _LB1 (OUT ^* ) using the driving intention membership functions μ _LS , μ _LB , μ _SS , μ _SB , μ _RS , μ _RB for the obtained normalized associative memory output ^{_{L1), μ SS1 (OUT *}} S1), μ SB1 (OUT * S1), μ RS1 (OUT * R1), μ RB1 (OUT * R1) ~μ LSN (OUT * LN), μ LBN (OUT * LN) , _ΜSSN (OUT ^* _SN ), _μSBN (OUT ^* _SN ), _μRSN (OUT ^* _RN ), and _μRBN (OUT ^* _RN ) are obtained.

  Based on the driving intention grade for the obtained N individuality data, the determination means 207a determines the driving intention and obtains an intention recognition result (left turn, straight ahead, right turn). The determination of the determination means 207a is made based on a predetermined determination condition (for example, the one having the largest value of the driving intention grade, that is, the one recalled most strongly) is selected from the N driving intention grades. An intention (for example, a right turn) that is previously associated with the determined driving intention grade is output as an intention recognition result.

  The determination conditions of the determination unit 207a are not limited to those described above. For example, when the difference between the largest driving intention grade and the next largest driving intention grade is small, the driving intention grade with the largest driving intention grade and the driving intention corresponding to the next largest driving intention grade are averaged. May be.

  Next, an example in which intention inference is performed by applying the human state estimation system shown in the above-described embodiment will be described.

First, acquisition of driver's face direction information will be described.
We focused on the driver's face movement as the main information for intention inference. In order to acquire accurate and highly robust information, the field-of-view image acquisition camera 21 is installed above and in front of the driver, and using the above-described I-Space, the facial motion information is converted into the facial direction information θ by the following process. Obtained as _D.

1) A fixed point to be tracked forward, right front, and left front is set by the field-of-view image acquisition camera 21, and the second face direction information _θD2 is acquired from the coordinate information sent from the I-Space.

2) The driver's face and lips are tracked by the vehicle-mounted camera 24 to obtain the first face direction information θ _D1 .

3) by the reconstruction information using care function to obtain the face direction provisional information theta _P, robustness, improve the accuracy.

  4) The parameters of the vehicle-mounted camera 24 are corrected using the Newton method, and the accuracy of the vehicle-mounted camera 24 alone is improved.

5) At the time of use, the first face direction information θ _D1 is obtained based on the image of the vehicle-mounted camera 24 using the parameter-corrected calculation formula, and the obtained first face direction information θ _D1 is used as the face direction information θ _D. Set to.

Next, intention inference will be described.
Stereo vision using the cameras 31 and 32, driver face direction information θ _D , and environmental information from the control unit 2d are acquired, and intention inference is performed. More specifically, an experiment was conducted on a road 41 with poor visibility as shown in FIG.

The vehicle 2 goes straight on the road 41 toward the intersection where the roads 41 and 42 intersect. The vehicle position information _VL of the vehicle 2 at this time is acquired by a stereo vision system including the cameras 31 and 32 installed at the intersection. At the same time, the driver's 10 face direction information θ _D and the vehicle speed information V _v are acquired and intentional inference is performed using a fuzzy associative memory system, whether the driver 10 is going to turn right or straight ahead, Recognize whether you are going to turn left. In addition, the recognition rate is improved by increasing the number of learning cases. In this example, as shown in FIG. 18, an experiment was performed using basic driving behavior data having characteristics of seven Japanese patterns. These cases were accumulated in the middle layer 53 shown in FIG.

  FIG. 19 shows the result obtained without using the context when data for which driving behavior has not been learned is input to the intention recognition model. In FIG. 19, each fuzzy label 521a to 521c, 522a to 522c, 523a to 523c represents the magnitude of the membership grade obtained based on the actually given input parameters, and each of the case nodes 531 to 539. Shows the activity value distribution in the already obtained cases. In the result of FIG. 19, the activity value distribution of the middle layer 53 is ambiguous, and it cannot be determined which driving intention. On the other hand, FIG. 20 shows a result using the context. The expressions of the fuzzy labels 521a to 521c, 522a to 522c, 523a to 523c and the case nodes 531 to 539 are the same as those in FIG. In the case of FIG. 20, the magnitudes of the values obtained for the nodes 541 to 543 of the upper layer 54 are also shown. The node activity value distribution converges to “left” by the BAM echo operation. Even when the input value or the initial active state is ambiguous in this way, the active value distribution converges to a context-dependent state by the echo operation.

  As described above, the human state estimation system according to the present embodiment is constructed as a natural interface related to human actions, and an intelligent network system that attaches a cooperation function to the network and places importance on interaction with the user. The city model is shown in FIG. FIG. 16 is a schematic diagram of a welfare disaster prevention cyber city. As shown in FIG. 16, a plurality of cameras 31 and 32 are arranged near an intersection or road in the environment around the station. When a plurality of vehicles 2 are parked, the vehicle 2 enters the intersection, or a pedestrian 1 comes out from the vicinity of the station, the human state is estimated based on images captured by these cameras 31 and 32. Is done.

  More specifically, as shown in FIG. 17, the two cameras 31, 32 as the distributed sensory intelligence and the vehicle 2a as another distributed sensory intelligence cooperate with each other to estimate the driver's state. By informing the vehicle 2a that the obstacle 172 or the dangerous object 171 is approaching in the traveling direction, or the vehicle 2b as another distributed sensory intelligence cooperates with the vehicle 2a to estimate the driver's state. It is possible to inform the driver of the vehicle 2b that the vehicle 2a is approaching the vehicle.

  In this way, by using distributed sensory intelligence that has a sensory part consisting of various sensors and intelligence realized by CPU arithmetic processing, each intelligent agent cooperates autonomously, Provide appropriate support and guides. By expanding such a distributed sensory intelligence space into a city space where people live, we can realize a city that is strong in disasters and has a real sense of security with a view from daily life to emergency assistance in the event of a disaster.

  The present invention is not limited to the above embodiment. The city space shown in FIG. 1 is only an example to which the human state estimation system of this embodiment is applied. Distributed sensory intelligence may be placed in other urban spaces. Also, the distributed sensory intelligence is not limited to that shown in the above embodiment. Any sensor that detects a physical quantity of the environment, and any processing calculation unit that generates environment information based on the detection result of this sensor can be applied as the distributed sensory intelligence. In this case, for example, the human action result is expressed not only from the direct action result of the human such as the face direction but also from any physical quantity obtained as a result of the action of the human driving the vehicle such as the vehicle speed and the vehicle position. Environmental information can be generated and used for intention recognition.

Further, as shown in the example of FIG. 2, in one distributed sensory intelligence, the sensor and the processing calculation unit do not need to be arranged at the same position in the space. For example, as the first distributed sensory intelligence for calculating the first face direction information θ _D1 , a vehicle-mounted camera 24 that functions as the sensor and a first face direction information calculation unit 203 that functions as a processing calculation unit are shown. . In addition, as the second distributed sensory intelligence for calculating the second face direction information θ _D2 , a visual field image acquisition camera 21 that functions as a sensor and a second face direction information calculation unit 204 that functions as a processing calculation unit are shown. Yes. Further, as a third dispersion sensory intelligence for calculating the speed information V _v, the vehicle speed detecting device 25 that functions as a sensor and processing operation section is shown. These first to third distributed sensory intelligences are all arranged in the vehicle 2 and are not spatially separated. On the other hand, in the fourth distributed sensory intelligence for calculating the vehicle position information _VL , the cameras 31 and 32 that function as sensors are installed at the intersection, and the vehicle position information calculation unit 202 that functions as a processing calculation unit thereof is the vehicle. 2 is arranged. In this way, the sensor and the processing calculation unit may be arranged at different positions. Of course, the vehicle position information calculation unit 202 may be installed on the intersection side together with the cameras 31 and 32.

In the example of FIG. 2, the information reconstruction unit 205 reconstructs information based on the first face direction information θ _D1 and the second face direction information θ _D2, and the value further corrected by the Newton method correction unit 206 Although the example in which the intention recognizing unit 207 performs the intention recognition process based on the above is shown, the present invention is not limited to this. For example, only the vehicle-mounted camera 24 or the field-of-view image acquisition camera 21 may be installed, and the intention recognition unit 207 may perform intention recognition processing based on face direction information obtained from one of them. In this case, it is not necessary to arrange the information reconstruction unit 205. Further, the Newton method correction unit 206 can be omitted. In addition to the Newton method correction unit 206, a correction unit that performs another correction method may be arranged, and the intention recognition unit 207 may perform intention recognition processing based on the correction value.

In the above embodiment, an example in which human intention recognition processing is performed based on the vehicle speed information V _v , the vehicle position information V _L, and the face direction information θ _D has been described, but the present invention is not limited to this. For example, the intention recognition process may be performed by adding handle angle information to the input parameters. Of course, other environmental information may be added to the input parameters.

  As described above, the present invention is used in a human support system that supports elderly people and persons with physical disabilities who have been marginalized from information technology living in an urban space with an information infrastructure and have not been able to fully benefit from it. The present invention is effective in the technical fields of human state estimation system, human state estimation method, and human state estimation program.

The figure which shows an example of the distributed sensory intelligence space in the city space where the human state estimation system which concerns on one Embodiment of this invention is applied. The figure which shows an example of each structure of the human state estimation system which concerns on the embodiment. The figure for _{demonstrating} the calculation method of the vehicle position _VL which concerns on the same embodiment. FIG. 2 is an overview diagram of a vehicle according to the embodiment. The figure which showed typically a part of function of the vehicle which concerns on the embodiment. The figure for demonstrating acquisition of the face direction information which concerns on the embodiment. The schematic diagram for demonstrating the example of calculation of the face direction information (theta) _D1 which concerns on the embodiment. The schematic diagram for _{demonstrating} the calculation method of face direction information (theta) _{D2 which concerns on} the embodiment. The schematic diagram for _{demonstrating} the calculation method of face direction information (theta) _{D2 which concerns on} the embodiment. The conceptual diagram for demonstrating the reconstruction process of the information reconstruction part which concerns on the embodiment. The conceptual diagram of the intention recognition model which concerns on the same embodiment. The figure which shows the concept of CFS (concept fuzzy set) which concerns on the same embodiment. The figure for demonstrating the concept of the bidirectional associative memory (BAM) which concerns on the same embodiment. The schematic diagram which expressed the fuzzy rule which concerns on the embodiment using the associative memory network. The figure which shows the knowledge required when constructing | assembling the network of the associative memory which concerns on the embodiment. The figure which shows an example of the city model to which the human state estimation system which concerns on the embodiment is applied. The figure which shows the function example of the human condition estimation system which concerns on the same embodiment. The figure which shows the example which experimented using the characteristic basic driving action data of the Japanese 7 pattern which concerns on the embodiment. The figure which shows the result obtained without using a context, when the data from which the driving action which concerns on the embodiment is not learned are input into the intention recognition model. The figure which shows the intention recognition result obtained using the context which concerns on the same embodiment. The figure for demonstrating the intention recognition determination process by the intention recognition part based on the fuzzy associative memory system actually constructed | assembled based on the embodiment. The figure for demonstrating the intention recognition determination process by the intention recognition part based on the fuzzy associative memory system actually constructed | assembled based on the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 2 ... Vehicle, 10 ... Driver, 21 ... Visual field image acquisition camera, 22 ... Display part, 24 ... Vehicle mounted camera, 31, 32 ... Camera, 41, 42 ... Road, 201 ... Communication control means, 202 ... Vehicle position information Calculation unit, 203 ... first face direction information calculation unit, 204 ... second face direction information calculation unit, 205 ... information reconstruction unit, 206 ... Newton method correction unit, 207 ... intention recognition unit

Claims (5)

A human state estimation system that estimates driving behavior that a driver of a vehicle will perform when the vehicle reaches a preset target point before the vehicle reaches the target point,
A vehicle position calculating device for calculating position information of the vehicle with respect to the target point;
A speed information detecting device for detecting speed information of the vehicle;
The driver's face image data is obtained by a camera mounted on the vehicle, and information representing the driver's face direction is detected from the face image data by detecting the center of gravity of a specific area of the face by image recognition processing. A face direction calculating device,
An input membership function preset for each of the vehicle position, speed and direction of the driver's face, an output membership function preset for each of a plurality of assumed driving intentions, and each of the input memberships An associative memory that stores a plurality of individuality data for associating a function with each output membership function in a matrix, and the vehicle position information calculated by the vehicle position calculation device is detected by the speed detection device. Speed information and information indicating the driver's face direction detected by the face direction calculation device are input to the associative memory as input parameters, and the associative memory is used to perform forward and reverse matrix calculations. Multiple output membership functions converged by repeatedly executing a predetermined number of times, and this output A driving intention grade corresponding to each of the plurality of output membership functions is selected, and the intention corresponding to the selected driving intention grade is output as the driver's intention recognition result. A human state estimation system comprising an intention recognition unit. The vehicle position calculation device includes:
A plurality of cameras arranged at the target point, imaging the vehicle from different directions and transmitting the image data to the vehicle;
The vehicle is installed in the vehicle, receives a plurality of vehicle image data sent from the plurality of cameras, and is represented by three-dimensional coordinates using a stereo vision function based on the received vehicle image data. The human state estimation system according to claim 1, further comprising a vehicle position information calculation unit that calculates position information of the vehicle. The face direction calculation device includes:
Means for detecting the center-of-gravity coordinates of the mouth region of the driver by image recognition processing from the face image data obtained by the camera;
An angle formed by a straight line Lo connecting the camera and the center of the driver's head and a straight line Lx connecting the camera and the center of gravity coordinates of the detected mouth area is θ _A , An angle formed by a straight line connecting the center and the center-of-gravity coordinates of the detected mouth area and the straight line Lx is θ _B , and a distance between the camera and the center of the driver's head is d, When the head radius determined by the head model is r and the angle of view of the camera is φ, the driver's face direction θ _D2 is
r / sin θ _A = d / sin θ _B
sin θ _B = (d / r) · sin θ _A
θ _B = 180−arcsin (sin θ _B ) · 180 / π
θ _D2 = 180− (θ _A + θ _B )
The human state estimation system according to claim 1, further comprising: A human state estimation method for estimating a driving action that a driver of a vehicle will perform when the vehicle reaches a preset target point before the vehicle reaches the target point,
Calculating position information of the vehicle relative to the target point;
Detecting the vehicle speed information;
The driver's face image data is obtained by a camera mounted on the vehicle, and information representing the driver's face direction is detected from the face image data by detecting the center of gravity of a specific area of the face by image recognition processing. The process of
An input membership function preset for each of the vehicle position, speed and direction of the driver's face, an output membership function preset for each of a plurality of assumed driving intentions, and each of the input memberships An associative memory for storing a plurality of individuality data for associating a function with each output membership function in a matrix, the calculated vehicle position information, the detected speed information, and the detected driving A plurality of output membership functions that have converged by inputting information representing the face direction of the person as an input parameter and repeatedly executing matrix operations in the forward direction and the reverse direction a predetermined number of times using this associative memory. The value is the highest among the driving intention grades corresponding to the output membership functions. Select heard what the human state estimation method characterized by comprising the steps of outputting the intent corresponding to the selected driving intention grade as intended recognition result of the driver. A human condition estimation program used to estimate a driving action that a driver of a vehicle will perform when the vehicle reaches a preset target point before the vehicle reaches the target point. ,
Processing for calculating position information of the vehicle with respect to the target point;
Processing for detecting speed information of the vehicle;
The driver's face image data is obtained by a camera mounted on the vehicle, and information representing the driver's face direction is detected from the face image data by detecting the center of gravity of a specific area of the face by image recognition processing. Processing to
An input membership function preset for each of the vehicle position, speed and direction of the driver's face, an output membership function preset for each of a plurality of assumed driving intentions, and each of the input memberships An associative memory for storing a plurality of individuality data for associating a function with each output membership function in a matrix, the calculated vehicle position information, the detected speed information, and the detected driving A plurality of output membership functions that have converged by inputting information representing the face direction of the person as an input parameter and repeatedly executing matrix operations in the forward direction and the reverse direction a predetermined number of times using this associative memory. The value is the highest among the driving intention grades corresponding to the output membership functions. Selecting a threshold and causing a computer provided in the vehicle to execute a process of outputting an intention corresponding to the selected driving intention grade as an intention recognition result of the driver. Estimation program.

Images