JP7617644B2 - Pedestrian movement direction determination system and pedestrian movement direction determination software - Google Patents

Description

The present invention relates to a pedestrian movement direction determination system and pedestrian movement direction determination software. More specifically, the present invention relates to a pedestrian movement direction determination system and pedestrian movement direction determination software for people with poor walking skills, such as the elderly and visually impaired.

For people with walking disabilities, going outside and walking along the road to their destination is extremely difficult, no matter how simple the path may be.

To help those with mobility issues, especially those who are visually impaired, there are braille blocks, guide bells, braille signs, and white canes, and guide dogs and walking training by walking trainers are also available.

In addition, various techniques have been proposed to assist people with walking disabilities.
For example, Patent Document 1 describes an invention in which a pedestrian terminal transmits a message including pedestrian information to a roadside unit from an ITS communication unit, and when the roadside unit receives the message transmitted from the pedestrian terminal using the ITS communication unit, it accumulates the pedestrian information included in the message in a memory unit as passage history information. Based on the passage history information in the memory unit, a control unit detects route guidance information (pedestrian support information) regarding evacuation routes in the event of a disaster, in particular people in need of assistance such as elderly people and people who frequently engage in risky behavior, generates route guidance information for these people, and transmits a message including that route guidance information to nearby pedestrian terminals.

JP 2019-53371 A

By the way, visually impaired people often worry that, for example, when crossing a pedestrian crossing, they may walk in a direction that strays from the crosswalk, and even when walking on a regular sidewalk, they may walk toward a curb or step.

The invention described in Patent Document 1 can generate information to assist pedestrians based on information collected from pedestrian terminals, but this is insufficient to alleviate the above-mentioned anxieties of visually impaired people.

The present invention was devised in light of the above points, and aims to provide a pedestrian movement direction determination system and pedestrian movement direction determination software that can determine the direction in which a pedestrian is safe and secure when moving forward.

In order to achieve the above object, the pedestrian travel direction determination system of the present invention includes an imaging unit capable of capturing an image at least in front of a pedestrian; an object detection unit capable of detecting a boundary line of an installed object that extends at an angle to the vertical direction of the captured image in at least a lower region of the captured image, in an image captured by the imaging unit and having an image center line that passes through the center point of the captured image and extends in the vertical direction of the captured image; a travel direction detection unit capable of detecting whether the boundary line intersects with the image center line and whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line; and a travel direction determination unit capable of determining that the pedestrian is facing normally when the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line.

Here, the direction of travel detection unit can detect whether a boundary line intersects with an image center line, and can detect whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, and can detect the direction of a pedestrian relative to an installed object having an inclined boundary line in the captured image.
Moreover, a "boundary line segment" means a straight line segment.

In addition, if the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, the travel direction determination unit, which can determine that the pedestrian's orientation is normal, can confirm whether the pedestrian's orientation relative to an installed object having an inclined boundary line is normal.
In addition, in the present invention, the "normal state" means a state in which a pedestrian is facing a direction in which he or she can proceed safely and securely.

In addition, in the pedestrian travel direction determination system of the present invention, the boundary line segment can be configured as a left boundary line segment whose upper end x-coordinate value in the coordinate system of the captured image is greater than the lower end x-coordinate value, and a right boundary line segment whose upper end x-coordinate value in the coordinate system of the captured image is smaller than the lower end x-coordinate value.

Here, the "coordinate system of the captured image" is an image coordinate system in which, when the captured image is rectangular, the upper left corner is the origin (0,0), the x-axis runs horizontally to the right from the origin, and the y-axis runs vertically downward from the origin.
In this case, when the travel direction detection unit detects that the image center line intersects with the left boundary line segment or the right boundary line segment, it is possible to determine whether the pedestrian is facing toward the left or right side of the installed object having the inclined boundary line segment, making it easy to correct the pedestrian's orientation.

The pedestrian travel direction determination system of the present invention further comprises a detected object information recording unit capable of recording installed object information, which is information relating to an installed object, and an object identification unit capable of identifying an installed object by comparing the information on the installed object detected by the object detection unit with the installed object information recorded by the detected object information recording unit. When the object identification unit identifies the installed object as a pedestrian crossing, the travel direction detection unit can be further configured to detect whether the image center line intersects with a pedestrian crossing arrival gate line segment that is the portion of a horizontal line segment having the smallest y coordinate value, which is the vertical coordinate value in the coordinate system of the captured image, of a crosswalk existence area rectangle corresponding to the entire crosswalk that is displayed in the captured image when the object detection unit detects an installed object, that is, a portion that is greater than the x coordinate value of the intersection with the left boundary line segment or smaller than the x coordinate value of the intersection with the right boundary line segment.

In this case, it becomes easier to check whether the direction of the pedestrian in relation to the crosswalk is normal.

In addition, in the pedestrian travel direction determination system of the present invention, the travel direction detection unit can be further configured to detect whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line is within a predetermined range when the object identification unit identifies an installed object as a pedestrian traffic light other than the installed object identified as a crosswalk by the object identification unit, and the travel direction determination unit can be further configured to determine that the pedestrian's direction is normal when the travel direction detection unit detects that the crosswalk arrival gate line intersects with the image center line and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line is within a predetermined range.

Here, the "shortest distance" refers to the distance (difference) between x coordinate values when the y coordinate values are the same in the coordinate system of the captured image.
In addition, the "predetermined range" here means a range greater than "0" and equal to or less than the shortest distance between the imaginary center line that runs through the center of the crosswalk and the coordinates of the pedestrian traffic light.
In this case, it is possible to check with even greater accuracy whether the direction of the pedestrian with respect to the crosswalk is normal.

The pedestrian travel direction determination system of the present invention further includes a notification unit capable of notifying the result of the determination made by the travel direction determination unit, and the travel direction determination unit can determine that the pedestrian's orientation is in a first abnormal state when the travel direction detection unit detects that the image center line intersects with the left boundary line segment, and can determine that the pedestrian's orientation is in a second abnormal state when the travel direction detection unit detects that the image center line intersects with the right boundary line segment, and the notification unit can be configured to notify information encouraging the pedestrian to turn right when the travel direction determination unit determines that the pedestrian is in the first abnormal state, and to notify information encouraging the pedestrian to turn left when the travel direction determination unit determines that the pedestrian is in the second abnormal state.

In this case, it is easier to correct the pedestrian's direction.

In addition, in the pedestrian travel direction determination system of the present invention, the notification unit can be a vibration notification unit having a main body and multiple vibrators, the main body has a longitudinal direction and can be detachably connected to one longitudinal side and the other side located opposite to the one side to form a ring shape, the vibrators are attached to one surface of the main body in a line in the longitudinal direction of the main body, and the vibration notification unit can vibrate the vibrator located on one side of the ring-shaped main body when notifying information encouraging a right turn, and can vibrate the vibrator on the other side located opposite to the vibrator on one side when notifying information encouraging a left turn.

In this case, one side of the main body is connected to the other side to form the main body into a ring shape, and the main body can be attached to the pedestrian's body so that, for example, one side of the main body is positioned on the right side of the pedestrian's body, and the notification can be made more quickly than by audio.

The pedestrian travel direction determination system of the present invention includes a notification unit capable of notifying the result of the determination made by the travel direction determination unit, and a route information setting unit capable of setting route information, which is information about the route between the departure point and the destination, based on information about the departure point and the destination, and the notification unit can be configured to notify a change of direction based on the route information set by the route information setting unit, and the travel direction determination unit can be configured to determine that the pedestrian is in a direction change state if the travel direction detection unit detects that the boundary line segment and the image center line intersect within a predetermined time from when the notification unit notifies the user of the change of direction based on the route information.

In this case, when the notification unit notifies the pedestrian to change direction based on the route information, for example, from the time when the notification unit notifies the pedestrian to turn right or left on the sidewalk until a predetermined time has elapsed, the travel direction determination unit determines that the pedestrian is in a direction change state rather than determining whether the pedestrian's direction is normal or not, so that the pedestrian can concentrate on changing direction.

Here, "a certain amount of time" generally refers to the time it takes for a pedestrian to change direction while moving forward, from when they start to finish changing direction, for example 3 to 10 seconds.

In order to achieve the above object, the pedestrian travel direction determination software of the present invention is a pedestrian travel direction determination software for determining the direction of a pedestrian when traveling, and is software for causing an information processing device to function as a means including: an object detection unit capable of detecting a boundary line of an installed object that is an installed object and extends at an angle to the vertical direction of the captured image in at least the lower region of the captured image in an image obtained by an imaging unit capable of imaging at least the area in front of the pedestrian, the boundary line passing through the center point of the captured image and extending in the vertical direction of the captured image; a travel direction detection unit capable of detecting whether the boundary line intersects with the image center line and whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line; and a travel direction determination unit capable of determining that the direction of the pedestrian is normal when the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line.

Here, the direction of a pedestrian relative to an installed object having an inclined boundary line in the captured image can be detected by a direction detection unit that can detect whether the boundary line intersects with the image center line and can detect whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line.

In addition, if the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, the travel direction determination unit, which can determine that the pedestrian's orientation is normal, can check whether the pedestrian's orientation is normal relative to the installed object that has an inclined boundary line.

The pedestrian travel direction determination system according to the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner.
The pedestrian travel direction determination software according to the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner.

1 is a schematic diagram showing an example of the configuration of a pedestrian travel direction determination system to which the present invention is applied. 1A is a schematic diagram showing a first example of an image captured in a pedestrian travel direction determination system to which the present invention is applied, and FIG. 1B is a schematic diagram showing an example of an augmented reality image based on the image captured in FIG. 1A. FIG. 2(a) is a schematic diagram showing an example of an augmented reality captured image at the next stage of the augmented reality captured image shown in FIG. 2(b), and FIG. 2(b) is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian's direction being determined by a direction determination unit of a pedestrian direction determination system to which the present invention is applied, based on the augmented reality captured image shown in schematic diagram (a). 1 is a schematic diagram illustrating an example of a vibration notification unit included in a pedestrian travel direction determination system to which the present invention is applied. FIG. 1A is a schematic diagram showing a second example of an image captured in a pedestrian travel direction determination system to which the present invention is applied, and FIG. 1B is a schematic diagram showing an example of an augmented reality image including the result of a pedestrian's direction being determined by a travel direction determination unit of the pedestrian travel direction determination system to which the present invention is applied based on an augmented reality captured image of the image shown in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings for the purpose of understanding the present invention.
FIG. 1 is a schematic diagram showing an example of the configuration of a pedestrian travel direction determination system to which the present invention is applied.

FIG. 2(a) is a schematic diagram showing a first example of a captured image in a pedestrian travel direction determination system to which the present invention is applied. FIG. 2(b) is a schematic diagram showing an example of an augmented reality captured image based on the captured image shown in FIG. 2(a).

Also, FIG. 3(a) is a schematic diagram showing an example of an augmented reality captured image at a next stage of the augmented reality captured image shown in FIG. 2(b). Also, FIG. 3(b) is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian's direction being determined by a direction determining unit of a pedestrian direction determining system to which the present invention is applied, based on the augmented reality captured image shown in FIG. 3(a).

The pedestrian travel direction determination system 1 of the present invention shown in FIG.
Here, the glasses device 2 has the shape of typical glasses, and the server 3 is capable of communicating with the glasses device 2 via the Internet 14 .
The glasses device 2 is worn by, for example, a visually impaired pedestrian.
The glasses device 2 is a device having a general information processing function.

The glasses device 2 also includes an imaging camera 4 capable of capturing an image of at least the area ahead of the pedestrian.
Here, the imaging camera 4 is an example of an imaging unit.

Furthermore, as long as the device has an imaging camera 4 capable of capturing an image at least in front of the pedestrian and is capable of communicating with the server 3 via the Internet 14, it does not necessarily have to be a glasses device 2; for example, a smartphone can be hung around the pedestrian's neck and placed at chest level to capture an image of the area in front of the pedestrian.
In other words, the imaging direction of the imaging camera 4 is the direction in which the pedestrian is facing.

The glasses device 2 also includes a captured image transmission unit 4A.
Here, the captured image transmission unit 4A is capable of transmitting the captured images captured by the imaging camera 4 to the server 3 via the Internet 14.

The captured image also has an image centerline that passes through the center point of the captured image and extends vertically through the captured image.

On the other hand, the server 3 includes a captured image receiving unit 4B.
Here, the captured image receiving section 4B is capable of receiving the captured image transmitted by the captured image transmitting section 4A.

The glasses device 2 also includes an object detection unit 5 .
Here, the object detection unit 5 is capable of detecting an installed object, which is an object that has been installed, in the captured image.

Furthermore, the object detection unit 5 of the glasses device 2 can also detect the boundary line of the installed object.
Moreover, this boundary line segment is a line segment based on the outer shape of the installation object, and is a straight line segment.
Furthermore, the boundary line segment detected by the object detection unit 5 is a boundary line segment that extends at an angle with respect to the vertical direction of the captured image in at least the lower region of the captured image.

Here, "a boundary line segment that extends at an angle" means a straight line portion of a paved general sidewalk that appears to be at an angle in the captured image, a straight line portion of a crosswalk that appears to be at an angle, a straight line portion of a boundary block that appears to be at an angle, and a straight line portion of a curb that appears to be at an angle.
Moreover, the "lower region of the captured image" refers to the lower half of the captured image.

Specific examples of installed objects include crosswalks, paved general sidewalks, boundary blocks lined up in the direction of a paved general sidewalk, pedestrian traffic lights, telegraph poles, and street lights.
In addition, the boundary line of a general sidewalk and the boundary line of a boundary block are usually the same.

In addition, the center line of the captured image is displayed on the image.
Furthermore, a rectangle is displayed at the location of the installed object detected in the captured image to indicate the presence of the installed object.
In addition, a boundary line segment that is detected in the captured image and extends at an angle is also displayed at a corresponding location on the installed object that appears to be at an angle.
An image in which the image center line, rectangles, inclined boundary lines, etc. are displayed is an "augmented reality captured image."

In this specification, an image in which an image center line, rectangle, inclined boundary line, etc. are displayed based on a captured image is referred to as an "augmented reality captured image," but an augmented reality captured image and a captured image are essentially the same.
Therefore, the "image center line of the augmented reality captured image" and the "image center line of the captured image" are synonymous with each other.

As shown in FIG. 2B, an image center line 22 that passes through a captured image center point 23 that is the center point of the captured image 20 and extends in the vertical direction of the captured image is displayed together with the captured image center point 23.
As shown in FIG. 2B, crosswalk fragment rectangles 25A are displayed at each of the locations of an installed object detected in the captured image 20 by the object detection unit 5, for example, a plurality of parallel white line areas of a crosswalk.
Then, as shown in FIG. 3( a), the object detection unit 5 detects the entire area in which the multiple crosswalk fragment rectangles 25A are displayed as a single installed object, and a crosswalk existence area rectangle 25 large enough to encompass the multiple crosswalk fragment rectangles 25A is displayed in place of the crosswalk fragment rectangles 25A.

Furthermore, as shown in FIG. 2B, a pedestrian traffic light presence area rectangle 29 is displayed at the location of an installed object, such as a pedestrian traffic light, detected in the captured image 20 by the object detection unit 5.
Furthermore, as shown in FIG. 3( a ), the inclined boundary lines of the crosswalk detected by the object detection unit 5 in the captured image 20, i.e., the left crosswalk boundary line segment 26 and the right crosswalk boundary line segment 27, are displayed at the inclined portion of the crosswalk.

Furthermore, the positions of rectangles, straight line segments, etc. are indicated in the coordinate system of the captured image.
That is, in the coordinate system of the captured image, the upper left corner of the captured image is the origin of the image coordinate system, and the x coordinate value increases as you move from the origin in the horizontal direction of the image, i.e., toward the right of the image, and the y coordinate value increases as you move from the origin in the vertical direction of the image, i.e., toward the bottom of the image.

Here, the left crosswalk boundary line segment 26 is a boundary line segment whose upper end has an x-coordinate value greater than its lower end in the coordinate system of the captured image 20 .
Further, the crosswalk right boundary line segment 27 is a boundary line segment whose upper end x-coordinate value is smaller than the lower end x-coordinate value in the coordinate system of the captured image 20 .

In this way, the image in which the image center line 22, rectangles such as the crosswalk existence area rectangle 25, and inclined straight line segments such as the crosswalk left boundary line segment 26 are displayed is the augmented reality captured image 21.

The object detection unit 5 also detects the left crosswalk boundary line 26, for example, as follows:

The image of the portion enclosed by the crosswalk existence area rectangle 25 is copied and cut out, and the cut out image is binarized.

Then, pre-processing is performed to label a plurality of parallel white line regions.
That is, in the frame portion of the region of parallel white lines, "0" is set to three lines on the top and bottom sides, and three pixels on the left and right sides are also set to "0", forcibly separating the inside from the frame portion.
In addition, contraction and expansion processing is performed 0 to 2 times to remove areas consisting of only thin line segments of 1 to 2 pixels.

Next, by performing a labeling process on the binarized image that has been pre-processed with labeling, the separated parallel white line regions are labeled and a circumscribing rectangle that encloses the region can be obtained.
From the obtained circumscribing rectangles, those whose length is equal to or greater than about 1/3 to half of the width direction length of the rectangular area and whose difference in the y direction is 2 or more are selected and extracted.

The selected and extracted rectangular data is expressed as follows:
{(x1d[i], y1d[i])-(x2d[i], y2d[i])}
Here, "i=0, 1, . . . Nd-1", and "Nd" is the number of labels.
In addition, these are assumed to have been subjected to sorting processing so that they are arranged in descending order of y1d[i] (from the bottom to the top of the image).

Then, the left boundary line is determined from the left edge data of the selected and extracted rectangular data. The data is expressed as follows:
{(x1d[i], y1d[i])}
Here, i = 0, 1, ..., Nd-1.

Also, the end point with the smallest x coordinate value is found, and the coordinate value at that point is set to (x0L, y0L).
Next, the coordinate values of a point whose x coordinate value is the same as "x0L" or is within a specified range from "x0L", whose y coordinate value is smaller than "y0L", and whose next point is further away from "x0L" than the specified value are defined as (x1L, y1L).

If the point (x1L, y1L) is the “i1L”th point in the left boundary candidate point sequence, the directional angle p of the left boundary line connecting the “i1L+1, Nd (number of labels)”th point (x1d[i], y1d[i]) and the point (x1L, y1L) is calculated using the following formula.
p=angle(x1L, y1L, x1d[i], y1d[i])

Here, the direction angle is the clockwise map direction angle with north being 0 degrees. When the image center line of the captured image overlaps with the north azimuth, the azimuth angle is 0 degrees, and when it overlaps with the east azimuth, the azimuth angle is 90 degrees.

Then, it is determined whether the calculated direction angle is within the direction angle range [pLmin, pLmax] that is acceptable for points on the left boundary line segment, and for points that are within the range, the average direction angle p2L is calculated using the following formula.
p2L=Σp"j"/mL
Here, "p'j'" is an angle within the direction angle range [pLmin, pLmax] that is acceptable for a point on the left boundary line segment, and "mL" is the number of such angles.

The object detection unit 5 detects the line segment that satisfies this average directional angle p2L as the left crosswalk boundary line segment 26.

The object detection unit 5 also detects the right crosswalk boundary line 27, for example, as follows:

The right boundary line is determined from the right edge data of the selected and extracted rectangular data. The data is expressed as follows:
{(x1d[i], y1d[i])}
Here, i = 0, 1, ..., Nd-1.

In addition, the end point with the maximum x coordinate value is found, and the coordinate value at that point is set to (x0R, y0R).
Next, the coordinate values of a point whose x coordinate value is the same as "x0R" or is within a specified range from "x0R", whose y coordinate value is smaller than "y0R", and whose next point is further away from "x0R" than the specified value are defined as (x1R, y1R).

If the point (x1R, y1R) is the “i1R”th point in the sequence of right boundary candidate points, the directional angle p of the right boundary line connecting the “i1R+1, Nd (number of labels)”th point (x1d[i], y1d[i]) and the point (x1R, y1R) is calculated using the following formula.
p=angle(x1R, y1R, x1d[i], y1d[i])

Here, the direction angle is the clockwise map direction angle with north being 0 degrees. When the image center line of the captured image overlaps with the north azimuth, the azimuth angle is 0 degrees, and when it overlaps with the west azimuth, the azimuth angle is 270 degrees.

Then, it is determined whether the calculated direction angle is within the direction angle range [pRmin, pRmax] that is acceptable for points on the right boundary line segment, and for points that are within the range, the average direction angle p2R is calculated using the following formula.
p2R=Σp"j"/mR
Here, "p'j'" is an angle within the direction angle range [pRmin, pRmax] that is acceptable for a point on the right boundary line segment, and "mR" is the number of such angles.

The object detection unit 5 detects the line segment that satisfies this average directional angle p2R as the right crosswalk boundary line segment 27.

The glasses device 2 also includes a traveling direction detection unit 6 .
Here, the traveling direction detection unit 6 can detect whether or not the boundary lines, that is, the left pedestrian crossing boundary line line 26 and the right pedestrian crossing boundary line line 27 , intersect with the image center line 22 .
Furthermore, the travel direction detection unit 6 can detect whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower end of the left crosswalk boundary line segment 26 and the lower end of the right crosswalk boundary line segment 27.

The glasses device 2 also includes a traveling direction determination unit 7 .
Here, when the traveling direction detection unit 6 satisfies the following two requirements, the traveling direction determination unit 7 can determine that the orientation of the pedestrian wearing the glasses device 2 is in the "normal state".
(1) The travel direction detection unit 6 detects that the left crosswalk boundary line portion 26 and the right crosswalk boundary line portion 27 do not intersect with the image center line 22, and (2) the travel direction detection unit 6 detects that the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line portion 26 and the upper end of the right crosswalk boundary line portion 27 than to the lower end of the left crosswalk boundary line portion 26 and the lower end of the right crosswalk boundary line portion 27.

The glasses device 2 also includes a notification unit 8 .
Here, the notification unit 8 can notify the result of the determination made by the travel direction determination unit 7, for example, the determination result that the vehicle is in a "normal state."
The notification method of the notification unit 8 may be, for example, at least one of notification by voice, notification by vibration, and notification by display on an image.

That is, as shown in FIG. 3( b ), the image center line 22 does not intersect with the left crosswalk boundary line segment 26 and the right crosswalk boundary line segment 27, and further passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower end of the left crosswalk boundary line segment 26 and the lower end of the right crosswalk boundary line segment 27. When the traveling direction detection unit 6 detects such a situation, the traveling direction determination unit 7 can determine that the orientation of the pedestrian wearing the glasses device 2 is “normal.”
The notification unit 8 can notify the user of the judgment result that the travel direction judgment unit 7 has judged to be “normal” by displaying a normal judgment result symbol 24A, such as “◎”, on the augmented reality captured image 21, as shown in Figure 3 (b).

In addition, at this time, the notification unit 8 can also notify the result of the determination by the travel direction determination unit 7 that the vehicle is in a "normal state" by, for example, a voice saying, "You are moving in the correct direction."

The glasses device 2 also includes a detected object information transmitting unit 5A.
Here, the detected object information transmission unit 5A is capable of transmitting information regarding the installed object detected by the object detection unit 5, such as an augmented reality captured image 21 displaying an existence area rectangle surrounding at least a portion of the installed object, to the server 3 via the Internet 14.

On the other hand, server 3 has a detected object information receiving unit 5B.
Here, the detected object information receiving unit 5B is capable of receiving information about the installed object transmitted from the glasses device 2.
That is, the detected object information receiving unit 5B is capable of receiving an augmented reality captured image 21 that displays a presence area rectangle that surrounds at least a portion of the installed object.

Server 3 also has a detected object information database 9. Detected object information database 9 is an example of a detected object information recording section.
Here, the detected object information database 9 is capable of recording installed object information, which is information relating to various objects that can be detected by the object detection unit 5, that is, installed objects.
Specifically, the installed object information is, for example, the name of the installed object and the external characteristics (shape, size, color, etc.) of the installed object.

The server 3 also includes an object identification unit 10 .
Here, the object identification unit 10 can identify the installed object by comparing the information of the installed object detected by the object detection unit 5 and received by the detected object information receiving unit 5B with the installed object information recorded in the detected object information database 9.

Specifically, the object identification unit 10 can identify an installed object by comparing the numerical values of the rectangle of an installed object detected by the object detection unit 5, such as vertical and horizontal numerical values, with the installed object information recorded in the detected object information database 9, such as a vertical and horizontal numerical range.
Here, the vertical value of the rectangle is the distance between the two horizontal lines of the rectangle, and the horizontal value of the rectangle is the distance between the two vertical lines of the rectangle, both of which are dimensional values calculated by image analysis or the like.

In addition, when the object identification unit 10, for example, compares the numerical value of the crosswalk fragment rectangle 25A of the installed object detected by the object detection unit 5 with the numerical range recorded by the detected object information database 9 and identifies the installed object as a "white line of a crosswalk," it can identify the installed object in which the crosswalk existence area rectangle 25 is displayed as a "crosswalk" based on the fact that multiple crosswalk fragment rectangles 25A corresponding to the "white line of a crosswalk" are displayed.

The server 3 also has a specific object information transmission unit 10A that can transmit information about the installed object identified by the object identification unit 10 to the glasses device 2 via the Internet 14.

On the other hand, the glasses device 2 has a specific object information receiving unit 10B that can receive information about the installation object identified by the object identification unit 10 from the server 3.

In addition, when the object identification unit 10 identifies the installed object as a “crosswalk,” the travel direction detection unit 6 can further detect whether the crosswalk arrival entrance line segment 28 intersects with the image center line 22.
Here, the crosswalk arrival entrance line segment 28 is the upper line segment of the crosswalk existence area rectangle 25, i.e., the horizontal line segment with the smallest y coordinate value of the crosswalk existence area rectangle 25 in the coordinate system of the captured image, whose x coordinate value is greater than the x coordinate value of the intersection of this horizontal line segment with the crosswalk left boundary line segment 26, or less than the x coordinate value of the intersection of this horizontal line segment with the crosswalk right boundary line segment 27.

In addition, if the object identification unit 10 identifies an installed object other than the one identified as a "crosswalk" as a "pedestrian traffic light," the travel direction detection unit 6 can further detect whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a predetermined range.

Furthermore, the traveling direction determination unit 7 can further determine that the orientation of the pedestrian wearing the glasses device 2 is “further normal” when the traveling direction detection unit 6 satisfies the following two requirements.
(1) The travel direction detection unit 6 detects that the crosswalk arrival gate line segment 28 and the image center line 22 intersect, and (2) the travel direction detection unit 6 detects that the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a specified range.

In addition, when the travel direction detection unit 6 detects that the image center line 22 intersects with the left crosswalk boundary line segment 26, the travel direction determination unit 7 can determine that the pedestrian's direction is in a "first abnormal state."
In addition, when the travel direction detection unit 6 detects that the image center line 22 intersects with the crosswalk right boundary line segment 27, the travel direction determination unit 7 can determine that the pedestrian's direction is in a "second abnormal state."

In addition, when the traveling direction determination unit 7 determines that the first abnormal state exists, the notification unit 8 can provide information encouraging the vehicle to turn right.
In addition, when the traveling direction determination unit 7 determines that the second abnormal state exists, the notification unit 8 can provide information encouraging the driver to turn left.

The pedestrian travel direction determination system 1 of the present invention may further include a vibration notification unit 30 as shown in FIG.
That is, FIG. 4 is a schematic diagram showing an example of a vibration notification unit provided in a pedestrian traveling direction determination system to which the present invention is applied.
Here, the vibration notification unit 30 is an example of the notification unit 8, and realizes a notification method using vibration.

The vibration notification unit 30 has a main body 31 and multiple vibrating bodies 32. The main body 31 is configured in a belt shape.

That is, the main body 31 has a longitudinal direction, and one side in the longitudinal direction and the other side positioned opposite to the one side can be detachably connected to form a ring shape.
The plurality of vibrating bodies 32 are attached to one surface of the main body 31 in a line in the longitudinal direction of the main body 31 .

When the travel direction determination unit 7 determines that the first abnormal state exists, the notification unit 8 can provide information encouraging the driver to turn right. In this case, the vibration notification unit 30, which is an example of the notification unit 8, can vibrate a vibrating body 32 located on one side of the annular main body 31 (e.g., the upper side in Figure 4).
In addition, when the travel direction determination unit 7 determines that a second abnormal state has occurred, the notification unit 8 can provide information encouraging the driver to turn left. In this case, the vibration notification unit 30 can vibrate the vibrator 32 on the other side (e.g., the lower side in Figure 4) of the annular main body 31, which is located opposite the vibrator 32 on one side of the main body 31.

The glasses device 2 also includes a voice transmitting unit 11 .
Here, the voice transmitting unit 11 can transmit voice generated by a pedestrian wearing the glasses device 2 to the server 3 via the Internet 14 .

The server 3 also includes a voice receiving unit 12 .
Here, the audio receiving unit 12 is capable of receiving audio transmitted from the glasses device 2 .

The server 3 also includes a route information setting unit 13 .
Here, the route information setting unit 13 can set route information, which is information about a route between a departure point and a destination, based on the voice related to the departure point and the destination received by the voice receiving unit 12.

Furthermore, when setting route information, the route information setting unit 13 can, of course, set the route information based on map information or the like as appropriate.
In this case, the server 3 has a map information recording section (map information database) capable of recording map information, and can set route information based on the map information recorded in this map information database, or can set route information based on map information recorded in a map information database held by another server other than the server 3 while communicating with the other server via the Internet.

For example, based on a voice received by the voice receiving unit 12 saying "From XX Park to the sea side of Kokura Station," the route information setting unit 13 sets route information for a pedestrian to travel, with "XX Park" as the starting point and "the sea side of Kokura Station" as the destination.

Furthermore, the route information setting unit 13 does not necessarily have to set route information based on the voice received by the voice receiving unit 12 .
If the departure point and destination are known, the route information setting unit 13 can set route information based on, for example, characters or images.
In this case, the glasses device 2 may have a function for transmitting characters and images to the server 3, for example, an e-mail function.

The server 3 also includes a route information transmission unit 13A.
Here, the route information transmission unit 13A can transmit the route information set by the route information setting unit 13 to the glasses device 2 via the Internet 14.

On the other hand, the glasses device 2 includes a route information receiving unit 13B.
Here, the route information receiving unit 13B is capable of receiving route information transmitted from the server 3 .

The notification unit 8 can also notify the route information received by the route information receiving unit 13B.
The notification unit 8 can also notify a change of direction based on the route information set by the route information setting unit 13 .

In addition, if the traveling direction detection unit 6 detects, for example, that the right crosswalk boundary line 27 and the image center line 22 intersect within a predetermined time, for example, 8 seconds, from when the notification unit 8 notifies the user of a change of direction based on route information, the traveling direction determination unit 7 can determine that the pedestrian is changing direction.

The pedestrian travel direction determination system 1 of the present invention does not necessarily have to include a detected object information database 9, i.e., a detected object information recording unit, nor does it necessarily have to include an object identification unit 10. Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction detection unit 6 does not necessarily have to be able to detect whether the crosswalk arrival gate line segment 28 and the image center line 22 intersect.

However, if the pedestrian travel direction determination system 1 of the present invention is equipped with a detected object information database 9 and an object identification unit 10, and is capable of identifying an installed object as a crosswalk, and the travel direction detection unit 6 is capable of detecting whether the crosswalk arrival entrance line segment 28 intersects with the image center line 22, this is preferable, since it makes it easier to check whether the direction of the pedestrian relative to the crosswalk is normal.

Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction detection unit 6 does not necessarily have to be able to detect whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a specified range.
Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian's direction is also normal when the travel direction detection unit 6 detects that the pedestrian crossing arrival gate line segment 28 and the image center line 22 intersect and detects that the shortest distance between the coordinates of a pedestrian traffic light and the image center line 22 is within a predetermined range.

However, if the travel direction detection unit 6 can detect whether the crosswalk arrival gate line segment 28 and the image center line 22 intersect, and can detect whether the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is within a predetermined range, and if the travel direction detection unit 6 detects that the crosswalk arrival gate line segment 28 and the image center line 22 intersect and that the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is within a predetermined range, then it is preferable if the travel direction determination unit 7 can determine that the pedestrian's orientation is also normal, because this allows for more accurate confirmation of whether the pedestrian's orientation relative to the crosswalk is normal.

Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian's orientation is in the "first abnormal state" when the travel direction detection unit 6 detects that the image center line 22 intersects with the left pedestrian crossing boundary line segment 26, and it does not necessarily have to be able to determine that the pedestrian's orientation is in the "second abnormal state" when the travel direction detection unit 6 detects that the image center line 22 intersects with the right pedestrian crossing boundary line segment 27.
In the pedestrian travel direction determination system 1 of the present invention, the notification unit 8 does not necessarily have to be able to notify information encouraging a turn to the right when the travel direction determination unit 7 determines that the first abnormal state is occurring, and does not necessarily have to be able to notify information encouraging a turn to the left when the travel direction determination unit 7 determines that the second abnormal state is occurring.

However, if the travel direction determination unit 7 is capable of determining that the first abnormal state and the second abnormal state are present, and if the notification unit 8 is capable of issuing information encouraging the pedestrian to turn right when the travel direction determination unit 7 determines that the first abnormal state is present, and if the notification unit 8 is capable of issuing information encouraging the pedestrian to turn left when the travel direction determination unit 7 determines that the second abnormal state is present, this is preferable because it makes it easier to correct the pedestrian's direction than if the travel direction determination unit 7 is not configured in this way.

The pedestrian travel direction determination system 1 of the present invention does not necessarily have to include a route information setting unit 13, and the notification unit 8 does not necessarily have to be able to notify the pedestrian of a change of direction based on the route information set by the route information setting unit 13. Furthermore, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian is changing direction when the travel direction detection unit 6 detects that the left pedestrian crossing boundary line 26 or the right pedestrian crossing boundary line 27 has intersected with the image center line 22 within a predetermined time from when the notification unit 8 notifies the pedestrian of a change of direction based on the route information.

However, when the pedestrian travel direction determination system 1 of the present invention includes a route information setting unit 13, and the travel direction detection unit 6 detects that the left pedestrian crossing boundary line 26 or the right pedestrian crossing boundary line 27 has crossed the image center line 22 within a predetermined time from when the notification unit 8 notifies the user of a change of direction based on the route information, if the travel direction determination unit 7 can determine that the pedestrian is changing direction, the travel direction determination unit 7 determines that the pedestrian is changing direction until the predetermined time has elapsed, rather than determining whether the pedestrian's direction is normal or not, so that the pedestrian can concentrate on changing direction, which is preferable.

The pedestrian movement direction determination software of the present invention for executing such a pedestrian movement direction determination system 1 is software for causing an information processing device to function as a glasses device 2 having the functions described above.

Moreover, the example shown in FIG. 1 is just an example, and the functions of the glasses device 2 and the server 3 are not limited to this.
For example, the server 3 may have an object detection unit 5, a traveling direction detection unit 6, and a traveling direction determination unit 7. In this case, the pedestrian traveling direction determination software of the present invention for executing the pedestrian traveling direction determination system 1 can cause the information processing device to function as a server having the same functions as the glasses device 2.

In addition, if the server 3 has an object detection unit 5, a traveling direction detection unit 6, and a traveling direction determination unit 7, the object detection unit 5, the traveling direction detection unit 6, and the traveling direction determination unit 7 perform processing based on the captured image received by the captured image receiving unit 4B of the server 3.

Next, we will explain an example of the process for determining the direction of travel of a pedestrian using the pedestrian travel direction determination system of the present invention.

When a pedestrian starts walking on the sidewalk, the pedestrian transmits to the server 3 information required to set the route along which the pedestrian plans to walk.
That is, when a pedestrian starts walking on the sidewalk, he or she speaks out loud information about the starting point, which is the location where the pedestrian is currently located, and information about the destination, and the voice transmission unit 11 of the glasses device 2 transmits the voice related to the starting point and destination to the server 3 via the Internet 14.

For example, if a pedestrian wants to walk to Kokura Station to go home after playing at "XX Park," the pedestrian can vocalize route information such as "From XX Park to the sea side of Kokura Station" to set the route he or she will take.

Then, the voice transmitting unit 11 of the glasses device 2 transmits the voice "From XX Park to the sea side of Kokura Station" to the server 3 via the Internet 14.
On the other hand, the voice receiving unit 12 of the server 3 receives the voice “From XX Park to the sea side of Kokura Station” transmitted from the glasses device 2 .

Then, based on the voice received by the voice receiving unit 12 saying "From XX Park to the sea side of Kokura Station," the route information setting unit 13 of the server 3 sets route information for the pedestrian to travel, with "XX Park" as the starting point and "the sea side of Kokura Station" as the destination.

Furthermore, the route information transmitting unit 13A of the server 3 transmits the route information set by the route information setting unit 13, "from XX Park to the sea side of Kokura Station", to the glasses device 2 via the Internet 14.
On the other hand, the route information receiving unit 13B of the glasses device 2 receives the route information transmitted from the server 3 .

Then, the notification unit 8 of the glasses device 2 notifies the pedestrian of the route information received by the route information receiving unit 13B, "From XX Park to the sea side of Kokura Station."

Also, for example, a pedestrian leaves XX Park and stands at the intersection in front of XX Park, relying on the tactile paving blocks.
At this time, the imaging camera 4 of the glasses device 2 captures an image ahead of the pedestrian, and a first example of an image captured by the imaging camera 4 is shown in FIG. 2( a ).
In addition, an example of an augmented reality captured image 21 when the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 20 shown in Figure 2 (a) is shown in Figure 2 (b), Figure 3 (a), and Figure 3 (b).

The route information you set also includes crossing the crosswalk at the intersection right in front of XX Park.

In addition, the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 20 .
Furthermore, the object detection unit 5 of the glasses device 2 detects the obliquely extending boundary line of the placed object.
When the object detection unit 5 detects an installed object or a boundary line segment, a rectangle or a boundary line segment is displayed at the location of the detected installed object or at a corresponding location of the installed object that appears tilted.

Furthermore, as shown in Figures 2(b), 3(a) and 3(b), the augmented reality captured image 21 displays an image center line 22 that passes through the captured image center point 23, which is the center point of the augmented reality captured image 21, and extends vertically in the captured image.
Here, since the augmented reality captured image 21 is a display of lines and rectangles on the screen based on the captured image 20 , the center point of the augmented reality captured image 21 and the center point of the captured image 20 are the same.

Furthermore, as shown in FIG. 2B, a crosswalk fragment rectangle 25A is displayed in each of a plurality of parallel white line areas of the crosswalk, which are installation objects detected by the object detection unit 5.
Then, the object detection unit 5 detects the entire area in which the multiple crosswalk fragment rectangles 25A are displayed as a single installed object, and as shown in Figure 3 (a), a crosswalk existence area rectangle 25 large enough to encompass the multiple crosswalk fragment rectangles 25A is displayed in place of the crosswalk fragment rectangles 25A.

Furthermore, as shown in FIG. 2B , a pedestrian traffic light presence area rectangle 29 is displayed on a pedestrian traffic light, which is an installed object detected by the object detection unit 5 .
Furthermore, as shown in FIG. 3A, the obliquely extending boundary segments of the crosswalk detected by the object detection unit 5, that is, a left crosswalk boundary segment 26 and a right crosswalk boundary segment 27, are displayed at an inclined portion of the crosswalk.

Then, the traveling direction detection unit 6 detects whether the left pedestrian crossing boundary line segment 26 and the right pedestrian crossing boundary line segment 27 intersect with the image center line 22 .
The travel direction detection unit 6 also detects whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower end of the left crosswalk boundary line segment 26 and the lower end of the right crosswalk boundary line segment 27.

As shown in Figures 2(b), 3(a), and 3(b), the image center line 22 does not intersect with either the left crosswalk boundary line segment 26 or the right crosswalk boundary line segment 27, so the travel direction detection unit 6 detects that the image center line 22 does not intersect with the left crosswalk boundary line segment 26 or the right crosswalk boundary line segment 27.
Furthermore, as shown in Figures 2(b), 3(a), and 3(b), the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower end of the left crosswalk boundary line segment 26 and the lower end of the right crosswalk boundary line segment 27, so the travel direction detection unit 6 detects that the image center line 22 passes through that position.

Then, the traveling direction determining unit 7 determines that the direction of the pedestrian wearing the glasses device 2 is "normal."
Furthermore, the notification unit 8 notifies the pedestrian of the determination result "normal state" determined by the travel direction determination unit 7 by displaying a determination result symbol on the augmented reality captured image 21, or by a voice, for example, saying "You are moving in the correct direction."

In this way, the pedestrian travel direction determination system 1 of the present invention can determine the direction of a pedestrian without identifying an installed object in the captured image 20, as long as it can detect the inclined boundary line of the installed object.

In addition to this process of determining the direction of travel, the detected object information transmission unit 5A also transmits an augmented reality captured image 21, which displays a presence area rectangle that surrounds at least a portion of the installed object, to the server 3 via the Internet 14.

Meanwhile, in the server 3 that receives such an augmented reality captured image 21, the object identification unit 10 compares the information of the installed object detected by the object detection unit 5 with the installed object information recorded in the detected object information database 9 to identify the installed object.

In addition, the object identification unit 10 identifies the installed object in which the crosswalk existence area rectangle 25 is displayed as a “crosswalk,” and the specified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2.
Therefore, when the notification unit 8 notifies by voice the result of the determination made by the traveling direction determination unit 7 that the vehicle is in a "normal state," the notification unit 8 also notifies the content of the specified installed object, for example, "You are proceeding in the correct direction on the crosswalk."

In addition, since the object identification unit 10 identifies the installed object as a "crosswalk," the travel direction detection unit 6 also detects whether the crosswalk arrival entrance line segment 28 intersects with the image center line 22.

In addition, the object identification unit 10 identifies an installed object in which the pedestrian traffic light existence area rectangle 29 is displayed, i.e., an installed object other than the installed object that the object identification unit 10 identified as a "crosswalk," as a "pedestrian traffic light," and the identified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2.
In addition, the direction of travel detection unit 6 further detects whether the shortest distance between the coordinates of the pedestrian traffic light, i.e., the coordinates of the detected installed object identified as a "pedestrian traffic light", in the coordinate system of the captured image, and the image center line 22 is within a predetermined range, i.e., greater than "0" and less than or equal to the shortest distance between a virtual center line passing through the center of the crosswalk and the coordinates of the pedestrian traffic light.

As shown in Figures 2(b), 3(a) and 3(b), the image center line 22 intersects with the crosswalk arrival gate line segment 28, and since the image center line 22 passes near the center of the crosswalk, the travel direction detection unit 6 detects that the crosswalk arrival gate line segment 28 intersects with the image center line 22, and that the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a specified range.

Therefore, in this case, the traveling direction determining unit 7 determines that the direction of the pedestrian wearing the glasses device 2 is "still in a normal state."
The notification unit 8 then notifies the pedestrian of the determination result, "still in a normal state," made by the travel direction determination unit 7 by displaying a normal determination result symbol 24 such as "◎" on the augmented reality captured image 21, as shown in Figure 3 (b), or by a voice message, for example, "You are moving sufficiently in the correct direction."

Furthermore, for example, if the image center line 22 shown in Figures 2(b), 3(a), and 3(b) is located a little further to the right and intersects with the pedestrian traffic light existence area rectangle 29, the image center line 22 intersects with the crosswalk arrival entrance line segment 28, but is not within the specified range because the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is "0".
Therefore, in this case, the travel direction determination unit 7 does not determine that the state is "still normal," so the notification unit 8 notifies the user by displaying a determination result symbol such as "O" on the augmented reality captured image 21, or by providing a voice message saying, "You are moving in the correct direction to a certain extent."

On the other hand, if route information is set for a pedestrian starting from "XX Park" and going to "the sea side of Kokura Station", when the pedestrian leaves XX Park and stands at the intersection in front of XX Park, relying on the tactile paving blocks, it is also possible that the pedestrian will stand facing diagonally to the crosswalk.

FIG. 5A is a schematic diagram showing a second example of an image captured in the pedestrian traveling direction determination system to which the present invention is applied.
FIG. 5(b) is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian's direction being determined by a direction determination unit of a pedestrian direction determination system to which the present invention is applied, based on the augmented reality captured image of the captured image shown in FIG. 5(a).

FIG. 5A shows a second example of an image captured by the imaging camera 4 of the glasses device 2 when a pedestrian stands facing diagonally toward the crosswalk.
FIG. 5B shows an example of an augmented reality captured image 41 when the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 40 shown in FIG. 5A.

The route information you set also includes crossing the crosswalk at the intersection right in front of XX Park.

Furthermore, the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 40 .
Furthermore, the object detection unit 5 of the glasses device 2 detects the obliquely extending boundary line of the placed object.
When the object detection unit 5 detects an installed object or a boundary line segment, a rectangle or a boundary line segment is displayed at the location of the detected installed object or at a corresponding location of the installed object that appears tilted.

As shown in FIG. 5B, the augmented reality captured image 41 displays an image center line 42 that passes through a captured image center point 43, which is the center point of the augmented reality captured image 41, and extends in the vertical direction of the captured image.
Here, the center point of the augmented reality captured image 41 and the center point of the captured image 40 are the same.

In addition, the object detection unit 5 detects the entire area in which multiple crosswalk fragment rectangles (not shown) are displayed as a single installed object, and a crosswalk presence area rectangle 45 large enough to encompass the multiple crosswalk fragment rectangles (not shown) is displayed in place of the crosswalk fragment rectangles (not shown), as shown in FIG. 5(b).

Furthermore, as shown in FIG. 5B , a pedestrian traffic light presence area rectangle 49 is displayed on a pedestrian traffic light, which is an installed object detected by the object detection unit 5 .
Also, as shown in FIG. 5B, a right crosswalk boundary line segment 47, which is an inclined boundary line segment of the crosswalk detected by the object detection unit 5, is displayed at an inclined portion of the crosswalk.
Here, the object detection unit 5 has not detected the left boundary line of the pedestrian crossing.

Then, the traveling direction detection unit 6 detects whether or not the crosswalk right boundary line segment 47 and the image center line 42 intersect.
The traveling direction detection unit 6 also detects whether the image center line 42 passes through a position closer to the upper end of the right pedestrian crossing boundary line segment 47 than to the lower end of the right pedestrian crossing boundary line segment 47 .

As shown in FIG. 5(b), the image center line 42 intersects with the right crosswalk boundary line segment 47, so the travel direction detection unit 6 detects that the right crosswalk boundary line segment 47 intersects with the image center line 42.

Also, as shown in FIG. 5( b ), the image center line 42 passes through a position closer to the bottom end of the crosswalk right boundary line segment 47 than to the top end of the crosswalk right boundary line segment 47; in other words, the image center line 42 does not pass through a position closer to the top end of the crosswalk right boundary line segment 47 than to the bottom end of the crosswalk right boundary line segment 47.
Therefore, the traveling direction detection unit 6 detects that the image center line 42 does not pass through a position closer to the upper end of the right crosswalk boundary line segment 47 than to the lower end of the right crosswalk boundary line segment 47 .

In this case, the travel direction determination unit 7 determines that the orientation of the pedestrian wearing the eyeglass device 2 is "not in a normal state," but since the travel direction detection unit 6 detects that the image center line 42 intersects with the right crosswalk boundary line segment 47, the travel direction determination unit 7 determines that the orientation of the pedestrian is "in a second abnormal state."

In addition, the notification unit 8 notifies the pedestrian of the determination result that "a second abnormal state has occurred" determined by the travel direction determination unit 7 by displaying an abnormality determination result symbol 44 such as "X" on the augmented reality captured image 41, as shown in Figure 5 (b).
Furthermore, if the travel direction determination unit 7 determines that the pedestrian's direction is in the "second abnormal state," the notification unit 8 further notifies the pedestrian by providing information encouraging the pedestrian to turn left, for example, by a voice saying, "You are moving to the right. Please turn a little further to the left."
Furthermore, in this case, a vibration notification unit 30 having a ring-shaped main body 31 worn like a belt around the pedestrian's waist vibrates a vibrator 32 located on one side of the ring-shaped main body 31, i.e., on the pedestrian's left flank, to notify the pedestrian of information encouraging them to turn left.

On the other hand, if the travel direction detection unit 6 detects that the image center line intersects with the left boundary line of the crosswalk, the travel direction determination unit 7 determines that the pedestrian's direction is in the "first abnormal state."

In this case, the notification unit 8 notifies the driver by issuing information encouraging the driver to turn right, for example, by a voice saying, "You are heading left. Please turn a little further to the right."
Furthermore, in this case, a vibration notification unit 30 having a ring-shaped main body 31 worn like a belt around the pedestrian's waist vibrates a vibrator 32 located on the opposite side of the ring-shaped main body 31, i.e., on the pedestrian's right side, to notify information encouraging the pedestrian to turn right.

In addition to this process of determining the direction of travel, the detected object information transmission unit 5A also transmits an augmented reality captured image 41, which displays a presence area rectangle that surrounds at least a portion of the installed object, to the server 3 via the Internet 14.

In addition, the object identification unit 10 identifies the installed object in which the crosswalk existence area rectangle 45 is displayed as a “crosswalk,” and the specified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2.
Therefore, when the notification unit 8 notifies by voice the result of the determination made by the travel direction determination unit 7 that "this is a second abnormal state", the notification unit 8 also notifies the content of the specified installed object, for example, "You are proceeding to the right on the crosswalk."

In addition, since the object identification unit 10 identifies the installed object as a "crosswalk," the travel direction detection unit 6 also detects whether the crosswalk arrival entrance line segment 48 and the image center line 42 intersect.

In addition, the object identification unit 10 identifies an installed object in which the pedestrian traffic light existence area rectangle 49 is displayed, that is, an installed object other than the installed object that the object identification unit 10 identified as a "crosswalk," as a "pedestrian traffic light," and the identified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2.
Furthermore, the travel direction detection unit 6 further detects whether or not the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 42 is within a predetermined range.

As shown in FIG. 5( b ), the image center line 42 does not intersect with the crosswalk arrival gate line segment 48, and in addition, the image center line 42 intersects with the crosswalk right boundary line segment 47, and passes through a position closer to the lower end of the crosswalk right boundary line segment 47 than to the upper end of the crosswalk right boundary line segment 47.
Therefore, the travel direction detection unit 6 detects that the crosswalk arrival entrance line segment 48 and the image center line 42 do not intersect, and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line 42 is not within a specified range.

On the other hand, a variant is also envisaged in which a pedestrian vocalizes route information such as "from X△ Park to △△ Building" to set the route they will take, and the route information setting unit 13 of the server 3 sets route information with "X△ Park" as the starting point and "△△ Building" as the destination.
The route information that has been set is to proceed to the right without crossing the crosswalk at the intersection in front of XX Park.

In this case, immediately after the pedestrian leaves the park, the pedestrian faces in approximately the same direction as the crosswalk extends, as shown in FIG. 2(a).
The notification unit 8 then notifies pedestrians of a rightward turn based on the route information set by the route information setting unit 13 .

When a pedestrian receives a notice of a rightward turn, he or she begins to move forward by turning their body to the right in order to proceed to the right.
At this time, the captured image of the area in front of the pedestrian changes from that shown in FIG. 2(a) to that shown in FIG. 5(a).

Furthermore, the object detection unit 5 of the glasses device 2 detects the left boundary line of the crosswalk and the right boundary line of the crosswalk.
Further, the traveling direction detection unit 6 detects whether or not the left boundary line of the crosswalk and the right boundary line of the crosswalk intersect with the image center line.

Therefore, when a pedestrian turns his/her body to the right and starts moving forward, the image center line will intersect with the right boundary line of the crosswalk, and the direction of travel detection unit 6 will detect that the right boundary line of the crosswalk and the image center line intersect.

Then, if the traveling direction detection unit 6 detects that a boundary line, such as the right boundary line of a pedestrian crossing, intersects with the image center line within a predetermined time, such as within 8 seconds, from when the notification unit 8 notifies the pedestrian of a rightward turn based on route information, the traveling direction determination unit 7 determines that the pedestrian is changing direction.

For example, from the time when a pedestrian is notified to turn right until 8 seconds have elapsed, even if the image center line intersects with a boundary line such as the right boundary line of the pedestrian crossing, the traveling direction determination unit determines that the pedestrian is changing direction rather than determining whether the pedestrian is facing normally, so that the pedestrian can concentrate on changing direction.

After turning right just before the crosswalk, pedestrians follow the route information and proceed along the general sidewalk alongside the roadway.
Then, the object detection unit 5 of the glasses device 2 detects the boundary line segments of the installed object that extend at an angle, i.e., the straight line segments of the curb that appear to be inclined between the roadway and the general sidewalk, and the straight line segments of the boundary blocks that appear to be inclined between the sidewalk and the site, as the left boundary line segment of the general sidewalk and the right boundary line segment of the general sidewalk.

Furthermore, the traveling direction detection unit 6 detects whether or not the left boundary line of the general sidewalk and the right boundary line of the general sidewalk intersect with the image center line.
In addition, the travel direction detection unit 6 detects whether the image center line passes through a position closer to the upper end of the general sidewalk left boundary line portion and the upper end of the general sidewalk right boundary line portion than the lower end of the general sidewalk left boundary line portion and the lower end of the general sidewalk right boundary line portion.

Then, when the travel direction detection unit 6 detects that the general sidewalk left boundary line segment and the general sidewalk right boundary line segment do not intersect with the image center line, and further detects that the image center line passes through a position closer to the upper end of the general sidewalk left boundary line segment and the upper end of the general sidewalk right boundary line segment than to the lower end of the general sidewalk left boundary line segment and the lower end of the general sidewalk right boundary line segment, the travel direction determination unit 7 determines that the pedestrian's direction is "normal."

As described above, the pedestrian travel direction determination system 1 of the present invention is equipped with a travel direction detection unit 6, which can detect whether the boundary segments, i.e., the left crosswalk boundary segment 26 and the right crosswalk boundary segment 27, intersect with the image center line 22, and further detect whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary segment 26 and the upper end of the right crosswalk boundary segment 27 than to the lower end of the left crosswalk boundary segment 26 and the lower end of the right crosswalk boundary segment 27. Therefore, the orientation of a pedestrian relative to an installed object having an inclined boundary segment in the captured image can be detected.

The pedestrian travel direction determination system 1 of the present invention also includes a travel direction determination unit 7, which is capable of determining that the pedestrian's orientation is "normal" when the travel direction detection unit 6 detects that the left pedestrian crossing boundary line segment 26 and the right pedestrian crossing boundary line segment 27 do not intersect with the image center line 22 and the travel direction detection unit 6 detects that the image center line 22 passes through a position closer to the upper end of the left pedestrian crossing boundary line segment 26 and the upper end of the right pedestrian crossing boundary line segment 27 than to the lower end of the left pedestrian crossing boundary line segment 26 and the lower end of the right pedestrian crossing boundary line segment 27. Therefore, it is possible to confirm whether the pedestrian's orientation with respect to an installed object having an inclined boundary line segment is normal.

Therefore, the pedestrian movement direction determination system 1 of the present invention can determine a safe and secure direction for pedestrians when moving forward.

The pedestrian movement direction determination software of the present invention can also cause an information processing device to function as a glasses device 2 having the same functions as the movement direction detection unit 6 and the movement direction determination unit 7 of the pedestrian movement direction determination system 1 of the present invention.

Therefore, the pedestrian movement direction determination software of the present invention can determine the direction in which a pedestrian is safe and secure when moving forward.

REFERENCE SIGNS LIST 1 Pedestrian travel direction determination system 2 Glasses device 3 Server 4 Imaging camera 4A Imaging image transmission unit 4B Imaging image reception unit 5 Object detection unit 5A Detected object information transmission unit 5B Detected object information reception unit 6 Travel direction detection unit 7 Travel direction determination unit 8 Notification unit 9 Detected object information database 10 Object identification unit 10A Specific object information transmission unit 10B Specific object information reception unit 11 Voice transmission unit 12 Voice reception unit 13 Route information setting unit 13A Route information transmission unit 13A Route information reception unit 14 Internet 20 Captured image 21 Augmented reality captured image 22 Image center line 23 Captured image center point 24 Normal determination result symbol 25 Crosswalk existence area rectangle 25A Crosswalk fragment rectangle 26 Crosswalk left boundary line segment 27 Crosswalk right boundary line segment 28 Crosswalk arrival entrance line segment 29 Pedestrian traffic light presence area rectangle 30 Vibration notification unit 31 Main body 32 Vibrator 40 Captured image 41 Augmented reality captured image 42 Image center line 43 Captured image center point 44 Abnormality determination result symbol 45 Pedestrian crossing presence area rectangle 47 Pedestrian crossing right boundary line segment 48 Pedestrian crossing arrival gate line segment 49 Pedestrian traffic light presence area rectangle

Claims (8)

An imaging unit capable of imaging at least an area in front of a pedestrian;
an object detection unit capable of detecting a boundary line of an installed object, which is an object installed in an image captured by the imaging unit and has an image center line passing through a center point of the captured image and extending in a vertical direction of the captured image, the boundary line extending at an angle with respect to the vertical direction of the captured image in at least a lower region of the captured image;
a travel direction detection unit capable of detecting whether the boundary line segment intersects with the image center line and detecting whether the image center line passes through a position closer to an upper end of the boundary line segment than to a lower end of the boundary line segment;
and a traveling direction determination unit that is capable of determining that the pedestrian is facing normally when the traveling direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to an upper end of the boundary line than to a lower end of the boundary line. 2. The pedestrian travel direction determination system according to claim 1, wherein the boundary line segment is a left boundary line segment whose upper end has an x-coordinate value that is greater than the x-coordinate value of the lower end in a coordinate system of the captured image, and a right boundary line segment whose upper end has an x-coordinate value that is smaller than the x-coordinate value of the lower end in the coordinate system of the captured image. a detection object information recording unit capable of recording installation object information which is information regarding the installation object;
an object identification unit capable of identifying an installed object by comparing information on the installed object detected by the object detection unit with the installed object information recorded by the detected object information recording unit;
3. The pedestrian travel direction determination system according to claim 2, wherein, when the object identification unit has identified the installed object as a crosswalk, the travel direction detection unit is further capable of detecting whether the image center line intersects with a crosswalk arrival gate line segment that is a portion of a horizontal line segment having a y coordinate value that is the smallest vertical coordinate value in the coordinate system of the captured image and that increases as the image progresses downward, of a crosswalk existence area rectangle corresponding to the entire crosswalk, which is displayed in the captured image when the object detection unit detects the installed object, and which is greater than the x coordinate value of a point of intersection with the left boundary line segment or smaller than the x coordinate value of a point of intersection with the right boundary line segment. the travel direction detection unit is further capable of detecting, when an installed object other than an installed object identified by the object identification unit as a pedestrian crossing is identified as a pedestrian traffic light by the object identification unit, whether or not a shortest distance between the coordinates of the pedestrian traffic light in a coordinate system of the captured image and the image center line is within a predetermined range;
4. The pedestrian travel direction determination system according to claim 3, wherein the travel direction determination unit is further capable of determining that the pedestrian's direction is normal when the travel direction detection unit detects that the pedestrian crossing arrival gate line segment and the image center line intersect and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line is within a predetermined range. A notification unit capable of notifying the result determined by the traveling direction determination unit,
the traveling direction determination unit is capable of determining that the orientation of the pedestrian is in a first abnormal state when the traveling direction detection unit detects that the image center line intersects with the left boundary line segment, and is capable of determining that the orientation of the pedestrian is in a second abnormal state when the traveling direction detection unit detects that the image center line intersects with the right boundary line segment,
3. The pedestrian travel direction determination system according to claim 2, wherein the notification unit is capable of providing information encouraging the pedestrian to turn right when the travel direction determination unit determines that a first abnormal state exists, and is capable of providing information encouraging the pedestrian to turn left when the travel direction determination unit determines that a second abnormal state exists. 6. The pedestrian travel direction determination system according to claim 5, wherein the notification unit is a vibration notification unit having a main body and a plurality of vibrators, the main body having a longitudinal direction and capable of detachably connecting one side of the longitudinal direction to the other side located opposite the one side to form a ring shape, the vibrators being attached in a line in the longitudinal direction of the main body to one face of the main body, the vibration notification unit being capable of vibrating the vibrator located on one side of the ring-shaped main body when notifying information urging a right turn, and being capable of vibrating the vibrator on the other side located opposite the vibrator on one side when notifying information urging a left turn. a notification unit capable of notifying a result determined by the traveling direction determination unit;
A route information setting unit capable of setting route information, which is information relating to a route between the departure point and the destination, based on information relating to the departure point and the destination;
The notification unit is capable of notifying a change of direction based on the route information set by the route information setting unit,
2. The pedestrian travel direction determination system according to claim 1, wherein the travel direction determination unit is capable of determining that the pedestrian is in a direction change state when the travel direction detection unit detects that the boundary line segment and the image center line intersect within a predetermined time from when the notification unit notifies the user of a direction change based on the route information. A pedestrian movement direction determination software for determining a direction of a pedestrian when moving, comprising:
Information processing equipment,
an object detection unit capable of detecting a boundary line of an installed object, which is an object that is installed in an image obtained by an imaging unit capable of imaging at least an area in front of a pedestrian, the boundary line passing through a center point of the image and extending in a vertical direction of the image, the boundary line extending at an angle with respect to the vertical direction of the image in at least a lower region of the image;
a travel direction detection unit capable of detecting whether the boundary line segment intersects with the image center line and detecting whether the image center line passes through a position closer to an upper end of the boundary line segment than to a lower end of the boundary line segment;
a traveling direction determination unit capable of determining that the pedestrian is facing normally when the traveling direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to an upper end of the boundary line than to a lower end of the boundary line;
Pedestrian direction determination software to function as a means including

Images