JP7580938B2 - CONTROL DEVICE, WALKING ASSISTANCE ... SYSTEM, CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a control device, a walking assistance device, a walking assistance system, a control method, and a program that assist the walking of visually impaired people.

Typically, visually impaired people walk along the road by swinging a white cane from side to side in front of them while detecting obstacles. In order to stay on the road, visually impaired people also tap their white cane at regular intervals against steps or block walls that separate the sidewalk from the road. However, in real roads, there are many sections where steps are interrupted, such as at corners and at the entrance to parking lots, making it difficult to stay on the road.

As a technology to assist visually impaired people in walking along the road, Patent Document 1 discloses a system in which a sensor that detects the magnetism of magnetic tape installed along the path is attached to the bottom end of a white cane, and vibrations or other messages are transmitted to the user depending on the detection result.

Patent document 2 also discloses a system in which an illumination unit that illuminates a guidance line with white light and an identification unit that identifies light reflected from the guidance line are provided at the tip of a white cane, and vibrations and sounds are transmitted to the user according to the identified color.

JP 2003-131556 A JP 2005-342218 A

However, the systems in Patent Documents 1 and 2 require detection of magnetic tape or guide lines, and therefore can only operate properly in passageways where magnetic tape or guide lines have been installed in advance.

In addition, in the systems of Patent Documents 1 and 2, the sensor is installed at the tip of the white cane, so there is a risk that the sensor may be damaged by obstacles.

The present invention aims to provide a control device, walking assistance device, walking assistance system, control method, and program that can assist walking even on passageways that do not have special markers.

The control device as one aspect of the present invention is characterized by having a setting means for setting walking guide lines indicating walkable areas in the image captured by the imaging means using information regarding the end of steps on the road in the image, an acquisition means for acquiring position information regarding the position of a specific object in the image, and a control means for instructing the notification means to perform a notification operation when the position information is located on the walking guide line .

Furthermore, a control method as another aspect of the present invention is characterized by having a step of setting a walking guide line indicating a walkable area in the image captured by the imaging means using information regarding the end of a step on a road in the image, a step of acquiring position information regarding the position of a specific object in the image, and a step of instructing the notification means to perform a notification operation when the position information is located on the walking guide line .

The present invention provides a control device, walking assistance device, walking assistance system, control method, and program that can assist walking even on passageways that do not have special markers.

1 is a block diagram showing an outline of the configuration of a walking assistance system according to an embodiment of the present invention. 4 is a flowchart showing a process of the imaging apparatus according to the first embodiment. FIG. 13 is a diagram showing an example of an image. FIG. 13 is a diagram showing a discontinuous section of a step. FIG. 2 is a diagram showing an example of walking guide lines. FIG. FIG. 13 is a diagram showing the position history of a white cane. 4 is a flowchart showing a process of the vibration device according to the first embodiment. 10 is a flowchart showing a process of the imaging apparatus according to the second embodiment. FIG. 13 is a diagram showing an example of walking guide lines for turning corners. 13 is a flowchart showing a process of the imaging apparatus according to the third embodiment. FIG. 13 is a diagram showing an example of walking guide lines for avoiding obstacles.

The following describes in detail an embodiment of the present invention with reference to the drawings. In each drawing, the same reference numbers are used for the same components, and duplicate descriptions are omitted.

FIG. 1(a) is a block diagram showing an outline of the configuration of a walking assistance system according to an embodiment of the present invention. The walking assistance system includes an imaging device (walking assistance device) 100 and a vibration device 120.

The imaging device 100 has a control unit 101, a memory 102, a non-volatile memory 103, an imaging unit (imaging means) 104, an image processing unit 105, a communication I/F 106, and a GPS (Global Positioning System) 107. Each unit is connected to an internal bus 110 and is configured to be able to exchange data with each other via the internal bus 110.

Memory 102 is composed of a volatile memory that uses semiconductor elements such as RAM (Random Access Memory).

The non-volatile memory 103 is composed of, for example, a hard disk (HD) or a ROM (Read Only Memory), and stores image data, audio data, other data, and various programs for the operation of the control unit 101.

The control unit 101 controls each unit of the imaging device 100 using the memory 102 as a work memory according to a program stored in, for example, the non-volatile memory 103. As shown in FIG. 1(b), the control unit 101 has a setting unit 101a, an acquisition unit 101b, and a control unit 101c. Note that, in this embodiment, the control unit 101 is mounted inside the imaging device 100, but may be configured as a control device separate from the imaging device 100.

The imaging unit 104 has a lens and an optical sensor that converts light from the lens into an electrical signal, captures an image under the control of the control unit 101, and stores the image in the memory 102. As the optical sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a CCD (Charge-Coupled Device) image sensor, or the like can be used.

Based on the control of the control unit 101, the image processing unit 105 performs various image processing such as resizing, trimming, edge detection, motion vector calculation, and matrix calculation on the image data stored in the memory 102. The image processing unit 105 may be configured with a dedicated circuit block for performing specific image processing. Furthermore, depending on the type of image processing, the control unit 101 may perform image processing according to a program.

The communication I/F 106 is an interface for communicating with external devices, the Internet, etc., and sending and receiving various data such as files and commands. For example, Bluetooth or Wi-Fi can be used as the communication I/F 106.

The GPS 107 receives and analyzes GPS signals transmitted from satellites to estimate the current position (latitude and longitude) of the imaging device 100. The current position may be estimated using WPS (WiFi Positioning System) or the like.

The vibration device 120 has a control unit 121, a memory 122, a non-volatile memory 123, a communication I/F 124, and a vibration unit 125. Each unit is connected to an internal bus 130 and is configured to be able to exchange data with each other via the internal bus 130.

Memory 122 consists of, for example, RAM.

The non-volatile memory 123 is composed of, for example, a hard disk (HD) or ROM, and stores various data and various programs for the control unit 121 to operate.

The control unit 121 controls each part of the vibration device 120 using the memory 122 as a work memory, for example, according to a program stored in the non-volatile memory 123.

The communication I/F 124 is an interface for communicating with external devices, the Internet, etc., and sending and receiving various data such as files and commands. For example, Bluetooth or Wi-Fi can be used as the communication I/F 124.

The vibration unit 125 is a vibration device such as a haptic device, and vibrates based on the control from the control unit 121. The user can use the vibration transmitted from the vibration unit 125 to know that they are moving in the correct direction. Note that in this embodiment, the vibration of the vibration unit 125 is used to perform a notification operation for the user, but another notification means may be used as long as it is possible to perform a notification operation for the user.

The operation of each embodiment of the present invention will be described below.

In this embodiment, we will explain a configuration that guides walking by notifying the user when a white cane, which is a walking aid, crosses a virtual line (walking guide line) that serves as a walking guide and is set in a section where the sidewalk is interrupted by steps or other obstacles.

First, the processing of the imaging device 100 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the processing of the imaging device 100 of this embodiment. Each process in the flowchart of FIG. 2 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it.

In step S201, the control unit 101 acquires an image captured by the imaging unit 104. FIG. 3 is a diagram showing an example of an image acquired in step S201. The image in FIG. 3 shows a road 301, steps 302 and 303, and a white cane 304.

In step S202, the control unit 101 (setting means 101a) acquires information about the steps on the road that serve as walking landmarks by analyzing the image acquired in step S201. In this embodiment, the control unit 101 acquires information about the steps 302 and 303. Methods for analyzing an image to acquire information about objects such as sidewalk steps include a method using machine learning such as Deep Learning and a method using SLAM (Simultaneous Localization and Mapping). In this embodiment, information about the steps on the road is acquired, but is not limited to this as long as it is predetermined information that can set a predetermined area to assist walking as described later. In this embodiment, information about the steps is acquired by the control unit 101 analyzing the image, but may be acquired using position information based on a signal from an artificial satellite. In addition, the image may be analyzed using the image processing unit 105.

In step S203, the control unit 101 (setting means 101a) uses the information about the step acquired in step S202 to determine whether or not the step has a discontinuous section. An example of a discontinuous section is section 401 between steps 302 and 303, which is indicated by the dashed line in FIG. 4. If a discontinuous section is present, the process proceeds to step S204; if not, the process ends.

In step S204, the control unit 101 (setting means 101a) sets a predetermined area in the image to assist walking, based on the results of the image analysis. In this embodiment, the control unit 101 (setting means 101a) sets a virtual line (walking guide line) indicating a walkable area so as to connect the steps, as an example of the predetermined area. FIG. 5 is a diagram showing an example of a walking guide line. Walking guide line 501, indicated by a dashed line, is generated parallel to steps 302, 303 in the section where steps 302, 303 are interrupted.

In step S205, the control unit 101 (acquisition means 101b) acquires information about the position of the white cane (specific object) in the image obtained by analyzing the image acquired in step S201. FIG. 6 is a schematic diagram showing a white cane. 601, 602, and 603 respectively represent the grip, handle, and ferrule of the white cane. 604 represents a painted part painted red for about 20 cm near the tip of the white cane. In this embodiment, the control unit 101 acquires information about the position of the white cane by detecting the painted part 504. Methods for detecting a part having a specific color from an image include a method using a color histogram for each divided block, and a method using machine learning such as Deep Learning, as in step S202. In this embodiment, information about the position of the white cane is acquired by analyzing the image, but information about the position of the white cane may also be acquired by installing a position identification transmitter in the white cane and acquiring information about the position of the white cane by the transmitted signal.

In step S206, the control unit 101 determines whether the position information regarding the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 determines whether the white cane has crossed the walking guide line (crossed the walking guide line) on the image. FIG. 7 is a diagram showing the history of the position of the white cane. In FIG. 7, the white circle represents the position of the white cane, and the position of the white cane changes in the order of (a) to (c). In FIG. 7(a) and (b), the white cane is located to the right of the walking guide line, and the control unit 101 determines that the white cane does not cross the walking guide line. In FIG. 7(c), the white cane has moved from the right to the left of the walking guide line. Therefore, the control unit 101 determines that the white cane has crossed the walking guide line at the time of FIG. 7(c). If the white cane has crossed the walking guide line, the process proceeds to step S207; if not, the flow ends.

In step S207, the control unit 101 (control means 101c) sends an instruction to the notification means to perform a notification operation for the user. In this embodiment, the control unit 101 sends an instruction to the vibration unit 125 via the communication I/Fs 106 and 124 to vibrate.

Next, the processing of the vibration device 120 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing the processing of the vibration device 120. Each process in the flowchart of FIG. 8 is realized by the control unit 121 expanding a program stored in the non-volatile memory 123 into the memory 122 and executing it.

In step S801, the control unit 121 determines whether or not an instruction has been received from the control unit 101. If an instruction has been received, the process proceeds to step S802; if not, the process of this step is repeated.

In step S802, the control unit 121 vibrates the vibration unit 125 to notify the user that the white cane has crossed the walking guide line.

As described above, according to the configuration of this embodiment, when the white cane crosses a walking guide line in a section where there are no steps, the vibration device attached to the white cane is vibrated. This makes it possible to assist the user in walking along the path, even in passages where no special markers are installed.

In this embodiment, we will explain a configuration that assists the user in walking when turning a corner.

Fig. 9 is a flowchart showing the processing of the imaging device 100 of this embodiment. Each process in the flowchart in Fig. 9 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. Note that the processes of steps S201, S205, and S207 in Fig. 9 are similar to the processes of steps S201, S205, and S207 in Fig. 2, respectively, and therefore will not be described.

In step S901, the control unit 101 checks the current position of the imaging device 100 acquired from the GPS 107 against map data stored in the non-volatile memory 103 or the memory 102 to determine whether or not the user is near a corner. If the user is near a corner, the process proceeds to step S902; if not, the process proceeds to step S905. Note that in this step, it may also be possible to use route information set in advance to determine whether or not the user is about to turn a corner.

In step S902, the control unit 101 (setting means 101a) determines whether or not there is a corner by analyzing the image acquired in step S201. As in the first embodiment, a method of determining whether or not there is a corner is to use machine learning such as Deep Learning. If there is a corner, the process proceeds to step S903, and if not, the process proceeds to step S905. Note that in this embodiment, information regarding the presence or absence of a corner is acquired by the control unit 101 analyzing the image, but may also be acquired using position information based on a signal from an artificial satellite.

In step S903, the control unit 101 (setting means 101a) sets a predetermined area on the image to assist walking, based on the analysis result of the image. In this embodiment, the control unit 101 (setting means 101a) sets a walking guide line for turning corners as an example of the predetermined area. FIG. 10 is a diagram showing an example of a walking guide line for turning corners. The image in FIG. 10 shows a road 1001 and steps 1002 and 1003. The walking guide line 1004 shown by a dashed line is a walking guide line for a user who has been walking along the step 1003 to turn the corner, and is set to go around the corner, starting from the point where the step 1003 ends.

In step S904, the control unit 101 determines whether the position information regarding the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 determines whether the white cane has crossed the walking guide line on the image. The processing in step S904 is similar to the processing in step S206, but in this step, the control unit 101 determines that the white cane has crossed the walking guide line when the white cane moves from the left side of the walking guide line to the right side. This allows the user to walk inside the walking guide line. If the white cane has crossed the walking guide line, the process proceeds to step S207, and if not, the process ends. Note that while FIG. 10 shows a case in which the user turns left, in the case of turning right, the control unit 101 may determine that the white cane has crossed the walking guide line when it moves from the right side to the left side.

In step S905, the control unit 101 performs the processing from step S202 onward in FIG. 2.

As described above, the configuration of this embodiment can assist the user in walking even when turning corners.

In this embodiment, we will explain a configuration that assists the user in walking when there is an obstacle on the road.

Fig. 11 is a flowchart showing the processing of the imaging device 100 of this embodiment. Each process in the flowchart of Fig. 11 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. Note that the processing of steps S201, S205, and S207 in Fig. 11 is similar to the processing of steps S201, S205, and S207 in Fig. 2, respectively, and therefore will not be described.

In step S1101, the control unit 101 (setting means 101a) determines whether or not there is an obstacle on the road by analyzing the image acquired in step S201. As in the first embodiment, a method of determining whether or not there is an obstacle is to use machine learning such as Deep Learning. If there is an obstacle, the process proceeds to step S1102, and if not, the process proceeds to step S1104. Note that in this embodiment, information regarding the presence or absence of an obstacle is acquired by the control unit 101 analyzing the image, but may also be acquired using position information based on a signal from an artificial satellite.

In step S1102, the control unit 101 (setting means 101a) sets a predetermined area on the image to assist walking based on the analysis result of the image. In this embodiment, the control unit 101 (setting means 101a) sets walking guide lines to avoid obstacles as an example of the predetermined area. FIG. 12 is a diagram showing an example of walking guide lines to avoid obstacles. The image in FIG. 12 shows a road 1201 and obstacles 1202 and 1203. The inner walking guide line 1204 and the outer walking guide line 1205, shown by dashed lines, are walking guide lines to walk while avoiding the obstacles 1202 and 1203. The inner walking guide line 1204 is set so as to follow the obstacle. The outer walking guide line 1205 is set so as to go around the obstacle.

In step S1103, the control unit 101 determines whether the position information regarding the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 determines whether the white cane has crossed the walking guide line on the image. The processing of step S1103 is similar to the processing of step S206, but in this step, the direction in which the control unit 101 determines that the white cane has crossed the walking guide line is different for the inner walking guide line 1204 and the outer walking guide line 1205. Specifically, the control unit 101 determines that the white cane has crossed the walking guide line when it moves from the right side to the left side of the inner walking guide line 1204, and determines that the white cane has crossed the walking guide line when it moves from the left side to the right side of the outer walking guide line 1205. In other words, the control unit 101 determines that the white cane has crossed the two walking guide lines when it moves from the inside to the outside. This enables the user to walk without hitting an obstacle with the white cane and without going too far into the roadway. If the white cane intersects with the walking guide line, proceed to step S207; if not, end this flow.

In step S1104, the control unit 101 performs the processing from step S901 onward in FIG. 9.

As described above, according to the configuration of this embodiment, it is possible to assist the user in walking even when there is an obstacle on the road.
[Other Examples]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

101 Control unit (control device)
101a Setting means 101b Acquisition means 101c Control means 104 Imaging unit (imaging means)
125 Vibration unit (notification means)

Claims (20)

a setting means for setting walking guide lines indicating walkable areas in an image captured by an imaging means, using information regarding the end of steps on a road in the image;
an acquisition means for acquiring position information relating to a position of a specific object in the image;
and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line. A setting means for setting a walking guide line indicating a walkable area in an image captured by the imaging means, using information about a corner of a road in the image;
an acquisition means for acquiring position information relating to a position of a specific object in the image;
and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line. a setting means for setting walking guide lines indicating walkable areas in an image captured by the imaging means, using information about obstacles on a road in the image;
an acquisition means for acquiring position information relating to a position of a specific object in the image;
and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line. 2. The control device according to claim 1 , wherein the setting means acquires information regarding the end of the step on the road by analyzing the image. 3. The control device according to claim 2, wherein the setting means acquires information regarding the corner of the road by analyzing the image. 4. The control device according to claim 3, wherein the setting means acquires information regarding an obstacle on the road by analyzing the image. 2. The control device according to claim 1 , wherein the setting means obtains information regarding the end of the step on the road by using position information based on a signal from a satellite. 3. The control device according to claim 2, wherein the setting means acquires information regarding corners of the road. 4. The control device according to claim 3, wherein the setting means acquires information regarding an obstacle on the road. The control device according to claim 1, characterized in that the setting means sets the walking guide line so as to connect the discontinuities. The control device according to claim 2, characterized in that the setting means sets the walking guide line so as to go around the corner. The control device according to claim 3, characterized in that the setting means sets a first walking guide line along the obstacle and sets a second walking guide line to go around the obstacle. 13. The control device according to claim 1, wherein the notification means is a vibration means. The control device according to any one of claims 1 to 12 , wherein the specific object is a walking stick. An imaging means for capturing an image;
A walking assistance device comprising the control device according to any one of claims 1 to 14 . A walking assistance device according to claim 15 ;
a notification device including notification means for performing a notification operation based on an instruction from the walking assist device. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information regarding the end of a step on a road in the image;
obtaining position information relating to a position of a specific object in the image;
and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information about corners of a road in the image;
obtaining position information relating to a position of a specific object in the image;
and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information about obstacles on a road in the image;
obtaining position information relating to a position of a specific object in the image;
and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation. A program for causing a computer to execute the control method according to any one of claims 17 to 19 .