JP7685911B2 - Information collection system, information collection method, and information collection program - Google Patents

Description

This disclosure relates to an information collection device, an information collection method, and an information collection program. In particular, it relates to a technology for collecting information in real time about an object that an occupant of an autonomous PMV (Personal Mobility Vehicle) pays attention to while traveling.

When an autonomous PMV is used to move within the grounds of various facilities such as theme parks, entertainment facilities, or elderly care facilities, it is necessary to collect information on what the passengers of the autonomous PMV pay attention to appropriately and in real time. The autonomous PMV is, for example, a mobile object such as an electric wheelchair or a senior car.
By collecting information about objects that passengers pay attention to within the premises of various facilities and analyzing the collected information, it can be used to improve the comfort and safety of autonomous PMVs.

Patent document 1 discloses a technology that reads the facial expressions of passengers and estimates whether their travel on a link while in motion is comfortable.

JP 2019-002789 A

In Patent Document 1, it is estimated whether travel on a link while traveling is comfortable from the facial expression of the passenger. Patent Document 1 only estimates whether the link is comfortable or uncomfortable, and does not specify what the passenger saw that made it comfortable or uncomfortable. Therefore, Patent Document 1 is unable to collect information on the objects that passengers focus on while traveling appropriately and in real time, and it is unable to use this information to improve the comfort and safety of autonomous PMVs.

The objective of this disclosure is to appropriately and in real time identify the object that a passenger in an autonomous PMV is paying attention to while the autonomous PMV is traveling.

The present disclosure relates to an information collection system for collecting information in a moving area where a moving object moves, comprising:
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
The vehicle is provided with an attention target identification unit that identifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger.

In the information collection system according to the present disclosure, the attention target identification unit identifies an object or event that is in the line of sight of the passenger as the attention target of the passenger, based on the position information of the moving body and the line of sight of the passenger. Therefore, the information collection system according to the present disclosure has the effect of being able to appropriately and in real time identify the object that the passenger of the moving body is paying attention to while the moving body is traveling in a travel area.

1 is a diagram showing an example of the overall configuration of an information collecting system according to a first embodiment; FIG. 2 is a diagram showing an example of a PMV according to the first embodiment; FIG. 1 is a diagram showing a configuration example of an information collecting device according to a first embodiment. FIG. 2 is a diagram showing a configuration example of a server device according to the first embodiment; FIG. 4 is a flow diagram showing an example of an attention target identification process according to the first embodiment. FIG. 4 is a flow diagram showing an example of an attention setting process according to the first embodiment. FIG. 4 is a diagram showing an example of an attention database according to the first embodiment. FIG. 4 is a flowchart showing an example of a map setting process according to the first embodiment. FIG. 4 is a diagram showing an example of map information according to the first embodiment. FIG. 4 is a flow diagram showing an example of a recommended information display process according to the first embodiment. FIG. 4 is a flow diagram showing an example of a warning notification process according to the first embodiment. FIG. 11 is a flow diagram showing an example of an attention target identification process according to a first modification of the first embodiment. FIG. 13 is a diagram showing a configuration example of an information collection device according to a third modification of the first embodiment. FIG. 13 is a diagram showing an example of a PMV according to a third modification of the first embodiment. FIG. 13 is a diagram showing a configuration example of an information collecting device according to a fourth modification of the first embodiment.

The present embodiment will be described below with reference to the drawings. In each drawing, the same or corresponding parts are given the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. Furthermore, the size relationships of the components in the drawings below may differ from the actual relationships. Furthermore, in the description of the embodiment, directions or positions such as up, down, left, right, front, back, front, and back may be indicated. These notations are given for the convenience of explanation and do not limit the arrangement, direction, or orientation of devices, instruments, parts, etc.

Embodiment 1.
***Configuration Description***
FIG. 1 is a diagram showing an example of the overall configuration of an information collection system 500 according to the present embodiment.
The information collection system 500 is a system that collects various types of information in a moving area 800 in which a moving object moves.

The PMV 200 is an example of a moving object that moves in the moving area 800. The PMV 200 is a PMV that performs autonomous driving. The PMV 200 is also called an autonomous driving PMV.
Although not shown, passenger 30 is a person who travels by riding in PMV 200. Passenger 30 can travel to the destination automatically without driving by setting the destination in the automatic driving system installed in PMV 200. Therefore, passenger 30 can travel while viewing the surrounding scenery or surrounding environment even while riding in the vehicle.

A PMV is a motorized vehicle in which occupants are not spatially enclosed as in a car, but are instead exposed to the environment. More specifically, a PMV refers to a mobile object such as an electric wheelchair or a senior car. In particular, such mobile objects that are equipped with an autonomous driving function are called autonomous PMVs, or simply PMVs.

The travel area 800 is assumed to be an area in which the PMV 200 travels by autonomous driving, such as the grounds of a senior care facility, an entertainment facility, or a theme park.
This embodiment can also be applied when the moving object moves in a moving area such as a general road or a sidewalk.

The information collection system 500 includes a PMV 200 and a server device 300. In this embodiment, the information collection device 100 is mounted on the PMV 200.
The PMV 200 and the server device 300 communicate with each other via a network. The server device 300 may be installed inside the moving area 800 or outside the moving area 800.

FIG. 2 is a diagram showing an example of a PMV 200 according to the present embodiment.
The PMV 200 is an example of a moving object.
The PMV 200 includes a position sensor 201 , a passenger sensor 202 , an imaging device 203 , an autonomous driving system 204 , and an information collection device 100 .

The position sensor 201 acquires position information of the PMV 200. More specifically, the position sensor 201 acquires position information and time of the PMV 200 during movement. The position sensor 201 is, for example, a GPS (Global Positioning System).

The passenger sensor 202 acquires the line of sight direction of the passenger 30 riding in the PMV 200. The passenger sensor 202 also acquires emotion information indicating the type of emotion of the passenger 30 toward the target of attention from the facial expression of the passenger 30.
The passenger sensor 202 is, for example, a smartphone or tablet terminal with a camera function. The passenger sensor 202 captures an image of the face of the passenger 30 using the camera function, and acquires the line of sight of the passenger 30 by analyzing the captured image of the face.
Furthermore, the passenger sensor 202 acquires emotional information representing the emotions of the passenger 30 by analyzing the facial image. The emotional information may be facial expression. A smiling expression is considered to mean an emotion such as comfort, enjoyment, or happiness. A stern expression means an emotion such as discomfort, anger, or sadness. The passenger sensor 202 acquires emotional information representing the facial expression or emotion from the facial image.

The imaging device 203 is equipped with an imaging device 203 capable of capturing images in all directions. The imaging device 203 captures, for example, an image in the line of sight of the passenger 30. The imaging device 203 captures an image in the line of sight of the passenger 30 in response to a command from the information collection device 100.

The autonomous driving system 204 includes a communication device that communicates with external devices via a network, an autonomous driving control unit that controls autonomous driving, and a display device that displays maps, etc.

The information collection device 100 acquires information from the position sensor 201 , the passenger sensor 202 , or the imaging device 203 , and identifies an object of interest that the passenger 30 is paying attention to while moving within the moving area 800 .
The information collection device 100 is a computer. In this embodiment, the information collection device 100 is installed in a smartphone or a tablet that includes a passenger sensor 202.
However, the information collection device 100 may be mounted on a computer separate from the passenger sensor 202. Alternatively, the position sensor 201, the passenger sensor 202, and the information collection device 100 may all be mounted on a single computer such as a smartphone or a tablet.

FIG. 3 is a diagram showing an example of a configuration of the information collecting device 100 according to the present embodiment.
The information collection device 100 is a computer. The information collection device 100 includes a processor 910, as well as other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to the other hardware via signal lines and controls the other hardware.

The information collection device 100 includes, as functional elements, an information acquisition unit 110, a sensor information receiving unit 120, an attention target identification unit 130, a display unit 140, and a memory unit 150. The memory unit 150 stores map information 51 and an attention database 52.

The functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 are realized by software. The storage unit 150 is provided in the memory 921. Note that the storage unit 150 may be provided in the auxiliary storage device 922, or may be provided separately in the memory 921 and the auxiliary storage device 922.

The processor 910 is a device that executes an information collection program. The information collection program is a program that realizes the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140.
The processor 910 is an integrated circuit (IC) that performs arithmetic processing. Specific examples of the processor 910 include a central processing unit (CPU), a digital signal processor (DSP), and a graphics processing unit (GPU).

The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 include a static random access memory (SRAM) and a dynamic random access memory (DRAM).
The auxiliary storage device 922 is a storage device that stores data. A specific example of the auxiliary storage device 922 is a HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, a CF, a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

The input interface 930 is a port that is connected to an input device such as a mouse, a keyboard, or a touch panel. Specifically, the input interface 930 is a USB (Universal Serial Bus) terminal. The input interface 930 may also be a port that is connected to a LAN (Local Area Network).

The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) (High Definition Multimedia Interface) terminal. Specifically, the display is an LCD (Liquid Crystal Display). The output interface 940 is also called a display interface.

The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, or a telephone line. Specifically, the communication device 950 is a communication chip or a NIC (Network Interface Card).

The information collection program is executed in the information collection device 100. The information collection program is loaded into the processor 910 and executed by the processor 910. The memory 921 stores not only the information collection program but also an OS (Operating System). The processor 910 executes the information collection program while executing the OS. The information collection program and the OS may be stored in the auxiliary storage device 922. The information collection program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the information collection program may be incorporated into the OS.

The information collection device 100 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the information collection program. Each processor is a device that executes the information collection program in the same way as the processor 910.

The data, information, signal values and variable values used, processed or output by the information collection program are stored in memory 921, auxiliary storage device 922, or in a register or cache memory within processor 910.

The "parts" of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 may be read as "circuits,""steps,""procedures,""processing," or "circuitry." The information collection program causes a computer to execute an information acquisition process, a sensor information receiving process, an attention information identification process, and a display process. The "processing" of the information acquisition process, the sensor information receiving process, the attention information identification process, and the display process may be read as a "program," a "program product," a "computer-readable storage medium storing a program," or a "computer-readable recording medium recording a program." The information collection method is a method performed by the information collection device 100 executing the information collection program.
The information collection program may be provided in a form stored in a computer-readable recording medium, or as a program product.

FIG. 4 is a diagram showing an example of a configuration of the server device 300 according to the present embodiment.
The server device 300 is a computer. The server device 300 includes a processor 910, as well as other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to the other hardware via signal lines and controls the other hardware.
In FIG. 4, for ease of explanation, the same reference numerals are used for hardware having the same functions as those in FIG. 3, but the information collection device 100 and the server device 300 each have their own separate hardware.

The server device 300 includes, as functional elements, a map setting unit 310, an attention setting unit 320, an information distribution unit 330, a notification unit 340, and a storage unit 350. The storage unit 350 stores map information 51, an attention database 52, and notification conditions 53.
The hardware of the server device 300 is similar to that described for the information collection device 100 .

*** Operation Description ***
Next, an operation of the information collecting system 500 according to the present embodiment will be described. The operation procedure of the information collecting system 500 corresponds to an information collecting method. Also, a program for realizing the operation of the information collecting system 500 corresponds to an information collecting program.

<Attention target identification process>
FIG. 5 is a flow diagram showing an example of an attention target identification process according to the present embodiment.
In this embodiment, the attention target identification process is performed by information collection device 100 mounted on PMV 200 .

In step S101, the information acquisition unit 110 of the information collection device 100 acquires the map information 51 and the attention database 52 distributed from the server device 300 via the communication device 950. Then, the information acquisition unit 110 stores the distributed map information 51 and the attention database 52 in the storage unit 150.
Server device 300 collects information on the target of attention from PMV 200 moving in moving area 800, and updates map information 51 and attention database 52. Server device 300 then distributes the latest map information 51 and attention database 52 to PMV 200 every time it updates map information 51 and attention database 52, or periodically.
Therefore, the process of step S101 does not necessarily have to be performed at the beginning of the attention target identification process, but may be performed at the time when the map information 51 and the attention database 52 are delivered.

In step S102, the sensor information receiving unit 120 of the information collection device 100 receives sensor information from various sensors mounted on the PMV 200.

Specifically, the sensor information receiving unit 120 receives the position information 41 of the PMV 200 from the position sensor 201 .
In addition, the sensor information receiving unit 120 receives the line of sight direction 42 of the passenger 30 riding in the PMV 200 from the passenger sensor 202 .
In addition, the sensor information receiving unit 120 receives emotion information 43 indicating the type of emotion of the occupant 30 from the occupant sensor 202 .
The position information 41, the gaze direction 42, and the emotion information 43 are linked by time.

Next, the attention target identification unit 130 of the information collection device 100 identifies an object or event present in the line of sight 42 of the passenger 30 as the attention target 45 that the passenger 30 is paying attention to based on the position information 41 of the PMV 200 and the line of sight direction 42 of the passenger 30.
For example, the attention target identification unit 130 identifies the attention target 45 from the map information 51 stored in the storage unit 150 .
Specifically, the following applies:

First, in step S103, the attention target identification unit 130 converts the line of sight 42 of the passenger 30 into an absolute coordinate system.
The position information 41 is obtained as a position in an absolute coordinate system by a GPS or the like. The line of sight direction 42 is obtained from an image of the face of the passenger 30 as a direction in a PMV coordinate system based on the traveling direction of the PMV 200.
The attention target identification unit 130 calculates the traveling direction of the PMV 200 from the position information 41 of the PMV 200. The traveling direction of the PMV 200 is acquired as a direction in an absolute coordinate system. The attention target identification unit 130 calculates in which direction the line of sight 42 is facing relative to the traveling direction of the PMV 200, and converts the line of sight 42 of the passenger 30 into the absolute coordinate system.

Then, in step S104, the attention target identification unit 130 identifies an object or event that is in the line of sight 42 of the passenger 30 as the attention target 45 from the map information 51 based on the position of the PMV 200 and the line of sight direction expressed in the absolute coordinate system.

The map information 51 includes surrounding environment information that indicates the surrounding environment of the map including the travel area 800 .
The surrounding environment information refers to objects or events that can be seen in the moving area 800, such as buildings, sculptures, attractions, plants, and animals set within the moving area 800.
Additionally, events such as parades or fireworks held within the moving area 800, and scenery that can be seen from the moving area 800 are also added to the map information 51 as surrounding environment information.
The attention target identification unit 130 identifies an object or event existing in the line of sight 42 of the passenger 30 as an attention target 45 from the surrounding environment information added to the map information 51 .

In step S105, the attention target identification unit 130 transmits attention target information 40 including the position information 41 of the PMV 200, the attention target 45, the line of sight direction 42, the position of the attention target 45, and emotion information 43 to the server device 300. Note that when the attention target 45 is a scene outside the movement area 800, the position of the attention target 45 does not need to be included in the attention target information 40.
The position information 41 of the PMV 200, the target of attention 45, the line of sight direction 42, the position of the target of attention 45, and the emotion information 43 included in the target of attention information 40 are linked by time. For this reason, it can be estimated that the emotion information 43 represents the emotion of the passenger toward the target of attention 45.

<Attention Setting Processing>
FIG. 6 is a flow diagram showing an example of the attention setting process according to the present embodiment.
The attention setting unit 320 of the server device 300 sets the position of the attention target 45 and the emotion information 43 of the passenger 30 with respect to the attention target 45 in the attention database 52 .

In step S201, the attention setting unit 320 receives the attention target information 40 from the information collection device 100 via the communication device 950.
In step S202, attention setting section 320 sets position information 41 of PMV 200, attention target 45, line of sight direction 42, position of attention target 45, and emotion information 43, which are included in attention target information 40, in attention database 52.

In step S203, the attention setting unit 320 calculates an attention level 48 that indicates the degree of attention to the attention target 45. For example, the attention setting unit 320 counts the total number of passengers who have passed a position where the attention target 45 can be seen in the past, counts the number of passengers who paid attention to the attention target 45 among the total number of passengers, and calculates the proportion of passengers who paid attention to the attention target 45 among the total number of passengers as the attention level 48. The attention level 48 only needs to be able to indicate the degree of attention to the attention target 45, and may be calculated by other calculation methods.
In step S204, the attention setting unit 320 sets the attention level 48 for the attention target 45 in the attention database 52.

FIG. 7 is a diagram showing an example of the attention database 52 according to the present embodiment.
In the attention database 52, a position range 521 of the PMV 200, an attention target 522, a line of sight direction 523, a position 524 of the attention target, emotion information 525, and an attention level 526 are set.
Position range 521 of PMV 200 is a range within movement area 800 in which attention target 522 can be seen. Movement area 800 is assumed to be divided into predetermined ranges in advance. Alternatively, a road or sidewalk along which PMV 200 moves in movement area 800 may be divided into predetermined ranges in advance. Attention setting unit 320 sets position range 521 and attention target 522 based on attention target 45 and position information 41 included in attention target information 40.
Line of sight direction 523 is a line of sight direction expressed in a PMV coordinate system based on the traveling direction of PMV 200. Attention setting section 320 sets line of sight direction 523 based on line of sight direction 42 included in attention target information 40.
The position 524 of the target of attention is the position of the target of attention included in the target of attention information 40 .
Emotion information 525 is emotion information 43 included in attention target information 40 .
The attention level 48 is calculated in step S203.

In step S205, the information distribution unit 330 distributes the attention database 52 to the PMV 200.

<Map setting process>
The map setting unit 310 of the server device 300 adds the emotion information 43 and the attention level 48 of the passenger 30 with respect to the attention object 45 to the map information 51 .

FIG. 8 is a flow diagram showing an example of a map setting process according to the present embodiment.
In step S301, the map setting unit 310 refers to the attention database 52.
In step S302, the map setting unit 310 extracts information on the attention object 522 that has been newly set in the attention database 52 by the attention setting process. The map setting unit 310 adds the attention object 522, emotion information 525 of the passenger 30 regarding the attention object 522, and an attention level 526 to the map information 51.

FIG. 9 is a diagram showing an example of map information 51 according to the present embodiment.
For example, a case will be described in which a "model house" installed in the moving area 800 is specified as a new attention target 45 in the attention database 52 of FIG.
7, the "model house" is set as the attention target 522, and a positional range 521 in which the "model house" is visible and a gaze direction 523 when the "model house" is identified as the attention target are set. In addition, the position 524 of the "model house" and "smile" which is emotion information 525 regarding the "model house" are set, and the attention level 526 of the "model house" are set to "0.8".
The map setting unit 310 adds a mark representing the attention target 45, a mark representing a "smile" and an attention level of "0.8" to the position of the "model house" in the map information 51.

<Recommended information display process>
The display unit 140 of the information collection device 100 displays, on the display device of the PMV 200, recommendation information for recommending the passenger 30 to look at the attention target 45 to which good emotion information has been assigned.

FIG. 10 is a flow diagram showing an example of the recommended information display process according to the present embodiment.
In step S 401 , the display unit 140 refers to the attention database 52 .
In step S402, the display unit 140 determines whether the PMV 200 is approaching a position corresponding to the attention target 45 to which good emotion information has been assigned. When the display unit 140 determines that the PMV 200 is approaching a position corresponding to the attention target 45 to which good emotion information has been assigned, the display unit 140 displays recommendation information 46 on the display device that recommends viewing the attention target 45.

<Warning notification process>
The notification unit 340 of the server device 300 issues a warning 47 to the manager of the moving area 800 when the attention target 45 satisfies a notification condition 53 for issuing a warning 47 to the manager of the moving area 800 .

FIG. 11 is a flow diagram showing an example of a warning notification process according to the present embodiment.
In step S 501 , the notification unit 340 refers to the attention database 52 .

In step S502, the notification unit 340 determines whether or not there is an attention target 45 in the attention database 52 that satisfies the notification condition 53.
For example, if the notification condition is that the emotion information is bad and the attention level is 0.4 or more, the notification unit 340 refers to the attention database 52 and searches for attention targets 45 that have bad emotion information and an attention level of 0.4 or more.
If there is an attention target 45 that satisfies the notification condition 53, the notification unit 340 proceeds to step S503, and if there is not, the notification unit 340 returns to step S501.

In step S503, the notification unit 340 notifies the administrator of the moving area 800 of the warning 47. The notification unit 340 may output the warning 47 to an output device such as a display device of the server device 300, or may send the warning 47 to a pre-registered email.

For example, as shown in FIG. 9, if there is a trash can overflowing with trash within the movement area 800, a passenger 30 of a PMV 200 passing nearby is likely to look at the trash can and make an unpleasant expression. In such a case, a warning 47 is sent to the manager in real time, so that the manager can immediately take measures against the discomfort factor within the movement area 800.

***Other configurations***
<Modification 1>
In this embodiment, the attention target identification unit 130 identifies the attention target 45 from the map information 51. In the first modification, a mode in which the attention target 45 is identified by another method will be described.

FIG. 12 is a flow diagram illustrating an example of an attention target identification process according to the first modification of the present embodiment.
In FIG. 12, the processes from step S102 to step S103 and step S105 are similar to those in FIG.
Furthermore, a new process of step S106 is added after step S103, and thereafter, step S104a is executed instead of step S104.

The PMV 200 is equipped with an imaging device 203 that can capture images in all directions. The imaging device 203 captures images in the line of sight of the passenger 30 in response to a command from the information collection device 100.

In step S106, the attention target identification unit 130 transmits an instruction to the image capturing device 203 to capture an image of the gaze direction converted into the absolute coordinate system in step S103. The image capturing device 203 captures an image of the gaze direction of the passenger 30, and outputs the image obtained by capturing the image as the gaze direction image 44.
In step S<b>104 a , the attention target identification unit 130 identifies the attention target 45 from the gaze direction image 44 .

In this way, by identifying the target of interest 45 from the line-of-sight direction image 44 captured by the imaging device 203, it is possible to identify the target of interest 45 that cannot be identified from the map information 51.

<Modification 2>
In the present embodiment, the information collection device 100 has been described as being mounted on a smartphone or tablet equipped with the passenger sensor 202 .
However, the information collection device 100 may be mounted on a computer separate from the passenger sensor 202. Alternatively, the position sensor 201, the passenger sensor 202, and the information collection device 100 may all be mounted on a single computer such as a smartphone or a tablet.
Further, some or all of the functions of the information collection device 100 may be installed in the server device 300 .

In addition, in this embodiment, the passenger sensor 202 analyzes the gaze direction and emotional information. However, the passenger sensor 202 may transmit only the facial image of the passenger to the information collection device 100, and the passenger's gaze direction and emotional information may be analyzed in the information collection device 100.

<Modification 3>
FIG. 13 is a diagram showing an example of a configuration of an information collection device 100 according to a third modification of the present embodiment.
FIG. 14 is a diagram showing an example of a PMV 200 according to a third modification of the present embodiment.
In the third modification, a safety assurance method for ensuring the safety of passengers 30 using sensors mounted on the PMV will be described.
The information collection device 100 according to the third modification of the present embodiment includes an analysis unit 160 in addition to the configuration shown in FIG.
2. Moreover, the PMV 200 according to the third modification of this embodiment includes a seating sensor 205 and a seat belt sensor 206 in addition to the components shown in FIG.

<<Example 1 of safety assurance method>>
The passenger sensor 202 periodically transmits a face image 491 .
The analysis unit 160 analyzes the face image 491 and determines whether the health condition of the passenger 30 is normal or abnormal. If the health condition is determined to be abnormal, the analysis unit 160 notifies the manager of the movement area 800 of the abnormality in the health condition of the passenger 30.
In the case of the example 1 of the safety ensuring method, the seating sensor 205 and the seat belt sensor 206 may be omitted.

<<Example 2 of safety assurance method>>
The seating sensor 205 detects the seating state of the passenger 30 .
The seat belt sensor 206 detects the state of the seat belt worn by the passenger 30 .
The analysis unit 160 judges whether the seating state of the passenger 30 is normal or abnormal based on the sensing information 492 from the seating sensor 205 and the seat belt sensor 206. If the seating state is judged to be abnormal, the analysis unit 160 notifies the manager of the movement area 800 of the abnormality in the seating state of the passenger.
In addition, an abnormality in the seating state may be detected by using either the seating sensor 205 or the seat belt sensor 206 alone.

The above-described safety assurance method makes it possible to detect danger or abnormality of passengers, such as the following:
・It's dangerous if you fall asleep.
Not looking in the direction you should be facing. For example, not looking at the traffic lights before a pedestrian crossing, or not facing an approaching object that may cause a collision.
- Looks obviously unwell, such as pale, swollen, or bleeding.
- The position of the face is strange, as if it looks like it might slip off or fall off to one side.

<Modification 4>
In this embodiment, the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 are realized by software. As a fourth modification, the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 may be realized by hardware.
Specifically, the information collection device 100 includes an electronic circuit 909 instead of a processor 910 .

FIG. 15 is a diagram showing an example of a configuration of an information collection device 100 according to the fourth modification of the present embodiment.
The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

The functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 may be realized by a single electronic circuit, or may be distributed across multiple electronic circuits.

As another variation, some of the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 may be realized by electronic circuits, and the remaining functions may be realized by software. Also, some or all of the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 may be realized by firmware.

Each of the processor and electronic circuitry is also called a processing circuitry. In other words, the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 are realized by the processing circuitry.

As with the information collection device 100, the functions of the information acquisition unit 110, the sensor information receiving unit 120, the attention target identification unit 130, and the display unit 140 of the server device 300 may be realized by hardware.

***Description of Effects of This Embodiment***
According to this embodiment, the following effects are obtained.
- For theme parks or entertainment venues, it can identify what to look out for and what people dislike.
-Tourist spots will no longer have to worry about not knowing what is popular.
- It is possible to determine in real time what is well received based on a smile, and whether a stern expression indicates a problem.
- From this information, it is possible to determine where a person's facial expression changes when they pass a certain location and turn in a specific direction.

Furthermore, according to this embodiment, the following effects are obtained.
- If something unpleasant is detected, an alert is sent to the center, making it easier to keep the movement area comfortable.
- You can obtain data such as how much attention sign B attracts when it is placed.

Furthermore, according to this embodiment, it is possible to detect whether there is a bias in the positions or times at which people smile.
If there is a bias in the location or time when people smile, by analyzing what location, direction and what they are looking at that makes them smile, it is possible to obtain basic data on what is popular.
On the other hand, if there is a bias in the position or time when the person's expression becomes stern, it can be expected that there is a problem.
Furthermore, according to this embodiment, it is possible to detect whether there is a bias in the facing direction.
If there is a bias in the direction of facing at a certain position, it means that there is something in that direction that is attracting a lot of attention. Conversely, if there is no bias, it means that the attention is low.
For example, if you create a large object for display in an amusement park, you can measure the attention it attracts by the number of people around it who are looking in that direction. If this is exactly the same as when the object is not there, you can determine that the object is not attracting any attention at all. This can be used to obtain data such as how much attention fireworks will attract.
In either case, if it is determined that there is a bias, by installing a camera that takes images in that direction, it is possible to simultaneously collect information about what the bias is.

The safety assurance method in the information collection system 500 according to this embodiment is particularly effective when the seat belt is not fastened.
One anticipated demand for self-driving PMVs is the automatic return home of drunk people or elderly people with impaired cognitive function. If such people are allowed in the vehicle, even if they are seated correctly at first, after a while they may lose their posture and fall out of the vehicle.
If the seating sensor detects that the seat is losing its posture and is about to fall off, the following measures can be taken to avoid the situation.
- Stop in a place that seems relatively safe even if you fall off.
- The rider is notified of the risk of falling off by sound, light, vibration, or a combination of these.
- Notify the administrator or guardian via communication that the child is on the verge of dropping out.

In addition, the safety assurance method in the information collection system 500 according to this embodiment makes it possible to identify locations where such people are likely to ride or where such situations are likely to occur. This allows PMVs to be dispatched with greater consideration given to safety, and problems can be avoided in advance.

In the above first embodiment, each part of the information collection system has been described as an independent functional block. However, the configuration of the information collection system does not have to be the same as that of the above-mentioned embodiment. The functional blocks of the information collection system may have any configuration as long as they can realize the functions described in the above-mentioned embodiment. Furthermore, the information collection system may be composed of multiple devices instead of one device.
In addition, a plurality of parts of the first embodiment may be combined to be implemented. Alternatively, one part of this embodiment may be implemented. In addition, this embodiment may be combined in any manner as a whole or in part to be implemented.
That is, in the first embodiment, the embodiments can be freely combined, or any of the components in each embodiment can be modified, or any of the components in each embodiment can be omitted.

The above-described embodiments are essentially preferred examples, and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiments can be modified in various ways as necessary.

40 Attention target information, 41 Position information, 42 Gaze direction, 43 Emotion information, 44 Gaze direction image, 45 Attention target, 46 Recommended information, 47 Warning, 48 Attention level, 491 Face image, 492 Sensing information, 51 Map information, 52 Attention database, 53 Notification condition, 30 Passenger, 100 Information collection device, 110 Information acquisition unit, 120 Sensor information receiving unit, 130 Attention target identification unit, 140 Display unit, 150 Memory unit, 160 Analysis unit, 200 PMV, 201 Position sensor, 202 Passenger sensor, 203 Imaging device, 204 Autonomous driving system, 205 Seat sensor, 206 Seat belt sensor, 300 Server device, 310 Map setting unit, 320 Attention setting unit, 330 Information distribution unit, 340 Notification unit, 350 Memory unit, 500 information collection system, 521 position range, 522 target of attention, 523 gaze direction, 524 position, 525 emotion information, 526 attention level, 800 movement area, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output interface, 950 communication device.

Claims (17)

An information collection system for collecting information in a moving area in which a moving object moves,
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
an attention target specifying unit that specifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger;
The occupant sensor includes:
acquiring emotion information representing a type of emotion of the passenger toward the target of attention from a facial expression of the passenger;
The information collection system includes:
An information gathering system comprising an attention setting unit that sets a position of the attention target and the emotion information of the passenger with respect to the attention target in an attention database. The attention setting unit is
The information collection system according to claim 1, further comprising: counting the number of passengers paying attention to an object of interest set in the attention database; calculating a degree of attention to the object of interest; and setting the degree of attention to the object of interest in the attention database. The information collection system includes:
map information in which surrounding environment information indicating a surrounding environment is added to a map including the travel area;
The attention target identification unit is
The information gathering system according to claim 2 , wherein the target object is identified from the map information. The information collection system includes:
The information gathering system according to claim 3 , further comprising a map setting unit that adds the emotion information of the passenger with respect to the attention object and the attention level to the map information. The information collection system includes:
an imaging device capable of imaging an entire circumference of the moving body,
The imaging device includes:
capturing an image of the passenger's line of sight and outputting the image obtained by capturing the image as a line of sight direction image;
The attention target identification unit is
The information gathering system according to claim 2 , wherein the target of interest is identified from the line-of-sight direction image. The information collection system includes:
6. The information collection system according to claim 5, further comprising a map setting unit that sets the target of attention on map information in which surrounding environment information representing the surrounding environment is added to a map including the travel area, and that sets the emotional information and the attention level of the passenger with respect to the target of attention. The information collection system includes:
The information collection system according to any one of claims 1 to 6, further comprising a display unit that displays, on a display device of the moving body, recommendation information for recommending to the passenger that he or she look at an attention target to which positive emotion information has been assigned. The information collection system includes:
8. The information collecting system according to claim 1, further comprising a notification unit that issues a warning to a manager of the moving area when the target of interest satisfies a notification condition for issuing a warning to the manager of the moving area. The passenger sensor periodically transmits a facial image;
The information collection system includes:
The information collection system according to any one of claims 1 to 8, further comprising an analysis unit that analyzes the facial image, determines whether the passenger's health condition is normal or abnormal, and, if the health condition is determined to be abnormal, notifies a manager of the travel area of the abnormality in the passenger's health condition. The information collection system includes:
a seating sensor for detecting the seating state of the passenger;
a seat belt sensor for detecting a state of a seat belt worn by the passenger;
The analysis unit includes:
The information collection system of claim 9, which determines whether the seating state of the occupant is normal or abnormal based on sensing information from the seating sensor and the seat belt sensor, and if the seating state is determined to be abnormal, notifies the manager of the movement area of the abnormality in the seating state of the occupant. 1. An information gathering method used in an information gathering system for gathering information in a moving area where a moving object moves, comprising:
Acquire location information of the moving object;
Acquire a line of sight direction of a passenger riding in the moving object,
Based on the position information of the moving body and the line of sight of the passenger, an object or event existing in the line of sight of the passenger is identified as an attention target of the passenger;
acquiring emotion information representing a type of emotion of the passenger toward the target of attention from a facial expression of the passenger;
The information gathering method includes setting the position of the target of attention and the emotion information of the passenger with respect to the target of attention in an attention database. An information collection program used in an information collection system for collecting information in a moving area where a moving object moves,
A location acquisition process for acquiring location information of the moving body;
A line of sight acquisition process for acquiring a line of sight direction of a passenger riding in the moving body;
an attention target identification process for identifying an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to, based on position information of the moving body and the line of sight of the passenger;
acquiring emotion information representing a type of emotion of the passenger toward the target of attention from a facial expression of the passenger;
and an attention setting process for setting the position of the attention target and the emotion information of the passenger with respect to the attention target in an attention database. An information collection system for collecting information in a moving area in which a moving object moves,
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
an attention target specifying unit that specifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger;
An information collection system comprising: a display unit that displays, on a display device of the vehicle, recommendation information for recommending to the passenger that he or she look at an attention target to which positive emotion information has been assigned. An information collection system for collecting information in a moving area in which a moving object moves,
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
an attention target specifying unit that specifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger;
and a notification unit that issues a warning to a manager of the moving area when the target of interest satisfies a notification condition for issuing a warning to the manager of the moving area. An information collection system for collecting information in a moving area in which a moving object moves,
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
an attention target specifying unit that specifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger;
The passenger sensor periodically transmits a facial image;
The information collection system includes:
The information collection system includes an analysis unit that analyzes the facial image, determines whether the passenger's health condition is normal or abnormal, and, if the health condition is determined to be abnormal, notifies the manager of the travel area of the abnormality in the passenger's health condition. An information collection system for collecting information in a moving area in which a moving object moves,
a position sensor for acquiring position information of the moving object;
A passenger sensor for acquiring a line of sight direction of a passenger riding on the moving body;
an attention target specifying unit that specifies an object or event present in the line of sight of the passenger as an attention target that the passenger is paying attention to based on position information of the moving body and the line of sight of the passenger;
an attention setting unit that counts the number of passengers paying attention to the object of attention, calculates an attention level for the object of attention, and sets the attention level for the object of attention in an attention database. The attention setting unit is
The information collection system of claim 16, which counts the total number of passengers passing a position from which the object of interest can be seen, counts the number of passengers among the total number of passengers who looked at the object of interest, and calculates the proportion of passengers among the total number of passengers who looked at the object of interest as the attention level for the object of interest.

Images