JP7778633B2 - Monitoring System - Google Patents

Description

The present invention relates to a monitoring system that can be used in various mobile objects such as vehicles.

Conventionally, vehicle control devices that employ technology known as a DMS (Driver Monitoring System) that detects abnormalities in the driver of a vehicle such as an automobile are known (see, for example, Patent Document 1). Furthermore, conversational systems known as chatbots are used to enable the operation of in-vehicle equipment in response to speech from the driver. Chatbots are thought to perform voice processing to recognize the driver's speech.

Japanese Patent Application Laid-Open No. 2005-62911

Recently, there has been a growing demand to combine DMS with voice processing technology, such as that used in chatbots. Image processing in DMS and voice processing in chatbots both require high-load computational processing. Therefore, high-precision computing devices are required to perform this image and voice processing, which results in high costs.

The present invention aims to provide a monitoring system that can accurately monitor pilots while reducing costs even when combining image processing technology and audio processing technology.

(1) The monitoring system of the present invention, which is provided to solve the above-mentioned problems, comprises an image acquisition unit that acquires images of the operator of a moving object, an audio reception unit that receives audio uttered by the operator, an audio output unit that outputs audio to the operator, and a control unit. The control unit comprises an image processing unit that can detect an abnormality occurring in the operator based on the images acquired by the image acquisition unit, and an audio processing unit that can realize audio dialogue processing for exchanging information with the operator via audio by executing processing that involves asking audio questions to the operator via the audio output unit and processing the audio received by the audio reception unit in response to the questions. The control unit is characterized in that, when an abnormality in the operator is detected, the image processing unit controls to restrict at least a portion of the image acquisition by the image acquisition unit and to allow the audio dialogue processing.

In the above-described monitoring system, the image processing unit controls the image acquisition unit to restrict at least a portion of image acquisition when an abnormality in the pilot is detected. That is, once an abnormality in the pilot is detected, the above-described monitoring system restricts image acquisition and switches to voice dialogue with the pilot. Therefore, the above-described monitoring system can appropriately realize voice dialogue with the pilot while reducing the image processing burden on the image acquisition unit and image processing unit. Furthermore, because the above-described monitoring system can reduce the image processing load on the image processing unit, a high-performance processing unit is not required even when both the image processing unit and the voice processing unit are used. This allows the above-described monitoring system to be simply configured using, for example, a single microcomputer (e.g., SoC: system-on-chip), which is expected to reduce costs. Furthermore, when an abnormality in the pilot occurs, the above-described monitoring system restricts image acquisition in the image acquisition unit and then allows voice dialogue processing, thereby preventing a loss of responsiveness in the voice dialogue processing unit.

(2) In the monitoring system of the present invention described above, the restriction on image acquisition by the image acquisition unit may be such that the image acquisition is stopped or the frequency of image acquisition is reduced.

By configuring the above-described monitoring system in this way, the load on image processing can be reduced. This increases the versatility of the processing unit that can be used in the above-described monitoring system. As a result, the above-described monitoring system does not require high computing performance from the control unit, which is expected to reduce the cost of the monitoring system. Here, the image acquisition frequency in the image acquisition unit is reduced by, for example, changing from 10 fps (initial acquisition frequency) to 1 fps (reduced acquisition frequency).

(3) In the monitoring system of the present invention described above, the image acquisition unit acquires images of at least the face of the operator of the moving object, and the image processing unit is preferably capable of detecting an abnormality occurring in the operator from a change in facial expression of the operator contained in the images acquired by the image acquisition unit.

By configuring the above-mentioned monitoring system in this way, it is possible to accurately identify abnormalities in the driver's facial expression based on changes in the driver's facial expression. As a result, when the above-mentioned monitoring system determines that the driver's facial expression has changed due to, for example, the driver dozing, yawning, or looking away, it can transition to voice dialogue processing and engage in a voice-based exchange (dialogue) with the driver. Therefore, the above-mentioned monitoring system can smoothly transition to voice dialogue when an abnormality occurs in the driver's facial expression.

(4) In the monitoring system of the present invention described above, it is preferable that the voice receiving unit releases some or all of the restrictions on image acquisition by the image acquiring unit on the condition that no speech is received from the pilot within a predetermined time.

By configuring the above-mentioned monitoring system in this way, it is possible to resume image acquisition if it is estimated that a serious abnormality has occurred in the pilot. This allows the above-mentioned monitoring system to reliably acquire the pilot's condition while reducing the image processing burden on the image processing unit. Here, examples of when the voice receiving unit does not receive a specified utterance from the pilot within a specified time include when the pilot falls asleep or becomes ill. The above-mentioned specified time can be set to various times depending on the speed of the moving object, etc., but it is recommended to set it to 1 to 3 seconds, for example.

(5) In the monitoring system of the present invention described above, the frequency of image acquisition by the image acquisition unit may be varied based on the abnormal state of the operator detected by the image processing unit.

By configuring the above-mentioned monitoring system in this way, it is possible to acquire and process images appropriately based on the operator's abnormal state. As a result, the above-mentioned monitoring system can reliably detect the operator's abnormal state while reducing the load on the control unit. As a result, the above-mentioned monitoring system does not require the control unit to have high computing performance, which is expected to reduce the cost of the monitoring system.

(6) In the monitoring system of the present invention described above, the audio processing unit may output an instruction to the audio output unit to issue an alarm if the abnormality of the pilot detected by the image processing unit is due to a change in the pilot's posture exceeding a predetermined range, or due to the pilot dozing for a predetermined period of time or more.

By configuring the above-described monitoring system in this way, it can detect when a serious abnormality occurs in the operator and issue an alarm from the audio output unit. This allows the above-described monitoring system to issue an alarm appropriately when a serious abnormality occurs in the operator. Furthermore, the above-described monitoring system can prevent unnecessary alarms from being issued, thereby preventing the issuance of alarms from becoming cumbersome.

(7) In the monitoring system of the present invention described above, the voice processing unit may have a voice determination unit that determines whether the voice received by the voice receiving unit contains a predetermined keyword, and the control unit may control changes to the voice output content of the voice output unit in accordance with the determination result of the voice determination unit.

By configuring the monitoring system described above, it is possible to output appropriate audio in response to the operator's speech. Any appropriate keyword can be used as the predetermined keyword, such as "take a break," "yes," "OK," or "understood."

(8) The monitoring system of the present invention described above may include a speed detection unit that detects the speed of the moving object, and a speed processing unit that is provided in the control unit and performs processing according to the speed detected by the speed detection unit, and the control unit may control at least one of restricting image acquisition by the image acquisition unit and changing the content of audio output by the audio output unit according to the speed processing by the speed processing unit.

By configuring the monitoring system described above, it is possible to perform appropriate image processing and audio processing according to the speed of the moving object. For example, when the speed of the moving object is high, the impact on the operator of any abnormalities that may occur can be significant, so it is advisable to reduce the restrictions on image acquisition by the image acquisition unit and to make the audio output content from the audio output unit more appealing to the operator (e.g., an alarm). On the other hand, when the speed of the moving object is slow, it is advisable to increase the restrictions on image acquisition by the image acquisition unit and to make the audio output content from the audio output unit more gentle.

The present invention can provide a monitoring system that can accurately monitor pilots, while reducing costs even when combining image processing technology and audio processing technology.

1 is a configuration diagram of a monitoring system according to an embodiment of the present invention. 1A is an explanatory diagram showing the abnormality judgment criteria (abnormality judgment A) for a pilot in a monitoring system according to one embodiment of the present invention, and FIG. 1B is an explanatory diagram showing the abnormality judgment criteria (abnormality judgment B) for a pilot in a monitoring system according to a modified example of the present invention. 1 is an explanatory diagram showing an example of the content of audio output in the monitoring system of the present invention. FIG. FIG. 1 is a flow diagram of a monitoring system of the present invention. FIG. 10 is a flow diagram of audio processing (subroutine) in the monitoring system of the present invention. FIG. 10 is a flow diagram of a monitoring system according to a modified example of the present invention.

A monitoring system 1 according to one embodiment of the present invention will be described below with reference to Figures 1 and 2. In this embodiment, the mobile object is an automobile (vehicle) as an example. In addition, this embodiment will be described on the assumption that a pilot (driver) is sitting in the driver's seat of the vehicle and driving the vehicle.

As shown in FIG. 1, the monitoring system 1 includes an image acquisition unit 10, an audio reception unit 20, and a speed detection unit 30. In addition to the above, the monitoring system 1 also includes an audio output unit 40, a control unit 50, etc.

The image acquisition unit 10 is composed of a capture device such as a drive recorder or camera (including a video camera). The image acquisition unit 10 is installed, for example, inside the vehicle cabin and can capture images of at least the driver's face. The image acquisition unit 10 preferably uses a camera capable of infrared photography so that it can capture images of the driver even at night or in dark surroundings. The images acquired by the image acquisition unit 10 are processed in the image processing unit 51, which will be described later.

The voice reception unit 20 is composed of, for example, a microphone, and is configured to receive voice uttered by the driver. The voice reception unit 20 is also disposed, for example, near the steering wheel of the vehicle so that it can detect voice uttered by the driver. The voice received by the voice reception unit 20 is processed by the voice processing unit 52, which will be described later.

The speed detection unit 30 is composed of, for example, a GPS sensor. The speed detection unit 30 detects a GPS signal (speed signal) input to the GPS sensor. The speed detected by the speed detection unit 30 is processed in the speed processing unit 55, which will be described later. The GPS sensor may be one installed in the vehicle's navigation system. Note that the speed detection unit 30 may also detect speed using a speed detection sensor installed in the vehicle instead of GPS.

The audio output unit 40 is configured, for example, as a speaker, and is located inside the vehicle. As will be described in more detail below, the audio output unit 40 outputs audio (such as speech or an alarm) related to a predetermined question to the driver depending on the result of the determination of the driver's abnormal state.

The control unit 50 is composed of, for example, a single microcomputer (e.g., SoC: System on Chip). As will be described in more detail below, in this embodiment, the control unit 50 uses a processing device with a minimum processing capacity for image processing and audio processing. The control unit 50 includes an image processing unit 51, an audio processing unit 52, a speed processing unit 55, etc. The control unit 50 controls the image processing unit 51, the audio processing unit 52, and the speed processing unit 55, and can also control the entire monitoring system 1.

The image processing unit 51 processes the images acquired by the image acquisition unit 10 and can detect changes in the driver's facial expression in the processed images. The image processing unit 51 can also detect abnormalities occurring in the driver from changes in the driver's facial expression. In this embodiment, as shown in Figure 2(a), an abnormality determination for the driver is made based on abnormality determination A. Abnormality determination A determines changes in facial expression due to, for example, dozing, yawning, or looking away, as well as changes in driving condition that are not manifested by facial expression, such as poor posture. Abnormality determination A is described in detail below.

"Drowsiness" is detected by detecting whether the driver's eyes are open or closed. More specifically, the image processing unit 51 determines that the driver is "drowsy" if the driver's eyes are closed for, say, two seconds. "Yawning" is detected by detecting whether the driver's mouth is open or closed. More specifically, the image processing unit 51 determines that the driver is "yawning" if the driver's mouth is open for, say, three seconds. "Inattentiveness" is detected by using AI to detect feature points (e.g., six points) on the driver's face and calculate the angle of the driver's face (left and right). More specifically, the image processing unit 51 determines that the driver is "looking away" if the driver's face is turned left or right for, say, two seconds. "Poor posture" is detected by using AI to detect feature points (e.g., six points) on the driver's face and calculate the angle of the driver's face (up and down, roll). More specifically, the image processing unit 51 determines that the driver has "poor posture" when the driver's facial angle is facing up or down.

As shown in FIG. 1, if the image processing unit 51 determines that an abnormality has occurred with the driver, it controls the image acquisition unit 10 to restrict image acquisition. Here, the restriction on image acquisition by the image acquisition unit 10 can be, for example, to stop image acquisition or to reduce the image acquisition frequency. Furthermore, the reduction in the image acquisition frequency by the image acquisition unit 10 is achieved, for example, by changing from 10 fps (initial acquisition frequency) to 1 fps (reduced acquisition frequency).

The voice processing unit 52 uses AI to analyze the voice uttered by the driver received by the voice reception unit 20, and performs voice dialogue processing based on the driver's abnormality determination made by the image processing unit 51. Specifically, the voice processing unit 52 executes processing that involves asking voice questions to the driver via the voice output unit 40 and processing the voice received by the voice reception unit 20 in response to the questions. In this way, the voice processing unit 52 performs voice dialogue processing (for example, a function known as a chatbot) that exchanges information with the driver via voice.

Here, voice dialogue processing is permitted on the condition that the image processing unit 51 detects an abnormality in the driver. That is, in the monitoring system 1 of the present invention, when an abnormality in the driver is detected, image acquisition by the image acquisition unit 10 is restricted, and voice dialogue processing by the voice processing unit 52 is initiated. In other words, it is possible to prevent image processing by the image processing unit 51, which has a high load, and voice processing by the voice processing unit 52, which has a high load, from operating in parallel. Therefore, the monitoring system 1 of the present invention can reduce the load on the control unit 50.

The voice determination unit 53 determines whether the voice received by the voice reception unit 20 contains a predetermined keyword. Any appropriate keyword can be used as the predetermined keyword, such as "take a break," "yes," "OK," or "I understand." The voice determination unit 53 (control unit 50) can control changes to the voice output content of the voice output unit 40 based on the determination result of the voice determination unit 53. Details of the voice output content will be explained in the operation flow of the monitoring system 1, which will be described later.

The speed processing unit 55 processes the vehicle speed detected by the speed detection unit 30. Specifically, the speed processing unit 55 classifies the vehicle speed detected by the speed detection unit 30 into an appropriate range (e.g., 50-59 km/h), and can control the image processing unit 51 and the audio processing unit 52 to change the judgment criteria (e.g., the time required to determine whether the driver is dozing) according to the classified speed range.

In this way, in this embodiment, the speed processing unit 55 (control unit 50) can control at least one of the following depending on the speed: limiting image acquisition in the image processing unit 51, and changing the audio output content in the audio output unit 40.

The monitoring system 1 described above can therefore perform appropriate image processing and audio processing according to the vehicle speed. For example, when the vehicle speed is high, the impact on the driver of any abnormalities that may occur may be significant, so it may be advisable to reduce the restrictions on image acquisition by the image acquisition unit 10 and to make the audio output by the audio output unit 40 more appealing to the driver (e.g., an alarm). On the other hand, when the vehicle speed is low, it may be advisable to increase the restrictions on image acquisition by the image acquisition unit 10 and to make the audio output by the audio output unit 40 more gentle.

The above is the configuration of the monitoring system 1 of the present invention. Next, the operational flow of the monitoring system 1 of the present invention will be explained below with reference to Figures 4 and 5.

As shown in Figure 4, when processing by the monitoring system 1 begins, the image acquisition unit 10 first acquires an image that includes at least the driver's face (step S10).

When an image of the driver is acquired by the image acquisition unit 10, the image is processed by the image processing unit 51. As the image processing unit 51 processes the image, an abnormality in the driver (pilot) is determined based on abnormality determination A (see Figure 2(a)) (step S11).

If it is determined in step S11 that there is nothing abnormal with the driver, no audio output is produced and processing returns to step S10 (step S12).

If it is determined in step S11 that the driver is experiencing an abnormality (for example, dozing off at the wheel), image acquisition by the image acquisition unit 10 is stopped (step S13). When the processing in step S13 is completed, the process proceeds to the audio processing subroutine (step S20).

As shown in Figure 5, when the audio processing in step S20 starts, audio is output in accordance with audio output A (see Figure 3) (step S21). For example, if it is determined that the driver has fallen asleep, audio such as "Thank you for driving. Are you sleepy now?" is output.

Next, the voice reception unit 20 determines whether the driver has spoken (step S22). If the voice reception unit 20 does not receive any speech from the driver within a predetermined time, the restriction on image acquisition by the image acquisition unit 10 (in this embodiment, a stopped state) is lifted. In other words, image acquisition by the image acquisition unit 10 is resumed (step S23). Examples of cases where the voice reception unit 20 does not receive any predetermined speech from the driver within a predetermined time include when the driver falls asleep or when the operator becomes ill. The predetermined time can be set to various times depending on the vehicle speed, etc., but is preferably set to 1 to 3 seconds, for example.

When image acquisition resumes in step S23, audio is output in accordance with audio output D (step S24). In step S24, for example, an audio message saying "wake up" or an alarm such as a buzzer is output.

When step S24 is completed, the subroutine processing in step S20 returns to the main routine (see Figure 4). When the main routine processing is returned to, processing returns to step S10.

As shown in Figure 5, if it is determined in step S22 that the driver has spoken within the predetermined time, a voice is output in accordance with voice output B (see Figure 3) (step S25). For example, if it is determined that the driver has fallen asleep, a voice is output saying, "We have detected that you are falling asleep, so we are calling out to you. Why don't you take a break and have a cup of hot coffee?"

Next, the voice receiving unit 20 begins receiving voice, and a determination is made as to whether the driver's utterance (voice) matches a predetermined keyword (step S26). In step S26, a determination is made as to whether the driver's utterance matches a keyword (e.g., "take a break," "yes," "OK," "I understand," etc.) that corresponds to the content of the question from the voice processing unit 52. In this way, the monitoring system 1 described above determines whether the driver's utterance matches a keyword, and can therefore output appropriate voice in accordance with the driver's utterance.

If the driver's utterance does not match a predetermined keyword in step S26, audio output is performed in accordance with audio output E (see Figure 3) (step S27). In step S26, for example, audio output such as "Please continue to drive safely from now on" is performed. When the processing of step S27 ends, the subroutine processing in step S20 ends and the process returns to the main routine (see Figure 4). When the processing of the main routine returns, the process returns to step S10.

Furthermore, in step S26, if the driver's utterance matches a predetermined keyword, audio output is performed in accordance with audio output C (see Figure 3) (step S28). In step S28, for example, audio output such as "Please take a good rest" is performed. When the processing of step S28 ends, the subroutine processing in step S20 ends and the process returns to the main routine (see Figure 4). When the processing of the main routine returns, the process returns to step S10.

In this embodiment, after the processing in the main routine is completed, processing returns to step S10. However, repeated processing from step S10 can be performed as needed, and the processing of the monitoring system 1 can also be terminated after the processing in the main routine is completed. The above is the operational flow of the monitoring system 1 according to one embodiment of the present invention. Next, the effects of the monitoring system 1 of the present invention will be described below.

As described above, in the monitoring system 1 of the present invention, the image processing unit 51 performs control to restrict at least a portion of the image acquisition by the image acquisition unit 10 on the condition that an abnormality in the operator (driver) is detected. In other words, once an abnormality in the operator is detected, the monitoring system 1 described above restricts image acquisition and switches to voice dialogue with the operator. Therefore, the monitoring system 1 described above can appropriately realize voice dialogue with the operator while reducing the image processing burden in the image acquisition unit 10 and image processing unit 51.

Furthermore, the monitoring system 1 described above can reduce the load caused by image processing in the image processing unit 51, so even if the image processing unit 51 and the audio processing unit 52 coexist, a high-performance arithmetic processing unit is not required. As a result, the monitoring system 1 described above can be simply configured using, for example, a single microcomputer (e.g., SoC: system-on-chip), which is expected to reduce the cost of the monitoring system 1. As described above, the monitoring system 1 of the present invention does not require the control unit to have high computing performance, which increases the versatility of the arithmetic processing unit that can be used in the monitoring system 1. Furthermore, when an abnormality occurs with the operator, the monitoring system 1 described above restricts image acquisition by the image acquisition unit 10 and then allows audio dialogue processing, so the responsiveness of audio dialogue processing in the audio processing unit 52 is not impaired.

Furthermore, when the monitoring system 1 described above determines based on the images that something is wrong with the driver, it transitions to voice dialogue processing and engages in dialogue with the driver. This allows the monitoring system 1 to restore the driver's awareness and concentration on driving when there is a possibility that the driver is unable to concentrate on driving (piloting) due to drowsiness or looking away. This makes the monitoring system 1 highly effective in preventing dangers caused by drowsiness or looking away in advance.

Furthermore, the above-mentioned monitoring system 1 is configured to lift some or all of the restrictions on image acquisition by the image acquisition unit 10 on the condition that the voice acceptance unit 20 does not accept any speech from the pilot within a specified period of time.

The above-described monitoring system 1 can therefore resume image acquisition if it is estimated that a serious abnormality has occurred in the operator. This allows the above-described monitoring system 1 to reliably acquire the operator's condition while reducing the image processing burden on the image processing unit 51.

In addition, in the monitoring system 1 described above, the audio processing unit 52 outputs an instruction to the audio output unit 40 to issue an alarm if the abnormality in the pilot detected by the image processing unit 51 is due to the pilot's posture changing by more than a predetermined range, or due to the pilot dozing off for more than a predetermined period of time.

The above-described monitoring system 1 can therefore detect when a serious abnormality occurs in the operator and issue an alarm from the audio output unit 40. This allows the above-described monitoring system 1 to issue an alarm appropriately when a serious abnormality occurs in the operator. Furthermore, the above-described monitoring system 1 can prevent unnecessary alarms from being issued, thereby preventing the issuance of alarms from becoming cumbersome.

Next, the operational flow of the monitoring system 1 according to a modified example of the present invention will be described below. Explanations of content similar to those in the above-described embodiment will be omitted.

<<Variations>>
6, when the operation of the monitoring system 1 according to the modified example is started, the image acquisition unit 10 acquires an image of the driver (step S100). The image acquisition frequency at this time is set to, for example, 10 fps.

Next, an abnormality determination of the driver is performed in accordance with abnormality determination A (see Figure 2(a)) (step S101). If it is determined in step S101 that there is no abnormality with the driver, the process returns to step S100 without audio output (step S102).

If it is determined in step S101 that the driver is abnormal (for example, dozing off), the image acquisition frequency in the image acquisition unit 10 is reduced (step S103). The image acquisition frequency at this time is set to, for example, 1 fps. In this way, in the monitoring system 1 according to the modified example, the image acquisition frequency in the image acquisition unit 10 is changed based on the abnormal state of the driver detected by the image processing unit 51.

When the processing in step S103 is completed, the process moves to parallel processing. In parallel processing, processing from step S104 is performed in parallel with the audio processing (step S20) in the above-mentioned embodiment. The audio processing (step S20) is the same as in the above-mentioned embodiment, so a description thereof will be omitted.

In step S104, image acquisition by the image acquisition unit 10 is performed at, for example, 1 fps.

Next, the driver's condition is assessed in accordance with abnormality assessment B (step S105). As shown in Figure 2(b), abnormality assessment B assesses whether the driver is experiencing an abnormality related to poor posture. First, prior to abnormality assessment B, the image processing unit 51 acquires the outline of the driver's face using AI, calculates the coordinates of the center position of the face from the outline of the face, and tracks the coordinates of the center position. Next, if the coordinates of the center position of the driver's face fall outside a predetermined range during tracking, it is determined that the driver has experienced poor posture.

As shown in Figure 6, if it is determined in step S105 that the driver has not lost their posture, processing returns to step S100.

If it is determined in step S105 that the driver has lost their posture, the parallel audio processing (step S20) is stopped (step S106). When the audio processing (step S20) is stopped in step S106, audio is output in accordance with audio output D (see Figure 3). That is, in step S106, if it is estimated that a serious abnormality has occurred that has caused the driver to lose their posture, audio output D corresponding to an alarm is output.

The above is the operational flow of the monitoring system 1 according to the modified example of the present invention. Next, the effects of the monitoring system 1 according to the modified example will be explained below.

As described above, in the monitoring system 1 according to the modified example, the frequency of image acquisition by the image acquisition unit 10 varies based on the abnormal state of the operator (driver) detected by the image processing unit 51.

The above-described monitoring system 1 is therefore able to acquire and process images appropriately based on the operator's abnormal state. As a result, the above-described monitoring system 1 can reliably detect the operator's abnormal state while reducing the load on the control unit 50. As a result, the above-described monitoring system 1 does not require the control unit 50 to have high computing performance, which is expected to reduce the cost of the monitoring system 1.

The above is the configuration and effects of the monitoring system 1 according to the embodiment and modifications of the present invention, but the monitoring system 1 of the present invention is not limited to the above-described embodiment and modifications, and various modifications are possible.

In this embodiment, the control unit 50 is equipped with an image processing unit 51, an audio processing unit 52, and a speed processing unit 55, but these processing units may be arranged separately. Furthermore, in this embodiment, the control unit 50 is equipped with a speed detection unit 30 and a speed processing unit 55, but the speed detection unit 30 and the speed processing unit 55 may be provided as needed, and the configuration may also omit the speed detection unit 30. Furthermore, the speed detection unit 30 may use various speed detection means, such as a GPS sensor or a speed sensor mounted on the vehicle.

In addition, while the present embodiment illustrates a case in which the image processing unit 51 and the audio processing unit 52 are formed as a single microcomputer (SoC), this is not limiting. For example, the image processing unit 51 and the audio processing unit 52 may each be provided independently. Furthermore, the image processing unit 51 and the audio processing unit 52 can utilize processing devices with various performance capabilities (including high-performance ones) taking into account the load imposed. The performance required for the image processing unit 51, the audio processing unit 52, etc. can be determined taking cost-effectiveness into consideration. The image acquisition unit 10 can utilize various drive recorders, cameras, video cameras, etc. capable of acquiring the driver's status. Furthermore, the image acquisition unit 10 can acquire various images capable of acquiring the driver's status instead of, or in addition to, facial images. For example, the image acquisition unit 10 can detect the driver's body tilt. Furthermore, the audio reception unit 20 can utilize various means, such as a microphone, capable of acquiring audio. Furthermore, the audio processing unit 52 can utilize a chatbot.

In addition, in this embodiment, the image processing unit 51 performs control to restrict at least a portion of the image acquisition by the image acquisition unit 10 when an abnormality in the pilot is detected, but this is not limited to this. Restrictions on the image acquisition unit 10 and the image processing unit 51 can be imposed using various means that can reduce the processing load on the image acquisition unit 10 and the image processing unit 51. For example, image processing in the image processing unit 51 may be restricted. In addition, while this modified example illustrates an example in which the image acquisition frequency in the image acquisition unit 10 is restricted from 10 fps to 1 fps, the image acquisition frequency can be changed as appropriate depending on the performance of the image acquisition unit 10 to be installed.

Furthermore, the conditions for determining whether a driver is abnormal, as detected in this embodiment, are not limited to the embodiment or modified example, and various conditions can be set. Furthermore, the content of the audio output from the audio output unit 40 is not limited to the embodiment or modified example, and various audio contents can be set.

In this embodiment and its variants, an alarm is issued if the pilot's abnormality is due to the pilot's "poor posture" or the pilot falling asleep for a predetermined period of time or longer, but this is not limited to this and an alarm can be issued based on various criteria.

In addition, in this embodiment and its modified examples, the voice determination unit 53 determines whether or not the voice contains a predetermined keyword, but various keywords can be used. Furthermore, the voice determination unit 53 can set various determination conditions that enable voice dialogue with the pilot, in addition to using keywords. For example, the voice determination unit 53 may use AI to determine the voice (utterance) of the pilot.

The foregoing are various embodiments and modifications of the monitoring system 1 according to the present invention, but the present invention is not limited to the above-described embodiments and modifications. Those skilled in the art will readily understand that other embodiments are possible within the scope of the claims and the teachings and spirit of the present invention.

The monitoring system of the present invention can be used to monitor operators (drivers) of various types of mobile vehicles, such as automobiles, airplanes, and trains.

1: Monitoring system 10: Image acquisition unit 20: Audio reception unit 30: Speed detection unit 40: Audio output unit 50: Control unit 51: Image processing unit 52: Audio processing unit 53: Audio determination unit 55: Speed processing unit

Claims (4)

an image acquisition unit that acquires an image related to a pilot of a moving object;
a voice receiving unit that receives voice uttered by the pilot;
a voice output unit that outputs voice to the operator;
A control unit;
Equipped with
The control unit
an image processing unit capable of detecting an abnormality occurring in the operator based on the image acquired by the image acquisition unit;
a voice processing unit capable of realizing voice dialogue processing for exchanging information with the operator via voice by executing a process involving issuing a voice question to the operator via the voice output unit and processing the voice received by the voice receiving unit in response to the question;
In addition to providing
A monitoring system characterized in that, when an abnormality in the pilot is detected, the image processing unit performs control to restrict at least a portion of the image acquisition by the image acquisition unit and control to allow the voice dialogue processing. The monitoring system of claim 1, wherein the restriction on image acquisition by the image acquisition unit is to stop image acquisition or reduce the frequency of image acquisition. The monitoring system described in claim 1 or 2, characterized in that the restrictions on image acquisition by the image acquisition unit are partially or completely lifted on the condition that the voice acceptance unit does not accept any speech from the operator within a predetermined time. The monitoring system described in claim 1 or 2, characterized in that the image acquisition frequency of the image acquisition unit is changed based on the abnormal state of the operator detected by the image processing unit.