JP7828184B2 - Monitoring system and monitoring method - Google Patents

Description

Embodiments of the present invention relate to a monitoring system and a monitoring method.

Demand is growing for systems that use camera footage to monitor people's behavior. Systems using such cameras have the advantage of being able to record accidents and crimes that occur within the camera's range, allowing for post-event review. However, in spaces with many people, such as on trains, it can be difficult to assess the situation from images alone. This can make it difficult to grasp unusual situations, such as accidents or crimes, in real time.

Patent No. 6487371

The problem that this invention aims to solve is to provide a monitoring system and monitoring method that can more easily grasp situations in which an abnormal condition has occurred.

The monitoring system of this embodiment includes a camera, a measuring device, and a monitoring device. The camera captures images of the space being monitored. The measuring device measures the state of a person in the image. Based on the measurements, the monitoring device generates notification information when the person in the image is in a predetermined state.

FIG. 1 is a diagram showing a schematic configuration of a monitoring system. FIG. 1 is a block diagram showing an example of the configuration of a monitoring system. FIG. 1 is a top view showing an example of the arrangement of multiple cameras and multiple transmitters and receivers in a vehicle. FIG. 4 is a diagram showing an example of a screen displayed on a display device by the monitoring device. 10A and 10B are diagrams showing an example of an irradiation range of a transmitter/receiver in a measurement device and an example of a specified area. 10 is a flowchart showing an example of processing by the imaging device. 10 is a flowchart showing an example of processing by the measurement device. 10 is a flowchart showing an example of processing by a monitoring device. FIG. 10 is a top view showing another example of the arrangement of multiple cameras and multiple transmitter/receivers in a vehicle. FIG. 10 is a block diagram showing an example of the configuration of a monitoring system according to a second embodiment. 10 is a flowchart showing an example of processing by the scream detection device. FIG. 10 is a block diagram showing the configuration of a monitoring system according to a third embodiment. FIG. 11 is a top view showing an example of an arrangement inside a vehicle according to a third embodiment. FIG. 2 is a diagram showing an example of an image captured by a camera. FIG. 11 is a diagram showing an example of a screen displayed on a display device by a monitoring device according to a third embodiment. 10 is a flowchart showing the processing of the scream detection device according to the third embodiment. 11 is a flowchart showing an example of processing by a monitoring device according to the third embodiment. FIG. 10 is a diagram showing a plurality of arrangement patterns. FIG. 10 is a diagram showing an example in which a sound collecting unit is further disposed at an intermediate position in the traveling direction of the vehicle. 1A and 1B are diagrams showing examples of vehicle layouts in which seats face the direction of travel or the opposite direction to the direction of travel. FIG. 10 is a diagram showing an example in which a camera and a sound collecting unit are arranged in the center of a vehicle in the longitudinal direction. FIG. 10 is a diagram showing an example of the arrangement of cameras, transmitter/receivers, and voice collectors in an elevator hall.

A monitoring system and a monitoring method according to an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the embodiment described below is merely an example of an embodiment of the present invention, and the present invention should not be construed as being limited to these embodiments. Furthermore, in the drawings referred to in this embodiment, identical parts or parts having similar functions are given the same or similar symbols, and repeated explanations may be omitted. Furthermore, for the sake of convenience, the dimensional proportions of the drawings may differ from the actual proportions, and some components may be omitted from the drawings.

(First embodiment)
FIG. 1 is a diagram showing the schematic configuration of a monitoring system 1 according to this embodiment. As shown in FIG. 1, the monitoring system 1 is a system capable of acquiring information on the status of a person in a monitored space, and includes a camera 11, a first measuring device 12, and a monitoring device 13. The monitoring system 1 is installed, for example, inside a passenger car such as a train or bus, or in an elevator hall. That is, the monitoring system 1 is installed, for example, on the ceiling of a railcar, a bus, or an elevator hall, and is a device capable of acquiring status information from above downward. The camera 11, the first measuring device 12, and the monitoring device 13 may be installed on the ceiling using separate fixtures. Alternatively, they may be fixed together using a pair of fixtures and installed on the ceiling. The structure of the fixture is arbitrary.

Here, an example configuration of the monitoring system 1 will be described using Figures 2 and 3. Figure 2 is a block diagram showing an example configuration of the monitoring system 1. As shown in Figure 2, the image capture device 11 is configured to have, for example, a CPU. This image capture device 11 is a device that can capture images of people and estimate their ages, and has a person information storage unit 110, multiple cameras 111a, b, a person detection unit 112, an age estimation unit 113, and a communication unit 134. Details of the image capture device 11 will be provided later.

The first measuring device 12 is configured to include, for example, a CPU. The first measuring device 12 is a device that can measure the heart rate and respiratory rate in a non-contact manner as status information of a person in an image captured by the imaging device 11, and has multiple transceivers 121, an amplifier 122, and a processing unit 123.

The monitoring device 13 is configured to have, for example, a CPU. The monitoring device 13 is a device capable of determining the state of a person, and has a memory unit 131, a determination unit 132, a firing unit 133, and a communication unit 134. Note that the monitoring system 1 according to this embodiment has multiple cameras 1111a, b, c and multiple transmitter/receivers 121, but is not limited to this. For example, the monitoring system 1 may be configured to have a single camera 111 and a single transmitter/receiver 121. Note that details of the first measuring device 12 and the monitoring device 13 will be described later.

Also shown in Figure 2 is a display device 200. The display device 200 is placed, for example, in the driver's cab of a train or in a central monitoring room. This display device 200 communicates with the monitoring device 13 via wired or wireless communication, and includes, for example, a monitor 201 and a speaker 202. The display device 200 also displays an image on the monitor 201 in response to an image signal transmitted from the monitoring device 13, and produces sound from the speaker 202 in response to the transmitted audio signal.

The person information storage unit 110 of the image capture device 11 is realized, for example, by RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc. The person information storage unit 110 stores, for example, information on normal pulse rate ranges by age or abnormal pulse rate ranges. For example, the abnormal pulse rate ranges are divided into a low pulse rate range and a high pulse rate range.

Figure 3 is a top view showing an example of the arrangement of multiple cameras 111a, b and multiple transmitter/receivers 121 inside vehicle 200. Line L200 indicates the middle of the opposing opening and closing doors. That is, vehicle 200 has, for example, eight opening and closing doors. Line L202 indicates the center line of vehicle 200 in the direction of travel, i.e., the center line of vehicle 200 in the longitudinal direction. Seats 202 are arranged inside vehicle 200. Multiple cameras 111a, b are cameras capable of capturing video.

As shown in FIG. 3, cameras 111a and 111b of the imaging device 11 are positioned on the short sides of vehicle 200. That is, cameras 111a and 111b are positioned at the end of vehicle 200 where it connects to other vehicles. As a result, camera 111a primarily captures images of the interior of vehicle 200 from the left end to the right, while camera 111b primarily captures images of the interior of vehicle 200 from the right end to the left. By positioning cameras 111a and 111b at both ends of vehicle 200 where it connects to other vehicles, it is possible to capture images of passengers from both the left and right sides, and capture images of at least half of the passengers' faces. As can be seen, by positioning cameras 111a and 111b at the end of vehicle 200 where it connects to other vehicles, it is possible to capture a wide depth in the direction of travel, thereby reducing the number of cameras 111a and 111b installed. Furthermore, in imaging device 11, the coordinates of the images captured by cameras 111a and 111b are pre-assigned to coordinates within vehicle 200, such as planar coordinates. Additionally, images captured by multiple cameras 111a and 111b are associated with the camera that captured the image and the time it was captured, and stored in the person information storage unit 114.

Furthermore, as shown in FIG. 3, each of the multiple transmitter/receivers 121 is positioned at the intersection of line L200 and line L202. This allows the multiple transmitter/receivers 121 to measure almost all passengers entering and exiting through the opening and closing doors, from above and diagonally downward. In other words, the multiple transmitter/receivers 121 are positioned in positions where they can measure passengers entering and exiting through the opening and closing doors.

Figure 4 shows an example screen displayed on the display device 200 by the monitoring device 13. Screen M10 shows the screen of the monitor 201 (see Figure 2), screen M12 within screen M10 is a top view of the vehicle 200, and marks mh10 and mh20 indicate the positions of people h10 and h20 in which some abnormality has been detected. Screen M14 within screen M10 alternately displays images captured by cameras 111a and 111b. Areas ag10 and ag20 indicate the areas of people h10 and h20 detected by the person detection unit 112, which will be described later. Furthermore, areas f10 and f20 indicate the face areas within areas ag10 and ag20 detected by the person detection unit 112, which will be described later.

As shown again in Figure 2, the person detection unit 112 of the image capture device 11 detects person areas ag10, ag20 and facial areas f10, f20 within these person areas ag10, ag20 from each image stored in the person information storage unit 114 (see M14 in Figure 4). A general algorithm can be used to detect these person areas ag10, ag20 and facial areas f10, f20. For example, the contours of moving objects are extracted from time-series difference images, and the person areas are extracted using labeling processing and color separation processing based on the extracted contours. Furthermore, the facial areas f10, f20 are extracted, for example, by recognizing the color and shape within the person areas ag10, ag20.

The person detection unit 112 also associates the coordinates of the detected person's areas ag10, ag20 and facial areas f10, f20 with planar coordinates within the vehicle 200. In this case, the image areas captured by the multiple cameras 111a may overlap with the image areas captured by the camera 111b. Therefore, the detected person's areas ag10, ag20 and the facial areas f10, f20 may also overlap. The person detection unit 112 then associates the coordinates of the detected person's areas ag10, ag20 and the facial areas f10, f20, and the planar coordinates within the vehicle 200 with the captured image, and stores them in the person information storage unit 110. In other words, the position information generated by the person detection unit 112 includes, for example, the coordinates of the circumscribing rectangle of the detected person's area, the position coordinates of the center of gravity of the circumscribing rectangle, and the position coordinates of the circumscribing rectangle and center of gravity of the detected facial area.

The age estimation unit 113 of the imaging device 11 performs age estimation processing on the image of the face region detected by the person detection unit 112. A general algorithm can be used for this age estimation processing. For example, the age estimation unit 113 has an image processing unit that generates multiple feature amounts related to wrinkles and the like for the image of the face region, and a neural network unit that uses these multiple feature amounts as input and outputs age. The age estimation unit 113 then associates the age estimated by the neural network unit with the captured image stored in the person information storage unit 114 and stores it. The age estimation unit 113 also supplies the position information, estimated age, and image capture time information of the detected person regions ag10 and ag20 to the storage unit 131 of the monitoring device 13.

The age estimation unit 113 according to this embodiment uses a so-called neural network for recognition, but is not limited to this. For example, the age estimation unit 113 may compare an image of the identified face region with an image of the face region stored in the person information storage unit 114, and estimate the age of the identified person image. This age estimation method, for example, compares the similarity of the images, and determines the age of the image with the closest similarity as the age of the identified person.

Note that some or all of the person detection unit 112 and age estimation unit 113 may be realized by hardware, such as a circuit board, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). In other words, the person detection unit 112 and the age estimation unit 113 may be configured as electronic circuits.

Figure 5 is a diagram showing an example of the irradiation range As10 of the transmitter/receiver 121 in the first measuring device 12 and the identified area As12. Area As12 is, for example, the chest area of person h10 identified by the processing unit 123.

Referring to Figure 5, and again as shown in Figure 2, the transmitter/receiver unit 121 of the first measuring device 12 is capable of measuring the distance to an object and the position of the object using a pair of two transmitter/receivers. For example, the processing unit 123 measures the position of, for example, person h10 using the principle of triangulation using the two distances to the object from the two transmitter/receivers and the distance between the two transmitters/receivers.

The transmitter/receiver unit 121 generates a modulated wave (transmission wave) and transmits it into the irradiation range As10. For example, this transmission wave is a millimeter wave, and the FMCW (frequency continuous modulation) method is used. The transmitter/receiver unit 121 then receives a reflected wave reflected from, for example, a person h10. The transmitter/receiver unit 121 mixes the transmission wave and the reception wave to generate an IF (intermediate frequency) signal and supplies it to the amplifier 122. The amplifier 122 amplifies the IF (intermediate frequency) signal, converts it to a digital signal in the A/D converter, and supplies it to the processing unit 123. Furthermore, the transmitter/receiver unit 121 sweeps and irradiates the modulated frequency of the irradiated radio waves, allowing it to irradiate radio waves with different phases to various targets, thereby making it possible to distinguish between reflections. In this case, the two transmitter/receivers of the transmitter/receiver unit 121 synchronize the modulation of the irradiated radio waves.

The processing unit 123 performs general signal processing based on the digital signal to generate the distance for each object. The processing unit 123 then generates the position of the object using the distance for each object from the two transmitter/receivers.

The processing unit 123 also generates a time series of distance fluctuations to the measurement target based on the digital signal as the amplitude of the measurement target. The processing unit 123 then calculates the time series fluctuations in the amplitude of the measurement target as a frequency. Note that the amplitude generation process can also use general processing (see, for example, Patent Document 1).

In the processing of the processing unit 123, the amplitude is generally 0 for stationary objects such as machines when there is no change in the distance between devices. On the other hand, in the case of the human body, the distance between devices fluctuates due to contractions of blood vessels and the heart (heart rate) and changes in the skin due to breathing, so the calculated amplitude fluctuates over time. The processing unit 123 then calculates the frequency corresponding to the heart rate/respiratory rate based on the periodic information of the amplitude that fluctuates over time. Furthermore, for example, the processing unit 123 can use so-called Doppler FFT to measure the frequency and calculate frequency information corresponding to the heart rate/respiratory rate based on the periodicity of the change in the speed of the object.

If the generated frequency is within the range of the heart rate/respiratory rate, the processing unit 123 determines that the measurement target is a person. Through this processing, the chest region of the identified person h10 is As12 (see Figure 5). The processing unit 123 supplies the monitoring device 13 with position information for the region As12 of the identified person h10, information on the frequency corresponding to the heart rate/respiratory rate, and the measurement time. The position information generated by the processing unit 123 includes, for example, the coordinates of the circumscribing rectangle of the detected person's chest region and the position coordinates of the center of gravity of the circumscribing rectangle.

The memory unit 131 of the monitoring device 13 is realized by, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc. As described above, the memory unit 131 stores the position information, age information, and time information of the person detected by the imaging device 11, as well as the position information, heart rate, respiratory rate, and time information of the person within the measurement range calculated by the first measuring device 12. This memory unit 131 stores, for example, information on the standard heart rate range by age, the heart rate range in an abnormal state, the standard respiratory rate range, and the respiratory rate range in an abnormal state.

The determination unit 132 determines whether or not there is something abnormal with a person within the monitoring range. More specifically, the determination unit 132 acquires from the storage unit 131 information on the standard heart rate range and respiratory rate range corresponding to the age information of the person detected by the image capture device 11. The determination unit 132 then acquires from the storage unit 131 the heart rate and respiratory rate associated with area As12 corresponding to the position information (area ag10) of these people h10 and h20 detected by the image capture device 11. The determination unit 132 then determines that person h10 is in an abnormal state if at least one of the heart rate and respiratory rate associated with area As12 is not within the standard range.

If an abnormality is detected, the determination unit 132 reads the captured images of persons h10 and h20 from the person information storage unit 110, writes the area of person h10 in the image, and outputs the image to the firing unit 133 together with notification information indicating the reason for the abnormality determination, such as "abnormal heart rate" or "abnormal breathing," and indicating the person's condition. In other words, if at least one of the heart rate and breathing rate of person h10 captured by the imaging device 11 is not within the standard range, the determination unit 132 generates notification information including information about the person's condition, such as "abnormal heart rate" or "abnormal breathing."

Furthermore, when there is an abnormality in person h10, i.e., when person h10 is in a predetermined state, the determination unit 132 writes the position coordinates of person h10 as marks mh10 and mh20 on the plan view of the train 200, and outputs this as notification information to the firing unit 133. For example, while people h10 and h20 are continuously being filmed as a video, the determination unit 132 writes the position coordinates of people h10 and h20 as marks mh10 and mh20 on the plan view of the train 200, and continues to output notification information to the firing unit 133. As described above, in this embodiment, person condition information such as "abnormal heart rate" and "abnormal breathing" is included in the notification information. For example, the determination unit 132 generates, as notification information, an image signal including an image written as marks mh10 and mh20 on a plan view of the train 200 (see M12 in FIG. 4), and an image captured by cameras 111a and 111b (see M14 in FIG. 4) in which person areas ag10 and ag20, face areas f10 and f20, and the person's condition such as "abnormal heart rate" and "abnormal breathing" have been written. At the same time, the determination unit 132 generates, as notification information, an audio signal including information about the person's condition such as "abnormal heart rate" and "abnormal breathing".

The firing unit 133 has a speaker. When notification information is received, the firing unit 133 fires a warning sound corresponding to the notification information. This makes it possible to notify people around the monitoring device 13 that an abnormality has occurred. Note that the audio signal may be configured to be output only once to the same person h10. Note that the firing unit 133 can change the content of the warning sound depending on the status information associated with the status of an object included in the notification information. For example, the warning sound may be fired to say "abnormal heart rate" or "abnormal breathing."

The firing unit 133 also transmits notification information including the audio signal and image signal supplied from the determination unit 132 to the display device 200. As described above, this audio signal includes information indicating that an abnormality has occurred. As a result, the monitor 202 of the display device 200 displays information about the vehicle with the abnormality (see M12 and 14) as shown in FIG. 4. The speaker 203 of the display device 200 also emits a warning sound in response to the audio signal indicating the person's condition, such as "abnormal heart rate" or "abnormal breathing." In this way, when an abnormality is detected, the determination unit 132 causes the display device 200 to display information about the vehicle with the abnormality (see M12 and 14 in FIG. 4) along with a warning sound, as shown in FIG. 4, via the firing unit 133. This allows the monitor of the display device 200 to easily determine the condition of the person inside the vehicle 200.

The determination unit 132 and firing unit 133 may also be implemented by hardware such as a circuit board, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). Furthermore, the imaging device 11, the first measuring device 12, and the monitoring device 13 may share a CPU, memory unit, etc. Alternatively, the imaging device 11, first measuring device 12, and monitoring device 13 may be configured by sharing a single circuit board such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array). The above is an explanation of an example configuration of the monitoring system 1, and below is an explanation of an example processing.

Figure 6 is a flowchart showing an example of processing by the image capture device 11. Multiple cameras 111a and 111b capture images of a range diagonally downward from their installation positions or a range in the horizontal direction, and store the images in the object information storage unit 114 (step S11). Each captured image is associated with the time of capture.

Next, the person detection unit 112 detects a person area and a face area within this person area from each image stored in the person information storage unit 114. If a person is detected, the coordinates of the rectangular area containing the person and the coordinates of the face area are generated (step S12).

Next, the person detection unit 112 calculates the two-dimensional coordinates of the center of gravity (the intersection of the diagonals) of the rectangle representing the person (step S13). The calculated coordinates indicate the position of the identified person on the image and become the reference coordinates of marks mh10, mh20 (see Figure 4), etc.

Next, the age estimation unit 113 generates feature values based on the image corresponding to the coordinates of the face area and estimates the age based on the feature values (step S14). The age estimation unit 113 transmits the location coordinates of the identified person and the estimated age to the storage unit 131 (step S15). The image capture device 11 performs and updates these processes at regular intervals.

Figure 7 is a flowchart showing an example of processing by the first measuring device 12. In step S21, the transmitter/receiver 121 emits radio waves downward relative to the installation direction and receives radio waves reflected within the irradiation range (step S21). The transmitter/receiver 121 mixes the transmitted and received waves to generate an IF (intermediate frequency) signal and supplies it to the amplifier 122. In addition, by sweeping the modulated frequency of the emitted radio waves, radio waves with different phases can be emitted to various targets, making it possible to distinguish reflections.

Next, the amplifier 122 amplifies the IF (intermediate frequency) signal, converts it into a digital signal in the A/D converter, and supplies it to the processing unit 123 (step S22).

Next, the processing unit 123 calculates the time-series fluctuations in the distance value to the measurement target as the amplitude of the reflecting target based on the digital signal (step S23). Next, the processing unit 123 calculates the frequency of the time-series amplitude fluctuations, and if the generated frequency is within the range of the heart rate/respiratory rate, determines that the measurement target is a person (step S24). In other words, the processing unit 123 determines that an object is a person if the calculated amplitude is not 0 and the frequency is within the range of the person's heart rate/respiratory rate.

Next, the processing unit 123 calculates the position of the object determined to be a person using the distance of each object from the two transmitting and receiving units. In this case, the processing unit 123 detects the entire continuous area that fluctuates with the same period as, for example, the chest area (step S24).

Next, the processing unit 123 supplies information on the position and area of the object determined to be a person, as well as the frequencies corresponding to the heart rate and respiratory rate, to the memory unit 131 of the monitoring device 13. The first measuring device 12 performs and updates these processes at regular intervals.

Figure 8 is a flowchart showing an example of the processing performed by the monitoring device 13. Here, the processing up to the point where an alarm is issued is explained. The determination unit 132 reads out the position information of the person identified from the image captured by the imaging device 11 and the position information of the person calculated by the first measuring device 12, both of which are stored in the memory unit 131 (step S31).

The determination unit 132 compares the respective position coordinates, searches for combinations of corresponding position coordinate values, and associates the heart rate/respiratory rate calculated by the first measuring device 12 with the person image captured by the imaging device 11 (step S32).

Next, the determination unit 132 reads out the standard heart rate/respiratory rate from the memory unit 131 based on the estimated age of the associated person image (step S33). The standard heart rate/respiratory rate of the human body varies depending on age; for example, the heart rate for young children (1 to 3 years old) is 120 to 140, and for adolescents it is 70 to 90.

Next, the determination unit 132 compares the person's heart rate/respiratory rate calculated by the first measuring device 12 with the standard heart rate/respiratory rate for a certain period of time, and determines whether they are within the set threshold range (step S34). If the determination result is determined to be normal (Yes in step S34), the determination unit 132 repeats the process from step S31.

On the other hand, if the judgment result indicates an abnormality (No in step S34), the judgment unit 132 sends notification information including status information to the firing unit 133 and fires a shot to notify the abnormality. At the same time, the location information of the person in question, the time, etc. are stored in the memory unit 131.

In this way, by monitoring the position and heart rate/respiratory rate of people within the measurement range, it is possible to quickly determine where on the image a person experiencing an abnormality in their physical condition is located. This allows for early response. In addition, because the gun is fired at the same time, it is possible to notify people in the monitoring room, driver's cab, and surrounding areas such as the monitoring device 13 that an abnormality has occurred. In particular, it is possible to notify the monitor that an abnormality has occurred along with image information via the display device 200 installed in the monitoring room, driver's cab, etc. This allows the monitor to quickly decide what to do depending on the situation, such as stopping the train or opening and closing the doors.

Figure 9 is a top view showing another example of the arrangement of multiple cameras 111a, b, and c and multiple transmitter/receivers 121 within a vehicle 200. In Figure 9, the imaging device 11 differs from the example arrangement in Figure 3 in that it includes multiple cameras 111c. The cameras 111c are, for example, cameras with fisheye lenses, and are arranged at the intersection of line L200 and line L202. This allows the multiple cameras 111c to capture images of almost all passengers entering and exiting through the opening and closing doors from above. In other words, the multiple cameras 111c are installed in positions where they can capture images of almost all passengers entering and exiting through the opening and closing doors from above. In addition, an imaging area A200 is assigned to each camera 111c. In this embodiment, coordinates in the image captured by the cameras 111c are previously associated with planar coordinates within the imaging area A200. Images captured by multiple cameras 111a, b, and c are associated with the camera that captured the image and the time of capture, and stored in person information storage unit 114. Furthermore, the image captured by multiple cameras 111c is an image directly below, and overlaps with the measurement ranges of multiple transmitter/receivers 121, allowing for more accurate person matching. This makes it possible to capture passengers in more detail even when the number of passengers in vehicle 200 increases, further improving the determination accuracy of monitoring system 1.

As explained above, according to this embodiment, the imaging device 11 captures an image of the space to be monitored, the first measuring device 12 measures the state of a person in the image, and the monitoring device 13 generates notification information when the person in the image is in a predetermined state. This uses the results of measuring the state of the person in the image in addition to the image information, making it possible to generate notification information that corresponds to the state of the person in the image with greater accuracy.

Second Embodiment
The monitoring system 1 according to the second embodiment differs from the monitoring system 1 according to the first embodiment in that it further includes a scream detection device (second measuring device) 14. The differences from the monitoring system 1 according to the first embodiment will be described below.

Figure 10 is a block diagram showing an example configuration of a monitoring system 1 according to the second embodiment. As shown in Figure 10, the monitoring system 1 according to the second embodiment further includes a scream detection device 14 in addition to a photographing device 11, a first measuring device 12, and a monitoring device 13. Note that the first measuring device 12 and the scream detection device (second measuring device) 14 according to this embodiment correspond to the measuring devices.

The scream detection device 14 has a sound collection unit 141, a sound information storage unit 142, a scream detection unit 143, and a second firing unit 144. The sound collection unit 141 is, for example, a microphone. The microphone may be directional or omnidirectional. Multiple microphones may be installed. The sound collection unit 141 records the collected voice of a person and surrounding sounds.

The audio information storage unit 142 is realized, for example, by RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc. The audio information storage unit 142 stores audio data of various types of screams recorded in the past.

The scream detection unit 143 compares the collected sound data with the scream data in the audio information storage unit 142 to detect a predetermined sound, such as a scream. A scream is, for example, a sound made by a person to warn of danger or crisis.

The scream detection unit 143 stores the time when the scream was detected and the position of the audio collection unit 141 where the scream was collected in the memory unit 131 of the monitoring device 13, and also supplies this to the determination unit 132 of the monitoring device 13. Note that the scream detection unit 143 may be realized by hardware such as a circuit board, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

When the scream detection unit 143 detects a scream, the second firing unit 144 transmits an audio signal indicating that a scream has been detected to the display device 200. The second firing unit 144 also has a speaker, which can emit audio indicating that a scream has been detected. Furthermore, the determination unit 132 of the monitoring device 13 may display an image of the vehicle, including the position of the audio collection unit 141 that collected the scream, on image M14 (see Figure 4) on the monitor 202. This allows the monitor of the display device 200 to more easily grasp the situation within the monitored space.

Figure 11 is a flowchart showing an example of the processing of the scream detection device 14. The sound collection unit 141 records sounds around the installation location (step S41). The recorded sounds are digitized using a processing method such as PCM (Pulse Code Modulation) or DSD (Direct Stream Digital) and transmitted to the scream detection unit 143.

Next, the scream detection unit 143 calculates a power spectrum from the digitized sound data and compares the calculated power spectrum with the sound data stored in the audio information storage unit 142 (step S42). The scream detection unit 143 then determines whether the recorded sound data is a scream (step S43). If the scream data stored in the audio information storage unit 142 is similar to the recorded sound data, the scream detection unit 143 determines that it is a scream (Yes in step S43). The scream detection unit 143 then sends a firing command to the second firing unit 144, causing the second firing unit 144 to fire an alarm (step S44).

On the other hand, if the scream data stored in the audio information storage unit 142 and the recorded sound data are not similar, the scream detection unit 143 determines that it is not a scream (No in step S43). If the scream detection unit 143 determines that it is not a scream, it repeats the process from step S41. Furthermore, based on the detection result of the scream detection unit 143, the determination unit 132 of the monitoring device 13 includes, for example, "scream" as status information of the person emitting the specified sound in the notification information.

In this way, by detecting the screams of a person within the measurement range, it is possible to quickly determine that an abnormality has occurred in the surrounding area, allowing for early response. In addition, by firing a gun at the same time, it is possible to notify the surrounding area that an abnormality has occurred.

As described above, according to this embodiment, when the scream detection device (second measurement device) 14 detects a predetermined sound (e.g., a scream), status information is generated that associates the person in the image with the state in which the person is making the predetermined sound. This makes it possible to generate notification information using the status information of the person making the predetermined sound in addition to the image information.

(Third embodiment)
The monitoring system 1 according to the third embodiment differs from the monitoring system 1 according to the second embodiment in that it further includes a scream generation position identification unit 145 that identifies the position of the source of the sound. The differences from the monitoring system 1 according to the second embodiment will be described below.

Figure 12 is a block diagram showing the configuration of a monitoring system 1 according to the third embodiment. As shown in Figure 12, the scream detection device 14 according to the third embodiment further includes a scream generation position identification unit 145 that identifies the position of the person who generated the scream.

Figure 13 is a top view showing an example of the arrangement of multiple cameras 111a, b, multiple transceiver units 121, and multiple sound collection units 141 within a vehicle 200 according to the third embodiment. Line L200 indicates the middle of the opposing opening and closing doors. That is, the vehicle 200 has, for example, eight opening and closing doors. Line L202 indicates the horizontal center line of the vehicle 200. As shown in Figure 13, two sound collection units 141a and b, and 141c and d are arranged in pairs, respectively.

Each of the multiple transmitter/receivers 121 is positioned at the intersection of line L200 and line L202, and also at intermediate locations. This makes it possible to measure almost all passengers in vehicle 200 in more detail from above and diagonally downward.

Figure 14 shows an example image captured by camera 111b. This image includes line L13, which passes through person h10 in Figure 13, and was captured from a direction roughly facing line L13. Sound areas Av10-16 are schematic examples of areas identified by the scream generation position identification unit 145.

Figure 15 shows an example of a screen displayed on the display device 200 by the monitoring device 13 according to the third embodiment. Screen M10 shows the screen of the monitor 201 (see Figure 12), screen M12 is a top view of the vehicle 200, and mark mh10 indicates the position of person h10 who uttered the scream. Screen M14 alternately displays images captured by cameras 111a and 111b. Area ag10 indicates the area of person h10 detected by the person detection unit 112. Furthermore, area f10 indicates the face area within area ag10 detected by the person detection unit 112.

As shown again in FIG. 12, the scream generation position identification unit 145 calculates the distance between the sound collection unit 141a and the sound collection unit 141b from the time difference between the screams recorded by the sound collection units 141a, b, 141c, and d (see FIG. 13) and the corresponding sound data, and estimates the position of the sound source using the principle of triangulation. Similarly, the scream generation position identification unit 145 calculates the distance between the sound collection unit 141c and the sound collection unit 141d, and estimates the position of the sound source using the principle of triangulation. More specifically, the scream detection unit 143 performs scream recognition processing on the recorded sounds of the sound collection units 141a, b, 141c, and d (see FIG. 13), and supplies the detection times corresponding to each of the sound collection units 141a, b, 141c, and d to the scream generation position identification unit 145.

The scream generation position identification unit 145 identifies the planar coordinates of the sound source on the vehicle 200 using, for example, the distance at which a difference in detection time occurs between the sound collection units 141a and 141b, and the distance between the sound collection units 141a and 141b, according to the principle of triangulation. Similarly, the scream generation position identification unit 145 identifies the planar coordinates of the sound source on the vehicle 200 using the distance at which a difference in detection time occurs between the sound collection units 141c and 141d, and the distance between the sound collection units 141c and 141d, according to the principle of triangulation. The scream generation position identification unit 145 supplies the position coordinates and time of the sound source to the memory unit 131 of the monitoring device 13.

Based on the position coordinates of the sound source, the determination unit 132 of the monitoring device 13 selects one sound area Av13 from the sound areas Av10-16 to which the position coordinates of the sound source belong. Then, the determination unit 132 identifies person h10 from the selected sound area Av13 whose coordinates are close to the position coordinates of the sound source.

The determination unit 132 reads the captured image of the identified person h10 from the person information storage unit 110, writes the area of the person h10 in the image, and outputs this as notification information to the firing unit 133 along with status information such as the reason for the abnormality determination, for example, "scream," "abnormal heart rate," or "abnormal breathing." Furthermore, if an abnormality is detected, the determination unit 132 writes the position coordinates of the person h10 as mark mh10 on the plan view of the train 200 and outputs this as an image signal to the firing unit 133. For example, while the person h10 continues to be captured as a video, the determination unit 132 writes the position coordinates of the person h10 as mark mh10 on the plan view of the train 200 and continues outputting an image signal to the firing unit 133.

The firing unit 133 transmits an image signal containing image information and an audio signal supplied from the determination unit 132 to the display device 200. This audio signal contains status information indicating the occurrence of an abnormality, as described above. Thus, when an abnormality is detected, the determination unit 132 causes the display device 200 to display information about the vehicle with the abnormality along with an audible warning via the firing unit 133, as shown in FIG. 15. In this case, the firing unit 133 and the second firing unit 144 may emit an audible warning. Firing a gun in this way also makes it possible to notify the monitoring device 13 and people around the display device 200 that an abnormality has occurred. Note that the audio signal can be configured to be output only once to the same person h10, for example. Furthermore, the monitor monitoring the display device 200 can easily grasp the situation inside the vehicle 200 from the image, making it easier to make decisions based on the situation, such as stopping the train or opening and closing the doors. Furthermore, if an abnormality occurs, all that is required is to check an image of the area where the abnormality occurred, so there is no need to constantly monitor images from inside the entire vehicle, which prevents interference with other operations, such as driving, during normal operation.

When images from both cameras 111a and 111b are available, the determination unit 132 may preferentially display the image from camera 111b, which is closer to line L13 (see Figure 13), on the display device 200. For example, when displaying images in chronological order, four images from camera 111b may be displayed consecutively, followed by one image from camera 111a. This makes it easier to grasp the situation.

Figure 16 is a flowchart showing the processing of the scream detection device 14 according to the third embodiment. Steps S41 to S43 perform the same processing as in Figure 11. If it is determined that a scream has occurred (Yes in step S43), the scream generation position identification unit 145 identifies the coordinates of the source of the scream in the vehicle 200 using, for example, the distance at which the detection time difference occurs between the sound collection units 141a and 141b, and the distance between the sound collection units 141a and 141b, according to the principle of triangulation (step S54).

The scream occurrence position identification unit 145 transmits information on the calculated scream occurrence coordinates to the memory unit 131 of the monitoring device 13. Then, the scream detection unit 142 transmits a firing command to the second firing unit 144, which then fires an alarm to warn of danger (step S55). Furthermore, the determination unit 132 of the monitoring device 13 causes the display device 200, via the firing unit 133, to emit a sound including status information and display image information, etc.

Figure 17 is a flowchart showing an example of processing by the monitoring device 13 according to the third embodiment. The determination unit 132 reads from the storage unit 131 the position coordinates of the person in the image calculated by the image capture device 11 and the scream occurrence position information calculated by the scream detection device 14 (step S61).

Next, the determination unit 132 searches for the combination that is closest to the position coordinates of the person in the image calculated by the image capture device 11 and the scream occurrence position calculated by the scream detection device 14, and if a matching combination is found, it determines that the person in question has screamed (step S62). Next, the determination unit 132 transmits the position information and time of the person who emitted the scream to the memory unit 131 (step S63). The determination unit 132 then transmits a firing command to the firing unit 133, which then fires a gun to warn of danger (step S64).

In this way, by monitoring the location of people within the measurement range and whether or not they are screaming, it is possible to quickly determine where on the image the person who is screaming is located, enabling an early response. In addition, because a shot is fired at the same time, it is also possible to notify the surrounding area of any abnormalities or dangers.

In addition, although a detailed explanation will be omitted, it is possible to link the results of the scream detection device 14 and the first measuring device 12 in addition to the results of the imaging device 11. This makes it possible to identify (location, person, condition), for example, a person who is experiencing a change in their physical condition and is crying out in distress, making it possible to respond to a variety of situations.

Here, using Figures 18 to 21, we will explain other examples of the placement of the camera 111, transceiver unit 121, and audio collection unit 141. Figure 18 shows pattern P18, in which the audio collection unit 141 is placed in the door section, and pattern P19, in which the audio collection unit 141 is placed at the intersection of line L200 and line L202.

In pattern P18, the camera 111 and transmitter/receiver 121 are placed in the center of the ceiling in the corridor near the entrance/exit door, and the image is taken directly below, allowing for more accurate correspondence with the sound source position. Similarly, by generating millimeter waves in the same range as the camera 111, it is possible to more accurately match the subject area of the image with the chest area of the heartbeat source. In addition, by placing the sound collection unit 141 at the end, it is possible to suppress the input of reflected waves.

On the other hand, pattern P19 differs from pattern P18 in that the sound collection unit 141 is also placed in the center of the ceiling. In pattern P19, the sound collection area of the sound collection unit 141, the imaging area of the camera 111, and the millimeter wave generation range of the transmitter/receiver 121 can be made equivalent, allowing for more accurate matching of the measurement target. Furthermore, because the sound collection unit 141 is placed in the center of the ceiling, direct sound can be collected more efficiently.

Figure 19 is a diagram showing an example in which a sound collection unit 141 is further placed in the middle of the vehicle 200 in the direction of travel. This differs from the placement pattern in Figure 13 in that a sound collection unit 141 is further placed in the middle. By placing the sound collection unit 141 in the center of the ceiling, direct sound can be collected more efficiently. In addition, the sound collected by the sound collection units 141 at the ends can be supplemented by the sound collection units 141 placed in the center of the ceiling, making it possible to identify the position of the sound source with greater accuracy.

Figure 20 is a diagram showing an example of the layout of a vehicle 200 in which seats face in the direction of travel or opposite the direction of travel. That is, Figure 20 is a diagram showing an example of the layout when there are no opening and closing doors on the side of the vehicle 200 along the direction of travel, such as in a Shinkansen vehicle. Camera 151 is a camera installed on the door side. The imaging device 11 can also refer to images from camera 151.

Camera 111c is located in the center of vehicle 200. Camera 111c, located in the center, complements the images captured by camera 111am, located near the doors. Audio collection units 141a-f are located near the doors between vehicles and in the center of vehicle 200. Audio collection units 141e and 141f, located in the center, complement the audio collected by audio collection units 141a-d, located near the doors. Because cameras 111a, b, and c, transceiver unit 121, and audio collection units 141a-f are located symmetrically with respect to the direction of travel of vehicle 200, fluctuations in measurement accuracy are suppressed even when the direction of travel changes.

Figure 21 shows an example in which camera 111 and audio collection unit 141 are placed in the longitudinal center of vehicle 200. Camera 111e near the door is a camera with a fisheye lens, and camera 111f is a camera with a normal lens. Furthermore, because four cameras 111e and 111f are placed above the aisle, the capture range of the captured images is approximately 5m apart, making it possible to capture seated people with higher accuracy. Furthermore, because aisle camera 111f can capture images directly below, if the position of the sound source can be identified using audio collection unit 141, the accuracy of matching can be improved.

The monitoring system 1 according to this embodiment has been described using a vehicle 200 as an example, but is not limited to this. For example, the monitoring system 1 can also be placed in a monitored space where an unspecified number of people enter and exit, such as an elevator hall.

Figure 22 is a diagram showing an example of the arrangement of cameras 111a, b, transceiver unit 121, and audio collection units 141a-d in an elevator hall 300. The arrangement of cameras 111a, b, and audio collection units 141a-d is the same as that shown in Figure 13, etc., and the monitoring system 1 can capture images of people in the elevator hall 300, identify their areas, and obtain location information for people who scream, etc. Furthermore, because the transceiver unit 121 is placed in front of the elevator doors, it is possible to measure the heart rates, etc. of people waiting for the elevator 302 with greater accuracy.

As described above, according to this embodiment, when the scream detection device (second measurement device) 14 detects a predetermined sound (e.g., a scream), the scream generation position identification unit 145 identifies the position of the sound source. This makes it possible to generate status information that associates the position of the sound source with the position of a person in the image. Therefore, the status information makes it possible to confirm the status of the sound source from the image.

While several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their variations are within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as set forth in the claims.

1...surveillance system, 11...imaging device, 12...first measuring device, 13...surveillance device, 14...scream detection device (second measuring device), 111a, b, c, e, f...camera, 112...person detection unit, 113...age estimation unit, 114...person information storage unit, 121...transmitting/receiving unit, 122...amplifier, 123...processing unit, 131...storage unit, 132...determination unit, 133...gun firing unit, 141a, b, c, d...audio collection unit, 142...scream detection unit, 143...audio information storage unit, 144...second gun firing unit, 145...scream generation location identification unit.

Claims (15)

an image capturing device that captures an image associated with coordinates in a space to be monitored , and outputs, based on the image, position information of a person area in the image and an estimated age of the person in the person area ;
a measurement device that outputs measurement area information of the person and at least one of a heart rate and a respiratory rate of the person associated with the measurement area information based on the received signal ;
A monitoring device ;
With
The monitoring device
a storage unit that stores position information of the person area output by the image capturing device, an estimated age of the person, and at least one of a heart rate and a respiratory rate of the person associated with the measurement area information of the person;
a determination unit that determines, using the information stored in the storage unit, that at least one of a heart rate and a respiratory rate of the person associated with the measurement area information corresponding to position information of the person area is abnormal when the heart rate and the respiratory rate are not within a standard range for the estimated age of the person;
A monitoring system having : The monitoring system according to claim 1 , wherein when the monitoring device determines that an abnormality has occurred, the monitoring device generates, as notification information, information relating to the position of the person in the image and a state of the person. The surveillance system according to claim 2 , wherein the image capturing device includes an age estimation unit that estimates an estimated age of the person in the image. the age estimation unit estimates the estimated age based on image information within the person area in the image,
The monitoring system according to claim 3 , wherein the monitoring device generates the notification information by associating the position information of the person area with the measurement area information. The measurement device detects a predetermined sound,
The monitoring system according to claim 1 , wherein the monitoring device generates notification information when the predetermined sound is detected. the predetermined sound is a scream that notifies a person of danger or crisis,
The monitoring system according to claim 5 , wherein when the monitoring device detects the scream, the monitoring device generates the notification information that associates the person in the image with a state in which the person is emitting the predetermined sound. The monitoring system according to claim 6 , wherein the monitoring device generates a sound in response to the notification information. The monitoring system according to claim 7 , wherein the monitoring device, when determining that an abnormality has occurred , causes a display device to display the image including the person. The surveillance system of claim 8, wherein the notification information includes coordinate information within a vehicle or elevator hall indicating the person's location. The surveillance system of claim 9, wherein the imaging device includes a camera that captures the image inside the vehicle or the elevator hall. The monitoring system of claim 10, wherein the camera is positioned at an end of the vehicle in the direction of travel. The monitoring system described in claim 10, wherein the camera is positioned within a range capable of capturing images of the vehicle's opening and closing doors. The surveillance system described in claim 11 or 12, wherein the camera captures images through a fisheye lens. The measuring device is
a transmitting/receiving unit that transmits and receives predetermined radio waves;
a processing unit that generates a fluctuation period of the distance to the person as at least one of a heart rate and a respiratory rate of the person based on the predetermined radio wave,
The monitoring system according to claim 9, wherein the transmitter/receiver is arranged within a range in which the radio waves can be transmitted to a door of the vehicle and within a range in which the radio waves can be transmitted to a door of an elevator in an elevator hall. The measuring device is
a voice collection unit that collects voice;
a detection unit that detects a predetermined sound from the sound;
an identification unit that identifies the position of a sound source that is emitting the sound,
The monitoring system according to claim 9 , wherein the sound collecting unit is disposed at least one of an end of the vehicle in a traveling direction and an end of an elevator hall.