JP7845557B2 - Display control device and monitoring system - Google Patents

Description

The present invention relates to a display control device and a monitoring system.
This application claims priority based on Japanese Patent Application No. 2021-167872, filed on 13 October 2021, the contents of which are incorporated herein by reference.

Conventionally, technologies related to remote monitoring devices have been disclosed (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2017-11698

One embodiment of the present invention is a display control device comprising: a receiving unit that receives event information relating to an event occurring at a monitored location; and a display unit that displays a history of the event information received by the receiving unit.

This figure shows an example of the device configuration of the monitoring system according to this embodiment. This figure shows an example of the functional configuration of the monitoring system of this embodiment. This figure shows an example of the configuration information for the monitoring system of this embodiment. This figure shows an example of the initial setup screen for this embodiment. This figure shows an example of the initial setup screen for this embodiment. This figure shows an example of the operation screen for the emergency call function of this embodiment. This figure shows an example of the display for each monitoring mode in this embodiment. This figure shows an example of the display for each monitoring mode in this embodiment. This figure shows an example of communication in the emergency response function of this embodiment. This figure shows an example of a pop-up display during an emergency response in this embodiment. This figure shows an example of the live view display in this embodiment. This figure shows an example of the corresponding selection screen in this embodiment. This figure shows an example of the display screen for the correspondence history in this embodiment. This figure shows another example of communication in the emergency response function of this embodiment. This figure shows an example of the event history display in this embodiment. This figure shows an example of the event selection screen for this embodiment. This figure shows an example of communication using the conversation function of this embodiment. This figure shows an example of communication for the live view function in the private mode of this embodiment. This figure shows an example of communication for the live view function in public mode of this embodiment. This figure shows an example of the operation of the monitoring system in public mode and private mode according to this embodiment. This figure shows an example of communication in the voltage drop notification function of this embodiment. This figure shows an example of the voltage drop notification screen of this embodiment. This figure shows an example of communication in the remote control device search function of this embodiment. This figure shows an example of communication in the remote control device out-of-range notification function of this embodiment.

[Monitoring System Configuration]
Figure 1 shows an example of the device configuration of the monitoring system 1 of this embodiment.
The monitoring system 1 comprises a terminal device 10, a server device 20, a communication device 30, and a remote control device 40. The monitoring system 1 provides a monitoring service for a monitor to monitor the status of a person being monitored.

In the following explanation, the person being monitored will also be referred to as the person being monitored or the person being monitored. The monitoring service provided by monitoring system 1 will also be referred to as the monitoring service. Furthermore, the person monitoring will also be referred to as the monitor, the user of the monitoring service, or the user of the monitoring service.
In other words, monitoring system 1 provides a monitoring service that allows the caregiver to monitor the condition and behavior of the person being monitored.
For example, the person being monitored is a non-resident family member of the caregiver. Monitoring system 1 provides a function that allows the caregiver to remotely monitor the situation of the person being monitored.

Terminal device 10 is a device used by the monitor, i.e., the user. Terminal device 10 is, for example, a smartphone, tablet, or personal computer, and communicates information with server device 20 via network N1.
The server device 20 is a computer device operated by the provider of the monitoring service. The server device 20 is responsible for various settings in the monitoring system 1, as well as for storing and providing information acquired by the monitoring system 1.
The communication device 30 includes, for example, an imaging unit 350 (imaging device) such as a camera, and takes images of the monitored area A. The communication device 30 communicates information with the server device 20 via the network N2.
The remote control device 40 is a device used by the person being monitored. The remote control device 40 communicates information with the communication device 30 using short-range wireless communication or infrared communication.

The monitoring system 1 may also include multiple terminal devices 10, communication devices 30, and remote control devices 40.
For example, the monitoring system 1 can be configured to allow for the setting of multiple caregivers, with each caregiver using one terminal device 10. For instance, if the caregivers have siblings, the system can be configured so that each caregiver (sibling) watches over the parent who is the person being monitored.
Furthermore, the monitoring system 1 can be configured to monitor the state and behavior of the person being monitored in each room of a building by installing a communication device 30 in each room. For example, if a building has multiple rooms, the monitoring system 1 can be configured to monitor the state and behavior of the person being monitored in each room, with the first room (e.g., a living room) designated as monitoring area A1 and the second room (e.g., a bedroom) designated as monitoring area A2.
Furthermore, the monitoring system 1 can be configured to allow multiple people being monitored who live together in a building to use the remote control device 40. For example, if the people being monitored are the parents of the person being monitored, the system can be configured so that the father and mother each use the remote control device 40.
Furthermore, the monitoring system 1 can monitor not just one building, but multiple buildings. For example, if the person being monitored has a spouse, the monitoring system 1 can be configured to monitor the state and actions of the person being monitored in each building, designating the first building where the person's parents reside as monitoring area A1, and the second building where the parents of the person's spouse reside as monitoring area A2.

In the following explanation, when distinguishing between multiple terminal devices 10, the first terminal device 10 will be referred to as terminal device 10-1, the second terminal device 10 as terminal device 10-2, and so on.
Furthermore, when distinguishing between multiple communication devices 30, the first communication device 30 is referred to as communication device 30-1, the second communication device 30 as communication device 30-2, and so on.
Next, with reference to Figure 2, the functional configuration of each device in the monitoring system 1 will be described.

[Functional Configuration of the Monitoring System]
Figure 2 shows an example of the functional configuration of the monitoring system 1 of this embodiment.

[Functional Configuration of Terminal Devices]
The terminal device 10 includes a terminal operation detection unit 110, a display unit 120, a terminal control unit 130, a terminal communication unit 140 (receiving unit), a terminal sound pickup unit 150, and a terminal sound generation unit 160.
The terminal operation detection unit 110 is equipped with an operation device such as a touch panel or an operation switch, and detects operations performed by the user of the terminal device 10 on the operation device. When the terminal operation detection unit 110 detects an operation, it generates operation information indicating the detected operation and outputs it to the terminal control unit 130.
The display unit 120, for example, includes a liquid crystal display and displays images based on the control of the terminal control unit 130.
The terminal control unit 130, for example, includes a microcomputer and controls each part of the terminal device 10.
The terminal communication unit 140 communicates information with the server device 20 based on the control of the terminal control unit 130.

The terminal sound pickup unit 150, for example, is equipped with a microphone and picks up sounds from the surrounding area of the terminal device 10. The terminal sound pickup unit 150 outputs the sound information it has picked up to the terminal control unit 130.
Furthermore, the terminal device 10 may operate various functions of the terminal control unit 130 (for example, by performing voice recognition) based on the sound it has received. In this case, the terminal sound collection unit 150 can be said to be an example of a terminal operation detection unit 110 that operates the terminal device 10 based on sound.
The terminal sound-emitting unit 160, for example, is equipped with a speaker and emits sound based on the control of the terminal control unit 130. The terminal device 10 may also use the emitted sound to alert the user (for example, by emitting an alarm sound). In this case, the terminal sound-emitting unit 160 can be said to be an example of a presentation unit 120 that presents information by sound.

[Server Device Functional Configuration]
The server device 20 includes a server communication unit 210 and a server storage unit 220.
The server communication unit 210 communicates information with the terminal device 10 and with the communication device 30.
The server storage unit 220 is equipped with a storage device such as semiconductor memory and stores various types of information. An example of the information stored in the server storage unit 220 will be explained with reference to Figure 3.

Figure 3 shows an example of the configuration information for the monitoring system 1 of this embodiment.
As described above, in the monitoring system 1, terminal device 10 can monitor multiple monitoring targets. In this example, among the multiple monitoring targets a to c, terminal devices 10-1 and 10-2 are set to monitor targets a to c.
Furthermore, while terminal device 10-3 has monitoring targets a to b set as monitoring targets, monitoring target c is not set as a monitoring target.
Furthermore, when multiple terminal devices 10 monitor a single target, it is possible to configure one terminal device 10 to function as the main monitoring device and another terminal device 10 to function as the sub-monitoring devices. For example, for target a, terminal device 10-1 functions as the main monitoring device, and terminal devices 10-2 and 10-3 function as sub-monitoring devices. In the monitoring system 1, the main monitoring device and the sub-monitoring devices can provide different functions to each other.
The server storage unit 220 stores configuration information as shown in Figure 3. This configuration information is accessed as needed by the terminal device 10 or the communication device 30.

Here, the user of the main monitoring device and the user of the secondary monitoring device may be given different operational permissions. For example, the user of the main monitoring device may be given the right to view real-time video regardless of whether the monitored person has given permission, while the user of the secondary monitoring device may be prevented from viewing real-time video without the monitored person's permission.
Furthermore, while the operation permissions were explained here as being divided into two levels, main and sub, this is not the only way to do so; operation permissions may be set up in three or more levels.
In other words, each user is granted at least one of two levels of operational authority, and the presentation unit 120 presents content based on the operational authority granted to the user.

This operational authority may be granted for each monitored area A (i.e., each monitored location).
In other words, there are multiple locations to be monitored, and the user is granted operational authority for each monitored location. The display unit 120 of the terminal device 10 displays content based on the operational authority granted to the user for each monitored location.

[Functional Configuration of Communication Devices]
Returning to Figure 2, the communication device 30 includes the following functions: a first communication unit 310, a second communication unit 320, a communication device control unit 330, a measurement unit 340, an imaging unit 350, a communication device sound pickup unit 360, a communication device sound generation unit 370, an operation detection unit 380, and a communication device storage unit 390.

The first communication unit 310 communicates information with the remote control device 40 based on the control of the communication device control unit 330.
The second communication unit 320 communicates information with the server device 20 based on the control of the communication device control unit 330.

As mentioned above, the second communication unit 320 communicates with the terminal device 10 via the server communication unit 210 of the server device 20. However, in the following description, the explanation that the server device 20 mediates communication between the terminal device 10 and the communication device 30 may be omitted, and it may be stated, for example, that the second communication unit 320 transmits the first operation information to the terminal device 10.

The measurement unit 340 measures various conditions in the monitored area A. For example, the measurement unit 340 may be equipped with a temperature sensor and a humidity sensor to measure the temperature and humidity of the monitored area A. Alternatively, the measurement unit 340 may be equipped with an infrared sensor to measure the movement of people in the monitored area A.
The imaging unit 350, for example, is equipped with a camera and, based on the control of the communication device control unit 330, takes images of the area A to be monitored.

The measurement unit 340 and the imaging unit 350 are collectively referred to as the detection unit. In other words, the communication device 30 detects various conditions in the monitored area A using the detection unit.
In addition to the imaging unit 350, the detection unit may also have a determination unit (not shown) that determines the state of the monitored object based on the video captured by the imaging unit 350. The determination unit, for example, analyzes the posture of the monitored object based on the results of machine learning and determines whether or not an abnormality has occurred to the monitored object, such as the object falling. The determination unit outputs the determination result to the communication device control unit 330.

The communication device control unit 330, for example, includes a microcomputer and controls various parts of the communication device 30, including the first communication unit 310, the second communication unit 320, and the imaging unit 350.

The communication device sound pickup unit 360, for example, is equipped with a microphone and picks up sounds from the vicinity of the communication device 30. The communication device sound pickup unit 360 outputs the sound information it has picked up to the communication device control unit 330.
The communication device sound generation unit 370, for example, is equipped with a speaker and emits sound based on the control of the communication device control unit 330.
The operation detection unit 380 detects operations on switches or other operating parts (not shown) provided by the communication device 30. For example, the communication device 30 is equipped with an operating mode switching switch (not shown). The operation detection unit 380 detects the setting state of the operating mode switching switch and outputs it to the communication device control unit 330.

The communication device storage unit 390 is equipped with a storage device such as a semiconductor memory and stores various types of information. For example, the communication device storage unit 390 stores information about images (e.g., moving images) captured by the imaging unit 350 and information about sounds captured by the communication device sound pickup unit 360.

[Functional Configuration of Remote Control Device]
The remote control device 40 includes an emergency button 410, an OK button 420, an NG button 480, a remote control communication unit 430, a remote control control unit 440, a remote control sound pickup unit 450, a remote control sound generation unit 460, and a voltage detection unit 470.
The OK button 420 is an operation button that indicates agreement, approval, or a positive response, and is also called the "yes button." The NG button 480 is an operation button that indicates disagreement, rejection, or a negative response, and is also called the "no button."

The emergency button 410 is a button that is operated by the person being monitored if any abnormality occurs to them.
The OK button 420 is a button that is operated by the person being monitored when they grant permission when a permission request is made to them by the monitoring system 1.

The remote control communication unit 430 communicates information with the communication device 30 based on the control of the remote control control unit 440.

The remote control unit 440, for example, is equipped with a microcomputer and controls various parts of the remote control device 40. For instance, when the remote control unit 440 detects an operation on the emergency button 410, it generates first operation information indicating that the emergency button 410 has been operated. The remote control unit 440 transmits the generated first operation information to the communication device 30 via the remote control communication unit 430. Similarly, when the remote control unit 440 detects an operation on the OK button 420, it generates second operation information indicating that the OK button 420 has been operated. The remote control unit 440 transmits the generated second operation information to the communication device 30 via the remote control communication unit 430.

The remote control sound pickup unit 450, for example, is equipped with a microphone and picks up sounds from the surrounding area of the remote control device 40. The remote control sound pickup unit 450 outputs the sound information it has picked up to the remote control control unit 440.
The remote control sound-emitting unit 460, for example, is equipped with a speaker and emits sound based on the control of the remote control control unit 440.
The voltage detection unit 470 detects the voltage value of the operating power supply of the remote control device 40. The voltage detection unit 470 outputs the detected voltage value of the operating power supply to the remote control control unit 440. If the voltage value detected by the voltage detection unit 470 is below a predetermined value, the remote control control unit 440 transmits voltage drop information to the communication device 30 via the remote control communication unit 430.

[Explanation of each function]
Next, the functions and operations of each feature provided by the monitoring system 1 will be explained with reference to the diagram.
In the following description, application software (hereinafter also simply referred to as "app") provided by the monitoring system 1 is installed on the terminal device 10. The terminal control unit 130 controls each part of the terminal device 10 based on the installed app. The display unit 120 of the terminal device 10 displays the screen defined by the app. The display unit 120 displays the initial setup screen when the app is first launched after it has been installed on the terminal device 10.

[Initial Setup]
Figures 4A to 4B show examples of the initial setup screens of this embodiment. Figure 4A shows an example of the notification settings screen. The monitoring system 1 provides a function to initially set the status of the person being monitored that should be notified to the terminal device 10. An example of a status to be notified is shown below.

- Emergency call: When emergency button 410 is pressed.
- Fall: If the person being monitored falls within the monitored area A.
- Temperature and humidity anomaly: This occurs when the temperature or humidity in monitored area A is outside the specified range.
- Safety confirmation abnormality: When there is no response to calls made to the person being monitored.
- Camera malfunction: If a malfunction or other abnormality occurs in the imaging unit 350.
- Routine deviation: When the behavior of the person being monitored differs from the usual.

The user operates the notification settings screen displayed on the display unit 120 of the terminal device 10 and selects a notification target from several types of situations. The terminal device 10 sends the selected notification settings to the server device 20. The server device 20 stores the received notification settings in the server storage unit 220. The monitoring system 1 notifies the terminal device 10 when the configured situation occurs.
In this example, notification settings are described as being stored in the server storage unit 220, but this is not the only option. Notification settings only need to be able to determine whether or not a user should be notified when a certain situation occurs, and they can be stored in any of the terminal device 10, server device 20, or communication device 30. The various types of information described below can also be stored in any of the devices, as long as the function of using that information can be fulfilled.

Figure 4B shows an example of the contact settings screen. The monitoring system 1 provides an emergency notification function that contacts pre-configured contacts with minimal operation (for example, tapping an icon displayed on the terminal device 10) in an emergency.

Figure 4C shows an example of the operation screen for the emergency call function. When the above-mentioned notification target situation occurs, the display unit 120 of the terminal device 10 displays the operation screen (emergency response screen) for the emergency call function shown in Figure 4C. For example, if three emergency call destinations are set, the display unit 120 will display emergency call destination icons P11 to P13. The contacts for this emergency call function can be set using the contact setting screen shown in Figure 4B.
The user operates the contact setting screen displayed on the display unit 120 of the terminal device 10 to set contacts. The terminal device 10 stores the set contacts in its storage unit (not shown).

[Monitoring Mode Settings]
Figures 5A to 5B show examples of displays for each monitoring mode in this embodiment.
The monitoring system 1 provides multiple monitoring modes, including a private mode and a public mode. Figure 5A shows an example of the screen displayed on the terminal device 10 in private mode. In private mode, no image of the monitored person is displayed; instead, an image P21 is displayed that shows the monitored person's status in text. Figure 5B shows an example of the screen displayed on the terminal device 10 in public mode. In public mode, an image P22 of the monitored area A captured by the imaging unit 350 of the communication device 30 is displayed. If there is a monitored person in the monitored area A, the image P22 displays a moving image (real-time image) showing the monitored person's current status. Note that in Figure 5B, for the sake of explanation, the moving image of the monitored person is represented by a line drawing. In this embodiment, the terminal device 10 displays a moving image of the monitored person actually captured by the imaging unit 350, rather than a line drawing.
The private mode and the public mode can be switched using the operating mode switching switch (not shown) of the communication device 30 described above.
The monitoring system 1 can provide a monitoring function that allows users to select the degree of privacy protection and the level of monitoring function according to the relationship between the person being monitored and the monitor, as well as the situation of the person being monitored, by making it possible to switch between monitoring modes.
Next, we will explain each function after the initial setup.

[Monitoring function (1) Emergency response function]
Figure 6 shows an example of communication in the emergency response function of this embodiment.
(Step S101) If any abnormality occurs in the person being monitored, the person being monitored operates the emergency button 410 on the remote control device 40. When the remote control device 40 detects that the emergency button 410 has been operated, it transmits first operation information to the communication device 30. The first operation information is information indicating that the emergency button 410 has been operated.

(Step S102) When the communication device control unit 330 receives information from the remote control device 40 indicating that the emergency button 410 has been operated, the second communication unit 320 transmits first operation information to the terminal device 10.
In the following explanation, the information that the communication device control unit 330 transmits to the terminal device 10 when the emergency button 410 is operated will also be referred to as "first operation information." This expression includes cases where the communication device control unit 330 directly transfers the first operation information received from the remote control device 40 to the terminal device 10, and cases where it transmits information generated in response to receiving the first operation information from the remote control device 40 to the terminal device 10.

In this case, if the communication device 30 includes an imaging unit 350, when the communication device control unit 330 receives first operation information from the remote control device 40, it transmits the image captured by the imaging unit 350 and the first operation information to the terminal device 10 from the second communication unit 320.
In addition, the communication device 30 may include a communication device sound pickup unit 360. In this case, when the communication device control unit 330 receives first operation information from the remote control device 40, it transmits the sound information picked up by the communication device sound pickup unit 360 and the first operation information to the terminal device 10.
Furthermore, the communication device control unit 330 may transmit sound information picked up by the communication device sound pickup unit 360 without transmitting the image captured by the imaging unit 350. Alternatively, the communication device control unit 330 may transmit both the image captured by the imaging unit 350 and the sound information picked up by the communication device sound pickup unit 360 to the terminal device 10.
Furthermore, when the communication device 30 transmits the image captured by the imaging unit 350 and the sound picked up by the communication device's sound pickup unit 360 immediately after the timing of image capture and sound pickup, that is, in real time, this is also referred to as "live view image and sound."

The following description assumes that the communication device 30 comprises an imaging unit 350 and a communication device sound pickup unit 360, and transmits live view images and audio to the terminal device 10.

Furthermore, the audio from the monitored area A can also be picked up by the remote control sound pickup unit 450, which is installed in the remote control device 40 operated by the person being monitored. In this case, the sound picked up by the remote control sound pickup unit 450 is transmitted from the remote control communication unit 430 to the communication device 30. However, after the emergency button 410 is operated, communication between the remote control device 40 and the communication device 30 may be interrupted for some reason. Because the communication device 30 is equipped with a communication device sound pickup unit 360, even if communication between the communication device 30 and the remote control device 40 is impossible, the communication device 30 can still pick up ambient sounds and provide live view images and audio to the terminal device 10.

(Step S103) When the terminal device 10 receives the first operation information and the live view image and audio from the communication device 30, it displays an image on the display unit 120 indicating that the emergency button 410 has been operated.

Figure 7 shows an example of a pop-up display during an emergency response in this embodiment. When the terminal device 10 receives the first operation information from the communication device 30, it displays a pop-up image P31 as shown in Figure 7. With the monitoring system 1 configured in this way, even if the user is not operating the terminal device 10 or is performing an operation other than the monitoring system 1's application, the user can be notified that the monitored person has pressed the emergency button 410.

Figure 8 shows an example of the live view display in this embodiment.
(Step S104) When the pop-up image P31 displayed in step S103 is operated (for example, tapped), the terminal device 10 displays the live view image P41, the corresponding selection icon P42, and the conversation icon P43.
Note that in Figure 8, for the sake of explanation, the video of the person being monitored in the live view image P41 is represented by a line drawing. In this embodiment, however, the terminal device 10 displays the video of the person being monitored that is actually being captured by the imaging unit 350, rather than a line drawing.

Figure 9 shows an example of the corresponding selection screen in this embodiment.
(Step S105) When the corresponding selection icon P42 is operated, the terminal device 10 displays the corresponding selection screen P51 shown in Figure 9.

Here, the first operation information is an example of event information indicating that the emergency button 410 was operated. The response selection screen P51 contains icons for the response content corresponding to the type of event.
In other words, the event information includes the type of event, and the presentation unit 120 presents content corresponding to the type of event included in the received event information.

When terminal device 10 receives the first operation information, that is, when the event type is "operation of emergency button 410", terminal device 10 will, for example,
- The user rushes to the location where the person being monitored is (Icon P511)
- Report to a public institution (fire/ambulance, police, public health center, local government office, etc.) (Icon P512)
- Contact a pre-configured contact (Icon P513)
- A selection screen (P51) is displayed, which includes various icons such as the option to launch the phone application (icon P514).
In other words, the response includes at least one of the following: the user moving to the location being monitored, notification to a public institution, or notification to other users. When any of the icons on the response selection screen P51 is operated, the terminal device 10 performs the action corresponding to the operated icon (for example, making a call to a designated contact). Furthermore, the terminal device 10 transmits the selected response to the server device 20.

In other words, when the terminal communication unit 140 receives event information, the display unit 120 displays an icon indicating the event information as a pop-up, and then, based on the user's actions after the icon is displayed, displays content corresponding to the type of event.

Furthermore, as mentioned above, if the communication device 30 has a judgment unit (not shown), and this judgment unit determines whether or not an abnormality has occurred based on the posture analysis results of the person being monitored, then the judgment result of the judgment unit, "the person being monitored has fallen," can also be considered an example of event information.

(Step S106) Returning to Figure 6, when the server device 20 receives the response information from the terminal device 10, it stores the received response information, along with the time information transmitted by the terminal device 10, in the server storage unit 220 as a response history. The server device 20 also transmits information to the communication device 30 indicating that the user has responded to the operation of the emergency button 410.
(Step S107) The communication device 30 releases the emergency state and returns to normal operation.

[Information sharing with other terminal devices]
(Step S108) Other terminal devices 10 (for example, terminal devices 10-2 and 10-3) acquire the correspondence history stored in step S106 from the server device 20.

Here, the monitoring system 1 provides a function that allows other terminal devices 10 to share the selection result of the response content selected in step S105. For example, if the terminal device 10 that selected the response content in step S105 is terminal device 10-1, other terminal devices 10 (for example, terminal devices 10-2 and 10-3) can obtain that selection result (i.e., the user's response content of terminal device 10-1 to the operation of the emergency button 410).

In other words, event information is transmitted to multiple users, and the display unit 120 displays the status of how one of the multiple users has responded to the event information.

Figure 10 shows an example of the display screen for the correspondence history in this embodiment. The terminal device 10 displays the correspondence history screen P61 acquired from the server device 20. This correspondence history screen P61 includes the name of the user who was corresponded, the time of the correspondence, and the content of the correspondence.

In other words, the display unit 120 of the terminal device 10 displays a screen for selecting from multiple options (option selection screen P51), and then displays the content of the option selected by one of the users.

As described above, multiple destination terminal devices 10 can be set for a single communication device 30. In this case, when the communication device 30 receives event information (for example, first operation information) from the remote control device 40, it transmits the event information to each of the configured terminal devices 10.
At this time, if a user of any terminal device 10 takes some action in response to event information (for example, rushing to the location of the person being monitored in response to the operation of the emergency button 410), this information is shared among the users of multiple terminal devices 10, allowing each user to determine whether further action is necessary. A monitoring system 1 configured in this way can provide users with a sense of security.

[Video display when the emergency button is pressed]
The monitoring system 1 may also provide a function to display images captured by the imaging unit 350 during the time period before and after the operation of the emergency button 410 on the video display screen P62 shown in Figure 10.

Figure 11 shows another example of communication in the emergency response function of this embodiment.
(Step S101) As explained with reference to Figure 6, if any abnormality occurs in the person being monitored, the person being monitored operates the emergency button 410 on the remote control device 40. When the remote control device 40 detects that the emergency button 410 has been operated, it transmits first operation information to the communication device 30.

(Step S201) The communication device 30 of this embodiment continuously performs imaging by the imaging unit 350 and sound collection by the communication device sound collection unit 360, regardless of whether the emergency button 410 is operated or not.
The communication device control unit 330 stores the captured image and the recorded sound in the communication device storage unit 390. Depending on the capacity of the communication device storage unit 390, the communication device control unit 330 may erase information captured and recorded before a predetermined period and store new images and sounds. Here, the predetermined period is, for example, 12 hours, 24 hours, or 7 days.
(Step S202) When the operation of the emergency button 410 is detected in step S101, the communication device 30 stores in the communication device storage unit 390, based on a predetermined recording time, the images and sounds from the most recent period among the past images and sounds stored in the communication device storage unit 390, as well as images and sounds captured and recorded after the operation of the emergency button 410 was detected. For example, if the predetermined recording time is 10 minutes, the communication device 30 stores in the communication device storage unit 390 the images and sounds from the 5 minutes before and 5 minutes after the operation of the emergency button 410 was detected.

(Step S203) The terminal device 10 requests images and audio from the communication device 30 before and after the operation of the emergency button 410 is detected. For example, the user taps the play button image on the video display screen P62 shown in Figure 10. When the terminal device 10 detects the tap on the play button image on the video display screen P62, it requests images and audio from the communication device 30 before and after the operation of the emergency button 410 is detected.
(Step S204) When the communication device 30 receives a request for images and sounds from before and after the operation of the emergency button 410 is detected, it transmits the images and sounds from the time period before and after the operation of the emergency button 410 was detected, which were saved in step S202, to the terminal device 10.
(Step S205) When the terminal device 10 receives images and sounds from the communication device 30 for the time period before and after the operation of the emergency button 410 is detected, it plays back the received images and sounds.

The information indicating the operation of the emergency button 410 (first operation information) is an example of event information. The event information includes information indicating a video within a time range including the time the event occurred, and the display unit 120 displays the video. Here, the time range including the time the event occurred is, for example, 10 minutes (5 minutes before and 5 minutes after the operation of the emergency button 410). Note that the time width and the ratio of before and after shown here are examples; they may be longer or shorter than 10 minutes, and the ratio of before and after time may be equal or unequal, with the before (or after) time being longer. Furthermore, the information indicating the video may be the video file itself, or it may be link information indicating the location where the video is saved.

With the monitoring system 1 configured in this way, the situation before the emergency button 410 was pressed—that is, the situation during the time period when the cause that led to the pressing of the emergency button 410 occurred—can be displayed to the terminal device 10. Therefore, the user can understand the cause of the abnormality that occurred in the monitored person and take more appropriate action.

[Display of emergency button operation history]
Figure 12 shows an example of the event history display in this embodiment. As described above, the terminal device 10 includes a display unit 120 and a terminal communication unit 140.
The terminal communication unit 140 receives event information regarding events that occurred at the monitored location. This event information includes, for example, first operation information indicating that the emergency button 410 was operated.
The display unit 120 displays the history of event information received by the terminal communication unit 140. For example, the display unit 120 of the terminal device 10 displays past events among the notifiable events on the event history screen P71, along with the date and time of their occurrence.
With the monitoring system 1 configured in this way, not only can notifications be sent when an event occurs (for example, when the emergency button 410 is operated), but past events can also be presented to the user.
Furthermore, the terminal device 10 may have a function to exclude (i.e., delete) events selected by the user from the event history from being displayed.
Furthermore, the terminal device 10 may display the timing of events in a calendar format, or display detailed information about events that occurred on a date selected from the calendar.

Here, the event information includes the time the event occurred. The display unit 120 displays the history of the event information based on the time the event occurred.
Here, the display unit 120 can also be configured to display a history of event information based on the time the event information is received by the terminal device 10. However, since the event information is transmitted from the remote control device 40 to the terminal device 10 via the communication device 30 and the server device 20, there may be a discrepancy between the time the event information is received and the time the event occurs, depending on the communication conditions. Also, for example, if multiple events occur, the order in which the events occur and the order in which the event information is received may be reversed.
According to the monitoring system 1 configured to present a history of event information based on the time of event occurrence, as in this embodiment, the event occurrence history can be presented more accurately regardless of the communication status.

Furthermore, as described above, a communication device 30 equipped with an imaging unit 350 is installed in the monitored area A. The event information transmitted to the terminal device 10 (in the example above, information indicating that the emergency button 410 was operated; i.e., first operation information) includes information about the image captured by the imaging unit 350 (for example, a thumbnail image or video). The presentation unit 120 presents this image information.

Furthermore, the terminal device 10 may mark events shown on the event history screen P71 as "unread" (P72) for events whose content has not been viewed by the user, i.e., unread events.
In other words, the display unit 120 may indicate that the event information has not been viewed by the user.

Furthermore, with respect to an unopened event, if a predetermined time has elapsed since the event occurred, the terminal device 10 may indicate to the user that the predetermined time has elapsed by changing the color of the unopened indicator P72, displaying a pop-up image, making a sound, vibrating, etc.
In other words, the display unit 120 indicates that a predetermined amount of time has elapsed since the event information was viewed by the user.

Furthermore, the terminal device 10 may change the color and shape of the unopened indicator P72 depending on the type of event. In other words, the display unit 120 may display different information depending on the type of event indicated by the event information.

Figure 13 shows an example of the event selection screen for this embodiment. The terminal device 10 may narrow down the notifications when an event occurs based on the user's settings on the event selection screen P81. The presentation unit 120 selects the event information to be presented based on at least one of the event occurrence time and event type.

With the monitoring system 1 configured as described above, users can easily understand the situation when an event occurs, and they can take more appropriate action in response to the event.

[Monitoring function (2) Conversation function using remote control device]
The monitoring system 1 provides a conversation function using the remote control device 40. For example, by tapping the conversation icon P43 shown in Figure 8 above, the user can have a conversation with the person being monitored.

As described above, the remote control device 40 is equipped with a remote control sound pickup unit 450. When a conversation request is received from the terminal device 10, the communication device control unit 330 transmits information about the sound picked up by the remote control sound pickup unit 450 to the terminal device 10 based on the second operation information. Here, the second operation information is information indicating that the OK button 420 has been pressed. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information.
Let's explain the more specific steps for using the conversation function.

Figure 14 shows an example of communication using the conversation function of this embodiment.
(Step S301) When the terminal device 10 detects a tap on the conversation icon P43, it sends a conversation request to the communication device 30.
(Step S302) When the communication device 30 receives a conversation request from the terminal device 10, the communication device sound-emitting unit 370 emits a conversation request sound. The person being monitored hears the conversation request sound, realizes that a conversation has been requested, and decides whether or not to allow the conversation. If the person being monitored allows the conversation, they operate the OK button 420 on the remote control device 40.
(Step S303) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated. Here, the information indicating that the OK button 420 has been operated is also called the second operation information. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information.

(Step S304) When the terminal device 10, server device 20, communication device 30, and remote control device 40 detect that permission to the conversation request has been granted in step S303, they start a two-way conversation. Here, the two-way conversation function is a function that enables two-way conversation between the user and the person being monitored by having the sound picked up by the terminal sound pickup unit 150 of the terminal device 10 be emitted from the communication device sound output unit 370 of the communication device 30, and having the sound picked up by the remote control sound pickup unit 450 of the remote control device 40 be emitted from the terminal sound output unit 160 of the terminal device 10. In the two-way conversation function, the monitoring system 1 may display the image of the person being monitored during the conversation, captured by the imaging unit 350, on the terminal device 10.
(Step S305) The communication device 30 captures an image of the monitored area A using the imaging unit 350.
(Step S306) The remote control device 40 receives the voice of the person being monitored using the remote control sound receiving unit 450.
(Step S307) The remote control device 40 transmits the collected audio as audio information to the communication device 30.
(Step S308) The communication device 30 transmits the image captured in step S305 and the audio information received from the remote control device 40 in step S307 to the terminal device 10.
(Step S309) The terminal device 10 presents the image and sound received from the communication device 30 to the user.
(Step S310) In parallel with steps S305 to S309, the terminal device 10 transmits the sound picked up by the terminal sound pickup unit 150 as voice information to the communication device 30.
(Step S311) The communication device 30 receives voice information from the terminal device 10 and emits it from the communication device sound-producing unit 370.

As described above, the monitoring system 1 uses the remote control sound pickup unit 450 of the remote control device 40 carried by the person being monitored to pick up the person's voice. Therefore, compared to, for example, using the communication device sound pickup unit 360 of the communication device 30 to pick up the voice, the voice can be picked up more clearly. Furthermore, the monitoring system 1 uses the communication device sound output unit 370 of the communication device 30 to produce sound. Therefore, compared to, for example, using the remote control sound output unit 460 of the remote control device 40, the sound can be produced at a louder volume or with better sound quality. Thus, the monitoring system 1 can improve the quality of conversations between the user and the person being monitored.

As described above, the remote control device 40 is equipped with an OK button 420. This OK button 420 is a different operation button from the emergency button 410. The remote control device 40 transmits second operation information indicating the operation of the OK button 420. The communication device control unit 330 of the communication device 30 controls the type of information to be transmitted to the terminal device 10 based on the receipt of the second operation information from the remote control device 40. In the example described above, when the OK button 420 is operated, the communication device control unit 330 controls the type of information to be transmitted to the terminal device 10 by transmitting an image and sound to the terminal device 10.

[Monitoring function (3) Live view function]
The monitoring system 1 provides a live view function that displays the current image and sound of the monitored area A to the terminal device 10 in real time. In the live view function, the terminal device 10 displays, for example, the live view image P41 shown in Figure 8 above.
In the aforementioned conversation function, the remote control sound pickup unit 450 of the remote control device 40 picked up the sound, but in the live view function, the communication device sound pickup unit 360 of the communication device 30 picked up the sound, which is a difference from the conversation function described above.
Furthermore, the live view function operates differently in private mode and public mode. In private mode, permission from the monitored person is required. An example of this procedure is described below.

<Live View Function: Private Mode>
Figure 15 shows an example of communication for the live view function in the private mode of this embodiment.
(Step S401) When the terminal device 10 detects an operation in which the user requests the live view function (for example, selecting the "Live" tab or the "Current Status" tab), it sends a live view request to the communication device 30.
(Step S402) When the communication device 30 receives a live view request from the terminal device 10, the communication device sound unit 370 emits a live view request sound. The person being monitored hears the live view request sound, realizes that a live view request has been made, and decides whether or not to grant permission for live view. If the person being monitored grants permission for live view, they operate the OK button 420 on the remote control device 40.
(Step S403) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated. Here, the information indicating that the OK button 420 has been operated is also called the second operation information. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information.
In addition, the remote control device 40 also transmits second operation information in the conversation function described above.
Depending on the required function, the meaning of the operation of the OK button 420, that is, the meaning of the second operation information, changes.

(Step S404) The communication device 30 captures an image of the monitored area A using the imaging unit 350. The communication device 30 also captures sound from the monitored area A using the sound collection unit 360.
(Step S405) The communication device 30 transmits the image and audio information to the terminal device 10.
(Step S406) The terminal device 10 presents the image and sound received from the communication device 30 to the user.

<Live View Function: Public Mode>
Figure 16 shows an example of communication for the live view function in public mode of this embodiment. In public mode, the communication device 30 continuously captures images of the monitored area A using the imaging unit 350.

(Step S501) When the terminal device 10 detects an operation by the user requesting the live view function (for example, selecting the "Live" tab or the "Current Status" tab), it sends a live view request to the communication device 30.
(Step S502) When the communication device 30 receives a live view request from the terminal device 10, the communication device sound unit 370 emits a live view request sound. The person being monitored hears the live view request sound, realizes that a live view request has been made, and decides whether or not to grant permission for live view. If the person being monitored grants permission for live view, they operate the OK button 420 on the remote control device 40.
(Step S503) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated.

(Step S504) The communication device 30 collects sound from the area A to be monitored using the communication device sound collection unit 360.
(Step S505) The communication device 30 transmits the image and audio information to the terminal device 10.
(Step S506) The terminal device 10 presents the image and sound received from the communication device 30 to the user.

A live view request is also called an image transmission request. That is, when an image transmission request is received from the terminal device 10, the communication device control unit 330 of the communication device 30 transmits the image captured by the imaging unit 350 to the terminal device 10 based on the second operation information.

Here, the differences in the operation of monitoring system 1 between public and private modes will be explained with reference to the diagram.

Figure 17 shows an example of the operation of the monitoring system 1 in public mode and private mode according to this embodiment. In Figure 17, "OFF" indicates that the button is not operated, and "ON" indicates that the button is operated.
Regarding images, in public mode, the image captured by the imaging unit 350 is displayed to the terminal device 10 regardless of whether the emergency button 410 or the OK button 420 is operated. On the other hand, in private mode, the image captured by the imaging unit 350 is displayed to the terminal device 10 when the emergency button 410 is operated, or when the OK button 420 is operated in response to a conversation request or live view request (i.e., when the person being monitored gives permission). In other words, in private mode, if neither the emergency button 410 nor the OK button 420 is operated (in the OFF position), no image is displayed.
Regarding audio, in both public and private modes, the recorded audio is presented to the terminal device 10 when the emergency button 410 is operated, or when the OK button 420 is operated in response to a conversation request or live view request (i.e., when the person being monitored gives permission). In other words, regarding audio, in both public and private modes, if neither the emergency button 410 nor the OK button 420 is operated (i.e., OFF), the audio is not presented.

Regarding audio, in both public and private modes, when the emergency button 410 is operated, the sound picked up by the communication device sound pickup unit 360 of the communication device 30 is displayed. Also, in both public and private modes, when the OK button 420 is operated (i.e., when the monitored person gives permission), the sound picked up by the remote control sound pickup unit 450 is displayed in the case of a conversation request, and the sound picked up by the communication device sound pickup unit 360 is displayed in the case of a live view request.

As described above, the meaning of the information indicating the operation of the OK button 420, that is, the second operation information, changes depending on the content of the user's request and the operating mode.
The second operation information is information indicating an operation that authorizes providing the terminal device 10 with at least one of the sound information and the captured image, in a non-public mode in which the image captured by the imaging unit 350 is not made public.

Furthermore, the second operation information indicates an operation that authorizes the provision of the sound information acquired by the imaging unit 350 to the terminal device 10 in the public mode, where the image captured by the imaging unit 350 is made public.

[Monitoring function (4) Voltage drop notification function for remote control device]
Figure 18 shows an example of communication in the voltage drop notification function of this embodiment.
(Step S601) The voltage detection unit 470 of the remote control device 40 detects the power supply voltage of the remote control device 40. If the detected power supply voltage is below a predetermined value, the remote control control unit 440 sends a voltage drop notification to the terminal device 10. That is, when the communication device control unit 330 receives voltage drop information from the remote control device 40, it sends information to the terminal device 10 indicating that the power supply voltage of the remote control device 40 has dropped.
(Step S602) The terminal device 10 receives a voltage drop notification.
(Step S603) The terminal device 10 notifies the user of the voltage drop.

Figure 19 shows an example of the voltage drop notification screen of this embodiment. In step S603, the terminal device 10 displays the voltage drop notification screen P91 shown in Figure 19.
Furthermore, if the power supply voltage of the remote control device 40 is restored by replacing the batteries or charging the device, the remote control device 40 may send a voltage restoration notification to the communication device 30 or terminal device 10. The communication device 30 or terminal device 10 may manage the period from when it receives the voltage drop notification until when it receives the voltage restoration notification as the power supply voltage drop period of the remote control device 40.

[Monitoring function (5) Remote control device search function]
The monitoring system 1 provides a remote control device search function that, if the remote control device 40 is lost, sounds the remote control device 40 to assist in the search for the remote control device 40.

Figure 20 shows an example of communication in the remote control device search function of this embodiment.
(Step S701) The communication device 30 is equipped with a remote control device search switch (not shown). The communication device 30 detects the operation of the remote control device search switch. This remote control device search switch is an example of an operation receiving unit that receives an operation to search for the remote control device 40.
(Step S702) When the communication device 30 detects that the remote control device search switch has been operated, it transmits a remote control device search instruction to the remote control device 40.
(Step S703) When the remote control device 40 receives a remote control device search instruction from the communication device 30, it emits a sound from the remote control sound-emitting unit 460 to notify the person being monitored or the user of its location.

Furthermore, the remote control device 40 may be equipped with a vibrating unit (not shown) and a light-emitting unit (not shown), and may notify its location by vibration or light when it receives a remote control device search instruction from the communication device 30.

[Monitoring function (6) Remote control device out of range notification function]
The monitoring system 1 provides a remote control device out-of-range notification function that notifies the terminal device 10 when the remote control device 40 is outside the communication range of the communication device 30.

Figure 21 shows an example of communication in the remote control device out-of-range notification function of this embodiment.
(Step S801) The communication device 30 determines whether or not it can communicate with the remote control device 40. The communication device 30 may perform this communication feasibility determination periodically or in response to a request from the terminal device 10. If the communication device 30 cannot communicate with the remote control device 40, it determines that the remote control device 40 is outside the communication range and transmits remote control range out-of-range information to the terminal device 10.
Furthermore, the communication device 30 may, in combination with the voltage drop notification function of the remote control device described above, determine that the remote control device 40 is outside the communication range if the power supply voltage of the remote control device 40 has not dropped and communication with the remote control device 40 is not possible.

(Step S802) The terminal device 10 receives remote control out-of-range information from the communication device 30.
(Step S803) The terminal device 10 informs the user that the remote control device 40 is outside the communication range of the communication device 30.

As explained above, monitoring system 1 can provide a highly convenient monitoring system for both monitors and non-monitored individuals.

While embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and modifications can be made as appropriate without departing from the spirit of the invention. Combinations of the configurations described in the above embodiments may also be used.

Furthermore, each component of the device in the above embodiment may be implemented by dedicated hardware, or it may be implemented by memory and a microprocessor.

Furthermore, each component of the device may consist of memory and a CPU (Central Processing Unit), and its functions may be realized by loading and executing programs into memory to implement the functions of each component.

Alternatively, programs for realizing the functions of each part of each device may be recorded on a computer-readable recording medium, and the programs recorded on this recording medium may be read by a computer system and executed to perform processing by each part of the control unit.
In this context, "computer system" includes hardware such as the operating system and peripheral devices.

Furthermore, "computer system" shall also include the homepage provisioning environment (or display environment) if the WWW system is being used.
Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. In addition, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system.

1…Monitoring system, 10…Terminal device, 110…Terminal operation detection unit, 120…Display unit, 130…Terminal control unit, 140…Terminal communication unit, 150…Terminal sound pickup unit, 160…Terminal sound generation unit, 30…Communication device, 310…First communication unit, 320…Second communication unit, 330…Communication device control unit, 340…Measurement unit, 350…Imaging unit, 360…Communication device sound pickup unit, 370…Communication device sound generation unit, 380…Operation detection unit, 390…Communication device memory unit, 40…Remote control device, 410…Emergency button, 420…OK button, 430…Remote control communication unit, 440…Remote control unit, 450…Remote control sound pickup unit, 460…Remote control sound generation unit, 470…Voltage detection unit, 480…NG button

Claims (13)

A receiving unit that receives an image captured by at least one of the first communication device or the second communication device, each of which is equipped with an imaging unit ,
A transmission unit that transmits the aforementioned image to a terminal device used by the user,
A storage unit that stores the user's viewing rights to the image, which are set for each of the first and second communication devices,
Equipped with,
A server device that displays content based on the access permissions to the terminal device . A receiving unit that receives an image captured by at least one of the first communication device or the second communication device, each of which is equipped with an imaging unit,
A transmission unit that transmits the aforementioned image to a terminal device used by the user,
A storage unit that stores the user's viewing rights to the image, set for each of the first location where the first communication device is installed and the second location where the second communication device is installed,
Equipped with,
The terminal device is a server device on which content based on the aforementioned viewing permissions is presented. The aforementioned access rights include at least two levels of access rights.
The server device according to claim 1 or 2. The aforementioned viewing rights include a first viewing right that does not restrict viewing of the image, and a second viewing right that restricts viewing of the image.
The server device according to claim 3. The first viewing permission is a viewing permission that allows real-time viewing of the image,
The second viewing permission is a viewing permission that does not allow real-time viewing of the image.
The server device according to claim 4. The aforementioned viewing permission includes not granting viewing permission to the user.
The server device according to claim 3. At least one of the first communication device or the second communication device includes a detection unit that detects the occurrence of an event based on an image captured,
The receiving unit receives event information relating to the event detected by the detection unit.
The transmitting unit transmits the event information to the terminal device.
The server device according to claim 1 or 2. The event information includes the type of the event,
The server device according to claim 7. The event information transmitted from the transmitting unit is displayed on the terminal device.
Based on the type of event information, the terminal device can select the event information to be presented.
The server device according to claim 8. A plurality of communication devices, including a first communication device and a second communication device, each equipped with an imaging unit,
A communication unit that transmits an image captured by at least one of the first communication device or the second communication device to a terminal device used by the user,
A storage unit that stores the user's viewing rights to the image, which are set for each of the first and second communication devices,
Equipped with,
A system in which the terminal device is presented with content based on the aforementioned viewing permissions. A plurality of communication devices, including a first communication device and a second communication device, each equipped with an imaging unit,
A communication unit that transmits an image captured by at least one of the first communication device or the second communication device to a terminal device used by the user,
A storage unit that stores the user's viewing rights to the aforementioned image,
Equipped with,
The aforementioned access rights are set for the first location where the first communication device is installed and for the second location where the second communication device is installed.
A system in which the terminal device is presented with content based on the aforementioned viewing permissions. A program executed by a computer, wherein the computer
A receiving unit that receives images captured by at least one of a first communication device and a second communication device, each having an imaging unit,
A communication unit that refers to the access permissions set for the first communication device and the second communication device,
A display unit where content based on the aforementioned viewing rights is presented,
A program that makes something function as such. A program executed by a computer, wherein the computer
A receiving unit that receives images captured by at least one of a first communication device and a second communication device, each having an imaging unit,
A communication unit that refers to the access rights set for the first location where the first communication device is installed and the second location where the second communication device is installed,
A display unit where content based on the aforementioned viewing rights is presented,
A program that makes something function as such.