JP7634901B2 - Systems and programs - Google Patents

Description

The present invention relates to a system and a program for processing images captured by a camera or the like attached to a vehicle or the like.

The following Patent Document 1 discloses a system applicable to drive recorders and the like. This system repeats a cycle of recording video data in a recording area. When a predetermined cycle end condition is met, the currently ongoing cycle ends, and a new cycle begins in which the video data recorded in the ended cycle is overwritten with new video data in the oldest first order. Based on at least one of the capacity of the video data recorded in the currently ongoing cycle and the capacity of the video data recorded in the immediately preceding cycle that has not been overwritten,
The information on the overwriting position is output to the output means.

JP 2018-6969 A

Images recorded by a drive recorder are effectively used to identify the cause of an accident, etc. It is desired that images captured by a camera mounted on a vehicle, etc., be used more effectively in identifying the cause of an accident than ever before, and not limited to identifying the cause of an accident.

The object of the invention of this application is not limited to this, and the intention is to obtain rights for configurations that aim to obtain effects from the configuration parts disclosed in this specification and drawings, etc., through divisional applications, amendments, etc. For example, this specification discloses a problem in which the phrase "can be" is read as "the problem is." Each problem is described as an independent one, and the configuration for solving this problem may also be filed separately through divisional applications, amendments, etc.
The applicant intends to obtain rights by amending the specification, etc. Even if the problem is implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in the specification in the scope of the claims in an amendment or divisional application. Problems that combine these independent problems are also disclosed.

(1) The system may have a processing means having a function of recording images captured by a camera attached to a vehicle, and a function of performing image analysis of the images captured by the camera and, based on the analysis results, executing processing corresponding to the analysis results.

Rather than simply recording images captured by a camera mounted on a vehicle, the images can be utilized more effectively.

Examples of the vehicle include a passenger car, a freight vehicle, a special vehicle, etc.
For example, it is preferable to capture at least one of the interior and exterior of the vehicle.
"Outside the vehicle" can be interpreted as follows, for example:

When the interior and exterior spaces of a vehicle are clearly separated by the vehicle body, the interior of the body is referred to as the "interior" and the exterior of the body is referred to as the "exterior". For example, in the case of a van in which the passenger space and the cargo space are continuous, the continuous passenger space and cargo space are referred to as the interior. In the case of a sedan in which the passenger space and the cargo space are separated, both the passenger space and the cargo space are referred to as the interior. In a covered freight vehicle with a sealed cargo bed, the passenger space and the interior of the sealed cargo bed are referred to as the interior. In an open-top freight vehicle (flatbed vehicle) with an open cargo bed, it is difficult to clearly separate the space that accommodates the cargo on the cargo bed from the space around it, but the space that accommodates the open cargo is also referred to as the interior. The passenger space of an open-top freight vehicle is also referred to as the interior.
In some special vehicles such as forklifts, both the passenger space and the cargo space are open, and it may be difficult to clearly distinguish the passenger space and the cargo space from the surrounding space. Even in such cases, both the passenger space and the cargo space can be called the interior of the vehicle. For example, the space above the seat where the passenger sits and the space above the forks that hold and lift the cargo can be called the interior of the vehicle. In this way, the space that can be considered to move with the vehicle as it moves can be called the interior of the vehicle, and the surrounding space can be called the outside of the vehicle.

As the "camera mounted on the vehicle", a camera whose relative position and attitude with respect to the vehicle is fixed in a normal use state may be used. Also, as the "camera mounted on the vehicle", a hemispherical or omnispherical 360-degree camera may be used. When a hemispherical or omnispherical 360-degree camera is used, images of a wider range can be acquired compared to when a normal camera is used. For example, images of both the inside and outside of the vehicle can be acquired with one camera. Note that, instead of the 360-degree camera, multiple normal cameras may be installed on the vehicle.

The image captured by the camera may be a moving image. When the image captured by the camera is a moving image, the processing means can easily determine the movement of the subject, the presence or absence of the subject before and after a certain point in time, etc. by performing image analysis.

The function of recording the images captured by the camera may be realized by a server that communicates with the vehicle-mounted device of the vehicle equipped with the camera, but it is more preferable to perform the function in the vehicle-mounted device. A removable medium such as an SD card can be used as the recording medium. When a removable medium is used as the recording medium, the removable medium on which the images are recorded can be removed from the vehicle-mounted device and the images can be displayed on another terminal such as a personal computer.

The function of analyzing the images captured by the camera may be realized by a server that communicates with the in-vehicle device on which the camera is mounted, but it is more preferable to perform the function in the in-vehicle device. When the in-vehicle device analyzes the images, the process corresponding to the analysis results can be quickly executed in the vehicle.

The processing means may be configured as one device or as multiple devices. For example, a drive recorder has a function of recording images captured by a camera. The function of the processing means of the present system may be realized by a microcomputer built into the drive recorder and an application program stored in the memory. Also, for example, the processing means may be configured as two devices, an in-vehicle device and a server, with the in-vehicle device recording images and the server analyzing images.

The system may also have a processing means having a function of recording images captured by a camera attached to an object other than the vehicle, and a function of analyzing the images captured by the camera and executing a process corresponding to the analysis result based on the analysis result. However, it is particularly preferable to use the method as in (1).

(2) The processing means may be configured as a system as described in claim 1, which has a function of image analysis to determine whether the movement of a person in an image corresponds to a specific movement registered in advance, and when the movement of the person corresponds to the specific movement, executes at least one of the following processes corresponding to the analysis result: recording an image capturing the specific movement of the person, transmitting the image capturing the specific movement to a server, reporting that the person has made the specific movement, and issuing an alarm.

For example, the recorded image or the image transmitted to the server can be used as evidence that a person has made a certain movement. For example, by reporting that a person has made a certain movement, an outsider can be informed that there is a person making a certain movement. For example, by issuing an alarm, a person who has been making a certain movement can be motivated to stop making the certain movement by noticing that an alarm has been issued to him/her.

The person performing a specific movement may be, for example, a driver of a vehicle.
For example, the actions may be actions other than driving. For example, the actions may be actions of looking at a mobile device such as a smartphone, eating food, looking away or distracted, or falling asleep.
The presence or absence of distraction may be determined based on the detection result of the driver's eye position. The recorded image may be used to improve the driver's awareness, analyze the cause of accidents, etc.
When it is detected that the driver is operating a mobile device such as a smartphone, an alarm may be issued to warn the driver not to carry the mobile device while driving. If the driver notices the alarm and stops the specific movement, danger can be avoided in advance. The server to which the data is transmitted may be the server of the vehicle manager or the server of the police. The vehicle manager can evaluate the driver's driving behavior based on the transmitted data. The police can evaluate the driver's driving behavior and analyze the cause of the accident based on the transmitted image.

A person making a specific movement may be, for example, a suspicious person outside the vehicle, and a specific movement may be, for example, an attempt to break into the vehicle, such as an attempt to break a glass window with a hammer, or an attempt to look into the vehicle. By issuing an alarm, a suspicious person attempting to break into the vehicle may be deterred from doing so. The recorded image or the image transmitted to the server may be used as useful information for arresting a thief after a crime involving vehicle burglary has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver may immediately return to the vehicle, which may prevent damage from vehicle burglary. The driver or owner who receives the notification may also report the crime to the police. In this way, this function may be useful for crime prevention.

A person who makes a specific movement may be, for example, a suspicious person in a vehicle, and a specific movement may be various methods of vehicle theft, such as illegal entry into a parked vehicle, a suspicious person's movement in a parked vehicle, or the removal of a device in the vehicle. If an alarm is issued and the thief retreats in a hurry, the damage from the vehicle theft is reduced. The recorded image or the image transmitted to the server can be used as useful information for arresting the thief after the vehicle theft crime occurs. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle or report the crime to the police, thereby reducing the damage from the vehicle theft. An example of reporting to the outside is sending an e-mail with an image of a suspicious person attached. The destination of the e-mail may be the manager, driver, owner, police, etc. of the vehicle.

The person making a specific movement may be, for example, a passenger in a commercial vehicle carrying unspecified people, and the specific movement may be various robbery methods, such as brandishing a knife, holding a gun, or using violence. By executing a process to issue an alarm by the processing means, the robber will be deterred from the crime, and damage can be prevented. The recorded image or the image transmitted to the server can be used as useful information for arresting the robber. By notifying the manager of the vehicle that a person has made a specific movement, the manager can notice that a robbery has occurred and notify the police. In addition, the processing means may have a function of notifying the police using a communication function.

An example of a person making a particular movement may be an infant riding in the car, and an example of a particular movement may be the movement when waking up from sleep. When the image analysis detects that the infant has woken up, the processing means may notify the guardian by e-mail or the like. There may be cases where an infant falls asleep in the car while driving, and the driver wants to go shopping or run an errand with the engine running. In such cases, the infant will get upset if they are woken up forcefully, so the guardian would prefer to leave the infant asleep. However, the guardian may want to wake the infant up when they are away from the vehicle,
If the parent uses this system, an e-mail is sent to the parent's mobile device when the baby wakes up, allowing the parent to immediately rush to the vehicle and deal with the situation.

The person making the specific movement may be, for example, a person requiring care outside the vehicle or a person requiring rescue, and the specific movement may be, for example, a movement specific to a person requiring care or a movement corresponding to a situation in which rescue is likely to be required. For example, an example of a movement specific to a person requiring assistance may be a movement of walking unsteadily with a bent back while leaning on a cane. An example of a movement corresponding to a situation in which rescue is likely to be required may be a movement corresponding to a situation in which a person has collapsed and foamed at the mouth.

The function of detecting a person performing a specific movement may include a function of selecting at least one of an image of the front of the vehicle, an image of the rear of the vehicle, and an image of the interior of the vehicle, and the person in question may be detected from the selected image. The selection of at least one image may be specified by the user. This prevents the processing means from analyzing unnecessary or less important images, and therefore eliminates the need for the processing means to have excessive processing power.

It would be good to add a function to detect the driver's drowsiness to the function to detect a person making a specific movement. For example, the driver's pulse rate is measured from the degree of change in the redness of the driver's skin, and based on the change in pulse rate,
It is advisable to determine whether the driver is drowsy or not. It is advisable to improve the accuracy of drowsiness detection by combining body temperature measurement with a thermal camera and pulse measurement with microwaves. If the processing means of this system detects drowsiness, it is advisable to lower the air conditioner's temperature setting or issue a warning by voice or sound.

(3) The processing means may have, as a function of the image analysis, a function of determining whether or not a person in the image has a specific characteristic that has been registered in advance, and may be a system that executes processing according to the specific characteristic possessed by the person in the image as processing corresponding to the analysis result.

Appropriate processing can be performed depending on the characteristics of the person captured in the image.

The function of determining whether a person in an image has a specific feature may include, for example, a face recognition function. For example, face recognition of a person sitting in the driver's seat may be performed, and if the person sitting in the driver's seat does not correspond to a person registered in advance, the processing means may access the vehicle and prevent the engine from starting. For example, the processing means may transmit a command to an engine control unit (ECU)
) to prohibit starting the engine. This allows the person who can drive the vehicle to be registered in advance, and prohibits unregistered people from driving the vehicle. As a result, the crime prevention function of the vehicle can be improved. It is also effective as a measure against children playing with the vehicle. In addition to the process of prohibiting the start of the engine, the accelerator pedal can be prevented from being depressed,
It may also be possible to perform processing such as preventing the handbrake (parking brake) from being released, preventing the shift lever from being placed in the drive position, sounding an alarm, and sending an email with an image to a registered email address.

In addition, for example, face recognition of a person outside the vehicle may be performed, and if the person outside the vehicle corresponds to a person registered in advance, the processing means may control the vehicle to unlock the door. For example,
If the vehicle is provided for a car sharing service, it is advisable to register members of the car sharing service in advance. Registered members can use the vehicle without carrying a card key or the like by approaching the vehicle and moving their face into the field of view of the vehicle's camera. This makes the vehicle of the car sharing service easier to use.

The vehicle is a commercial vehicle in which unspecified persons board, and the processing means may determine whether the passenger is a wanted criminal or not by a facial recognition function. A facial photograph of the wanted criminal may be transmitted from a police database to the processing means of the system by communication via LTE, wireless LAN, or the like, and may be registered in the processing means in advance. If the processing means determines that the passenger is a wanted criminal, it may notify the manager of the vehicle, the police, or the like, via a communication device and a communication network. Furthermore, the processing means may notify the driver in a manner that is not apparent to the passenger. The manager of the vehicle, the police, the driver, etc., may recognize that a wanted criminal has boarded the vehicle and may take appropriate action promptly.

For example, since taxis are closed spaces with only the driver and passengers aboard, this system is likely to be in high demand in terms of ensuring the safety of the driver. When the processing means detects that a criminal has boarded a taxi, it is advisable to notify the taxi company and the police of the criminal's boarding. In addition, this system can detect the criminal's boarding at the time he or she gets into the vehicle before he or she actually commits a crime inside the vehicle. This makes it possible to prevent crimes from occurring.

The function of determining whether or not a person in an image has a specific characteristic may include, for example, a function of comparing the appearance of the person in the image with a specific appearance registered in advance and determining whether or not the two match (hereinafter referred to as an appearance authentication function). As the specific appearance, it is preferable to register in advance appearances specific to wanderers, appearances specific to criminals, etc. Examples of appearances specific to criminals are as follows:
For example, carrying a knife on the street, carrying a dangerous object, etc. In addition, it is preferable to combine images from an infrared camera, an ultraviolet camera, etc. with images from a normal camera to detect people carrying illegal drugs. When the processing means detects a person with a specific appearance in the image, it is preferable to notify the police or the vehicle manager via the communication device and communication network.

When it is desired to find a specific missing person or wanderer, for example, the appearance of the person at the time of going missing can be individually registered as search information in the processing means. For example, individual search information such as gender, height, type of clothing, color of clothing, whether or not the person is wearing a hat, whether or not they are carrying a cane, and whether or not they are bent over can be registered. In addition, the area where the person is thought to be wandering (search target area) can be registered. When the vehicle is located in the search target area, the processing means can execute a process to detect people matching the individual search information. Such individual search information can be transmitted to the processing means of this system from a database (server) of the police, a local support center, etc. The on-board device can perform a function of determining whether or not a person in an image has a specific characteristic,
When the vehicle enters the search area, search information may be downloaded from the server to the vehicle's processing means.

The function of determining whether a person in an image has a specific feature may include a function of determining whether a person sitting in the passenger seat or the back seat is wearing a seat belt (hereinafter referred to as an appearance recognition function). When the processing means of the present system detects a person not wearing a seat belt, it may issue a message urging the person to wear a seat belt.

Especially when the person in the back seat is not wearing their seat belt and they are approaching the entrance to the highway.
A message encouraging the driver to fasten his/her seat belt may be issued. For example, when the distance from the vehicle's current position to the expressway entrance becomes equal to or less than a reference distance, it may be determined that the vehicle is approaching the expressway entrance. In addition, when a signal for checking if an ETC card has been forgotten to be inserted is received, it may be determined that the vehicle is approaching the expressway entrance. In addition, when an image of the front of the vehicle is analyzed and the expressway entrance is detected in the image, it may be determined that the vehicle is approaching the expressway entrance.

(4) The processing means may be a system that has a function of counting the number of people in an image as a function of the image analysis, and executes a process of recording or transmitting data on the number of people in the image as a process corresponding to the analysis result.

The present system can count the number of people without using any special device for counting the number of people.

For example, a camera may be attached to a bus, and an image of the inside of the vehicle may be used as the image on which the number of people is counted. The processing means may display the counting result and inform the driver. The driver can easily grasp the number of passengers (passengers) on board the vehicle. This function may be used in place of a roll call for tours. When all passengers should disembark and the passenger count result does not reach zero, the driver can know that there is someone left behind on the vehicle. For example, it may be possible to detect people who are unable to express their will, such as babies, or people who are unable to disembark on their own, such as disabled people, who are left behind.

For example, the vehicle on which the camera is installed may be a type of vehicle with a predetermined number of passengers, such as a tourist bus or a long-distance express bus, and the predetermined number of passengers may be registered in the processing means of the present system. The processing means may compare the number of passengers counted with the registered number, and if there is an excess or shortage, may warn the driver of the surplus or shortage. This prevents the vehicle from departing with an insufficient number of passengers. It may also be possible to easily determine if an unauthorized person has boarded the vehicle and encourage them to disembark.

For example, a camera (e.g., a hemispherical camera on the ceiling) may be installed inside a route bus, and the processing means may perform image analysis to count the number of passengers on board, standing passengers, and seated passengers.Furthermore, the processing means may acquire the current location information of the bus, the bus door opening/closing signal, the bus alighting buzzer, the touch of an IC card when getting on or off the bus, and the signal from the fare box, and record each of these as a trigger.This information may be stored in a data lake,
It will be possible to perform big data analysis to find out where passengers got on and off, and what the seating situation was in the car during that time. Such data can be valuable information for consulting companies that handle urban planning, etc. Furthermore, it would be a good idea to collect information transmitted over in-vehicle networks such as CAN, as well as the driver's vital signs. When an abnormality occurs in a vehicle, this information can be used to determine the causal relationship, such as whether the abnormality is due to driving factors, passenger factors, driver factors, or vehicle factors.

For example, an image of the outside of the vehicle may be used as the image on which the number of people is counted. The processing means of the present system may record the counting results as number of people data. Based on the recorded number of people data, locations where there were many people can be easily extracted. The counting results may be used as a trigger to record the image on which the counting was based. For example, an image may be recorded when the number of people is equal to or greater than a predetermined threshold value.

(5) The processing means may have, as its image analysis function, a function of detecting the display content of a display device of the vehicle or the operation status of an operating device of the vehicle from images inside the vehicle, and may execute processing corresponding to the analysis results in accordance with the display content of the display device or the operation status of the operating device.

The present system can acquire data on which the display contents of the display device are based, or data corresponding to the operation status of the operating device, without retrofitting such data. For example, such data can be acquired without connecting an additional device to the OBD connector. In particular, even if the vehicle manufacturer prohibits connecting third-party products to the OBD connector, the present system can acquire such data. The display devices may include various gauges, warning lights in the vehicle, a head-up display that projects images on the windshield, etc.

The display content of the display device to be detected may be, for example, the vehicle speed displayed on a speedometer. The processing means may acquire vehicle speed data by performing pattern recognition on the position of the needle of an analog speedometer and character recognition on the numbers on the dial. Also, the processing means may acquire vehicle speed data by performing character recognition on the numbers displayed on a digital speedometer. The acquired vehicle speed data may be used for various processes. An example of a usage form of the vehicle speed data will be described below.

The processing means may have a function of automatically detecting the speedometer by image recognition. This function enables the processing means to detect the speedometer in various vehicles, regardless of the vehicle type. In addition, the position and shape of the speedometer in the image of the vehicle in which the processing means is installed may be registered in advance in the processing means. This allows the processing means to easily detect the speedometer.

A stand-alone car navigation device uses GPS position information to obtain the current position. In cases where the GPS signal cannot be received, such as in a tunnel, the current position of the vehicle must be determined by another method. As an example, the current position of the vehicle in the map is moved for five minutes at a constant speed of about 80% of the speed just before the GPS signal was lost. The mode in which such processing is performed is called tunnel mode. The tunnel mode is effective not only when the vehicle enters a tunnel, but also when the vehicle is traveling under an elevated road. However, if the vehicle encounters a traffic jam while processing in tunnel mode, the error in the current position becomes excessively large in a method that assumes that the vehicle is traveling at a constant speed, and the method is practically useless. If an acceleration sensor is attached to the vehicle and acceleration data from the acceleration sensor is used, the accuracy of the current position information in tunnel mode can be improved. However, it is difficult and expensive for the user to attach an acceleration sensor, so this is not practical.

It is advisable to read the vehicle speed from the speedometer display and use the map matching function implemented in the car navigation system to update the vehicle's current location information based on the read speed, thereby improving the accuracy of the vehicle's current location information.

In a place where a road inside a tunnel and a ground road run parallel to each other, or where an elevated road and a ground road under the elevated road run parallel to each other, it is preferable to select which road to match based on whether or not the tunnel mode is activated. For example, when the tunnel mode is activated, the vehicle position may be mapped to the road inside the tunnel or the ground road under the elevated road, and when the tunnel mode is not activated, the vehicle position may be mapped to the ground road or the elevated road running parallel to the road inside the tunnel.

The display contents of the display device to be detected may be, for example, the remaining fuel displayed on the fuel gauge. The processing means may detect the remaining fuel by performing image recognition of the fuel gauge. When the processing means detects that the remaining fuel falls below a predetermined reference value, it may notify the driver of a message encouraging refueling. For example, by linking with the current location information of the vehicle, when the remaining fuel is less than a reference value, for example, 1/4 of a full tank, when the vehicle returns from an outside location to near the vehicle storage area at home or at the company, the processing means may issue a warning and encourage refueling. In addition, by linking a car navigation device with this system, the processing means may compare the distance to the destination with the remaining fuel, and encourage refueling at the time of departure or on the way to the destination based on the comparison result. For example, it may output a message encouraging refueling at a gas station near the current location near a gas station on the way.

The function of extracting the speedometer and fuel gauge from the image may be realized by artificial intelligence (AI) installed in the processing means of the present system. If it is difficult to extract the speedometer and fuel gauge from the image using artificial intelligence, the positions of the speedometer and fuel gauge in the image and image information may be registered in advance in the processing means.

The processing means reads a fuel gauge of a vehicle provided to the car sharing service,
When it is detected that the remaining fuel level is less than a predetermined reference value, the driver may be notified that refueling will allow him/her to receive benefits from the car sharing service. For example, a message such as "If you refuel now, you can receive a discount equivalent to 15 minutes" may be issued.

The display contents of the display device to be detected may be, for example, whether or not a warning light in the instrument panel is on. The warning lights to be detected may include a headlight indicator, a turn signal indicator, a high beam indicator (high beam warning light), a parking brake indicator, a half-open door warning light, a fuel warning light, etc.
The processing means of the present system can detect the vehicle state corresponding to the warning lights by determining whether or not these warning lights are on.

The display content of the display device to be detected may be, for example, warning information displayed by a head-up display. The processing means may capture changes in the image displayed by the head-up display to detect the warning content and execute a predetermined process according to the warning content. For example, the warning may be issued by sound or voice. By converting a visual warning into auditory information and issuing the warning, it is possible to obtain the effect of making the driver more aware of the warning.

For example, in cooperation with brightness information outside the vehicle and current vehicle position information, the processing means of the present system may notify the driver to turn on the headlights when it detects that the headlight indicator is not on even though the headlights should be turned on. Situations in which the headlights should be turned on include, for example, a situation in which the outside of the vehicle is darker than a certain brightness, a situation in which the current position of the vehicle is inside a tunnel or inside a culvert, etc. When the headlight indicator remains on even though the situation in which the headlights should be turned on has been resolved, the processing means of the present system may notify the driver to turn off the headlights. Since there are tunnels whose insides are bright due to lighting, if it is determined whether or not the headlights should be turned on based only on brightness information outside the vehicle, a situation may occur in which it is erroneously determined that the headlights should not be turned on even though the vehicle is inside a tunnel. By determining whether or not the vehicle is inside a tunnel based on the current vehicle position information, such erroneous determination can be avoided. In addition, if it is detected that the side light indicator is not lit even though not only the headlights but also the side light (lower light) should be turned on, the driver may be notified to turn on the side light.

The processing means of this system determines whether there is an oncoming vehicle from the image in front of the vehicle, and if it detects that the high beam indicator is on even though there is an oncoming vehicle, it is preferable to warn the driver to change to low beam. Since the operation of switching between high and low beams differs depending on the vehicle model, when a driver is driving a vehicle model different from the one he or she normally uses, it is easy for an incident to occur in which the driver intends to switch to low beam but the high beam remains. In such a case, the function of issuing this warning is particularly effective.

The processing means of the system detects that the turn signal is flashing from an image of the outside of the vehicle when it is dark outside the vehicle, and recognizes an indicator flashing inside the vehicle in sync with the flashing period of the turn signal as the turn signal indicator, so that the processing means can easily find the turn signal indicator in the instrument panel.

When the processing means of the present system detects from the image of the inside of the vehicle that no one is inside the vehicle but detects that the headlights or width lights are on, it is preferable to notify the driver outside the vehicle or the vehicle manager that the lights have been forgotten to be turned off. This notification may be made, for example, by sending an e-mail to a pre-registered e-mail address. In addition, the processing means may:
It is preferable to notify the driver by e-mail or the like of forgetting to turn off the hazard lamps or room lights. It is preferable to detect the lighting status of the hazard lamps from the blinking status of the turn signal indicator. It is preferable to detect the lighting status of the room lights from the brightness of the room lights captured in the image.

There are generally three types of room light settings: a constant light mode, a constant off mode, and a door open/close state linked mode (hereinafter simply referred to as the linked mode).
These three modes are cyclically updated each time the room light on/off switch is pressed. Some vehicles have a function that turns on the room light when the engine is stopped in a dark place (at night) and automatically turns it off after a while (for example, after a few seconds have passed). There are also cases where you may want to keep the room light on for a longer period of time after turning off the engine in order to prepare to get off the vehicle. For example, after turning off the engine, you may want to keep the room light on for a period of time while you organize and remove purchased items and take out your mobile device. In such cases, the driver switches the room light setting to linked mode or constant light mode to prevent the room light from automatically turning off a certain period of time after the engine is stopped. However,
When the door is open, it is difficult to determine whether the current room light setting is the constant-on mode or the linked mode. If a person gets off the vehicle thinking that the current room light setting is the linked mode even though the current room light setting is the constant-on mode, the room light will remain on. By using this system, the driver or manager can realize that they have forgotten to turn off the room light and turn it off, thereby preventing the vehicle battery from running out due to the room light being on for a long time. This function of the system is particularly useful in vehicles with an operation switch structure that makes it impossible to know what mode the room light is currently set to just by looking at the operation switch, for example, a vehicle with an operation switch structure that switches the room light setting cyclically each time a push button is pressed.

When the room light is set to linked mode, if any door is open or ajar, the room light will remain on. When the driver or manager is notified that the room light is on, they can be aware that the door may be ajar or that a door on the hard-to-see side may be open.

Some vehicles have a function to keep the room light on until a certain time (light-off delay time) has elapsed after the door is closed when the room light is set to the linked mode. In such vehicles, the processing means of the present system may not determine whether the light has been forgotten to be turned off until the light-off delay time has elapsed after detecting that the door is closed. The light-off delay time may be set by the present system based on an input from the user.

In addition, when the processing means detects that the room light remains on even after a certain period of time has passed since no one has left the vehicle, the processing means may transmit a signal to the vehicle controller to turn off the room light.

The processing means of the present system may have a function of issuing an alarm notifying the driver that any of the lights are still on when it detects that the vehicle door has been opened while the headlights, width lights, hazard lights, interior lights, etc. are on or flashing. This allows the driver to realize that he or she has forgotten to turn off the lights before getting out of the vehicle, and thus prevents the driver from forgetting to turn them off.

The processing means of the present system may detect that the parking brake is applied based on the illumination of the parking brake indicator, and may not issue an alarm even if it detects that the driver is asleep from the image of the driver. In addition, it may not issue an alarm even if it detects that the driver is performing an action that would be considered dangerous if driving, such as looking at a mobile terminal, while the parking brake is applied.

The processing means of the present system may have a function of notifying the vehicle's dealer or a registered repair shop of the condition of the vehicle when it detects that a warning light indicating the need for early inspection or repair has remained on for a certain period of time. For example, a user who is not interested in automobiles may not notice the warning light on if the vehicle is still drivable even though it is on, and
For example, if the engine warning light comes on, the car will usually be unable to drive, but if the engine oil warning light comes on, the car can still be driven. Also, if the O2 sensor is abnormal, the fuel economy will be poor, but the car can still be driven.
If the vehicle is still drivable, continuing to drive without inspection and repair may result in a minor breakdown becoming more severe. For dealers and repair shops, the notification of the warning light being on is useful information for sales activities for inspection and repair to the owner of the vehicle with the warning light on. In addition, the owner of the vehicle can prevent the breakdown from becoming more severe by having the vehicle inspected and maintained early.

The display content of the display device to be detected may be, for example, the temperature displayed in a temperature display area in an instrument panel that displays a temperature measurement value measured by a temperature sensor that measures the temperature outside the vehicle. The processing means may have a function of reading the temperature by character recognition from an image displayed in the temperature display area of the instrument panel as an image analysis function. The processing means may execute a process of transmitting temperature information to an external server via a communication device and a communication network as a process corresponding to the analysis result.

For example, Weathernews Inc. and others provide a service that installs sensors in the homes of general users to obtain data on pollen dispersion, collects measurement data from the sensors, and then predicts detailed pollen dispersion conditions and distributes pollen dispersion information. In addition, outdoor photos posted by general users and images collected from outdoor live cameras installed in general homes are incorporated into weather forecasts. In addition, services that distribute these images (for example, Weathernews SOLiVE24, etc.) are provided. Systems that provide these weather information services can use many vehicles as temperature measurement probes by collecting outside air temperature information measured by many vehicles. When many vehicles that transmit temperature information are distributed over a wide area, the temperature information collected from many vehicles can be used as spot information. For example, it becomes possible to issue spot heatstroke warnings that cannot be expressed by general weather information.

For companies that provide such weather information services, the advantage is that they can collect temperature information over a wide area without installing their own temperature sensors. The company that receives the temperature information can offer the paid weather information service that it provides free of charge or at a discount as a token of appreciation to the vehicle owners who provided the temperature information.

Outside temperature information can be obtained not only from temperature information read from the temperature display area in the instrument panel, but also by character recognition of numbers displayed on roadside temperature signs captured in images captured by the camera. Temperature signs can be detected, for example, by recognizing the words "Current Temperature" on the sign. The obtained outside temperature information is
The processing means of the present system may transmit the temperature information to an external server in the same manner as the temperature information obtained from the temperature display area in the instrument panel.

The function of detecting the operation status of the operation device by image analysis may include a function of detecting the position of the driver's feet relative to the accelerator pedal and the brake pedal. As a process according to the operation status of the operation device, a process of notifying the driver that the driver's feet are on the accelerator pedal or the brake pedal may be executed so that the driver can easily recognize. For example, the driver may be notified by changing the color of light emitted by the light-emitting element when the foot is on the accelerator pedal or the brake pedal. The system may be installed in a vehicle having a function of detecting whether the foot is on the accelerator pedal or the brake pedal and displaying the detection result as visual information on a display, and the processing means of the system may image-analyze the contents displayed on the display and detect the position of the driver's feet relative to the accelerator pedal and the brake pedal.

Furthermore, if an obstacle is detected ahead by image analysis but the driver's foot is on the accelerator pedal, a warning may be issued, for example by sound. By letting the driver notice the warning, it is possible to prevent the driver from mistaking the accelerator pedal for the brake pedal.

The function of detecting the operation status of the operation device by image analysis may include a function of detecting the rotation angle of the steering wheel. As a process according to the operation status of the operation device, a process of calculating the traveling direction and the travel distance of the vehicle based on the rotation angle of the steering wheel and the vehicle speed may be executed. The processing result may be provided to, for example, a stand-alone car navigation device. The stand-alone car navigation device can calculate the traveling route of the vehicle based on the information on the traveling direction and the travel distance provided by the system even in a situation where it cannot receive a GPS signal.

The image analysis function may include a function to detect whether the vehicle is in autonomous driving (including autonomous driving levels 1 to 5) or manual driving (autonomous driving level 0). This detection may be performed, for example, by recognizing from an image the display state of an indicator displayed on an instrument panel that indicates whether various autonomous driving functions are in operation. As a process corresponding to the analysis result, for example, when it is detected that the autonomous driving function has been deactivated, a process to notify the driver that the autonomous driving function has been deactivated may be executed. In addition, when it is detected that the driver has not placed his/her hands on the steering wheel despite the autonomous driving function being deactivated, the processing means may issue a warning to alert the driver to operate the steering wheel.
Furthermore, the processing means may record images captured by the camera when the automatic driving function is deactivated and the driver has no hands on the steering wheel.
The recorded images can be used as evidence that the driver was operating improperly.

In a vehicle equipped with a radar detector, the processing means may disable the radar detector function when the autonomous driving function is enabled. This can prevent unnecessary alarms from being generated by the radar detector. When the autonomous driving state is disabled, the processing means may enable the radar detector function.

It is advisable to determine whether the vehicle is being driven autonomously or manually, and what level (mode) of autonomous driving it is in, from indicators, the movement of the steering wheel, the position of the driver's hands, the direction the driver's head is facing, etc.

(6) The processing means may have, as the image analysis function, a function of recognizing characters displayed on a sticker affixed to the vehicle from an image of the interior of the vehicle, and, as a processing corresponding to the analysis results, may execute a processing to issue a warning based on the information indicated by the recognized characters.

By receiving the warning, the driver, manager or owner of the vehicle will be able to pay attention to the text information on the sticker attached to the vehicle and take appropriate action in accordance with the text information.

For example, the stickers to be subjected to image analysis may include an inspection mark (vehicle inspection sticker). The inspection mark on the inside of the vehicle indicates the date on which the vehicle inspection expires. The processing means may read the information on the date on which the vehicle inspection expires by character recognition, and when the period from the present time to the expiration date of the vehicle inspection (the deadline for the vehicle inspection) becomes shorter than a predetermined period, may issue a warning to have the vehicle inspected. For example, the processing means may issue a warning by emitting a voice saying, "The expiration date of the vehicle inspection is approaching." When the period until the expiration date of the vehicle inspection becomes even shorter, for example, the processing means may issue a warning by a voice saying, "Please have your vehicle inspected."

For example, the stickers to be subjected to image analysis may include a maintenance inspection sticker. The maintenance inspection sticker indicates the deadline for the next regular inspection in years, months, and days. The processing means may read the date information indicating the deadline for the regular inspection by character recognition, and when the period from the present time until the deadline for the regular inspection becomes shorter than a predetermined period, may issue a warning to have the regular inspection. For example, the processing means may issue a warning by emitting a voice saying, "The deadline for the regular inspection is approaching." When the period until the deadline for the regular inspection becomes even shorter, for example, the processing means may issue a warning by voice saying, "Please have the regular inspection."

The image analysis function may include a function for determining whether or not the inspection mark is captured in the image captured by the camera. As a process corresponding to the analysis result, if the inspection mark is not captured in the image, the processing means may execute a process for calling attention to adjust the mounting position and attitude of the camera so that the inspection mark is captured. For example, the processing means may issue a voice saying, "The inspection mark is not captured. Please adjust the position and attitude of the camera so that the inspection mark is captured." This allows the position and attitude of the camera to be easily adjusted when it is installed. It is further preferable that the processing means has a function for reading the contents displayed on a display device in the vehicle, such as a speedometer, a fuel gauge, etc., and has a function for calling attention so that the display device in the vehicle is captured in the image.

The image analysis function may include a function for determining whether or not stickers other than those permitted by law are attached to the windshield and the driver's and passenger's side windows. If it is determined that a sticker other than those permitted by law is attached, a process may be executed to notify the driver or the like to remove the sticker as a process corresponding to the analysis result.

(7) The processing means may include, as the image analysis function, a function for analyzing an image of an area in which the passenger seat is reflected, and the system may execute processing according to the situation of the passenger seat as processing corresponding to the analysis results.

For example, appropriate processing can be executed depending on whether or not a person is in the passenger seat and whether or not an object is placed on the passenger seat.

The processing means may be installed in a vehicle having a function of issuing a warning when a person sitting in the passenger seat is not wearing a seat belt. The processing means may have, as an image analysis function, a function of determining whether or not a person is sitting in the passenger seat from an image of an area in which the passenger seat is captured. As a process corresponding to the analysis result, the processing means may disable the function of issuing a warning that the passenger seat belt is not fastened when it detects that no person is sitting in the passenger seat. The processing means may have, as an image analysis function, a function of determining whether or not an object is placed on the passenger seat. As a process corresponding to the analysis result, the processing means may execute a process of disabling the function of issuing a warning that the passenger seat belt is not fastened when an object is placed on the passenger seat.

When an object equivalent to a human weight is placed on the passenger seat, the vehicle may mistakenly determine that a person is in the vehicle and issue a warning that the seat belt is not fastened. When the present system is installed in a vehicle, the vehicle will not issue a warning that the seat belt is not fastened when an object is placed on the passenger seat. This prevents the driver from being confused by the warning. This function is particularly effective in vehicles that are designed to keep sounding a warning buzzer when a heavy object is placed on the passenger seat. To prevent the vehicle from issuing a warning that the seat belt is not fastened, for example, the processing means may transmit a signal to the vehicle controller via the in-vehicle network (CAN) to disable issuance of the warning.

Regarding the rear seats, in a vehicle having a similar function, when it is detected that an object is placed on the rear seat, the processing means may cancel the seat belt unfastened warning function.

(8) The processing means may have, as its image analysis function, a function of comparing the current image of the interior of the vehicle with a past image of the interior of the vehicle, and, as a process corresponding to the analysis results, may execute a process of notifying when the degree of discrepancy between the current image of the interior of the vehicle and the past image of the interior of the vehicle exceeds a predetermined level.

The owner or manager of the vehicle can be notified that the situation inside the vehicle has changed beyond a certain level. As the past image of the interior of the vehicle, it is preferable to use an image of the vehicle in a state where nothing other than the items that come with the vehicle has been brought into the vehicle. By comparing this image with the current image of the interior of the vehicle, it is possible to determine whether anything has been brought into the vehicle.

The time during which the object has remained in the vehicle may be calculated by comparing the images taken at multiple past times with the current image. The processing means may have a function of notifying the user of an object that has been left in the vehicle for a long period of time.

For example, when there is no baggage in the vehicle, and preferably only the driver is in the vehicle, the initial setting button of the system is pressed and the vehicle is driven. The processing means recognizes the part in the image with the optical flow corresponding to the vehicle speed as a window. The other part with movement is recognized as the driver. The part other than the part recognized as the window and the driver is recognized as an area where an object may be placed, and is recorded as an image of a no-baggage state (no-baggage state video) in which no baggage is loaded in the vehicle. The initial setting process is completed by the process up to this point. Thereafter, when the engine is started, when the engine is stopped, etc., the difference between the no-baggage state image and the current image is taken to identify the presence of an object brought into the vehicle. If the position of the object identified as having been brought into the vehicle does not change for a certain period of time, a warning is issued. This allows the driver, etc. to recognize an object that has been left in the vehicle for a long period of time. This function is particularly useful for people whose car quickly becomes full of garbage.

(9) The processing means may have, as a function of the image analysis, a function of determining whether or not a pre-registered object is captured in an image of the interior of the vehicle, and, as a process corresponding to the analysis result, may execute a process of issuing an alarm when a pre-registered object is captured in an image of the interior of the vehicle.

The alarm can alert the driver or the like to the presence of a pre-registered item in the vehicle. For example, an image of an item (a specific bag, wallet, etc.) that should be checked for as lost can be registered in the processing means. When the processing means detects that a registered item is captured in the image when no one is left in the vehicle, it can issue an alarm or notify the driver that the item has been lost. An example of issuing an alarm is to sound a buzzer or the like installed in the vehicle. An example of notifying the driver of the lost item is to wirelessly notify the driver or the like of the smartphone, mobile phone, etc. by email or the like.

The processing means may, for example, issue an alarm when it detects an item left behind inside the vehicle and detects the movement of opening the door. The alarm allows the driver or the like to realize that the item has been left behind before leaving the vehicle. The processing means may, for example, notify the driver or the like by e-mail or the like when it detects an item left behind inside the vehicle and detects the movement of closing the door. The driver or the like can realize that the item has been left behind after leaving the vehicle. The detection of the movement of opening and closing the door may be performed by:
For example, this can be done by detecting optical flow.

The processing means may have a function of detecting other movements in the vehicle by detecting the optical flow. For example, the processing means may detect the movement of opening and closing the doors, the movement of opening and closing the windows, the operation of the turn signal, the operation of the steering wheel, the operation of raising and lowering the sun visor, etc., as a function of image analysis. The present system is connected to a connector (
This information can be obtained without connecting to a vehicle (e.g., an OBD connector).

When the sun visor is down, particularly when the driver's side is imaged using a 360-degree camera attached to the windshield, the sun visor gets in the way and no image in that direction can be obtained. Therefore, when the processing means detects that the sun visor is down even though the brightness in front of the vehicle is such that the sun visor is not necessary, it is preferable that the processing means issues a warning. It is preferable that the processing means judges whether the sun visor is down based on the proportion of the image that the sun visor area occupies.

(10) The processing means may have, as the image analysis function, a function of capturing an image of an object brought close to the camera by the user and recognizing the image, and, as a processing corresponding to the analysis results, may execute processing according to information written on the object brought close to the camera by the user.

By bringing an object that a user is carrying close to the camera, a process corresponding to the information written on the object can be executed.

For example, the object that can be brought close to the camera may be a driver's license. The processing means may perform character recognition of the validity date printed on the driver's license, and when the period from the current time to the validity date is shorter than a predetermined period, may notify the user that the license renewal date is approaching.

For example, a receipt from a gas station may be brought close to the camera. The processing means may perform character recognition of the date, amount of fuel filled, and amount shown on the receipt, record this information, and create a database.

For example, an identification card (e.g., a driver's license, an employee ID card, a resident registration card, etc.) may be brought close to the camera. The processing means performs character recognition of a name, etc., written on the identification card,
It is preferable to determine whether the person is a person registered in advance. If the person on the identification card is not registered, it is preferable to perform a process to prohibit the start of the engine. In this way, the people who can drive the vehicle can be limited to those who are registered in advance. Furthermore, the processing means
It is advisable to record the name of the person on the identification card, etc. The recorded information can be used to identify the driver when managing the operation of the vehicle.

The processing means may have, as the image recognition function, a function of recognizing an object brought close to the camera by the user outside the vehicle, and may further have a function of recognizing an object brought close to the camera by the user inside the vehicle. For example, the processing means may recognize that an object that is not present in the normal image is brought close to the camera when the object occupies a larger area than a predetermined range within the angle of view of the camera than a normal image. In particular, when the processing means recognizes that the user is holding an object that occupies a larger area than a predetermined range within the angle of view of the camera, it may determine that the user has brought the object close to the camera.

(11) The processing means may be a system that has a function of performing image analysis of an object captured in an image ahead as the image analysis function, and that executes a process corresponding to the analysis result to provide the driver with information that should be noted regarding the driving of the vehicle.

By alerting the driver according to the situation ahead, it is possible to prevent future accidents that may occur as the vehicle moves forward.

The image analysis function may include a function for calculating at least one of the following distance and the relative speed with respect to the vehicle ahead. Based on the following distance or the relative speed, a situation where there is a high risk of collision may be detected, and the driver may be notified of the high risk of collision by issuing an alarm or the like. The following distance may be calculated from information such as the size of the vehicle ahead in the image, such as the size of the license plate, the size of the tread (the distance between the tires), and the width of the tail lamp. The relative speed may be calculated from the rate of change in the following distance.

The image analysis function may include a function for detecting the color of a traffic light ahead. When the traffic light is red, the processing means may notify the driver to stop.

The image analysis function may include a function for recognizing road signs ahead. The processing means may alert the driver according to the information indicated by the road sign. For example, if the road sign indicates a speed limit, the processing means may obtain the speed limit at the current location by character recognition and notify the driver of the speed limit. If the road sign indicates a no entry sign, the processing means may
If the vehicle attempts to enter a road where entry is prohibited, an alarm may be issued to notify the driver. For example, when it is detected that the vehicle is attempting to travel in violation of a traffic rule indicated by a road sign or is traveling in violation of a traffic rule, the driver may be notified of the violation. This makes it possible to prevent or suppress the occurrence of violations of traffic rules. For example, when the processing means detects a sign indicating that the section currently being traveled is a section where an automatic speed camera is installed by recognizing a character string on a road sign, the processing means may notify the driver to be careful not to drive too fast.

The image analysis function may include a function for detecting congestion in lanes adjacent to the lane in which the vehicle is traveling. For example, whether congestion is occurring may be determined based on the average traveling speed of vehicles in adjacent lanes and the distance between vehicles. For example, when the traveling speed of vehicles in adjacent lanes is below a predetermined reference value, and a predetermined number of vehicles are lined up with the distance between vehicles being below a predetermined reference value,
It is preferable to determine that a traffic jam has occurred. As a process corresponding to the analysis result, when it is detected that a traffic jam has occurred in a lane adjacent to the lane in which the vehicle is traveling, a process is executed to warn the driver to be careful not to jump out between the vehicles in the traffic jam. The lanes adjacent to the lane in which the vehicle is traveling should include an oncoming lane and an adjacent lane traveling in the same direction as the vehicle. In particular,
In places where people gather, such as shopping malls, theme parks, and stations, and places where vehicles enter and exit, such as gas stations, there is a high possibility that the lane adjacent to the lane in which the vehicle is traveling in the same direction will also be congested. In such cases, it is effective to detect congestion in the adjacent lane traveling in the same direction as the vehicle. This can prevent accidents caused by vehicles jumping out from the road. For example, when the vehicle's traveling speed is less than a predetermined speed, for example less than 30 km/h, it is advisable not to issue a warning sound to warn of vehicles jumping out from the road. This can avoid the generation of unnecessary warning sounds.
In addition, it is advisable to detect situations where there is a high risk of vehicles rushing out from the front image and warn the driver of such a situation. For example, it is advisable to warn the driver of vehicles rushing out from the front image when there are a series of parked vehicles on the side of the road, or when there are vehicles parked in a parking lot with a parking meter.

For example, some car navigation devices are equipped with a function that alerts the driver to driving caution based on traffic jam probe information, general road and highway driving information, and surrounding facility information (residential areas, schools, etc.) By using this system, it is possible to alert the driver to traffic jams even in places where traffic jam probe information cannot be obtained.

The image analysis function detects animals (dogs, cats, raccoons, wild boars, etc.) around the vehicle before starting off.
When the processing means detects an animal or a person around the vehicle,
It is a good idea to inform the driver that there are animals around the vehicle, which can help prevent accidents from occurring immediately after starting off.

The image analysis function may include a function for detecting that the vehicle is moving forward slowly from the flow of the surrounding scenery. The processing means may issue an alarm when it detects that the vehicle is moving forward slowly. For example, when the vehicle stops once and then starts to move little by little due to creeping caused by loosening of the force on the brake pedal, the driver can be notified that the vehicle is moving and that there is a risk of rear-end collision. For example, when the vehicle is moving at a speed that would cause the vehicle to move forward due to creeping, it may be determined that the vehicle is moving forward slowly. The speed of the flow of the surrounding scenery may be detected by detecting optical flow.

The image analysis function may include a function for determining whether the vehicle ahead is a large vehicle, such as a bus or a box-shaped truck with an aluminum cargo bed.
When it detects that the vehicle is approaching a traffic light, it is advisable to notify the driver of the presence of the traffic light. For example,
It is advisable to issue a voice message such as "Watch out for the traffic lights." The presence or absence of traffic lights should be registered in advance in the processing means as POI information. For example, when a large vehicle is traveling ahead, it is difficult to see the traffic lights in the direction of travel. It is preferable to look at the traffic lights in the opposite lane and traffic lights for pedestrians to predict the presence of traffic lights, or to keep a sufficient distance between vehicles so that the traffic lights in the direction of travel can be easily seen. However, if such care is neglected, there is a risk of overlooking the traffic lights in the direction of travel. Also, in rural areas, there are cases where traffic lights in the opposite lane and traffic lights for pedestrians are not installed. By notifying the driver of the presence of a traffic light when the vehicle ahead is a large vehicle,
This reduces the risk that the driver will miss a traffic light. Also, instead of directly informing the driver of the presence of a traffic light, the processing means may warn the driver to drive in a way that reduces the risk of missing the traffic light. For example, a voice message such as "Please keep a sufficient distance between vehicles" may be issued. For example, the warning may be issued when the distance to an intersection where a traffic light is installed is longer than the safe stopping distance of the vehicle. The safe stopping distance of the vehicle is determined based on the vehicle speed,
It depends on the weather (rainy or not). It is advisable to set an appropriate distance as the safe stopping distance based on the vehicle speed and the weather (rainy or not). For example, if the safe stopping distance in fine weather is 50 m, the processing means may issue a warning when the current weather is fine, the distance to the large vehicle ahead is less than 50 m, the vehicle is outside the urban area as defined by the car navigation device, and there is an intersection with a traffic light within 100 m in the direction of travel.

The image analysis function may include a function to determine whether the road ahead is a slope, and if so, to detect the degree of the slope. Depending on the degree of slope, the processing means may warn the driver not to go too fast, reduce speed, or set the vehicle to a hill climb mode.

The vehicle may be a vehicle that handles luggage, such as a forklift or truck, and the processing means may have a function of recognizing a character string, a barcode, or other marking on the luggage from the captured image and determining whether or not the luggage is a luggage that should be handled by the vehicle. Information indicated by the marking on the luggage that should be handled by the vehicle may be registered in advance in the processing means. When the processing means detects that the vehicle is attempting to load luggage that is not a luggage that should be handled by the vehicle, it may execute a process of issuing an alarm.

The processing means may have a function to detect a window area in an image from the optical flow. Furthermore, the processing means may have a function to determine, from the optical flow, whether the vehicle has entered a tunnel, whether the vehicle has exited a tunnel, and whether the vehicle is traveling inside a tunnel. In addition, the processing means may have a function to determine, from the optical flow, whether the vehicle is traveling under an elevated road (under a road that runs parallel to one another). Furthermore, the processing means may determine, from the optical flow,
It would be good to have a function to determine whether or not guardrails are present and whether or not caution is required regarding gutters.

(12) The processing means may have, as a function of image analysis, a function of recognizing characters written on the license plates of other vehicles captured in an image outside the vehicle, and may be a system that performs at least one of a process of recording the license plate information identified by character recognition and a process of transmitting the information via a network as a process corresponding to the analysis result.

It is possible to identify the license plate information of other vehicles that are traveling or parked near a vehicle equipped with an on-board device having the function of the processing means of this system. The processing means may transmit the current location information of the vehicle equipped with this on-board device and the date and time information when the license plate was imaged in association with the identified license plate information. The license plate information acquired together with the location information and date and time information, together with the license plate information acquired by an automatic license plate reader (N system) installed on the road, is useful information for knowing the travel routes of many vehicles. In this way, by linking the vehicle number with the GPS information, a system that can be called a wide-area mobile N system can be constructed.

It is preferable to construct a center system that collects location information, date and time information, and license plate information in association with each other from the present system installed in each of a plurality of vehicles via a network. This center system is preferable to have a function of extracting the location information and date and time information of a vehicle having specific license plate information and identifying the travel route of the vehicle.

(13) The processing means may have, as a function of the image analysis, a function of determining whether or not a window glass edge is present in the window portion of the vehicle, and as a process corresponding to the analysis result, may execute a notification process when the edge of the window glass is detected in the window portion.

The presence of a window glass edge in the window portion means that the window glass has been left closed. The driver or the like can realize that the window glass has been left closed by receiving a notification from this system. The window glass edge may be detected, for example, by detecting discontinuity in an image caused by a difference in refractive index between the window glass and the atmosphere. The notification process may be, for example, a wireless notification to the driver's mobile terminal or a notification by email. Alternatively, the notification process may be performed by sounding a buzzer attached to the vehicle. A tinted film is attached to the window of the vehicle equipped with the processing means, and the processing means may detect that the window is open based on a difference in image brightness between the open window area and the tinted window area.

(14) The processing means may have, as the image analysis function, a function of determining whether it is raining or not by detecting whether the wipers are operating or whether the number of raindrops on the windshield is increasing or decreasing, and as a process corresponding to the analysis result, if it is determined that it is raining, the system may execute a process of transmitting information from a communication device notifying that it is raining.

By receiving information transmitted from this system, the vehicle can know whether it is raining in the area in which the vehicle is traveling. The processing means may notify an external device such as a server that it is raining, and may also notify the external device of the vehicle's current location information. The external device can create a map of the area in which it is raining by collecting information from multiple vehicles. Furthermore, the processing means may notify the external device that it is raining, and may also notify the external device of the current time information. This allows the external device to obtain information regarding changes over time in the area in which it is raining.

The external device may be, for example, a server of an organization that provides weather information. The processing means may upload to the server an image of a time when the wipers are not in the image, for example, when the wipers are down. From the image uploaded to the server, information about rainfall, for example, the amount of rain, can be obtained. The server may use the data such as the amount of rain collected from the vehicle as big data in various application fields. For example, it may be used to predict localized torrential rain. The processing means may determine that the rain has stopped when there are raindrops on the windshield but the amount (number) of them is not increasing or decreasing.
When the processing means determines that the rain has stopped, it may notify the server that the rain has stopped. The server may obtain more detailed information regarding the rainfall.

(15) The processing means may have, as a function of the image analysis, a function of determining the weather from the sky conditions reflected in the image, and may be a system that executes a process of transmitting information regarding the weather from a communication device as a process corresponding to the analysis results.

Weather information collected from multiple vehicles can be used as big data related to weather information. For example, the weather can be judged as clear, sunny, cloudy, etc. based on the color of the sky captured in the image, the presence or absence of clouds, etc. Furthermore, the strength of the morning sun or afternoon sun can be detected based on the height of the sun and the brightness of the sky, and information that the morning sun or afternoon sun is strong can be transmitted. Current location information of the vehicle can be transmitted together with the weather information. An external device that has collected this information can provide the information that the morning sun or afternoon sun is strong, as well as the current location information of the vehicle, to drivers of other vehicles heading to that location. Drivers who receive this information can realize that they need to be careful of the morning sun or afternoon sun.

(16) The processing means may have a function of detecting sound information from a microphone mounted on the vehicle, and may associate the detection results of the sound information with the analysis results of the image analysis function, and may perform processing according to the results of the association.

Compared to processing based only on the results of sound detection or image analysis,
By associating the two, more appropriate processing can be performed.

The processing means has a function of detecting a change in the state of a display unit of a device having a display unit mounted in the vehicle as a function of image analysis, and as a process corresponding to the analysis result, detects sound information and
When the change in the state of the display unit is synchronized, a process may be executed to associate the detected sound information with the device whose state of the display unit has changed and store the information.

When the processing means detects sound information from the microphone, it can identify the device that produced the sound based on the stored information in which the sound information is associated with a change in the state of the display unit.
For example, the processing means may have a voice recognition function, determine whether the sound information from the microphone is generated from a specific device, and cancel the voice recognition function if the sound information is generated from a specific device. This makes it possible to avoid unnecessary execution of voice recognition processing. Furthermore, when the processing means determines that the sound information from the microphone is generated from a specific device, the processing means may have a function of adding information that allows the area of the device that generated the sound to be distinguished from other areas in the image inside the vehicle and recording the area. For example, an image of a rectangle or a closed curve of any shape that surrounds the area in which the device that generated the sound is shown may be superimposed on the original image and recorded. Alternatively, a semi-transparent geometric pattern may be superimposed on the area in which the device that generated the sound is shown and recorded.

The processing means may have, as an image analysis function, a function of determining whether or not an emergency vehicle, such as an ambulance, a fire engine, or a police car, is captured in the image. When the processing means detects the sound of an emergency vehicle siren from a microphone and detects an emergency vehicle in an image, the processing means may associate the two and determine that an emergency vehicle is present near the vehicle. When the processing means determines that an emergency vehicle is present, the processing means may notify the driver of a message to warn him or her.

(17) There are multiple cameras installed inside the vehicle.
The processing means may have, as a function of the image analysis, a function for detecting the positions of other cameras captured in the image, and may be a system that executes a process for identifying the relative positions of multiple cameras as a process corresponding to the analysis results.

It is possible to obtain images of a wide area inside the vehicle that is difficult to capture with a single camera. When the positional relationship of the multiple cameras is specified, the positional relationship of the images captured by each camera can be specified.

The processing means may have a function of identifying the models of other cameras captured in the image and identifying the angle of view from the models. The processing means may have a function of synthesizing images captured by multiple cameras. By synthesizing the images, the situation inside the vehicle can be grasped more intuitively than when individual images from the cameras are displayed.

For example, at least a part of the functions of the processing means may be realized by a viewer such as a personal computer.
It is advisable to display a sign such as a camera icon in the displayed image so that the position of the camera can be identified. It is difficult to identify the position of the camera in the vehicle from the image captured by the camera (self). It is advisable to detect the position of the camera in the vehicle from the image of the other camera in which the camera is captured. For example, when images captured by three cameras are synthesized and displayed, it is advisable to display the positions of the three cameras in the synthesized image with icons or the like. It is advisable to use different icons depending on the model or angle of view of the camera.

The processing means may have a function of displaying the image captured by each camera, as well as the position of the camera and the angle of view of the camera (the area captured by the camera) in the image. For example, the angle of view of the camera may be displayed in the image depending on the function of the camera, such as a 360-degree spherical camera, a 360-degree semi-spherical camera, or a wide-angle camera. A user who sees the image can intuitively understand which area each camera is capturing.

(18) The camera has an image circle smaller than the diagonal length of the imaging surface.
It is a 60 degree camera,
The processing means may be a system having, as the image analysis function, a function of transforming a basic pattern for calculating image features in accordance with a position within the image circle.

The basic pattern may be used to calculate the image feature amount without correcting the image distortion in the image circle. This reduces the load on the processing means. The basic pattern is defined for an image obtained by capturing an image using a standard lens (a general wide-angle lens). Projection methods for obtaining an image circle include, for example, orthogonal projection, equidistant projection, stereoscopic projection, and equisolid angle projection, and the basic pattern may be deformed according to these projection methods. To obtain an image circle, a circular fisheye lens may be used. The imaging surface of the camera is rectangular, and the diameter of the image circle may be shorter than the long side, and more preferably, equal to or shorter than the short side.

For example, a template pattern of an image to be detected may be prepared in advance as a basic pattern. In this case, the template pattern may be deformed according to the position in the image circle. Alternatively, for example, when recognizing an image using a Hare-like feature, each of a plurality of rectangular patterns for obtaining a Hare-like feature may be deformed according to the position in the image circle.

When performing image recognition, it is advisable to limit the area in the image circle in which a specific object to be detected is captured, and perform image analysis within the limited area. For example, it is advisable to realize the function of the in-vehicle device of this system with a drive recorder. In general, since the recommended installation position of the drive recorder is determined in advance, the area in the image circle in which the driver is captured, the area in which the traffic light is captured, the area in which the person in the back seat is captured, etc. are almost fixed. Therefore, depending on the person or object to be detected, it is advisable to limit the area in which image analysis is performed to detect the person or object to the area in which the person or object is captured. For example, when analyzing an image in front of the vehicle, it is advisable to analyze the area in which the windshield is captured.

Alternatively, for example, a specific area of the image circle may be converted into a planar image (parallel projection image), and matching may be performed on the converted planar image using a normal basic pattern.

Also, the image in the image circle may be mapped onto a sphere using a predetermined transformation function, a portion of the image mapped onto the sphere may be zoomed in, and image analysis may be performed as a planar image while ignoring distortion. If a specific object, such as a road sign, is to be detected, the area in which the specific object is captured may be zoomed in and image analysis may be performed. If a road sign is to be detected, the area in the forward image in which the road sign is captured may be zoomed in and image analysis may be performed.

(19) The processing means has, as a function of the image analysis, a function of identifying an area in the image that should not be made public from the viewpoint of privacy protection as a privacy protection area,
As a process corresponding to the analysis results, the system may execute a process of generating an image from an image captured by the camera such that a person appearing in the privacy protection area cannot be identified by appearance.

By making it impossible to identify a person, the privacy of the person can be protected. For example, a region in which the inside of the vehicle is reflected may be set as a region that should not be made public. In particular, when an image is acquired using a spherical camera, the inside of the vehicle is captured, but there may be cases where the image of the inside of the vehicle is not desired to be retained. In such a case, for example, the inside of the vehicle may be painted black or the inside of the vehicle may be pixelated, and the image outside the vehicle may be recorded as it is. The processing means may distinguish between a region in which the inside of the vehicle is reflected and a region in which the outside of the vehicle is reflected, for example, by performing image analysis. For example, a region in which optical flow occurs at a speed equivalent to the vehicle speed may be set as the outside of the vehicle. The processing means may have a function that allows the user to set a privacy protection region in advance.

a normal mode for recording images capable of identifying people within the privacy protection area;
A privacy mode in which an image is recorded in which a person captured in the privacy protection area cannot be identified by appearance may be selected by the user. The user can switch between the normal mode and the privacy mode depending on the purpose and situation of using the vehicle.

The privacy area may be an area in which a person is photographed. For example, the processing means may detect a person by performing image analysis, and the area surrounded by the contour of the detected person may be the privacy area. Alternatively, only the face of the detected person may be the privacy area. In this case, the person may be displayed as a silhouette. Alternatively, only the contour of the person may be displayed, and an image in which there is no person inside the contour may be displayed. In this case, the person appears transparent. For example, the edge of a moving object may be displayed as the contour of the person.

The processing means may display an avatar in place of the person in an area in which the person is captured. The avatar to be displayed may be similar to the original person. For example, an avatar reflecting the characteristics of the person may be displayed according to the gender, body shape, facial shape, etc. The person and the avatar may be associated in advance, and the processing means may identify the person captured in the image and display the avatar associated with the identified person. An avatar may be created in advance for each person. The avatar may be moved in accordance with the movement of the person. For example, when the person raises their hand, the avatar may be made to raise its hand, and when the person turns to the side, the avatar may also be made to turn to the side.
The facial expression of the avatar may be changed according to the facial expression of the person. For example, when the processing means detects that the person is smiling, sad, angry, etc., the processing means may change the facial expression of the avatar according to the following:
The facial expression may be changed to a smiling, sad, or angry expression, respectively. The processing means may read the person's emotion and change the color of the avatar according to the person's emotion. For example, when it is detected that the person is angry, the color of the avatar may be changed to a color representing anger, such as red.

The processing means may generate an image in which a person appearing in the privacy protection area cannot be identified by appearance from the image captured by the camera before recording the image, and record the generated image. Also, when playing back the image captured by the camera, the processing means may convert the image into an image in which a person appearing in the privacy protection area cannot be identified by appearance and display the image.

(20) The processing means may be a system that executes a process of converting the results of the image analysis into text data or a code and recording the results as a process corresponding to the analysis results.

By converting the image data into text data or encoding it becomes possible to easily find a desired image from among the recorded images. For example, when a person is detected in an image, text data such as "person detected" can be generated, and when a red traffic light is detected in an image, text data such as "red traffic light" can be generated. The generated text data can be recorded in association with the image from which the text data was generated. Instead of text data, the image analysis results can be associated with a code and encoded (converted into binary data) as the image analysis results.

(21) The processing means may have, as a function of the image analysis, a function of detecting areas of crushed black in an image, and, as a process corresponding to the analysis results, may execute a process of brightening the crushed black areas of the image and displaying them on the display screen.

A user who sees the image can easily recognize objects or people in the area with blackout. In particular, when an image is acquired using a spherical camera, the bright scenery outside the vehicle and the dark interior of the vehicle are simultaneously captured in the image, so the area in which the interior of the vehicle is reflected is prone to blackout. The process of detecting the blackout area and restoring it to a brighter state for display is particularly effective when an image is acquired using a spherical camera. As the process of restoring the image to a brighter state, for example, gamma correction, color correction, conversion to a black and white image, contrast enhancement, etc. may be performed.

(22) The processing means may have a function for allowing a user to divide an imaging range captured by the camera into a plurality of sections and a function for allowing a user to weight each section, and may have a function for determining imaging conditions based on the brightness of each section and the weight assigned to the section, and for causing the camera to capture an image under the determined imaging conditions.

A user can obtain an image under imaging conditions suitable for the area he wants to see by increasing the weight of the area he wants to see and decreasing the weight of the area he does not want to see. A user can divide the imaging range into any shape according to the shape of the area he wants to see and the area he does not want to see. This makes it possible to perform weighting that is more suitable for the user than, for example, a case where weighting is uniformly performed on areas divided into a square lattice shape regardless of the content of the image.

In particular, when a 360-degree camera is used for imaging, the difference in the positions within the imaging range where parts of the vehicle are captured becomes greater for each vehicle model compared to when a general camera is used. This increases the possibility that the relationship between the general-purpose weighting table set in advance in the firmware of the processing means and the actual expected division of the imaging range will not match. The function of determining weighting using PC browser software, etc., rather than using a general-purpose weighting table, is
This is particularly effective when using a round camera. For example, an image captured by a camera without weighting can be analyzed using AI software on a PC to estimate what is in which part of the image range, and a weight of "5" can be assigned to the road, a weight of "3" to the face, a weight of "1" to the instrument panel, and a weight of "0" to the outside of the round image range.

The processing means may determine, for example, gain, exposure time, etc., as the imaging conditions. The back faces of two semi-spherical cameras may be opposed to each other to form a spherical camera, and the imaging conditions of the two semi-spherical cameras may be set to be the same. This provides the effect that when images captured by the two semi-spherical cameras are combined, the boundary between the two images becomes difficult to distinguish.

When the processing means has the user divide the area, the processing means may have the user surround a part of the image capture range with a closed curve, and the inner area surrounded by the closed curve may be regarded as one section. When the user divides the inner area surrounded by the closed curve into two with a curve, each of the divided areas may be regarded as one section. The processing means may have the user input a weight as a numerical value for each set section. The numerical value to be input may be selected from a list such as a pull-down or pop-up.

The processing means may have a function for inputting a type of a section instead of or in addition to the function for inputting a weight as a numerical value. A weight is associated with a type of a section in advance, and when a type of a section is input, the processing means may associate the weight associated with the type with the section. As the type of a section, for example, types such as a driver, a meter panel, a car navigation screen, a windshield, a rear seat, a passenger seat, etc. may be provided.
The processing means may have a function that allows the user to change the correspondence between the type of the partition and the weight.The list of partition types may be registered in advance by the user.

The processing means may have a function for allowing the user to assign weights to each section, for example, a function for setting sections by having the user specify an area within the imaging range that the user wants to see. The weight of the section that the user wants to see, specified by the user, may be greater than the weight of the other areas. This function corresponds to an example in which the section is divided into two types, that is, sections that the user wants to see and other sections, in the function for allowing the user to input the type of section.

The processing means may allow the user to divide the imaging range in an interactive manner (wizard format).
For example, the processing means may output a message such as "Please surround the unnecessary area" or "Please fill in the unnecessary area," and recognize the area surrounded or filled in by the user as an "unnecessary area." Alternatively, the processing means may output a message such as "Please fill in the driver's area" or "Please fill in the road portion," and recognize the filled in area as a section having a driver type or a section having a road type. The operation of surrounding with a closed curve or the operation of filling in may be performed as follows:
For example, this may be done using a touch panel or a pointing device such as a mouse.

The processing means may use artificial intelligence (AI) to realize a function of dividing the imaging range into a plurality of sections based on the captured image. For example, the image may automatically distinguish between the inside and outside of the vehicle, the seats, the dashboard, the instrument panel, etc., and divide the imaging range for each of the automatically determined components. The processing means may display the division results on a display screen and allow the user to determine whether the division is acceptable. If the user determines that the division made by the AI is incorrect, the processing means may have a function of allowing the user to correct the division. If the user determines that the division and the weight or type corresponding to the division are incorrect, the processing means may allow the user to select the correct type from a pull-down or pop-up list.

(23) The processing means may have a function for allowing a user to divide the imaging range captured by the camera into a plurality of sections and a function for allowing the user to weight each section, and the system may execute a process corresponding to the analysis results in which compression is performed by making blocks in sections with relatively small weights larger than blocks in sections with relatively large weights.

A user can increase the compression level of low-importance sections by increasing the weight of high-importance sections and decreasing the weight of low-importance sections. As a result, the volume of image data can be reduced while maintaining high image quality in high-importance sections compared to compressing the entire image uniformly. For example, the processing means may compress high-weight sections into blocks of 8×8 pixels and low-weight sections into blocks of 16×16 pixels.

An image of the lens barrel may appear in the area outside the circular image formed on the rectangular imaging surface of the camera. Since the image outside the circular image is unnecessary, it is advisable to paint this area black. This makes it possible to reduce the volume of image data including the pixels outside the circular image. It is also advisable to obtain image data by raster scanning only the pixels inside the circular image, excluding the pixels outside the circular image. This makes it possible to reduce the volume of image data.

(24) The processing means may be a system that has, as the image analysis function, a function to detect a following vehicle and a function to detect the distance to the following vehicle, and that executes, as a processing corresponding to the analysis result, a processing to detect aggressive driving by the following vehicle based on the distance to the following vehicle.

The processing means executes processing according to the judgment result, thereby making it possible to take appropriate measures against aggressive driving. For example, when the same following vehicle continues to exist within a predetermined judgment distance for a predetermined judgment time or more even though the vehicle speed is equal to or greater than a predetermined reference value, it may be judged that the following vehicle is aggressive driving.

The camera may have a function of measuring the distance to an object for each pixel, for example, a back-illuminated time-of-flight distance image sensor may be used as the image sensor of the camera.

When the processing means determines that the following vehicle is aggressive, it may perform processing according to the determination result, such as notifying the driver, recording an image of the rear, advising the driver, and reporting to a traffic enforcement agency (e.g., the police). When reporting to a traffic enforcement agency, it may be possible to attach an image of the aggressive vehicle and its driver. When the processing means detects through image analysis that the aggressive vehicle has overtaken the vehicle and gone around in front of the vehicle, it may notify the driver that there is a possibility of harassment by sudden stops.

The processing means may detect people and vehicles around the vehicle and control the vehicle according to the distance to the people and the vehicle. For example, the processing means may control the vehicle to stop when it determines that there is a high possibility of contact with a person based on the speed of the vehicle, the distance to a person ahead, and the determination result of whether or not a person is on the planned driving route of the vehicle.

(25) The present invention may be provided as a program for causing a computer to realize the functions of the processing means described in any one of (1) to (24) above.

The inventions shown in (1) to (24) above can be combined in any way. For example, at least a part of the configuration of at least one of the inventions from (2) onwards may be added to all or a part of the configuration of the invention shown in (1). In particular, the invention shown in (1) may be combined with at least a part of the configuration of (2).
It is preferable to add at least a part of the configuration of at least one of the following inventions. In addition, any configuration may be extracted from the inventions shown in (1) to (24) and the extracted configurations may be combined. The applicant of this application intends to obtain rights to the invention including these configurations.

Rather than simply recording images captured by a camera mounted on a vehicle, the images can be utilized more effectively.

The effects of the present invention are not limited to those described above, and the effects achieved by the configurations disclosed in the present specification and drawings are also disclosed. The configurations achieving these effects are also included in the divisional application and the
The applicant intends to obtain rights by amendment, etc. For example, the phrase "can..." in this specification clearly indicates the effect to be achieved, and there are parts that indicate the effect even without the phrase "can...". In addition, there are effects that can be understood by the configuration even without such a phrase.

FIG. 1 is a diagram showing the windshield, dashboard, etc. inside a vehicle in a state in which a drive recorder as a system according to a first embodiment is mounted in the vehicle. FIG. 2 is a block diagram of a drive recorder as a system according to the first embodiment. FIG. 3 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. FIG. 4 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by an in-vehicle device and an external device such as a personal computer. FIG. 5 is a diagram of an imaging surface for explaining a method of dividing an imaging range into a plurality of sections in a method according to a comparative example. FIG. 6 is a diagram of an imaging surface for explaining a method of dividing an imaging range into a plurality of sections in a system according to the twenty-second embodiment.

[First embodiment]
A system according to a first embodiment of the present invention will be described with reference to Figures 1 and 2. The functions of this system are realized by a drive recorder installed in a vehicle.

FIG. 1 is a diagram showing the windshield, dashboard, etc., inside a vehicle in a state in which a drive recorder 1 is mounted on the vehicle. The vehicle in which the drive recorder 1 is mounted is a passenger car. The drive recorder 1 is attached to a position on the passenger side adjacent to a room mirror 3, which is located at the upper part of a windshield 2 of the vehicle and near the center in the left-right direction. The drive recorder 1 is attached and fixed to the inner surface of the windshield 2 by a mounting member such as double-sided tape. The drive recorder 1 is connected to a cigarette lighter 5 of the vehicle via a power cable 4. When an accessory power source of the vehicle is turned on, power is supplied to the drive recorder 1 from the cigarette lighter 5. The drive recorder 1 includes a main body 10 and a 360-degree spherical camera 11. The camera 11 has a fixed position and attitude relative to the vehicle in a normal use state.

FIG. 2 is a block diagram of a drive recorder 1 as a system according to the first embodiment.
The drive recorder 1 includes a processing unit 20 , a camera 11 , a GPS receiver 12 , an SD card reader 13 , an acceleration sensor 14 , a speaker 15 , a display 16 , and an operation button 17 .
Strictly speaking, the SD card reader 13 should be called an "SD card reader/writer", but in this specification it is simply called an "SD card reader".

The speaker 15 and the display 16 function as output means for notifying the user of various information. The speaker 15 issues warnings and various information by sound or voice under the control of the processing unit 20. The display 16 displays various information as images under the control of the processing unit 20. The operation button 17 functions as input means for the user to give various commands to the drive recorder 1.

The camera 11 is a 360-degree spherical camera that captures video of the entire area of the vehicle in the forward/backward, up/down, and left/right directions. Therefore, the camera 11 captures images of the outside of the vehicle through the windshield, rear windshield, and side windows, as well as the inside of the vehicle, including the driver's seat, passenger seat, rear seat, instrument panel, center console, etc. The video captured by the camera 11 is input to the processing unit 20 as image data. The processing unit 20 processes the video captured by the camera 11.
Based on the image data input from 1, an image is recorded on an SD card connected to an SD card reader 13.

The GPS receiver 12 detects the position information of the vehicle at the current time based on an instruction from the processing unit 20. The position information includes the time, the speed of the vehicle, and
This information includes longitude, latitude, altitude, etc. The processing unit 20 performs processing to record the history of this position information.

Under the control of the processing unit 20, the SD card reader 13 reads data from an SD card inserted into an SD card slot, or records data on the SD card.

A three-axis type sensor that detects acceleration and tilt in each of three axes (x-axis, y-axis, z-axis) is used as the acceleration sensor 14. The acceleration sensor 14 constantly outputs detection values to the processing unit 20. The processing unit 20 acquires three-axis acceleration information, for example, every 10 ms.

The processing unit 20 includes a known CPU 20a, memories such as a ROM 20b and a RAM 20c, a timer 20d, and other peripheral circuits. Various programs are stored in the ROM 20b of the processing unit 20. The processing unit 20 executes these programs to realize various functions. The various programs include an operating system (OS), a GPS information processing program, an image analysis program, and the like.

The processing unit 20 executes a GPS information processing program to store the GPS information received by the GPS receiver 12 in an SD card connected to the SD card reader 13. The processing unit 20 executes an image processing program to continuously record image data of moving images captured by the camera 11 associated with time information and location information in a predetermined recording area of the SD card.

The processing unit 20 executes an image analysis program to perform image analysis of the image captured by the camera 11. Furthermore, the processing unit 20 executes various processes corresponding to the analysis results.

By mounting the system according to the first embodiment on a vehicle, images captured by a camera attached to the vehicle can be not only recorded but also more effectively utilized. If the images captured by the camera 11 are converted into a video, image analysis of the video makes it possible to easily determine the movement of the subject, the presence or absence of the subject before and after a certain point in time, and the like.

Next, a modification of the first embodiment will be described.
In the first embodiment, the drive recorder 1 is installed in a passenger car, but in a modified example, it is installed in other types of vehicles, such as a freight car or a special purpose vehicle.
In the modified example, the camera 11 captures images of at least one of the inside and outside of the vehicle. The terms "inside the vehicle" and "outside the vehicle" may be interpreted as follows, for example.

When the interior and exterior spaces of a vehicle are clearly separated by the vehicle body, the interior of the body is referred to as the "interior" and the exterior of the body is referred to as the "exterior". For example, in the case of a van in which the passenger space and the cargo space are continuous, the continuous passenger space and cargo space are referred to as the interior. In the case of a sedan in which the passenger space and the cargo space are separated, both the passenger space and the cargo space are referred to as the interior. In a covered freight vehicle with a sealed cargo bed, the passenger space and the interior of the sealed cargo bed are referred to as the interior. In an open-top freight vehicle (flatbed vehicle) with an open cargo bed, it is difficult to clearly separate the space that accommodates the cargo on the cargo bed from the space around it, but the space that accommodates the open cargo is also referred to as the interior. The passenger space of an open-top freight vehicle is also referred to as the interior.
In some special vehicles such as forklifts, both the passenger space and the cargo space are open, and it may be difficult to clearly distinguish the passenger space and the cargo space from the surrounding space. Even in such cases, both the passenger space and the cargo space can be called the interior of the vehicle. For example, the space above the seat where the passenger sits and the space above the forks that hold and lift the cargo can be called the interior of the vehicle. In this way, the space that can be considered to move with the vehicle as it moves can be called the interior of the vehicle, and the surrounding space can be called the outside of the vehicle.

In the first embodiment, a 360-degree omnidirectional camera is used as the camera 11, but a 360-degree semi-spherical camera or a normal wide-angle camera may also be used. When using a wide-angle camera, it is preferable to attach multiple cameras to the vehicle so that a wide area can be captured.

In the first embodiment, the function of recording the image captured by the camera 11 is realized by the in-vehicle device, but it may be realized by a server that communicates with the in-vehicle device. Also, in the first embodiment, the function of analyzing the image captured by the camera is realized by the in-vehicle device, but it may be realized by a server that communicates with the in-vehicle device.

As described above, the processing unit 20 of the present system may be configured as one in-vehicle device or as multiple devices. As in the first embodiment, the function of the processing unit 20 is the same as that of the drive recorder 1.
The processing unit 20 may be realized by a microcomputer and an application program stored in the memory.
It is preferable to realize the functions of the processing unit 20 by a single device, with the on-board device recording the images and the server analyzing the images. It is also preferable to realize the functions of the processing unit 20 by a vehicle-mounted device and an external device such as a personal computer, with the on-board device recording the images and the external device analyzing and displaying the images.

3 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. The in-vehicle drive recorder 1 includes a communication device 21. The server 30 includes a processing unit 31, a communication device 32, an input device 33, and a display 34. The communication device 21 of the drive recorder 1 and the communication device 32 of the server 30 are both connected to a communication network 40, and transmit and receive data to and from other devices via the communication network 40.
The processing unit 20 of the drive recorder 1 and the processing unit 31 of the server 30 are connected to a communication network 40.
Data communication is carried out between the two devices via this.

The input device 33 is operated by a user to input various commands and data to the processing unit 31. The processing unit 31 causes the display 34 to display the processing results and various messages to the user.

4 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by an in-car device and an external device 50 such as a personal computer.
, an input device 53, and a display 54. The SD card 60 on which image data is recorded is inserted into the SD card reader 52 of the external device 50 by the SD card reader 13 of the drive recorder 1.
By inserting the SD card 60 into the external device 50 , the processing unit 51 of the external device 50 can access the image data recorded on the SD card 60 .

The processing unit 51 reads image data recorded in the SD card 60 and performs image analysis or image processing based on a command input from the input device 53. The processing unit 51 causes the display 54 to display the analysis result or processing result.

Next, the second to twenty-fourth embodiments will be described. Below, a duplicated description of the configuration common to the system according to the first embodiment will be omitted. In addition, the systems according to the various embodiments described below are realized, as an example, in any of the forms shown in the block diagrams in FIG. 2, FIG. 3, and FIG. 4.

[Second embodiment]
Next, a second embodiment will be described. In the second embodiment, the processing unit of the system has a function of determining whether or not the movement of a person in an image corresponds to a specific movement registered in advance as a function of image analysis. When the movement of the person corresponds to the specific movement, the processing unit executes at least one of the following processes corresponding to the analysis result: recording an image capturing the specific movement of the person, transmitting the image capturing the specific movement to the server 30 (FIG. 3), reporting that the person has made the specific movement, and issuing an alarm.

The recorded image or the image sent to the server can be used as evidence that a person has made a specific movement. By reporting that a person has made a specific movement, it is possible to inform an outsider that there is a person making a specific movement. By issuing an alarm, the person who was making the specific movement is motivated to stop making the specific movement by noticing that an alarm has been issued to him/her.

Next, an example of a "person making a specific movement" will be described.
An example of a person making a specific movement is a driver of a vehicle. An example of a specific movement is a movement other than the driver's driving. For example, the movement is a movement of looking at a mobile terminal such as a smartphone, a movement of eating food, a movement of looking away or distracted, a movement of falling asleep, etc. For example, the presence or absence of a movement of distracted driving is determined based on the detection result of the position of the driver's eyes. The recorded image can be used to improve the driver's awareness, analyze the cause of an accident, etc. When the processing unit detects that the driver is operating a mobile terminal such as a smartphone, it issues an alarm to warn the driver not to hold the mobile terminal while driving. If the driver notices the alarm and stops the specific movement, it is possible to avoid danger in advance. The server 30 to which the data is transmitted is, for example, a server of the manager of the vehicle or a server of the police. The manager of the vehicle can evaluate the driver's driving attitude based on the transmitted data. The police can evaluate the driver's driving attitude, analyze the cause of the accident, etc. based on the transmitted image.

A person performing a specific movement may be, for example, a suspicious person outside the vehicle. A specific movement may be, for example, a movement of trying to break into the vehicle, for example, a movement of trying to break a glass window with a hammer,
The actions include peering into the vehicle. By issuing an alarm, a suspicious person attempting to break into the vehicle can be deterred from doing so. The recorded images or the images transmitted to the server 30 can be used as useful information for arresting a thief after a vehicle burglary occurs. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle, thereby preventing damage from being caused by a vehicle burglary. The driver or owner who receives the notification can also report the incident to the police. In this way, this function can be used to help with crime prevention.

An example of a person making a specific movement is a suspicious person in a vehicle. Examples of specific movements are various methods of vehicle theft, such as illegal entry into a parked vehicle, suspicious movements in a parked vehicle, and the removal of devices from inside the vehicle. If an alarm is issued and the thief retreats in a hurry, the damage from vehicle theft is reduced. The recorded image or the image transmitted to the server 30 can be used as useful information for arresting the thief after a vehicle theft crime has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle or report the incident to the police, thereby reducing the damage from vehicle theft. An example of reporting to the outside is sending an email with an image of a suspicious person attached. The email is sent to the following address:
For example, vehicle managers, drivers, owners, police, etc.

An example of a person who moves in a specific manner is a passenger in a commercial vehicle carrying unspecified people.
The specific movements are various robbery methods, such as brandishing a knife, aiming a gun, or using violence. If the processing unit executes a process to issue an alarm, the robber will think about committing the crime and damage can be prevented. The recorded images or the images sent to the server can be used as useful information for arresting the robber.
By notifying the vehicle manager that a person has made a specific movement, the manager can notice that a robbery has occurred and notify the police. The processing unit also has a function of notifying the police of the occurrence of a robbery using the communication function.

An example of a person making a specific movement is an infant riding in a car. An example of a specific movement is a movement when waking up from sleep. When the image analysis detects that the infant has woken up, the processing unit notifies the guardian by e-mail or the like. There are cases where the infant falls asleep in the car while driving, and the parent wants to go shopping or run an errand while the engine is running. In such cases, the parent wishes to leave the infant asleep, since forcing the infant to wake up will make the infant angry. However, the guardian is concerned about whether the infant has woken up when they are away from the vehicle. When this system is used, an e-mail is sent to the guardian's mobile device as soon as the infant wakes up, so the parent can immediately rush to the vehicle to deal with the infant's awakening.

An example of a person making a specific movement is a person outside the vehicle who needs care or a person who needs rescue. An example of a specific movement is a movement that is specific to a person who needs care, or a movement that corresponds to a situation where rescue is likely to be required. An example of a movement that is specific to a person who needs assistance is a movement of walking unsteadily with a bent back while leaning on a cane. An example of a movement that corresponds to a situation where rescue is likely to be required is a movement that corresponds to a situation where a person has collapsed and is foaming at the mouth.

The processing unit has a function of detecting a person making a specific movement by selecting at least one image from an image of the front of the vehicle, an image of the rear of the vehicle, and an image of the interior of the vehicle, and a function of detecting the person in the selected image. Furthermore, the processing unit has a function of allowing a user to specify at least one image. This prevents the processing unit from analyzing unnecessary or less important images, eliminating the need for the processing unit to have excessive processing power.

The processing unit has a function for detecting drowsiness in a driver as a function for detecting a person making a specific movement. For example, the processing unit measures the pulse rate from the fluctuation in the redness of the driver's skin, and judges whether the driver is drowsy or not based on the fluctuation in the pulse rate. Furthermore, the processing unit has a function for improving the accuracy of drowsiness detection by combining body temperature measurement using a thermal camera and pulse measurement using microwaves. When the processing unit of this system detects drowsiness, it performs a process of lowering the air conditioner's temperature setting,
Or execute a process to issue a warning by voice or sound.

[Third Example]
Next, a third embodiment will be described. The processing unit of the system according to the third embodiment has an image analysis function of determining whether or not a person in an image has a specific characteristic that has been registered in advance, and executes a process according to the specific characteristic possessed by the person in the image as a process corresponding to the analysis result.

The system according to the third embodiment can carry out appropriate processing depending on the characteristics of a person appearing in an image.

The processing unit has a face recognition function as a function for determining whether a person in an image has a specific feature. The processing unit performs face recognition of a person seated in the driver's seat, and if the person seated in the driver's seat does not correspond to a person registered in advance, the processing unit accesses the vehicle and prevents the engine from starting. For example, the processing unit transmits a signal to an engine control unit (ECU) via an in-vehicle network, such as a CAN, to prohibit the engine from starting. This makes it possible to register people who can drive in advance and prohibit unregistered people from driving the vehicle. As a result, the crime prevention function of the vehicle can be improved. It is also effective as a measure against children's mischief.

According to a modified example of the third embodiment, the processing unit has a function of performing at least one of the following processes: preventing the accelerator from being depressed, preventing the handbrake (parking brake) from being released, preventing the shift lever from being placed in the drive position, sounding an alarm, and sending an email together with an image to a registered email address.

According to another variation of the third embodiment, the processing unit has a function of performing face authentication of a person outside the vehicle, and controlling the vehicle to unlock the door if the person outside the vehicle corresponds to a person registered in advance. For example, if the vehicle is provided for a car sharing service, registered members of the car sharing service are registered in the processing unit in advance. The registered members can approach the vehicle and move their faces within the field of view of the vehicle's camera, thereby enabling them to use the vehicle even if they do not have a card key or the like. This improves the usability of the vehicles of the car sharing service.

According to yet another variation of the third embodiment, the vehicle is a commercial vehicle in which unspecified persons board, and the processing unit has a function of determining whether or not the passenger is a wanted criminal by a facial recognition function. A facial photo of the wanted criminal is transmitted from a police database to the processing unit of the present system by communication via LTE, wireless LAN, or the like. The processing unit has a function of registering the received facial photo. If the processing unit determines that the passenger is a wanted criminal, it notifies the manager of the vehicle, notifies the police, and the like, via the communication device 21 (FIG. 3) and the communication network (FIG. 3). Furthermore, the processing unit has a function of notifying the driver in a manner that is not known to the passenger.
The vehicle manager, police, driver, etc. will be able to recognize that a wanted criminal has gotten into the vehicle and can take appropriate action quickly.

For example, taxis are closed spaces with only the driver and passengers, so this system is in high demand from the perspective of ensuring the safety of the driver. When the processing unit detects that a criminal has boarded a taxi, it notifies the taxi company and the police. In addition, this system can detect the criminal's boarding at the time he or she gets into the vehicle before he or she actually commits a crime inside the vehicle. This makes it possible to prevent crimes from occurring.

According to yet another modification of the third embodiment, the processing unit has a function of determining whether a person in an image has a specific characteristic by comparing the appearance of the person in the image with a specific appearance registered in advance and determining whether the two match (hereinafter referred to as an appearance authentication function). As the specific appearance, an appearance specific to a wanderer, an appearance specific to a criminal, etc. are registered in the processing unit in advance. Examples of the appearance specific to a criminal include carrying a knife on the street and carrying a dangerous object. In addition, the processing unit has a function of detecting a person who possesses illegal drugs by combining images from an infrared camera, an ultraviolet camera, etc. with images from a normal camera. When the processing unit detects a person with a specific appearance in the image, it notifies the police or the vehicle manager via the communication device 21 (FIG. 3) and the communication network 40 (FIG. 3).

According to yet another modification of the third embodiment, the processing unit has a function of individually registering the appearance of a specific missing person, wanderer, etc., at the time of disappearance as search information. For example, the processing unit has a function of registering individual search information such as gender, height, type of clothing, color of clothing, whether or not the person is wearing a hat, whether or not the person is carrying a cane, and whether or not the person is bent over. Furthermore, the processing unit has a function of registering an area where the person is thought to be wandering (search target area). When the vehicle is located in the search target area, the processing unit executes a detection process of a person matching the individual search information. Such individual search information is received by the processing unit of the present system from a database (server) of the police, a local support center, etc. In this system, the processing unit of the on-board device executes a function of determining whether or not a person in an image has a specific characteristic, and when the vehicle enters the search area, the search information is downloaded from the server 30 (FIG. 3) to the processing unit of the vehicle.

According to yet another modification of the third embodiment, the processing unit has a function of determining whether a person in an image has a specific feature, that is, a function of determining whether a person sitting in the passenger seat or the back seat is wearing a seat belt (hereinafter referred to as an appearance authentication function). When the processing unit of this system detects a person not wearing a seat belt, it issues a message urging the person to wear a seat belt.

Furthermore, the processing unit has a function of issuing a message urging a person in the back seat to fasten their seat belt when the person approaches the entrance to the expressway without fastening their seat belt. The processing unit determines that the vehicle is approaching the entrance to the expressway when the distance from the current position of the vehicle to the entrance to the expressway is equal to or less than a reference distance. As another example, the processing unit determines that the vehicle is approaching the entrance to the expressway when it receives a signal for checking whether the ETC card has been inserted. As yet another example,
The processing unit analyzes an image of the area in front of the vehicle, and when it detects an entrance to a highway in the image, it determines that the vehicle is approaching the entrance to the highway.

[Fourth embodiment]
Next, a fourth embodiment will be described. The processing unit of the system according to the fourth embodiment has a function of counting the number of people in an image as a function of image analysis, and executes a process of recording or transmitting data on the number of people in the image as a process corresponding to the analysis result.

The system according to the fourth embodiment makes it possible to count the number of people without using a special device for counting the number of people.

According to a modification of the fourth embodiment, a camera is attached to the bus, and the processor uses an image of the inside of the bus as the image on which the number of people is counted. The processor displays the counting result on the display 16 (FIG. 2) and informs the driver. The driver can easily know the number of passengers (passengers) on board the vehicle. This function can be used in place of a roll call for tours. When all passengers should disembark, if the passenger counting result does not reach 0, the driver can know that there is someone left behind on the bus. For example, it is possible to detect people who cannot express their will, such as babies, or people who cannot disembark by themselves, such as disabled people, who are left behind.

According to another modification of the fourth embodiment, the vehicle on which the camera is mounted is a type of vehicle in which the number of passengers is predetermined, such as a tourist bus, a long-distance express bus, etc., and the processing unit of this system has a function of registering the predetermined number of passengers. The processing unit compares the result of the headcount with the registered number, and if there is an excess or shortage, warns the driver of the excess or shortage.
This warning is given by emitting a warning sound from the speaker 15 (Fig. 2) or displaying a warning message on the display 16 (Fig. 2). This makes it possible to prevent the train from departing with an insufficient number of passengers. It also makes it easy to know when an unauthorized person has boarded the train and encourages them to get off.

According to yet another variation of the fourth embodiment, a camera (e.g., a hemispherical camera on the ceiling) is attached inside a route bus, and the processing unit performs image analysis to count the number of passengers on board, standing passengers, and seated passengers. Furthermore, the processing unit acquires the current location information of the bus, the bus door opening/closing signal, the bus alighting buzzer, the touch of an IC card when getting on or off the bus, and the signal from the fare box, and has the function of recording these events and images at the time of the event occurrence in association with each of these as triggers. The server 30 (Figure 3) accumulates this information in a data lake. Using this data lake, it becomes possible to perform big data analysis on where people got on and off, where they got off, and what the seating situation was inside the bus during that time. Such data,
This information can be valuable for consulting companies that handle urban planning, etc. Furthermore, the processing unit has the function of acquiring information transmitted through in-vehicle networks such as CAN, as well as vital information on the driver. When some abnormality occurs in the vehicle, this information can be used to identify causal relationships, such as whether the abnormality is due to driving factors, passenger factors, factors of the driver himself, or vehicle factors.

According to yet another variation of the fourth embodiment, the processing unit employs an image of the outside of the vehicle as the image that is the basis for counting the number of people. The processing unit of this system has a function of recording the counting result as number of people data. Based on the recorded number of people data, it is possible to easily extract locations where there were many people. The processing unit has a function of recording the image that was the basis for the counting, triggered by the counting result. For example, it has a function of recording the image when the number of people is equal to or greater than a predetermined threshold value.

[Fifth Example]
Next, a fifth embodiment will be described. The processing unit of the system according to the fifth embodiment has an image analysis function of detecting the display contents of a display device of the vehicle or the operating status of an operating device of the vehicle from an image of the inside of the vehicle, and executes processing according to the display contents of the display device or the operating status of the operating device as processing corresponding to the analysis result.

This system can acquire data on which the display contents of the display device are based, or data corresponding to the operating status of the operating device, without retrofitting such data. For example, this data can be acquired without connecting an additional device to the OBD connector. In particular, this system can acquire such data even if the vehicle manufacturer prohibits connecting third-party products to the OBD connector. Various gauges, warning lights in the vehicle, etc., can be used as display devices for the vehicle.
and head-up displays that project images onto the windshield, etc.

In a modification of the fifth embodiment, the processing unit of the system has a function of detecting the vehicle speed displayed on the speedometer as the display content of the display device. The processing unit has a function of performing pattern recognition on the position of the needle of the analog speedometer and character recognition on the numbers on the dial to obtain vehicle speed data. Furthermore, the processing unit has a function of performing character recognition on the numbers displayed on the digital speedometer to obtain vehicle speed data.
The acquired vehicle speed data can be used for various processes. An example of a form in which the vehicle speed data is used will be described below.

The processing unit has a function to automatically detect the speedometer by image recognition.
The processing unit can detect the speedometer in various vehicles, regardless of the vehicle type. In addition, the processing unit has a function of registering in advance the position and shape of the speedometer in the image of the vehicle in which the processing unit is installed. This allows the processing unit to easily detect the speedometer.

A typical standalone car navigation device uses GPS to obtain the current location.
It uses location information from PS. When GPS signals cannot be received, such as in tunnels,
The current position of the vehicle must be determined in another way. As an example, the current position of the vehicle is moved within the map at a constant speed of about 80% of the speed just before the GPS signal was lost for 5 minutes.
The mode in which such processing is performed is referred to as tunnel mode. Tunnel mode is effective not only when entering a tunnel, but also when traveling under an overpass. However, if the vehicle encounters a traffic jam while processing in tunnel mode, the error in the current position becomes excessively large in a method that assumes that the vehicle is traveling at a constant speed, making it practically useless. If an acceleration sensor is attached to the vehicle and acceleration data from the acceleration sensor is used, the accuracy of the current position information in tunnel mode can be improved. However, it is difficult and expensive for the user to attach an acceleration sensor, so this is not practical.

This system has a function of a standalone car navigation system. The processing unit has a function of reading the vehicle's speed from the speedometer display, and updating the vehicle's current position information using the map matching function based on the read speed information. This makes it possible to improve the accuracy of the vehicle's current position information.

The processing unit has a function of selecting which road to match based on whether or not the tunnel mode is activated in a place where a road inside a tunnel and a ground road run parallel to each other, or where an elevated road and a ground road under the elevated road run parallel to each other. For example, when the tunnel mode is activated, the processing unit has a function of mapping the vehicle position to the road inside the tunnel or the ground road under the elevated road, and when the tunnel mode is not activated, the processing unit has a function of mapping the vehicle position to the ground road or the elevated road running parallel to the road inside the tunnel.

In another modification of the fifth embodiment, the processing unit detects the display content of the display device to be detected as follows:
It has a function to detect the remaining fuel level displayed on the fuel gauge. The processing unit detects the remaining fuel level by performing image recognition of the fuel gauge. When the processing unit detects that the remaining fuel level has fallen below a predetermined reference value, it notifies the driver of a message encouraging him or her to refuel. For example, by linking with the current location information of the vehicle, when the driver returns from an outside location to near the vehicle storage area at home or at work,
If the remaining fuel level falls below a reference level, for example, 1/4 of the full tank, the processor issues a warning and urges the driver to refuel. In addition, the system is linked to a car navigation device, and the processor compares the distance to the destination with the remaining fuel level, and based on the comparison result, urges the driver to refuel at the time of departure or on the way to the destination. For example, near a gas station on the way, a message is output that urges the driver to refuel at a gas station near the current location.

The function of extracting the speedometer and fuel gauge from the image is realized by artificial intelligence (AI) installed in the processing unit of this system. The processing unit also has a function of registering the positions and image information of the speedometer and fuel gauge in the image in advance. If it is difficult to extract the speedometer and fuel gauge from the image using the artificial intelligence, the processing unit uses the registered image information to extract the speedometer and fuel gauge from the image.

The processing unit has a function of reading the fuel gauge of a vehicle provided for a car sharing service, and when it detects that the remaining fuel level is below a predetermined reference value, notifying the driver that he/she can receive benefits of the car sharing service by refueling. For example, it may issue a message such as "If you refuel now, you can receive a discount equivalent to 15 minutes."

In yet another modification of the fifth embodiment, the processing unit has a function of determining whether or not a warning light in the instrument panel is on. The warning lights for which the lighting status determination is performed include a headlight indicator, a turn signal indicator, a high beam indicator (high beam warning light), a parking brake indicator, a door ajar warning light, a fuel warning light, etc. The processing means of this system can detect the vehicle status corresponding to the warning light by determining whether or not these warning lights are on.

In yet another modification of the fifth embodiment, the processing unit has a function of detecting warning information displayed by the head-up display. The processing unit detects the content of the warning by capturing a change in the image displayed by the head-up display, and executes a predetermined process according to the content of the warning. For example, the warning is issued by sound or voice. By converting the visual warning into auditory warning and issuing it, it is possible to obtain an effect that the driver can easily notice the warning.

In yet another modification of the fifth embodiment, the processing unit has a function of linking with brightness information outside the vehicle and current vehicle position information, and when it detects that the headlight indicator is not lit even though the headlights should be turned on, notifying the driver to turn on the headlights. Situations in which the headlights should be turned on include, for example, a situation where the outside of the vehicle is darker than a certain brightness level, or a situation where the vehicle is currently located inside a tunnel or culvert. When the headlight indicator remains lit even though the situation in which the headlights should be turned on has been resolved, the processing unit has a function of notifying the driver to turn off the headlights. Since there are tunnels whose interiors are bright due to lighting,
If the process determines whether the vehicle is in a tunnel and the headlights should be turned on only from the brightness information outside the vehicle, the process may erroneously determine that the headlights should not be turned on even though the vehicle is in a tunnel. The process unit can avoid such erroneous determination by determining whether the vehicle is in a tunnel and the headlights should not be turned on based on the current position information of the vehicle.
If the system detects that the side light indicator is not lit when not only the headlights but also the side light (lower light) should be turned on, it has the function of notifying the driver to turn on the side light.

In yet another modification of the fifth embodiment, the processing unit has a function of determining whether there is an oncoming vehicle from an image in front of the vehicle, and warning the driver to change to low beam when it detects that the high beam indicator is on despite the presence of an oncoming vehicle. Since the operation of switching between high and low beams differs depending on the vehicle model, when the driver is driving a vehicle model different from the one he or she normally uses, it is easy for an incident to occur in which the driver intends to switch to low beam but the high beam remains on. In such a case, the function of issuing this warning is particularly effective.

In yet another modification of the fifth embodiment, the processor has a function of detecting that a turn signal is flashing from an image of the outside of the vehicle when it is dark outside the vehicle, and recognizing an indicator flashing inside the vehicle in synchronization with the flashing period of the turn signal as a turn signal indicator. In this way, the processor can easily find the turn signal indicator in the instrument panel.

In yet another modification of the fifth embodiment, when the processing unit detects that the headlights or width lights are on even though it detects from an image of the interior of the vehicle that no one is inside the vehicle, the processing unit has a function of notifying the driver outside the vehicle or the vehicle manager that the lights have been forgotten to be turned off. This notification is performed, for example, by sending an email to a pre-registered email address. In addition, the processing unit has a function of notifying the driver by email or the like that the hazard lights and room lights have been forgotten to be turned off. The lighting status of the hazard lights is detected from the blinking status of the turn signal indicator. The lighting status of the room lights is detected from the brightness of the room lights captured in the image.

There are generally three types of room light settings: a constant light mode, a constant off mode, and a door open/close state linked mode (hereinafter simply referred to as the linked mode).
These three modes are cyclically updated each time the room light on/off switch is pressed. Some vehicles have a function that turns on the room light when the engine is stopped in a dark place (at night) and automatically turns it off after a while (for example, after a few seconds have passed). There are also cases where you may want to keep the room light on for a longer period of time after turning off the engine in order to prepare to get off the vehicle. For example, after turning off the engine, you may want to keep the room light on for a period of time while you organize and remove purchased items and take out your mobile device. In such cases, the driver switches the room light setting to linked mode or constant light mode to prevent the room light from automatically turning off a certain period of time after the engine is stopped. However,
When the door is open, it is difficult to determine whether the current room light setting is the constant-on mode or the linked mode. If a person gets off the vehicle thinking that the current room light setting is the linked mode even though the current room light setting is the constant-on mode, the room light will remain on. By using this system, the driver or manager can realize that they have forgotten to turn off the room light and turn it off, thereby preventing the vehicle battery from running out due to the room light being on for a long time. This function of the system is particularly useful in vehicles with an operation switch structure that makes it impossible to know what mode the room light is currently set to just by looking at the operation switch, for example, a vehicle with an operation switch structure that switches the room light setting cyclically each time a push button is pressed.

When the room light is set to linked mode, if any door is open or ajar, the room light will remain on. When the driver or manager is notified that the room light is on, they can be aware that the door may be ajar or that a door on the hard-to-see side may be open.

Some vehicles have a function to keep the room light on until a certain time (light-off delay time) has elapsed after the door is closed when the room light is set to the linked mode. In such vehicles, the processing unit of the present system has a function to not judge whether the light has been forgotten to be turned off until the light-off delay time has elapsed after detecting that the door is closed. The processing unit has a function to set the light-off delay time based on an input from the user.

In addition, the processing unit has a function of transmitting a signal to the vehicle controller to turn off the room light when it detects that the room light remains on even after a certain time has passed since no one has left the vehicle.

In yet another variation of the fifth embodiment, the processing unit has a function of issuing an alarm notifying the driver that any of the lights remains on when it detects that the vehicle door has been opened while the headlights, width lights, hazard lights, interior lights, etc. are on or flashing.
This allows the driver to realize that he or she has forgotten to turn off the lights before getting out of the vehicle, and thus makes it possible to prevent the driver from forgetting to turn off the lights.

In yet another modification of the fifth embodiment, the processing unit has a function of not issuing an alarm when it detects that the parking brake is applied based on the illumination of the parking brake indicator, even if it detects that the driver is asleep from the image of the driver. In addition, even if it detects that the driver is performing an action that would be considered dangerous if driving, such as looking at a mobile terminal, while the parking brake is applied,
It has a function that does not issue an alarm.

In yet another modification of the fifth embodiment, the processing unit has a function of notifying the dealer or a registered repair shop of the vehicle of the condition of the vehicle when it detects that a warning light indicating the need for early inspection or repair has remained on for a certain period of time. For example,
Users who are not interested in automobiles often do not notice that the warning light is on and leave it alone if the vehicle is still drivable even when the warning light for the engine is on. For example, if the warning light for the engine itself comes on, the vehicle is normally unable to be driven, but the vehicle can still be driven even if the warning light for the engine oil is on. Also, if there is an abnormality in the O2 sensor, the vehicle will be able to be driven, although fuel economy will be poor. If the vehicle is still drivable and the vehicle is continued to be driven without inspection and repair, a minor malfunction may become serious. Notification of the warning light being on is useful information for dealers and repair shops to use in sales activities for inspection and repair to owners of vehicles with the warning light on. Also,
By having vehicle owners inspect and maintain their vehicles early, they can prevent breakdowns from becoming serious.

In yet another modification of the fifth embodiment, the processing unit has a function of detecting a temperature displayed in a temperature display area in an instrument panel that displays a temperature measurement value measured by a temperature sensor that measures the temperature outside the vehicle. The processing unit has a function of reading a temperature by character recognition from an image displayed in the temperature display area of the instrument panel as an image analysis function. The processing unit executes a process corresponding to the analysis result, which is a process of transmitting temperature information to an external server via a communication device and a communication network.

For example, Weathernews Inc. and others provide a service that installs sensors in the homes of general users to obtain data on pollen dispersion, collects measurement data from the sensors, and then predicts detailed pollen dispersion conditions and distributes pollen dispersion information. In addition, outdoor photos posted by general users and images collected from outdoor live cameras installed in general homes are incorporated into weather forecasts. In addition, services that distribute these images (for example, Weathernews SOLiVE24, etc.) are provided. Systems that provide these weather information services can use many vehicles as temperature measurement probes by collecting outside air temperature information measured by many vehicles. When many vehicles that transmit temperature information are distributed over a wide area, the temperature information collected from many vehicles can be used as spot information. For example, it becomes possible to issue spot heatstroke warnings that cannot be expressed by general weather information.

For companies that provide such weather information services, the advantage is that they can collect temperature information over a wide area without installing their own temperature sensors. The company that receives the temperature information can offer the paid weather information service that it provides free of charge or at a discount as a token of appreciation to the vehicle owners who provided the temperature information.

The processing unit further has a function of acquiring outside temperature information by character recognition of numbers displayed on roadside temperature signs captured in images acquired by the camera, as well as temperature information read from the temperature display area in the instrument panel. Temperature signs are detected by recognizing characters such as "current temperature" on the signs. The acquired outside temperature information is sent by the processing unit to an external server, in the same way as temperature information acquired from the temperature display area in the instrument panel.

In yet another modification of the fifth embodiment, the processing unit has a function of detecting the position of the driver's feet on the accelerator pedal and the brake pedal as a function of detecting the operation status of the operation device by image analysis. As a process according to the operation status of the operation device, the processing unit executes a process of notifying the driver whether the driver's foot is at the accelerator pedal position or the brake pedal position so that the driver can easily recognize. For example, when the driver's foot is at the accelerator pedal position and when the driver's foot is at the brake pedal position, the light-emitting element changes the light-emitting color to notify the driver.

In yet another modification of the fifth embodiment, the system is installed in a vehicle having a function of detecting whether the driver's foot is on the accelerator pedal or the brake pedal and displaying the detection result as visual information on a display. The processing unit of the system has a function of image-analyzing the content displayed on the display and detecting the position of the driver's feet relative to the accelerator pedal and the brake pedal.

In yet another modification of the fifth embodiment, the processing unit has a function of issuing a warning, such as a sound, when the driver's foot is on the accelerator pedal even though an obstacle is detected ahead by image analysis. By letting the driver notice the warning, it is possible to prevent the driver from mistaking the accelerator pedal for the brake pedal.

In yet another modification of the fifth embodiment, the processing unit has a function of detecting the rotation angle of the steering wheel (handle) as a function of detecting the operation status of the operation device by image analysis. The processing unit executes a process of determining the traveling direction and travel distance of the vehicle based on the rotation angle of the steering wheel and the vehicle speed as a process according to the operation status of the operation device.
The processing results are provided to a stand-alone car navigation device, which is capable of calculating the vehicle's travel route based on the information on the travel direction and distance provided by the system even in a situation where it cannot receive a GPS signal.

In yet another modification of the fifth embodiment, the processing unit has a function of detecting whether the vehicle is in automatic driving (including automatic driving levels 1 to 5) or manual driving (automatic driving level 0) as a function of performing image analysis. This detection is performed by the processing unit recognizing from an image the display state of an indicator displayed on the instrument panel indicating whether various automatic driving functions are in operation. When the processing unit detects that the automatic driving function has been deactivated as a process corresponding to the analysis result, the processing unit executes a process of notifying the driver that the automatic driving function has been deactivated. Furthermore, when the processing unit detects that the driver has not placed his hands on the steering wheel despite the automatic driving function being deactivated, the processing unit issues a warning to alert the driver to operate the steering wheel. Furthermore, the processing unit has a function of recording the image captured by the camera as an event record separate from the continuous record, triggered by the deactivation of the automatic driving function and the driver not placing his hands on the steering wheel. The recorded image can be used as evidence that the driver's operation was inappropriate.

The processing unit has a function of disabling the radar detector function when the autonomous driving function is enabled in a vehicle equipped with a radar detector. This makes it possible to prevent unnecessary alarms from being generated by the radar detector. When the autonomous driving state is disabled, the processing unit enables the radar detector function.

The processing unit determines whether the vehicle is being driven autonomously or manually, and what level (mode) of autonomous driving it is, based on indicators and steering wheel movement, the position of the driver's hands, the direction the driver's head is facing, etc.

[Sixth Example]
Next, a sixth embodiment will be described. The processing unit of the system according to the sixth embodiment has an image analysis function that recognizes characters displayed on a sticker attached to a vehicle from an image of the vehicle interior, and executes a process corresponding to the analysis result to call attention to information indicated by the recognized characters.

By receiving the warning, the driver, manager or owner of the vehicle will be able to pay attention to the text information on the sticker attached to the vehicle and take appropriate action in accordance with the text information.

In a modified example of the sixth embodiment, the processing unit has a function of recognizing characters displayed on the inspection emblem (vehicle inspection sticker). The inspection emblem on the inside of the vehicle has the date on which the validity period of the vehicle inspection expires displayed on it. The processing unit reads the information on the date on which the validity period of the vehicle inspection expires by character recognition, and when the period from the present time to the expiration date of the validity period of the vehicle inspection (the deadline for the vehicle inspection) becomes shorter than a predetermined period, it issues a warning to have the vehicle inspected. For example, the processing unit issues a warning by emitting a voice message saying, "The expiration date of the validity period of the vehicle inspection is approaching."
When the period until the expiration date of the validity period of the vehicle inspection becomes even shorter, for example, the processing unit issues a warning in the form of a voice saying, "Please have your vehicle inspected."

In another modification of the sixth embodiment, the processing unit has a function of recognizing characters displayed on a maintenance inspection sticker. The maintenance inspection sticker displays the deadline for the next regular inspection in years, months, and days. The processing unit reads the date information indicating the deadline for the regular inspection by character recognition, and when the period from the present time until the deadline for the regular inspection becomes shorter than a predetermined period, it issues a warning to have the regular inspection. For example, the processing unit issues a warning by emitting a voice saying, "The deadline for the regular inspection is approaching." When the period until the deadline for the regular inspection becomes even shorter, for example, the processing unit issues a warning by voice saying, "Please have the regular inspection."

In yet another modification of the sixth embodiment, the processing unit has an image analysis function that determines whether or not an inspection mark is captured in an image captured by a camera. If the inspection mark is not captured in the image, the processing unit executes a process corresponding to the analysis result, which calls for the user to adjust the mounting position and attitude of the camera so that the inspection mark is captured in the image.
For example, the processing unit may output a voice message saying, "The inspection mark is not visible. Please adjust the position and attitude of the camera so that the inspection mark is visible." This allows the position and attitude of the camera to be easily adjusted when it is installed.
For example, it has the function of reading the information displayed on the speedometer, fuel gauge, etc., and has the function of drawing attention to the fact that display devices inside the vehicle are included in the image.

In yet another modification of the sixth embodiment, the processing unit has an image analysis function that determines whether or not a sticker other than the sticker permitted by law is attached to the windshield and the driver's side window and the passenger's side window. As a process corresponding to the analysis result, when the processing unit determines that a sticker other than the sticker permitted by law is attached, the processing unit executes a process of notifying the driver or the like to remove the sticker.

[Seventh Example]
Next, a seventh embodiment will be described. The processing unit of the system according to the seventh embodiment has an image analysis function that analyzes an image of an area in which the passenger seat is captured, and executes processing according to the situation of the passenger seat as processing corresponding to the analysis result.

For example, appropriate processing can be executed depending on whether or not a person is in the passenger seat and whether or not an object is placed on the passenger seat.

In a modified example of the seventh embodiment, the drive recorder 1 (FIG. 1) is mounted on a vehicle having a function of issuing a warning when a person sitting in the passenger seat is not wearing a seat belt. The processing unit has a function of judging whether or not a person is sitting in the passenger seat from an image of an area in which the passenger seat is captured, as an image analysis function. As a process corresponding to the analysis result, the processing unit disables the function of issuing a warning that the passenger seat belt is not fastened when it detects that no person is sitting in the passenger seat. The processing unit further has a function of judging whether or not an object is placed on the passenger seat, as an image analysis function. As a process corresponding to the analysis result, the processing unit executes a process of disabling the function of issuing a warning that the passenger seat belt is not fastened when an object is placed on the passenger seat.

When an object equivalent to a human weight is placed on the passenger seat, the vehicle may mistakenly determine that a person is in the vehicle and issue a warning that the seat belt is not fastened. When this system is installed in a vehicle, the vehicle will not issue a warning that the seat belt is not fastened when an object is placed on the passenger seat. This prevents the driver from being confused by the warning. This function is particularly effective in vehicles that are designed to keep sounding a warning buzzer when a heavy object is placed on the passenger seat. In order to prevent the vehicle from issuing a warning that the seat belt is not fastened, the processing unit transmits a signal to the vehicle controller via the in-vehicle network (CAN) to disable the issuance of the warning.

In a vehicle having a similar function for the rear seats, when the processing unit detects that an object has been placed on the rear seat, the processing means cancels the seat belt unfastened warning function.

[Eighth Example]
Next, an eighth embodiment will be described. The processing unit of the system according to the eighth embodiment has, as an image analysis function, a function of comparing a current image of the interior of the vehicle with a past image of the interior of the vehicle, and, as a process corresponding to the analysis result, executes a process of notifying when the degree of discrepancy between the current image of the interior of the vehicle and the past image of the interior of the vehicle exceeds a predetermined level.

The vehicle owner or manager can be notified when conditions inside the vehicle change beyond a certain level.

In a modified example of the eighth embodiment, the processing unit has the function of using an image of the past interior of the vehicle in which nothing other than items already attached to the vehicle has been brought into the vehicle, and by comparing this image with the image of the interior of the vehicle at the current time, determining whether anything has been brought into the vehicle.

The processing unit further has a function of calculating the time that the object brought into the vehicle has remained in the vehicle by comparing the images taken at multiple past points in time with the current image. When the processing unit detects that an object has remained in the vehicle for a long period of time, the processing unit has a function of notifying the user of the object.

Next, specific processing of the processing unit will be described. First, with no luggage loaded in the vehicle, and preferably with only the driver on board, the initial setting button of the system is pressed and the vehicle is driven. The processing unit recognizes the parts in the image that have an optical flow corresponding to the vehicle speed as windows. Any other parts with movement are recognized as the driver. Any parts other than the parts recognized as windows and the driver are recognized as areas where objects may be placed, and are recorded as an image of an unladen state (unladen state video) in which no luggage is loaded in the vehicle. With the processing up to this point, the initial setting processing is completed.

Thereafter, when the engine is started, when the engine is stopped, etc., the processing unit calculates the difference between the image of the no-baggage state and the current image, and identifies the presence of an object brought into the vehicle. If the position of an object identified as having been brought into the vehicle does not change for a certain period of time, the processing unit issues a warning.
This function allows the driver or other person to recognize objects that have been left in the vehicle for a long period of time. This function is particularly useful for people whose vehicles tend to become cluttered with trash.

[Ninth Example]
Next, a ninth embodiment will be described. The processing unit of the system according to the ninth embodiment has, as an image analysis function, a function of determining whether or not a pre-registered object is captured in an image of the interior of the vehicle, and executes, as a process corresponding to the analysis result, a process of issuing an alarm when a pre-registered object is captured in an image of the interior of the vehicle.

The alarm makes the driver or the like aware that a pre-registered object is present inside the vehicle.

In a modification of the ninth embodiment, the processing unit has a function of registering an image of an object (such as a specific bag or wallet) that should be checked for lost property. When the processing unit detects that a registered object is captured in the image when no one is in the vehicle, the processing unit issues an alarm or notifies the driver that the object has been lost. An example of issuing an alarm is to sound a buzzer or the like installed in the vehicle. An example of notifying the driver that the object has been lost is to wirelessly notify the driver by email or the like to a smartphone or mobile phone of the driver or the like.

In another modification of the ninth embodiment, the processing unit has a function of issuing an alarm when it detects an item left behind in the vehicle and a movement to open the door. The alarm allows the driver or the like to realize that the item has been left behind before leaving the vehicle. The processing unit further has a function of notifying the driver or the like of the left behind item by e-mail or the like when it detects an item left behind in the vehicle and a movement to close the door. The driver or the like can realize that the item has been left behind after leaving the vehicle. The movement to open and close the door is detected, for example, by detecting optical flow.

In yet another modification of the ninth embodiment, the processing unit has a function of detecting other movements inside the vehicle by detecting the optical flow. For example, the processing unit has a function of detecting, as an image analysis function, movements of opening and closing the doors, opening and closing the windows, operation of the turn signals, operation of the steering wheel, operation of raising and lowering the sun visors, etc.
This information can be obtained without connecting to a vehicle's in-vehicle network connector (e.g., an OBD connector).

When the sun visor is down, particularly when capturing an image of the driver's side using a 360-degree camera attached to the windshield, the sun visor gets in the way and makes it impossible to capture any image in that direction. Therefore, when the processing unit detects that the sun visor is down even though the brightness in front of the vehicle is such that the sun visor is deemed unnecessary, it issues a warning. The processing unit determines whether the sun visor is down based on the proportion of the image that the sun visor area occupies.

[Tenth Example]
Next, a tenth embodiment will be described. The processing unit of the system according to the tenth embodiment has an image analysis function of capturing an image of an object brought close to the camera by the user and recognizing the image, and executes processing according to information written on the object brought close to the camera by the user as processing corresponding to the analysis result.

By bringing an object that a user is carrying close to the camera, a process corresponding to the information written on the object can be executed.

In a modification of the tenth embodiment, the processing unit has a function of performing image recognition of the driver's license of the driver. The processing unit performs character recognition of the validity date printed on the driver's license, and notifies the user that the license renewal date is approaching when the period from the current time to the validity date is shorter than a predetermined period.

In another modification of the tenth embodiment, the processor has a function of performing image recognition on receipts from gas stations. The processor performs character recognition on the date, amount of fuel filled, and amount shown on the receipt, and records this information to create a database.

In yet another modified example of the tenth embodiment, the processing unit has a function of performing image recognition of an identification card (e.g., a driver's license, an employee ID card, a resident registration card, etc.). The processing unit performs character recognition of the name, etc. written on the identification card to determine whether or not the person is a pre-registered person. If the person on the identification card is not registered, the processing unit performs a process to prohibit the start of the engine. This makes it possible to limit the people who can drive the vehicle to those who are pre-registered. Furthermore, the processing unit records the name, etc. of the person on the identification card. The recorded information can be used to identify the driver when managing the operation of the vehicle.

In yet another modified example of the tenth embodiment, the processing unit has, as the image recognition function, a function of recognizing an object that the user brings close to the camera outside the vehicle, and a function of recognizing an object that the user brings close to the camera inside the vehicle. For example, the processing unit may compare an image with a normal image and:
When something not present in a normal image occupies a larger area than the camera's field of view,
In particular, when the processing unit recognizes that the user is holding an object that occupies a larger portion of the camera's angle of view than a predetermined range, it determines that the user has brought the object closer to the camera.

[Eleventh embodiment]
Next, an eleventh embodiment will be described. The processing unit of the system according to the eleventh embodiment has an image analysis function that performs image analysis of an object captured in an image ahead, and executes a process corresponding to the analysis result to provide the driver with information that should be noted regarding the driving of the vehicle.

By alerting the driver according to the situation ahead, it is possible to prevent future accidents that may occur as the vehicle moves forward.

In a modified example of the eleventh embodiment, the processing unit has a function of calculating at least one of the following distance and the relative speed with respect to the vehicle ahead as a function of image analysis. The processing unit detects a situation where there is a high risk of collision based on the following distance or the relative speed, and notifies the driver of the high risk of collision by issuing an alarm or the like. The following distance is calculated from information such as the size of the vehicle ahead in the image, for example, the size of the license plate, the size of the tread (the distance between the tires), and the width of the tail lamp. The relative speed is calculated from the rate of change in the following distance.

In another modification of the eleventh embodiment, the processing unit has a function of detecting the color of a traffic light ahead as a function of image analysis. When the traffic light is red, the processing unit notifies the driver to urge him to stop.

In yet another modified example of the eleventh embodiment, the processing unit has a function of recognizing a road sign ahead as a function of image analysis. The processing unit warns the driver according to information indicated by the road sign. For example, if the road sign indicates a speed limit, the processing unit obtains the speed limit at the current location by character recognition and notifies the driver of the speed limit. If the road sign indicates a no-entry road, the processing unit issues an alarm to notify the driver if the vehicle attempts to enter a no-entry road. When the processing unit detects that the vehicle is attempting to run against the traffic rules indicated by the road sign or is running against the traffic rules, the processing unit notifies the driver that the vehicle is violating the rules. This makes it possible to prevent or suppress the occurrence of violations of traffic rules. When the processing unit detects a sign indicating that the section currently being traveled is a section equipped with an automatic speed camera by recognizing the character string of the road sign, the processing unit notifies the driver to be careful not to drive too fast.

In yet another modification of the eleventh embodiment, the processing unit has a function of detecting a congestion state of a lane adjacent to the lane in which the vehicle is traveling as a function of image analysis. The processing unit determines whether a congestion has occurred based on the average traveling speed of the vehicles in the adjacent lanes and the distance between the vehicles. When the traveling speed of the vehicles in the adjacent lanes is equal to or less than a predetermined reference value, and a predetermined number of vehicles are lined up with the distance between the vehicles being equal to or less than a predetermined reference value, the processing unit determines that a congestion has occurred. When the processing unit detects that a congestion has occurred in a lane adjacent to the lane in which the vehicle is traveling as a process corresponding to the analysis result, the processing unit executes a process of calling the driver's attention to be careful of jumping out from between the vehicles in the congestion. The lanes adjacent to the lane in which the vehicle is traveling include an oncoming lane and an adjacent lane traveling in the same direction as the vehicle. In particular, in places where people gather, such as shopping malls, theme parks, and stations, and places where vehicles enter and exit, such as gas stations, there is a high possibility that the same-direction lane adjacent to the lane in which the vehicle is traveling is also congested. In such a case, it is effective to detect a congestion in the adjacent lane traveling in the same direction as the vehicle. This makes it possible to prevent accidents caused by sudden jumping out of the road.
When the distance is less than m, a process is executed not to emit a warning sound for warning against sudden sudden jumping out. This makes it possible to avoid unnecessary warning sounds. In addition, the processing unit has a function of detecting a situation where there is a high risk of sudden sudden jumping out from an image of the front and issuing a warning. For example, the processing unit issues a warning against sudden sudden jumping out when there are consecutive parked vehicles on the side of the road, when there is a parked vehicle in a parking lot where a parking meter is installed, etc.

For example, some car navigation devices are equipped with a function that alerts the driver to driving caution based on traffic jam probe information, general road and highway driving information, and surrounding facility information (residential areas, schools, etc.) By using this system, it is possible to alert the driver to traffic jams even in places where traffic jam probe information cannot be obtained.

In yet another modification of the eleventh embodiment, the processing unit has an image analysis function that detects animals (dogs, cats, raccoon dogs, wild boars, etc.) and people around the vehicle before starting. When the processing unit detects animals or people around the vehicle, it notifies the driver that there are animals, etc. around the vehicle. This makes it possible to prevent accidents from occurring immediately after starting.

In yet another modification of the eleventh embodiment, the processing unit has a function of detecting that the vehicle is moving forward slowly from the flow of the surrounding scenery as a function of image analysis. When the processing unit detects that the vehicle is moving forward slowly, it issues an alarm. For example, when the vehicle stops once and then starts to move little by little due to creeping caused by loosening of the force applied to the brake pedal, the driver can be informed that the vehicle is moving and that there is a risk of rear-end collision. When the vehicle is moving at a speed that is similar to the speed at which the vehicle moves forward due to creeping, the processing unit determines that the vehicle is moving forward slowly. The speed of the flow of the surrounding scenery is detected by the processing unit by detecting optical flow.

In yet another modification of the eleventh embodiment, the processing unit has, as an image analysis function, a function of determining whether the vehicle ahead is a large vehicle, such as a bus or a box-shaped truck with an aluminum cargo bed. When the processing unit detects that the vehicle ahead is a large vehicle approaching a traffic light,
The presence of a traffic light is notified to the driver. For example, a voice message such as "Watch out for the traffic light" is issued. The presence or absence of a traffic light is registered in advance as POI information in the processing unit. For example, when a large vehicle is traveling ahead, the traffic light in the direction of travel is difficult to see. It is preferable to look at the traffic lights in the opposite lane and traffic lights for pedestrians to predict the presence of a traffic light, or to keep a sufficient distance between vehicles so that the traffic light in the direction of travel can be easily seen. However, if such care is neglected, there is a risk of overlooking the traffic light in the direction of travel. Also, in rural areas, there are cases where traffic lights in the opposite lane and traffic lights for pedestrians are not installed. By notifying the driver of the presence of a traffic light when the vehicle ahead is a large vehicle, the risk of the driver overlooking the traffic light can be reduced. In addition to the function of directly notifying the driver of the presence of a traffic light, the processing unit has a function of calling attention to driving in a way that reduces the risk of overlooking the traffic light. For example, the processing unit may be configured to
Please keep a sufficient distance between vehicles." For example, a warning is issued when the distance to an intersection with traffic lights is longer than the safe stopping distance of the vehicle. The safe stopping distance of the vehicle varies depending on the vehicle speed and weather (rainy or not). The processing unit sets an appropriate distance as the safe stopping distance based on the vehicle speed and weather (rainy or not). For example,
If the safe stopping distance in fine weather is 50 m, the processing unit issues a warning when the current weather is fine, the distance to the large vehicle ahead is less than 50 m, the vehicle is outside what is considered an urban area in terms of car navigation devices, and there is an intersection with traffic lights within 100 m in the direction of travel.

In yet another modification of the eleventh embodiment, the processing unit has an image analysis function that determines whether the road ahead is a slope, and if so, detects the degree of slope. Depending on the slope of the slope, the processing unit warns the driver not to go too fast, to slow down, to switch to hill climb mode, etc.

In yet another modification of the eleventh embodiment, the vehicle is one that handles luggage, such as a forklift or truck, and the processing unit has a function of recognizing a character string, a barcode, or other marking on the luggage from a captured image and determining whether or not the luggage is a luggage that should be handled by the vehicle. Information indicated by the marking on the luggage that should be handled by the vehicle is registered in advance in the processing unit. When the processing unit detects that the vehicle is attempting to load luggage that is not a luggage that should be handled by the vehicle, it executes a process of issuing an alarm.

In yet another modified example of the eleventh embodiment, the processing unit has a function of detecting a window region in an image from the optical flow. Furthermore, the processing unit has a function of judging, from the optical flow, whether the vehicle has entered a tunnel, whether the vehicle has exited a tunnel, and whether the vehicle is traveling in a tunnel. In addition, the processing unit may detect, from the optical flow, whether the vehicle is under an overpass (
The processing unit has a function to determine whether the vehicle is traveling on a road parallel to the road above or below the road. Furthermore, the processing unit has a function to determine the presence or absence of guardrails and the need to be careful of gutters from the optical flow.

[Twelfth embodiment]
Next, a twelfth embodiment will be described. The processing unit of the system according to the twelfth embodiment has an image analysis function of recognizing characters written on the license plates of other vehicles captured in an image outside the vehicle, and executes at least one of a process of recording the license plate information identified by the character recognition and a process of transmitting the information via a network as a process corresponding to the analysis result.

It is possible to identify license plate information of other vehicles traveling or parked nearby a vehicle equipped with an on-board device having the processing function of this system. The processing unit transmits the current position information of the vehicle equipped with this on-board device and the date and time information when the license plate was imaged, in association with the identified license plate information. The license plate information acquired together with the position information and date and time information, together with license plate information acquired by an automatic license plate reader (N system) installed on the road, is useful information for knowing the travel routes of many vehicles. In this way, the vehicle license plate and GP
By linking it with S information, it is possible to construct a system that can be called a wide-area mobile N system.

In a modification of the twelfth embodiment, a center system is constructed that collects location information, date and time information, and license plate information in association with each other via a network from the present system installed in each of a plurality of vehicles. This center system has a function of extracting the location information and date and time information of a vehicle having specific license plate information and identifying the travel route of the vehicle.

[Thirteenth embodiment]
Next, a thirteenth embodiment will be described. The processing unit of the system according to the thirteenth embodiment has, as an image analysis function, a function of determining whether or not the edge of the window glass is present in the window portion of the vehicle, and, as a process corresponding to the analysis result, executes a notification process when the edge of the window glass is detected in the window portion.

The presence of the edge of the window glass in the window area means that the window glass has been left open. The driver or the like can be made aware of this by receiving a notification from the system. The processing unit detects discontinuities in the image due to the difference in refractive index between the window glass and the atmosphere,
The edge of the window glass is detected. As a notification process, the processing unit performs processes such as wireless notification to a mobile terminal of the driver or notification by email. In addition, the processing unit performs notification processing by sounding a buzzer installed in the vehicle. When a tinted film is attached to the window of the vehicle on which the processing unit is installed, the processing unit detects that the window is open based on the difference in image brightness between the open window area and the tinted window area.

[Fourteenth embodiment]
Next, a fourteenth embodiment will be described. The processing unit of the system according to the fourteenth embodiment has an image analysis function that detects whether the wipers are operating or not, or whether the number of raindrops on the windshield is increasing or decreasing, to determine whether it is raining or not, and, as a process corresponding to the analysis result, executes a process of transmitting information notifying that it is raining from a communication device when it is determined that it is raining.

By receiving information transmitted from this system, the vehicle can know whether it is raining in the area in which the vehicle is traveling. The processing unit has a function of notifying an external device such as a server that it is raining, and also notifying the current location information of the vehicle. The external device can create a map of the area where it is raining by collecting information from multiple vehicles. Furthermore, the processing unit has a function of notifying that it is raining, and also notifying the current time information. This allows the external device to obtain information regarding changes over time in the area where it is raining.

In a modified example of the fourteenth embodiment, the external device is a server of an organization that provides weather-related information. The processing unit has a function of uploading an image of a timing when the wiper is not in the image, for example, when the wiper is down, to the server. The server has a function of acquiring information about rainfall, for example, the amount of rainfall, from the uploaded image. The server has a function of utilizing data such as the amount of rainfall collected from the vehicle as big data in various application fields. For example, the server utilizes the data for predicting localized torrential rain. The processing unit determines that the rain has stopped when there are raindrops on the windshield but the amount (number) of raindrops does not increase or decrease. When the processing unit determines that the rain has stopped, it notifies the server that the rain has stopped. The server can acquire more detailed information about the rainfall.

[Fifteenth embodiment]
Next, a description will be given of a fifteenth embodiment. The processing unit of the system according to the fifteenth embodiment has, as an image analysis function, a function of determining the weather from the state of the sky reflected in the image, and executes a process of transmitting information related to the weather from a communication device as a process corresponding to the analysis result.

Weather information collected from multiple vehicles can be used as big data on meteorological information. The server that collects this information determines the weather, such as clear, sunny, or cloudy, based on the color of the sky and the presence or absence of clouds captured in the image. Furthermore, the processing unit has the function of detecting the strength of the morning sun or the afternoon sun based on the height of the sun and the brightness of the sky, and transmitting information that the morning sun or the afternoon sun is strong. The processing unit transmits the current location information of the vehicle along with the weather information.
The server that collects this information has the function of providing information about the strong morning or afternoon sun, as well as the current location of the vehicle, to drivers of other vehicles heading to that location. Drivers who receive this information can be aware that they need to be careful of the morning or afternoon sun.

[16th embodiment]
Next, a description will be given of Example 16. The processing unit of the system according to Example 16 has a function of detecting sound information from a microphone mounted on a vehicle, associates the detection result of the sound information with the analysis result of the image analysis function, and executes processing according to the result of the association.

Compared to processing based only on the results of sound detection or image analysis,
By associating the two, more appropriate processing can be performed.

In a modified example of the sixteenth embodiment, the processing unit has, as an image analysis function, a function for detecting a change in the state of a display unit of an equipment having a display unit mounted in the vehicle, and, as a process corresponding to the analysis results, executes a process for associating and storing the detected sound information with the equipment whose display unit state has changed when the detected sound information and the change in the state of the display unit are synchronized.

In another modification of the sixteenth embodiment, when the processing unit detects sound information from the microphone,
The processing unit has a function of identifying the device that has emitted the sound based on information that is stored in association with the sound information and the change in the state of the display unit. For example, the processing unit has a voice recognition function, and determines whether the sound information from the microphone has been emitted from a specific device, and cancels the voice recognition function if the sound information has been emitted from a specific device. This makes it possible to avoid unnecessary execution of voice recognition processing. In addition, when the processing unit determines that the sound information from the microphone has been emitted from a specific device, the processing unit has a function of adding information that allows the area of the device that has emitted the sound to be distinguished from other areas in the image inside the vehicle and recording the added information. For example, an image of a rectangle or a closed curve of any shape that surrounds the area in which the device that has emitted the sound is captured is recorded by superimposing it on the original image.
Alternatively, a translucent geometric pattern is superimposed on the area showing the device that produced the sound.

In yet another modification of the sixteenth embodiment, the processing unit has, as an image analysis function, a function of determining whether or not an emergency vehicle, such as an ambulance, a fire engine, or a police car, is captured in an image. When the processing unit detects the sound of an emergency vehicle siren from a microphone and detects an emergency vehicle in an image, the processing unit associates the two and determines that an emergency vehicle is in the vicinity of the vehicle.
If it determines that an emergency vehicle is present, it will send a message to the driver to warn them.

[17th Example]
Next, a seventeenth embodiment will be described. In the seventeenth embodiment, a plurality of cameras are installed inside the vehicle, and the processing unit has a function of detecting the positions of other cameras captured in an image as an image analysis function, and executes a process of identifying the positional relationship between the plurality of cameras as a process corresponding to the analysis result.

It is possible to obtain images of a wide area inside the vehicle that is difficult to capture with a single camera. When the positional relationship of the multiple cameras is specified, the positional relationship of the images captured by each camera can be specified.

According to a modification of the seventeenth embodiment, the processing unit has a function of identifying the models of other cameras captured in the image and identifying the angle of view from the models. The processing unit further has a function of synthesizing images captured by the multiple cameras. By synthesizing the images, the situation inside the vehicle can be grasped more intuitively than when images from each camera are displayed.

According to another modification of the seventeenth embodiment, at least some of the functions of the processing unit are realized by a viewer such as a personal computer. The processing unit displays a sign such as a camera icon in the image to be displayed so that the position of the camera can be known. It is difficult to identify the position of the camera in the vehicle from the image captured by the camera (self). The processing unit detects the position of the camera in the vehicle from the image of the other camera in which the camera is captured. For example, when images captured by three cameras are synthesized and displayed, the processing unit displays the positions of the three cameras in the synthesized image using icons or the like. Furthermore, the icons are made different depending on the model or angle of view of the camera.

According to yet another modification of the seventeenth embodiment, the processing unit has a function of displaying an image captured by each camera, and displaying the position of the camera and the angle of view of the camera (the area captured by the camera) in the image. For example, the angle of view of the camera is displayed in the image according to the function of the 360-degree spherical camera, the 360-degree semi-spherical camera, the wide-angle camera, etc. A user who sees the image can intuitively grasp whether each camera is capturing the area.

[Eighteenth embodiment]
Next, an 18th embodiment will be described. In the 18th embodiment, the camera mounted on the vehicle is a 360-degree camera having an image circle smaller than the diagonal length of the imaging surface, and the processing unit has, as an image analysis function, a function of transforming a basic pattern for calculating image feature amounts according to a position within the image circle.

The processing unit has a function of calculating the image feature amount using the basic pattern without correcting the image distortion in the image circle. This can reduce the load on the processing unit. The basic pattern is defined for an image obtained by capturing an image using a standard lens (a general wide-angle lens). Projection methods for obtaining an image circle include, for example, orthogonal projection, equidistant projection, stereoscopic projection, and equisolid angle projection, and the processing unit transforms the basic pattern according to these projection methods. A circular fisheye lens is used as the camera lens to obtain an image circle. The imaging surface of the camera is rectangular, and the diameter of the image circle is shorter than the long side and equal to or shorter than the short side.

A template pattern of an image to be detected is registered in advance in the processing unit as a basic pattern. The processing unit deforms the template pattern according to a position in the image circle. In addition, the processing unit has a function of recognizing an image using a Haar-like feature, for example, and a function of deforming each of a plurality of rectangular patterns for obtaining a Haar-like feature amount according to a position in the image circle.

In a modified example of the 18th embodiment, the processing unit has a function of limiting the area in the image circle in which a specific object to be detected is captured when performing image recognition, and performs image recognition within the limited area. This function is realized by a drive recorder. In general, the recommended installation position of the drive recorder is determined in advance, so the area in the image circle in which the driver is captured, the area in which the traffic light is captured, the area in which the person in the back seat is captured, etc. are almost determined. Therefore, depending on the person or object to be detected, the area in which image analysis is performed to detect the person or object is limited to the area in which the person or object is captured. For example, when analyzing an image in front of the vehicle, the processing unit performs analysis on the area in which the windshield is captured.

In another modification of the eighteenth embodiment, the processing unit converts a specific area of the image circle into a planar image (parallel projection image), and performs matching on the converted planar image using a normal basic pattern.

In yet another modification of the eighteenth embodiment, the processing unit maps the image in the image circle onto a sphere using a predetermined transformation function, zooms into a portion of the image mapped onto the sphere, and performs image analysis on the planar image while ignoring distortion. When it is desired to detect a specific object, such as a road sign, the processing unit zooms into an area in which the specific object appears and performs image analysis. When it is desired to detect a road sign, the processing unit zooms into an area in the forward image in which the road sign appears and performs image analysis.
[19th Example]
Next, a 19th embodiment will be described. In the 19th embodiment, the processing unit has a function of identifying an area of an image that should not be made public from the viewpoint of privacy protection as a privacy protection area as an image analysis function, and executes a process corresponding to the analysis result to generate an image in which a person reflected in the privacy protection area cannot be identified by appearance from an image captured by a camera.

By making it impossible to identify a person, the privacy of that person can be protected. The processing unit has a function of dividing an area into areas in which the interior of the vehicle is captured and areas that should not be made public.
In particular, when an image is acquired using a spherical camera, the inside of the vehicle is captured, but there may be cases where the image of the inside of the vehicle is not desired to be retained. In such cases, the processing unit has a function of blacking out the area inside the vehicle, or of pixelating the area inside the vehicle, and recording the image outside the vehicle as is. The processing unit has a function of distinguishing between an area in which the inside of the vehicle is captured and an area in which the outside of the vehicle is captured by performing image analysis. As an example, the processing unit has a function of determining that an area in which optical flow occurs at a speed equivalent to the vehicle speed is the outside of the vehicle. The processing unit has a function of allowing the user to set a privacy protection area in advance.

In a modification of the nineteenth embodiment, the processing unit has a function that allows the user to select between a normal mode in which an image that allows a person in the privacy protection area to be identified and a privacy mode in which an image that does not allow a person in the privacy protection area to be identified by appearance is recorded. The user can switch between the normal mode and the privacy mode depending on the purpose and situation of using the vehicle.

According to another modification of the nineteenth embodiment, the processing unit has a function of determining an area in which a person is photographed as a privacy area. The processing unit has a function of detecting a person by performing image analysis, and determining an area surrounded by the outline of the detected person as a privacy area. Furthermore, the processing unit
The system has a function of setting only the face of a detected person as a privacy area. In this case, the processing unit has a function of displaying the person as a silhouette. Alternatively, the processing unit has a function of displaying only the outline of the person, with no person inside the outline, in an image. In this case, the person appears transparent. The processing unit has a function of displaying the edges of moving objects as the outline of the person.

In yet another modification of the nineteenth embodiment, the processing unit has a function of displaying an avatar in place of a person in an area in which the person is photographed. The avatar to be displayed is one that resembles the original person. For example, an avatar that reflects the characteristics of the person, such as gender, body shape, and facial shape, is displayed.
The person and the avatar are associated in advance, and the processing unit identifies the person appearing in the image,
The system has the function of displaying the avatar associated with the identified person. The avatar is
A character sequence is created for each person in advance and registered in the processing unit. The processing unit has a function to move the avatar in accordance with the movements of the person. For example, when a person raises their hand, the avatar raises their hand, and when the person turns to the side, the avatar also turns to the side. The processing unit has a function to change the avatar's facial expression in accordance with the person's facial expression. For example, the processing unit detects when a person is smiling,
When it detects a sad or angry expression, it changes the avatar's expression to a smiling, sad, or angry expression, respectively. The processing unit has a function of reading the person's emotions and changing the color of the avatar according to the person's emotions. For example, when it detects that the person is angry, it changes the color of the avatar to a color that represents anger, such as red.

As yet another modification of the 19th embodiment, the processing unit has a function of generating an image from an image captured by the camera, in which a person appearing in the privacy protection area cannot be identified by appearance, and recording the generated image, before recording the image. Furthermore, the processing unit has a function of converting the image captured by the camera into an image in which a person appearing in the privacy protection area cannot be identified by appearance when playing back the image.

[Twentieth Example]
Next, a twentieth embodiment will be described. In the twentieth embodiment, the processing unit executes a process corresponding to the analysis result, which is to convert the result of the image analysis into text data or code and record it.

By converting the image into text data or encoding it becomes possible to easily find a desired image from among the recorded images. For example, the processing unit generates text data such as "person detected" when a person is detected in the image, and generates text data such as "red light" when a red light is detected in the image. The processing unit has a function of recording the generated text data in association with the image that was the source of the text data generation. In yet another variation, the processing unit has a function of encoding (converting to binary data) the image analysis result by associating the image analysis result with a code instead of text data.

[Twenty-first embodiment]
Next, a description will be given of Example 21. In Example 21, the processing unit has an image analysis function of detecting areas of black crush in an image, and executes a process corresponding to the analysis result to brighten and restore the areas of black crush in the image and display them on the display screen.

A user who sees the image can easily recognize objects or people in the area with blackout. In particular, when an image is acquired using a spherical camera, the bright scenery outside the vehicle and the dark interior of the vehicle are simultaneously captured in the image, so the area in which the interior of the vehicle is reflected is prone to blackout. The process of detecting the blackout area and restoring it to a brighter state for display is particularly effective when an image is acquired using a spherical camera. As the process of restoring the image to a brighter state, the processing unit performs, for example, gamma correction, color correction, conversion to a black and white image, contrast enhancement, and the like.

[Twenty-second Example]
Next, a 22nd embodiment will be described. In the 22nd embodiment, the processing unit has a function of allowing a user to divide an imaging range captured by a camera into a plurality of sections and a function of allowing the user to weight each section, and has a function of determining imaging conditions based on the brightness of each section and the weight assigned to the section, and causing the camera to capture an image under the determined imaging conditions.

By increasing the weight of the section that the user wants to see in the image and decreasing the weight of the section that the user does not need to see, the user can acquire an image under imaging conditions suitable for the area that the user wants to see. The user can divide the imaging range into any shape according to the shape of the area that the user wants to see and the area that the user does not need to see.

5 shows an image plane diagram for explaining a method of dividing an image capture range into a plurality of sections in a comparative example. An image circle 73 is formed on a rectangular or square image capture plane 71 of a 360-degree camera for the front, and an image circle 74 is formed on a rectangular or square image capture plane 72 of a 360-degree camera for the rear. In the image circle 73 for capturing an image of the front, the view ahead is reflected through the front window, and the dashboard, instrument panel, display screen of the car navigation system, etc. in the vehicle are also reflected. In the image circle 74 for capturing an image of the rear, the driver, passenger seat, rear seat, and ceiling in the vehicle are also reflected, and the views to the sides and rear of the vehicle are reflected through the side windows and rear window.

The imaging surfaces 71 and 72 are divided into a plurality of sections by a rectangular lattice. The user can assign a weight to each section.

Fig. 6 shows a diagram of an imaging plane for explaining a method of dividing an imaging range into a plurality of sections in a system according to the 22nd embodiment. As in the example shown in Fig. 5, an image circle 73 is formed on an imaging plane 71 of a 360-degree camera for the front, and an image circle 74 is formed on an imaging plane 72 of a 360-degree camera for the rear.

When the user divides the area, the processing unit 51 (FIG. 4) divides a part of the image capturing range into a closed curve 7
The user then surrounds the area with the closed curve 75 and sets the area inside the closed curve 75 as one section.
A plurality of sections are set by drawing a plurality of closed curves 75. When the user divides an inner area surrounded by the closed curves into two with curves, each of the divided areas becomes one section.

Compared to the comparative example shown in FIG. 5, where weighting is uniformly applied to sections divided into a square lattice shape regardless of the content of the image, it is possible to perform weighting that is more appropriate for the user.

In particular, when a 360-degree camera is used for imaging, the difference in the position within the imaging range where the vehicle parts are captured becomes larger for each vehicle model compared to when a general camera is used. Therefore, there is a high possibility that the relationship between the general-purpose weighting table set in advance in the firmware of the processing unit and the actual expected division of the imaging range does not match. The function of determining the weighting using PC browser software or the like, rather than using a general-purpose weighting table, is particularly effective when a 360-degree camera is used.

The processing unit determines, as the imaging conditions, for example, gain, exposure time, etc. In a configuration in which two hemispherical cameras are arranged with their backs facing each other to form a celestial sphere camera, the processing unit makes the imaging conditions of the two hemispherical cameras the same. This provides the effect that when images captured by the two hemispherical cameras are combined, the boundary between the two images becomes difficult to distinguish.

In a modification of the 22nd embodiment, a processing unit (e.g., AI software of a PC) has a function of analyzing an image captured by a camera without weighting and estimating what is in which part of the image range. For example, the processing unit assigns a weight of "5" to the road part, a weight of "3" to the face part, a weight of "1" to the instrument panel part, and a weight of "0" to the outside of the range of the circle image.

When the processing means has the user divide the image, the processing means may have the user enclose a part of the image capture range with a closed curve, and the inner area enclosed by the closed curve may be regarded as one section. The processing means may have the user input a weighting value for each section that has been set. The input value may be selected from a list such as a pull-down or pop-up.

In another modification of the twenty-second embodiment, the processing unit has the following function instead of allowing the user to input the weight as a numerical value:
In addition to the function of inputting a numerical value, the processing unit has a function of inputting the type of the section. A weight is associated with the type of section in advance, and when the type of section is input, the processing unit associates the weight associated with the type with the section. As the type of section, for example, types such as driver, meter panel, car navigation screen, windshield, rear seat, passenger seat, etc. are provided. The processing unit has a function of allowing the user to change the correspondence between the type of section and the weight. A list of the types of section can be registered in the processing unit in advance by the user.

In yet another modification of the twenty-second embodiment, the processing unit has a function for having the user specify an area within the imaging range that the user wants to see as a function for weighting each area, thereby setting the area. The weight of the area that the user wants to see, specified by the user, is made larger than the weight of the other areas. This function is performed by inputting the type of area by the user, and by dividing the area into the area that the user wants to see,
This corresponds to an example of dividing the area into two types, the other areas.

In yet another modified example of the twenty-second embodiment, the processing unit has a function of having the user divide the imaging range in an interactive format (wizard format). The processing unit outputs a message such as "Please surround the unnecessary area" or "Please fill in the unnecessary area", and recognizes the area surrounded or filled in by the user as an "unnecessary area". Furthermore, the processing unit outputs a message such as "Please fill in the driver's area" or "Please fill in the road area", and recognizes the filled in area as a section having a driver type or a section having a road type. The processing unit has a function of having the user perform an operation of surrounding with a closed curve or an operation of filling in using a pointing device such as a touch panel or a mouse.

In yet another modification of the twenty-second embodiment, the processing unit realizes a function of dividing the imaging range into a plurality of sections based on the captured image by artificial intelligence (AI). The processing unit has a function of automatically distinguishing between the inside and outside of the vehicle, the seats, the dashboard, the instrument panel, etc. from the image, and dividing the imaging range for each automatically distinguished component part. The processing unit displays the division result on the display screen, and allows the user to determine whether the division is acceptable. The processing unit has a function of allowing the user to correct the division. If the user determines that the division made by the AI is incorrect, the user can correct the division. The processing unit has a function of allowing the user to select the correct type from a pull-down or pop-up list. If the user determines that the weight or type corresponding to the section is incorrect, the user can easily correct the weight or type.

[Twenty-third embodiment]
Next, a 23rd embodiment will be described. In the 23rd embodiment, the processing unit has a function for allowing the user to divide the imaging range captured by the camera into a plurality of sections and a function for allowing the user to weight each section, and executes a process corresponding to the analysis result in which a block in a section with a relatively small weight is compressed to a larger size than a block in a section with a relatively large weight.

A user can increase the compression level of low-importance sections by increasing the weight of high-importance sections and decreasing the weight of low-importance sections. As a result, the volume of image data can be reduced while maintaining high image quality in high-importance sections compared to compressing the entire image uniformly. For example, the processing unit compresses high-weight sections into blocks of 8×8 pixels and low-weight sections into blocks of 16×16 pixels.

In a modification of the twenty-third embodiment, the processing unit has a function of painting the area outside the image circle black. An image of the lens barrel may be captured in the area outside the image circle formed on the rectangular imaging surface of the camera. By painting the unnecessary image in the area outside the image circle black, the capacity of the image data including the pixels outside the image circle can be reduced. In another modification, the processing unit has a function of acquiring image data by raster scanning only the pixels inside the image circle, excluding the pixels outside the image circle. This makes it possible to reduce the capacity of the image data.

[Twenty-fourth embodiment]
Next, a 24th embodiment will be described. In the 24th embodiment, the processing unit has, as the image analysis function, a function for detecting a following vehicle and a function for detecting the distance to the following vehicle, and executes a process for detecting aggressive driving by the following vehicle based on the distance to the following vehicle as a process corresponding to the analysis result.

The processing unit executes processing according to the determination result, thereby making it possible to take appropriate measures against aggressive driving. For example, when the same following vehicle continues to exist within a predetermined determination distance for a predetermined determination time or more even though the vehicle speed is equal to or greater than a predetermined reference value, the processing unit determines that the following vehicle is aggressive driving.

In a modification of the twenty-fourth embodiment, a camera having a function of measuring the distance to an object for each pixel is used. For example, a back-illuminated time-of-flight distance image sensor is used as the image sensor of the camera.

In another modification of the 24th embodiment, when the processor determines that the following vehicle is aggressive, it performs processing according to the determination result, such as notifying the driver, recording an image of the rear, advising the driver, and reporting to a traffic enforcement agency (e.g., the police). When reporting to a traffic enforcement agency, the processor attaches an image of the aggressive vehicle and its driver. When the processor detects through image analysis that the aggressive vehicle has overtaken the vehicle and gone around in front of the vehicle, it notifies the driver that there is a possibility of harassment by sudden stops.

In yet another modification of the twenty-fourth embodiment, the processor has a function of detecting people and vehicles around the vehicle and controlling the vehicle according to the distance to the people and the vehicle. For example, the processor performs control to stop the vehicle when it determines that there is a high possibility of contact with a person based on the speed of the vehicle, the distance to a person ahead, and the determination result of whether or not a person is on the planned driving route of the vehicle.

The scope of the present invention is not limited to the configurations expressly described in the specification, but includes combinations of various aspects of the present invention disclosed herein. The configurations of the present invention that are sought to be patented are specified in the appended claims, but it is the intention to claim configurations disclosed in this specification in the future, even if they are not currently specified in the claims.

The present invention is not limited to the configurations described in the above-mentioned embodiments. The components of the above-mentioned embodiments and modifications may be arbitrarily selected and combined. Any component of each embodiment or modification may be arbitrarily combined with any component described in the Summary for Solving the Problems of the Invention or any component embodying any component described in the Summary for Solving the Problems of the Invention. We intend to obtain rights to these as well through amendments to this application or divisional applications, etc.

In addition, we intend to obtain rights to the overall design or partial design by converting the application to a design application. The drawings show the entire device in solid lines, but they include not only the overall design but also partial designs claimed for a portion of the device. For example, some components of the device may be included as partial designs, and some parts of the device may be included as partial designs regardless of the components. A portion of the device may be a component of the device, or it may be a part of that component. We intend to obtain rights to the overall design, as well as partial designs in which any part of the solid line portion of the drawings is shown as a broken line.

REFERENCE SIGNS LIST 1 Drive recorder 2 Windshield 3 Rearview mirror 4 Power cable 5 Cigarette lighter socket 10 Drive recorder body 11 Camera 12 GPS receiver 13 SD card reader 14 Acceleration sensor 15 Speaker 16 Display 17 Operation buttons 20 Processing unit 21 Communication device 30 Server 31 Processing unit 32 Communication device 33 Input device 34 Display 40 Communication network 50 External device 52 SD card reader 53 Input device 54 Display 60 SD card 71, 72 Imaging surface 73, 74 Image circle 75 Closed curve

Claims (6)

A system having a processing means having a function of performing image analysis of an image captured by a camera mounted on a vehicle and executing a process corresponding to the analysis result based on the analysis result,
The processing means has, as a function of the image analysis, a function of detecting a change in the state of a display unit of an apparatus having a display unit mounted in the vehicle, and, as a process corresponding to the analysis result, executes a process of associating the detected sound information with the apparatus whose display unit state has changed, and storing the detected sound information and the apparatus whose display unit state has changed, when the detected sound information and the change in the state of the display unit are synchronized;
The system is characterized in that the processing means has a function of, when detecting sound information from a microphone, identifying the device that emitted the sound based on the association between the stored sound information and a change in the state of the display unit. 2. The system according to claim 1, further comprising a function for recording images captured by the camera. 3. The system according to claim 1 or 2, wherein the processing means has a voice recognition function, determines whether the sound information from the microphone is generated from a specific device, and cancels the voice recognition function if the sound information is generated from a specific device. The system according to any one of claims 1 to 3, characterized in that the processing means has a function of recording an image surrounding the area of the image inside the vehicle in which the device that generated the sound is shown , superimposed on the original image, when it is determined that the sound information from the microphone is generated from a specific device. The system according to any one of claims 1 to 3, characterized in that the processing means has a function of recording a translucent pattern over an area of the image inside the vehicle in which the device that generated the sound is shown when it is determined that the sound information from the microphone is generated from a specific device. A program for causing a computer to realize the functions of the processing means of the system of any one of claims 1 to 5.

Images