JP7742638B2 - Information processing systems and programs - Google Patents

Description

The present invention relates to an information processing system, a program, etc.

JP 2018-160135 A (Patent Document 1) is a background technology in this technical field. This publication states, "The vehicle attention warning device 10 is a system mounted on an autonomous vehicle, and is equipped with a microphone 22 for capturing sound outside the front vehicle, a microphone 23 for capturing sound outside the rear vehicle, a front camera 24, a rear camera 25, a camera 26, a front speaker 31 for sound outside the front vehicle, a rear speaker 32 for sound outside the rear vehicle, a first display 33, and a second display 34, and is used to warn passengers inside or outside the vehicle when they exit the vehicle." (See Abstract.)

Japanese Patent Application Laid-Open No. 2018-160135

The conventional system has various problems, for example, the Patent Document 1 does not consider a mechanism for capturing images of the interior of the vehicle from the rear side.
Therefore, the present invention provides a system that is superior to conventional systems, such as a mechanism for photographing the interior of a vehicle from the rear side.

The objectives of the present invention are not limited to this, and the applicant intends to obtain rights, through divisional applications, amendments, etc., for configurations that aim to achieve the effects achieved by the components disclosed in this specification and drawings, etc. For example, this specification discloses problems in which phrases such as "can do" and "is possible" are read as "the problem is." Each problem is described as an independent entity, and the applicant intends to obtain rights for the configurations that solve each problem separately through divisional applications, amendments, etc. Even if the problems are implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in this specification within the scope of the patent claim through amendments or divisional applications. Furthermore, configurations that solve problems that combine these independent problems have also been disclosed, and the applicant intends to obtain rights for them.

In order to solve the above problems, for example, the configurations described in the claims are adopted.
The present application includes multiple means for solving the above-mentioned problems, and one example is characterized by processing video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle, for example, by processing video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle and video data of the exterior of the vehicle taken from the rear of the vehicle.

According to the present invention, it is possible to provide a system that is superior to conventional systems, such as a system for photographing the interior of a vehicle from the rear side.
Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments.

The effects of the present invention are not limited to these; the effects achieved by the components disclosed in the present specification and drawings are also disclosed, and we intend to obtain rights to the components that achieve these effects through divisional applications, amendments, etc. For example, in this specification, phrases such as "can" and "is possible" are descriptions that clearly state the effects achieved, and there are also parts that demonstrate effects even without phrases such as "can" and "is possible." Furthermore, there are effects that can be understood from the components in question even without such phrases.

FIG. 1 is an explanatory diagram illustrating an example of the installation positions and shooting directions of a photographing device 101R and a rear camera 102 for photographing the interior of a vehicle 10 according to this embodiment. FIG. 2 is an example of a connection diagram of the image capturing devices 101F and 101R, the rear camera 102 for capturing images inside the vehicle, the user terminal 103, and the external server 104. FIG. 3 shows an example of the hardware configuration of the image capturing device 101R connected to the rear camera 102 for capturing images inside the vehicle. FIG. 4 shows an example of a processing flow 400 for an image captured by the rear camera 102 for capturing images inside the vehicle. FIG. 5 shows an example in which two cameras are installed. FIG. 6 shows an embodiment in which the photographing direction of the camera in the embodiment of FIG. 1 can be changed. FIG. 7 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle shown in FIG. 1 are configured as a single wide-angle camera. FIG. 8 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of an ambulance vehicle. FIG. 9 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of a welfare vehicle. FIG. 10 is a diagram illustrating an image captured by the rear camera 102 for capturing images inside the vehicle. FIG. 11 is a diagram illustrating an image captured by the rear camera 102 for capturing images inside the vehicle. FIG. 12 is an example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. FIG. 13 is another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. FIG. 14 is yet another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. FIG. 15 shows an example of a variation of the attachment. FIG. 16 shows an example of another variation of the mounting fixture. FIG. 17 shows an example of a rear camera 102 for photographing the interior of a vehicle, which has a pin plug. FIG. 18 is an example of an enlarged view of another anti-rotation structure of the camera of FIG. FIG. 19 shows an embodiment in which the camera of FIG. 17 is attached to a cylindrical photographing device. FIG. 20 is an explanatory diagram illustrating an example of a procedure for attaching the camera of FIG. 17 to a digital camera type photographing device. FIG. 21 shows an embodiment in which two cameras shown in FIG. 17 are attached to a digital camera type photographing device using a branching cable 1804. FIG. 22 shows another example of a wide-angle camera 100. FIG. 23 shows an example of a camera that blocks the light from a projector. FIG. 24 is a diagram showing an example of the external configuration of the photographing device 101F. FIG. 25 is a diagram showing the configuration of the bracket 600. As shown in FIG. FIG. 26 is a block diagram showing the electrical configuration of the photographing device 101F. FIG. 27 is an explanatory diagram of a screen displayed by the photographing device 101F. FIG. 28 is an explanatory diagram of an example of a viewer. FIG. 29 is an explanatory diagram of display elements included in the viewer of FIG. FIG. 30 is an explanatory diagram of display elements included in the viewer of FIG. FIG. 31 is an explanatory diagram of display elements included in the viewer of FIG. FIG. 32 is an explanatory diagram of an example of a viewer. FIG. 33 is a diagram showing another example of the rear camera 102 for photographing the interior of the vehicle, different from that shown in FIG.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment shown below is one example of the present invention, and the content of the present invention should not be construed as being limited based on the following description.
In addition, the following explanation will specifically describe the individual components of an information processing system that processes video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle and video data of the exterior of the vehicle taken from the rear of the vehicle. However, the components of this system are not limited to these, and other configurations and forms may be used as long as they can perform the functions described in relation to the individual components.
Furthermore, the video data may be data that constitutes either a still image or a moving image, but moving images are particularly preferable. In the following explanation, the term "video" may refer to a still image (also called a still image) or a moving image (also called a moving image) that can be seen by the human eye, but it may also be interpreted as "video data,""image," or "image information" as long as it is consistent with the technical content.

FIG. 1 is an explanatory diagram illustrating an example of the installation positions and shooting directions of a photographing device 101R and a rear camera 102 for photographing the interior of a vehicle 10 according to this embodiment.
FIG. 1( a ) is a side view of a vehicle 10 , and FIG. 1( b ) is a top view of the vehicle 10 .

As shown in Figures 1(a) and 1(b), vehicle 10 has imaging device 101R at the rear of the passenger compartment. Imaging device 101R is an electronic device with imaging capabilities, and may be a camera installed for security, surveillance, or other sensing purposes, but is preferably a rear camera that works in conjunction with a drive recorder, and most preferably is the drive recorder itself.

1 shows the interior rear camera 102 mounted on the image capture device 101R, side by side on the left and right of the vehicle, to capture images of the interior of the vehicle from behind. The interior rear camera 102 is mounted on the rear side of the vehicle 10, preferably rearward of the center of the interior of the vehicle 10, as viewed from the front to the rear of the vehicle 10. It is particularly advantageous to position the interior rear camera 102 in the area between the rear seat headrests and seatbacks and the rear window, and it is particularly preferable to mount the camera on the rear window. However, the method of mounting the interior rear camera 102 on the image capture device 101R is not limited to this. The image capture device 101R and the interior rear camera 102 can be mounted in various ways, as described below. For example, the interior rear camera 102 and the image capture device 101R can be connected using a mounting fixture.

In the example of Figure 1, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are shown directly adjacent to each other, but they may also be connected via a mounting bracket.In this case, if the cables for both the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are pulled out from between the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using, for example, an L-shaped cable, and then combined into a single cable and connected to an existing line that connects to a power source, etc., the rear camera 102 for photographing the interior of the vehicle can be added without the wiring being noticeable and without detracting from the aesthetic appearance of the interior of the vehicle.
Similarly, even if there is additional front-side imaging equipment inside the vehicle and a cable connection for power and data communication exists between the rear-side imaging equipment 101R and the front-side imaging equipment, it is possible to integrate both cables pulled out from the space and connect them to the connection cable with the front-side imaging equipment.

The two cameras, one for the rear rear and one for the interior of the vehicle, may be integrated. This configuration allows the imaging device 101R and the interior rear camera 102 to be handled as a single unit, which offers the advantage of ease of installation and convenience for the user. For example, the lens for the interior of the vehicle may be positioned higher than the lens for the rear rear (e.g., vertical camera configuration). In particular, when the lens for the rear rear is installed in a position where it can capture rearward images through the rear window, the lens for the interior of the vehicle may be attached using a mounting bracket that is positioned so that its back surface is in the non-glass portion of the upper part of the rear window, particularly in the area where it cannot capture rearward images through the rear window.
Furthermore, in such a configuration, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle may be configured so that the central axes of the respective photographed images are aligned in advance, for example, during the design and manufacturing stages. In this way, it is easy to perform processes such as comparing the images from each camera.
The photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are vertically arranged and integrated, and the central axes of the two are aligned in advance, so that the user can obtain an image of the interior of the vehicle and an image of the exterior of the vehicle behind the vehicle with their centers aligned with each other simply by placing one camera with an integrated vertical camera arrangement.

In another embodiment of the two-camera integrated type, the interior rear camera 102 for taking pictures inside the vehicle may be rotated vertically to change the vertical angle of the shooting direction. Furthermore, if the interior rear camera 102 for taking pictures inside the vehicle is rotated horizontally to change the angle of the shooting direction up, down, left, and right, the desired shooting direction can be set over a wider range.
Furthermore, the user may use his/her own user terminal, a user terminal installed in the vehicle, a remote control, etc. to adjust the angle of the shooting direction of the rear camera 102 for shooting inside the vehicle while checking the image on the display of the user terminal, etc.

In another embodiment, the image capture device 101R and the interior rear camera 102 may be configured as a single wide-angle camera, and the image captured by this configuration may capture both the interior of the vehicle and the area behind the vehicle from rear to front. Such an embodiment is described in more detail in Figure 7.

In another embodiment, the photographing device 101R and the interior-photography rear camera 102 may be configured as two physically independent rear cameras (for example, configured as separate housings) and mounted in different positions on the rear side of the vehicle 10. In particular, the two rear cameras may be mounted near either side of the interior of the vehicle, with one mounted on one pillar at the rear of the vehicle 10 and the other mounted on the other pillar at the rear of the vehicle 10. The pillar position may be on the passenger compartment side of the pillar. In this way, the camera is less likely to be visible to occupants of the vehicle 10, and also less likely to be visible to occupants such as the driver of a vehicle behind the vehicle 10.

In either embodiment, it is advantageous to use the image capture device 101R and the rear camera 102 for capturing images inside the vehicle cabin to record both images captured from the rear toward the front and images of the rear of the vehicle. It is also advisable to record images of the exterior to the sides of the vehicle and the distant exterior behind the vehicle.

The above describes examples of the placement and shooting direction of the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle interior. Below, we will explain examples of the functions and roles of the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle interior.

The interior rear camera 102 is connected to an image capturing device 101R installed at the rear of the vehicle, which performs image processing and compression using an ISP (Image Signal Processor). Using this image capturing device 101R as the main unit is significantly more advantageous than using a front-facing camera attached to the front of the vehicle, for example, the windshield, as the main unit. The front-facing camera is installed at the front of the vehicle 10 and captures images ahead of the vehicle 10.
Due to its mounting location, the front-facing camera is susceptible to the effects of direct sunlight and the resulting heat. On the other hand, because the front-facing camera is installed on the windshield, etc., it is functionally limited by the need to have an open field of view in front of the vehicle, making it difficult to adopt effective shading measures.

In contrast, a rearview camera mounted on the rear window of a vehicle has the advantage of being more effective at reducing the effects of direct sunlight and the resulting heat at its installation location than a frontview camera. Furthermore, compared to the windshield, the rear window can be treated with a certain degree of shading, further reducing the effects of direct sunlight and heat.

In addition, in many vehicles, such as minivans, the rear window is closer to vertical than the windshield. Therefore, there is an advantage that direct sunlight (especially direct sunlight during hot hours when the sun is high in the sky) is less likely to enter the drive recorder when it is located on the rear window than when it is located on the windshield. The rear rearview camera is installed on the rear side of the vehicle 10 and is used to capture images behind the vehicle 10.
The rear rear camera, which is less affected by direct sunlight and heat, is used as the main unit, and this rear rear camera processes and records images from the additionally installed camera for photographing inside the vehicle and the front front camera, making it possible to make the entire system less susceptible to breakdowns and to reduce malfunctions due to heat.

By using the rear camera 102 for capturing images inside the vehicle as shown in FIG. 1, the following types of images can be captured.
The rear camera 102 for photographing the interior of the vehicle may be configured to photograph the interior of the vehicle from behind, so that it can photograph, analyze, and detect people's actions while protecting their privacy and not photographing their faces.
The rear camera 102 for photographing inside the vehicle interior may be configured to photograph the outside of the vehicle through the side window glass from the rear of the vehicle interior to detect the situation around the vehicle.
The rear camera 102 for photographing the interior of the vehicle may be configured to detect the state of an object installed on the ceiling of the vehicle.
The rear camera 102 for photographing the interior of the vehicle may be configured to detect luggage on the luggage rack, on the seats, etc.
The rear camera 102 for photographing inside the vehicle interior may be configured to detect the state of a display unit of the vehicle, such as a monitor provided on the back of a seat or a monitor provided on the ceiling.
The rear camera 102 for photographing the interior of the vehicle may be configured to detect whether or not there is a person inside the vehicle, particularly in the rear space of the vehicle.
The rear camera 102 for photographing inside the vehicle may be configured to detect the parking state based on whether or not there is a person inside the vehicle.
These points are explained in more detail in connection with FIG.

When using an infant-only or rear-facing child car seat, conventional front-facing cameras are unable to see the front seats, making it extremely difficult to check the condition of the infant in the child car seat. However, the interior rear camera 102, which captures the interior of the vehicle from the rear, can capture excellent images of the infant in the child car seat. In fact, by adjusting the capture direction of the image captured by the interior rear camera 102 while checking the image captured by the camera on a handheld device, as described above, it is even possible to place the infant in the center of the image captured by the interior rear camera 102.
This allows the control unit 303 of the photographing device 101R to accurately identify the baby in the child car seat based on the image from the rear camera 102 for photographing inside the vehicle interior, and to reliably detect the occurrence of abandonment, etc. Such abandonment detection and associated warning can be more reliably carried out by a configuration including a human detection sensor, which will be described later.

The drive recorder may also be provided with a function for combining the image from the interior rear camera 102 with images from other cameras to connect images from the rear rear, left and right sides of the rear interior, left and right sides of the front interior, and front of the vehicle to generate images that can be rotated around the vehicle (for example, a full circle). Alternatively, a viewer program for a personal computer or an information processing device such as a smartphone or tablet that plays back data from a recording medium on which these images are recorded may be provided. The drive recorder may also be provided with a function for generating images that can be rotated around the vehicle (for example, a full circle) from images from the rear interior and the front interior.
For example, in a configuration in which the rear rear camera 101R and the rear camera 102 for interior photography each cover a 180° photography range, it is preferable to configure the camera 101R to continuously record sudden approaches from behind and rearward tailgating, as photographed by the rear camera 101R, and side tailgating and dangerous overtaking, as photographed through the window glass of the vehicle interior by the rear camera 102.

The video captured by the interior rear camera 102 should be configured with an angle of view and installation position that not only allows the interior of the vehicle to be seen from the rear, but also captures the driver's seat and passenger seat. This makes it possible to record aggressive driving from the side, or intimidating or violent behavior by getting out of the car. In such situations, the situation could worsen if it becomes known that recording has started. Therefore, it is advantageous to configure the camera to either automatically start recording when it detects that such a situation has occurred, or to use a weak remote control or IR remote control to record events so that the user does not easily notice that they have started recording.

The interior rear camera 102 photographs the interior of the vehicle from the rear (rear side) of the vehicle, so compared to conventional interior cameras that photograph the interior from the front (front side) of the vehicle, the opportunities for frontal photography of people in the driver's seat, passenger seat, and other seats are significantly reduced. This makes it possible to protect the privacy of people inside the vehicle. This is particularly advantageous when used in taxis, buses, etc.

Effects such as highly accurate detection of abandoned vehicles, seamless recording of dangerous driving by other vehicles, and privacy protection can also be achieved by installing a wide-angle camera, such as a 360° camera, on the rear.
For example, even if a vehicle behind approaches from the rear, changes lanes, and cuts to the side of the vehicle, it is advisable to use a 360° camera to record the changing situation without interruption.
In addition, while a 360° camera typically has a large viewing angle and produces small images, it is advisable to configure the camera to have a viewing angle that is as wide as possible to match the interior of the vehicle (enables viewing outside the driver's seat and passenger seat).

For example, with conventional front-side interior cameras installed near the front drive recorder, there is a problem in that the front mirror occupies an unnecessarily large area of the screen, which sometimes makes it difficult to use interior cameras with wide-angle or ultra-wide-angle shooting ranges.
In the above embodiment, the rear camera 102 for photographing inside the vehicle cabin, which acquires video data inside the vehicle cabin, and the photographing equipment 101R for acquiring video data outside the vehicle can be designed separately and more flexibly depending on the purpose of photographing, etc.

The interior rear camera 102 may also be configured to capture images of the left and right front and rear of the vehicle through the windows. The image capturing device 101R may be configured to perform processing by combining these images with images from other cameras. For example, when a vehicle behind approaches or overtakes dangerously, the image captured by the image capturing device 101R, which captures the rear from the rear, may be combined with an image from the perspective of a rear seat passenger to perform a more multifaceted risk assessment than in the past.
In particular, unlike cameras for monitoring the exterior of the vehicle that are attached to the left and right sides of the vehicle outside, the rear camera 102 for photographing the interior of the vehicle can photograph both the state of tailgating from the side and the state of the interior of the vehicle at that time.

The user terminal 103 in FIG. 1 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or it may be a wearable terminal such as glasses, a wristwatch, or clothing. It may also be a stationary or portable computer, or a server located on the cloud or a network. In terms of functionality, it may also be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of multiple of these terminals. For example, a combination of one smartphone and one wearable terminal can logically function as a single terminal. It may also be any other information processing terminal.

The user terminal 103 includes a processor that executes an operating system, applications, programs, etc.; a main storage device such as RAM (Random Access Memory); a secondary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), or flash memory; a communication control unit such as a network card, wireless communication module, or mobile communication module; input devices such as a touch panel, keyboard, mouse, voice input, or input based on motion detection captured by a camera; and output devices such as a monitor, display, or speaker. Note that the output device may also be a device or terminal that transmits information to be output to an external monitor, display, speaker, printer, or other device.

In FIG. 1, the rear camera 102 for recording images inside the vehicle is electrically connected to the image capturing device 101R at the rear of the vehicle cabin, and transmits video data captured from the rear of the vehicle toward the front of the vehicle to the image capturing device 101R. The image capturing device 101R and the rear camera 102 for recording images inside the vehicle cabin may also be connected to the front image capturing device 101F and the front vehicle cabin camera.

In such a configuration, the rear camera 102 for photographing the interior of the vehicle may further be configured to have a different aspect ratio from the front and rear photographing devices 101F, 101R.
For example, the front and rear image capturing devices 101F, 101R may be arranged horizontally (horizontal position in camera terminology) to capture images, and the rear camera 102 for capturing images inside the vehicle may be arranged vertically (vertical position in camera terminology). In other words, the rear camera 102 for capturing images inside the vehicle may be configured to capture images so that the long side of the image is the vertical direction of the image (i.e., the vertical direction) and the short side is perpendicular to the vertical direction. For example, the rear camera 102 for capturing images inside the vehicle may be configured to capture images from the center of the rear left and right directions.

On the other hand, the rear exterior, front exterior, and front interior of the vehicle should preferably be configured in a horizontal arrangement, i.e., the cameras capturing the rear exterior, front exterior, and front interior of the vehicle should capture images such that the longitudinal direction of the images is the horizontal direction of the images.
The rear camera 102 for capturing images inside the vehicle interior should be configured to capture images in a vertically positioned manner using an image sensor with a 4:3 aspect ratio for a standard vehicle or a 16:9 aspect ratio for a vehicle with a long cabin such as a van.
The front interior camera, which takes pictures of the interior from the front, can be positioned vertically. However, a camera that takes pictures of the interior from the rear is preferable because it is easier to install in the center. By installing it in the center, it is possible to configure a seven-seater vehicle so that it can take pictures of a wide range all the way to the front through the center aisle.

In a configuration where both front and rear interior cameras are installed, the front interior camera may be configured to record images that are less clear than the rear interior camera, thereby protecting privacy and making it easier to understand the state of the interior of the vehicle by viewing the recording.
This is an example:
・The rear camera for taking pictures inside the vehicle has a higher resolution (more pixels) than the front camera for taking pictures inside the vehicle.
The front interior camera is a thermal imaging camera, and the rear interior camera is a visible light camera.
- This can be achieved by adopting measures such as using a black and white camera (infrared camera) for the front interior camera and a visible light camera for the rear interior camera.
The in-car camera may also be configured to record (also called audio recording) the sounds inside the car.

When using an interior camera mounted at the front of a vehicle to capture the interior of the vehicle from front to rear, part of the field of view is blinded by the rearview mirror, making it impossible or very difficult to check the condition of luggage piled up behind the driver's seat or passenger seat.
The rear camera 102 for photographing the interior of the vehicle photographs the interior of the vehicle from the rear to the front, so it is desirable to configure it to photograph areas of the interior of the vehicle that are blind spots for conventional front interior cameras such as those behind the driver's seat and passenger seat or for the rearview mirror.
It is also preferable to configure the vehicle interior rear camera 102 so that the shooting direction can be changed to the rear of the vehicle. Such an embodiment will be described later with reference to FIG.

FIG. 2 is an example of a connection diagram of the photographing devices 101F and 101R, the rear camera 102 for photographing the interior of the vehicle, the user terminal 103, and the external server 104.
The photographing devices 101F and 101R, the user terminal 103, and the external server 104 are each connected via a network 105. The network 105 may be wired or wireless, and each user terminal 103 may be configured to send and receive information via the network 105. The network 105 is not limited to the Internet, and may also be a combination of networks with different protocols.

The image capturing devices 101F and 101R are devices that are mounted on, for example, a vehicle and record (also referred to as video recording) images while driving, etc.
However, although the vehicle to be installed is a four-wheeled automobile in this embodiment, it is not limited to a four-wheeled automobile, and any vehicle capable of installing the image capturing devices 101F and 101R may be used. For example, large transport vehicles with four or more wheels, such as buses and trucks, two-wheeled vehicles such as motorcycles and bicycles, and other vehicles may be installed. Furthermore, for example, transportation vehicles such as trains, monorails, and linear motor cars may also be installed.
The photographing devices 101F and 101R may be configured to transmit the acquired various information and information relating to the results of event determination (described later) to the user terminal 103 via the network 105.

The user terminal 103 is a terminal used by, for example, a user who drives a car. The user terminal 103 is preferably configured to be connected to the image capturing devices 101F and 101R via a mobile data communication network or an in-vehicle network such as Wi-Fi (registered trademark) in the car, and to operate the image capturing devices 101F and 101R.
The user terminal 103 may be configured to be connected to the external server 104 via the network 105 and transmit various types of information to the external server 104 .

FIG. 3 shows an example of the hardware configuration of the image capturing device 101R connected to the rear camera 102 for capturing images inside the vehicle.
The photographing device controller 304 (DVR controller 304) is connected to and controls a camera 302 that mainly captures rearward video outside the vehicle, including the area immediately behind the vehicle, an acceleration sensor 305, an SD card 306, a GNSS sensor 307, an input/output unit 315, and other sensors (not shown). Based on information received from these elements, the photographing device controller 304 generates status information indicating the state of the photographing device controller 304. The photographing device controller 304 also transmits the generated status information to the microcomputer 301.
The photographing device controller 304 is connected via the input/output unit 315 to a rear camera 102 for photographing the interior of the vehicle, which is different from the camera 302 and photographs the interior of the vehicle from the rear toward the front.

The microcomputer 301 performs the following various processes, for example.
Controls the communication processing unit 314.
The communication processing unit 314 notifies the photographing device controller 304 of the acquired location information.
The status information acquired from the image capture device controller 304 and the location information acquired from the communication processing unit 314 are converted into, for example, 16-byte data.
- Performs sleep control when powered by a secondary battery.
- Execute test mode during manufacturing and testing.
The microcomputer 301 is programmable, and stores programs for executing the various processes described above. The various processes described above are realized by executing these programs with a processing unit of the microcomputer 301. In this embodiment, the processing unit of the microcomputer 301 or the microcomputer 301 itself may also be referred to as a control unit.

The photographing device controller 304, microcomputer 301, and communication processing unit 314 can each be implemented as an integrated circuit (chip) such as a SoC (System on a Chip), or multiple of these may be implemented together on a single chip. In particular, in this embodiment, the photographing device controller 304 and microcomputer 301 operate in conjunction with each other, so they may be implemented together on a single chip.

Alternatively, the microcomputer 301 may be configured to perform some of the functions of the photographing device controller 304, or the photographing device controller 304 may be configured to perform some of the functions of the microcomputer 301. In this embodiment, the photographing device controller 304 and the microcomputer 301 are collectively referred to as the control unit 303. The control unit 303 may be a single chip, or may be configured to consist of multiple chips that separate the functions of the photographing device controller 304 and the microcomputer 301.

It is preferable to equip the image sensor of the camera 302 with an AI processing function for preprocessing, so that the following output can be performed in addition to or instead of generating a normal captured image. The AI processing function has a function for performing image recognition based on the video captured by these cameras.
- Output metadata of the object from the image.
- Outputs images in ISP (Image Signal Processor) output format, for example, YUV or RGB.
・Output an image that only includes a specific area.
The AI processing function may be configured to perform similar processing on video data transmitted from the rear camera 102 for capturing images inside the vehicle.

By performing preprocessing on the image sensor side in this way, it is possible to reduce the amount of data and enable real-time tracking of objects using high-speed AI processing.
For example, if the photographing equipment 101R is equipped with an AI processing function that performs preprocessing on the image sensor of the camera 302, it is desirable to configure the photographing equipment 101R to independently perform video processing and image processing, and to create image information of ideal image quality.
Similar processing can also be performed on video data transmitted from the rear camera 102 for recording inside the vehicle interior, and the configuration may be such that by updating the recording equipment 101R, processing can be added according to the model and type of the rear camera 102 for recording inside the vehicle interior that is connected.

The acceleration sensor 305 detects acceleration and generates acceleration information. For example, when a sudden change in acceleration occurs due to an impact, sudden steering, sudden stop, etc., the control unit 303 detects the occurrence of an event. The control unit 303 may be configured to analyze the change in acceleration information and estimate the circumstances of an accident or the like that has occurred to a vehicle, etc.
The SD card 306 is a storage device that records various information such as image information and recording information corresponding to the image information. Note that the SD card is not limited to an SD card, and any configuration that can store data will suffice, but a flash memory with high vibration resistance is preferable for in-vehicle use.

The communication processing unit 314 transmits information about the photographing device 101R to the user terminal 103 via the wireless communication antenna 308 and the network.
The GNSS sensor 307 has a location information acquisition unit implemented as an integrated circuit or the like, and acquires location information based on a signal received from a GNSS antenna compatible with a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System). The location information acquisition unit may also be configured to acquire time information, moving speed information, and moving direction information.

The photographing device 101R has a power supply 311 connected to the vehicle power supply 310.
The power supply 311 supplies power received (or sometimes referred to as power reception) from the vehicle power supply 310 to the photographing device controller 304 and also charges the secondary battery 312. Note that an input/output unit 315 (described later) may also be configured to supply power. In this case, the photographing device 101R receives power from the vehicle power supply 310 via the input/output unit 315.

The secondary battery 312 is a rechargeable battery such as a lithium-ion battery, nickel-cadmium battery, nickel-metal hydride battery, or lead-acid battery. However, a non-rechargeable primary battery may also be used. If the accessory power supply (ACC power supply) is turned off, or if the power supply from the vehicle power supply 310 to the power supply 311 of the photographing device 101R is interrupted due to a fault such as a broken or disconnected wire caused by an accident, or if the power supply 311 fails and is no longer able to supply power, the secondary battery transmits (or sometimes called power transmission or power supply) power to the microcomputer 301, communication processing unit 314, etc., to maintain at least some of the functions of the photographing device 101R.

The control unit 303 acquires image information from the camera 302 of the photographing device 101R and stores it in a storage device such as an SD card 306. Note that in this specification, "storage" is sometimes referred to as "recording."

The input/output unit 315 inputs and/or outputs video data captured by the camera 402 or the rear camera 102 for capturing images inside the vehicle interior, and/or video data stored in a storage unit such as the SD card 306. The input and/or output of video data is basically performed in a digital format, but is not limited to this.
The input/output unit 315 is connected to the photographing device controller 304, but may be connected to the microcomputer 301 instead.
The input/output unit 315 is directly connected to the network 105, but may be configured to be connected to the network 105 via the rear camera 102 for photographing the interior of the vehicle.
As described above, the input/output unit 315 may be configured to be able to input and output power from the vehicle power supply 310 .

The rear camera 102 for photographing inside the vehicle is an additional camera connected to the photographing device 101R via the input/output unit 315.
The vehicle interior rear camera 102 transmits the video data captured by the vehicle interior rear camera 102 to the image capturing device controller 304 of the image capturing device 101R via the input/output unit 315.

The rear camera 102 for photographing the interior of the vehicle may be configured to output the video data input from the photographing device 101R to the network 105. The rear camera 102 for photographing the interior of the vehicle may be configured to output the power input from the vehicle power supply 310.
By configuring the rear camera 102 for taking photographs inside the vehicle interior to be able to input and output video data and power, it is possible to pass through the video data and power to the outside simply by connecting the rear camera 102 for taking photographs inside the vehicle interior to the photographing device 101R having the input/output unit 315, which makes it possible to add a new camera without having to change the configuration of the existing photographing device 101R, which is convenient.

FIG. 4 shows an example of a process flow 400 for processing images captured by the rear camera 102 for capturing images inside the vehicle.
The control unit 303 of the photographing device 101R executes the following process in accordance with the process flow 400. In the following description, a description of specific algorithms and settings for image processing and video processing by the control unit 303 of the photographing device 101R will be omitted.

The control unit 303 acquires an image (for example, video data such as a still image or a moving image) from the rear camera 102 for capturing images inside the vehicle (S410) and analyzes the image (S420).
Based on the analysis results, the control unit 303 makes a determination about each of the set situations, which will be described later (S430).
The control unit 303 performs output, which will be described later, based on the determination result (S430).

In the following explanation, an example will be described in which the control unit 303 is the main processing unit in each of steps S410 to S440, but each step may also be performed by, for example, an individual module within the control unit 303 or in a data connection state with the control unit 303.
For example, the image acquisition module may perform the image acquisition processing step S410, the image analysis module may perform the image analysis step S420, the judgment module may perform the judgment step S430, and the output module may perform the output processing step S440, and the control unit 303 may work in conjunction with each module.
Furthermore, each module may be implemented in hardware or software.

A specific example of processing that can be executed according to the processing flow 400 will be described below.
First example: Expected case: Take a picture of the inside of a car from behind, detect people, and apply a filter.
Input example: Image in which a person can be seen Output example: Image in which a person has been subjected to a video filter, etc.

The control unit 303 has a function of recognizing the face position (body position) of the image captured from the rear of the vehicle interior and applying an image filter.
The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines in S430 whether a person is captured, whether the face of the person captured can be identified, or whether only the back of the person is captured.
In S440, when the control unit 303 identifies the face of the person in the image, it creates and outputs an image in which a filter with, for example, an animal's face motif is applied to the area of the face.

In addition to the above process of applying a filter to the person's face in S440, if the control unit 303 recognizes that the video captured inside the vehicle from the rear was taken from behind the person's face (body position), it is preferable to create and output video that has been processed to overlay the back of the face (body) of an animal such as a cat as a filter.

In other words, the image is analyzed, and a filter of the animal's front face is applied to the image taken from the front, and a filter of the animal's back of the head is applied to the image taken from behind.
It is advisable for the control unit 303 to be configured to overlay a 3D model over the entire face of a person in the video (like a mask) and change the display position depending on the direction of the face, but since such processing requires a high computational load, the computational load for processing can be significantly reduced by configuring the camera to overlay a front image on the front side for a camera from the front and a back image on the back side for a camera from the rear.

In this case, by filming the interior of the car from behind, the chances of people's faces being captured are reduced, and by applying a filter to the face area in the photograph, it is possible to film and record the actions of people inside the car while protecting their privacy more than ever before.
Furthermore, by applying a filter to people's faces, etc., it is possible to create video that is even more considerate of the privacy of people inside the vehicle. This can reduce the resistance felt by passengers of buses, taxis, etc., about having their own video recorded. Such resistance can be further reduced by, for example, displaying the filtered video on an in-vehicle monitor so that passengers can check it themselves. With this configuration, it is also possible to set it so that the video without the filter can be viewed if a request from the police or other authorities is granted in the event of an incident or accident.

Second example: Expected case: Detects the situation around the vehicle and issues notifications and warnings based on that situation.
Input example: Images of the vehicle's surroundings Output example: Notifications and warnings about the vehicle's surroundings

In this case, the image capturing device 101R may be configured to capture and record the surroundings of the vehicle by capturing an image of the outside from the rear of the vehicle through a side window (on the side of the vehicle).
The control unit 303 may be configured to analyze the video data captured from the rear of the vehicle interior by the rear camera 102, and perform, for example, side vehicle detection, human detection, and obstacle detection. In this configuration, the rear camera 102 may be a wide-angle camera, and may also be a 180° camera, a hemispherical camera, or a celestial spherical camera.
For example, by specifying the outline of the window glass in advance, the control unit 303 can distinguish between the state inside the vehicle and the state around the vehicle (the state outside the vehicle), and by analyzing only the video data of each part depending on the situation, it is possible to identify the situation with a lower computational load.

The control unit 303 acquires images captured by the rear camera 102 for capturing images inside the vehicle interior at S410, analyzes the captured video data at S420, and determines which parts of the captured video data are images inside the vehicle and which parts are images outside the vehicle at S430, whether other vehicles are captured in the images outside the vehicle, whether dangerous driving by other vehicles is observed, and whether there are any other situations that would trigger a pre-set notification or warning.

Examples of situations that may trigger pre-defined notifications or warnings include:
- When you turn on the turn signal, there are cars, people, or obstacles on the side,
- A situation where someone approaches your vehicle unnaturally while it is parked with no one inside,
- Situations where such a person is looking into the car,
- When starting the engine, slowing down, or driving, there are people, especially children, around the vehicle, approaching or playing with the vehicle.
- Situations where the vehicle is being tailgated from behind or the side,
etc.

In S440, when the determined situation occurs, the control unit 303 outputs a notification or warning by video or audio. Also, the control unit 303 may output a notification or warning about the state around the vehicle that is customized according to each determined situation.
With an in-vehicle camera installed at the front, the front seats (driver's seat, passenger seat) get in the way, making it difficult to take side shots through the rear seat window, making it difficult to make highly accurate judgments. On the other hand, with a configuration in which the front is photographed from the in-vehicle rear camera 102 installed at the rear of the vehicle, as in this embodiment, the sides of the vehicle and the front side can be photographed well, making it possible for the control unit 303 to judge these situations with high accuracy.

For example, if the control unit 303 determines that another vehicle is tailgating the vehicle by analyzing the video data, the control unit 303 will output a warning sound from the in-vehicle speaker or a voice message such as "Your vehicle is being tailgated from behind." A configuration that allows the driver to immediately recognize the occurrence of tailgating from behind the vehicle is extremely advantageous from the perspective of safe driving, as it gives the driver more time and psychological leeway.

The control unit 303 may be configured to analyze the video data to determine the presence of a suspicious individual approaching the parked vehicle while the driver or passengers are away from the vehicle, and then send a notification or warning about the approaching or suspicious individual to a user terminal such as a smartphone. In particular, the control unit 303 may be configured to determine if a suspicious individual is peering into the vehicle interior in the blind spot of the front camera located in a straight line between the front cabin camera and the front seats (driver's seat, passenger seat). This processing may be performed in combination with, for example, the seventh example described below.

A configuration in which a wide-angle rear camera 102 for capturing images inside the vehicle is installed at the rear of the vehicle not only allows for recording of the interior of the vehicle, but also allows for clear recording of the surrounding area through the window glass on both sides. With this configuration, images can be captured outside the driver's seat, passenger seat, and rear seat windows without the front seats getting in the way.

Third Example: Expected Case: Detect the status of an object installed on the ceiling of a vehicle, and issue a notification or warning based on that status.
Input example: Images of objects installed on the vehicle ceiling. Output example: Notifications and warnings about the status of objects installed on the vehicle ceiling.

The control unit 303 may be configured to analyze the captured video data to identify the status of objects installed on the vehicle ceiling, such as room lamps, nets, projectors, flip-down monitors, etc., installed on the vehicle ceiling.
The control unit 303 acquires an image taken by the rear camera 102 for photographing the interior of the vehicle in S410, analyzes the taken image in S420, and determines in S430 whether an object installed on the vehicle ceiling is captured, whether the captured object on the vehicle ceiling is in an appropriate position or state, whether the object installed on the vehicle ceiling is in contact with passengers, particularly their heads, or whether there is a risk of contact.
In S440, if the control unit 303 determines that an object installed on the vehicle ceiling is not in an appropriate position or state, or that the object installed on the vehicle ceiling is in contact with or at risk of contact with a passenger, particularly a passenger's head, it outputs a warning about the situation.

For example, the control unit 303 may be configured to analyze video data to determine whether the interior lamp is on and whether there is anyone inside the vehicle. If the control unit 303 determines that the interior lamp remains on even after the door has been closed and that there is no one inside the vehicle, it will issue a warning from a speaker inside the vehicle or from a user terminal carried by the driver. This can effectively prevent the battery from running out.

In another example, the control unit 303 may be configured to analyze the video data to determine the condition of the net attached to the ceiling, for example, whether the net is sagging due to being overloaded with luggage, or whether the left and right fasteners have come loose. When the control unit 303 determines that the net is sagging due to being overloaded with luggage, or that the left and right fasteners have come loose, it outputs a warning sound or a warning voice such as "The net is in an abnormal state" from speakers inside the vehicle. This allows the user to adjust the net to an optimal state in advance before it breaks, before luggage falls off the net, or before the net comes loose unexpectedly.

In yet another example, the control unit 303 may be configured to analyze the video data to determine the orientation and state of a projector mounted on the ceiling, the position and state of the projector's image, and the relative positions of the projector and passengers in the rear seats.
For example, the control unit 303
- The projector is adjusted to the desired position or direction.
- Whether the projector is running when there are no passengers on board, or conversely, whether the projector is running when there are passengers on board,
- When the image is being projected, is the positional relationship between the image and the screen appropriate?
- Check whether passengers are hitting their heads on the projector.
It is preferable to have a configuration that determines the above.
When the control unit 303 determines that one of these situations has occurred, the control unit 303 may output a warning sound, a warning voice such as "The projector is in an abnormal state," or a warning image from a speaker or monitor in the vehicle.

The projector in this example may be configured to project advertisements or the like onto a screen placed on the side window glass of a vehicle such as a taxi.
Based on these warnings, the driver can adjust the projector's position, status, settings, etc. as appropriate. For example, when there are no passengers, the projector can be used to reliably project advertising images toward the outside of the vehicle, and when there are passengers inside, the projector can be stopped to ensure passenger comfort. Also, when there are no passengers, the projector can be used to reliably project advertising images toward the outside of the vehicle, and when there are passengers inside, the projector can be used to project images aimed at those passengers, thereby increasing the effectiveness of advertising.

In yet another example, the control unit 303 may be configured to analyze the video data to determine the open/closed state of a flip-down monitor mounted on the ceiling, its activation state (whether or not video is being displayed), and the like.
For example, the control unit 303
- The flip-down monitor is in the stopped or closed position when passengers are on board.
Conversely, the flip-down monitor is playing video when there are no passengers on board, or the flip-down monitor is in the open position.
It is preferable to have a configuration that determines the above.
When the control unit 303 determines that such a situation has occurred, the control unit 303 outputs a warning sound or a warning voice or image such as "Please check the status of the flip-down monitor" from the speaker, monitor, etc. in the vehicle.

Based on such warnings, the driver can appropriately adjust the state and settings of the flip-down monitor. This allows, for example, a taxi to stop video playback on the flip-down monitor when there are no passengers, thereby reducing power consumption, while still reliably playing advertising videos and the like when there are passengers in the vehicle.
In addition, the rear camera 102 for photographing the interior of the vehicle, which is installed on the rear side of the vehicle, can clearly recognize the images displayed on the flip-down monitor, so the control unit 303 can record the passenger's level of interest and concern for each advertising image by analyzing data such as the length of time that the passenger spent facing the flip-down monitor for each image.
The control unit 303 may also be configured to use the results of analyzing such records to play advertising videos that are likely to be of greater interest to passengers, or to transmit the analysis results via the network 105 to an external server 104 of a company that provides the advertising videos.

Fourth Example: Expected Case: Detect the state of luggage on the luggage rack or on the seats, etc., and issue notifications or warnings based on that state.
Input example: Images showing the state of the luggage Output example: Notifications and warnings about the stability of the luggage, etc.

In this case, the control unit 303 may be configured to detect the state of luggage placed behind the driver's seat, for example, by analyzing the video data.
The control unit 303 acquires an image captured by the interior camera 102 in S410, analyzes the captured image in S420, and determines in S430 the state of luggage placed or stacked on the luggage rack or seats, for example, the relative position of the luggage relative to the luggage rack or seats, the estimated position of the center of gravity of the luggage, the relative position of the centers of gravity of multiple luggage, the speed and acceleration of the vehicle and changes in the position of the luggage related to these changes, etc. The center of gravity of the luggage may be simply estimated, for example, as the center point of a two-dimensional shape recognized as luggage on the screen, but may also be estimated by other methods.

In S440, if the control unit 303 determines that there is a risk of luggage falling off the loading platform or seats, that there is a risk of stacked luggage collapsing, or that the luggage is slipping more than a set value on the loading platform or seats, the control unit 303 outputs a warning about that situation. The control unit 303 may output such a warning while the vehicle is in operation, or may detect the status of the luggage when a person gets into the vehicle and output a notification according to that status.

Interior images taken from behind the vehicle can provide particularly good images of the condition of luggage placed near the rear seats. Therefore, if the vehicle is involved in an accident involving another vehicle, such interior images can serve as strong evidence that the other vehicle involved in the accident was responsible for the damage to the luggage, allowing for smooth and accurate calculation of damages from the traffic accident. In particular, if the control unit 303 is configured to determine whether the luggage was placed securely and stably inside the vehicle, the reliability of the evidence is improved, providing further advantages.

Furthermore, in combination with the second case, for example, the control unit 303 may be configured to analyze the video data to determine the correlation between the collapse of luggage and tailgating or contact by another vehicle filmed from behind through the window inside the vehicle.
For example, if a driver turns the steering wheel to avoid a collision with another vehicle when another vehicle suddenly tailgates them from behind, but the luggage loaded in the vehicle collapses and causes damage, the video from the rear camera used to record footage of the other vehicle rapidly approaching from the side, the driver's driving in response to it, the driving trajectory of the other vehicle at the time, and the timing at which the luggage collapsed will be recorded in a single piece of video data, which is extremely useful for analyzing accidents and events that have occurred.

There have been many cases in the past where drivers have been concerned about the collapse of purchased items, resulting in accidents. For example, a person who went out to buy lunch for work became concerned about the lunch and the moment they looked at it, caused an accident. This is a situation that could happen to many drivers all too easily. Therefore, the above configuration can detect when luggage is in danger of collapsing and notify the driver in advance, providing the user with a safer driving environment and allowing them to concentrate on driving.

The control unit 303 may also be configured to detect the condition of the luggage and the condition of the person by analyzing the video data. For example, the control unit 303 may detect the correlation between the condition of the luggage and the condition of the person, and output a warning, notification, or the like based on the detection results. For example, the control unit 303 may detect that the luggage has collapsed in the direction of the person. For example, the control unit 303 may detect that the distance between the collapsed luggage and the person is less than a predetermined distance. Even better, the control unit 303 may detect both of these, for example, that the distance between the collapsed luggage and the person is less than a predetermined distance and that the luggage has collapsed in the direction of the person. The control unit 303 may change the content of the output of the warning, notification, or the like based on the relative positions of the luggage and the person. For example, the higher the degree of impact on the person, such as the degree of danger posed by the luggage, the higher the level of the output of the warning, notification, or the like. If the output destination of the warning, notification, or the like is a person who may be hit by the collapsed luggage, the control unit 303 can notify the person of the danger and encourage them to take appropriate action.

Fifth Example: Possible Case: Detect the status of the vehicle's display unit and issue notifications, commands, etc. based on that status.
Input example: Image of the vehicle's display Output example: Notification of the display status, commands according to the display status, etc.

In this case, the display unit is a display unit located in the vehicle, particularly a monitor mounted on the back of a seat or a monitor mounted on the ceiling, and the control unit 303 detects the status of these units by analyzing the video data. It is also desirable to configure the system to capture clear images of the display content and whether or not an operation is being performed on a monitor (hereinafter referred to as the "center monitor") installed near the center of the front of the vehicle between the driver's seat and the passenger seat. The display unit is not limited to a display unit fixedly attached to the vehicle, but may also be a user terminal 103 that displays information about the vehicle.

The center monitor screen tends to be in a blind spot from the position of the interior camera installed at the front. In contrast, the interior rear camera 102 installed at the rear of the vehicle is very suitable for capturing images of the center monitor screen from behind the vehicle.

The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior at S410, analyzes the captured image at S420, and determines the status of the vehicle's display units at S430, particularly the status of the monitors mounted on the backs of the seats and the monitors mounted on the ceiling, the display content of the center monitor and whether or not it has been operated, and whether or not there are passengers in the rear seats of the vehicle.

In S440, if the control unit 303 determines that, for example, there is no passenger in the rear seat of the vehicle and a monitor installed on the back of the seat is playing video, the control unit 303 outputs a warning sound, a report voice such as "The rear monitor is on," or a warning image from the speaker or monitor inside the vehicle.
In addition, in combination with the ninth example described below, the control unit 303 can also output a command to temporarily suspend or hide the display on the monitor when it determines that a child in the back seat is asleep, for example.

This embodiment provides even more advantageous effects when the center monitor is a touch panel display that accepts inputs related to vehicle operation and control. When the control unit 303 determines that part of a person's body is overlapping the center monitor, it can use this determination result as a trigger to start recording of in-vehicle video. For example, if video is recorded for approximately 10 seconds before and after the touch panel display operation (monitor operation), even if an accident occurs, if it can be proven that the timing of the driver's monitor operation was different from the timing of the accident, it can be proven that the monitor operation was not the cause of the accident.

Alternatively, the user can use the absence of remaining video as a means of proving that at least no action that could be determined to be monitor operation occurred while driving at the time of the accident. In this case, it is even better if the control unit 303 is configured to record a deletion history of recorded video, etc.

The control unit 303 may be configured to determine and record the specific details of the operation. In particular, in a configuration in which the center monitor also serves as an operation console for an autonomous vehicle, it is more preferable that the recorded video be configured to clearly capture and record the state in which the screen indicates whether the vehicle is being driven autonomously or manually at the time the video was recorded.
Therefore, if a driver is involved in an accident while driving properly, the photographing device 101R of the information processing system:
- The accident did not occur while operating the device,
- The accident was not caused by incorrect operation of the device,
- Whether the accident occurred during automated or manual driving,
It is advisable to provide objective evidence of the above.
Furthermore, a configuration in which the above-mentioned conditions are comprehensively judged from images captured by cameras inside the vehicle at the front and rear is even more advantageous, as it can improve the accuracy of the judgment and increase the amount of objective evidence.

Furthermore, if the control unit 303 determines that an accident has occurred, it is advisable to configure it so that video data from before and after the accident is sent to an external server. This eliminates any suspicion that the video data has been tampered with, further improving the evidentiary value of the video data. It is best if the external server is operated by a neutral organization that is unrelated to the accident, such as the company that manufactured and sold the imaging equipment.

Sixth Example: Possible Case: Detects a situation where the head is too often visible in the captured video, and issues a notification or warning based on that situation.
Input example: Images showing people, especially their heads. Output example: Notifications and warnings for people whose heads are too often visible in the footage.

For example, if the majority of the screen is the head, the control unit 303 issues a warning to that effect, or issues a warning to change the installation position.
The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether a head is captured on the screen of the captured image in S430, what proportion of the screen the captured head occupies, etc., or determines whether the installation state is such that such an event can be anticipated in advance, etc.

The proportion of the screen occupied by the head can be set to, for example, 50% or more of the entire screen. It can also be configured to issue a notification or warning if the head occupies 50% or more of the entire screen for a total of 5 minutes or more within a time interval set to, for example, 10 minutes. Furthermore, since the head tends to take up a larger proportion of the screen when getting in or out of a vehicle, it is recommended that this determination be suspended while the vehicle door is open.

In S440, if the control unit 303 determines that a head is in the image and that the proportion of the head in the image is equal to or greater than a set value, it outputs a notification to warn that the proportion of the head in the captured image is large, or that the rear camera 102 for capturing images inside the vehicle is installed in a state in which such an event is anticipated.
This allows the control unit 303 to provide the user with an opportunity to correct the state of the image captured by the vehicle interior rear camera 102 to a desired state at an appropriate timing.

In this case, depending on the car model and installation location, when someone sits in the back seat, it is possible to avoid the back of the person's head being constantly displayed on the screen (the entire screen being completely black).

Seventh example: Possible case: The parking status is detected, and a command to switch the shooting mode is issued based on that status.
Input example: Image of a person Output example: Command to switch to parking monitoring mode, etc.

The control unit 303 has a function of detecting whether a person is inside the vehicle from an image of the person, and automatically switching to parking monitoring mode when it detects that the person has left the vehicle.
The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether there is a person in the captured image in S430, whether the vehicle is parked, etc.
In S440, if the control unit 303 determines that the vehicle is in a parked state, it outputs a command to switch the imaging mode of the rear camera 102 for imaging inside the vehicle to a parking monitoring mode.

In the parking monitoring mode, the control unit 303 prioritizes processing to identify suspicious approaching individuals, for example, from within the area of the vehicle's windshield, rear window, or side window, and stops processing to filter out people such as those in the first example above.
This configuration can be combined with the eighth example of abandoned vehicle detection described below, and abandoned vehicle detection can be activated in the parking monitoring mode. Also, for example, it can be combined with the second example described above, and vehicle surroundings detection can be activated in the parking monitoring mode.

As a result, when the vehicle is parked, the control unit can prioritize allocating computing power to processing for identifying suspicious individuals approaching from within the area of the vehicle's windshield, rear window, or side window, for example, allowing the user to use a more efficient information processing system.

Example 8: Expected Case: Detects situations such as a person being left behind in a vehicle, and issues notifications or warnings based on that situation.
Input example: Image of a person Output example: Notifications and warnings about abandonment, etc.

The control unit 303 may be configured to determine whether a person, such as a child or a disabled person, has been left behind in the vehicle from the image of the person, and to issue a warning if such a person has been left behind. This processing may be performed in conjunction with the processing in the seventh example.
The control unit 303 acquires an image taken by the rear camera 102 for taking pictures inside the vehicle interior in S410, analyzes the image taken in S420, and determines whether there is a person in the image taken in S430, whether the person in the image is a child or a disabled person, whether there is a person other than a child or a disabled person in the vehicle, whether the vehicle is parked or abandoned, etc.

In S440, if the control unit 303 determines that the person is abandoned, it outputs an alarm or an alarm image from a device carried by the driver or a speaker or monitor of the vehicle, notifying the person that the person is abandoned. The control unit 303 may change the output destination or output method of the alarm based on the elapsed time since the person was abandoned and the state of the person who was abandoned. For example, the control unit 303 may be configured to initially output the alarm only to the device carried by the driver, and then, if the abandoned state is not resolved, output the alarm audio or image to both the device carried by the driver and the speaker or monitor of the vehicle.
This means that in situations where the driver is able to deal with the situation, the situation can be resolved with minimal warning, and in situations where the driver is unable to deal with the situation, the driver can notify those around them that the vehicle has been left behind.

In this configuration, the interior rear camera 102, which is a rear camera for photographing the interior of the vehicle, may further include a sensor other than the camera that can detect people, and may be configured to detect whether children, disabled people, etc. have been left behind based on the image from the interior rear camera and the sensor that can detect people.

Examples of sensors capable of detecting people include pyroelectric infrared sensors and thermosensors. A configuration using a sensor capable of detecting people can identify children, disabled people, and the like with a lower load than AI processing of video data. Since constantly processing video data requires a large computational load, it is advantageous from the perspective of reducing computational load and achieving highly accurate human detection to detect the presence of people using a separate sensor capable of detecting people and to use the result of the detection of people inside the vehicle by the sensor capable of detecting people as a trigger for starting human detection processing based on images from the rear camera 102 for capturing images inside the vehicle interior.
For example, it is advisable to limit the detection range of a sensor capable of detecting people to the rear area of the vehicle or the vicinity of the rear seats, and to configure the system so that the driver and passengers can be distinguished with high accuracy.

The above configuration makes it possible to detect people, distinguish between children, disabled persons, etc. and the driver, and determine whether children, disabled persons, etc. have been left behind with extremely high accuracy using video captured from the rear of the vehicle, thereby providing the vehicle with a highly accurate rear-end abandonment prevention function.

As already explained, the interior camera 102, which captures the interior of the vehicle from the rear, can also capture and identify babies in rear-facing child seats installed behind the front seats. In such situations, in this embodiment, the control unit 303 can provide a highly accurate rear-end baby abandonment prevention function. Furthermore, if the human detection sensor is a thermal sensor, it can accurately detect the baby's body temperature and issue a warning with the utmost caution if necessary.

As a variation of the eighth example, detecting a person such as a child from video captured from the rear of the vehicle may be used to make a determination other than whether the person has been abandoned. For example, if the control unit 303 of the imaging device 101R recognizes a child's head protruding from the child seat based on the image captured by the interior rear camera 102, it may output a warning or notification. The protruding position may be, for example, a position above the child seat. Suppose the interior rear camera 102 is installed so that the child's head does not appear in the captured image when the child is sitting in the child seat. In this case, if the child's head appears in the captured image, it is possible that the child is standing up, which could be a dangerous situation for the child.

The warning or notification may be output to a passenger inside the vehicle 10, or to an external terminal such as a terminal carried by the driver (e.g., user terminal 103). The warning may be an audible warning, or may be output using light or a display. The control unit 303 may determine whether to output a warning or notification, or change the content of the warning or notification, based on the direction or amount of movement of the head that has protruded from the child seat. According to this variation of the eighth example, an image captured from the rear side can be used to alert passengers to the dangerous situation of a child standing up.

Ninth Example: Expected Case: Detect whether a person in the vehicle is dozing or asleep, and based on that, issue a notification or warning.
Input example: Images of people Output example: Notifications, warnings, etc. about dozing or sleeping

The control unit 303 may be configured to determine the state of people inside the vehicle, particularly whether they are drowsy or asleep, from the video captured from behind, and issue a warning or notification.
The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether there is a person in the captured image in S430, whether the person in the image is the driver or a passenger, whether the driver's head is captured, whether the state of the head shows a warning pattern for drowsiness, such as nodding, whether the face or head of a passenger in the rear seat is captured, and whether the state of the face or head shows a sleeping pattern, etc.

In conventional video captured by an in-vehicle camera installed at the front of a vehicle, it has been difficult to determine the sleep state of passengers in the rear seats, especially, because the front seats are in the blind spot. In this embodiment, the control unit 303 is configured to effectively perform this determination by using video captured from behind. To determine whether a passenger is dozing off or asleep from behind, the control unit 303 stores sleep patterns indicated by a frontal video of the face, as well as sleep patterns indicated by head movement, body movement, and the positional relationship between the body and the vehicle, such as the head being in constant contact with the window glass.

In S440, if the control unit 303 analyzes the video data and detects that it matches a stored sleep pattern and determines that the driver is dozing, it uses this as an event trigger and outputs an alarm from the vehicle's speaker or the like to warn that the driver is dozing. With this configuration, the control unit 303 can also more accurately determine whether the driver is dozing when combined with a lane departure warning system (LDWS).

Furthermore, if the control unit 303 determines in S440 that a child in the back seat is asleep, it may be configured to output a notification voice such as "Your child is sleeping" from the vehicle's speaker or the like. This allows the control unit 303 to provide the driver with an opportunity to start driving carefully so as not to wake up a sleeping child. In another case, if the control unit 303 determines that a child in the back seat is asleep, it may be configured to output a command to change the driving mode of the autonomous vehicle to a gentler driving mode with gentler acceleration and deceleration.

10th example Expected case: Taking a photo when there is a person in the vehicle. Input example: Information to determine whether there is a person in the vehicle, such as an image in which a person can be identified. Output example: Image information of the captured image.

The control unit 303 has a function for capturing images when a person is in the vehicle. In particular, the control unit 303 preferably has a function for capturing images when a person is in the vehicle, regardless of whether the vehicle engine is on or off or whether the accessory power supply is on or off. Capturing images regardless of whether the vehicle engine is on or off or whether the accessory power supply is on or off can be referred to as capturing images in either case. Power control for this purpose can be, for example, as follows: The imaging device 101R is electrically connected to a battery unit (an example of a power supply device) (not shown) that has a charging function.

The photographing device 101R may operate by receiving power from the vehicle's battery while the vehicle's engine is on or the accessory power is on. The photographing device 101R may operate by receiving power from the battery unit while the vehicle's engine is off or the accessory power is off. The battery unit is electrically connected between the photographing device 101R and the vehicle's battery. The battery unit may charge based on the power received from the vehicle's battery while receiving power from the battery.

If the battery unit continues to supply power to the photographing device 101R after the power supply from the vehicle has stopped, it may supply the stored power to the photographing device 101R. The photographing device 101R may have a function equivalent to the battery unit described above as an internal power source. If the vehicle is equipped with a continuous power source (also called +B), the photographing device 101R may receive power from the continuous power source and operate regardless of whether the vehicle engine or accessory power source is on or off. The photographing device 101R may also have an internal power source such as a secondary battery 312, and may use this internal power source to operate regardless of whether the vehicle engine or accessory power source is on or off.

When the control unit 303 detects the presence of a person inside the vehicle cabin, it may determine that a person is in the vehicle. Information for detecting the presence of a person inside the vehicle cabin may be, for example, an image in which a person inside the vehicle cabin can be identified, such as an image captured by the vehicle interior rear camera 102. The control unit 303 acquires image information of the image captured by the vehicle interior rear camera 102 in S410, analyzes the captured image in S420, and determines whether a person is captured in S430. The control unit 303 may detect the presence of a person inside the vehicle cabin when, for example, it recognizes that a person's head, face, or part of a body is captured in the image.

In S440, if the control unit 303 determines that a person is in the vehicle, it may output image information indicating an image captured by the interior-photography rear camera 102 to a predetermined output destination. For example, the control unit 303 may output the image information to a storage device such as an SD card 306 as an output destination and record the image information in the storage device. This recording of image information may be considered event recording triggered by the determination that a person is in the vehicle.

When image information is output to a specified output destination, audio information of sound picked up by a microphone may also be output to that output destination. For example, when recording image information (i.e., video recording), audio information may also be recorded at the same time. This also applies to other configuration examples unless otherwise specified. The microphones may be positioned or numbered to capture sound from within the vehicle cabin. The image capture device 101R may have a microphone, or the microphone may be located outside the image capture device 101R. The control unit 303 may set an external device as the output destination. For example, the control unit 303 may use the communication processing unit 314 to transmit (i.e., transmit and output) the image information to an external device. The external device may be, for example, a server device for recording image information, a user terminal 103, or a computer device at a management center that manages image information captured by multiple vehicles.

If the control unit 303 has not determined that a person is in the vehicle, or if it has determined that no person is in the vehicle, in S440, it may be configured not to output image information showing an image captured by the interior-photography rear camera 102 to a specified output destination.

This makes it easier to understand the situation when a person is in the vehicle from the captured image. Since trouble or some other incident often occurs in the vehicle's interior when there is someone in the vehicle, making it easier to understand the situation when there is someone in the vehicle is useful for confirming whether or not such an incident has occurred and the circumstances under which it occurred. The person in the vehicle may be a legitimate user of the vehicle, such as the driver or a passenger, or may be someone who has entered the vehicle illegally (e.g., a thief). In particular, when there is someone in the vehicle, there is a possibility that trouble or some other incident may occur in the vehicle's interior regardless of whether the vehicle's drive source or vehicle power source is on or off. Therefore, making it easier to understand the situation when there is someone in the vehicle, regardless of whether the vehicle engine or vehicle accessory power source is on or off, is useful for more reliably confirming whether or not such an incident has occurred and the circumstances under which it occurred.

Furthermore, the following configuration may be adopted.
The control unit 303 may output image information only for the period during which it is determined that a person is in the vehicle to a predetermined output destination, but it is more preferable to output image information for a period before, after, or both of these periods (hereinafter referred to as the "period before and after"). The period before and after may be a fixed time length (e.g., 5 minutes before and 5 minutes after), or the image capturing device 101R may be configured so that the user can set it. In this way, image information before and after the period during which a person is in the vehicle is also output, making it possible to grasp, for example, what it was like when a person got in or out of the vehicle from the captured image.

Information other than an image that can identify a person may be used as information for determining whether a person is in the vehicle. For example, a detection signal from the human detection sensor described above may be used. The control unit 303 may determine whether a person is in the vehicle based on this detection signal.

The control unit 303 may stop capturing images using the interior rear camera 102 during a period when it has not determined that a person is in the vehicle, or during a period when it has determined that no person is in the vehicle. In this case, when the control unit 303 determines that a person is in the vehicle, it may start capturing images using the interior rear camera 102, acquire image information, and output the image information to a predetermined output destination. Furthermore, up to this point, we have described a function that captures images when it is determined that a person is in the vehicle, regardless of whether the vehicle engine is on or off or whether the accessory power is on or off. However, the function may also be such that it captures images only while the vehicle engine is on or the accessory power is on.

Example 11: Expected case: Taking a picture when a person speaks. Input example: Information to determine whether a person has spoken, such as a person's voice. Output example: Image information of the captured image.

The control unit 303 has a function to capture an image when a person speaks. "A person spoke" may be interpreted as a person speaking, or a human voice being detected, etc. The control unit 303 may capture an image when a person speaks regardless of whether the vehicle engine is on/off or the accessory power is on/off, or may capture an image only while the vehicle engine or accessory power is on. If this is to be done regardless of whether the vehicle engine or accessory power is on/off, it is preferable to perform power control similar to the tenth example.

The information used to determine whether a person has spoken may be, for example, audio information of a voice input to a microphone placed inside the vehicle cabin. The microphone may be located in a position suitable for picking up audio inside the vehicle cabin. The image capturing device 101R may have a microphone, or the microphone may be placed outside the image capturing device 101R. The control unit 303 may acquire audio information indicating the audio input to the microphone in S410, analyze the audio information in S420, and determine whether a person has spoken in S430. There is no particular restriction on the method for determining whether a person has spoken, but the control unit 303 may make the determination based on, for example, the signal waveform of the audio information, or based on the frequency band components of the audio information to which the human voice belongs.

The control unit 303 may determine that a person has spoken when it detects the voice of one person, such as talking to oneself, or may determine that a person has spoken when it detects the voices of two or more people, such as a conversation (dialogue). The condition for determining that a person has spoken may be the recognition of voice that can be recognized as language, or in particular the recognition of human voice in a dialogue format, for example the detection of an utterance and a response thereto. The condition for determining that a person has spoken may also be the detection of voice (for example, voice keywords) that suggests the occurrence of harassment such as power harassment, sexual harassment, or moral harassment, or the utterance of abusive language, or the occurrence of other trouble.

In S440, if the control unit 303 determines that a person has spoken inside the vehicle cabin, it may output image information showing an image captured by the vehicle cabin rear camera 102 to a predetermined output destination. The predetermined output destination may be the same as in the tenth example. For example, if the control unit 303 records image information to a storage device such as an SD card 306 as an output destination, the recording of this image information may be considered event recording triggered by the determination that a person has spoken inside the vehicle cabin. If the control unit 303 has not determined that a person has spoken inside the vehicle cabin, or if it has determined that a person has not spoken inside the vehicle cabin, it may not output the image information to the predetermined output destination in S440.

In this way, it becomes easier to understand from captured images what happens when someone in the vehicle speaks. For example, the interior of a vehicle can be considered a closed room, and there is a possibility that trouble such as harassment or some other incident may occur. In this case, there is a possibility that someone in the vehicle may speak about such an incident. Also, for example, if a group of friends go on a trip, they may want to record the period of time they were talking so that the captured images can be kept as a memory. In this way, making it easier to understand what happens inside the vehicle when someone speaks is useful for confirming whether or not such an incident has occurred and the circumstances under which it occurred.

Furthermore, the following configuration may be adopted.
The control unit 303 may output image information of a period before, after, or both of a period when a person spoke in the vehicle cabin. The period before and after may be a fixed time length, or the image capturing device 101R may be configured so that the user can set the period. In this way, image information before and after the period when a person spoke in the vehicle cabin is also output, making it possible to grasp the situation before and after the speaking period from the captured image.

Information other than human voice may be used as information for determining whether a person has spoken. For example, the control unit 303 may analyze an image captured by the interior-photography rear camera 102, and determine that a person has spoken if it recognizes the action of a person speaking.

The control unit 303 may output image information to a predetermined output destination regardless of who speaks, or may output image information to a predetermined output destination when a specific person speaks, and not output image information to a predetermined output destination when any other person speaks. In this case, the control unit 303 may use voice analysis technology to identify the person who spoke. The specific person may be a legitimate user, such as the driver or his/her passenger, or may be a person other than a legitimate user (for example, a thief or other person who has boarded the vehicle illegally). If an image of the speaker's face can be obtained, the control unit 303 may analyze the image to identify the speaker.

During a conversation with multiple passengers, if the control unit 303 uses the above-mentioned voice analysis technology to determine that the voice contains specific information assigned to various types of harassment, such as voice that is semantically recognized as harassment or voice that is identified as harassment based on its waveform, the control unit 303 may add information about the speaker and the assigned harassment to the image information and output it. It is also preferable that the specific information assigned to various types of harassment can be set, selected, or added by the user.

12th example Expected case: Taking a picture when a person inside the car reaches out their hand. Input example: Information to determine that a person inside the car has reached out their hand, such as an image in which the person can be identified. Output example: Image information of the captured image.

The control unit 303 has a function to capture an image when a person extends their hand inside the vehicle cabin. A person extending their hand may be interpreted as a person extending their arm. The control unit 303 may capture an image when a person extends their hand inside the vehicle cabin regardless of whether the vehicle engine is on/off or the accessory power is on/off, or may only capture an image when the vehicle engine or accessory power is on. If the image is captured regardless of whether the vehicle engine or accessory power is on/off, power control similar to that in the tenth example may be performed.

The information used to determine whether a person has reached out inside the vehicle cabin may be, for example, an image captured by the vehicle interior rear camera 102. In S410, the control unit 303 acquires image information from an image in which a person can be identified, for example, an image captured by the vehicle interior rear camera 102, analyzes the captured image in S420, and determines whether a person inside the vehicle has reached out in S430. The control unit 303 may determine that a person has reached out when the person sitting in the driver's seat (i.e., the driver) reaches out their hand toward the passenger seat, or may determine that a person has reached out when the person sitting in the passenger seat reaches out their hand toward the driver's seat, or may do both. Hands are relatively long parts of the human body. The control unit 303 may determine that a hand has been reached out based on, for example, the characteristics of that part of the body.

The control unit 303 may determine that a person has reached out when the person sitting in the driver's seat (i.e., the driver) reaches out their hand or something they are holding toward the passenger seat, for example, beyond the movable range of the shift lever or the range of the center console. Similarly, the control unit 303 may determine that a person has reached out when the person sitting in the passenger seat reaches out their hand or something they are holding toward the driver's seat, for example, beyond the movable range of the shift lever or the range of the center console.
As described above, the range of movement of the shift lever and the range of the center console can be used as criteria for determining whether a person has reached out, but it is also desirable that the user can use images captured by the rear camera 102 for photographing the interior of the vehicle to independently set criteria for determining whether a person has reached out, for example, to any position between the driver's seat and the passenger seat, and furthermore, that criteria can be changed or adjusted as needed.

In S440, if the control unit 303 determines that a person has reached out their hand inside the vehicle cabin, it outputs image information showing the image captured by the vehicle interior rear camera 102 to a predetermined output destination. The output destination may be the same as in the tenth example. For example, if the control unit 303 records the image information to a storage device such as an SD card 306 as the output destination, the recording of this image information may be considered event recording triggered by the determination that a person has reached out their hand inside the vehicle cabin. If the control unit 303 does not determine that a person has reached out their hand inside the vehicle cabin, or if it determines that a person inside the vehicle has not reached out their hand, it may be configured not to output the image information to the predetermined output destination in S440.

This makes it easier to understand from captured images what happens when a person inside the vehicle reaches out their hand. For example, the interior of a vehicle can be considered a closed space, and if a problem such as harassment or some other incident occurs, there is a possibility that violent or threatening acts may occur. If a person is seen extending their arm, it is suspected that an act such as hitting may have occurred. Making it easier to understand what happens inside the vehicle when a person extends their hand is useful for confirming whether such an incident has occurred and the circumstances under which it occurred.

Furthermore, the following configuration may be adopted.
The control unit 303 may output image information of a period before, after, or both of the period when a person in the vehicle reaches out their hand. The period before and after may be a fixed time length, or the image capturing device 101R may be configured so that the user can set the period. In this way, image information before and after the period when a person in the vehicle reaches out their hand is also output, making it possible to grasp the situation before and after the period when the person reaches out their hand.

The control unit 303 may output image information to a predetermined output destination regardless of who extends their hand, but may also output image information to a predetermined output destination when a specific person extends their hand, and not output image information to a predetermined output destination when any other person extends their hand. The control unit 303 may identify the person using a method as described in the 11th example.

Furthermore, a person may reach out even when simply picking up or placing an object. Therefore, the control unit 303 may use audio picked up by the microphone and other information to identify the situation in which a person is reaching out within the vehicle cabin, and may determine whether to output image information to a specified output destination depending on the situation. For example, the control unit 303 may output image information if it determines that a specified audio related to the occurrence of a problem has been detected, or if it has acquired other information for understanding the occurrence of a problem, but may not output image information in other cases.

A person reaching out in the vehicle cabin may be interpreted as someone reaching out from the back seat to the driver's seat or passenger seat. For example, in a taxi, a person pretending to be a passenger may commit violent or threatening acts against the driver. This makes it easier to understand the situation when a problem occurs from the captured image. A person reaching out in the vehicle cabin may be interpreted as someone sitting in the driver's seat or passenger seat reaching out towards the back seat, and various variations on the person's position are possible.

In the tenth, eleventh, and twelfth examples, the control unit 303 may be configured to further issue a warning or notification when performing the processing of S440. The warning or notification may be issued, for example, by illuminating a light-emitting element provided in the image capture device 101R or illuminating a light-emitting element provided in the image capture device 101F, but the warning or notification may be issued in any manner that is perceptible by humans. However, if the image capture device 101R is used for purposes such as crime prevention, the warning or notification may be directed to the outside of the vehicle, or a higher level warning or notification may be issued.

FIG. 5 shows an example in which two cameras are installed.
In the embodiment of Fig. 5(a), in addition to the photographing device 101R, two rear cameras 102a and 102b for photographing the interior of the vehicle are installed on the vehicle 10. In the embodiment of Fig. 5(b), in addition to the photographing device 101R, two rear cameras 102c and 102d for photographing the interior of the vehicle are installed on the vehicle 10.

In both the embodiments shown in Figures 5(a) and 5(b), the interior rear cameras 102a, 102b or the interior rear cameras 102c, 102d capture images in a direction from the rear of the vehicle to the front of the vehicle, but one or both of them can also be directed rearward. One of the interior rear cameras 102 may be replaced with imaging device 101R. Instead of two interior rear cameras 102, only one of the cameras may be used. It is recommended that the interior rear cameras 102c, 102d be attached to the pillars of the vehicle 10 (in this case, the area on the cabin side).

Furthermore, it is advantageous to mount the two interior rear cameras 102a, 102b or the interior rear cameras 102c, 102d at the rear of the vehicle, particularly in the range from behind the rear seats to the rear window.
One photographing device 101R and two additional rear cameras 102a, 102b or 102c, 102d for photographing the interior of the vehicle are connected via a wired or wireless connection, and the video data photographed by the two additional rear cameras 102a, 102b or 102c, 102d for photographing the interior of the vehicle may be subjected to image processing by the photographing device 101R.

In the embodiment of FIG. 5(b), the images from the vehicle interior rear cameras 102c and 102d may each be sent directly to the image capture device 101R, or, for example, the image from the vehicle interior rear camera 102c may be sent to the image capture device 101R via the vehicle interior rear camera 102d.

A camera mounted on the rear of a vehicle, such as a bus, can capture clear images of passengers sitting in the rearmost seats.
The rear camera may be equipped with a connector designed to extend the electrical wires from the glass. Such a connector should be positioned so that the cables are less visible inside the vehicle, and it is even more preferable to have a configuration in which the mounting position can be adjusted depending on the type and design of the vehicle.

The interior rear cameras 102c, 102d may be configured as, for example, spherical cameras (semi-spherical cameras), 180° cameras, 360° cameras, etc. In such a configuration, the interior rear cameras 102c, 102d may be configured to capture images of the vehicle interior captured from the rear toward the front of the vehicle, as well as images of the vehicle exterior from the rear of the vehicle, without changing their orientation. Images captured with such a configuration may be combined with images captured by the front camera for analysis.
If this video data is transmitted using the fifth generation mobile communication system (5G), data transmission can be carried out smoothly even if, for example, many celestial cameras are installed.

Figure 6 shows an embodiment in which the shooting direction of the rear camera 102 for shooting inside the vehicle interior of the embodiment in Figure 1 can be changed. Note that the rear camera 102e for shooting inside the vehicle interior in Figure 6(a) and the rear camera 102d in Figure 6(b) show the same camera facing in different shooting directions.

As shown in FIG. 6, for example, the interior rear camera 102e can be rotated 180° around its longitudinal axis relative to the imaging device 101R to change its imaging direction to that of the interior rear camera 102d. By changing the imaging direction of the interior rear camera 102d in this way and configuring it to capture images of the rear of the vehicle 10, images of the rear of the vehicle can be captured using two cameras, the imaging device 101R and the interior rear camera 102d, and the accurate distance to an object behind, such as a vehicle behind, can be measured.

In this embodiment, accurate distance measurement is possible even if the two cameras are separated by just 1 mm. Furthermore, extremely high-precision distance measurement is possible when the distance is approximately 10 cm. In this configuration, the image from the interior rear camera 102d can be used to accurately detect the distance to the vehicle behind, so it is desirable to have a configuration that can more accurately determine tailgating, dangerous lane changes by the vehicle behind, etc. based on the accurately detected distance.

In this configuration, by adding one rear camera 102 for capturing images inside the vehicle to the imaging device 101R, it becomes possible to capture images of the rear of the vehicle with two cameras, providing significant expandability to the in-vehicle camera system.

Note that accurate distance measurements such as those described above can also be performed on vehicles to the side by using two cameras that capture images of the side. For example, this can be used in the second example above to more accurately determine the risk of side-tailgating.

The rear camera 102 for photographing the interior of the vehicle may be equipped with a floodlight such as an LED light (e.g., an infrared LED light) whose light projection direction is linked to the adjustment of the camera direction, and when the rear camera 102 for photographing the interior of the vehicle is pointed outside the vehicle, the floodlight's light may be electrically blocked (e.g., by turning off a switch) or mechanically shielded (e.g., by covering the light-emitting part with a cover) so that the light from the floodlight cannot be seen outside the vehicle.

Furthermore, with this configuration, when the interior rear camera 102 is facing inward, the light from the photography light can capture clear images of the interior of the vehicle even in dark environments such as at night, or the light from the indicator light can be used to check the operating status of the interior rear camera 102. On the other hand, when the interior rear camera 102 is facing outside the vehicle, the emission of light from the floodlights such as the photography light and indicators can be stopped, preventing unnecessary light from leaking outside the vehicle.

FIG. 20 shows an example of a camera that blocks the light from a projector.
10A shows an overview of the rear camera 102 for photographing the interior of a vehicle, and includes a lens 2004 on a housing 2001, and two LED lamps 2003 attached to the left and right of the lens 2004. Note that the location of the LED lamps is not limited to this location, and they may be located in another position on the housing 2001.

(b) is an example diagram showing the state in which a cover 2002 that blocks the light from the floodlight in this embodiment is attached. The rear camera 102 for photographing the interior of the vehicle, which is attached to the rear window or the like, can rotate the direction of its lens 2004 180 degrees toward the rear of the vehicle, as shown in Figures 10 and 16. In this way, when photographing the outside of the vehicle behind the vehicle, the cover 2002 is configured to cover the LED lamp 2003 so that the light from the LED lamp 2003 is blocked. By mechanically covering the LED lamp 2003 in this way, it is possible to prevent light from being emitted toward the outside of the vehicle when the lens 2004 is facing outside the vehicle.

(c-1) and (c-2) are diagrams showing examples of a configuration in which the lens 2004 is rotated 180 degrees relative to (b) to capture images of the interior of the vehicle. (c-1) is a diagram of this configuration as seen from the rear of the vehicle, with the lens 2004 located at the back of the page and capturing images of the interior of the vehicle. (c-2) is a diagram of this configuration as seen from inside the vehicle, with the lens 2004 capturing images of the interior of the vehicle, and the LED lamps 2003 on the left and right of the lens 2004 not being blocked, so that they project light into the interior of the vehicle.

Even if the rear camera 102 for capturing images inside the vehicle interior is not configured to allow the camera direction to be adjusted, the above-mentioned floodlight blocking configuration can be applied. For example, the camera's floodlight can be attached so that it faces the interior of the vehicle, but not the exterior. Alternatively, even if the camera's floodlight is attached so that it faces the exterior of the vehicle, it can be prevented from emitting light toward the rear of the vehicle 10. For example, a configuration can be adopted in which the floodlight's light is electrically blocked or mechanically blocked.

It is preferable that the interior rear cameras 102e and 102d are configured to be able to rotate by 270° or more. With such a configuration, the shooting direction can be flexibly adjusted depending on the shape of the vehicle and the installation positions of the imaging device 101R and the interior rear camera 102.

FIG. 7 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle shown in FIG. 1 are configured as a single wide-angle camera.
The wide-angle camera 700 in Figure 7(a) has a wide-angle lens. A wide-angle lens has a wider angle of view than a standard lens. A wide-angle lens has a shorter focal length than a standard lens. The wide-angle camera 700 has an imaging means for capturing images using the wide-angle lens. The imaging means of the wide-angle camera 700 captures images and generates image data representing the captured images.

The image may be either a moving image or a still image, but is typically a moving image. Images captured using a wide-angle lens contain distortion (also known as image distortion) caused by the wide-angle lens. In this image, the distortion increases as you move from the center of the image to the edges. In this embodiment, the wide-angle lens has a 240° angle of view on both sides of a vertically downward direction when wide-angle camera 700 is mounted in the orientation that it should be mounted on the rear window or rear roof of a vehicle. This allows wide-angle camera 700 to capture a wide area on both sides of a vertically downward direction.

The wide-angle camera 700 in Figure 7(a) is a hemispherical camera with a field of view X° of 240°. This allows the hemispherical camera to capture images of the interior of the vehicle taken from the rear of the vehicle toward the front, images immediately behind the vehicle, images in front of the vehicle, and images of the vehicle in the elevation angle direction in front of and behind the vehicle, all as a single piece of video data. By selecting a hemispherical camera with an appropriate field of view depending on the vehicle model and placing it in an appropriate position, it is possible to record the status of traffic lights in front of and behind the vehicle in the above video data.

The angle of view X° is greater than 180°, and may be, for example, 220°, 270°, 360°, or another angle of view. The wide-angle lens 114 may be a fisheye lens. The angle of view of the wide-angle lens does not need to be uniform in each direction; for example, the lens may have characteristics such that the angle of view in the longitudinal direction of the vehicle is greater than the angle of view in the lateral direction of the vehicle.

FIG. 7(b) shows an enlarged view of the wide-angle camera 700 of FIG. 7(a).
Wide-angle camera 700 has a projector 701 in the direction of the lens, and the orientation of projector 701 changes in conjunction with the orientation of the lens. The rear half of wide-angle camera 700 is covered with a filter 702. In the embodiment shown in FIG. 7B, filter 702 is attached to a housing cover that covers wide-angle camera 700.

The light from floodlight 701 is expected to be auxiliary light for taking bright photographs of the interior of the vehicle, or light from an indicator LED light, but when the lens is facing the rear of the vehicle, it is preferable that filter 702 be configured to block the light emitted from floodlight 701, thereby reliably preventing the light from being seen by people outside the vehicle through the rear window.
If the light from the projector 701 is infrared light and the wide-angle camera 700 is an IR camera, the interior of the vehicle can be clearly photographed even at night.

The wide-angle camera 700 can also be attached to the rear or front of the vehicle. The wide-angle camera 700 can also be a hemispherical camera and installed in a diagonal corner of the vehicle rather than in the center of the windshield or rear window, allowing for a full view of both the inside and outside of the vehicle.

19 is another example of wide-angle camera 100, showing the positional relationship between the flat surface of operation surface 120 and optical axis 114A of wide-angle lens 114 when wide-angle camera 100 is viewed from the side. In the state shown in FIG. 19, wide-angle camera 100 is in a normal posture.
The normal to the flat surface of operation surface 120 is oriented in horizontal direction 120H, and the optical axis 114A of wide-angle lens 114 is tilted at a predetermined angle θ with respect to the vertically downward direction toward the side opposite to the side where operation surface 120 exists.

In this embodiment, the optical axis 114A of the wide-angle lens 114 is tilted by a predetermined angle θ toward the rear, where the vehicle's rear window is, with respect to the vertically downward direction, which is the direction that the user can visually confirm. This orientation when the case 116 is attached to the vehicle's rear window is the normal orientation. Furthermore, the angle of view X1° is the angle of view on the side where the operation surface 120 is located with respect to the vertically downward direction when the case 116 is in the normal orientation, and the angle of view X2° is the angle of view on the side opposite the side where the operation surface 120 is located with respect to the vertically downward direction when the case 116 is in the normal orientation. The angle of view X°, the angle of view X1°, and the angle of view X2° satisfy the relationship X° = X1° + X2°. The angle of the wide-angle lens 114 is fixed with respect to the case 116.

The angle θ satisfies the following formula (1) for the angle of view X° of the wide-angle lens 114.
θ=((X-200)×0.5)°...(1)
By setting a predetermined angle θ so as to satisfy equation (1), wide-angle camera 100 tilts optical axis 114A of wide-angle lens 114 by ((X-200) x 0.5) degrees relative to the downward vertical direction toward the side opposite to the side where operation surface 120 is located, and can capture a wide area both vertically and horizontally, including the area of (X-190) degrees above horizontal direction 120H.

For example, when the angle of view X° of the wide-angle lens 114 is 240°, the predetermined angle θ is 20°, the angle of view X1° is 100°, and the angle of view X2° is 140°, and the wide-angle camera 100 can capture a wide area both vertically and horizontally, including an area 50° above the horizontal direction 120H. When capturing an image with the wide-angle lens 114, the edges of the captured image are distorted, so even if an angle of view 10° smaller is set to capture an image with less distortion, the wide-angle camera 100 can still capture an area up to 40° above the horizontal direction 120H. When the wide-angle camera 100 is in the upright position, the capturing area includes the area behind the vehicle and the interior of the vehicle. It is even better if the capturing area also includes areas to the sides of the vehicle.

The wide-angle lens 114 may be capable of adjusting its angle relative to the case 116 within a predetermined variable angle range on both sides of the vertically downward direction. For example, in the case of a wide-angle camera 100 mounted on the rear window of a vehicle, the two sides of the vertically downward direction may be the front-to-rear direction of the vehicle. The predetermined variable angle range may be, for example, a variable angle range set with the optical axis 114A of the wide-angle lens 114 as the center in relation to the angle of view X° of the wide-angle lens 114, so that the wide-angle camera 100 can capture a wide area both vertically and horizontally. For example, when the angle of view of the wide-angle lens 114 is 240° and the center of the optical axis 114A of the wide-angle lens 114 is tilted by 20°, the variable angle range may be -20° < variable angle ≦ 20°. In this case, the wide-angle lens 114 can be adjusted within a range of 0° to 40° on the side opposite the side where the operation surface 120 is located, relative to the case 116, in the vertically downward direction.

The variable angle range may be any range that includes both sides of the vertically downward direction, and may be, for example, an angle range that is asymmetric between the front and rear of the vehicle with the vertically downward direction in between. Also, the variable angle range may include only the range from the vertically downward direction to the front of the vehicle, and thus, both sides of the vertically downward direction may include a case where one of the both sides is in the vertically downward direction.
By implementing the rear side with one camera in this way, costs can be reduced, installation effort can be reduced, and the device can be made smaller than if the rear side had two cameras, one inside the room and one behind.

FIG. 8 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of an ambulance vehicle.
8(a) shows the state in which the rear hatch of the ambulance is closed, and FIG. 8(b) shows the state in which the rear hatch of the ambulance is open. In the state in FIG. 8(b), the rear camera 102 for photographing the interior of the vehicle can clearly photograph the movement of a stretcher or gurney into the ambulance vehicle and the surrounding environment while the ambulance is stopped.

A rear hatch is an example of a rear opening/closing part that opens and closes an opening at the rear of a vehicle. A rear hatch is also called a hatchback door, rear hatch gate, or back door. A rear hatch closes the opening when closed, and opens the opening when open. The opening allows access between the inside and outside of the vehicle and is used, for example, for passengers to get in and out, and for loading and unloading luggage. The rear hatch shown in Figure 8 is also called a flip-up rear hatch, and is the type that opens upward.

In FIG. 8, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are shown, but instead of these, a celestial camera or a wide-angle camera similar to that described in FIG. 7 may be used.
It would be good to mount, for example, a spherical camera or a similar wide-angle camera on the inside of a flip-up door such as a rear hatch, so that when the hatch is open, it can take an image of the state of a stretcher or litter placed on the road from above.

It is also advisable to configure the camera so that the scene of the person (patient) being loaded into the vehicle on a stretcher or gurney is also captured as a hemispherical image.
In this embodiment, for example, the rear camera 102 for taking photos inside the vehicle interior is not operated from the time the ambulance is dispatched until it arrives at the scene, and the control unit 303 may be configured to start recording using the rear camera 102 for taking photos inside the vehicle interior as a trigger when it detects that the hatch has been opened, or to change the shooting settings or the lens to be used.

When loading the patient, the patient's identity and accident/condition are confirmed on the road or on the stretcher, but this information is often not communicated accurately, and in many cases, the accident situation is often communicated incorrectly until the very end.
By transmitting the video captured by the in-vehicle rear camera 102 to a hospital or fire department via 5G or the like, it becomes possible to convey accurate information in the form of video and audio. Automatic transcription using voice recognition is also effective.

It would be even more advantageous if the interior rear camera 102 had sensors for detecting various biological reactions. In such a configuration, the control unit 303 could be configured to quickly transmit the patient's body temperature data, detected by a thermosensor, to the medical monitor 802, even before various medical monitor devices are attached to the patient. It would also be even more advantageous if the control unit 303 stored this biological reaction data in synchronization with the video data.

It is preferable to use the rear camera 102 for photographing inside the vehicle to photograph the entire process from the state in which the hatch is closed, to the state in which the hatch is opened and the stretcher is lowered, the patient is placed on the stretcher, the patient is carried into the vehicle, the hatch is closed, and the vehicle is started to transport the patient. This makes it possible to continuously photograph the entire situation, even if questions are asked while the patient is in the vehicle.
In addition, it can also capture images of people who were nearby at the time of the accident, which can be used to identify the cause of the accident and the culprit.

When the ambulance arrives at the hospital and the patient is removed from the vehicle, it will be possible to verify later whether the procedure was carried out properly, and it will also be possible to verify whether any information was passed on to the hospital appropriately, making it easier to identify the party responsible if any problems arise later.

In a configuration in which the celestial sphere camera is attached to a flip-up hatch, it is preferable that when the hatch is closed, it photographs a 180-degree celestial sphere with the interior of the vehicle as its central axis, and when the hatch is open, it photographs a 180-degree celestial sphere with the downward direction from the ceiling as its central axis.
The control unit 303 may have a function to determine whether the hatch is open or closed and record the time point. Preferably, the control unit 303 may be configured to determine whether the hatch is being opened or closed and record the time point.

Furthermore, it is advisable to configure the system so that the recording method and camera settings are switched depending on whether the hatch is open or closed. In other words, it is preferable to switch so that outdoor recording method and camera settings are applied when the hatch is open, and indoor recording method and camera settings are applied when the hatch is closed. This makes it possible to reliably record high-quality video data even when there are major changes in the shooting environment and shooting conditions.

In the case of a spherical camera, it is best to mount it on the hatch so that one camera is in the transparent window that can capture the outside from the hatch window. For example, the main body should be in the interior body of the car just below the window, and the camera should pop out above and be in the transparent window. It is best to mount it in the center left and right.

In this way, when the hatch is closed (Figure 8(a)), one camera will take pictures of the interior of the vehicle, while the other camera will take pictures of the exterior, such as the rear. In this case, the thickness of the hatch will be close to the boundary between the two cameras, making it easier for each camera to take clear images, with appropriate exposure, for example. In this case, it is best not to match the exposure of the two cameras.

Also, when the hatch is open, one of the cameras can capture the surroundings, including what is happening below the hatch.
The other camera will take pictures of the sky, so in the event of a fire or jumping, it can capture the situation at the scene and weather conditions in the air.

In the configuration of FIG. 8, it is also possible to use a hemispherical camera.
In the case of a hemispherical camera, it is recommended that it be attached to the hatch just below the hatch window, in the center in the left-right direction.
In this way, when the hatch is open, it is less susceptible to the effects of wind and rain, and it is possible to photograph the surroundings, including what is happening below the hatch.

Data from the rear camera 102 for capturing images inside the vehicle interior can be synchronized with other cameras (cameras mounted on the vehicle ceiling, wearable devices worn by personnel, stretchers, etc.) and recorded in a reproducible manner or transmitted via 5G or other means.

Note that while the interior rear camera 102 may also be used to photograph the exterior rear of the vehicle, it is referred to as a "rear camera for interior photography" because it is used at least to photograph the interior of the vehicle. Such an interior rear camera may be understood as being "dual-purpose for both interior and exterior rear of the vehicle." Furthermore, the interior rear camera 102 may be understood as the first camera, and camera 402 as the second camera. Note that the camera of imaging device 101F may be understood as the third camera.

The above explanation assumes that the vehicle is an ambulance, but the vehicle in this embodiment may also be, for example, a commercial vehicle with a flip-up door, such as a rear hatch, to carry luggage or passengers, or a caravan. Commercial vehicles include, for example, welfare vehicles used to transport users (e.g., elderly people or people with disabilities) of designated facilities such as day care centers, emergency vehicles such as police cars, and postal vehicles. Furthermore, rear hatches are sometimes installed on vehicles for general users, as exemplified by caravans.

The rear hatch may be installed in vehicles such as minivans, hatchbacks, and station wagons. In such cases, it is advisable to replace "patient" with "luggage" or "passengers other than patients, such as the driver." For example, if the vehicle is a welfare vehicle, "passengers other than patients, such as the driver," may be users of a designated facility. The attributes of the passengers vary depending on the use of the vehicle.
Even in these configurations, it is advisable to record the situation when luggage is loaded, changes in the state of the loaded luggage, the situation when passengers other than the patient, such as the driver, are on board, and the state of work being done with the rear hatch open.

<Another embodiment of the photographing device provided in the rear hatch>
For example, the following embodiment is based on a concept common to the embodiment described in Fig. 8. Next, referring to Fig. 9, an embodiment in which a vehicle 500, which is a welfare vehicle, is used to transport a day care service user U1 will be described. In Fig. 9, the vehicle 500 is a welfare vehicle equipped for wheelchairs. The camera of the image capture device 101R is a hemispherical camera or a wide-angle camera equivalent thereto, and the camera range is described as an example in which the camera range is the same as that shown in Figs. 8(a) and (b). The vehicle 500 may preferably have a mechanism for smoothly loading a wheelchair onto the vehicle 500 using a ramp or lift, but this is not shown in the figure.

Figure 9 (a) is a diagram showing the user U1 before getting into the vehicle 500. User U1 is sitting in a wheelchair 501. Supporter U2 is, for example, a staff member at a facility such as a nursing home or hospital. Supporter U2 is, for example, a caregiver, attendant, or nurse. Supporter U2 supports user U1 by pushing the wheelchair 501 or by helping user U1 get into the vehicle 500 together with the wheelchair 501. Supporter U2 moves, for example, from user U1's home to the rear of the vehicle 500 while pushing the wheelchair 501.

As explained in Figure 8, the photographing device 101R is attached to the passenger compartment side of the rear window 11 provided on the rear hatch of the vehicle 500. When the rear hatch is closed, the interior camera 102 of the photographing device 101R faces towards the interior of the vehicle 500, specifically capturing images of the front of the vehicle from the rear of the vehicle. By capturing images of the interior of the vehicle from the rear, the interior camera 102 also takes privacy into consideration by capturing images of people's actions while protecting their privacy without capturing images of their faces. As explained in Figure 8, this embodiment is also used to actively capture and record people's behavior.

9(b) shows the situation when supporter U2 is trying to get user U1 into vehicle 500 (i.e., getting in). When the rear hatch of vehicle 500 is opened, the shooting direction of interior camera 102 changes. When the rear hatch is open, interior camera 102 shoots the rear exterior of vehicle 500 facing downward. Therefore, interior camera 102 shoots from above, capturing images of user U1 and supporter U2, who are located just behind vehicle 500. In this way, when supporter U2 is letting user U1 into vehicle 500, the rear hatch is open, and the camera automatically captures an appropriate range for capturing and recording the situation at the time of boarding. Such captured images are useful, for example, for verifying whether user U1 struggled and fell, or whether supporter U2's response was appropriate.

Note that Figures 9(a) and 9(b) illustrate the case where the door hatch is opened when supporter U2 attempts to get user U1 into vehicle 500, but there are also cases where the rear hatch is already open at the stage shown in Figure 9(a). In this case, the rear camera 102 for photographing the interior of the vehicle may be able to photograph user U1 and supporter U2 at an earlier stage.

Figure 9(c) shows the situation after user U1, seated in wheelchair 501, has been placed into vehicle 500. Wheelchair 501 is placed on platform 502. Platform 502 is provided with a mechanism (not shown) for locking (i.e., securing) the wheelchair. By securing wheelchair 501 to platform 502 with this mechanism, it is prevented from moving while vehicle 500 is moving. After user U1 has been placed into vehicle 500, the rear hatch of vehicle 500 is closed and placed in a closed state.

As described above, when the rear hatch is closed, the interior camera 102 takes pictures facing the interior of the vehicle 500. This shooting range includes at least user U1. Therefore, the photographing device 101R can use the interior camera 102 to photograph and record the state of user U1 while traveling. For example, by photographing and recording the state inside the vehicle 500 while traveling to a facility (e.g., a nursing home or hospital), it is possible to record interactions such as conversations between user U1 and supporter U2. Therefore, even if some kind of trouble occurs while traveling in the vehicle 500, such as abusive language or harassment, the photographed images can be used later to investigate the trouble.

In the embodiment described in FIG. 9, the interior rear camera 102 of the photographing device 101R attached to the rear hatch can capture an appropriate range depending on the situation at the time. Here, the control when loading the user U1 into the vehicle 500 has been described, but the same applies when unloading the user U1 from the vehicle 500 after arrival at a facility, for example. In this case, the photographing device 101R can use the interior rear camera 102 to capture and record the situation when unloading the user U1.

Furthermore, the following configuration may be adopted.
9(b) and 9(c), that is, during the period when user U1 and wheelchair 501 are in vehicle 500, the engine of vehicle 500 may be on or off. Even in such a case, by configuring the power supply control described in the above-mentioned tenth example to be performed, for example, photographing device 101R can photograph and record using rear camera 102 for photographing inside the vehicle interior regardless of whether the engine of vehicle 500 is on or off or whether the accessory power supply is on or off.

(B) For example, as in the case where the image capturing device 101R performs continuous recording, it may be configured to capture and record images over the period described in Figures 9(a) to 9(c), or it may be configured to capture and record images from the time the driver attempts to enter the vehicle 500 onwards, as in the case described in Figures 9(b) to 9(c). In the latter case, the control unit 303 in the image capturing device 101R may record image information of the image captured by the interior camera 102 when the rear hatch changes from a closed state to an open state.

If the power of the photographing device 101R is off before the recording starts, or if the power is on but photographing and recording cannot be started immediately (for example, in power saving mode), the control unit 303 may first turn on the power of the photographing device 101R or transition it to a state in which photographing and recording can be started (for example, in normal operation mode). The control unit 303 may then start photographing and recording. If the power of the photographing device 101R has already been turned on and it is in a state in which recording can be started immediately, the control unit 303 may start the recording immediately.

A method for determining whether the rear hatch has changed from a closed state to an open state will now be described. The control unit 303 may determine that the rear hatch has changed from a closed state to an open state, for example, based on a rear hatch door open detection signal. The rear hatch door open detection signal is a signal output from the OBD II ("II" is the Roman numeral for "2") connector when the rear hatch is detected to be open while the accessory power of the vehicle 500 is turned on. Alternatively, the control unit 303 may determine that the rear hatch has changed from a closed state to an open state by, for example, configuring a specific circuit external to or internal to the control unit 303.

The control unit 303 may determine that the rear hatch has changed from a closed state to an open state, for example, based on the acceleration detected by the acceleration sensor 305. The control unit 303 may acquire acceleration information and determine that the rear hatch has changed from a closed state to an open state if the acceleration information indicates the acceleration when the rear hatch is open, or if the acceleration information indicates a change in acceleration over time when the rear hatch changes from a closed state to an open state. By using a method that does not rely on a door open detection signal in this way, the control unit 303 can photograph and record when the rear hatch changes from a closed state to an open state, even if the accessory power is turned off. For example, even if the accessory power is turned off before the rear hatch opens, it is possible to photograph and record the situation when user U1 gets in.

The following configuration may be employed to determine whether the rear hatch has changed from a closed state to an open state. A G sensor is provided in the rear camera 102 for photographing the interior of the vehicle, and a G sensor (e.g., acceleration sensor 305) is provided in the photographing device 101R, a rear-facing camera attached to the same rear hatch. The control unit 303 may then determine whether the rear hatch is open or closed based on the degree of agreement between the direction of gravitational acceleration of the G sensor (e.g., measured assuming the vehicle is stopped) when the rear hatch is closed and when it is open, and the time during which this agreement is maintained (i.e., the stable time). Alternatively, the control unit 303 may determine whether the rear hatch has opened or closed based on a change in a characteristic G sensor value when the rear hatch is opened or closed. The characteristic G sensor value may be a three-dimensional value, such as acceleration in three axes.

In the configurations described in (C) and (B), when the control unit 303 determines that the rear hatch has changed from a closed state to an open state, it may start by turning on the power of the photographic equipment 101R. However, it is preferable to have the photographic equipment 101R turned on in advance, as this allows recording to begin more quickly. To achieve this, the photographic equipment 101R may utilize a recording standby function. The recording standby function is a function that waits for the photographic equipment 101R to start recording captured images (hereinafter sometimes referred to as "recording during the designated time period") when the photographic equipment 101R is turned on during a designated time period (hereinafter referred to as the "designated time period") and a specified trigger is detected.

When using the recording standby function, if the accessory power changes from on to off, any recording that was in progress (for example, continuous recording) will stop, but the power to the photographing device 101R will not be turned off, at least within the specified time period. To keep the power on even when the accessory power is off, the photographing device 101R should receive power from a continuous power source (i.e., +B operation), but an internal power source such as a secondary battery 312 or an external battery unit may also be used instead, or both.

In this embodiment, the designated time period for the recording standby function can be specified (i.e., set) in advance to include the time period during which user U1 is picked up or dropped off. The control unit 303 then operates the recording standby function, and if it determines that the rear hatch has changed from a closed state to an open state within the designated time period, it can begin recording captured images. In this way, the change of the rear hatch from a closed state to an open state is used as a trigger to promptly begin recording captured images, making it possible to more reliably capture the situation during the pick-up or drop-off.

In place of or in addition to the configurations (D) and (C), the following configuration may be used to more reliably capture and record the situation during pick-up and drop-off. When the control unit 303 is recording captured images while the accessory power is on (for example, continuous recording), if the accessory power is turned off, the control unit 303 does not immediately end recording of images captured by the interior rear camera 102, but rather extends the recording of images by a specified length of time (hereinafter referred to as "specified time extended recording"). The starting point of the specified time may be, for example, the point when the accessory power is turned off. The inventors have considered that this method may be advantageous in reducing adverse effects on the vehicle 500 battery.

Next, an actual usage scenario will be described using an example. When vehicle 500 arrives at user U1's home, the accessory power of vehicle 500 is changed from on to off. In this case, the control unit 303 of imaging device 101R is triggered by the accessory power being turned off to start extended recording for a specified time. In this way, the situation of user U1 and supporter U2 about to get in or out of the vehicle can be captured and recorded using the rear camera 102 for capturing images inside the vehicle. The specified time may be specified in advance, for example, at the time of product shipment.

For example, the specified time may be set taking into consideration the time required from when the accessory power is turned off, to when user U1 gets into vehicle 500, and until the accessory power is turned back on. The specified time may be set taking into consideration the time required for standard boarding and disembarking (e.g., the time required to stop the vehicle with the engine turned off). For example, a stop time during a day care service pick-up and drop-off may be approximately 5 to 10 minutes. However, there may be various factors that could extend the time, such as creating a wheelchair ramp and operating the lock to secure the wheelchair after boarding and disembarking, or talking with the user UI's family. Therefore, a configuration may be adopted in which the specified time is specified to the imaging device 101R by a facility staff member, such as supporter U2.

The specified time extended recording may be performed as follows. If the control unit 303 determines that the rear hatch has changed from a closed state to an open state within the extended specified time, it may perform a specified time extended by a first specified time length. If the rear hatch remains closed and does not open, it may perform a specified time extended by a second specified time length that is shorter than the first specified time. The specified time extended recording performed for the second specified time may be performed by interrupting the specified time extended recording for the first specified time. This is because if the rear hatch does not open, it is assumed that no passengers are getting in or out of the vehicle using the opening at the rear of the vehicle, and it can be said that this is a situation in which it is not necessary to record images captured by the interior camera 102.

In the configuration of (E), (C), or (D), when recording within a specified time period or recording for an extended specified time, the control unit 303 may associate identification information for identifying whether the recording was within a specified time period or an extended specified time period with the image information of the captured image (e.g., by adding it to the image information) and record it. Software with the function of playing back recorded video, such as a viewer, may have a function to output information to notify the user whether the recording was within a specified time period or an extended specified time period based on the identification information. The information for notifying the user may be, for example, display information, and may be displayed overlaid on or on the same screen as the played back video. Alternatively, this software may have a function to enable searching for images from the period of recording within a specified time period or an extended specified time period.

In the configurations (F) and (D), the image capture device 101R may be configured to allow the user to set whether or not to perform extended recording for a specified time after the accessory power is turned off. The control unit 303 controls whether or not to perform extended recording for a specified time after the accessory power is turned off in accordance with this setting. Setting this in advance reduces the burden on the user, but it may also be possible for the user to set this each time the accessory power is turned off.

Under the configurations (G) and (D), when the rear hatch is changed from a closed state to an open state and the control unit 303 determines that a person is exiting the vehicle through the opening, the control unit 303 may use the rear camera 102 for recording inside the vehicle interior to perform extended recording for a specified time. In the case of picking up or dropping off a user, this is useful because the rear camera 102 for recording inside the vehicle interior can be used to record the image of the supporter U2 pushing the wheelchair 501 in which the user U1 is sitting. This is based on the idea that the subject that was being recorded inside the vehicle interior by the rear camera 102 for recording inside the vehicle interior can continue to be recorded (i.e., viewed) in the rear, outside the vehicle direction.

The control unit 303 may determine that a person is exiting the vehicle through the opening when the rear hatch changes from a closed state to an open state, or from an open state to a closed state, while the vehicle 500 is stopped with the accessory power on. Furthermore, as described below, the control unit 303 may determine that a person is exiting the vehicle through the opening based on a signal from a mechanism provided in the vehicle 500, which is a welfare vehicle.

The control unit 303 may differentiate the manner of the specified time extension recording when it determines that a person has exited the vehicle through the opening from when it determines that a person has entered the vehicle. For example, the control unit 303 may set different recording times for the two cases, and in particular, it may set a shorter recording time for when a person has entered the vehicle than when a person has exited the vehicle. This is based on the idea that, since there are people outside the vehicle when a person has exited the vehicle and the possibility of an accident or other incident occurring is relatively high, it is necessary to ensure that such recording is more reliable. Furthermore, the control unit 303 may record higher quality images when a person has entered the vehicle than when a person has exited the vehicle. Examples of quality include frame rate and resolution.

The door that is determined to have changed from a closed state to an open state when exiting the vehicle, or from an open state to a closed state when exiting the vehicle, may be a door other than the rear hatch, such as the driver's side door or the passenger side door. In this way, extended recording for a specified time can be performed based on the latter of the supporter U2, who is also a passenger, which may more reliably capture the state of user U1 getting in and out of the vehicle.

In the configurations (H) and (D), the control unit 303 may have a function to set a designated time. The setting of the designated time here is so-called automatic setting. The control unit 303 may, for example, identify the situation inside or around the vehicle when the user disembarks by recognizing images captured by the interior rear camera 102. For example, the control unit 303 may determine whether the rear hatch of the vehicle 500 is open or closed by performing image recognition on the images captured by the interior rear camera 102. The control unit 303 may also determine whether the user U1 and supporter U2 are getting in and out of the vehicle, or whether that action has been completed, by performing image recognition on the images captured by the interior rear camera 102. The control unit 303 may set the designated time for designated time extended recording based on these determination results.

(I) The control unit 303 may recognize images captured by the interior-photography rear camera 102 and record information regarding the presence or absence of an occupant in the interior of the vehicle 500 by associating it with (e.g., adding) the information to the recorded image information. The control unit 303 may also add information indicating the position of the occupant in the image to the recorded image information. This position information may be, for example, a graphic such as an arrow pointing to the occupant or a circle surrounding the occupant's position. Furthermore, even if the occupant is in a fixed position, for example, because the wheelchair 501 is fixed to the base 502, the position of the occupant in the image will differ depending on whether the rear hatch is open or closed. Therefore, if information indicating the actual position of the occupant (the position of the wheelchair 501) is recorded or displayed, it will be easier for the user to understand the occupant's situation from the image. The occupant's position may be detected using the human detection sensor described above.

(J) The condition for ending recording of a specified time period or extended specified time period in the imaging device 101R is preferably that the specified time period has elapsed or that the extended specified time period has elapsed, but other conditions may also be used. For example, a condition may be that the door has changed from an open state to a closed state within the specified time period or the extended specified time period. This type of operation may be used when the interior of the vehicle is not being photographed or recorded. Furthermore, the recording time for recording of a specified time period may be specified taking into account the time required for user U1 to get into the vehicle 500 and for the accessory power to be turned on again. For example, the recording time may take into account the time it takes for standard entry and exit (for example, the time spent stopping with the engine turned off).

(K) With regard to the recording of the designated time period described in (C), the control unit 303 may set the designated time period. If the pattern of round trips of the vehicle between different bases, such as between the user UI's home and a facility, is fixed, and the vehicle operation schedule is set, or if reservation data for operation reservations exists, the control unit 303 may set the designated time period to include the estimated time of arrival at the base based on the schedule or reservation data.

The recording during designated time periods described in (L) and (C) can also be applied to applications other than the transportation mentioned above. For example, it would be useful for applications where the vehicle stops at a specific location (e.g., a facility) at a set time or approximate time. For example, a delivery person who delivers packages may make deliveries along a set route. In this case, the rear hatch is opened when the delivery person retrieves the package, and even if the delivery person leaves the rear hatch open while delivering the package, the situation behind the vehicle can be photographed and recorded, which is desirable for package security reasons. Note that the vehicle in this case could also be a cart or a bicycle with a carrier that the delivery person pushes around. In such a configuration, the rear hatch could be interpreted as a cover or lid attached to the cart or carrier.

(M) Figures 10 and 11 are diagrams illustrating images captured by the rear camera 102 for photographing the interior of the vehicle. Figures 10(a) and 11 show examples of images captured when the rear camera 102 for photographing the interior of the vehicle is attached to the rear hatch of a light wagon as described in Figures 8 and 9. Figure 10(a) is a diagram illustrating an image captured when the rear hatch is closed. Figure 10(b) is a diagram illustrating the photographing range when the rear hatch is closed.

As shown in Figure 10(a), when the rear hatch is closed, the interior of the vehicle can be photographed from the rear facing the interior, allowing for the person's actions to be captured while protecting their privacy without capturing their face. The person's head is visible above the driver's seat, allowing the captured image to grasp their condition. As shown in Figure 10(b), when the rear hatch is open, the optical axis of the interior camera 102 extends in a nearly vertical direction, ensuring a roughly conical capture range. However, because the floor of the vehicle is higher than the ground inside the vehicle, the capture range is limited to floor height. As can be seen from Figure 10, even when the rear hatch is open, the capture device 101R is less susceptible to wind and rain, and the interior camera 102 can be used to capture the exterior rear of the vehicle, including the view below the rear hatch.

Figure 11 shows an image captured by the rear camera 102 for capturing images inside the vehicle interior when the rear hatch is open. Figure 11(a) shows the rear hatch when it is open (horizontal dimension), Figure 11(b) shows the rear hatch when it is open (vertical dimension outside the vehicle), and Figure 11(c) shows the rear hatch when it is open (vertical dimension inside the vehicle). As can be seen, when it is open, the area below the rear hatch and the area towards the rear of the vehicle interior are included in the capture range. In this way, the rear camera 102 for capturing images inside the vehicle interior can be used to capture the area below the rear hatch outside the vehicle and the area towards the rear of the vehicle interior.

With interior cameras and hemispherical cameras that shoot from the front of the vehicle, the images are taken from in front of the driver, which can give passengers the feeling that they are being watched. Furthermore, because vehicles are equipped with a rearview mirror, the images are not visible or are difficult to see through the side windows. To overcome this, the interior of the vehicle is filmed from the rear. With the interior rear camera 102, passengers are filmed from behind, so they do not feel like they are being watched, and images from inside the vehicle can be recorded, while images from the side windows outside can be captured clearly with little interference from other objects.

Furthermore, as shown in Figures 10 and 11, the rear camera 102 for capturing images inside the vehicle interior may have a narrower viewing angle than a 360-degree camera or the like. Therefore, the images of the outside the driver's seat and passenger seat windows captured by the rear camera 102 for capturing images inside the vehicle interior are larger and have higher resolution than when a 360-degree camera or the like is used, which may make it easier for the user to see the contents of the captured image.

(N) Regarding the appeal regarding the rear camera 102 for photographing the interior of the vehicle mounted on the rear hatch, for example, the appeal regarding the rear camera 102 for photographing the interior of the vehicle mounted on the rear hatch is as follows.

Vehicles fitted with the imaging device 101R (in-vehicle rear camera 102) may include emergency vehicles such as police cars, parcel delivery vehicles (delivery vehicles), and postal vehicles, in addition to the ambulances and welfare vehicles mentioned above. In this way, when the rear hatch is opened in a vehicle where people or cargo are being unloaded at the rear of the vehicle, the unloading of the parcel or people will be photographed. Furthermore, even if the rear hatch is opened when a parcel delivery person retrieves a package, and the delivery person leaves the hatch open while delivering the package, the situation outside the rear of the vehicle can be photographed and recorded, which is desirable for package security. Furthermore, in the event of a theft while a postal worker is unloading a package, the footage can be recorded, which can contribute to the apprehension of the perpetrator.

In addition, general users can also take pictures in the event of a snatch theft while unloading luggage. For example, when carrying luggage in a supermarket parking lot, a user may leave the rear hatch open and go to a remote location. For example, a general user may go to a remote location to pick up luggage, go to dispose of cardboard boxes they brought from home, or return from the supermarket and carry luggage to their vehicle. In such cases, the rear hatch is often left open for several minutes. Even in such cases, the presence of the imaging device 101R can be expected to provide a crime prevention effect, so the benefits of so-called consumer use are also fully anticipated.

(O) Furthermore, the following configuration may be adopted.
(O-1) The camera 402 of the image capturing device 101R, the rear camera 102 for capturing images inside the vehicle interior, and the camera (not shown) of the image capturing device 101R may each record images at a resolution of 1080P, for example, as full HD image quality. In this way, the three cameras can record uniform images, for example, high-resolution images.

(O-2) In a viewer that has the function of time-synchronizing and playing back images from cameras with different angles of view, such as the camera 402 of the imaging device 101R, the rear camera 102 for capturing images inside the vehicle, and the imaging device 101R, it is desirable to provide a viewer that has a button that adjusts the angle of view when playing back images from the other cameras to that of the camera with the narrowest angle of view.

Assuming that camera 402 (for capturing images behind the vehicle) has the narrowest angle of view among these three cameras, the viewer can change the angle of view of the images captured by camera 101F and the rear camera 102 for capturing images inside the vehicle to the angle of view of camera 402 by pressing a button. This solves the problem of it being difficult to understand the correspondence between the visible ranges of each camera.

(O-3) It is desirable that the viewer be one that has the function of changing the angle of view of the video being played, and that it be provided with the function of changing the angle of view of the video from another camera with a single button to the angle of view of the video from the camera that is currently set to the narrowest angle of view.

(O-4) In the above explanation, the word "viewer" may be interpreted as a rearview mirror-type drive recorder body that is attached to a rearview mirror, and (O-1) to (O-3) are applicable to various devices.

(P) Changing the settings for capturing or recording video using the opening/closing operation of a vehicle door as a trigger Regarding changing the settings for capturing or recording video using the opening/closing operation of a vehicle door as a trigger as described above, the following configuration may be adopted: Regarding the settings for capturing or recording video, there are two points to consider: whether to record, i.e., which camera should record the images captured by the camera, and the quality of the recording, i.e., which camera should record from and at what quality, such as frame rate and resolution.

Situations include: (P-1) opening and closing the rear hatch; (P-2) the vehicle is moving; (P-3) when you want to record the front of the vehicle; (P-4) when the vehicle engine is on or the accessory power is on; (P-5) when the vehicle engine is off or the accessory power is off but you want to record within a specified time period; (P-6) when recording for a specified extended period after the vehicle engine is off or the accessory power is off; (P-7) when the vehicle engine is on or the accessory power is on but you want to record within a specified time period.

The control unit 303 may have a function for configuring settings for capturing or recording video for all or some of these situations. The control unit 303 may also have a function for configuring common settings for two or more of these situations. The settings for capturing or recording video may be configured manually by the user, or may be configured in advance or after product shipping using a method other than manual user configuration (e.g., automatic configuration).

When configuring settings using a method other than manual user configuration, it is preferable to configure video capture or recording settings as follows, for example. In the following description, if there are two cameras that record at relatively high quality, or two cameras that record at relatively low quality, the quality should be the same, but they may also be different. In the following, the control unit 303 is responsible for control of recording, such as determining quality, but this may also be performed by a control unit of the imaging device 101F (for example, the control unit 5511 described below) or another entity.

(P-1) Rear Hatch Opening and Closing Operation When the rear hatch is open, there may be situations where you want to check the area around the rear hatch later, so it is advisable to record the area around the rear hatch with high quality. To achieve this, it is advisable to record with a relatively high quality using the rear camera 102 for recording inside the vehicle. For example, when the rear hatch is open, it is advisable to use the camera 101F to record the front and the camera 101R to record the rear at a relatively low quality (e.g., coarse), while the rear camera 102 for recording inside the vehicle at a relatively high quality (e.g., detailed). By setting the cameras 402 of the camera devices 101F and 101R to record at a lower quality than the rear camera 102 for recording inside the vehicle, it is possible to check the situation around the vehicle over a wider area.

At least one of the image capture devices 101F and 101R may stop recording. Because the camera 402 of the image capture device 101R is a device that captures images of the exterior rear of the vehicle, it may continue to record its captured images at a lower quality than the rear camera 102 for capturing images inside the vehicle. Alternatively, because the image capture direction of the image capture device 101R is approximately directly above, recording of images captured by the camera 402 of the image capture device 101R may be stopped, while image capture device 101F continues recording.

(P-2) While the Vehicle is Running While the vehicle is running, there is a high need to later check the outside of the vehicle, in this embodiment, the front, rear, or both of the vehicle. Therefore, it is preferable to record the outside of the vehicle with high quality. To achieve this, it is preferable to record images captured by the cameras 402 of the image capturing devices 101F and 101R with relatively high quality. For example, while the vehicle is running, it is preferable to use the image capturing device 101F to capture the front, and the camera 402 of the image capturing device 101R to capture the rear with relatively high quality (e.g., detailed).

When the vehicle is moving, the rear hatch is typically closed. Therefore, the rear camera 102 for capturing images of the interior of the vehicle may capture images of the interior of the vehicle at a relatively low quality (e.g., coarse). The control unit 303 may determine whether the vehicle is moving based on the vehicle speed. For example, the control unit 303 may determine the vehicle speed based on changes in position information acquired using the GNSS sensor 307, and determine that the vehicle is moving if the vehicle speed is equal to or greater than a predetermined speed (e.g., 8 km). Furthermore, if information can be acquired from the vehicle, the control unit 303 may acquire the vehicle speed via the OBDII connector.

(P-3) When it is desired to capture images of the front side of the vehicle The front side of the vehicle can be the front area inside the vehicle's cabin, the front area outside the vehicle, or an area including both. For example, there may be cases where it is desired to later check the situation around the driver's seat and passenger seat of the vehicle, or/and the front area outside the vehicle. Therefore, it is desirable to record the front side of the vehicle with high quality. Therefore, when it is desired to record the front area inside the vehicle with relatively high quality, it is desirable that the interior camera 102 for capturing images record images with higher quality than the cameras 402 of the image capturing devices 101F and 101R.

When it is desired to record the area outside the vehicle forward with relatively high quality, the image capture device 101F may record images of higher quality than the rear camera 102 for capturing images inside the vehicle interior and the camera 402 of the image capture device 101R. When it is desired to record the area outside the vehicle forward with relatively high quality, the image capture device 101F and the rear camera 102 for capturing images inside the vehicle interior may record images of higher quality than the camera 402 of the image capture device 101R. The image capture device 101R may stop recording, but may record at a relatively lower quality.

(P-4) When the vehicle engine is on or the accessory power is on When the vehicle engine is on or the accessory power is on, if the vehicle is not running, it is likely that the vehicle is stopped, for example, parked. Therefore, it is desirable to record the exterior of the vehicle with high quality. To achieve this, it is desirable to record images captured by the cameras 402 of the image capturing devices 101F and 101R at a relatively high quality. The rear camera 102 for capturing images inside the vehicle may stop recording, but it is desirable to capture images of the interior of the vehicle at a relatively low quality.

(P-5) When recording within a specified time period even when the vehicle engine or accessory power is off When recording within a specified time period even when the vehicle engine or accessory power is off, as described above, there may be cases where the driver wants to check the situation around the rear hatch later for purposes such as transportation, so it is advisable to record high-quality images of the area around the rear hatch. To achieve this, it is advisable to record images captured by the rear camera 102 for capturing images within the vehicle interior at a relatively high quality. For example, it is advisable to have the rear camera 102 for capturing images with a relatively high quality, and the image capturing device 101F and camera 402 record images with a relatively low quality. Recording of images captured by at least one of the image capturing devices 101F and camera 402 may be stopped.

Also, since the vehicle engine or accessory power is off, in order to reduce power consumption, at least one camera may be set to record at a lower quality than in the case described in (P-4), thereby achieving lower power consumption.

(P-6) In the case of extended recording for a specified time after the vehicle engine or accessory power is off Even when performing extended recording for a specified time after the vehicle's drive source or accessory power is off, as mentioned above, there may be cases where you want to check the situation around the rear hatch later for purposes such as transportation, so it is advisable to record the area around the rear hatch with good quality. The recording control in this case may be the same as (P-5), for example, but may be different.

(P-7) When recording within a designated time period while the vehicle engine or accessory power is on Even when recording within a designated time period while the vehicle engine or accessory power is on, as described above, there may be cases where the driver wants to check the situation around the rear hatch later for purposes such as transportation, so it is desirable to record the area around the rear hatch with high quality. In this case, the recording control may be the same as, for example, (P-5), but may be different. Alternatively, if the vehicle's drive source or accessory power is on during the designated time period, power can be supplied from the vehicle, so it is possible to record with higher quality than in the case of (P-6) for at least some of the cameras.

In this way, it is preferable that the control unit 303 has the function of varying the video quality based on various combinations. Note that if two or more of the multiple situations described in (P-1) to (P-7) are met simultaneously, the control unit 303 may perform control according to one of the situations, with or without user specification. Alternatively, if two or more situations are met simultaneously, the control unit 303 may prioritize recording of higher quality.

(Q) It is preferable that microphones for picking up audio inside the vehicle cabin are installed in the front body of the image capturing device 101F, and in one or both of the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle cabin. The control unit 303 may then configure the audio to be recorded for each image capturing device or camera. The settings for the audio to be recorded may be configured manually by the user, or may be configured in advance or after product shipping by a method other than manual user configuration (e.g., automatic configuration). When configuring in advance or after product shipping by a method other than manual user configuration, the settings may be configured, for example, as follows.

(Q-1) When the control unit 303 determines that a person is in the vehicle, it is recommended to set it to record at a higher quality (for example, by recording in greater detail) than when it determines that no person is in the vehicle or that no person is in the vehicle. This allows for high-quality recording of human voices, making it easier to check the content of the voices later.

(Q-2) Based on the configuration of (Q-1), the control unit 303 may be configured to record with higher quality when it determines that a person is present within a predetermined range from the position of the camera or microphone, compared to when it does not determine that a person is present within that predetermined range, or when it determines that no person is present. In this way, even if the vehicle interior is large, high-quality recording can be made in a situation where human voices can be picked up.

(Q-3) A program such as a viewer with video playback functionality should simultaneously record audio input from all microphones, and when playing back recorded video, should have the ability to select which microphone to use to output the recorded audio. Selection can be manual by the user, or selection can be made without manual selection (e.g., automatic selection). In the latter case, the program should select and output the audio from the microphone that produced the highest quality recording.

(Q-4) When the rear hatch is open, recording should be centered on the microphone of the camera 402 of the image capture device 101R or the microphone of the rear camera 102 for capturing images inside the vehicle interior. When the rear hatch is closed, particularly when the vehicle is moving, recording should be centered on the microphone of the image capture device 101F. While central recording may be performed using only that microphone, recording may also be performed using other microphones, with the audio level of the central microphone being set relatively high. Note that although adjustment of the recording level has been described, the level may also be adjusted during playback.

FIG. 12 is an example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture.
The mounting fixture 901 has a first mounting portion 902 , a second mounting portion 903 , and a fixing means 904 .

The first attachment portion 902 has a C-shaped cross section, and is preferably configured to clamp the photographing device 101R based on the elastic force of the C-shaped portion.
The second mounting portion 903 is formed in a pipe shape, and its inner diameter is formed to be slightly larger than the outer diameter of the interior-of-vehicle rear camera 102. The direction of the interior-of-vehicle rear camera 102 inserted into the second mounting portion 903 can be easily adjusted in the circumferential direction and the up-down direction, and the user can fix the interior-of-vehicle rear camera 102 in a desired state using fixing means 904 such as a screw.

In this configuration, the user can very easily attach and fix the rear camera 102 for photographing inside the vehicle to the existing photographing device 101R.
Furthermore, since the mounting portion 902 is formed in a pipe shape, it is possible to change the direction of the mounting fixture 901 by rotating the mounting fixture 901 around the axis of the pipe, for example, for the photographic device 101R adhered to the rear window.
Furthermore, since the mounting portion 903 is formed in a pipe shape, the photographing direction can be changed by rotating the rear camera 102 for photographing the interior of the vehicle relative to the mounting fixture 901 around the axis of the pipe.

FIG. 13 is another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture.
The illustrated pipe-shaped mounting fixture 1001 has three joint portions (openings) 1002, and it is advantageous to set the outer diameter of at least one end of each rear camera, which has an arc portion, to match or be slightly larger than the inner diameter of the pipe joint so that a cylindrical camera can be inserted into each of these joint portions 1002. This allows the rear camera to fit into the hole. For example, two rear cameras can be attached to a pipe-shaped mounting fixture with two holes, and two or three rear cameras can be attached to a pipe-shaped mounting fixture with three holes.
As in the case of FIG. 12, the position and direction of the rear camera 102 for photographing the interior of the vehicle can be easily adjusted in the circumferential direction and the up-down direction.

In the embodiment of Figure 13, the so-called pipe joint method can be used to attach the interior-of-vehicle rear camera 102 to the imaging device 101R, allowing the user to easily and intuitively attach the additional interior-of-vehicle rear camera 102 to the information processing system. Furthermore, by changing the relative position of the additional interior-of-vehicle rear camera 102 with respect to the imaging device 101R or the imaging direction of the additional interior-of-vehicle rear camera 102, it is possible to easily adjust the system to capture the desired image.

A cable that provides electrical connection between the cameras may be housed inside the pipe-shaped fixture 1001. This makes it possible to prevent the cable from coming loose due to accidental contact, etc., compared to a configuration in which the cable is exposed.
The fixture 1001 can be designed with various shapes and positional relationships of the openings, and an extension pipe can also be used depending on the desired installation state. A preferred shape of the fixture 1001 is shown in FIG.

The pipe-shaped mounting fixture 1001 can be configured to be fixed directly with screws or adhesive, but it is even better to configure it with a part (e.g., a ball joint) that connects to the mounting member from somewhere other than the opening for the joint. For example, a ball joint can be installed where the screw is in the diagram.

FIG. 14 is yet another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture.
14 is a pan-tilt-shaped mounting fixture 1101, and it is particularly advantageous to have a configuration in which the photographing device 101R and the interior-of-vehicle rear camera 102 are connected to each other via the pan-tilt-shaped mounting fixture 1101. In this way, a camera configuration can be achieved in which only the photographing device 101R and the interior-of-vehicle rear camera 102 attached to the vehicle 10 appear to rotate around.

The mounting fixture 1101 has dowel screw portions 1102 and 1103, which can be threaded into dowel screw holes 1105 and 1106 provided on the housings of the imaging device 101R and the interior-mounted rear camera 102, respectively, to connect the imaging device 101R, the interior-mounted rear camera 102, and the mounting fixture 1101 to each other. After adjusting the relative positions of the imaging device 101R and the interior-mounted rear camera 102, the two can be fixed together by tightening the fixing portion 1104.

This embodiment makes it easy to adjust the shooting direction by rotating the interior rear camera 102, and to adjust the shooting direction in the elevation or depression direction. Even if the interior rear camera 102 itself has a function for adjusting the shooting direction (lens direction), the mounting fixture 1101 can be used to further expand the adjustable range of the shooting direction.
Furthermore, even when the rear camera 102 for photographing inside the vehicle interior is moved from an interior photographing position to an exterior photographing position such as the rear or left or right, the mounting state of the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior can be stably maintained.

FIG. 15 shows an example of a variation of the attachment.
It is preferable to use the fixtures 1201-1208 shown in Fig. 15 to provide a configuration in which multiple cylindrical cameras can be simply and easily attached and combined. In particular, by standardizing part of the outer shape of the cylindrical cameras and the shapes of the openings 1210 and 1220 of the fixtures 1201-1208, it is possible to provide a wide range of expandability for the camera system at low cost. Some of the pipe-shaped fixtures 1201-1208 have circular openings 1210 where the cylindrical cameras are attached, while others have openings on the sides of the pipe, such as opening 1220 (for example, a C-shaped cross section).

The inner diameter of the openings 1210, 1220 of the pipe-shaped fittings 1201 to 1208 may be the same as or slightly larger than the outer diameter of the cylindrical camera.
Furthermore, the circular openings 1210 of the fittings 1201 to 1208 may be shaped like cutouts in a portion of a pipe.
In the structure using the pipe-shaped mounting fixtures 1201 to 1208, the angle and position of the attached camera can be adjusted and can be fixed after adjustment.

By using pipe-shaped mounting fixtures 1201 to 1208 as so-called pipe joints, an easy and simple configuration and good expandability can be applied to mounting an in-vehicle camera.
Furthermore, it is preferable to have a cylindrical camera or drive recorder attached to (or integrated with) the pipe cap for the pipe joint.
By standardizing the pipe joint standards in this way, it becomes possible to easily install a camera in a factory, exhibition, etc. by simply removing the cylindrical drive recorder or rear camera 102 for taking pictures inside the vehicle and inserting it into a shelf, fence (enclosure), cart, or other pipe that is made up of pipes, and reusing the rear camera 102 for taking pictures inside the vehicle.
In addition, there is no need to attach a pipe cap to the location where the camera is installed.

As described above, a configuration in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle both have cylindrical housings is advantageous because it makes it easier and simpler to attach the rear camera 102 for photographing the interior of the vehicle to the photographing device 101R.
The mounting fixture has multiple joint parts (openings) depending on its type, and the outer diameter of at least one end of each rear camera, which has an arc portion, is set to match or be slightly larger than the inner diameter of the mounting fixture so that different rear cameras can be inserted into each of the multiple joint parts (openings).

FIG. 16 shows an example of another variation of the mounting fixture.
FIG. 16(a) illustrates an alternative fixture 1301 for connecting two cameras.
FIG. 16(b) illustrates yet another fixture 1311 for connecting two cameras.
A cylindrical camera can be attached to each of the openings of the other fixture 1301 and the further fixture 1311.

Another mounting fixture 1301 in FIG. 16( a ) is composed of a first fixing part 1302 , a second fixing part 1303 , and a connection joint 1310 that rotatably connects them.
FIG. 16(a-1) shows a state in which the openings of the first fixing portion 1302 and the second fixing portion 1303 face in the same direction.
FIG. 16( a - 2 ) shows another mounting fixture 1301 of FIG. 16( a - 1 ) rotated by 90° around a rotation axis 1307 .

The connection joint 1310 has a screw or the like attached in the direction of the rotation axis 1307, and by tightening the screw, the first fixed part 1302 and the second fixed part 1303 can be fixed so that they cannot rotate relative to each other.
The first fixing part 1302 and the second fixing part 1303 are each composed of two fixing members 1305a, 1305b, 1306a, and 1306b, and a pivotal connecting part 1304 that pivotally connects them to each other. The camera can be attached to the first fixing part 1302 and the second fixing part 1303 by loosening the screws of the pivotal connecting part 1304, opening the fixing members 1305a, 1305b, 1306a, and 1306b, attaching the camera, and then closing the fixing members 1305a, 1305b or 1306a and 1306b again.

Another mounting fixture 1311 in FIG. 16(b) is made up of a first fixing part 1312, a second fixing part 1313, and a pivotal connecting part 1314 that rotatably connects them.
FIG. 16(b-1) shows a state in which the openings of the first fixing portion 1312 and the second fixing portion 1313 face in the same direction.
Fig. 16(b-2) shows a state in which the first fixing portion 1312 of the mounting fixture 1311 in Fig. 16(b-1) has been rotated around the pivotal connection portion 1314. The user can fix the first fixing portion 1312 and the second fixing portion 1313 so that they cannot rotate relative to each other by screwing the pivotal connection portion 1314 at a desired angle.

When the pivoting connection part 1314 is opened, one of the cameras that was facing in the same direction will face the opposite direction by 180 degrees. Therefore, for example, at one time, one camera can be facing the rear of the vehicle and one camera can be facing the interior of the vehicle, and at other times, two cameras can be installed facing the rear of the vehicle.
The cameras can be attached and secured to the openings of the first and second fixing portions 1312 and 1313 in the manner described in connection with FIG.

FIG. 17 shows an example of a rear camera 102 for photographing the interior of a vehicle, which has a pin plug.
In the embodiment of FIG. 17, the rear camera 102 for photographing the interior of the vehicle has a pin plug 1401, which can be inserted into a plug jack of the photographing device 101R such as a drive recorder or other additional camera.

An O-ring 1403 is attached to the pin plug 1401, and when the pin plug 1401 is inserted into the pin jack, the frictional force of the O-ring 1403 ensures that the rear camera 102 for photographing the interior of the vehicle is securely fixed in place.
It is more preferable that the rear camera 102 for taking pictures inside the vehicle interior has a triangular protrusion on the inner circumference as a rotation prevention structure 1402 so that it can engage with the mating surface of the photographing device 101R, for example. If the rotation prevention structure 1402 is configured to engage with the photographing device 101R at a predetermined angle such as 0°, 30°, 45°, 90°, or 180°, the angle of the rear camera 102 for taking pictures inside the vehicle interior can be easily changed, which is convenient.

Furthermore, the rear camera 102 for photographing the interior of the vehicle may be configured to have a triangular protrusion on the outer circumference as a rotation prevention structure 1405, for example, on the joint surface with another additional camera.
These configurations are advantageous in preventing unwanted rotation due to vibrations.

When attaching another interior camera 102 of the same type to the interior camera 102, it is advantageous if the rotation prevention structure 1402 of the interior camera 102 and the rotation prevention structure 1405 of the other interior camera 102 engage with each other.

FIG. 18 is an example of an enlarged view of another anti-rotation structure of the camera of FIG.
FIG. 18 shows an enlarged view of the periphery of the rotation prevention structure when the rear camera 102 for photographing the interior of the vehicle shown in FIG. 17 is attached to the photographing device 101R.
In FIG. 18, the camera rotation prevention structure 1402 in FIG. 17 is illustrated as a rotation prevention structure 1501 on the side of the rear camera for photographing inside the vehicle, and further illustrated as a rotation prevention structure 1502 on the side of the photographing device 101R that engages with it.

The rotation prevention structures 1501, 1502 on the rear camera side for photographing inside the vehicle interior, which prevent rotation, consist of a sawtooth-shaped photographing equipment side uneven portion 1504 formed on the photographing equipment 101R side, and a camera side claw portion 1503 made of an elastic material formed on the rear camera side for photographing inside the vehicle interior and engaged with the photographing equipment side uneven portion 1504.The camera side claw portion 1503 engaged with the recess of the photographing equipment side uneven portion 1504 is held in the recess unless a rotational torque greater than a predetermined value is applied to the rear camera 102 for photographing inside the vehicle interior, and when a rotational torque greater than the predetermined value is applied, the camera side claw portion 1503 can move from the recess of the photographing equipment side uneven portion 1504 to the adjacent recess, overcoming the protrusion.
This structure is suitable for easily holding the rear camera 102 for photographing the interior of the vehicle at any rotational position.

In this way, it is possible to both securely fix the interior camera 102 and easily change the shooting direction by rotating the interior camera 102. In this case, the interior camera 102 is held within a predetermined range of 360°, allowing the same subject to be captured at different angles by the camera 101R that captures the rear of the vehicle interior.

FIG. 19 shows an embodiment in which the camera of FIG. 17 is attached to a cylindrical photographing device.
When the rear camera 102 for photographing the interior of the vehicle is rotated with a force equal to or greater than a certain level from the attached state shown in FIG. 19(a), it becomes in the state shown in FIG. 19(b).

In the embodiment of Figure 19, the interior rear camera 102 is connected to the imaging device 101R, which captures images of the area behind the vehicle, in the manner described in connection with Figures 17 and 18. In Figure 19(b), the interior rear camera 102 is fixed with its lens 1601 facing rearward, just like the imaging device 101R. Conversely, in Figure 19(a), the lens 1601 (not shown) is directed toward the interior of the vehicle, capturing images of the interior from the rear of the vehicle toward the front of the vehicle.

The rear camera 102 for photographing inside the vehicle interior has the anti-rotation structure 1602 described in relation to Figures 17 and 18. This anti-rotation structure 1602 is preferably used to achieve a configuration in which the rear camera 102 for photographing inside the vehicle interior and the photographing device 101R are rotatable yet securely fixed.

The photographing device 101R of FIG. 19 includes a first housing 1603, a second housing 1604, a lens 1605, a bracket 1606, a non-slip surface 1607, support legs 1608, a mounting surface 1609, a mounting plate 1610, and the like.
If the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle have cylindrical housings with the same outer diameter, the sense of unity in appearance is improved, which is preferable in terms of appearance.
Although the configuration has been described in which the rear camera 102 for photographing the interior of the vehicle is attached externally to the photographing device 101R, the camera shown in FIG. 19 can also be configured as a single device that is integrally formed in advance and whose lens direction can be changed.

A similar rear camera 102 for photographing inside the vehicle can also be attached to the front photographing device 101F. Also, the rear camera 102 for photographing inside the vehicle according to Figures 17 to 19 may be configured so that it can be attached to various types of cameras, such as digital cameras and cylindrical cameras, that have a pin jack into which a pin plug can be inserted.
As mentioned above, the in-vehicle camera can be connected to either the front or rear camera, and it is preferable that the connected camera is detected so that image adjustments can be set to match the in-vehicle image from the front or the in-vehicle image from the rear.
In particular, it is advantageous if the rear camera 102 for photographing inside the vehicle outputs data that has been subjected to IPS processing and compression processing, but it may also be one that outputs RAW data as is.

It is advantageous if the pin shape and pin arrangement of the pin jack for the in-vehicle camera are compatible with the pin shape and pin arrangement of the pin jack for the existing rear camera, so that the user can easily install the in-vehicle rear camera 102 via the pin jack of the existing photographing device 101R already installed in the vehicle.
In addition, it is preferable to select a narrower angle of view for the rear camera 102 for photographing the interior of the vehicle than the angle of view for the rear-facing camera of the photographing device 101R. In this configuration, the state inside the vehicle interior can be photographed more clearly.

In the above configuration, it is even more advantageous if the drive recorder is provided with plug jacks into which the compatible connectors can be inserted on different surfaces of the drive recorder, particularly different sides or bottom surfaces, and is capable of recording all of the images from cameras inserted into the compatible plug jacks on the different surfaces in a time-synchronized manner so that they can be played back.
With such a configuration, users can easily achieve the desired expansion of the shooting range, shooting purpose, and control functions depending on the total number of cameras to be added, the shooting direction, angle of view, shooting method, and other camera specifications.

In Figures 17 to 19, the rear camera 102 for taking interior photographs has a male round pin plug protruding from the main body, but this may be a normal female and a male-male round pin plug may be used to connect the rear camera 102 for taking interior photographs to other cameras, such as the main camera, photographing device 101R. In this configuration, it is preferable that the male-male round pin plug not only provides an electrical connection between the rear camera 102 for taking interior photographs and photographing device 101R, but also serves as a mechanical connection and fixation.

Using the above-described Z-shaped male-male round pin plug allows the interior camera 102 to be rotated and positioned relative to the imaging device 101R on a trajectory with a larger rotation radius. Because the interior camera 102 is lightweight, no additional mounting hardware is required to attach, position, and secure the interior camera 102 to the imaging device 101R, in addition to the plug.

Furthermore, the configuration using a male-male round pin plug means that the pin plug does not protrude from the interior camera 102, preventing damage during transport and storage due to electrical shocks such as static electricity at the terminals or physical shocks such as bending of the terminals. It also prevents the end of the connecting cable from becoming a female connector and becoming larger when the interior camera 102 is placed separately in a separate location without being directly connected to another camera. It also makes it easy to use the interior camera 102 as an exterior camera.

Another possible connection and fastening device is a fastener that allows the terminals of the image capture device 101R and the in-car camera to be connected in positions that are not coaxial (collinear), but that allows for rotationally adjustable and (semi-)fixed positions of the two cameras. This offsets the axes of the I-shaped ends of the cable, allowing the I-plugs to be positioned parallel to each other and reducing the distance between them. In other words, it is desirable to offset the axes of the male portions, which are normally designed to be aligned with each other. For example, the male round pin plugs could be configured in a Z-shape, handle shape, or crank shape. These connection and fastening devices can reduce the distance between the in-car camera and the main camera. A similar configuration can also be used for the I portion of the L-shaped terminal.

FIG. 20 is an explanatory diagram illustrating an example of a procedure for attaching the camera shown in FIG. 17 to the digital camera type photographing device 101R.
20(a) shows a state in which the imaging device 101R, which is a digital camera type drive recorder that captures images of the rear of a vehicle, is installed inside the vehicle and connected to a power source and a front camera by a cable 1703. It is preferable to configure the cable 1703 to be a single cable that serves the roles of supplying and receiving power and transmitting and receiving data, which is advantageous in that it allows for simplified wiring.

Figure 20(b) shows the state immediately before the connections between the photographing device 101R, cable 1703, power supply, and front camera in Figure 20(a) are disconnected and the rear camera 102 for photographing the interior of the vehicle according to Figure 17 is attached between the photographing device 101R and cable 1703.

The pin plug 1705 and pin jack 1706 of the rear camera 102 for photographing inside the vehicle correspond to the standards of the pin jack 1704 of the photographing device 1701 and the pin plug 1707 of the cable 1703, respectively.
In FIG. 20, the pin plug 1707 of the cable 1703 is shown with a larger outer diameter than the cable 1703, but this is simply for ease of viewing, and the outer diameter of the pin plug 1707 may be smaller than the outer diameter of the cable 1703.

Figure 20(c) shows the state in which an additional rear camera 102 for photographing the interior of the vehicle is installed between the photographing device 1701 and the cable 1703, as shown in Figure 17. The photographing device 101R and the additional rear camera 102 for photographing the interior of the vehicle are powered by a power source via the cable 1703, and are able to send and receive photographed data with the front camera. In other words, in addition to transmitting digital data, the cable 1703 also serves to supply power to each camera. Therefore, adding an additional rear camera 102 for photographing the interior of the vehicle does not require new wiring, and simple routing is possible using a single cable. It is particularly preferable to use a coaxial cable for this cable.

In Figure 20(c), the rear camera 102 for taking pictures inside the vehicle interior can be easily attached to an existing photographing device 101R. Existing photographing devices 101R generally have standardized pin jacks, so by providing a standardized pin plug that corresponds to the rear camera 102 for taking pictures inside the vehicle interior, various photographing devices 101R can be used in this information processing system.

In Figure 20(c), only one additional rear camera 102 for photographing the interior of the vehicle is attached between the photographing device 101R and the cable 1703, but it is also possible to attach multiple additional rear cameras 102 for photographing the interior of the vehicle in series. Furthermore, the user may attach multiple additional rear cameras 102 for photographing the interior of the vehicle using a branching cable; such a configuration will be described in Figure 21.

In the configuration of FIG. 20(c), the photographing direction of the photographing device 101R is set so as to photograph the outside rear of the vehicle, and the rear camera 102 for photographing the inside of the vehicle is set so as to photograph the inside of the vehicle.
The rear camera 102 for photographing inside the vehicle interior is preferably configured to receive video data from the front camera via the cable 1703, and send both the received video data from the front camera and video data photographed by the rear camera 102 for photographing inside the vehicle interior to the photographing device 101R. As described above, it is even more preferable that the cable 1703 can also be used to supply power to the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior.

The data sent from the front camera and the rear camera 102 for photographing the interior of the vehicle to the photographing equipment 101R may be data that has been processed or compressed by an ISP (Image Signal Processor) in each of the front camera and the rear camera 102 for photographing the interior of the vehicle, or it may be RAW data.
In a configuration in which the front camera and the rear camera 102 for photographing the interior of the vehicle transmit RAW data as is, it is preferable that the photographing device 101R further performs ISP processing and compression processing on the RAW data.

The interior rear camera 102 is preferably configured to receive video data from the image capture device 101R and send both the received video data from the image capture device 101R and the video data captured by the interior rear camera 102 itself to the front camera. In such a configuration, the data sent from the image capture device 101R and the interior rear camera 102 to the front camera is preferably data that has been processed and compressed by the ISP at the image capture device 101R and the interior rear camera 102, respectively, but may also be RAW data.

In a configuration in which the image capture device 101R and the rear camera 102 for capturing images inside the vehicle transmit RAW data as is, the front camera can also perform processing and compression processing by the ISP on the RAW data.

In a configuration in which the image capture device 101R, the interior rear camera 102, and the front camera each perform IPS processing and compression processing, ISP processing and the like are not concentrated on one camera, so the control unit 303 of the image capture device 101R, for example, can allocate more processing power to the image analysis and situation assessment described in relation to Figure 4. This allows image analysis and situation assessment to be performed faster or with higher accuracy.

In a configuration in which the rear camera 102 and front camera for capturing images inside the vehicle send RAW data to the image capture device 101R, which then performs all IPS processing and compression processing of the video data from the individual cameras that make up the in-vehicle camera system, it is possible to make the main unit, the image capture device 101R, high-performance and configure the other units with inexpensive units, allowing the entire system to be manufactured inexpensively.

In the above configuration, it is further preferable that the video recording medium be selectable from three options: front only, front and rear, or rear only.
In a configuration in which recording media are provided on both the front and rear, since the recording media are separated into both, even if one of the cameras is damaged in the event of a major accident, it is possible to prevent any data from being left behind.

When there are two recording media, a front and a rear, it is preferable that the frames are synchronized with each other by communication, electrical signals, etc. Note that the synchronization in this configuration may be off to some extent.
If there are two recording media, one front and one rear, it is advisable to have a function to synchronize the time of the images on each media.
If there are two recording media, one front and one rear, it is preferable to detect the event trigger on both.Furthermore, in this configuration, it is preferable that the event trigger is determined and designed using conditions such as OR and AND.

FIG. 21 shows an embodiment in which two cameras shown in FIG. 17 are attached to a digital camera type photographing device using a branching cable 1804.
In the embodiment of Fig. 21, it is possible to transmit images from two or more cameras to another device. In the embodiment of Fig. 21, the other device is, for example, a digital camera-type drive recorder, which is an image capturing device 101R. Two rear cameras 102 for capturing images inside the vehicle are connected to the image capturing device 101R via a bifurcated branch cable 1804.
The rear cameras 102 for photographing the interior of the vehicle are attached to desired positions of the vehicle, particularly to the rear area of the vehicle, by position fixing means (not shown).

21, two rear cameras 102 for photographing the interior of the vehicle are provided, namely, a first additional camera 1802a and a second additional camera 1802b. For example, it is advantageous to configure the first additional camera 1802a to photograph the interior of the vehicle from the rear toward the front, and the second additional camera 1802b to photograph the side of the vehicle from the rear of the vehicle.
The imaging device 1801 may have an additional pin jack, and in such a configuration, two additional cameras can be connected using two branching cables.
The branch cable is not limited to a two-branched cable, and may be branched into three or more branches.

In this embodiment, the additional rear camera 102 for photographing inside the vehicle interior may be configured to pass through to the outside, for example, the video data photographed by the rear camera 102 for photographing inside the vehicle interior itself and the video data from the photographing device 101R.
In addition, the first additional camera 1802a can pass through the power supply from the cable 1803, and is preferably configured to provide power to the second additional camera 1802b and the photographing device 101R via a branch cable 1804.

The rear camera 102 for recording inside the vehicle interior can send video data recorded by the rear camera 102 itself, video data recorded by the imaging device 101R, and power to the outside via a cable, and can also send video data received from the outside via a cable, video data recorded by the rear camera 102 itself, and power to the imaging device 101R.

The configuration in which the rear camera 102 for photographing the interior of the vehicle is an IR camera that lights up an IR LED when photographing the interior of the vehicle can clearly photograph the interior of the vehicle even at night, etc. The configuration in which the camera for photographing the interior of the vehicle is an IR camera can also be used effectively for photographing outside the vehicle/at night.
In the above configuration, it is more preferable to use a lens that automatically inserts/removes an IR filter.

The camera for capturing images inside the vehicle can be an RGB-IR camera, and can be configured to switch between RGB and IR capture modes as needed. This allows the camera for capturing images inside the vehicle to use IR capture mode in low-light conditions, such as at night, and switch to RGB capture mode in conditions with sufficient light.

The rear camera 102 for photographing inside the vehicle is an in-vehicle photographing camera equipped with a camera that can also be directed outside the vehicle and a floodlight having a light projection direction linked to adjustment of the camera direction, and is preferably configured so that when the camera is directed outside the vehicle, the light of the floodlight can be made invisible outside the vehicle by electrically blocking or mechanically blocking it.
For example, it is preferable that the projector is mechanically or electrically blocked when the camera turns backward in conjunction with the rotation of the camera.
This configuration, on the one hand, makes it possible to secure the amount of light necessary for photographing the interior of the vehicle, and, on the other hand, achieves a configuration in which the light from the floodlight cannot be seen from outside the vehicle when photographing the exterior of the vehicle.

Furthermore, it is desirable to have a configuration in which the angle of view of the captured image can be changed, for example by changing the lens used in conjunction with the rotation of the camera. For example, it is possible to change from standard to medium telephoto (e.g., a focal length of 50mm to 80mm in 35mm film camera terms) when shooting inside the vehicle and ultra-wide angle (e.g., a focal length of 10mm to 20mm in 35mm film camera terms) when shooting outside the vehicle, or to change from wide angle (e.g., a focal length of 24mm to 28mm in 35mm film camera terms) when shooting inside the vehicle and ultra-wide angle (e.g., a focal length of 10mm to 20mm in 35mm film camera terms) when shooting outside the vehicle.

The control unit 303 can execute these measures as a rear capture mode. The control unit 303 switches to the rear capture mode, for example, in conjunction with the rotation of the camera, which may electrically or mechanically block the light from a projector such as an infrared LED, or change the lens. The control unit 303 may also be configured to output, for example, information to a user terminal notifying the user that the mode has been switched to the rear capture mode or that the mode is the rear capture mode.

It is preferable that the above-mentioned in-car camera be configured with a filter in front of the lens when facing the rearward direction, which allows a clear image to be seen through the glass. It is more advantageous to use an ND filter, PL filter, or the included Posica film as the filter.

The above filter can be applied to the rear portion of a hemispherical camera attached to the rear of the vehicle, or even just the rear half. Even with this configuration, it is possible to ensure the amount of light necessary for capturing images inside the vehicle, and when capturing images outside the vehicle, it is possible to achieve a configuration in which the light from the floodlight cannot be seen from outside the vehicle.

Furthermore, the following may be done:
There is a demand for further strengthening of safety measures against accidents that occur when a vehicle is backing up. In this regard, the following may be used as an application of an information processing system that captures images of the area behind the vehicle 10 from the rear side of the vehicle 10 using two cameras and obtains distances. The following configuration can be applied to, for example, an information processing system that captures images of the area behind the vehicle using the above-mentioned two cameras, the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle (hereinafter, sometimes referred to as "cameras that capture images of the rear side"), and measures the distance to an object behind the vehicle, such as a vehicle behind.

1. The information processing system should be a system in which a camera installed near the rear of the vehicle that captures the rear view is connected directly or indirectly via wire or wireless to a device with a display installed in a position visible to the driver, and the image from the camera capturing the rear view is acquired and displayed so that it can be seen by the driver.

The installation location of the device having a display may be on the dashboard, the rearview mirror, the A-pillar, or the sun visor, and the device may be provided with mounting portions that can be attached to at least three of the same devices having the display.
It is also preferable that the at least three attachments that can be attached to the device and the device are shipped packed together in a packaging material.
It is advisable to package the camera, the device, and the at least three mounting fixtures together.

It is preferable to provide a function for inputting images from a plurality of different cameras and selecting the images from the cameras to be displayed on the display or combining them into one screen.
It is desirable to have a function that, even when an image from any camera or a composite image is being displayed, can transition by voice recognition or one touch to a state where the image from the camera capturing the rear view is displayed on one screen, and the images from the other cameras are not displayed on the same screen.
It is preferable to provide a function of storing the state before the transition and returning to the stored selection state of the camera before the transition after a certain period of time has elapsed.

The vehicle may be provided with a function of detecting that the vehicle has been switched to a reverse state, and when the vehicle has been switched to a reverse state, the display may be provided with a function of displaying an image having a range of photography of at least the area behind the vehicle from the rear side of the vehicle.
However, at this timing, in many cases the image behind the vehicle is displayed on the screen only after the start of backing up is detected, so there are cases where it cannot be used to check the situation when attempting to back up.

Therefore, it is preferable to provide a function that causes the display to display an image covering at least the area behind the vehicle from the rear side of the vehicle when the driver inputs an instruction to reverse the vehicle.
The driver's command to reverse the vehicle may be input by voice recognition or by detecting the pressing of a button.
The lighting of the hazard lamps may be detected, or the gear operation by the driver may be detected.
In particular, when either of these is detected and the vehicle is "stopped," it is desirable to have a function that causes the display to display an image with a range of coverage at least from the rear of the vehicle to the rear of the vehicle.

The detection of whether the hazard lights are on can be performed by detecting the driver's operation of the hazard light on button in the image captured by a camera facing the interior of the vehicle (from the rear), or by detecting the lighting status of the lamp indicating the lighting status of the hazard lights, for example, by detecting the lighting of the turn signal lamps on both the left and right sides.
The driver's gear operation may be detected by detecting the driver's gear operation in an image captured by a camera facing the interior of the vehicle (from the rear).
In many vehicles, the hazard lamp operation buttons, gear operation levers, etc. are located in the center of the left and right sides of the vehicle, so their status can be easily obtained using a camera that photographs the interior of the vehicle from the rear.

2. The information processing system should have the function of detecting when the vehicle is switched into reverse, and the function of recording video of the interior of the vehicle including the time before the trigger corresponding to the switch into reverse, and video of the area behind the vehicle taken from the rear including the time before the trigger corresponding to the switch into reverse. This video should be recorded in association with information that can be distinguished from other events when played back.

3. The function of detecting that the vehicle has switched to reverse, such as in 1.2. above, may use any one of the following methods: a method using a change in the direction of acceleration in the forward/backward direction of a G-sensor; a method using a change in the direction of travel of the vehicle obtained from a GPS or the like; or a method using a change in the image captured by a camera, and it is particularly preferable to use at least two of these methods for detection.
The camera image may be one that captures images in front, behind, or to the side, but it is particularly preferable to use a camera that includes images to the side in its capture area. In particular, it is preferable to use changes in the direction of image flow around the periphery of a hemispherical camera installed facing downward, and it is particularly preferable to use a configuration that detects changes in the direction of image flowing in the circumferential direction.

4. The information processing system may have a function to determine the distance to an object behind the vehicle from an image captured by a camera installed on the rear side of the vehicle, and to issue a warning based on the distance.
The information processing system may be provided with a switching function that switches between estimating the distance to an obstacle behind the vehicle by distance estimation using a monocular camera when video of the area behind the vehicle is acquired from one camera, and estimating the distance to an obstacle behind the vehicle by distance estimation using a stereo camera when video of the area behind the vehicle is acquired from two cameras.
This switching can be set in advance using a setting screen or a switch on the device, and the device can switch depending on this setting, but it is better to have a function that automatically switches depending on the state of the input video and the installation state of the device.
The information processing system may be provided with a function for selecting whether to switch automatically or manually.

5. The information processing system may have a function for discriminating and determining the next area from an image captured by a camera capturing the interior of the vehicle (for example, from the image area of one frame, or from the area of one still image).
(1) Areas of the vehicle's structures (seats, pillars, etc.), (2) areas outside the vehicle (for example, areas visible through the windows), and (3) areas of movable objects such as people and luggage inside the vehicle.The information processing system has the function of detecting people inside the vehicle, retains images when it determines that there are no people inside, and determines areas (1) and (2) from the difference between the retained images.
In particular, it is advisable to provide a function for detecting people inside the vehicle from sources other than camera images and to perform this processing based on this.

It is preferable that a thermal image sensor, a pyroelectric sensor, or the like is installed inside the vehicle, and the information processing system has the function of detecting whether or not there is a person inside the vehicle by utilizing the property that such sensors do not detect people through glass.
The information processing system may have the function of inferring the image captured by a camera capturing the interior of the vehicle using a pre-trained learning model to obtain segmentation results and determine the areas (1) to (3) above.
The information processing system should have an object detection function that is not segmentation, a function to distinguish whether a person or object is outside or inside the vehicle, and a function to decide whether to issue an alarm or change the manner of the alarm (e.g., the content or level of the alarm) depending on whether the person or object is inside or outside the vehicle.

6. The information processing system should have a function to automatically estimate the imaging range of the rear camera. In particular, the information processing system should have a function to determine the rear range within a single image area and display the image of that range on the display.

7. The information processing system should have a function to determine the distance to a vehicle behind from video footage taken from the rear of the vehicle, a function to issue an alarm using the distance to the vehicle behind when backing up (reversing), and a function to detect aggressive driving by a vehicle behind based on the distance to the vehicle behind when driving forward.

8. The information processing system may have a function for calculating the distance to an object behind the vehicle from an image of the rear side of the vehicle captured from the rear side of the vehicle, and a function for changing the content of the calculation process for the distance to the object behind the vehicle depending on whether the vehicle is moving forward or backing up (reversing).
The information processing system may be provided with a function to change the range of the detection range, for example, by setting the range of the detection range so that it is calculated accurately within a range of 0 to several meters when backing up, and by setting the range of the detection range so that it is calculated accurately within a range of several meters to several tens of meters when moving forward.

The above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. The above-mentioned configurations, functions, etc. may also be realized in software by a processor interpreting and executing programs that realize the respective functions. Information such as programs, tables, and files that realize the respective functions can be stored in memory, storage devices such as hard disks and SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

Other vehicles and other locations where the present invention may be applied include:
This is a shuttle bus for kindergartens and special needs schools. A teacher is on board to pick up the children, but in the event of an accident or trouble, the driver, teacher, and children's movements can be monitored. This is expected to help prevent accidents where kindergarten children are left behind on the bus.
Installed on vehicles that transport people, such as tourist buses, hotel shuttle buses, and trains.
Inside an airplane. It is possible that the state of the passenger cabin cannot be seen from the cockpit. In such cases, if the images from cameras installed in the cabin and cargo hold could be viewed in the cockpit or galley (for example, the CA area), it would be useful in preventing trouble or incidents before they occur. These applications are similar to those of surveillance cameras.

While many rear hatches are flip-up types that open upward, some vehicle models have rear hatches that swing open to the side or double doors. With such rear hatches, the interior rear camera 102 captures images of the rear of the vehicle 500 from the side when the hatch is open. The rear hatch may be opened and closed manually or electrically. Furthermore, in the above-described configurations, the description of the rear hatch as an example of a rear opening and closing unit that opens and closes an opening at the rear of the vehicle may be interpreted as an opening and closing unit that opens and closes an opening in the vehicle, as appropriate. For example, a similar configuration may be used for an opening and closing unit located in a location other than the rear of the vehicle.

As already explained, "The invention is characterized by processing video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, for example, by processing video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle and video data of the exterior of the vehicle captured from the rear of the vehicle," and as can be seen from the explanation of the embodiments so far, the imaging device installed on the rear side of the vehicle can be understood as imaging device that has at least the function of capturing images from the rear of the vehicle toward the front of the vehicle, or, for example, as imaging device that does not have the function of capturing images of the exterior of the vehicle behind the vehicle from the rear of the vehicle. For example, imaging device 101F may not have camera 402, and may be configured to process video data of the interior of the vehicle captured using rear camera 102 for capturing images of the interior of the vehicle.

[a. Configuration example of the photographing device 101F]
The photographing device 101F may have the following configuration, for example.
<a-1. External configuration of the photographing device 101F>
FIG. 24 is a diagram showing an example of the external configuration of the photographing device 101F. FIG. 24( a) is a view of the photographing device 101F viewed from the front, diagonally upward to the right. FIG. 24( b) is a view of the photographing device 101F viewed from the rear, diagonally upward to the right. The photographing device 101F has a housing 1010. The housing 1010 is a rectangular parallelepiped that is longer in the left-right direction than in the up-down direction and has a relatively small thickness. The housing 1010 has a top surface 1011 that faces upward when installed in a vehicle, a second side surface 1012, a first side surface 1013, a third side surface 1014 located opposite the first side surface 1013, and a fourth side surface 1015 located opposite the second side surface 1012.

The top surface 1011 is provided with a joint rail 552 and a camera jack 5591. The joint rail 552 allows for a detachable bracket (e.g., bracket 600, described below) to be attached to the joint rail 552 for mounting the image capture device 101F at a predetermined mounting position on the vehicle. The mounting position may be, for example, the vehicle windshield (e.g., near the top edge of the windshield), the vehicle's rearview mirror, or the ceiling of the vehicle interior. When the image capture device 101F is mounted on the vehicle, the second side surface 1012 faces the front of the vehicle. In this case, the first side surface 1013 faces the right side when viewed from the rear of the vehicle. The third side surface 1014 faces the left side when viewed from the rear of the vehicle. The fourth side surface 1015 faces the rear of the vehicle.

The camera jack 5591 is a terminal to which one end of a cable for electrical connection with the photographing device 101R is connected. The cable has, for example, a signal line for transmitting and receiving various signals (e.g., video signals) between the photographing device 101R and the photographing device 101F, and a power line for supplying power from the photographing device 101F to the photographing device 101R. For example, the photographing device 101F may obtain a video signal specifying an image captured by the photographing device 101R from the photographing device 101R via the cable, or may supply power to the photographing device 101R via the cable. The camera jack 5591 may, for example, be compatible with the USB Type C standard, and may also be a terminal for the photographing device 101F to communicate with the photographing device 101R according to the Ethernet standard.

The second side surface 1012 is provided with an imaging lens 5551, a sound output hole 553, and a microphone hole 554. The imaging lens 5551 is a focusing lens of the imaging unit (camera 5515, described below; an example of a third camera) of the photographing device 101F. The sound output hole 553 is provided above the imaging lens 5551 and is a hole that transmits sound output from the audio output unit (audio output unit 5514, described below) of the photographing device 101F from the inside to the outside of the housing 1010. The microphone hole 554 is provided below the imaging lens 5551 and is a hole that transmits external sound from the outside to the inside of the housing 1010. The sound that has transmitted into the inside of the housing 1010 is input to a microphone (microphone 5521, described below) of the photographing device 101F.

An event recording button 5522 is provided on the first side 1013. The event recording button 5522 is provided on the first side 1013 facing the driver's seat so that it is easy for the driver of a right-hand drive vehicle to operate. The event recording button 5522 is an operating means for instructing the start or end of recording of images captured by the camera 5515 (i.e., video recording). When the user operates the event recording button 5522 while event recording (also referred to as event recording) is not in progress, the image capturing device 101F starts event recording. Event recording will be described in more detail below.

The event recording button 5522 functions as a button for switching the image displayed on the display surface 5531 while an image stored in the storage medium 5500 (see FIG. 26) is being played back (in other words, the image is being displayed). In this case, when the event recording button 5522 is operated, the images displayed on the display surface 5531 are switched in the following order (loop): an image captured by the camera 5515 of the image capturing device 101F (image indicated as "front" in FIG. 27(a)), an image captured by the camera 402 of the image capturing device 101R (image indicated as "rear rear" in FIG. 27(a)), an image captured by the rear camera 102 for capturing images inside the vehicle of the image capturing device 101R (image indicated as "rear interior" in FIG. 27(a)), an image captured by the camera 5515 of the image capturing device 101F (image indicated as "front" in FIG. 27(a)), and so on.

The third side surface 1014 is provided with a terminal 5592 and a storage medium insertion port 5581. The terminal 5592 is a terminal for receiving power from an external device. The terminal 5592 is, for example, a DC jack. A connector on one end of a power cord (for example, a cigarette lighter plug cord) is connected to the terminal 5592. The connector on the other end of the power cord is connected to a power supply terminal (for example, a cigarette lighter socket) provided on the vehicle, for example.

Terminal 5592 may be connected to an OBDII adapter that can be connected to the vehicle's OBDII ("II" is the Roman numeral for "2") connector. The OBDII connector is an example of a diagnostic connector that is connected to the vehicle's ECU (Engine Control Unit) and outputs various vehicle information at predetermined intervals (e.g., every 0.5 seconds). By connecting terminal 5592 to the OBDII connector using an OBDII adapter (not shown), the image capture device 101F receives operating power from the vehicle and acquires vehicle information from the vehicle.

Vehicle information is information related to the state of the vehicle. Vehicle information may include at least one of the following: vehicle speed (vehicle speed), engine RPM, engine load factor, throttle degree, ignition timing, percentage of remaining fuel, intake manifold pressure, intake air flow rate (MAF), injection open time, engine coolant temperature (coolant temperature), temperature of air drawn into the engine (intake temperature), air temperature outside the vehicle (outside air temperature), amount of fuel remaining in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, accelerator position, turn signal information (operation of left and right turn signals (ON/OFF)), brake position, steering wheel rotation angle, gear position, and door open/close status.

Storage medium insertion slot 5581 is an insertion slot for inserting storage medium 5500, which serves as external storage means, into photographing device 101F. Storage medium 5500 is, for example, a storage medium on which images captured by photographing device 101F or photographing device 101R are recorded, such as an SD card. SD cards include any of various shapes, such as SD memory cards, miniSD cards, and microSD cards. Storage medium 5500 may also store a viewer (e.g., a dedicated viewer) program for playing back stored images on an information processing device such as a personal computer.

The fourth side surface 1015 is provided with an operation unit 5523, a display surface 5531, and a light-emitting unit 5525. The operation unit 5523 has a first button 5523A, a second button 5523B, a third button 5523C, and a fourth button 5523D. The first button 5523A, the second button 5523B, the third button 5523C, and the fourth button 5523D are arranged vertically along one side of the right side of the display surface 5531. The functions that can be assigned to each of these buttons include, for example, the following functions:

The first button 5523A functions as a button for switching images when pressed and held, and as a button for instructing formatting of the storage medium 5500 when pressed and held.

When pressed and held, the image displayed on display surface 5531 is, for example, the image currently being captured by photographing device 101F and part or all of the image currently being captured by photographing device 101R (for example, camera 402 and rear camera 102 for capturing images inside the vehicle interior). When pressed and held, as shown in FIG. 27(b), the images are switched in the following order: an image taken by the camera 5515 of the photographing device 101F (image shown as "front" in FIG. 27(b)), an image taken by the camera 402 of the photographing device 101R (image shown as "rear rear" in FIG. 27(b)), an image taken by the rear camera 102 of the photographing device 101R for photographing the interior of the vehicle (image shown as "rear interior" in FIG. 27(b)), images taken by these three cameras (image shown as "front + rear" in FIG. 27(b)), a screen with the display turned off and no images taken by these three cameras (screen shown as "screen OFF" in FIG. 27(b)), the camera 5515 of the photographing device 101F (image shown as "front" in FIG. 27), ... (hereinafter, a loop). In the image shown as "front + rear" in Figure 27(b), the display area of the image captured by camera 5515 is larger than the display areas of the images captured by camera 402 and the interior rear camera 102, and is identified by an L-shaped area. The display areas of the images captured by camera 402 and the interior rear camera 102 are each rectangular and may be, for example, of the same size.

Formatting the storage medium 5500 means initializing the storage medium 5500, and can be understood as at least one of the following: erasing data such as images stored on the storage medium 5500; writing setting information indicating the contents of the operational settings to the storage medium 5500 so that the photographing device 101F can use the storage medium 5500 (e.g., so that images can be recorded and read); and putting the storage medium 5500 into a specific file state.

The second button 5523B is a button for displaying a selection screen for selecting the image to be played back by the photographing device 101F. The third button 5523C is a button for displaying a menu related to the settings of the photographing device 101F and the photographing device 101R. The fourth button 5523D is a button for instructing the start and stop of image recording. For example, if the fourth button 5523D is pressed briefly during recording using the continuous recording function described below, the recording is paused. If the fourth button 5523D is pressed briefly during this pause, the recording of images using the continuous recording function is resumed. If the fourth button 5523D is pressed and held down, the frame rate at which images are recorded can be changed.

The display surface 5531 is an area where an image displayed by a display unit (display unit 5513, described later) of the imaging device 101F is displayed. The display surface 5531 is, for example, a rectangular or square area. A touch sensor 5524 is provided over the display surface 5531 to detect touch operations by the user.

The light-emitting unit 5525 is located above the first button 5523A and emits light in a specified color.

<a-2. Bracket configuration>
Figure 25 is a diagram showing the configuration of the bracket 600. Figure 25(a) is a diagram of the bracket 600 viewed from the front, diagonally upward right direction. Figure 25(b) is a diagram of the bracket 600 viewed from the side. Figure 25(c) is a diagram of the bracket 600 viewed from the front, diagonally downward right direction. The bracket 600 is an example of a mounting member for mounting the photographing device 101F to a vehicle.

Bracket 600 is an attachment component that uses a ball joint mechanism. In bracket 600, flat base portion 610 has an attachment surface 611 that is attached to the vehicle's windshield. Base portion 610 is inclined at a predetermined angle relative to the support of ball stud 620. An adhesive member such as double-sided tape is attached to attachment surface 611, and the bracket is attached to the vehicle's windshield or the like via this adhesive member.

The ball stud 620 is a portion of the base portion 610 that rises from the surface opposite the mounting surface 611. The ball portion 621 of the ball stud 620 is attached to the socket portion 630. The nut 640 is detachably attached to the periphery of the socket portion 630. A thread groove is formed on the outer periphery of the socket portion 630. The nut 640, which fits into the thread groove, is attached to the outer periphery of the socket portion 630. With the ball portion 621 of the ball stud 620 attached to the socket portion 630, before the nut 640 is tightened, the socket portion 630 can rotate in the desired direction along the circumferential surface of the ball portion 621, thereby changing the attitude and position of the base portion 650. When the nut 640 is tightened, the attitude and position of the base portion 650 are fixed.

The base portion 650 is paired with the socket portion 630 and is the portion for mounting to the photographic device 101F. The base portion 650 has a pair of guide rails 651 on its underside. The pair of guide rails 651 are configured to be slidable along the joint rails 5502 of the photographic device 101F. A claw-shaped tip portion 652 is provided at the tip of the base portion 650. The tip portion 652 is hooked onto the joint rails 5502 near the front end of the photographic device 101F, thereby attaching the bracket 600 to the photographic device 101F.

<a-3. Electrical configuration of the photographing device 101F>
26 is a block diagram showing the electrical configuration of the photographing device 101F. The control unit 5511 controls each unit of the photographing device 101F. The control unit 5511 is a computer including, for example, a processor 5511A and a memory 5511B. The processor 5511A includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application-Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array). The memory 5511B is a main storage device having, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor 5511A temporarily stores a program read from the ROM of the memory 5511B in the RAM. The RAM of the memory 5511B provides a working area for the processor 55511A. The processor 5511A performs various controls by temporarily storing data generated during program execution in the RAM while performing arithmetic processing. The control unit 5511 further includes a clock unit 5511C that keeps track of time. The clock unit 5511C is, for example, a real-time clock. The clock unit 5511C may be mounted on the motherboard of the processor 5511A or may be external to the processor 5511A.

The input unit 5512 accepts information input from the user. The input unit 5512 includes, for example, the microphone 5521, event recording button 5522, operation unit 5523, and touch sensor 5524 described above. The microphone 5521 converts sound incident through the microphone hole 554 or the like into an electrical signal. The microphone 5521 is, for example, a condenser microphone. The touch sensor 5524 detects the position touched by the user on the display surface 5531. The touch sensor 5524 is, for example, a capacitance type.

The display unit 5513 displays an image on the display surface 5531. The display unit 5513 is, for example, a liquid crystal display (LCD).

The audio output unit 5514 outputs audio. This audio may include, for example, notification sounds, background music, audio messages, etc. The audio output unit 5514 includes, for example, an audio processing circuit and a speaker.

The camera 5515 takes images and generates the resulting video data. The camera 5515 has, for example, an imaging lens 5551 and an imaging element that captures the light collected by the imaging lens 5551. The imaging element is, for example, a CMOS (Complementary MOS) or CCD (Charge Coupled Device). The camera 5515 generates a color (multi-color) image consisting of, for example, red (R), green (G), and blue (B) color components.

The communication unit 5516 communicates with external devices. The communication unit 5516 has a communication circuit for wirelessly communicating with external devices via, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless LAN (Local Area Network) communication or short-range wireless communication. The communication unit 5516 may also have a communication circuit for communicating in accordance with mobile communication system standards such as LTE (Long Term Evolution), 4G, and 5G.

The sensor unit 5517 has various sensors. The sensor unit 5517 has, for example, at least one of an acceleration sensor, a gyro sensor, a barometric pressure sensor, and an illuminance sensor. The acceleration sensor is, for example, a three-axis acceleration sensor that detects the vehicle's acceleration in the forward/backward, left/right, and up/down directions. The gyro sensor is a sensor that detects the tilt of the image capture device 101F. The acceleration sensor and gyro sensor may be used, for example, to estimate the vehicle's position using autonomous navigation when signals from GNSS satellites cannot be received. The barometric pressure sensor measures barometric pressure. The barometric pressure sensor is used, for example, to detect differences in elevation and determine whether the vehicle is on a highway or an ordinary road. The illuminance sensor is a sensor that detects illuminance, which indicates the brightness inside the vehicle cabin, which is the area around the image capture device 101F. The illuminance sensor is used, for example, to adjust the display brightness of the display unit 5513.

The reader/writer 5518 functions as a medium holding unit that holds the storage medium 5500 inserted into the photographing device 101F through the storage medium insertion port 5581. The reader/writer 5518 writes data to the storage medium 5500 and reads data from the storage medium 5500. The reader/writer 5518 may hold only one storage medium 5500, or may be configured to be able to hold two or more storage media 5500 simultaneously.

The terminal unit 5519 has a terminal for electrically connecting to an external device. The terminal unit 5519 has the camera jack 5591 and terminal 5592 described above. An external battery may be used as a device connected to the terminal unit 5519 so that the image capture devices 101F and 101R can operate even without power supply from the vehicle. The device connected to the terminal unit 5519 may be, for example, a device having a function to assist the user in safe driving. Examples of such devices include a device that captures an image of the driver (e.g., their face) and detects and notifies the driver of their state, such as distracted driving or drowsy driving, and a device that detects and notifies the driver of obstacles around the vehicle (e.g., a device used to detect vehicles for forward vehicle collision warning systems (FCWS)). Other devices connected to the terminal portion 5519 may also be in-vehicle devices such as radar detectors, laser detectors, car navigation devices, and display devices.

The location information acquisition unit 5520 acquires location information indicating the location (more specifically, the current location) of the photographic device 101F. The location of the photographic device 101F can be considered the same as the location of the vehicle in which the photographic device 101F is located, the location of the photographic device 101R located in the vehicle, and the location of the driver or other people in the vehicle. The location information acquisition unit 5520 acquires location information (latitude information and longitude information) of the photographic device 101F, for example, based on signals from GPS, which is one type of GNSS. The location information acquisition unit 5520 may also use Michibiki as a QZSS (Quasi-Zenith Satellite System).

The light-emitting unit 5525 emits light in a predetermined color. The light-emitting unit 5525 has, for example, a light-emitting diode.

The power supply control unit 5526 controls the supply of power to each unit of the photographing device 101F and to the photographing device 101R. The power supply control unit 5526 has, for example, a power switch and a power supply control circuit. The power supply control unit 5526 supplies power supplied from the vehicle side via the terminal unit 5519 to each unit of the photographing device 101F and to the photographing device 101R. The power supply control unit 5526 may further have a secondary battery, button battery, or electric double layer capacitor (also called a supercapacitor) as a power storage means.

The photographing device 101F may further include an auxiliary storage device, such as a flash memory (e.g., eMMC, SSD), as an internal storage means. Various types of storage media, such as optical storage media, magnetic storage media, and semiconductor storage media, can be used as the auxiliary storage device.

<a-4. Recording function of the photographing device 101F>
The image capturing device 101F may have one or more of the following functions as a recording function. The recording function is a function for recording (i.e., recording) images captured by the image capturing device 101F as video data in a predetermined file format. The video data may be video data in a moving image format, such as an MPEG (Moving Picture Experts Group) format (e.g., MPEG2, MPEG4), AVI, MOV, WMV, etc.

(a-4-1. Continuous recording function)
The continuous recording function (also referred to as the continuous recording function) is a function that, while the photographing device 101F is in operation, continuously (i.e., constantly) records images captured by the camera 5515 of the photographing device 101F, and by some or all of the camera 402 of the photographing device 101R and the rear camera 102 for capturing images inside the vehicle. When the continuous recording function is being executed, the control unit 5511 stores the video data captured from the start to the stop of the vehicle engine.

(a-4-2. Event recording function)
The event recording function (also referred to as the event recording function) is a function that records images captured by the camera 5515 of the image capturing device 101F, and some or all of the camera 402 of the image capturing device 101R and the interior rear camera 102 in response to the occurrence of a specific event. An event is an occurrence that serves as a condition (i.e., a trigger) for recording an image. For example, an event may be a user's sudden steering or sudden braking while the vehicle is in motion, or the vehicle's collision with another object. The control unit 5511 determines that an event has occurred based on, for example, a measurement value obtained by the acceleration sensor of the sensor unit 5517. Specifically, the control unit 5511 determines that an event has occurred when the measurement value obtained by the acceleration sensor exceeds a predetermined threshold value or when the measurement value indicates a predetermined change over time. The conditions for determining the occurrence of an event are not limited to these. The control unit 5511 may also determine that an event has occurred based on vehicle information, for example, when the vehicle state, such as vehicle speed or steering state, satisfies a predetermined condition. The control unit 5511 may determine the occurrence of an event by analyzing images captured by at least one of the camera 5515, the camera 402, and the interior rear camera 102. The control unit 5511 may determine that an event has occurred, for example, when detecting dangerous driving of the vehicle or another vehicle (e.g., tailgating, approaching, or abnormally approaching). The control unit 5511 also determines that an event has occurred when the event recording button 5522 is operated.

When the control unit 5511 determines that an event has occurred, it records images captured during a predetermined period before and after the event (hereinafter referred to as the "event recording period") in the storage medium 5500. The control unit 5511, for example, temporarily records images captured by the camera 5515, the camera 402, and the interior rear camera 102 in memory 5511B (e.g., RAM). When the control unit 5511 determines that an event has occurred, it may record images captured during the event recording period read from memory 5511B in the storage medium 5500. For example, the control unit 5511 may generate a single file containing images for a total of 40 seconds, consisting of 20 seconds before and 20 seconds after the event, for each of the camera 5515, the camera 402, and the interior rear camera 102. This event recording period is an example and may vary depending on the type of event. The image capturing devices 101F and 101R may also be configured to allow the user to change the event recording period. The control unit 5511 may set different event recording periods for some or all of the camera 5515, camera 402, and interior rear camera 102, or may set the same length for all of them. The control unit 5511 may further associate information specifying the type of event, values measured by the sensor unit 5517 during the event recording period (e.g., acceleration in each of the three axes), and location information acquired by the location information acquisition unit 5520 with the image and record them in the storage medium 5500.

The control unit 5511 may vary the conditions for performing event recording for some or all of the camera 5515, camera 402, and the interior rear camera 102. For example, when a value measured by the sensor unit 5517 satisfies a predetermined condition (e.g., exceeds a threshold), the control unit 5511 may vary the conditions for performing event recording for some or all of the camera 5515, camera 402, and the interior rear camera 102. The condition may be that any one of the three-axis accelerations, or the absolute value of the three-axis acceleration, exceeds a threshold. The conditions for performing event recording may be configured so that the user can set them for each of the camera 5515, camera 402, and the interior rear camera 102. Acceleration occurring on the photographing device 101F side (specifically, camera 5515) is measured using sensor unit 5517, and acceleration occurring on the photographing device 101R side (specifically, camera 402 and rear camera 102 for photographing the interior of the vehicle) is measured using acceleration sensor 305. This makes it possible to more accurately measure the acceleration occurring at the position of each camera, and to perform event recording using the measurement results as a trigger.

(a-4-3. Parking monitoring function)
The parking monitoring function is a function for recording images captured by the camera 5515 of the image capturing device 101F, and some or all of the camera 402 and the interior rear camera 102 of the image capturing device 101R while the vehicle is parked. The parking monitoring function is a function for monitoring the interior of the vehicle or the exterior surrounding the vehicle while parked. When not receiving power from the vehicle, the control unit 5511 may receive power from an external battery and record images in the storage medium 5500. The control unit 5511 may determine whether the vehicle is parked based on one or more of the following: the accessory power source is turned off; the engine is turned off; power supply from the external battery has started; the vehicle speed is 0 km/h or a predetermined speed or less; and the location information acquired by the location information acquisition unit 5520 is predetermined location information (e.g., location information of the home, workplace, or parking lot). However, the control unit 5511 may determine whether the vehicle is parked by analyzing an image captured by the rear camera 102 for capturing images inside the vehicle, as already explained.

The parking monitoring function may have a time lapse mode and a motion detection mode. In this case, when the user selects time lapse mode, the control unit 5511 may record at a frame rate lower than other image recording functions such as the continuous recording function and event recording function. For example, while the frame rate of other image recording functions may be 20 to 30 frames per second, the frame rate of time lapse mode may be 1 frame per second.

Moving object detection mode is a mode in which images are recorded in response to the detection of a moving object (i.e., the movement of an object). Examples of moving objects include people and other vehicles. Specifically, when the user selects moving object detection mode, the control unit 5511 may record images captured within a predetermined period before and after the detection when it detects a moving object based on changes in images captured by at least one of the cameras: camera 5515, camera 402, and interior rear camera 102. When the control unit 5511 detects a moving object based on images captured by at least one camera, recording images captured by all cameras (camera 5515, camera 402, and interior rear camera 102) allows for later review of a wide range of the situation, including the area in front of, behind, and inside the vehicle. When the control unit 5511 records images captured by only the camera that detected the moving object, it is possible to conserve storage capacity on the storage medium 5500 while recording images of the moving object. It is recommended that the frame rate be the same as that used for continuous recording and event recording functions.

In motion detection mode, the conditions for recording captured images (i.e., trigger conditions) may be different for some or all of camera 5515, camera 402, and interior rear camera 102. For example, the control unit 5511 may vary the motion detection sensitivity for some or all of camera 5515, camera 402, and interior rear camera 102. The image recording conditions may be configured so that the user can set them for each of camera 5515, camera 402, and interior rear camera 102. For example, the motion detection mode may be configured so that the user can select "OFF" (i.e., no image recording in response to motion detection), "low," "normal," or "high." The latter setting makes it easier for even the slightest movement of a moving object to be recorded. By default, the sensitivity may be set to "normal" for each of camera 5515, camera 402, and interior rear camera 102.

Note that the photographing device 101F may not have a display unit 5513. The housing of the photographing device 101F does not have to be rectangular, and may be, for example, cylindrical. The process of recording data on the storage medium 5500 may be replaced with or combined with, for example, a process of recording data on a storage device other than the storage medium 5500 or a process of transmitting data to an external device.

[b. Viewer and related functions]
FIG. 28 is a diagram showing an example of a viewer screen for playing back images captured by the camera 5515 of the image capturing device 101F, the camera 402 of the image capturing device 101R, and the in-vehicle rear camera 102. This viewer may be realized by a viewer program for a personal computer, smartphone, tablet, or other information processing device, as described above, or may be realized by a viewer program used to display images captured by the image capturing device 101F. FIGS. 29 to 31 are diagrams illustrating the display elements included on the viewer screen. The circled numbers in FIG. 29 correspond to the numbers in the "No." column shown in the tables of FIGS. 30 and 31. Each display element included on the viewer screen has a display name shown in the tables of FIGS. 30 and 31 and performs the function described in FIGS. 30 and 31. 30 and 31, the "front camera" corresponds to the camera 5515, the "rear rear camera" corresponds to the camera 402, and the "rear interior camera" corresponds to the rear camera 102 for photographing the interior of the vehicle.

As shown in FIG. 28, the image captured by camera 5515 capturing the view in front of the vehicle is displayed in rectangular area T1 at the largest size. To the right of area T1, smaller images captured by camera 402 of imaging device 101R are displayed in rectangular area T2, and images captured by interior-mounted rear camera 102 are displayed in rectangular area T3. The viewer may have a function for synchronously playing back images captured by camera 5515, camera 402, and interior-mounted rear camera 102. This allows the user to simultaneously check the view of the front, rear, and interior of the vehicle at each point in time. As shown in FIG. 32(a), a playback list L is displayed on the viewer screen by default. However, depending on the user's operation, a map M showing the shooting location may be displayed instead of playback list L, as shown in FIG. 32(b).

The viewer program should also have two modes for displaying images captured by camera 402: one that displays the captured image as is (i.e., a mode that displays a normal image that looks exactly as it does from top to bottom and left to right), and one that displays an image that is flipped left to right (i.e., a mode that displays a mirror image). Camera 402 and the interior rear camera 102 may capture images in opposite directions. In this case, the left-right relationship between the images captured by these cameras will be reversed. Therefore, the viewer program should be configured to allow the user to select whether to display the captured image as is (i.e., display a normal image) or display a mirror image to align the left and right.

It is preferable that the images from each camera recorded on storage medium 5500 by imaging device 101F be fixed as normal images, but it is also possible to configure it so that, for example, mirror images captured by camera 5515 or interior rear camera 102 can be recorded. Also, while images captured by interior rear camera 102 are fixed as normal images, the viewer program may be provided with a mode that displays mirror images that determine left and right. This is expected to have the effect of improving the flexibility of image recording and display.

[c. Other examples of images captured by the vehicle interior rear camera 102]
Figure 33 is a diagram showing another example of an image captured by the interior rear camera 102, different from that shown in Figure 11. As shown in Figure 33, the interior rear camera 102 can capture and record a wide-angle image of the left and right conditions when viewed from the rear of the vehicle toward the front of the vehicle. Furthermore, as shown in areas T4 and T5, it can capture and record images of the left and right conditions of the vehicle, more specifically, the surrounding conditions such as side roads and adjacent lanes located on the outside in the vehicle width direction. Furthermore, as shown in area T6, it is possible to capture images that respect the privacy of passengers, with the faces of the passengers not or hardly visible in the images.

[d. Relationship between the forward vehicle approach warning function and recording of captured images]
The photographing device 101F may have a forward vehicle approach warning function. The forward vehicle approach warning function is a function that warns the vehicle when it approaches a forward vehicle (i.e., a leading vehicle) while the vehicle is traveling. The forward vehicle approach warning function is a function for avoiding, for example, unintentional tailgating by the driver against the forward vehicle or a collision with the forward vehicle. The detection of the approach of the forward vehicle may be performed, for example, by the control unit 5511 analyzing an image captured by the camera 5515 or by using a sensor such as a laser ranging sensor. The warning may be interpreted as an alarm, a notification, or the like, and may be performed by a sound, voice, display, light emission, or other method that can be perceived by a human (e.g., the driver).

Under this configuration, when issuing a warning using the forward vehicle approach warning function, the control unit 5511 of the photographing device 101F may record images captured by the camera 5515, the camera 402, and the rear camera 102 for capturing images inside the vehicle interior. This makes it possible to later check the situation in a wide range, including the area in front of the vehicle, the area behind the vehicle, and the interior of the vehicle when approaching a forward vehicle. The period for recording images may coincide with the period during which the forward vehicle approach warning function is issuing a warning, but it may also be set to include a predetermined period before, after, or both before and after that period. This makes it possible to later check the situation in front of, the area behind the vehicle, and the interior of the vehicle around the time of approaching a forward vehicle. The length of the recording period for images captured by camera 5515, camera 402, and interior rear camera 102 may be the same, or at least some of them may be different. For example, if the length of the recording period for camera 5515 is made longer than the length of the recording period for camera 402 and interior rear camera 102, this will conserve recording capacity on storage medium 5500, etc., while also being useful for later understanding the situation in front of the vehicle in detail.

Note that while the relationship between the forward vehicle approach warning function and the recording of images captured by camera 5515, camera 402, and interior rear camera 102 has been described above, the forward vehicle approach warning function can be replaced with or combined with the lane departure warning function or the forward vehicle departure warning function. The lane departure warning function is a function that issues a warning if the host vehicle deviates from its lane while in motion. The forward vehicle departure warning function is a function that issues a warning if the host vehicle does not start moving even though the vehicle in front has started moving while the host vehicle is stopped. The lane departure warning function may detect lane departure, or the forward vehicle departure warning function may detect that the vehicle in front has started moving while the host vehicle is stopped, by having the control unit 5511 analyze images captured by camera 5515, or by other well-known methods.

[e. Relationship between rear vehicle approach recording/warning function and recording of captured images]
The image capturing device 101F may have a rear vehicle approach recording and warning function. The rear vehicle approach recording and warning function is a function that issues a warning when the vehicle approaches a rear vehicle (i.e., a following vehicle) while traveling. The rear vehicle approach recording and warning function is a function that detects dangerous driving, such as tailgating (failure to maintain a safe distance) by a rear vehicle, and records at least an image captured by the camera 402 and issues a warning. The detection of the approach of a rear vehicle may be performed, for example, by the control unit 5511 analyzing an image captured by the camera 402 or by using a sensor such as a laser ranging sensor. The warning may be interpreted as an alarm, a notification, or the like, and may be performed by a sound, voice (e.g., outputting a message such as "Dangerous driving behind has been detected!"), a display, light, or other method that can be perceived by a human (e.g., the driver).

Under this configuration, when issuing a warning using the rear vehicle approach recording and warning function, the control unit 5511 of the photographing device 101F may record images captured by the camera 5515, the camera 402, and the rear camera 102 for capturing images inside the vehicle interior. This makes it possible to later check the situation in a wide range, including the area in front of the vehicle, the area behind the vehicle, and the interior of the vehicle, when a rear vehicle approaches. The period for recording images may coincide with the period during which the warning using the rear vehicle approach recording and warning function is issued, but it may also be set to include a predetermined period before, after, or both before and after that period. This makes it possible to later check the situation in front of, the area behind the vehicle, and the interior of the vehicle around the time the rear vehicle approached. The recording periods for images captured by camera 5515, camera 402, and interior rear camera 102 may be the same in length, or at least some of them may be different. For example, making the recording periods for camera 402 and interior rear camera 102 longer than that of camera 5515 saves recording capacity on storage medium 5500, etc., while being particularly useful for later understanding the situation behind the vehicle in detail. Note that although the function has been described as a "rear vehicle approaching recording and warning function," this may be replaced with a "rear vehicle approaching recording function" that does not provide a warning.

In the above-described embodiments, the terms "engine" and "accessory power source" may be replaced with "drive source" and "vehicle power source." Examples of vehicles include internal combustion engine vehicles that use an engine as a drive source, hybrid vehicles that use an engine and a traction motor as drive sources, and electric vehicles that use a traction motor as a drive source.

In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.
The above-described embodiments disclose at least the configurations described in the claims.

The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed herein. The configurations of the present invention for which a patent is sought are specified in the accompanying claims. However, it is the intention of the present inventors to include configurations disclosed in this specification that are not currently specified in the claims in the future.

The present invention is not limited to the configurations described in the above-described embodiments. The components of each of the above-described embodiments and variations may be arbitrarily selected and combined. Furthermore, any component of each embodiment or variation may be arbitrarily combined with any component described in the Summary of the Invention or any component embodying any component described in the Summary of the Invention. It is our intention to obtain rights to these as well through amendments to this application or divisional applications, etc. Even if a statement such as "in the case of..." or "when..." is included, it is not intended to describe a configuration that is limited to that case or time. Configurations that are not limited to those cases or times are also disclosed, and we intend to obtain rights to them. Furthermore, any descriptions that specify an order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and we intend to obtain rights to them.

Furthermore, by converting the drawings to a design registration application, it is intended to obtain rights to the overall design or partial design. The drawings depict the entire device using solid lines, but they also include partial designs claimed for parts of the device. For example, not only can a partial design be made of a part of the device, but the drawings also include a part of the device as a partial design regardless of the part. A part of the device could be a part of the device, or it could be part of that part. It is intended to obtain rights not only to the overall design, but also to partial designs in which any part of the solid line portion of the drawings is made into a dashed line. Furthermore, the modules, parts, components, images, etc. inside the device's casing, all of which are shown in the drawings, are subject to trade independently, and it is intended to obtain rights to them similarly by converting the drawings to a design registration application.

The above-described embodiment discloses at least the following configurations.
[Configuration 1]
This information processing system
It is preferable to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle.
In this way, it is possible to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle. For example, in conventional video data of the interior of the vehicle captured from the front of the vehicle toward the rear of the vehicle, objects, elements, events, etc. that are located in blind spots and are impossible or difficult to process can be detected, identified, recognized, judged, etc. by capturing the interior of the vehicle from a direction different from the conventional one, i.e., by capturing the image from the rear of the vehicle toward the front of the vehicle, and thus, the user can utilize newly provided functions based on this processing.
[Configuration 2]
This information processing system
It is preferable to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, and video data of the exterior of the vehicle captured from the rear of the vehicle toward the outside behind the vehicle.
In this way, it is possible to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, and video data of the exterior of the vehicle captured from the rear of the vehicle. For example, in conventional video data of the interior of the vehicle captured from the front of the vehicle toward the rear of the vehicle, objects, elements, events, etc. that are located in blind spots and are impossible or difficult to process can be detected, identified, recognized, judged, etc. by capturing the interior of the vehicle from a direction different from the conventional one, i.e., by capturing the image from the rear of the vehicle toward the front of the vehicle, and therefore, the user can use newly provided functions based on this processing.
[Configuration 3]
The information processing system according to the present invention comprises:
It is preferable that the vehicle control device has a control unit that analyzes the video data of the inside of the vehicle cabin and determines the situation based on the analysis results.
In this way, the control unit can analyze video data of the interior of the vehicle captured from a different direction than before, making it possible to perform new or more accurate judgment processing for various situations than before, allowing the user to take advantage of new functions based on new, more accurate judgments.
[Configuration 4]
The information processing system according to the present invention comprises:
The control unit may determine the presence or absence of a person in the vehicle cabin based on the video data of the vehicle cabin, and create and output video data that has been filtered to remove the person.
In this way, by filming the interior of the vehicle from the rear toward the front of the vehicle, the chances of people's faces being photographed are reduced, and furthermore, by applying a filter to the people photographed, the actions of people inside the vehicle can be photographed and recorded with greater consideration given to their privacy than before. This reduces the resistance felt by users, for example, by passengers of buses and taxis, about having their images recorded.
[Configuration 5]
The information processing system according to the present invention comprises:
The control unit may determine the state of an object outside the vehicle that is captured in the video data based on the video data of the interior of the vehicle, and output information relating to the state of the object.
In this way, the control unit can determine, for example, when a turn signal is turned on, a situation where there is a car, person, or obstacle on the side, a situation where someone is approaching the vehicle unnaturally when the vehicle is parked with no one inside, a situation where such a person is looking into the vehicle, a situation where people, particularly children, are present, approaching, or playing around the vehicle when the engine is started, when the vehicle is slowing down, when the vehicle is traveling, etc., a situation where the vehicle is being tailgated from the side from behind, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc. depending on the results of the determination of these situations, so that the user can quickly notice and respond to abnormalities or dangers in or around the vehicle.
[Configuration 6]
The information processing system according to the present invention comprises:
The control unit may determine an abnormality in an object installed on the ceiling of the vehicle based on the video data of the interior of the vehicle compartment, and output information related to the abnormality.
In this way, the control unit can determine, for example, the lighting status of the room lamp and the presence or absence of a person in the vehicle, whether the net is sagging due to excessive luggage, whether the left and right fasteners have come loose, the orientation and status of the projector, the position and status of the projector's image, whether the relative positions of the projector and rear seat passengers are appropriate, whether the flip-down monitor is open or closed, whether the activation status (whether an image is displayed or not) is appropriate, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver or the like according to the results of the determination of these conditions, so that the user can easily maintain the vehicle in an appropriate state and can quickly notice any abnormalities in the vehicle.
[Configuration 7]
The information processing system according to the present invention comprises:
The control unit may determine a state of luggage in the vehicle based on the video data of the interior of the vehicle, and output information relating to the state of the luggage.
In this way, the control unit can determine, for example, the state of the luggage placed behind the driver's seat, the positional relationship between the luggage and the luggage bed or seat, the estimated position of the center of gravity of the luggage, the positional relationship between the centers of gravity of multiple luggage, the vehicle speed and acceleration and changes therein, the risk of luggage falling from the luggage bed or seat, the risk of stacked luggage collapsing, the degree of slippage of luggage on the luggage bed or seat, and the relationship between the collapse of luggage and tailgating or contact by another vehicle photographed from behind through a vehicle window. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of the determination of the luggage state. This allows the driver to concentrate on driving without worrying about the luggage collapsing.
[Configuration 8]
The information processing system according to the present invention comprises:
The control unit may determine the state of a display unit disposed in the vehicle based on the video data of the interior of the vehicle, and output information relating to the state of the display unit.
In this way, the display unit can determine the status of, for example, a monitor installed on the back of a seat, the status of a monitor installed on the ceiling, the display content and presence or absence of an operation on a center monitor installed near the center of the front between the driver's seat and the passenger seat, the specific operation performed on the display unit, whether the vehicle was in autonomous driving or manual driving at the time of video recording, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of determining these conditions. Furthermore, if the user is involved in an accident, etc. while the driver is operating the vehicle appropriately, the user can provide objective evidence that the accident did not occur while operating the terminal, that the accident was not due to a terminal misoperation, and whether the accident occurred during autonomous driving or manual driving, etc.
[Configuration 9]
The information processing system according to the present invention comprises:
The control unit may determine the presence of a person's head based on the video data, and output information relating to the proportion of the video data that is occupied by the head.
In this way, the display unit can determine, for example, whether a head is visible on the screen, what proportion of the screen the visible head occupies, whether the installation state is such that such an event is anticipated, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of the determination of these situations. This allows the user to avoid recording only video data in which the back of a person's head is constantly visible on the entire screen when sitting in the back seat.
[Configuration 10]
The information processing system according to the present invention comprises:
The control unit may determine whether the vehicle is in a parked state based on the video data, and if the vehicle is in the parked state, may output a command to switch to a parking monitoring mode.
In this way, the control unit can determine whether a person is inside the vehicle, whether a person has left the vehicle, whether the vehicle is parked, etc. If the control unit determines that the vehicle is parked, it outputs a command to switch the shooting mode to parking monitoring mode. As a result, when the vehicle is parked, the control unit can preferentially allocate computing power to processing for identifying a suspicious person approaching from within the area of the vehicle's windshield, rear window, or side window, for example, allowing the user to use a more efficient information processing system.
[Configuration 11]
The information processing system according to the present invention comprises:
The control unit may determine the state of the person inside the vehicle based on the video data, and output information relating to the state of the person inside the vehicle.
In this way, the control unit can determine the status of people inside the vehicle, especially those behind the vehicle, with higher accuracy than before, and the user can receive appropriate notifications and warnings depending on the results of the determination of the person's status.
[Configuration 12]
The information processing system according to the present invention comprises:
The control unit may determine whether a person in the vehicle is abandoned based on the video data, and output information relating to the abandoned state.
In this way, the control unit determines whether there is a person in the video, whether the person in the video is a child or a disabled person, whether there is a person other than a child or a disabled person in the vehicle, whether the vehicle is parked, whether a child, a disabled person, or the like has been left behind, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, the surroundings of the vehicle, etc., depending on the determination result of the abandonment situation. This allows the user to use the highly accurate rear-side abandonment prevention function.
[Configuration 13]
The information processing system according to the present invention comprises:
The control unit may determine whether a person in the vehicle is dozing or sleeping based on the video data, and output information relating to the dozing or sleeping state.
In this way, the control unit determines whether there is a person in the video, whether the person in the video is the driver or a passenger, whether the driver's head is visible, whether the state of the driver's head indicates a warning pattern for drowsiness, whether the face or head of a passenger in the rear seat is visible, and whether the state of the face or head indicates a sleeping pattern, etc. Furthermore, the control unit can output appropriate notifications or warnings to the driver or the surroundings of the vehicle, etc., depending on the determination result of the abandoned vehicle situation. This makes it easier for the driver to check for signs of drowsiness himself or herself and to take into consideration the sleeping state of the passenger when driving.
[Configuration 14]
The information processing system according to the present invention comprises:
It is preferable that a human detection sensor is provided, and a detection signal from the human detection sensor is at least used as a trigger for starting the determination or for the determination.
In this way, for example, by using a pyroelectric infrared sensor or a thermosensor, it is possible to identify children, disabled persons, etc. with a lower load than AI processing of video data. Since constantly executing AI processing requires a large computational load, it is advantageous to detect the presence of people using a separate human-detection sensor and use the result of the human-detection sensor's detection of a person inside the vehicle as a trigger to start human detection processing based on video data, in terms of reducing the computational load and achieving high-accuracy human detection. Furthermore, for example, by limiting the detection range of the human-detection sensor to the rear area of the vehicle or near the rear seats, it is possible to distinguish between the driver and passengers with high accuracy. This allows users to reduce the risk to passengers.
[Configuration 15]
The information processing system according to the present invention comprises:
It is preferable to photograph the interior of the vehicle from the rear of the vehicle toward the front of the vehicle at a plurality of different positions.
In this way, for example, by combining video data captured at multiple positions using a wide-angle camera, the user can get a full view of the interior and exterior of the vehicle.
[Configuration 16]
The information processing system according to the present invention comprises:
It is preferable that the opening and closing operation of the opening/closing part of the vehicle is used as a trigger to start taking or recording images, or to change settings for taking or recording images.
In this way, by triggering the opening or closing of a hatch or the like to start shooting or recording video or change the video shooting or recording settings, the user can reliably obtain video data that has been shot with appropriate settings even when there are major changes in the shooting environment or shooting conditions.
[Configuration 17]
The information processing system according to the present invention comprises:
In addition to the opening and closing operation of the opening and closing part of the vehicle, the vehicle running state, the state of the drive source of the vehicle, and the state of the power source of the vehicle are used as triggers,
Begin filming or recording footage; or
It is advisable to change the settings for taking or recording video.
In this way, in addition to the opening and closing operation of the vehicle's opening and closing part, it is possible to start taking or recording images or change settings in a manner that corresponds to at least one of the following conditions: the vehicle's running state, the state of the vehicle's drive source, and the state of the vehicle's power supply.
[Configuration 18]
The information processing system according to the present invention comprises:
Recording the sounds collected by at least one of the sound collection means provided at a plurality of locations in the vehicle together with the captured video;
It is preferable that the settings of the audio to be recorded be changed using the opening and closing operation of the opening and closing part of the vehicle as a trigger.
In this way, the settings for the audio to be recorded can be changed using the opening and closing operation of the opening and closing part of the vehicle as a trigger.
[Configuration 19]
The information processing system according to the present invention comprises:
The opening/closing unit of the vehicle includes a rear opening/closing unit that opens and closes an opening provided at the rear of the vehicle,
The camera of the photographing device provided in the rear opening/closing section photographs the outside behind the vehicle when the opening/closing section is in the open state, and photographs from the rear of the vehicle toward the front of the vehicle when the rear opening/closing section is in the closed state.
In this way, the shooting direction changes depending on the open/closed state of the opening provided at the rear of the vehicle, and when the rear opening/closing part is in the open state, the outside behind the vehicle is photographed, and when it is in the closed state, the front of the vehicle is photographed from the rear of the vehicle, so that the appropriate direction can be photographed depending on the open/closed state.
[Configuration 20]
The information processing system according to the present invention comprises:
The rear opening/closing portion opens upward,
The camera of the photographing device may take photographs of the exterior of the rear of the vehicle facing downward when the rear opening/closing section is in an open state, and may take photographs from the rear of the vehicle facing forward when the rear opening/closing section is in a closed state.
In this way, the photographing direction changes depending on the open/closed state of the opening that opens upward, and when the rear opening/closing part is in the open state, the outside behind the vehicle is photographed facing downward, and when it is in the closed state, the front of the vehicle is photographed from the rear of the vehicle, so that the appropriate direction can be photographed depending on the open/closed state.
[Configuration 21]
The information processing system according to the present invention comprises:
When the rear opening/closing section changes from a closed state to an open state, it is preferable to start recording images captured by the camera of the photographing device in a predetermined manner.
In this way, when the rear opening/closing section changes from the closed state to the open state, the predetermined recording of the image captured by the photographing device can be started. When the rear opening/closing section changes from the closed state to the open state, it is conceivable that a person is getting on or off, so it is possible to photograph and confirm an event that occurs when a person is present.
[Configuration 22]
The information processing system according to the present invention comprises:
the photographing device is operable by receiving power from an external source or by an internal power source when a drive source or a power source of the vehicle is turned off;
When the rear opening/closing section is opened while the drive source or power source of the vehicle is turned off, it is preferable to start recording the images captured by the photographing device in a predetermined manner.
In this way, when the rear opening/closing section changes from the closed state to the open state, the image captured by the photographing device can be recorded in a predetermined manner. When the rear opening/closing section changes from the closed state to the open state, it is thought that an event such as people getting on or off or putting something in or taking something out may occur, and it is possible to photograph such an event.
[Configuration 23]
The information processing system according to the present invention comprises:
The photographing device is in standby mode with its power turned on during the designated time period,
If the rear opening/closing section changes from a closed state to an open state within the specified time period, the predetermined recording may be started.
In this way, when the rear opening/closing section changes from a closed state to an open state, the specified recording of the image captured by the photographing device can be started immediately, and the situation when the rear opening/closing section changes to the open state can be more reliably photographed and recorded.
[Configuration 24]
The information processing system according to the present invention comprises:
The photographing device takes pictures with the camera when the rear opening/closing section is in the closed state and the drive source or power supply of the vehicle is on, and when the rear opening/closing section is in the open state, the photographing by the camera is extended for a specified length of time and the captured images are recorded even after the drive source or power supply of the vehicle is turned off.
In this way, the image is taken from the rear of the vehicle toward the front of the vehicle, and when the rear opening/closing part changes from a closed state to an open state, the image is recorded for an extended period of time by a specified length of time, so that even after the drive source or power source of the vehicle is turned off, the situation when the rear opening/closing part changes to the open state can be photographed and recorded.
[Configuration 25]
The information processing system according to the present invention comprises:
When the rear opening/closing section is in the open state, if the photographing device determines that a person is getting off through the opening, it is preferable that the photographing device record the captured image for an extended period of time by the specified length of time.
In this way, the object photographed by the rear camera inside the vehicle can be continuously photographed in the rearward direction outside the vehicle.
[Configuration 26]
The information processing system according to the present invention comprises:
It is preferable that the image data of the inside of the vehicle and the image data of the outside of the vehicle are captured by a single camera.
In this way, at least an image of the interior of the vehicle taken from the rear of the vehicle toward the front and an image of the exterior of the vehicle from the rear of the vehicle can be captured as a single image data. By selecting an appropriate angle of view and mounting position, the image data can also include images of the area immediately behind the vehicle, images of the area in front of the vehicle, and images of the areas in front of and behind the vehicle at an elevation angle. This allows users to add various functions to their vehicles by installing a single wide-angle camera at the rear of the vehicle. Furthermore, implementing a single rear camera reduces costs, reduces installation effort, and enables miniaturization compared to using two rear cameras, one for the interior and one for the rear [Configuration 27].
The information processing system according to the present invention comprises:
The one camera may be a semi-spherical camera.
In this way, it is possible to cover a wider range of image capture than a general wide-angle camera. For example, a hemispherical camera with a field of view of 240° or more can capture not only images of the interior of the vehicle taken from the rear of the vehicle toward the front, images immediately behind the vehicle, and images in front of the vehicle, but also images in the elevation direction in front of and behind the vehicle as a single image data. Therefore, from the image data, the user can check, for example, the status of traffic lights in front of and behind the vehicle at a desired time [Configuration 28].
The information processing system according to the present invention comprises:
When it is determined that a person is in the vehicle, image information of an image taken inside the vehicle cabin may be output to a predetermined output destination.
In this way, it is possible to easily grasp the state of the vehicle when a person is in the vehicle from the captured image. For example, trouble or some other event often occurs in the vehicle cabin when a person is in the vehicle. Making it easy to grasp the state of the vehicle when a person is in the vehicle is useful in confirming whether or not such an event has occurred and the circumstances under which it has occurred.
[Configuration 29]
The information processing system according to the present invention comprises:
If it is determined that a person is in the vehicle, image information of an image taken inside the vehicle cabin may be output to the specified output destination regardless of whether the vehicle's drive source or the vehicle's power source is on or off.
In this way, it is possible to easily grasp the state of a vehicle when a person is in the vehicle from the captured image, regardless of whether the vehicle's drive source or the vehicle's power source is on or off. For example, if a person is in the vehicle, there is a possibility that a problem or some kind of event will occur in the vehicle's cabin, regardless of whether the vehicle's drive source or the vehicle's power source is on or off. Therefore, being able to easily grasp the state of a vehicle when a person is in the vehicle, regardless of whether the vehicle's drive source or the vehicle's power source is on or off, is useful in determining whether or not such an event will occur and the circumstances under which it will occur.
[Configuration 30]
The information processing system according to the present invention comprises:
When it is determined that a person has spoken inside the vehicle cabin, image information of an image captured inside the vehicle cabin may be output to a predetermined output destination.
In this way, it is possible to easily grasp from the captured image what is happening inside the vehicle when a person is speaking. For example, if a problem or some other event occurs inside the vehicle, the person may speak about it. Being able to easily grasp what is happening inside the vehicle when a person is speaking is useful in confirming whether or not such an event has occurred and the circumstances under which it occurred.
[Configuration 31]
The information processing system according to the present invention comprises:
When it is determined that a person has reached out their hand in the vehicle, image information of an image taken of the interior of the vehicle may be output to a predetermined output destination.
In this way, it is possible to easily grasp from the captured image what happens when a person in the vehicle reaches out their hand. For example, if a problem or some other incident occurs in the vehicle interior, it is conceivable that a violent or threatening act may be committed. If a person is seen extending their arm, it is suspected that an act such as hitting may have occurred. Making it easier to grasp what happens in the vehicle interior when a person extends their hand is useful in confirming whether or not such an incident has occurred and the circumstances under which it occurred.
[Configuration 32]
The information processing system according to the present invention comprises:
It is preferable to include a photographing device having a first camera that captures video data of the inside of the vehicle and a second camera that captures video data of the outside of the vehicle.
This allows the camera that captures video data inside the vehicle and the imaging device that captures video data outside the vehicle to be designed separately and more flexibly according to the purpose of the imaging, etc. Also, the user can more flexibly adjust the positional relationship between the camera and the imaging device.
[Configuration 33]
The information processing system according to the present invention comprises:
The photographing direction of the first camera, which photographs the interior of the vehicle from the rear of the vehicle toward the front of the vehicle, may be changeable to a direction from the rear of the vehicle toward the outside rear of the vehicle.
In this way, the camera for recording inside the vehicle can be used together with the camera for recording behind as a camera that can accurately measure the distance to the vehicle behind. By adding one camera to this information processing system, the user can add a new function of analyzing and judging images outside the vehicle in addition to the function of analyzing and judging images inside the vehicle.
[Configuration 34]
The information processing system according to the present invention comprises:
It is preferable that the photographing direction of the first camera be changeable from a direction from the rear of the vehicle toward the passenger compartment to a direction from the rear of the vehicle toward the outside rear of the vehicle by rotating the first camera.
In this way, the user can easily and quickly change the shooting direction and use the camera for shooting inside the vehicle together with the camera for shooting behind as a camera that can accurately measure the distance to the vehicle behind. By adding one camera, this information processing system can provide a new function of analyzing and judging images outside the vehicle in addition to the function of analyzing and judging images inside the vehicle.
[Configuration 35]
The information processing system according to the present invention comprises:
The first camera may be provided with a floodlight having a light projection direction that is linked to the adjustment of the shooting direction, and the light from the floodlight may be electrically or mechanically blocked in response to a change in the shooting direction.
In this way, it is possible to capture clear images of the interior of the vehicle even in dark environments such as at night, while the user can use the vehicle with confidence that the floodlight does not emit light that can be seen by people outside the vehicle.
[Configuration 36]
The information processing system according to the present invention comprises:
The first camera may be attached to the photographing device by including a mounting fixture having a cylindrical portion and inserting the first camera into the cylindrical portion.
In this way, it becomes possible to use the so-called pipe joint method for attaching the camera to the camera and photographing equipment, and the user can attach an additional camera to the information processing system in a complete and intuitive manner.
[Configuration 37]
The information processing system according to the present invention comprises:
It is preferable that the relative position of the first camera with respect to the photographing device or the photographing direction of the first camera can be changed while the first camera is inserted inside the cylindrical portion.
In this way, not only can the user attach an additional camera to the information processing system in a complete and intuitive manner, but by changing the relative position of the additional camera to the imaging equipment or the imaging direction of the first camera, the user can easily adjust the camera to capture the desired image.
[Configuration 38]
The information processing system according to the present invention comprises:
The first camera may have a pin plug that can be inserted into the photographing device.
In this way, it becomes possible to easily attach the additional first camera to an existing photographic device. Since existing photographic devices generally have standardized pin jacks, by providing a standardized pin plug corresponding to the additional first camera, users can adopt various photographic devices to this information processing system.
[Configuration 39]
The information processing system according to the present invention comprises:
It is preferable that other video data captured by a third camera can be input from the third camera to the first camera, and that the image data captured by the first camera and the other video data captured by the third camera can be output from the first camera to the photographing device.
In this way, the first camera can not only output the video data that it has captured itself to the filming equipment, but can also output other video data that has been captured by another third camera to the filming equipment in the same way, allowing the user to introduce this system with a simple setup using a single cable.
[Configuration 40]
The program to achieve the above objectives is
It is preferable to provide a program for realizing the above-mentioned functions in an information processing system.
In this way, the above functions can be provided by providing a program to an existing information processing system.

The inventions shown in [Configuration 1] through [Configuration 40] above can be combined in any way. For example, a configuration may be created by adding at least a portion of the configuration of at least one of the inventions from [Configuration 2] onward to all or a portion of the configuration of the invention shown in [Configuration 1]. In particular, an invention may be created by adding at least a portion of the configuration of at least one of the inventions from [Configuration 2] onward to the invention shown in [Configuration 1]. Furthermore, any configuration may be extracted from the inventions shown in [Configuration 1] through [Configuration 40] and combined. The applicant of this application intends to obtain rights to inventions that include these configurations. Furthermore, even if a description is made of "in the case of..." or "when...," it is not intended to describe a configuration that is limited to that case or time. These are merely examples of better configurations, and the applicant intends to obtain rights to configurations that are not in these cases or times. Furthermore, parts that are described in order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and the applicant intends to obtain rights to them.

10... vehicle, 101F, 101R... photographing equipment, 102... rear camera for photographing inside the vehicle, 103... user terminal, 104... external server, 105... network

Claims (7)

It processes the video data of the interior of the vehicle taken from the rear of the vehicle towards the front,
a control unit that analyzes the video data of the vehicle interior and determines the situation based on the analysis result;
The control unit has a function of determining whether or not a human head is present based on the video data of the interior of the vehicle, and outputting information regarding the proportion of the head in the video data.
Information processing system. The information processing system according to claim 1 , wherein the control unit has a function of outputting a notification or a warning when the head is captured in the video data of the interior of the vehicle . The information processing system according to claim 1 or 2, wherein the control unit has a function of stopping the process of determining whether or not the head is present when the head is visible in the video data of the interior of the vehicle and while the door of the vehicle is open . It processes the video data of the interior of the vehicle taken from the rear of the vehicle towards the front,
a control unit that analyzes the video data of the vehicle interior and determines the situation based on the analysis result;
the control unit has a function of determining the area of the person's head based on the video data of the interior of the vehicle and outputting information relating to the proportion of the video data that is occupied by the head.
Information processing system. The information processing system according to claim 4 , wherein the control unit has a function of outputting a notification or a warning when a proportion of the head captured in the video data of the vehicle interior within the screen is equal to or greater than a predetermined proportion . The information processing system according to claim 4 or 5, wherein the control unit has a function of outputting a notification or a warning when the proportion of the head captured in the video data of the vehicle interior within the screen is equal to or greater than a predetermined proportion for a predetermined period of time or more within a set period of time. A program for causing an information processing system to realize each function according to any one of claims 1 to 6 .