JP7657480B2 - Systems and programs - Google Patents

Description

The present invention relates to a system, a program, etc.

Systems that are equipped with cameras and record images from the cameras are known.

JP 2018-196066 A

For example, there have been various problems with systems equipped with functions such as recording images from such cameras. One of the objectives of the present invention is to provide a system that is superior to conventional systems.

The purpose of the invention of this application is not limited to this, and the applicant intends to obtain rights to configurations that aim to obtain effects from parts of the configuration disclosed in this specification and drawings, etc., through divisional applications, amendments, etc. For example, this specification discloses problems in which the phrase "can be" in this specification is read as "the problem is that...". The problems are described as independent, and the applicant intends to obtain rights to the configurations for solving each problem separately through divisional applications, amendments, etc. Even if the problems are implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in this specification in the scope of the patent claim by amendment or divisional application. In addition, the applicant has disclosed a configuration that solves a problem that combines these independent problems, and the applicant intends to obtain rights to it.

The present invention can be understood as the following aspects, for example.
(1) A system having a camera and a camera image recording circuit that compresses images captured by the camera and stores them in a memory means, and preferably the system also has a signal distribution means that distributes signals transmitted between the camera and the camera image recording circuit to electronic components that realize specified functions based on the signals.

In this way, the image captured by the camera can be compressed and stored in the storage means, and the electronic components that realize the specified function can realize the specified function based on the signal distributed between the camera and the camera image recording circuit.

The camera may be of various types, but may be a CCD (Charge Coupled Device) camera, and may be a CMOS (Complementary Metal-Oxide-Semiconductor) camera in particular. The camera may be a camera for capturing still images, but may be a camera for capturing moving images in particular. The camera may be a special camera such as an infrared camera, but may be a camera for capturing visible light, and may be a visible light camera in particular. The camera may be a camera that can capture monochrome images, but may be a camera that can capture color images. The camera may be, for example, a camera with red (R), green (G), and blue (B) color filters arranged on a plane. The camera resolution may be VGA or higher.

The camera may refer to a part that includes an image sensor, for example, but it may also refer to the image sensor itself.

In particular, the camera may be a camera installed in the vehicle. In particular, the camera may be a camera that is retrofitted after the vehicle is shipped by the manufacturer.

The video compression section in the camera video recording circuit can use various algorithms, but it is preferable to use an encoder with a standardized algorithm, and in particular, to have at least one of an H.264 encoder or an H.265 encoder. In particular, the video encoder in the camera video recording circuit should be configured with hardware, not with software processing. It is preferable that the encoder is a SoC (System-on-a-Chip) rather than a simple FPGA (Field-Programmable Gate Array). In this way, the camera video recording circuit may be a camera video recording chip. It is preferable that the camera video recording circuit is configured with a hard macro, not a soft macro.

It is particularly preferable that the camera video recording circuit is sold as a dedicated chip for camera video recording. The camera video recording circuit may be, for example, an ASSP for camera video recording, an SoC for camera video recording, or an LSI (Large Scale Integrated circuit) for camera video recording, and in particular, the camera video recording circuit may be an SoC for a drive recorder.

The storage means is preferably a removable storage means, and more preferably a card-shaped medium such as an SD (registered trademark) card or a micro SD (registered trademark) card.

The signal transmitted between the camera and the camera video recording circuit may be a control signal, but should preferably include at least one of a video signal and a video synchronization signal. The video synchronization signal should preferably be at least one of a pixel clock, a vertical synchronization signal, or a horizontal synchronization signal, and should preferably be at least two of these. In particular, it should preferably be a pixel clock and a vertical synchronization signal.

The video signal transmitting the camera image may be transmitted, for example, in a serial manner with the image synchronized on a video channel basis, or in a parallel manner. Alternatively, serial and parallel transmission may be combined. In particular, it is advisable to overlay data such as the current time, event information, and sensor information on a video frame basis, or to overlay the data on the video. Data such as the current time, event information, and sensor information may be overlaid on the synchronized units, or may be overlaid on the video.

When distributing a video signal, for example, a video signal of a specific protocol may be input, and distributed to multiple output destinations using the same protocol as the input. For example, if the camera outputs a MIPI CSI-2 signal as the video signal, the signal distribution means may be a MIPI CSI-2 signal splitter chip.

When distributing video signals, it is preferable to input a video signal of a specific protocol and output signals of the same protocol to multiple output destinations, which are different from the input protocol. For example, in the case of a camera that outputs an FPD-LINK III signal as a video signal, the signal distribution means should be a circuit with ports that input the FPD-LINK III signal and output it as multiple MIPI CSI-2 signals (for example, a dual port output deserializer hub with one port input and two ports output). In particular, the port that outputs the MIPI CSI-2 signal should have the function of outputting a replica of one port to another port. In particular, it is preferable to use a deserializer hub chip.

The electronic component that realizes the specified function may preferably include at least one of a camera separate from the camera or an image recognition circuit. When the electronic component that realizes the specified function includes a camera, the specified function may preferably be a function for capturing an image of a separate area that includes a range different from that of the camera. When the electronic component that realizes the specified function includes an image recognition circuit, the specified function may preferably be an image recognition function.

The signal distribution means may be, for example, a simple wiring branch, but it is particularly preferable that the means has a function of distributing signals so that signals transmitted between the multiple distributed signal lines can be synchronized, and it is particularly preferable that the means is a circuit (for example, a chip) having a function of distributing signals so that signals transmitted between the multiple distributed signal lines can be synchronized.

The camera and the camera video recording circuit may be configured to be housed in the same housing, but it is preferable to configure them to be housed in separate housings and connect the two housings with a connection means such as a cable or connector. It is preferable to configure the camera and the camera video recording circuit to be connected via a cable of at least one meter. In particular, it is preferable for the camera and the camera video recording circuit to be connected via a cable that is routed inside the vehicle.

(2) The electronic component may have another camera different from the camera, the signal may include a video synchronization signal generated by a synchronization signal generating means, and the video signals of the camera and the other camera may be output to the camera video recording circuit based on the distributed synchronization signal.

In this way, a video signal synchronized between the camera and another camera can be obtained on the camera video recording circuit side. For example, a video signal synchronized between the camera and another camera can be input to the camera video recording circuit, and video based on the input synchronized multiple video signals can be compressed and recorded. For example, when the recorded multiple video signals are played back simultaneously, multiple synchronized videos can be viewed. Note that the video may be compressed into separate files for each of the multiple cameras, but it is preferable to compress the video into a single file.

(3) The camera may have the synchronization signal generating means, and the signal distribution means may be configured to distribute the synchronization signal to the other camera.

In this way, the video signal output from the other camera will be a video signal based on the synchronization signal generated within the camera, making it easier to synchronize the two cameras. It is particularly effective to adopt this configuration and have the camera and the other camera housed in the same housing. This is particularly effective when the synchronization signal has to be routed over a short distance.

In the camera, it is especially good to generate it within the image sensor. This way, there is no need to use an image sensor capable of external synchronization that captures images in sync with the input synchronization signal, and it can be configured easily and at low cost.

(4) The synchronization signal generating means may be provided outside the camera and the other camera, and the signal distribution means may be configured to distribute the synchronization signal to the camera and the other camera.

This makes it easier to synchronize the video signal output from the other camera with the video signal output from the other camera.

In particular, it is advisable to adopt this configuration and have the camera and the other camera housed in different housings. This is particularly effective when the synchronization signal has to be routed over a relatively long distance.

(5) The camera video recording circuit may have the synchronization signal generating means, and the signal distribution means may be configured to distribute the synchronization signal to the camera and the other camera.

This increases the likelihood of obtaining synchronized video signals from the camera and another camera, such as at a time that matches the video recording process.

(6) A first video signal conversion circuit having a first video signal conversion means for converting signals output from the camera and the other camera into video signals for transmission via a cable, and a second video signal conversion circuit having a second video signal conversion means for converting the video signals transmitted via the cable into video signals that can be input to the camera video recording circuit, the synchronization signal generation means being provided within the first video signal conversion circuit or the second video signal conversion circuit, and the signal distribution means being provided within the first video signal conversion circuit or the second video signal conversion circuit, and distributing the synchronization signal to the camera and the other camera.

In this way, the video signals from both the camera and the other camera that are synchronized can be easily input to the camera video recording circuit via a cable and recorded.

The cable may be relatively long. By relatively long, it is preferable that the cable be longer than the length required for wiring within a camera unit that includes at least one of the camera and the other camera, and in particular, it is preferable that the cable be longer than one meter.

The first video signal conversion means may be, for example, a serializer, and the second video signal conversion means may be, for example, a deserializer. The first video signal conversion circuit may particularly be a serializer chip, and the second video signal conversion circuit may particularly be a deserializer chip. The transmission distance between the first video signal conversion circuit and the second video signal conversion circuit may be, particularly, several meters when the camera and the other camera are in-vehicle cameras.

For example, the vehicle may be equipped with a first camera unit having the camera and a first first video signal conversion circuit, a second camera unit having the other camera and a second first video signal conversion circuit, and a recording unit having a second video signal conversion circuit and a camera video recording circuit, and the first camera unit and the recording unit may be connected by a first cable and a second cable between the second camera unit and the recording unit to transmit the respective video signals. For example, the vehicle may be equipped with a first camera unit installed so that the camera (first camera) attached to the windshield of the vehicle captures the image of the front of the vehicle, and a second camera unit installed so that the other camera (second camera) captures the image of the rear of the vehicle, and a recording unit installed on the dashboard or under the passenger seat is connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

Also, for example, a first camera unit having the camera and a first first video signal conversion circuit and a second camera unit having the other camera and a second first video signal conversion circuit may be connected by a first cable, and a recording unit having a second video signal conversion circuit and a camera video recording circuit and the second camera unit may be connected by a second cable, and the respective video signals may be transmitted.
Also, for example, one camera unit may be provided with a plurality of cameras including the camera and the other camera, and a first video signal conversion circuit, and the first video signal conversion circuit may convert the video signals of the plurality of cameras to flow through one cable, and the video signals may be transmitted to a recording unit provided with a second video signal conversion circuit and a camera video recording circuit via the single cable, so that the video images of the plurality of cameras are recorded in the recording means. For example, a camera unit may be attached to the windshield of a vehicle, and may be configured to connect a camera unit including the camera (first camera) that captures an image in front of the vehicle and the other camera (second camera) that captures an image behind the vehicle, to a recording unit provided on the dashboard or under the passenger seat, etc., with a cable.

For example, the second video signal conversion circuit may convert the video signals of the multiple cameras that have been converted by the first video signal conversion circuit and transmitted over a relatively long distance via the single cable into signals for transmission over a relatively short distance, which is necessary for wiring within the recording unit.

For example, the second video signal conversion circuit may be configured to output video signals from multiple cameras to the camera video recording circuit so that they are sent over a single transmission path, which is then input by the camera video recording circuit. For example, the second video signal conversion circuit may divide the video signals from two cameras into packets of a predetermined data size, and transmit the packets alternately. Video signals from a larger number of cameras may be output in an interleaved manner over a single transmission path.

Alternatively, for example, the second video signal conversion circuit may be configured to output video signals from multiple cameras to the camera video recording circuit so that they are sent over different transmission paths for each camera, and the camera video recording circuit inputs these. In this way, processing can be easily performed on a camera-by-camera basis in the different transmission paths for each camera, or in the camera video recording circuit connected to these. For example, it becomes relatively easy to record and output information separate from the video by superimposing it on each camera frame for each camera.

It is best to use as few signal lines as possible in a single cable, for example a twisted pair cable.

The transmission path should preferably have as few signal lines as possible, for example, a few signal lines (e.g., printed wiring on a substrate). The camera, the other camera, and the first video signal conversion circuit should be connected via at least one of printed wiring or a flexible cable. The second video signal conversion circuit and the camera video recording circuit should be connected via at least one of printed wiring or a flexible cable.

(7) It is preferable that the lengths of the signal wiring from the signal distribution means to the camera and the other camera are approximately the same.

This increases the likelihood that the video signal from the camera and the video signal from the other camera will be synchronized.

(8) The electronic components may include an image recognition circuit that is provided separately from the camera video recording circuit and performs a predetermined image recognition process, the signal to be transmitted may include a video signal, the image recognition circuit may output information regarding the results of the image recognition process performed based on the distributed video signal to the camera video recording circuit, and the camera video recording circuit may compress the video captured by the camera based on the information regarding the results and store the compressed video in the storage means.

In this way, it is possible to reliably compress the camera image and store it in the storage means, and it is also possible to record based on the results of image recognition processing of the same camera image.

Camera video recording circuits are relatively inexpensive, but because they are designed specifically for compressing and storing video, there are problems with adding features such as image recognition (e.g., little information disclosure, few remaining resources, etc.). For example, SoCs for drive recorders are inexpensive, but because they are designed for drive recorders, there are problems with adding features such as image recognition (e.g., little information disclosure, few remaining resources, etc.).

On the other hand, there is a problem that there is no image recognition circuit used for image recognition that is highly reliable as software for compressing and recording video. The image recognition circuit should be a general-purpose SoC (SoC for image recognition) that can be used by compiling, for example, OSS (Open Source Software). In particular, it is recommended to use an SoC equipped with an accelerator circuit for image recognition or an accelerator circuit for deep learning. There is also the problem that it is costly to mount an image encoder such as H.264 on a general SoC or, in particular, an image recognition circuit that is an FPGA. In addition, even if a general SoC is equipped with an image encoder, there is a problem that there is a lack of reliable software for drive recorders, especially when trying to use it as a drive recorder, and there is also the issue that it is extremely costly to create it from scratch in-house.

For example, a MIPI CSI-2 splitter (e.g., a splitter chip) is used to distribute and input the same camera video signal to the camera video recording circuit and the image recognition circuit. The camera video recording circuit and the image recognition circuit are connected by a communication line, and the image recognition circuit performs image recognition of the state of the target object, and if the result of the recognition indicates that recording is necessary, a trigger signal is passed to the camera video recording circuit via this communication line. When the camera video recording circuit receives a trigger signal from the image recognition circuit, it is preferable for it to event-record the video signal it has received in a storage means.

The communication line may be a parallel communication line or a serial communication line to transmit trigger data. For example, information on the type of recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs (General-purpose input/output). For example, the GPIO output port of the image recognition SoC is connected to the trigger input port of the drive recorder SoC, and when the result of image recognition by the image recognition SoC is that recording is required, the signal of the GPIO output port of the image recognition SoC is changed, and when there is a change in the signal of the trigger input port of the drive recorder SoC, the drive recorder SoC records the image for a predetermined number of seconds before and after the trigger occurrence in the recording means. Furthermore, signals from sensors connected to the camera image recording circuit may be transmitted to the image recognition circuit. For example, if the camera video recording circuit is a SoC for a drive recorder and the image recognition circuit is a SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the information received by the SoC for the drive recorder from the acceleration sensor, switch, and GPS module is output in real time to the SoC for image recognition. The SoC for image recognition may receive this information in real time from the SoC for the drive recorder, use it for image recognition, and output the results to the SoC for the drive recorder. The functions of both SoCs may be realized by executing programs for realizing these functions stored in each SoC.

For example, each of the signals from four cameras is serialized by a serializer and input to a deserializer hub. The deserializer hub outputs serial signals that are interleaved with the signals from the same four cameras to the first and second ports. For example, it is recommended to connect a camera video recording circuit to the first port and an image recognition circuit to the second port.

For example, if the image recognition circuit recognizes a no entry sign but detects an intrusion beyond the sign, it outputs a no entry trigger to the camera video recording circuit. When the camera video recording circuit receives a no entry trigger from the image recognition circuit, it encodes and records video for 15 seconds before and after the no entry trigger in the storage means as no entry trigger video.

In addition, for example, when the image recognition circuit detects that a person has entered the vicinity of the forklift, it outputs a surrounding person recognition trigger to the camera video recording circuit. For example, when the camera video recording circuit receives a surrounding person recognition trigger from the image recognition circuit, it encodes and records video for 30 seconds before and after the trigger in the storage means as surrounding person recognition trigger video.

For example, the interleaved signals from four cameras are processed to be separated into four camera images by an image recognition circuit and a camera video recording circuit. The image recognition circuit may perform processing to determine whether or not to output a trigger from one of the images from the four cameras, or may output a trigger if a condition requiring a trigger is recognized in any of the images. Alternatively, it may perform processing to determine whether or not to activate a trigger overall. It is preferable to be able to set which of these methods is to be adopted.

In particular, it is a good idea to provide a synchronization signal to the deserializer hub, camera video recording circuit, and image recognition circuit to achieve synchronization. The synchronization signal can be a pixel clock or a vertical synchronization signal. The deserializer hub can synchronize and output a clock to the four serializers. Alternatively, the deserializer hub can generate a clock and provide this clock to the camera video recording circuit and image recognition circuit to achieve synchronization. The image recognition circuit can be an FPGA instead of a SoC, or it can be any other type of computer.

The above example shows a deserializer hub with a MIPI CSI-2 branching function, but the branching device can also be external.

The function for setting the camera (and/or the other camera) (e.g., a part of the ISP (Image Signal Processor) function that includes a part for controlling the camera) may be, for example, a configuration that uses the one on the camera image recording circuit side (Pattern 1), a configuration that uses the one on the image recognition circuit side (Pattern 2), or a configuration that switches between the two (Pattern 3). Also, it may be a configuration that uses a SoC with a built-in ISP or a configuration that uses an external ISP. The external ISP may be installed on either the camera image recording circuit side. However, since the image creation required for image recognition is different from the image creation that is compressed and recorded on a recording means for playback, etc., the question arises as to how the camera should be controlled, such as the shutter speed, frame rate, and exposure time. In this regard, it is particularly preferable to use (9).

(9) The camera image recording circuit may configure the camera.

In this way, it is possible to prevent conflicts in settings, and to relatively effectively balance camera image recording with image recognition based on the camera image. For example, the inventors have found that a configuration that employs the above-mentioned (Pattern 1) produces excellent results. In many cases, the images used for image recognition are of low quality relative to the image quality required for recording. It is particularly good to have a function for recognizing images using deep learning, and in such a configuration, the requirements for the number of pixels, frame rate, etc. are generally low. It is particularly good to have the camera image recording circuit inside the camera image recording circuit.

However, with regard to the focus and exposure functions of the ISP, it may be better to set the camera to match the requirements for image processing on the image recognition circuit side. Therefore, it is a good idea to have a system in which the camera settings on the camera image recording circuit side are set in advance, taking into account the settings related to exposure and focus required on the image recognition circuit side. For example, it is a good idea to set the ISP on the camera image recording circuit side (for example, the ISP in the camera image recording circuit) in advance, taking into account the settings related to exposure and focus required on the image recognition circuit side, and adjusting them to a range where the camera image does not become too bad.

(10) The camera settings may be adjusted in a coordinated manner between the camera video recording circuit and the image recognition circuit.

This makes it easier to both compress and record camera footage and improve the accuracy of image recognition processing.

It is advisable for the image recognition circuit and the camera video recording circuit to communicate with each other and have the ability to mutually understand the current state of camera control.

For example, instead of setting the camera settings in advance, or in addition to setting them in advance, a system may be provided in which an exposure or focus adjustment request signal is generated on the image recognition circuit side and output to the camera video recording circuit, and when this is received within the camera video recording circuit, the settings are made using the setting function on the camera video recording circuit side. For example, instead of setting the camera settings in advance, or in addition to setting them in advance, a system may be provided in which an exposure or focus adjustment request signal is output to the camera video recording circuit side, and when this is received within the camera video recording circuit, the ISP settings on the camera video recording circuit side are adjusted accordingly. It is particularly preferable to configure the camera settings information set on the camera video recording circuit side to be sent to the image recognition circuit, which receives this and uses it in the image recognition processing.

For example, when recognizing signs using an image recognition circuit, the area in the image that corresponds to the shoulder of the road can be calculated as the priority area for exposure calculation. In this case, the camera is basically controlled from the camera image recording circuit, but the exposure priority of this area can be set slightly higher in advance, and if the image recognition circuit finds that the sign area is underexposed, the image recognition circuit can output an exposure correction signal to the camera image recording circuit, and when the camera image recording circuit receives this signal, it can output a signal to increase the camera exposure to the camera side.

In this way, it is advisable to configure the camera itself so that the camera video recording circuit is the main controller, and the image recognition circuit does not directly control the camera. It is also advisable to configure the image recognition circuit to output an adjustment signal to the camera video recording circuit, and the image recognition circuit that receives this adjustment signal indirectly controls the camera. For example, it is advisable to configure the camera itself so that the camera video recording circuit is the main controller, and the image recognition circuit does not directly control the camera. It is also advisable to configure it to control indirectly.

It is recommended that you set the camera to pan focus so that focus settings are not necessary.

As another example, as in the above-mentioned (Pattern 3), the camera control entity may be switched in a time-division manner. In particular, the video recording frame rate may be set to a frame rate sufficient for video recording and playback (for example, a frame rate that is not stuttering to human vision (e.g., about 24 to 30 fps)), while the camera frame rate may be set to a higher frame rate (e.g., 120 fps), and when the frames output by the camera are numbered in order as "1", "2", "3", "4", "1", "2", "3", "4", etc., "1" may be the camera video recording circuit side, and "3" the image recognition circuit side. For example, "2" and "4" may be the switching times.

For example, the switching can be done by physically switching both the ISP on the camera video recording circuit side and the ISP on the image recognition circuit side, but it is better to switch the settings of the ISP on the camera video recording circuit side in a time-division manner. It is better to have all the multiple cameras controlled by the camera video recording circuit side, and not have the image recognition circuit side directly control any of the multiple cameras. It is better to have them controlled indirectly.

The cameras that the image recognition circuit indirectly controls should be a subset of multiple cameras. These subset of cameras should be those where there is a large discrepancy between the requirements for recognition and the requirements for recording and playing back images (for humans to view the images).

You can use just one camera. For example, you can use a MIPI CSI-2 splitter to split the camera signal into two.

(11) It is preferable to provide a recognition video signal generating means for generating a signal with a reduced amount of information compared to the video signal input to the camera video recording circuit, and the image recognition circuit is configured to perform the image recognition processing based on the video signal with the reduced amount of information.

In this way, the amount of information handled by the image recognition circuit can be reduced in advance, making the recognition process faster and less expensive, while also allowing the image to be compressed with excellent image quality and recorded on the recording means.

For example, the signal from the camera may not be branched, but may be input to a camera video recording circuit, which may then reduce the amount of information by thinning out the video signal from the camera, before outputting it to the image recognition circuit from another port such as a USB port. Image thinning may be achieved, for example, by thinning out pixels or frames. In general, with image recognition, and particularly with deep learning, the amount of image information required for recognition can be significantly smaller than when a human checks an image.

In this example, the thinning is performed by the camera video recording circuit, but there is a possibility that the load on the camera video recording circuit will be high or that resources will be insufficient. In addition, it may be impossible to make changes to the camera video recording circuit in the first place. Therefore, it is a good idea to branch the video signal from the camera and input one of the branched video signals directly to the video recording circuit, while inputting the thinned video signal to the image recognition circuit. For example, this function can be configured using an FPGA. Alternatively, instead of an FPGA, a standalone ISP chip can be used and the output port switching function of the ISP chip can be used to switch the output destination for each frame.

(12) The image recognition circuit may record information based on the video signal distributed from the signal distribution means in a second recording means.

In this way, information based on the video signal distributed from the signal distribution means is recorded on the second recording means.

The second recording means may be a recording means separate from the first recording means. In particular, it may be a removable recording medium separate from the second recording means. The information based on the video signal distributed from the distribution means may be information obtained by processing the video signal distributed and input to the image recognition circuit. For example, video information that surrounds the recognized object and overlays its attributes as text information may be recorded. In addition to or instead of such video information, a trigger signal sent to a camera video recording circuit may be recorded along with the time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with the time.

The inventions shown in (1) to (12) above can be combined in any way. For example, the invention may be configured by adding at least a part of the configuration of at least one of the inventions after (2) to all or a part of the configuration of the invention shown in (1). In particular, the invention may be configured by adding at least a part of the configuration of at least one of the inventions after (2) to the invention shown in (1). In particular, the invention may be configured to have both the configuration of (2) and the configuration of (8). In addition, any configuration may be extracted from the inventions shown in (1) to (12) and the extracted configurations may be combined. The applicant of this application intends to acquire rights to the inventions including these configurations. In addition, even if there is a description such as "in the case of" or "when", this is not intended to describe the configuration as being limited to that case or time. These are examples of better configurations, and the applicant intends to acquire rights to configurations other than these cases or times. In addition, the parts described in order are not limited to this order. Configurations in which some parts have been deleted or the order has been changed are also disclosed, and the applicant intends to acquire rights.

The present invention provides a system that is superior to conventional systems.

The effects of the invention of this application are not limited to this, and effects achieved from the configuration parts disclosed in this specification and drawings, etc. are also disclosed, and it is the intention to obtain rights to the configuration that achieves said effects through divisional applications, amendments, etc. For example, in this specification, the phrase "can..." is a description that clearly indicates the effect achieved, and there are also parts that show the effect even without the phrase "can...". Also, there are effects that can be understood from the configuration even without such a description.

1 is a block diagram showing a configuration of a system according to an embodiment; FIG. 13 is a block diagram showing a modified example of the system configuration. FIG. 13 is a block diagram showing a modified example of the system configuration. FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. FIG. 2 is a diagram illustrating an example of a connection between a processing device and a recording device.

The embodiments will be described in detail below with reference to the drawings. The embodiments described below are examples of embodiments of the present disclosure, and the present disclosure is not limited to these embodiments. In the drawings referred to in the present embodiment, identical parts or parts having similar functions are given the same or similar symbols (symbols consisting of a number followed by A, B, etc.), and repeated explanations may be omitted. In addition, in each figure referred to in the following description, the scale may be different from the actual scale in order to make each component, each area, etc. recognizable.

Figure 1 is a block diagram showing the configuration of system 1 according to one embodiment. System 1 may be an on-board device mounted in a vehicle, and in particular may be an on-board device that functions as a drive recorder. The vehicle may be a forklift, but may also be a work vehicle other than a forklift, or a vehicle other than that of a worker (e.g., a private or public car, bus, or train).

Below, we first explain the transmission of video signals and the processing using the video signals in system 1, and then explain the transmission of synchronization signals and the processing using the synchronization signals.

The system 1 includes cameras 10-1 to 10-4, first video signal conversion circuits 20-1 to 20-4, second video signal conversion circuit 30, camera video recording circuit 40, image recognition circuit 50, storage medium 60, information input unit 70, and display unit 80. Hereinafter, when it is not necessary to distinguish between the cameras 10-1 to 10-4, they will be collectively referred to as "cameras 10", and when it is not necessary to distinguish between the first video signal conversion circuits 20-1 to 20-4, they will be collectively referred to as "first video signal conversion circuits 20". In this embodiment, there are four cameras 10 and four first video signal conversion circuits 20, and they handle video signals of four video channels. However, there may be three or less cameras 10 and five or more cameras 10 and five or more video channels, respectively, and they may handle video signals of three or less video channels. Each circuit of the above-described system 1 is realized, for example, by a physically separated circuit (e.g., a chip), but two or more circuits may be realized by circuits fabricated on the same substrate (e.g., the same chip).

The camera 10 is an electronic component that captures an image and generates a video signal that indicates the captured image. The camera 10 can be any type of camera, but can be a CCD camera, and is preferably a CMOS camera. The camera 10 can be a camera for capturing still images, but is preferably a camera for capturing moving images. The camera 10 can be a special camera such as an infrared camera, but is preferably a camera that captures visible light (for example, light in the wavelength range of 400 to 700 nm), and is preferably a visible light camera. The camera 10 may be a camera that obtains monochrome images, but is preferably a camera that obtains color images. The camera 10 may be, for example, a camera in which red (R), green (G), and blue (B) color filters are arranged on a plane. The resolution of the camera is preferably VGA or higher. The camera 10 may refer to, for example, a part equipped with an image sensor, but may also refer to the image sensor itself. In particular, the camera 10 is preferably a camera installed in a vehicle. In particular, it is preferable for camera 10 to be a camera that is retrofitted after the vehicle is shipped by the manufacturer.

In this embodiment, the video signal generated and output by the camera 10 is digital data including the gradation values of each of a plurality of pixels. The video signal preferably includes 8 bits (256 gradations) of data for each of the red (R), green (G), and blue (B) color components, but the color components and number of gradations are not limited to this.

Camera 10-1 is communicatively connected to the first video signal conversion circuit 20-1. Camera 10-2 is communicatively connected to the first video signal conversion circuit 20-2. Camera 10-3 is communicatively connected to the first video signal conversion circuit 20-3. Camera 10-4 is communicatively connected to the first video signal conversion circuit 20-4. Each of the cameras 10-1 to 10-4 and each of the first video signal conversion circuits 20-1 to 20-1 may be connected by either a wired or wireless transmission path, but it is preferable to connect using a wired transmission path. In the example of FIG. 1, camera 10-1 outputs video signals A1, A2, A3, and A4 in chronological order. Camera 10-2 outputs video signals B1, B2, and B3 in chronological order. Camera 10-3 outputs video signals C1, C2, C3, and C4 in chronological order. Camera 10-4 outputs video signals D1 and D2 in chronological order. Cameras 10-1 to 10-4 output video signals in ascending order of the numbers suffixed to "A" to "D". Video signals with the same number are output simultaneously to cameras 10-1 to 10-4. For example, video signals A1, B1, C1, and D1 are output simultaneously. Each of video signals A1 to A4, B1 to B3, C1 to C4, D1, and D2 is a video signal of a predetermined unit, and may be, for example, a video signal showing an image of one frame.

The cameras 10-1 to 10-4 may each capture a different area. The area captured by the camera 10 may include, for example, either the inside of the vehicle or the outside of the vehicle, or both. The area outside the vehicle may be any of the following: the front (front), right diagonally forward, right (directly to the side), right diagonally rear, rear (directly behind), left diagonally rear, left (directly to the side), and left diagonally forward, but may be other areas. The cameras 10-1 to 10-4 may, for example, include at least a camera (first camera) attached to the windshield of the vehicle, and a camera (second camera) installed to capture an image of the rear of the vehicle.

The first video signal conversion circuit 20 includes a first video signal conversion means that receives a video signal input from the camera 10 and converts it into a video signal in a serial format. The first video signal conversion circuit 20 may be a serializer, particularly a serializer chip. A serial format video signal is a signal obtained by converting the video signal input from the camera 10 into a serial data string. The first video signal conversion circuit 20-1 converts the video signal from the camera 10-1, the first video signal conversion circuit 20-2 converts the video signal from the camera 10-2, the first video signal conversion circuit 20-3 converts the video signal from the camera 10-3, and the first video signal conversion circuit 20-4 converts the video signal from the camera 10-4. The first video signal conversion circuits 20-1 to 20-4 output the interleaved converted video signals to the second video signal conversion circuit 30. Each of the first video signal conversion circuits 20-1 to 20-4 and the second video signal conversion circuit 30 may be connected either wired or wirelessly, but it is preferable to connect them using a single wired cable. The first video signal conversion circuits 20-1 to 20-4 convert the video signals into video signals for transmission via such a cable. This cable should be a relatively long-distance cable. The relatively long distance should be a distance longer than the length required for wiring within a camera unit that includes at least one of the camera 10 and another camera 10, and should particularly be one meter or more.

When the camera 10 is an in-vehicle camera, it is preferable to use a circuit capable of transmitting signals between the first video signal conversion circuit 20 and the second video signal conversion circuit 30 over a distance of several meters.

The second video signal conversion circuit 30 includes a second video signal conversion means for converting the video signals of the cameras 10-1 to 10-4 transmitted from each of the first video signal conversion circuits 20-1 to 20-4 via cables into video signals that can be input to the camera video recording circuit 40. The second video signal conversion circuit 30 may be a deserializer, particularly a deserializer chip. The second video signal conversion circuit 30 also functions as a signal distribution means for distributing the video signals transmitted between the cameras 10-1 to 10-4 and the camera video recording circuit 40 to the image recognition circuit 50. For example, the second video signal conversion circuit 30 may have a MIPI CSI-2 branching function. In this way, the video captured by the camera 10 can be compressed and stored in the storage medium 60, and the image recognition function can be realized in the image recognition circuit 50 based on the distributed signal between the camera 10 and the camera video recording circuit 40.

The second video signal conversion circuit 30 has a first port 31 and a second port 32. A camera video recording circuit 40 is communicatively connected to the first port 31, and an image recognition circuit 50 is communicatively connected to the second port 32. Such a second video signal conversion circuit 30 can also be identified as a deserializer hub.

The second video signal conversion circuit 30 may be connected to the camera video recording circuit 40 and the image recognition circuit 50 either by wire or wirelessly, but it is preferable to connect them using a wired cable. The second video signal conversion circuit 30 outputs a video signal S1 to the camera video recording circuit 40 and a video signal S2 to the image recognition circuit 50 based on the interleaved video signals from the cameras 10-1 to 10-4. The second video signal conversion circuit 30 distributes the video signals from the cameras 10-1 to 10-4 by outputting the same image signals S1 and S2 from the first port 31 and the second port 32 at the same timing. The second video signal conversion circuit 30, for example, inputs a video signal of a predetermined protocol and distributes and outputs the signal to multiple output destinations using the same protocol as the input protocol. For example, if the camera 10 outputs a MIPI CSI-2 signal as a video signal, the second video signal conversion circuit 30 may be a MIPI CSI-2 signal splitter chip. The second video signal conversion circuit 30 may, for example, input a video signal of a predetermined protocol and output signals of the same protocol to multiple output destinations that are different from the input protocol. For example, in the case of a camera 10 that outputs an FPD-LINK III signal as a video signal, the second video signal conversion circuit 30 may be a chip equipped with a port that inputs an FPD-LINK III signal and outputs it as, for example, multiple MIPI CSI-2 signals (for example, a dual port output deserializer hub with one port input and two ports output). In particular, the port that outputs the MIPI CSI-2 signal may have a function of outputting a replica of one port to another port. In particular, it may be a deserializer hub chip.

In system 1, a single cable should have as few signal lines as possible, for example a twisted pair cable. A transmission path should have as few signal lines as possible, for example a few signal lines (for example a printed wiring on a board). A connection between a camera 10 and multiple cameras 10, including another camera 10, and the first video signal conversion circuit 20 should be made via at least one of a printed wiring or a flexible cable. A connection between the second video signal conversion circuit 30 and the camera video recording circuit 40 should be made via at least one of a printed wiring or a flexible cable.

The second video signal conversion circuit 30 selects the video signals of the cameras 10-1, 10-2, 10-3, and 10-4 in a predetermined order, and outputs the selected video signals from each of the first port 31 and the second port 32. Here, in the example of FIG. 1, the second video signal conversion circuit 30 outputs the video signals S1 and S2 by switching the video signals on a frame-by-frame basis in the order of the cameras 10-1, 10-2, 10-3, 10-4, 10-1, 10-2, 10-3, 10-4, etc. Specifically, the second video signal conversion circuit 30 outputs the video signals S1 and S2 from the first port 31 and the second port 32 in the order of the video signals A1, B1, C1, D1, A2, B2, C2, D2, A3, B3, C3, A4, and C4. In addition, the 30th does not need to have the function of distributing the video signal (MIPI CSI-2 branching function), in which case an external distributor may be used.

The camera video recording circuit 40 has a function of compressing the video signal S1 from the first port 31 and storing it in the storage medium 60. The video compression section (not shown) in the camera video recording circuit 40 can use various algorithms, but it is preferable to use an encoder with a standardized algorithm, and it is particularly preferable to have at least one of an H.264 encoder or an H.265 encoder. In particular, the video encoder in the camera video recording circuit 40 is preferably configured with hardware rather than software processing. The camera video recording circuit 40 is preferably an FPGA, and more preferably an SoC. In addition, it is preferable that the camera video recording circuit 40 is configured with a hard macro rather than a soft macro.

It is particularly preferable that the camera video recording circuit 40 is sold as a dedicated chip for camera video recording. The camera video recording circuit 40 may be, for example, an ASSP for camera video recording, an LSI for camera video recording such as an SoC for camera video recording, and in particular, the camera video recording circuit may be an SoC for a drive recorder.

When the trigger signal TG is input from the image recognition circuit 50, the camera video recording circuit 40 performs event recording by storing the video signal for a period corresponding to the input in the storage medium 60. This period may be, for example, 30 seconds before and after the event occurs, but may be another period.

The image recognition circuit 50 functions as an electronic component that realizes a predetermined function using the signal distributed by the second video signal conversion circuit 30 and transmitted between the camera 10 and the camera video recording circuit 40. The image recognition circuit 50 recognizes the image indicated by the video signal S2 from the second port 32 and determines whether an event that triggers event recording has occurred. If the image recognition circuit 50 determines that the event has occurred, it outputs a trigger signal TG that triggers the start of event recording to the camera video recording circuit 40. For example, the image recognition circuit 50 recognizes the state of an object in the image, and if the result of the recognition indicates that event recording is necessary, it outputs the trigger signal TG to the camera video recording circuit 40. In this way, the image recognition circuit 50 can comprehensively determine whether to generate a trigger signal TG from the images of the four cameras 10-1 to 10-4.

In this way, the image recognition circuit 50 outputs information about the results of the image recognition process performed based on the video signal S2 to the camera video recording circuit 40. The image recognition circuit 50 may output information indicating the position of a recognized object in the video to the camera video recording circuit 40. The camera video recording circuit 40 compresses the video represented by the video signal S1 based on the information about the results of the image recognition process and stores it in the storage medium 60. In this way, it is possible to reliably compress the video from the camera 10 and store it in the storage medium 60, and it is also possible to record based on the results of the image recognition process of the same video from the camera 10.

Here, an example of event recording will be described. For example, when the image recognition circuit 50 recognizes a no-entry sign based on the image, but detects an intrusion beyond the sign, the image recognition circuit 50 outputs a trigger signal TG1 indicating a no-entry trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG1, it encodes and records the video signal showing the video for 15 seconds before and after the trigger signal TG1 as a no-entry trigger video in the storage medium 60. For example, when the image recognition circuit 50 recognizes that a person has intruded into the vicinity of the forklift, it outputs a trigger signal TG2 indicating a surrounding person recognition trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG2, it encodes and records the video signal showing the video for 30 seconds before and after the trigger signal TG2 as a surrounding person recognition trigger video in the storage medium 60. Here, only the video of the video channel that triggered the output of the trigger signal TG1 or TG2 may be recorded, or the video of all the video channels may be recorded. Note that various variations are possible regarding the event that triggers the output of a trigger signal and the period that is the subject of event recording. For example, event recording may be performed when information input via the information input unit 70 satisfies a predetermined condition.

For example, a MIPI CSI-2 splitter (e.g., a splitter chip) is used to distribute and input the same camera video signal to the camera video recording circuit 40 and the image recognition circuit 50. The camera video recording circuit 40 and the image recognition circuit 50 are connected by a communication line, and the image recognition circuit 50 performs image recognition of the state of the target object, and if the result of the recognition indicates that recording is necessary, a trigger signal is passed to the camera video recording circuit via this communication line. When the camera video recording circuit receives a trigger signal from the image recognition circuit, it is preferable to event-record the video signal it has received in a storage means.

The communication line may be a parallel communication line or a serial communication line to transmit the trigger signal TG and other data. For example, information on the type of recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs. For example, the GPIO output port of the image recognition SoC is connected to the trigger input port of the drive recorder SoC, and when the result of image recognition by the image recognition SoC is that recording is required, the signal of the GPIO output port of the image recognition SoC is changed, and when there is a change in the signal of the trigger input port of the drive recorder SoC, the drive recorder SoC records the image of a predetermined number of seconds before and after the trigger occurrence in the recording means. Furthermore, signals from sensors connected to the camera image recording circuit 40 may be transmitted to the image recognition circuit 50. For example, if the camera video recording circuit 40 is a SoC for a drive recorder and the image recognition circuit 50 is a SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the information received by the SoC for the drive recorder from the acceleration sensor, switch, and GPS module is output in real time to the SoC for image recognition. The SoC for image recognition may receive this information in real time from the SoC for the drive recorder, use it for image recognition, and output the results to the SoC for the drive recorder. The functions of both SoCs may be realized by executing programs for realizing these functions stored in each SoC.

The image recognition circuit 50 may record information based on the video signal distributed from the second video signal conversion circuit 30 in the second recording means. In this way, information based on the video signal distributed from the second video signal conversion circuit 30 is recorded in the 22nd recording means. The second recording means may be a recording means different from the first recording means. In particular, it may be a removable recording medium different from the first recording means. If the first recording means is a storage medium 60, the second recording means is a recording means different from the storage medium 60. The information based on the distributed video signal may be information obtained by processing the video signal distributed and input to the image recognition circuit 50. For example, it may record video information that surrounds the recognized object and superimposes its attributes as text information. In addition to such video information, or instead of such video information, for example, the trigger signal TG output to the camera video recording circuit 40 may be recorded together with the time. For example, information on the coordinates and type of the recognized object may be recorded in association with the time.

The camera video recording circuit 40 may have a function for continuous recording in addition to the function for event recording. In this case, the camera video recording circuit 40 records video in a specified memory area secured in the storage medium 60, and when the memory area becomes full, records video by overwriting the oldest video first. To prevent the loss of video related to event recording due to the continuous recording function, these memory areas may be separated, or overwriting may be prohibited using an overwrite prohibition flag.

The storage medium 60 functions as a storage means for storing the images recorded by the camera image recording circuit 40. The storage medium 60 may be either an internal storage means of the system 1 or an external storage means, but is preferably a removable external storage means, and in particular a card-shaped medium such as an SD card or a microSD card. The images stored in the storage medium 60 can be displayed on the display unit 80 or using an external terminal. The display unit 80 is, for example, a liquid crystal display or some other type of display device.

The information input unit 70 accepts input of information from the outside. The information input unit 70 may include various switches including, for example, a switch for turning on or off the power of the system 1. The information input unit 70 may include a sensor for a drive recorder. The information input unit 70 may include, for example, an acceleration sensor (functioning as a G sensor) for detecting an impact that triggers event recording. The information input unit 70 may also include a GPS (Global Positioning System) sensor for measuring the current position. Various signals according to the information input via the information input unit 70 may be exchanged between the camera video recording circuit 40 and the image recognition circuit 50.

Next, the transmission of the synchronization signal and the processing using the synchronization signal will be described. The signal transmitted between the camera 10 and the camera video recording circuit 40 includes a video synchronization signal generated by the synchronization signal generating means. The system 1 is preferably configured to output the video signals of the cameras 10-1 to 10-4 to the camera video recording circuit 40 based on the synchronization signal distributed together with the video signals.

In this way, the camera video recording circuit 40 can obtain video signals synchronized between the cameras 10-1 to 10-4. For example, the camera video recording circuit 40 can input video signals synchronized between the cameras 10-1 to 10-4, and compress and record video based on the input synchronized video signals. For example, when the recorded video signals are played back simultaneously, the synchronized videos can be viewed. Note that the video may be compressed into separate files for each of the multiple cameras, but it is preferable to compress the video into a single file.

In the system 1, in order to synchronize the operations of the multiple cameras 10, the first video signal conversion circuits 20-1 to 20-4, the camera video recording circuit 40, and the image recognition circuit 50, the synchronization signal is also distributed to the image recognition circuit 50. In this way, the signal transmitted between the camera 10 and the camera video recording circuit 40 may be a control signal, but it is preferable to have at least one of a video signal or a video synchronization signal. The synchronization signal may be at least one of a pixel clock, a vertical synchronization signal, and a horizontal synchronization signal, and it is preferable to have at least two of them. It is preferable to have a pixel clock and a vertical synchronization signal. The second video signal conversion circuit 30 may output a clock to the four first video signal conversion circuits 20-1 to 20-4 in synchronization with each other. The following (Method 1) or (Method 2) may be adopted as the generator and distribution method of the synchronization signal.

(Method 1) A synchronization signal is generated with one camera 10 as the parent, and distributed to another camera 10, the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

In (Method 1), one of the cameras 10 has a synchronization signal generating means. Then, that one camera 10, acting as a signal distribution means, distributes this synchronization signal to another camera 10, the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

In this way, the video signal output from the other camera 10 becomes a video signal based on the synchronization signal generated within the first camera 10, making it easier to synchronize these cameras 10. In particular, it is advisable to adopt this configuration and have the first camera 10 and the other camera 10 housed in the same housing. This is particularly effective when the synchronization signal is routed a short distance. In the camera 10, it is particularly advisable to have the signal generated within the imaging element. In this way, there is no need to use an imaging element capable of external synchronization that captures images in synchronization with the input synchronization signal, and a low-cost and simple configuration can be achieved.

(Method 2) Either the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, or the image recognition circuit 50 generates a synchronization signal and distributes it to each of the other elements.

In (Method 2), a synchronization signal generating means is provided outside the camera 10, and the synchronization signal generated by this synchronization signal generating means is distributed to the cameras 10-1 to 10-4 and other elements.

(Method 2) makes it easier to synchronize the video signal output from the other camera 10 with the video signal output from the other camera 10. In particular, it is advisable to adopt this configuration and have the camera 10 and the other camera 10 housed in different housings. This is particularly effective when the synchronization signal has to be routed over a relatively long distance.

In (Method 2), the camera video recording circuit 40 may have a synchronization signal generating means. In this case, the camera video recording circuit 40, which serves as a signal distribution means, distributes the synchronization signal to the cameras 10-1 to 10-4 and other elements. In this way, it is possible to increase the possibility of obtaining synchronized video signals from the camera and another camera at a timing that matches the video recording process, etc.

In (Method 2), a synchronization signal generating means may be provided in either the first video signal conversion circuit 20 or the second video signal conversion circuit 30. In this case, the first video signal conversion circuit 20 or the second video signal conversion circuit 30, which serves as a signal distribution means, distributes the synchronization signal to the cameras 10-1 to 10-4 and other elements. In this way, it is easy to input the video signals synchronized between the camera and the other camera to the camera video recording circuit via a cable and record them.

In particular, a synchronization signal may be provided to the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 to achieve synchronization. The synchronization signal may be a pixel clock or a vertical synchronization signal. The second video signal conversion circuit 30 may synchronize and output a clock to the four first video signal conversion circuits 20-1 to 20-4. Alternatively, a clock may be generated in the second video signal conversion circuit 30, and this clock may be provided to the camera video recording circuit 40 and the image recognition circuit 50 to achieve synchronization. The image recognition circuit 50 may be an FPGA instead of a SoC, or may be any other type of computer.

The method of processing based on the synchronization signal is not limited to either (Method 1) or (Method 2), and for example, a synchronization signal generating means may be provided separately from the elements of the system 1 described above.

The above signal distribution means may be, for example, a simple wiring branch, but it is preferable that the means has a function of distributing signals so that signals transmitted between the multiple distributed signal lines can be synchronized, and it is preferable that the means is a circuit (for example, a chip) having a function of distributing signals so that signals transmitted between the multiple distributed signal lines can be synchronized.

Although the camera 10 and the camera video recording circuit 40 may be configured to be housed in the same housing, it is preferable to configure them to be housed in separate housings and to connect the two housings with a connecting means such as a cable or connector. It is preferable to configure the camera 10 and the camera video recording circuit 40 to be connected via a cable of 1 meter or more. In particular, it is preferable to connect the camera 10 and the camera video recording circuit 40 via a cable that is routed inside the vehicle.

The configuration described in FIG. 1 may be modified as shown in FIG. 2. In FIG. 2, the configuration described in FIG. 1 is mainly shown with different parts from FIG. 1. In this example, the video signal from the camera 10 is input to the camera video recording circuit 40 without being distributed. The camera video recording circuit 40 supplies the video signals from the cameras 10-1 to 10-4 to the image recognition circuit 50. The camera video recording circuit 40 has a recognition video signal generating means for generating a signal with a reduced amount of information compared to the input video signal, and reduces the amount of information by thinning the video signal, and outputs it to the image recognition circuit 50 via another port such as a USB (Universal Serial Bus) port. The video signal may be thinned out by, for example, thinning out pixels or thinning out frames. This is because the amount of information required for image recognition in general image recognition, particularly in the case of deep learning, is significantly smaller than that required for humans to check the video.

The configuration described in FIG. 2 may be further modified as shown in FIG. 3. In FIG. 3, the configuration described in FIG. 1 is shown with a focus on the parts different from FIG. 1. In the example of FIG. 2, the thinning is performed by the camera video recording circuit 40, but this may increase the processing load or cause a resource shortage. Therefore, as shown in FIG. 3, an image processing unit 90 may be provided between the camera 10 and the camera video recording circuit 40 and the image recognition circuit 50. The image processing unit 90 branches the video signal from the camera, outputs the video signal as is to the camera video recording circuit 40, and outputs the thinned video signal to the image recognition circuit 50. The image processing unit 90 is, for example, an FPGA, but may also be configured using a single ISP chip. The image processing unit 90 may be configured to switch the output destination for each frame using the output port switching function of the ISP chip.

In system 1, there may be only one camera 10. For example, the camera signal may be split into two using a splitter such as a MIPI CSI-2 splitter. In this case, it is equivalent to a configuration in which one of the cameras 10-1 to 10-4 described in FIG. 3 is used, and the image processing unit 90 is used as a splitter.

In system 1, the video signal transmitting the video from camera 10 may be transmitted in serial or parallel, for example, with the video synchronized on a video channel basis. Or, it may be transmitted in a combination of serial and parallel. In particular, it is advisable to overlay data such as the current time, event information, sensor information, etc. on a video frame basis, or to overlay it on the video. Data such as the current time, event information, sensor information, etc. may be overlaid on the synchronized units, or may be overlaid on the video.

It is also advisable to make the lengths of the signal wiring from the signal distribution means to the camera 10 and to the other camera 10 approximately the same. This increases the likelihood that the video signal from the camera 10 and the video signal from the other camera 10 will be synchronized.

Next, referring to Figs. 4 to 7, an example of a connection between the cameras 10-1 to 10-4 and the processing device 100 will be described. The processing device 100 is a device that performs predetermined processing based on the video signals output by the cameras 10-1 to 10-4. The processing device 100 may be, for example, either the camera video recording circuit 40 or the image recognition circuit 50.

In the example of FIG. 4, cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs a video signal Sa1 to the second video signal conversion circuit 30 via a single cable. The first video signal conversion circuit 20 outputs a video signal Sa2, which is the same as the video signal Sa1, to the processing device 100 via a single cable. The video signal Sa2 is a signal that retains the serial format of the video signal Sa1. The processing device 100 performs processing using the video signal Sa2 input via the single cable.

In the example of FIG. 5, multiple cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs video signal Sb1 to the second video signal conversion circuit 30 via one cable. Video signal Sb1 is the same signal as video signal Sa1. The second video signal conversion circuit 30 outputs video signals Sb21, Sb22, Sb23, and Sb24 using different cables for each of the cameras 10-1 to 10-4. The video signals Sb21 to Sb24 are signals obtained by converting the video signal Sb1, which is in serial format, into parallel format. The processing device 100 performs processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

In the example of FIG. 6, multiple cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. Each of the first video signal conversion circuits 20-1 to 20-4 outputs video signals Sc11, Sc12, Sc13, and Sc14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs video signal Sc2 to the processing device 100 via a single cable. The processing device 100 performs processing using the video signal Sc2 input via the cable. The video signal Sc2 is a signal obtained by converting the parallel video signals Sc1 to Sc4 into a serial format. The system 1 in FIG. 1 corresponds to this connection example.

In the example of FIG. 7, a single camera unit is provided with multiple cameras 10-1 to 10-4, which are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. The system 1 in FIG. 1 corresponds to this connection example. Each of the first video signal conversion circuits 20-1 to 20-4 outputs video signals Sd11, Sd12, Sd13, and Sd14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs video signals Sd21, Sd22, Sd23, and Sd24 using different cables for each of the cameras 10-1 to 10-4. The video signals Sd21 to Sd24 are parallel signals that are the parallel versions of the video signals Sd11 to Sd14. The processing device 100 executes processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

1 and 6 and 7, a system may be configured that includes a first camera unit having a camera 10 and a first video signal conversion circuit 20, a second camera unit having another camera 10 and a second first video signal conversion circuit 20, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40, with a first cable connecting the first camera unit and the recording unit, and a second cable connecting the second camera unit and the recording unit, and transmitting the respective video signals. For example, a first camera unit is installed so that the camera (first camera) attached to the windshield of the vehicle captures the image of the front of the vehicle, and a second camera unit is installed so that the other camera (second camera) is installed to the rear windshield of the vehicle captures the image of the rear of the vehicle, and a recording unit installed on the dashboard or under the passenger seat is connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

For example, a first camera unit having a camera 10 and a first first video signal conversion circuit 20 may be connected to a second camera unit having another camera 10 and a second first video signal conversion circuit 20 by a first cable, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40 may be connected to the second camera unit by a second cable, and the respective video signals may be transmitted.

Based on the configurations of Figures 1, 4 and 5, a system may be configured in which one camera unit includes a camera 10 and a plurality of cameras 10, including another camera 10, and a first video signal conversion circuit 20, and the first video signal conversion circuit 20 converts the video signals of the plurality of cameras 10 to flow through a single cable, and transmits the signals to a recording unit including a second video signal conversion circuit 30 and a camera video recording circuit 40 via the single cable, thereby recording the images of the plurality of cameras 10 on a storage medium 60. For example, a camera unit that is attached to the windshield of a vehicle and includes a camera 10 (first camera) that captures an image in front of the vehicle and another camera 10 (second camera) that captures an image behind the vehicle may be connected to a recording unit installed on the dashboard or under the passenger seat, etc., via a cable.

For example, the second video signal conversion circuit 30 may convert the video signals of the multiple cameras 10 that have been converted by the first video signal conversion circuit 20 and transmitted over a single cable over a relatively long distance into signals for transmission over a relatively short distance, which is necessary for wiring within the recording unit.

Based on the configurations of Figures 1, 4 and 5, for example, the second video signal conversion circuit 30 may be configured to output video signals from multiple cameras 10 to the camera video recording circuit 40 so that they are sent over a single cable, which is then input to the camera video recording circuit 40. For example, the second video signal conversion circuit 30 may divide the video signals from the two cameras into packets of a predetermined data size and transmit the packets alternately. Video signals from a larger number of cameras may be output in an interleaved manner over a single transmission path.

Alternatively, based on the configurations of Figures 1, 6 and 7, for example, the second video signal conversion circuit 30 may be configured to output video signals from multiple cameras 10 to the camera video recording circuit 40 so that they are sent over different cables for each camera, and these are input to the camera video recording circuit 40. In this way, processing can be easily performed for each camera 10 in the different transmission paths for each camera 10 or in the camera video recording circuit 40 connected to these. For example, it becomes relatively easy to record and output information separate from the video by superimposing it on each camera frame for each camera 10.

The camera video recording circuit 40 is relatively inexpensive, but is designed specifically for compressing and storing video, which makes it difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.). For example, SoCs for drive recorders are inexpensive, but are designed specifically for drive recorders, which makes it difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.).

On the other hand, there is a problem that the image recognition circuit 50 used for image recognition does not have a highly reliable software for compressing and recording images. The image recognition circuit may be a general-purpose SoC (SoC for image recognition) that can be used by compiling, for example, OSS. In particular, it may be an SoC equipped with an accelerator circuit for image recognition or an SoC equipped with an accelerator circuit for deep learning. There is also a problem that it is costly to mount an image encoder such as H.264 on a general SoC or, in particular, an image recognition circuit that is an FPGA. In addition, even if a general SoC has an image encoder, there is a problem that there is a lack of reliable software for drive recorders, especially when it is used as a drive recorder, and there is also a problem that it is extremely costly to create it from scratch in-house.

In contrast, in the system of this embodiment, the camera video recording circuit 40 and the image recognition circuit 50 are implemented as different circuits such as SoCs, and the video signal and synchronization signal are distributed as described above, providing a system that is superior to conventional systems.

In addition, the following configuration may be adopted:

(Camera control example 1)
The function for setting the camera 10 (and/or another camera 10) (e.g., a part of the ISP's functions that includes a part for controlling the camera) may be, for example, a configuration using the one on the camera video recording circuit 40 side (Pattern 1), a configuration using the one on the image recognition circuit 50 side (Pattern 2), or a configuration using both sides by switching (Pattern 3). Also, it may be a configuration using either a SoC built into the ISP or a configuration using an external ISP. The external ISP may be installed on either the camera video recording circuit 40 side. However, since the image creation required for image recognition is different from the image creation that is compressed and recorded on the storage medium 60 and can be played back, it is a problem as to what state the camera should be controlled in, such as the shutter speed, frame rate, and exposure time. Therefore, the camera video recording circuit 40 may have a function for setting the camera 10.

In this way, it is possible to prevent conflicts in settings, and to relatively effectively balance the recording of the image from the camera 10 with image recognition based on the image from the camera 10. For example, the inventors have found that a configuration that adopts the above-mentioned (Pattern 1) produces excellent results. In many cases, the images used for image recognition are of low quality relative to the image quality required for recording. It is particularly good to have a function for recognizing images through deep learning, and in such a configuration, the requirements for the number of pixels, frame rate, etc. are generally low. On the camera image recording circuit 40 side, it is particularly good to have it within the camera image recording circuit 40.

However, with regard to the focus and exposure functions of the ISP, it may be better to set the camera on the image recognition circuit 50 side according to the requirements for image processing. Therefore, it is a good idea to have a system in which the camera settings on the camera image recording circuit 40 side are set in advance, taking into account the settings related to exposure and focus required on the image recognition circuit 50 side. For example, it is a good idea to set the ISP on the camera image recording circuit 40 side (for example, the ISP within the camera image recording circuit 40) in advance, taking into account the settings related to exposure and focus required on the image recognition circuit 50 side, and adjusting the camera image to a level that does not make it too bad.

(Camera control example 2)
The camera settings are adjusted in a coordinated manner between the camera image recording circuit 40 and the image recognition circuit 50. In this way, it becomes easier to achieve both the compression and recording of the camera image and the improvement of the accuracy of the image recognition process.

As shown in Figures 8(a) and (b), it is preferable that the image recognition circuit 50 and the camera image recording circuit 40 communicate with each other to mutually understand the state of the control of the camera 10.

For example, instead of setting the camera 10 in advance, or in addition to setting it in advance, the image recognition circuit 50 generates an exposure or focus adjustment request signal and outputs it to the camera video recording circuit 40. When the camera video recording circuit 40 receives this, the settings are made using the setting function on the camera video recording circuit 40 side. For example, instead of setting the camera settings in advance, or in addition to setting it in advance, the image recognition circuit 50 outputs an exposure or focus adjustment request signal to the camera video recording circuit 40. When the camera video recording circuit 40 receives this, it is preferable for the ISP settings on the camera video recording circuit 40 side to be adjusted accordingly. Note that information regarding the camera settings set on the camera video recording circuit 40 side is sent to the image recognition circuit 50. It is particularly preferable for the image recognition circuit 50 to receive this and use it in the image recognition processing.

For example, when recognizing a sign using the image recognition circuit 50, the area in the image that corresponds to the shoulder of the road can be calculated as the priority area for exposure calculation. In this case, the camera is basically controlled by the camera image recording circuit 40, but the exposure priority of this area can be set slightly higher in advance, and if the image recognition circuit 50 finds that the sign area is underexposed, the image recognition circuit 50 can output an exposure correction signal to the camera image recording circuit 40, and when the camera image recording circuit 40 receives this signal, it can output a signal to increase the camera exposure to the camera side.

In this way, it is preferable to configure the camera 10 itself so that the camera video recording circuit 40 is the main controller, and the image recognition circuit 50 does not directly control the camera. It is also preferable to configure the image recognition circuit 50 to output an adjustment signal to the camera video recording circuit 40, and the image recognition circuit 50 that receives this adjustment signal indirectly controls the camera. For example, it is preferable to configure the camera 10 itself so that the camera video recording circuit 40 is the main controller, and the image recognition circuit 50 does not directly control the camera. It is also preferable to configure it to control indirectly.

In addition, it is recommended that you use a pan focus camera to eliminate the need for focus settings.

As another example, as in the above-mentioned (Pattern 3), the camera control entity may be switched in a time-division manner. In particular, the video recording frame rate may be set to a frame rate sufficient for recording and playing back the video (for example, a frame rate that does not stutter to the human eye (for example, about 24 to 30 fps)), while the camera frame rate may be set to a higher frame rate (for example, 120 fps), and when the frames output by the camera are numbered in order as "1", "2", "3", "4", "1", "2", "3", "4", etc., "1" may be the camera video recording circuit 40 side and "3" the image recognition circuit 50 side. For example, "2" and "4" may be the switching time.

For example, the switching may be done by physically switching both the ISP on the camera video recording circuit 40 side and the ISP on the image recognition circuit 50 side, but it is preferable to switch the ISP settings on the camera video recording circuit 40 side in a time-division manner. It is preferable that the camera video recording circuit 40 side controls all of the multiple cameras 1-0, and that the image recognition circuit 50 side does not directly control any of the multiple cameras. It is preferable to control them indirectly.

The cameras that are indirectly controlled by the image recognition circuit 50 may be a subset of multiple cameras. These subset of cameras may be those for which there is a large discrepancy between the requirements for recognition and the requirements for recording and playing back images (for humans to view images).

In the configuration described in FIG. 8, the image recognition circuit 50 may be replaced with a processing device that performs processing other than image recognition processing.

The functions of system 1 described in the above-mentioned embodiment may be realized by one or more programs and by the cooperation of one or more hardware resources. When the functions of system 1 are realized by using a program, the program for realizing the functions may be provided in a state stored in a computer-readable recording medium such as various magnetic recording media, optical recording media, magneto-optical recording media, semiconductor memory, etc. Furthermore, the program may be distributed via a network. The present invention may also be understood as a signal processing method.

The scope of the present invention is not limited to the configurations expressly described in the specification, but includes combinations of the various aspects of the invention disclosed herein. The configurations of the invention that are sought to be patented are specified in the accompanying claims, but it is the intention to claim configurations disclosed in this specification in the future, even if they are not currently specified in the claims.

The present invention is not limited to the configurations described in the above-mentioned embodiments. The components of each of the above-mentioned embodiments and variations may be arbitrarily selected and combined. Any of the components of each of the embodiments and variations may be arbitrarily combined with any of the components described in the means for solving the invention or any of the components that embody any of the components described in the means for solving the invention. We intend to acquire rights to these as well through amendments to this application or divisional applications, etc. Furthermore, even if there is a description such as "in the case of" or "when", this is not intended to describe a configuration that is limited to that case or time. We also disclose configurations that are not in these cases or times, and we intend to acquire rights to them. Furthermore, the parts that are described in order are not limited to this order. We also disclose configurations in which some parts are deleted or the order is changed, and we intend to acquire rights to them.

In addition, we intend to obtain rights to the overall design or partial design by filing a conversion application for a design application. The drawings show the entire device in solid lines, but they include not only the overall design but also partial designs claimed for some parts of the device. For example, not only can some components of the device be made into partial designs, but the drawings also include some parts of the device as partial designs regardless of the components. A part of the device may be a part of the device, or it may be part of that component. We intend to obtain rights to not only the overall design, but also partial designs in which any part of the solid line parts of the drawings is made into dashed lines.

1: System 1, 10-1 to 10-4: Cameras, 20, 20-1 to 20-4: Second video signal conversion circuit, 30: Deserializer, 31: First port, 32: Second port, 50: Image recognition circuit, 60: Storage medium, 70: Information input unit, 80: Display unit, 90: Image processing unit, 100: Processing device

Claims (2)

A system including a plurality of cameras, an SoC for image recognition, and an SoC for a drive recorder,
the SoC for the drive recorder receives an input of a video signal indicating an image captured by each of the plurality of cameras, converts the video signal, outputs information obtained by converting the video signal to the SoC for image recognition, and outputs predetermined information received from at least one of an acceleration sensor, a switch, and a GPS module in real time to the SoC for image recognition;
the SoC for image recognition receives information obtained by converting the video signal from the SoC for the drive recorder, receives the predetermined information in real time from the SoC for the drive recorder, performs a predetermined image recognition process using the information obtained by converting the video signal and the predetermined information, and outputs information regarding a result of the image recognition process to the SoC for the drive recorder;
The SoC for the drive recorder records video based on the video signals output from the multiple cameras based on the information about the results. The plurality of cameras includes a first camera mounted on a windshield of the vehicle and a second camera mounted on a rear windshield of the vehicle.
The system of claim 1 .

Images