JP7717359B2 - Electronic equipment, a server that converts still images into animation, and programs - Google Patents

Description

The present invention relates to an electronic device, a server that animates still images sent from the electronic device, and a program that enables a computer to implement the functions of the electronic device and server.

In recent years, in fields such as IoT (Internet of Things), low-power wide area networks (LPWA) known as LPWA (Low Power Wide Area) or LPWAN (Low Power Wide Area Network) have come into use as communication functions. LPWA uses relatively low frequencies, enabling communication over relatively long distances with relatively little power.

JP 2020-088803 A (Patent Document 1) is a background technology in this technical field. This publication states, "A remote monitoring control device includes a sensor, a low-power wide-area communication transmitter that transmits sensor data, a low-power wide-area communication receiver that receives the sensor data, and a remote monitoring control device that receives the sensor data from the receiver, wherein the remote monitoring control device associates the sensor data with sensor data acquisition information and stores them in a sensor data storage unit, determines a specific state based on the sensor data stored in the sensor data storage unit, and acquires from a video file storage unit a video file corresponding to the sensor data acquisition location indicated by the sensor data acquisition information associated with the sensor data determined to be in the specific state" (see Abstract).

JP 2019-129328 A (Patent Document 2) is a background technology in this technical field. This publication states, "The high-definition video generation device 20 includes an original video receiving unit 21 that receives an original video 1, which is a high-definition video; a low-bit-rate video receiving unit 24 that receives a low-bit-rate video 2; a learning model generation unit 22 that generates a learning model M using the original video 1 received by the original video receiving unit 21 as training data; and a high-definition video generation unit 25 that generates a high-definition video 3 from the low-bit-rate video 2 received by the low-bit-rate video receiving unit 24 using the learning model M corresponding to the low-bit-rate video 2 from the learning model M generated by the learning model generation unit 22" (see Abstract).

Japanese Patent Application Laid-Open No. 2020-088803 Japanese Patent Application Laid-Open No. 2019-129328

The present invention provides feasible and effective electronic devices, servers, programs, etc. for transmitting video information and creating videos.

The applicant also intends to obtain rights to configurations that achieve the effects achieved by the components disclosed in this specification and drawings, etc., through divisional applications, amendments, etc. For example, this specification discloses problems in which phrases such as "can" and "is possible" are read as "the problem is." Each problem is described as an independent entity, and the applicant intends to obtain rights to the configurations that solve each problem separately through divisional applications, amendments, etc. Even if the problems are implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in this specification within the scope of the patent claim through amendments or divisional applications. Furthermore, configurations that solve problems that combine these independent problems have also been disclosed, and the applicant intends to obtain rights to them.

In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problem, but one example is an electronic device that includes a communications processing unit and a control unit that processes image information captured by a camera and transmits multiple still images via the communications processing unit, and is characterized in that the control unit controls the transmission of the multiple still images based on at least one of the vehicle's running state and the transmission state of the still images.

The present invention provides feasible and effective electronic devices, servers, programs, etc. for transmitting video information and creating videos.

Problems, configurations, and effects other than those described above will become clear from the description of the embodiments below. The effects of the present invention are not limited to these; the effects achieved by the configurations disclosed in this specification and drawings are also disclosed, and it is the intention to obtain rights to the configurations that achieve these effects through divisional applications, amendments, etc. For example, in this specification, phrases such as "can" and "is possible" are descriptions that clearly state the effects achieved, and there are also parts that demonstrate effects even without phrases such as "can" and "is possible." Furthermore, there are effects that can be understood from the configuration even without such phrases.

FIG. 1 shows an example of the overall configuration of a DVR management system 1. FIG. 2 shows an example of the connection configuration of the management server 102. FIG. 3 shows an example of the hardware configuration of the drive recorder 101. FIG. 4 shows an example of the hardware configuration of the management server 102. FIG. 5 shows an example of the hardware configuration of the user terminal 103. FIG. 6 shows an example of the data structure of DVR status information 600. FIG. 7 is a conceptual diagram showing a case where a video stream is transmitted to a management server during actual driving. FIG. 8 is a conceptual diagram showing a case where a plurality of still images are transmitted to the management server during actual driving. FIG. 9 is a graph showing changes in driving speed, reception strength (RSRP), transmission speed, and reception speed during actual driving. FIG. 10 shows an example of the data structure when vehicle information and some of a plurality of still images are transmitted to the management server during actual driving. 11 shows examples of still images taken during actual driving of No. 1 to No. 8 in FIG. 12 shows examples of still images taken during actual driving of No. 9 to No. 16 in FIG. 13 shows examples of still images taken during actual running of No. 17 to No. 24 in FIG. 10. Fig. 14 shows examples of still images No. 15 to No. 18 in Fig. 10 taken during actual driving and still images extracted from them. Fig. 15 shows examples of still images No. 19 to No. 22 in Fig. 10 taken during actual driving and still images extracted from them. FIG. 16 shows an example of the data structure when ping, vehicle information, and some of a plurality of still images are transmitted to the management server during actual driving. FIG. 17, following FIG. 16, shows an example of the data structure when ping, vehicle information, and some of a plurality of still images are transmitted to the management server during actual driving. FIG. 18 shows an example of the transmission speed of a set of ping, vehicle information, and a still image when the transmission state is good. FIG. 19 shows an example of the transmission speed of a set of ping, vehicle information, and still image when the transmission state deteriorates. FIG. 20 shows an example of the transmission speed of a set of ping, vehicle information, and still image, in the state of FIG. 19, where control is performed to improve the transmission state of the still image in particular. FIG. 21 is a conceptual diagram showing an example of image analysis performed on a still image in (A) and (B).

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment shown below is one embodiment of the present invention, and the content of the present invention should not be interpreted as being limited based on the following description.
1 shows an example of the overall configuration of a drive recorder (DVR) management system 1. The DVR management system 1 includes a drive recorder 101, a management server 102, a user terminal 103, a data center 104, and a management terminal 105, all of which are connected via a network. The network may be wired or wireless, and each terminal can send and receive information via the network. The network is not limited to the Internet, and may also be a combination of networks with different protocols.

The drive recorder 101 is a device that is mounted on, for example, an automobile and records video images while the automobile is driving. The drive recorder 101 is an example of an in-vehicle device.
However, the installation target is not limited to automobiles, and may be any moving body in which the drive recorder 101 can be installed. For example, the installation target may be automobiles, buses, trucks, motorcycles, etc. Furthermore, the installation target may also be transportation vehicles such as trains, monorails, linear motor cars, etc.
The management server 102 is a server that stores and manages various information such as video information and status information acquired by the drive recorder 101. The information acquired by the drive recorder 101 may be configured to be sent directly from the drive recorder 101 to the management server 102, or may be configured to be sent to the management server 102 via the data center 104.

The user terminal 103 is a terminal used by, for example, a user who drives a car. The user terminal 103 is connected to the drive recorder 101 via a mobile data communication network or an in-vehicle network such as Wi-Fi (registered trademark) of the car, and can operate the drive recorder 101.
Furthermore, the user terminal 103 is connected to the management server 102 via these networks and can display various information stored in the management server 102 .

The data center 104 receives various information acquired by the drive recorder 101 and transmits it to the management server 102. For example, the data center 104 is connected to the drive recorder 101 via a low-power wide area (LPWA) network, receives various information transmitted from the drive recorder 101 via the LPWA network, and transmits the received information to the management server 102 via the Internet. Note that LPWA wireless communication is low-power and enables long-distance communication. However, limitations on the amount of data that can be transmitted per unit time can be an issue. For example, transmitting large amounts of data can result in significant communication delays. Examples of LPWA include ELTRES, LoRa, LoRaWAN, LTE-MTC, NB-IoT, NB-Fi Protocol, RPMA, GreenOFDM, DASH7, RPMA, Wi-SUN, EnOcean Long Range, Weightless-P, SIGFOX, LTE Cat.0, and LTE Cat.M1.
The management terminal 105 is a terminal for operating and managing the management server 102 .

Each terminal of the DVR management system 1 and the management server 102 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, a wristwatch, or clothing. They may also be stationary or portable computers, or servers located on the cloud or a network. In terms of functionality, they may be VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, or MR (Mixed Reality) terminals. Or they may be a combination of multiple of these terminals. For example, a combination of one smartphone and one wearable terminal can logically function as a single terminal. Other information processing terminals may also be used.

Each terminal and management server 102 of the DVR management system 1 includes a processor that executes an operating system, applications, programs, etc.; a primary storage device such as RAM (Random Access Memory); a secondary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), or flash memory; a communication control unit such as a network card, wireless communication module, or mobile communication module; input devices such as a touch panel, keyboard, mouse, voice input, and input based on motion detection captured by the camera unit; and an output device such as a monitor or display. The output device may also be a device or terminal that transmits information to be output to an external monitor, display, printer, or other device.

The main memory stores various programs, applications, etc. (called modules), and the processor executes these programs and applications to realize each functional element of the overall system. Each of these modules may be implemented in hardware, such as by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

In this specification, each module is described as a subject that performs processing, but in reality, the processing is carried out by a processor that processes various programs, applications, etc. (modules).
The auxiliary storage device stores various databases (DBs). A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate arbitrary data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The method of implementing the database is not limited, and may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.

FIG. 2 shows an example of the connection configuration of the management server 102.
In this connection configuration example, the drive recorder 101 transmits various information to the management server 102 via the data center 104 .
The drive recorder 101 transmits various types of information about the drive recorder 101 to the data center 104 via a wireless communication network such as an LPWA network. The customer-side management server 202 receives and stores various types of information from the data center 104 via the Internet using a protocol such as MQTT (Message Queue Telemetry Transport). The management terminal 105 and the user terminal 103 access the customer-side management server 202 via the Internet or a mobile communication network, receive and display various types of information.

The same function can also be performed via the service provider's management server 201 of the service provider that manages the DVR management system 1. Once various information about the drive recorder 101 is stored in the service provider's management server 201, it is sent to the management terminal 105 or the user terminal 103 via the customer's management server 203 or directly.
The various information transmitted and received may be encoded and decoded by each server.
It should be noted that the drive recorder 101 may be configured to transmit various pieces of information directly to the management server 102 rather than via the data center 104 .

FIG. 3 shows an example of the hardware configuration of the drive recorder 101.
The camera 304 captures images of the front, rear, or 360 degrees, and generates video information such as moving images and still images.
By equipping the image sensor of the camera 304 with an AI processing function that performs preprocessing, the following output can be generated in addition to or instead of generating a normal captured image. The AI processing function has a function of performing image recognition based on the video captured by the camera 304.
- Output metadata of the object from the image.
- Outputs images in ISP (Image Signal Processor) output format, for example, YUV or RGB.
・Output an image that only includes a specific area.

Performing preprocessing on the image sensor side in this way reduces the amount of data and enables real-time tracking of objects using high-speed AI processing. A particular benefit is that it reduces the amount of data (i.e., communication volume) sent from the drive recorder 101 to the management server 102. This is because, even when sending data via an LPWA network as in this embodiment, it is expected that data related to AI processing functions can be sent with low latency, for example.

The acceleration sensor 305 detects acceleration and generates acceleration information. For example, if a sudden change in acceleration occurs due to an impact, sudden steering, sudden stopping, etc., the DVR controller 303 detects the occurrence of an event. By analyzing this change in acceleration information, it is possible to infer the circumstances of an accident, such as an automobile accident.
The SD card 306 is a storage device for recording video information and various other information. Note that the SD card is not limited to an SD card, and any other configuration that can store data will suffice, but a flash memory with high vibration resistance is preferable for in-vehicle use.

The DVR controller 303 is connected to and controls the camera 304, acceleration sensor 305, SD card 306, and other sensors (not shown). Based on information received from these elements, the DVR controller 303 generates status information indicating the state of the DVR controller 303. The DVR controller 303 also transmits the generated status information to the microcomputer 301.
The microcomputer 301 performs the following various processes.
Controls the communication processing unit 302.
The communication processing unit 302 notifies the DVR controller 303 of the acquired location information.
The status information acquired from the DVR controller 303 and the position information acquired from the communication processing unit 302 are converted into, for example, 16-byte data.
- Performs sleep control when powered by a secondary battery.
- Execute test mode during manufacturing and testing.
The microcomputer 301 is programmable, and stores programs for executing the various processes described above. The various processes described above are realized by the execution of these programs by a processing unit of the microcomputer 301. In this embodiment, the processing unit of the microcomputer 301 or the microcomputer 301 itself may also be referred to as a control unit.

The communication processing unit 302 transmits information from the drive recorder 101 to the management server 102 via a wireless communication antenna 308, either directly or via the data center 104. The communication processing unit 302 performs wireless communication using the LPWA method, which is an example of narrowband communication. Although the LPWA method is described below, this may be replaced with another method of narrowband communication, or may be replaced with a communication method other than narrowband communication.
The communication processing unit 302 also has a location information acquisition unit implemented as an integrated circuit or the like, and acquires location information based on a signal received from a GNSS antenna 307 compatible with a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System). The location information acquisition unit can also acquire time information, moving speed information, and moving direction information.

It should be noted that the communication processing unit 302 does not necessarily have to have the location information acquisition unit, and the GNSS antenna 307 may be connected to the microcomputer 301, and these may have the location information acquisition unit.
Alternatively, the location information acquisition unit may be configured to exist independently and be connected to the microcomputer 301 or the communication processing unit 302 .
Furthermore, although the time information is acquired by the position information acquisition unit, the microcomputer 301 may be configured to acquire the position information.

The drive recorder 101 has a power supply 311 connected to a vehicle power supply 310. The power supply 311 supplies power received (or sometimes referred to as power reception) from the vehicle power supply 310 to the DVR controller 303 and also charges the secondary battery 312.
The secondary battery 312 is a rechargeable battery such as a lithium-ion battery, a nickel-cadmium battery, a nickel-metal hydride battery, or a lead-acid battery. However, a non-rechargeable primary battery may also be used. When the accessory power supply (ACC power supply) is turned off, or when the supply of power from the vehicle power supply 310 to the power supply 311 of the drive recorder 101 is interrupted due to a fault such as a broken wire or a disconnected wire caused by an accident, or when the power supply 311 is unable to supply power due to a malfunction or the like, the secondary battery transmits (or may also be called power transmission or power supply) power to the microcomputer 301, the communication processing unit 302, etc., thereby maintaining at least some of the functions of the drive recorder 101.

The voltage supplied from the vehicle power supply 310 of an automobile is generally 12 V. The power supplied from the power supply 311 or the secondary battery 312 is stepped down to a voltage of, for example, 2 V via a DC/DC converter 313 and supplied to the microcomputer 301 and the communication processing unit 302 to drive them.
The DVR controller 303, microcomputer 301, and communication processing unit 302 can each be implemented as an integrated circuit (chip) such as a SoC (System on a Chip), or a configuration in which a plurality of these components are integrated into a single chip may be used. In particular, in this embodiment, the microcomputer 301 and communication processing unit 302 operate in cooperation with each other, so they may be integrated into a single chip.

In this embodiment, the microcontroller 301 and the communication processing unit 302 are configured so that their power supply is separated from the DVR controller 303, and location information and various other information can be sent to the management server 102 by simply supplying power to the microcontroller 301 and the communication processing unit 302 from the secondary battery without starting the DVR controller 303, thereby reducing power consumption.
In particular, the drive recorder 101 can perform long-distance communication with low power consumption by using LPWA wireless communication, which allows the drive recorder 101 to transmit status information of the drive recorder 101 to the remote management server 102 while maximizing the duration of the secondary battery in the drive recorder 101.
Furthermore, since the drive recorder 101 transmits status information to the management server 102 using LPWA communication, it is desirable to perform communication frequently or at short intervals while keeping the amount of data to a minimum.
Therefore, the status information that the drive recorder 101 sends to the management server 102 is also based on the knowledge of the inventors of the present application and takes into consideration factors such as the characteristics of wireless communication using the LPWA method and the priority of the information to be sent to the management server 102.

Furthermore, by configuring the communication processing unit 302 to include a location information acquisition unit and a GNSS antenna 307, it becomes possible to acquire location information and speed information by simply driving the low-power communication processing unit 302 without starting the DVR controller 303.
Furthermore, by keeping the driving voltage of the microcontroller 301 and the communication processing unit 302 at a low voltage, such as 2 V, compared to the driving voltage of the drive recorder 101, which is, for example, 12 V, the power consumption used to generate and transmit the DVR status information 600 can be kept low.

By installing a small-capacity rechargeable secondary battery as in this embodiment and notifying the management server 102 of the status of the drive recorder 101 when powered by the secondary battery, it is possible to notify the administrator even if the user intentionally unplugs the drive recorder 101 while driving. It is also possible to notify of a malfunction even when the main unit cannot be turned on due to a malfunction. It is also possible to track theft.

FIG. 4 shows an example of the hardware configuration of the management server 102.
The management server 102 is configured, for example, by a server deployed on a cloud.
The main memory device 401 stores programs and applications such as a DVR management module 410 and a video information management module 411, and the processor 403 executes these programs and applications to realize each functional element of the management server 102.

The DVR management module 410 acquires DVR status information indicating the status of the drive recorder 101 from the drive recorder 101, decodes it, and stores and manages it in the DVR status information 600 in the auxiliary storage device 402.
The DVR management module 410 also transmits some or all of the information obtained by decoding the DVR status information 600 to the user terminal 103 or the management terminal 105 via push distribution or in response to a request from the user terminal 103 or the management terminal 105.

The video information management module 411 acquires the video information stored in the drive recorder 101 from the drive recorder 101, stores it in the video information 420 in the auxiliary storage device 402, and manages it.
The video information management module 411 also transmits video information 420 to the user terminal 103 or the management terminal 105 by push distribution or in response to a request from these terminals.
The video creation module 412 performs processing to create a video by integrating a plurality of still images received from the drive recorder 101 or the like.
The auxiliary storage device 402 stores the DVR status information 600, the video information 420, and the like.

FIG. 5 shows an example of the hardware configuration of the user terminal 103.
The user terminal 103 is configured as a terminal such as a smartphone, a tablet, a notebook PC, or a desktop PC.
The main memory device 501 stores programs and applications such as a management server collaboration module 510 and a DVR collaboration module 511, and the processor 503 executes these programs and applications to realize each functional element of the user terminal 103.

The management server cooperation module 510 cooperates with the management server 102, acquires the DVR status information 600 and the video information 420 stored in the management server 102, and displays this information on an output device 505 such as a display.
Furthermore, the management server cooperation module 510 cooperates with the management server 102 and can perform various settings of the management server 102 .
The DVR collaboration module 511 operates the drive recorder 101 and configures the drive recorder 101 when the drive recorder 101 and the user terminal 103 are connected via an in-vehicle network such as Wi-Fi (registered trademark).

The management terminal 105 can also be configured in the same way as the user terminal 103, and can operate and configure the drive recorder 101 and management server 102 using a DVR collaboration module and a management server collaboration module stored in the main memory device.

FIG. 6 shows an example of the data structure of DVR status information 600.
DVR status information 600 is a diagram that explains the data structure when the microcomputer 301 of the drive recorder 101 transmits status information obtained from the DVR controller 303 and location information obtained from the location information acquisition unit to the management server 102.
The DVR status information 600 may also be referred to as status information regarding the drive recorder 101 .

In this embodiment, the microcomputer 301 generates 16-byte data of the DVR status information 600, and the communication processing unit 302 stores it in the payload and transmits it to the management server 102 based on the LPWA protocol.
The protocol used for transmission is not limited to LPWA, and other communication methods and protocols may be used.

Item 610 indicates the item of the DVR status information 600 to be transmitted. Type 1 (620), Type 2 (630), and Type 3 (640) indicate three types of transmitted data content, and the numbers listed in each table indicate the data bit length. The sum of the data bit lengths of each Type is 128 bits, or 16 bytes.
In addition, there is a free format spare data configuration other than that described in item 610 as Type 4 (not shown).

In this embodiment, when the vehicle power supply 310 is being supplied, the communication processing unit 302 transmits the DVR status information 600 to the management server 102 at one-minute intervals. The microcomputer 301 acquires the position information, speed information, information indicating the recording status, etc. at three different times during this one interval, stores the position information, etc. for these three points in time in a 16-byte payload, and can send the information to the management server 102 in a single transmission process.
Type 1 is a format for transmitting location information etc. at one point in time. Type 2 is a format for transmitting location information etc. at two points in time together. Type 3 is a format for transmitting information etc. at three points in time together.

Referring again to FIGS. 1 and 3, one embodiment will be described in detail.
1 shows a drive recorder 101 as a specific example of the embodiment. However, the embodiment is not limited to the drive recorder 101. For example, the embodiment can be implemented as other in-vehicle devices such as a detector (radar). In a broad sense, the embodiment can relate to an electronic device that acquires video information.
Hereinafter, a drive recorder 101 as an in-vehicle device will be described as an example of an electronic device according to this embodiment.

The drive recorder 101 uses the camera (photography unit) 304 to capture video information of the surrounding scenery (for example, the front, rear, side, diagonally forward, diagonally backward, or any angle range up to 360° around the vehicle) while the vehicle is moving or stopped. The video information or image information is captured in the form of a video or multiple still images.

The drive recorder 101 is an example of an electronic device that incorporates a camera 304. For example, the drive recorder 101 can incorporate a camera that captures the surrounding scenery and records (stores) the video information. On the other hand, if the electronic device is implemented as a detector (radar), an electronic device with an external camera, or a car navigation system, it can communicate with or connect to an external camera to obtain video information captured by other in-vehicle devices.

The camera 304 generates a file in video format and acquires captured still images from the file. Alternatively, the camera 304 may capture still images (photographs) from the beginning and acquire the still images directly. The still images may be in, for example, JPEG format, but may be in other formats. The video may be in, for example, MPEG format (MPEG2, MPEG4, etc.), but may be in other formats. Note that a video can be understood as a chronological arrangement of multiple still images, and therefore, below, still images will be described as an example of image information captured by the camera 304.
The camera 304 is not limited by differences in the size of the camera (small, large, etc.), the color of the still image captured by the camera (black and white, full color, etc.), or the camera's shooting method (normal shooting, infrared shooting, etc.).

The drive recorder 101 preferably uses an LPWA communication processing unit 302 to transmit captured still image data to the external management server 102. Using LPWA enables relatively long-distance communication with relatively low power consumption. On the other hand, LPWA has the drawback of being slow in data transmission speed and unsuitable for exchanging large amounts of data.
Therefore, the drive recorder 101 performs special control to utilize the advantages of the LPWA while suppressing its disadvantages, and this control can be performed by, for example, the microcomputer 301 and/or the DVR controller 303, etc.

The electronic device (for example, the drive recorder 101) includes at least a communication processing unit 302 and a control unit (such as a microcomputer 301 and/or a DVR controller 303) that processes image information captured by a camera 304 and transmits a plurality of still images via the communication processing unit 302. Hereinafter, the microcomputer 301 and/or the DVR controller 303 will be collectively referred to simply as the control unit 350.
The still image is transmitted to an external management server 102, and the control on the management server 102 side will be described later.

"Video transmission test (Test 1)"
In order to understand the characteristics of transmitting data using the LPWA method, the applicant carried out the following tests 1 to 4.
First, the applicant conducted a test in which the drive recorder 101 transmitted a video stream captured by the camera 304 to the external management server 102 via the communication processing unit 302. As will be understood by those skilled in the art, in order to transmit a video stream in real time using the communication processing unit 302, the video stream needs to be transmitted stably. Generally, in order to transmit a video stream in real time, it is necessary to transmit a video stream with a bandwidth of 300 Kbps or more (ideally, about 1 Mbps).

The applicant therefore conducted a test in which a video stream with a bandwidth of 300 Kbps captured by the drive recorder 101 was transmitted in an actual driving vehicle via a communications processing unit 302 compatible with LPWA Cat. M1. As a result, degradation of the playback screen and frequent disconnections occurred on the receiving side (the management server 102 in this embodiment).

FIG. 7 is a diagram for explaining the outline of the contents of the above-mentioned test 1.
With the horizontal axis representing time series, a video stream with a bandwidth of 300 Kbps was captured and attempted to be transmitted, as shown within the rectangular areas 710 and 720. As illustrated in FIG. 7, the transmission speed varies depending on the surrounding environment. When the transmission speed of the first video stream, indicated by reference numeral 710, deteriorated, the file transfer failed and the transmission was interrupted (see reference numeral 712). When the connection was retried (see reference numeral 722), the deterioration in the transmission speed of the second video stream 720 did not result in a transmission interruption, but the image became distorted (see reference numeral 724).

As described above, a stable communication speed environment is essential for the drive recorder 101 to transmit the video stream captured by the camera 304 in real time. However, it has been found that the LPWA Cat. M1 specifications as of January 2021 pose practical challenges when transmitting the video stream captured by the camera 304 as is.

In relation to this, Patent Document 1 states that "the LPWA communication method used by LPWA communication system 1 does not ensure sufficient communication capacity, and therefore has the problem of not being able to transmit large amounts of video data, such as high-resolution still images and videos, in real time (see paragraph 0023)." In other words, Patent Document 1 states that because video information captured by an on-board camera cannot be transmitted satisfactorily in LPWA format, other means must be used.

Patent Document 2 also describes uploading low-bit-rate video to the cloud in real time using narrowband communications such as LPWA. However, this requires converting the original video captured by the in-vehicle camera into low-bit-rate video, and then converting this low-bit-rate video into high-definition video. Drive recorders are intended to provide evidence in the event of a traffic accident, among other purposes. For this reason, it is undesirable for there to be no strict one-to-one correspondence between the original video captured and the resulting high-definition video.

"Test of sending still images (Test 2)"
Next, the applicant conducted a test in which the drive recorder 101 transmitted multiple still images captured by the camera 304 to the external management server 102 via the communication processing unit 302. Specifically, the applicant conducted a test in which still images (JPEG files) of approximately 60 Kbytes captured by the drive recorder 101 were transmitted over a Cat. M1 connection HTTP (TCP/IP) in a vehicle that was actually running. As a result, the receiving side (management server 102) did not experience any interruptions in transmission (or the frequency of interruptions was extremely low), and satisfactory results were obtained from a practical standpoint.

FIG. 8 is a diagram for explaining the outline of the contents of the above-mentioned test 2.
With the horizontal axis representing time series, multiple still images 810 to 860 of approximately 60 Kbytes were captured and transmitted. It was confirmed that transmission was possible in approximately 3.5 seconds when transmission conditions were good (see reference numeral 810). When transmission conditions deteriorated, a delay of approximately 8 seconds in transmission speed was observed (see reference numeral 830), but even in this case, transmission was not interrupted. In this way, when multiple still images were transmitted while actually driving, it was confirmed that the time until transmission was completed varied depending on the line conditions, but the images could be reliably transmitted to the receiving side (management server 102) without causing file corruption or the like.

The results of these two tests, 1 and 2, confirmed the effectiveness of sending multiple still images taken by the camera 304 periodically to an external management server 102, rather than simply transmitting the video stream captured by the camera 304, and then converting these still images into a video on the management server 102 side.

The management server 102 aligns multiple still images in chronological order and creates a video. Leaving gaps between each still image (for example, from about one second to several seconds, e.g., about five seconds) reduces the smoothness of the video. However, this is not a major problem, as the role required of the drive recorder 101 is to secure recorded images in the event of a traffic accident, etc., and to confirm the actual route traveled. In fact, by collecting discrete still images in chronological order and creating a video, the overall data volume can be reduced.

"Test of sending still images and vehicle information (Test 3)"
Furthermore, the applicant conducted a test of still image transmission under different conditions, in which still images (relatively large data size) captured by the drive recorder 101 and vehicle information (relatively small data size) such as location information were separately transmitted from a vehicle in actual motion to the external management server 102.
The information transmitted from the drive recorder 101 to the external management server 102 includes not only still images captured by the camera 304, but also other vehicle information (here, text data) that similarly requires real-time performance, such as G-sensor events detected by an acceleration sensor 305, vehicle position information received from a GNSS antenna 307, and vehicle speed (see Figures 3 and 6).

Test 3 was conducted under different transmission conditions than Test 2 above. Multiple still images (each still image was approximately 50 Kbytes in size) taken by camera 304 while the vehicle was traveling were transmitted in LPWA format to an external management server 102. As a result, it was confirmed that the time required to complete transmission of each still image varied from 3 to 10 seconds depending on the line conditions, and that stable transmission at intervals of several seconds was not always possible. It was confirmed that when line conditions were particularly poor, the transmission time could be longer, potentially reducing real-time performance.

In contrast, vehicle information such as location information is relatively small in size compared to still images, and therefore transmission speeds are faster. In Test 3, for vehicle information with a small data size consisting of characters and numbers, transmission was completed in a few hundred milliseconds under the same conditions, confirming good real-time performance. In this way, it was confirmed that the probability of transmission failure is lower for vehicle information compared to still images, due to the relatively small data size.

Therefore, the results of actual tests 1 to 3 confirmed the effectiveness of separately transmitting the still images captured by the camera 304 and the vehicle information to the external management server 102. By transmitting the still images and the vehicle information separately, the volume of each piece of data transmitted by the communication processing unit 302 can be minimized.
Furthermore, by focusing on the transmission speed of the vehicle information transmitted independently, it is possible to determine whether the transmission status of the communication processing unit 302 is good or not. For this purpose, it is possible to use a ping (a smaller message) for checking data reachability.

"Test of reception strength, transmission speed and reception speed (Test 4)"
Furthermore, the applicant conducted tests to measure changes in the running speed, reception strength (RSRP: Reference Signal Receive Power), transmission speed, and reception speed in a vehicle that was actually running.
9 shows the change in communication speed (iperf every 30 seconds) corresponding to the CAT.M1 during actual driving, which was carried out by the applicant.

The vehicle speed during the test is shown at the bottom of Fig. 9 (see reference numeral 910). The vehicle speed changes depending on the surrounding conditions, etc., and the received signal strength (RSRP) shown second from the bottom changes accordingly (see reference numeral 920).
It was found that when the reception strength (RSRP) decreases, the reception speed (see reference numeral 930) shown second from the top and the transmission speed (see reference numeral 940) shown at the top decrease.
On the other hand, when the vehicle is stopped (travel speed is 0 km/h), the reception strength (RSRP) is stable, and the transmission speed and reception speed are also stable.
Note that the graph shows the average sending and receiving speeds over approximately 15 seconds measured using the communication performance measurement tool Iperf, so there is a possibility that speeds below 100 kbps may momentarily be close to zero.

Based on the results of tests 1 to 4 above, this embodiment utilizes special techniques to minimize the drawbacks of the LPWA used in the communication processing unit 302 while taking advantage of its advantages. That is, rather than transmitting all of the still images captured by the camera 304, the control unit 350 of the drive recorder 101 controls the transmission of multiple still images based on at least one of (a) the vehicle's driving state and (b) the transmission status of the still images (in this case, only a portion of the multiple still images is transmitted).

Preferably, vehicle information such as location information may be transmitted in its entirety from the drive recorder 101 to the management server 102, as it is relatively small in size. This allows for comprehensive recording of time-series changes in the vehicle's position and speed. On the other hand, depending on the vehicle's driving conditions, not all still images are necessarily required. The role required of the drive recorder 101 is to secure recorded images in the event of a traffic accident, etc., and to confirm the actual driving route, so some still images can be thinned out during normal driving when no traffic accidents or other events have occurred.

(a) Control based on vehicle driving conditions
Preferably, when the control unit 350 of the drive recorder 101 detects at least one of the following: that the vehicle speed is within a predetermined speed range or that the change in the vehicle's position is within a predetermined distance range, the control unit 350 selects and transmits a predetermined still image from among the multiple still images taken.

For example, when a vehicle is traveling at a relatively high legal speed (for example, 60 km/h or faster), it can be predicted that each still image captured by camera 304 will capture a different scene. In this case, thinning out some of the still images will cause the traveling route to suddenly change, which is likely to cause a sense of discomfort to the viewer.
On the other hand, when the vehicle is temporarily stopped due to traffic congestion or the like (for example, the vehicle's traveling speed is 0 km/h or the change in the vehicle's position is 0 km), the still images captured by camera 304 may each capture a substantially similar scene. In this case, even if some of the multiple still images are selectively removed and the remaining still images are animated, the continuity of the scenery is relatively not lost, and the viewer is unlikely to feel uncomfortable.

Generally, low image quality reduces the reliability of individual still images, so there is a risk in reducing the number of still images recorded as evidence. However, as shown in Figure 9, when the vehicle is stopped (here, the traveling speed is 0 km/h), the reception strength (RSRP) is stable, and the transmission and reception speeds are also stable. In this state, the image quality of the captured still images is relatively good (Test 4). For this reason, it can be said that there is less risk in thinning out some of the multiple still images captured by a stopped vehicle. This is because the reliability of each still image is high, further reducing the need to prepare spares.

Therefore, in one embodiment, when the control unit 350 of the drive recorder 101 determines that the vehicle is traveling at a low speed or at a stop, it selects only some still images from the multiple images captured by the camera 304 and transmits them to the external management server 102. The control unit 350 may determine that the vehicle is traveling at a low speed or at a stop when the vehicle speed is within a predetermined speed range or a predetermined movement range. The vehicle's traveling at a low speed may be determined, for example, when the vehicle speed is 2 km/h or less. The vehicle's stop may be determined, for example, when the vehicle speed is 0 km/h.
This embodiment will be described below with reference to FIGS.

"Test of sending still images (Test 5)"
From this perspective, the applicant conducted a test in which a plurality of still images taken by the camera 304 were selectively transmitted from a vehicle in motion.
Referring to FIG. 10, the results of Test 5 above are shown in a table.

The leftmost column in Fig. 10 shows the order in which each still image was captured over time. In this example, a total of 24 still images were captured. These 24 still images are specifically shown in Figs. 11 to 13. As can be seen from Figs. 11 to 13, this embodiment can be effectively applied to checking the route a vehicle is traveling, etc.
The second column from the left in Fig. 10 shows the file names of the still images. For example, in Example No. 1, the still image was captured at 02:41:55 on December 17, 2020. The control unit 350 automatically selected the file name "20201217-024155" to match the capture time.
In this way, preferably, each still image is sent to the management server 102 with the shooting date and time as the file name, which allows the management server 102 to smoothly manage the data of each still image.

The third and fourth columns from the left in Figure 10 show the latitude and longitude of the vehicle at the time each still image was captured. This information may be obtained, for example, based on signals received from the GNSS antenna 307.
The fifth column from the left in Figure 10 shows the vehicle speed at the time each still image was captured. For example, the first example in No. 1 displays "21," which means 21 km/h (kilometers per hour). The unit may be meters per second, an SI unit. Alternatively, miles per hour or feet per second may be used.

The sixth column from the left in Figure 10 shows the still image time difference as the time interval between two adjacent still images. In the example shown in Figure 10, each still image was basically taken five seconds apart. This interval is not necessarily constant and can be longer, such as six seconds, or shorter, such as four seconds, or other intervals. It is also possible to take each still image at different intervals. By recording these still image time differences, the relative relationships between these still images can be understood. For example, when organizing multiple still images into a chronological order, they can be aligned to match the corresponding shooting times.

The rightmost column in Fig. 10 indicates which still images were not transmitted by the control unit 350. Specifically, in the example of Fig. 10, focusing on the vehicle's traveling speed, it can be seen that still images No. 7, No. 8, No. 10, No. 11, No. 13, No. 14, No. 16, No. 17, and No. 19 were not transmitted.
As described above, the still image and the vehicle information (latitude, longitude, etc.) are transmitted separately by the control unit 350.

Next, with reference to Figures 14 and 15, we will specifically explain the case where, among multiple still images (No. 15 to No. 22) within the range indicated by reference numeral 1000 in Figure 10, still images that meet predetermined conditions (No. 15, No. 18, No. 20, No. 21, No. 22) are selectively transmitted.

The left columns of Figures 14 and 15 show examples of still images taken within the range of No. 15 to No. 22 in Figure 10. The still images were taken continuously from top to bottom at a predetermined interval (approximately every 5 seconds).
14 and 15 show cases where still images that meet predetermined conditions are selected from the plurality of still images. Still images that are not selected (No. 16, No. 17, No. 19) are not sent to the management server 102. For ease of explanation, they are referred to as "no image."

Specifically, the area between No. 15 and No. 19 corresponds to a situation where the vehicle is stopped at an intersection at a speed of 0 km/h (or with a position change of 0 m) while waiting for the traffic light to change. Next, the area between No. 20 and No. 22 corresponds to a situation where the vehicle has restarted, entered the intersection at a low speed, and is making a left turn.

When a vehicle is stopped, there is generally little relative change between the vehicle and the surrounding scenery. However, this does not include the movement of other vehicles or people in the vicinity. Therefore, when the control unit 350 of the drive recorder 101 determines that the vehicle is traveling at a low speed or is stopped, it selects only some still images that meet predetermined conditions from the multiple images captured by the camera 304 and transmits them to the external management server 102. In this example, of the multiple still images (No. 15 to No. 19), still images (No. 15 and No. 18) are selected and transmitted.

The still images that meet the predetermined conditions are still images that are selected every predetermined number of still images from a plurality of still images that are continuously shot.
Preferably, the still image that meets the predetermined condition is at least one still image out of at least three still images that are taken consecutively.
This allows the control unit 350 to regularly thin out still images for each group of a predetermined number of images, thereby reducing the burden on the system during transmission. This is particularly effective when the communication processing unit 302 transmits data using the LPWA method, where the data volume during transmission is an issue. Furthermore, by deleting two out of every three images, the difference in playback speed with the video footage of the vehicle running can be eliminated, effectively compressing the video data size.

More preferably, the still image that meets the predetermined condition is the most recent still image among a plurality of still images that have been taken in succession.
Alternatively, the still image that meets the above-mentioned specified conditions is the most recent still image among at least three still images taken in succession, and in the example of Figures 14 and 15, one still image is transmitted every 15 seconds.
This allows the control unit 350 to more regularly thin out still images for each group of a predetermined number of images, so that when the selected still images are converted into a video, the unnaturalness caused by missing still images can be relatively reduced. Also, accidents and the like generally occur immediately after a vehicle waiting at a traffic light at an intersection starts to move. Therefore, by keeping the most recent still image, the still image immediately after the vehicle starts to move is selected.

More preferably, the still image that satisfies the above-mentioned predetermined condition is the most recent still image among a plurality of still images taken continuously when the vehicle speed is continuously at or below a low speed.
For example, in the embodiment shown in Figures 10 to 15, the predetermined condition is set as follows: "If the vehicle speed is continuously low (for example, 2 km/h) or less for three consecutive files, the third file, which is the most recent in terms of time, is transmitted, and the remaining two files are not transmitted."

More preferably, the still image that meets the above-mentioned predetermined condition is a still image, among multiple still images captured in succession, that is captured at or after the vehicle's start of movement is detected. For example, in the example shown in Figures 10 to 15, in No. 20, the vehicle's speed changes from 0 km/h to 7 km/h (i.e., 2 km/h or more), and the vehicle's start of movement is detected. Therefore, still image No. 20, which corresponds to the time at or after the vehicle's start of movement is detected, is selected and transmitted.

It is generally said that traffic accidents are most likely to occur at intersections, etc., immediately after a vehicle begins to move. By performing the above control, it is possible to preferentially extract still images that show the surrounding conditions immediately after the vehicle begins to move. For example, the second image from the top in Figure 15 (No. 20 in Figure 10) records a vehicle entering an intersection and a bicycle or other vehicle attempting to cross the crosswalk at the intersection. Therefore, the results of Test 5 above show that this embodiment does not simply thin out still images periodically, but is also able to preferentially extract still images that capture changes in the surrounding conditions, making it suitable for a drive recorder 101 that is required to record evidential images in the event of a traffic accident, etc.

As described above, the drive recorder 101 prioritizes the extraction of highly necessary images while periodically thinning out less necessary images, thereby reducing the overall required image capacity (amount of transmission or recording) (see Figures 14 and 15). In actual test 5, it was confirmed that by performing the above control, data reduction of more than 20% was possible for driving in town. This demonstrates that the drive recorder 101 is suitable for transmitting data using the LPWA method.

However, when an event related to a vehicle collision or the like is detected, the control unit 350 of the drive recorder 101 records or transmits all still images within a predetermined period of time. Such an event can be detected, for example, by a G sensor or the like.
For example, in the column of still images shown on the left side of Figures 14 and 15, if an event related to a vehicle collision or the like occurs, all still images No. 15 to No. 22 may be selected and transmitted.

When an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images within a specified period of time, from a time point in the past prior to the time of detection to a time point in the future prior to the time of detection.
For example, when an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images from a past point in time that is a predetermined period (e.g., a few seconds, tens of seconds, one minute, several minutes, or more) before the time of detection to a future point in time that is a predetermined period (e.g., a few seconds, tens of seconds, one minute, several minutes, or more) after the time of detection.

In this case, still images that are not transmitted by the control unit 350 may not be deleted immediately, but may be temporarily stored in a transmission buffer and then deleted after a time lag (for example, several seconds, several tens of seconds, one minute, several minutes, or more).
This makes it possible to record all still images taken during the period before and after the accident, including the moment of the accident, when a traffic accident occurs.

Strictly speaking, the time when a still image is captured and the time when the still image is transmitted do not coincide. For example, the time when the still image is transmitted is slightly (approximately 1 second) later than the time when the still image was captured. For this reason, the control unit 350 may transmit a still image if the vehicle speed at the time of transmission or the speed at the time when the still image was captured does not satisfy the low speed condition.

Furthermore, with this method, if there is a slight delay between the timing of still image capture and transmission due to internal processing time or the like (or transmission is intentionally delayed), and a stopped (or low-speed) state is confirmed by GNSS or vehicle speed pulses at both times, the control unit 350 may discard the transmission of the still image two out of three times, thereby reducing the amount of transmitted data and lowering communication charges.

Regarding the "predetermined speed range": Preferably, the "predetermined speed range" relates to when the vehicle is traveling at low speed or when the vehicle is stopped.
Compared to the normal legal speed limit (e.g., approximately 60 km/h), when driving at low speeds, the surrounding scenery changes more slowly, increasing the tendency for duplicated data and unnecessary data to accumulate. In particular, when the drive recorder 101 communicates wirelessly using LPWA, it is preferable to reduce the transmission load. Furthermore, the incidence of traffic accidents has been decreasing in Japan in recent years. Therefore, video of a vehicle traveling at low speeds is selectively recorded.

In the above example, the predetermined speed range is for when the vehicle is traveling at a low speed of 2 km/h or less, or when the vehicle is stopped at 0 km/h, but this embodiment is not limited to this speed range. The vehicle's traveling speed varies depending on changes in the surrounding conditions, etc. The upper limit of the predetermined speed range can be changed in various ways depending on the embodiment.

Preferably, the specified speed range should include a stopped state (e.g., 0 km/h). Generally, vehicles may come to a temporary complete stop (0 km/h) while waiting for a traffic light to change at an intersection, when waiting for a train to pass at a railroad crossing, or during heavy traffic congestion. In such cases, leading and following vehicles may not be able to stop, making rear-end collisions more likely to occur.

Preferably, the specified speed range may be extremely slow driving (several kilometers per hour or less, for example, 2 kilometers per hour or less). Generally, when a vehicle is driving at extremely slow speeds, there are situations, such as traffic jams, where the driver is forced to drive slowly to maintain an appropriate distance from the vehicle ahead. In such cases, it is believed that the driver is more likely to be distracted or not pay attention to the road ahead due to, for example, operating a car navigation system or a mobile phone.

The specified speed range may also be slow driving (for example, 10 km/h or less). Generally, when a vehicle is traveling slowly (for example, 10 km/h or less), there are situations that require careful driving, such as when traveling on narrow roads or entering intersections. At these speeds, it is said that the vehicle's stopping distance can vary by several meters depending on whether the driver is anticipating danger while driving.

Furthermore, the specified speed range may be low-speed driving (e.g., 30 km/h or less). It is generally said that traffic accidents are more likely to occur when a vehicle is traveling at low speeds (e.g., 30 km/h or less). For example, statistically, in Japan, three-quarters of all traffic accidents occur at speeds of 30 km/h or less.

Therefore, in consideration of the role of the drive recorder 101 in capturing video footage of traffic accidents, the upper limit of "low speed driving" may be 30 km/h or less, or 10 km/h or less, or preferably a few kilometers per hour or less (for example, 2 km/h or less), and the lower limit is preferably 0 km/h.

"Vehicle begins to move"
A change in the speed or acceleration of the vehicle can be used as a trigger for detecting the start of vehicle movement. For example, the control unit 350 may determine that the vehicle has started to move when an acceleration of the vehicle that exceeds a predetermined threshold (e.g., 0) from 0 is detected based on an input signal from a speed sensor or an acceleration sensor of the vehicle.

For example, the control unit 350 may calculate the vehicle speed based on the signal received by the GNSS antenna 307. In this case, depending on the reception environment of the signal, a positioning error may occur, and the vehicle speed may be calculated as being faster by the amount of this positioning error. For example, even if the vehicle speed is actually 0 km/h, it may be calculated as a speed other than 0 km/h. Therefore, the control unit 350 may determine a predetermined speed range taking this positioning error into account. For example, the control unit 350 may determine that the vehicle is stopped when the speed is 7 km/h or less or 8 km/h or less.

The control unit 350 can also use a change in the vehicle's position as a trigger to detect when the vehicle has started to move. For example, the control unit 350 may determine that the vehicle has started to move from the position information acquired based on a signal received from the GNSS antenna 307. For example, the latitude and longitude information of a vehicle from GPS generally can contain an error of about 7 km or 8 km. Therefore, the control unit 350 may recognize that the vehicle has started to move when a change in the vehicle's position of more than 7 km or 8 km is detected.

The above-mentioned changes in vehicle speed or acceleration and changes in vehicle position may be used in combination. While it is difficult to detect the start of vehicle movement with high accuracy based on signals received from the GNSS antenna 307, combining the vehicle speed or acceleration can increase reliability. For example, the control unit 350 may increase the reliability of the information by comparing (cross-checking) changes in vehicle speed or acceleration over time with changes in vehicle position over time.

The trigger for detecting the start of vehicle movement is not limited to when the vehicle actually starts moving and at least changes its position. The trigger for detecting the start of vehicle movement can also include the moment when the vehicle state changes from a stopped state to a moving state, or the state immediately before that, based on a signal sent from the in-vehicle device, even if the vehicle's position has not actually changed.

For example, in the case where the vehicle is a multi-wheeled vehicle including a four-wheeled vehicle, if changes in the operation of one or more of the accelerator pedal, brake pedal, clutch pedal, parking brake lever, automatic transmission change lever, and manual transmission shift lever can be supplied as signals to the drive recorder 101, the control unit 350 can use one or more of these as a trigger to detect the start of movement of the vehicle.

For example, when a vehicle restarts after a temporary stop at an intersection, etc., the brake pedal is generally released from a predetermined depression amount, and the depression amount of the accelerator pedal of the vehicle increases by a predetermined amount from zero. Therefore, the start of movement of the vehicle may be detected based on the depression amount of the accelerator pedal and/or the brake pedal.
When restarting the vehicle on a slope, the parking brake lever may be further operated. Therefore, the start of vehicle movement may be detected based on the operation of the parking brake lever.

In the case of a manual transmission vehicle, when restarting from a temporary stop at an intersection, etc., in addition to braking, the clutch and the manual transmission shift lever may be operated, so the start of vehicle movement may be detected based on these operations.
In the case of an automatic transmission vehicle, the automatic transmission change lever may be operated when restarting the vehicle after a temporary stop at an intersection, etc. Therefore, the start of movement of the vehicle may be detected based on the operation of the automatic transmission change lever.

Furthermore, if the vehicle is a motorcycle, sidecar, or trike (three-wheeled vehicle), the control unit 350 can use one or more of the accelerator grip, front wheel brake lever, rear wheel brake lever, clutch lever, and shift pedal as a trigger to detect the start of movement, if changes in the operation of one or more of these can be supplied as signals to the drive recorder 101.

For example, when a vehicle restarts after a temporary stop at an intersection, etc., the amount of grip of the accelerator pedal of the vehicle may increase by a predetermined amount from zero, or the front wheel brake lever and/or the rear wheel brake lever may be released from a predetermined amount of depression. Therefore, the start of movement of the vehicle may be detected based on the accelerator grip, the front wheel brake lever and/or the rear wheel brake lever.
Unlike automatic transmission vehicles, manual transmission vehicles may require clutch and shift pedal operation in addition to brake operation when restarting from a temporary stop at an intersection, etc. Therefore, the start of vehicle movement may be detected based on these operations.

In addition, the control unit 350 may detect a change in the state of the vehicle, such as a temporary stop or restart, based on a signal from any device installed in the vehicle. For example, in the case of a vehicle equipped with an idling stop function that stops the engine while the vehicle is parked or stopped, waiting at a traffic light, etc., if the control unit 350 can receive a signal indicating the start or end of the idling stop function, it may detect a temporary stop or restart of the vehicle based on that signal.

Furthermore, if the control unit 350 can receive a signal indicating, for example, that the steering wheel has been turned, it may detect a temporary stop or restart of the vehicle based on that signal. Furthermore, when the control unit 350 detects that the vehicle has started moving again based on the signal sent from the above-mentioned in-vehicle device, it can transmit a still image at that time (without a time lag) or after a delay from that time (with a time lag of, for example, one second or more, or several seconds or more).

(b) Control based on the transmission status of still images
Furthermore, in another embodiment, the control unit 350 of the drive recorder 101 selects only some of the still images from the multiple images captured by the camera 304 based on the transmission status of the still images and transmits them to the external management server 102.
This embodiment will be described below with reference to FIGS.

"Test of sending still images (Test 6)"
Furthermore, the applicant conducted a test to control (in this case, selectively transmit) the transmission of multiple still images based on the transmission status of the still images.
The results of Test 6 are shown in the form of tables in Figures 16 and 17. Although Figures 16 and 17 together form one table, they are shown in two parts for ease of viewing.
18 to 20 provide a more detailed explanation of three specific examples of this table.

In this embodiment, three types of data are transmitted by the control unit 350 from the drive recorder 101 to the management server 102. The first type of data is a captured still image (the largest image data), the second type of data is the vehicle's latitude and longitude location information (relatively small text data), and the third type of data is a ping (the smallest data) used to confirm data reachability.

16 and 17, the leftmost column shows the order in which still images are taken over time. In this example, a total of 37 still images are taken.
The second column from the left shows the file names of the still images.
The third and fourth columns from the left show the latitude and longitude of the vehicle at the time each still image was taken.
The fifth column from the left shows the vehicle speed at the time each still image was taken.
In the sixth column from the left, the still image time difference is shown as the time interval between two adjacent still images.

16 and 17, the seventh column from the left further shows the response time of a ping for checking data reachability.
The eighth column from the left shows the time it takes for the latitude and longitude data (vehicle information) to reach the server (management server 102).
The ninth column from the left shows the time it takes for a still image to reach the server (management server 102).
The rightmost column shows the transmission/reception status of still image data.

Next, a specific example of Figures 16 and 17 will be described with reference to Figures 18 to 20. In this example, the control unit 350 determines the transmission status of the communication processing unit 302 based on the number of still images transmitted and the response time from transmission of the still image to completion of transmission.

Referring to FIG. 18, the transmission state of data within the range indicated by reference numeral 1500 in FIG. 16 is conceptually shown.
FIG. 18 shows a case where the data transmission status is good, and each of the above three types of data sets is transmitted at a predetermined interval of 5 seconds.
Ping measures the round trip time (RTT) between sending a message from the drive recorder 101 as the sender to the management server 102 as the destination and receiving a response from the destination. Of the three types of data above, Ping has the smallest data capacity and is therefore the most suitable for transmission.

Generally, under good transmission conditions, a ping response will be returned within a few hundred milliseconds. For this reason, ping is used for verification purposes to objectively check the communication status.
Since the vehicle information (latitude and longitude data) is several hundred bytes in size, it takes less than one second for the management server 102 to receive it.
Since the still image data has a file size of 50K to 60K bytes, it takes 3 to 4 seconds for the management server 102 to receive it.

In the example of FIG. 18, the first ping takes 67.7 milliseconds to transmit data, and the next ping takes 124.3 milliseconds to transmit data.
Vehicle information has a larger data capacity than ping, and therefore requires a longer transmission time, with the first vehicle information taking less than one second, and the next vehicle information taking less than one second.
Since still images have the largest data volume, they require the longest transmission time, with the first still image requiring 4 seconds, and the next still image requiring 4 seconds.
The predetermined interval for transmitting each data set is 5 seconds, so that each data transmission is completed within the predetermined interval without causing any delay.
The predetermined interval between each still image can be determined taking into consideration the normal transmission time of a still image (for example, 5 seconds, but is not limited to this).

Next, referring to FIG. 19, a conceptual diagram of the transmission state of data within the range indicated by reference numeral 1600 in FIG. 16 is shown.
FIG. 19 shows a case where the transmission condition deteriorates, and similarly illustrates a situation where three types of data sets are transmitted at predetermined intervals of five seconds.
The first ping takes 133.1 milliseconds to transmit data, and the second ping takes over a second to transmit data.
That is, the communication state was good at the time of Ping No. 7, but the long data arrival time suggests that the communication state deteriorated immediately after the Ping was sent. At the time of No. 8, the communication state remained poor, and the Ping transmission time became even longer. In this way, the control unit 350 can determine whether the transmission state is good or deteriorating by comparing the time required to send the Ping with a predetermined threshold.

Preferably, the control unit 350 determines the transmission status of the communication processing unit based on the response time of data that is transmitted separately from the still image and is smaller in size than the still image, but this small data is not limited to Ping. For example, the response time of vehicle information may be used to determine the transmission status.

As the transmission conditions worsen, the first vehicle information takes one second, and the next vehicle information takes nine seconds. Because the predetermined interval for transmitting each data set is five seconds, the second vehicle information to be transmitted cannot be completed within five seconds. This may affect the transmission of subsequent vehicle information.
In particular, for still images, the first vehicle information takes 15 seconds, and the next vehicle information takes 21 seconds. Because the predetermined interval for transmitting each data set is 5 seconds, none of the still images transmitted in each case can be completed within 5 seconds. This may affect the transmission of subsequent still images. In this situation, there is a concern that the impact of the deterioration of communication conditions may accumulate.

Next, with reference to FIG. 20, a control for improving data transmission under the deteriorated transmission conditions illustrated in FIG. 19 will be described.
FIG. 20 conceptually shows two ranges, particularly the range 1700 in FIG. 16 and the range 1710 in FIG.

In Figure 20, similar to Figure 19, the control unit 350 determines that the transmission status (line status) is not good when two or more consecutive still images are being transmitted (transmission is incomplete) (see symbol A in Figure 20). In this case, the control unit 350 temporarily suspends the transmission of still images following the still image whose transmission time has been extended, and stores that data in the storage device (memory) of the drive recorder 101 (see symbol B in Figure 20). This temporarily halts the transmission of the still image with the largest data volume, reducing the overall burden of data transmission. As a result, it is possible to prevent the cumulative impact of deteriorated transmission status from affecting subsequent data transmissions, contributing to the early improvement of deteriorated transmission status.

Furthermore, the control unit 350 continuously monitors the number of still images being transmitted, and if the number of still images being transmitted is zero and the most recent Ping response time is less than a predetermined threshold (specified time) (e.g., less than 900 milliseconds) (No. 25), it can determine that the communication status has improved. In this case, the control unit 350 begins sequentially transmitting the still images stored in the temporary storage memory (see symbol C in Figure 20).

20, if we calculate backwards from the fact that it took 87 seconds for the still image No. 9 to reach the management server 102, we can see that transmission from the drive recorder 101 started at the same time as the transmission of data No. 26.
In the specific example of Fig. 20, still image transmission is not performed after No. 26 due to the low speed caused by the still image transmission thinning function (see Fig. 17). Therefore, after No. 26, the state of waiting for completion of still image transmission continues to be zero, and the control unit 350 can sequentially transmit the still images stored in the temporary storage memory.

As described above, the control unit 350 of the drive recorder 101 controls the transmission of a plurality of still images captured by the camera 304 based on at least one of the vehicle's running state and the still image transmission state.
(a) As illustrated in FIGS. 10 to 15, the control unit 350 may control the transmission of a plurality of still images captured by the camera 304 based on the traveling state of the vehicle.
(b) As illustrated in FIGS. 16 to 20, the control unit 350 may control the transmission of a plurality of still images captured by the camera 304 based on the transmission status of the still images.
(a) and (b) may be performed separately or in combination with each other.

The above-described controls (a) and (b) may be started at any timing while the vehicle is running or stopped. Preferably, the controls may be started at the moment when the engine of a stopped vehicle is started and the vehicle is ready to run after the engine is turned off. In other words, the controls may be started in synchronization with the control of the drive recorder 101.
The above-described controls (a) and (b) may be terminated at any timing while the vehicle is running or stopped. Preferably, the control may be terminated the moment the vehicle stops, the engine is turned off, and the vehicle becomes unable to run. In other words, the control may be terminated in synchronization with the control of the drive recorder 101.

In at least one of the cases of "(a) control based on the vehicle's driving state" and "(b) control based on the still image transmission state," when the control unit 350 delays the transmission of a still image due to poor radio wave conditions, etc., it temporarily stores the still image in the transmission buffer. With this configuration, when retransmitting a still image from the transmission buffer, the control unit 350 may dynamically change the priority of each still image so that the most recent still image is always transmitted with the highest priority. Specifically, if still image A, still image B, and still image C are stored in the transmission buffer in descending order of their shooting time, the transmission priority is highest in descending order of their shooting time, in this case still image A, still image B, and still image C.

If the latest still image Z cannot be sent from this state due to reasons such as radio wave conditions, the control unit 350 stores still image Z in the transmission buffer and changes the transmission priority to still image Z, still image A, still image B, and still image C. In this way, the control unit 350 performs control on a last-in, first-out basis. Furthermore, when the transmission buffer reaches its upper limit, the control unit 350 may perform processing to discard still images from the transmission buffer in order of lowest priority. This ensures that the latest still image is always transmitted when the radio wave conditions improve and it becomes possible to transmit still images. Therefore, even when monitoring images captured by the drive recorder 101 remotely in real time, it is easy to check the latest situation.

The number of cameras 304 used in the drive recorder 101 is not limited to one.
For example, the drive recorder 101 can use, as the camera 304, at least one of a camera that takes pictures in front of the vehicle, a camera that takes pictures behind the vehicle, a camera that takes pictures to the side of the vehicle, a camera that takes pictures diagonally in front of the vehicle, a camera that takes pictures diagonally behind the vehicle, and a camera that takes pictures at any angle range up to 360° around the vehicle.

For example, the cameras 304 of the drive recorder 101 may be provided on the front and rear sides of the vehicle, respectively. In this case, the camera 304F on the front side of the vehicle and the camera 304B on the rear side of the vehicle may independently capture images of the front side and the rear side, respectively.
The cameras 304F and 304B may synchronize the timing of capturing video images under the control of the same or different control units 350. When selecting an appropriate still image from among a plurality of still images captured by each camera based on the vehicle's running state or the like (controls a and b above), the cameras 304F and 304B may synchronously extract still images in front of and behind the vehicle based on the same trigger.

As described above, the drive recorder 101 can transmit original still images captured by the camera 304 to the management server 102 based on the LPWA format. In the above example, each still image is captured at 5-second intervals, but this interval can be changed in other embodiments to accommodate the data capacity of each still image captured by the camera 304 and future changes in communication technologies such as LPWA.

"Control on the Management Server 102 Side"
Next, referring again to FIGS. 1 and 4,
The control on the management server 102 side will be described.

The management server 102 has at least a receiving unit (e.g., the communication control unit 406 in FIG. 4) that receives multiple still images transmitted from the drive recorder 101 that controls the above-mentioned content, and a video creation unit 412 (e.g., the video creation module 412 executed by the processor 403 in FIG. 4) that integrates the multiple received still images to create a video. Note that the video creation unit 412 may be implemented as a sub-module of the video information management module 411.

For example, the management server 102 may perform a method including the following steps.
First, as a first step, the communication control unit 406 of the management server 102 sequentially receives each of the plurality of still images transmitted from the drive recorder 101 side.
Next, at this time, the communication control unit 406 or the video creation unit 412 of the management server 102 may combine into one group multiple still images transmitted at predetermined time intervals (e.g., several minutes, 10 minutes, 30 minutes, one hour, or more). Alternatively, the communication control unit 406 or the video creation unit 412 of the management server 102 may combine into one group multiple still images transmitted at predetermined numbers (e.g., several, 10, 30, 100, hundreds, thousands, or more).

Next, in the second step, the video creation unit 412 of the management server 102 applies an animation effect to each of the still images integrated into one group, in which multiple still images are sequentially switched over over time, like a flip book, to create a video. At this time, the video may be created to have a special effect, as described below. At this time, the created video may be stored in the storage device 401 or 402 of the management server 102.
Next, the communication control unit 406 of the management server 102 transmits the created video to the user terminal 103 .

For example, when a plurality of still images are transmitted from the drive recorder 101 to the management server 102, the management server 102 converts the plurality of still images into a moving image consisting of, for example, two still images per second. At this time, the moving image creation unit 412 of the management server 102 may perform control to reduce the amount of data used for creating the moving image.
For example, the management server 102 stores and manages still images (video information 420) transmitted from the drive recorder 101 in the auxiliary storage device 402. In this case, if the management server 102 stores the data in the form of still images one after another, the amount of data to be managed will be large, resulting in a large capacity. Therefore, when the management server 102 converts multiple still images into a video, it may store only the animated data and delete the data of the still images used in the video from the auxiliary storage device 402.

When creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view.
For example, the management server 102 may animate each received still image by aligning their time axes (according to the actual shooting time). However, in this case, the still images are updated every few seconds (e.g., every 5 seconds), resulting in a drawn-out video and a large video size. Therefore, when integrating multiple received still images to create a video, the video creation unit 412 of the management server 102 may create a video that is played back faster than the actual chronological order. For example, when animate each still image taken at 5-second intervals, the video creation unit 412 may display two still images per second. This improves the ease of viewing the video.
In this case, a video shot at one frame every five seconds is converted into a video shot at one frame every 0.5 seconds at a fast speed.

Additionally, when creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view.
Ideally, the drive recorder 101 captures multiple still images at regular time intervals. However, on the management server 102 side, delays can occur between still images depending on the line conditions, so the intervals are not necessarily regular. If this difference is too large, viewers may feel uncomfortable when the video is played back.

For example, if the delay is within the range of a few seconds to a dozen seconds, the viewer will not feel uncomfortable even if the video is animated and played back on a fixed timeline. When the vehicle is moving, the image is constantly changing, so even if there is a slight delay in the playback timeline, the viewer will not feel uncomfortable. Also, when the vehicle is stopped while waiting for a traffic light or railroad crossing to change, or when stopped in traffic, there are a series of still images with little change, so the viewer will not feel uncomfortable even if there is a delay in the timeline.

Therefore, if there are a series of still images with little change, the video creation unit 412 can create a video by thinning out the still images with little change. For example, if there are nine consecutive still images of waiting at a traffic light, the video creation unit 412 can create a video from the three still images by deleting two out of the nine still images.
By using this configuration, the playback time of the video can be further compressed, and the time spent checking parts with little change when watching can be saved.

On the other hand, if the time axis deviation occurs at a larger time interval than a predetermined threshold (for example, if a deviation of one minute or more occurs), viewers may feel uncomfortable. For example, depending on the surrounding conditions of the traveling vehicle, such as inside a tunnel, the function of the communication processing unit 302 may be reduced or may temporarily stop functioning.
Therefore, when creating a video from a plurality of received still images, if the still images have an interval that exceeds a predetermined threshold (for example, one minute), the video creation unit 412 may create a video by duplicating a still image that precedes this interval. Alternatively, the video creation unit 412 may create a video by generating a new still image based on the still images before and after this interval.
In this way, when still images are continuously missing, they may be copied at a rate corresponding to the missing time and inserted to create a moving image.

However, in the above case, the video creation unit 412 creates a video that displays information indicating that the original image data is unavailable. For example, the video creation unit 412 may add a caption (text information, etc.) indicating that there is no image data at the location where the copied still image has been inserted, and display this caption together with the caption when the video is played back. This makes it possible to clearly distinguish between images that were actually captured and images that were created to facilitate video creation, when images captured by the drive recorder 101 are used later.

Additionally, the video creation unit 412 may change the control when the vehicle is traveling at high speed compared to when the vehicle is traveling at low speed or is stopped as described above.
For example, suppose each still image is captured at an interval of 5 seconds. In this case, if the vehicle is traveling at a speed of about 100 km/h, when the captured images are converted into a video, the viewer may perceive the video as being played back in fast-forward mode. Therefore, the video creation unit 412 may change the frequency of fast-forwarding when the vehicle is traveling at a low speed and when the vehicle is traveling at a high speed.

Generally, when a series of still images are played back consecutively to create the appearance of a moving image, it is necessary to align the distance traveled by the vehicle with the playback time axis. However, it has been found that when the vehicle is traveling at a high speed, duplicating the still images to increase the number of images before creating a moving image makes the corresponding portion of the video appear to be traveling at a relatively slow speed during playback. In other words, even if the vehicle's actual traveling speed is relatively fast, the playback video may appear to be traveling at a slow speed, which may cause discomfort to the viewer.
Therefore, when the vehicle is traveling at a low speed or stopped, the video creation unit 412 may copy still images and increase the number of images before creating a video, as described above, while when the vehicle is traveling at a high speed, the video creation unit 412 may avoid the control of copying still images and increasing the number of images before creating a video.

Additionally, when creating a video from a plurality of still images, the video creation unit 412 can perform control to improve the convenience of the video.
That is, in addition to a plurality of still images, vehicle information such as vehicle position information is transmitted from the drive recorder 101 to the management server 102. The still images and vehicle information may each have information on the time of capture. Therefore, even if the management server 102 receives the still images and vehicle information separately, they can be linked together.

Therefore, on the management server 102 side, when the communication control unit 406 receives from the drive recorder 101 a plurality of still images and the vehicle's position information at the time the plurality of still images were created, the video creation unit 412 may embed the vehicle's position information in the video when integrating the received plurality of still images to create a video. This may enable the user terminal 103 that plays the video to display the vehicle's position information in synchronization with the video.
For example, when creating a video from multiple still images, the video creation unit 412 embeds vehicle information as metadata in the video file. As a result, when playing a video using a dedicated application on the user terminal 103 side, location information may be synchronized with the video data only and displayed on a map.

The video creation unit 412 may also transmit the video created from the still images and the vehicle information separately to the user terminal 103. Vehicle information can be transmitted faster and more efficiently than video. Therefore, the application software on the user terminal 103 that displays this information prioritizes displaying vehicle information such as location information on a map, and displays (updates) the video as soon as it is received, thereby preventing the user from perceiving any delays in data transmission overall.

Furthermore, the video creation unit 412 may include still image file names and checksum values in the vehicle information. This allows individual data stored on the management server 102 to be synchronized and displayed later, or still image data that cannot be received or is corrupted for some reason to be skipped and displayed in the application software.

"Vehicle position estimation using still images"
As described above, the management server 102 receives vehicle information from the drive recorder 101 and can know the latitude, longitude, etc. of the vehicle based on the information.
However, there may be cases where the vehicle's location information cannot be received sufficiently depending on the surrounding conditions of the traveling vehicle, communication conditions, etc. Therefore, the management server 102 may perform image analysis on the received still image to estimate the vehicle's location.

21A and 21B are diagrams schematically illustrating an example in which the management server 102 performs image analysis on a still image.
As shown by reference numeral 1810 in Fig. 21A, on general roads, there are road signs near intersections, etc. On expressways, there are exit guide boards, etc. These road signs, etc., usually have text information 1810 indicating the name of a place or other local area (for example, Tokyo).
Furthermore, as shown by reference numeral 1820 in FIG. 21A, these road signs and the like usually have text information 1820 indicating the distance to the reference area in numbers (e.g., 4 km).
Furthermore, these road signs and the like are usually marked with text information (not shown) indicating the road number of a national road, a prefectural road, or the like in numbers (for example, National Route 6, not shown).

Therefore, the video creation unit 412 of the management server 102 may perform image analysis of the still image, particularly based on the text information 1810 and 1820. The position of the vehicle may be estimated by performing image analysis on the still image based on the names or numbers of roads or areas.
For example, the video creation unit 412 of the management server 102 scans the still image to extract text such as letters or numbers. At this time, the video creation unit 412 may use an "OCR" function that recognizes and converts characters from the still image.

Normally, road information used on road signs, etc. (for example, area names and numbers for national highways, etc.) is known, so it is possible to comprehensively create a database of various information such as maps or road information in advance and store it in storage device 401 or 402.
Therefore, the video creation unit 412 transcribes the image data and temporarily stores the text information in the storage device 401 or 402. The vehicle position may then be estimated by comparing the information with a database that stores various types of information such as maps or road information.

Furthermore, the video creation unit 412 may perform image analysis of the still image based on characteristic shapes, figures, contours, colors, or the like.
For example, as shown in (A) of Fig. 21, there are cases where two or more roads temporarily overlap in the vertical direction (such as at a junction), and in such cases, it may not be possible to determine which of the two vertically overlapping roads the vehicle is traveling on, based only on the latitude and longitude information of the vehicle.
In such a case, the video creation unit 412 may perform image analysis on the received still image to distinguish, for example, between a case where there is some kind of structure above the road, as illustrated by reference numeral 1830 in Figure 21 (A), and a case where there is nothing above the road, as illustrated by reference numeral 1840 in Figure 21 (B).

For example, the video creation unit 412 may perform image analysis on the received still image, focusing particularly on the area above the road, based on changes in shape, contour, color, etc., to determine whether or not a characteristic target 1830 such as a structure can be extracted (see (A) and (B) in Figure 21).
In this case, the video creation unit 412 may classify the received still images by pixel or pixel group, and then perform image analysis by identifying the shape or color of each pixel or pixel group to identify characteristic targets.

The video creation unit 412 may perform image analysis training on various images in advance using AI machine learning. In particular, based on various specific examples, actual image analysis is performed to train the system to identify and extract targets (letters, numbers, shapes, contours, lines, vertices, colors, etc.) from images. Training is performed on, for example, hundreds, thousands, or more images of different targets, and various results are accumulated. In this case, image processing may be performed under various conditions, such as images of roads only, images of roads with multiple vehicles traveling on them, images of roads during the day, and images of roads at night, and statistical data may be calculated and generated from the results. After a high recognition rate, for example, a recognition rate of approximately 99%, 99.9%, or higher, is obtained through training, the video creation unit 412 may actually perform image processing to identify targets from images.

In the training performed by the AI-based machine learning, various image information may be used as training data. For example, a determination model may be generated by machine learning, with a still image as input and information on the contours, vertices, colors, letters, numbers, etc. of characteristic objects as output. By inputting a new still image into this determination model, a characteristic target object may be obtained as output.

For example, the video creation unit 412 may perform image analysis training in advance on still images of various tunnels or the top and bottom of elevated roads, and accumulate the statistical information. For example, the video creation unit 412 may store the shapes, colors, etc. of characteristic objects inside various tunnels or on elevated roads. Then, when performing image analysis on the still images, the video creation unit 412 may identify the vehicle's position based on the detection of these characteristic objects.

In this way, the video creation unit 412 may use image analysis and AI to estimate the position of a vehicle from received still images when the vehicle's position cannot be determined using GNSS, such as inside a tunnel or above or below an elevated road, and use this information for vehicle management and driving records. Furthermore, the video creation unit 412 may estimate the position inside a tunnel from text on road signs, exit guide boards, etc. Furthermore, for an elevated road, the video creation unit 412 may determine whether the vehicle is traveling above or below the road signs, based on the presence or absence of structures at the top of the screen.

The video creation unit 412 of the management server 102 may determine the position of a moving vehicle with greater accuracy by combining vehicle information (latitude, longitude, etc.) sent from the drive recorder 101 with target information extracted based on image analysis performed on still images. This position information may be used when creating a video from a still image and displaying the created video on a user terminal.

As described above, the control of creating a video from a plurality of still images performed by the management server 102 may be started at any timing.
For example, the management server 102 may control the creation of a plurality of still images into a video after reception of the still images has started and then after the reception of the still images has stopped. In other words, the video creation unit 412 may start controlling the creation of a video after reception of a series of still images is completed, so that still images from the entire process of the vehicle's travel are compiled into a single video. This means that the moment when reception of still images stops at intervals of, for example, one minute, several minutes, five minutes, ten minutes, or more may be used as a trigger to determine that the vehicle has finished traveling, and video creation may start immediately after that timing.

Alternatively, the management server 102 may control the creation of a video from multiple still images after a predetermined period of time has elapsed since the start of reception of still images transmitted from a moving vehicle. For example, the video creation unit 412 may start creating a video several minutes, five minutes, ten minutes, 30 minutes, one hour, or several hours after the start of reception of still images. This allows videos to be created at predetermined time intervals or with a predetermined size. The video creation unit 412 may create one video if the vehicle's travel is completed in a short period of time, and may create multiple videos in succession if the vehicle's travel is extended over a long period of time.
The control of animation of a plurality of still images performed on the management server 102 side may end when the animation is created and transmitted to the user terminal 103. Alternatively, the control of animation of a plurality of still images performed on the management server 102 side may end when the animation is created and saved in a storage device on the management server 102 side.

The drive recorder 101 and management server 102 have been described above. Furthermore, the above control may be implemented as a program that can be executed on the drive recorder 101 and management server 102.

For example, this embodiment may be a computer program product for causing a computer to implement the present invention so as to enable the drive recorder 101 to control the transmission of still images and/or vehicle information as described above with reference to Figures 1, 3, and 7 to 20.
The computer program product may be implemented as a program or feature or function or routine or executable object.

Preferably, the computer program product comprises a program code enabling execution of a method for controlling the drive recorder 101 including the above-mentioned controls a and b.
The program is stored in a storage device connected to a control unit such as the microcomputer 301 or the DVR controller 303 of the drive recorder 101. The storage device may be built into the drive recorder 101 or may be externally connected. The program controls the control unit of the drive recorder 101 so as to enable control of transmission of still images.

Furthermore, it may be a computer program product that causes a computer to execute the control of animation of the contents described above with reference to FIGS. 1, 4 and 21 on the management server 102 .
The computer program product may be implemented as a program or feature or function or routine or executable object.

Preferably, the computer program product comprises program code enabling the execution of a method for controlling the management server 102 including said animation.
The program is stored in a storage device (401 or 402, etc.) connected to the video creation unit 412 of the management server 102. The storage device may be built into the management server 102 or may be externally connected. The program controls the video creation unit 412 of the management server 102 so as to enable control of animation.

Furthermore, this embodiment may be a computer program product for causing a computer to implement the control of the playback of the video described above with reference to FIG. 1 on the user terminal 103 .
The computer program product may be implemented as a program or feature or function or routine or executable object.

Preferably, the computer program product comprises program code enabling the execution of the method for controlling a user terminal 103 including video playback as described above.
The program is stored in a video playback unit such as the processor 503 of the user terminal 103 or in a storage device (such as 501 or 502) connected to the video playback unit. The storage device may be built into the user terminal 103 or may be externally connected. The program controls the video playback unit of the user terminal 103 so as to enable control of video playback.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.
In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

A computer program product, such as a computer program means, may be implemented as a memory card, USB stick, CD-ROM, DVD, or as a file downloadable from a server in a network. For example, such a file may be provided by transferring a file containing the computer program product over a wireless communication network.

The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed herein. The configurations of the present invention for which a patent is sought are specified in the accompanying claims. However, it is the intention of the present inventors to include configurations disclosed in this specification that are not currently specified in the claims in the future.

The present invention is not limited to the configurations described in the above-described embodiments. The components of each of the above-described embodiments and variations may be arbitrarily selected and combined. Furthermore, any component of each embodiment or variation may be arbitrarily combined with any component described in the Summary of the Invention or any component embodying any component described in the Summary of the Invention. It is our intention to obtain rights to these as well through amendments to this application or divisional applications, etc. Even if a statement such as "in the case of..." or "when..." is included, it is not intended to describe a configuration that is limited to that case or time. Configurations that are not limited to those cases or times are also disclosed, and we intend to obtain rights to them. Furthermore, any descriptions that specify an order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and we intend to obtain rights to them.

The above describes embodiments relating to various controls implemented in relation to the drive recorder (electronic device or in-vehicle device) 101, management server 102, and user terminal 103. These inventions can be combined in any manner. Additionally, any configurations may be extracted from these inventions and combined. The applicant of this application intends to acquire rights to inventions that include these configurations. Furthermore, even if a description is made of "in the case of..." or "when...," it is not intended to describe a configuration that is limited to that case or time. These are merely examples of better configurations, and the applicant intends to acquire rights to configurations that are not limited to those cases or times. Furthermore, sections that are described in order are not limited to this order. Configurations in which some sections are deleted or the order is changed are also disclosed, and the applicant intends to acquire rights to them.

Furthermore, by converting the drawings to a design registration application, it is intended to obtain rights to the overall design or partial design. The drawings depict the entire device using solid lines, but they also include partial designs claimed for parts of the device. For example, not only can a partial design be made of a part of the device, but the drawings also include a part of the device as a partial design regardless of the part. A part of the device could be a part of the device, or it could be part of that part. It is intended to obtain rights not only to the overall design, but also to partial designs in which any part of the solid line portion of the drawings is made into a dashed line. Furthermore, the modules, parts, components, etc. inside the device's casing, all of which are shown in the drawings, are subject to trade independently, and it is intended to obtain rights to them similarly by converting the drawings to a design registration application.

The above-described embodiment discloses at least the following configurations.
(1)
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
The control unit controls the transmission of the plurality of still images based on at least one of a vehicle running state and a still image transmission state.
This configuration reduces the burden on the electronic device during transmission because still images are transmitted, compared to when a video stream is directly transmitted. Alternatively, still images can be transmitted under optimal conditions by controlling the transmission of still images based on the vehicle's running state and the transmission state of the still images.
(2)
The control unit, regarding the running state of the vehicle,
When it is detected that the speed of the vehicle is within a predetermined speed range or that the change in the position of the vehicle is within a predetermined distance range, a predetermined still image may be selected from the plurality of captured still images and transmitted.
This configuration reduces the burden on the electronic device during transmission compared to when all still images are transmitted. In this case, by thinning out still images while paying attention to changes in the vehicle's traveling speed or position, still images suitable for creating a video can be preferentially retained and thinned out.
(3)
The predetermined still images may be still images selected every predetermined number of the plurality of still images captured continuously.
With this configuration, still images are periodically thinned out for each group, so that it is possible to avoid localized culling of still images.
(4)
The predetermined still image may be the most recent still image among the plurality of still images captured in succession.
With this configuration, still images are periodically thinned out at the beginning of each group, so that moving images can be generated regularly.
(5)
The predetermined still image may be a still image taken at or after the start of movement of the vehicle is detected.
By configuring in this way, it is possible to record the surroundings when a vehicle that has been temporarily stopped at an intersection or the like starts moving again.
(6)
The running state of the vehicle is
a change in the speed or acceleration of the vehicle;
A change in the position of the vehicle, or in the case where the vehicle is a multi-wheeled vehicle including a four-wheeled vehicle, operation of one or more of an accelerator pedal, a brake pedal, a clutch pedal, a parking brake lever, an automatic transmission change lever, and a manual transmission shift lever,
The determination may be based on at least one of the following:
This configuration allows the system to detect when a vehicle starts moving and transmit still images based on that information. Furthermore, by combining this information with changes in speed and position, the reliability of the detection results can be improved.
(7)
The control unit may transmit all still images within a predetermined period, including the predetermined still image, when an event related to a collision of the vehicle is detected.
By configuring in this way, in the event of a traffic accident or the like, it is possible to comprehensively record original still images taken by the vehicle-mounted camera for the period before and after the accident.
(8)
The predetermined speed may be a speed when the vehicle is traveling at a low speed or a speed when the vehicle is stopped.
This configuration allows for the still images to be thinned out when the vehicle's running position changes relatively slowly and similar scenes are continuously captured.Also, when the reception strength (RSRP), transmission speed, and reception speed are relatively stable and the need for spare images decreases, the still images can be thinned out.
(9)
The control unit may determine the transmission state of the communication processing unit based on the number of transmitted still images and a response time from transmission of the still image to completion of transmission.
This configuration makes it possible to monitor changes in the communication status of a moving vehicle depending on the location and speed of the vehicle, etc. This allows still images to be transmitted under relatively good communication conditions.
(10)
The control unit may determine the transmission state of the communication processing unit based on a response time of data that is transmitted separately from the still image and has a size smaller than the still image.
By configuring in this manner, the communication status of a moving vehicle can be determined in a relatively short time based on a ping or vehicle information, etc., which has a small data capacity, before transmitting a still image, which has a large data capacity.
(11)
When the response time of two consecutive still images exceeds a predetermined threshold, the control unit may store the subsequent still images in a storage device without transmitting the subsequent still images.
With this configuration, it is possible to avoid the cumulative effect of delays in still image transmission caused by constantly transmitting still images under worsening transmission conditions.
(12)
The control unit may transmit the still image stored in the storage device if the number of still images currently being transmitted is 0 and the response time of data smaller in size than the still image that is transmitted separately from the still image is below a predetermined threshold.
With this configuration, it is possible to transmit all selected still images while avoiding the cumulative effect of delays in transmitting still images.
(13)
The communication processing unit may transmit data in a Low Power Wide Area (LPWA) format.
With this configuration, by using a relatively low frequency, communication can be performed over a relatively long distance with a relatively small amount of power.
(14)
The electronic device may be a drive recorder.
With this configuration, the surroundings while driving can be recorded using the video captured and recorded by the drive recorder.
(15)
a receiving unit that receives the plurality of still images transmitted from the electronic device;
a video creation unit that combines the received still images to create a video;
A management server having:
This configuration reduces the burden on the electronic device of sending still images and eliminates the burden of creating videos. Furthermore, since the management server has relatively high processing specifications, it can create videos based on the transmitted still images. For example, the management server can create more advanced videos, such as videos with altered timelines or videos that supplement still images, compared to when videos are created on the electronic device.
(16)
When creating the moving image by integrating the received still images, the moving image creation unit may create a moving image that is played back faster than an actual chronological order.
By configuring in this way, it is possible to avoid giving the viewer an impression that the video is drawn out when played back, and to create a video that allows the viewer to quickly check the driving record, etc.
(17)
When creating the video from the received still images, if a time interval between the still images exceeds a predetermined threshold, the video creation unit:
The moving image may be created by duplicating still images from before the interval, or by generating new still images based on still images from before and after the interval.
By configuring in this way, it is possible to create a moving image that can avoid giving an unnatural impression to the viewer when the moving image is played back due to the occurrence of missing still images.
(18)
When a still image is duplicated or a new still image is generated, the video creation section may create the video displaying information indicating that the original image data does not exist.
By configuring in this way, it is possible to create a video that can be used when a traffic accident occurs or when checking a driving route, by clearly distinguishing between still images that were actually taken and those that were not.
(19)
the receiving unit receives, from the electronic device, the plurality of still images and position information of the vehicle at the time when the plurality of still images were created;
When the video creation unit creates a video by integrating the received multiple still images, it may embed the vehicle's location information in the video, thereby enabling the vehicle's location information to be displayed in synchronization with the video.
With this configuration, when the video is played back on the user terminal side, the vehicle position can be known using only the video.
(20)
A program running on at least one computer,
It may also be a program that enables the above method to be executed.
With this configuration, the above control can be performed by providing a program to an existing electronic device or a management server.

101...Drive recorder (electronic device or in-vehicle device), 102...Management server, 103...User terminal, 301...Microcomputer, 302...Communication processing unit, 303...DVR controller (control unit), 304...Camera (photographing unit), 350...Control unit

Claims (15)

An electronic device mounted on a moving body,
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
the control unit has a function of controlling transmission of the plurality of still images based on a traveling state of the moving object,
a function of selecting and transmitting a predetermined still image from among the plurality of captured still images when it is detected that the speed of the moving object is within a predetermined speed range;
The electronic device , wherein the predetermined speed is a speed when the mobile body is traveling at a low speed or a speed when the mobile body is stopped . An electronic device mounted on a moving body,
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
the control unit has a function of controlling transmission of the plurality of still images based on a transmission state of the still images ;
The control unit has a function of determining a transmission state of the communication processing unit based on the number of still images transmitted and a response time from transmission of the still images to completion of transmission. An electronic device mounted on a moving body,
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
the control unit has a function of controlling transmission of the plurality of still images based on a transmission state of the still images ;
The control unit is configured to determine the transmission state of the communication processing unit based on a response time of data that is transmitted separately from the still image and has a size smaller than the still image. An electronic device mounted on a moving body,
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
the control unit has a function of controlling transmission of the plurality of still images based on a transmission state of the still images ;
The electronic device wherein the control unit has a function of storing the subsequent still images in a storage device without transmitting them when the response time of two consecutive still images exceeds a predetermined threshold. 5. The electronic device according to claim 4, wherein the control unit has a function of transmitting the still image stored in the storage device when the number of still images being transmitted is 0 and the response time of data smaller than the still image that is transmitted separately from the still image is below a predetermined threshold . The electronic device according to claim 1 , wherein the communication processing unit has a function of transmitting data in a Low Power Wide Area (LPWA) format. The electronic device according to any one of claims 1 to 6 , wherein the electronic device is a drive recorder. a management server,
a communication processing unit;
a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit;
Equipped with
the control unit has a function of controlling transmission of the plurality of still images based on at least one of a running state of the mobile body and a transmission state of the still images; and a receiving unit that receives the plurality of still images transmitted from an electronic device mounted on the mobile body ;
a video creation unit that combines the received still images to create a video;
A management server having: a management server,
a receiving unit that receives the plurality of still images transmitted from the electronic device according to any one of claims 1 to 7 ;
a video creation unit that combines the received still images to create a video;
A management server having: The management server according to claim 8 , wherein the video creation unit creates a video that is played back faster than an actual timeline when creating the video by integrating the received still images. When creating the video from the received still images, if a time interval between the plurality of still images exceeds a predetermined threshold, the video creation unit:
creating the video by duplicating still images from before the interval, or creating new still images based on still images from before and after the interval;
10. The management server according to claim 8 or 9 . The management server according to claim 11, wherein the video creation unit creates the video displaying information indicating that the original image data is not available when the still image is duplicated or when the new still image is generated. the receiving unit receives, from the electronic device, the plurality of still images and location information of the moving object at the time when the plurality of still images were created;
The management server described in claim 8 or 9, wherein when the video creation unit integrates the received still images to create a video, it embeds the location information of the moving object in the video, thereby enabling the location information of the moving object to be displayed in synchronization based on the video . A program for causing a control unit of an electronic device to realize the functions described in any one of claims 1 to 7 . A program for causing a management server to function as the receiving unit and the video creating unit according to any one of claims 8 to 13 .