JP7343997B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

Conventionally, a technology has been proposed in which, for example, when a predetermined obstacle is detected from image data captured around a vehicle, the obstacle is displayed on an in-vehicle monitor in a pop-up display (for example, see Patent Document 1). .

JP2013-005233A

However, in the conventional technology, there is room for improvement in, for example, providing high value-added video.

The present invention has been made in view of the above, and an object of the present invention is to provide an image processing device and an image processing method that can provide images with high added value.

In order to solve the above-mentioned problems and achieve the purpose, an image processing device according to an embodiment includes an acquisition section and a detection section. The acquisition unit is provided in each moving body and acquires image data captured by a plurality of imaging devices having different imaging directions. The detection unit detects a detection target set for each imaging device from the image data.

According to the present invention, videos with high added value can be provided.

FIG. 1A is a diagram showing an overview of an image processing device. FIG. 1B is a diagram showing an overview of the image processing method. FIG. 2 is a block diagram of the image processing device. FIG. 3 is a diagram showing an example of a detection target. FIG. 4 is a diagram showing an example of the layout of the playback screen. FIG. 5 is a flowchart showing the processing procedure executed by the image processing device. FIG. 6 is a schematic diagram of an image processing device according to a modification.

Hereinafter, an image processing apparatus and an image processing method according to an embodiment will be described with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below. In addition, although the case where a moving object is a vehicle is demonstrated below, a moving object is not restricted to a vehicle, A moving object other than a vehicle, for example, a motorcycle, a ship, or an airplane may be sufficient as it.

First, an overview of an image processing apparatus and an image processing method according to an embodiment will be described using FIG. 1A and FIG. 1B. FIG. 1A is a diagram illustrating an example outline of an image processing device. FIG. 1B is a diagram showing an overview of the image processing method.

As shown in FIG. 1A, the image processing system S according to the embodiment includes an image processing device 1 and an in-vehicle device 50. The in-vehicle device 50 is, for example, a storage device that stores image data captured on a vehicle. In the example of FIG. 1A, the in-vehicle device 50 is connected to a front camera 10f and a rear camera 10b that captures an image of the rear of the vehicle, and stores image data captured by the front camera 10f and the rear camera 10b in respective internal storage media. Remember.

The front camera 10f is a camera that images the front of the vehicle, and the rear camera 10b is a camera that images the rear of the vehicle. In this embodiment, the front camera 10f and the rear camera 10b each generate image data at a predetermined frame rate. Note that the front camera 10f and the rear camera 10b each correspond to an example of an imaging device. Further, the imaging device is not limited to the front camera 10f and the rear camera 10b, but may include a side camera that captures an image of the side of the vehicle and an in-vehicle camera that captures an image of the inside of the vehicle. Furthermore, hereinafter, when there is no need to distinguish between the front camera 10f and the rear camera 10b, they may be referred to as the camera 10.

In the example shown in FIG. 1A, the image processing device 1 is a computer having so-called viewer software that reproduces image data acquired from the in-vehicle device 50. The image processing device 1 is an information processing device such as a mobile phone including a smartphone, a tablet terminal, a desktop PC, a notebook PC, or a PDA (Personal Digital Assistant). The image processing device 1 also includes a wearable device that is a glasses-type or watch-type information processing terminal.

For example, as shown in FIG. 1A, the image processing device 1 can acquire image data from the in-vehicle device 50 via the portable memory M or the network N.

By the way, conventionally, a technique has been proposed in which a predetermined obstacle is detected from image data captured by a camera mounted on a vehicle, and the obstacle is displayed in an emphasized manner. For example, in the conventional technology, only a predetermined obstacle is detected from image data and the user is notified, and there is room for improvement in providing a video with high added value.

Therefore, in the image processing method according to the embodiment, an object to be detected is set for each camera 10, and the detection object set for each camera 10 is detected from image data. That is, in the image processing method according to the embodiment, attention is paid to the fact that the object that the user wants to check differs depending on the imaging direction of the camera 10.

Specifically, as shown in FIG. 1B, for example, in the image processing method, a front image D1 captured by the front camera 10f and a rear image D2 captured by the rear camera 10b are simultaneously synchronized at their imaging times. Reproduce.

Then, as shown in FIG. 1B, when a target object set for each camera 10 is detected from the corresponding front image D1 or rear image D2, an emphasized image De in which the target object is highlighted is displayed. In the example shown in FIG. 1B, the target object T is a wagon displayed in the front image D1, and the target object T is enlarged and displayed as the emphasized image De, and additional information I regarding the target object T is displayed in a balloon. Indicate the case. Further, in the example of FIG. 1B, a case is shown in which the supplementary information I is a vehicle type or the like.

On the other hand, even if a wagon is shown in the rear image D2, if the wagon is not set as an object of the rear camera 10b, the wagon shown in the rear image D2 will not be a detection target.

In this way, in the image processing method according to the embodiment, the object to be detected that is set for each camera 10 is detected from the corresponding image data. Thereby, the target object can be detected according to the user's needs. Therefore, according to the image processing method according to the embodiment, a video with high added value can be provided.

Next, a configuration example of the image processing device 1 according to the embodiment will be described using FIG. 2. FIG. 2 is a block diagram of the image processing device 1. As shown in FIG. Note that FIG. 2 also shows the display device 100. The display device 100 is a display device that displays a video signal input from the image processing device 1. Furthermore, the display device 100 has an operation unit (not shown) and can receive user operations.

As shown in FIG. 2, the image processing device 1 according to the embodiment includes a control section 2 and a storage section 3. The control unit 2 includes an acquisition unit 21, a detection unit 22, a generation unit 23, and a reproduction unit 24. The storage unit 3 includes an image information storage unit 31 , a travel information storage unit 32 , a model information storage unit 33 , an additional information storage unit 34 , and an object information storage unit 35 .

Here, the control unit 2 includes, for example, a computer or various circuits having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input/output ports, etc. include.

The CPU of the computer functions as the acquisition section 21, the detection section 22, the generation section 23, and the reproduction section 24 of the control section 2, for example, by reading and executing a program stored in the ROM.

Further, the storage unit 3 corresponds to, for example, a RAM or an HDD. The RAM and HDD have an image information storage section 31, a travel information storage section 32, a model information storage section 33, an incidental information storage section 34, and an object information storage section 35, and can also store information on various programs. . Note that the image processing device 1 may acquire the above-mentioned programs and various information via another computer or a portable recording medium connected via a wired or wireless network.

The image information storage section 31 is a storage section that stores the above-mentioned front image D1 and rear image D2. For example, the front image D1 and the rear image D2 are stored in the image information storage unit 31 in chronological order.

The driving information storage unit 32 is a storage unit that stores driving information of the vehicle. The traveling information storage unit 32 stores, for example, information regarding the traveling speed of the vehicle, the acceleration acting on the vehicle, the steering angle of the vehicle, and the like.

The model information storage unit 33 is a storage unit that stores a model for detecting a target object. For example, the model information storage unit 33 stores a model for each object.

The supplementary information storage unit 34 is a storage unit that stores supplementary information regarding the object. The supplementary information storage section 34 stores supplementary information corresponding to the model in the model information storage section 33. The object information storage section 35 is a storage section that stores information regarding the object detected by the detection section 22. For example, the object information storage unit 35 stores information regarding image data in which a target object is detected, information regarding the above-mentioned emphasized image De, and the like.

The control unit 2 acquires image data from a plurality of cameras 10 having different imaging directions, and detects a target object set for each camera 10 from the image data. Furthermore, the control unit 2 generates an emphasized image De that highlights the detected object.

The acquisition unit 21 is provided in each moving object and acquires image data captured by a plurality of cameras 10 having different imaging directions. In this embodiment, as shown in FIG. 1A, the acquisition unit 21 acquires image data from the in-vehicle device 50 via the portable memory M or the network N, and transfers the acquired image data to the image information storage unit 31. Store in. Further, the acquisition unit 21 acquires vehicle travel information from the in-vehicle device 50 and stores the acquired travel information in the travel information storage unit 32 .

The detection unit 22 detects a target object set for each camera 10 from the image data acquired by the acquisition unit 21. For example, the detection unit 22 selects a model of the target set for each camera 10, Image data captured by the corresponding camera 10 is input.

The detection unit 22 then detects the object based on the score output from the model. Specifically, the detection unit 22 determines that the image data includes a target object when the above score is equal to or greater than a predetermined value.

FIG. 3 is a diagram showing a specific example of the object. “Camera type” shown in FIG. 3 indicates the type of camera 10, and “object” indicates an object set for each camera 10.

The example shown in FIG. 3 shows a case where the camera types are classified into a front camera 10f, a rear camera 10b, and a front camera 10f/rear camera 10b. The front camera 10f and the rear camera 10b indicate objects set to the front camera 10f and the rear camera 10b, respectively.

In addition, the front camera 10f/rear camera 10b is configured to move an object cross-sectionally from the image data of the front camera 10f to the image data of the rear camera 10b, or to cross-transversely move the image data from the image data of the rear camera 10b to the image data of the front camera 10f. Indicates a moving object. That is, the front camera 10f/rear camera 10b indicates the object that appears in the image data of both the front camera 10f and the rear camera 10b, respectively.

In this way, by detecting an object moving across image data captured by a plurality of cameras 10, it is possible to provide enhanced images De regarding the object viewed from various angles.

In the example shown in FIG. 3, the front camera 10f is set to include a sign, a swaying vehicle, a pedestrian, a fallen object, an obstacle, etc., and the rear camera 10b is set to a vehicle driving in a tail-swept manner, a falling object from the own vehicle, etc. This shows the case where is set as the target object.

Further, in the example shown in FIG. 3, a case is shown in which an emergency vehicle, a specific vehicle, a landmark, etc. are set as objects for the front camera 10f/rear camera 10b. Here, the emergency vehicle refers to an ambulance, a police car, or a fire engine, and the specific vehicle refers to a preset specific vehicle (light car, passenger car, truck, sedan, SUV, etc.). Note that the specific vehicle may be, for example, a specific model of vehicle. Furthermore, landmarks refer to tourist facilities, business facilities, and the like.

The objects for each camera 10 can be arbitrarily set by the user, for example. Moreover, the target object shown in FIG. 3 is an example, and it is also possible to change it arbitrarily. For example, additional conditions may be set as the object detection conditions, such as adding weather conditions or conditions related to the travel section.

Returning to the explanation of FIG. 2, the generation unit 23 will be explained. The generation unit 23 generates an emphasized image De in which the object detected by the detection unit 22 is emphasized. Specifically, the generation unit 23 generates an emphasized image De that enlarges and displays the object detected by the detection unit 22. The generation unit 23 also generates an emphasized image De for several seconds before and after the point where the object was detected by the detection unit 22.

For example, the generation unit 23 extracts a predetermined range including the object detected by the detection unit 22 from the image data, and enlarges the extracted range to generate the emphasized image De. Note that in this case, the generation unit 23 may extract only the target object by a predetermined trimming process and generate the emphasized image De.

Subsequently, the generation unit 23 generates an emphasized image De to which additional information I regarding the object is added, as shown in FIG. 1B. For example, the generation unit 23 acquires additional information I regarding the object from the additional information storage unit 34 and adds it to the emphasized image De. At this time, the generation unit 23 may change the attached information I to be provided depending on the user.

That is, it is also possible to change the supplementary information I, for example, based on the user's attributes (age, gender, etc.) and interests. In this case, appropriate additional information I can be provided by the user. Further, after generating the emphasized image De, the generating section 23 stores the emphasized image De in the object information storage section 35 .

In this way, the generation unit 23 generates the emphasized image De and the supplementary information I. By checking the emphasized image De and the supplementary information I, the user can easily understand the object.

The reproduction unit 24 simultaneously reproduces the front image D1 and the rear image D2 by synchronizing their imaging times, and also reproduces the emphasized image De. Specifically, the reproduction section 24 reads out the front image D1 and the rear image D2 from the image information storage section 31 at any time and reproduces them. Moreover, the reproduction unit 24 can reproduce the front image D1, the rear image D2, and the emphasized image De, respectively, by appropriately reading out the emphasized image De from the object information storage unit 35.

Note that the reproduction unit 24 can also reproduce only the emphasized image De. In this case, it is possible to display a list of information regarding the emphasized image De and reproduce the emphasized image De selected by the user.

FIG. 4 is a diagram showing an example of a playback screen. As shown in FIG. 4, for example, the playback screen D includes a front image area Rf, an enlarged area Rf2 corresponding to the front image D1, a rear image area Rb1, and an enlarged area Rb2 corresponding to the rear image D2.

Furthermore, the example in FIG. 4 shows a case where the playback screen D includes a tag area Rt, a map image area Rm, and a sensor image area Rs.

The front image area Rf is an area where the front image D1 is displayed, and the enlarged area Rf2 is an area where the emphasized image De of the object detected in the front image D1 is displayed.

Similarly, the rear image area Rb1 is an area where the rear image D2 is displayed, and the enlarged area Rb2 is an area where the emphasized image De of the object detected from the rear image D2 is displayed.

Further, the tag region Rt is, for example, a region in which tag information regarding the emphasized image De (object) is displayed. For example, by selecting the tag information displayed in the tag area Rt, the user can skip to the image data in which the target object was detected.

Further, the map image area Rm is an area where a map image indicating the imaging positions of the front image D1 and the rear image D2 being reproduced in the front image area Rf and the rear image area Rb is displayed. For example, the playback unit 24 generates a map image by reading information regarding the imaging position from the travel information storage unit 32, for example. Note that an icon is displayed at the position of the vehicle in the map image displayed in the map image area Rm.

The sensor image area Rs is an area for displaying a sensor image indicating the detection result of the vehicle sensor at the imaging time of the front image D1 and the rear image D2 reproduced in the front image area Rf and the rear image area Rb.

For example, as the sensor image, a speed image showing the traveling speed, an acceleration image showing the acceleration generated in the vehicle, etc. can be displayed in the sensor image area Rs. Note that the layout of the playback screen D shown in FIG. 4 is an example, and may be changed arbitrarily.

Next, a processing procedure executed by the image processing apparatus 1 according to the embodiment will be described using FIG. 5. FIG. 5 is a flowchart showing the processing procedure executed by the image processing device 1.

As shown in FIG. 5, the image processing device 1 first acquires image data (step S101), and then detects a target object set for each camera 10 (step S102). Subsequently, in step S102, the image processing device 1 determines whether or not there is a detected object (step S103).

In the determination process of step S103, if the image processing device 1 detects a target object (step S103, Yes), the image processing device 1 extracts supplementary information I from the supplementary information storage unit 34 for each detected target object (step S104). ), an emphasized image De is generated (step S105), and the process ends.

Moreover, in the determination process of step S103, if the image processing apparatus 1 has not detected the target object (step S103, No), the image processing apparatus 1 ends the process.

In the embodiment described above, the image processing device 1 includes the acquisition section 21 and the detection section 22. The acquisition unit 21 is provided in each vehicle (an example of a moving object) and acquires image data captured by a plurality of cameras 10 having different imaging directions. The detection unit 22 detects a target object set for each camera 10 from the image data. Therefore, according to the image processing device 1 according to the embodiment, a video with high added value can be provided.

Incidentally, in the embodiment described above, a case has been described in which the image processing device 1 acquires image data from the in-vehicle device 50, but image data may also be acquired from the camera 10. In this case, the image processing device 1 can also function as the in-vehicle device 50.

FIG. 6 is a schematic diagram of an image processing device according to a modification. As shown in FIG. 6, a front camera 10f and a rear camera 10b are connected to the image processing device 1A. In this case, the control unit 2-1 of the image processing device 1A acquires image data from the front camera 10f and the rear camera 10b, and detects the object set for each camera 10 from the image data.

The control unit 2-1 can then cause the storage unit 3-1 to store image data regarding the detected object. Here, storing in the storage unit 3-1 may mean storing only image data related to the object, or if the storage unit 3-1 is a ring buffer storage medium, This includes prohibiting overwriting of image data related to objects. In other words, it refers to storing the image data regarding the object so that it remains in the storage unit 3-1 without fail.

Therefore, in the image processing device 1A, the image data related to the object set for each camera 10 can be reliably stored in the storage unit 3-1, so that images with high added value can be provided.

Further advantages and modifications can be easily deduced by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments shown and described above. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

1 Image processing device 10 Camera 10f Front camera 10b Rear camera 21 Acquisition unit 22 Detection unit 23 Generation unit 24 Playback unit

Claims (2)

Obtaining image data captured by a plurality of imaging devices provided on one moving body and having different imaging positions,
detecting a target object settable by the user for each imaging device from the acquired image data ;
comprising a control unit that emphasizes the detected target object and generates an emphasized image with additional information regarding the target object;
The control unit includes:
Selecting the additional information stored in an additional information storage unit that stores the additional information based on the user's attributes or the user's interests, and changing the additional information included in the emphasized image.
An image processing device characterized by: Obtaining image data captured by a plurality of imaging devices provided on one moving body and having different imaging positions,
detecting a target object settable by the user for each imaging device from the acquired image data;
a control step of emphasizing the detected object and generating an emphasized image with additional information about the object;
including;
The control step includes:
Selecting the additional information stored in an additional information storage unit that stores the additional information based on the user's attributes or the user's interests, and changing the additional information included in the emphasized image.
An image processing method characterized by:

Images