JP7537188B2 - IMAGE RECOGNITION DEVICE, IMAGE RECOGNITION METHOD, COMPUTER PROGRAM, AND IMAGE RECOGNITION SYSTEM - Google Patents

Description

The present disclosure relates to an image recognition device, an image recognition method, a computer program, and an image recognition system.

Conventionally, a system has been proposed that recognizes road signs and the like from images of the surroundings of a moving object, such as an automobile, captured by a camera mounted on the moving object, and controls the traveling of the moving object based on the recognition results (see, for example, Patent Document 1).

JP 2019-220199 A

For controlling the travel of a moving object, it is necessary to accurately recognize objects in an image and to process the image without delay.

In order to process images without delay, it is necessary to extract an image of a region of interest that contains a specific object from the image, and then perform intensive object recognition processing on the image of the extracted region of interest.

However, images captured by a camera may become unclear due to the effects of night, rain, fog, dusk, backlight, shadows of temporary objects, power outages, etc. In such cases, objects such as road signs and traffic signals may appear unclear, and extraction of the image of the attention area may fail, such as not being able to extract the image of the attention area or extracting an image of an incorrect position. This may result in the object not being recognized accurately. In addition, if the image of the attention area cannot be extracted from the image, recognition processing is performed on the entire area of the image as a processing target, making it difficult to process the image without delay.
Such problems arise not only in automobiles, but also in the driving control of other moving objects such as mobile robots.

This disclosure has been made in light of these circumstances, and aims to provide an image recognition device, an image recognition method, a computer program, and an image recognition system that can accurately recognize objects in an image without delay.

An image recognition device according to one aspect of the present disclosure includes an acquisition unit that acquires an attention area in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores an attention area in an image in association with the shooting position of the image, a cutout unit that cuts out an image of the attention area from the image of the surroundings of the moving body, and a recognition processing unit that performs a predetermined recognition process based on the image of the attention area.

An image recognition method according to another aspect of the present disclosure includes the steps of acquiring an attention area in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image, cutting out an image of the attention area from the image of the surroundings of the moving body, and performing a predetermined recognition process based on the image of the attention area.

A computer program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires an area of interest in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores an area of interest in an image in association with the shooting position of the image, a cutout unit that cuts out an image of the area of interest from the image of the surroundings of the moving body, and a recognition processing unit that performs a predetermined recognition process based on the image of the area of interest.

An image recognition system according to another aspect of the present disclosure includes a camera mounted on a moving object, an acquisition unit that acquires an area of interest in an image of the surroundings of the moving object captured by the camera based on the position of the moving object from a storage unit that associates and stores an area of interest in the image with the shooting position of the image, an extraction unit that extracts an image of the area of interest from the image of the surroundings of the moving object, and a recognition processing unit that performs a predetermined recognition process based on the image of the area of interest.

It goes without saying that the computer program can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet. The present disclosure can also be realized as a semiconductor integrated circuit that realizes part or all of the image recognition device.

According to this disclosure, objects in an image can be recognized accurately and without delay.

FIG. 1 is a diagram showing an overall configuration of a driving assistance system according to a first embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example of a configuration of a control system according to the first embodiment of the present disclosure. FIG. 3 is a block diagram showing the functional configuration of the processor. FIG. 4 is a diagram showing an example of attention area information stored in the server. FIG. 5 is a diagram showing an example of a region of interest. FIG. 6 is a flowchart of a process executed by the control system according to the first embodiment of the present disclosure. FIG. 7 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the first embodiment of the present disclosure. FIG. 8 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the second embodiment of the present disclosure. FIG. 9 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the third embodiment of the present disclosure.

[Summary of the embodiment of the present disclosure]
First, an overview of the embodiments of the present disclosure will be listed and described.
(1) An image recognition device according to one embodiment of the present disclosure includes an acquisition unit that acquires an area of interest in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores an area of interest in an image in association with the shooting position of the image, an extraction unit that extracts an image of the area of interest from the image of the surroundings of the moving body, and a recognition processing unit that performs a predetermined recognition processing based on the image of the area of interest.

According to this configuration, information on the area of interest corresponding to the position of the moving object is acquired from the storage unit, and an image of the area of interest indicated by the acquired information is cut out from the image. In addition, a predetermined recognition process is executed based on the cut-out image of the area of interest. For example, if the moving object is a mobile robot that travels autonomously within a factory, the positions of display devices, road signs, factory corridors, etc. within the factory do not change. Therefore, the area of interest in images captured at the same position is the same. Therefore, by acquiring information on the area of interest from the storage unit, it is possible to accurately cut out the image of the area of interest. This allows objects in the image to be recognized accurately and without delay.

(2) Preferably, the device further includes an extraction unit that extracts a predetermined region of interest from an image of the surroundings of the moving object, a determination unit that determines whether the extraction unit has succeeded in extracting the region of interest, and a selection unit that selects at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a determination result of the determination unit, and the cutout unit cuts out an image of the region of interest selected by the selection unit.

According to this configuration, if extraction of the area of interest from an image fails, information on the area of interest corresponding to the position of the moving object is acquired from the storage unit, and an image of the area of interest indicated by the acquired information is cut out from the image. In addition, a predetermined recognition process is executed based on the image of the cut-out area of interest. For example, if the moving object is a mobile robot that travels autonomously within a factory, the positions of display devices, road signs, factory passageways for pedestrians, etc. within the factory do not change. Therefore, the area of interest in images taken at the same position is the same. Therefore, by acquiring information on the area of interest from the storage unit, it is possible to accurately cut out an image of the area of interest even if extraction of the area of interest fails due to reasons such as the image being unclear. This allows objects in the image to be recognized accurately and without delay.

(3) More preferably, the determination unit determines whether the extraction of the region of interest was successful based on the result of the extraction of the region of interest.

According to this configuration, for example, if it is not possible to extract a region of interest from an image, it can be determined that extraction of the region of interest has failed.

(4) The determination unit may also determine whether the extraction of the region of interest has been successful based on a degree of confidence indicating the likelihood of the extraction of the region of interest.

According to this configuration, for example, if the certainty of the region of interest is lower than a predetermined standard, it can be determined that extraction of the region of interest has failed. As a result, if the certainty of the extracted region of interest is equal to or greater than the predetermined standard, recognition processing is performed based on the image of the region of interest. Also, if the certainty of the extracted region of interest is less than the predetermined standard, recognition processing is performed based on the image of the region of interest corresponding to the position of the moving object. This allows the object in the image to be accurately recognized.

(5) The determination unit may also determine whether the extraction of the region of interest was successful based on abnormal state information at the time of capturing the image.

With this configuration, for example, if only a blurred image is obtained outdoors due to abnormal weather conditions such as rain or snow, or if only a blurred image is obtained indoors (e.g., in a factory) due to an abnormal condition such as a power outage or disaster (e.g., smoke caused by a fire), it can be determined that extraction of the attention area has failed. As a result, in normal conditions such as when it is sunny or when no abnormalities have occurred, recognition processing is performed based on the image of the attention area extracted. Also, in abnormal conditions such as when it is raining, snowing, or a disaster, recognition processing is performed based on the image of the attention area corresponding to the position of the moving object. As a result, objects in the image can be accurately recognized.

(6) The selection unit may further select at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a selection instruction from a user.

According to this configuration, the area of interest is selected by the user. This allows the area of interest to be switched depending on the situation. For example, if the user determines that the recognition accuracy of the recognition process based on the image of the area of interest extracted by the extraction unit has decreased, it is possible to switch to recognition processing based on the image of the area of interest corresponding to the position of the moving object.

(7) The cropping unit may further crop out an image of a non-attention area including an area other than the attention area from the image, and the image recognition device may further include a reduction unit that reduces the amount of data of the image of the non-attention area, and the recognition processing unit may perform the predetermined recognition processing based on the image of the attention area and the image of the non-attention area with the amount of data reduced.

According to this configuration, a data reduction process such as thinning is performed on the image of the non-attention area, and then a predetermined recognition process is executed. The non-attention area is an area other than the object to be recognized, such as the sky or buildings on the roadside. This reduces the processing load of the recognition process on the non-attention area, and allows the image to be processed without delay.

(8) The storage unit may further store the image in association with the time when the image was captured, and the acquisition unit may acquire the area of interest from the storage unit based on the position of the moving body and the current time.

With this configuration, for example, it is possible to obtain an area of interest extracted from an image captured at the time closest to the current time. As a result, for example, when a mobile body is traveling on a reversible lane whose traveling direction changes depending on the time of day, it is not necessary to obtain an area of interest extracted from an image captured by a mobile body traveling on the reversible lane at a time when the traveling direction is reversed. Also, for example, when the current time is nighttime, it is possible to preferentially obtain an area of interest extracted from an image captured during the day. This makes it possible to obtain an accurate area of interest.

(9) The storage unit may further store the image in association with the size of the image, and the acquisition unit may further acquire the size of the image corresponding to the area of interest, and correct the acquired area of interest based on the acquired size and the size of the image captured by the camera.

With this configuration, even if the size of the image captured by the camera mounted on the moving object differs from the size of the image stored in the storage unit that was used to extract the area of interest, it is possible to extract an image of the area of interest at an accurate position.

(10) The storage unit may further store the image in association with an angle of view, and the acquisition unit may further acquire an angle of view of the image corresponding to the region of interest, and correct the acquired region of interest based on the acquired angle of view and the angle of view of the image captured by the camera.

With this configuration, even if the angle of view of the image captured by the camera mounted on the moving object differs from the angle of view of the image that was used to extract the area of interest stored in the storage unit, it is possible to extract an image of the area of interest at an accurate position.

(11) An image recognition method according to another embodiment of the present disclosure includes the steps of acquiring an area of interest in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the area of interest in an image in association with the shooting position of the image, cutting out an image of the area of interest from the image of the surroundings of the moving body, and performing a predetermined recognition process based on the image of the area of interest.

This configuration includes steps that are characteristic of the image recognition device described above. Therefore, this configuration can achieve the same effects and advantages as the image recognition device described above.

(12) A computer program according to another embodiment of the present disclosure causes a computer to function as an acquisition unit that acquires an area of interest in an image of the surroundings of a moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores an area of interest in an image in association with the shooting position of the image, an extraction unit that extracts an image of the area of interest from the image of the surroundings of the moving body, and a recognition processing unit that performs a predetermined recognition process based on the image of the area of interest.

This configuration allows the computer to function as the image recognition device described above. This allows the same actions and effects to be achieved as the image recognition device described above.

(13) An image recognition system according to another embodiment of the present disclosure includes a camera mounted on a moving body, an acquisition unit that acquires an area of interest in an image of the surroundings of the moving body captured by the camera from a storage unit that associates and stores the area of interest in the image with the shooting position of the image, based on the position of the moving body, a cut-out unit that cuts out an image of the area of interest from the image of the surroundings of the moving body, and a recognition processing unit that performs a predetermined recognition process based on the image of the area of interest.

According to this configuration, information on the area of interest corresponding to the position of the moving object is acquired from the storage unit, and an image of the area of interest indicated by the acquired information is cut out from the image. In addition, a predetermined recognition process is executed based on the cut-out image of the area of interest. For example, if the moving object is a mobile robot that travels autonomously within a factory, the positions of display devices, road signs, factory corridors, etc. within the factory do not change. Therefore, the area of interest in images captured at the same position is the same. Therefore, by acquiring information on the area of interest from the storage unit, it is possible to accurately cut out the image of the area of interest. This allows objects in the image to be recognized accurately and without delay.

[Details of the embodiment of the present disclosure]
Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are examples and do not limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims are components that can be added arbitrarily. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.
In addition, the same components are denoted by the same reference numerals, and since their functions and names are also the same, their explanations will be omitted as appropriate.

<Embodiment 1>
[Overall configuration of driving assistance system]
FIG. 1 is a diagram showing an overall configuration of a driving assistance system according to a first embodiment of the present disclosure.

Referring to FIG. 1, the driving assistance system 1 includes a plurality of mobile objects 2 traveling on a road where wireless communication is possible, one or more base stations 6 that wirelessly communicate with the mobile objects 2, and a server 4 that communicates with the base stations 6 via a network 5 such as the Internet in a wired or wireless manner.

Base station 6 includes a macrocell base station, a microcell base station, and a picocell base station.

The mobile object 2 is, for example, a mobile robot such as a transport robot that transports luggage while autonomously traveling within a factory, or a monitoring robot that monitors a factory while autonomously traveling within the factory. In this embodiment, the mobile object 2 is a mobile robot. However, the mobile object 2 is not limited to a mobile robot that travels within a factory. The mobile object 2 includes, for example, not only normal passenger cars that travel on roads, but also public vehicles such as route buses and emergency vehicles. Furthermore, the mobile object 2 may be not only a four-wheeled vehicle, but also a two-wheeled vehicle (motorcycle).

Each moving body 2 is equipped with a control system 3 including a camera as described below, and transmits the position of the area of interest in an image obtained by capturing an image of the surroundings of the moving body 2 with the camera, and at least the camera's shooting position and shooting direction, to the server 4 via the network 5. The camera's shooting position is the same as the position of the moving body 2 when the image was captured by the camera. In this embodiment, the optical axis of the camera is assumed to face forward of the moving body 2. For this reason, the traveling direction of the moving body 2 is assumed to be the shooting direction of the camera.

In this embodiment, the attention area is an area that includes information important for the autonomous driving of the mobile body 2, and is an area that includes static information or quasi-static information. For example, the attention area is an area that includes, as static information, display devices in a factory, road signs, factory passageways, and the like that are constantly present. In addition, the attention area is an area that includes, as quasi-static information, temporary road construction sites, traffic accident sites, and the like.

The server 4 receives sets of positions of areas of interest and associated data from multiple moving bodies 2 via the network 5, and stores them as area of interest information in the storage unit of the server 4.

The server 4 receives at least the position information of the moving body 2 from the control system 3 via the network 5, and transmits the position information of the area of interest corresponding to the position of the moving body 2 to the control system 3 via the network 5.

The control system 3 receives position information of the attention area from the server 4 via the network 5. Based on the received position information of the attention area, the control system 3 cuts out an image of the attention area from the image captured by the control system 3. The control system 3 performs a predetermined recognition process by applying a predetermined image processing to the cut-out image of the attention area. For example, the control system 3 recognizes an object included in the image of the attention area and performs processing according to the recognition result. As an example, if the object is a road sign indicating a stop, the control system 3 performs braking control of the moving object 2 to safely stop the moving object 2.

[Configuration of moving body 2]
FIG. 2 is a block diagram showing an example of the configuration of the control system 3 according to the first embodiment of the present disclosure.

As shown in FIG. 2, the control system 3 of the moving object 2 includes a camera 31, a communication unit 32, a clock 33, a control unit (ECU: Electronic Control Unit) 34, a GPS (Global Positioning System) receiver 37, a gyro sensor 38, and a speed sensor 39.

The camera 31 is composed of an image sensor that captures images of the surroundings of the moving body 2 (particularly, the area in front of the moving body 2). The camera 31 is monocular. However, the camera 31 may be compound-eyed. The images are composed of multiple images in a time series.

The communication unit 32 is, for example, a wireless communication device capable of communication processing compatible with 5G (fifth generation mobile communication system). The communication unit 32 may be a wireless communication device already installed in the mobile body 2, or may be a mobile terminal brought to the mobile body 2 by the passenger.

The passenger's mobile terminal is temporarily connected to the vehicle's in-vehicle LAN (Local Area Network) and becomes an in-vehicle wireless communication device.

Clock 33 keeps track of the current time.

The control unit 34 is a computer device that controls each of the devices 31-33, 37-39 of the mobile unit 2. The control unit 34 determines the mobile unit position of the mobile unit itself from the GPS signals periodically acquired by the GPS receiver 37. The control unit 34 may complement the GPS signal or correct the mobile unit position of the mobile unit itself by also using a GPS complement signal or a GPS augmentation signal received by a receiver of a signal transmitted from a quasi-zenith satellite (not shown). The mobile unit position preferably has a positional accuracy sufficient to distinguish between lanes.

The control unit 34 complements the position and direction of the moving body 2 based on the input signals of the gyro sensor 38 and the speed sensor 39, and grasps the accurate current position and direction of the moving body 2. Here, the current position of the moving body 2 is indicated, for example, by latitude and longitude. In addition, the direction (travel direction) of the moving body 2 is indicated, for example, by an angle ranging from 0 degrees to 360 degrees clockwise, with north being 0 degrees.

The GPS receiver 37, gyro sensor 38, and speed sensor 39 are sensors that measure the current position, direction, and speed of the moving body 2.

The control unit 34 includes a processor 35 and a memory 36 .
The processor 35 is an arithmetic processing device such as a microcomputer that executes a computer program stored in the memory 36 .

Memory 36 is composed of a volatile memory element such as SRAM (Static RAM) or DRAM (Dynamic RAM), a non-volatile memory element such as flash memory or EEPROM (Electrically Erasable Programmable Read Only Memory), or a magnetic storage device such as a hard disk. Memory 36 stores computer programs executed by control unit 34 and data generated when computer programs are executed by control unit 34.

[Functional configuration of processor 35]
FIG. 3 is a block diagram showing the functional configuration of the processor 35. As shown in FIG.

Referring to FIG. 3, the processor 35 includes, as functional processing units realized by executing a computer program stored in the memory 36, an image acquisition unit 51, an area of interest extraction unit 52, an extraction success/failure determination unit 53, an area of interest acquisition unit 54, a moving object position determination unit 55, an area of interest selection unit 56, a cropping unit 57, a data amount reduction unit 58, and a recognition processing unit 59.

The image acquisition unit 51 sequentially acquires images of the area ahead of the moving body 2 captured by the camera 31 in chronological order. The image acquisition unit 51 sequentially outputs the acquired images to the attention area extraction unit 52 and the cropping unit 57.

The attention area extraction unit 52 acquires an image from the image acquisition unit 51 and extracts an attention area from the acquired image. For example, the attention area extraction unit 52 acquires an attention area by inputting the acquired image into a learning model that has been machine-learned using the image and the attention area in the image as learning data. The attention area extraction unit 52 also acquires a confidence level indicating the likelihood of the attention area from the learning model. The learning model is, for example, a CNN (Convolution Neural Network), an RNN (Recurrent Neural Network), an AutoEncoder, or the like, and each parameter of the learning model is determined by a machine learning method such as deep learning.

In this embodiment, the region of interest is a rectangle. The region of interest is defined by the coordinates of the upper left corner of the rectangle and the widths of the rectangle in the X and Y directions. However, the position of the region of interest is not limited to the above. For example, the region of interest may be defined by the coordinates of the upper left corner and the coordinates of the lower right corner of the rectangle. The region of interest may also be a shape other than a rectangle, such as an ellipse.

The attention area extraction unit 52 outputs the extracted attention area and the confidence level to the extraction success/failure determination unit 53 and the attention area selection unit 56.

The extraction success/failure determination unit 53 determines whether the extraction of the region of interest by the region of interest extraction unit 52 has been successful. Specifically, the extraction success/failure determination unit 53 determines whether the extraction of the region of interest has been successful based on the confidence level extracted by the region of interest extraction unit 52. For example, the extraction success/failure determination unit 53 determines that the extraction of the region of interest has been successful if the confidence level is equal to or greater than a predetermined threshold value TH, and determines that the extraction of the region of interest has failed if the confidence level is less than the predetermined threshold value TH.

When an image contains multiple regions of interest, the success or failure of extraction of the regions of interest may be determined based on multiple confidence levels. For example, the extraction success or failure determination unit 53 may determine that extraction of the region of interest has been successful if the smallest confidence level among the multiple confidence levels is equal to or greater than a predetermined threshold TH, and may determine that extraction of the region of interest has failed if the smallest confidence level is less than the predetermined threshold TH. Alternatively, the average of the multiple confidence levels may be used instead of the smallest confidence level. The extraction success or failure determination unit 53 may determine the success or failure of extraction for each region of interest.

The extraction success/failure determination unit 53 outputs the determination result to the attention area extraction unit 52, the attention area acquisition unit 54, the moving object position determination unit 55, and the attention area selection unit 56.

When it is determined that the extraction of the area of interest has been successful, the area of interest extraction unit 52 transmits the position of the area of interest and at least the shooting position and shooting direction of the camera 31 to the server 4 via the communication unit 32. The server 4 stores the information received from the moving object 2 as area of interest information 41.

The attention area acquisition unit 54 acquires an attention area from the attention area information 41 stored in the server 4 based on the position of the mobile object 2 determined by the mobile object position determination unit 55 described later.

FIG. 4 is a diagram showing an example of attention area information 41 stored in the server 4. As shown in FIG.
The attention area information 41 includes a moving object ID, a shooting time, a shooting position, an attention area, a moving direction, an angle of view, and an image size. Each moving object 2 uploads this information to the server 4.

The mobile object ID is identification information of the uploaded mobile object 2. The mobile object ID may be an ID unique to the mobile object 2 or an ID unique to a user of the mobile object 2.
The photographing time indicates the photographing time of the camera 31 mounted on the moving object 2 .
The photographing position is the photographing position of the camera 31, and is equal to the position of the moving object 2 when the image is photographed. The photographing position is indicated by, for example, north latitude N and east longitude E.

The region of interest indicates the position (X, Y, W, H) of the region of interest extracted from the image, where X and Y indicate the X and Y coordinates of the upper left corner of the region of interest, respectively, and W and H indicate the widths of the region of interest in the X and Y directions, respectively.
The moving direction indicates the traveling direction of the moving body 2 .
The angle of view indicates the angle of view of the camera 31 .

The image size indicates the size of the image captured by the camera 31. For example, 4K indicates 3840 x 2160 (pixels), and 8K indicates 7680 x 4320 (pixels).

For example, five areas of interest extracted from an image captured by moving object 2 with moving object ID "C001" at shooting position (N1, E1) at shooting time "9:00:03" are (X1, Y1, W1, H1), (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4), and (X5, Y5, W5, H5). The direction of moving object 2 is 45°, the angle of view of the image is 60°, and the image size is 4K.

The area of interest extracted from an image captured by moving object 2 with moving object ID "C001" at shooting position (N2, E2) at shooting time "9:01:45" is (X6, Y6, W6, H6). The direction of moving object 2 is 48°, the angle of view of the image is 60°, and the image size is 4K.

The area of interest extracted from an image captured by moving object 2 with moving object ID "C002" at shooting position (N1, E1) at shooting time "20:02:55" is (X7, Y7, W7, H7). The direction of moving object 2 is 45°, the angle of view of the image is 120°, and the image size is 8K.

Figure 5 is a diagram showing an example of a region of interest. Figure 5 shows a rectangular region of interest on an image captured by camera 31. For example, the region of interest (X1, Y1, W1, H1) is set to include a road sign, and the regions of interest (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4) and (X5, Y5, W5, H5) are set to include a factory passageway.

Referring again to FIG. 3, for example, when the position of the moving body 2 determined by the moving body position determination unit 55 is (N2, E2), the attention area acquisition unit 54 acquires an attention area (X6, Y6, W6, H6) corresponding to the shooting position (N2, E2) from the attention area information 41.

However, the attention area acquisition unit 54 may acquire the attention area from the attention area information 41 based on the position of the moving object 2 and the current time. For example, when the position of the moving object 2 determined by the moving object position determination unit 55 is (N1, E1), the attention area information 41 includes the attention areas (X1, Y1, W1, H1), (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4), and (X5, Y5, W5, H5) at the shooting time "9:00:03" as the attention area corresponding to the shooting position (N1, E1), and the attention area (X7, Y7, W7, H7) at the shooting time "20:02:55". Therefore, the attention area acquisition unit 54 acquires from the attention area information 41 the attention area corresponding to the time closest to the current time among these. For example, when the current time is "9:30:00", the attention area acquisition unit 54 acquires attention areas (X1, Y1, W1, H1), (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4), and (X5, Y5, W5, H5) corresponding to the shooting time "9:00:03". When the current time is "19:00:00", the attention area acquisition unit 54 acquires attention area (X7, Y7, W7, H7) corresponding to the shooting time "20:02:55".

It is preferable that the attention area acquisition unit 54 acquires the attention area by considering the direction of the moving body 2 in addition to the position of the moving body 2. This makes it possible to acquire attention area information obtained from a moving body 2 traveling in the same direction without acquiring attention area information obtained from a moving body 2 traveling in the opposite direction, even if they are in the same position. However, if the moving body 2 is traveling on a one-way road or if the traveling lane of the moving body 2 can be identified from the position of the moving body 2, it is not necessary to consider the direction of the moving body 2.

The mobile object position determination unit 55 determines the position of the mobile object 2. Specifically, the mobile object position determination unit 55 obtains the position of the mobile object 2 from the GPS signal periodically acquired by the GPS receiver 37. The mobile object position determination unit 55 complements the position and direction of the mobile object 2 based on the input signals of the gyro sensor 38 and the speed sensor 39, and grasps the accurate current position and direction of the mobile object 2. The mobile object position determination unit 55 also performs map matching processing to associate the obtained position of the mobile object 2 with a position on the road, with reference to map information 61 pre-stored in the memory 36, thereby determining the accurate position of the mobile object 2.

The attention area selection unit 56 selects at least one of the attention area extracted by the attention area extraction unit 52 and the attention area acquired by the attention area acquisition unit 54 based on the determination result of the extraction success/failure determination unit 53.

For example, if it is determined that the extraction of the region of interest has been successful, the region of interest selection unit 56 selects the region of interest extracted by the region of interest extraction unit 52, and if it is determined that the extraction of the region of interest has failed, it selects the region of interest acquired by the region of interest acquisition unit 54. Note that if it is determined that the extraction of the region of interest has been successful, the region of interest selection unit 56 may select both the region of interest extracted by the region of interest extraction unit 52 and the region of interest acquired by the region of interest acquisition unit 54.

The cropping unit 57 crops out an image of the region of interest (region of interest image) and an image outside the region of interest (region of non-interest image) from the image acquired by the image acquisition unit 51. The cropping unit 57 outputs the region of interest image to the recognition processing unit 59 and outputs the region of non-interest image to the data amount reduction unit 58.

The data amount reduction unit 58 performs a process of reducing the data amount of the non-attention area image. For example, the data amount reduction unit 58 may perform a downscaling process to reduce the size of the non-attention area image, or may reduce the number of bits indicating the luminance value of each pixel of the non-attention area image to reduce the gradation (color depth). The data amount reduction unit 58 may also perform a temporal thinning process of the non-attention area image (for example, a process of deleting non-attention areas obtained from even-numbered images in a chronological order). The data amount reduction unit 58 outputs the non-attention area image after the data amount reduction process to the recognition processing unit 59.

The recognition processing unit 59 executes a predetermined recognition process based on the attention area image received from the cutout unit 57 and the non-attention area image received from the data amount reduction unit 58. For example, the recognition processing unit 59 performs a predetermined image processing on the attention area image to recognize a display device in a factory, a stop road sign, the presence or absence of a pedestrian on a factory passage, and the like. The recognition result of the recognition processing unit 59 is used, for example, for controlling the automatic driving of the mobile body 2. For example, the recognition processing unit 59 obtains a recognition result of the attention area image by inputting the attention area image into a learning model that has been machine-learned using the image and a label indicating the recognition result as learning data. The learning model is, for example, a CNN, an RNN, an AutoEncoder, or the like, and each parameter of the learning model is determined by a method such as deep learning.
The recognition processing unit 59 may also execute the same process on non-attention area images.

[Processing flow of control system 3]
FIG. 6 is a flowchart of a process executed by the control system 3 according to the first embodiment of the present disclosure.
The image acquisition unit 51 acquires an image from the camera 31 (step S1).
The attention area extraction unit 52 extracts an attention area and a certainty factor of the attention area based on the image acquired by the image acquisition unit 51 (step S2).
The extraction success/failure determination unit 53 determines whether extraction of the region of interest has been successful (step S3).

FIG. 7 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the first embodiment of the present disclosure.
The extraction success/failure determination unit 53 determines whether the certainty of the region of interest extracted by the region of interest extraction unit 52 is equal to or greater than a predetermined threshold value TH (step S21).
If the certainty is equal to or greater than the threshold value TH (YES in step S21), the extraction success/failure determination unit 53 determines that extraction of the region of interest has been successful (step S22).
If the certainty is less than the threshold value TH (NO in step S21), the extraction success/failure determination unit 53 determines that extraction of the region of interest has failed (step S23).

If multiple regions of interest are extracted in step S2, the processes of steps S21 to S23 are executed based on, for example, the smallest degree of certainty. If no regions of interest are extracted, it is determined that extraction of the regions of interest has failed.

Referring again to Figure 6, if it is determined that the extraction of the area of interest has been successful (YES in step S4), the area of interest extraction unit 52 transmits area of interest information including the position of the area of interest and at least the shooting position and shooting direction of the camera 31 to the server 4 via the communication unit 32 (step S5).
Moreover, the attention region selection unit 56 selects the attention region extracted in step S2, and outputs it to the cut-out unit 57 (step S6).

If it is determined that extraction of the attention area has failed (NO in step S4), the moving object position determination unit 55 determines the position and direction of the moving object 2 (step S7).
Based on the determined position and direction of the moving object 2, the attention area acquisition unit 54 reads out attention area information corresponding to the position and direction from the server 4 (step S8).

The attention area acquisition unit 54 corrects the attention area based on the read attention area information (step S9). For example, if the image size of the camera 31 mounted on the target moving body 2 is 4K and the image size indicated by the read attention area information is 8K, the attention area values indicated by the attention area information are halved to correct them to values of the attention area of the 4K image. Similarly, if the angle of view of the camera 31 mounted on the target moving body 2 is 45° and the angle of view indicated by the attention area information is 60°, the attention area acquisition unit 54 converts the value of the attention area indicated by the attention area information to a value on an image with an angle of view of 45°. The conversion process is performed, for example, using a conversion table that indicates the correspondence between positions between different angles of view.

The attention area selection unit 56 selects the attention area acquired in step S8 and corrected in step S9, and outputs it to the cropping unit 57 (step S10).

The cutout unit 57 cuts out an image of an area of interest from the image acquired in step S1 (step S11). The cutout unit 57 also cuts out an image of an area of non-interest from the image acquired in step S1 (step S12).

The data amount reduction unit 58 performs a data amount reduction process on the non-attention area image cut out in step S12 (step S13).

The recognition processing unit 59 performs recognition processing of the attention area image and the non-attention area image based on the attention area image cut out in step S11 and the non-attention area image subjected to data volume reduction processing in step S13, and performs driving control processing of the mobile body 2 based on the recognition result (step S14).

[Effects of the First Embodiment]
As described above, according to the first embodiment of the present disclosure, information on the attention area corresponding to the position of the moving object 2 is acquired from the server 4, and an image of the attention area indicated by the acquired information is cut out from the image. In addition, a predetermined recognition process is executed based on the cut-out image of the attention area. For example, the positions of display devices, road signs, factory passages, etc. in a factory do not change. Therefore, the attention area in an image taken at the same position is the same. Therefore, by acquiring information on the attention area from the server 4, the image of the attention area can be accurately cut out. This allows the object in the image to be accurately recognized without delay.

Furthermore, if extraction of the area of interest from the image fails, information on the area of interest corresponding to the position of the moving body 2 is obtained from the server 4, and an image of the obtained area of interest is cut out from the image. This makes it possible to accurately cut out an image of the area of interest even if extraction of the area of interest fails due to, for example, the image being unclear.

The extraction success/failure determination unit 53 determines that extraction of the area of interest has failed if the certainty of the area of interest is less than the threshold TH, and determines that extraction of the area of interest has been successful if the certainty is equal to or greater than the threshold TH. As a result, if the certainty of the extracted area of interest is equal to or greater than the threshold TH, recognition processing is performed based on the image of the area of interest. Also, if the certainty of the extracted area of interest is less than the threshold TH, recognition processing is performed based on the image of the area of interest corresponding to the position of the moving body 2. This allows the object in the image to be accurately recognized.

In addition, a process to reduce the amount of data, such as thinning, is performed on the non-attention area images before a predetermined recognition process is performed. Non-attention areas are areas other than the object to be recognized, such as the sky or buildings on the roadside. This reduces the processing load of the recognition process on non-attention areas, and allows images to be processed without delay.

In addition, since the attention area acquisition unit 54 acquires the attention area based on the position of the moving body 2 and the current time, it is possible to acquire, for example, an attention area extracted from an image captured at the time closest to the current time. This prevents, for example, when the moving body 2 is traveling on a reversible lane whose traveling direction changes depending on the time of day, acquisition of an attention area extracted from an image captured by another moving body 2 traveling on the reversible lane at a time when the traveling direction is opposite. Also, for example, if the current time is nighttime, it is possible to acquire preferentially an attention area extracted from an image captured during the day. Images are clearer during the day than at night. This allows an accurate attention area to be acquired.

The attention area acquisition unit 54 also corrects the attention area acquired from the server 4 based on the image size. Therefore, even if the size of the image captured by the camera 31 mounted on the moving body 2 differs from the size of the image stored in the server 4 that was used to extract the attention area, it is possible to cut out an image of the attention area at an accurate position.

The attention area acquisition unit 54 also corrects the attention area acquired from the server 4 based on the angle of view. Therefore, even if the angle of view of the image captured by the camera 31 mounted on the moving body 2 differs from the angle of view of the image stored in the server 4 that was used to extract the attention area, it is possible to cut out an image of the attention area at an accurate position.

<Embodiment 2>
In a second embodiment of the present disclosure, another example of a method for determining whether extraction of a region of interest has been successful will be described.
The configuration of the driving assistance system 1 is the same as that of the first embodiment. The following description will focus on the differences from the first embodiment.

FIG. 8 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the second embodiment of the present disclosure.
The extraction success/failure determination unit 53 determines whether or not there is an extraction region extracted by the attention region extraction unit 52 (step S31).

If any region of interest has been extracted (YES in step S31), the extraction success/failure determination unit 53 determines that the extraction of the region of interest has been successful (step S32).
If no region of interest has been extracted (NO in step S31), the extraction success/failure determination unit 53 determines that extraction of the region of interest has failed (step S33).

According to embodiment 2 of the present disclosure, for example, if a region of interest cannot be extracted from an image, it can be determined that extraction of the region of interest has failed.

<Embodiment 3>
In a third embodiment of the present disclosure, another example of a method for determining whether extraction of a region of interest has been successful will be described.
The control system 3 is connected via a network 5 to a weather server (not shown) that provides meteorological information for the position of the mobile unit 2 .
Therefore, the extraction success/failure determination unit 53 determines whether the extraction of the attention area has been successful based on meteorological information about the position of the moving body 2 at the time the image is captured by the camera 31 .

FIG. 9 is a flowchart showing details of the extraction success/failure determination process (step S3 in FIG. 6) according to the third embodiment of the present disclosure.
The extraction success/failure determination unit 53 acquires meteorological information about the position of the moving object 2 at the time when the image is captured by the camera 31 from the weather server via the communication unit 32 (step S41).

If the acquired weather information indicates rain or snow (YES in step S41), the extraction success/failure determination unit 53 determines that extraction of the region of interest has failed (step S43).

If the acquired weather information indicates something other than rain or snow (NO in step S42), the extraction success/failure determination unit 53 determines that the extraction of the region of interest has been successful (step S44).

According to embodiment 3 of the present disclosure, for example, when only a blurred image is obtained outdoors due to abnormal weather conditions such as rain or snow, it can be determined that extraction of the attention area has failed. As a result, when it is sunny, recognition processing is performed based on the image of the extracted attention area. Also, when it is raining or snowing, recognition processing is performed based on the image of the attention area corresponding to the position of the moving body 2. As a result, objects in the image can be accurately recognized.

[Additional Notes]
Although the driving assistance system 1 according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

For example, the attention area selection unit 56 may select at least one of the attention area extracted by the attention area extraction unit 52 and the attention area acquired by the attention area acquisition unit 54 based on a selection instruction from the user. This allows the attention area to be switched depending on the situation. For example, if the user determines that the recognition accuracy of the recognition process based on the image of the attention area extracted by the attention area extraction unit 52 has decreased, it is possible to switch to recognition process based on the image of the attention area corresponding to the position of the moving body 2.

The extraction success/failure determination unit 53 may also determine whether the extraction of the area of interest has been successful based on the time at which the image was captured by the camera 31. For example, the extraction success/failure determination unit 53 may determine that the extraction of the area of interest has failed if the image was captured at night, and may determine that the extraction of the area of interest has been successful if the image was captured during the day.

The extraction success/failure determination unit 53 may also determine whether the extraction of the area of interest has been successful based on the image captured by the camera 31 and acquired by the image acquisition unit 51. For example, the extraction success/failure determination unit 53 may detect an abnormal condition such as a power outage or disaster (e.g., smoke generated by a fire) indoors (e.g., in a factory) by performing recognition processing on the image, and determine that the extraction of the area of interest has failed if an abnormal condition has occurred. The extraction success/failure determination unit 53 may also determine that the extraction of the area of interest has been successful if no abnormal condition has occurred.

Furthermore, the object on which the control system 3 is mounted is not limited to a moving object such as a mobile robot. For example, the control system 3 may be mounted on other moving objects such as automobiles and drones.

Furthermore, the purpose of the recognition processing performed by the recognition processing unit 59 is not limited to automatic driving. For example, the control system 3 may be mounted on a mobile robot that moves within a factory and used for monitoring purposes to prevent workers from engaging in dangerous acts.
Furthermore, some or all of the components constituting the control system 3 may be configured from one or more semiconductor devices such as a system LSI.

The above-mentioned computer program may be distributed by recording it on a computer-readable non-transitory recording medium, such as a HDD, a CD-ROM, a semiconductor memory, etc. In addition, the computer program may be distributed by transmitting it via a telecommunication line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.
Furthermore, at least some of the above-described embodiments and modifications may be combined in any manner.

The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims.

1 Driving assistance system 2 Vehicle 3 Control system (image recognition system)
4. Server (storage unit)
5 Network 6 Base station 31 Camera 32 Communication unit 33 Clock 34 Control unit 35 Processor (image recognition device)
36 Memory 37 GPS receiver 38 Gyro sensor 39 Speed sensor 41 Attention area information 51 Image acquisition unit 52 Attention area extraction unit (extraction unit)
53 Extraction success/failure determination unit (determination unit)
54 Attention area acquisition unit (acquisition unit)
55 Moving object position determination unit 56 Attention area selection unit (selection unit)
57 Cutting unit 58 Data amount reduction unit (reduction unit)
59 Recognition processing unit 61 Map information

Claims (12)

an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
An extraction unit that extracts a predetermined area of interest from an image of the surroundings of the moving object;
a determination unit that determines whether extraction of the region of interest by the extraction unit has been successful;
a selection unit that selects at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a determination result of the determination unit;
a cutout unit that cuts out an image of the region of interest selected by the selection unit from an image of the surroundings of the moving object;
and a recognition processing unit that performs a predetermined recognition process based on the image of the cut-out area of interest. The image recognition device according to claim 1 , wherein the determination unit determines whether extraction of the region of interest has been successful based on a result of extraction of the region of interest. The image recognition device according to claim 1 , wherein the determination unit determines whether extraction of the region of interest has been successful based on a degree of certainty indicating a likelihood of extraction of the region of interest. The image recognition device according to claim 1 , wherein the determination unit determines whether the extraction of the attention area is successful or not based on abnormal state information when an image of the surroundings of the moving body is captured by a camera mounted on the moving body . 5. The image recognition device according to claim 1 , wherein the selection unit further selects at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a selection instruction by a user. An image recognition device,
an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
a cutout unit that cuts out an image of the region of interest from an image of the surroundings of the moving object;
a recognition processing unit that performs a predetermined recognition process based on the image of the attention area,
The cropping unit further crops out an image of a non-attention region including a region other than the attention region from the image,
The image recognition device further includes a reduction unit that reduces an amount of data of the image of the non-attention area,
The recognition processing unit performs the predetermined recognition processing based on an image of the region of interest and an image of the non-region of interest with a reduced amount of data. an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
a cutout unit that cuts out an image of the region of interest from an image of the surroundings of the moving object;
a recognition processing unit that performs a predetermined recognition process based on the image of the attention area,
The storage unit further stores the images stored in the storage unit in association with the times at which the images were captured ;
The acquisition unit acquires the area of interest from the storage unit based on the position of the moving object and the current time. an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
a cutout unit that cuts out an image of the region of interest from an image of the surroundings of the moving object;
a recognition processing unit that performs a predetermined recognition process based on the image of the attention area,
The storage unit further stores the image in association with a size of the image,
The acquisition unit further acquires a size of the image corresponding to the area of interest, and corrects the acquired area of interest based on the acquired size and the size of the image captured by the camera. an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
a cutout unit that cuts out an image of the region of interest from an image of the surroundings of the moving object;
a recognition processing unit that performs a predetermined recognition process based on the image of the attention area,
The storage unit further stores the image in association with an angle of view of the image,
The acquisition unit further acquires an angle of view of the image corresponding to the area of interest , and corrects the acquired area of interest based on the acquired angle of view and the angle of view of the image captured by the camera. acquiring an area of interest in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the area of interest in the image in association with the shooting position of the image;
Extracting a predetermined area of interest from an image of the surroundings of the moving object;
a step of determining whether extraction of the region of interest has been successful in the step of extracting the region of interest;
selecting at least one of the extracted region of interest and the acquired region of interest based on a result of determining whether extraction of the region of interest has been successful;
cutting out an image of the selected region of interest from an image of the surroundings of the moving object;
and performing a predetermined recognition process based on the extracted image of the region of interest. Computer,
an acquisition unit that acquires an attention area in an image of the surroundings of the moving body captured by a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the attention area in the image in association with the shooting position of the image;
An extraction unit that extracts a predetermined area of interest from an image of the surroundings of the moving object;
a determination unit that determines whether extraction of the region of interest by the extraction unit has been successful;
a selection unit that selects at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a determination result of the determination unit;
a cutout unit that cuts out an image of the region of interest selected by the selection unit from an image of the surroundings of the moving object;
A computer program for causing the computer to function as a recognition processing unit that performs a predetermined recognition process based on the image of the cut-out area of interest. A camera mounted on a moving object;
an acquisition unit that acquires an attention area in an image of the surroundings of the moving object captured by the camera based on the position of the moving object from a storage unit that stores the attention area in the image in association with the shooting position of the image;
An extraction unit that extracts a predetermined area of interest from an image of the surroundings of the moving object;
a determination unit that determines whether extraction of the region of interest by the extraction unit has been successful;
a selection unit that selects at least one of the region of interest extracted by the extraction unit and the region of interest acquired by the acquisition unit based on a determination result of the determination unit;
a cutout unit that cuts out an image of the region of interest selected by the selection unit from an image of the surroundings of the moving object;
and a recognition processing unit that performs a predetermined recognition process based on the image of the cut-out area of interest.

Images