JP7777472B2 - Repair Support System - Google Patents

Description

The present invention relates to a repair assistance system that assists in repair work on mobile objects, particularly vehicles.

A vehicle repair assistance system is known that reduces the burden on vehicle repair workers while enabling them to properly repair vehicles (see, for example, Patent Document 1). The vehicle repair assistance system in Patent Document 1 includes a communications terminal that acquires vehicle information from the vehicle, and a server that receives the vehicle information from the communications terminal and assists in vehicle repair.

The server interviews the repair requester about the vehicle's condition, including product classification, symptoms, detailed symptoms, a summary of the problem, the environment, and the circumstances at the time of the problem. Based on the interview results, the server searches for information about similar cases with symptoms similar to those of the vehicle being repaired, and notifies the communication terminal of the search results. This allows the worker to use the communication terminal to obtain detailed information about the selected similar case.

Japanese Patent Application Laid-Open No. 2014-225216

However, there is a desire to realize a sustainable transportation system that takes into consideration the most vulnerable people among transportation participants. To this end, vehicles are now being equipped with a variety of sensors, such as radar and lidar, to further improve traffic safety.

However, there are cases where the sensor is covered by an exterior material and acquires external information through a transparent area set in the exterior material. For example, if the sensor is a radar, such a transparent area is defined as the area through which radio waves emitted from the radar pass. Therefore, workers cannot visually determine the location or area of the transparent area. Repairing such a transparent area using methods other than those specified could impair the function of the sensor, so there is a strong demand for technology that allows workers to easily identify the transparent area.

In light of the above background, the present invention provides a repair work support system that assists repair work on mobile objects equipped with sensors that acquire external information through transparent areas set in exterior components, and aims to enable workers to easily grasp the transparent areas. This will ultimately contribute to the development of sustainable transportation systems by improving the safety of mobile objects.

In order to solve the above problem, one aspect of the present invention is a repair assistance system (1, 51) for assisting in the repair of a mobile object (3) equipped with a sensor (5) that acquires external information through a transparent area set in an exterior member (7). The system includes an imaging device (27) that acquires an image (28) of the mobile object's exterior surface, a display device (29, 69) that can present to a worker an image (3A) of the mobile object with an image showing a specific area of the exterior surface superimposed thereon, a storage device (25, 37) that stores specific information (38, 38A, 38B, 38C) for identifying a portion of the captured image that corresponds to the transparent area, and a processor (21, 33) that processes the captured image. The processor identifies a portion of the captured image that corresponds to the transparent area based on the specific information, and causes the display device to display the portion (8A) corresponding to the transparent area superimposed on the image (3A) of the mobile object.

According to this aspect, the portion corresponding to the transparent area is superimposed on the image of the moving object and displayed on the display device. Therefore, even if the sensor is covered by an exterior member and not directly exposed to the outside, the worker can easily grasp the position and range of the transparent area of the exterior member.

In the above aspect, preferably, the display device (29) displays the captured image by superimposing the portion corresponding to the transparent region.

According to this aspect, workers can easily grasp the position and range of the transparent area by visually checking the display device.

In the above aspect, preferably, the display device (69) is configured to be able to superimpose information onto a transparent lens, and the display device displays the image of the moving object visible through the lens by superimposing the portion corresponding to the transparent area.

In this embodiment, by visually observing the moving object through the lens, the worker can easily grasp the position and range of the transparent area.

In the above aspect, preferably, the processor extracts an image of an abnormal area that may reduce the detection sensitivity of the sensor from the captured image, and the display device displays an image of the moving object by superimposing the portion corresponding to the transparent area and an icon (39) indicating the abnormal area.

This allows the operator to easily identify any abnormalities.

In the above aspect, preferably, the processor identifies the cause of the abnormality corresponding to the abnormal location based on the captured image, and the display device displays the cause of the abnormality.

This allows the operator to easily identify the cause of the abnormality.

In the above aspect, the abnormality factors preferably include scratches and dirt.

This aspect makes it possible to properly identify abnormalities that may reduce the sensor's detection sensitivity.

In the above aspect, preferably, the processor determines whether the abnormal part can be repaired based on the captured image, and if it is determined that the abnormal part cannot be repaired, the display device notifies the worker that replacement is necessary (45), and if it is determined that the abnormal part can be repaired, the display device notifies the worker that repair is required (43).

This aspect improves the efficiency of repair work.

In the above aspect, preferably, the display device updates and displays the portion corresponding to the transparent area in accordance with updates to the captured image.

According to this aspect, even if the worker moves while capturing images, the display device updates and displays the portion corresponding to the transparent area in real time. This allows the worker to more easily understand the position and range of the transparent area and grasp it more accurately.

In the above aspect, preferably, the processor identifies the portion corresponding to the transparent area based on the surface shape of the exterior member.

This aspect allows for more accurate identification of the portion corresponding to the transmission area compared to when it is not based on the surface shape of the exterior member.

In the above aspect, preferably, when the processor cannot identify a portion of the captured image that corresponds to the transparent region, it notifies the worker (40) to take an image.

According to this aspect, if the portion corresponding to the transparent region cannot be identified using the captured image, the worker can be prompted to capture an image that will enable identification.

In the above aspect, preferably, the processor identifies the portion corresponding to the transparent area based on the positions of multiple components mounted on the moving body in addition to the exterior component.

This aspect allows the portion corresponding to the transparent area to be identified more accurately.

The above aspects provide a repair work assistance system that assists repair work on a mobile object equipped with a sensor that acquires external information through a transparent area set in an exterior member, allowing the worker to easily grasp the transparent area.

FIG. 1 is a schematic diagram illustrating an overview of a repair assistance system according to a first embodiment; Repair support system block diagram 1 shows an example of a screen display on a terminal in a repair support system, in which (A) a notification of a repairable abnormality and (B) a notification of an unrepairable abnormality are displayed on a mobile device. Vehicle information table example Flowchart of abnormality identification process Flowchart of area data generation process Example of a screen display with arrows indicating the direction of movement FIG. 10 is a perspective view of a terminal of a repair support system according to a second embodiment; Example of screen display of a terminal of the repair support system according to the second embodiment

Hereinafter, with reference to the drawings, we will explain an embodiment in which the mobile object repair assistance system of the present invention is applied, in particular, to a system for assisting in the repair of vehicles, i.e., a vehicle repair assistance system.

<<First Embodiment>>
A vehicle (hereinafter referred to as a vehicle to be repaired 3) to be repaired by a worker assisted by the vehicle repair assistance system 1 is a four-wheeled automobile and is equipped with a sensor 5. The sensor 5 acquires external information such as the position, size, direction, and distance of objects around the vehicle to be repaired 3 relative to the vehicle to be repaired 3. The sensor 5 may be, for example, a radar that emits radio waves toward a predetermined irradiation area 6 (also referred to as a detection area) and measures the radio waves reflected from objects located within the irradiation area 6 and incident on the sensor 5, thereby detecting the position and size of objects within the irradiation area 6.

As shown in FIG. 1, an exterior member 7 may be provided on the vehicle exterior side of the sensor 5. In this embodiment, a bumper is provided as the exterior member 7 on the vehicle exterior side of the sensor 5. In this case, the sensor 5 acquires external information through a transmission area 8 set in the exterior member 7. The transmission area 8 corresponds to the overlapping portion between the irradiation area 6 from which the sensor 5 emits radio waves (or the area where radio waves are incident on the sensor 5) and the exterior member 7.

If an exterior member 7 is installed on the exterior side of the vehicle and the sensor 5 is covered by the exterior member 7, the worker cannot easily grasp the position of the sensor 5 or the transmission area 8.

When repairing a vehicle 3 to be repaired, in which the sensor 5 is covered by an exterior member 7, repairing the transparent area 8 in a manner other than that specified could impair the function of the sensor 5, so the worker must accurately understand the transparent area 8. The vehicle repair assistance system 1 is a system that assists workers in repairing a vehicle 3 to be repaired, which is equipped with a sensor 5 that acquires external information through the transparent area 8, by making the transparent area 8 easier for workers to understand.

As shown in Figures 1 and 2, the vehicle repair assistance system 1 includes a terminal 11 and a server 13.

As shown in FIG. 2, the terminal 11 includes a processor (CPU, central processing unit; hereinafter referred to as terminal processor 21), a main memory 23 such as ROM or RAM, a storage device 25 (flash memory, etc.), an imaging device 27, and a touch panel 29. The terminal 11 is configured to be able to communicate with the server 13 via a network 31 such as the Internet. The terminal 11 may be a smartphone or a tablet terminal.

The imaging device 27 includes a single or multiple cameras 27A. As shown in Figures 3(A) and (B), the imaging device 27 captures an image of the exterior of the vehicle 3 to be repaired, thereby obtaining an image (captured image 28) including an image 3A of the vehicle 3 to be repaired. In this embodiment, the imaging device 27 is composed of multiple cameras 27A (compound camera).

The touch panel 29 functions as an input/output device that includes a display device that displays images, text, etc., and an input device that accepts input. The touch panel 29 can display predetermined patterns and text superimposed on the captured image 28 acquired by the imaging device 27. For example, when the vehicle 3 to be repaired is reflected in the captured image 28, the touch panel 29 can superimpose an image showing a specific area on the exterior of the vehicle 3 to be repaired on the image 3A of the vehicle 3 to be repaired, thereby presenting the position and size of that area to the worker.

The processor of the terminal 11 (hereinafter referred to as the terminal processor 21) can display input fields, etc. on the touch panel 29 and accept input from the user.

The terminal processor 21 displays an input field on the touch panel 29 and receives information related to the vehicle 3 to be repaired from the worker. The information received by the terminal processor 21 from the worker (hereinafter referred to as repair vehicle information) includes the vehicle name (the vehicle's unique name), model, etc. of the vehicle 3 to be repaired. The repair vehicle information may also include the manufacturer's name, vehicle number, registration date, etc. The terminal processor 21 transmits the received repair vehicle information to the server 13.

The terminal processor 21 transmits the captured image 28 acquired by the imaging device 27 to the server 13. In this embodiment, the terminal processor 21 is configured to be able to transmit the captured image 28 acquired by the imaging device 27 to the server 13 in real time. In addition, the terminal processor 21 can display images and text data transmitted from the server 13 on the touch panel 29. The terminal processor 21 may also have a function to use three-dimensional data transmitted from the server 13 and superimpose it on the captured image 28 for display.

As shown in FIG. 2, the server 13 is a computer equipped with a processor (CPU, MPU, etc., also called a processing unit; hereinafter referred to as the server processor 33), main memory 35 such as ROM and RAM, and a storage device (HDD or SSD, also called storage; hereinafter referred to as the server storage device 37).

The server storage device 37 stores identification information 38 for identifying the portion of the captured image 28 that corresponds to the transparent area 8. The identification information 38 includes a vehicle information table 38A, a vehicle appearance data group 38B, and a transparent area data group 38C. As shown in FIG. 4, the vehicle information table 38A records the vehicle name, model, vehicle appearance data name, and overlap area data name in association with each other.

The vehicle exterior data group 38B includes vehicle exterior data. The vehicle exterior data is data used to estimate the position and attitude of the imaging device 27 from images acquired by the imaging device 27 and determine the position and attitude of the terminal 11. The vehicle exterior data may be, for example, data indicating the exterior surface of the vehicle with the center or center of gravity of the vehicle as the origin, or may also be data indicating the positions of multiple feature points.

The transparent area data group 38C includes transparent area data. This transparent area data indicates the irradiation range of the bumper (exterior member 7) through which radio waves emitted from the sensor 5 mounted on the vehicle penetrate, for a vehicle of the name and model corresponding to the transparent area data name in the vehicle information table 38A. In other words, the transparent area data indicates the overlapping portion of the sensor 5's detection area and the bumper, i.e., the position, size, and range of the transparent area 8. The server storage device 37 stores the transparent area data as data (model data) including a three-dimensional point sequence that indicates the shape of an object in three-dimensional space. However, the server storage device 37 may also store the transparent area data as data indicating a point cloud that constitutes the outer surface of the overlapping portion of the sensor 5's detection area and the bumper.

When the server processor 33 acquires the repair vehicle information from the terminal 11, it extracts the vehicle name and model included in the repair vehicle information. The server processor 33 then references the vehicle information table 38A stored in the server storage device 37 and extracts the corresponding transparent area data name. The server processor 33 then identifies the transparent area data corresponding to the transparent area data name from the transparent area data group 38C, i.e., the transparent area data indicating the transparent area 8 set on the exterior member 7 of the vehicle 3 to be repaired.

When the server processor 33 acquires repair vehicle information from the terminal 11, it further refers to the vehicle information table 38A to acquire vehicle exterior data corresponding to the vehicle 3 to be repaired.

Then, when the server processor 33 acquires the captured image 28 from the terminal 11, it executes, in parallel, an abnormality identification process that processes the captured image 28 to identify the abnormal location, the cause of the abnormality, etc., and an area data generation process that processes the captured image 28 to superimpose on the captured image 28 an image showing the transparent area 8 set in the exterior member 7. Below, details of the abnormality identification process will first be described with reference to Figure 5.

In the first step ST1 of the anomaly identification process, the server processor 33 determines whether or not there are any abnormalities, such as scratches or dirt on the vehicle body, from the captured image 28 acquired by the imaging device 27. The server processor 33 may determine whether or not there are any abnormalities by extracting abnormalities from the image captured by the imaging device 27 based on a known machine learning method using a pre-constructed learning model. The server processor 33 may also determine whether or not there are any abnormalities by performing enhancement processing, filtering processing, etc. on the image captured by the imaging device 27, and extracting abnormalities based on, for example, shading or shape, etc.

If the server processor 33 cannot identify any abnormalities from the captured image 28 acquired by the imaging device 27, it may send a request signal to the terminal 11 requesting further images. When the terminal 11 receives the request signal, the terminal processor 21 displays a notification on the touch panel 29 urging the worker to take further images using the imaging device 27. At this time, the touch panel 29 of the terminal 11 may display, for example, "The image is unclear. Please take another image." When the worker takes another image using the imaging device 27, the terminal processor 21 transmits the captured image 28 acquired by the imaging device 27 to the server processor 33. The server processor 33 then determines whether or not any abnormalities exist based on the newly transmitted captured image 28.

If an abnormality is found, the server processor 33 proceeds to step ST2; if no abnormality is found, the server processor 33 ends the abnormality identification process.

In step ST2, the server processor 33 identifies the location of the abnormality and generates data (hereinafter, "abnormality location data") for displaying an icon 39 indicating the location and size of the abnormality on the touch panel 29 of the terminal 11. The icon 39 may be, for example, a frame surrounding the abnormality, and the abnormality location data may be numerical data indicating the shape and position of the frame corresponding to the icon 39 in the image captured by the imaging device 27. Once acquisition of the abnormality location data is complete, the server processor 33 executes step ST3.

In step ST3, the server processor 33 analyzes the captured images 28 acquired by the imaging device 27 and identifies the abnormality factor corresponding to each abnormal location. Abnormality factors include scratches, dirt, etc. The server processor 33 may identify the abnormality factor for each abnormal location based on a known machine learning method using a pre-constructed learning model. Alternatively, the server processor 33 may perform enhancement processing or filtering on the images acquired by the imaging device 27 and identify the factor corresponding to the abnormal location based on shading, shape, etc. The server processor 33 identifies the abnormality factor for each abnormal location and acquires abnormality factor data indicating the abnormality factor for each abnormal location.

In this embodiment, if the abnormality is a scratch or stain, the server processor 33 calculates a level (hereinafter referred to as the abnormality level) indicating the degree of the abnormality (for example, the depth of the scratch or the ease of removing the stain).

Once the abnormality cause and level have been acquired for each abnormality location, the server processor 33 executes step ST4.

In step ST4, the server processor 33 extracts abnormal locations where the cause of the abnormality is a defect, and if an abnormal location where the cause of the abnormality is a defect is extracted, it determines whether the abnormal location can be repaired. For example, when an abnormal location where the cause of the abnormality is a defect is extracted, the server processor 33 may determine whether the abnormal location can be repaired based on the abnormality level of the abnormal location. The server processor 33 acquires data indicating whether the abnormal location can be repaired (hereinafter referred to as repairability data) for each abnormal location where the cause of the abnormality is a defect. Once acquisition of the repairability data has been completed, the server processor 33 executes step ST5.

In step ST5, the server processor 33 transmits the abnormality location data, abnormality cause data, and repair possibility data to the terminal 11. Once transmission is complete, the server processor 33 ends the abnormality identification process.

Next, the area data generation process will be explained in detail with reference to Figure 6.

In step ST11, the server processor 33 determines the position and orientation of the terminal 11 relative to the vehicle 3 to be repaired. In this embodiment, the server processor 33 determines the position and orientation of the terminal 11 based on the vehicle exterior data and the captured image 28 acquired by the imaging device 27.

Specifically, the server processor 33, for example, identifies an image of the exterior component 7 of the vehicle 3 to be repaired from an image acquired by the imaging device 27, and acquires its surface shape. The server processor 33 then identifies the position and attitude of the terminal 11 by comparing the acquired surface shape with the vehicle exterior data.

In addition, the server processor 33 may determine an image of the vehicle 3 to be repaired and extract feature points from the image. The server processor 33 may then identify the position and orientation of the terminal 11 by comparing the positions of the acquired feature points with the positions of feature points included in the vehicle exterior data. The server processor 33 may identify the position and orientation of the terminal 11 based on the positions of multiple components mounted on the vehicle 3 to be repaired, in addition to the exterior components 7, as appropriate.

In this embodiment, if the server processor 33 cannot determine the position and orientation of the terminal 11 from the image acquired by the imaging device 27, it transmits a request signal to the terminal 11 requesting further images. When the terminal 11 receives the request signal, the terminal processor 21 displays a notification 40 on the touch panel 29 prompting the terminal 11 to take further images using the imaging device 27. At this time, the touch panel 29 of the terminal 11 may display, for example, "Please take a photo of the car to be repaired from a different angle," as shown in FIG. 7(A). When the worker again captures an image using the imaging device 27, the terminal processor 21 transmits the captured image to the server processor 33. The server processor 33 then determines the position and orientation of the terminal 11 based on the newly transmitted image.

Once the position and orientation of the terminal 11 have been determined, the server processor 33 executes step ST12.

In step ST12, the server processor 33 reads the transparent area data identified based on the repair vehicle information. Then, based on the position and attitude of the terminal 11 identified in step ST11, the server processor 33 determines whether the transparent area 8 is located within the range imaged by the imaging device 27. If the transparent area 8 is located within the range imaged by the imaging device 27, step ST13 is executed; if it is not located within the range, step ST14 is executed.

In step ST13, the server processor 33 generates an image (hereinafter, "transparent area image 8A") showing a portion corresponding to the transparent area 8 in the captured image 28 acquired by the imaging device 27 based on the position and orientation of the terminal 11 and the transparent area data. The transparent area image 8A may be frame-shaped, and the inside of the frame may be colored. The server processor 33 may generate the transparent area image 8A as point data for constructing a frame corresponding to the transparent area image 8A. Once generation of the transparent area image 8A is complete, the server processor 33 executes step ST15.

In step ST14, the server processor 33 acquires data (hereinafter, movement direction data) indicating the movement direction of the terminal 11 required to include the transparent area 8 within the imaging area of the imaging device 27. The server processor 33 may acquire the movement direction data based on the position and attitude of the terminal 11 and the transparent area data. Once acquisition of the movement direction data is complete, the server processor 33 transmits the movement direction data to the terminal 11. Once transmission of the movement direction data is complete, the server processor 33 terminates the area data generation process.

In step ST15, the server processor 33 transmits the transmission area image 8A to the terminal 11. Once transmission is complete, the server processor 33 ends the area data generation process.

Upon receiving the abnormality location data, abnormality cause data, and repair feasibility data, the terminal processor 21 displays on the touch panel 29, based on the abnormality location data, an image captured by the imaging device 27, particularly an image 3A of the vehicle 3 to be repaired, with an icon 39 indicating the abnormality superimposed on it. Based on the abnormality cause data, the terminal processor 21 acquires the abnormality cause corresponding to each abnormality location displayed on the touch panel 29, and displays information indicating the abnormality cause on the touch panel 29 using text 41, graphics, etc. Furthermore, based on the repair feasibility data, the terminal processor 21 displays whether each abnormality location displayed on the touch panel 29 where the abnormality cause is a defect can be repaired. In this embodiment, for example, for an abnormality location that can be repaired, the terminal processor 21 displays a notification 43 instructing repair (also referred to as repair), as shown in FIG. 3(A). For an abnormality location that cannot be repaired, the terminal processor 21 displays a notification 45 on the touch panel 29 indicating the need for replacement, as shown in FIG. 3(B). When contact with an icon 39 indicating an abnormality displayed on the touch panel 29 is detected, the terminal processor 21 may display a corresponding notification 43, 45. Furthermore, as shown in FIG. 3(A), when the terminal processor 21 displays notification 43 instructing repair, it may also display the repair method (repair method).

When the terminal processor 21 acquires the transmission area image 8A, it superimposes a frame corresponding to the transmission area image 8A on the captured image 28 acquired by the imaging device 27 to generate a composite image, and displays the composite image on the touch panel 29. When the terminal processor 21 acquires abnormal area position data, it may simultaneously superimpose a frame corresponding to the transmission area image 8A and an icon 39 indicating the abnormal area on the captured image 28 acquired by the imaging device 27 and display them on the touch panel 29.

As shown in FIG. 7(B), when the terminal processor 21 acquires movement direction data, it displays a notification 47 on the touch panel 29 indicating the movement direction corresponding to the movement direction data. The movement direction notification 47 may be, for example, an arrow indicating the movement direction corresponding to the movement direction data. In this embodiment, the terminal processor 21 may superimpose an arrow indicating the movement direction on the captured image 28 along with the text "Please move in the direction of the arrow."

Next, we will explain the operation and effects of the vehicle repair assistance system 1 configured in this manner.

When a worker holds the terminal 11, takes an image of the vehicle 3 to be repaired, and sends it to the server 13, the server processor 33 starts the abnormality identification process and the area data generation process.

In the abnormality identification process, the server processor 33 determines whether or not there are any abnormalities, such as scratches or dirt on the vehicle body, from the images acquired by the imaging device 27, and if there are any abnormalities, it transmits abnormality location data, abnormality cause data, and repair possibility data to the terminal 11.

Furthermore, in the area data generation process, the server processor 33 identifies the position and orientation of the terminal 11, and if a transparent area 8 is included in the imaging area of the imaging device 27, transmits a transparent area image 8A to the terminal 11.

Upon receiving the abnormality location data, abnormality cause data, repairability data, and transparent area image 8A, the terminal 11 (terminal processor 21) superimposes an icon 39 indicating the abnormality and the transparent area image 8A on the image captured by the imaging device 27, and displays it on the touch panel 29, as shown in Figures 3(A) and (B). That is, the server processor 33 and terminal processor 21 work together to identify the portion of the captured image 28 that corresponds to the transparent area 8 based on the identification information 38, and superimposes the transparent area image 8A, which is the portion corresponding to the transparent area 8, on the image 3A of the vehicle 3 to be repaired in the captured image 28, and displays it on the touch panel 29.

As a result, an image of the transparent area 8 (transparent area image 8A), which cannot be seen with the naked eye in the real world, is superimposed on the image 3A of the vehicle 3 to be repaired in the captured image 28 and displayed on the touch panel 29. Therefore, even if the sensor 5 is covered by the exterior member 7 and not directly exposed to the outside, the worker can easily identify the part of the exterior member 7 that is within the detection range of the sensor 5 by visually checking the display on the touch panel 29.

In this way, the vehicle repair assistance system 1 uses virtual reality technology (AR (Augmented Reality) technology) to superimpose areas that cannot be seen in the real world on the captured image 28 and display them on the touch panel 29. This allows the worker to grasp the transparent area 8 without having to manually mark the vehicle 3 to be repaired to indicate the radar radio wave irradiation range (transparent area 8). This significantly reduces the time required for repair work. Furthermore, because the transparent area 8 is displayed by calculation, the transparent area 8 can be presented to the worker more accurately than if it were marked manually. Furthermore, because it is configured using a smartphone, the vehicle repair assistance system 1 can be configured relatively inexpensively without using a separate imaging device 27, etc.

In addition, the server processor 33 extracts images of abnormal areas that may reduce the detection sensitivity of the sensor 5 from the captured image 28, and an icon 39 indicating the abnormal area is displayed on the touch panel 29 superimposed on the captured image 28. Therefore, the worker can easily identify the abnormal area by checking the display on the touch panel 29.

Furthermore, the server processor 33 acquires the cause of the abnormality corresponding to the abnormal location from the captured image 28, and information (text 41) indicating the cause of the abnormality is displayed on the touch panel 29. The display on the touch panel 29 allows the worker to easily understand the cause of the abnormality at the abnormal location. This reduces the burden on the worker performing the repair. Furthermore, because the cause of the abnormality includes scratches and dirt, the worker can understand whether the cause of the abnormality at the abnormal location is scratches or dirt. This allows the worker to understand the display on the touch panel 29 to identify abnormal locations such as scratches and dirt that reduce the detection sensitivity of the sensor 5, and to take appropriate action against the abnormal location.

The terminal processor 21 displays on the touch panel 29 text 41 indicating the cause of the abnormality, along with whether or not each abnormality location where the cause of the abnormality is a defect can be repaired. The display on the touch panel 29 allows the worker to understand whether or not the location can be repaired, improving the efficiency of repair work.

For abnormal parts that can be repaired, the terminal processor 21 displays a notification 43 instructing repair, and for abnormal parts that cannot be repaired, the touch panel 29 displays a notification that replacement is required. This allows workers to easily understand the work that needs to be done, further improving work efficiency.

The terminal processor 21 is configured to be able to transmit images captured by the imaging device 27 to the server 13 in real time. Therefore, when the worker moves while holding the terminal 11 and taking images, the terminal processor 21 transmits the images captured by the imaging device 27 to the server 13 in accordance with the worker's movement. When the server 13 acquires images from the terminal 11, the server processor 33 transmits abnormality location data, abnormality cause data, repairability data, and transmission area image 8A to the terminal 11 in real time.

As a result, the touch panel 29 of the terminal 11 displays the transmission area image 8A and an icon 39 indicating the abnormal area superimposed in real time on the captured image 28 captured by the imaging device 27. Therefore, even if the worker moves while capturing an image, the transmission area image 8A and the icon 39 indicating the abnormal area are updated and displayed in real time on the touch panel 29, allowing the worker to more easily understand the position and range of the transmission area 8. Furthermore, because the worker can view the transmission area image 8A from various directions, the transmission area 8 can be more accurately grasped.

As shown in Figure 7 (B), when the terminal processor 21 acquires movement direction data, it displays the movement direction corresponding to the movement direction data on the touch panel 29. This allows the worker to grasp the position and direction of the sensor 5, and easily understand the areas that need attention when performing repair work on the exterior of the vehicle 3 to be repaired.

Furthermore, when the server processor 33 cannot determine the presence or absence of an abnormality from the image acquired by the imaging device 27, or when it is unable to determine the position and orientation of the terminal 11, a notification 47 prompting the operator to re-image the image is displayed on the touch panel 29 of the terminal 11. As a result, even when the image is unclear and the abnormality cannot be identified, or when the position and orientation of the terminal 11 cannot be identified and a transparent area image 8A corresponding to the overlapping portion cannot be acquired, the operator can re-image the image and send it, making it possible to identify the abnormality or overlapping portion.

In this way, the server processor 33 identifies the position and orientation of the terminal 11 by comparing the acquired surface shape with the vehicle exterior data. This allows the transparent area 8 to be identified more accurately than if it were not based on the surface shape of the exterior component 7. Furthermore, the server processor 33 acquires the position and orientation of the terminal 11 based on the positions of multiple components mounted on the vehicle 3 to be repaired, in addition to the exterior component 7, as appropriate, and identifies the transparent area 8. This allows the transparent area 8 to be identified more accurately.

<<Second Embodiment>>
The vehicle repair assistance system 51 according to the second embodiment differs from the first embodiment in the configuration of the terminal 61. The other configurations are the same as those of the first embodiment, and therefore a description thereof will be omitted. However, in the following description, it is assumed that the repair vehicle information relating to the vehicle 3 to be repaired is separately acquired by the server 13.

As shown in Figure 8, the terminal 61 in the second embodiment is a terminal that can be worn by a worker, i.e., a wearable terminal. In this embodiment, the terminal 61 is a glasses-type wearable terminal (AR wearable device) that uses augmented reality (AR) technology. The terminal 61 is configured as so-called smart glasses. The terminal 61 includes a frame 63 that is fastened to the worker's ears, and two lenses 65 (also called glasses) that are attached to the frame 63 and positioned in front of the worker's eyes when wearing the terminal 61.

The terminal 61 is equipped with an imaging device 27, a display device 69, a speaker 71, and a microcomputer 73.

The imaging device 27 is mounted on the frame 63 and is composed of a camera 27A that captures images in front of the worker wearing it. The display device 69 is a device that can display information such as images and text semi-transparently within the worker's field of vision. In this embodiment, the display device 69 displays images and text semi-transparently superimposed on a background (i.e., overlay display) on the lens 65. The speaker 71 notifies the worker wearing the terminal 61 by voice.

Each microcomputer 73 is equipped with a processor (terminal processor 21), main memory 23, and storage device 25, similar to those in the first embodiment. The microcomputer 73 is configured to be able to communicate with the server 13 via the network 31. In this embodiment, the microcomputer 73 is provided in the frame 63. However, this is not limited to this configuration, and the microcomputer 73 may also be connected to the imaging device 27 and display device 69 via cables connected to the frame 63.

The processor of the microcomputer 73 (hereinafter referred to as the terminal processor 21) transmits the images captured by the imaging device 27 to the server 13, as in the first embodiment.

Each server 13 is equipped with a processor (server processor 33), main memory 35, and storage device (server storage device 37), similar to those in the first embodiment. As in the first embodiment, the server 13's processor (hereinafter referred to as server processor 33) performs anomaly identification processing and area data generation processing, and when an abnormality location and transmission area image 8A are acquired, it transmits the abnormality location position data, abnormality cause data, repairability data, and transmission area image 8A.

As a result, as shown in FIG. 9, the terminal processor 21 displays, on the display device 69, the image of the vehicle 3 to be repaired, which is visible through the lens 65, a transparent area image 8A and an icon 39 indicating the abnormality location, superimposed on the image. Furthermore, as in the first embodiment, the terminal processor 21 may also superimpose on the display device 69 the cause of the abnormality corresponding to the abnormality location and whether or not the vehicle can be repaired.

When the terminal processor 21 receives a request signal requesting further images, it may output a voice message such as "Please move" from the speaker 71 to prompt the imaging device 27 to take further images from another position.

Next, we will explain the effects of the vehicle repair support system 51 configured in this manner.

The lens 65 displays an image 3A of the vehicle 3 to be repaired, which can be seen through the lens, with a transparent area image 8A and an icon 39 indicating the abnormality superimposed on it. As a result, even if the sensor 5 is covered by the exterior member 7 and not directly exposed to the outside, the worker can easily identify the portion of the exterior member 7 that is within the detection range of the sensor 5, i.e., the transparent area 8, by looking at the vehicle 3 to be repaired through the lens 65. In particular, in this embodiment, because the terminal 61 is a wearable terminal in the form of glasses, the worker can perform repair work with both hands while looking at the transparent area image 8A and the icon 39 indicating the abnormality, which is expected to further improve work efficiency.

When the terminal processor 21 receives a request signal requesting further images, it sends a notification prompting the imaging device 27 to take an image from another position. As a result, even if it is difficult for the imaging device 27 to acquire the abnormality location or the position and orientation of the terminal 61, the terminal 61 can be moved, allowing the server processor 33 to acquire the abnormality location and the position and orientation of the terminal 61.

In this way, the vehicle repair assistance system 1, 51 according to the present invention improves the efficiency of repair work by workers and further enhances the safety of moving objects (vehicles). Therefore, the present invention is expected to contribute to the development of sustainable transportation systems.

This concludes the explanation of specific embodiments, but the present invention is not limited to the above embodiments and modifications, and can be implemented in a wide variety of ways.

In the above embodiment, an example was described in which the mobile object was a four-wheeled automobile (vehicle), but the mobile object is not limited to a four-wheeled automobile. The mobile object may be, for example, a motorcycle, a tricycle, a robot, a drone, a work machine, a railroad vehicle, a ship, an airplane, etc.

In the above embodiment, the sensor 5 was radar, but it is not limited to this and may be any device that transmits and receives radio waves, sound waves, light, etc. For example, the sensor 5 may be a sonar that is covered by an exterior member 7 such as a bumper and transmits and receives sound waves toward the outside of the vehicle.

In the first embodiment described above, the terminal processor 21 generates a composite image by superimposing the transparent area image 8A on the image acquired by the imaging device 27, but this is not limited to this configuration. The server processor 33 may superimpose the transparent area image 8A on the image acquired by the imaging device 27 and transmit the composite image to the terminal 61, and the terminal processor 21 may display the composite image transmitted from the server 13 on the touch panel 29.

In the above embodiment, the server storage device 37 stores the specific information 38, but this is not limited to the embodiment, and the storage device 25 of the terminal 11 may store the specific information 38. Alternatively, the server storage device 37 and the storage device 25 of the terminal 11 may each store the specific information 38 separately. Alternatively, some of the specific information 38 may be stored by multiple servers different from the server 13, and the remaining information of the specific information 38 may be stored by the server storage device 37 and the storage device 25 of the terminal 11.

In the first embodiment described above, the captured image 28, the transmission area image 8A, and the icon 39 indicating the abnormality were superimposed and displayed in real time on the touch panel 29 of the terminal 61, but this is not limited to a real-time superimposed display. Alternatively, the terminal 11 may superimpose the transmission area image 8A on the captured image 28 a predetermined time after the imaging device 27 acquires the captured image 28.

Furthermore, when the server processor 33 determines that a transparent region 8 is present in the captured image 28 transmitted in real time, a notification that a still image will be captured may be displayed on the terminal 11. The notification that a still image will be captured may be any notification that prompts the worker to take a photo, and may be, for example, a text message such as "Please take a photo." After that, when the captured image 28 is transmitted from the terminal 11 to the server 13, the server 13 may transmit a transparent region image 8A, and the terminal 11 may superimpose the transparent region image 8A on the captured image 28.

In addition, when the terminal processor 21 determines that the abnormality can be repaired, it may send a signal to the server 13 requesting an explanation of the repair method. The server 13 may send text or images indicating the corresponding repair method to the terminal 61, and the terminal processor 21 may display the acquired text or images indicating the repair method on the touch panel 29.

In the first embodiment described above, the server processor 33 was configured to acquire data indicating the movement direction of the terminal 61 in step ST14. However, in addition to the movement direction of the terminal 61, the server processor 33 may also acquire the rotation direction (direction of turning) and movement distance of the terminal 61 required to include the transparent area 8 within the imaging area of the imaging device 27. In this case, the server processor 33 may transmit displacement data related to the movement direction, rotation direction, and movement distance to the terminal 61. Upon receiving the displacement data, the terminal 61 may display the corresponding movement direction, rotation direction, and movement distance on the touch panel 29.

In the above embodiment, the server processor 33 was configured to extract all abnormal areas within the captured image, but this is not limited to this configuration. The server processor 33 may extract the portion of the image captured by the imaging device 27 that is located within the transparent area 8 (i.e., that overlaps with the transparent area image 8A), and extract abnormal areas only from this extracted portion. Because the area through which the radio waves and sound waves from the sensor 5 pass is mostly made up of relatively flat surfaces, it becomes easier to extract abnormal areas, and the amount of calculation required by the server processor 33 can be reduced.

In the second embodiment described above, the terminal 61 was a glasses-type wearable device, but this is not limited to this. For example, the terminal 61 may be configured as a contact lens-type wearable device. The terminal 61 may also be configured as a wristwatch-type wearable device (a so-called smart watch). The imaging device 27 may also be provided on a remotely operable robot. In this case, the transparent area image 8A may be displayed superimposed on the image 3A of the vehicle 3 to be repaired on a device (e.g., a touch panel 29) that presents information to the worker remotely operating the robot.

1: Vehicle repair assistance system (repair assistance system according to the first embodiment)
3: Vehicle to be repaired (mobile body)
3A: Image (image of the vehicle to be repaired)
5: Sensor 7: Exterior member 8: Transmitting area 8A: Transmitting area image 21: Terminal processor (processor)
23: Main memory 25: Storage device 27: Imaging device 28: Captured image 29: Touch panel (display device)
33: Server processor (processor)
37: Server storage device (storage device)
38: Identification information 38A: Vehicle information table 38B: Vehicle appearance data group 38C: Transparent area data group 39: Icon 40: Notification 43: Notification 45: Notification 47: Notification 51: Vehicle repair assistance system (repair assistance system according to the second embodiment)
65: Lens 69: Display device

Claims (11)

A repair assistance system for assisting repair of a mobile object having a sensor that acquires external information through a transparent area set in an exterior member,
an imaging device for capturing an image of the exterior of the moving object;
a display device capable of superimposing an image showing a specific area of the outer surface on an image of the moving body and presenting the image to a worker;
a storage device that stores identification information for identifying a portion of the captured image corresponding to the transmission region;
a processor for processing the captured image,
The processor identifies a portion of the captured image that corresponds to the transparent region based on the identification information, and displays the portion that corresponds to the transparent region on the display device by superimposing it on an image of the moving body. The repair support system described in claim 1, wherein the display device displays the captured image by superimposing the portion corresponding to the transparent area. the display device is configured to be able to superimpose information on a transparent lens,
The repair assistance system according to claim 1 , wherein the display device displays the image of the moving object visually recognized through the lens by superimposing the portion corresponding to the transparent region on the image. The processor extracts, from the captured image, an image of an abnormality that may reduce the detection sensitivity of the sensor;
The repair support system according to any one of claims 1 to 3, wherein the display device displays an image of the moving body, superimposed on it, a portion corresponding to the transparent area and an icon indicating the abnormality. The processor identifies an abnormality factor corresponding to the abnormal location based on the captured image,
The repair support system according to claim 4 , wherein the display device displays the cause of the abnormality. The repair support system described in claim 5, wherein the abnormality factors include scratches and dirt. The processor determines whether or not the abnormality can be repaired based on the captured image,
A repair support system according to any one of claims 4 to 6, wherein when it is determined that the abnormal part cannot be repaired, the display device notifies the worker that replacement is necessary, and when it is determined that the abnormal part can be repaired, the display device notifies the worker that repairs are required. The repair support system described in any one of claims 1 to 7, wherein the display device updates and displays the portion corresponding to the transparent area in accordance with updates to the captured image. The repair support system described in any one of claims 1 to 8, wherein the processor identifies the portion corresponding to the transparent area based on the surface shape of the exterior member. The repair support system described in any one of claims 1 to 9, wherein the processor notifies the worker to take an image when it cannot identify a portion corresponding to the transparent area in the captured image. The repair support system described in any one of claims 1 to 10, wherein the processor identifies the portion corresponding to the transparent area based on the positions of multiple components mounted on the moving body in addition to the exterior component.