JP7743974B2 - Display System - Google Patents

Description

The present invention relates to a display system .

A system is known that uses an onboard camera to capture and display images of the vehicle's surroundings (see, for example, Patent Document 1).

JP 2010-166196 A

If a system that captures the surroundings, such as a system that captures the area around a car using an onboard camera, could provide location information of surrounding objects, it could help users understand the location of surrounding objects.

One example of the objective of the present invention is to provide a display system, image generation device, image generation method, and program that can present position information of surrounding objects based on captured images of the surrounding area.

According to a first aspect of the present invention, the display system includes an image capturing device mounted on a ship for capturing images, and a horizontal relative position of an object whose image is included in an image captured by the image capturing device relative to a reference position, the horizontal relative position of the object being calculated by multiplying the distance from the ship to the horizon, which is calculated based on the radius of the Earth and the height of the image capturing device from the water surface, by the ratio of the distance from the ship to the distance from the object to the distance from the ship to the horizon, which is calculated based on the positional relationship between an origin set at the center of the bottom edge of the captured image, the image of the object, and the image of the horizon. The apparatus is equipped with a position calculation unit that performs calculations based on distance and, based on time-series information on the orientation of the imaging device, performs corrections to cancel out the effect of changes in the orientation of the imaging device on the relative position by rotating the coordinates for each time data by an angle that is the difference between the actual orientation of the imaging device and a reference orientation; and a display unit that displays an image in which the position of the object is plotted in two-dimensional coordinates whose intersection of coordinate axes indicates the reference position, and also displays an image in which the trajectory of the object's position after correction is plotted in two-dimensional coordinates of position coordinates and time coordinates to cancel out the effect of changes in the orientation of the imaging device .

The position calculation unit may estimate the relative position of an object whose relative position cannot be calculated from the captured image by performing linear interpolation in a three-dimensional coordinate space using the coordinate axes of the two-dimensional coordinates and a time axis based on the time series data of the captured image.

The position calculation unit may set the position at each time on a trajectory obtained by linearly interpolating the position of the imaging device as the reference position based on time series information of the position of the imaging device , and the display unit may display an image in which the trajectory of the position of the object is plotted on two-dimensional coordinates of position coordinates and time coordinates, with the position at each time on the trajectory obtained by linearly interpolating the position of the imaging device being the reference position .

This invention makes it possible to present location information of surrounding objects based on captured images of the surrounding area.

1 is a schematic block diagram illustrating an example of the configuration of a display system according to an embodiment. 1A and 1B are diagrams showing examples of images captured by the imaging device 10. 3A and 3B are diagrams showing examples of images displayed by a display unit 120. FIG. 10 is a diagram showing an example of the position of an object image in a captured image. FIG. 10 is a diagram showing an example of a three-dimensional coordinate space in which a time coordinate axis is added to a two-dimensional horizontal coordinate axis. FIG. 10 is a diagram showing an example of interpolation of a ship's trajectory by the position calculation unit 193. 10 is a diagram showing an example of an image displayed by the display unit 120, in which the trajectory of an object's position is plotted on two-dimensional coordinates of position coordinates and time coordinates. FIG. FIG. 10 is a diagram showing an example of the display of the ship's trajectory by the display unit 120 when the ship's own situation affects the position calculation by the position calculation unit 193. FIG. 10 is a diagram showing an example of a processing procedure in which the processing device 30 displays an image showing the position of an object such as a ship. FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.

The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
Fig. 1 is a schematic block diagram showing an example configuration of a display system according to an embodiment. In the configuration shown in Fig. 1, the display system 1 includes an imaging device 10, a sensor 20, and a processing device 30. The processing device 30 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a control unit 190. The control unit 190 includes an image acquisition unit 191, an orientation information acquisition unit 192, a position calculation unit 193, an image generation unit 194, and a display control unit 195.

The display system 1 displays an image showing the positions of objects around the display system 1 itself. In the following, an example will be described in which the display system 1 is mounted on a ship and displays an image showing the horizontal positions of surrounding objects, such as surrounding ships. The ship on which the display system 1 is mounted is referred to as the ship itself.
However, the installation location of the display system 1 is not limited to a ship. For example, the display system 1 may be installed on a moving body other than a ship, such as an automobile. The moving body referred to here may be various moving bodies, such as an automobile, a railway vehicle, an aircraft, a ship, or an unmanned guided vehicle. Alternatively, the display system 1 may be fixed in position, such as installed on a lighthouse. Furthermore, the various parts of the display system 1 may be installed on different bodies, such as an image capture device 10 installed on an unmanned ship and a processing device 30 installed at a land-based control base.

The photographing device 10 takes photographs. The photographing device 10 may take moving images. Alternatively, the photographing device 10 may take still images repeatedly, for example, at predetermined intervals.
The image capturing device 10 may be configured to capture images of the entire surroundings (all directions in the horizontal direction). In this case, the image capturing device 10 may be configured using an all-around camera. Alternatively, the image capturing device 10 may be configured to include multiple cameras, and the all-around images may be captured using a combination of these multiple cameras.
Alternatively, the image capturing device 10 may capture images in only a portion of the entire surrounding area. For example, the image capturing device 10 may capture images within a 180-degree field of view centered on the front of the ship (90-degree field of view on both the left and right sides of the front of the ship).

Fig. 2 is a diagram showing an example of an image captured by the image capturing device 10. In the example of Fig. 2, a line L111 indicates an image of the horizon.
2 includes an image of a ship 911 and an image of a ship 912. The images of the ships 911 and 912 correspond to examples of images included in the image captured by the image capturing device 10. The ships 911 and 912 correspond to examples of objects whose images are included in the image captured by the image capturing device 10.

The sensor 20 measures the position and orientation of the image capture device 10. For example, the image capture device 10 and the sensor 20 may be fixedly installed on the hull of the ship. The position and orientation measured by the sensor 20 itself may be used as the position and orientation of the ship, and may also be used as the position and orientation of the image capture device 10. The orientation measured by the sensor 20 includes not only the horizontal orientation (yaw angle), but also left-right tilt (roll angle) and fore-and-aft tilt (pitch angle).

For example, the sensor 20 may be configured to include a 9-axis sensor. The sensor 20 may then measure the orientation of the image capture device 10 using the 9-axis sensor and calculate the position of the image capture device 10 based on the measurement values of the 9-axis sensor. However, the configuration of the sensor 20 is not limited to a specific one. For example, the sensor 20 may be configured to include a GNSS (Global Navigation Satellite System) terminal device in addition to or instead of the 9-axis sensor, and positioning may be performed using the GNSS terminal device.

The processing device 30 generates and displays an image indicating the position of an object whose image is included in the captured image based on the image captured by the imaging device 10. The processing device 30 may be configured using a computer such as a workstation or a personal computer (PC).
The processing device 30 is an example of an image generating device.

The communication unit 110 communicates with other devices. For example, the communication unit 110 communicates with the image capturing device 10 to receive image data of an image captured by the image capturing device 10. The communication unit 110 also communicates with the sensor 20 to receive measurement data of the position and orientation of the image capturing device 10.
The display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images under the control of the display control unit 195. For example, the display unit 120 displays an image showing the position of an object whose image is included in the captured image. Specifically, the display unit 120 displays an image in which the relative position of the object whose image is included in the captured image is plotted on two-dimensional coordinates in which the intersection of the coordinate axes indicates a reference position.

Fig. 3 is a diagram showing an example of an image displayed by the display unit 120. Fig. 3 shows an example in which the display unit 120 displays an image in the format of a radar display.
3 correspond to examples of coordinate axes indicating a certain horizontal direction. The intersection of lines L211 and L212 indicates the reference position. For example, the reference position may be set to the position of the ship, with line L211 indicating the left-right coordinate of the ship and line L212 indicating the fore-aft coordinate of the ship.
Lines L221, L222, L223, and L224 indicate concentric circles from a reference position.
Points P111 and P112 respectively indicate the positions of ships photographed by the image capturing device 10 (i.e., ships whose images are included in the images captured by the image capturing device 10).

However, the screen displayed by the display unit 120 is not limited to a specific format. For example, the lines L211 and L212 indicating the coordinate axes may not be displayed. Alternatively, when the lines L211 and L212 indicating the coordinate axes are displayed, scales may be displayed on the coordinate axes. Furthermore, the lines L221 to L224 indicating the concentric circles may not be displayed.
Furthermore, the display of the position of an object such as a ship is not limited to a point display, and for example, an image of the object such as a ship may be displayed at the position of the object on the coordinate plane.

The operation input unit 130 includes an input device such as a touch sensor that is provided on the display screen of the display unit 120 and forms a touch panel, or an operation panel, and accepts user operations. For example, the operation input unit 130 may accept a user operation that specifies the format of the display screen.
The storage unit 180 stores various types of data. For example, the storage unit 180 may store time-series information on the positions of objects around the ship. The storage unit 180 may also store time-series information on the position and orientation of the ship.
The storage unit 180 is configured using a storage device provided in the processing device 30 .

The control unit 190 executes various processes by controlling each unit of the processing device 30. The functions of the control unit 190 are executed, for example, by a CPU (Central Processing Unit) included in the processing device 30 reading and executing a program from the storage unit 180.
The image acquisition unit 191 acquires an image captured by the image capturing device 10. For example, the image acquisition unit 191 extracts image data of the image captured by the image capturing device 10 from the data received by the communication unit 110 from the image capturing device 10.
The orientation information acquisition unit 192 acquires information indicating the position and orientation of the image capture device 10. For example, the orientation information acquisition unit 192 extracts measurement data of the sensor 20 from data received by the communication unit 110 from the sensor 20.

The position calculation unit 193 calculates the horizontal relative position with respect to a reference position of an object whose image is included in the image captured by the imaging device 10. For example, the position calculation unit 193 calculates the position of the object in two-dimensional coordinates in the horizontal direction, such as the coordinates exemplified in FIG.
The position calculation unit 193 may detect an image of an object from the image captured by the image capture device 10 and calculate the position of the detected object. In this case, the position calculation unit 193 may calculate the relative position of the object with respect to the ship based on the position of the image of the object in the image captured by the image capture device 10.

FIG. 4 is a diagram showing an example of the position of an object image in a captured image.
4, in the example of the captured image shown in FIG. 2, an origin O is set at the center of the bottom edge of the image. A line L311 indicates a horizontal line. A line L321 indicates a vertical line passing through the origin O.
Distance r11 indicates the distance from the origin O to the image of the ship 911. Angle θ11 indicates the direction (rotation angle) of the image of the ship 911 as viewed from the origin O when the line L321 is used as the reference (0°). Distance r12 indicates the distance from the origin O to the image of the ship 912. Angle θ12 indicates the direction of the image of the ship 912 as viewed from the origin O when the line L321 is used as the reference.

The position calculation unit 193 may calculate the relative position of the ship 911 with respect to the ship itself based on the distance r11 and the angle θ11. The position calculation unit 193 may also calculate the relative position of the ship 912 with respect to the ship itself based on the distance r12 and the angle θ12.
In this case, the position calculation unit 193 may calculate the distance from the ship to the ship based on the positional relationship between the image of the ship and the image of the horizon (line L311).
For example, the distance l from the ship to the horizon is expressed as in equation (1).

R represents the radius of the Earth, and h represents the height of the imaging device 10 from the water surface.
4, the image of the ship 911 is at the position of the image of the horizon. In this case, the position calculation unit 193 may calculate the distance from the ship to the ship 911 as distance 1.
Furthermore, the position calculation unit 193 may calculate the ratio of the distance from the ship to the ship 912 to the distance from the ship to the horizon based on the positional relationship between the origin O, the image of the ship 912, and the image of the horizon.The position calculation unit 193 may then calculate the distance from the ship to the ship 912 by multiplying the distance l from the ship to the horizon by the calculated ratio.

However, the method by which the position calculation unit 193 calculates the relative position of an object with respect to the reference position is not limited to a method of calculating directly from a captured image. For example, the ship may be equipped with a distance measurement device such as a laser rangefinder, and the position calculation unit 193 may use this distance measurement device to calculate the relative position of an object with respect to the ship. If the reference position is set to a position other than the ship's position, the position calculation unit 193 may further calculate the relative position of an object with respect to the reference position based on the relative position of the object with respect to the ship.

Alternatively, the position calculation unit 193 may be equipped with a learning model that receives input image data and outputs coordinate information for the position of an object whose image is included in the captured image, thereby calculating the position of the object directly from the captured image. In this case, the learning model may be configured using a convolutional neural network (CNN), but is not limited to this.

The image generation unit 194 generates an image indicating the position of an object whose image is included in the image captured by the imaging device 10. For example, the image generation unit 194 generates an image in which the relative position of the object with respect to the reference position calculated by the position calculation unit 193 is plotted on a two-dimensional coordinate plane in which the intersection of the coordinates indicates the reference position, as in the image illustrated in FIG.
The display control unit 195 controls the display unit 120 to display various images. In particular, the display control unit 195 causes the display unit 120 to display the images generated by the image generation unit 194.

The position calculation unit 193 may calculate the position of the object in three-dimensional coordinates obtained by adding time to two-dimensional coordinates in the horizontal direction.
5 is a diagram showing an example of a three-dimensional coordinate space in which a time coordinate axis is added to the two-dimensional horizontal coordinate axes. The time coordinate axis is also called the time axis.
In the example of FIG. 5, the x-axis and y-axis indicate two-dimensional coordinate axes in the horizontal direction.

In Fig. 5, the positions of ships at times t11, t12, and t13 are shown in the form of images displayed by the display unit 120. In Fig. 5, the positions of three ships are shown. These three ships are referred to as the first ship, the second ship, and the third ship.
In the image taken at time t11, point P211 indicates the position of the first ship, point P212 indicates the position of the second ship, and point P213 indicates the position of the third ship. In the image taken at time t12, point P221 indicates the position of the first ship, point P222 indicates the position of the second ship, and point P223 indicates the position of the third ship. In the image taken at time t13, point P231 indicates the position of the first ship, point P232 indicates the position of the second ship, and point P233 indicates the position of the third ship.
Line L411 shows the position history of the first vessel. Line L412 shows the position history of the second vessel. Line L413 shows the position history of the third vessel. The position history is also called a trajectory.

The control unit 190 can perform various analyses by analyzing data in a three-dimensional coordinate space including a time axis. For example, the control unit 190 may predict the time and distance at which a ship approaching the host ship will come closest to the host ship.
Furthermore, the position calculation unit 193 may analyze data in a three-dimensional coordinate space including a time axis, thereby estimating the relative position of an object whose relative position cannot be calculated from the images captured by the image capture device 10, based on the time-series data of the captured images. For example, the position calculation unit 193 may interpolate the trajectory of an object, such as a ship, in a three-dimensional coordinate space including a time axis for a time period when the image of the object is not included in the images captured by the image capture device 10.

Fig. 6 is a diagram showing an example of interpolation of the ship's trajectory by the position calculation unit 193. Fig. 6 shows an example in which the angle of view of the image capturing device 10 is limited to a triangular area (area A11) shown in the image at each time in the example shown in Fig. 5. The angle of view of the image capturing device 10 here is the capturing range of the image capturing device 10.
In the example of FIG. 6, the x-axis and y-axis indicate two-dimensional coordinate axes in the horizontal direction.

As in Figure 5, Figure 6 also shows the positions of the ships at times t11, t12, and t13 in the form of images displayed by the display unit 120. However, while Figure 6 shows the positions of the first ship and the second ship, it does not show the position of the third ship. The third ship is located outside the angle of view of the imaging device 10 from time t11 to t13, and is therefore not included in the position display.

As in Figure 5, the position of the first vessel is indicated by point P211 in the image taken at time t11, by point P221 in the image taken at time t12, and by point P231 in the image taken at time t13. Of these positions, points P221 and P231 are located within area A11, which is the imaging range of the imaging device 10. On the other hand, point P211 is located outside area A11.

The position calculation unit 193 interpolates the position of the first vessel when it is located outside the area A11, such as at time t11, based on the position of the first vessel when it is located within the area A11, such as at time t12 and t13. In the example of Figure 6, the position calculation unit 193 calculates the position of the first vessel by linear interpolation (straight-line interpolation).
In this way, the position calculation unit 193 can calculate the position of a ship that was not within the field of view of the imaging device 10 by going back in time.

The position of the second vessel is indicated by point P212 in the image taken at time t11, by point P222 in the image taken at time t12, and by point P232 in the image taken at time t13. Of these positions, points P212 and P222 are located within area A11, which is the imaging range of the imaging device 10. On the other hand, point P232 is located outside area A11.

The position calculation unit 193 interpolates the position of the second vessel when it is located outside area A11, such as at time t13, based on the position of the second vessel when it is located within area A11, such as at times t11 and t12. In the example of Figure 6, the position calculation unit 193 calculates the position of the second vessel by linear interpolation.

In this way, the position calculation unit 193 can calculate the position of a ship that is not within the angle of view of the imaging device 10 based on the past positions of that ship. For example, at time t13, the position calculation unit 193 can calculate in real time the current position of the first ship that is not within the angle of view of the imaging device 10. Furthermore, at time t12, the position calculation unit 193 can calculate the position of the first ship at a future time t13.
In this way, the position calculation unit 193 can estimate the relative position of any object, past, present, or future, for which the relative position cannot be calculated from the image captured by the imaging device 10, based on the time series data of the captured image.

In addition to displaying an image in which the position of an object is plotted on two-dimensional coordinates in which the intersection of the coordinate axes indicates a reference position, as illustrated in Figure 3, the display unit 120 may also display an image in which the trajectory of the object's position is plotted on two-dimensional coordinates of position coordinates and time coordinates.
FIG. 7 is a diagram showing an example of an image displayed by the display unit 120, in which the locus of an object's position is plotted on a two-dimensional coordinate system of position coordinates and time coordinates.

The horizontal axis (x-axis) in Figure 7 represents position coordinates. For example, the horizontal axis represents the left-right direction of the ship. The vertical axis (t-axis) represents the time coordinate. Figure 7 shows an example of the history of the left-right positions of a ship located ahead of the ship, as seen from the ship itself.
Lines L511 and L521 show the position history of the fourth ship. The position calculation unit 193 calculates the position of the fourth ship on line L511 from the position of the image of the fourth ship in the image captured by the photographing device 10. On the other hand, for line L521, the image captured by the photographing device 10 does not include an image of the fourth ship, and the position calculation unit 193 calculates the position of the fourth ship shown on line L521 by linear interpolation based on the position of the fourth ship shown on line L511.

Lines L512, L522, and L532 indicate the position of the fifth ship. The position calculation unit 193 calculates the position of the fifth ship indicated by lines L512 and L532 from the position of the image of the fifth ship in the image captured by the imaging device 10. On the other hand, for line L522, the image captured by the imaging device 10 does not include an image of the fifth ship, and the position calculation unit 193 calculates the position of the fifth ship indicated by line L522 by linear interpolation based on the position of the fifth ship indicated by lines L512 and L532.

In this way, even when the display unit 120 displays an image of two-dimensional coordinates of position coordinates and time coordinates, the position calculation unit 193 can display the position calculated by interpolation for an object whose position cannot be calculated from the image captured by the imaging device 10.
Examples of cases in which an object such as a ship is not captured by the imaging device 10 include, but are not limited to, when the object is located outside the imaging range of the imaging device 10 as in the example of Figure 6, when the object is temporarily hidden by waves, when the object is blocked by another object and cannot be seen from the ship, and when the object cannot be seen due to weather conditions such as fog.

If the ship's conditions, such as small corrections to the ship's orientation or position, or the ship's swaying, affect the images captured by the imaging device 10, the position calculation unit 193 may be configured to cancel out the influence of the ship's conditions when calculating the position of an object such as a ship based on the images captured by the imaging device 10.

FIG. 8 is a diagram showing an example of the trajectory of the ship displayed on the display unit 120 when the situation of the ship affects the position calculation by the position calculation unit 193.
The horizontal axis (x-axis) in Figure 8 indicates position coordinates. For example, the horizontal axis indicates the left-right direction of the ship. The vertical axis (t-axis) indicates the time coordinate. Figure 8 shows, for example, an example of the history of the left-right positions of a ship located ahead of the ship, as seen from the ship itself.

Line L611 shows an example of the ship's trajectory before the position calculation unit 193 makes any corrections. For example, if the camera device 10 is installed with a fixed orientation relative to the ship, and the ship changes orientation slightly in response to the tide, and the orientation of the camera device 10 changes slightly, the position calculation unit 193 may calculate the trajectory of a ship proceeding in a straight line at an approximately constant speed as a wavy trajectory as shown by line L611. In this way, changes in the orientation of the camera device 10 may affect the position of the ship calculated by the position calculation unit 193, making it difficult to grasp the ship's actual movement.

The position calculation unit 193 may therefore perform a correction based on time-series information on the orientation of the camera device 10 to offset the effect of changes in the orientation of the camera device 10 on the relative position of an object such as a ship with respect to a reference position. For example, the position calculation unit 193 may use the current orientation of the ship as the reference orientation and correct the relative position of an object such as a ship to the relative position that would be achieved if the ship were always facing the reference orientation. Such a correction is performed, for example, by the position calculation unit 193 rotating the coordinates for the data at each time by the angle of difference between the actual orientation of the camera device 10 and the reference orientation.

Line L612 shows an example of the ship's trajectory after the position calculation unit 193 has performed the correction. Line L612 shows a trajectory in which the ship is moving straight ahead at an approximately constant speed. This is expected to enable crew members to understand the actual movement of the ship by referring to the image displayed by the display unit 120.

It is considered that not only the orientation of the image capturing device 10 but also the position of the image capturing device 10 affects the position of an object such as a ship calculated by the position calculation unit 193 .
Therefore, the position calculation unit 193 may set the reference position based on time-series information on the position of the image capturing device 10 .

For example, if the ship is meandering, the position calculation unit 193 may calculate a trajectory by linearly interpolating the ship's trajectory, and use that trajectory as a reference position to calculate the relative position of the ship, etc. Alternatively, the position calculation unit 193 may set a predetermined position as a common reference position at each time, and calculate the relative position of the ship, etc. with respect to that reference position. The position calculation unit 193 may set the ship's current position as a common reference position at each time, and calculate the relative position of the ship, etc. with respect to that reference position.

Figure 9 is a diagram showing an example of a processing procedure in which the processing device 30 displays an image showing the position of an object such as a ship. The processing device 30 may perform the processing of Figure 9 at predetermined intervals, such as every second. The processing device 30 may also perform the processing of Figure 9 when a user operation instructing the display of an image is received.

9, the image acquisition unit 191 acquires an image captured by the image capturing device 10 (step S111). For example, the image acquisition unit 191 extracts image data of the image captured by the image capturing device 10 from the data received by the communication unit 110 from the image capturing device 10.
Next, the orientation information acquisition unit 192 acquires information indicating the position and orientation of the image capture device 10 (step S112). For example, the orientation information acquisition unit 192 extracts measurement data of the position and orientation of the image capture device 10 measured by the sensor 20 from the data received by the communication unit 110 from the sensor 20.

Next, the position calculation unit 193 detects the image of the object in the image captured by the image capture device 10 and calculates the position of the object based on the position of the detected image in the image capture (step S113). If multiple object images are detected in the image captured by the image capture device 10, the position calculation unit 193 calculates the position of each detected object.

Then, the position calculation unit 193 updates the history information of the object positions stored in the storage unit 180 (step S114). For example, the position calculation unit 193 adds information linking the position of each object calculated in step S113 with the time to the history information stored in the storage unit 180.
Next, the position calculation unit 193 determines whether or not the history information of the object's position needs to be corrected (step S115). For example, the position calculation unit 193 may determine whether or not a user setting instructing correction has been made.

If it is determined that correction is necessary (step S115: YES), the position calculation unit 193 corrects the position of the object in the history information based on the information indicating the position and orientation of the image capture device 10 acquired by the orientation information acquisition unit 192 in step S112 (step S116). As described above, the position calculation unit 193 may perform correction to cancel out the effects of changes in the orientation of the image capture device 10. As also described above, the position calculation unit 193 may set a reference position and perform correction to change the position of each object to a relative position from the reference position. The position calculation unit 193 may perform both of these two types of correction.

Next, the position calculation unit 193 determines whether or not there is an object to be interpolated (step S117). For example, the position calculation unit 193 may determine whether or not there is an object whose position is indicated at some time and whose position is not indicated at other time during a predetermined time period in which the storage unit 180 stores the history information of the object's position, and if it determines that a corresponding object exists, it may determine that there is an object to be interpolated.
Furthermore, the position calculation unit 193 may determine whether or not a user operation instructing interpolation has been performed, and if a user operation instructing interpolation has not been performed, determine that there is no object to be interpolated.

If it is determined that there is an object to be interpolated (step S117: YES), the position calculation unit 193 performs a process of interpolating the object's position using the object's position history information stored in the memory unit 180, as described above (step S118).
Next, the display unit 120 displays an image showing the position of the object under the control of the display control unit 195 (step S119). The display unit 120 may display an image showing the position of the object in two-dimensional coordinates, as exemplified in Fig. 3. Alternatively, the display unit 120 may display an image showing the position of the object in three-dimensional coordinates consisting of two position coordinates and one time coordinate, as exemplified in Fig. 5. Alternatively, the display unit 120 may display an image showing the position of the object in two-dimensional coordinates consisting of position coordinates and time coordinates, as exemplified in Fig. 7.
After step S119, the processing device 30 ends the process of FIG.

On the other hand, if it is determined in step S115 that correction is not necessary (step S115: NO), the process proceeds to step S117.
On the other hand, if it is determined in step S117 that there is no object to be interpolated (step S117: NO), the process proceeds to step S119.

As described above, the image capturing device 10 captures images. The position calculation unit 193 calculates the horizontal relative position with respect to a reference position of an object whose image is included in an image captured by the image capturing device 10. The display unit 120 displays an image in which the position of the object is plotted on a two-dimensional coordinate system in which the intersection of the coordinate axes indicates the reference position.
According to the display system 1, it is possible to present position information of surrounding objects based on captured images of the surroundings.

Furthermore, for an object whose relative position cannot be calculated from the captured image, the position calculation unit 193 estimates the relative position based on the time-series data of the captured image.
This allows the display system 1 to estimate the position of objects whose relative position cannot be calculated from the captured image, such as objects whose image is not included in the image captured by the imaging device 10, and present the position to the user.

In addition to displaying an image in which the position of an object is plotted on two-dimensional coordinates where the intersection of the coordinate axes indicates the reference position, the display unit 120 also displays an image in which the trajectory of the object's position is plotted on two-dimensional coordinates of position coordinates and time coordinates.
This allows the processing device 30 to display the history of the position of an object such as a ship.

Furthermore, the position calculation unit 193 performs correction based on time-series information on the orientation of the image capturing device 10 to cancel out the influence of changes in the orientation of the image capturing device 10 on the relative position.
This allows a user such as a passenger to relatively easily understand the trajectory of the photographed object.

In addition, the position calculation unit 193 sets the reference position based on time-series information about the position of the imaging device 10. This allows the display system 1 to relatively easily grasp the trajectory of the photographed object even when the ship is meandering.

FIG. 10 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.
In the configuration shown in FIG. 10, a computer 700 includes a CPU 710 , a main memory device 720 , an auxiliary memory device 730 , and an interface 740 .

The processing device 30 described above may be implemented in a computer 700. In this case, the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-mentioned storage units in accordance with the program. Communication between each device and other devices is carried out by the interface 740, which has a communication function, and which communicates under the control of the CPU 710.

When the processing device 30 is implemented in a computer 700, the operation of the control unit 190 and each of its components is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

Furthermore, the CPU 710 allocates a storage area corresponding to the storage unit 180 in the main storage device 720 in accordance with the program.
Communication with other devices via the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710 .
The display on the display unit 120 is implemented by the interface 740 having a display device and displaying various images under the control of the CPU 710 .
The operation input unit 130 receives a user operation by the interface 740 having input devices such as a keyboard and a mouse, and outputs information indicating the received user operation to the CPU 710 .

It is also possible to record a program for executing all or part of the processing device 30 on a computer-readable recording medium, and have a computer system load and execute the program to perform the processing of each part. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.
Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment and includes design modifications and the like that do not deviate from the gist of the present invention.

REFERENCE SIGNS LIST 1 display system 10 imaging device 20 sensor 30 processing device 110 communication unit 120 display unit 130 operation input unit 180 storage unit 190 control unit 191 image acquisition unit 192 attitude information acquisition unit 193 position calculation unit 194 image generation unit 195 display control unit

Claims (3)

an imaging device mounted on the ship for taking images;
a position calculation unit that calculates the horizontal relative position of an object whose image is included in an image captured by the camera with respect to a reference position based on the distance from the camera to the horizon, which is calculated by multiplying the distance from the camera to the horizon, which is calculated based on the radius of the Earth and the height of the camera from the water surface, by the ratio of the distance from the camera to the distance from the camera to the horizon, which is calculated based on the positional relationship between an origin set at the center of the bottom edge of the image, the image of the object, and the image of the horizon, and that performs correction to cancel out the effect of changes in the orientation of the camera on the relative position by rotating coordinates for each data point by the angle of difference between the actual orientation of the camera and the reference orientation, based on time-series information of the orientation of the camera;
a display unit that displays an image in which the position of the object is plotted on two-dimensional coordinates whose intersections of coordinate axes indicate the reference position, and also displays an image in which a trajectory of the position of the object after correction that cancels out the influence of a change in the orientation of the image capturing device is plotted on two-dimensional coordinates of position coordinates and time coordinates;
A display system comprising: the position calculation unit estimates the relative position of the object whose relative position cannot be calculated from the captured image by performing linear interpolation in a three-dimensional coordinate space using the coordinate axes of the two-dimensional coordinate system and a time axis based on the time-series data of the captured image.
The display system of claim 1 . the position calculation unit sets , as the reference position, a position at each time on a trajectory obtained by linearly interpolating the position of the image capturing device based on time-series information of the position of the image capturing device ;
The display unit displays an image in which a trajectory of the position of the object is plotted in two-dimensional coordinates of position coordinates and time coordinates, with the reference position being a position at each time on a trajectory obtained by linearly interpolating the position of the image capturing device.
3. The display system according to claim 1 or 2 .