JP7791913B2 - Image generation device, image generation method, and image generation program - Google Patents

Description

The present invention relates to an image generation device, an image generation method, and an image generation program.

Various technologies for assisting ship movement, also known as navigation systems, have been known for some time. For example, Patent Document 1 describes a technology in which a composite image generated based on display information including the ship's planned route, structures located on the sea such as buoys, and other ships, as well as the ship's position and attitude, is superimposed on an image captured from the ship and displayed in 3D on a display. The technology described in Patent Document 1 generates a composite image to be superimposed on an image captured from the ship according to the ship's attitude, thereby enabling the creation of natural, highly realistic augmented reality images.

Japanese Patent Application Laid-Open No. 2021-184614

However, while the technology described in Patent Document 1 makes it possible to view structures and other ships located on the sea, it does not display the seabed topography, which is not visible to the operator operating the navigation system. When the technology described in Patent Document 1 overlays an image showing the seabed topography on a landscape image, the topographical image showing the seabed topography is overlaid over the entire surface of the ocean, covering the entire displayed screen and potentially making it difficult for the operator to view structures on the sea. Furthermore, when a topographical image showing the topography is overlaid over the entire surface of a landscape image captured from a moving body other than a ship, such as a vehicle, much of the landscape image may be covered by the topographical image, making it difficult to view the landscape image.

The present invention therefore aims to provide an image generation device that reduces the risk of landscape images becoming difficult to view when a terrain image is superimposed on the landscape image.

The image generating device according to the embodiment includes an information acquisition unit that acquires terrain information indicating the terrain in a first direction, an image information generation unit that generates image information indicating an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a landscape image captured from a single point, and an output unit that outputs the image information.

In the image generation device according to the embodiment, the information acquisition unit further acquires current location information indicating the current location and first direction information indicating the first direction, and it is preferable that the first location be a location separated from the current location by a first distance in the direction of travel, and the second location be a location separated from the first location by a predetermined display length in the first direction.

In the image generating device according to the embodiment, it is preferable to determine whether there is a risk that the bottom of the body of a vehicle traveling in a first direction will come into contact with the surface of the terrain based on the vertical distance between the bottom of the body of the vehicle traveling in a first direction and the bottom of the wheels that movably support the body, as well as the difference in elevation of the terrain, and to output a warning signal when it is determined that there is a risk that the bottom of the body will come into contact with the surface of the terrain.

In the image generating device according to the embodiment, it is preferable that the image information generating unit generates image information representing an image by superimposing a superimposed image on a landscape image captured by an imaging device located at a single point.

In the image generating device according to the embodiment, it is preferable that the image information generating unit generates superimposed image information that indicates the superimposed image so that it can be viewed at an elevation angle corresponding to the tilt of the imaging device.

An image generation method according to an embodiment includes acquiring terrain information indicating the terrain in a first direction, generating image information indicating a perspective image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a landscape image captured from a single point, and outputting the image information.

The image generation program according to the embodiment acquires terrain information indicating the terrain in a first direction, generates image information indicating an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a landscape image captured from a single point, and outputs the image information.

The image generation device of the present invention can reduce the risk of the landscape image becoming difficult to see when a terrain image is superimposed on the landscape image and displayed.

1 is a perspective view of a ship equipped with a monitoring image generation system according to a first embodiment. FIG. 2 is a diagram showing a monitoring image generating device mounted on the ship shown in FIG. 1 . FIG. 3 is a functional block diagram of the monitoring image generating device shown in FIG. 2 . 3 is a flowchart showing an information setting process executed by the monitoring image generating device shown in FIG. 2. FIG. 3A is a flowchart showing the monitoring image generating process executed by the monitoring image generating device shown in FIG. 2, and FIG. 3B is a flowchart showing more detailed processing of the processing of S205 shown in FIG. 3A. 5(b) is a diagram (part 2) for explaining the processing of S301 shown in FIG. 5(b); and FIG. 5(c) is a diagram (part 3) for explaining the processing of S301 shown in FIG. 5(b). 6 is a flowchart showing the process of S206 shown in FIG. 5A in more detail. 7A is a diagram for explaining the processing of S401 shown in FIG. 7, (b) is a diagram for explaining the processing of S402 shown in FIG. 7, (c) is a diagram for explaining the processing of S403 shown in FIG. 7, (d) is a diagram for explaining the processing of S404 shown in FIG. 7, (e) is a diagram for explaining the processing of S405 shown in FIG. 7, and (f) is a diagram for explaining the processing of S406 shown in FIG. 7. 6 is a flowchart showing the detailed process of step S207 shown in FIG. 5A. 4 is a diagram showing an example of a monitoring image displayed on the display unit shown in FIG. 3. FIG. FIG. 2 is a functional block diagram of the vessel monitoring device. FIG. 12 is a flowchart showing an image display process executed by the vessel monitoring device shown in FIG. 11 . 12 is a diagram showing an example of a display image displayed on the monitoring display unit shown in FIG. 11 . FIG. FIG. 10 is a perspective view of a vehicle equipped with a monitoring image generation system according to a second embodiment. FIG. 15 is a diagram showing a monitoring image generating device mounted on the vehicle shown in FIG. 14. FIG. 16 is a functional block diagram of the monitoring image generating device shown in FIG. 15. FIG. 16 is a flowchart showing an information setting process executed by the monitoring image generating device shown in FIG. 15 . 16A is a flowchart showing the monitoring image generating process executed by the monitoring image generating device shown in FIG. 15, and FIG. 16B is a flowchart showing more detailed processing of the processing of S704 shown in FIG. 18A is a flowchart showing the process of S705 shown in FIG. 18A in more detail, and FIG. 18B is a flowchart showing the process of S706 shown in FIG. 18A in more detail.

The image generation device, image generation method, and image generation program according to the present invention will be described below with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions set forth in the claims and their equivalents.

(Configuration and Function of a Surveillance Image Generation System Having a Surveillance Image Generation Device According to the First Embodiment)
FIG. 1 is a perspective view of a ship equipped with a surveillance image generation system according to a first embodiment, FIG. 2 is a diagram showing a surveillance image generation device mounted on the ship shown in FIG. 1, and FIG. 3 is a functional block diagram of the surveillance image generation device shown in FIG. 2.

The surveillance image generation system 100 includes a GNSS receiver 101, a satellite communication receiver 102, and a surveillance image generation device 1, which is an example of an image surveillance device according to the present invention, and is mounted on a ship 110. Based on signals received from the GNSS receiver 101 and the satellite communication receiver 102, the surveillance image generation system 100 displays a landscape image and a superimposed image superimposed on the landscape image on a screen visible to an operator 112 on board the ship 110.

The GNSS receiver 101 receives signals from a Global Navigation Satellite System (GNSS) such as the Global Positioning System (GPS). In response to receiving signals from the GNSS, the GNSS receiver 101 outputs a current position signal indicating the current position of the ship 110 and a heading signal indicating the direction in which the ship 110 is traveling to the monitoring image generation device 1. The satellite communication receiver 102 receives signals from a communication satellite that can connect to the Internet network via a gateway earth station. The satellite communication receiver 102 receives a tide level signal indicating the tide level at the current position of the ship 110, which is extracted from information on tide levels provided by the Marine Information Department of the Japan Coast Guard via the Internet network, and outputs the received tide level signal to the monitoring image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, AIS signals containing the vessel's identification code, type, position, course, speed, navigation status, and other safety-related information of an AIS vessel located near the vessel 110 and equipped with an AIS signal transceiver, and outputs the received AIS signals to the surveillance image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, non-AIS signals indicating the vessel's position, course, speed, navigation status, and other safety-related information of a non-AIS vessel located near the vessel 110 and not equipped with an AIS signal transceiver, and outputs the received non-AIS signals to the surveillance image generation device 1. A server that outputs the AIS and non-AIS signals is connected to the GNSS receiver 101 via the Internet network.

The surveillance image generation device 1 is placed inside the cabin 111 of the ship 110 so that it is visible to the operator 112 and so that the built-in imaging unit 16 can capture images of the ship's 110's direction of travel through the porthole 113. The surveillance image generation device 1 has a communication unit 11, a memory unit 12, an operation unit 13, a display unit 14, an audio output unit 15, an imaging unit 16, an inclination detection unit 17, and a processing unit 20. The communication unit 11, the inclination detection unit 17, and the processing unit 20 are connected via a bus 18.

The communication unit 11 is configured to enable the surveillance image generation device 1 to communicate with the GNSS receiver 101 to the satellite communication receiver 102, and includes a communication interface circuit. The communication interface circuit is, for example, a wired LAN, wireless LAN, LTE (Long Term Evolution) or other communication interface circuit. The communication unit 11 supplies signals input from the GNSS receiver 101 to the satellite communication receiver 102 to the processing unit 20, and outputs signals supplied from the processing unit 20 to the GNSS receiver 101 to the satellite communication receiver 102.

The storage unit 12 is configured to store data and programs, and includes, for example, semiconductor memory. The storage unit 12 stores operating system programs, driver programs, application programs, data, etc. used in processing by the processing unit 20. For example, the storage unit 12 stores, as an application program, an information setting program that causes the processing unit 20 to execute an information setting process that sets various information used when executing an image generation process that generates images, including topographical images showing the seabed topography. The storage unit 12 also stores, as an application program, an image generation program that causes the processing unit 20 to execute the image generation process. The information setting program and image generation program are installed into the storage unit 12 from computer-readable, non-transitory, portable storage media such as a CD (Compact Disc)-ROM (Read Only Memory) or a DVD (Digital Versatile Disc)-ROM. The storage unit 12 also stores various data and information used when executing the information setting process and image generation process. For example, the memory unit 12 stores elevation data, also referred to as topographical information, which indicates the topography of the seabed in terms of elevation, as elevation tiles 12a.

Elevation tiles 12a have a 256 x 256 matrix structure, with the coordinates of the seafloor located in each matrix element. Elevation data is defined in elevation tiles at various intervals depending on the zoom level. The structure and function of elevation tiles 12a are well known, so a detailed description will be omitted here.

The memory unit 12 also stores structure information indicating structures included in the scenery in the direction of travel of the ship 110, including land-based structures such as lighthouses and port facilities, and offshore structures such as buoys and offshore wind power generation facilities. The structure information includes the name of the structure, shape data indicating the shape of the structure, and the latitude and longitude of the structure. The memory unit 12 also stores route information indicating areas that should be visible when maneuvering the ship 110, including route lines indicating routes along which the ship 110 can proceed, and no-entry areas into which the ship 110 is prohibited from entering. The route information includes the names of areas including the route lines and no-entry areas, shape data indicating the shape of the areas, and the latitude and longitude of the areas. The memory unit 12 also stores alarm threshold information indicating an alarm threshold used when determining whether the bottom of the ship 110 will come into contact with the seabed.

The operation unit 13 is configured to accept user operations on the surveillance image generation device 1 and includes, for example, a touch panel. The operation unit 13 generates a signal in response to the operation of the operator 112 and supplies the generated signal to the processing unit 20.

The display unit 14 is configured to display images and includes a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 14 generates and displays images based on signals supplied from the processing unit 20.

The audio output unit 15 is configured to output audio and includes a speaker. The audio output unit 15 outputs audio based on the signal supplied from the processing unit 20.

The imaging unit 16 is configured to capture images of the area around the monitoring image generating device 1 and includes a camera. The camera includes an imaging optical system for forming an image on the light-receiving surface, a photoelectric conversion element, and an image generation circuit for generating an image based on the output of the photoelectric conversion element. The photoelectric conversion elements included in the camera are arranged two-dimensionally on the light-receiving surface and are CMOS (Complementary Metal Oxide Semiconductor) and CCD (Charge Coupled Device) sensors or the like that output electrical signals according to the amount of incident light. The imaging unit 16 captures and acquires landscape images showing the scenery in the direction of travel of the ship 110, and supplies a landscape image signal showing the acquired landscape image to the processing unit 20.

The tilt detection unit 17 includes, for example, a three-axis acceleration sensor or a gyro sensor, and detects the tilt of the monitoring image generation device based on the indication of the three-axis acceleration sensor or gyro sensor, and supplies a tilt signal indicating the detected tilt to the processing unit 20.

The processing unit 20 is configured to comprehensively control the operation of the surveillance image generation device 1 and includes one or more processors and their peripheral circuits. The processing unit 20 includes, for example, a CPU (Central Processing Unit). The processing unit 20 may also include a DSP (Digital Signal Processor), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. The processing unit 20 executes processing based on programs stored in the memory unit 12. The processing unit 20 also controls the operation of each component of the surveillance image generation device 1 so that processing is executed appropriately.

The processing unit 20 has the following functional blocks: an information acquisition unit 21, a condition setting unit 22, an image information generation unit 23, a collision determination unit 24, and an output unit 25. These functional blocks are functional modules implemented based on a program executed by the processing unit 20. These functional blocks may also be firmware implemented in the surveillance image generation device 1.

Figure 4 is a flowchart showing the information setting process executed by the surveillance image generation device 1. The information setting process shown in Figure 4 is realized by the processing unit 20 working in cooperation with each component of the surveillance image generation device 1 based on a program stored in the memory unit 12. The information setting process shown in Figure 4 is executed each time the surveillance image generation device 1 is deployed on the ship 110.

First, the information acquisition unit 21 acquires first distance information indicating a first distance in response to operation of the operation unit 13 by the operator 112 (S101) and stores the acquired first distance information in the memory unit 12. The first distance is the distance from the ship 110 to the near clip plane of the view frustum. The first distance, which is the distance to the near clip plane of the view frustum, is used when generating, as a view frustum, a terrain image indicating the seabed terrain included in the superimposed image to be superimposed on the landscape image captured by the imaging unit 16. Next, the condition setting unit 22 sets the first distance acquired in the processing of S101 as information indicating the first distance to be used when generating the terrain image (S102). The condition setting unit 22 sets the first distance by associating the first distance information acquired in the processing of S101 with a predetermined identifier.

Next, the information acquisition unit 21 acquires display length information indicating the display length, which is the distance between the near clip plane and the far clip plane of the terrain image generated as a viewing frustum, in response to operation of the operation unit 13 by the operator 112 (S103). The information acquisition unit 21 stores the acquired display length information in the memory unit 12. Next, the condition setting unit 22 sets the display length acquired in the processing of S103 as information indicating the display length to be used when generating the terrain image (S104). The condition setting unit 22 sets the display length by associating the display length information acquired in the processing of S103 with a predetermined identifier.

Next, the information acquisition unit 21 acquires tilt information indicating the tilt of the monitoring image generation device 1 from a horizontal plane in response to operation of the operation unit 13 by the operator 112 (S105). When instructed to acquire tilt information via the operation unit 13, the information acquisition unit 21 requests the tilt detection unit 17 to output a tilt signal, and when the tilt signal is input, acquires tilt information indicating the tilt corresponding to the input tilt signal and stores the acquired tilt information in the storage unit 12. Next, the condition setting unit 22 sets the tilt information acquired in the processing of S105 as information indicating the tilt to be used when generating a superimposed image (S106). The condition setting unit 22 sets the tilt by associating the tilt information acquired in the processing of S105 with a predetermined identifier.

Next, the information acquisition unit 21 acquires height information indicating the height from sea level of the location where the monitoring image generation device 1 is located in response to operation of the operation unit 13 by the operator 112 (S107). The information acquisition unit 21 stores the acquired height information in the memory unit 12. Next, the condition setting unit 22 sets the height information acquired in the process of S107 as information indicating the height to be used when generating a superimposed image (S108). The condition setting unit 22 sets the height by associating the height information acquired in the process of S107 with a predetermined identifier. The height set in the process of S108 is the height of the imaging unit 16 from sea level.

Next, the information acquisition unit 21 acquires draft information indicating the draft of the vessel 110 in response to operation of the operation unit 13 by the operator 112 (S109), and stores the acquired draft information in the memory unit 12. Next, the condition setting unit 22 sets the draft information acquired in the processing of S109 as information indicating the draft to be used in the collision determination processing that determines whether the vessel 110 will collide with the seabed (S110). The condition setting unit 22 sets the draft by associating the draft information acquired in the processing of S109 with a predetermined identifier.

Figure 5(a) is a flowchart showing the surveillance image generation process executed by the surveillance image generation device 1. The surveillance image generation process shown in Figure 5(a) is realized by the processing unit 20 working in cooperation with each component of the surveillance image generation device 1 based on a program stored in the storage unit 12. The surveillance image generation process shown in Figure 5(a) is executed at a predetermined execution interval.

First, the image information generation unit 23 starts capturing a landscape image using the imaging unit 16 in response to operation of the operation unit 13 by the operator 112 (S201). When the operator 112 operates the operation unit 13 and receives an instruction to start capturing images via the operation unit 13, the image information generation unit 23 starts operation of the imaging unit 16, causing the imaging unit 16 to start capturing a landscape image in the direction of travel of the ship 110. The image information generation unit 23 acquires landscape image information indicating the landscape image captured by the imaging unit 16, and outputs the acquired landscape image information to the display unit 14. The display unit 14 displays a landscape image corresponding to the input landscape image information.

Next, the information acquisition unit 21 acquires current position information indicating the current position of the ship 110 (S202). The information acquisition unit 21 requests the GNSS receiver 101 to output a current position signal via the communication unit 11, and when the current position signal is input to the communication unit 11, acquires current position information indicating the current position corresponding to the input current position signal.

Next, the information acquisition unit 21 acquires heading information indicating the direction of travel of the ship 110 (S203). The information acquisition unit 21 requests the GNSS receiver 101 to output a heading signal via the communication unit 11, and when the heading signal is input to the communication unit 11, acquires heading information indicating the heading corresponding to the input heading signal.

Next, the information acquisition unit 21 acquires tide level information indicating the tide level at the current position of the vessel 110 (S204). The information acquisition unit 21 requests the satellite communication receiver 102 to output a current position signal via the communication unit 11, and when a tide level signal indicating the tide level at the current position of the vessel 110 is input to the communication unit 11, the information acquisition unit 21 acquires tide level information indicating the tide level corresponding to the input tide level signal.

Next, the image information generation unit 23 generates terrain image information showing a point on the ship 110, i.e., a terrain image, which is an image of the seabed terrain in the direction of travel of the ship 110 as seen through the monitoring image generation device 1 (S205), and stores the generated terrain image information in the memory unit 12.

Figure 5(b) is a flowchart showing the processing of S205 in more detail.

First, the image information generation unit 23 sets the range for generating a terrain image (S301) using the first distance, display length, and inclination set in the processes of S102, S104, and S106, as well as the current position information and traveling direction information acquired in the processes of S202 and S203. The range for generating a terrain image is called the terrain image range.

First, as shown in FIG. 6(a), the image information generation unit 23 sets a quadrangular pyramid R1 including a viewing frustum indicating the range for generating a terrain image from the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203. The apex P of the quadrangular pyramid R1 is the latitude and longitude indicating the current position corresponding to the current position information acquired in the processing of S202, and the extension direction of the normal L1 of the base SL of the quadrangular pyramid R1 is the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

Next, as shown in FIG. 6(b), the image information generation unit 23 sets the near clip plane S1 to a plane that is a first distance L1, set in the processing of S102, away from the vertex P of the pyramid R1 in the extension direction of the normal L1. The point at which the near clip plane S1 is located is also referred to as the first point. The near clip plane S1 is the end of the terrain image range closest to the current position. Next, as shown in FIG. 6(c), the image information generation unit 23 sets the far clip plane S2 to a plane that is a display length LD, set in the processing of S104, away from the near clip plane S1 in the extension direction of the normal L1. The point at which the far clip plane S2 is located is also referred to as the second point. The far clip plane S2 is the end of the terrain image range farthest from the current position.

Next, the image information generation unit 23 acquires elevation data included in the near clip plane S1 of the terrain image range determined in the processing of S301 (S302). The image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain superimposed on the near clip plane S1 at a position separated by a first distance from the current position in the direction of travel, and stores the acquired elevation data in the memory unit 12 as a first terrain data group.

Next, the image information generation unit 23 acquires elevation data contained in a plane parallel to the near clip plane S1 and spaced a predetermined plot interval from the near clip plane S1 in the normal line L1 direction (S303). Similar to the processing in S302, the image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain superimposed on the plane spaced a predetermined plot interval from the near clip plane S1, and stores the acquired elevation data in the memory unit 12 as a second terrain data group.

Next, the image information generation unit 23 determines whether the distance between the acquisition point, which is separated in the normal line L1 direction from the surface on which the elevation data was acquired in the processing of S303 by the plot interval, and the near clip plane S1 is longer than the display length LD set in the processing of S104 (S304). If it is determined that the distance between the acquisition point and the near clip plane S1 is not longer than the display length LD (S304-NO), the processing returns to S303. Thereafter, the processing of S303 and S304 is repeated until it is determined that the distance between the acquisition point and the near clip plane S1 is longer than the display length LD (S304-YES). By repeating the processing of S303 and S304, multiple terrain data groups are stored in the memory unit 12, with the data group numbers sequentially increasing, starting with the third terrain data group.

When the image information generation unit 23 determines that the distance between the acquisition point and the near clip plane S1 is longer than the display length (S304-YES), it stores the first to Nth terrain data groups acquired in the processes of S302 to S304 in the memory unit 12 as terrain image information (S305). The image information generation unit 23 stores the terrain image information in the memory unit 12, and the process of S205 ends.

Following the processing of S205, the image information generation unit 23 generates superimposed image information indicating a point on the ship 110, i.e., a superimposed image that can be superimposed on the landscape image captured by the imaging unit 16 (S206), and stores the generated superimposed image information in the memory unit 12.

Figure 7 is a flowchart showing the processing of S206 in more detail.

First, the image information generation unit 23 sets the imaging range to be captured by the imaging unit 16 (S401). As shown in FIG. 8(a), the image information generation unit 23 sets an imaging range R2, which is a quadrangular pyramid that indicates the range for generating a terrain image, based on the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

Next, the image information generation unit 23 places the terrain image corresponding to the terrain image information generated in the processing of S205 in the imaging range R2 set in the processing of S401 (S402). As shown in Figure 8 (b), the image information generation unit 23 acquires the terrain image information stored in the memory unit 12, and places the terrain image I1, which is a point cloud image indicating each point representing the elevation data included in the terrain data group corresponding to the acquired terrain image information, in the imaging range R2.

The image information generation unit 23 extracts structure information including structures included in the imaging range R2 based on the latitude and longitude included in the structure information stored in the memory unit 12. As shown in Figure 8 (c), the image information generation unit 23 places a structure image I2 showing the shape included in the structure information extracted based on the latitude and longitude in the imaging range R2.

The image information generation unit 23 then arranges the route information included in the imaging range in the imaging range R2 estimated in the processing of S401 (S404). The route information indicates route lines indicating routes along which the ship 110 can proceed, and areas that are preferably visible when maneuvering the ship 110, including no-entry areas into which the ship 110 is prohibited. The image information generation unit 23 extracts route information indicating the area included in the imaging range R2 based on the latitude and longitude included in the route information stored in the memory unit 12. As shown in FIG. 8(d), the image information generation unit 23 arranges an area image I3 indicating the shape included in the route information extracted based on the latitude and longitude in the imaging range R2.

Next, the image information generation unit 23 places ships included in the imaging range R2 in the imaging range R2 estimated in the processing of S401 (S405). The image information generation unit 23 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input AIS signal. The image information generation unit 23 extracts AIS ships included in the imaging range R2 from the acquired signal related to the ship, and stores the ship's name, captain, speed, and position from the AIS signal of the extracted AIS ship in the memory unit 12. The image information generation unit 23 also requests the server that transmits the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input non-AIS signal. The image information generation unit 23 extracts non-AIS ships within the imaging range R2 from the acquired ship-related signals, and stores the ship's name, speed, and position from the extracted non-AIS signal in the memory unit 12. As shown in FIG. 8(e), the image information generation unit 23 places ship images according to their distance from the ship 110 at the positions of the AIS and non-AIS ships stored in the memory unit 12, and places images including the names and speeds of the AIS and non-AIS ships in association with the ship image I4. Note that the images placed in association with the ship image may include the captain of the AIS ship and the distance between the AIS and non-AIS ships and the ship 110, in addition to the names and speeds of the AIS and non-AIS ships.

Next, the image information generation unit 23 tilts the imaging range in which the undersea topography, etc., was located in steps S402 to S405, in accordance with the tilt of the monitoring image generation device 1, so that it is visible at an elevation angle that corresponds to the tilt of the monitoring image generation device 1 (S406). As shown in FIG. 8(f), the image information generation unit 23 tilts the imaging range R2 by a tilt θ in accordance with the tilt that corresponds to the tilt information set in step S106.

Next, the image information generation unit 23 sets the height of the imaging range R2 from sea level so that the height of the imaging range R2 from sea level matches the height of the monitoring image generation device 1 from sea level (S407). The image information generation unit 23 sets the height of the imaging range R2 so that it matches the height set in the processing of S108.

Next, the image information generation unit 23 corrects the height of the imaging range R2 set in the processing of S407 in accordance with the tide level at the current location (S408). The information acquisition unit 21 corrects the height of the imaging range R2 in accordance with the tide level corresponding to the tide level information acquired in the processing of S204.

Next, the image information generation unit 23 generates a direction image indicating the direction (S409). The image information generation unit 23 generates an isosceles triangular image as a direction image, with a slope corresponding to the angle difference between the direction of travel corresponding to the travel direction information acquired in the processing of S203 and the north direction, and stores direction image information indicating the generated direction image. The direction image may be an image other than an isosceles triangular image.

Next, the image information generation unit 23 generates superimposed image information by combining the azimuth image generated in the processing of S406 with the imaging range R2 (S410), and stores the generated superimposed image information in the storage unit 12. The processing of S206 ends when the image information generation unit 23 stores the superimposed image information in the storage unit 12.

Following the processing of S206, the collision determination unit 24 executes collision determination processing (S207).

Figure 9 shows more detailed processing of the collision detection process shown in S207.

First, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography at a point a predetermined distance away in the direction of travel obtained in the processing of S203 (S501), and stores elevation information indicating the extracted elevation in the memory unit 12. For example, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography between the near clip plane S1 and the far clip plane S2.

Next, the collision determination unit 24 calculates the water depth at the point whose elevation was extracted in S501 from the tide level information acquired in S204 and the elevation extracted in S501 (S502). The collision determination unit 24 calculates the water depth by adding the tide level corresponding to the tide level information to the tide level information acquired in S204 and the elevation extracted in S501.

Next, the collision determination unit 24 calculates the separation distance between the bottom of the vessel 110 and the seabed from the draft set in the processing of S110 and the water depth calculated in the processing of S502 (S503). The collision determination unit 24 stores the calculated separation distance in the memory unit 12.

Next, the collision determination unit 24 determines whether there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504). The collision determination unit 24 acquires warning threshold information stored in the memory unit 12. The collision determination unit 24 compares the warning threshold corresponding to the acquired warning threshold information with the separation distance calculated in the processing of S503, and determines whether the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold. If the separation distance calculated in the processing of S503 is less than the warning threshold, the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES). On the other hand, if the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold, the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO).

If the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES), it outputs a collision warning signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed to the display unit 14 and the audio output unit 15. The display unit 14 displays a collision area image indicating the area where the bottom of the vessel 110 is likely to come into contact with the seabed, superimposed on a landscape image. In response to the input of the collision warning signal, the audio output unit 15 outputs an audio signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed. If the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO), the processing of S207 ends. Note that instead of displaying the collision area image, text indicating a warning by type, such as "Caution: Contact with ground" or "Caution: Entering no-approach area," may be displayed.

Following the processing of S207, the information acquisition unit 21 acquires scenery image information indicating the scenery image captured by the imaging unit 16 (S208). The information acquisition unit 21 requests the imaging unit 16 to output a scenery image signal indicating the scenery image captured by the imaging unit 16, and acquires scenery image information indicating the scenery image captured by the imaging unit 16.

Next, the image information generation unit 23 superimposes the superimposed image corresponding to the superimposed image information generated in the process of S206 onto the scenery image corresponding to the scenery image information acquired in the process of S208, thereby generating monitoring image information (S209).

Next, the output unit 25 outputs the surveillance image information generated in the processing of S209 to the display unit 14 (S210). In response to the input of the surveillance image information, the display unit 14 displays a surveillance image corresponding to the input surveillance image information.

Figure 10 shows an example of a monitoring image displayed on the display unit 14.

The monitoring image 120 includes a terrain image 121, a structure image 122, a route image 123, and a ship image 124, and is displayed on the display unit 14 superimposed on a landscape image captured by the imaging unit 16. The terrain image 121 is formed from a point cloud and shows the topography of the seabed. The structure image includes multiple buoys and shows structures located on land and at sea. The route image 123 includes a route line image 125 and a no-entry area image 126. The ship image 124 includes a ship image 127 showing a ship and a ship information image 128 showing information about the ship. A collision warning area image 129 corresponding to a collision warning signal is also superimposed on the monitoring image 120. Furthermore, a warning image 130 indicating various warnings is also superimposed on the monitoring image 120. The route image 123 may also include a navigable width corresponding to the width of the ship sailing toward the ship 110. The monitoring image generating device 1 may also output an alarm signal when the ship 110 enters the navigable width included in the route image 123.

Following the processing of S208, the image information generation unit 23 determines whether an instruction to end the image generation processing has been received (S211). When a processing end signal is input in response to operation of the operation unit 13 by the operator 112, the image information generation unit 23 determines that an instruction to end the image generation processing has been received (S211-YES). If the image information generation unit 23 determines that an instruction to end the image generation processing has not been received (S211-NO), the processing returns to S202. Thereafter, the processing of S202 to S211 is repeated until the image information generation unit 23 determines that an instruction to end the image generation processing has been received (S211-YES).

If the image information generation unit 23 determines that an instruction to end the image generation process has been issued (S211-YES), the capturing of the landscape image by the imaging unit 16 is terminated in response to operation of the operation unit 13 by the operator 112 (S212).

(Operational effects of the monitoring image generating device according to the first embodiment)
The monitoring image generating device 1 can reduce the risk of ships and structures on the sea becoming difficult to see by superimposing a topographical image of the seabed between the near clip plane S1 and the far clip plane S2 on the landscape image, rather than displaying a topographical image of the entire seabed included in the landscape image. For example, by making the display length LD very short and positioning the near clip plane S1 and the far clip plane S2 close to each other, it is possible to display the cross-sectional shape of the seabed at a point separated by the first distance L1 from the ship 110 as a topographical image.

In addition, the monitoring image generation device 1 displays an image of the seabed topography between the near clip plane S1 and the far clip plane S2, which are positioned in the direction of travel of the ship 110, allowing the operator 112 to easily grasp the topography of the seabed in the direction of travel of the ship 110.

In addition, by using the elevation tiles 12a, the monitoring image generation device 1 can quickly acquire the seabed topography. Furthermore, since the monitoring image generation device 1 generates a topographical image by forming the seabed topography as a point cloud, it can quickly generate and display a topographical image of the seabed.

In addition, the monitoring image generation device 1 corrects the position at which the seabed topography image is superimposed on the landscape image according to the tide level at the ship 110's current location, so the seabed topography image can be superimposed on the landscape image with high accuracy regardless of changes in the tide level.

In addition, the surveillance image generation device 1 generates a superimposed image that includes ships and land and sea structures contained in the landscape image, so the operator 112 can easily see the structures even when the landscape image captured by the imaging unit 16 is unclear due to nighttime conditions, thick fog, etc.

In addition, the surveillance image generation device 1 generates superimposed image information so that it is visible at an elevation angle that corresponds to the tilt of the surveillance image generation device 1, so the superimposed image can be superimposed on the landscape image regardless of the tilt of the surveillance image generation device 1.

In addition, the monitoring image generating device 1 outputs an alarm signal when there is a risk that the vessel 110 will come into contact with the seabed, allowing the operator 112 to maneuver the vessel 110 in a way that reduces the risk of contact with the seabed.

In addition, the monitoring image generating device 1 determines whether there is a risk of the ship 110 coming into contact with the seabed based on the tide level at the ship 110's current location, in addition to the ship's draft and the seabed elevation, so it can more accurately determine whether the ship 110 is at risk of coming into contact with the seabed.

(Modification of the monitoring image generating device according to the first embodiment)
Although the surveillance image generating device 1 arranges on the superimposed image not only the topographical image but also land and sea structures, route information, and other ships, in the surveillance image generating device according to the embodiment, only the topographical image may be arranged in the superimposed image. Also, in the surveillance image generating device according to the embodiment, the superimposed image may arrange the topographical image and any of structures, route information, and other ships.

Furthermore, while the surveillance image generation device 1 generates terrain images and superimposed images as a viewing frustum, the surveillance image generation device according to the embodiment may also generate terrain images and superimposed images by performing image processing such as affine transformation and projective transformation on the seabed terrain, etc.

Furthermore, while the surveillance image generation device 1 generates terrain image information using elevation tiles, the surveillance image generation device according to the embodiment may also generate terrain image information using information acquired using a shape detection sensor such as sonar. Furthermore, while the surveillance image generation device 1 generates terrain images by forming the seabed terrain as a point cloud, the surveillance image generation device according to the embodiment may also generate terrain images by forming the seabed terrain as a line drawing of contour lines.

Furthermore, while the surveillance image generation device 1 corrects the position at which the superimposed image is superimposed on the landscape image according to the tide level at the current location of the ship 110, the surveillance image generation device according to the embodiment does not need to perform correction according to the tide level. Further, while the surveillance image generation device according to the embodiment performs collision detection processing using the tide level at the current location of the ship 110, the surveillance image generation device according to the embodiment may perform collision detection processing using the ship's draft and the seabed elevation without using the tide level.

Furthermore, while the surveillance image generation device 1 has an operation unit 13 to an imaging unit 16, the surveillance image generation device according to the embodiment may not have any of the operation unit 13 to the imaging unit 16, and the operation unit 13 to the imaging unit 16 may be arranged as devices separate from the surveillance image generation device.

Furthermore, while the surveillance image generation system 100 having the surveillance image generation device 1 superimposes a superimposed image on a landscape image visible from a single ship 110, the surveillance image generation system according to the embodiment may be implemented on multiple ships. When the surveillance image generation system according to the embodiment is implemented on multiple ships, the landscape image captured by the surveillance image generation system and the superimposed image superimposed on the landscape image may be displayable by the ship monitoring device.

(Ship monitoring device according to the embodiment)
FIG. 11 is a functional block diagram of a vessel monitoring device capable of displaying a landscape image captured by a monitoring image generating system and a superimposed image superimposed on the landscape image.

The vessel monitoring device 150 has a monitoring communication unit 151, a monitoring memory unit 152, a monitoring operation unit 153, a monitoring display unit 154, a monitoring audio output unit 155, and a monitoring processing unit 160, and displays landscape images and superimposed images captured by the monitoring image generation device 1 mounted on multiple vessels 110. Each of the multiple vessels 110 is equipped with a monitoring image generation device 1. The monitoring communication unit 151, the monitoring audio output unit 155, and the monitoring processing unit 160 are connected via a bus 156.

The monitoring communication unit 151 is configured to enable communication with the server that outputs AIS and non-AIS signals, and the monitoring image generation device 1 via the Internet, and is equipped with a communication interface circuit. The communication interface circuit is, for example, a wired LAN, wireless LAN, LTE, or other communication interface circuit. It supplies signals input from the server that outputs AIS and non-AIS signals, and the monitoring image generation device 1 to the monitoring processing unit 160, and outputs signals supplied from the monitoring processing unit 160 to the server that outputs AIS and non-AIS signals, and to the monitoring image generation device 1.

The monitoring memory unit 152 is configured to store data and programs, and includes, for example, semiconductor memory. The monitoring memory unit 152 stores operating system programs, driver programs, application programs, data, and the like used in processing by the monitoring processing unit 160. For example, the monitoring memory unit 152 stores, as an application program, an image display program that causes the monitoring processing unit 160 to execute image display processing that displays monitoring images generated by the monitoring image generation devices 1 installed on multiple ships 110. The image display program is installed into the monitoring memory unit 152 from a computer-readable, non-transitory, portable storage medium such as a CD-ROM or DVD-ROM. The monitoring memory unit 152 also stores various data and information used when executing the image display processing.

The monitoring operation unit 153 is configured to accept user operations on the vessel monitoring device 150 and includes, for example, a keyboard. The monitoring operation unit 153 generates signals in response to the user operations and supplies the generated signals to the monitoring processing unit 160.

The monitoring display unit 154 is configured to display images and includes a display such as a liquid crystal display or an organic EL display. The monitoring display unit 154 generates and displays images based on signals supplied from the monitoring processing unit 160.

The monitoring audio output unit 155 is configured to output audio and includes a speaker. The monitoring audio output unit 155 outputs audio based on a signal supplied from the monitoring processing unit 160.

The monitoring processing unit 160 is configured to comprehensively control the operation of the ship monitoring device 150, and includes one or more processors and their peripheral circuits. The monitoring processing unit 160 includes, for example, a CPU. The monitoring processing unit 160 may also include a DSP, LSI, ASIC, FPGA, etc. The monitoring processing unit 160 executes processing based on programs stored in the monitoring memory unit 152. The monitoring processing unit 160 also controls the operation of each component of the ship monitoring device 150 so that processing is executed appropriately.

The monitoring processing unit 160 has the following functional blocks: a monitoring information acquisition unit 161, a monitoring decision unit 162, a monitoring image information generation unit 163, and a monitoring output unit 164. These functional blocks are functional modules implemented based on a program executed by the monitoring processing unit 160. These functional blocks may also be firmware implemented in the monitoring image generation device 1.

Figure 12 is a flowchart showing the image display processing executed by the vessel monitoring device 150. The information setting processing is realized by the monitoring processing unit 160 working in cooperation with each component of the vessel monitoring device 150 based on a program stored in the monitoring memory unit 152. The image display processing shown in Figure 12 is executed at a predetermined execution cycle.

First, the monitoring information acquisition unit 161 acquires AIS ship information (S501) and stores the acquired AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the monitoring communication unit 151, it acquires AIS ship information, including the ship's name and location, from the input AIS signal.

Next, the monitoring information acquisition unit 161 acquires non-AIS ship information (S502) and stores the acquired non-AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the monitoring communication unit 151, it acquires non-AIS ship information, including the ship's name and location, from the input non-AIS signal.

Next, the monitoring determination unit 162 determines the monitored ship to be monitored from the AIS ship information acquired in the processing of S501 and the non-AIS ship information acquired in the processing of S502 (S503). The monitoring determination unit 162 may determine the monitored ship to be a ship located within the port, or may determine the monitored ship to be a ship located within a specified area of the port.

Next, the monitoring information acquisition unit 161 acquires monitoring image information from the ship determined to be the monitored ship in the processing of S503 (S504). The monitoring information acquisition unit 161 requests the monitoring image generation device 1 mounted on the ship 110 determined to be the monitored ship to output a monitoring image signal, and when the monitoring image signal is input to the monitoring communication unit 151, acquires monitoring image information from the input monitoring image signal. The monitoring image information acquired in the processing of S504 may be a moving image or a still image. When the monitoring image information acquired in the processing of S504 is a still image, the performance and communication speed of the monitoring image generation device 1 and the monitoring image 120 can be lower than when the monitoring image information acquired in the processing of S504 is a moving image.

Next, the monitoring image information generation unit 163 generates display image information (S505) and stores the generated display image information in the monitoring storage unit 152. The monitoring output unit 164 then outputs the display image information generated in the processing of S506 to the monitoring display unit 154. In response to the input of the display image information, the monitoring display unit 154 displays a display image corresponding to the display image information.

Figure 13 shows an example of a display image displayed on the monitoring display unit 154.

Display image 170 has a first split image 171, a second split image 172, a third split image 173, and a fourth split image 174. First split image 171 to third split image 173 display landscape images and superimposed images acquired from three ships included in the monitored area indicated by arrow A in fourth split image 174, along with monitored object information 171a, 172a, and 173a. Monitored object information 171a, 172a, and 173a includes the name of the monitored ship and the time the displayed image was captured.

The fourth divided image 174 displays the monitored area and the position information of ships located near the monitored area in real time as a two-dimensional planar image.

(Configuration and Function of Surveillance Image Generation System Having Surveillance Image Generation Device According to Second Embodiment)
FIG. 14 is a perspective view of a vehicle equipped with a surveillance image generation system according to the second embodiment, FIG. 15 is a diagram showing a surveillance image generation device mounted on the vehicle shown in FIG. 14, and FIG. 16 is a functional block diagram of the surveillance image generation device shown in FIG. 15.

The surveillance image generation system 200 includes a GNSS receiver 201, a wireless communication device 202, an operating device 203, a display device 204, an audio output device 205, an imaging device 206, and a surveillance image generation device 2, which is another example of an image surveillance device according to the present invention, and is mounted on a vehicle 210, which is another example of a moving object. Based on signals received from the GNSS receiver 201, the wireless communication device 202, and the imaging device 206, the surveillance image generation system 200 displays a landscape image and a superimposed image on the landscape image on a screen visible to the driver of the vehicle 210.

The GNSS receiver 201 receives signals from a satellite positioning system such as a global positioning system. In response to receiving signals from the GNSS, the GNSS receiver 201 outputs to the monitoring image generation device 2 a current position signal indicating the current position of the vehicle 210 and a heading direction signal indicating the direction in which the vehicle 210 is traveling. The wireless communication device 202 is a communication device that can access the Internet network via the LTE network, and receives signals indicating various information. For example, the wireless communication device 202 receives point cloud signals indicating three-dimensional point cloud data collected by the Ministry of Land, Infrastructure, Transport and Tourism from the website of the Japan Digital Road Map Association (https://www.drm.jp/pointcloud/).

The operation device 203 is configured to accept user operations on the surveillance image generation device 2 and includes, for example, a touch panel. The operation device 203 generates a signal in response to the driver's operation and outputs the generated signal to the surveillance image generation device 2.

The display device 204 is configured to display images and includes a display such as a liquid crystal display or an organic EL display. The display device 204 generates and displays images based on signals input from the monitoring image generation device 2.

The audio output device 205 is configured to output audio and includes a speaker. The audio output device 205 outputs audio based on the signal input from the monitoring image generation device 2.

The imaging device 206 is configured to capture images of the area around the surveillance image generation device 2, and is equipped with a camera. It is positioned so that it is visible to the driver and can capture images in the direction of travel of the vehicle 210. The camera includes an imaging optical system for forming an image on the light-receiving surface, a photoelectric conversion element, and an image generation circuit that generates an image based on the output of the photoelectric conversion element. The photoelectric conversion elements included in the camera are CMOS and CCD sensors, etc., that are arranged two-dimensionally on the light-receiving surface and output electrical signals corresponding to the amount of incident light. The imaging device 206 captures and acquires landscape images showing the scenery in the direction of travel of the vehicle 210, and outputs landscape image signals showing the acquired landscape images to the surveillance image generation device 2.

The monitoring image generating device 2 has a communication unit 31, a memory unit 32, and a processing unit 40, and is arranged inside the cabin 213 of the vehicle 210 so that it is visible to the driver. The communication unit 31, memory unit 32, and processing unit 40 are communicatively connected via a bus 33.

The communication unit 31 is configured to enable the monitoring image generation device 2 to communicate with the GNSS receiver 201 through the imaging device 206, and includes a communication interface circuit. The communication interface circuit is, for example, a wired LAN, wireless LAN, LTE, or other communication interface circuit. The communication unit 31 supplies signals input from the GNSS receiver 201 through the imaging device 206 to the processing unit 40, and outputs signals supplied from the processing unit 40 to the GNSS receiver 201 through the imaging device 206.

The memory unit 32 is configured to store data and programs, and includes, for example, a semiconductor memory. The memory unit 32 stores operating system programs, driver programs, application programs, data, etc. used in processing by the processing unit 40. For example, the memory unit 32 stores, as an application program, an information setting program that causes the processing unit 40 to execute an information setting process that sets information used when executing an image generation process that generates an image including a terrain image showing the terrain of the road on which the vehicle 210 travels. The memory unit 32 also stores, as an application program, an image generation program that causes the processing unit 40 to execute the image generation process. The information setting program and image generation program are installed into the memory unit 32 from a computer-readable, non-transitory, portable storage medium such as a CD-ROM or DVD-ROM. The memory unit 32 also stores warning threshold information that indicates a warning threshold used when determining whether the body of the vehicle 210 will come into contact with the road. The memory unit 32 also stores various data and information used when executing the information setting process and the image generation process. For example, the storage unit 32 stores point cloud data indicating the topography of roads, also referred to as topographical information, as point cloud information 32a.

Point cloud information 32a includes information indicating the latitude, longitude, and altitude of each point included in the point cloud. For example, it is available from the website of the Japan Digital Road Map Association, a general incorporated foundation. The structure and functions of point cloud information 32a are well known, so a detailed explanation will be omitted here.

The processing unit 40 is configured to comprehensively control the operation of the surveillance image generation device 2, and includes one or more processors and their peripheral circuits. The processing unit 40 includes, for example, a CPU. The processing unit 40 may also include a DSP, LSI, ASIC, FPGA, etc. The processing unit 40 executes processing based on programs stored in the storage unit 32. The processing unit 40 also controls the operation of each component of the surveillance image generation device 2 so that processing is executed appropriately.

The processing unit 40 has the following functional blocks: an information acquisition unit 41, a condition setting unit 42, an image information generation unit 43, a collision determination unit 44, and an output unit 45. These functional blocks are functional modules implemented based on a program executed by the processing unit 40. These functional blocks may also be firmware implemented in the surveillance image generation device 2.

Figure 17 is a flowchart showing the information setting process executed by the surveillance image generation device 2. The information setting process is realized by the processing unit 40 working in cooperation with each component of the surveillance image generation device 2 based on a program stored in the storage unit 32. The information setting process shown in Figure 17 is executed each time the surveillance image generation device 2 is placed in the vehicle 210.

First, the information acquisition unit 41 acquires first distance information indicating a first distance in response to the driver's operation of the operating device 203 (S601) and stores the acquired first distance information in the storage unit 32. The first distance is the distance from the vehicle 210 to the near clip plane of the viewing frustum of a terrain image indicating the terrain included in the superimposed image superimposed on the landscape image captured by the imaging device 206. Next, the condition setting unit 42 sets the first distance acquired in the processing of S601 as information indicating the first distance to be used when generating the terrain image (S602). The condition setting unit 42 sets the first distance by associating the first distance information acquired in the processing of S601 with a predetermined identifier.

Next, the information acquisition unit 41 acquires display length information indicating the display length, which is the distance between the near clip plane and the far clip plane of the terrain image generated as a viewing frustum, in response to the driver's operation of the operating device 203 (S603). The information acquisition unit 41 stores the acquired display length information in the storage unit 32. Next, the condition setting unit 42 sets the display length acquired in the processing of S603 as information indicating the display length to be used when generating the terrain image (S604). The condition setting unit 42 sets the display length by associating the display length information acquired in the processing of S603 with a predetermined identifier.

Next, the information acquisition unit 41 acquires tilt information indicating the tilt of the imaging device 206 from the horizontal plane in response to the driver's operation of the operation device 203 (S605). The information acquisition unit 41 stores the acquired tilt information in the storage unit 32. Next, the condition setting unit 42 sets the tilt information acquired in the processing of S605 as information indicating the tilt to be used when generating a superimposed image (S606). The condition setting unit 42 sets the tilt by associating the tilt information acquired in the processing of S605 with a predetermined identifier.

Next, the information acquisition unit 41 acquires imaging height information indicating the height from the ground of the location where the imaging device 206 is located in response to the driver's operation of the operating device 203 (S607). The information acquisition unit 41 stores the acquired imaging height information in the storage unit 32. Next, the condition setting unit 42 sets the imaging height information acquired in the processing of S607 as information indicating the height to be used when generating a superimposed image (S608). The condition setting unit 42 sets the height by associating the imaging height information acquired in the processing of S607 with a predetermined identifier.

Next, the information acquisition unit 41 acquires ground distance information indicating the vertical distance between the bottom of the body 211 of the vehicle 210 and the bottom of the wheels 212 that movably support the body 211 in response to the driver's operation of the operating device 203 (S609). The information acquisition unit 41 stores the acquired ground distance information in the memory unit 32. Next, the condition setting unit 42 sets the ground distance information acquired in the processing of S609 as information indicating the ground distance to be used in the collision determination processing that determines whether the body 211 will collide with the ground surface (S610). The condition setting unit 42 sets the ground distance by associating the ground distance information acquired in the processing of S609 with a predetermined identifier.

Figure 18(a) is a flowchart showing the surveillance image generation process executed by the surveillance image generation device 2. The surveillance image generation process is realized by the processing unit 40 working in cooperation with each component of the surveillance image generation device 2 based on a program stored in the storage unit 32. The surveillance image generation process shown in Figure 18(a) is executed at a predetermined execution interval.

First, the image information generation unit 43 starts capturing a scenery image using the imaging device 206 in response to the driver's operation of the operation device 203 (S701). When the driver operates the operation device 203 and receives an instruction to start capturing images via the operation device 203, the image information generation unit 43 starts operation of the imaging device 206, causing the imaging device 206 to start capturing a scenery image in the traveling direction of the vehicle 210. The image information generation unit 43 acquires scenery image information indicating the scenery image captured by the imaging device 206, and outputs the acquired scenery image information to the display device 204. The display device 204 displays a scenery image corresponding to the input scenery image information.

Next, the information acquisition unit 41 acquires current position information indicating the current position of the vehicle 210 (S702). The information acquisition unit 41 requests the GNSS receiver 201 to output a current position signal via the communication unit 31, and when the current position signal is input to the communication unit 31, acquires current position information indicating the current position corresponding to the input current position signal.

Next, the information acquisition unit 41 acquires traveling direction information indicating the traveling direction of the vehicle 210 (S703). The information acquisition unit 41 requests the GNSS receiver 201 to output a traveling direction signal via the communication unit 31, and when the traveling direction signal is input to the communication unit 31, acquires traveling direction information indicating the traveling direction corresponding to the input traveling direction signal.

Next, the image information generation unit 43 generates terrain image information showing a point on the vehicle 210, i.e., a terrain image, which is an image of the ground surface and structures in the direction of travel of the vehicle 210 as viewed from the imaging device 206 (S704), and stores the generated terrain image information in the memory unit 32.

Figure 18(b) is a flowchart showing the processing of S704 in more detail.

First, the image information generation unit 43 sets the range for generating a terrain image (S801) using the first distance, display length, and inclination set in the processes of S602, S604, and S606, as well as the current position information and traveling direction information acquired in the processes of S702 and S703. The range for generating a terrain image is referred to as the terrain image range. The process of S801 is similar to the process of S301, so a detailed description will be omitted here.

Next, the image information generation unit 43 acquires information indicating the height of each point included in the near clip plane of the terrain image range determined in the processing of S801 (S802). The image information generation unit 43 references the point cloud information 32a to acquire information indicating the height of the terrain superimposed on the near clip plane at a position separated by a first distance from the current position in the direction of travel, and stores the acquired information in the storage unit 32 as a first terrain data group.

Next, the image information generation unit 43 acquires information indicating the height of each point included in a plane that is separated from the near clip plane by a predetermined plot interval in the direction of travel and is parallel to the near clip plane (S803). The image information generation unit 43 references the point cloud information 32a and performs processing similar to that of S802 to acquire information indicating the height of the terrain superimposed on the plane separated from the near clip plane by a predetermined plot interval, and stores the acquired information in the storage unit 32 as a second terrain data group.

Next, the image information generation unit 43 determines whether the distance between the acquisition point, which is separated in the normal line L1 direction from the surface for which information was acquired in the processing of S803 by the plot interval, and the near clip plane is longer than the display length set in the processing of S604 (S804). If it is determined that the distance between the acquisition point and the near clip plane is not longer than the display length (S804-NO), processing returns to S803. Thereafter, the processing of S803 and S804 is repeated until it is determined that the distance between the acquisition point and the near clip plane is longer than the display length (S804-YES). By repeating the processing of S803 and S804, multiple terrain data groups are stored in the memory unit 32, with the data group numbers sequentially increasing, starting with the third terrain data group.

If the image information generation unit 43 determines that the distance between the acquisition point and the near clip plane is longer than the display length (S804-YES), it stores the first to Nth terrain data groups acquired in the processes of S802 to S804 in the storage unit 32 as terrain image information (S805). The image information generation unit 43 stores the terrain image information in the storage unit 32, and the processing of S704 ends.

Following the processing of S704, the image information generation unit 43 generates superimposed image information indicating a superimposed image that can be superimposed on a point on the vehicle 210, i.e., a landscape image captured by the imaging device 206 (S705), and stores the generated superimposed image information in the storage unit 32.

Figure 19(a) is a flowchart showing the processing of S705 in more detail.

First, the image information generation unit 43 sets the imaging range to be captured by the imaging device 206 (S901). Next, the image information generation unit 43 arranges the terrain image corresponding to the terrain image information generated in the processing of S704 in the imaging range set in the processing of S901 (S902). Next, the image information generation unit 43 tilts the imaging range in which the terrain is arranged in the processing of S902, in accordance with the tilt of the imaging device 206, so that it is visible at an elevation angle corresponding to the tilt of the imaging device 206 (S903). The processing of S901 to S903 is similar to the processing of S401, S402, and S406, so detailed description will be omitted here.

Next, the image information generation unit 43 sets the height of the imaging range so that the height of the imaging range from the ground surface matches the height of the imaging device 206 from the ground surface (S904). The image information generation unit 43 sets the height of the imaging range so that it matches the height set in the processing of S608.

Next, the image information generation unit 43 generates a direction image indicating the direction (S905). The image information generation unit 43 generates an isosceles triangular image as a direction image, with a slope corresponding to the angle difference between the direction of travel corresponding to the direction of travel information acquired in the processing of S703 and the north direction, and stores direction image information indicating the generated direction image.

Next, the image information generation unit 43 generates superimposed image information by combining the azimuth image generated in the processing of S905 with the imaging range (S906), and stores the generated superimposed image information in the storage unit 32. The processing of S705 ends when the image information generation unit 43 stores the superimposed image information in the storage unit 32.

Following the processing of S705, the collision determination unit 44 executes collision determination processing (S706).

Figure 19(b) shows a more detailed view of the collision detection process shown in S706.

First, the collision determination unit 44 extracts information indicating the height of the terrain at a point a predetermined distance away in the direction of travel obtained in the processing of S703 from the point cloud information 32a (S1001), and stores the extracted information in the storage unit 32. For example, the collision determination unit 44 extracts information indicating the height of the terrain between the near clip plane and the far clip plane from the point cloud information 32a.

Next, the collision determination unit 44 calculates the separation distance between the body 211 of the vehicle 210 and the ground surface (S1002). The separation distance is calculated by subtracting the value indicating the vertical direction of the ground surface height information from the value indicating the vertical direction of the coordinates extracted in the processing of S1001, from the length corresponding to the ground surface distance information set in the processing of S610. The collision determination unit 44 stores the calculated separation distance in the memory unit 32.

Next, the collision determination unit 44 determines whether there is a risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003). The collision determination unit 44 acquires warning threshold information stored in the memory unit 32. The collision determination unit 44 compares the warning threshold corresponding to the acquired warning threshold information with the separation distance calculated in the processing of S1002, and determines whether the separation distance calculated in the processing of S1002 is equal to or greater than the warning threshold. If the separation distance calculated in the processing of S1002 is less than the warning threshold, the collision determination unit 44 determines that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-YES). On the other hand, if the separation distance calculated in the processing of S1002 is equal to or greater than the warning threshold, the collision determination unit 44 determines that there is no risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-NO).

If the collision determination unit 44 determines that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-YES), it outputs a collision warning signal indicating that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground to the display device 204 and the audio output device 205. The display device 204 displays a collision area image indicating the area where the body 211 of the vehicle 210 is at risk of coming into contact with the ground, superimposed on a landscape image. In response to input of the collision warning signal, the audio output device 205 outputs an audio signal indicating that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground. If the collision determination unit 44 determines that there is no risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-NO), the processing of S706 ends.

Following the processing of S706, the information acquisition unit 41 acquires scenery image information indicating the scenery image captured by the imaging device 206 (S707). The information acquisition unit 41 requests the imaging device 206 via the communication unit 31 to output a scenery image signal indicating the scenery image captured by the imaging device 206, and when the scenery image signal is input to the communication unit 31, acquires scenery image information indicating the scenery image corresponding to the input scenery image signal.

Next, the image information generation unit 43 superimposes the superimposed image corresponding to the superimposed image information generated in the processing of S705 onto the scenery image corresponding to the scenery image information acquired in the processing of S707, thereby generating monitoring image information (S708).

Next, the output unit 45 outputs the monitoring image information generated in the processing of S708 to the display device 204 (S709). In response to the input of the monitoring image information, the display device 204 displays a monitoring image corresponding to the input monitoring image information.

Next, the image information generation unit 43 determines whether an instruction to end the image generation process has been received (S710). When a processing end signal is input in response to the driver's operation of the operation device 203, the image information generation unit 43 determines that an instruction to end the image generation process has been received (S710-YES). If the image information generation unit 43 determines that an instruction to end the image generation process has not been received (S710-NO), the process returns to S702. Thereafter, the processes of S702 to S710 are repeated until the image information generation unit 43 determines that an instruction to end the image generation process has been received (S710-YES).

If the image information generation unit 43 determines that an instruction to end the image generation process has been issued (S710-YES), the imaging device 206 ends capturing the scenery image in response to the driver's operation of the operation device 203 (S711).

(Operational effects of the monitoring image generating device according to the second embodiment)
The surveillance image generating device 2 can reduce the risk of the scenery image becoming difficult to view by superimposing a terrain image of the terrain between the near clip plane and the far clip plane on the scenery image, rather than a terrain image of the terrain included in the scenery image. For example, by very shorting the display length and positioning the near clip plane and the far clip plane close to each other, it is possible to display the cross-sectional shape of the terrain at a point a first distance away from the vehicle 210 as a terrain image.

In addition, the monitoring image generating device 2 displays a terrain image of the terrain between the near clip plane and the far clip plane, which are located in the direction of travel of the vehicle 210, allowing the driver to easily grasp the terrain in the direction of travel of the vehicle 210.

Furthermore, by using the point cloud information 32a, the monitoring image generation device 2 can acquire the terrain accurately and quickly. Furthermore, because the monitoring image generation device 2 forms the terrain using a point cloud to generate a terrain image, it can generate and display the terrain image with high accuracy and quickly.

In addition, the surveillance image generating device 2 generates superimposed image information so that it is visible at an elevation angle corresponding to the tilt of the imaging device 206, so the superimposed image can be superimposed on the landscape image regardless of the tilt of the surveillance image generating device 2.

In addition, the monitoring image generating device 2 outputs a warning signal when there is a risk that the body 211 of the vehicle 210 will come into contact with the ground, allowing the driver to drive the vehicle 210 in a way that reduces the risk of the body 211 coming into contact with the ground.

(Modification of the monitoring image generating device according to the second embodiment)
The surveillance image generating device 2 generates terrain images and superimposed images as a viewing cone, but the surveillance image generating device according to the embodiment may also generate terrain images and superimposed images by performing image processing such as affine transformation and projective transformation on the terrain.

Furthermore, while the surveillance image generation device 2 generates terrain image information using point cloud information, the surveillance image generation device according to the embodiment may also generate terrain image information using information acquired using a shape detection sensor such as a laser scanner.

Furthermore, while the surveillance image generating device 2 is a separate device from the operation device 203 to the imaging device 206, in the surveillance image generating device according to the embodiment, the operation device 203 to the imaging device 206 may be arranged as an integrated device with the surveillance image generating device.

Furthermore, while surveillance image generation devices 1 and 2 generate surveillance image information that shows surveillance images, the image generation devices according to the embodiments may also generate image information that shows images other than surveillance images. For example, the image generation devices according to the embodiments may generate image information that shows an image that includes pipes buried underground.

In addition, while surveillance image generation devices 1 and 2 acquire topographical information about the topography in the direction of travel of a moving object, image generation devices according to embodiments may also acquire topographical information indicating the topography in a desired first direction. Furthermore, image generation devices according to embodiments may be, for example, a mobile terminal held by a worker performing a specified task, or a fixed imaging device that is fixed to one point and can capture images in a desired direction.

When acquiring terrain information indicating the terrain in a first direction, the image generating device according to the embodiment generates image information indicating an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a landscape image captured from the single point.

1, 2 Monitoring image generating device 21, 41 Information acquisition unit 22, 42 Condition setting unit 23, 43 Image information generating unit 24, 44 Collision determination unit 25, 45 Output unit

Claims (7)

an information acquisition unit that acquires topographical information indicating the topography in the first direction;
an image information generating unit that generates image information showing an image obtained by superimposing a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, viewed from a single point on a scenery image captured from the single point;
an output unit that outputs the image information,
An image generating device characterized in that the superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane . the information acquisition unit further acquires current location information indicating a current location and first direction information indicating the first direction;
the first point is a point separated from the current location by a first distance in the first direction,
The image generating device according to claim 1 , wherein the second point is a point spaced apart from the first point in the first direction by a predetermined display length. determining whether or not there is a risk that the bottom of the body of the vehicle traveling in the first direction will come into contact with the surface of the terrain based on a vertical distance between the bottom of the body of the vehicle traveling in the first direction and the bottom of the wheels that movably support the body and a difference in elevation of the terrain;
outputting an alarm signal when it is determined that there is a risk that the bottom of the body will come into contact with the surface of the terrain;
The image generating device according to claim 2 , further comprising a collision determination unit. The image generating device according to claim 2 or 3, wherein the image information generating unit generates image information representing the image by superimposing the superimposed image on the landscape image captured by the imaging device located at the single point. The image generating device of claim 4, wherein the image information generating unit generates superimposed image information indicating the superimposed image so that it can be viewed at an elevation angle corresponding to the inclination of the imaging device. acquiring topographical information indicating the topography in a first direction;
generating image information showing an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a scenery image captured from the one point;
outputting the image information;
An image generation method, characterized in that the superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane . acquiring topographical information indicating the topography in a first direction;
generating image information showing an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a scenery image captured from the one point;
outputting the image information;
The superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane .