JP7677854B2 - Exposed image generating device, exposed image generating method, and program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. [Meeting name] Sewerage Exhibition '21 Osaka [Date] August 17-20, 2021

The present invention relates to an unfolded image generating device, an unfolded image generating method, and a program, and more particularly to image generation to assist in the inspection of tubular structures such as sewer pipes.

Inspections are being conducted on tubular structures such as water supply and sewerage pipes, cable pipes, and tunnels to check for damage and to check their condition. In these inspections, the inner wall surface of the pipe is photographed, and the image is then cut open in a flat plane so that the pipe runs longitudinally, creating an unfolded image using a computer. Workers then visually inspect the unfolded image to determine whether there is any damage and to prepare an inspection report.

Patent Document 1 describes an inner pipe wall image development system that creates an unfolded image based on all-sky image data of the inside of a tubular object captured while moving along the inside of the tubular object. The inner pipe wall image development system of Patent Document 1 holds cables, wires, etc. connected to an image capture means running inside the pipe so that they do not slacken, detects the position of the image capture means from the amount of cable, wire, etc. fed out to obtain position information Sp, and inserts the position information Sp at the time when image data Sv is input from the image capture means into the image data Sv as a digital signal to generate an unfolded image without distortion.

Publication No. 2008-090782

However, as mentioned above, to measure the position of the image capture means traveling inside the pipe, not only is hardware such as cables and wires required, but slack in the cables and wires is particularly likely to affect the image capture when it starts. In response to this, for example, there is a method of estimating the position information at the start of image capture by assuming that the image capture means is moving at a constant speed, but because there are obstacles such as dirt and puddles on the road surface inside the pipe, even if the image capture means is made to travel at a constant speed, it is not necessarily moving at a constant speed, which causes the problem of distorting the image.

The present invention has been made in consideration of the above problems, and aims to provide an unfolded image generating device etc. that can estimate the position information of the unfolded image generated during the investigation of a tubular structure without the need for hardware equipment such as cables or wires.

A first invention for solving the above-mentioned problems is an unfolded image generating device comprising: an image acquisition means for acquiring a direct-view image, which is an image captured by a wide-angle camera while moving inside a tubular structure in a pipeline direction; an unfolded image generating means for generating an unfolded image based on the direct-view image acquired by the image acquisition means; a position estimation means for detecting feature points included in the direct-view image, estimating the speed of the camera based on movement of the detected feature points, and estimating position information of the camera in the pipeline direction from the estimated speed; and a position information adding means for adding the position information estimated by the position estimation means to the unfolded image, further comprising an encoder for measuring a distance traveled by the camera; and a selection means for accepting a selection of whether the position information to be added to the unfolded image is position information based on the travel distance measured by the encoder, or position information estimated by the position estimation means, and the position information adding means is characterized in that the position information selected by the selection means is added to the unfolded image .

According to the unfolded image generating device of the first invention, since the position information of the camera in the pipe direction can be estimated based on the movement of the feature points included in the direct view image and added to the unfolded image, hardware equipment such as an encoder (cable, wire) is not required. Also, it can be used in combination with an encoder, making it possible to add more accurate position information.

It is also preferable to further include a setting means for displaying an unfolded circle object, the shape and center of which can be edited, overlaid on the displayed direct-view image, and setting pixels to be replaced based on an operation on the unfolded circle object. By providing such a user interface, it becomes possible to easily align the position of the inner wall surface of the pipe, which is to be replaced, while viewing the direct-view image on the display screen.

In addition, when the feature points appear periodically, it is desirable to further provide a correction means for correcting the position information estimated by the position estimation means based on a distance determined based on the periodicity. In this way, for feature points that appear periodically, such as pipe joints, more accurate position information can be added by correcting the position information based on pipe length information.

A second invention is a method for generating an unfolded image executed by a computer, comprising the steps of acquiring a direct-view image, which is a video image captured by a wide-angle camera while traveling inside a tubular structure in the pipeline direction, generating an unfolded image based on the acquired direct-view image, detecting feature points contained in the direct-view image, estimating the speed of the camera based on movement of the detected feature points, and estimating position information of the camera in the pipeline direction from the estimated speed, and adding the estimated position information to the unfolded image, and further comprising a step of accepting a selection of whether the position information to be added to the unfolded image is position information based on a distance traveled measured by an encoder that measures the distance traveled by the camera, or the estimated position information, and the adding step is characterized in that the position information selected in the step of accepting the selection is added to the unfolded image .

According to the second invention, the position information of the camera in the pipe direction can be estimated based on the movement of the feature points included in the direct view image and added to the unfolded image, so that hardware equipment such as an encoder (cable, wire) is not required. Also, it can be used in combination with an encoder, so that more accurate position information can be added.

A third invention is a program that functions as an image acquisition means for acquiring a direct-view image, which is an image captured by a wide-angle camera while moving inside a tubular structure in the pipeline direction; an unfolded image generation means for generating an unfolded image based on the direct-view image acquired by the image acquisition means; a position estimation means for detecting feature points included in the direct-view image, estimating the speed of the camera based on movement of the detected feature points, and estimating position information of the camera in the pipeline direction from the estimated speed; and a position information addition means for adding the position information estimated by the position estimation means to the unfolded image, and further functions as a selection means for accepting a selection of whether the position information to be added to the unfolded image is position information based on a distance traveled measured by an encoder that measures the distance traveled by the camera, or position information estimated by the position estimation means, and the position information addition means is characterized in that it adds the position information selected by the selection means to the unfolded image .

The third invention allows a computer to function as the unfolded image generating device in the first invention.

The present invention provides an unfolded image generating device that can estimate the position information of unfolded images generated during the investigation of tubular structures without the need for hardware equipment such as cables or wires.

FIG. 1 is a diagram showing an example of the overall configuration of an exfoliated image generating system 1. A block diagram showing the functional configuration of the photographing vehicle 2 and the PC 3 (developed image generating device) A flowchart showing the process executed by the application FIG. 6 is a diagram showing an example of a construction information input screen 61. FIG. 13 is a diagram showing an example of a development circle setting screen 62. FIG. 13 is a diagram showing a state in which a development circle is aligned with a joint on the development circle setting screen 62. FIG. 13 is a diagram showing an example of a development screen 63. Flowchart showing the flow of the developed image generation process A FIG. 13 is a diagram showing an example of a confirmation screen 64. Flowchart showing the flow of correction processing FIG. 1 is a diagram showing an example of the overall configuration of an exfoliated picture generating system 1A. A block diagram showing the functional configuration of the photographing vehicle 2 and the PC 3 (developed image generating device) A flowchart showing the process executed by the application FIG. 13 is a diagram showing an example of a development screen 63A. Flowchart showing the flow of the developed image generation process B

The following describes in detail a preferred embodiment of the present invention with reference to the drawings.

[First embodiment]
FIG. 1 is a diagram showing the overall configuration of a development image generating system 1 according to the present invention. In the following description, as an example, as shown in FIG. 1, a photographing vehicle 2 and a PC (Personal Computer) functioning as a development image generating device are used.
The following describes a developed image generating system 1 in which the PC 3 and the photographing vehicle 2 are connected via a wired or wireless communication interface. Note that the system configuration example in Fig. 1 is merely an example, and the present invention is not limited to this. For example, the PC 3 and the photographing vehicle 2 may be connected to a server (not shown) via a network, and photographed data may be shared via the server.

The photographing vehicle 2 photographs the inside of the tubular structure 10 (such as water supply and sewerage pipes, intake and exhaust pipes, cable pipes, tunnels, etc., hereafter referred to as pipes 10) to be inspected with a wide-angle camera 21 while traveling along the pipe. As shown in FIG. 2, the photographing vehicle 2 has a wide-angle camera 21 and a traveling vehicle drive unit 22.

The wide-angle camera 21 is equipped with a wide-angle lens and imaging elements such as CCD (Charge Coupled Devices) and CMOS (Complementary Metal Oxide Semiconductor), and is capable of photographing the inner wall surface around the wide-angle camera 21 by photographing the longitudinal direction of the tube 10 as the depth direction. If an ultra-wide-angle lens with a photographing angle of approximately right angles (diagonal 180°) or more to the optical axis of the lens is used, it is possible to photograph the inside of the tube 10 including the inner wall surface around the wide-angle camera 21. If optical fiber is used for the wide-angle camera 21, it is possible to photograph the inner wall surface of a tube 10 with a small diameter. It is preferable that the photographing vehicle 2 is provided with lighting (not shown) for illuminating the inside of the tube 10.

The vehicle drive unit 22 includes wheels, a motor, a power source, and the like for driving the imaging vehicle 2. It may also be equipped with a floating body to deal with puddles inside waterways and pipes 10.

PC3 is a computer terminal that acquires images (hereinafter, direct-view images) captured using the wide-angle camera 21 of the photographing vehicle 2 and generates an unfolded image based on the acquired direct-view images.

The unfolded image is an image of the inside of the pipe 10 that is cut open in the pipe line direction and unfolded into a flat surface. The method for generating the unfolded image will be described later.

As shown in FIG. 2, PC3 is a computer in which a control unit 31, a memory unit 32, a communication unit 33, an input unit 34, a display unit 35, and a peripheral device I/F (interface) unit 36 are connected via a bus. In addition to PC3, a tablet, a smartphone, etc. may be used as the unfolded image generating device, and the configuration of PC3 can be changed as appropriate. An unfolded image generating program is installed in PC3, and the control unit 31 executes processing in accordance with the unfolded image generating program to realize each function described below.

The control unit 31 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The CPU loads programs stored in the memory unit 32, ROM, etc. into a work memory area on the RAM and executes them, and drives and controls each unit connected via the bus (memory unit 32, communication unit 33, input unit 34, display unit 35, peripheral device I/F unit 36). The ROM permanently stores programs such as the boot program and BIOS, data, etc. The RAM temporarily stores loaded programs and data, and also provides a work area used by the control unit 31 to perform various processes.

The storage unit 32 is a storage device such as a flash memory or a hard disk, and stores the acquired image data, the input damage information, the determined pipe structure information, etc. The storage unit 32 also stores processing programs related to each function described below (collectively referred to as "unfolded image generation programs" or "apps").

The communication unit 33 has a WiFi antenna, a wireless communication unit such as Bluetooth, or a communication port for a wired communication unit such as a LAN, and a communication control device, and is an interface that mediates communication with external devices.

The input unit 34 includes, for example, a touch panel, a keyboard, a pointing device such as a mouse, an input device such as a numeric keypad, and the like, and inputs the input data to the control unit 31.

The display unit 35 is composed of a display such as a liquid crystal panel and a logic circuit (such as a video adapter) for executing display processing in cooperation with the display, and displays the display data input under the control of the control unit 31. The display unit 35 may be a touch panel display in which an input device (input unit 34) such as a touch panel is integrally provided on the display screen.

The peripheral device I/F unit 36 is a port for connecting peripheral devices, and the control unit 31 transmits and receives data to and from the peripheral devices via the peripheral device I/F unit 36. The peripheral device I/F unit 36 is configured using a USB (Universal Serial Bus) or the like. The connection with the peripheral devices may be wired or wireless.

Next, the functional configuration of the PC 3 (exfoliated picture generating device) will be described with reference to FIG.
The PC 3 has, as its functional units, an image acquisition unit 311, an unfolded image generation unit 312, a position estimation unit 313, and a position information addition unit 314. These functional units are realized by the CPU of the control unit 31 reading a processing program (unfolded image generation program) stored in the storage unit 32, calling it up into a work memory area on the RAM, and executing it.

The image acquisition unit 311 acquires the direct-view image captured by the wide-angle camera 21 of the photographing vehicle 2 via the communication unit 33 or the peripheral device I/F unit 36. Alternatively, the image acquisition unit 311 may acquire the direct-view image by reading it from a storage medium, or the communication unit 33 may acquire the direct-view image stored in another computer or the direct-view image stored on a storage server (not shown) via a network such as a LAN or the Internet, and import it into the control unit 31.

The unfolded image generating unit 312 generates an unfolded image based on the direct-view image acquired by the image acquiring unit 311. The specific procedure for generating the unfolded image will be described later.

The position estimation unit 313 detects feature points included in the direct view image, estimates the speed of the wide-angle camera 21 (photographing vehicle 2) based on the movement of the detected feature points, and estimates the position of the wide-angle camera 21 (photographing vehicle 2) in the pipe direction from the estimated speed. The technology for tracking the movement of corresponding points between image frames is known as optical flow. Optical flow is a vector (movement vector) that indicates which point in one image frame at a certain coordinate (x, y) has moved to in the other image, and the X-direction component and the Y-direction component are calculated for one image coordinate (x, y). A method for estimating the speed of a moving vehicle from a monocular camera image is described in, for example, a literature (Morimoto Masakazu, Kaneko Hiroki, Miyamoto Naoki, "Estimating the Speed of a Moving Vehicle from Monocular Vehicle-Mounted Camera Images," Journal of the Institute of Visual Media Studies, Vol. 68, No. 1, pp. j47j54 (2014)).

The position information adding unit 314 adds the position information estimated by the position estimation unit 313 to the unfolded image generated by the unfolded image generating unit 312.

Next, a method for generating an unfolded image using the unfolded image generating system 1 will be described. The processing of each step in the flowchart of FIG. 3 is executed by the control unit 31 of the PC 3.

The worker transports the camera vehicle 2 and PC 3 to the investigation site and prepares to take a video of the inside of the pipe 10. The worker starts the application related to the generation of the unfolded image on the PC 3 and starts the process. In this embodiment, as an example, the unfolded image is generated in real time while the direct-view image is being taken.

When the app is launched (step S101), the control unit 31 displays a construction information input screen 61 as shown in FIG. 4 on the display unit 35 and accepts the input of construction information (step S102). The construction information input screen 61 includes an input field 61a for inputting information such as the construction name, route, upstream manhole, downstream manhole, pipe type, and pipe diameter, and a "Go to expansion circle setting" button 61b that is operated to move to the next screen. When information is input into the input field 61a via the input unit 34 and the "Go to expansion circle setting" button 61b is operated, the control unit 31 displays the expansion circle setting screen 62 shown in FIG. 5 and accepts the setting of the expansion circle and the camera from which the image is to be input (step S103).

The unfolding circle setting screen 62 is provided with a camera image display area 62a, a camera information input field 62b, and a "Go to unfold" button 62c for moving to the next step. The camera image display area 62a is an area for displaying images input from the camera. The camera information input field 62b is an input field for specifying which camera to capture images from. The camera information input field 62b displays a list of cameras that are connected for communication via the communication unit 33 or the peripheral device I/F unit 36 of the PC 3 and are in a communicable state, and can be selected. When the camera from which images are to be captured is input, the PC 3 captures the image of the specified camera and displays it in the camera image display area 62a. In the camera image display area 62a, an unfolding circle object 4 for aligning the camera image (direct view image) is displayed superimposed on the camera image (direct view image).

The developed circle object 4 is for specifying the position of the inner wall surface of the pipe 10 on the direct view image, and is an elliptical or circular object equipped with four pointers 41a, 41b, 41c, and 41d for specifying four points of the pipe 10, i.e., the pipe top, the pipe bottom, the left end, and the right end, and a center mark 43 for specifying the center position. Each pointer 41 and center mark 43 can be moved by operating a mouse or the like. The control unit 31 draws an ellipse or circle (developed circle) whose vertical diameter is the length between the pointers 41a to 41b and whose horizontal diameter is the length between the pointers 41c and 41d. In other words, the shape of the developed circle object 4 can be changed by moving each of the pointers 41a to 41d. When the center mark 43 of the developed circle object 4 is moved by operating a mouse or the like, the position of the developed circle object 4 is moved. In the development process in the later step, the control unit 31 replaces the pixels of the direct view image that overlap with the developed circle object 4 as pixels of the inner wall surface on the developed image plane.

The worker aligns the unfolded circle object 4 while viewing the image (direct view image) displayed in the camera image display area 62a. In aligning, for example, as shown in FIG. 6, the positions of the pointers 41a, 41b, 41c, 41d and the center mark 43 are adjusted so that the joint part of the pipe 10 shown in the direct view image displayed in the camera image display area 62a overlaps with the unfolded circle object 4.

Then, when the "To Expansion" button 62c is operated, the PC 3 displays the expansion screen 63 shown in FIG. 7 and starts the expansion image generation process (step S104). The expansion screen 63 is provided with a camera image display area 63a, an expansion image display area 63b, a status display area 63c, a "Start/Stop Expansion" button 63d, an "End" button 63e, a confirmation button 63f, and the like. The camera image display area 63a displays the image (direct view image) input from the wide-angle camera 21. The expansion image display area 63b displays the generated expansion image. The "Start/Stop Expansion" button 63d displays "Start Expansion" in the initial state, and when the button is operated in this state, the expansion process starts. During the expansion process, the display of the "Start/Stop Expansion" button 63d displays "Stop Expansion", and when the button is operated in this state, the expansion process is paused. The status display area 63c displays the status of the camera image, such as "Playing", "Paused", and the like. The "End" button 63e is a button that is operated when ending the expansion process. The confirmation button 63f is the button that is operated to move to the next step, the confirmation screen 64.

The unfolded image generation process in step S104 will be described with reference to FIG. 8. FIG. 8 shows a flowchart of unfolded image generation process A, which is the processing flow in the first embodiment.

When the "Start/Stop Deployment" button 63d is operated on the deployment screen 63 and a shooting start operation is performed on the shooting vehicle 2 (step S201; Yes), the control unit 31 acquires a direct-view image, which is an image captured by the wide-angle camera 21 (step S202). The control unit 31 extracts feature points from the acquired direct-view image (step S203). The feature point is, for example, a point (e.g., the upper left point) in an area extracted from an edge of an image (such as a contour line or a brightness boundary), such as dust, a pattern, or a joint. When the control unit 31 extracts the feature point, it tracks the movement of the feature point and estimates the moving speed of the wide-angle camera 21 (shooting vehicle 2) (step S204). The process of step S204 uses a known method such as the above-mentioned method of estimating the speed of a traveling vehicle using optical flow. Then, the position of the wide-angle camera 21 in the pipe direction is calculated from the speed of the shooting vehicle 2 (step S205).

The control unit 31 replaces the pixel values of the pixels overlapping the unfolded circle object 4 in the direct view image frame _fn at time t= _tn on the unfolded image plane (step S206). Specifically, the pixels overlapping the circumference of the unfolded circle object 4 are rearranged in a row and inserted into the nth column of the unfolded image plane. The control unit 31 adds the information of the pipeline direction position (position information) at time t= _tn calculated in step S205 as additional information of the direct view image frame _fn , and also adds it as position information of the nth column of the unfolded image (step S207). The control unit 31 adds the generated image in the nth column to the already generated unfolded image and updates the display (step S208). Similarly, for the direct view image frame fn ₊₁ at the next time t= _tn+1 , the pixels on the circumference of the unfolded circle object 4 are rearranged in a row and inserted into the n+1th column of the unfolded image plane. Moreover, the information on the pipeline direction position (position information) at time t=tn ₊₁ calculated in step S205 is added as additional information of the direct-view image frame fn ₊₁ , and is added as position information of the n+1th column of the exfoliated image, thereby updating the display of the exfoliated image. As time passes, newly generated exfoliated images are sequentially added to the right end of the exfoliated image display area 63b in Fig. 7, and the exfoliated images in the leftmost column are sequentially replaced with images in the adjacent column to the right.

The control unit 31 repeats the processes of steps S202 to S208. When the "Start/Stop Deployment" button 63d or the "End" button 63e is operated (step S209; Yes), the control unit 31 stops or ends the process. The series of images input from the wide-angle camera 21 are temporarily stored as direct-view images in the RAM or memory unit 32 of the PC 3, and the generated deployment images are also temporarily stored in the RAM or memory unit 32 of the PC 3.

When the unfolded image generation process A is completed, the control unit 31 proceeds to step S105 in the flowchart of FIG. 3. When the confirmation button 63f on the unfolded screen 63 is operated (step S105; Yes), the control unit 31 displays the confirmation screen 64 shown in FIG. 9.

On the confirmation screen 64, the control unit 31 displays the direct view image, which is an image captured by the wide-angle camera 21, in the display area 64a, and displays the generated unfolded image (part or all) in the display area 64b (step S106). The unfolded image is displayed so that it can be scrolled left and right (in the pipe direction), and the display range can be changed by operating the scroll bar 64d with a mouse or the like. In addition, the direct view image at a position corresponding to the unfolded image displayed in the display area 64b is displayed in the display area 64a, and the two images can be compared and confirmed. In addition, when the play/stop button 64c below the display area 64a is operated, the control unit 31 plays the direct view image as a video, and displays the unfolded image in the display area 64b so that it includes the unfolded image at a position corresponding to the displayed direct view image.

When the data save button 64e is operated (step S107; Yes), the direct-view image and the unfolded image are stored in the storage unit 32 of the PC 3 (step S108). If the unfolded image generation is to be redone (step S107; No), the process returns to step S103.

By the above processing, position information estimated based on the movement of the feature points is added to the direct view image and the unfolded image. This makes it possible to estimate and add the pipe direction position of the wide-angle camera 21 without using hardware equipment such as an encoder, improving the work efficiency of the inspection of sewer pipes, etc.

In addition to the above-mentioned position estimation, it is also possible to add more accurate position information by correcting the estimated position based on periodically appearing features such as the joint position of the pipe 10. FIG. 10 is a flowchart explaining the correction process. Note that the correction process in FIG. 10 may be performed at any time after the development image is generated.

The control unit 31 of the PC 3 acquires the unfolded image data generated in the unfolded image generation process of FIG. 8 (step S301), and displays the unfolded image on the display unit 35 (step S302). The control unit 31 accepts an input operation of the joint position of the pipe 10 on the unfolded image (step S303).

When a ready-made or standardized product is used, the pipe 10 is manufactured to a predetermined length. The control unit 31 obtains information on the pipe 10 standard or the length of the pipe 10 from the memory unit 32, accepts input if not entered, and corrects the position information added in the unfolded image generation process A based on the information on the length of the pipe 10 (step S304). For example, when the pipe type is PVC pipe and the pipe diameter is 200 [mm], the joints are spaced 1.5 [m] apart. The control unit 31 corrects the joint positions entered in the unfolded image based on the information on the length of the pipe 10, or inserts the joint positions into the unfolded image. It also corrects the position information added in the unfolded image generation process based on the joint positions.

Thereafter, the control unit 31 accepts a selection as to whether or not to reflect the correction made in step S304. If the correction is to be reflected (step S305; Yes), the position information is updated and stored. If the correction is not to be reflected (step S305; No), the original position information is left unchanged.
In this way, in the case of a periodic feature point (such as a joint), it is possible to more accurately add the position information to the unfolded image, etc. by correcting the position information estimated from the movement of the feature point based on information based on the periodicity, for example, known information such as the length of the tube 10. Periodic features are not limited to joints, and include, for example, patterns, etc.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, position information of the unfolded image is added only by position estimation based on the movement of feature points without using the encoder 23 (cable, wire) at all, but the encoder 23 may be used in combination. For example, the encoder 23 is used within a range where it can be used accurately, and in a place where accurate distance measurement is not possible due to slackness in the cable, the first embodiment (position estimation based on the movement of feature points) is used as an auxiliary.

Fig. 11 is a diagram showing the overall configuration of the unfolded image generating system 1A of the second embodiment, and Fig. 12 is a diagram showing the functional configuration of the unfolded image generating system 1A. As shown in the figure, the unfolded image generating system 1A includes an encoder 23 in addition to the photography vehicle 2 and PC 3 of the unfolded image generating system 1 of the first embodiment. The encoder 23 has a cable 24 for measuring the distance from the entrance of the pipe 10 to the wide-angle camera 21. The cable 24 is connected to the photography vehicle 2, and the distance information measured by the encoder 23 (the extension length of the cable 24) is output to the PC 3 via the communication interface of the encoder 23.

In addition to the same configuration as in the first embodiment, the PC 3 has an interface for acquiring the distance measured by the encoder 23. Similarly to the first embodiment, the control unit 31 of the PC 3 has functional units such as an image acquisition unit 311, an unfolded image generation unit 312, a position estimation unit 313, and a position information addition unit 314, but in the second embodiment, the unfolded image generation unit 312 allows the selection of an estimation mode or a cable mode.

In the estimation mode, as in the first embodiment, the pipeline direction position is calculated based on the movement of the feature points in the unfolded image generation process, and is added to the unfolded image and the direct-view image. In the cable mode, the distance information measured by the encoder 23 is added to the unfolded image and the direct-view image as pipeline direction position information.

The unfolded image generating method in the second embodiment will be described below. Note that the system configuration examples in Figs. 11 and 12 are merely examples, and the present invention is not limited thereto. In the following description, the same reference numerals are used for the same parts as in the first embodiment, and duplicated descriptions will be omitted.

Next, a method for generating an unfolded image using the unfolded image generating system 1A will be described. The processing of each step in the flowchart of FIG. 13 is executed by the control unit 31 of the PC 3.

Steps S401 to S403 are the same as steps S101 to S103 of the unfolded image generating method of the first embodiment. That is, when an application related to unfolded image generation is launched on the PC 3 and processing is started (step S401), the control unit 31 displays a construction information input screen 61 as shown in FIG. 4 on the display unit 35 and accepts input of construction information (step S402). When each piece of information is entered in the input field 61a and the "Go to unfolded circle setting" button 61b is operated, the control unit 31 displays the unfolded circle setting screen 62 shown in FIG. 5 and accepts camera settings and unfolded circle settings (step S403).

When the camera settings and the deployment circle settings are completed and the "Go to deployment" button 62c is operated, the control unit 31 of the PC 3 displays the deployment screen 63A shown in FIG. 14. The deployment screen 63A in FIG. 14 includes a camera video display area 63a, a deployment image display area 63b, a status display area 63c, a "Start/Stop deployment" button 63d, an "End" button 63e, a confirmation button 63f, and a mode selection section 63g that is operated to switch between estimation mode and cable mode.

When the user operates the mode selection unit 63g and selects the estimation mode (step S404; estimation mode), the control unit 31 executes the unfolded image generation process A (FIG. 8) described in the first embodiment (step S405). When the cable mode is selected (step S404; cable mode), the control unit 31 executes the unfolded image generation process B shown in the flowchart of FIG. 15 (step S406). As described in the first embodiment (FIG. 8), the unfolded image generation process A is a process that estimates the speed of the photographing vehicle 2 based on the movement of feature points between image frames of the direct-view image, calculates the position of the photographing vehicle 2 from the estimated speed, and adds it to the unfolded image and the direct-view image. On the other hand, the unfolded image generation process B is a process that adds the distance information obtained from the encoder 23 as position information to the unfolded image and the direct-view image.

When using the encoder 23, if the cable 24 of the encoder 23 is loose, an accurate distance cannot be obtained. For this reason, the estimation mode is selected for a few meters from the entrance of the pipe 10, where the cable 24 is likely to become loose (step S404; estimation mode), and an unfolded image is generated (unfolded image generation process A; step S405). Then, after the cable 24 is taut and not loose, the mode is switched to the cable mode (step S407; Yes), and an unfolded image is generated using the distance information from the encoder 23 (unfolded image generation process B; step S406). This is the usage mode that can be adopted.

The unfolded image generation process B in step S406 will be described with reference to FIG. 15. When the "unfold start/stop" button 63d is operated on the unfolded screen 63A and a shooting start operation is performed on the shooting vehicle 2 (step S501; Yes), the control unit 31 acquires a direct-view image, which is an image captured by the wide-angle camera 21 (step S502). At the same time, the control unit 31 acquires distance information from the encoder 23 (step S503).

The control unit 31 replaces the pixel values of the pixels overlapping the developed circle object 4 in the direct-view image frame _fn at time t= _tn with the developed image (step S504). The control unit 31 adds the distance information at time t= _tn acquired in step S503 as position information as additional information of the direct-view image frame _fn , and also adds it as position information of the nth column of the developed image (step S505). The control unit 31 adds the generated image in the nth column to the displayed developed image and updates the display (step S506). Similarly, for the direct-view image frame fn ₊₁ at the next time t=tn ₊₁ , the control unit 31 rearranges the pixels on the circumference of the developed circle object 4 in a row and inserts it into the n+1th column of the developed image plane. Also, the control unit 31 adds the distance information at time t=tn ₊₁ acquired in step S503 as additional information of the direct-view image frame fn ₊₁ , and also adds it as position information of the n+1th column of the developed image, and updates the display of the developed image. As time passes, newly generated exfoliated images are added in sequence to the right end of the exfoliated image display area 63b in FIG. 14, and at the same time, the exfoliated images in the leftmost column are successively replaced with images in the adjacent column to the right.

In this manner, the processes of steps S502 to S506 are repeated. When the "Start/Stop Expansion" button 63d or the "End" button 63e is operated (step S507; Yes), the control unit 31 stops or ends the process. The series of images input from the wide-angle camera 21 are temporarily stored as direct-view images in the RAM or memory unit 32 of the PC 3, and the generated expanded images are also temporarily stored in the RAM or memory unit 32 of the PC 3.

When the unfolded image generation process B is completed, the control unit 31 proceeds to step S408 in the flowchart of FIG. 13. When the confirmation button 63f on the unfolded image screen 63A is operated (step S408; Yes), the control unit 31 displays the confirmation screen 64 shown in FIG. 9. The operation and display example on the confirmation screen 64 are the same as those in the first embodiment (steps S105 to S108 in FIG. 3). That is, on the confirmation screen 64, the control unit 31 displays the direct-view image, which is the image captured by the wide-angle camera 21, in the display area 64a, and displays the generated unfolded image (part or all) in the display area 64b (step S409). When the data save button 64e is operated (step S410; Yes), the direct-view image and the unfolded image are stored in the storage unit 32 of the PC 3 (step S411). Note that if the unfolded image generation is to be redone (step S410; No), the process returns to step S403.

As described above, the unfolded image generating system 1A equipped with the encoder 23 allows a selection between the estimation mode and the cable mode. When the cable mode is selected, position information based on the travel distance measured by the encoder 23 is added to the unfolded image. When the estimation mode is selected, position information estimated by the position estimation unit 313 is added to the unfolded image. As a result, near the entrance of the pipe 10 where the cable 24 is likely to slacken, the estimation mode is used to add position information estimated from the movement of feature points in the direct view image to the unfolded image. After the cable 24 is taut without slack, the mode is switched to the cable mode, and the distance information obtained from the encoder 23 is added to the unfolded image as position information.

After the processing shown in the second embodiment, the correction processing shown in FIG. 10 may be performed.

Although the preferred embodiment of the present invention has been described above with reference to the attached drawings, the present invention is not limited to the examples. For example, the layout of each screen, the display size of each image, the arrangement of various buttons, etc. are merely examples, and other layouts, sizes, and arrangements may be adopted. In addition, it is clear that a person skilled in the art could come up with various modified or revised examples within the scope of the technical ideas disclosed in this application, and it is understood that these also naturally fall within the technical scope of the present invention.

1 .... Development image generating system 2 .... Photographing vehicle 21 .... Wide-angle camera 22 .... Traveling vehicle driving unit 3 .... PC (development image generating device)
31: Control unit 311: Image acquisition unit 312: Developed image generation unit 313: Position estimation unit 314: Position information addition unit 4: Developed circle object 10: Tubular structure (tube)

Claims (5)

an image acquisition means for acquiring a direct-view image, which is a moving image captured by a wide-angle camera while traveling inside the tubular structure in a pipe line direction;
an unfolded image generating means for generating an unfolded image based on the direct-view image acquired by the image acquiring means;
a position estimation means for detecting a feature point included in the direct-view image, estimating a speed of the camera based on a movement of the detected feature point, and estimating position information of the camera in a pipeline direction from the estimated speed;
a position information adding means for adding the position information estimated by the position estimating means to the unfolded image;
Equipped with
An encoder for measuring a travel distance of the camera;
a selection unit that accepts a selection of whether the position information to be added to the unfolded image is to be position information based on a travel distance measured by the encoder or position information estimated by the position estimation unit,
The position information adding means adds the position information selected by the selecting means to the expanded image.
13. An exfoliated image generating device comprising: The exfoliated image generating device according to claim 1, further comprising a setting means for displaying an exfoliated circular object, the shape and center point of which can be edited, overlaid on the displayed direct-view image, and setting pixels to be replaced based on an operation on the exfoliated circular object. 3. The unfolded picture generating device according to claim 1, further comprising a correction means for correcting position information estimated by the position estimation means on the basis of a distance determined on the basis of the periodicity when the feature points appear periodically. 1. A computer -implemented method for generating an unfolded image, comprising the steps of :
acquiring a direct-view image, which is a moving image captured by a wide-angle camera while moving inside the tubular structure in a pipe line direction;
generating an unfolded image based on the acquired direct-view image;
detecting feature points included in the direct-view image, estimating a speed of the camera based on a movement of the detected feature points, and estimating position information of the camera in a pipeline direction from the estimated speed;
adding the estimated position information to the unfolded image;
Including,
receiving a selection of whether the position information to be added to the deployed image is to be position information based on a travel distance measured by an encoder that measures a travel distance of the camera, or the estimated position information,
The adding step adds the position information selected in the selection receiving step to the expanded image.
A method for generating an expanded image comprising the steps of: Computer,
an image acquisition means for acquiring a direct-view image, which is a moving image captured by a wide-angle camera while moving inside the tubular structure in the pipe line direction;
an unfolded image generating means for generating an unfolded image based on the direct-view image acquired by the image acquiring means;
a position estimation means for detecting a feature point included in the direct-view image, estimating a speed of the camera based on a movement of the detected feature point, and estimating position information of the camera in a pipeline direction from the estimated speed;
a position information adding means for adding the position information estimated by the position estimating means to the unfolded image;
A program that functions as
a selection unit that accepts a selection of whether the position information to be added to the deployed image is to be position information based on a travel distance measured by an encoder that measures a travel distance of the camera, or position information estimated by the position estimation unit,
The position information adding means adds the position information selected by the selecting means to the expanded image.
A program characterized by:

Images