JP7778351B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program for processing image data.

Technology for displaying an augmented reality image corresponding to a specified position based on image data captured by a camera installed in electronic goggles on the display of the electronic goggles is known (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2019-166405

In an image processing device such as the electronic goggles described in Patent Document 1, the position of the image processing device can be identified by comparing the pattern of feature points of a subject in captured image data generated in advance by a camera installed in the image processing device with a pattern of feature points previously associated with a position in space. As one example, the image processing device can display an augmented reality image corresponding to the identified position.

However, when a person inside a moving vehicle uses such an image processing device to simultaneously view the internal and external spaces of the moving vehicle, the pattern of feature points of the subject in the captured image data is compared with the pattern of feature points previously associated with positions in the external space, but these patterns do not match, and the position of the image processing device cannot be identified.

The present invention was made in consideration of these points, and aims to enable a person inside a moving vehicle to identify the position of an image processing device even when the person inside the moving vehicle is looking at both the internal and external spaces of the moving vehicle at the same time.

The image processing device of a first aspect of the present invention is a wearable image processing device that includes an image data acquisition unit that acquires captured image data generated by the image processing device worn by a person riding on a moving body by capturing images of the internal space of the moving body and the external space of the moving body; a boundary identification unit that identifies the boundary between internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; and a position identification unit that identifies the position of the image processing device in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and identifies the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data.

The position identification unit may identify the position of the image processing device in the internal space based on the relationship between the positions of the multiple feature points included in the latest internal space image data and the positions of the multiple feature points included in the immediately preceding internal space image data, and may identify the position of the image processing device in the external space based on the relationship between the positions of the multiple feature points included in the latest external space image data and the positions of the multiple feature points included in the immediately preceding external space image data.

The position identification unit may identify the position of the image processing device in the internal space at a slower cycle than the cycle at which the process for identifying the position of the image processing device in the external space is executed.

The image processing device may further include an internal image display processing unit that causes a display unit of the image processing device to display an internal augmented reality image corresponding to the position of the image processing device in the internal space identified by the position identification unit, and an external image display processing unit that causes the display unit to display an external augmented reality image corresponding to the position of the image processing device in the external space identified by the position identification unit.

The internal image display processing unit may display the internal augmented reality image on the display unit at a position relative to a specified subject in the internal space in the captured image data.

The internal image display processing unit may cause the display unit to display the internal augmented reality image of a size determined based on a pattern of multiple feature points that indicate the characteristics of one or more subjects in the captured image data.

The external image display processing unit may display the external augmented reality image on the display unit at a position that is pre-stored in a storage unit in association with the position of the image processing device in the external space.

The image data acquisition unit may acquire the captured image data at predetermined time intervals, and the boundary identification unit may identify the boundary based on the rate of change in the position of each of multiple subjects included in the captured image data acquired at different times.

An image processing method according to a second aspect of the present invention includes the steps of: acquiring captured image data generated by an image processing device worn by a person riding on a moving body by capturing images of the interior space of the moving body and the exterior space of the moving body, executed by a processor; identifying internal space image data corresponding to the interior space in the captured image data and external space image data corresponding to the exterior space in the captured image data; identifying the position of the image processing device in the interior space based on the positions of multiple feature points of the subject included in the internal space image data; and identifying the position of the image processing device in the exterior space based on the positions of multiple feature points of the subject included in the external space image data.

A third aspect of the present invention provides a program that causes a processor to execute the following steps: acquiring captured image data generated by an image processing device worn by a person riding on a moving body capturing images of the interior space of the moving body and the exterior space of the moving body; identifying internal space image data corresponding to the interior space in the captured image data and external space image data corresponding to the exterior space in the captured image data; identifying the position of the image processing device in the interior space based on the positions of multiple feature points of the subject included in the internal space image data; and identifying the position of the image processing device in the exterior space based on the positions of multiple feature points of the subject included in the external space image data.

The present invention has the advantage of being able to identify the position of an image processing device even when a person inside a moving object is simultaneously viewing the space inside and outside the moving object.

FIG. 1 is a diagram illustrating an overview of an image processing device. FIG. 10 is a diagram schematically illustrating an example of an image viewed by a user wearing the image processing device. FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus. 10 is an example of point cloud data created by a boundary specifying unit. 10 is a flowchart showing a processing flow in the image processing device.

[Overview of image processing device 1]
Fig. 1 is a diagram for explaining an overview of an image processing device 1. The image processing device 1 is a terminal that can be worn by a user U (e.g., a pilot) riding on a moving object P, and has the shape of, for example, glasses or goggles. The moving object P is any object that can move with a person on board, such as an airplane, ship, train, or vehicle. Fig. 1 schematically shows a user U in the cockpit of an airplane operating a control stick H.

Through the lens of the image processing device 1, the user U can see the space inside the cockpit (hereinafter referred to as the "internal space") and the space outside the window W (hereinafter referred to as the "external space"). The user U can see objects such as the control stick H, meters M1, M2, etc. in the internal space, as well as the runway in the external space. Furthermore, while viewing the internal and external spaces, the user U can also see augmented reality images (hereinafter referred to as "AR images") displayed on the lens by the image processing device 1.

Figure 2 is a diagram showing a schematic example of an image viewed by user U while wearing image processing device 1. Region R1 in Figure 2 shows a runway in the external space, and region R2 shows the cockpit in the internal space. Figure 2(a) shows an AR image displayed superimposed on the internal and external spaces viewed by user U while sitting upright, and Figure 2(b) shows an AR image displayed superimposed on the internal and external spaces viewed by user U while tilted relative to the vertical direction.

The image shown in Figure 2(a) displays an internal AR image G1 and an external AR image G2. The internal AR image G1 shows the speed of the airplane and is displayed above the control stick H. The external AR image G2 shows the direction the airplane should head and is displayed at a predetermined position on the runway.

Similarly, the image shown in Figure 2(b) also displays an internal AR image G1' and an external AR image G2'. The internal AR image G1 and the internal AR image G1' are displayed in the same orientation above the control stick H. In contrast, the external AR image G2 and the external AR image G2' are displayed in different orientations. Specifically, in Figure 2(a), the pilot is not leaning, so the runway appears vertical, and the external AR image G2 is made up of straight lines in the vertical and horizontal directions. In contrast, in Figure 2(b), the pilot is leaning, so the runway appears diagonal, and the external AR image G2' is made up of multiple straight lines in the diagonal direction.

The image processing device 1 displays an AR image containing information related to the internal space at a position determined based on the position of the image processing device 1 in the internal space, and displays an AR image containing information related to the external space at a position determined based on the position of the image processing device 1 in the external space. In order to display the AR image corresponding to the internal space and the AR image corresponding to the external space at appropriate positions, the image processing device 1 identifies the position of the image processing device 1 in the internal space and the position of the image processing device 1 in the external space based on the captured image data captured and generated by the image processing device 1.

Specifically, the image processing device 1 identifies the position of the image processing device 1 in the internal space based on internal space image data corresponding to the internal space in the captured image data, and identifies the position of the image processing device 1 in the external space based on external space image data corresponding to the external space in the captured image data. In this way, the image processing device 1 can identify the position of the image processing device 1 even when a person inside the moving body P simultaneously views the internal and external spaces of the moving body P, and can display an AR image appropriate for the state of the internal space and an AR image appropriate for the state of the external space based on the identified position.

[Configuration of image processing device 1]
3 is a diagram showing the hardware configuration of the image processing device 1. FIG. 4 is a diagram showing the functional configuration of the image processing device 1.

As shown in FIG. 3, the image processing device 1 has an imaging unit 11, a right lens 121, a left lens 122, a right projection unit 123, and a left projection unit 124. The right lens 121, the left lens 122, the right projection unit 123, and the left projection unit 124 constitute the display unit 12 shown in FIG. 4. The display unit 12 displays an AR image so that the user U can see it while viewing subjects in the internal space and the external space. In the display unit 12, the right projection unit 123 projects a projection image L1 toward the right lens 121, and the left projection unit 124 projects a projection image L2 toward the left lens 122, thereby displaying the AR image on the right lens 121 and the left lens 122.

The imaging unit 11 has an imaging element that creates captured image data by capturing an image of a subject. In the example shown in FIG. 3, the imaging unit 11 is located near the center of the image processing device 1, but the imaging unit 11 may be located in any position as long as it can create captured image data corresponding to the area viewed by the user U. The imaging unit 11 may have multiple imaging elements located at different left and right positions, and may function as a stereo camera that can generate data in the depth direction.

The right lens 121 corresponds to the right eye of the user U and is located in front of the user U's right eye when the user U is wearing the image processing device 1. The right lens 121 is optically transparent, allowing the user U to see surrounding subjects through the right lens 121. The right lens 121 also displays the AR image projected by the right projection unit 123 in a manner that is visible to the user U. The right lens 121 has, for example, a light guide plate that propagates light projected by the right projection unit 123 onto a portion of the right lens 121 and then refracts the light toward the user U's right eye.

The left lens 122 has a configuration similar to that of the right lens 121. The left lens 122 corresponds to the left eye of the user U and is positioned in front of the user U's left eye when the user U is wearing the image processing device 1. The left lens 122 is optically transparent, allowing the user U to see surrounding subjects through the left lens 122. The left lens 122 also displays the AR image projected by the left projection unit 124 in a manner that can be seen by the user U. The left lens 122 has, for example, a light guide plate that propagates light projected by the left projection unit 124 onto a portion of the left lens 122 and then refracts the light toward the user U's right eye.

The right projection unit 123 projects an AR image onto the right lens 121. The right projection unit 123 projects an AR image for the right eye input from the display processing unit 166 onto the right lens 121. The right projection unit 123 and the left projection unit 124 project the AR image onto the right lens 121 and the left lens 122 while the user is viewing the subject through the right lens 121 and the left lens 122.

The left projection unit 124 projects an AR image onto the left lens 122. The left projection unit 124 projects an AR image for the left eye input from the display processing unit 166 onto the left lens 122.

Next, the internal configuration and functions of the image processing device 1 will be described with reference to Figure 4. As shown in Figure 4, the image processing device 1 has an image capturing unit 11 and a display unit 12, as well as an operation unit 13, a communication unit 14, a memory unit 15, and a control unit 16. The control unit 16 has an image data acquisition unit 161, a boundary identification unit 162, a position identification unit 163, an internal image display processing unit 164, an external image display processing unit 165, and a display processing unit 166.

The operation unit 13 is a device for receiving operations of the image processing device 1 by the user U. The operation unit 13 may be, for example, a touch switch or push switch, or a short-range wireless communication device that receives operation data from an external terminal such as a smartphone or tablet. The operation data is, for example, data indicating the operation content to start or stop the display of the AR image, data indicating the model name of the mobile object P, or data indicating the location to which the mobile object P is moving (data indicating the name of an airport, port, railway line, or road). The operation unit 13 notifies the control unit 16 of the operation data indicating the operation content of the user U.

The communication unit 14 has a communication interface for sending and receiving data to and from an external server that provides data used in processing by the control unit 16. The communication unit 14 has, for example, a wireless communication controller for communicating using a wireless communication line. The communication unit 14, for example, acquires, from the external server, reference point cloud data indicating the feature points of various objects in the space near where the mobile body P on which the user U is riding is moving, and AR image data to be displayed on the display unit 12, and stores the acquired reference point cloud data and AR image data in the memory unit 15.

The communication unit 14 acquires multiple internal reference point cloud data consisting of feature points of the subject included in images captured at multiple positions in the internal space. The communication unit 14 also acquires multiple external reference point cloud data consisting of feature points of the subject included in images captured at multiple positions in the external space.

The AR image data is associated with one or more internal reference point cloud data or external reference point cloud data. When point cloud data based on feature points of a subject included in an area corresponding to the internal space in the captured image data matches any of the internal reference point cloud data and the AR image data is associated with the internal reference point cloud data, the image processing device 1 displays an AR image based on the AR image data. Similarly, when point cloud data based on feature points of a subject included in an area corresponding to the external space in the captured image data matches any of the external reference point cloud data and the AR image data is associated with the external reference point cloud data, the image processing device 1 displays an AR image based on the AR image data.

The internal reference point cloud data and the internal AR image data corresponding to the internal reference point cloud data may be associated with the model name of the mobile object P. Furthermore, the external reference point cloud data and the external AR image data corresponding to the external reference point cloud data may be associated with location information indicating latitude, longitude, and altitude, or information indicating an area. As an example, the external reference point cloud data and the external AR image data may be associated with the name of an airport, a port, a railway line, or a road.

The image processing device 1 may use reference point cloud data and AR image data stored in advance in the storage unit 15, rather than acquiring various data from an external server. In this case, the image processing device 1 may not need to have a communication unit 14.

The storage unit 15 has storage media such as ROM (Read Only Memory) and RAM (Random Access Memory). The storage unit 15 stores programs executed by the control unit 16. The storage unit 15 also stores reference point cloud data and AR image data received by the communication unit 14 from an external server in association with each other. The storage unit 15 may store multiple pieces of reference point cloud data and AR image data in association with location information or information indicating an area.

The storage unit 15 stores, as reference point cloud data, multiple pieces of internal reference point cloud data composed of feature points of the subject included in images captured at multiple positions in the internal space. The storage unit 15 may store the internal reference point cloud data in association with a position and orientation in the internal space. A position in the internal space is indicated by a distance and orientation relative to a reference position in the internal space. An orientation in the internal space is indicated by an orientation relative to a reference orientation in the internal space (e.g., forward).

The memory unit 15 stores, as reference point cloud data, multiple pieces of external reference point cloud data composed of feature points of subjects included in images captured at multiple positions in external space. The memory unit 15 may store the external reference point cloud data in association with a position and orientation in external space. The position in external space is indicated by a position (coordinates) relative to a reference position. The reference position is, for example, the location of an airport, port, railway line, or road where the image processing device 1 is located. The reference position may be expressed by latitude and longitude, with a position where latitude and longitude are 0 degrees as the reference position, and altitude relative to sea level. The orientation in external space is indicated by the orientation relative to a reference orientation in external space (for example, facing north).

Furthermore, the storage unit 15 stores internal AR image data in association with each of the multiple internal reference point cloud data. For example, the storage unit 15 stores internal AR image data showing different content in association with each of the multiple internal reference point cloud data, or stores internal AR image data of the same content but different sizes in association with each of the multiple internal reference point cloud data. Furthermore, the storage unit 15 may store internal AR image data in association with the state of the moving object P (e.g., speed, altitude, type of problem that has occurred).

The storage unit 15 also stores external AR image data in association with each of the multiple external reference point cloud data. For example, the storage unit 15 stores external AR image data showing different content in association with each of the multiple external reference point cloud data, or stores external AR image data with the same content but different sizes in association with each of the multiple external reference point cloud data. The storage unit 15 may also store external AR image data in association with a position and orientation in external space.

The storage unit 15 may store external reference point cloud data in association with the name of an airport, a port, a railway line, or a road. For example, the storage unit 15 stores a large number of external reference point cloud data created based on image data generated by photographing multiple points within Airport A in multiple orientations as external reference point cloud data for Airport A.

The storage unit 15 may also store the internal reference point cloud data in association with moving body identification information (e.g., model name) for identifying the moving body P. The storage unit 15 stores, as internal reference point cloud data for model B, a large number of internal reference point cloud data created based on captured image data generated by photographing, for example, multiple positions in the cockpit of model B in multiple orientations.

The control unit 16 has a CPU (Central Processing Unit) as a processor that processes various types of data by executing programs stored in the memory unit 15. By executing the programs stored in the memory unit 15, the control unit 16 functions as an image data acquisition unit 161, a boundary identification unit 162, a position identification unit 163, an internal image display processing unit 164, an external image display processing unit 165, and a display processing unit 166.

The image data acquisition unit 161 acquires captured image data generated by the image processing device 1 worn by the user U riding on the moving body P, capturing images of the interior space of the moving body P and the exterior space of the moving body P. That is, the image data acquisition unit 161 acquires captured image data generated by the imaging unit 11 while the user U is wearing the image processing device 1. The image data acquisition unit 161 acquires captured image data at predetermined time intervals. The predetermined time interval is arbitrary, but is, for example, 33.3 milliseconds (corresponding to a frame rate of 30 fps). The image data acquisition unit 161 inputs the acquired captured image data to the boundary identification unit 162 and the display processing unit 166.

The boundary identification unit 162 identifies the boundary between the area corresponding to the internal space and the area corresponding to the external space in the captured image data input from the image data acquisition unit 161. Based on the position of the boundary in the captured image data, the boundary identification unit 162 creates internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data. The boundary is the position indicated by the dashed line in the example shown in Figure 2.

The boundary identification unit 162 may identify the boundary in any manner, but for example, the boundary may be identified by machine learning (e.g., deep learning), the boundary may be identified based on the difference in the rate of change of feature points, or the boundary may be identified based on the difference in distance to feature points.

When the boundary identification unit 162 identifies boundaries using machine learning, for example, it inputs the captured image data acquired by the image data acquisition unit 161 into a machine learning model that learns captured image data including data indicating boundary positions as training data and outputs boundary position data indicating the boundary position when the captured image data is input. The boundary identification unit 162 identifies boundaries in the captured image data based on the boundary position data output from the machine learning model.

When identifying boundaries based on differences in the rate of change of feature points, the boundary identification unit 162 identifies boundaries based on, for example, the rate of change of the positions of multiple subjects included in multiple pieces of captured image data acquired at different times. Specifically, the boundary identification unit 162 creates first point cloud data consisting of feature points of multiple subjects included in captured image data generated at a first time, and second point cloud data consisting of feature points of multiple subjects included in captured image data generated at a second time following the first time. Feature points are points located on the contour line of a subject, on the boundary line of an uneven surface, or on the boundary line of an area of a different color.

The boundary identification unit 162 identifies the difference in position between multiple feature points included in the first point cloud data and multiple corresponding feature points in the second point cloud data. The greater this difference, the greater the rate of change of the feature points. Therefore, the boundary identification unit 162 classifies the multiple feature points into two groups based on the rate of change, and identifies areas containing feature points with a relatively slow rate of change as internal space, and areas containing feature points with a relatively fast rate of change as external space.

Figure 5 is an example of point cloud data created by the boundary identification unit 162. The point cloud data shown in Figure 5 corresponds to the image shown in Figure 2. In Figure 5, feature points corresponding to the internal space are indicated by white squares, and feature points corresponding to the external space are indicated by black circles.

The feature points of the internal space indicated by white squares have a slower rate of change than the feature points of the external space indicated by black circles, so the boundary identification unit 162 can identify the boundary between the internal space containing the feature points indicated by white squares and the external space containing the feature points indicated by black circles based on the rate of change. For example, the boundary identification unit 162 identifies, as the boundary, the line connecting multiple feature points that have a relatively slow rate of change and that are closest in the vertical and horizontal directions to feature points that have a relatively fast rate of change.

Similarly, when identifying a boundary based on the difference in distance to a feature point, the boundary identification unit 162 identifies a feature point that corresponds to a distance within a predetermined distance (e.g., 1 m) corresponding to the internal space as a feature point that corresponds to the internal space. Furthermore, the boundary identification unit 162 identifies a feature point that corresponds to a distance greater than the predetermined distance as a feature point that corresponds to the external space. For example, since the distance to the feature point indicated by the white square in Figure 5 is shorter than the distance to the feature point indicated by the black circle, it is possible to identify the boundary between the internal space that includes the feature point indicated by the white square and the external space that includes the feature point indicated by the black circle.

The boundary identification unit 162 creates internal space point cloud data corresponding to the internal space based on the internal space image data created based on the identified boundary, and creates external space point cloud data corresponding to the external space based on the external space image data. The boundary identification unit 162 inputs the created internal space point cloud data and external space point cloud data to the position identification unit 163. The boundary identification unit 162 may also divide the captured image data into internal space image data and external space image data, and input the internal space image data and external space image data to the position identification unit 163.

Note that if the user U does not move frequently within the internal space, the position of the boundary is likely to change very little. Therefore, the boundary identification unit 162 may identify the boundary within a predetermined time period after the moving object P starts moving, and store the identified position of the boundary in the storage unit 15. The boundary identification unit 162 may then create internal space point cloud data and external space point cloud data based on the position of the boundary stored in the storage unit 15.

The boundary identification unit 162 may also identify the boundary using other methods. As one example, the boundary identification unit 162 may identify the boundary by displaying an image based on the captured image data on an information terminal (e.g., a smartphone) used by the user U, and acquiring data indicating the boundary that the user U has input into the information terminal via the operation unit 13.

The position identification unit 163 identifies the position of the image processing device 1 in the internal space based on the internal space point cloud data input from the boundary identification unit 162, and identifies the position of the moving object P in the external space based on the external space point cloud data. That is, the position identification unit 163 identifies the position of the image processing device 1 in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and identifies the position of the image processing device 1 in the external space based on the positions of multiple feature points of the subject included in the external space image data. The position identification unit 163 may further identify the orientation of the image processing device 1 in the internal space and the orientation of the image processing device 1 in the external space.

As an example, the position identification unit 163 compares multiple internal reference point cloud data stored in the storage unit 15 with the internal space point cloud data input from the boundary identification unit 162 to identify the internal reference point cloud data that is closest to the internal space point cloud data. The position identification unit 163 identifies the position or orientation stored in the storage unit 15 in association with the internal reference point cloud data as the position or orientation of the image processing device 1 in the internal space. The position identification unit 163 notifies the internal image display processing unit 164 of the identified position or orientation of the image processing device 1.

The position identification unit 163 may identify the position and orientation of the image processing device 1 based on the relationship between the positions of multiple first feature points included in the latest first captured image data and the positions of multiple second feature points included in the immediately preceding second captured image data. Specifically, the position identification unit 163 identifies the amount of change between the positions of the multiple first feature points and the positions of the second feature points corresponding to each of the multiple first feature points, and identifies the position and orientation changed from the previously identified position and orientation of the image processing device 1 by a distance and orientation equivalent to the identified amount of change as the latest position and orientation of the image processing device 1.

In this case, if the feature points used by the position identification unit 163 to identify the position of the image processing device 1 include a mixture of feature points in the internal space and feature points in the external space, each of which has a different rate of change, the amount of change will differ depending on the feature point, and the position identification unit 163 will not be able to identify the space in which the image processing device 1 exists. As a result, the position identification unit 163 will be unable to identify the latest position and orientation of the image processing device 1 based on the position and orientation of the image processing device 1 identified immediately before.

In response to this, the position identification unit 163 identifies the position of the image processing device 1 in internal space based on the relationship (i.e., the amount of change in position) between the positions of multiple first feature points included in the latest internal space image data and the positions of multiple second feature points included in the immediately preceding internal space image data. Furthermore, the position identification unit 163 identifies the position of the image processing device 1 in external space based on the relationship between the positions of multiple first feature points included in the latest external space image data and the positions of multiple second feature points included in the immediately preceding external space image data. By operating in this manner, the position identification unit 163 can allow the image processing device 1 to correctly identify the position in internal space and the position in external space, even if the captured image data includes both internal space image data and external space image data.

The position identification unit 163 may identify the position of the image processing device 1 in the internal space using internal reference point cloud data corresponding to the model name of the moving object P indicated by the operation data input via the operation unit 13. By operating in this manner, the position identification unit 163 does not need to refer to a large number of internal reference point cloud data corresponding to a large number of models, thereby reducing the time required to identify the position of the image processing device 1 in the internal space.

Similarly, the position identification unit 163 compares multiple external reference point cloud data stored in the storage unit 15 with the external space point cloud data input from the boundary identification unit 162 to identify the external reference point cloud data that is closest to the external space point cloud data. The position identification unit 163 identifies the position or orientation stored in the storage unit 15 in association with the external reference point cloud data as the position or orientation of the image processing device 1 in external space. The position identification unit 163 notifies the external image display processing unit 165 of the identified position or orientation of the image processing device 1.

The position identification unit 163 may identify the position of the image processing device 1 in external space using external reference point cloud data corresponding to the location of the moving body P indicated by the operation data input via the operation unit 13. By operating in this manner, the position identification unit 163 does not need to refer to a large number of external reference point cloud data corresponding to a large number of locations, thereby reducing the time required to identify the position of the image processing device 1 in external space.

However, internal space image data changes less than external space image data. Therefore, the position identification unit 163 may identify the position of the image processing device 1 in the internal space at a slower cycle than the cycle at which the process of identifying the position of the image processing device 1 in the external space is executed. By operating the position identification unit 163 in this manner, the processing load can be reduced, and the time required for the position identification unit 163 to identify the position of the image processing device 1 in the external space can be shortened.

The internal image display processing unit 164 references the multiple internal AR image data stored in the memory unit 15 and instructs the display processing unit 166 to display on the display unit 12 an internal AR image corresponding to the position of the image processing device 1 in the internal space identified by the position identification unit 163. The internal image display processing unit 164 may also display an internal AR image corresponding to the orientation of the image processing device 1 in the internal space. The internal image display processing unit 164 may identify the internal reference point cloud data that is closest to the internal point cloud data from the multiple internal reference point cloud data stored in the memory unit 15, and display on the display unit 12 the internal AR image data stored in association with the identified internal reference point cloud data.

The internal image display processing unit 164 instructs the display processing unit 166 to display an internal AR image based on the internal AR image data at a position on the display unit 12 that is stored in the memory unit 15 in association with the identified internal AR image data. The internal image display processing unit 164 displays the internal AR image at a position determined based on the position of a specific subject (e.g., the joystick H) within the internal space in the captured image, such as the internal AR image G1 shown in FIG. 2(a) and the internal AR image G1' shown in FIG. 2(b). If the meter M2 is a speedometer, the internal image display processing unit 164 may display an internal AR image including information about speed at a position determined based on the position of the speedometer.

In this case, the internal image display processing unit 164 may display the internal AR image in a direction determined based on the direction of the display unit 12, regardless of the tilt of the image processing device 1 (i.e., the tilt of the user U). If the internal AR image contains content that does not need to be linked to the tilt of the subjects in the internal and external images, the internal image display processing unit 164 operating in this manner makes it easier for the user U to grasp the content of the internal AR image even when their posture is tilted. Note that if the internal AR image contains information indicating the direction in which the subject in the internal space is to be operated, the internal image display processing unit 164 may display the internal AR image in a state in which the internal AR image is tilted relative to the direction of the display unit 12 in accordance with the tilt of the image processing device 1.

The internal image display processing unit 164 may also display an internal AR image on the condition that the state of the internal space has reached a predetermined state. As an example, when the internal image display processing unit 164 detects a change in the state of an indicator light (e.g., an alarm light) in the cockpit based on the internal space image data, it may display an internal AR image containing information indicating what action should be taken at a predetermined position relative to the indicator light.

However, when the user U leans back, objects in the internal space (for example, the control stick H and meters M1 in the cockpit) appear smaller. Therefore, the internal image display processing unit 164 may change the size of the internal AR image to match the size of these objects. Specifically, the internal image display processing unit 164 may cause the display unit 12 to display an internal augmented reality image of a size determined based on a pattern of multiple feature points that indicate the characteristics of one or more objects in the captured image data.

More specifically, the internal image display processing unit 164 identifies internal AR image data of a size stored in the storage unit 15 in association with the internal reference point cloud data that is closest to the internal space point cloud data created by the position identification unit 163, and displays the identified internal AR image on the display unit 12. By operating in this manner, the internal image display processing unit 164 changes size depending on the posture of the user U, making it appear as if the internal AR image is present in the internal space, enhancing the sense of realism.

The external image display processing unit 165 references multiple external AR image data stored in the memory unit 15 and instructs the display processing unit 166 to display on the display unit 12 an external AR image corresponding to the position of the image processing device 1 in external space identified by the position identification unit 163. The external image display processing unit 165 displays the external AR image on the display unit 12 at a position that is stored in advance in the memory unit 15 in association with the position of the image processing device 1 in external space. The external image display processing unit 165 may also display an internal AR image that corresponds to the orientation of the image processing device 1 in external space. The external image display processing unit 165 may identify external AR image data stored in the memory unit 15 in association with external reference point cloud data that is closest to the external space point cloud data created by the position identification unit 163, and display on the display unit 12 an external AR image based on the identified external AR image data.

The display processing unit 166 displays the internal AR image data input from the internal image display processing unit 164 and the external AR image data input from the external image display processing unit 165 on the display unit 12. The display processing unit 166 may acquire captured image data from the image data acquisition unit 161, and display an image based on image data obtained by combining the acquired captured image data with the internal AR image data and external AR image data on the display unit 12.

[Processing flow in image processing device 1]
Fig. 6 is a flowchart showing the flow of processing in the image processing device 1. The flowchart shown in Fig. 6 starts when the image processing device 1 is powered on.

The image data acquisition unit 161 acquires captured image data from the imaging unit 11 at predetermined time intervals (S11). The boundary identification unit 162 creates point cloud data by extracting feature points of the subject contained in the acquired captured image data, and stores the created point cloud data in the storage unit 15 in association with time (S12).

The boundary identification unit 162 identifies the rate of change of the position of each feature point by comparing the created point cloud data with point cloud data corresponding to the immediately preceding captured image data (S13). The boundary identification unit 162 classifies the identified rate of change to identify the boundary between the area corresponding to the internal space and the area corresponding to the external space in the captured image data (S14). Based on the identified boundary, the boundary identification unit 162 creates internal space point cloud data corresponding to the internal space and external space point cloud data corresponding to the external space (S15). The boundary identification unit 162 associates the created internal space point cloud data and external space point cloud data with time and stores them in the storage unit 15.

Next, the position identification unit 163 identifies the internal reference point cloud data that is most similar to the internal space point cloud data created by the boundary identification unit 162 from the multiple internal reference point cloud data stored in the storage unit 15, and identifies the position stored in the storage unit 15 in association with the identified internal reference point cloud data as a position in the internal space of the image processing device 1 (S16). The internal image display processing unit 164 displays, on the display unit 12, an internal AR image corresponding to the position in the internal space identified by the position identification unit 163 (S17).

Similarly, the position identification unit 163 identifies the external reference point cloud data that is most similar to the external space point cloud data created by the boundary identification unit 162 from among the multiple external reference point cloud data stored in the storage unit 15, and identifies the position stored in the storage unit 15 in association with the identified external reference point cloud data as a position in the external space of the image processing device 1 (S18). The external image display processing unit 165 displays on the display unit 12 an external AR image that corresponds to the position in the external space identified by the position identification unit 163 (S19). Note that the order in which S16 and S17 and S18 and S19 are executed is arbitrary.

The image processing device 1 repeats the processes from S11 to S19 until an operation to stop displaying the AR image is performed on the operation unit 13 (NO in S20).

[Effects of image processing device 1]
As described above, the position identifying unit 163 identifies the position of the image processing device 1 in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and identifies the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data. By configuring the image processing device 1 in this way, it is possible to correctly identify the position of the image processing device 1 even when a person riding on the moving body P looks at the internal space and the external space of the moving body P at the same time.

Furthermore, by the position identification unit 163 identifying the position of the image processing device 1 in the internal space, the internal image display processing unit 164 can cause the display unit 12 to display an internal AR image corresponding to the position of the image processing device 1 in the internal space at a predetermined position in the internal space or external space viewed by the user. Furthermore, the external image display processing unit 165 can cause the display unit 12 to display an external AR image corresponding to the position of the image processing device 1 in the external space at a predetermined position in the internal space or external space viewed by the user U.

By configuring the internal image display processing unit 164 and the external image display processing unit 165 in this manner, when the user U is viewing an internal space and an external space that change at different speeds, it is possible to present the user U with internal AR images and external AR images that are appropriate for the state of the internal space and the state of the external space, respectively. Therefore, the image processing device 1 is suitable for applications that require the display of AR images while the user U is wearing and using the device while riding on a moving object P.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and alterations are possible within the spirit of the invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of new embodiments resulting from the combination will also have the effects of the original embodiments.

REFERENCE SIGNS LIST 1 Image processing device 11 Imaging unit 12 Display unit 13 Operation unit 14 Communication unit 15 Storage unit 16 Control unit 121 Right lens 122 Left lens 123 Right projection unit 124 Left projection unit 161 Image data acquisition unit 162 Boundary identification unit 163 Position identification unit 164 Internal image display processing unit 165 External image display processing unit 166 Display processing unit

Claims (9)

An image processing device in the form of glasses that can be worn by a person,
an image data acquisition unit that acquires captured image data generated by the image processing device worn by a person riding on the moving body by capturing images of the interior space of the moving body and the exterior space of the moving body;
a boundary specifying unit that specifies a boundary between internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data;
a position specifying unit that specifies a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data, and that specifies a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data;
an internal image display processing unit that displays, on a display unit of the image processing device, an internal augmented reality image corresponding to the position of the image processing device in the internal space identified by the position identifying unit;
and
When the internal augmented reality image includes content that does not need to be linked to the tilt of the subject within the internal space, the internal image display processing unit displays the internal augmented reality image in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device; when the internal augmented reality image includes information indicating the direction in which the subject within the internal space is to be operated, the internal image display processing unit displays the internal augmented reality image in a state in which the internal augmented reality image is tilted relative to the direction of the display unit in conjunction with the tilt of the image processing device.
Image processing device. the position identification unit identifies a position of the image processing device in the internal space based on a relationship between positions of the plurality of feature points included in the latest internal space image data and positions of the plurality of feature points included in the immediately preceding internal space image data, and identifies a position of the image processing device in the external space based on a relationship between positions of the plurality of feature points included in the latest external space image data and positions of the plurality of feature points included in the immediately preceding external space image data;
The image processing device according to claim 1 . the position specifying unit specifies the position of the image processing device in the internal space at a cycle slower than a cycle for executing a process for specifying the position of the image processing device in the external space;
3. The image processing device according to claim 1 or 2. an external image display processing unit that causes an external augmented reality image corresponding to the position of the image processing device in the external space identified by the position identification unit to be displayed on the display unit;
The image processing device according to claim 1 . the internal image display processing unit displays the internal augmented reality image on the display unit, the internal augmented reality image having a size determined based on a pattern of a plurality of feature points that indicate features of one or more subjects in the captured image data.
The image processing device according to claim 1 . the external image display processing unit displays the external augmented reality image on the display unit at a position that is stored in advance in a storage unit in association with a position of the image processing device in the external space;
The image processing device according to claim 4 . the image data acquisition unit acquires the captured image data at predetermined time intervals;
the boundary specifying unit specifies the boundary based on a speed of change in the position of each of a plurality of subjects included in the plurality of pieces of captured image data acquired at different times;
The image processing device according to claim 1 . The processor executes
acquiring captured image data generated by an eyeglass-shaped image processing device worn by a person riding on a moving body by capturing an image of an interior space of the moving body and an exterior space of the moving body;
Identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data;
Identifying a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data;
specifying a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data;
displaying an internal augmented reality image corresponding to the specified position of the image processing device in the internal space on a display unit of the image processing device;
and
In the displaying step, if the internal augmented reality image contains content that does not need to be linked to the tilt of the subject within the internal space, the internal augmented reality image is displayed in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device, and if the internal augmented reality image contains information indicating the direction in which the subject within the internal space is to be operated, the internal augmented reality image is displayed in a state in which it is tilted relative to the direction of the display unit in accordance with the tilt of the image processing device . The processor
acquiring captured image data generated by an eyeglass-shaped image processing device worn by a person riding on a moving body by capturing an image of an interior space of the moving body and an exterior space of the moving body;
Identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data;
Identifying a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data;
specifying a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data;
displaying an internal augmented reality image corresponding to the specified position of the image processing device in the internal space on a display unit of the image processing device;
Execute
In the displaying step, if the internal augmented reality image contains content that does not need to be linked to the tilt of the subject within the internal space, the internal augmented reality image is displayed in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device, and if the internal augmented reality image contains information indicating the direction in which the subject within the internal space is to be operated, the internal augmented reality image is displayed in a state in which the internal augmented reality image is tilted relative to the direction of the display unit in conjunction with the tilt of the image processing device.