JP7726966B2 - Image processing system and image processing method - Google Patents

Description

Embodiments of the present invention relate to an image processing device, an image processing method, and an image processing system.

Mixed reality, a technology that uses stereoscopic technology to combine real and virtual spaces, is known. For example, a method has been proposed that generates a stereoscopic image using left-eye and right-eye images captured by a stereo camera and a virtual model (virtual object). Another method has also been proposed that takes the user's viewpoint into account and superimposes a virtual model onto a real-space image.

JP 2012-94102 A Japanese Patent Application Laid-Open No. 2020-24497

With the advancement of the information society, manufacturing sites and other facilities are now creating virtual product models using 3D Computer Aided Design (CAD), and checking and reviewing the product's shape and design during the design phase using a personal computer (PC) or a goggle-type display device. Furthermore, in plant construction work, the status of plant equipment is checked and reviewed using virtual models on a PC during the design phase.

In the past, during on-site construction, construction supervisors and contractors would mentally compare the design drawings with the actual site conditions to create construction plans. However, nowadays, goggle-type holographic display devices can be used to display a virtual model of the completed state superimposed on a real-world image.

However, existing goggle-type holographic display devices do not have a means for multiple display devices to share and review the same virtual model. Therefore, in order to share and review the same virtual model across multiple display devices, each display device must store virtual model data, and the multiple display devices must place the virtual model in the same position on the on-site space image to allow for confirmation and review.

According to this technique, the shape of the virtual model seen by a user of one display device and the shape of the virtual model seen by a user of the other display device are the same model shape. However, the data used for display on the former display device and the data used for display on the latter display device are independent data. Therefore, even if one user changes the display content of the virtual model, this change cannot be reflected in the display content of the virtual model for the other user. For example, even if the former user rotates the virtual model, the latter user cannot see the rotation.

Therefore, embodiments of the present invention provide an image processing device, an image processing method, and an image processing system that enable multiple devices to conveniently share a virtual model.

According to one embodiment, an image processing system includes a plurality of image processing devices, each of which includes an alignment unit that aligns a coordinate system of a three-dimensional model space, in which a virtual object generated based on a design member designed by computer-aided design is placed and in which marker position coordinates are set, with a coordinate system of a real space , based on marker position information obtained by capturing an image of a marker placed at a position in the real space corresponding to the marker position coordinates and the marker position coordinates in the three-dimensional model space . Each of the plurality of image processing devices further includes an image processing unit that generates an image of the virtual object observed from a user's viewpoint position based on the alignment, in correspondence with a real space image observed from the user's viewpoint position. Each of the plurality of image processing devices further includes a superimposition display unit that superimposes the real space image and the virtual object . Furthermore , a first image processing device among the plurality of image processing devices and a second image processing device different from the first image processing device acquire virtual object data , which is data of the common virtual object, and relative position information representing a relative position in a coordinate system of the three-dimensional model space in which the virtual object is placed relative to the marker , and align the coordinate system of the three-dimensional model space with the coordinate system of the real space based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space .

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment; FIG. 2 is a diagram for explaining a model space in the first embodiment. 1 is a schematic diagram illustrating a configuration of an image processing system according to a first embodiment. FIG. 10 is a schematic diagram illustrating a configuration of an image processing system according to a second embodiment. FIG. 10 is a block diagram for explaining the operation of an image processing system according to a second embodiment. FIG. 10 is a schematic diagram showing the configuration of an image processing system according to a third embodiment. FIG. 10 is a schematic diagram showing the configuration of an image processing system according to a fourth embodiment. FIG. 10 is a block diagram for explaining the operation of an image processing system according to a fourth embodiment.

Embodiments of the present invention will now be described with reference to the drawings. In Figures 1 to 8, identical components are designated by the same reference numerals, and duplicate descriptions will be omitted.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of an image processing apparatus 1 according to the first embodiment.

The image processing device 1 of this embodiment is a device that aligns a real space captured by a camera with a three-dimensional model space, and displays a superimposed image of the real space and an image of a virtual object (virtual model). As shown in Figure 1, the image processing device 1 of this embodiment includes a head-mounted video unit 10, a design unit 20, and a display unit 30. Figure 1 also shows a marker M provided in real space for alignment purposes.

The head-mounted video unit 10 is a device worn on the head of an operator (user), and is, for example, a wearable computer equipped with a CPU (Central Processing Unit). In this embodiment, the image processing device 1 is a goggle-type holographic display device, and the head-mounted video unit 10 is goggles that function as a wearable computer. In the following description, the image processing device 1 will also be referred to as the "display device 1" as appropriate. The detailed configuration of the head-mounted video unit 10 will be described later.

The design unit 20 is a functional block that places design components in the coordinate system of the design space, and is, for example, a PC equipped with a CPU. In this embodiment, the design unit 20 is a CAD device that generates design components using a design support tool with a 3D CAD model and places the design components in the 3D model space for design. The detailed configuration of the design unit 20 will be described later.

The display unit 30 is a functional block that displays various types of information on a screen, and is, for example, a display device. In this embodiment, the display unit 30 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display unit 30 displays various types of information generated by the design unit 20. The display unit 30 also displays a real-space image and a virtual object superimposed on each other.

The marker M is a mark used to associate the coordinate system of the real space with the coordinate system of the three-dimensional model space used for design. In this embodiment, the marker M is, for example, a two-dimensional geometric pattern printed on a sheet of paper or the like.

Next, the detailed configuration of the design unit 20 will be described. The design unit 20 comprises an attribute information assignment unit 202, a design processing unit 204, and an input operation unit 206.

The attribute information assigning unit 202 assigns attribute information to the design components generated by the design processing unit 204. For example, the attribute information assigning unit 202 assigns 3D CAD model attribute information for selecting a 3D CAD model to the design components.

The design processing unit 204 generates design components using a design support tool and places the design components in the three-dimensional model space for design. The design processing unit 204 generates design components based on the attribute information assigned by the attribute information assigning unit 202 and places the design components in the three-dimensional model space for design. For example, the design processing unit 204 selects design components from the three-dimensional CAD model for design based on the three-dimensional CAD model attribute information assigned by the attribute information assigning unit 202.

The input operation unit 206 is used to input information required for operating the head-mounted imaging unit 10 and the design unit 20 into the image processing device 1 of this embodiment. The input operation unit 206 is composed of, for example, a keyboard, a mouse, a pointing device, etc.

Next, we will explain the detailed configuration of the head-mounted video unit 10. The head-mounted video unit 10 includes multiple cameras 102, a microphone 104, an input processing unit 106, a data storage unit 107, a recognition processing unit 108, an image control processing unit 110, an LCD screen 112, and a speaker 114.

The camera 102 outputs an image of real space as image data. The camera 102 is, for example, a color digital camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. The image data captured by the camera 102 is output to the image control processing unit 110 via the input processing unit 106.

The microphone 104 outputs audio collected near the head-mounted video unit 10 as audio data. For example, the microphone 104 collects audio as gestures made by the operator and outputs audio data about the gestures. This audio data is output to the recognition processing unit 108 via the input processing unit 106. Note that the image processing device 1 of this embodiment may further capture images of the operator's gestures using the camera 102, and output image data about the gestures from the camera 102 to the recognition processing unit 108 via the input processing unit 106.

The input processing unit 106 performs processing to input image data and audio data to the recognition processing unit 108 and image processing control unit 110. The input processing unit 106 includes an alignment unit 106a and a conversion unit 106b.

The alignment unit 106a aligns the coordinate system of the three-dimensional model space, in which the virtual object generated based on the design components is placed, with the coordinate system of real space. In this embodiment, the alignment unit 106a recognizes the position and orientation of the head-mounted video unit 10 using information about the marker M captured by the camera 102, and aligns the coordinate system of the three-dimensional model space with the coordinate system of real space. For example, the alignment unit 106a sets the position of the marker M to the coordinate origin of the coordinate system of the three-dimensional model space. In this case, the marker M is positioned in advance in a position in real space that corresponds to the coordinate origin of the coordinate system of the three-dimensional model space, and the position and orientation of the head-mounted video unit 10 is recognized based on the image data capturing the marker M.

The conversion unit 106b converts, into virtual objects, information about the design members generated by the design processing unit 204. The conversion unit 106b of this embodiment uses information about the design members arranged in the three-dimensional model space, for example, information about the three-dimensional CAD model, to convert the design members into virtual objects arranged in the three-dimensional model space.

The data storage unit 107 stores various types of data. The data storage unit 107 is composed of, for example, a magnetic recording device such as a hard disk drive (HDD) and/or semiconductor memory such as a random access memory (RAM), a read only memory (ROM), or a solid state drive (SSD). In this embodiment, the data storage unit 107 is used, for example, to store data and information used for alignment by the alignment unit 106a.

The recognition processing unit 108 performs various types of recognition processing. The recognition processing unit 108 includes a voice recognition processing unit 108a, an image recognition processing unit 108b, and a three-dimensional space recognition processing unit 108c.

The voice recognition processing unit 108a processes voice data acquired from the microphone 104 via the input processing unit 106. For example, the voice recognition processing unit 108a converts the voice data collected by the microphone 104 into an instruction code. In this case, the head-mounted video unit 10 performs a control operation in accordance with this instruction code.

The image recognition processing unit 108b processes image data acquired from the camera 102 via the input processing unit 106. For example, the image recognition processing unit 108b acquires distance information between the coordinates in the real-space image in the superimposed image and the coordinates on the image of the virtual object.

The three-dimensional space recognition processing unit 108c also processes image data acquired from the camera 102 via the input processing unit 106. For example, the three-dimensional space recognition processing unit 108c calculates the three-dimensional coordinates of natural feature points extracted from an image in real space based on the image data using SfM (Structure from Motion), and stores the calculated coordinates as landmark data. In this case, the three-dimensional space recognition processing unit 108c estimates the position and orientation of the head-mounted image unit 10 based on the correspondence between the landmark data and the natural feature points. This makes it possible to estimate the position and orientation of the head-mounted image unit 10 in the coordinate system in real space even when an image of the marker M is not captured.

The image processing control unit 110 controls image processing and other processes. The image processing control unit 110 includes a control unit 110a, an image processing unit 110b, and an output processing unit 110c.

The control unit 110a controls various operations of the image processing device 1 of this embodiment. For example, the control unit 110a controls the operation of the head-mounted video unit 10, and through control of the head-mounted video unit 10, controls the operation of the design unit 20 and display unit 30.

Based on the alignment performed by the alignment unit 106a, the image processing unit 110b combines a real-space image observed from the user's viewpoint with an image of the virtual object observed from the user's viewpoint. After the alignment performed by the alignment unit 106a is complete, the image processing unit 110b changes the position and shape of the virtual object generated based on the design components based on the position and orientation of the head-mounted video unit 10 estimated by the three-dimensional space recognition processing unit 108c, and superimposes it on the real-space image. The image processing unit 110b also generates a movable object of the virtual object and superimposes it on the real-space image. In this way, the virtual object generated by the image processing unit 110b is superimposed on the real-space image.

The output processing unit 110c converts the data format of data input from the image processing unit 110b and other units, and outputs the converted data. The converted data is output to, for example, the display unit 30, the LCD screen 112, or the speaker 114. The real-space image and virtual object superimposed by the image processing unit 110b are output from the output processing unit 110c and displayed in a superimposed manner on the display unit 30 or the LCD screen 112. The output processing unit 110c is an example of a superimposed display unit.

The LCD screen 112 displays various information at a position where it can be seen by a user wearing the head-mounted video unit 10. For example, the LCD screen 112 displays a superimposed image of real space observed from the user's viewpoint and an image of a virtual object observed from the user's viewpoint. The LCD screen 112 of this embodiment is a head-mounted display. The LCD screen 112 may achieve display using an optical see-through method. Such display is achieved, for example, by displaying an image of a virtual object generated according to the position and posture of the observer's viewpoint on an optical see-through LCD display. In this case, the image of the virtual object is superimposed so as to be superimposed on the actual real space rather than on the image of real space. In this way, the superimposed display of this embodiment may be performed on an image of real space or on the actual real space.

The speaker 114 outputs audio to the ears of a user wearing the head-mounted video unit 10. For example, the speaker 114 outputs audio based on the recognition results by the recognition processing unit 108, such as audio based on an instruction code output by the voice recognition processing unit 108a. This audio may be used as an instruction for the design support tool.

Note that communication between the head-mounted video unit 10 and the design unit 20 may be wireless or wired. Similarly, communication between the head-mounted video unit 10 and the display unit 30 may be wireless or wired. Furthermore, the design unit 20 may be located inside the head-mounted video unit 10 instead of being located outside the head-mounted video unit 10. An example of a design unit 20 located outside the head-mounted video unit 10 is a PC, and an example of a design unit 20 located inside the head-mounted video unit 10 is an MC (Micro Controller). When the design unit 20 is located inside the head-mounted video unit 10, the image processing device 1 does not need to be equipped with a display unit 30.

Furthermore, the image processing device 1 of this embodiment may generate design components using a design support tool and place the design components in a three-dimensional model space for design while the operator is wearing the head-mounted video unit 10. This makes it possible to observe the design components designed on-site as images of virtual objects. In this case, the UI (User Interface) for operating the design support tool may be realized by displaying a virtual keyboard or virtual numeric keypad on the LCD screen 112. This makes it possible to check, verify, and modify virtual objects of design components that overlap with the actual real space while performing the design process, further improving design efficiency. As a result, it becomes possible to eliminate the need for 3D laser scanning, leading to significant process shortening.

Figure 2 is a diagram explaining the model space of the first embodiment.

Figure 2 shows a lattice model MDL representing the model space and the arrangement of design members A1 to A4. Figure 2 shows a portion of a plane (horizontal plane) within the three-dimensional lattice point model MDL. The attribute information assignment unit 202 (Figure 1) generates a three-dimensional lattice point model MDL as a three-dimensional model space corresponding to the real space to be constructed, based on the actual dimensions of the real space to be constructed.

The lines X1 to X4 and Y1 to Y3 shown in Figure 2 indicate the X and Y coordinates within the lattice point model MDL. For example, the lines X1 to X4 and Y1 to Y3 correspond to the positions of the grid lines in real space. Furthermore, marker position M' indicates the position where marker M (Figure 1) is placed. For example, design members A1 to A3 are one flange pipe and two pipes at both ends of it, and design member A4 is one straight pipe.

The attribute information assigning unit 202 assigns the position at which the design member is to be placed within the three-dimensional model space to the design member as attribute information. For example, the attribute information assigning unit 202 assigns the position at which the design member is to be placed to the design member as attribute information based on coordinates on the lattice point model MDL specified by the input processing unit 106 (Figure 1). The attribute information assigning unit 202 can also assign the position at which the design member is to be placed to the design member as attribute information, corresponding to the position of the center line on the lattice point model MDL specified by the input processing unit 106. The attribute information assigning unit 202 may also place the design member within the three-dimensional model space using distance information acquired by the image recognition processing unit 108b (Figure 1).

Figure 3 is a schematic diagram showing the configuration of the image processing system of the first embodiment.

The image processing system of this embodiment is composed of two display devices (image processing devices) 1a and 1b. Each of display devices 1a and 1b has the same configuration as display device (image processing device) 1 shown in Figure 1. Each of display devices 1a and 1b of this embodiment is a goggle-type holographic display device. Figure 3 shows user Pa wearing the goggles (head-mounted image unit 10) of display device 1a, and user Pb wearing the goggles (head-mounted image unit 10) of display device 1b. One of display devices 1a and 1b is an example of a first device, and the other of display devices 1a and 1b is an example of a second device. Note that the image processing system of this embodiment may be composed of three or more display devices 1.

Figure 3 shows a real space 301 where users Pa and Pb are located, and a virtual model (virtual object) 302 superimposed on the real space image on the liquid crystal screen 112 (Figure 1) of display devices 1a and 1b. Figure 3 shows a cube as an example of a three-dimensional virtual model 302. Users Pa and Pb can see the virtual model 302 projected onto real space 301. Figure 3 also shows a marker M placed in real space 301.

The alignment unit 106a (Figure 1) provided in each of the display devices 1a and 1b will be described below.

In each of the display devices 1a and 1b, the alignment unit 106a aligns the coordinate system of the three-dimensional model space in which the virtual model 302 generated based on the design components is placed with the coordinate system of the real space 301. For example, the alignment unit 106a uses information about the marker M captured by the camera 102 (Figure 1) to recognize the position and orientation of the head-mounted video unit 10, and aligns the coordinate system of the three-dimensional model space with the coordinate system of the real space 301.

In each of the display devices 1a and 1b of this embodiment, the alignment unit 106a performs the above alignment based on virtual model data (virtual object data), which is data on the virtual model 302, and relative position information that indicates the relative position between the marker M and the virtual model 302. Figure 3 schematically shows virtual model data Da used in the display device 1a and virtual model data Db used in the display device 1b. On the other hand, an example of the relative position information is information regarding the position of the origin of the coordinate system of the three-dimensional model space in which the virtual model 302 is placed, relative to the position of the marker M.

In this embodiment, the alignment unit 106a of display device 1a and the alignment unit 106a of display device 1b acquire the same virtual model data and relative position information, i.e., virtual model data and relative position information common to display device 1a and display device 1b. Therefore, virtual model data Da and virtual model data Db are data with the same content. Then, the alignment unit 106a of display device 1a performs the above alignment based on the acquired virtual model data and relative position information. Similarly, the alignment unit 106a of display device 1b also performs the above alignment based on the acquired virtual model data and relative position information.

The alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b can obtain the same virtual model data and relative position information by, for example, obtaining the virtual model data and relative position information from the same storage. This storage may be provided within the display device 1a, within the display device 1b, or external to the display devices 1a and 1b. Furthermore, the display devices 1a and 1b may each store the virtual model data and relative position information obtained from the same storage in advance in the data storage unit 107 (Figure 1) of the display devices 1a and 1b, and read the virtual model data and relative position information from the data storage unit 107 of the display devices 1a and 1b when displaying the virtual model 302. Examples of such storage will be described later in the second to fourth embodiments.

As described above, display devices 1a and 1b display virtual model 302 based on the same virtual model data. This makes it possible for display devices 1a and 1b to display virtual models 302 of the same shape. In addition, display devices 1a and 1b perform the above-mentioned alignment based on the same virtual model data and relative position information. This makes it possible for display devices 1a and 1b to place virtual models 302 of the same shape in the same position and orientation in the real space image. This is because the reference position for placing virtual model 302 can be made common between display devices 1a and 1b. Therefore, according to this embodiment, display devices 1a and 1b can share the same display content of the same virtual model 302.

In Figure 3, user Pa is located to the right of virtual model 302, and user Pb is located to the left of virtual model 302. Therefore, user Pa observes virtual model 302 from the right side of virtual model 302 on the LCD screen 112, and user Pb observes virtual model 302 from the left side of virtual model 302 on the LCD screen 112. According to this embodiment, users Pa and Pb can simultaneously observe the same virtual model 302 from any position. In Figure 3, users Pa and Pb observe the same virtual model 302 from different angles.

Second Embodiment
FIG. 4 is a schematic diagram showing the configuration of an image processing system according to the second embodiment.

As shown in FIG. 4, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. FIG. 4 also shows a network connector 401, a network 402, a server 403, storage 404, a cloud 405, and storage 406 that can be used by the image processing system of this embodiment. Storages 404 and 406 are examples of storage external to display devices 1a and 1b.

The display devices 1a and 1b can be connected to a network 402 via a network connector 401. The network connector 401 is, for example, a communications router or a PC. The network 402 is, for example, an on-premise network such as a VPN (Virtual Private Network). The display devices 1a and 1b can access the storage 404 of the server 403 via the network 402. The storage 404 is, for example, configured with an HDD and/or SSD. Figure 4 schematically shows virtual model data D1 stored in the storage 404. The virtual model data D1 is data of the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 404.

The display devices 1a and 1b can also access storage 406 on the cloud 405 via the network connector 401. The storage 406 is configured, for example, with an HDD and/or SSD. Figure 4 schematically shows virtual model data D2 stored in the storage 406. Like the virtual model data D1, the virtual model data D2 is data on the virtual model 302. In this embodiment, relative position information indicating the relative position between the marker M and the virtual model 302 is also stored in the storage 406.

In this embodiment, the alignment unit 106a of display device 1a and the alignment unit 106a of display device 1b acquire the virtual model data D1 and relative position information from storage 404, thereby acquiring the same virtual model data D1 and relative position information. This makes it possible for display device 1a and display device 1b to perform the above-mentioned alignment based on the same virtual model data D1 and relative position information. In this case, the virtual model data D1 acquired by display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D1 acquired by display device 1b corresponds to the virtual model data Db in FIG. 3.

In this embodiment, the alignment unit 106a of the display device 1a and the alignment unit 106a of the display device 1b may acquire the same virtual model data D2 and relative position information by acquiring the virtual model data D2 and relative position information from the storage 406. In this case, the above alignment can be performed on the display device 1a and the display device 1b based on the same virtual model data D2 and relative position information. In this case, the virtual model data D2 acquired by the display device 1a corresponds to the virtual model data Da in FIG. 3, and the virtual model data D2 acquired by the display device 1b corresponds to the virtual model data Db in FIG. 3.

For example, if a company uses display devices 1a and 1b, storage 404 and 406 may be located at the company's headquarters, and display devices 1a and 1b may be used on-site. This makes it possible to provide virtual model data and relative position information to display devices 1a and 1b after they are transported to the site. However, storage 404 and 406 may be located somewhere other than the headquarters, and display devices 1a and 1b may be used somewhere other than the site.

Display devices 1a and 1b can also communicate with each other via network connector 401. An example of such communication is described below.

In the image processing system of this embodiment, display device 1a functions as a parent device, and display device 1b functions as a child device. A user Pa of display device 1a, which is the parent device, can input operations related to the virtual model 302 to display device 1a. For example, user Pa can measure the dimensions of the virtual model 302 displayed on the LCD screen 112, thereby displaying the measurement results on the LCD screen 112. Figure 4 shows the measured value L of the length of one side of a cube. User Pa can input operations related to the virtual model 302 to display device 1a by moving a hand-shaped cursor H on the LCD screen 112. For example, when measuring the length of one side of a cube, the length of the side can be measured by specifying the start and end points of the side with cursor H. User Pa can also input operations to display device 1a to move or rotate the virtual model 302 within the real-space image, thereby moving or rotating the virtual model 302 on the LCD screen 112.

In this way, the display device 1a, which is the parent device, can change the display content of the virtual model 302 in various ways. The image processing unit 110b of the display device 1a changes the display content of the virtual model 302 by accepting operations performed by the user Pa. This function of the image processing unit 110b is an example of a display control unit. In addition, the output processing unit 110c of the display device 1a transmits operation information related to operations performed by the user Pa to the display device 1b, which is the child device, via the network connector 401. This function of the output processing unit 110c is an example of an information providing unit.

When the image processing unit 110b of the display device 1b receives operation information from the display device 1a, it changes the display content of the virtual model 302 on the LCD screen 112 of the display device 1b based on the operation information. As a result, when the user Pa measures the length of one side of the cube, the measurement value L is displayed not only on the LCD screen 112 of the display device 1a but also on the LCD screen 112 of the display device 1b. Furthermore, when the user Pa performs an operation to rotate the virtual model 302, the virtual model 302 rotates not only on the LCD screen 112 of the display device 1a but also on the LCD screen 112 of the display device 1b. Therefore, according to this embodiment, changes to the display content of the virtual model 302 can be shared between the display devices 1a and 1b, and further, changes to the display content can be shared in real time. The user Pa is the operator of the virtual model 302, and the user Pb is the viewer of the operation results. The above operation information is an example of predetermined information provided from the parent device to the child device.

Note that the operation information may be in any format, as long as it is information that can identify the content of the operation performed by user Pa and the changes in the display content that accompany the operation. For example, operation information related to an operation to measure the length of one side of a cube can be constructed using information related to the start point of the side, the end point of the side, and the measurement value L.

Figure 5 is a block diagram illustrating the operation of the image processing system of the second embodiment.

As shown in FIG. 5, display devices 1a and 1b can acquire virtual model data D1 and relative position information from storage 404 via network connector 401 (FIG. 4). Furthermore, display device 1a, which is the parent device, can transmit data D to display device 1b, which is the child device, via network connector 401. An example of data D is the operation information described above.

In this case, data D may be transmitted from display device 1a to storage 404 and stored in storage 404, or data D stored in storage 404 may be transmitted to display device 1b. This makes it possible to store data D such as operation information in storage 404 together with virtual model data D1 and relative position information. In this case, storage 406 (Figure 4) may be used instead of storage 404.

According to this embodiment, it is possible to share changes to the display content of the virtual model 302 by transmitting operation information, which generally has a small amount of data, from display device 1a to display device 1b, rather than transmitting virtual model data, which generally has a large amount of data, from display device 1a to display device 1b. This makes it possible to reduce the amount of communication between display devices 1a and 1b. For example, it is believed that the amount of communication when transmitting operation information from display device 1a to display device 1b is significantly reduced compared to the amount of communication when transmitting video data of the virtual model 302 from display device 1a to display device 1b. Furthermore, according to this embodiment, security can be improved by limiting the data to be communicated between display devices 1a and 1b.

In the image processing system of this embodiment, display device 1a may always function as the parent device and display device 1b may always function as the child device, or the display device 1 functioning as the parent device and the display device 1 functioning as the child device may be switchable. Alternatively, display device 1a may always function as the child device and display device 1b may always function as the parent device. In this way, the parent-child relationship between display devices 1a and 1b may be fixed or variable. Further details of the parent device and child device will be described later in the third and fourth embodiments.

(Third embodiment)
FIG. 6 is a schematic diagram showing the configuration of an image processing system according to the third embodiment.

As shown in Figure 6, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. Similar to Figure 4, Figure 6 also shows a network connector 401 that can be used by the image processing system of this embodiment.

In this embodiment, the parent-child relationship between display devices 1a and 1b is fixed. Therefore, of display devices 1a and 1b, only display device 1a functions as the parent device, and of display devices 1a and 1b, only display device 1b functions as the child device. User Pa of display device 1a operates the virtual model 302, and user Pb of display device 1b views the results of the operation. According to this embodiment, as in the second embodiment, it is possible for display devices 1a and 1b to share changes to the display content of the virtual model 302.

In this embodiment, the virtual model data Da and relative position information are stored in the data storage unit 107 (Figure 1) of the display device 1a. Therefore, the alignment unit 106a of the display device 1a acquires the virtual model data Da and relative position information from the data storage unit 107 of the display device 1a. On the other hand, the data storage unit 107 of the display device 1b does not store the virtual model data and relative position information. Therefore, the alignment unit 106a of the display device 1b acquires the virtual model data Da and relative position information from the data storage unit 107 of the display device 1a via the network connector 401. This makes it possible for the display devices 1a and 1b to perform alignment based on the same virtual model data Da and relative position information.

According to this embodiment, even when the network 402 or cloud 405 is unavailable or when the storages 404 and 406 are unavailable, it is possible for the display devices 1a and 1b to share display content and changes thereto. On the other hand, according to the second embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to this embodiment.

In this embodiment, display device 1a is equipped with a data storage unit 107 that stores virtual model data Da and relative position information, while display device 1b is not equipped with a data storage unit 107 that stores virtual model data and relative position information. To achieve this configuration, display device 1b of this embodiment may not be equipped with a data storage unit 107 itself, or may be equipped with a data storage unit 107 that is not used to store virtual model data and relative position information.

(Fourth embodiment)
FIG. 7 is a schematic diagram showing the configuration of an image processing system according to the fourth embodiment.

As shown in Figure 7, the image processing system of this embodiment has the same components as the image processing system of the first embodiment. Similar to Figures 4 and 6, Figure 7 also shows a network connector 401 that can be used by the image processing system of this embodiment.

In this embodiment, not only can user Pa of display device 1a be the operator of virtual model 302 and user Pb of display device 1b be the viewer of the operation results, but user Pb of display device 1b can also be the operator of virtual model 302 and user Pa of display device 1a can be the viewer of the operation results. This makes it possible to change the display content of virtual model 302 more freely.

In this embodiment, the virtual model data Da and relative position information are stored in the data storage unit 107 ( FIG. 1 ) of the display device 1a. Therefore, the alignment unit 106a of the display device 1a acquires the virtual model data Da and the relative position information from the data storage unit 107 of the display device 1a. Furthermore, in this embodiment, the virtual model data Db and the relative position information are stored in the data storage unit 107 of the display device 1b. Therefore, the alignment unit 106a of the display device 1b acquires the virtual model data Db and the relative position information from the data storage unit 107 of the display device 1b. In this embodiment, the same virtual model data and relative position information are stored in advance in the data storage unit 107 of the display device 1a and the data storage unit 107 of the display device 1b. This makes it possible to perform alignment between the display device 1a and the display device 1b based on the same virtual model data and relative position information.

According to this embodiment, even when the network 402 or cloud 405 cannot be used, or when the storages 404 and 406 cannot be used, it is possible for the display devices 1a and 1b to share display content and changes thereto. Furthermore, according to this embodiment, it is possible to reduce the amount of communication between the display devices 1a and 1b compared to the third embodiment. This is because, in this embodiment, there is no need to provide the virtual model data Da and relative position information from the display device 1a to the display device 1b. In general, when the amount of virtual model data Da is large, it is desirable to adopt this embodiment, and when the amount of virtual model data Da is small, it is desirable to adopt the third embodiment.

In this embodiment, the parent-child relationship between display devices 1a and 1b is variable. That is, it is possible to switch between display device 1 functioning as a parent device and display device 1 functioning as a child device. This will be explained in more detail with reference to Figure 8.

Figure 8 is a block diagram illustrating the operation of the image processing system of the fourth embodiment.

Figure 8(a) shows an image processing system in which display device 1a functions as a parent device and display device 1b functions as a child device. In this embodiment, display devices 1a and 1b each include a switching unit 501 for switching the parent-child relationship. For example, the switching unit 501 of display device 1a sets display device 1a as a parent device, and the switching unit 501 of display device 1b sets display device 1b as a child device in response to a request from the switching unit 501 of display device 1a, thereby allowing display devices 1a and 1b to become a parent device and a child device, respectively. Switching of the parent-child relationship may be performed in response to an operation by user Pa or user Pb, or may be performed automatically by display device 1a or display device 1b. Furthermore, the switching unit 501 may be provided in only one of display devices 1a or 1b.

In Figure 8(a), display device 1a, which is the parent device, transmits data D, such as operation information, to display device 1b, which is the child device, via network connector 401. This makes it possible for changes to the display content made by user Pa to be reflected not only on the LCD screen 112 of display device 1a, but also on the LCD screen 112 of display device 1b.

Figure 8(b) shows an image processing system in which display device 1b functions as a parent device and display device 1a functions as a child device. The function of the switching unit 501 of display devices 1a and 1b is as described above.

In Figure 8(b), display device 1b, which is the parent device, transmits data D, such as operation information, to display device 1a, which is the child device, via network connector 401. This makes it possible for changes to the display content made by user Pb to be reflected not only on the LCD screen 112 of display device 1b, but also on the LCD screen 112 of display device 1a.

In each of the second to fourth embodiments, the image processing system may be configured with N display devices 1 (N is an integer equal to or greater than 3). In this case, one display device 1 may function as a parent device, and the remaining N-1 display devices 1 may function as child devices.

Although several embodiments have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. The novel devices, methods, and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and modifications may be made to the forms of the devices, methods, and systems described herein without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications that fall within the scope and spirit of the invention.

1, 1a, 1b: image processing device (display device),
10: Head-mounted video department, 20: Design department, 30: Display department,
102: camera, 104: microphone, 106: input processing unit,
106a: alignment unit, 106b: conversion unit, 107: data storage unit,
108: Recognition processing unit, 108a: Voice recognition processing unit,
108b: image recognition processing unit, 108c: three-dimensional space recognition processing unit,
110: image processing control unit, 110a: control unit, 110b: image processing unit,
110c: output processing unit, 112: liquid crystal screen, 114: speaker,
202: attribute information assigning unit, 204: design processing unit, 206: input operation unit,
301: Real space, 302: Virtual model,
401: network connector, 402: network, 403: server,
404: Storage, 405: Cloud, 406: Storage, 501: Switching unit

Claims (9)

An image processing system comprising a plurality of image processing devices,
Each of the plurality of image processing devices
a positioning unit that aligns a coordinate system of a three-dimensional model space in which virtual objects generated based on design members designed by computer-aided design are arranged and in which marker position coordinates are set, with a coordinate system of a real space, based on position information of the marker obtained by capturing an image of a marker arranged at a position in the real space corresponding to the marker position coordinates, and the marker position coordinates in the three-dimensional model space;
an image processing unit that generates an image of the virtual object observed from a viewpoint position of the user in correspondence with a real space image observed from a viewpoint position of the user based on the alignment;
a superimposition display unit that displays the real space image and the virtual object in a superimposed manner;
Equipped with
a first image processing device among the plurality of image processing devices and a second image processing device different from the first image processing device acquire virtual object data that is data of the common virtual object and relative position information that indicates a relative position in a coordinate system of the three-dimensional model space in which the virtual object is to be placed with respect to the marker, and align the coordinate system of the three-dimensional model space with the coordinate system of the real space based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space;
At least one of the first image processing device and the second image processing device further
a display control unit that changes the display content of the virtual object by the superimposed display unit by receiving an operation performed by the user;
an information providing unit that provides operation information relating to the operation performed by the user to other image processing devices;
Equipped with
the other image processing device changes the display content of the virtual object by its own image processing device based on the provided operation information;
Image processing system. The image processing system of claim 1, wherein each of the first image processing device and the second image processing device is a goggle-type holographic display device. The image processing system of claim 1, wherein the first image processing device and the second image processing device acquire the virtual object data and the relative position information stored in storage external to the first image processing device and the second image processing device. The image processing system of claim 3, wherein the first image processing device and the second image processing device acquire the virtual object data and the relative position information from the storage via a network, or acquire the virtual object data and the relative position information from the storage on a cloud. The image processing system according to claim 1 , wherein each of the first image processing device and the second image processing device includes a data storage unit that stores the virtual object data and the relative position information. one of the first image processing device and the second image processing device functions as a parent device that outputs the operation information regarding the virtual object;
the other of the first image processing device and the second image processing device functions as a slave device that acquires the operation information output by the master device.
The image processing system according to claim 1 . only the first image processing device of the first image processing device and the second image processing device functions as the parent device;
of the first image processing device and the second image processing device, only the second image processing device functions as the child device;
The image processing system according to claim 6 . The image processing system according to claim 6 , wherein the first image processing device and the second image processing device are switchable between functioning as the parent device and functioning as the child device. In each of a plurality of image processing devices that perform image processing,
an alignment unit aligns a coordinate system of a three-dimensional model space in which a virtual object generated based on a design member designed by computer-aided design is arranged and in which marker position coordinates are set, with a coordinate system of a real space, based on position information of the marker obtained by capturing an image of a marker arranged at a position in the real space corresponding to the marker position coordinates and the marker position coordinates in the three-dimensional model space;
an image processing unit generates an image of the virtual object observed from the viewpoint position of the user, corresponding to a real space image observed from the viewpoint position of the user, based on the alignment;
a superimposed display unit that superimposes the real space image and the virtual object;
a first image processing device among the plurality of image processing devices and a second image processing device different from the first image processing device acquire virtual object data that is data of the common virtual object and relative position information that indicates a relative position in a coordinate system of the three-dimensional model space in which the virtual object is to be placed with respect to the marker, and align the coordinate system of the three-dimensional model space with the coordinate system of the real space based on the position information of the marker obtained by each alignment unit capturing an image of the common marker at each position and the marker position coordinates in the three-dimensional model space;
At least one of the first image processing device and the second image processing device further
a display control unit that changes the display content of the virtual object by the superimposed display unit by accepting an operation performed by the user;
an information providing unit providing operation information relating to the operation performed by the user to another image processing device;
the other image processing device changes the display content of the virtual object by its own image processing device based on the provided operation information;
Image processing methods.