KR20240035363A - Digital twin-based realistic content production method and system - Google Patents

Abstract

An objective of the present invention is to provide a method and a system for simulating situations in which various conditions are applied in a metaverse space implemented using digital twin technology and implementing a building, a distance, a person, and the like by fusing it with a realistic 3D graphic technology. A system for providing a simulation result performed in a metaverse environment comprises: an electronic device; a metaverse server for providing a metaverse environment to the electronic device; a database server for providing implementation information for implementing the metaverse environment to the metaverse server; and an external device for providing observation information on the real world to the metaverse server or the database server.

Description

Digital twin-based realistic content production method and system}

The technical idea of the present disclosure relates to a method and system for providing a metaverse implemented based on a digital twin, and more specifically, implemented based on a digital twin to confirm result information that satisfies the conditions set by the user. It relates to a method and system for producing digital twin-based realistic content by simulating in the metaverse space and using the result information according to the simulation results.

The content described in this part simply provides background information on an embodiment of the present invention and does not constitute prior art.

A representative service that can provide virtual reality (VR), augmented reality (AR), or mixed reality (MR) to users is Metaverse. Metaverse is a compound word of Meta, meaning processing and abstraction, and Universe, meaning the real world. It refers to a three-dimensional virtual world and is an advanced concept than the existing term virtual reality environment.

Meanwhile, recently, technologies such as the Internet of Things (IoT), big data, artificial intelligence, and cyber physical systems (CPS) have been widely used, and digital twin technology is an integration of these technologies. This is attracting attention. Digital Twin is a technology that creates twins of objects in reality on a computer and predicts the results in advance by simulating situations that may occur in reality on a computer. Digital twin technology is expected to be applied to many industrial fields, including aviation, machinery manufacturing, and construction.

The present inventors developed digital twin-based realistic content technology and completed the present invention.

The purpose of the present invention is to simulate situations where various conditions are applied in a metaverse space implemented using digital twin technology and to implement buildings, streets, people, etc. by fusing this with realistic 3D graphic technology.

In particular, the present invention aims to implement a digital twin-based metaverse and meta-commerce system, realistic content for education, or realistic content for industrial use.

In addition, the present invention provides the optimal building form to prevent sunlight, view, and ground problems by utilizing metaverse space based on digital twin technology prior to constructing a new building to prevent problems with sunlight, view, and ground in the real world. The purpose is to present methods and systems that contribute to new urban planning, construction, operation, and development by predicting and constructing buildings in the real world.

A system that provides simulation results performed in a metaverse environment according to an embodiment disclosed in this document includes an electronic device, a metaverse server that provides a metaverse environment to the electronic device, and a metaverse environment that is provided to the metaverse server. It may include a database server that provides implementation information to be implemented, and an external device that provides observation information about the real world to the metaverse server or the database server. The electronic device establishes a wireless communication channel with the metaverse server through a communication circuit of the electronic device, receives implementation information implementing the metaverse environment from the metaverse server through the communication circuit, and receives implementation information implementing the metaverse environment from the metaverse server through the communication circuit. Based on the implementation information that implements the environment, output a metaverse screen through the display of the electronic device, and while outputting the metaverse screen, to check result information that satisfies the standard conditions set by the user, Obtain a user input to perform a simulation on a metaverse server, and in response to obtaining the user input, transmit a command for performing the simulation to the metaverse server, and the command includes information about the reference condition. may include. In response to receiving the command from the electronic device, the metaverse server uses artificial intelligence for the implementation information and the observation information to perform simulation to confirm result information that satisfies the reference condition, Based on the simulation results, result information that satisfies the standard conditions may be transmitted to the electronic device.

In the present invention, the implementation information that implements the metaverse environment is map information about the real environment corresponding to buildings, cities, streets, tourist attractions, event halls, performance halls, meta-commerce systems, educational realistic content, or industrial realistic content in the real world. , may include at least one of building information, terrain information, weather information, ground characteristics information, traffic signal information, traffic volume information, tourism information, financial information, or graphic image quality.

According to various embodiments disclosed in this document, buildings, streets, people, etc. are implemented using digital twin technology and realistic 3D graphic technology, and digital twin-based metaverse and meta-commerce systems, realistic content for education, or industrial realistic content are implemented. The realistic content for education and industrial realistic content are implemented as first-person participatory content to provide a high level of immersion, provide digital twin content through online data streaming, and provide high-quality graphics through character rendering data. can be provided.

In addition, various effects that can be directly or indirectly identified through this document may be provided.

Figure 1 shows a system that provides a metaverse space applying digital twin.
Figure 2 shows a flowchart of a method for providing a metaverse space applying digital twin.
Figure 3 shows a flowchart of a method of simulation using a metaverse space applying digital twin.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to “include” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary. In addition, terms such as "◎ unit", "◎ unit", and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware, software, or a combination of hardware and software. there is.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 shows a system 100 that provides a metaverse space applying digital twin.

Referring to FIG. 1, a system 100 that provides a metaverse space to which a digital twin is applied may provide the metaverse space to a user (eg, a user of the electronic device 101). A user (e.g., a user of the electronic device 101) uses an electronic device 101 (e.g., a smart phone, a wearable electronic device) capable of providing the metaverse space, and accesses the metaverse server to experience the metaverse space. You can access it at (103). The user can access the metaverse server 103, interact with other users in the metaverse space, and experience a metaverse environment that reflects the real world using a digital twin. For example, a metaverse environment that reflects the real world using a digital twin may include buildings, streets, people, etc., and can be applied to create a digital twin-based metaverse and meta-commerce system, realistic content for education, or industrial reality. type content can be implemented.

According to one embodiment, the user can simulate various situations (conditions) using the metaverse server 103. For example, a user can simulate the real world using the metaverse server 103 to provide a sense of immersion to the user prior to producing digital twin-based realistic content. The metaverse server 103 can implement a metaverse space reflecting the real world environment using measurement data about the real world environment provided from the database server 103, and uses machine learning, deep learning, or artificial intelligence. Thus, realistic content that can provide the best sense of immersion can be implemented by simulating realistic content (for example, realistic content for education, realistic content for industrial use). The metaverse server 103 can implement a metaverse space reflecting the real world environment using measurement data about the real world environment provided from the database server 103, and uses machine learning, deep learning, or artificial intelligence. By simulating various building forms, the optimal building form can be derived.

According to one embodiment, the system 100 that provides a metaverse space applying a digital twin may include an electronic device 101, a metaverse server 103, a database server 105, and an external device 107. You can. The system 100, which provides a metaverse space applying digital twin, includes the components shown in FIG. 1 (e.g., electronic device 101, metaverse server 103, database server 105, and external device ( 107)) may not be limited to this. For example, the system 100 may further include an external server that simulates various situations in the metaverse space. The external server can simulate various situations using machine learning or deep learning.

According to one embodiment, the electronic device 101 may be an electronic device that provides a metaverse environment so that the user of the electronic device 101 can access the metaverse server 103 and experience the metaverse space to the user. . For example, the electronic device 101 may include, but is not limited to, a smart phone or a wearable device. The wearable device may include, but is not limited to, a head mounted display (HMD) device or augmented reality (AR) glass.

According to one embodiment, the electronic device 101 may include a communication circuit, a display, and a processor. The electronic device 101 is not limited to the above-mentioned components and may further include other components. For example, the electronic device 101 may further include memory, sensors, microphones, or speakers.

According to one embodiment, the electronic device 101 may perform wireless communication with at least one of the metaverse server 103, the database server 105, or the external device 107 through the communication circuit. For example, the electronic device 101 may receive information constituting the metaverse environment from the metaverse server 103 through the communication circuit. The electronic device 101 may display a metaverse space implemented in 3D graphics through the display using the information constituting the received metaverse environment. For example, users can experience the metaverse space displayed through a display and implemented in 3D graphics. For another example, the electronic device 101 may transmit a command to start a simulation to the metaverse server 103 through the communication circuit and receive simulation results from the metaverse server 103.

According to one embodiment, the processor may be operatively and/or electrically connected to the communication circuit and the display. The processor may obtain commands from components of the electronic device 101 (e.g., communication circuit, display), interpret and/or process the commands, and control the electronic device 101.

According to one embodiment, the metaverse server 103 may implement a metaverse environment using digital twin technology. The metaverse server 103 may transmit information about the implemented metaverse environment to the electronic device 101. The electronic device 101 may use the information received from the metaverse server 103 to output the metaverse environment implemented using the digital twin technology through the display of the electronic device 101. The metaverse environment may be composed of various contents that reflect the real world, such as buildings, streets, and people. Implementation information for implementing a metaverse environment that reflects the real world using a digital twin may be information provided from the database server 105. For example, the implementation information may be at least one of map information, building information, weather information, ground information, traffic signal information, traffic volume information, or tourist information about buildings, streets, etc., and faces, expressions, and emotions about people, etc. , skin color, gender, etc.

According to one embodiment, the database server 105 may transmit the implementation information to the metaverse server 103. The database server 105 can update the implementation information. For example, the database server 105 may update the implementation information in real time based on real-time data received from the external device 107. The external device 107 may refer to an electronic device that can observe the real world. That is, the external device 107 may be a device that performs observation of the real world using at least one of aerial photography, drone 3D photography, or lidar photography. For example, the external device 107 may include a drone device, CCTV, a camera attached to a car, a device capable of aerial photography, and a weather observation device. The drone device can photograph the real world in real time using the camera of the drone device, and transmit the data captured in real time to the database server 105 or the metaverse server 103. For example, the drone device can photograph buildings, roads, or terrain in the real world in 3D and transmit the photographed data to the database server 105 or the metaverse server 103. The database server 105 can update the implementation information in real time using the captured data. The CCTV, the camera attached to the car, and the device capable of aerial photography can capture buildings, roads, or terrain in the real world in real time, and transmit the captured data in real time to the database server 105 or the metaverse server 103. ) can be transmitted. The weather observation device can measure weather information in the real world in real time and transmit the measured data to the database server 105 or the metaverse server 103. The metaverse server 103 can update the implementation information in real time using the captured data and measured data. The metaverse server 103 reflects the captured data or measured data in the implementation information and derives the optimal road shape and optimal building shape using at least one of machine learning, deep learning, or artificial intelligence. A simulation can be performed to do this.

Figure 2 shows a flowchart of a method for providing a metaverse space applying digital twin.

Below, with reference to FIG. 2, a method of providing a metaverse environment implemented using a digital twin to a user and allowing the user to simulate various situations using the metaverse environment will be described. Parts that overlap with what was explained with reference to FIG. 1 will be omitted and described.

In operation 201, the database server 105 may transmit implementation information implementing the metaverse environment to the metaverse server 103. The implementation information may be at least one of map information, building information, weather information, ground information, traffic signal information, traffic volume information, or tourist information for buildings, streets, etc., and may include faces, expressions, emotions, skin color, and gender for people, etc. It may be information such as:

In operation 203, the external device 107 can observe a real situation in the real world and transmit the observed information to the metaverse server 103. For example, a drone device can photograph buildings, roads, or terrain in the real world in 3D and transmit the photographed data to the metaverse server 103. For another example, a weather observation device may measure weather information in the real world and transmit the measured data to the metaverse server 103. The weather information may include information about sunlight, precipitation, wind strength and direction, etc. Although not shown in FIG. 2, the photographed data or the measured data can also be transmitted to the database server 105, and the database server 105 uses the photographed data or the measured data to create a metaverse. Implementation information that implements the environment can be updated in real time.

In operation 205, the metaverse server 103 can update implementation information implementing the metaverse environment received from the database server 105 in real time using photographed data or measured data about the real world. there is. The metaverse server 103 may store the updated implementation information. The metaverse server 103 may compare implementation information received from the database server 105 with observed information (eg, the captured data or the measured data) received from the external device 107. The metaverse server 103 may update the implementation information when the difference between the comparison result data is greater than or equal to a reference value. The metaverse server 103 may update the implementation information when the number of times observed information has been received from the external device 107 is greater than the standard number. The metaverse server 103 may update the implementation information when a reference time has elapsed since the observation information was received from the external device 107.

According to one embodiment, the database server 105 stores implementation information stored in the database server 105 and observed information (e.g., the photographed data or the measured data) received from the external device 107. You can compare. The database server 105 may update the implementation information when the difference in data as a result of the comparison is greater than or equal to a reference value. The database server 105 may update the implementation information when the number of times observed information is received from the external device 107 is greater than the standard number. The database server 105 may update the implementation information when a reference time has elapsed since the observation information was received from the external device 107.

In operation 207, the electronic device 101 may connect to the metaverse server 103 so that the user of the electronic device 101 can experience the metaverse environment. Although operation 207 is shown in FIG. 2 to be performed after operation 205, operation 207 may be performed at any time during operations 201 to 205.

According to one embodiment, the electronic device 101 may perform user authentication in response to detecting an input for accessing the metaverse server 103. The electronic device 101 can access the metaverse server 103 when the user is authenticated. The electronic device 101 may include a sensor and may authenticate a user through the sensor. The sensor may include a biometric sensor. The electronic device 101 can acquire biometric information through the biometric sensor and authenticate the user based on the biometric information. The biometric sensor may include a fingerprint sensor, ultrasonic fingerprint sensor, iris sensor, heart rate sensor, etc.

In operation 209, the metaverse server 103 may transmit information about the metaverse environment to the electronic device 101. The information about the metaverse environment may mean information for creating a metaverse environment by processing implementation information that implements the metaverse environment. The information about the metaverse environment may mean updated implementation information by reflecting observation information received from the external device 107.

In operation 211, the electronic device 101 may use information about the metaverse environment received from the metaverse server 103 to output a screen representing the metaverse environment through the display of the electronic device 101. The user of the electronic device 101 can experience the metaverse environment through a screen output through the display using information about the metaverse environment.

In operation 213, while outputting the metaverse environment through the display, the electronic device 101 may obtain user input for performing a simulation to confirm result information that satisfies the reference conditions in the metaverse environment. there is. For example, the electronic device 101 is similar to the real world and is suitable for developing first-person participatory content to provide a sense of immersion by providing map information about buildings, streets, etc., building information, weather information, ground information, traffic signal information, and traffic volume. In order to check information or tourist information, user input that allows simulation to be performed in the metaverse environment can be obtained. For another example, the electronic device 101 is configured to determine whether the sunlight and view rights of buildings, streets, etc. in the real world are greater than or equal to a certain standard value, and to check the shape of the building to prevent ground problems in the building or street, in the metaverse environment. User input that allows the simulation to be performed can be obtained.

The user input may include at least one of touch input, drag input, swipe input, button input, gesture input, or voice input.

In operation 215, the electronic device 101 may transmit a command for performing the simulation to the metaverse server 103 in response to obtaining the user input.

In operation 217, the metaverse server 103 may perform a simulation in response to receiving a command to perform the simulation from the electronic device 101. The metaverse server 103 configures the implementation information received from the database server 105 and the measurement information received from the external device 107 using at least one of machine learning, deep learning, or artificial intelligence to enable user settings. To confirm result information that satisfies a standard condition, a simulation can be performed. For example, the metaverse server 103 uses the implementation information and the measurement information at least one of machine learning, deep learning, or artificial intelligence to conduct a simulation to confirm a road shape that minimizes traffic congestion in the real world. can be performed. For another example, the metaverse server 103 uses the implementation information and the measurement information at least one of machine learning, deep learning, or artificial intelligence to calculate the sunlight and view rights of buildings and streets in the real world as standard values. As above, simulations can be performed to confirm the shape of the building to prevent ground problems in the building, or to implement a real world suitable for providing immersion by developing first-person participatory content.

According to one embodiment, the metaverse server 103 may perform a simulation using the implementation information and the measurement information using a learning model of an artificial neural network (ANN). The artificial neural network may include a Deep Feedforward Network (DNF), a Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM).

According to one embodiment, when there is no result information that satisfies the standard condition as a result of performing a simulation, the metaverse server 103 automatically changes the standard condition within a set range and satisfies the changed standard condition. A system that performs simulation to confirm the resulting information. The set range may be set by the user and may mean a level range in which the essential characteristics of the reference condition do not change.

In operation 219, the metaverse server 103 may transmit the simulation results to the electronic device 101. The metaverse server 103 may transmit the simulation result or a notification indicating the result to another electronic device (eg, a wearable device) of a user wirelessly connected to the electronic device 101.

Figure 3 shows a flowchart of a method of simulation using a metaverse space applying digital twin.

In operation 301, the metaverse server 103 may confirm reference conditions for the simulation in response to receiving a command to perform a simulation from the electronic device 101. The command may include information about the reference condition. For example, the reference condition may be in the form of a street to minimize traffic congestion in the real world. For another example, the standard condition may be the location and shape of a building in the real world where the sunlight and view rights of the building are greater than or equal to standard values, and the ground characteristics of the building are greater than or equal to standard values. The form of a building may include the height, width, and shape of the building. In addition, the above standard condition may be graphic quality where the graphic quality characteristics exceed the standard value so that digital twin content can be produced through online data streaming in order to develop a real world suitable for providing immersion by developing first-person participatory content. In addition, the character rendering data may be rendering data that exceeds a standard value so that high-quality graphics can be provided through the character rendering data.

In addition, in order to develop a real world suitable for providing a sense of immersion by developing first-person participatory content, the first rendered data through cloud computing is received on a PC and images are expressed, thereby minimizing the data processing amount of the PC and providing high-quality By displaying photorealistic graphics, realistic graphics can be delivered in real time.

In operation 303, the metaverse server 103 may perform simulation using at least one of machine learning, deep learning, artificial intelligence, or artificial neural network to satisfy the above reference conditions. The simulation operation may be performed on a server other than the metaverse server 103. For example, the metaverse server 103 transmits information about the reference conditions to an external server, and the external server uses one of machine learning, deep learning, artificial intelligence, or artificial neural network based on the information about the reference conditions. Simulation can be performed using at least one. The external server may transmit the simulation results to the metaverse server 103.

According to one embodiment, the metaverse server 103 uses at least one of machine learning, deep learning, artificial intelligence, or artificial neural network based on information about traffic volume, information about the signal system, and information about the location of the building. By using this, the sunlight and view rights of buildings and streets in the real world are greater than the standard values, and the shape of the building can be confirmed to prevent ground problems, or it can be developed as first-person participatory content to provide a sense of immersion in the real world. can be implemented. The information about the traffic volume may be data captured by an external device 107 (e.g., drone device, CCTV, camera installed in cars, aerial photography device). The information about the signaling system may be information obtained from a government agency that controls the signaling system. The information about the location of the building may be data captured by an external device 107 or a Geographic Information System (GIS).

According to one embodiment, the metaverse server 103 performs machine learning and deep learning based on at least one of information about sunlight, precipitation, wind strength and direction, information about ground characteristics, and information about the shape of the building. , using at least one of artificial intelligence, or artificial neural networks, it is possible to determine the form of a building in the real world where the sunlight and view rights are above standard values and the ground characteristics are above standard values. The information about the amount of sunlight, precipitation, and wind strength and direction may be information observed by an external device 107 or information obtained from a government agency that collects weather information. The information on ground characteristics may be information observed by an external device 107 or information obtained from a government agency that collects information on ground characteristics. The information about the shape of the building may include information about the height, width, and shape of the building.

In operation 305, the metaverse server 103 can confirm simulation results that satisfy the above reference conditions.

The embodiments of the present invention described above are not only implemented through devices and methods, but can also be implemented through programs that implement functions corresponding to the configurations of the embodiments of the present invention or recording media on which the programs are recorded.

As above, exemplary embodiments have been disclosed in the drawings and specification. In this specification, embodiments have been described using specific terms, but this is only used for the purpose of explaining the technical idea of the present disclosure and is not used to limit the meaning or scope of the present disclosure described in the claims. Therefore, those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present disclosure should be determined by the technical spirit of the attached claims.

Claims (10)

In a system that provides simulation results performed in a metaverse environment,
The system includes an electronic device, a metaverse server that provides a metaverse environment to the electronic device, a database server that provides implementation information for implementing the metaverse environment to the metaverse server, and observation information about the real world. Includes a metaverse server or an external device provided to the database server,
The electronic device:
Establishing a wireless communication channel with the metaverse server through a communication circuit of the electronic device,
Receiving implementation information implementing the metaverse environment from the metaverse server through the communication circuit,
Based on implementation information implementing the metaverse environment, output a metaverse screen through the display of the electronic device,
While outputting the metaverse screen, obtain user input to perform a simulation on the metaverse server to confirm result information that satisfies the reference conditions set by the user,
In response to obtaining the user input, transmitting a command for performing the simulation to the metaverse server, the command including information about the reference condition,
The metaverse server is:
In response to receiving the command from the electronic device, using artificial intelligence for the implementation information and the observation information, perform a simulation to confirm result information that satisfies the reference condition,
Based on the simulation results, the system transmits result information satisfying the reference conditions to the electronic device.
In claim 1,
The artificial intelligence includes machine learning, deep learning, and artificial neural network (ANN), and the artificial neural network includes deep feedforward network (DNF), recurrent neural network (RNN), and long short-term memory (LSTM). system, including.
In claim 1,
The metaverse server updates the implementation information based on observation information received from the external device.
In claim 3,
The metaverse server is:
Compare the observation information and the implementation information to determine a difference value,
A system for updating the implementation information when the difference value is greater than or equal to a reference value.
In claim 3,
The metaverse server is:
Identifying the number of times the observation information has been received from the external device,
A system for updating the implementation information when the number of times is greater than or equal to a reference number.
In claim 3,
The metaverse server is:
Obtain time information corresponding to the point in time when observation information is received from the external device,
Based on the time information, the system updates the implementation information when a reference time or more has passed from the point in time.
In claim 1,
The metaverse server is:
As a result of performing the simulation, if there is no result information that satisfies the standard conditions, the standard conditions are automatically changed within a set range,
A system that performs simulation to confirm result information that satisfies the changed reference conditions.
In claim 1,
The database server is:
Receive the observation information from the external device, and
Based on the observation information, update implementation information stored in the database server,
A system that transmits the updated implementation information to the metaverse server.
In claim 1,
The external device is a device that performs observation of the real world using at least one of aerial photography, drone 3D photography, or lidar photography.
In claim 1,
Implementation information that implements the metaverse environment includes map information about the real environment, buildings, buildings, cities, streets, tourist attractions, event halls, performance halls, meta-commerce systems, educational realistic content, or industrial realistic content corresponding to the real world. A system including at least one of information, topographic information, weather information, ground characteristics information, traffic signal information, traffic volume information, tourism information, financial information, or graphic image quality.