JP7685262B2 - SYSTEM AND METHOD FOR DISPLAYING ENHANCED ANATOMICAL FEATURES - Patent application - Google Patents

Description

This PCT international application claims priority to U.S. Patent Application No. 17/062,085, filed October 2, 2020, which is a continuation-in-part of U.S. Patent Application No. 16/514,163, filed July 17, 2019, the contents of which are incorporated herein by reference in their entireties.

This disclosure generally relates to methods, systems, and user devices for displaying internal anatomical structures.

This section provides background information related to the present disclosure and is not necessarily prior art.

Augmented reality technology has the ability to modify or augment a user's view of the surrounding environment by overlaying computer-generated imagery on top of the user's view of the real world, creating a composite view of both real and virtual elements. Augmented reality provides a richer experience for the user by enhancing the user's perception of their environment and their immediate surroundings through overlaid digital content. Users can augment their view through a variety of electronic devices, such as wearable technology (e.g., headsets, glasses, smart watches, etc.), tablets, laptops, mobile devices, or other devices. Users can use these electronic devices to augment their perception of their environment, for example, by overlaying information about the user's surroundings or overlaying graphical images to enhance the user's perception of their current environment.

Augmented reality can be used in a variety of environments and by a variety of users to educate each user about their surroundings. For example, rail yard staff could wear augmented reality glasses that allow them to view information about the trains in the yard, or biologists could use augmented reality to help them identify different plant species around them.

Health professionals such as doctors and nurses are constantly in need of technological assistance to treat their patients. In particular, health professionals constantly need to obtain and accumulate data about their patients to assess the best treatment plan for the patient. Health professionals would greatly benefit from the use of augmented reality in collecting data about their patients. Although known augmented reality technologies are being used by health professionals to collect patient data, there is still a need for continuous improvement in the associated technology.

U.S. Patent Application Serial No. 16/514,163 U.S. Patent Application Serial No. 17/062,085

This section provides a general overview of the disclosure and is not a comprehensive disclosure of its complete scope or all of its features.

The current disclosure provides various methods, user devices, and systems for displaying augmented anatomical features. One aspect of the disclosure provides a method. The method includes detecting a target individual. The target individual has a body. The method further includes tracking the target individual. The method further includes displaying a visual representation of the body on a display. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at a processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of anatomical features. The method further includes displaying, on the display, a graphical representation of the anatomical features overlaid on the visual representation of the body. The graphical representation of the anatomical features is positioned on the visual representation of the body based on the anatomical profile.

Disclosed embodiments may further include one or more of the following features: In some embodiments, the plurality of reference markers correspond to at least one of an umbilicus, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, a muscle, or a blood vessel.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some embodiments, the anatomical profile of the target individual includes a default anatomical profile that is modified based on multiple reference markers.

According to another aspect of the disclosure, a user device is provided. The user device includes a display. The user device further includes data processing hardware in communication with the display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. According to this aspect, the operations perform a method. The method includes detecting a target individual. The target individual has a body. The method further includes tracking the target individual. The method further includes displaying, on the display, a visual representation of the body. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of anatomical features. The method further includes displaying, on the display, a graphical representation of the anatomical features overlaid on the visual representation of the body. The graphical representation of the anatomical features is positioned on the visual representation of the body based on the anatomical profile.

This aspect may further include one or more of the following features: In some implementations, the plurality of reference markers correspond to at least one of an umbilicus, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, a muscle, or a blood vessel.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some embodiments, the anatomical profile of the target individual includes a default anatomical profile that is modified based on multiple reference markers.

According to another aspect of the disclosure, a system is provided. The system includes a user device. The user device includes a first display. The user device further includes data processing hardware in communication with the first display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. According to this aspect, the operation performs a method. The method includes detecting a target individual. The target individual has a body. The method further includes tracking the target individual. The method further includes displaying a visual representation of the body on the first display. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of anatomical features. The method further includes displaying, on the first display, a graphical representation of the anatomical features overlaid on the visual representation of the body. The graphical representation of the anatomical features is positioned on the visual representation of the body based on the anatomical profile. The system further includes a wearable portion in communication with the user device. The wearable portion includes a second display configured to display the graphical representation of the anatomical features overlaid on the visual representation of the body. The graphical representation of the anatomical features is positioned on the visual representation of the body based on the anatomical profile.

This aspect may further include one or more of the following features: In some implementations, the plurality of reference markers correspond to at least one of an umbilicus, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, a muscle, or a blood vessel.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some implementations, the wearable portion is further configured to detect and track target individuals independently of the user device.

In some embodiments, the anatomical profile of the target individual includes a default anatomical profile that is modified based on multiple reference markers.

According to another aspect of the disclosure, a method is provided. The method includes detecting a target individual. The target individual has a body. The method further includes displaying a visual representation of the body on a display. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of internal anatomical features. The method further includes selecting a medical procedure. The method further includes determining a future state anatomical profile corresponding to the selected medical procedure. The method further includes displaying, on the display, a graphical representation of the visual representation of the body modified with the internal anatomical features based on the selected medical procedure.

According to another aspect of the disclosure, a method is provided. The method includes detecting a target individual. The target individual has a body. The method further includes displaying, on a display, a visual representation of the body. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of internal anatomical features. The method further includes displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features.

The disclosed embodiments may further include one or more of the following features: In some embodiments, the plurality of reference markers correspond to at least one of the navel, a portion of the sternum, a portion of the buttocks, a portion of the clavicle, a portion of the shoulder, the nose, the corners and heads of the eyes, the tips of the ears, the chin, etc.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, or a muscle.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some embodiments, the method further includes selecting a medical procedure and determining a future state anatomical profile corresponding to the selected medical procedure, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the selected medical procedure. In such embodiments, the graphical representation of the body may be modified with external anatomical features based on the selected medical procedure. In such embodiments, the method further includes providing a graphical representation of the body modified with external anatomical features based on the selected medical procedure. In such embodiments, the selected medical procedure is a procedure affecting at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers correspond to a body structure such as an umbilicus, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, a corner of a mouth, a tip of a nose, or an ear. In such embodiments, the anatomical profile of the target individual may further be based on a plurality of data corresponding to the body. In such embodiments, the plurality of reference markers may be identified by a user interacting with the display.

In some embodiments, the method may include identifying at least one of a disease, a genetic condition, or an anatomical variation, and determining a plurality of reference markers and an anatomical profile corresponding to at least one of the identified disease, the identified genetic condition, or the identified anatomical variation, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the determined anatomical profile. In such embodiments, the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may correspond to at least one of an umbilicus, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, a lip, a corner of a mouth, a tip of a nose, or an ear. In such embodiments, the anatomical profile of the target individual may further be based on a plurality of data corresponding to the body. In such embodiments, the plurality of internal anatomical features may include at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may be identified by a user interacting with the display.

According to another aspect of the disclosure, a user device is provided. The user device includes a display. The user device further includes data processing hardware in communication with the display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. According to this aspect, the operations perform a method. The method includes detecting a target individual. The target individual has a body. The method further includes displaying, on the display, a visual representation of the body. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of internal anatomical features. The method further includes displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features.

This aspect can include one or more of the following features: In some implementations, the plurality of reference markers correspond to at least one of the navel, a portion of the sternum, a portion of the buttocks, a portion of the clavicle, a portion of the shoulder, the lips, the corners of the mouth, the tip of the nose, or the ears.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, or a muscle.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some implementations, the multiple reference markers are identified by data processing hardware.

In some embodiments, the anatomical profile of the target individual includes a default anatomical profile that is modified based on multiple reference markers.

In some embodiments, the operations further include selecting a medical procedure and determining a future state anatomical profile corresponding to the selected medical procedure, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the selected medical procedure. In such embodiments, the selected medical procedure is a procedure that affects at least one of an organ, a bone, or a muscle.

In some embodiments, the operation further includes identifying at least one of a disease, a genetic condition, or an anatomical variation, and determining an anatomical profile corresponding to the plurality of reference markers and the at least one of the identified disease, the identified genetic condition, or the identified anatomical variation, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the determined anatomical profile. In such embodiments, the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may correspond to at least one of an umbilicus, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, a lip, a corner of a mouth, a tip of a nose, or an ear. In such embodiments, the anatomical profile of the target individual may further be based on a plurality of data corresponding to the body. In such embodiments, the plurality of internal anatomical features may include at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may be identified by a user interacting with the display.

According to another aspect of the disclosure, a system is provided. The system includes a user device. The user device includes a first display. The user device further includes data processing hardware in communication with the first display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. According to this aspect, the operation performs a method. The method includes detecting a target individual. The target individual has a body. The method further includes displaying a visual representation of the body on the first display. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, an anatomical profile of the target individual. The anatomical profile of the target individual is determined based on the plurality of reference markers. The anatomical profile includes a plurality of internal anatomical features. The system further includes a wearable portion in communication with the user device. The wearable portion includes a second display configured to display a graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features.

This aspect can include one or more of the following features: In some implementations, the plurality of reference markers correspond to at least one of an umbilicus, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.

In some embodiments, the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.

In some embodiments, the plurality of anatomical features includes at least one of an organ, a bone, or a muscle.

In some implementations, the multiple reference markers are identified by a user interacting with the display.

In some implementations, the wearable portion is further configured to detect and map the target individual independent of the user device.

In some embodiments, the selected medical procedure is a procedure that affects at least one of an organ, a bone, or a muscle.

In some embodiments, the method further includes selecting a medical procedure and determining a future state anatomical profile corresponding to the selected medical procedure, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the selected medical procedure.

In some embodiments, the method further includes identifying at least one of a disease, a genetic condition, or an anatomical variation, and determining a visual representation of the body modified with a plurality of reference markers and an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation, where the graphical representation includes a visual representation of the body modified with internal anatomical features based on the determined anatomical profile. In such embodiments, the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may correspond to at least one of an umbilicus, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, a lip, a corner of a mouth, a tip of a nose, or an ear. In such embodiments, the anatomical profile of the target individual may further be based on a plurality of data corresponding to the body. In such embodiments, the plurality of internal anatomical features may include at least one of an organ, a bone, or a muscle. In such embodiments, the plurality of reference markers may be identified by a user interacting with the display.

According to another aspect of the disclosure, a method is provided. The method includes mapping a target individual. The target individual has a body. The method further includes generating an initial three-dimensional representation of the body based on the mapping, the three-dimensional representation including a plurality of anatomical features of the body. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method further includes determining, at the processor, a preferred anatomical profile of the target individual. The preferred anatomical profile of the target individual is determined based on the plurality of reference markers. The preferred anatomical profile is a preferred three-dimensional representation of the body. The method includes three-dimensionally modifying, at the processor, the initial three-dimensional representation of the body to have a shape of the preferred anatomical profile, and displaying, on the display, the modified initial three-dimensional representation of the body.

In some embodiments, the multiple reference markers correspond to at least one of the navel, a portion of the sternum, a portion of the buttocks, a portion of the clavicle, a portion of the shoulder, the lips, the corners of the mouth, the tip of the nose, or the ears.

The disclosed embodiments may further include one or more of the following features: In some implementations, the method includes selecting a medical procedure associated with a desired body part, replacing the desired body part with a corresponding desired body part taken from the preferred anatomical profile to generate a future anatomical image, and displaying the future anatomical image.

In some embodiments, the medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.

In some embodiments, the future anatomical images are displayed in three dimensions.

According to another aspect of the disclosure, a user device is provided. The user device includes a display. The user device further includes data processing hardware in communication with the display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. According to this aspect, the operations perform a method. The method includes mapping a target individual. The target individual has a body. The method further includes generating an initial three-dimensional representation of the body based on the mapping. The method further includes identifying a plurality of reference markers on the initial three-dimensional representation of the body. The method further includes determining, at a processor, a preferred anatomical profile of the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of the body, and three-dimensionally modifying, at the processor, the initial three-dimensional representation of the body to have a shape of the preferred anatomical profile. The method further includes displaying, on the display, a graphical representation of the modified initial three-dimensional representation of the body.

In some embodiments, the multiple reference markers correspond to at least one of the navel, a portion of the sternum, a portion of the buttocks, a portion of the clavicle, a portion of the shoulder, the lips, the corners of the mouth, the tip of the nose, or the ears.

The disclosed embodiments may further include one or more of the following features: In some implementations, the method includes selecting a medical procedure associated with a desired body part, replacing the desired body part with a corresponding desired body part taken from the preferred anatomical profile to generate a future anatomical image, and displaying the future anatomical image.

In some embodiments, the medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.

In some embodiments, the future anatomical images are displayed in three dimensions.

According to another aspect of the disclosure, a system is provided. The system includes a user device. The user device includes a first display. The user device further includes data processing hardware in communication with the first display. The user device further includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. According to this aspect, the operation performs a method. The method includes mapping a target individual. The target individual has a body. The method further includes generating an initial three-dimensional representation of the body based on the mapping. The method further includes identifying a plurality of reference markers on the initial three-dimensional representation of the body. The method further includes determining, at the processor, a preferred anatomical profile of the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of the body. The method further includes three-dimensionally modifying, at the processor, the initial three-dimensional representation of the body to have a shape of the preferred anatomical profile. The system further includes a wearable portion in communication with the user device. The wearable portion includes a second display configured to display a graphical representation of the modified initial three-dimensional representation of the body.

This aspect may further include one or more of the following features: In some implementations, the plurality of reference markers correspond to at least one of an umbilicus, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.

In some embodiments, the multiple reference markers correspond to at least one of the navel, a portion of the sternum, a portion of the hips, a portion of the clavicle, or a portion of the shoulder.

Disclosed implementations may further include one or more of the following features. In some implementations, the method includes selecting a medical procedure associated with a desired body part, replacing the desired body part with a corresponding desired body part taken from the preferred anatomical profile to generate a future anatomical image, and displaying the future anatomical image.

In some embodiments, the medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.

In some embodiments, the future anatomical images are displayed in three dimensions.

Details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

The drawings described herein are for illustrative purposes only of selected configurations, are not intended to illustrate all possible embodiments, and are not intended to limit the scope of the present disclosure.

1 illustrates an example augmented reality system in accordance with the principles of the present disclosure. 1B illustrates one example use of a user device of the augmented reality system of FIG. 1A. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1B illustrates one example use of the augmented reality system of FIG. 1A in accordance with the principles of the present disclosure. 1 is a block diagram illustrating an example augmented reality system according to the principles of the present disclosure. 1 is a block diagram illustrating one example method for displaying enhanced anatomical features in accordance with the principles of the present disclosure. FIG. 1 is a block diagram illustrating one example method for displaying modified internal anatomical features of a patient based on a selected medical procedure. FIG. 2 is a block diagram illustrating one example method for displaying a three-dimensional representation of a patient having a body of a preferred anatomical profile. 1 is a schematic diagram of an example electronic device executing instructions for displaying augmented anatomical features according to the principles of the present disclosure.

Like numbers in the various drawings represent like elements.

Some of the embodiments of the disclosure will be described more fully with reference to the accompanying drawings. Example configurations are provided so that this disclosure will be thorough and the scope of the disclosure will be fully conveyed to those skilled in the art. Specific details, such as examples of specific components, examples of specific devices, and examples of specific methods, are described to provide a thorough understanding of the configurations of the present disclosure. As will be apparent to those skilled in the art, specific details need not be employed and the example configurations may be embodied in many different forms, and the specific details and example configurations should not be considered as limiting the scope of the disclosure.

Exemplary embodiments provide methods, user devices, and systems for displaying augmented anatomical features. An augmented reality device, such as an augmented reality headset or other electronic device (e.g., a phone, tablet computing device, or other computer), may be used to overlay a computer-generated or virtual image onto a view of the real world. In particular, when a target individual, such as a patient, is within the field of view of a health care professional, such as a doctor or nurse, the health care professional can use the augmented reality device to view a virtual image of the anatomical features of the human body overlaid on the target individual. The augmented reality device can project a virtual image onto a display of the augmented reality device such that the virtual image of the anatomical features approximates one or more characteristics (e.g., size, position, shape, etc.) of the actual anatomical features of the target individual. For example, the augmented reality device can project a virtual image onto a display of the augmented reality device such that the virtual image is overlaid and positioned in an approximate appropriate location of the actual anatomical features of the target individual according to the anatomical structure of the target individual. The virtual images can assist health care professionals in more accurately assessing treatment plans or otherwise treating patients by enhancing the health care professional's visualization of the patient's body.

In another aspect, an augmented reality device, such as a tablet or mobile device, captures an image of the patient. The augmented reality device can have a software application configured to identify a plurality of reference markers on the image of the patient and to determine an anatomical profile including a plurality of internal anatomical features of the target individual based on the plurality of reference markers. The software application is further configured to display, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features.

In another aspect, the software application includes a list of medical procedures to choose from. The software application can further access a database populated with a plurality of future-state anatomical profiles corresponding to the selected medical procedures, and a graphical representation of the future-state anatomical profiles is overlaid on an image of the patient, and the internal anatomical features are modified based on the selected medical procedure. As a result, the image capture device displays how the selected medical procedure will affect the patient's internal anatomical features.

In another aspect, the image capture device is configured to map the patient to generate an initial three-dimensional representation of the body. The software application can access a database populated with a plurality of preferred anatomical profiles corresponding to a plurality of reference markers on the initial three-dimensional representation of the patient. The preferred anatomical profile is a profile of a person having a preferred body type. The software application three-dimensionally overlays anatomical features based on the selected procedure with the preferred anatomical profile and displays a graphical representation of the preferred three-dimensional representation of the body overlaid on the initial three-dimensional representation of the body. As a result, the patient can see what they would look like if they had the preferred body type. The body is mapped and three-dimensionally generated so that the patient can view the preferred body type in three dimensions.

1A, an example augmented reality system 10 is shown that includes one or more augmented reality devices 102. As described in more detail below, a user 104 can use the augmented reality device(s) 102 to enhance the user's view of a target individual 106 in a healthcare environment 100. For example, the user 104 may be a doctor, the target individual 106 may be a patient, and the healthcare environment 100 may be a doctor's office where the doctor can examine the patient. In another example, the user 104 may be a paramedic providing emergency treatment to the target individual 106 in the healthcare environment 100 in an ambulance. Although the user 104 is generally depicted and described herein as being a healthcare professional (e.g., a doctor, a nurse, a physical therapist or trainer, a paramedic, a medical assistant, a pharmacist, etc.) and the target individual 106 is generally depicted and described herein as being a healthcare patient, the user 104 or the target individual 106 may include a variety of other persons within the scope of the present disclosure. For example, the individual 106 may be an athlete, a student, or other individual having a body and being the subject of a study or research by another user 104. In this regard, the augmented reality device(s) 102 may be used in a wide variety of situations by a variety of users 104 to examine the target individual 106 in a variety of environments, including the healthcare environment 100.

The augmented reality device 102 may include an image capture device 108 and a display 110. In use, as described in more detail below, the image capture device 108 may capture data about the healthcare environment 100, and in particular about a target individual 106 located within the healthcare environment 100, and the display 110 may display a composite view of the healthcare environment 100 overlaid with a virtual image generated by the augmented reality device 102 for viewing by the user 104. In some implementations, the system 10 includes a first augmented reality device 102a and a second augmented reality device 102b. The first augmented reality device 102a may include a smartphone, tablet computer, or other suitable mobile computing device, and the second augmented reality device 102b may include an augmented reality headset. References herein to the augmented reality device 102 should be understood to apply equally to the first augmented reality device 102a and/or the second augmented reality device 102b.

The first augmented reality device 102a may include an image capture device 108a (e.g., a camera) and a display 110a (e.g., a screen). In use, the image capture device 108a may capture an image of the healthcare environment 100, and in particular an image of the target individual 106. The screen 110a of the first augmented reality device 102a may be used to display a composite view of the healthcare environment 100, i.e., a view overlaid with a virtual image captured by the camera 108a and generated by the first augmented reality device 102a. The first augmented reality device 102a may include a keyboard, a mouse, a microphone, a camera 108a, or a touch screen to allow the user 104 to input data into the first and/or second augmented reality devices 102a, 102b.

The second augmented reality device 102b may include an image capture device 108b (e.g., a camera) and a display 110b (e.g., an eyepiece). In use, the image capture device 108b may capture an image of the healthcare environment, and in particular an image of the target individual 106. The display 110b may be adapted to display a composite view of the healthcare environment 100, i.e., a view captured by the camera 108b and/or the camera 108a and overlaid with a virtual image generated by the second augmented reality device 102b. The second augmented reality device 102b may include a track pad 111, a camera 108b, a microphone, an eye tracking device, or a gesture tracking device to enable the user 104 to input data into the first and/or second augmented reality devices 102a, 102b. For example, the user 104 can input data and otherwise interact with the second augmented reality device 102b by touch via the trackpad 111, by spoken commands via a microphone, by eye gestures via the camera 108b, by positional tracking of a hand or other body part via the camera 108b, by hand gesture tracking via the camera 108b, or by positional tracking of an object such as a pointing stick, stylus, pointer, or glove via the camera 108b.

It should be noted that while the illustrated example depicts the augmented reality device 102 as a first augmented reality device 102a or a second augmented reality device 102b, the augmented reality device 102 may be any device (e.g., augmented reality glasses, an augmented reality helmet, a tablet, etc.) capable of overlaying computer-generated or virtual images onto a view of the real world.

1B, an example augmented reality device is shown. Note that while FIG. 1B depicts the augmented reality device 102 as a first augmented reality device 102 displaying visual data on a screen 110a, a second augmented reality device 102b can similarly display visual data on an eyepiece display 110b. The augmented reality device 102 can detect a target individual 106 by using an image capture device 108. The augmented reality device 102 can then display a visual representation 105 of the body of the target individual 106 on a display 110. In some implementations, the visual representation 105 is a live (e.g., real-time) image of the target individual 106. In other implementations, the visual representation 105 is a still image (e.g., a photograph) of the target individual 106. In some implementations, the image capture device 108 includes an infrared camera using, for example, infrared laser scattering beam technology to create a three-dimensional visual representation 105 of the target individual 106.

The augmented reality device 102 identifies (e.g., assigns) one or more reference markers 112 on the visual representation 105. As described above, in some implementations, the reference markers 112 are identified on the three-dimensional visual representation 105 created using infrared laser scattering beam technology. Each reference marker 112 may correspond to a particular part of the body of the target individual 106 or a particular location on the body. In some implementations, the augmented reality device 102 assigns the reference marker(s) 112 by detecting an input (e.g., a touch, a hand gesture, etc.) from the user 104 that corresponds to one or more particular parts of the body of the target individual 106. In particular, the reference markers 112 can be identified by the user 104's interaction with the augmented reality device 102. For example, in some implementations, the user 104 touches the screen 110a at a location corresponding to each reference marker 112. In another embodiment, the augmented reality device 102b receives input from the user 104 via the camera 108b or the trackpad 111 corresponding to each reference marker 112. For example, the camera 108b can capture the position of the user 104's hand placed at a location corresponding to each reference marker 112.

In some implementations, the augmented reality device 102 recognizes one or more specific parts of the body of the target individual 106 (e.g., facial features) and assigns the reference marker(s) 112 thereto. For example, the image capture device 108 may include an infrared camera, e.g., using infrared laser scatter beam technology, to recognize one or more parts of the body of the target individual 106 (e.g., facial features) and assign the reference marker(s) 112 thereto. In particular, the image capture device 108 may be capable of creating a three-dimensional reference map of the face of the target individual 106 and comparing the three-dimensional reference map to reference data stored in a storage resource of the augmented device 102, such as the storage device 430 (FIG. 4). The augmented reality device 102 may use the infrared camera of the image capture device 108 to identify the reference marker 112 on the face of the target individual 106. The augmented reality device 102 can identify reference markers 112 on the lips, corners of the mouth, tip of the nose, or ears of the target individual 106. For example, the augmented reality device 102 can identify the reference markers 112 based on input from the user 104 (e.g., touch, hand gestures, etc.). As described in more detail below, in some implementations, the augmented reality device 102 uses identification information from the infrared camera in conjunction with the identified reference markers 112 based on the input from the user 104 to transmit data corresponding to the location of the reference markers 112 to a processing module (e.g., processor 410 of FIG. 4), advantageously enabling the augmented reality device 102 to provide a more personalized and unique estimate of the location of various anatomical features on the body (e.g., face) of the target individual 106, including underlying blood vessels, nerves, and muscles.

In some implementations, the augmented reality device 102 identifies and assigns the reference marker(s) 112 by using a machine learning or artificial intelligence algorithm that identifies a particular part of the body of the target individual 106. The augmented reality device 102 may be adapted to assign the location of the reference marker 112 on the target individual 106 based on the locations of similar reference markers 112 on one or more other target individuals 106. The augmented reality device 102 may be adapted to identify the target individual 106 as a human body by using a machine learning or artificial intelligence algorithm to detect a silhouette of the target individual 106, recognize a body part (e.g., a limb, a crotch, an armpit, or a neck) of the detected silhouette, and then determine the location of the reference marker 112 based on the recognized body part and assign the reference marker 112.

In the illustrated example, the first reference marker 112a corresponds to the navel of the target individual 106. The second reference marker 112b corresponds to a portion of the right shoulder of the target individual 106. The third reference marker 112c corresponds to a portion of the left shoulder of the target individual 106. The fourth reference marker 112d corresponds to a portion of the clavicle of the target individual 106. The fifth reference marker 112e corresponds to a portion of the left hip of the target individual 106. The sixth reference marker 112f corresponds to a portion of the right hip of the target individual 106. The seventh reference marker 112g corresponds to a portion of the sternum of the target individual 106. The reference markers 112a-112g do not represent an exhaustive list of all reference markers, but rather an exemplary list of reference markers that may be identified by the augmented reality device 102. Additionally, the augmented reality device 102 may identify other reference markers, such as reference markers corresponding to the neck, nose, eyes, mouth, knees, ankles, hip joints, shoulder blades, wrists, or elbows, in addition to the reference markers 112a-112g, without departing from the teachings herein. The augmented reality device 102 may also omit one or more of the reference markers 112a-112g without departing from the teachings herein.

The augmented reality device 102 can determine an anatomical profile of the target individual 106. The anatomical profile may include a number of characteristics corresponding to the individual 106. In some implementations, the anatomical profile includes or is based on a number of target data, such as the age or gender of the target individual 106. In some implementations, the augmented reality device 102 determines the anatomical profile based on input from the user 104 (e.g., touch, hand gestures, etc.). In other implementations, the augmented reality device 102 uses machine learning or artificial intelligence algorithms to determine the anatomical profile. For example, the augmented reality device 102 may be adapted to determine the anatomical profile based on a number of target data (e.g., a number of reference markers 112) received by the augmented reality device 102.

1C, an exemplary augmented reality device is shown. The augmented reality device 102a, 102b can display a graphical representation 113 of the target individual 106. In some implementations, the augmented reality device 102 can display the graphical representation 113 even when the target individual 106 is fully clothed. This is advantageous because it saves time since the target individual 106 does not need to remove his or her clothing. The graphical representation 113 of the target individual may include one or more virtual images 114a-114c of anatomical features. For example, the virtual images 114a-114c may be virtual images of internal anatomical features (e.g., bones or other organs) or external anatomical features (e.g., skin or its contours). In some implementations, the augmented reality device 102 uses three-dimensional data corresponding to the target individual 106 to generate the graphical representation 113. In some implementations, the graphical representation 113 includes a virtual image 114 of detailed underlying anatomical structures of the face of the target individual 106.

The method of displaying the graphical representation 113 may include various steps. For example, the method of displaying the graphical representation 113 may include identifying one or more reference markers 112 (FIG. 1B) using the augmented reality device 102 (e.g., 102a, 102b). The augmented reality device 102 may further determine a distance from each reference marker 112 to each of the other reference markers 112 and transmit data corresponding to the distances to a processor (e.g., processor 410, FIG. 4). The augmented reality device 102 may further perform a lookup in a database (e.g., database 218, FIG. 2) having data corresponding to the reference markers 112 (e.g., memory 420, FIG. 4). The augmented reality device 102 uses the data corresponding to the reference markers 112 and, in some implementations, a plurality of target data to determine data corresponding to characteristics (e.g., size, location, etc.) of the anatomical feature. For example, the augmented reality device 102 can use the reference markers 112 and the three-dimensional visual representation 105 of the target individual 106 created from infrared laser scattering beam technology of the image capture device 108 to create a graphical representation 113 including virtual images 114a-114c of anatomical features. In particular, the augmented reality device 102 can transmit data corresponding to the characteristics of the anatomical features to a processor (e.g., processor 410 of FIG. 4) and display the graphical representation 113 including the virtual images 114a-114c of anatomical features on the display 110 at a location corresponding to the target individual 106 (see FIG. 1C).

As depicted in FIG. 1C, the second augmented reality device 102b can display a graphical representation 113 on the eyepiece display 110b, and the first augmented reality device 102a can display a graphical representation 113 on the screen 110a. As previously described, the graphical representation 113 displayed on the screen 110 and overlaid on the visual image 105 of the target individual 106 may include computer-generated virtual images (e.g., virtual images 114a-114c of anatomical features). In the illustrated embodiment, the augmented reality device 102 displays on the display 110 a graphical representation 113 that includes virtual images 114a-114c, each of which represents an organ, bone, or structure within the human body.

The first virtual image 114a represents the rib cage of a human body. The virtual image 114a is overlaid on the visual representation 105 of the target individual 106 at a location that approximates where the rib cage of the target individual 106 would be located. The second virtual image 114b represents the intestines of a human body. The virtual image 114b is overlaid on the visual representation 105 of the target individual 106 at a location that approximates where the intestines of the target individual 106 would be located. The third virtual image 114c represents the pancreas of a human body. The virtual image 114c is overlaid on the visual representation 105 of the target individual 106 at a location that approximates where the pancreas of the target individual 106 would be located. The virtual images 114a-114c do not represent an exhaustive list of all virtual images, but rather an illustrative list of virtual images that may be displayed in the graphical representation 113 by the augmented reality device 102. Additionally, the augmented reality device 102 may display other virtual images in addition to the virtual images 114a-114c, or may omit one or more of the virtual images 114a-114c, without departing from the teachings herein.

The graphical representation 113 combined with the visual representation 105 enhances the view and experience of the user 104 by creating a composite view of both real and virtual images on the display 110. Through the display 110, the user 104 can view the visual representation 105 of the target individual 106 in conjunction with virtual images 114a-114c of organs and other anatomical features represented by the graphical representation 113 of the target individual 106. The anatomical features may include organs, bones, muscles, blood vessels, tendons, ligaments, or nerves. In some implementations, the virtual images 114a-114c are not actual images of the internal organs, bones, or other body structures of the target individual 106, but rather are representative depictions (e.g., illustrations) of those body structures. Advantageously, allowing the augmented reality device 102 to store representative virtual images 114a-114c that can be used for any target individual 106 requires less space (e.g., memory 420) to store the virtual images 114a-114c, allowing the augmented reality device 102 to operate faster and display the virtual images 114a-114c.

In some real terrains, the augmented reality device 102 can use the image capture device 108 to take a still image or record a video of the target individual 106. The augmented reality device 102 may then overlay a virtual image 114 of anatomical features onto the still image or video of the target individual 106. The augmented reality device may then display the still image or video on the display 110 along with the overlaid virtual image 114. This is particularly convenient for educational purposes, as it can help a health care professional to teach a patient about their own anatomy by showing the patient approximately how their own individual anatomy functions. In other implementations, the augmented reality device 102 may be able to take or record a picture or video of the target individual 106 along with a visual representation 105 that includes the virtual images 114 of organs and other anatomical features. The augmented reality device 102 may store the picture or video in a storage resource, such as a storage device 430 (FIG. 4).

1D-1F, in some implementations, the augmented reality device 102 (e.g., 102a, 102b) can display an anatomical profile of a target (e.g., target individual 106) based on certain expected changes to the target individual 106. FIG. 1D shows a visual representation 105' of an alternative target individual 106' displayed on a display 110. The augmented reality device 102 can determine the visual representation 105' of the alternative target individual 106' using an image capture device 108. As depicted in FIG. 1E, similar to the visual representation 105 of FIG. 1B, the augmented reality device 102 can be adapted to identify a number of reference markers 112 on the visual representation 105'. In some implementations, the augmented reality device 102 can be adapted to determine an anatomical profile of the alternative target individual 106'. The anatomical profile can include a number of characteristics corresponding to the alternative target individual 106'. As described above with reference to FIG. 1B, the anatomical profile may be based on a plurality of goal data, input from the user 104, or machine learning or artificial intelligence algorithms. In some implementations, the anatomical profile may also be based on certain changes that the alternate target individual 106' may undergo. For example, the anatomical profile may be based on future weight loss or weight gain that the alternate target individual 106' will undergo or desires to undergo.

1E and 1F show an augmented reality device 102 displaying a visual representation 105' of an alternative target individual 106' on a display 110. The augmented reality device 102 displays a graphical representation 113 of the alternative target individual 106', the graphical representation 113 including a virtual image 114 of anatomical features. In the example shown in FIG. 1F, the augmented reality device 102 displays a visual representation 105' of the alternative target individual 106' if the alternative target individual 106' had lost weight (FIG. 1F). The augmented reality device further overlays the graphical representation 113 including the virtual image 114 of anatomical features. The virtual image 114 of anatomical features can be based on the anatomical features of the alternative target individual 106' if the alternative target individual 106' had lost weight. Also, the visual representation 105' of the alternative target individual 106' (FIG. 1F) can be based on the alternative target individual 106' having lost weight.

1F depicts the augmented reality device 102 displaying a virtual image 114 of anatomical features based on the alternate target individual 106' losing weight, it should be noted that the augmented reality device may be adapted to display virtual images 114 based on other specific changes, such as gaining weight, becoming pregnant, undergoing reconstructive or cosmetic surgery, or other changes the individual's body may undergo. The augmented reality device 102 may also be adapted to display other virtual images 114 that show how musculoskeletal features would function under certain changes.

1E and 1F, another aspect of the augmented reality device 102 is provided. The augmented reality device 102 is configured to detect a target individual 106 by using an image capture device 108 and to display a visual representation 105 of the body of the target individual 106. The augmented reality device 102 is configured to identify one or more reference markers 112 as described above with reference to FIG. 1B. The reference markers 112 are processed to identify an anatomical profile including a plurality of internal anatomical features 114. The augmented reality device 102 is further configured to access a database 218. The database 218 may be stored in the augmented reality device 102 or may be stored in a remote server, such as a server located in the network 400 or a server accessible through the network 400.

The augmented reality device 102 is configured to display a graphical representation of internal anatomical features on the display 110, as shown in FIG. 1G, over the visual representation of the body. Thus, a user, such as a health care professional, can identify the location of the internal anatomical features without performing a conventional procedure, such as an X-ray, MRI, or other medical scanning operation. Such a display is useful for assisting the health care professional in identifying, within a reasonable degree of accuracy, a medical condition, a location for an incision, a specific portion of an organ, a muscle or tendon, or any other internal anatomical structure that is not visible from the outside. As an example, the augmented reality device 102 captures an image of the patient body 106 and displays a visual representation 105 of the body. The augmented reality device 102 identifies the reference markers 112a-112g and searches a database 218 to find an anatomical profile 216 that corresponds to the spatial orientation and relationship of the identified reference markers 112a-112g. The graphical representation of the internal anatomical features is overlaid onto the visual representation 105 of the body.

A patient may be experiencing abdominal pain and may point to the location of the pain on the body. A health care professional may then be able to more clearly identify the problem by seeing where the pain is in relation to the location of internal anatomical features (displayed in the graphical representation 105), such as by identifying that the pain is in the duodenum as opposed to the pancreas, pancreatic duct, etc. Thus, the augmented reality device 102 assists the surgeon in treatment, diagnosis, and patient education without subjecting the patient to an X-ray, MRI, or other internal invasive scanning procedure.

1G, another aspect of the augmented reality device 102 is provided. The database 218 includes a plurality of future state anatomical profiles 216' corresponding to a selected medical procedure 226. It is to be understood that the database 218 may include thousands of future state anatomical profiles 216'. The augmented reality device 102 is further configured to receive the selected medical procedure 226. For example, the augmented reality device 102 may be configured to display a menu 118 providing a selection of a plurality of medical procedures 216, such as procedures affecting an organ, bone, or muscle. Such medical procedures include liposuction, gastrointestinal bypass, rhinoplasty, etc.

In operation, the augmented reality device 102 processes the reference markers 112 to determine which anatomical profiles 216 stored in the database 218 match the identified reference markers 112. The user then selects a medical procedure 226 and retrieves the future state anatomical profile 216' corresponding to the selected medical procedure 225. The augmented reality device 102 then modifies the internal anatomical features 114 based on the selected medical procedure 226. As an example, FIG. 1B depicts the augmented reality device 102 identifying multiple reference markers 112 on the target individual 106.

1G, the augmented reality device 102 searches the database 218 to find an anatomical profile corresponding to the reference markers 112. For example, the augmented reality device 102 can use the distance between the reference markers 112 to find an anatomical profile having an equivalent distance between the reference markers. For example, if the patient is a 5'10" male with left and right shoulders 112c and 112b spaced 18" apart from each other and left and right hips 112f and 112e spaced 19" apart from each other, the augmented reality device 102 will search the database 218 to find an anatomical profile for a 5'10" male with equally spaced reference markers 112. It should be understood that more reference markers may be used to determine corresponding anatomical profiles other than just the left and right shoulders 112c and 112b and the left and right hips 112f and 112e. The internal anatomical features are then displayed on the graphical representation as shown in FIG.

In another aspect, the augmented reality device 102 may be further configured to scale the internal anatomical features of the selected anatomical profile based on the size of the target individual 105 relative to the selected anatomical profile. The augmented reality device 102 may be adapted to determine whether the target individual is larger or smaller than the selected anatomical profile. The augmented reality device 102 may be further configured to scale the internal anatomical features associated with the selected anatomical profile to fit the image and the reference markers 112. As an example, for a target individual 105 that is larger than the selected anatomical profile, the internal anatomical features of the selected anatomical profile are scaled up. Such an aspect may be beneficial when the database 218 is populated with a discrete number of anatomical profiles. Thus, the augmented reality device 102 is configured to display the internal anatomical features on the visual representation 105 of the target individual 106 using fewer processing resources compared to an aspect where the database 218 is populated with hundreds of anatomical profiles.

In one example, the database 218 may be populated with only eight anatomical profiles each for infant, child, adult, and geriatric male and female versions, for a total of 64 anatomical profiles. In such an embodiment, the augmented reality device 102 processes the 64 anatomical profiles to select the anatomical profile that most closely matches the spatial dimensions of the reference marker 112. For example, the target individual 106 may be a 5'11" male adult. The database 218 may be populated with a 5'8" male adult and another 6'2" male adult. The augmented reality device 102 selects the 6'2" male adult anatomical profile if the 6'2" male adult anatomical profile more closely matches the reference marker 112 than the 5'8" male adult anatomical profile. The augmented reality device 102 scales the internal anatomical features of the selected anatomical profile to fit within the visual representation 105 of the body of the target individual 106. Using the same example, the augmented reality device 102 would shrink the internal anatomical features of the selected anatomical profile to fit within the smaller visual representation 105 of the target individual 106.

It should be understood that not all of the internal anatomical features of the selected anatomical profile need to be scaled. For example, the augmented reality device 102 may be further configured to determine which parts of the target individual's body are larger or smaller than the selected anatomical profile, in which case the internal anatomical features of the body parts that are larger or smaller than the selected anatomical profile are scaled. As an example, in the case where the target individual 106 has a chest cavity that is larger than the selected anatomical profile, the augmented reality device 102 may zoom in on the lungs of the selected anatomical profile. The zooming is done to overlay the lungs in proportion to the visual representation 105 of the target individual 106.

In yet another aspect, the augmented reality device 102 may be further configured to process the body type along with the reference marker 112. For example, it may be assumed that the target individual 106 shown in FIG. 1A and FIG. 1B is considered to be a normal body type, the target individual 106 shown in FIG. 1D and FIG. 1E is considered to be an athlete body type, and the target individual 106 shown in FIG. 1I and FIG. 1K is considered to be an overweight body type. The augmented reality device 102 processes the body type using the silhouette of the target individual 106 and the selected anatomical profile to provide an overlay of other anatomical features such as fat layers, muscle thickness, etc. The augmented reality device 102 may be further configured to process the silhouette of the target individual and the selected anatomical profile to overlay other features of the internal anatomical structure of the individual target 106, such as blood vessels, nerves, and organs corresponding to the selected anatomical profile. It should be understood that such features provide additional details to assist health care professionals in providing care, education, performing diagnoses, etc.

For illustrative purposes, an operation of an augmented reality device that processes a body type and silhouette in conjunction with a selected anatomical profile is provided. The augmented reality device 102 detects a target individual 106 by using an image capture device 108 and processes a visual representation 105 of the target individual to determine a silhouette. The silhouette can be determined by implementing any now known or later developed image processing technique for edge detection, now known or later developed.

In one aspect, the augmented reality device 102 may process the silhouette to determine the body type. The body type may be determined by processing the spatial relationship between the reference markers or opposing edges of the silhouette. As an example, the augmented reality device 102 may determine that the target individual has an overweight body type by determining that the distance between opposing edges around the waist exceeds the distance between opposing edges of each shoulder in images taken from the same viewpoint. As another example, the augmented reality device may determine that the target individual 106 has an athlete body type by determining that the distance between opposing edges around the waist is less than the distance between opposing edges of each shoulder in images taken from the same viewpoint. Alternatively, the body type may simply be entered using a drop-down menu, a keyboard, or any other input device now known or later developed.

If the augmented reality device 102 determines through image processing or input that the body type is overweight, the augmented reality device 102 may scale the internal anatomical features to be proportionate to the body type. For example, the fat layer may be made thicker relative to the fat layer of a standard body type, in which case the internal anatomical features of the selected anatomical profile are scaled to fit behind the thickened fat layer. The thickening of the fat layer may be proportionate to the silhouette and further refined by processing the reference markers 112. In the case where the body type is an athlete, the fat layer may be made thinner relative to the fat layer of a standard body type and the muscles may be made larger relative to the standard body type. It is understood that the examples provided herein are illustrative and not limiting, and other internal anatomical features may be adjusted based on the determined body type.

Although the description of the operation of the augmented reality device 102 according to the above aspect has been provided in a broad manner, it should be understood that the augmented reality device 102 may be operated in a granular manner specific to a particular body part. For example, the adjustment of the fat layer may be only for parts of the body that are deemed to deviate from the standard body type. For example, the augmented reality device 102 may determine, based on the silhouette, that only the waist is overweight and the rest of the body is standard. In such an aspect, the augmented reality device 102 may be configured to thicken the fat layer only around the waist and display the fat layer of the remaining body parts as standard. It should be further understood that the adjustment of the anatomical features based on the body type may be performed automatically or based on input from a medical professional. That is, the augmented reality device 102 may be configured to operate without processing the silhouette. As a result, the graphical representation of the internal anatomical features is overlaid on the visual representation of the body without any adjustment to the internal anatomical features such as fat layers, muscles, etc. The health care professional can then select an option or otherwise indicate that they would like to modify the graphical representation with a silhouette of the determined body type.

1H, the graphical representation 113 is modified relative to FIG. 1G. In particular, the augmented reality device 102 is configured to receive a selected medical procedure 226. For illustrative purposes, the medical procedure 226 is selected via a drop-down menu 118 (shown in FIG. 1G). However, other means for inputting the selected medical procedure 226 may be adapted for use herein, including, by way of example, a keyboard, voice input, and the like. For illustrative purposes, the selected medical procedure 226 is a gastric bypass. The augmented reality device 102 selects a future state anatomical profile 216' corresponding to the selected medical procedure 226. In this instance, the augmented reality device 102 modifies the internal anatomical feature 114 (i.e., the stomach) by overlaying the future state anatomical profile 216' determined by the augmented reality device 102 on the stomach shown in FIG. 1G, as shown in FIG. 1H. Thus, in this aspect, the patient is able to see the impact of the surgical procedure on the patient's internal anatomical features. In particular, patients will be able to see what their internal anatomical features look like before the medical procedure, shown in Figure 1G, and after the medical procedure, shown in Figure 1H.

In another aspect of the augmented reality device 102, the database 218 further includes a plurality of diseases, genetic conditions, and/or anatomical variations. As an example, the database may store different types of diseases, such as cancer, heart disease, diabetes, etc.; genetic conditions, such as cystic fibrosis, Down's syndrome, etc.; anatomical variations, which may include anatomical variations of anatomical features, such as the gallbladder, kidney, liver, etc. The augmented reality device 102 is further configured to determine an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation, and to generate a graphical representation, which is a visual representation of the body modified with the internal anatomical features based on the determined anatomical profile. As such, the augmented reality device 102 may be useful in showing the patient and/or surgeon the effect of the disease or genetic condition on the internal anatomical features.

In another aspect, the database 218 includes anatomical profiles having at least one of a disease, a genetic condition, and/or an anatomical variant. In such an aspect, the augmented reality device 218 simply selects an anatomical profile corresponding to the identified reference marker 112a-112g, possibly having the identified disease, genetic condition, and/or anatomical variant, and displays the internal anatomical features associated with the selected anatomical profile. In such an embodiment, it is preferred that the patient identifies the disease, genetic condition, or anatomical variant, possibly. This is done by an input field, which may be entered using conventional input means such as a keyboard, voice input, mouse, etc.

As an example, a surgeon or nurse inputs a disease into the augmented reality device 218. The augmented reality device 218 detects the target individual and displays a visual representation of the body. The augmented reality device 218 processes the reference markers 112a-112g to identify an anatomical profile that has the same spatial dimensions as the reference markers and the disease input into the augmented reality device 218. The internal anatomical features of the selected anatomical profile are mapped onto the visual representation of the body.

1I, another aspect of the augmented reality device 102 is provided. FIG. 1I depicts an augmented reality device 102 mapping a target object 106. The augmented reality device 102 includes an image capture device 108 configured to generate an initial three-dimensional representation 113' of a body. For example, the image capture device 108 may be a LIDAR camera or an infrared laser scatter beam device. FIG. 1I depicts the initial three-dimensional representation 113' as a still image taken from the front of the target object 106 and from a side of the target object. However, it should be understood that the initial three-dimensional representation 113' depicts the entire target object 106 and thus may be generated by taking multiple still images or videos. In one aspect, the target object 106 may stand on a platform 120 that rotates 360 degrees as indicated by the arrow. The augmented reality device 102 is placed at a predetermined distance from the target object 106, and the image capture device 108 is activated to map the target object. FIG. 1I depicts images taken while rotating the target object 106. It should be understood that the initial three-dimensional representation 113' of the target object 106 may be generated in other ways, for example, with the target object 106 standing stationary and the augmented reality devices 102 rotating around the target object 106. Alternatively, multiple augmented reality devices 102 may be positioned at different locations around the target object 106, and the captured images from each of the augmented reality devices 102 may be compiled together and processed to generate the initial three-dimensional representation 113'.

The augmented reality device 102 identifies one or more reference markers 112 as described above with reference to FIG. 1B. It should be understood that the reference markers 112 may be identified in real time or after the three-dimensional representation of the body is generated. The augmented reality device 102 is further configured to determine a preferred anatomical profile 216'' based on the plurality of reference markers 112. The preferred anatomical profile 216'' may be a profile representative of a fit and healthy person based on anatomical features extrapolated from the reference markers 112. The anatomical features include not only the reference markers 112 but also aspects such as height and body shape of the target individual 106.

Now referring to FIG. 1J, the augmented reality device 102 is configured to three-dimensionally modify the initial three-dimensional representation 113' of the body with the preferred anatomical profile 216'' to generate a modified initial three-dimensional representation 228. The modified initial three-dimensional representation 228 is the initial three-dimensional representation 113' of the body that is shaped to match the shape of the preferred anatomical profile 216''. The augmented reality device 102 displays a graphical representation of the modified initial three-dimensional representation 228 of the body.

The augmented reality device 102 may further be configured to receive a selected medical procedure 226 related to a desired body part, where the desired body part of the initial three-dimensional representation 113' of the body is replaced with a corresponding body part taken from the preferred anatomical profile 216''. The augmented reality device 102 displays the initial three-dimensional representation 113' of the body with the corresponding desired body part taken from the preferred anatomical profile 216''.

In operation, the augmented reality device 102 processes the reference markers 112 to determine which preferred anatomical profile 216" stored in the database 218 matches the initial three-dimensional representation 113' of the body. The augmented reality device 102 then processes the initial three-dimensional representation 113' of the body to adjust the contours of the initial three-dimensional representation 113' of the body to match the contours of the displayed preferred anatomical profile 216". In this way, the patient can see how he or she would actually look if their body was adjusted to the preferred anatomical profile. The resulting image is not simply a preferred profile overlaid on the patient, but rather a modified, stretched, shrunk, etc., body shape of the patient. Thus, since the patient's skin will be modified to exhibit the preferred anatomical profile 216", it is preferable for the patient to have as much skin as possible exposed when the initial three-dimensional representation 113' of the body is generated.

1K, the augmented reality device 102 searches the database 218 to find a preferred anatomical profile 216'' based on the multiple reference markers 112. As described above, this may be done by determining the distance between the reference markers. It should be understood that the database may contain hundreds of preferred anatomical profiles 216'' representative of a 5'10" male. For example, one preferred profile is a 5'10" male with left and right shoulders 112c and 112b spaced 18" apart from each other and left and right hips 112f and 112e spaced 19" apart from each other. Another preferred profile is a 5'10" male with left and right shoulders 112c and 112b spaced 17" apart and left and right hips 112f and 112e spaced 17" apart. As described above, the preferred anatomical profile 216" is a three-dimensional representation of a fit and healthy person. The augmented reality device 102 processes the initial three-dimensional body representation 113' to adjust the contours of the initial three-dimensional body representation 113' to match the contours of the preferred anatomical profile to generate a modified initial three-dimensional representation 228.

1J illustrates the patient shown in FIG. 1I after the initial three-dimensional representation 113' of the body has been adjusted to the contours of the preferred anatomical profile 216''. The modified initial three-dimensional representation 228 of the body may be displayed on the augmented reality device 102 or may be projected onto another display, preferably a larger one, so that the patient can view the modified initial three-dimensional representation 228. The modified initial three-dimensional representation 228 may be rotated so that the patient can fully evaluate the changes to his or her body. Furthermore, by having the initial three-dimensional representation 113' of the body adjusted, the user can evaluate the changes to his or her skin. That is, the initial three-dimensional image 113' is adjusted and not replaced by the preferred anatomical profile 216''.

FIG. 1K illustrates an embodiment in which only the selected body part is adjusted. In one aspect, the augmented reality device 102 may be configured to process the selected medical procedure 226. As mentioned above, the selected medical procedure 226 may be entered through the drop-down menu 118 or a keyboard, voice input, etc. For illustrative purposes, the selected medical procedure 226 is shown as liposuction of abdominal fat. The display 110 then depicts the initial three-dimensional representation 113' modified with the abdomen of the preferred anatomical profile 216''. In another aspect, the user may simply select the body part to be replaced by the corresponding body part of the preferred anatomical profile 216'' of the body. The user may select the body part by touching the body part displayed on the touch screen display 110, through voice commands, or via other known input devices such as a mouse or stylus. Thus, in this aspect, the patient can see what their actual body would look like if the body was modified to have the shape and contours of the preferred anatomical profile. In other words, the modified initial three-dimensional representation 228 is a modified version of the selected body parts of the initial three-dimensional representation 113' with the corresponding body parts of the preferred anatomical profile 216". Thus, FIG. 1K illustrates that the patient's arms, neck, legs, etc. remain the same as shown in FIG. 1I, and only the abdomen has been modified to match the abdomen of the preferred anatomical profile 216". In this way, the patient can see how they will look after the selected medical procedure has been performed, which can help the patient decide whether to have other procedures performed.

2, in some implementations, an exemplary system 200 for displaying augmented anatomical features (e.g., virtual images 114a-114c) provides a user 104 (e.g., a health care provider) with access to an augmented reality module 204 that enhances the user's 104 view of a target individual 106. The system 200 may include a network 208 that provides access to the augmented reality module 204 that provides for the performance of services on a remote device. Thus, the network 208 enables interaction between the user 104 and the augmented reality module 204. For example, the augmented reality module 204 provides the user 104 with access to the augmented reality module 204 and receives event data entered by the user 104 related to the user's 104 interaction with the augmented reality module 204. The augmented reality module 204 may then store the event data in a storage resource.

The network 208 may include any type of network that allows for the transmission and reception of communication signals, such as a wireless telecommunications network, a cellular phone network, a time division multiple access (TDMA) network, a code division multiple access (CDMA) network, a Global System for Mobile Communications (GSM), a third generation (3G) network, a fourth generation (4G) network, a satellite communication network, and other communication networks. The network 208 may include one or more of a wide area network (WAN), a local area network (LAN), and a personal area network (PAN). In some embodiments, the network 208 includes a combination of data networks, a combination of telecommunications networks, or a combination of data networks and telecommunications networks. The augmented reality device 102 and the augmented reality module 204 communicate with each other by transmitting and receiving signals (wired or wireless) over the network 208. In some embodiments, the network 208 provides access to cloud computing resources, which may be elastic/on-demand computing resources and/or storage resources available on the network 208. The term "cloud" services generally refers to services that are not performed locally on a user's device (e.g., device 102) but rather are delivered from one or more remote devices accessible via one or more networks 208.

The augmented reality device 102 may include, but is not limited to, a portable electronic device (smartphone, cellular phone, personal digital assistant, personal computer, or wireless tablet device), a wearable augmented reality device, or any other electronic device capable of capturing images and overlaying computer-generated or virtual images (e.g., virtual images 114a-114c) onto a real-world view (e.g., visual representation 105). The augmented reality device 102 includes data processing hardware (a computing device that executes instructions), memory hardware, and a display 110 in communication with the data processing hardware. An input module 214 provides the user 104 with access to interact with the augmented reality module 204 through the augmented reality device 102. In some examples, the input module 214 includes a keyboard, a touchpad, a mouse, a microphone, an eye tracking device, a gesture tracking device, and/or a camera to allow the user 104 to input data. In addition to or instead of the display 110, the augmented reality device 102 may include one or more speakers that output audio data to the user 104.

In some implementations, the user 104 can interact with the input module 214 by inputting data corresponding to the reference markers 224. The reference markers 224 may correspond to locations on the target individual 106. The data corresponding to the reference markers 224 is then transmitted to the augmented reality module 204. The augmented reality module 204 can communicate with the anatomy module 216. For example, the augmented reality module 204 may transmit anatomical data 222 corresponding to the reference markers 224 to the anatomy module 216. The augmented reality module 204 may then request data corresponding to the visual representation of the anatomical features (e.g., the virtual images 114a-114c) from the anatomy module 216. The anatomy module 216 may then retrieve data corresponding to the visual representation of the anatomical features, the future state anatomical profile, or the preferred anatomical profile from the anatomical database 218. The anatomy module 216 may then generate an anatomical profile 220 to be displayed on the display 110.

The augmented reality device 102 may include an image capture device 108 having a detection module 210 and a tracking module 212. The detection module 210 and the tracking module 212 may obtain visual data corresponding to the target individual 106 and transmit it to the augmented reality module 204. The visual data is data corresponding to the current real-world view of the image capture device 108 and may include data such as the distance between the target individual 106 and the augmented reality device 102, whether the target individual 106 is moving or stationary, or any other data corresponding to the visualization of the target individual 106 by the augmented reality device 102. The image capture device 108 may transmit data from the detection module 210 and the tracking module 212 to the augmented reality module 204. The augmented reality module 204 may use the data corresponding to the real-world view of the image capture device 108 from the detection module 210 and the tracking module 212 in conjunction with the virtual image of the anatomical profile 216 to create a composite enhanced view of the target individual 106 on the display 110.

FIG. 3A is a flow chart illustrating a method 300 for displaying an augmented anatomical feature (e.g., virtual images 114a-114c) according to an example implementation of the disclosed technology. According to one embodiment, the method 300 can be performed by an electronic device such as an augmented reality device 102. The method 300 begins at block 302, where the augmented reality device detects a target individual (e.g., target individual 106) and tracks the target individual. The target individual has a body. For example, at block 302, the image capture devices 108a, 108b can detect and/or track the target individual 106 via the detection module 210 and/or the tracking module 212.

The method includes displaying a visual representation of the body on a display (e.g., display 110) at block 304. For example, at block 302, displays 110a, 110b may receive and display visual representations 105, 105' from augmented reality module 204. In some implementations, after block 304, the method may return to block 302. In other implementations, the method may proceed to block 306.

The method includes, at block 306, identifying a plurality of reference markers (e.g., reference markers 112) on the visual representation of the body (e.g., visual representations 105, 105'). For example, the augmented reality device 102 can receive the reference markers 112 from a user's interaction with the displays 110a, 110b, from machine learning, or from another method described above. In some implementations, the augmented reality device 102 can receive input of the reference markers 112 from the user 104 and receive and generate the graphical representation 113 from the augmented reality module 204.

At block 308, the method includes determining, at a processor, an anatomical profile (e.g., anatomical profile 216) of the target individual based on the plurality of reference markers. The anatomical profile may include a plurality of characteristics (e.g., age, gender, etc.) corresponding to the target individual. In some implementations, the augmented reality device 102 receives the anatomical profile based on the reference markers 112. For example, the anatomy module 216 may transmit the anatomical profile to the augmented reality device 102 through the augmented reality module 204.

The method includes, at block 310, displaying on the display a graphical representation of the anatomical features (e.g., virtual images 114a-114c) overlaid on the visual representation of the body 105, 105'. The graphical representation of the anatomical features may be positioned on the visual representation of the body based on the anatomical profile. In some implementations, the augmented reality device generates and displays a graphical representation of the anatomical profile overlaid on the visual representation of the target individual. In one aspect, the anatomical profile includes internal anatomical features, and a graphical representation of the internal anatomical features of the anatomical profile is overlaid on the visual representation of the target individual. Following block 310, the method 300 may end.

FIG. 3B is a flow chart illustrating a method 300B for displaying modified internal anatomical features of a patient based on a selected medical procedure. According to one embodiment, the method 300B can be performed by an electronic device such as an augmented reality device 102. The method 300B begins at block 302B, where the augmented reality device detects a target individual (e.g., the target individual 106). The target individual has a body. For example, at block 302, the image capture devices 108a, 108b can detect the target individual 106 via the detection module 210 and/or the tracking module 212.

The method includes displaying a visual representation of the body on a display (e.g., display 110) at block 304B. For example, at block 302, displays 110a, 110b may receive and display visual representations 105, 105' from augmented reality module 204. In some implementations, after block 304, the method may return to block 302. In other implementations, the method may proceed to block 306B.

The method includes, at block 306B, identifying a plurality of reference markers (e.g., reference markers 112) on the visual representation of the body (e.g., visual representations 105, 105'). For example, the augmented reality device 102 can receive the reference markers 112 from a user's interaction with the displays 110a, 110b, from machine learning, or from another method described above. In some implementations, the augmented reality device 102 can receive input of the reference markers 112 from the user 104 and receive and generate the graphical representation 113 from the augmented reality module 204.

At block 308B, the method includes determining, at a processor, an anatomical profile (e.g., anatomical profile 216) of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features. The anatomical profile may include a plurality of characteristics (e.g., age, gender, etc.) corresponding to the target individual. In some implementations, the augmented reality device 102 receives the anatomical profile based on the reference markers 112. For example, the anatomy module 216 may transmit the anatomical profile to the augmented reality device 102 through the augmented reality module 204.

The method includes, at block 310B, selecting a medical procedure. The medical procedure may be selected by choosing from a drop-down menu or may be entered verbally into the augmented reality device 102 using a keyboard, voice commands, etc. The medical procedure is a procedure that affects an organ, bone, or muscle. Such medical procedures include liposuction, gastrointestinal bypass, rhinoplasty, etc.

The method includes, at block 312B, determining a future-state anatomical profile corresponding to the selected medical procedure. For example, the future-state anatomical profile may be one of thousands of future-state profiles stored in database 218 that correspond to the anatomical profile.

The method includes, at block 314B, displaying on the display a graphical representation (e.g., virtual images 114a-114c) of the visual representation of the body modified with internal anatomical characteristics based on the selected medical procedure.

FIG. 3C is a flow chart illustrating a method 300C for displaying a visual three-dimensional representation of a patient having a body with a preferred anatomical profile. According to one embodiment, the method 300C can be performed by an electronic device such as an augmented reality device 102. The method 300C begins at block 302C, where the augmented reality device maps a target individual (e.g., the target individual 106) to generate an initial three-dimensional representation of the body. The target individual has a body. For example, at block 302, the image capture devices 108a, 108b can detect the target individual 106 via the detection module 210 and/or the tracking module 212.

The method includes, at block 304C, identifying a plurality of reference markers (e.g., reference markers 112) on the visual representation of the body (e.g., visual representations 105, 105'). For example, the augmented reality device 102 can receive the reference markers 112 from a user's interaction with the displays 110a, 110b, from machine learning, or from another method described above. In some implementations, the augmented reality device 102 can receive input of the reference markers 112 from the user 104 and receive and generate the graphical representation 113 from the augmented reality module 204.

At block 306C, the method includes determining, at a processor, a preferred anatomical profile (e.g., anatomical profile 216) for the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of a body that is believed to be that of a fit and healthy person. The anatomical profile may include a plurality of characteristics (e.g., age, gender, etc.) corresponding to the target individual. In some implementations, the augmented reality device 102 receives the preferred anatomical profile based on the reference markers 112. For example, the anatomy module 216 may transmit the preferred anatomical profile to the augmented reality device 102 through the augmented reality module 204.

The method includes three-dimensionally modifying the initial three-dimensional representation 113' of the body to have the shape of the preferred anatomical profile at block 308C. The method includes displaying the modified initial three-dimensional representation of the body at block 310C. In this way, the patient can see how they would actually look if their body were adjusted to the preferred anatomical profile. The resulting image is a modified version of the patient's body shape, not simply a preferred profile overlaid on the patient.

FIG. 4 is a schematic diagram of an example computing device 400 that may be used to implement the systems and methods described in this document. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions illustrated herein are intended to be merely exemplary and are not intended to limit the implementation of the inventions described and/or claimed in this document.

The computing device 400 includes a processor 410, a memory 420, a storage device 430, a high-speed interface/controller 440 that connects to the memory 420 and the high-speed expansion port 450, and a low-speed interface/controller 460 that connects to the low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and can be implemented on a common motherboard or in other suitable manners. The processor 410 can process instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430, to display graphical information for a graphical user interface (GUI) on an external input/output device, such as a display 480, coupled to the high-speed interface 440. In other implementations, multiple processors and/or multiple buses can be used, as appropriate, with multiple memories and multiple memory types. Additionally, multiple computing devices 400 may be connected, each providing different parts of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

The memory 420 stores information non-transiently within the computing device 400. The memory 420 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

The storage device 430 can provide mass storage for the computing device 400. In some implementations, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product stores instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 420, the storage device 430, or a memory on the processor 410.

The high-speed controller 440 manages bandwidth-intensive operations for the computing device 400, while the low-speed controller 460 manages less bandwidth-intensive operations. Such allocation of operational responsibilities is merely exemplary. In some implementations, the high-speed controller 440 is coupled to the memory 420, to the display 480 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 450 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 460 is coupled to the storage device 430 and to the low-speed expansion port 490. The low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, pointing device, scanner, or networking devices such as switches and routers, for example, through a network adapter.

The computing device 400 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 400a, or more likely as a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

Among other advantages, the present disclosure discloses methods, user devices, and systems for displaying augmented anatomical features. The augmented reality device can overlay a virtual image of anatomy onto a human body that depicts an approximation of the structures, tissues, or organs that lie beneath the surface of an individual, such as a target individual, in front of a user, such as a health care professional. The virtual image can be adjusted to fit the target individual. Using the augmented reality device, a user can identify specific anatomical reference points on the body of the target individual and use those points to fixate and adjust the virtual image over the target individual. The virtual image may be representative of a person's anatomy, such as a person of the same age or gender.

Among other advantages, the present disclosure further provides methods, user devices, and systems that do not require input of data files from external imaging modalities (e.g., x-ray, magnetic resonance imaging, computed tomography scans, etc.) that may have incompatible formats, be large and unwieldy, or require vast amounts of processing power for each target individual.

Among other advantages, the present disclosure further provides methods, user devices, and systems that may be of general use. In this regard, the use of the augmented reality device is not limited to certified health care providers. Additionally, the capabilities of the augmented reality device may be to output or display computer-generated approximations of representative human anatomy.

Among other advantages, the present disclosure also provides broad applicability: the augmented reality device can be used consistently and quickly from target individual to target individual, and without the need for external data input.

Various implementations of the systems and techniques described herein may be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementations in one or more computer programs executable and/or translatable on a programmable system including at least one programmable processor, whether dedicated or general purpose, coupled to receive and transmit data and instructions to and from a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and/or object-oriented programming languages and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable media" refer to any computer program product, non-transitory computer-readable medium, apparatus, and/or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification may be performed by one or more programmable processors, sometimes referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and dedicated microcomputers, and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, the computer will further include one or more mass storage devices, such as magnetic disks, magneto-optical disks, optical disks, for data storage, or be operatively coupled to receive data from such mass storage devices or transmit data to such mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer programs, instructions, and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for user interaction, one or more aspects of the disclosure may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and a pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices may be used to provide for user interaction as well, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer may interact with the user by sending and receiving documents to and from the device being used by the user, for example, by sending web pages to a web browser in response to a request received from the web browser on the user's client device.

A software application (i.e., a software resource) may refer to computer software that enables a computing device to perform tasks. In some embodiments, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

For example, embodiments of the present invention are as follows.
[Embodiment 1]
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining, in a processor, an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to assist in identifying the internal anatomical features;
selecting a medical procedure and determining a future-state anatomical profile corresponding to the selected medical procedure, wherein the visual representation of the body is modified with the graphical representation of the internal anatomical features based on the selected medical procedure;
A method for providing the above.
[Embodiment 2]
13. The method of claim 1,
A method, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, tip of a nose, or ears.
[Embodiment 3]
13. The method of claim 1,
The method, wherein the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.
[Embodiment 4]
13. The method of claim 1,
The method, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle.
[Embodiment 5]
13. The method of claim 1,
A method, wherein the plurality of reference markers are identified by a user interacting with the display.
[Embodiment 6]
13. The method of claim 1,
The method further includes providing a graphical representation of the body modified with external anatomical features based on the selected medical procedure.
[Embodiment 7]
13. The method of claim 1,
The method, wherein the selected medical procedure is a procedure affecting at least one of an organ, a bone, or a muscle.
[Embodiment 8]
The method of embodiment 1, further comprising:
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation;
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the graphical representation comprises a visual representation of the body modified using the internal anatomical features based on the determined anatomical profile.
[Embodiment 9]
9. The method of claim 8, further comprising:
The method, wherein the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle.
[Embodiment 10]
9. The method of claim 8, further comprising:
A method, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, tip of a nose, or ears.
[Embodiment 11]
9. The method of claim 8, further comprising:
The method, wherein the anatomical profile of the target individual is further based on a plurality of data corresponding to the body.
[Embodiment 12]
9. The method of claim 8, further comprising:
The method, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle.
[Embodiment 13]
9. The method of claim 8, further comprising:
A method, wherein the plurality of reference markers are identified by a user interacting with the display.
[Embodiment 14]
The method of embodiment 1, further comprising:
scaling the internal anatomical features to fit the visual representation of the body.
[Embodiment 15]
A display and
data processing hardware in communication with said display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
selecting a medical procedure and determining a future-state anatomical profile corresponding to the selected medical procedure, wherein the visual representation of the body is modified with a graphical representation of the internal anatomical features based on the selected medical procedure; and
displaying, on a display, the graphical representation of the internal anatomical feature over the visual representation to aid in identifying the internal anatomical feature; and
1. A user device comprising:
[Embodiment 16]
16. The user device of claim 15,
A user device, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.
[Embodiment 17]
16. The user device of claim 15,
The anatomical profile of the target individual is further based on a plurality of data corresponding to the body, the user device.
[Embodiment 18]
16. The user device of claim 15,
A user device, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle.
[Embodiment 19]
16. The user device of claim 15,
A user device, wherein the plurality of reference markers are identified by a user interacting with the display.
[Embodiment 20]
16. The user device of claim 15,
The plurality of reference markers are identified by the data processing hardware.
[Embodiment 21]
16. The user device of claim 15,
A user device, wherein the selected medical procedure is a procedure that affects at least one of an organ, a bone, or a muscle.
[Embodiment 22]
16. The user device of claim 15,
The operations further include:
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation; and
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the graphical representation includes a visual representation of the body modified using the internal anatomical features based on the determined anatomical profile.
[Embodiment 23]
23. The user device of claim 22, further comprising:
A user device, wherein the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle.
[Embodiment 24]
16. The user device of claim 15,
The operations further include scaling the internal anatomical features to fit the visual representation of the body, the user device.
[Embodiment 25]
A user device, including:
display,
data processing hardware in communication with said display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining, at a processor, an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features; and
selecting a medical procedure and determining a future-state anatomical profile corresponding to the selected medical procedure, wherein the visual representation of the body is modified with a graphical representation of the internal anatomical features based on the selected medical procedure;
a user device including:
a wearable in communication with the user device, the wearable including a second display configured to display the graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features;
A system comprising:
[Embodiment 26]
26. The system of claim 25,
The system, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.
[Embodiment 27]
26. The system of claim 25,
The anatomical profile of the target individual is further based on a plurality of data corresponding to the body.
[Embodiment 28]
26. The system of claim 25,
The plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle.
[Embodiment 29]
26. The system of claim 25,
The system, wherein the plurality of reference markers are identified by a user interacting with the display.
[Embodiment 30]
26. The system of claim 25,
The wearable portion is further configured to detect and map the target individual independent of the user device.
[Embodiment 31]
26. The system of claim 25,
The selected medical procedure is a procedure that affects an organ,system.
[Embodiment 32]
26. The system of claim 25,
The operations further include:
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation; and
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the graphical representation includes a visual representation of the body modified using the internal anatomical features based on the determined anatomical profile.
[Embodiment 33]
26. The system of claim 25,
The operations further include scaling the internal anatomical features to fit the visual representation of the body.
[Embodiment 34]
generating an initial three-dimensional representation of the body of a target individual;
identifying a plurality of reference markers on the initial three-dimensional representation of the body;
determining, in a processor, a preferred anatomical profile of the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of the body;
three-dimensionally modifying, in the processor, the initial three-dimensional representation of the body to have the shape of the preferred anatomical profile;
displaying the modified initial three-dimensional representation of the body on the display;
A method for providing the above.
[Embodiment 35]
35. The method of claim 34,
A method, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, tip of a nose, or ears.
[Embodiment 36]
35. The method of embodiment 34, further comprising:
selecting a medical procedure associated with a desired body part;
replacing the desired body portion with a corresponding desired body portion taken from the preferred anatomical profile;
and displaying the initial three-dimensional representation of the body having the corresponding desired body part.
[Embodiment 37]
37. The method of claim 36,
The method, wherein the medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.
[Embodiment 38]
38. The method of embodiment 37, comprising:
A method in which future anatomical images are displayed in three dimensions.
[Embodiment 39]
A display and
data processing hardware in communication with said display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
generating an initial three-dimensional representation of the body of a target individual;
identifying a plurality of reference markers on the initial three-dimensional representation of the body;
determining, at a processor, a preferred anatomical profile of the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of the body;
three-dimensionally modifying, in the processor, the initial three-dimensional representation of the body to have a shape of the preferred anatomical profile; and
displaying the modified initial three-dimensional representation of the body on the display; and
1. A user device comprising:
[Embodiment 40]
40. The user device of claim 39,
A user device, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder.
[Embodiment 41]
40. The user device of claim 39,
The operations further include:
Selecting a medical procedure associated with a desired body part;
replacing the desired body part with a corresponding desired body part taken from the preferred anatomical profile to generate a future anatomical image; and
and displaying the future anatomical image.
[Embodiment 42]
42. The user device of claim 41,
The medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.
[Embodiment 43]
43. The user device of claim 42,
The future anatomical image is displayed in three dimensions on a user device.
User devices, including
[Embodiment 44]
A user device, including:
A first display;
data processing hardware in communication with the display; and
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
generating an initial three-dimensional representation of the body of a target individual;
identifying a plurality of reference markers on the initial three-dimensional representation of the body;
determining, at a processor, a preferred anatomical profile of the target individual based on the plurality of reference markers, the preferred anatomical profile being a preferred three-dimensional representation of the body;
memory hardware storing instructions that cause the processor to perform operations including three-dimensionally modifying the initial three-dimensional representation of the body to have the shape of the preferred anatomical profile;
a user device including:
a wearable in communication with the user device, the wearable including a second display configured to display the modified initial three-dimensional representation of the body on a second display;
A system comprising:
[Embodiment 45]
45. The system of claim 44,
The system, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, a tip of a nose, or ears.
[Embodiment 46]
45. The system of claim 44,
The operations further include:
Selecting a medical procedure associated with a desired body part;
replacing the desired body part with a corresponding desired body part taken from the preferred anatomical profile to generate a future anatomical image; and
displaying the future anatomical image.
[Embodiment 47]
47. The system of claim 46,
The medical procedure is one selected from the list consisting of liposuction, breast augmentation, and tummy tuck.
[Embodiment 48]
48. The system of claim 47,
The future anatomical images are displayed in three dimensions.
[Embodiment 49]
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining, in a processor, an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to assist in identifying the internal anatomical features;
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation;
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the visual representation of the body is modified with the graphical representation of the internal anatomical features based on the determined anatomical profile;
A method for providing the above.
[Embodiment 50]
A display and
data processing hardware in communication with said display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
displaying on said display a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to assist in identifying the internal anatomical features;
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation; and
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the graphical representation includes a visual representation of the body modified with the internal anatomical features based on the determined anatomical profile; and
1. A user device comprising:
[Embodiment 51]
A user device, including:
A first display;
data processing hardware in communication with the first display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining, at a processor, an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation;
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein a graphical representation includes a visual representation of the body modified with the internal anatomical features based on the determined anatomical profile;
a user device including:
a wearable in communication with the user device, the wearable including a second display configured to display the graphical representation of the internal anatomical features over the visual representation of the body to aid in identifying the internal anatomical features;
A system comprising:
A number of embodiments have been described. It is understood, nevertheless, that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the appended claims.

10 Augmented reality system 100 Healthcare environment 102 Augmented reality device 102a First augmented reality device 102b Second augmented reality device 104 User 105, 105' Visual representation 106, 106' Target individual 108 Image capture device 108a, 108b Camera 110, 110a, 110b Display 111 Trackpad 112, 112a-112g Reference markers 113 Graphical representation 113' Initial three-dimensional representation 114, 114a-114c Virtual image 118 Menu 120 Rotating platform 200 System 204 Augmented reality module 208 Network 210 Detection module 212 Tracking module 214 Input module 216 Anatomy module 216' Future state anatomical profile 216'' Preferred anatomical profile 218 Anatomical database 220 Anatomical profile 222 Anatomical data 224 Reference markers 226 Selected medical procedure 228 Corrected three-dimensional representation 400 Computing device 400a Server 400b Laptop computer 400c Rack server system 410 Processor 420 Memory 430 Storage device 440 High speed interface/controller 450 High speed expansion port 460 Low speed interface/controller 470 Low speed bus 480 Display 490 Low speed expansion port

Claims (24)

displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining, in a processor, an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
displaying, on the display, a graphical representation of the internal anatomical features over the visual representation of the body to assist in identifying the internal anatomical features;
selecting a medical procedure and determining a future state anatomical profile corresponding to the selected medical procedure;
displaying, on a display displaying the visual representation of the body, modified internal anatomical features based on the future state anatomical profile;
A method for providing the above. 10. The method of claim 1 ,
A method, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, tip of a nose, or ears. 10. The method of claim 1 ,
The method, wherein the anatomical profile of the target individual is further based on a plurality of data corresponding to the body. 10. The method of claim 1 ,
The method, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle. 10. The method of claim 1 ,
A method, wherein the plurality of reference markers are identified by a user interacting with the display. 10. The method of claim 1 ,
The method further includes providing a graphical representation of the body modified with external anatomical features based on the selected medical procedure. 10. The method of claim 1 ,
The method, wherein the selected medical procedure is a procedure affecting at least one of an organ, a bone, or a muscle. 10. The method of claim 1 further comprising:
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation;
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical variation and the identified plurality of reference markers, wherein the graphical representation comprises a visual representation of the body modified using the internal anatomical features based on the determined anatomical profile. 9. The method of claim 8,
The method, wherein the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle. 9. The method of claim 8,
A method, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a buttocks, a portion of a clavicle, a portion of a shoulder, lips, corners of a mouth, tip of a nose, or ears. 9. The method of claim 8,
The method, wherein the anatomical profile of the target individual is further based on a plurality of data corresponding to the body. 9. The method of claim 8,
The method, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle. 9. The method of claim 8,
A method, wherein the plurality of reference markers are identified by a user interacting with the display. 10. The method of claim 1 further comprising:
scaling the internal anatomical features to fit the visual representation of the body. A display and
data processing hardware in communication with said display;
memory hardware in communication with said data processing hardware, instructions which, when executed on said data processing hardware, cause said data processing hardware to perform operations including:
displaying, on a display, a visual representation of the body of the target individual;
identifying a plurality of reference markers on the visual representation of the body;
determining an anatomical profile of the target individual based on the plurality of reference markers, the anatomical profile including a plurality of internal anatomical features;
selecting a medical procedure and determining a future state anatomical profile corresponding to the selected medical procedure; and
displaying, on a display displaying the visual representation of the body, modified internal anatomical features based on the future state anatomical profile; and
1. A user device comprising: 16. The user device of claim 15,
A user device, wherein the plurality of reference markers correspond to at least one of a navel, a portion of a sternum, a portion of a hip, a portion of a clavicle, or a portion of a shoulder. 16. The user device of claim 15,
The anatomical profile of the target individual is further based on a plurality of data corresponding to the body, the user device. 16. The user device of claim 15,
A user device, wherein the plurality of internal anatomical features includes at least one of an organ, a bone, or a muscle. 16. The user device of claim 15,
A user device, wherein the plurality of reference markers are identified by a user interacting with the display. 16. The user device of claim 15,
The plurality of reference markers are identified by the data processing hardware. 16. The user device of claim 15,
A user device, wherein the selected medical procedure is a procedure that affects at least one of an organ, a bone, or a muscle. 16. The user device of claim 15,
The operations further include:
identifying at least one of a disease, a genetic condition, or an anatomical variation, wherein the anatomical profile comprises the disease, the genetic condition, or the anatomical variation; and
determining an anatomical profile corresponding to the identified disease, the identified genetic condition, or the identified anatomical mutation and the identified plurality of reference markers, wherein the graphical representation of the internal anatomical features includes a visual representation of the body modified with the internal anatomical features based on the determined anatomical profile. 23. The user device of claim 22,
A user device, wherein the disease, the genetic condition, and the anatomical variation affect at least one of an organ, a bone, or a muscle. 16. The user device of claim 15,
The operations further include scaling the internal anatomical features to fit the visual representation of the body, the user device.

Images