JP7730699B2 - Smart glasses equipped with lenses that can control light transmittance - Google Patents

Description

The present invention relates to smart glasses technology. In particular, it relates to see-through smart glasses as an augmented reality wearable computer.

Smart glasses are worn like eyeglasses in front of the user's (wearer's) field of vision and are equipped with a projection display on the lenses. Unlike virtual reality, which displays an unreal space, smart glasses display information superimposed on the wearer's field of vision in real space.

In recent years, a decline in the quality of nursing care services at elderly care facilities and medical institutions has become a social issue due to an increase in the number of people receiving care and a shortage of caregivers. Caregivers, in particular, need to provide different care for each care recipient. However, in facilities with a large number of care recipients, it is natural that caregivers will not be able to remember all of the personal information of each care recipient (for example, information on symptoms that are necessary for care).

In response to this, there is technology that allows the caregiver to wear smart glasses and display the care recipient's personal information hands-free on the lenses (see, for example, non-patent document 1). With this technology, the care recipient can be identified from an image of their face captured by the smart glasses' camera, and their personal information can be immediately displayed on the lenses. This means that the care recipient does not need to wear a wireless tag or marker for personal identification.

Figure 1 is an explanatory diagram showing the field of view through smart glasses.

For example, suppose a caregiver is wearing smart glasses 1 at a nursing home. At this time, the caregiver can see the care recipient through the lenses of smart glasses 1, and at the same time, can see the care recipient's "personal information" displayed on the lenses. This allows the caregiver to carry out care work while reading the care recipient's personal information hands-free.

The smart glasses 1 may be configured to store personal information in advance for each care recipient ID (standalone type), or may be configured to access a server via a network and download the information (server-client type).
Examples of personal information of care recipients include the following:
(basic information)
Name, age, level of care required, date of birth, room number, care plan, etc.
(Behavioral information)
Wake-up time, meals (done/not yet), hydration (done/not yet), medication (done/not yet), excretion (yes/no), bathing (done/not yet), etc.
(Transfer information)
Health condition, body temperature, blood pressure (diastolic and diastolic), pulse, treatment information, symptoms, treatment information, number of days without excretion, excretion time, abnormal excretion (yes/no), breakfast/lunch/supper time, amount of breakfast/lunch/supper amount, time of hydration, amount of fluid, time of taking medication, time of bathing, abnormal bathing (yes/no), etc.

However, as shown in Figure 1, there was a problem in that the information displayed on the lenses of the smart glasses 1 was very difficult to see for the wearer (caregiver). Because they are see-through, black text information in particular becomes visually very faded. According to the technology described in Non-Patent Document 1, sunglasses are used as the lenses of the smart glasses 1.

Figure 2 is an explanatory diagram showing the field of view through smart sunglasses.

As shown in Figure 2, when viewed from the wearer (caregiver) of the smart glasses 1, the displayed information projected on the lenses appears bright. This makes it easier for the caregiver wearing the smart glasses 1 to read the personal information of the person being cared for. In other words, by using lenses with low light transmittance, such as sunglasses (dark), the wearer's visibility of the displayed information can be improved.

KDDI Research and Zenkokai Press Release, "KDDI Research and Zenkokai Develop Hands-Free Nursing Care Work Support System Using AR Glasses" (February 2, 2021), [online], [Retrieved August 30, 2021], Internet <URL: https://www.kddi-research.jp/newsrelease/2021/020201.html> Ryota Tsukawaki et al., "The Effect of Wearing Glasses on Interpersonal Impressions," Hiroshima University Psychology Research, Vol. 10, 2010, [online], [Retrieved August 30, 2021], Internet <URL: https://ir.lib.hiroshima-u.ac.jp/files/public/3/31326/20141016181135840232/HPR_10_321.pdf> Liquid crystal light control film, [online], [searched August 30, 2021], Internet <URL: https://www.toppan.co.jp/electronics/new_products/lc_magic/> Instant photochromic sunglasses, [online], [searched August 30, 2021], Internet <URL: https://eshades.visionup.jp/> 0.1-second instant photochromic sunglasses, [online], [searched August 30, 2021], Internet <URL: https://www.wicue.jp/product/wicue01/>

In the case of the above-mentioned Non-Patent Document 1 (FIG. 2), from the perspective of the care recipient, the caregiver (wearer) appears to be wearing sunglasses.
However, some academic papers have suggested that sunglasses "reduce social desirability and gentleness" in terms of interpersonal impressions (see, for example, Non-Patent Document 2). In particular, in nursing care facilities, care recipients may have an uneasy interpersonal impression (a sense of resistance or dislike) of caregivers wearing smart glasses, thinking, "They look scary because they look like they're wearing black sunglasses." One reason for this is that it is difficult for care recipients to see the caregiver's "eyes" and "gaze."

In response to this, the inventors of the present application wondered whether it would be possible to control the brightness of the lenses in accordance with the surrounding conditions of the smart glasses. In other words, they wondered whether it would be possible to control the light transmittance of the lenses in smart glasses when it is necessary to ensure that the wearer of the smart glasses can reliably read the displayed information or when it is necessary to avoid creating an uncomfortable impression on others.

Therefore, the present invention aims to provide smart glasses that can control the light transmittance of the lenses depending on the surrounding conditions of the smart glasses.

According to the present invention, there is provided a see-through smart glass that can be worn in a user's field of vision,
a lens whose light transmittance can be controlled by a light control film and which has a display function of projecting display information into the user's field of vision;
an integrated or externally connected camera that captures an image of the user's field of view;
an object recognition means for recognizing a predetermined object from the video;
The device is characterized by having a control means for lowering the light transmittance of the light control film of the lens when display information is projected onto the lens, and for controlling the light transmittance of the light control film of the lens to be higher when a predetermined object is recognized.

According to another embodiment of the smart glasses of the present invention,
It is also preferable that the control means includes a means for controlling the light control film of the lens to a light transmittance lower than the current light transmittance when display information is projected onto the lens.

According to another embodiment of the smart glasses of the present invention,
It is also preferable that the control means controls the light control film of the lens so as to return the light transmittance to the original current light transmittance when a predetermined time that is readable by the user has elapsed after the display information is projected onto the lens.

According to another embodiment of the smart glasses of the present invention,
The object recognition means detects an image area of a predetermined object from the video;
It is also preferable that the control means controls the light transmittance of the light control film of the lens when the size of the image area exceeds a predetermined range.

According to another embodiment of the smart glasses of the present invention,
The object recognition means detects an image area of a predetermined object from the video;
It is also preferable that the control means controls the light control film of the lens to a light transmittance higher or lower than the current light transmittance when the size of the image area exceeds a predetermined range.

According to another embodiment of the smart glasses of the present invention,
It is also preferable that the object recognition means recognizes a human face area as an image area, and that the control means controls the light control film of the lens to have a light transmittance higher than the current light transmittance when the size of the face area becomes equal to or larger than a predetermined range.

According to another embodiment of the smart glasses of the present invention,
It is also preferable that the control means controls the light control film of the lens so as to return the light transmittance to the original current light transmittance when the size of the face area is no longer within the predetermined range.

According to another embodiment of the smart glasses of the present invention,
Further, the device has a wireless communication means for communicating with an external device,
The control means controls the light transmittance of the light control film of the lens when a predetermined signal is received from an external device.
It is also preferable.
As an embodiment not dependent on the present invention, there is provided a see-through type smart glass that can be worn in a user's field of vision,
a lens whose light transmittance can be controlled;
a wireless communication means capable of communicating with an external device;
The optical transmission device is characterized by having a control means for controlling the light transmittance of the lens when a predetermined signal is received from an external device.

According to another embodiment of the smart glasses of the present invention,
The wireless communication means receives radio waves from a wireless tag as an external device, or communicates with a wireless base station or an access point;
It is also preferred that the control means controls the light management film of the lens to a light transmittance higher or lower than the current light transmittance when it receives a predetermined signal including a predetermined address or ID.

According to another embodiment of the smart glasses of the present invention,
Further comprising a positioning means for detecting a current position,
The control means controls the light transmittance of the light control film of the lens when the current position stays within a predetermined position range.
It is also preferable.
As an embodiment not dependent on the present invention, there is provided a see-through type smart glass that can be worn in a user's field of vision,
a lens whose light transmittance can be controlled;
a positioning means for detecting a current location;
The present invention is characterized by comprising a control means for controlling the light transmittance of the lens when the current position stays within a predetermined position range.

According to another embodiment of the smart glasses of the present invention,
The control means controls the light transmittance of the light control film of the lens when the current time falls into a predetermined time zone.
It is also preferable.
As an embodiment not dependent on the present invention, there is provided a see-through type smart glass that can be worn in a user's field of vision,
a lens whose light transmittance can be controlled;
The present invention is characterized by comprising a control means for controlling the light transmittance of the lens when the current time falls into a predetermined time zone.

The smart glasses of the present invention can control the light transmittance of the lenses depending on the surrounding conditions of the smart glasses.

FIG. 1 is an explanatory diagram showing the field of view through smart glasses. This is an explanatory diagram showing the field of view through smart sunglasses. FIG. 1 is a diagram showing the external hardware configuration of smart glasses according to the present invention. FIG. 1 is a diagram illustrating a software functional configuration of smart glasses according to the present invention. 10 is an explanatory diagram illustrating how the light transmittance of a lens is controlled depending on whether or not display information is displayed. FIG. FIG. 10 is an explanatory diagram illustrating how the light transmittance of a lens is controlled depending on the size of a face area photographed by a camera. FIG. 10 is an explanatory diagram illustrating control of the light transmittance of a lens by receiving a predetermined signal from the outside. FIG. 10 is an explanatory diagram illustrating how the light transmittance of the lens is controlled depending on the position where the lens is positioned. FIG. 10 is an explanatory diagram illustrating control of the light transmittance of a lens depending on the time of day.

The following describes in detail the embodiments of the present invention using the drawings.

FIG. 3 is a diagram showing the external hardware configuration of smart glasses according to the present invention.
FIG. 4 is a diagram showing the software functional configuration of the smart glasses according to the present invention.

The smart glasses 1 are see-through type that can be worn in the user's field of vision.
According to FIG. 3, the smart glasses 1 are equipped with a lens 10, a camera 12, a wireless communication unit 13, and a positioning unit 14 as hardware.
4, the smart glasses 1 include, as software, a control unit 11 and an object recognition unit 15. These functional components are realized by executing a program that causes a computer installed in the smart glasses to function.

[Lens 10]
The lens 10 has a controllable light transmittance. The lens 10 uses, for example, a liquid crystal light control film, and can increase (make transparent) or decrease (make dark) the light transmittance (see, for example, Non-Patent Documents 3 to 5). The lens 10 can change the light transmittance in accordance with the control current output from the control unit 11.
The lens 10 also has a display function that projects display information into the field of view of the wearer of the smart glasses 1. This allows the wearer to visually recognize the display information.
The specific range of high and low light transmittance (control range) may be separately set by the wearer of the smart glasses 1 themselves.

[Control unit 11]
The control unit 11 projects display information onto the lens 10. The display information is pre-stored according to predetermined conditions and may be text, images, or videos. The display information may be pre-stored and received from an external device connected to the network via the wireless communication unit 13. Furthermore, the control unit 11 may project the display information onto the lens 10 when it receives the display information via the wireless communication unit 13.
According to the present invention, the control unit 11 controls the light transmittance of the lens 10 depending on whether or not display information is projected onto the lens 10 .

Figure 5 is an explanatory diagram showing how the light transmittance of the lens is controlled depending on whether or not display information is displayed.

(S1) Under normal circumstances, the lenses 10 of the smart glasses 1 have high light transmittance (transparent). The eyes and gaze of the wearer of the smart glasses 1 can be seen by other people, and the impression is not unnatural.
(S2) Now, suppose that a trigger (timing) occurs to display display information on the lens 10 of the smart glasses 1. When projecting display information onto the lens 10, the control unit 11 controls the lens 10 to have a light transmittance lower than the current light transmittance. That is, the lens 10 of the smart glasses 1 is darkened like sunglasses. Although the eyes and line of sight of the wearer of the smart glasses 1 become difficult for other people to see, it is important to enhance the visibility of the display information for the wearer for a certain period of time.
(S3) After the display information is projected onto the lens 10, a predetermined time period during which the information can be read by the user has elapsed. Specifically, this is set to, for example, about 5 seconds. In other words, by minimizing the time during which the lens 10 is darkened, the uncomfortable impression on the user is reduced even if only slightly.
(S4) Then, the control unit 11 controls the lens 10 to return to the original current light transmittance. The lens 10 of the smart glasses 1 is again set to high light transmittance (transparent), so that the eyes and line of sight of the wearer of the smart glasses 1 are visible to other people. Also, the display information that would have been visible to the wearer is hidden.

Figure 6 is an explanatory diagram showing how the light transmittance of the lens is controlled based on the size of the face area captured by the camera.

[Camera 12]
The camera 12 may be integrated with the smart glasses 1 or may be externally connected to the smart glasses 1. The camera 12 captures video of the wearer's field of vision. The video may include a person (e.g., a care recipient) or may include only an object such as facility equipment. The camera 12 constantly outputs the captured video to the object recognition unit 15.

[Object recognition unit 15]
The object recognition unit 15 recognizes a predetermined object from the image captured by the camera 12 .
The object recognition unit 15 is a general machine learning engine that is applied to image classification and face recognition based on deep learning, and recognizes specific classes (categories such as people and objects) that appear in the video.

The object recognition unit 15 has an "image area detection function" that detects the image area of an object, and an "object detection function" that detects the object in that image area.

<Image area detection function>
The image region detection function detects the image region of a specific object from a video. This involves extracting the object region (e.g., a bounding box) from the image of the video frame. Specifically, R-CNN (Regions with Convolutional Neural Networks) or SSD (Single Shot Multibox Detector) is used.
R-CNN combines rectangular object regions with convolutional neural network features to find a subset of object regions (region proposals). It then extracts CNN features from the region proposals. It then refines the bounding boxes of the region proposals using a support vector machine pre-trained with the CNN features.
SSD is a generic object detection algorithm that uses machine learning to determine rectangular bounding boxes called default boxes. Multiple default boxes of different sizes are overlaid on an image, and a predicted value is calculated for each box. For each default box, the system can predict its position based on how far it is from the object and how different its size is.

<Object detection function>
The object detection function detects a predetermined object from an image area. The predetermined object may be a human body or a face, or may be an object such as a book.
As an embodiment of the present invention is intended for use in a care facility, the object detection function recognizes a human face area from an image and identifies a user ID (identifier) from the face area.
The object detection function stores in advance feature amounts of the facial image of the actual user to be identified. For example, feature amounts of the facial image of a care recipient are stored as the user.
The object detection function then uses a facial recognition model to convert the facial region, which is the extracted image area, into 128-, 256-, or 512-dimensional features (Euclidean distance). Specifically, the facial recognition model can use the Google® Facenet® algorithm. This allows the facial region to be converted into a multidimensional vector feature.
Then, the object recognition unit 15 compares the collected data with a set of features of the user's face, and identifies the user ID with the feature that has the shortest distance or is equal to or smaller than a predetermined threshold.

<Functions of the control unit 11 based on the object recognition unit 15>
When a predetermined object is recognized by the object recognition unit 15, the control unit 11 controls the light transmittance of the lens 10 to a value higher or lower than the current light transmittance. Here, the predetermined object is assumed to be a human face.

In the case of a face area, the size and distance of the face area reflected in the image are proportional. When the size of the face area exceeds a predetermined range, the control unit 11 controls the lens 10 to have a light transmittance higher than the current light transmittance (transparent).

(S1) The smaller the size of the face area, the greater the distance between the subject user and the wearer.
If the size of the face area is smaller than a predetermined range, the control unit 11 determines that the distance to the subject user (care recipient) is far. Even in this case, the object detection function is used to identify the user ID of the subject user from the face area. Then, display information (personal information) corresponding to the user ID is displayed in advance on the lens 10. At this time, because the distance between the subject user and the wearer is far, the light transmittance of the lens 10 may be low (dark). The wearer can read the subject user's personal information before approaching the subject user.

(S2) The larger the size of the face area, the closer the distance between the subject user and the wearer.
When the size of the face area is equal to or larger than a predetermined range, the control unit 11 determines that the distance between the user (person receiving care) and the smart glasses 1 is close. In this case, the control unit 11 controls the lens 10 to have a higher light transmittance than the current light transmittance. The lens 10 of the smart glasses 1 becomes transparent (high light transmittance). In this case, when the caregiver (wearer) and the care recipient (person receiving care) are face to face in close proximity, the lens 10 of the smart glasses 1 worn by the caregiver has a high light transmittance (transparent), allowing the care recipient to see the caregiver's eyes. This prevents the care recipient from feeling uncomfortable about the caregiver's interpersonal impression, as would be the case if the caregiver were wearing sunglasses.

Furthermore, when the control unit 11 determines that the distance to an object such as a book (subject) is short, it may control the lens 10 to have a higher light transmittance (transparent) than the current light transmittance. This makes it possible to avoid the dark, difficult-to-read image, as with sunglasses, when reading a book.

(S3) Thereafter, when the size of the face area is no longer within the predetermined range (when the distance becomes farther), the control unit 11 controls the lens 10 to return to the original current light transmittance. When the wearer's field of vision becomes farther away from the other person, the smart glasses 1 may become dark like sunglasses.

In another embodiment, the display information displayed on the lens 10 of the smart glasses 1 may be information about an "object" shown in the image captured by the camera 12.

The object recognition unit 15 has a lightweight learning model so that it can be implemented in small IoT devices. The amount of calculation is significantly reduced by selecting intermediate layer features from the trained model (deleting layers) and replacing it with a lightweight architecture. Specifically, by using a C++ native library, it is possible to identify 10,000 people in one second, even on a small device such as smart glasses.

Figure 7 is an explanatory diagram showing how the light transmittance of the lens is controlled by receiving a specific signal from the outside.

[Wireless communication unit 13]
The wireless communication unit 13 may be an interface for short-range wireless communication, short-range wireless communication, and/or mobile phone communication, which is capable of communicating with an external device.
The control unit 11 controls the lens 10 to change the light transmittance to a value higher or lower than the current light transmittance when a predetermined signal is received from an external device via the wireless communication unit 13. Here, the predetermined signal from the external device may be transmitted by a third party operating the external device.

The short-range wireless communication interface may be, for example, one that receives wireless signals transmitted from a BLE (Bluetooth Low Energy) tag (wireless tag). BLE tags are pre-placed at various locations within the facility where the wearer travels. For example, wireless tags can be pre-placed at locations where the light transmittance of the lens 10 of the smart glasses 1 should be controlled.

The short-range wireless communication interface may also receive wireless signals from a wireless LAN access point. Packets received from the access point are assigned a MAC (Media Access Control) address or an IP address. The specified signal may be a specified address of the data packet. The access points are assumed to be pre-installed at various locations within the facility where the wearer travels.

Furthermore, the short-range wireless communication interface may receive a predetermined signal from an external device via an access point, access network, or the Internet. At that point, the control unit 11 controls the lens 10 to have a higher or lower light transmittance than the current light transmittance. The predetermined signal is a specific data packet sent from a terminal or server connected to the Internet.

Furthermore, the mobile phone communication interface may be one that receives push messages from the center. If the smart glasses 1 also have telephone functionality like existing smartphones, the center can send a push message to that phone number. At that point, the control unit 11 controls the lens 10 to have a light transmittance higher or lower than the current light transmittance.

Display information may also be received via wireless communication unit 13 from a terminal or center, which is an external device connected to a mobile phone network or the Internet. The received display information may be stored within control unit 11, or may be displayed on lens 10. When display information is displayed on lens 10, the present invention essentially involves control unit 11 controlling lens 10 to have a light transmittance higher or lower than the current light transmittance.

Figure 8 is an explanatory diagram showing how the light transmittance of the lens is controlled depending on the position.

[Positioning unit 14]
The positioning unit 14 detects the current location. For example, it may receive radio waves from a GPS (Global Positioning System) to determine the position, or it may determine the position from the reception strength of radio waves from base stations and access points.
In response to this, the control unit 11 controls the light transmittance of the lens when the current position stays within a predetermined position range.

In both the wireless communication unit 13 in FIG. 7 and the positioning unit 14 in FIG. 8, the control unit 11 can darken or clear the lenses 10 when the smart glasses 1 are in a predetermined position. For example, in dark surrounding areas, the lenses 10 can be forced to be clear, and in bright surrounding areas, the lenses 10 can be forced to be dark like sunglasses.

Figure 9 is an explanatory diagram showing how the light transmittance of the lens is controlled depending on the time of day.

As shown in Figure 9, the control unit 11 controls the light transmittance of the lens when the current time falls within a specified time period. This allows the lens 10 to be forced to become transparent, for example, during nighttime hours.

As explained in detail above, the smart glasses of the present invention can control the light transmittance of the lenses depending on the surrounding conditions of the smart glasses. In particular, the light transmittance of the lenses in the smart glasses can be controlled when it is necessary to ensure that the wearer of the smart glasses can read the displayed information reliably or when it is necessary to avoid creating an uncomfortable interpersonal impression.

Furthermore, this will, for example, "reduce the sense of discomfort felt by caregivers when wearing smart glasses," thereby contributing to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Ensure healthy lives and promote well-being for all at all ages."

With regard to the various embodiments of the present invention described above, those skilled in the art will be able to easily make various changes, modifications, and omissions that fall within the scope of the technical spirit and principles of the present invention. The foregoing description is merely illustrative and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents.

REFERENCE SIGNS LIST 1 Smart glasses 10 Lens 11 Control unit 12 Camera 13 Wireless communication unit 14 Positioning unit 15 Object recognition unit

Claims (11)

See-through smart glasses that can be worn in a user's field of vision,
a lens whose light transmittance can be controlled by a light control film and which has a display function of projecting display information into the user's field of vision;
an integrated or externally connected camera that captures an image of the user's field of view;
an object recognition means for recognizing a predetermined object from the video;
Smart glasses characterized by having a control means that reduces the light transmittance of the lens's light-adjusting film when display information is projected onto the lens, and controls the light transmittance of the lens's light-adjusting film to increase when a specified object is recognized. The smart glasses according to claim 1, wherein the control means controls the light-control film of the lens to have a light transmittance lower than the current light transmittance when display information is projected onto the lens. The smart glasses according to claim 2, characterized in that the control means controls the light-adjusting film of the lens so as to return the light transmittance to its original current value after a predetermined time that is readable by the user has elapsed after the display information is projected onto the lens. The object recognition means detects an image area of a predetermined object from the video;
The smart glasses according to any one of claims 1 to 3, characterized in that the control means controls the light transmittance of the light control film of the lens when the size of the image area becomes equal to or larger than a predetermined range. The object recognition means detects an image area of a predetermined object from the video;
The smart glasses according to claim 4 , characterized in that the control means controls the light-controlling film of the lens to a light transmittance higher or lower than the current light transmittance when the size of the image area exceeds a predetermined range. The object recognition means recognizes a human face area as the image area;
The smart glasses described in claim 5 , characterized in that the control means controls the lens's light-adjusting film to a light transmittance higher than the current light transmittance when the size of the face area becomes larger than a predetermined range. The smart glasses according to claim 6 , characterized in that the control means controls the light-control film of the lens so as to return to the original current light transmittance when the size of the face area is no longer within a predetermined range. Further, the device has a wireless communication means for communicating with an external device,
The smart glasses according to any one of claims 1 to 7 , wherein the control means controls the light transmittance of the light control film of the lens when a predetermined signal is received from an external device. The wireless communication means receives radio waves from a wireless tag as an external device, or communicates with a wireless base station or an access point;
The smart glasses according to claim 8, characterized in that the control means controls the light control film of the lens to a light transmittance higher or lower than the current light transmittance when it receives a predetermined signal including a predetermined address or ID. Further comprising a positioning means for detecting a current position,
The smart glasses according to any one of claims 1 to 9 , wherein the control means controls the light transmittance of the light control film of the lens when the current position stays within a predetermined position range. The smart glasses according to any one of claims 1 to 10 , wherein the control means controls the light transmittance of the light control film of the lens when the current time falls within a predetermined time period.